IRC Log

Tuesday October 26, 2010

[Time] NameMessage
[08:13] mikko good morning
[08:13] mikko sustrik: how was van puffelen?
[09:09] ptrb sustrik: thanks!
[10:14] sustrik mikko: nice, thanks for the tip
[10:36] Steve-o Looking for high performance Windows Servers for testing, can anyone squeeze more than 30,000pps out of Microsoft's OS? :P
[10:37] mikko Steve-o: is that like looking for an unicorn?
[10:37] guido_g hehehe
[10:38] Steve-o I get the impression sometimes, we were lucky to get 12,000pps from 2003/XP before.
[11:59] BooTheHamster Hi. I have got client and server applications. The client use REQ socket, server - REP socket. I start client without server, send message, then close socket via zmq_close(). Then I start server and recieve message, but I don't need that message. How I can clear send messages list?
[12:00] BooTheHamster I use 0MQ version 2.1 (from git)
[12:03] keffo Wouldn't it be both easier and less work to just not send it in the first place?
[12:04] BooTheHamster I test situation when server is lost
[12:07] keffo I dont get it.. You're free to discard data after you've received it..
[12:07] keffo recv(..), and then just toss it
[12:10] keffo with req/rep, a client and server are locked in an implicit send/recv cycle, initiated by the client..
[12:10] keffo open, send, close on a req socket is not correct
[12:11] BooTheHamster What type of socket it is correct?
[12:11] sustrik PUSH/PULL?
[12:11] BooTheHamster no. I need answer from server
[12:13] BooTheHamster My logic: I send request, get timeout, recreate socket and send another request and so on
[12:15] mato sustrik: yo
[12:15] sustrik mato: hi
[12:16] mato sustrik: re OpenPGM, can I commit & push Steve's patch plus the updated 5.x tarball to master?
[12:16] mikko pieter_hintjens: there?
[12:16] mikko pieter_hintjens: did you push the ICC build fix as well?
[12:16] sustrik BooTheHamster: set SO_LINGER to 0
[12:16] sustrik mato: sure
[12:17] mato sustrik: ok, coming up shortly then...
[12:17] sustrik mikko: he's not online right now
[12:17] sustrik mato: thanks
[12:17] mikko ok
[12:19] ntelford hi guys
[12:19] sustrik hi
[12:20] ntelford just looking through the 0mq guide and I notice that it recommends null terminating received strings by allocating a new buffer and copying the received bytes in to it - isn't this a somewhat expensive operation?
[12:20] sustrik depends on the length of the string :)
[12:20] mato ntelford: that's just the guide being newbie-friendly... there is no requirement like that in 0MQ itself
[12:21] mato ntelford: if in doubt, the reference at api.zeromq.org is your friend (although it's only up to date for 2.0.x right now)
[12:21] ntelford ok, fair enough
[12:21] ntelford I take it that most clients employ a more efficient means of properly terminating the strings?
[12:22] mato it's entirely up to you if you want to use length-prefixed or asciiz or something else entirely
[12:22] mato 0mq is just the transport
[12:22] ntelford ok, I see - encoding is up to the application, 0mq just deals with bytes; should've remembered that
[12:22] ntelford I think the tutorial is a shade confusing in that respect, as it implies that you need to worry about it
[12:23] mato ntelford: raise an issue with pieter hintjens, the tutorial is his baby
[12:23] ntelford ok :)
[12:24] BooTheHamster zmq_setsockopt(m_pSocket, SO_LINGER, NULL, 0) is correct?
[12:25] sustrik it's ZMQ_LINGER sorry
[12:25] sustrik and you have to pass the value in
[12:25] BooTheHamster value may be a BYTE?
[12:26] sustrik man zmq_setsockopt
[12:27] sustrik it's int
[12:27] BooTheHamster I found in man zmq_setsockopt :)
[12:29] matman Hey all, I just found zeromq a few days ago. I am happy with how fast it is and inproc is an awesome feature!
[12:32] BooTheHamster sustrik: thanx, ZMQ_LINGER helps
[12:33] sustrik :)
[12:33] sustrik matman: thanks
[12:34] matman has anyone looked at yarp in comparison?
[12:37] BooTheHamster sustrik: I forget to tell - when I build 0MQ library in VS2005 i get error "C3861: '__rdtsc': identifier not found". Need to add in clock.cpp something like:
[12:37] BooTheHamster #if defined ZMQ_HAVE_WINDOWS
[12:37] BooTheHamster #include <intrin.h>
[12:37] BooTheHamster #else
[12:37] BooTheHamster #include <sys/time.h>
[12:37] BooTheHamster #endif
[12:37] BooTheHamster maybe :)
[12:37] mato wasn't there a patch for that pushed some time ago?
[12:38] sustrik it wasn't signed off
[12:38] mato ah
[12:38] sustrik the guy haven't replied when i asked him to sign it off
[12:38] keffo It's simple enough to do in asm
[12:38] mato it was also not clear if intrin.h is a MSVC-only thing
[12:38] keffo with an ifdef for msvc
[12:39] sustrik even asm is msvc-specific
[12:39] sustrik msvc uses intel notation, while say gcc uses at&t notation
[12:39] keffo exactly, hence the guard :)
[12:39] sustrik BooTheHamster: would you submit a patch for the problem?
[12:40] BooTheHamster that guy be me :) I don't undestand correctry now to sing-off patch
[12:40] sustrik ha :)
[12:40] sustrik when you commit the patch, just use -s option
[12:40] mato sustrik: we know about the problem, you can just patch it yourself in your own name
[12:40] mato sustrik: in any case it needs the various MSVC-specific bits
[12:40] BooTheHamster mato: Yes
[12:41] sustrik i would prefer someone who actually has access to msvc2005 to fix it
[12:41] sustrik it works ok with msvc2008
[12:41] BooTheHamster sustrik: I make git commit -s but my ubuntu don't add Signed-Off in patch :(
[12:42] sustrik mato: any idea why that can happen?
[12:42] BooTheHamster I have MSVC2005 on work ...
[12:42] CIA-17 zeromq2: 03Steven McCoy 07master * re1debf1 10/ (3 files in 2 dirs):
[12:42] CIA-17 zeromq2: Update to OpenPGM 5.0.91.
[12:42] CIA-17 zeromq2: Signed-off-by: Steven McCoy <steven.mccoy@miru.hk> - http://bit.ly/9FEVnD
[12:42] mato sustrik: no, sorry
[12:43] sustrik Boo: never mind, just write "signed-off by <your name>" by hand
[12:43] mato sustrik: and is Boo's patch OK? i.e. it uses _MSC_VER instead of ZMQ_HAVE_WINDOWS?
[12:43] sustrik we've discussed that already :)
[12:43] sustrik the thing is that __rdtsc is msvc specific thing
[12:44] sustrik not win32 specific thing
[12:44] mato right
[12:44] mato __anything is generally compiler-dependent
[12:44] sustrik right
[12:45] BooTheHamster I send that patch file:
[12:45] BooTheHamster >From 89eeb3af58b4095e47c68c84a4b569fd6a6858dc Mon Sep 17 00:00:00 2001
[12:45] BooTheHamster From: Oleg Sevostyanov <oleg.sev@gmail.com>
[12:45] BooTheHamster Date: Wed, 13 Oct 2010 09:18:18 +0400
[12:45] BooTheHamster Subject: [PATCH] add header file define __rdtsc () function in MSVC build
[12:45] mato :-)
[12:46] sustrik Boo: yes, i know, what's needed is only your sign-off
[12:46] mato sustrik: he got kicked off for pasting too much to the channel
[12:46] sustrik it's used to track the proveninence of the code
[12:46] sustrik mato: who?
[12:46] sustrik ah
[12:46] sustrik i see
[12:46] mato sustrik: 14:45 -!- BooTheHamster [~BooTheHam@mail.elsystems.ru] has quit [Excess Flood]
[12:47] mato sustrik: that happens if you try to flood the channel (#zeromq) with too much data
[12:47] mato sustrik: e.g. pasting something
[12:47] sustrik i see
[12:47] sustrik let him reconnect
[12:47] mato yup
[13:13] sustrik BooTheHamster: hi
[13:14] sustrik it looks like pasting too much text kicks you out of the chatroom
[13:14] BooTheHamster hi, again. I undestand :)
[13:14] sustrik anyway, just resend your patch to the mailing list
[13:14] sustrik and add
[13:15] sustrik "Signed-off-by: your name <your email address>"
[13:15] sustrik and that's it
[13:15] sustrik i'll apply it
[13:16] BooTheHamster in patch's header?
[13:16] sustrik whereever
[13:16] sustrik the only point is to have the reference somewhere
[13:16] sustrik in a persistent fashion
[13:17] sustrik i.e. email rather than irc
[13:17] BooTheHamster oh ... bureaucrats ... :)
[13:17] sustrik licensing paranoia :)
[13:20] sustrik thanks!
[13:20] mato BooTheHamster: not wherever, the patch has a canonical format; see the contributions page
[13:21] sustrik mato: it's on the mailing list
[13:21] sustrik looks good
[13:21] sustrik and the format doesn't really matter as i apply it by hand anyhow
[13:21] mato sustrik: well, you shouldn't, but that's another story
[13:21] sustrik right
[13:21] mato sustrik: anyway, the patch it still bogus
[13:21] mato sustrik: i'll reply to the email...
[13:22] sustrik you mean the _MSC_VER thing?
[13:22] mato yes
[13:22] sustrik i'll patch that
[13:22] mato no
[13:22] sustrik ?
[13:22] mato sustrik: we discussed that; either you apply the patch as-is or it goes back to the contributor for review
[13:22] sustrik i appy it as is
[13:23] sustrik and fix it in another commit
[13:23] mato why?
[13:23] sustrik why not?
[13:23] mato that way you have a broken commit
[13:23] sustrik it's 10 secs
[13:23] mato i dunno
[13:23] mato bureaucracy
[13:23] mato :-)
[13:23] BooTheHamster boys, just make your own correct patch :)
[13:23] mato well, that was what i suggested to sustrik in the 1st place :-)
[13:23] mato sustrik: anyway, up to you ...
[13:24] sustrik sounds unfair to the original author of the patch
[13:24] mato sustrik: it's a trivial patch, honestly
[13:24] sustrik true
[13:24] sustrik if oleg is ok with it, i'll fix the problem myself
[13:24] mato the point is, you should get into the habit of not applying patches that are wrong
[13:24] mato *at all*
[13:25] BooTheHamster sustrik: ok, fix it.
[13:25] sustrik thx
[13:25] mato for trivial patches you can always just say, "thanks for the info, i did it myself differently"
[13:25] BooTheHamster mato: yes, I think it's good
[13:25] mato for actual *work* it should be bounced back to the original contributor for fixing...
[13:25] sustrik ok, i'll at least mention Oleg in credits section
[13:25] mato BooTheHamster: glad you agree....
[13:25] mato sustrik: sure
[13:27] BooTheHamster boys, you will be perfect bureaucrats ... "level 1" (C) Futurama :)
[13:27] mato :-)
[13:27] mato BooTheHamster: hey, i got yelled at for weeks for not providing a contribution process, so now we'll all stick to it unless it doesn't work for some reason :-)
[13:37] sustrik BooTheHamster: Ok, patch committed, thanks!
[13:38] BooTheHamster sustrik: thanks to you. I don't need to merge in next update :)
[13:38] CIA-17 zeromq2: 03Martin Sustrik 07master * r401986e 10/ (AUTHORS src/clock.cpp):
[13:38] CIA-17 zeromq2: Build with MSVC2005 fixed
[13:38] CIA-17 zeromq2: To use __rdtsc() function MSVC2005 requires including intrin.h
[13:38] CIA-17 zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/degonQ
[13:41] BooTheHamster I add into clock.cpp:
[13:41] BooTheHamster #include <stddef.h>
[13:41] BooTheHamster #if defined _MSC_VER
[13:41] BooTheHamster #include <intrin.h>
[13:41] BooTheHamster #endif
[13:41] BooTheHamster #if !defined ZMQ_HAVE_WINDOWS
[13:41] BooTheHamster #include <sys/time.h>
[13:41] BooTheHamster and rebuild. All ok
[13:42] BooTheHamster Only warnings
[13:42] BooTheHamster warning C4355: 'this' : used in base member initializer list
[13:42] sustrik yes, i am aware of those
[14:03] CIA-17 zeromq2: 03Dhammika Pathirana 07master * r465da7b 10/ (src/session.cpp src/zmq_init.hpp):
[14:03] CIA-17 zeromq2: drop connection requests with duplicate peer identity
[14:03] CIA-17 zeromq2: Signed-off-by: Dhammika Pathirana <dhammika@gmail.com> - http://bit.ly/cRHlbt
[14:45] DerGuteMoritz is it intentional that ipc sockets don't get cleaned up after the sockets are closed and the context they belong to are terminated? or is my clean-up code fubar?
[14:45] DerGuteMoritz I mean the named pipes, they stick around
[14:47] sustrik you mean AF_UNIX sockets?
[14:48] DerGuteMoritz I mean the named pipes
[14:48] DerGuteMoritz created by ipc://
[14:53] Nate75Sanders I'm using python (pyzmq 2.08dev with libzmq 2.09), I've set up a PUSHer and a PULLer, I'm using zmq.IDENTITY (pusher and puller) and zmq.HWM (pusher), and when I leave the pusher running and occasionally break and restart the puller, the puller loses exactly 1 message. Is this expected behavior? Is my combination of library versions bad?
[15:01] sustrik DerGuteMoritz: What OS are you on?
[15:02] sustrik Nate75Sanders: it's expected behaviour
[15:02] sustrik when application crashes, all the messages on the fly at the moment are lost
[15:02] Nate75Sanders sustrik: That's what I was afraid of
[15:03] DerGuteMoritz sustrik: Linux
[15:03] Nate75Sanders sustrik: So I need to add in my own durability to handle crashes?
[15:04] sustrik well, yes, but we warned: it's impossible to do
[15:04] sustrik :)
[15:04] sustrik DerGuteMoritz: It's AF_UNIX sockets then
[15:05] sustrik they are handled by the same code as TCP is
[15:05] sustrik so if you are seeing something with IPC that doesn't happen with TCP, it's kind of strange
[15:07] DerGuteMoritz hmm I'm not familiar with AF_UNIX, does it use filenames as addresses?
[15:07] sustrik yes
[15:07] DerGuteMoritz alright
[15:07] sustrik probably the same thing as mkfifo under the cover
[15:07] DerGuteMoritz that's what I figured
[15:08] DerGuteMoritz but the files stick around after terminating the zmq processes
[15:08] DerGuteMoritz so I was wondering whether this is intended
[15:08] DerGuteMoritz but I guess it is
[15:08] sustrik dunno, it wasn't me who wrote the code
[15:08] DerGuteMoritz ah alright
[15:09] sustrik ask on the mailing list
[15:09] DerGuteMoritz thanks for your input, anyway!
[15:34] matman has anyone implemented a system to auto select between tcp/ipc/inproc? maybe a lookup service or distributor system...
[15:35] sustrik there were some zeroconf discussions on the mailing list
[15:35] sustrik try to check that
[15:35] matman okay thanks
[16:21] ntelford is there an API call to inspect the number of pending messages in a sockets queue?
[16:24] sustrik no
[16:24] ntelford hmm
[16:24] ntelford wouldn't that be useful for monitoring an application's throughput?
[16:25] sustrik just count the messages and divide the figure by time
[16:25] sustrik that gives you the throughput
[16:26] ntelford ok, poor example
[16:27] ntelford I'm working on automated controllers that tweak various setting (thread pool size, batching factor etc.) based on various metrics; would be useful to know the size of the socket queues to monitor bottlenecks
[16:29] cremes ntelford: there is no way to tell; messages may be in 0mq's buffer, a kernel buffer, on the wire, on the receive side's kernel buffer, etc
[16:29] cremes you can do your runtime tweaking by using another set of "out of band" sockets and passing information amongst your distributed processes
[16:30] cremes plus, that is *way* more flexible
[16:30] ntelford I don't quite follow the "out of band" side of things
[16:30] ntelford just seemed like a nice idea to use 0mq's message queue instead of having to queue it all again within the application
[16:31] sustrik afaics your problem is you want to know when message queue grows ad infinitum so you can identify there's a bottlenect somewhere down the stream
[16:31] ntelford correct
[16:32] sustrik what i would propose rather is setting the queue limit to N messages
[16:32] ntelford importantly, so code can act on that bottleneck (dynamically allocating more resources etc.)
[16:32] sustrik then when the limit is reached next attempt to send will block/fail
[16:32] sustrik and you can handle the message in a different way
[16:33] ntelford hmm
[16:33] ntelford I may have bitten off more than I can chew here
[16:33] ntelford for context, I'm investigating whether I can have 0mq form the network I/O and stage queuing component in a SEDA architecture
[16:34] ntelford and one of the most important aspects of that is having the ability to dynamically control stage resource allocation based on performance metrics, like the size of the stage queue
[16:34] ntelford it may well be that 0mq just isn't a good fit for this
[16:35] sustrik what metrics to do exactly need?
[16:35] ntelford depends on the controller, but at a minimum the output throughput (easily calculable) and the current size (periodically polled) of the stage's input queue
[16:36] mikko i still reckon that 'messages-in-0mq send queue' would be a useful metric :)
[16:36] mikko just the messages pending in the IO threads
[16:36] ntelford mikko: I think that would also be a useful metric
[16:36] sustrik i still believe its impossible to get any sensible estimate
[16:36] ntelford both pending send and pending recv queue lengths would be nice
[16:37] sustrik given all the levels the data can be queued on
[16:37] ntelford sustrik: I'm under the impression that once the message has been buffered and assembled in to a single coherant object it's placed in a message queue for that socket that the zmq_recv() call retrieves from?
[16:37] sustrik anyway, afaiu ntelford is basically trying to implement custom load-balancing
[16:38] ntelford yeah, that about sums it up
[16:38] sustrik which is hard on top of 0mq as it has its own load-balancing
[16:38] ntelford only for the I/O surely?
[16:38] sustrik when you have multiple peers to say REQ socket
[16:38] ntelford 0mq has no control over how the application processes messages, afaik
[16:38] sustrik XREQ
[16:39] sustrik then individual requests are load balanced among the peers
[16:39] sustrik no, but it has control over message dispatching
[16:40] ntelford hmm
[16:41] ntelford worst case scenario, I can always pipe ZMQ messages in to an application-managed queue
[16:41] sustrik sure
[16:41] ntelford just seemed a little convoluted to move messages from one queue to another
[16:41] mikko pending recv queue length is almsot impossible
[16:41] ntelford mikko: how so?
[16:42] mikko ntelford: what is recv queue?
[16:42] mikko messages fully received by 0MQ and waiting in io thread?
[16:42] mikko messages in-flight between publisher and consumer?
[16:42] ntelford from the perspective of the application, messages that have been fully received and are awaiting consumption
[16:43] sustrik received by what layer?
[16:43] ntelford 0mq
[16:43] sustrik it's hard to figure out any sane semantics for the metric
[16:43] sustrik why not TCP?
[16:44] sustrik or the NIC?
[16:44] ntelford depends on your perspective
[16:44] sustrik i would say the only *real* metric would be "all the messages on the fly"
[16:44] ntelford ok, what's the high water mark set on?
[16:44] sustrik 0mq queue
[16:44] sustrik then you have SND/RCVBUF
[16:45] sustrik which are set on TCP level
[16:45] ntelford so there are multiple "limit" options, right?
[16:45] sustrik the NIC settings (if configurable) are set be ethtool and similar
[16:45] ntelford one of queue length, one on send/receive buffers etc.
[16:45] ntelford s/of/on/
[16:45] sustrik one per layer of the stack
[16:45] ntelford right
[16:46] ntelford so wouldn't it make sense to have multiple "current size/length" inspection methods too?
[16:46] ntelford and since the NIC limits are set through the OS etc. it'd make little sense to expose that in 0mq
[16:46] mikko can you get that info from TCP stack?
[16:46] sustrik nope
[16:46] sustrik you also have queues on middleboxes
[16:46] sustrik such as switches and routers
[16:47] ntelford right
[16:47] sustrik anyway, all the layers allow to set max limit
[16:47] mikko i feel that we've had this discussion before
[16:47] ntelford so it's not feasible to get the number of messages en-route
[16:47] sustrik but not to retrieve the actual size of the queue
[16:47] sustrik so you can set the size of TCP buffer usign SO_SNDBUF
[16:47] ntelford sustrik: but at least one layer must do, surely
[16:47] sustrik which one?
[16:47] mikko why can't we expose the queue size on layers that we can expose?
[16:47] mikko with a disclaimer
[16:48] ntelford the one that has the complete messages assembled waiting for the application to call zmq_recv() so it can be dispatched
[16:48] ntelford mikko: yes, exactly
[16:48] sustrik because it's a metric with no semantics
[16:48] sustrik = useless
[16:48] ntelford not really
[16:48] sustrik sure, you need it for load balancing
[16:49] mikko it's a metric with clear semantics. messages in 0MQ io-threads waiting to be sent
[16:49] ntelford mikko: or received
[16:49] sustrik so the load balancing should be done on the right layer
[16:49] sustrik say, load-balancing of IP packets is done on IP layer
[16:49] sustrik load balancing of 0MQ messages should be done on 0MQ layer
[16:49] sustrik etc.
[16:51] ntelford I guess I can see your point, but given that the 0mq layer has no control over the application threads, how can it load balance messages?
[16:51] sustrik imaging an PUSH socket
[16:51] sustrik it has 3 TCP connection underneath
[16:52] sustrik so, when you send a message it has to choose which connection it should send it to
[16:52] sustrik so there's a load-balancing algorithm to decide
[16:52] ntelford right, that seems reasonable to me
[16:53] ntelford what about for PULL sockets?
[16:53] sustrik there's "fair queueing" going on there
[16:53] sustrik i.e. allocating fair amount of resource to each connected peer
[16:53] ntelford ok, I get that
[16:54] ntelford but when a socket receives messages, it's up to the application to handle them (each call to zmq_recv())
[16:54] sustrik right
[16:54] ntelford if the socket receives messages faster than the application can deal with them, it'll eventually hit the high water mark
[16:54] sustrik right
[16:54] ntelford I'm suggesting it would be nice for the application to know things like that
[16:55] sustrik that's where load balancing mechanism kicks in
[16:55] mikko would that be more of a thing for the sys:// transport?
[16:55] mikko HWM reached
[16:55] sustrik if a HWM is reached for particular connection
[16:55] sustrik no more messages are dispatched to that connection
[16:55] ntelford ok, I see that
[16:56] sustrik mikko: kind of
[16:56] ntelford but what if the application can automatically tune things so that it can increase message throughput
[16:56] sustrik it's about monitoring
[16:56] sustrik such as?
[16:56] mikko btw, on a complete different topic: does the sys transport send structured information or strings?
[16:56] ntelford creating more threads to process the messages (in a thread pooled application)
[16:57] ntelford let me rephrase: increasing the size of the thread-pool to handle the additional load
[16:57] sustrik mikko: it doesn't send anything yet :)
[16:57] sustrik nobody using it at the moment
[16:57] mikko sustrik: i mean the plan
[16:57] sustrik ntelford: right
[16:57] sustrik why not use HWM for that?
[16:58] sustrik when send fails
[16:58] ntelford because that would reject/discard the new messages??
[16:58] sustrik you know the queues are full
[16:58] sustrik if you do non-blocking send
[16:58] sustrik it say "sorry, EAGAIN"
[16:58] sustrik so you can launch new peer
[16:58] sustrik or whatever
[16:59] ntelford how would the client inform the server it needs to increase the thread-pool size?
[16:59] sustrik you can change that in runtime
[16:59] sustrik can't
[16:59] ntelford change what?
[16:59] sustrik thread pool size
[16:59] ntelford depends on your application
[16:59] sustrik i meant 0mq thread pool size
[16:59] sustrik sorry
[17:00] ntelford I'm not talking 0mq I/O thread pool here
[17:00] sustrik i see
[17:00] ntelford talking about the application thread pool for processing the messages
[17:00] sustrik yes, i understand
[17:00] sustrik i don't know actually
[17:01] sustrik the assumption so far was that services offer constant amount of resources
[17:01] ntelford I see
[17:02] sustrik ntelford: is that a research project?
[17:02] ntelford nothing formal
[17:03] ntelford I'm looking in to it to form the basis of our data processing platform
[17:03] sustrik the main problem i see is the mislayering
[17:03] ntelford oh?
[17:03] sustrik you are trying to do load-balancing on top of layer that already does load-balancing
[17:04] sustrik so the solution IMO would be to move the functionality to 0MQ itself
[17:04] sustrik but that in turn requires a lot of work etc.
[17:04] ntelford I really don't see how it could
[17:04] sustrik neither do i
[17:04] ntelford I understand your argument about layering
[17:05] sustrik let me understand your problem
[17:05] sustrik are you not happy with how 0mq load balances messages
[17:06] ntelford to be clear, everything I'm talking about pertains only to sockets that receive messages
[17:06] ntelford the load balancing seems fine
[17:06] sustrik or is the load-balancing ok and what you need rather is dynamic response to the congestion on the side of your application?
[17:06] ntelford yes, that's it exactly
[17:06] sustrik i see
[17:07] sustrik what about doing it this way:
[17:07] sustrik call recv in non-blocking way
[17:07] ntelford the idea is to automatically maximize throughput by monitoring the input and output and modifying application runtime parameters for maximum throughput
[17:07] sustrik if it returns EAGAIN it means there's nothing to process
[17:07] sustrik so you are ok
[17:07] ntelford ok
[17:08] sustrik if it returns the message it's running at full speed
[17:08] sustrik and allocating more resource may be useful
[17:08] sustrik the algorithm may be tweaked in different ways
[17:09] sustrik such as "if we haven't seen EAGAIN last 1000x recvs"
[17:09] ntelford thing is, it really calls for policies like: "if there are > 100 messages, allocate more threads, if there are > 1000 messages, block new messages (HWM)"
[17:09] ntelford s/messages/pending messages/
[17:09] ptrb "it" being your app or ZMQ?
[17:09] sustrik how do you set the numbers?
[17:09] ntelford the app
[17:09] sustrik the limits i mean?
[17:10] ntelford sustrik: numbers are arbitrary, application dependent
[17:10] sustrik how does the application know what number to use?
[17:10] ntelford configured manually
[17:11] sustrik how does the admin know what number to use?
[17:11] sustrik :)
[17:11] sustrik it's just a heuristic, right?
[17:11] ntelford yeah
[17:12] sustrik when you look at standard networking stack, it has pretty strict policy about not exporting heuristics
[17:12] sustrik say, TCP has an estimate of round-trip time
[17:12] sustrik but there's no way to get it from layer above
[17:12] ntelford right
[17:13] ntelford I actually gtg, getting seriously hungry :)
[17:13] sustrik you can still tweak 0mq to get what you need
[17:13] ntelford for reference btw, the architecture I'm trying to hammer 0mq in to is SEDA: http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf
[17:13] sustrik it's open source
[17:13] sustrik ok, i'll have a look
[17:13] sustrik bon appettit
[17:13] ntelford my goal is to implement the stage event queues using 0MQ, allowing stages to run anywhere (inproc, ipc, remotely etc.)
[17:13] ntelford cheers :)
[17:14] ntelford later guys
[17:14] ntelford btw, feel free to drop me messages if you have any further thoughts, I'll see them tomorrow
[17:19] ngerakines morning
[17:21] Ghpu hello
[17:21] Ghpu I would like to know if there is some efforts ongoing to try and port zeromq to android platform
[17:22] sustrik Ghpu_: i have no experience with android
[17:22] Ghpu I was able to compile libuuid with the crystax version of android ndk, but haven't suceeded yet in compiling zeromq
[17:22] sustrik what was the problem?
[17:23] Ghpu 1/ android does not directly support the STL
[17:23] Ghpu which is fortunately corrected by using the ndk-crystax
[17:23] Ghpu 2/ 64-bits types are not correctly defined in the ndk
[17:24] sustrik you mean uint64 and such?
[17:24] Ghpu and last I got some problems with non-existent pthread-cancel
[17:24] Ghpu yes
[17:24] sustrik you can try to typedef them yourself
[17:24] sustrik that's what's done for win32 for example
[17:25] sustrik as for pthead_cancel 0mq doesn't use it
[17:25] Ghpu good then
[17:25] sustrik interesting it complains about it
[17:25] Ghpu I have only begun trying to port it today, and I just wanted to know whether someone has already tried it before
[17:25] sustrik people have already asked about it
[17:26] sustrik but i don't think anyone have done it yet
[17:26] Ghpu in fact the complaint came from the stl port, which cannot correctly guess the android platform specifics
[17:26] sustrik :(
[17:26] Ghpu ok, well, I'll keep you informed if/when I succeed
[17:26] Ghpu thanks!
[17:27] matman Question. When you use the zmq filter. Does it filter on the sender or the receiver side?
[17:27] sustrik ok, thanks!
[17:27] sustrik on the receiver side
[17:27] sustrik it should forward subscription to the sender side
[17:27] sustrik but that's not yet implemented
[17:28] sustrik it's definitely on the roadmap
[21:25] Lizito Hey folks. I'm evaluating 0MQ for a windows project. Question: Anyone have experience with 0MQ and the windows message loop used in GUIs and single-threaded apartment COM objects?
[22:02] jhawk28 hello
[22:03] jhawk28 would anyone mind if I submitted a patch to add a VS 2010 version of the ZMQ build?
[22:03] mikko jhawk28: i don't think anyone minds :)
[22:04] jhawk28 mikko: as in it can easily be ignored ;)
[22:04] mikko just sign the patch and send to mailing-list wiht title "[PATCH] short tag-line here"
[22:06] jhawk28 yeah, Im on the list - just getting some feedback if others actually wanted it
[22:07] mikko most of the devs are in europe so it might take a couple hours for anyone to respond
[22:07] mikko but im sure patches are always welcome
[22:08] mikko time to sleep
[22:08] mikko nn

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy