Sunday February 13, 2011

[Time] NameMessage
[00:08] rbitcon Howdy, I am kooking to be pointed in the right direction for help
[00:09] rbitcon I am using zeromq with Python in a REQ/RESP model with one server and many "workers". I see that the zmq library take up hude amounts of RAM
[00:11] rbitcon I read the docs about socket.setsockopt(zmq.HWM, 125), but it is not that clear
[00:11] rbitcon any pointers?
[00:52] rbitcon Solved my problem with memory usage.
[00:53] rbitcon I downloaded pyzmq and dis a plan python.exe install, previously i did a development build and install or 2.0.10. This is all under windows7 and using python2.6
[01:10] tjgillies if im behind NAT can i use zeromq to do udp hole punching?
[03:35] Seta00 anyone from the pyzmq team here?
[03:35] Seta00 in fact, anyone with python 2.7 pyzmq builds for windows here?
[03:35] Seta00 it just won't install
[08:50] sustrik zedas: just set ZMQ_LINGER option to zero for all the sockets and everything should work as with 2.0.x
[08:52] sustrik Guthut: wow!
[09:06] CIA-21 zeromq2: 03Martin Sustrik 07master * rba26cc9 10/ src/zmq.cpp :
[09:06] CIA-21 zeromq2: Build problem with OpenPGM and MSVC fixed
[09:06] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[11:30] Guthur Sustrik: ping
[11:33] DeadZen whats zeromq?
[11:38] mikko DeadZen:
[11:44] Guthur in like the '0MQ in 100 words' in the guide as a description
[11:44] Guthur
[11:49] Guthur mikko, would be possible to have tests run as part of a bindings build, on the build server?
[11:51] mikko sure
[11:53] Guthur cool, I don't have the tests at the moment, but it is something I would like to add in the future for clrzmq2
[12:04] mikko it should be a fairly painless process to add them to builds
[12:20] mikko i to the b
[12:45] Guthur I've been looking into IOCP for named pipes connection under windows, the main issue I have found is differentiating between the read and write operations. It is possible to pass in a handle to an event object to writefile function (as part of the overlapped struct), which will allow differentiation.
[12:46] Guthur The only issue with that is that both the poller and and the writer need the handle to that event object.
[12:47] Guthur or at least for the poller to know it is a write event object, it will be matched to the right pipe as part of the return from IOCP
[15:25] sustrik Guthur: pong
[15:30] Guthur sustrik, did that stuff i posted earlier make much sense?
[15:30] sustrik i have no idea how IOCP works :|
[15:30] sustrik anyway, what you need for integration with 0mq
[15:31] sustrik is a polling mechanism that triggers an event when particular "socket" is readable and/or writeable
[15:31] sustrik it has to be able to handle multiple "sockets" in parallel
[15:31] Guthur oh yeah IOCP does multiple sockets very well
[15:31] sustrik and all the event handlers have to be invoked within a single thread
[15:32] Guthur that's no problem as well
[15:32] sustrik then it should be easy, no?
[15:32] Guthur the only complication as far as I can see is differentiating between what event is fired
[15:33] sustrik can't you use 2 different events?
[15:33] sustrik event objects i mean
[15:34] Guthur if we can pass a different event object when writing to the pipe then yes
[15:35] Guthur 2 secs, just going to ask someone about it now
[15:36] Guthur when one creates the name pipe one passes in an overlapped struct with an event object, this is set active by all events as far as I can see
[15:39] sustrik aha
[15:40] sustrik so how you are supposed to know whether the socket is readable or writeable?
[15:40] Guthur you mean like a connected event?
[15:41] sustrik hm
[15:41] sustrik CreateNamedPipe doesn't seem to accept OVERLAPPED struct
[15:42] Guthur writefile does
[15:43] Guthur i'm talking to someone more knowledgeable now
[15:43] Guthur 2 secs
[15:48] mikko Guthur: steve-o might be able to help you with IOCP
[15:48] mikko he did some tests with it some time ago
[15:48] Guthur mikko, did he rule it out for any technical reason?
[15:49] mikko Guthur: i think it wasnt as fast as he expected
[15:50] sustrik the problem with 0mq is a bit different
[15:50] Guthur oh, interesting
[15:50] sustrik namely, select() function that's currently used for polling
[15:50] sustrik is limited to POSIX-like TCP sockets (winsock)
[15:50] sustrik it doesn't work with anything else afaik
[15:51] sustrik ie. named pipes etc.
[15:51] mikko and kelly brock has done some work
[15:51] mikko "For those interested, I highly
[15:51] mikko suggest reading the change lists from V1 to V2rc of libevent, most changes
[15:51] mikko were required because IOCP is nearly the opposite of all other async
[15:51] mikko solutions."
[15:52] sustrik the first question imo is whether IOCP is the only option to poll on TCP sockets and named pipes
[15:52] sustrik maybe there's a different (and maybe less complex) way
[15:53] mikko or whether named pipes could be emulated on windows using shared memory
[15:53] mikko not sure if that makes sense as you would busy wait
[15:53] sustrik you still to have poll in I/O thread
[15:53] sustrik if you emulate the named pipes
[15:53] sustrik you still have to event the I/O thread
[15:53] sustrik which can be done via TCP socket (slow)
[15:54] sustrik other option would be something like Event object
[15:54] sustrik but once again, you can't poll on Event object using select()
[15:54] mikko thinking out loud: optional dependency to libevent?
[15:55] mikko new poller type which uses libevent or libev
[15:55] mikko disabled by default but can be enabled during build
[15:55] sustrik that would be more work than supporting IOCP natively IMO
[15:56] mikko i really don't know whats involved in adding support for IOCP
[15:56] Guthur does steve-o come on irc
[15:56] mikko yes
[15:57] mikko he is here usually during the week
[15:57] mikko he's on NYC timezone
[15:57] mikko EST?
[15:57] sustrik still in NY?
[15:57] mikko i thought so
[15:57] sustrik he's from hong kong
[15:57] mikko yeah, i thought he was going to be in NYC for a longer period
[15:57] mikko maybe i misunderstood
[15:57] sustrik ah, no idea
[15:58] mikko he was in NYC and referred to "buying a house" during that time. i thought he might be there for a while
[15:58] mikko anyway, he does come here occasionally during the week
[16:00] Guthur there is a way to getting this IOCP working I'm sure
[16:01] Guthur I'm missing something regarding the identification of the various ops, I can know that an something is happening on a pipe, just not what that something is
[16:01] Guthur there has to be a way
[16:02] sustrik Guthur: my impression so far is that event object is associated with particular I/O operation (Read/Write)
[16:03] sustrik rather than Pipe as such
[16:03] Guthur yeah, that's correct enough
[16:03] Guthur sorry if I drew you to a different conclusion before
[16:04] sustrik ok, maybe i misunderstood what the problem is then
[16:05] Guthur `Orum on #winapi is posting some IOCP code, it might shed some light
[16:21] zchrish I am performing a test to perform repeated zmq_init and zmq_term w/o performing any bind; is that permissible? I am noticing a memory leak after a continued run of this test.
[16:22] sustrik zchrish: yes, it's permissible
[16:22] sustrik what's the leak?
[16:23] zchrish I am performing a "cout" of the context pointer and it is giving me a different value over time. I am watching "top" and the amount of memory required by my process is slowing growing. I am performing a zmq_init/zmq_term avery 1-2 seconds.
[16:24] mikko zchrish: have you checked in valgrind?
[16:24] zchrish Let me check; thanks.
[16:35] zchrish Not sure whether the output of valgrind is related to the way I am using my program. I am creating threads which zmq_init and then zmq_term using a pthread_cleanup_push concept. After 20 iterations, I close the master thread which closes the main program. But I don't know whether this closing affects the slave threads which perform the zmq_init/zmq_term.
[16:36] zchrish ==22197== LEAK SUMMARY:
[16:36] zchrish ==22197== definitely lost: 0 bytes in 0 blocks
[16:36] zchrish ==22197== indirectly lost: 0 bytes in 0 blocks
[16:36] zchrish ==22197== possibly lost: 12,584 bytes in 43 blocks
[16:36] zchrish ==22197== still reachable: 4,552 bytes in 5 blocks
[16:36] zchrish ==22197== suppressed: 0 bytes in 0 blocks
[16:37] zchrish I have 3 threads: (A) master and (B),(C) slave threads. In (B), (C), I peform zmq_init and zmq_term based on a simulated incident which indicates a problem.
[16:38] sustrik run the valgrind with --leak-check=full
[16:38] sustrik to find out what's leaking
[16:39] zchrish Anyway, it seems like the amount lost isn't that much, if it is from 0mq, so I won't worry about it at this point. I don't expect to perform a zmq_term and zmq_init very often.
[16:43] zchrish Hmm. I did this but I am not proficient enough with valgrind to properly interpret the results. The program is simple enough but I am not sure whether someone is interested in re-creating the situation. If so, I will upload somewhere.
[16:47] sustrik yes. do so. i'll have a look
[16:53] zchrish I remember watching discussions about uploading to a website but I don't remember; would you please let me know. Thank you.
[16:54] zchrish I will upload it.
[16:55] sustrik
[16:56] sustrik place it there
[16:56] sustrik post the link here
[16:57] Guthur sustrik, re: IOCP, what we can do is pass a custom overlapped struct to Connect, Read, Write etc, and this struct will be returned by IOCP
[16:58] Guthur so the custom struct can have a member containing and enum of the op type, for example
[16:58] sustrik how would you physically perform the polling?
[16:59] sustrik WaitForMultiple objects or what?
[16:59] Guthur GetQueuedCompletionStatus
[16:59] Guthur GetQueuedCompletionStatusEx
[16:59] zchrish Not sure if this is the output result; this is my first gist upload.... gist: 824833
[16:59] sustrik can you prpvide the link?
[17:00] zchrish
[17:00] zchrish Is that right?
[17:00] zchrish I think so.
[17:01] sustrik ah
[17:01] sustrik i meant, can you provide the valgrind output?
[17:01] zchrish Oh, sorry. Let me do another.
[17:03] sustrik Guthur: I see
[17:03] sustrik so, a completion port
[17:03] sustrik is actually a queue of pointers to OVERLAPPED structures
[17:05] zchrish
[17:05] zchrish is the valgrind output
[17:07] sustrik thanks
[17:08] Guthur sustrik, essentially yes
[17:08] Guthur there is also a completion key
[17:08] Guthur it's quite common to place the handle of the socket in that
[17:08] Guthur or some other context struct
[17:09] sustrik ack
[17:09] sustrik that seems reasonable
[17:09] Guthur sustrik, does it sound like it will fullfil 0MQs needs?
[17:09] sustrik definitely
[17:09] Guthur sweet
[17:10] sustrik you'll simply have a I/O thread loop with GetIOCompletionStatus
[17:10] sustrik should be relativelty easy
[17:14] Guthur so, do you want me put some code together for an implementation of epoll et al. using IOCP
[17:15] sustrik definitely
[17:15] sustrik try to hack something
[17:15] sustrik feel free to ask about 0mq details in the process
[17:16] Guthur will do, cheers to all for the help so far
[17:16] Guthur I need to do something else tonight, for MSc thesis, but I'll continue with the IOCP this week
[17:18] sustrik great
[17:18] Guthur just as an aside: according to wiki IOCP is also available on Solaris 10+
[17:19] Guthur is ZMQ on solaris?
[17:19] sustrik sure
[17:20] sustrik but i think solaris I/O completion ports are a different beast
[17:20] Guthur yeah probably
[17:20] Guthur not like I have a solaris box hanging around anyway, hehe
[17:21] sustrik i recall we've investigated polling mechanisms in solris quite a long time ago
[17:21] sustrik and devided to go with /dev/poll rather than completion ports
[17:23] Guthur if I can help bring IOCP on windows to 0MQ i'll be happy enough
[18:16] sustrik zchrish: it looks like you are not terminating your threads cleanly
[18:44] ianbarber m to the k (delayed response goodness)
[18:51] zchrish I see; any pointer is appreciated. Thanks.
[19:28] sustrik zchrish: pthread_cancel basically means you kill the thread
[19:28] sustrik what you should do instead is to pthread_join it
[21:54] eut cremes, i think that the delay or whatever in nonblocking packet delivery is manageable. if things on my end are designed properly then it shouldn't be a problem.
[21:55] bitweiler hey using a publisher, socket should the publisher receive some request?
[21:55] eut bitweiler, no
[21:55] bitweiler eut: why is that?
[21:56] eut see summary of ZMQ_PUB characteristics. it is unidirectional and send only
[21:56] bitweiler okay thanks
[22:01] bitweiler what is meant by: The zmq_send() function is not implemented for this socket type. where socket type is ZMQ_SUB
[22:04] eut did you bind or connect to this socket?
[22:04] bitweiler yes, I bind in the server side and connect on the client side
[22:05] eut whoops
[22:05] bitweiler but I'm getting nothing on the client side in response
[22:05] mikko bitweiler: are you subscribing
[22:05] eut see that page i linked to earlier for description of ZMQ_SUB
[22:05] mikko thats the most common thing causing messages not to be received
[22:06] eut ZMQ_SUB is unidirectional receive only... you cannot send...
[22:07] bitweiler mikko: yes I'm subscribing
[22:08] eut i have a question about the xrep/xreq sockets. on the xreq side i have set the socket identity to some 4 byte long string ("111\0"). when i send a message, on the xrep side, i'm getting a 17 byte long response in the first received part.
[22:08] mikko eut: you get three messages
[22:09] mikko one with identity, empty message and the actual message
[22:09] mikko you should send in xreq side in that fashion as well
[22:10] mikko identity (ZMQ_SNDMORE set), empty message (ZMQ_SNDMORE set), message
[22:10] eut i thought that zmq takes care of prepending the socket identity and empty message when sending from a xreq socket
[22:11] mikko let me check the code, it's been a while since i used xreq/xreq
[22:14] eut i hope it doesnt matter... but the xrep side is in lua and the xreq side is c
[22:18] mikko yes, client side prepends
[22:18] mikko on xrep side you should receive multiple messages
[22:18] mikko check for ZMQ_RCVMORE
[22:21] eut yes
[22:21] mikko does the lua binding expose that?
[22:21] eut it exposes getsockopt
[22:22] eut i'm able to see that more messages parts are there
[22:22] mikko how are you replying with xreq?
[22:22] eut i'm not replying at all yet, i'm just trying to interpret the received messages correctly.
[22:23] mikko can i see the xrep code?
[22:23] mikko trying to focus on too many things at once
[22:24] eut sure
[22:26] eut
[22:27] mikko what about xreq?
[22:27] eut whoops, this is more correct xrep code:
[22:27] eut a moment for xreq
[22:35] eut xreq code:
[22:37] Guthur are you not suppose to set the identity before connecting
[22:38] eut this is the output i'm getting and what i expect should happen:
[22:38] eut the first part of the multipart message is 17 bytes for some reason...
[22:39] eut the identity was set to "222" so i dont understand what is going on
[22:40] Guthur from guide: If you want to set an identity you must do it before connecting or binding the socket.
[22:40] Guthur you connect then set the identity in that code
[22:40] Guthur the one for the xreq
[22:41] eut :D
[22:41] eut it works. thanks very much
[22:41] Guthur you're welcome
[22:41] Guthur it actually should mention that in the manual
[22:42] Guthur i'm surprised it doesn't actually
[22:42] eut it would make a nice addition to the zmq_socket api page
[22:42] eut err, setsockopt
[22:42] Guthur yeah I agree
[22:43] eut do all options need to be set prior to connecting? or are some more flexible?
[22:43] Guthur no identity is special
[22:44] Guthur it makes sense that it needs to be before connecting, in a way
[22:44] Guthur though it might be nice to be able to change an identity
[22:45] Guthur after connecting, but things could get confusing then, and packet delivery might go terribly wrong
[22:45] eut right
[22:45] eut i think its fine the way it is, but it would help to have the api docs updated
[22:46] Guthur yeah, I'll submit a patch
[22:46] Guthur there was another thing I wanted to submit for the docs anyway
[22:46] eut ok sure, thanks for the help
[22:47] Guthur sure, anytime
[22:48] eut is the development branch
[22:49] mikko eut: yes
[22:49] mikko sorry, had to jet for a minute
[22:50] mikko 17 part response is UUID most likely, should've catched that
[22:50] mikko 17 char part*
[22:50] eut ah, and of course its binary data so it wont display as normal text
[22:54] Guthur mikko, do you think the api documentation for setsockopt identity should mention that it has to be set before bind/connect?
[22:54] Guthur and if so should it be a caution note?
[22:54] mikko Guthur: i think majority of them should mention it
[22:54] mikko like HWM affects only subsequent binds/connects
[22:55] sustrik all of them, actually
[22:55] Guthur oh, I thought some didn't matter
[22:55] eut is the delimiter message necessary if only going directly from xrep <-> xreq? it seems like it is only useful for chains longer than two nodes
[22:55] Guthur good to know
[22:55] sustrik it should be mentioned in the docs, you are right
[22:56] mikko i ended up doing a patch for jenkins after running an issue with build slaves
[22:56] mikko hopefully they'll accept it
[22:56] sustrik congrats!
[22:56] sustrik what was wrong there
[22:56] sustrik ?
[22:56] mikko it adds ssl between slave and master
[22:56] Guthur is there any opts it doesn't matter for?
[22:56] mikko rather than having to use stunnel
[22:57] mikko good for WAN connected slaves
[22:57] sustrik ack
[22:57] sustrik Guthur: the system applies to all the options
[22:57] mikko what are the big todo items for 2.1.x stable ?
[22:57] Guthur sustrik, ack
[22:58] mikko i think it might be useful at some point to look into the todo items and sort them by value and effort
[22:58] Guthur except for subscription string though, correct?
[22:58] Guthur strings*
[22:59] mikko that way people needing those features would have a place to check "rough estimation" and possibly chip in
[22:59] mikko currently todo-list is github tickets mainly (?)
[23:01] mikko
[23:01] mikko interesting, as this seems like the feature that would allow integrating udt with minimal pain
[23:09] Guthur I updated the docs on a my fork, should I just submit a pull request?
[23:09] mikko i think currently things go as signed patches over the mailing lists
[23:10] Guthur ah ok
[23:36] Guthur I have two minor doc updates, should they be separate patches?