ZeroMq IRC Log

Saturday September 4, 2010

[Time] Name	Message
[01:19] lestrrat	sustrik: I'm going to be out for the day, so won't be able to check for another 8hrs or so
[01:19] lestrrat	but will do when I get back
[06:03] joevandyk	hello!
[06:04] joevandyk	i'm using the ruby zmq gem. i'm using the req/res sockets. When i try to send a message after a client has disconnected and i've specified no blocking, the sender just blocks forever.
[06:13] joevandyk	I think I see the problem. When the service disconnects, the client still tries to send a message, and is waiting for a response.
[06:13] joevandyk	is there a way around that?
[06:14] joevandyk	i wonder if this is a bug in the ruby library
[06:14] lestrrat	sustrik: I sent a message to the mailing list, but I messed my From: field :/
[06:15] lestrrat	sustrik: anyway, I all I wanted to say was that the eintr branch worked :)
[07:51] sustrik	lestrrat: great, thanks!
[07:51] sustrik	joevandyk: it's the way 0mq works
[07:52] joevandyk	seems odd that if the service disconnects that the client is hung forever
[07:52] sustrik	it'll continue working once service becaomes available
[07:53] sustrik	if you want to explicitly exit in case your task haven't been completed in a predefined time, use timeouts
[07:59] joevandyk	sustrik: hm, i don't see how it can become unblocked once it's waiting for a recv
[07:59] sustrik	use zmq_poll
[07:59] sustrik	there's a timout parameter there
[07:59] joevandyk	sustrik: what i'm not understanding is why the client sends a message when there's no servers to receive it
[08:00] sustrik	what does "there's no server" mean?
[08:00] sustrik	there's no such thing in networking
[08:01] sustrik	what you have otoh is "the transfer haven't been acknowledged in time"
[08:01] sustrik	which is the timeout
[08:02] joevandyk	sorry -- when the REP socket disconnects. The REQ client had been happily sending data to the REP. But then the REP disconnected. The REQ tried to send data again to the REP and the send suceeded, but there was nothing to send it to.
[12:14] Tasser	cremes, got me the big picture for zmqmachine?
[12:33] mato	re
[12:33] sustrik	hi
[12:34] mato	sustrik: list of patches to cherry-pick along with instructions coming up shortly
[12:34] sustrik	thx
[12:34] mato	sustrik: just need to do some housekeeping (also delete and recreate FD_SETSIZE issue in tracker since I found that bug, not pieter)
[12:34] sustrik	ok
[12:42] mato	ok, i can't delete it, have commented with the current situation and removed the "maint" label
[12:44] sustrik	fine
[12:49] mato	sustrik: mumble... pieter also deleted the "empty" zmq_forwarder/streamer/queue documentation
[12:49] mato	sustrik: this has packaging implications
[12:50] mato	sustrik: how would you like me to resolve this?
[12:50] sustrik	get it back
[12:50] mato	ok, what about the zmq_device documentation he added?
[12:50] mato	are devices still experimental in 2.0.x?
[12:51] mato	or would you prefer to revert his commit entirely?
[12:51] mato	the problem is it has other stuff mixed up in it
[12:51] mato	like changes to asciidoc.conf which are half-broken
[12:52] sustrik	is there any problem with device documentation
[12:52] mato	i don't know, but you were keeping it as undocumented
[12:52] mato	there is a problem with the other parts of the commit
[12:53] mato	like the asciidoc footer only being present for manpages, not for html
[12:53] sustrik	what's easier for you?
[12:54] mato	less work right now would be just to revert the whole commit
[12:54] mato	and pieter can resubmit his changes
[12:55] mato	hmm, maybe
[12:55] sustrik	it's up to you
[14:03] CIA-20	zeromq2: 03Martin Lucina 07master * r2673a84 10/ (34 files): (log message trimmed)
[14:03] CIA-20	zeromq2: Merge branch 'maint'
[14:03] CIA-20	zeromq2: * maint:
[14:03] CIA-20	zeromq2: doc: Update zmq_socket(3) for 2.0.8 API changes
[14:03] CIA-20	zeromq2: Revert "Added man page for the zmq_device method"
[14:03] CIA-20	zeromq2: Revert "Added clean target that deletes generated man pages"
[14:03] CIA-20	zeromq2: Revert "Various changes to documentation project:"
[14:03] CIA-20	zeromq2: 03Martin Lucina 07maint * ree3444f 10/ doc/zmq_socket.txt : doc: Update zmq_socket(3) for 2.0.8 API changes (+8 more commits...) - http://bit.ly/aRdJ8z
[14:25] CIA-20	zeromq2: 03Martin Lucina 07master * rf850190 10/ include/zmq.h : zmq.h: Fix typo and use of C99 comment - http://bit.ly/95mVlM
[14:25] CIA-20	zeromq2: 03Martin Lucina 07master * r51a84c1 10/ (src/select.cpp src/zmq.cpp):
[14:25] CIA-20	zeromq2: zmq::select_t, zmq_poll(): assert if FD_SETSIZE reached
[14:25] CIA-20	zeromq2: Ensure that 0MQ does not attempt to call select() on more than FD_SETSIZE
[14:25] CIA-20	zeromq2: file descriptors. - http://bit.ly/ap5axW
[14:25] CIA-20	zeromq2: 03Martin Lucina 07master * rca17612 10/ (include/zmq.h src/select.cpp src/zmq.cpp):
[14:25] CIA-20	zeromq2: Merge branch 'maint'
[14:25] CIA-20	zeromq2: * maint:
[14:25] CIA-20	zeromq2: zmq::select_t, zmq_poll(): assert if FD_SETSIZE reached
[14:25] CIA-20	zeromq2: zmq.h: Fix typo and use of C99 comment
[14:25] CIA-20	zeromq2: Conflicts:
[14:25] CIA-20	zeromq2: src/zmq.cpp - http://bit.ly/9kFuS4
[14:25] CIA-20	zeromq2: 03Martin Lucina 07maint * rf850190 10/ include/zmq.h : zmq.h: Fix typo and use of C99 comment - http://bit.ly/95mVlM
[14:25] CIA-20	zeromq2: 03Martin Lucina 07maint * r51a84c1 10/ (src/select.cpp src/zmq.cpp):
[14:25] CIA-20	zeromq2: zmq::select_t, zmq_poll(): assert if FD_SETSIZE reached
[14:25] CIA-20	zeromq2: Ensure that 0MQ does not attempt to call select() on more than FD_SETSIZE
[14:25] CIA-20	zeromq2: file descriptors. - http://bit.ly/ap5axW
[15:12] CIA-20	zeromq2: 03Ivo Danihelka 07maint * rae567be 10/ (AUTHORS src/zmq.cpp): improved null checking in zmq_term - http://bit.ly/anJS6s
[15:12] CIA-20	zeromq2: 03Jon Dyte 07maint * rc2f3b3b 10/ (src/forwarder.cpp src/queue.cpp src/streamer.cpp): forwarder and streamer devices handle multi-part messages correctly - http://bit.ly/aRxGoy
[15:12] CIA-20	zeromq2: 03Dhammika Pathirana 07maint * r1022789 10/ src/zmq_decoder.cpp : assert on malformed messages - http://bit.ly/907fWH
[15:12] CIA-20	zeromq2: 03Martin Sustrik 07maint * rdb7fe85 10/ include/zmq.h : Broken device numbering reverted - http://bit.ly/cncpxn
[15:20] CIA-20	zeromq2: 03Ivo Danihelka 07master * rae567be 10/ (AUTHORS src/zmq.cpp): improved null checking in zmq_term - http://bit.ly/anJS6s
[15:20] CIA-20	zeromq2: 03Jon Dyte 07master * rc2f3b3b 10/ (src/forwarder.cpp src/queue.cpp src/streamer.cpp): forwarder and streamer devices handle multi-part messages correctly - http://bit.ly/aRxGoy
[15:20] CIA-20	zeromq2: 03Dhammika Pathirana 07master * r1022789 10/ src/zmq_decoder.cpp : assert on malformed messages - http://bit.ly/907fWH
[15:20] CIA-20	zeromq2: 03Martin Sustrik 07master * rdb7fe85 10/ include/zmq.h : Broken device numbering reverted - http://bit.ly/cncpxn
[15:20] CIA-20	zeromq2: 03Martin Sustrik 07master * r76f2e5d 10/ include/zmq.h : (log message trimmed)
[15:20] CIA-20	zeromq2: Merge branch 'maint'
[15:20] CIA-20	zeromq2: * maint:
[15:20] CIA-20	zeromq2: Broken device numbering reverted
[15:20] CIA-20	zeromq2: assert on malformed messages
[15:20] CIA-20	zeromq2: forwarder and streamer devices handle multi-part messages correctly
[15:20] CIA-20	zeromq2: improved null checking in zmq_term
[15:34] AndrewBC	I'm getting a failed assertion with a somewhat altered example from the user guide: specifically "Assertion failed: msg_->flags & ZMQ_MSG_MORE (..\..\..\src\req.cpp:225)", The code is here: http://andrewbc.pastebin.com/gcYPMPqM -- I found this thread: http://www.mail-archive.com/zeromq-dev@lists.zeromq.org/msg02531.html but I don't see any reason why the reply to my REQ socket would be ill-formed... Maybe I'm missing something,
[15:34] AndrewBC	and could use a fresh pair of eyes.
[15:35] AndrewBC	The actual example is just the multithreaded server part: http://www.zeromq.org/docs:user-guide#toc26 I pretty much hacked up a single thread for the client
[15:36] AndrewBC	I also don't see anything pertaining to this in the buglist
[15:46] AndrewBC	That gibberish output is looking more and more sinister, you know
[15:47] AndrewBC	I wonder what that's about
[15:50] CIA-20	zeromq2: 03Martin Lucina 07master * ra6d3629 10/ (Makefile.am configure.in ChangeLog): (log message trimmed)
[15:50] CIA-20	zeromq2: build: Generate ChangeLog in 'make dist', ZIP automatically
[15:50] CIA-20	zeromq2: Change 'make dist' to generate the Git ChangeLog file, that way it doesn't
[15:50] CIA-20	zeromq2: have to be manually updated nor kept in Git which causes unnecessary work.
[15:50] CIA-20	zeromq2: Also change 'make dist' to invoke 'dist-zip' automatically to generate a
[15:50] CIA-20	zeromq2: ZIP as well as a .tar.gz.
[15:50] CIA-20	zeromq2: Thanks to http://live.gnome.org/Git/ChangeLog for the inspiration to
[15:50] CIA-20	zeromq2: 03Martin Lucina 07master * r32fd916 10/ doc/asciidoc.conf :
[15:50] CIA-20	zeromq2: doc: Add 0MQ version to XHTML11 backend footer
[15:50] CIA-20	zeromq2: Thanks to Matt Weinstein for the suggestion. - http://bit.ly/d1r24H
[15:50] CIA-20	zeromq2: 03Martin Lucina 07master * r1e84519 10/ .gitignore : Update .gitignore - http://bit.ly/bH7Jhp
[15:50] CIA-20	zeromq2: 03Martin Lucina 07master * rd4c8de5 10/ (5 files in 2 dirs):
[15:50] CIA-20	zeromq2: Merge branch 'maint'
[15:50] CIA-20	zeromq2: * maint:
[15:50] CIA-20	zeromq2: Update .gitignore
[15:50] CIA-20	zeromq2: doc: Add 0MQ version to XHTML11 backend footer
[15:54] AndrewBC	Okay, I tracked it down to the zmq_recv() in my client. Perhaps there's something special in the flags I need to be aware of since it's on the other side of an XREP
[15:54] AndrewBC	So I'm reading the API docs now
[16:01] AndrewBC	beginning to suspect that I should be checking for multipart messages
[16:06] sccolbert	Perhaps a simple problem, but I appear to have a mem leak on the SUB side of a PUB/SUB pattern
[16:06] sccolbert	when I connect the SUB to the PUB, any messages not matching the filter cause wasted memory
[16:07] sccolbert	a simple test send say, 1 million short messages that dont match any filter on the SUB, will cause the memory usage of the SUB to explode
[16:07] sccolbert	has anyone else experienced this?
[16:08] AndrewBC	I assume that since the filtering is done on the SUB side, the messages are indeed being transmitted to the socket on that end
[16:09] sccolbert	this is true, but their memory is never freed
[16:10] AndrewBC	Are you using a binding, or what?
[16:10] sccolbert	PyZMQ
[16:10] sccolbert	on ubuntu 10.04
[16:11] sccolbert	so if I just connect the SUB and dont set any filter, the SUB process quickly locks up the system
[16:11] sccolbert	which I cant see as being related to PyZMQ
[16:12] AndrewBC	I'm not sure, since I'm not a dev of zmq. It's supposed to handle the memory just fine, if wishes were horses...
[16:12] AndrewBC	It could be either, though. All it takes is a circular reference for python to not garbage collect objects
[16:13] AndrewBC	circular, cyclic, bah
[16:14] sccolbert	this is true, though I would think that since I havent actually called recv() on the sub (since its filtering all messages) that python isnt even aware a message arrived
[16:21] AndrewBC	Wish I could help you. I'm banging my head up against a bizzare failed assertion myself
[16:38] CIA-20	zeromq2: 03Martin Sustrik 07maint * rebf3089 10/ NEWS : NEWS updated for 2.0.9 - http://bit.ly/doVZJ4
[16:56] pieter_hintjens	sccolbert: what version of 0MQ are you using?
[16:58] sccolbert	pieter_hintjens: 2.0.7
[16:58] pieter_hintjens	when you stop the publisher does the memory consumption of the subscriber go down again?
[16:59] sccolbert	hmm havent tried that
[16:59] sccolbert	gimme a sec
[17:01] sccolbert	no
[17:01] sccolbert	killing the publisher does not free the memory in the SUB
[17:04] pieter_hintjens	hmm... so the next steps are to test with the latest 2.0.9 release
[17:04] pieter_hintjens	and then make a minimal test case that reproduces this, and log an issue
[17:04] sccolbert	k, lemme grab the new one along with the new PyZMQ
[17:04] sccolbert	will only take me a couple minutes
[17:04] sccolbert	a different question:
[17:04] pieter_hintjens	if you can make a parallel test case in C or C++ that will help identify whether it's in PyZMQ or 0MQ
[17:05] sccolbert	sure
[17:05] pieter_hintjens	shoot...
[17:05] sccolbert	in a pub/sub pattern, if i send a multipart message, and the first part doesnt match the sub filter, does the sub still recieve the rest of the message?
[17:06] sccolbert	and then just dump it
[17:06] pieter_hintjens	i actually added this to the guide today but did not publish it yet
[17:06] pieter_hintjens	a multipart message is one message
[17:06] pieter_hintjens	think of the first part as the envelope
[17:06] pieter_hintjens	it's very convenient to put the pubsub key there
[17:06] pieter_hintjens	if the key does not match any subscription the entire message is discarded / ignored
[17:07] sccolbert	is there any performance improvement to be had with that, over just joining my key to the beggininng of my message?
[17:07] pieter_hintjens	using a pubsub envelope like this means you can do zerocopy on the data part
[17:07] sccolbert	ah, great
[17:07] pieter_hintjens	there is no significant performance cost and you can save on formatting the full message
[17:08] pieter_hintjens	plus you are guaranteed the key remains separate from the data
[17:08] pieter_hintjens	it's a Good Thing
[17:08] sccolbert	yep
[17:09] sccolbert	also, in a general pub/sub pattern do you recommend epgm over tcp?
[17:09] CIA-20	zeromq2: 03Martin Sustrik 07maint * r01c463c 10/ (builds/msvc/platform.hpp configure.in): Version number incremented to 2.0.10 - http://bit.ly/9vL8xS
[17:09] sccolbert	I wasnt able to find much literature on the pro's vs cons of each
[17:10] pieter_hintjens	you'd only use pgm/epgm if you have a performance issue with tcp
[17:10] pieter_hintjens	there is ongoing work to make tcp pubsub much more efficient for large fanouts
[17:10] pieter_hintjens	in general use tcp unless you know why you need multicast
[17:10] sccolbert	cool
[17:10] CIA-20	zeromq2: 03Martin Sustrik 07master * rebf3089 10/ NEWS : NEWS updated for 2.0.9 - http://bit.ly/doVZJ4
[17:10] CIA-20	zeromq2: 03Martin Sustrik 07master * r01c463c 10/ (builds/msvc/platform.hpp configure.in): Version number incremented to 2.0.10 - http://bit.ly/9vL8xS
[17:10] CIA-20	zeromq2: 03Martin Sustrik 07master * rb4740c1 10/ NEWS : (log message trimmed)
[17:10] CIA-20	zeromq2: Merge branch 'maint'
[17:10] CIA-20	zeromq2: * maint:
[17:10] CIA-20	zeromq2: Version number incremented to 2.0.10
[17:10] CIA-20	zeromq2: NEWS updated for 2.0.9
[17:10] CIA-20	zeromq2: Conflicts:
[17:10] CIA-20	zeromq2: builds/msvc/platform.hpp
[17:11] Steve-o	usually tcp up to 4-5 peers is better than multicast for almost all scenarios
[17:11] sccolbert	good to know
[17:11] sccolbert	thanks
[17:13] Steve-o	usually NICs give you some form of TCP acceleration compared to none for UDP multicast
[17:16] sustrik	uh, 2.0.9 is out
[17:47] sccolbert	pieter_hintjens: tested new version in C and Python
[17:47] sccolbert	same issues
[17:48] sccolbert	where should I log the issue?
[17:50] sccolbert	ah, ok, it only happens in C when the sub hasnt set any filter
[17:50] sccolbert	setting a filter that doesnt match anything from the PUB solves the issue
[17:50] sccolbert	in C at least
[17:51] sccolbert	i will file this with Brian
[19:01] twcc	Hey all, I'm trying to figure out if zeromq ist the right tool for me. I want to have one "Server" and multiple client, which pushes messages to the server. I can't find a pattern which fits really good - Request-reply would fit best, but as I understood I have to send a reply. - Is this case so trivial, that I should fall back to standard socket programming?
[19:37] Steve-o	twcc: sounds like PUB/SUB
[19:43] twcc	Because of the unidirectional manner this would fit, but AFAIK the Publisher starts the socket and the subscriber has to connect?! - This could run into problems with a changing number of publisher, or can I connect them to one socket, and zmq handles this?
[19:53] Steve-o	You can connect multiple to one zmq socket as detailed in zmq_socket(3)
[19:54] Steve-o	there is currently no bus topology socket pattern
[19:55] Steve-o	you can jump down to raw PGM for that
[19:57] twcc	maybe i'm stuck to much to socket programming ... - so I can bind multiple publisher to the same socket/port and have one subscriber which processes the messages? That would be great and solves all my problems :-)
[20:01] kooroo	hey folks, I'm having trouble compiling zeromq on rh5 x86_64
[20:01] kooroo	.libs/libzmq_la-txwi.o: relocation R_X86_64_PC32 against `pgm_rs_create' can not be used when making a shared object; recompile with -fPIC is what I'm getting.
[20:02] kooroo	has anyone come across this before?
[20:03] Steve-o	twcc: for each subscriber you call zmq_connect for each publisher on one zmq socket
[20:04] Steve-o	kooroo: it is a bug in RHEL GCC, you can disable PGM or there is a flag you can set to disable visibility on the PGM internal API
[20:07] kooroo	Steve-o, is this the -DPGM_GNUC_INTERNAL flag?
[20:07] Steve-o	kooroo: yes, it needs to be "-DPGM_GNUC_INTERNAL=" for broken GCC, default "-DPGM_GNUC_INTERNAL=G_GNUC_INTERNAL"
[20:08] Steve-o	I think someone found a backported GCC version that works too on the list before
[20:10] kooroo	Steve-o: do you know offhand if upgrading gcc works?
[20:11] Steve-o	4.1.2 is the affected version
[20:11] Steve-o	here's the thread from before: http://www.mail-archive.com/zeromq-dev@lists.zeromq.org/msg01646.html
[20:12] kooroo	Steve-o: thanks, Imma upgrade gcc and try it again.
[20:12] twcc	Hm, maybe I explained it bad - I have a changing number of processes (publisher) which have to send messages to one sink (daemon/subscriber), which handles these messages. A topologie where the daemon/subscriber has to connect to the publisher isn't feasible. - So the client/server application would fit best, except that the server has to response to a message.
[20:13] kooroo	twcc, could you have them all subscribe to the same multicast stream and only the subscriber react to messages?
[20:14] kooroo	not the most efficient way, I admit.
[20:17] twcc	hm, funny idea, didn't thought of it
[20:19] twcc	multicast is only possible on network connections?, because all the processes are running on the same computer - so i would like to use ipc
[20:24] Steve-o	multicast requires a network for reliable delivery, it's pretty useless for ipc.
[20:30] Steve-o	I think as IPC is currently implemented using local sockets you would have to add a shared memory transport for localhost broadcast
[20:32] twcc	I think the multicast idea would only be a workaround - I would like to have a non-blocking "Request-reply" without reply pattern, but I can't figure out, how to realize it with zeromq
[20:36] twcc	Thank you for your help - I will try if the request-reply pattern satisfy the requirements and handle the non-blocking by myself
[20:43] Steve-o	post a message on the list too, it sounds like a good idea for v3, I think 29West have a broadcast IPC transport too
[22:35] DasIch	i started playing around with zeromq using pyzmq from here https://launchpad.net/~chris-lea/+archive/zeromq, however if i create 2 contexts i get this message http://paste.pocoo.org/show/258660/ what's this about?
[22:37] DasIch	(obviously what i really did was a bit more complex but that what it boils down to)
[22:40] lestrrat	sustrik: so the eintr thins is going to be in 2.1?
[23:01] lestrrat	sustrik: And the ZMQ_FD stuff to get at the file descriptor -- is that going to be in 2.1 as well?