Wednesday December 29, 2010

[Time] NameMessage
[00:40] s0undt3ch hello ppl
[00:41] s0undt3ch I'm getting "Assertion failed: !pgm_supported() ...." when trying to run one of the pyzmq examples
[01:02] bgranger s0undt3ch: what version of pyzmq and zeromq are you running?
[01:03] s0undt3ch bgranger: well the ones I get from the ppa repo are pyzmq 2.0.7 and zmq 2.0.10
[01:03] s0undt3ch bgranger: I'm now trying to build local versions of the libraries to see if I can get over that
[01:03] bgranger This is a known bug that has been fixed in 2.0.10 of pyzmq. Can you give that a shot?
[01:03] bgranger Yep, that would solve it.
[01:03] s0undt3ch oh sorry
[01:04] bgranger But, just so you know, the library should work fine even though the tests fail.
[01:04] s0undt3ch I think I was using 2.0.10 of pyzmq
[01:04] s0undt3ch let me clean up the mess first
[01:04] bgranger OK
[01:05] bgranger Which example were you running.
[01:05] s0undt3ch heartbeat/
[01:05] s0undt3ch I'm building 2.0.10 now
[01:07] s0undt3ch ok, I now have the 2.0.10 pyzmq
[01:07] s0undt3ch bgranger: same error
[01:08] s0undt3ch ok, both packages libzmq0 and pyzmq are at 2.0.10
[01:09] s0undt3ch bgranger: yep, I've added a print statement at the top of
[01:09] s0undt3ch print zmq.__version__, zmq.__file__
[01:10] s0undt3ch output is:
[01:10] s0undt3ch 2.0.10 /home/vampas/projects/.virtual_python/lib/python2.6/site-packages/zmq/__init__.pyc
[01:10] s0undt3ch Assertion failed: !pgm_supported () (zmq.cpp:240)
[01:10] s0undt3ch Aborted
[01:12] bgranger Can you comment out line 17 of the and retry?
[01:12] bgranger The one where the Context is created.
[01:13] s0undt3ch bgranger: yeah, the Context() was what was causing it
[01:16] bgranger The issue is that when pgm is enabled (multicast), only 1 context can be created. The ThreadDevice creates one (which is fine), but so did line 17. The one in L17 was never used though.
[01:16] s0undt3ch bgranger: zmq.Context() wasn't doin' anything there right?
[01:17] bgranger This is a general limitation of zeromq - that with pgm, only 1 context can be used.
[01:17] bgranger But, pyzmq+zeromq should be working fine for you.
[01:17] s0undt3ch bgranger: but instead of completely failing can't it just warn or ignore and continue?
[01:18] bgranger As far as pyzmq goes, we don't have a clean way of doing that yet in general. We would need zeromq to handle it differently, but zeromq is limited by the initialization model of pgm.
[01:19] bgranger Tough to get around.
[01:19] bgranger At least, that is how I remember the situation for now.
[01:19] s0undt3ch well Thanks!
[01:19] s0undt3ch :)
[01:19] bgranger Sure
[01:19] bgranger At least it is working.
[01:20] s0undt3ch anyway, now for some implementation thought
[01:20] s0undt3ch *thoughts
[01:20] s0undt3ch here's my problem
[01:20] bgranger Sure
[01:21] s0undt3ch I need to build a system that listens to audio streams, analyses them, and sends events to clients
[01:21] s0undt3ch I created this already using twisted, not that good though
[01:21] bgranger OK
[01:22] s0undt3ch what I'm trying to achieve now is use green threads, ie, eventlet or gevent
[01:22] bgranger Are the audio streams coming over zeromq as well?
[01:22] s0undt3ch the issue with this is that gstreamer the audio lib I'll use is not integrated with either eventlet nor gevent
[01:23] s0undt3ch bgranger: nope, zmq was going at first to be used for the events messaging
[01:23] bgranger OK
[01:23] bgranger So you have 1 process that is using gstreamer to look at audio streams and you want to fire off event to clients.
[01:23] bgranger How many clients?
[01:23] s0undt3ch what I've been advised by some eventlet ppl is to run each gstreamer audio stream in it's own process
[01:24] s0undt3ch at most 100 clients at least for now
[01:24] bgranger OK
[01:24] s0undt3ch this is going to be a tool to be used inside the company
[01:24] bgranger OK
[01:24] bgranger So you actually have N gstreamer processes and M clients
[01:24] s0undt3ch yes
[01:24] bgranger OK
[01:24] bgranger And you want all M clients to get the events from all N streaming processes?
[01:25] s0undt3ch so what I'm thinking now is, also use zmq to communicate between the gstreamer processes and a "core daemon" which will get all messages and deliver them to clients
[01:25] s0undt3ch well, all clients should receive all messages
[01:25] s0undt3ch so
[01:25] s0undt3ch yes
[01:26] s0undt3ch "all M clients to get the events from all N streaming processes"
[01:26] s0undt3ch since the core daemon and the gstreamer process will be on the same machine I though of using ipc
[01:27] s0undt3ch but as you can see there are several needs for the zmq lib which I'm just starting to get aquainted
[01:27] s0undt3ch oh
[01:27] bgranger Yes
[01:27] s0undt3ch the gstreamer processes will need to be controlable
[01:27] bgranger In what sense?
[01:27] s0undt3ch ie, I'll need to pass "commands" to it
[01:28] s0undt3ch ie, should it stop, change the stream configuration variables, etc
[01:28] bgranger OK, let's start with the basics of getting the events to the clients.
[01:29] s0undt3ch ok
[01:29] bgranger Let's call the gstreamer processes source processes.
[01:29] s0undt3ch sure
[01:29] bgranger For each source process, you will have a PUB socket.
[01:29] bgranger Each of those PUB sockets will connect to a FORWARDER device that runs in a separate process.
[01:30] bgranger The job of the FOWARDER process is to collect all the events and publish them to all clients. The clients will each have a SUB socket that is connected to the other end of the FORWARDER process.
[01:31] s0undt3ch ok
[01:31] s0undt3ch so far, I'm understanding :)
[01:32] bgranger You could do all of this w/o the FORWARDER in the middle, but having it will make it much easier for all the clients to get the events of all the sources.
[01:32] s0undt3ch ok
[01:32] bgranger I would start out by prototyping that.
[01:32] s0undt3ch yeah, get some example code going on right?
[01:33] bgranger Then if you want to enable the clients to control the sources, I would use REQ/REP or XREQ/XREP sockets separately
[01:33] bgranger Yes
[01:33] bgranger The overall idea is to use different socket pairs for each type of communication pattern in your distributed application. In some of our more complicated apps, we have many different zeromq sockets.
[01:34] s0undt3ch what are the diferences between the req/rep and x version of them
[01:36] bgranger There are a number of differences, I would check out the guide:
[01:36] bgranger
[01:36] s0undt3ch I will
[01:36] bgranger In most cases find myself using the X variants as they are more flexible and handle multiple peers asynchronously.
[02:13] s0undt3ch nice reading
[02:13] s0undt3ch so far I'm not thinking about scalability, at least at the sources level
[02:14] s0undt3ch the only part that should be scalable is client handling
[02:14] s0undt3ch anyway, let cook up some code...
[02:14] s0undt3ch *let's
[08:49] jugg Having an xrep (bind)/req (connect) setup, with xrep implementation using zmq 2.1.x and the req implementation using 2.0.10, and the req implementation being a single process with multiple threads, each thread having a req-socket. It is always the case that eventually the req-socket after send'ing blocks on recv never getting a response. Upgrading the req implementation to 2.1.0 greatly reduces this occurrence frequency, but it still happens.
[08:50] jugg Sometimes instead of locking on recv, zmq asserts while blocking in recv with this error (zmq v2.1.0): Assertion failed: zmq_msg_size (msg_) == 0 (req.cpp:87).
[08:51] jugg Has that use case xrep(bind)/req(connect) been tested much?
[08:54] sustrik the assert looks like there's an malformed message being received on the wire
[08:56] jugg Perhaps the xrep isn't filtering the id part correctly, and sending the separator message?
[08:56] sustrik dunno, can you make a minimal test case?
[08:56] jugg I'm still working on that...
[08:57] jugg I jumped in here where I found that changing the req implementation zmq library version changed the behavior... I had been expecting to find the issue on the xrep side.
[08:59] jugg is the use case known to be robust, or does it have little known exposure?
[09:03] sustrik which part are you speaking about now? the blocking on recv or the assert?
[09:09] jugg one xrep(bind)/many req(connect) pair.
[09:11] sustrik what behaviour change?
[09:12] sustrik the assert?
[09:12] jugg the blocking on recv frequency diminished moving from 2.0.10 to 2.1.0.
[09:13] jugg The assert is random as ever with both versions.
[09:13] sustrik are you closing the req peer at that moment?
[09:15] jugg no, it happens at various points in the set of messages/responses I'm processing.
[09:16] sustrik are you setting HWMs?
[09:16] jugg no
[09:16] sustrik then it should not happen imo
[09:17] sustrik looks like a bug
[09:18] jugg If I can reproduce the issue on a simple test case, I'll pass it along.
[09:18] sustrik thanks
[09:31] s0undt3ch hello ppl, are there any issue using pyzmq inside a chroot?
[09:31] s0undt3ch
[14:30] CIA-21 jzmq: 03Gonzalo Diethelm 07master * ree2d21e 10/ (6 files in 2 dirs):
[14:30] CIA-21 jzmq: Added support for unregistering sockets from poller.
[14:30] CIA-21 jzmq: Applied standard indentation everywhere on Java files. -
[14:43] jgarbers Hi! I'm trying to build zmq under Visual Studio 2008 -- but the current source release seems to have VS2010 .vcproj files in its solution, so VS2008 can't read them. Is there a straightforward way to use VS2008 instead of VS2010?
[14:45] sustrik are they vs2010 project files?
[14:46] sustrik i recall i built it with vs2008 last time i tried
[14:46] jgarbers seems so, or maybe i'm doing something wrong
[14:47] jgarbers the VS2008 reports an error when opening the .sln
[14:47] jgarbers let me get the details again
[14:47] jgarbers i'm hoping this is a simple pilot error that will make me feel stupid for 5 minutes and let me get back to work :-)
[14:48] jgarbers says "...libzmq\libzmq.vcproj' cannot be opened because its project type (.vcproj) is not supported by this version of Visual Studio."
[14:48] jgarbers get that same error for each of the projects
[14:50] sustrik hm, no idea, sorry
[14:51] jgarbers darn. thanks anyway!
[15:01] jgarbers okay, it was indeed pilot stupidity. my VS2008 doesn't have C++ -- just C# and VB. Cripes. Off to go get the appropriate compiler, which *may* help.
[16:49] shykes hello
[16:49] shykes I'm seeing a weird issue when trying to PUB via a higher-latency link
[16:50] cremes what do you see?
[16:51] shykes ... they never arrive *if sent from a home DSL link*
[16:51] shykes PUB and PUSH messages *do* arrive at the same location if sent from the same datacenter
[16:52] cremes are you both working on the same issue? interleaving your messages like that is confusing....
[16:52] cremes ok
[16:53] cremes what does the "on tcp sockets btw" comment mean? you are using both 0mq and your own sockets?
[16:54] cremes ok, tcp transport in 0mq-speak
[16:54] cremes so to recap, req/rep works but pub & push do not
[16:55] sustrik strange
[16:55] cremes do *all* messages disappear or do you get *some* of them?
[16:55] sustrik have a look whether the underlying tcp connection exists
[16:56] sustrik netstat or something
[16:56] cremes in your client program with the SUB socket, do you use setsockopt with ZM_SUBSCRIBE to set your subscription?
[16:56] cremes if you don't, all messages are dropped
[16:56] cremes good
[16:58] sustrik i mean -- there may be a problem establishing the connection is particular direction
[16:58] sustrik due to firewalls
[16:58] sustrik the same way as in your use case?
[16:59] sustrik it may depend on the direction of connection establishment
[16:59] sustrik i.e. where you do bind and where you do connecty
[17:00] sustrik same as in your use case?
[17:00] sustrik anyway, start you app and check whether the TCP connection is established
[17:01] sustrik that would be the best point to start the investigation
[17:03] sustrik it's PUSH/PULL, right?
[17:04] sustrik let's focus on push/pull now
[17:04] sustrik can you check whether data are passed on the wire?
[17:05] sustrik using wireshark or tcp dump?
[17:09] shykes Ok, we just found the problem ;)
[17:09] shykes (taking over from seb`, sorry for the interleaving)
[17:09] shykes in order to give you netstat information, we had the client stay open and wait for manual input to send
[17:10] shykes instead of sending a pre-defined message and exiting instnatly
[17:10] shykes when the client stays connected for a second or two, the messages start arriving
[17:11] shykes looks like there's no guarantee that a connect() is ready when we start sending?
[17:11] shykes I suppose in the context of a never-ending stream of messages, it's no big deal
[17:11] sustrik right
[17:11] sustrik zmq_connect is async
[17:11] shykes So our problem is a non-problem :)
[17:11] sustrik great :)
[17:12] shykes out of curiosity, any way to hook into connect?
[17:12] shykes if we *do* need to wait for it to succeed? (ie for an automated test)
[17:12] sustrik not really, because the connection can be dropped and reestablished in the background
[17:12] sustrik so connect can happen even when no zmq_connect is being executed
[17:13] shykes I see
[17:14] sustrik pollout will succeed even though the connection is not established yet
[17:14] sustrik there's a message buffer
[17:15] sustrik if there's still space in the buffer
[17:15] sustrik you can send messages
[17:15] sustrik they will be passed to the peer as soon as possible
[17:15] sustrik (i.e. when connection is established)
[17:16] sustrik you are welcome
[22:39] s0undt3ch hello ppl, here's a gdb session where I'm having troubles daemonizing my app -> <- seems to be related to pyzmq
[22:57] mikko s0undt3ch: do you fork() ?
[22:57] mikko and if yes do you init zmq before or after fork?
[22:57] s0undt3ch mikko: I'm using python-daemon and yes, it forks
[22:57] s0undt3ch mikko: and yes, I think zmq is setup before the fork
[22:57] s0undt3ch work?
[22:57] s0undt3ch *wrong
[22:57] s0undt3ch *wrong?
[22:57] mikko i remember someone having issues with that
[22:58] mikko try initializing all 0mq stuff after fork
[22:59] s0undt3ch mikko: hmm, zmq is beeing used to message that the app has forket, but I'll try to cook something up
[23:02] mikko are you forking just to daemonize?
[23:05] s0undt3ch mikko: hmm, lemme look at python-daemon's code
[23:06] s0undt3ch mikko: yes, it's forking to daemonize
[23:06] s0undt3ch mikko:
[23:13] s0undt3ch mikko: if an app is chrooted, can it still reach the outside through tcp?
[23:13] s0undt3ch ie, will zmq connected to a tcp socket work?
[23:13] s0undt3ch doesn't seem to
[23:18] s0undt3ch hmm, I aparently have to mount proc inside the chroot, have a proper resolv.conf
[23:18] s0undt3ch so, aparently I have to solve both the daemonizing issues and also the chroot issues
[23:22] s0undt3ch hmm, how the hell does mogrel2 chroot
[23:53] mikko s0undt3ch: im not sure whether the fork actually causes the issues
[23:53] mikko but i think you are supposed to init zmq context for each fork