ZeroMq IRC Log

Wednesday December 29, 2010

[Time] Name	Message
[00:40] s0undt3ch	hello ppl
[00:41] s0undt3ch	I'm getting "Assertion failed: !pgm_supported() ...." when trying to run one of the pyzmq examples
[01:02] bgranger	s0undt3ch: what version of pyzmq and zeromq are you running?
[01:03] s0undt3ch	bgranger: well the ones I get from the ppa repo are pyzmq 2.0.7 and zmq 2.0.10
[01:03] s0undt3ch	bgranger: I'm now trying to build local versions of the libraries to see if I can get over that
[01:03] bgranger	This is a known bug that has been fixed in 2.0.10 of pyzmq. Can you give that a shot?
[01:03] bgranger	Yep, that would solve it.
[01:03] s0undt3ch	oh sorry
[01:04] bgranger	But, just so you know, the library should work fine even though the tests fail.
[01:04] s0undt3ch	I think I was using 2.0.10 of pyzmq
[01:04] s0undt3ch	let me clean up the mess first
[01:04] bgranger	OK
[01:05] bgranger	Which example were you running.
[01:05] s0undt3ch	heartbeat/heart.py
[01:05] s0undt3ch	I'm building 2.0.10 now
[01:07] s0undt3ch	ok, I now have the 2.0.10 pyzmq
[01:07] s0undt3ch	bgranger: same error
[01:08] s0undt3ch	ok, both packages libzmq0 and pyzmq are at 2.0.10
[01:09] s0undt3ch	bgranger: yep, I've added a print statement at the top of heart.py
[01:09] s0undt3ch	print zmq.__version__, zmq.__file__
[01:10] s0undt3ch	output is:
[01:10] s0undt3ch	2.0.10 /home/vampas/projects/.virtual_python/lib/python2.6/site-packages/zmq/__init__.pyc
[01:10] s0undt3ch	Assertion failed: !pgm_supported () (zmq.cpp:240)
[01:10] s0undt3ch	Aborted
[01:12] bgranger	Can you comment out line 17 of the heart.py and retry?
[01:12] bgranger	The one where the Context is created.
[01:13] s0undt3ch	bgranger: yeah, the Context() was what was causing it
[01:16] bgranger	The issue is that when pgm is enabled (multicast), only 1 context can be created. The ThreadDevice creates one (which is fine), but so did line 17. The one in L17 was never used though.
[01:16] s0undt3ch	bgranger: zmq.Context() wasn't doin' anything there right?
[01:17] bgranger	This is a general limitation of zeromq - that with pgm, only 1 context can be used.
[01:17] bgranger	But, pyzmq+zeromq should be working fine for you.
[01:17] s0undt3ch	bgranger: but instead of completely failing can't it just warn or ignore and continue?
[01:18] bgranger	As far as pyzmq goes, we don't have a clean way of doing that yet in general. We would need zeromq to handle it differently, but zeromq is limited by the initialization model of pgm.
[01:19] bgranger	Tough to get around.
[01:19] bgranger	At least, that is how I remember the situation for now.
[01:19] s0undt3ch	well Thanks!
[01:19] s0undt3ch	:)
[01:19] bgranger	Sure
[01:19] bgranger	At least it is working.
[01:20] s0undt3ch	anyway, now for some implementation thought
[01:20] s0undt3ch	*thoughts
[01:20] s0undt3ch	here's my problem
[01:20] bgranger	Sure
[01:21] s0undt3ch	I need to build a system that listens to audio streams, analyses them, and sends events to clients
[01:21] s0undt3ch	I created this already using twisted, not that good though
[01:21] bgranger	OK
[01:22] s0undt3ch	what I'm trying to achieve now is use green threads, ie, eventlet or gevent
[01:22] bgranger	Are the audio streams coming over zeromq as well?
[01:22] s0undt3ch	the issue with this is that gstreamer the audio lib I'll use is not integrated with either eventlet nor gevent
[01:23] s0undt3ch	bgranger: nope, zmq was going at first to be used for the events messaging
[01:23] bgranger	OK
[01:23] bgranger	So you have 1 process that is using gstreamer to look at audio streams and you want to fire off event to clients.
[01:23] bgranger	How many clients?
[01:23] s0undt3ch	what I've been advised by some eventlet ppl is to run each gstreamer audio stream in it's own process
[01:24] s0undt3ch	at most 100 clients at least for now
[01:24] bgranger	OK
[01:24] s0undt3ch	this is going to be a tool to be used inside the company
[01:24] bgranger	OK
[01:24] bgranger	So you actually have N gstreamer processes and M clients
[01:24] s0undt3ch	yes
[01:24] bgranger	OK
[01:24] bgranger	And you want all M clients to get the events from all N streaming processes?
[01:25] s0undt3ch	so what I'm thinking now is, also use zmq to communicate between the gstreamer processes and a "core daemon" which will get all messages and deliver them to clients
[01:25] s0undt3ch	well, all clients should receive all messages
[01:25] s0undt3ch	so
[01:25] s0undt3ch	yes
[01:26] s0undt3ch	"all M clients to get the events from all N streaming processes"
[01:26] s0undt3ch	since the core daemon and the gstreamer process will be on the same machine I though of using ipc
[01:27] s0undt3ch	but as you can see there are several needs for the zmq lib which I'm just starting to get aquainted
[01:27] s0undt3ch	oh
[01:27] bgranger	Yes
[01:27] s0undt3ch	the gstreamer processes will need to be controlable
[01:27] bgranger	In what sense?
[01:27] s0undt3ch	ie, I'll need to pass "commands" to it
[01:28] s0undt3ch	ie, should it stop, change the stream configuration variables, etc
[01:28] bgranger	OK, let's start with the basics of getting the events to the clients.
[01:29] s0undt3ch	ok
[01:29] bgranger	Let's call the gstreamer processes source processes.
[01:29] s0undt3ch	sure
[01:29] bgranger	For each source process, you will have a PUB socket.
[01:29] bgranger	Each of those PUB sockets will connect to a FORWARDER device that runs in a separate process.
[01:30] bgranger	The job of the FOWARDER process is to collect all the events and publish them to all clients. The clients will each have a SUB socket that is connected to the other end of the FORWARDER process.
[01:31] s0undt3ch	ok
[01:31] s0undt3ch	so far, I'm understanding :)
[01:32] bgranger	You could do all of this w/o the FORWARDER in the middle, but having it will make it much easier for all the clients to get the events of all the sources.
[01:32] s0undt3ch	ok
[01:32] bgranger	I would start out by prototyping that.
[01:32] s0undt3ch	yeah, get some example code going on right?
[01:33] bgranger	Then if you want to enable the clients to control the sources, I would use REQ/REP or XREQ/XREP sockets separately
[01:33] bgranger	Yes
[01:33] bgranger	The overall idea is to use different socket pairs for each type of communication pattern in your distributed application. In some of our more complicated apps, we have many different zeromq sockets.
[01:34] s0undt3ch	what are the diferences between the req/rep and x version of them
[01:36] bgranger	There are a number of differences, I would check out the guide:
[01:36] bgranger	http://zguide.zeromq.org/chapter:all#toc41
[01:36] s0undt3ch	I will
[01:36] bgranger	In most cases find myself using the X variants as they are more flexible and handle multiple peers asynchronously.
[02:13] s0undt3ch	nice reading
[02:13] s0undt3ch	so far I'm not thinking about scalability, at least at the sources level
[02:14] s0undt3ch	the only part that should be scalable is client handling
[02:14] s0undt3ch	anyway, let cook up some code...
[02:14] s0undt3ch	*let's
[08:49] jugg	Having an xrep (bind)/req (connect) setup, with xrep implementation using zmq 2.1.x and the req implementation using 2.0.10, and the req implementation being a single process with multiple threads, each thread having a req-socket. It is always the case that eventually the req-socket after send'ing blocks on recv never getting a response. Upgrading the req implementation to 2.1.0 greatly reduces this occurrence frequency, but it still happens.
[08:50] jugg	Sometimes instead of locking on recv, zmq asserts while blocking in recv with this error (zmq v2.1.0): Assertion failed: zmq_msg_size (msg_) == 0 (req.cpp:87).
[08:51] jugg	Has that use case xrep(bind)/req(connect) been tested much?
[08:54] sustrik	the assert looks like there's an malformed message being received on the wire
[08:56] jugg	Perhaps the xrep isn't filtering the id part correctly, and sending the separator message?
[08:56] sustrik	dunno, can you make a minimal test case?
[08:56] jugg	I'm still working on that...
[08:57] jugg	I jumped in here where I found that changing the req implementation zmq library version changed the behavior... I had been expecting to find the issue on the xrep side.
[08:59] jugg	is the use case known to be robust, or does it have little known exposure?
[09:03] sustrik	which part are you speaking about now? the blocking on recv or the assert?
[09:09] jugg	one xrep(bind)/many req(connect) pair.
[09:11] sustrik	what behaviour change?
[09:12] sustrik	the assert?
[09:12] jugg	the blocking on recv frequency diminished moving from 2.0.10 to 2.1.0.
[09:13] jugg	The assert is random as ever with both versions.
[09:13] sustrik	are you closing the req peer at that moment?
[09:15] jugg	no, it happens at various points in the set of messages/responses I'm processing.
[09:16] sustrik	are you setting HWMs?
[09:16] jugg	no
[09:16] sustrik	then it should not happen imo
[09:17] sustrik	looks like a bug
[09:18] jugg	If I can reproduce the issue on a simple test case, I'll pass it along.
[09:18] sustrik	thanks
[09:31] s0undt3ch	hello ppl, are there any issue using pyzmq inside a chroot?
[09:31] s0undt3ch	http://paste.pocoo.org/show/311220/
[14:30] CIA-21	jzmq: 03Gonzalo Diethelm 07master * ree2d21e 10/ (6 files in 2 dirs):
[14:30] CIA-21	jzmq: Added support for unregistering sockets from poller.
[14:30] CIA-21	jzmq: Applied standard indentation everywhere on Java files. - http://bit.ly/fcb5Pi
[14:43] jgarbers	Hi! I'm trying to build zmq under Visual Studio 2008 -- but the current source release seems to have VS2010 .vcproj files in its solution, so VS2008 can't read them. Is there a straightforward way to use VS2008 instead of VS2010?
[14:45] sustrik	are they vs2010 project files?
[14:46] sustrik	i recall i built it with vs2008 last time i tried
[14:46] jgarbers	seems so, or maybe i'm doing something wrong
[14:47] jgarbers	the VS2008 reports an error when opening the .sln
[14:47] jgarbers	let me get the details again
[14:47] jgarbers	i'm hoping this is a simple pilot error that will make me feel stupid for 5 minutes and let me get back to work :-)
[14:48] jgarbers	says "...libzmq\libzmq.vcproj' cannot be opened because its project type (.vcproj) is not supported by this version of Visual Studio."
[14:48] jgarbers	get that same error for each of the projects
[14:50] sustrik	hm, no idea, sorry
[14:51] jgarbers	darn. thanks anyway!
[15:01] jgarbers	okay, it was indeed pilot stupidity. my VS2008 doesn't have C++ -- just C# and VB. Cripes. Off to go get the appropriate compiler, which may help.
[16:49] shykes	hello
[16:49] shykes	I'm seeing a weird issue when trying to PUB via a higher-latency link
[16:50] cremes	what do you see?
[16:51] shykes	... they never arrive if sent from a home DSL link
[16:51] shykes	PUB and PUSH messages do arrive at the same location if sent from the same datacenter
[16:52] cremes	are you both working on the same issue? interleaving your messages like that is confusing....
[16:52] cremes	ok
[16:53] cremes	what does the "on tcp sockets btw" comment mean? you are using both 0mq and your own sockets?
[16:54] cremes	ok, tcp transport in 0mq-speak
[16:54] cremes	so to recap, req/rep works but pub & push do not
[16:55] sustrik	strange
[16:55] cremes	do all messages disappear or do you get some of them?
[16:55] sustrik	have a look whether the underlying tcp connection exists
[16:56] sustrik	netstat or something
[16:56] cremes	in your client program with the SUB socket, do you use setsockopt with ZM_SUBSCRIBE to set your subscription?
[16:56] cremes	if you don't, all messages are dropped
[16:56] cremes	good
[16:58] sustrik	i mean -- there may be a problem establishing the connection is particular direction
[16:58] sustrik	due to firewalls
[16:58] sustrik	the same way as in your use case?
[16:59] sustrik	it may depend on the direction of connection establishment
[16:59] sustrik	i.e. where you do bind and where you do connecty
[17:00] sustrik	same as in your use case?
[17:00] sustrik	anyway, start you app and check whether the TCP connection is established
[17:01] sustrik	that would be the best point to start the investigation
[17:03] sustrik	it's PUSH/PULL, right?
[17:04] sustrik	let's focus on push/pull now
[17:04] sustrik	can you check whether data are passed on the wire?
[17:05] sustrik	using wireshark or tcp dump?
[17:09] shykes	Ok, we just found the problem ;)
[17:09] shykes	(taking over from seb`, sorry for the interleaving)
[17:09] shykes	in order to give you netstat information, we had the client stay open and wait for manual input to send
[17:10] shykes	instead of sending a pre-defined message and exiting instnatly
[17:10] shykes	when the client stays connected for a second or two, the messages start arriving
[17:11] shykes	looks like there's no guarantee that a connect() is ready when we start sending?
[17:11] shykes	I suppose in the context of a never-ending stream of messages, it's no big deal
[17:11] sustrik	right
[17:11] sustrik	zmq_connect is async
[17:11] shykes	So our problem is a non-problem :)
[17:11] sustrik	great :)
[17:12] shykes	out of curiosity, any way to hook into connect?
[17:12] shykes	if we do need to wait for it to succeed? (ie for an automated test)
[17:12] sustrik	not really, because the connection can be dropped and reestablished in the background
[17:12] sustrik	so connect can happen even when no zmq_connect is being executed
[17:13] shykes	I see
[17:14] sustrik	pollout will succeed even though the connection is not established yet
[17:14] sustrik	there's a message buffer
[17:15] sustrik	if there's still space in the buffer
[17:15] sustrik	you can send messages
[17:15] sustrik	they will be passed to the peer as soon as possible
[17:15] sustrik	(i.e. when connection is established)
[17:16] sustrik	you are welcome
[22:39] s0undt3ch	hello ppl, here's a gdb session where I'm having troubles daemonizing my app -> http://paste.pocoo.org/show/311220/ <- seems to be related to pyzmq
[22:57] mikko	s0undt3ch: do you fork() ?
[22:57] mikko	and if yes do you init zmq before or after fork?
[22:57] s0undt3ch	mikko: I'm using python-daemon and yes, it forks
[22:57] s0undt3ch	mikko: and yes, I think zmq is setup before the fork
[22:57] s0undt3ch	work?
[22:57] s0undt3ch	*wrong
[22:57] s0undt3ch	*wrong?
[22:57] mikko	i remember someone having issues with that
[22:58] mikko	try initializing all 0mq stuff after fork
[22:59] s0undt3ch	mikko: hmm, zmq is beeing used to message that the app has forket, but I'll try to cook something up
[23:02] mikko	are you forking just to daemonize?
[23:05] s0undt3ch	mikko: hmm, lemme look at python-daemon's code
[23:06] s0undt3ch	mikko: yes, it's forking to daemonize
[23:06] s0undt3ch	mikko: http://paste.pocoo.org/show/311724/
[23:13] s0undt3ch	mikko: if an app is chrooted, can it still reach the outside through tcp?
[23:13] s0undt3ch	ie, will zmq connected to a tcp socket work?
[23:13] s0undt3ch	doesn't seem to
[23:18] s0undt3ch	hmm, I aparently have to mount proc inside the chroot, have a proper resolv.conf
[23:18] s0undt3ch	so, aparently I have to solve both the daemonizing issues and also the chroot issues
[23:22] s0undt3ch	hmm, how the hell does mogrel2 chroot
[23:53] mikko	s0undt3ch: im not sure whether the fork actually causes the issues
[23:53] mikko	but i think you are supposed to init zmq context for each fork