ZeroMq IRC Log

Thursday July 15, 2010

[Time] Name	Message
[01:05] sd88g93	in what circumstances is it appropriate to have more than one context ? i.e. passing a value greater than 1 into zmq_init( )
[02:09] cremes	sd88g93: if you are producing under 1 Gb/sec, there is no reason for multiple i/o threads
[02:09] sd88g93	oh ok
[02:10] sd88g93	do you know of a scenario when they are useful ? just when over 1 gb/sec ?
[04:30] ditou	Is it possible to have multiple publishers connected to the same endpoint in PUB/SUB scheme?
[04:30] ditou	I bind a first publisher then connect another and the connect two subscribers.
[04:30] ditou	Then I use second publisher to send message. I'm neither receiving any messages nor getting any errors.
[04:31] ditou	Any ideas?
[04:32] ditou	...using C API.
[06:46] mikko	ditou: no, not directly
[07:31] parsu	Hi, with pyzmq I am trying to use multiple zmq contexts in same application. It crashes when it tries to creates second context. the zmq api doc says we can create multiple contexts in 1 application. Is multiple context thing put in later release? I am using zeromq version 2.0.7. Or I am missing on something.
[07:33] sustrik	it shouldn't crash
[07:33] sustrik	still, there should be no point is using multiple contexts
[07:33] sustrik	how does it crash?
[07:36] parsu	wait I will provide sample code and crahs details
[07:54] parsu	hi
[07:54] parsu	this is sample code
[07:54] parsu	import zmq from threading import Thread class Ser(Thread): def __init__(self, url): Thread.__init__(self) self.ctx = zmq.Context(1) self.sock = self.ctx.socket(zmq.REP) self.url=url self.sock.bind(url) def run(self): while True: print 'watiting for read on: %s ' % self.url msg = self.sock.recv() print msg self.sock.send(self.url)
[07:54] parsu	ohh sorry
[07:54] parsu	import zmq
[07:55] parsu	from threading import Thread
[07:55] parsu	should I copy pase code here?
[07:56] parsu	or send you somewhere?
[07:56] parsu	this is the error printed before crash
[07:56] parsu	Assertion failed: !pgm_supported () (zmq.cpp:239)
[07:58] sustrik	aha, that's the problem with OpenPGM library
[07:58] sustrik	it cannot be initialised twice
[07:59] sustrik	thus, when compiling with --with-pgm, you cannot open two contexts
[07:59] sustrik	this have been fixed in new version of OpenPGM but these were not yet merged into 0mq
[07:59] sustrik	anyway, you shouldn't open two contexts
[07:59] sustrik	why are you doing that?
[08:00] parsu	actually this is sample code
[08:00] parsu	I am calling some other libs
[08:00] parsu	using ctypes
[08:00] parsu	which uses zmq there
[08:00] parsu	so we cant share the contexts
[08:01] parsu	thats where we have stuck
[08:01] sustrik	i see
[08:01] sustrik	do you need PGM?
[08:01] parsu	so if I ermove --with-pgm
[08:01] parsu	it should work ?
[08:01] sustrik	yes
[08:01] parsu	nopes
[08:01] parsu	cool
[08:01] parsu	I will do that
[08:02] parsu	Also can you tell me what is impact of no, of io threas we pass to zmq_init
[08:02] sustrik	in most cases use 1
[08:02] parsu	one of our eng. said he got crashes when he used more than 1
[08:02] parsu	ohh ok
[08:02] sustrik	more I/O threads may be useful if there's a lot of connections involved
[08:02] sustrik	hundreds, thousands
[08:03] parsu	ohk
[08:03] parsu	great
[08:03] parsu	I got my answers
[08:03] parsu	thanks a lot
[08:03] sustrik	parsu: if you get crashes, please do report it
[08:03] sustrik	otherwise it won't get fixed
[08:03] parsu	yups
[08:03] parsu	sure
[08:03] parsu	I will ask him to give me core
[08:03] parsu	I will pass it to you
[08:03] sustrik	thx
[08:03] parsu	thanks a lot :)
[08:04] sustrik	you are welcome
[08:46] parsu	<sustrik> thus, when compiling with --with-pgm, you cannot open two contexts [13:28] <sustrik> this have been fixed in new version of OpenPGM but these were not yet merged into 0mq
[08:46] parsu	Do we have any plans to merge in 0mq?
[09:02] sustrik	yes, eventually
[09:04] parsu	any estimated date?
[10:55] sustrik	parsu: not really
[10:55] sustrik	do you need it?
[11:17] parsu	yaa
[11:28] sustrik	parsu: try asking about the openpgm integration roadmap on the mailing list then
[11:33] parsu	ohk
[11:33] parsu	thanks a lot
[11:33] parsu	for now this is ok, we can use tcp fanout instead
[11:33] parsu	but ya we will need it later
[11:34] parsu	thank bye
[11:58] jugg	hmm, maybe I have erlang bindings working with zeromq finally...
[17:06] sd88g93	hello. ok, i simplified my problem down to a simple case program, and managed to replicate the same error i am getting. here is the code: http://pastebin.com/uk57UAB9
[17:06] sd88g93	basically, the subscriber recieves the first of the published messages from the publisher process, but none of the following ones .
[17:08] sd88g93	It is a pretty simple case, nothing fancy.
[17:35] sustrik	sd88g93: are use using single socket from multiple threads?
[17:36] sd88g93	sustrik: yes, but this example uses just 1 thread.
[17:36] sd88g93	its a publisher socket, so its a global variable accessed by a mutex
[17:37] sustrik	never use single socket from multiple threads
[17:37] sustrik	but if it fails even when accessed from a single thread, then it's a bug
[17:37] sustrik	i'll have a look
[17:37] sd88g93	even if its a publisher socket ?
[17:39] sd88g93	do you have any other suggestions as far as publishing message to one publisher socket, from multiple threads ? the ZMQ_FORWARDER device , or the QUEUE device doesnt seem to work for publish sockets
[17:43] sustrik	ZMQ_FORWARDER should
[17:43] sd88g93	it doesnt seem to like multiple pipes conneting to one pub socket
[17:43] sustrik	?
[17:43] sustrik	do you have the code?
[17:43] sd88g93	yes, in the pastebin
[17:44] sustrik	i mean the one with forwarder device?
[17:44] sd88g93	not using the forwarder device, i'll have to alter that code to try to use the forwarder then
[17:44] sd88g93	i have it but its not a simple case
[17:44] sustrik	just paste it
[17:45] sustrik	if it doesn't work there's a bug somewhere
[17:45] sd88g93	yeah, i think the existing code uses the method in the pastebin right now, i'll alter the example and see if i can get it to work wit the forwarder device, and then paste that
[17:46] sd88g93	it'll just take awhile
[17:46] sd88g93	actually, that code in the pastebin uses 2 threads, the main thread opens the socket, and the thread publishes to the socket, i guess that counts as 2 then
[17:46] sd88g93	i mean, just one thread to do the publishing
[17:57] sustrik	you cannot migrate sockets between threads
[17:57] sustrik	anyway, try with forwarder device and paste it
[18:15] sd88g93	sustrik: here's the code with the forwarder device: http://pastebin.com/gZJjYCky
[18:15] sd88g93	it creates the first thread, but the other threads cannot bind to it. It continues to publish messages with that one thread, but the subscriber doesnt recive it.
[18:48] sd88g93	ok, also tried binding both ends in the main thread and then calling connect() in the worker threads, same result.
[21:13] sd88g93	i think i found a real bug in the forwarding device
[21:13] sd88g93	here's the code that replicates the issue: http://pastebin.com/gZJjYCky
[22:40] veritos	In tcp_socket.cpp:213 of 2.0.7, ECONNRESET and ECONNREFUSED are handled, but not ETIMEDOUT. Timeouts thus cause the assertion failure a couple lines down. Doesn't this seem a bit harsh when just adding the test for ETIMEDOUT is so easy?
[22:45] veritos	Oh, hey, it's fixed in git :)
[22:48] sd88g93	ok, this problem appears not to be confined to just the forwarder device, but publishing in general. when i publish a series of messages, only the first one gets through, the second one is all jumbled, i reproduced the code in simple form here: http://pastebin.com/SrmAz2V2
[22:59] sd88g93	ohhhh ok, you have to re init the msg and close each time you send, i was calling zmq_msg_init_data once and just changing the variables