Thursday July 15, 2010

[Time] NameMessage
[01:05] sd88g93 in what circumstances is it appropriate to have more than one context ? i.e. passing a value greater than 1 into zmq_init( )
[02:09] cremes sd88g93: if you are producing under 1 Gb/sec, there is no reason for multiple i/o threads
[02:09] sd88g93 oh ok
[02:10] sd88g93 do you know of a scenario when they are useful ? just when over 1 gb/sec ?
[04:30] ditou Is it possible to have multiple publishers connected to the same endpoint in PUB/SUB scheme?
[04:30] ditou I bind a first publisher then connect another and the connect two subscribers.
[04:30] ditou Then I use second publisher to send message. I'm neither receiving any messages nor getting any errors.
[04:31] ditou Any ideas?
[04:32] ditou ...using C API.
[06:46] mikko ditou: no, not directly
[07:31] parsu Hi, with pyzmq I am trying to use multiple zmq contexts in same application. It crashes when it tries to creates second context. the zmq api doc says we can create multiple contexts in 1 application. Is multiple context thing put in later release? I am using zeromq version 2.0.7. Or I am missing on something.
[07:33] sustrik it shouldn't crash
[07:33] sustrik still, there should be no point is using multiple contexts
[07:33] sustrik how does it crash?
[07:36] parsu wait I will provide sample code and crahs details
[07:54] parsu hi
[07:54] parsu this is sample code
[07:54] parsu import zmq from threading import Thread class Ser(Thread): def __init__(self, url): Thread.__init__(self) self.ctx = zmq.Context(1) self.sock = self.ctx.socket(zmq.REP) self.url=url self.sock.bind(url) def run(self): while True: print 'watiting for read on: %s ' % self.url msg = self.sock.recv() print msg self.sock.send(self.url)
[07:54] parsu ohh sorry
[07:54] parsu import zmq
[07:55] parsu from threading import Thread
[07:55] parsu should I copy pase code here?
[07:56] parsu or send you somewhere?
[07:56] parsu this is the error printed before crash
[07:56] parsu Assertion failed: !pgm_supported () (zmq.cpp:239)
[07:58] sustrik aha, that's the problem with OpenPGM library
[07:58] sustrik it cannot be initialised twice
[07:59] sustrik thus, when compiling with --with-pgm, you cannot open two contexts
[07:59] sustrik this have been fixed in new version of OpenPGM but these were not yet merged into 0mq
[07:59] sustrik anyway, you shouldn't open two contexts
[07:59] sustrik why are you doing that?
[08:00] parsu actually this is sample code
[08:00] parsu I am calling some other libs
[08:00] parsu using ctypes
[08:00] parsu which uses zmq there
[08:00] parsu so we cant share the contexts
[08:01] parsu thats where we have stuck
[08:01] sustrik i see
[08:01] sustrik do you need PGM?
[08:01] parsu so if I ermove --with-pgm
[08:01] parsu it should work ?
[08:01] sustrik yes
[08:01] parsu nopes
[08:01] parsu cool
[08:01] parsu I will do that
[08:02] parsu Also can you tell me what is impact of no, of io threas we pass to zmq_init
[08:02] sustrik in most cases use 1
[08:02] parsu one of our eng. said he got crashes when he used more than 1
[08:02] parsu ohh ok
[08:02] sustrik more I/O threads may be useful if there's a lot of connections involved
[08:02] sustrik hundreds, thousands
[08:03] parsu ohk
[08:03] parsu great
[08:03] parsu I got my answers
[08:03] parsu thanks a lot
[08:03] sustrik parsu: if you get crashes, please do report it
[08:03] sustrik otherwise it won't get fixed
[08:03] parsu yups
[08:03] parsu sure
[08:03] parsu I will ask him to give me core
[08:03] parsu I will pass it to you
[08:03] sustrik thx
[08:03] parsu thanks a lot :)
[08:04] sustrik you are welcome
[08:46] parsu <sustrik> thus, when compiling with --with-pgm, you cannot open two contexts [13:28] <sustrik> this have been fixed in new version of OpenPGM but these were not yet merged into 0mq
[08:46] parsu Do we have any plans to merge in 0mq?
[09:02] sustrik yes, eventually
[09:04] parsu any estimated date?
[10:55] sustrik parsu: not really
[10:55] sustrik do you need it?
[11:17] parsu yaa
[11:28] sustrik parsu: try asking about the openpgm integration roadmap on the mailing list then
[11:33] parsu ohk
[11:33] parsu thanks a lot
[11:33] parsu for now this is ok, we can use tcp fanout instead
[11:33] parsu but ya we will need it later
[11:34] parsu thank bye
[11:58] jugg hmm, maybe I have erlang bindings working with zeromq finally...
[17:06] sd88g93 hello. ok, i simplified my problem down to a simple case program, and managed to replicate the same error i am getting. here is the code:
[17:06] sd88g93 basically, the subscriber recieves the first of the published messages from the publisher process, but none of the following ones .
[17:08] sd88g93 It is a pretty simple case, nothing fancy.
[17:35] sustrik sd88g93: are use using single socket from multiple threads?
[17:36] sd88g93 sustrik: yes, but this example uses just 1 thread.
[17:36] sd88g93 its a publisher socket, so its a global variable accessed by a mutex
[17:37] sustrik never use single socket from multiple threads
[17:37] sustrik but if it fails even when accessed from a single thread, then it's a bug
[17:37] sustrik i'll have a look
[17:37] sd88g93 even if its a publisher socket ?
[17:39] sd88g93 do you have any other suggestions as far as publishing message to one publisher socket, from multiple threads ? the ZMQ_FORWARDER device , or the QUEUE device doesnt seem to work for publish sockets
[17:43] sustrik ZMQ_FORWARDER should
[17:43] sd88g93 it doesnt seem to like multiple pipes conneting to one pub socket
[17:43] sustrik ?
[17:43] sustrik do you have the code?
[17:43] sd88g93 yes, in the pastebin
[17:44] sustrik i mean the one with forwarder device?
[17:44] sd88g93 not using the forwarder device, i'll have to alter that code to try to use the forwarder then
[17:44] sd88g93 i have it but its not a simple case
[17:44] sustrik just paste it
[17:45] sustrik if it doesn't work there's a bug somewhere
[17:45] sd88g93 yeah, i think the existing code uses the method in the pastebin right now, i'll alter the example and see if i can get it to work wit the forwarder device, and then paste that
[17:46] sd88g93 it'll just take awhile
[17:46] sd88g93 actually, that code in the pastebin uses 2 threads, the main thread opens the socket, and the thread publishes to the socket, i guess that counts as 2 then
[17:46] sd88g93 i mean, just one thread to do the publishing
[17:57] sustrik you cannot migrate sockets between threads
[17:57] sustrik anyway, try with forwarder device and paste it
[18:15] sd88g93 sustrik: here's the code with the forwarder device:
[18:15] sd88g93 it creates the first thread, but the other threads cannot bind to it. It continues to publish messages with that one thread, but the subscriber doesnt recive it.
[18:48] sd88g93 ok, also tried binding both ends in the main thread and then calling connect() in the worker threads, same result.
[21:13] sd88g93 i think i found a real bug in the forwarding device
[21:13] sd88g93 here's the code that replicates the issue:
[22:40] veritos In tcp_socket.cpp:213 of 2.0.7, ECONNRESET and ECONNREFUSED are handled, but not ETIMEDOUT. Timeouts thus cause the assertion failure a couple lines down. Doesn't this seem a bit harsh when just adding the test for ETIMEDOUT is so easy?
[22:45] veritos Oh, hey, it's fixed in git :)
[22:48] sd88g93 ok, this problem appears not to be confined to just the forwarder device, but publishing in general. when i publish a series of messages, only the first one gets through, the second one is all jumbled, i reproduced the code in simple form here:
[22:59] sd88g93 ohhhh ok, you have to re init the msg and close each time you send, i was calling zmq_msg_init_data once and just changing the variables