IRC Log


Thursday June 3, 2010

[Time] NameMessage
[07:24] sbeaulois hello
[07:27] versificateur hello everyone!
[07:28] versificateur I have some difficulties to reach messges sent by zeromq
[07:28] versificateur when I connect to hosts other than localhost
[07:29] versificateur the connection is opened but i get no message from the server
[07:30] versificateur however it's works fine in local
[07:30] guido_g show source in paste-bin
[07:31] guido_g all other things are guessing :)
[07:34] versificateur http://pastebin.com/5pWd2mFc
[07:34] versificateur this is the server
[07:36] versificateur http://pastebin.com/NvCZ3157
[07:36] versificateur and the test i wrote based on pyzmq example
[07:37] guido_g ok
[07:38] guido_g can you check if the server is listening on all ips of the server?
[07:38] guido_g on linux netstat would do the job
[07:38] guido_g netstat -tanp
[07:38] versificateur yes it's listening
[07:38] sustrik i think the problem is that you exit the sender before it's able to send the message
[07:39] sustrik try sleeping for a while before you exit the sender
[07:41] versificateur but the same code works well when zeromq subscriber and publisher are on the same machine
[07:41] guido_g yes, because there is no network involved, completely different behaviour
[07:41] sustrik i assume that's because loopback is faster than inter-machine communication
[07:42] sustrik so it local case the message manages to get through before the application quits
[07:43] versificateur okay i'll try sleeping
[07:43] versificateur after sendin??
[07:43] guido_g yes
[07:44] guido_g just to make sure that ømq has time to send
[07:44] versificateur i'll start with huge value and decrease gradually
[07:45] sustrik one second should do
[07:50] versificateur i tried with 5 second but received nothing
[07:52] sustrik another possibility:
[07:52] sustrik do you start the subscriber fast enough?
[07:52] sustrik i see there's a sleep in the code to allow subscriber to connect
[07:53] sustrik do you start it within 1 second after starting the publisher?
[07:54] sustrik versificateur: let me explain how pub/sub is supposed to work
[07:55] sustrik the idea is that publisher is a long-lived application that broadcast continuous stream of messages
[07:55] sustrik the subscribers connect to the publisher and start receiving the stream at some point
[07:56] versificateur it's what publisher do
[07:56] versificateur my publisher do
[07:57] sustrik does it run in loop?
[07:57] sustrik my impression is that it publishes a message and quits
[07:58] sustrik rather than transmitting a continuous stream
[07:58] versificateur not really . it open a connection (bind)
[07:58] versificateur and send every message
[07:59] versificateur he receive from an upstream application
[08:00] sustrik where should i look for the loop?
[08:01] sustrik it's clear up to line 34
[08:01] sustrik then i get lost
[08:02] versificateur http://pastebin.com/pdNkax6c
[08:02] versificateur okay i'll resume basically what the application do
[08:03] versificateur it receive a GET request through a WSGI python server
[08:03] guido_g ahhh it's running in web-server
[08:04] versificateur yes it's the last code i send
[08:05] versificateur but before running the WSGI server it start the zeromq publisher
[08:06] versificateur with import bmpMessageHandler on line 4
[08:07] sustrik ok, so it does publish a continuous stream of messages
[08:08] versificateur and everytime a GET request is send it run the method that send message
[08:08] sustrik can you check whether there's data over the wire (tcpdump/wireshark)
[08:08] sustrik ?
[08:08] versificateur i'll try
[08:09] versificateur on the publisher or subscriber?
[08:09] sustrik should be the same
[08:09] sustrik it's up to you
[09:35] mikko good morning
[09:35] mikko man it's hard to get back to work after hols
[09:36] mikko sustrik: are there any upcoming changes to the polling api?
[09:36] sustrik hi
[09:36] sustrik you mean the waitfd thing?
[09:36] mikko i mean the pollset changes
[09:36] sustrik ah, no
[09:36] mikko adding / removing items to/from pollset
[09:37] mikko ok
[09:37] sustrik it'll stay as is in 2.0.7
[09:37] mikko so making a pollset myself is a sensible thing to do
[09:37] sustrik yes
[09:37] mikko cool, because i wrote one some time ago
[09:37] mikko just need to polish it and push in
[09:37] sustrik great
[09:37] mikko thanks
[09:38] mikko a lot of improvements in 2.0.7
[09:41] versificateur i finally locate the bug
[09:42] versificateur it's in the deamon i run to start the WSGI server
[09:42] versificateur i'l try to fix it
[09:42] versificateur thanks sustrik
[09:43] sustrik you are welcome
[09:43] sustrik versificateur: is it an open source project you are working on?
[09:46] versificateur not yet but it could become
[09:47] sustrik if so, it would make sense to link it from zeromq.org site
[09:48] sustrik webserver<->application server communication is a thing many people would love to use
[09:54] versificateur of course... and i think it has a lot of sens in a JSON-like oriented data consumption world
[10:05] sustrik ack
[10:06] sustrik so if you go open source just ping me and i'll link to your project
[11:31] umesh I wan to send multicast messages using zeromq
[11:32] umesh I wrote the sample code for send and recv
[11:32] umesh namely mc_send and mc_recv
[11:32] umesh first I run mc_recv and then mc_send ...
[11:32] umesh but message is sent properly from mc_send ... but not received in mc_recv
[11:33] umesh then I add the delay of 15 seconds in mc_send between zmq_connect and zmq_send api
[11:33] umesh then mc_recv the message correctly ...
[11:34] umesh can anyone tell me why this delay of x seconds is required
[11:34] guido_g the message is put into the ømq io-thread, which runs asynchronoulsly to the main thread
[11:34] sustrik yea, but the receiver is run first
[11:35] guido_g so if the main thread exits, the io-thread is terminated and therefore nothing is sent
[11:35] sustrik ah
[11:35] sustrik umesh: that may be the problem
[11:36] umesh guido_g : ok got ...
[11:36] sustrik the 15 sec slep before send may not be required
[11:36] guido_g it's sort of *the* standard problem
[11:36] sustrik try sleeping after the send
[11:36] umesh then is there any way to avoid the sleep ...
[11:36] sustrik is it in faq?
[11:36] guido_g don't know
[11:36] sustrik umesh: sending in loop
[11:37] umesh means how i can do io-thread synchronously ...
[11:37] sustrik 0mq is async
[11:37] sustrik you can try using openpgm directly
[11:37] umesh i want to do it using zeromq ..
[11:38] sustrik sorry, it's inherently async (that's what MQ means = message queueing)
[11:39] guido_g sustrik: nope, it's not in the faq
[11:39] sustrik guido_g: no mention of it in FAQ
[11:39] sustrik :)
[11:39] guido_g :)
[11:39] umesh one more question : does zeromq send the more than one packter for multicast ?
[11:39] sustrik packet?
[11:40] sustrik guido_g: let's think of a faq question...
[11:40] umesh I am asking this because : when I have no sleep and no messages received ... I still can see some multicast messages in wireshark ...
[11:40] sustrik those are SPM's right?
[11:40] sustrik (check the details in wireshark)
[11:41] umesh SPM means ?
[11:41] sustrik type of PGM packet
[11:42] guido_g the protocol used to control the message flow
[11:42] guido_g like the special packtes used by tcp
[11:42] sustrik yesh, there are packets like SPM/NACK/NCF/ODATA/RDATA...
[11:43] umesh ok
[11:43] umesh how we can decide these are SPM packtes ? because wireshark does not tell anything related to SPM ...
[11:44] sustrik guido_g: what avout thisL
[11:44] sustrik "I send a message to PUB socket, then I quit. The message does not appear at the connected SUB socket. What's wrong?"
[11:44] umesh like it does for TCP ACK/FIN etc ...
[11:44] sustrik it should
[11:44] sustrik PGM dissector is part of wireshark out-of-the-box
[11:44] sustrik what does it say?
[11:44] guido_g sustrik: ok, maybe adding "using pgm/epgm"
[11:45] sustrik umesh: ah, you are using epgm?
[11:45] umesh yup
[11:45] sustrik guido_g: it can happen with tcp as well
[11:45] sustrik umesh: ah, that's encapsulated PGM, wireshark can dissect that
[11:45] sustrik try using pgm
[11:45] guido_g oh well, yes
[11:45] sustrik (you'll have to run your apps as root)
[11:46] guido_g i constantly forget that pub/sun is possible via tcp as well :)
[11:46] umesh ok
[11:46] sustrik what should the answer be?
[11:46] umesh but does pgm or epgm has anything to do to related to sleep issue ?
[11:46] guido_g no
[11:46] sustrik no, it's the same thing
[11:46] umesh or it will behave same for both pgm and epgm ?
[11:46] sustrik almost
[11:46] umesh ok
[11:46] guido_g sustrik: what are differences (for ømq usage)?
[11:47] sustrik epgm doesn't have to be run as root
[11:47] sustrik it's not a standard though
[11:47] sustrik pgm has to be run as root but it's a standard (RFC3208)
[11:48] sustrik the consequence, for example, is that wireshark can parse pgm but not epgm
[11:48] guido_g or with the capabilities set, if you're not root :)
[11:48] sustrik right
[11:49] umesh (process:27705): Pgm-WARNING **: PGM protocol requires CAP_NET_RAW capability, e.g. sudo execcap 'cap_net_raw=ep'
[11:49] umesh this error i am getting when I use pgm instead of epm ...
[11:49] sustrik "yes
[11:49] sustrik are you running it as root?
[11:49] umesh i are run as root ... then no error
[11:50] sustrik the whole point is that PGM is defined to be stacked on top of IP directly
[11:50] sustrik same way as TCP or UDP is
[11:50] sustrik so the user-space implementation has to use raw IP sockets to implement it
[11:51] umesh ok .. now its showing spm in wire shark
[11:51] sustrik but these can be used only be root
[11:51] sustrik ok
[11:51] sustrik spm is like a heartbeat
[11:51] sustrik more or less
[11:51] sustrik so what happens is that you start the publisher
[11:51] sustrik and sleep for 15 secs
[11:51] sustrik during those 15 secs, publisher is sending SPMs once in a while
[11:52] sustrik everything works as expected
[11:52] sustrik then you send a message
[11:52] sustrik but before it gets to the wire you shut down the publisher
[11:52] sustrik that's it
[11:54] umesh also this 15s is not predictable time
[11:54] umesh some time it can be more or sometime it can be less ...
[11:54] sustrik it's your code, no?
[11:54] umesh so what can be the ideal solution to avoid 15 s or sleep
[11:54] sustrik sleep (15);
[11:54] sustrik drop the 15 sec sleep, it's not needed
[11:55] sustrik what you need to do is to sleep _after_ you send the message
[11:55] sustrik so the publisher has some time to push the message to the wire
[11:55] umesh for how much time ?
[11:55] umesh ok
[11:55] sustrik 1 ms
[11:55] sustrik or something
[11:55] umesh got so even 1 ms will also work ... right eor even less ...
[11:55] umesh eor* = or
[11:55] umesh ok
[11:55] umesh got
[11:55] sustrik the latency is in orders of microseconds
[11:56] sustrik so millisecond should be more than enough
[11:58] umesh even i micro seconds is also working now ...
[11:59] umesh also more question ... are REQ/REPLY type of sockets in 0mq syncronous ?
[11:59] umesh because I haven't seen this issue of sleep for that ...
[12:00] sustrik they are not, but you cannot experience the problem simply because you wait for reply
[12:00] sustrik so you cannot quit before the request was sent
[12:10] umesh ok
[12:23] lvh What's the recommended thing for debian stable packages?
[12:27] mato sustrik: ping
[12:27] lvh I'll compile it if I have to
[12:27] mato sustrik: will the following do for an update of zmq_tcp(7) for 2.0.7?
[12:27] mato The flags field consists of a single octet containing various control flags:
[12:27] mato Bit 0: To Be Continued (TBC). A TBC value of 0 indicates that there are no more message parts to follow; or that the message being sent is not a multi-part message. A TBC value of 1 indicates that the message being sent is a multi-part message and more message parts are to follow.
[12:27] mato Bits 1-7: Reserved. Bits 1-7 are reserved for future expansion and MUST be set to zero.
[12:27] mato or do you want to actually explain it in more detail?
[12:28] mato oh, and of course we can call it MORE if that's the canonical name, I just liked "TBC"
[12:37] mato sustrik: are you there?
[12:50] versificateur hi i've a problem again
[12:51] versificateur http://pastebin.com/QZJ672jT
[12:53] versificateur i've this error -> Assertion failed: !inpipe && !outpipe (pair.cpp:47)
[12:55] versificateur following the muttithread server sustrik wrote i tried to pass a SUBCRIBER message to a thread of P2P
[12:55] versificateur but it seems that inproc doesn't accept P2P
[12:55] versificateur if anyone has an idea ...???
[15:23] versificateur is there anyone tohelp me?
[15:42] cremes versificateur: i'm not as fluent in C as i used to be, so keep that in mind...
[15:42] cremes lines 71 and 77 look wrong to me
[15:42] cremes i don't think you can use P2P/PAIR sockets with the queue device
[15:42] cremes i think you need to make your clients use REQ and your workers use REP sockets
[19:22] sustrik mato: the text is OK IMO
[19:23] sustrik MORE is used now, TBC sounded too much like a contagious disease
[20:13] sustrik mato: would it be possible to remove pgm examples from the build system?
[22:15] CIA-17 rbzmq: 03Brian Buchanan 07master * r72d81eb 10/ rbzmq.cpp : add ZMQ.select() wrapper for zmq_poll() function (+6 more commits...) - http://bit.ly/chNoHH