IRC Log

Thursday September 15, 2011

[Time] NameMessage
[06:24] khigia hi, newby here, let me know if my question is not for this channel;
[06:24] khigia in async example on zguide http://zguide.zeromq.org/page:all#toc54
[06:24] khigia zframe_send is used, then zframe_destroy is called
[06:25] khigia the czmq doc say that zframe_send destroy the frame
[06:25] khigia so, do we need to call zframe_destroy because of the flag ZFRAME_REUSE?
[06:25] khigia where can I find doc a ZFRAME_REUSE?
[06:40] MickeM is there a reason for all the "... just silently ignore the message." ?
[06:46] sustrik MickeM: what exactly do you mean?
[06:50] MickeM sustrik: for instance if a message subscriber was not found, why not return an error in the send() operation...
[06:50] sustrik that's the nature of pub/sub
[06:50] MickeM I was mainly curious if there was a reason for silently ignore... (I noticed in the srouce code it is a common practice)
[06:50] sustrik publisher doesn't know about subscribers
[06:51] MickeM but in this case they do right? (as you have an if checking so it would be possible to return an error state?
[06:51] MickeM but, ok... then it is the messaging pattern which drives the design...
[06:51] sustrik for example, with multicast transport, the publisher isn't even aware of subscribers
[06:52] sustrik the only component that knows about subscribers is the network switch then
[06:52] MickeM aha, oki...
[06:53] MickeM another question then, is there a way to tunnel more then one "connection" through the same "tcp connection" ?
[06:54] MickeM for instance to have req/rep going both ways through the same connection?
[06:55] khigia doesn't the simple req/rep use only one zmq socket? (so I'm assuming one tcp connection if socket is tcp)
[06:56] khigia I meant, both client and server use the same "tcp://host:port"
[06:56] MickeM khigia: yes.. but I want to have a dual req/rep one from s1 -> s2 and one from s2->s1 but it is simpler to have a single port through the firewall...
[06:57] khigia right, got your question, thanks
[06:58] khigia workaround would then be a router socket on both side
[06:58] khigia seems complex
[06:59] MickeM yeah... but thats (a lot) more code for me :) would be sweet if I could just create two connection from a "single connection" and have zeromq do all the work for me :P
[07:00] khigia agree; can device help reduce the code size? unless there is a way to reuse "connection"
[07:03] khigia for info, found ZFRAME_REUSE doc in zframe man page; sorry for noise
[07:29] sustrik MickeM: there's such feature in the development trunk
[07:29] sustrik but it's still bleeding edge
[08:08] CIA-121 libzmq: 03Martin Sustrik 07master * rf78d9b6 10/ (52 files): Session class separated into socket-type-specific sessions ...
[08:40] someone-noone hello! I'm trying to understand how ZMQ_FD and ZMQ_EVENTS works. I want to integrate libzmq to own event loop. What steps should i do?
[08:41] someone-noone that how i'm understanding it now:
[08:42] someone-noone for client side: create zmq_socket
[08:42] someone-noone 2) call zmq_getsockopt(ZMQ_FD) and then put new FD to my event loop for polling
[08:43] someone-noone 3) when i receive event on the new FD, than call zmq_getsockopt(ZMQ_EVENTS) and read\write data from to that FD?
[08:44] guido_g not to the fd, to the socket
[08:46] someone-noone guido_g: thanks, and how should i call connect?
[08:46] guido_g what?
[08:47] someone-noone i mean for which FD..
[08:47] someone-noone should i call zmq_connect or connect manually...
[08:48] guido_g you don't use the fd, as documented
[08:48] guido_g connect the socket
[08:48] guido_g or bind
[08:48] someone-noone sorry, but i don't understand completely
[08:50] someone-noone create zmq_socket, call zmq_getsockopt(ZMQ_FD), call zmq_connect? or... ?
[08:55] sustrik ZMQ_FD is only for polling
[08:55] sustrik never use it for anything else
[08:56] sustrik the pseudocode for integrating to your event loop is:
[08:56] sustrik fd = getsockopt (s, ZMQ_FD);
[08:57] sustrik ... poll for POLLIN on the fd inside your event loop ...
[08:57] sustrik ... once POLLIN is signaled:
[08:57] sustrik events = getsockopt (s, ZMQ_EVENTS);
[08:57] sustrik if (events & ZMQ_POLLIN) ...
[08:58] sustrik if (events & ZMQ_POLLOUT) ...
[08:58] sustrik etc.
[08:58] someone-noone sustrik: thanks, that is clear for me
[08:58] someone-noone but not clear where should i call connect\bind ?
[08:59] sustrik where it's appropriate :)
[08:59] sustrik in most cases people call connect/bind first
[08:59] sustrik then enter the event loop
[08:59] someone-noone as normal for libzmq? zmq_connect\zmq_bind. Or should i do it manually ?
[08:59] sustrik nothing prevents you binding/connecting while in the event loop though
[08:59] sustrik as normal
[08:59] someone-noone okay, now it seems to be clear for me
[09:00] someone-noone now i should to check how to do it for all polling interfaces implemented in my framework..
[10:06] MickeM sustrik: oki, any sort of ETA when it might be out? I am in no rush so I guess I can wait for it to become stable...
[10:06] someone-noone sustrik: should i call zmq_init with 0 parameter?
[10:06] someone-noone double attack :)
[10:06] sustrik :)
[10:06] sustrik MickeM: depends on how much will people test it and report problems
[10:07] sustrik anyway, you can go forward even without it
[10:07] MickeM sustrik: ok...
[10:07] sustrik once it's there, you just change the connection string from tcp to vtcp
[10:07] sustrik someone-noone: you can
[10:08] sustrik however, that means you can use only inproc transport
[10:08] sustrik tcp and alike need a worker thread to handle traffic asynchronously
[10:08] someone-noone sustrik: does it mean i can't use libzmq without spawning additional threads? (for tcp)
[10:09] MickeM sustrik: ahhh, that pretty sweet... then I wont bother to develop anything like it myself...
[10:09] sustrik no you can't
[10:10] someone-noone it's bad
[10:10] sustrik however, the additional threads are invisible to the user
[10:10] sustrik why do you care?
[10:10] sustrik no async system can possibly work without a worker thread
[10:11] someone-noone i know
[10:11] sustrik what's the problem then?
[10:11] sustrik no pthreads on your platform?
[10:12] someone-noone but i want to handle events by myself only in 1 thread. Now i will use 2 threads. In general there is no problem here... but additional thread for task that should be done in 1
[10:13] someone-noone the only platform that is not supporting pthreads i know is windows :D
[10:19] sustrik we have special #ifdefs for that :)
[10:19] sustrik anyway, why can't you handle events in 1 thread?
[10:20] sustrik the thread inside libzmq is completely invisible for you
[10:20] someone-noone because i spawn 1 more in zmq_init
[10:20] someone-noone i know
[10:20] sustrik are you awere that you are spawning kernel threads when opening TCP sockets?
[10:21] sustrik the libzmq thread is pretty similar to those
[10:22] someone-noone i understand, that in general there is no problem. 2 threads can live and don't know anything about each other. But the only thing i don't like is that there will be 2 threads where it can be only 1 thread (that doesn't mean that it's bad in general. But i'm little bit aware)
[10:23] sustrik i see
[10:23] sustrik the point is that if you want to have queueing going on in background, you need an additional thread
[10:23] sustrik if you don't care about queueing you can as well implement the protocol on top of a simple TCP socket
[10:24] sustrik in that case there are still additional threads, but they are in kernel space, so you are not aware of them
[10:26] someone-noone my goal is to make very simple (in logic meaning) appliaction. Crtmpserver handles video data and doesn't handle logic at all. That was done for handling video afap that it will not delay on client side for some reason (blocking).
[10:27] someone-noone All blocking and computing operations should be done at another server
[10:27] someone-noone i want to control that crtmpserver with 0mq
[10:27] sustrik then you can simply implement the 0mq protocol and ignore libzmq
[10:27] someone-noone that it could send and receives jobs (messages, doesn't metter)
[10:28] sustrik however, be aware that you can never avoid blocking
[10:28] someone-noone why?
[10:29] sustrik for example, if the network is slow, send can block because tcp tx buffer is full
[10:29] someone-noone that means that instances should be scaled (in my case)
[10:29] someone-noone instance*
[10:29] someone-noone so it's not a problem
[10:29] sustrik how would that help if it's the network that is slow?
[10:30] sustrik say a network congestion happens
[10:31] someone-noone then my company will ring to vps-provide with "wtf?" question :_
[10:31] sustrik :)
[10:33] someone-noone so one more time. What do you recommend for me? Using libzmq or implementing protocol. If second (as you saw yesterday i had completed send_msg functionality) how much should i implement for full "client"?
[10:35] someone-noone my things is that having one more thread is not so bad
[10:35] someone-noone are *
[10:36] sustrik depends on what functionality you need
[10:36] sustrik 0mq as is took some 10 manyears to get into shape
[10:37] sustrik you would probably need just a subset of that
[10:37] sustrik but it can still take rather long
[10:38] sustrik multiple the time required to implement it by your salary and ask whether having 1 thread less is worth of it
[10:38] sustrik that's about it
[10:40] someone-noone agreed with you :)
[10:45] someone-noone one more awareness i have. If i use ZM_EVENTS at that time where i will call socket.recv(&reply) will it be 100% enough data for receiving that reply? or it may block?
[10:49] sustrik it means that there's at least one message to receive
[10:50] someone-noone but message can be empty
[10:50] someone-noone and without reply
[10:50] someone-noone so socket.recv will read empty message and will wait for message
[10:50] someone-noone right?
[10:50] sustrik ?
[10:50] sustrik empty message is a message
[10:50] sustrik what's the difference?
[10:50] someone-noone rep\req is 2 msg
[10:51] someone-noone not 1
[10:51] sustrik ah, it's 1 composed of 2 parts
[10:51] sustrik messages are atomic
[10:51] sustrik so you either get all parts
[10:51] sustrik or none
[10:51] someone-noone that is good news :)
[10:51] sustrik if you get first part
[10:52] sustrik you should be able to recv remaining parts without blocking
[10:52] someone-noone nice design..
[11:19] someone-noone sustrik: from what version ZMQ_FD is avaliable? currently i don't see this constant
[11:22] sustrik 2.1 iirc
[11:22] someone-noone yeah
[11:22] someone-noone thanks
[11:25] someone-noone and is it possible to know version of zmq from cmake script?
[11:29] sustrik the autotools script parses zmq.h file to get the version
[11:30] sustrik https://raw.github.com/zeromq/libzmq/908b39bf8309044913a34c632cd432958dcf6960/version.s
[11:31] sustrik sorry
[11:31] sustrik https://raw.github.com/zeromq/libzmq/908b39bf8309044913a34c632cd432958dcf6960/version.sh
[11:31] someone-noone thx
[12:19] someone-noone probably this channel is not related to such questions, but maybe i'm misunderstanding some fundamental things of libzmq. I've downloaded and installed latest stable sources from git. Now i'm trying to link with libzmq and getting following:
[12:20] someone-noone probably this channel is not related to such questions, but maybe i'm misunderstanding some fundamental things of libzmq. I've downloaded and installed latest stable sources from git. Now i'm trying to link with libzmq and getting following:
[12:21] someone-noone error while loading shared libraries: libzmq.so.1: cannot open shared object file: No such file or directory
[12:21] someone-noone i've linked with -lzmq option
[12:22] someone-noone and i don't have such file (libzmq.so.1) why it's trying to link with it, not just to libzmq.so ?
[12:26] sustrik libzmq.so is a symbolic link pointing to libzmq.so.1
[12:27] sustrik it's a system that allows for having multiple versions of the same library installed at the same time
[12:27] sustrik the problem you are seing is presumably that the lib is installed in /usr/local/lib
[12:27] sustrik while your path doesn't point there
[12:28] someone-noone okay... so i will reinstall it to /usr/lib
[12:28] someone-noone will it help?
[12:28] sustrik depends on your paths
[12:32] someone-noone that solved the problem
[12:32] someone-noone thanks
[12:39] calvin I updated clrzmq2, or rather i created a version of it, that is compatible with zmq3.
[12:39] calvin 3.0*
[12:42] calvin Is there a way I can contribute this to the community through git?
[12:44] calvin I'm not sure if clrzmq's version numbers are based on the underlying zmq version numbers, but if so i think this is more like clrzmq3 since it isn't exactly backwards compatible with zmq2 (and I thin kthat's for the best since zmq2.x isn't compatible with zmq3.0)
[12:44] sustrik calvin_: I recall the maintainer of clrzmq2 is micheal compton
[12:45] calvin is it best just to e-mail him?
[12:45] sustrik so i would suggest either sending a pull request
[12:45] sustrik or sending him an emial
[12:45] sustrik email
[12:46] sustrik in any case feel free to write to the mailing list a post a link to your project
[12:47] sustrik there have already been people interested in C# + 0MQ/3.0
[12:49] calvin ok, is there anywhere you recommend uploading it to? I've never contributed to open source before (i've worked exclusively on private sector projects in the past)
[12:57] sustrik calvin_: welcome then! :)
[12:57] sustrik you can place it on github for example
[12:58] sustrik www.github.com
[12:58] sustrik are you familiar with git?
[12:59] calvin thanks! i've only used subversion
[13:00] calvin i just signed up on github
[13:00] calvin i guess i could just fork clrzmq2 and then upload my code?
[13:01] sustrik calvin_: yes
[13:01] sustrik moreover it'll allow you to send pull request to clrzmq2 maintainer
[13:08] calvin ok sounds good thanks
[14:15] someone-noone sustrik: i continue playing with ZMQ_FD, ZMQ_EVENTS and got next problem:
[14:16] someone-noone i'm receiving events in my event_loop, but i can't read data, only write
[14:16] someone-noone here's a code of what i'm doing : http://pastebin.com/hawdtS2h
[14:19] someone-noone did i miss smth?
[14:19] someone-noone i receive both ZMQ_POLLIN and ZMQ_POLLOUT msg, but i don't see any received data (only emtpy messages)
[14:20] someone-noone and i see result on server side
[14:21] shales instead of calling zmq_recv once when you see ZMQ_POLLOUT, try calling it in a loop until you get EAGAIN
[14:22] shales pass ZMQ_NOBLOCK as the flag
[14:22] guido_g if (flags | ZMQ_POLLIN) <- what should this be?
[14:23] someone-noone shales: thx, i will try this and report
[14:23] someone-noone guido_g: this mean i have ZMQ_POLLING flag set up
[14:23] guido_g no it doesn't
[14:23] guido_g it's or but schould be and
[14:25] someone-noone guido_g: you're right... i'm blind
[14:25] someone-noone and stupid :_
[14:28] someone-noone so now (after changing or to and) i'm receivng only ZMQ_POLLOUT event
[14:28] someone-noone and succesfully sending it to server side
[14:28] someone-noone but of course no reply
[14:29] someone-noone shales: i
[14:29] someone-noone shales: i've read in specification, that ZMQ_POLL* events guarantees you that you will read message without blocking
[14:30] someone-noone so i don't think that i should to change socket flags to ZMQ_NOBLOCK
[14:31] shales you'll definitely be able to recv one message without blocking. However if there are 2 msgs waiting, and you read only one, then I don't think poll will necessarily signal you that the FD is readable again.
[14:32] shales that's why I suggested calling recv in a loop to ensure you've got all the messages.
[14:33] someone-noone shales: ok,sounds you're right. But currently, i don't rcv even 1 message... even ZMQ_POLLIN event at all
[14:33] someone-noone so currently problem is not here
[14:34] shales ya
[14:35] shales do you get the reply if you put the block recv right after the send in the client (ignoring the FD and EVENTS entirely for the moment)
[14:35] shales ?
[14:37] someone-noone shales: :)
[14:37] someone-noone yes i do
[14:37] someone-noone so i'mnot checking flags correctly
[14:37] someone-noone or?
[14:38] guido_g you should check the return values of send and receive
[14:39] someone-noone guido_g i'm doing it by know
[14:39] someone-noone i receive 0
[14:39] someone-noone in both cases
[14:39] sustrik also be aware that ZMQ_FD is edge-triggered, bo level-triggered
[14:39] someone-noone sustrik: what does it mean>?
[14:39] sustrik meaning that ZMQ_FD signals only when something changes
[14:40] sustrik if there were messages queued and they are still queued -> no signal
[14:40] sustrik if you read the last message
[14:40] sustrik the check ZMQ_FD
[14:41] sustrik it signals when the first message arrives
[14:41] sustrik if you don't read it
[14:41] sustrik and poll on ZMQ_FD again
[14:41] sustrik => no signal
[14:41] sustrik etc.
[14:43] shales someone-noone: are you calling send, recv or getsockopts(ZMQ_EVENTS) for that socket anywhere else other than in that OnEvent method?
[14:43] shales on the client side, I mean
[14:44] someone-noone sutrik: okay, it's clear and it's what was said by shales. But as you may see, i'm entering OnEvent func first to send data. And i think i should receive new signal for reply.
[14:44] someone-noone shales: no, i don't do this in other methods
[14:44] someone-noone only there
[14:47] sustrik maybe the message was already queued when you've invoked the poll?
[14:47] calvin thanks for everything sustrik, i just pushed my changes and sent in a pull request
[14:47] sustrik try checking ZMQ_EVENTS first
[14:47] sustrik calvin_: where's the project?
[14:47] someone-noone sustrik: ok
[14:48] calvin this is my fork: https://github.com/calvindevries/clrzmq2
[14:48] sustrik thanks!
[14:51] someone-noone sustrik: you were right. Here is a working code http://pastebin.com/1MnaX0fn
[14:51] someone-noone i don't want to change socket type to NON_BLOCK. How do you think will it never block?
[14:51] sustrik great
[14:52] sustrik if you poll for POLLIN first, you are guaranteed to be able to read at least one message without blocking
[14:52] sustrik similar with POLLOUT and send()
[14:53] someone-noone what does "poll" mean in your context?
[14:54] sustrik poll(2)
[14:54] sustrik epoll(2)
[14:54] sustrik select(2)
[14:54] someone-noone clear
[14:55] someone-noone just to be sure
[14:55] sustrik wait a sec
[14:55] sustrik you do the above
[14:55] sustrik then get the ZMQ_EVENTS
[14:55] sustrik if events contain ZMQ_POLLIN you are guaranteed to be able to read at least one message without blocking
[14:56] sustrik similar with ZMQ_POLLOUT and send()
[14:58] someone-noone but when i call ZMQ_EVENTS again (after handling received data) will it know that "i've just read\write some data and flags should be recalced". Or i must wait next event?
[14:58] sustrik ZMQ_EVENTS always returns real state
[14:58] sustrik whenever you call it
[14:59] sustrik so, if POLLIN is set you can read a message etc.
[14:59] sustrik ZMQ_FD is to be used only if there's nothing to process and you want to wait without resorting to a busy loop
[15:00] someone-noone so it's safe to check ZMQ_EVENTS in a loop (like in a last pb)
[15:00] sustrik i guess i should write a ZMQ_FD tutotial
[15:00] someone-noone :)
[15:00] sustrik yes, it's safe
[15:00] sustrik however, if there are no messages, you'll end up with a busy lopp
[15:00] sustrik meaning 100% CPU load
[15:00] guido_g works great, just did it w/ the java bindings on zmq 2.1.8
[15:01] someone-noone you mean because of ZMQ_POLLOUT ?
[15:03] someone-noone it's just a "proof of concept". In real implementation it will put all out msgs into queue
[15:03] sustrik i mean, if you just check ZMQ_EVENTS
[15:04] sustrik in loop
[15:04] sustrik without polling on ZMQ_FD
[15:04] sustrik the loop will just just loop on and on if there's nothing to read
[15:05] sustrik while (true)
[15:05] sustrik {
[15:05] someone-noone i understood
[15:05] sustrik ok
[15:05] someone-noone of course, i will not do infinite loop.
[16:52] Steve-o So, any interesting 0mq projects in NYC?
[17:35] calvin in 3.0, when XSUB sends its subscriptiong message to XPUB it is just the value 1 + whatever the filter would be?
[17:35] calvin and then XPUB filters messages?
[17:44] sustrik yes
[17:45] calvin so when i send messages from the XPUB, how does it know if my message complies with the filter?
[17:45] calvin should i be sending the filter as a separate envelope? then the rest of my message?
[17:46] calvin (as in zmq2.0)
[17:48] sustrik it's exactly the same as in 2.x
[17:48] sustrik no change whatsoever
[17:49] calvin would the subscriber still get the entire message?
[17:49] calvin or would the xpub pull off that first envelope as if it were a dealer
[17:50] sustrik it would get message exactly as it was sent
[17:50] calvin ok cool, thanks
[17:50] sustrik no change from 2.x behaviour
[17:50] calvin i was just confused from one of the white papers where it was proposing multiple layers of xpub -> xsub, xpub -> sub
[17:51] calvin and wasn't sure if the xpub / xsub were pulling off the first envelope each time as a "routing tag"
[17:51] calvin as if they were dealers
[17:52] sustrik no, it's not that way
[17:52] calvin ok thanks a lot
[17:52] sustrik even if you have multiple intermediate nodes
[17:52] sustrik the message passes the topology unchanged
[17:52] calvin ok great, that's good to konw
[17:52] calvin know*
[19:45] nicolas is there a way to use zmq_poll() with resolution lower than 1 msec ?
[19:45] nicolas when it blocks undefinetely
[19:50] cremes nicolas: libzmq 2.1.x has microsecond resolution, but as of 3.x that was changed to milliseconds only
[19:57] mikko nicolas: why do you need to have less than msec?
[19:58] nicolas just for test purposes
[19:58] nicolas and curiosity
[20:12] Steve-o If I recall none of the event APIs offer resolution finer grain than 1ms
[20:15] sustrik nicolas: Steve-o is right
[20:16] sustrik the microsecond precision would be a fake
[20:16] sustrik as undelying system offers just millisecond resolution anyway in most cases
[20:19] Steve-o Anyone attending the HPC conf. this Monday in NYC?
[20:20] mikko Steve-o: you talking?
[20:21] Steve-o A TIBCO guy just pointed it out and looks like I can visit
[20:24] Steve-o I've never given a talk outside of TIBCO, maybe someday ... :-)
[20:34] sustrik you've used to work for tibco?
[20:45] Steve-o yup
[20:46] sustrik do you know derek collison by chance?
[20:47] Steve-o not familiar
[20:47] sustrik he was one of the guys who wrote the original tibco system iirc
[20:47] sustrik never mind
[20:47] Steve-o Tekmon then? That's before me
[20:47] Steve-o Teknekron
[20:48] sustrik must have been sometimes back then
[20:48] sustrik not sure
[20:48] sustrik he's working for vmware now, really smart guy
[20:49] Steve-o They're walking distance apart in Palo Alto
[20:49] sustrik yep, been there
[20:49] sustrik accidentally got to tibco instead of vmware :)
[20:56] Steve-o so, looking around NYC for a job
[21:02] guido_g ah ok, danke
[21:02] guido_g argh... sorry, wrong window
[21:13] guido_g ahhh nu hab ich das glaube ich
[21:14] guido_g again wrong window
[21:14] Steve-o :P
[21:14] guido_g focus follows mouse...
[21:32] Steve-o waffle about low latency NICs: http://www.scs.stanford.edu/~rumble/papers/latency_hotos11.pdf
[23:57] rb What is the best way to implement a 'bus' kind of topoloty using 0mq so that client and server become one ?

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy