IRC Log

Friday September 2, 2011

[Time] NameMessage
[05:16] Dude909 Is IPV6 supported by ZMQ?
[06:09] sustrik Dude909: there's support in the trunk
[06:09] sustrik not in released versions though
[06:10] Dude909 ok thanks
[06:10] sustrik are you interested in that?
[06:10] sustrik it would benefit from some testing
[06:10] sustrik as it's a new feature
[06:10] Dude909 yes I am interested.
[06:11] sustrik great, if there are any problems let us know, we'll try to fix it
[06:12] sustrik one thing we know is creaky is resolution of hostnames and interface names
[06:12] sustrik so maybe start with using literal addresses
[06:14] Dude909 I will only use adresses anyway. By trunk, do you mean the libzmq repository?
[06:17] count http://zguide.zeromq.org/page:all#Getting-the-Message-Out <-- Is there a way for 0mq to queue messages until subscribers take them off theq ueue, rather than dropping them if there are no subscribers?
[06:18] sustrik Dude909: yes
[06:18] count I'm trying to build a job queue, with multiple workers - I can't afford for a job to actually just get lost if the workers aren't running
[06:18] count Or am Doing It Wrong?
[06:18] sustrik Dude909: to make IPv6 work, you have to set ZMQ_IPV4ONLY socket option to 0
[06:19] sustrik count: shouldn't you use push/pull instead?
[06:20] Dude909 ok thanks
[06:21] count sustrik: haha, ok, that's the next chapter I guess - I missed it's intent with 'divide and conqueor'
[06:22] count sustrik: that looks like it, thanks
[06:22] sustrik np
[06:48] Alec I'm trying to compile my first example in C#, but the code makes reference to a ZMessage class. Does anybody know where can I find it. Thanks
[06:51] sustrik clrzmq binding?
[06:52] Alec There is no ZMsg namespace defined
[06:52] Alec I'm assuming that ZMessage resides in ZMsg
[07:00] sustrik no idea, sorry
[07:00] count sustrik: push/pull works great, and man, that's freaking easy to use! thanks again
[07:00] sustrik Alec_: try asking on the mailing list
[07:00] sustrik count :)
[07:01] Alec thanks
[07:19] Alec sustrik: found it thanks
[08:30] ohmy hi all
[08:53] pieterh hi guys
[10:18] ohmy i have just started reading zeromq documentation, i'd like to have your opinion in the meanwhile if some of you would like to share their opinions
[10:20] pieterh ohmy: hi
[10:20] pieterh sustrik: https://zeromq.jira.com/browse/LIBZMQ-217 is done, no?
[10:20] ohmy i'm workin on an embedded platform (not yet released but planned to be gpl) for an infotainement system including media players, navigation engine web browser and so on. I've been using d-bus after corba since months ago and i have many performances issues on some low end devices. I know that we can not compare apples with oranges but the general idea that i've got today is that zeomq is more designed for computers then embedded devices
[10:20] ohmy hi pieterh
[10:22] ohmy The middleware approach i've choosen is that middleware exposes a set of API to D-Bus and user interface (application layer) calls these API to update the display, the approach as simple unless like i say i released that i have many performances issues
[10:22] sustrik let me see
[10:22] guido_g ohmy: if you use dbus on the device, ømq is no problem at all
[10:22] sustrik yup, i've documented it iirc
[10:22] sustrik maybe check the grammar
[10:23] guido_g ohmy: there arm and ios builds out there, so even small "devices" are no problem
[10:23] pieterh i'm doing some issue cleanup, closing issues that are incomplete or won't fix
[10:25] ohmy guido_g: thanks, i'll come back after reading all the documentation, and after playing with some samples, hopefully i can reach the performances i need with zeomq :)
[10:26] guido_g ohmy: using dbus is asking for performance trouble
[10:26] sustrik pieterh: great
[10:27] ohmy guido_g: i passed many white nights trying to shortcut some features and i completely agree with you
[10:29] guido_g ha! i said something agreeable! (good day :)
[10:33] pieterh sustrik: is there any reason why apps would use SNDLABEL? just devices?
[10:33] pieterh IMO APIs should not have functions "for internal use"
[10:36] sustrik pieterh: no reason
[10:36] sustrik it's only for devices
[10:37] pieterh there's no clear dividing line between "devices" and "apps" though
[10:38] pieterh IMO if you want to make this distinction meaningful you have to almost define two separate APIs
[10:38] sustrik ack
[10:38] pieterh I'd prefer to have a single API that's clearly documented
[10:38] pieterh with no parts marked "for internal (device) use"
[10:38] sustrik well, single API (BSD) with two protocol families
[10:38] pieterh yes, but they are explicit
[10:39] pieterh this clarity is absent here
[10:39] sustrik the problem is the pattern mess we have today
[10:39] sustrik if all the patterns were implemented inside 0mq
[10:39] pieterh indeed, but IMO marking parts of the API as "internal" makes it worse, not better
[10:40] sustrik the devices would use X- socket, the applications non-X sockets
[10:40] pieterh ack
[10:40] sustrik any idea how to make it better?
[10:40] pieterh hang on a sec, phone call, brb
[10:42] mikko sustrik: hi
[10:42] mikko got idea
[10:42] mikko does this go against the pattern:
[10:42] mikko there is a current minor issue that you always seem to lose first message in pub/sub
[10:44] sustrik do you?
[10:44] sustrik what version is that?
[10:44] mikko well, timing
[10:44] mikko if you start both at the same time
[10:44] sustrik ah
[10:44] sustrik right
[10:44] sustrik what's the idea?
[10:44] mikko so, it feels non-deterministic
[10:45] mikko socket option
[10:45] mikko was my idea
[10:46] mikko that would tell you if there are consumers
[10:46] sustrik the problem is that you don't know
[10:46] sustrik say, if there's multicast transport
[10:46] sustrik you are not aware of the consumers
[10:47] sustrik the only party that is aware of the consumers is the network switch
[10:47] sustrik and afaik you can query it for the info
[10:47] sustrik can't
[10:47] mikko true
[10:48] mikko there is no subscription forwarding on pgm, right?
[10:48] sustrik nope
[10:48] mikko because it's highly indeterministic process
[10:48] mikko you can connect to publisher
[10:48] mikko but you dont know if you are connected
[10:48] sustrik yes
[10:48] mikko so that causes:
[10:49] mikko PULL socket on pub to wait on "Consumer ready message"
[10:49] mikko publish one message
[10:49] mikko wait for ack from consumer
[10:49] mikko start publishing
[10:49] sustrik wait a sec, PULL?
[10:49] mikko for one way ack
[10:49] sustrik are we talking pub/sub or push/pull?
[10:49] mikko both
[10:50] mikko you need to have separate socket for ack
[10:50] mikko that consumer is there
[10:51] sustrik in pub/sub?
[10:51] mikko yes
[10:51] sustrik yes, you need to provide a presence mechanism of your own
[10:51] sustrik however, presence is never 100% reliable
[10:51] mikko yes
[10:51] mikko im not aiming for 100% reliable
[10:52] sustrik ok
[10:52] mikko just not-really-uncomfortable approach
[10:52] mikko at the moment i am using XREP sockets for that
[10:52] mikko and that feels wrong as well
[10:52] sustrik the problem is still the first message missed in pub/sub?
[10:52] mikko looping through consumers and send the message to all
[10:52] mikko yes
[10:53] sustrik why not simply sleep for a second when starting publisher?
[10:53] mikko what about if there is network latency?
[10:53] sustrik so that consumers that are already running have a chance to connect?
[10:53] mikko latency in starting the consumer?
[10:53] mikko non-deterministic
[10:54] sustrik you mean like a consumder over satellite link?
[10:54] mikko well, it could be latency caused by anything
[10:54] pieterh sustrik: re
[10:54] sustrik in such case it is not probable that consumer and publisher will be started at the same time
[10:54] pieterh for the current API docs I'd recommend just properly documenting everything
[10:55] pieterh no "internal" pieces
[10:55] sustrik pieterh: ack
[10:55] mikko sustrik: another option i guess would be just publish PING messages
[10:55] pieterh then over time we can package up functional packages better, e.g. XPUB + ZMQ_SNDLABEL
[10:55] mikko and when consumer receives ping they send a request on another socket
[10:55] mikko to tell that "IM HERE"
[10:56] pieterh mikko: is this the infamous missing message scenario?
[10:56] mikko pieterh: yes
[10:56] mikko i don't like sleep(1)
[10:56] pieterh mikko: in reality, starting a publisher and subscriber at the same time is so rare
[10:57] mikko pieterh: you'd think so
[10:57] mikko but i saw this in wild
[10:57] pieterh so the gap is typically about 100msec
[10:57] sustrik the problem is how to get historical messages
[10:58] pieterh one solution would be for the publisher application to be able to know when there is 1+ subscriber
[10:58] sustrik so, consumer connects and says "i want last 5 secs of messages"
[10:58] pieterh it's the same problem with push/pull
[10:58] mikko pieterh: i got a ticket open for that as well
[10:58] pieterh the first pull socket can get all the messages before the others have finished connecting
[10:58] mikko ACK
[10:58] mikko LIBZMQ-160
[10:58] pieterh so if a socket would report number of connections, that would solve it
[10:58] sustrik which is doable, but actually has the same affect as sleep(5) in the consumer
[10:58] pieterh sustrik: is this feasible?
[10:59] pieterh sustrik: sleep(n) is not reliable
[10:59] pieterh what if it takes 6 seconds?
[10:59] sustrik you miss the messages
[10:59] pieterh exactly
[10:59] sustrik you cannot store messages on publisher forever
[10:59] sustrik you have to specify a limit
[10:59] pieterh that's not the problem here IMO
[10:59] pieterh it's simply delaying the first pub send until the subscriber connect is ready
[11:00] sustrik which subscriber?
[11:00] mikko just thinking out loud
[11:00] mikko it would be nice if you could configure time buffer on publisher, let's say ten seconds
[11:00] mikko and backlog recv size on consumer
[11:00] mikko so consumer would receive messages from the last N seconds
[11:01] mikko maybe ill do such a device
[11:01] pieterh mikko: let's just try adding a getsockopt for number of connections...?
[11:01] mikko pieterh: pgm
[11:01] pieterh pgm doesn't have the connect loss issue
[11:01] sustrik mikko: my point is that the time buffer is equivalent to sleep(N)
[11:01] mikko and possible any future unconnected streams
[11:01] pieterh mikko: for those cases, use something like the Clone pattern
[11:01] mikko sustrik: the time buffer would be there all the time
[11:01] mikko sustrik: like if consumer joins 10 minutes after starting
[11:02] mikko it would receive last 5 seconds
[11:02] sustrik yes
[11:02] mikko how is that equal to sleep(5) ?
[11:02] sustrik at the startup
[11:02] sustrik either you delay sending by 5 secs
[11:02] mikko i mean you have ten consumers, then eleventh joins
[11:02] mikko eleventh woud receive last five seconds
[11:03] sustrik in which case messages are held at the application level
[11:03] mikko it would be continuous ubuffer
[11:03] pieterh sustrik: ok, on separate thread, could you write a line of doc for ZMQ_SNDLABEL?
[11:03] sustrik or you add a backlog buffer in which case they are held at 0mq layer
[11:03] pieterh also, I don't see any mention of labels in zmq_recv
[11:03] sustrik pieterh: ok, let me do that
[11:04] pieterh sustrik: thanks, this should be aimed at device authors and it can state which socket types it's meant for, if relevant...
[11:04] pieterh just email it to me, I'll add it to the docs and then send a patch to you
[11:04] sustrik ok
[11:04] pieterh making a few other small fixes too
[11:07] ohmy does zeromq support both synchronoous and asynchronous messaging or only asynchronous ?
[11:09] guido_g b)
[11:10] guido_g sync is req/rep but it's based on the async part
[11:10] guido_g the core lib itself will do everything asynchronously
[11:10] guido_g even connects, which leads to "strange" experiences when not considering it
[11:17] sustrik pieterh: the whole LABEL documentation is in the trunk
[11:17] pieterh ok, great
[11:18] pieterh sustrik: where? zmq_send.txt still has the "for internal use" text...
[11:20] ohmy guido_g: thank you
[11:22] sustrik pieterh: ah, missed that
[11:22] sustrik would something like this do:
[11:22] pieterh sustrik: I can copy it from getsockopt...
[11:23] sustrik "Labels are used internally by messaging patterns. The exect semantic of the label is defined by specific pattern."
[11:23] pieterh nope, in all three places, still says "Labels are used internally by 0MQ."
[11:23] pieterh your explanation confuses me more than it helps :-)
[11:24] pieterh can you explain to me when I'd use labels in my (device) code?
[11:24] sustrik the problem is that there's no semantics associated with label as such
[11:24] pieterh they cannot be "used internally" and also documented, it's contradictory
[11:24] sustrik the semantics are defined by the pattern
[11:24] sustrik for example
[11:24] sustrik req/rep uses labels to build routeback stack
[11:25] pieterh then the LABEL semantics should be defined in zmq_socket
[11:25] sustrik yes, possibly
[11:25] pieterh you cannot have an API feature that is explained as "for internal use", that does not work
[11:26] sustrik ok, let's add "label semantics" section to each pattern in zmq_socket
[11:26] pieterh indeed
[11:27] pieterh it's xpub, xsub, xrep, req, rep I assume
[11:27] pieterh I'll fix the other pages
[11:27] sustrik it's per pattern, not per socket type
[11:27] pieterh right
[11:27] pieterh how close is the 3.0 and 4.0 doc?
[11:28] sustrik almost the same
[11:28] pieterh I'll patch the 4.0 doc then, and we can backport that to 3.0...
[11:28] sustrik ok
[11:28] sustrik so, for all patterns except req/rep it can say
[11:29] sustrik "this pattern does not use labels"
[11:29] pieterh xsub/xpub doesn't?
[11:29] sustrik no need
[11:29] pieterh ok
[11:29] sustrik there's no special information to be conveyed in the message
[11:29] sustrik maybe we'll need to add something in the future
[11:29] pieterh so labels are in fact for addressing, and used currently only in req-rep pattern
[11:29] sustrik yes, exactly
[11:30] pieterh are labels by definition multipart fragments?
[11:30] sustrik kind of
[11:30] pieterh so message consists of 0..n labels, 0..n parts?
[11:30] pieterh atomic
[11:30] sustrik yes
[11:31] sustrik where labels are at 0mq level and parts at application layer
[11:31] pieterh ok... and xrep will deliver labels before it delivers message parts?
[11:31] sustrik yes
[11:31] pieterh :-) there is no "0mq level"...
[11:31] sustrik ok, the X- level
[11:31] pieterh when you say "internally for 0MQ" that means (to me) "inside the library"
[11:31] pieterh especially since 0MQ no longer includes devices at all
[11:32] pieterh ok, I'll propose some text, see how it flies...
[11:32] pieterh I think it's clear
[11:32] sustrik ok
[11:35] pieterh sustrik: I'm also going to clarify a little the distinction between "message" and "message part"
[11:35] sustrik ok
[11:36] pieterh just to be clear, ZMQ_RCVMORE will be set on a label part?
[11:41] sustrik no
[11:41] sustrik just ZMQ_RCVLABEL
[11:41] pieterh ok, so code should test first for ZMQ_RCVLABEL and second for ZMQ_RCVMORE, if it need to?
[11:42] sustrik yes, a message is composed of 0..N labels and 1..N message parts
[11:42] pieterh ok, claro
[11:42] pieterh a label has to be a subclass of message part
[11:43] pieterh otherwise descriptions like "zmq_send - send a message on a socket" are very vague
[11:43] pieterh I need to tighten up the language
[11:45] CIA-32 libzmq: 03Martin Sustrik 07master * r2910a72 10/ (src/msg.cpp src/msg.hpp): msg_t::rm_refs closes the message when number of refs drops to 0 (issue 245) ...
[11:48] pieterh sustrik: 2910a72 has an issue? what was the problem here?
[11:48] sustrik 245
[11:49] sustrik the one raised by emmanuel yesterday evening
[11:49] pieterh ok, applied to 3-0
[12:18] sustrik pieterh: why have you closed 218?
[12:19] pieterh sustrik: I'm closing issues that appear dead, but feel free to reopen
[12:19] pieterh there 200 open issues, kind of needs cleaning up
[12:20] sustrik if the only issue there is setting an option on socekts, i can fix that
[12:20] sustrik will re-open
[12:20] pieterh this specific issue looks like a user architecture problem
[12:20] pieterh mixing forks and threads is just weird
[12:21] sustrik ack
[12:21] sustrik but setting an option is a trivial fix
[12:21] pieterh IMO it's better to steer people away from fragile design patterns instead of encouraging them
[12:23] sustrik note that the guy actually tried to avoid fragile design
[12:23] pieterh also if this issue is/was really important and there was a trivial fix, I'd expect to see a patch
[12:23] pieterh no patch + no further discussion = not important
[12:24] pieterh but that's probably too brutal...
[12:24] pieterh ack
[12:24] sustrik seems to make sense
[12:24] sustrik the fd are duplicated when fork is done
[12:25] sustrik the user does exec()
[12:25] pieterh ok, should work, indeed
[12:25] sustrik and at that point the duplicates are closed
[12:25] sustrik -- if we set the option of course
[12:25] pieterh i'd really like to have a test case though
[12:26] pieterh ah, there's a test case
[12:26] pieterh I'll run that, add it to the issues repo, let you know if I can reproduce it...
[12:26] sustrik ok
[12:37] sustrik mikko: hi, are you there?
[12:52] pieterh sustrik: ok, 218 reproduced with minimal test case, in issues repo
[12:52] sustrik thx
[12:57] pieterh sustrik: added 218 test case for 3.0/4.0...
[12:57] sustrik ok
[13:36] CIA-32 libzmq: 03Martin Sustrik 07master * r8b7ac4c 10/ (8 files): Close file descriptors on exec (issue 218) ...
[13:47] eintr in zmq 4.x, is the idea that a socket will have an automatically generated (and queryable / getsockoptable) identity?
[13:48] eintr or does it only happen as now, when a dealer connects to a router?
[13:49] pieterh sustrik: minimal fix for 218 is just to add FD_CLOEXEC when it's available...?
[13:49] pieterh I don't want to make too complex backports to 2.1
[13:49] pieterh doesn't patch cleanly to 3.0 even...
[14:10] sustrik it's just the CLOEXEC stuff
[14:10] sustrik it should be applied to all fds though
[14:10] eintr rephrasing.. how would you address a given destination peer in 4.x?
[14:10] eintr (on a ROUTER socket)
[14:10] sustrik ah
[14:11] sustrik that's the universal socket
[14:11] sustrik we've changed the semantics of ROUTER in 4.0 so that it can do basically anything
[14:11] sustrik 1. it reports connections/disconnection
[14:11] sustrik 2. each inbound message is tagged by the source
[14:12] sustrik 3. each outbound message is routed to specified peer
[14:12] eintr ah ok, I get it. that's awesome.
[14:15] eintr that's so much simpler overall. luvit
[14:16] pieterh sustrik: ok, 218 backported...
[16:31] mikko sustrik: am now
[16:31] mikko was getting my passport renewed
[18:39] jefferai hey there, I have a question about the IPC transport...on the man page it shows:
[18:39] jefferai rc = zmq_bind(socket, "ipc:\/\@@//@@tmp/feeds/0"); assert (rc == 0);
[18:39] jefferai I don't really understand that address -- is that some bit of markdown or something that slipped through?
[18:41] jefferai ah -- the actual man page fine; it's the web page that has the @@@@ in it
[19:16] mikko jefferai: looks like extra @@ signs there
[19:16] mikko should be ipc:///tmp/feeds/8
[20:14] minrk1 are there docs on precisely how XREP/ROUTER change in libzmq-3.0/4.0?
[20:15] minrk1 I'm trying to figure out if the fact that messages are no longer atomic is a libzmq bug or if I'm just doing it wrong
[20:15] xristos can i share connected sockets between threads?
[20:16] xristos with a lock maybe?
[20:56] jefferai mikko: yep -- seems to be some problem parsing the man page into html
[21:00] michelp xristos from the guide: "First, do not try to use the same socket from multiple threads. No, don't explain why you think this would be excellent fun, just please don't do it."
[21:01] xristos michelp: i have a multithreaded webserver and requests are made that send zeromq messages to the backend
[21:02] xristos it kind of defeats the purpose of zeromq if i can't use zeromq to talk to a zeromq server
[21:03] xristos i lock access to the socket, so effectively all threads are serialized there
[21:03] xristos don't know why that wouldn't be ok
[22:25] Dude909 I use the lzmq from the repo. this is what I get when I execute the hello World example from thr guide, after making the necessary changes:
[22:25] Dude909 Connecting to hello world server… Sending Hello 0… vi Assertion failed: rc == 0 (tcp_connecter.cpp:62) Aborted
[22:25] Dude909 I have an ipv- address
[22:25] Dude909 ipv6
[22:57] mikko Dude909: which version are you using?
[22:57] Dude909 I pulled from git
[22:58] Dude909 4
[23:04] Dude909 this is the ipv6 I used i case I made a silly error, tcp://[fd87:d87e:eb43:7c48:c42b:ef43:1d26:515e]:63001
[23:30] shykes Howdy
[23:32] shykes So I'm looking at a package using pyzmq's IOLoop
[23:32] shykes And I'm wondering if it would work with gevent_zeromq ?
[23:32] shykes Anybody have a hint for me?
[23:39] minrk1 I'm not certain, but I would guess the answer is no
[23:40] minrk1 it's generally somewhere between annoying and impossible to use multiple event loops simultaneously in Python
[23:41] minrk1 if it is to work, you may want to look into running the zmq/tornado eventloop only one step when an event is detected by gevent
[23:41] minrk1 (or vice versa)
[23:48] xristos shykes_: it doesn't work
[23:48] xristos i run ioloop on it's own thread
[23:48] xristos the webserver on a different thread
[23:49] xristos and messaging for communication
[23:49] xristos which is a joke but that's python
[23:52] shykes that's too bad
[23:52] shykes so ioloop can't be monkey-patched by gevent?
[23:53] shykes If I set some time aside to contribute a patch, would it be more constructive to a) extend gevent-zeromq to patch ioloop (if that's even possible, maybe I'm saying something stupid), or b) re-implement said package (pyzmq-mdp) on gevent instead of ioloop?
[23:53] shykes oh wait
[23:53] shykes b) is actually as simple as "use regular REQ/REP sockets instead of ioloop"
[23:55] minrk1 Then b) sounds like the way to go
[23:56] shykes sweet. I'll try it right now

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy