Sunday February 20, 2011

[Time] NameMessage
[02:16] Jay7 Hey I'm new to ZMQ and tried installing in on a Mac OSX 10.6.3; but it appears --with-pgm is not supported currently on darwin. Is this expected in the near future by chance?
[02:28] cremes Jay7_: i'm not a pgm expert, but apparently the openpgm package (a separate project from 0mq) doesn't have good/any osx support
[02:29] cremes Jay7_: check the mailing list archives for more discussion on this
[02:29] Jay7 @cremes_: ah, okay. Thank you. I'll look through those.
[07:51] pieterh sustrik: hi
[07:53] rem7 Hi, I was going over the zmq guide, I noticed the examples use strings as messages. If I want to send something like floats is it ok to just memcpy the float, or should I use something like protobuffers?
[07:53] pieterh rem7, depends on your needs
[07:54] pieterh sending data as-such is easy and cheap but doesn't give you any kind of leverage
[07:54] rem7 I just need to pass float arrays... using a PUSH/PULL system.
[07:55] pieterh if it's a throw-away app, go for it
[07:55] pieterh if it's something you are more serious about, make formal 'contracts'
[07:55] pieterh i.e. serialize your data properly (portably), document the message formats, etc.
[07:55] pieterh it's a lot more work up front but lets you work more cheaply long term
[07:56] pieterh e.g. plug in a worker in a different language
[07:56] rem7 would using google proto buffers take care of that?
[07:57] rem7 I don't have much experience with this kinda of stuff...
[08:15] pieterh rem7, yes, it would
[08:18] rem7 pieterh_: I guess I'll start looking into that. Thanks.
[08:18] pieterh let us know how it goes
[08:20] guido_g smaller alternative might be
[08:22] rem7 guido_g: thanks, I'll look into it
[08:42] sustrik pieterh_: morning
[08:56] pieterh sustrik, hi
[08:58] pieterh do you think the red theme is readable enough for the man pages?
[10:28] pieter_hintjens anyone looked at distributed hash tables on top of 0MQ, e.g. Chord?
[10:52] travlr hi all. i've got a question about interfacing zmq in Qt having read up a lot on what's going on I'm still a little confused about zmq's edge-trigger....
[10:53] travlr Qt has a QSocketNotifier class that provides a signal when there is activity on a profided FD
[10:53] travlr like from zmq_getsockopt(... ZMQ_FD...)
[10:54] travlr I'm assuming that the Qt sig is level triggered.
[10:56] travlr so when I get a signal from QSocketNotifier, I check the ZMQ_EVENTS for ZMQ_POLLIN (in this case) and ....
[10:56] travlr what do I do exactly when I get EAGAIN?
[11:02] travlr I'm guessing that I need to keep calling zmq_recv() until I do get the EAGAIN condition.
[11:06] pieter_hintjens travlr: sorry, can't really help, not sure what the edge trigger vs. level trigger is about
[11:06] pieter_hintjens try invoking sustrik... :-)
[11:07] travlr hi pieter.. ok, thanks just the same
[11:42] private_meta Is it possible to download a certain language bound example-set (or all of them if necessary) without having to install the entire git suite?
[11:42] travlr private_meta: each binding has its own git repo.. i think
[11:43] private_meta I meant that one
[11:43] private_meta and/or the dedicated subdirectories/trees
[11:43] private_meta brb
[11:44] travlr no. but the that download isn't that big anyway
[12:05] sustrik travlr: yes
[12:05] sustrik you have to call recv() till you get EAGAIN
[12:06] private_meta travlr: I don't want to download 40 files per language, it's a hassle
[12:15] mikko good morning all
[12:16] sustrik morning
[12:24] mikko build failure on s390x
[12:24] mikko but that is due to shutdown_stress
[12:35] pieter_hintjens mikko: quick question
[12:39] pieter_hintjens private_meta: it takes about 30 seconds to clone the git
[12:40] pieter_hintjens and it's nice that people get the git because they can then translate the examples into other languages
[12:42] pieter_hintjens private_meta: ah, you can click on 'Downloads' on the github page and get a .zip or .tar.gz. file
[12:45] mikko pieter_hintjens: sure
[12:45] mikko what is the question?
[12:45] pieter_hintjens mikko: I have scripts that will rebuild from the git repo
[12:46] pieter_hintjens bash/perl/python
[12:46] mikko do you want to run it automatically on daily builds?
[12:46] pieter_hintjens at least for master, I think, yes
[12:46] mikko of course
[12:46] pieter_hintjens also for any branch that is being maintained...
[12:46] mikko is the script available somewhere?
[12:47] mikko thats perfectly fine, the box is idle apart from the 5am/pm builds
[12:47] pieter_hintjens I guess the question is whether the scripts should be in their own git, the zeromq2 git, or just floating around somewhere
[12:47] pieter_hintjens eventually we could do the same for the zfl man pages
[12:47] mikko well, i think we could have git for random infra stuff
[12:47] pieter_hintjens ztools?
[12:47] mikko i got some scripts on the build server that could be there as well
[12:48] mikko which are not available anywhere now etc
[12:48] pieter_hintjens indeed
[12:49] pieter_hintjens so a new git that collects all the random tools we make?
[12:49] mikko it wouldn't probably something usable for people out of the box
[12:49] mikko but just to make sure that everything is available
[12:49] pieter_hintjens right
[12:49] pieter_hintjens in the zeromq organization would be best
[12:50] mikko yeah, probably
[12:50] pieter_hintjens
[12:51] mikko cool, i'll add things there later
[12:51] pieter_hintjens I'll whip up a README, I suggest one directory per tool
[12:51] mikko yeah
[12:51] mikko makes sense
[12:51] mikko lunch, brb
[12:53] pieter_hintjens ok, you should have rw access
[12:53] pieter_hintjens cyl
[12:53] private_meta pieter_hintjens: the download page was empty for me
[12:54] pieter_hintjens private_meta: ? it should pop up a window asking if you want a .zip or .tar.gz...
[12:54] pieter_hintjens what URL did you get?
[12:54] pieter_hintjens You want to be here:
[12:54] pieter_hintjens then click on Downloads
[12:55] pieter_hintjens sorry!
[12:55] pieter_hintjens
[12:56] private_meta pieter_hintjens: "should" is such a nice word, covers all kinds of possibilities, mostly the "not" possibility for me
[12:56] Guthur it works fine here
[12:57] pieter_hintjens private_meta: please explain in more detail what you do, and what happens
[12:57] pieter_hintjens "it doesn't work" is not useful...
[13:01] pieter_hintjens ok, lunch, bbl
[13:04] private_meta yay... demonstrative effect... had "no downloads" or something like that for half a week, now I ASK about it nd it shows downloads
[13:05] private_meta I hate that
[13:07] Guthur well, at least it's working, hehe
[13:13] private_meta I'm not sure anymore, someone might have told me already, but where in the guide can I find some info about how to implement heartbeating. according to the guide it has to do something with zmq_poll, but I don't find anything helpful on zmq_poll. I want to do something like heartbeat pings, but I'm not sure how to do it exactly
[13:19] mikko pieter_hintjens:
[13:23] mikko private_meta: normally you would send a heartbeat request to client
[13:23] mikko zmq_poll for incoming messages
[13:24] mikko no message in given timeout -> consider dead
[13:25] private_meta I don't quite understand zmq_poll and how to use it yet. I'm reading the man page, but i need to see how to be able to use the zmq_poll system.
[13:26] private_meta So the heartbeat request would be a normal message that's recognizable as a heartbeat for the other side I assume?
[13:26] mikko see "Handling Multiple Sockets
[13:26] mikko in the zguide
[13:27] mikko yes, or even a separate heartbeat socket
[13:27] mikko depends on the application
[13:28] private_meta A separate socket might be an interesting idea, thanks. I'll read up the multiple sockets first
[13:29] mikko for example if you got PUSH/PULL as main socket you can't really heartbeat over it
[13:31] private_meta So do push/pull stand for dedicated one-way-communication endpoints?
[13:32] mikko
[13:32] mikko you can see the characteristics of each socket type from here
[13:32] private_meta i hate surfing on gprs
[13:32] private_meta thanks
[13:32] mikko whether they force a pattern, uni-directional/bi-directional
[13:32] mikko if you got zeromq installed the man pages should work as well
[13:33] private_meta thought as much
[13:35] mikko XREP/XREQ is probably the most suitable socket for heartbeating
[13:37] private_meta Well, I have 1 server, multiple clients, and I need to address these clients separately and each of these clients need to check heartbeats, or at least know if a connection terminates
[13:37] private_meta while these connections are bidirectional
[13:39] mikko yep, definitely XREQ/XREP
[13:39] mikko i think zguide has an example of custom routing
[13:39] mikko which is needed here
[13:39] private_meta so the normal req/rep is balanced automatically all the time?
[13:40] private_meta or is xreq and xrep as well, normally?
[13:41] private_meta I have a slight difficulty understanding the terminology for the custom routing I find in the guide. dealer, mama, papa, router. Assuming I need one of those, I don't know what these terms actually mean
[13:42] private_meta ah...think i found the explanation
[13:44] private_meta so i'd need router-to-router,
[13:45] mikko private_meta: xrep/xrep doesnt force request-reply pattern as strictly as req/rep
[13:45] mikko in your case you might want "req client 1, req client 2, req client 3", "wait responses from all"
[13:45] mikko i guess
[13:45] private_meta uhm... train station coming up, brb
[13:45] mikko not necessarily req client 1, wait for rep client 1, req client 2
[13:45] mikko etc
[13:46] mikko pieterh_: snapshots for zfl are now generated during builds
[13:57] private_meta mikko: According to some of the terminology: Server-to-N-Clients connection, no forced request-to-reply pattern and standard-async-but-sync-by-choice (by higher level wrapper)
[13:57] private_meta Is what I need
[13:58] private_meta and non-balanced directly addressed messaging
[14:04] private_meta mikko: concerning multiple sockets, Actually I want the connections incoming at the SAME socket, like all connection incoming at tcp port (e.g.) 5555. Still I want to address all connected clients separately
[14:10] pieterh private_meta, see LRU routing in Ch3 of the Guide
[14:10] pieterh mikko: nice, snapshots
[14:12] private_meta pieterh_: you mean as in "lru uses several clients/workers connecting to the same socket"?
[14:12] pieterh private_meta, yes, it gives you the basic framework
[14:12] pieterh you will want to do your own routing, not LRU
[14:13] private_meta Yes
[14:13] private_meta pieterh_: do you know why the c++ version uses void* instead of char* for messages??
[14:14] pieterh private_meta, no idea...
[14:14] private_meta kk
[14:18] private_meta hmm... 3g networks are way too spotty here outside of towns... it's a hassle working in trains :/
[14:21] private_meta pieterh_: not quite sure I understand correctly, for demonstrative purposes, in the lru example the client and server code is in one file and only executed in threads so you don't have separate server/client files?
[14:22] pieterh private_meta, yes, it's just to simplify the exercise
[14:22] private_meta ok
[14:22] pieterh the code in each thread is the same as it would be in a separate process
[14:22] pieterh with a little less wrapping
[14:22] private_meta Thanks!
[14:30] private_meta hmm... the lru example on git seems pretty different from the guide example
[14:32] pieterh private_meta, what URL?
[14:32] pieterh are you looking at the same two languages?
[14:33] private_meta Ah, let me look a second
[14:34] private_meta Ah, i thought i was looking at the lruqueue example in the guide, it was rtmama
[14:34] private_meta Sorry
[14:35] private_meta mama is not ideal for me i guess, everyone should be able to initiate a message
[16:12] mikko private_meta: it's void* because it's not really an array of characters
[16:12] mikko rather a blob of data of size X
[16:42] private_meta mikko: "just because you are sending strings over a line most of the time doesn't mean everyone does"
[16:42] private_meta mikko: something like that?
[16:43] private_meta I guess representing something other than a string AS a string might be a severe waste of space...
[16:48] mikko private_meta: it's also conceptual thing
[16:48] mikko private_meta: using the inproc transport you might send a pointer to a struct
[16:48] mikko or just about anything
[16:48] mikko rather than char*
[16:50] mikko private_meta: but yes, you are right
[16:51] private_meta I didn't think of using a message queue from an external library for in-process messaging
[16:51] private_meta ah, between threads it might be handy i guess
[16:53] mikko private_meta:
[16:53] mikko related post
[17:24] Guthur imho inproc messaging is a big win
[17:24] Guthur the multithreading magic article explains it best though
[17:47] andrewvc anyone know if zmq_queue supports xreq/xrep or just rep/req
[17:56] sustrik andrewvc: just xreq/xrep :)
[17:56] andrewvc oh, awesome :)
[18:00] travlr sustrik: thanks for the help earlier
[18:02] sustrik you are welcome
[18:02] sustrik travlr: if you feel the reference is unclear, feel free to suggest a patch
[18:05] travlr i will. i'm just coming back to my work with zmq after having to totally focus on other stuff for a while.
[18:05] travlr I also have to get the internal api bot working again at
[18:06] travlr as its broken atm
[18:07] Guthur is the new API doc only beta at the moment
[18:07] Guthur I'm just wondering why it's not linked from the main site
[18:08] sustrik well, pieter have written the generator yesterday
[18:08] sustrik that's why
[18:08] Guthur ah, ok
[18:09] Guthur it does seem very good though
[18:10] travlr nice! l like the the new reference links from a simple index.
[18:10] sustrik what's particularly nice is that all the released versions are covered
[18:10] travlr yup
[18:20] andrewvc Are there any examples of the XML for zmq_queue?
[18:21] andrewvc the manpage is empty of course, and my google-fu is failing me
[18:26] andrewvc nm, figured it out
[18:53] pieterh sustrik, mikko and I will arrange to rebuild the docs once a day or so
[18:55] sustrik pieterh_: great
[18:56] sustrik if you do it twice a day, the docs will be in sync with latest snapshots
[18:56] pieterh we've made a ztools git for random tools, incl the rebuild shells scripts
[18:56] sustrik ack
[18:56] pieterh let's see if people have more comments on the look & feel
[18:57] pieterh i already toned it down, darker red, no logo on man pages, etc.
[18:57] sustrik let me see
[18:57] sustrik nice
[18:57] sustrik btw, what about the stable branch, do you have any schedule for doing the fork?
[18:58] pieterh next week sometime, I was thinking
[18:58] sustrik great
[18:59] pieterh ok, good!
[18:59] pieterh people will be happy to see a new stable 2.1.x version
[18:59] pieterh so many new goodies :-)
[18:59] sustrik i think so
[18:59] sustrik so will I
[18:59] pieterh I'll keep the 2.0.x version of the Guide online as well
[19:00] sustrik i like that all the versions are available
[19:00] sustrik ah, the guide
[19:00] pieterh yes, people have asked for that...
[19:00] pieterh so I will update that for 2.1, and keep the 2.0 version for reference
[19:00] sustrik i've checked it lately and it seemed to cover both 2.0 and 2.1
[19:01] pieterh in several places it covers things that were changed
[19:01] pieterh e.g. socket migration, socket shutdown
[19:01] pieterh needs fixing for 2.1
[19:01] sustrik right
[19:01] pieterh hmm, perhaps better is a section with explicit differences
[19:01] sustrik it's up to you
[19:02] pieterh i think as a programmer, what I want is "what changed in 2.1 and what does this mean for my code?"
[19:02] pieterh expanded release notes with some jokes
[19:02] sustrik sounds good
[19:02] pieterh ok!
[19:03] mikko good stuff
[19:03] mikko do you have the build script online already?
[19:03] mikko the documentation one
[19:04] mikko my todo-list is almost empty now
[19:04] mikko apart from testing openpgm on solaris
[19:06] pieterh mikko: I'm documenting / cleaning it up, will publish it asap
[19:07] mikko pieterh_: cool
[19:07] mikko any other periodic things that should be automated?
[19:08] pieterh mikko the machine
[19:08] pieterh no todo list can survive
[19:08] Guthur pieterh_, my first impression of the new api doc is that it is very good
[19:08] Guthur well done
[19:08] pieterh Guthur, I'm glad you like it...
[19:09] Guthur I'd imagine most will have similar impressions
[19:09] pieterh not too painful to read? it's the usual cortex red scheme but toned down a little
[19:09] Guthur nope the styling is very good, for my tastes
[19:10] mikko lestrrat: are you there?
[19:10] Guthur must dash though
[19:13] pieterh mikko: you'll need to get an API key from Wikidot
[19:13] pieterh it's tied to your account
[19:13] pieterh if you don't already have a account, register and tell me your account name
[19:14] pieterh then I'll get them to give you API access and a key
[19:14] mikko mkoppanen
[19:17] pieterh ok
[19:49] pieterh mikko: ok, the tool is documented and packaged for reuse
[20:42] jobytaffey Hi, I was wondering if anyone can help out. I'm new to 0MQ. I'm trying to write a TCP server which accepts many incoming connections, reads fixed sized packets from TCP and generates 0MQ messages for handling somewhere else. I'm using libev and the receive side seems fine. But, for sending (receive 0MQ message then put it onto a TCP socket), it looks like I need to zmq_poll() when libev gives me EV_WRITE, is there a better way of doing this? A zmq ca
[20:43] mikko jobytaffey: yes
[20:43] mikko jobytaffey: you should schedule libev EV_WRITE only if you got message coming from zeromq
[20:44] mikko so roughly the following (assuming zeromq 2.1.x)
[20:44] mikko you schedule constant read event for incoming tcp messages and incoming zeromq messages
[20:44] mikko if you get tcp message in you schedule a EV_WRITE for zeromq out socket
[20:45] mikko and if you get zeromq message in you schedule EV_WRITE for tcp socket
[20:45] jobytaffey Ok, that makes sense. I didn't see anything in the API for getting at zeromq's underlying sockets though?
[20:46] mikko jobytaffey: it's a feature in 2.1.x
[20:46] mikko give me a sec
[20:46] mikko jobytaffey: see ZMQ_FD and ZMQ_EVENTS there
[20:48] jobytaffey That looks like what I need. Thanks.
[20:49] mikko it doesn't make sense to schedule EV_WRITE event unless you know that you have something to write
[22:52] lestrrat wassup?
[23:05] lestrrat oh I get it, there's a build failure.
[23:05] lestrrat mikko: I didn't change anything in the last few days
[23:48] mikko lestrrat: ok, will rerun to see
[23:49] mikko succeeds
[23:49] mikko probably an intermittent failure