IRC Log

Sunday September 5, 2010

[Time] NameMessage
[05:40] sustrik lestrrat: both are going to be in 2.1
[05:40] sustrik ZMQ_FD as already there
[05:41] sustrik i'll write an email about EINTR & 2.1 today
[06:00] sustrik DasIch: That's a bug in OpenPGM library
[06:00] sustrik if you don't need PGM, just compile 0MQ without it
[06:01] sustrik it have been fixed in newer OpenPGM releases
[06:01] sustrik but haven't got into 0MQ as for now
[06:01] sustrik anway, creating multiple contexts is pretty useless
[06:02] sustrik why do you do that?
[06:32] lestrrat sustrik: by "it's already there" you mean the github repo, right?
[06:34] sustrik yes
[06:34] sustrik the master branch
[06:34] sustrik EINTR on eintr branch, FD & EVENTS on master branch
[06:35] sustrik if i have some spare time today, i'll merge EINTR fix to master as well
[08:39] pieter_hintjens sustrik: hi
[09:54] sustrik pieter_hintjens: hi
[09:57] pieter_hintjens hi martin... i was wondering about different ways to submit patches
[09:57] pieter_hintjens people are using at least three different routes
[09:57] pieter_hintjens patches to zeromq-dev, issues, and pull requests
[09:58] sustrik i am using patches from the mailing list
[09:58] sustrik are there any pending patches anywhere else?
[09:59] pieter_hintjens i saw one pull request on the ML today
[09:59] sustrik yup
[09:59] sustrik that's a packaging script thing, it's up to mato to manage that one
[10:00] pieter_hintjens :-) it's more of a consistency thing than anything else
[10:00] pieter_hintjens ok, totally different thing, i have a trivial inproc example that refuses to work
[10:00] sustrik can you paste is somewhere?
[10:00] pieter_hintjens yes, i raised an issue (62) with an example and analysis
[10:01] sustrik let me see
[10:01] pieter_hintjens i've spent several hours trying different things, ended up doing a sleep... :-/
[10:03] sustrik i would say it's the inproc reconnect thing
[10:03] sustrik inproc reconnect is still unimplemented
[10:03] sustrik thus doing connect before bind does not work
[10:04] sustrik so, if step1 connects before step2 binds
[10:04] sustrik :|
[10:04] pieter_hintjens ah, it's not just for PAIR sockets
[10:05] sustrik it's inproc as well
[10:05] pieter_hintjens with PUSH/PULL it works a little better, say 90% of the time
[10:05] sustrik it's not that visible as inproc connection cannot break
[10:05] pieter_hintjens with PAIR it dies almost immediately
[10:05] sustrik it's a timing issue
[10:05] pieter_hintjens ok, if this is a known restriction that's fine, I can document it
[10:06] sustrik Connecting a socket
[10:06] sustrik When connecting a socket to a peer address using zmq_connect() with the inproc transport, the endpoint shall be interpreted as an arbitrary string identifying the name to connect to. The name
[10:06] sustrik must have been previously created by assigning it to at least one socket within the same 0MQ context as the socket being connected.
[10:06] pieter_hintjens there is no technical reason it could not work, right?
[10:06] sustrik no
[10:06] pieter_hintjens hmm, ok
[10:06] sustrik just not implemented
[10:06] pieter_hintjens excellent, that is all I need
[10:06] pieter_hintjens i'll close the issue
[10:06] pieter_hintjens thx
[10:06] sustrik np
[10:08] pieter_hintjens "ZeroMQ is quickly becoming an even bigger hammer in the premature optimization planet of Newbo-Thumbia."
[10:08] pieter_hintjens vs. "Then they'd better find a new rite of passage, because ZeroMQ is turning into a category killer."
[10:08] sustrik :)
[10:08] pieter_hintjens that is an excellent discussion...
[10:09] sustrik yes, i liked it
[10:09] pieter_hintjens i'm wondering if it's worth resurrecting the old amqp file transfer semantics as an example over 0MQ
[10:09] pieter_hintjens should be nice and easy to implement
[10:09] sustrik if you have resources to do that
[10:09] sustrik an example?
[10:10] pieter_hintjens well, it's a question of what examples are the most valuable
[10:10] pieter_hintjens would you rather have a name resolution service or a file transfer service over 0MQ?
[10:10] sustrik ah, in the guide?
[10:10] pieter_hintjens yeah
[10:10] sustrik afaiu the examples in the guide are to be simple
[10:10] sustrik file transfer is rather complex
[10:10] pieter_hintjens sustrik: with 0MQ everything is simple :-)
[10:11] sustrik i mean chopping the files
[10:11] sustrik reassemblig them
[10:11] sustrik etc.
[10:11] pieter_hintjens maybe
[10:12] pieter_hintjens actually it's not that complex, really
[10:12] sustrik it's up to you
[10:12] pieter_hintjens the sophisticated part is the stage / delivery semantics but that's a neat demo of 0MQ IMO
[10:12] pieter_hintjens we'll see...
[10:12] sustrik optinally, it can be a tutorial
[10:12] pieter_hintjens yes
[10:12] pieter_hintjens it reminded me that someone said "80% of messaging is still file transfer"
[10:13] pieter_hintjens and it kind of seems that by covering the extreme other side of messaging (from low-latency) we prove the generality of the model
[10:14] pieter_hintjens also it fits into the notion of distributed device management via shuffling config files around
[10:14] sustrik ok
[10:14] pieter_hintjens we will need a simple tutorial on how to make and submit git patches
[10:14] sustrik enumerating the steps won't do?
[10:15] pieter_hintjens that would be a simple tutorial, then
[10:15] pieter_hintjens it would do, yes
[10:15] sustrik gitt diff, send as a main, put [PATCH] in the subject, state it's under MIT
[10:15] sustrik that's it, no?
[10:15] sustrik git, mail*
[10:16] pieter_hintjens roughly, but some of the details cause problems
[10:16] pieter_hintjens "Honestly? I spent too much time being stuck. I ended up on some crappy tutorial that said to type "git-format-patch", when of course it was "git format-patch". I got stuck on that until I trial-and-error'd."
[10:16] sustrik so far we don't require formatted patches
[10:17] sustrik so simple diff would do
[10:17] sustrik not even git diff is needed
[10:17] sustrik but it would be nice if people sent formatted patches
[10:17] pieter_hintjens mato was saying something about signed-off-by
[10:17] sustrik yup
[10:17] pieter_hintjens indeed
[10:17] pieter_hintjens for later...
[10:18] sustrik would be nice to document it though
[10:18] pieter_hintjens yes
[10:18] pieter_hintjens did you update the contract page or shall I do that
[10:18] pieter_hintjens i thought the way you expressed it in the email was very clear
[10:19] pieter_hintjens but it needs to be written down, or it'll get lost
[10:20] sustrik ok, i'll do that
[10:20] pieter_hintjens what we found, in FFII work, was that it was usually better to edit the page first, then post to the list explaining the change
[10:21] pieter_hintjens this is kind of a standard pattern to avoid losing stuff in the email list
[10:25] sustrik ok
[10:25] sustrik btw, the inproc example
[10:26] sustrik you can fix it by launch step1's thread from step2
[10:26] sustrik that should synchronise it
[10:27] pieter_hintjens hmm, nice solution...
[10:31] pieter_hintjens ack, that works
[10:31] sustrik great
[10:35] guido_g hi
[10:35] guido_g just saw the patch thing
[10:36] guido_g I used git format-patch
[10:36] guido_g but i had my changes in a local branch
[10:37] sustrik guido_g: hi
[10:37] sustrik i am not an expert on git
[10:38] sustrik do you need a local branch to be able to format a patch?
[10:38] guido_g i'm no expert neither, but will have a look
[10:39] sustrik looks like there's a "single-commit" mode
[10:42] guido_g git format-patch -k --stdout R1..R2 <- from the examples of git help format-patch
[10:43] guido_g seems that it works for commits and ranges of commits
[10:43] sustrik nice
[10:43] guido_g git format-patch -3 <- sweet
[10:43] guido_g last three commits as patches
[10:44] sustrik bingo
[10:44] sustrik that's what most people need
[10:44] guido_g uh
[10:46] sustrik they should, but you don't expect newbies submitting patches to deal with branching
[10:46] sustrik installing git and learning how to commit is hard enough
[10:46] sustrik that's why i am happy with applying simple git diffs for now
[10:47] sustrik or even dumb old diffs
[10:47] guido_g ok
[10:48] sustrik step-by-step would be nice though
[10:48] guido_g so I spend hours for nothing...
[10:48] guido_g :)
[10:48] sustrik i mean, that's the goal
[10:48] sustrik we have to educate people
[10:48] guido_g right
[10:48] sustrik do we have a patch process described anywhere?
[10:48] sustrik hm
[10:49] pieter_hintjens gudio_g: i think branches are orthogonal to patches
[10:50] pieter_hintjens in the simplest case, you clone a repo, make changes, do a diff or git diff, and get a patch
[10:50] pieter_hintjens well, that's the medium hard case
[10:50] pieter_hintjens in the simplest case you take the source code package, make changes, do a diff and get a non-git patch
[10:51] pieter_hintjens but that only applies to projects that make source code packages
[10:51] sustrik ack, and the hard case requires a topc branch
[10:51] sustrik topic*
[10:52] pieter_hintjens yes, exactly
[10:53] pieter_hintjens for projects like zguide, there is no source package
[10:53] pieter_hintjens we have to educate people, as you say
[10:54] sustrik we are at the beginning of the road, it'll take some time to make this work
[10:54] sustrik but it's worth of doing
[10:55] sustrik guido_g: can you write down your experience with format-patch?
[10:56] sustrik it could be used as a basis for the patch process tutorial
[10:57] guido_g hmmm
[10:58] guido_g i did it by book, three steps and done
[10:58] guido_g but if you really want it...
[10:58] sustrik yes, please
[10:58] sustrik you'll spare me an hour :)
[10:58] pieter_hintjens guido_g: yes please, we'll have to document the whole thing
[10:58] guido_g *sigh*
[10:58] pieter_hintjens how to submit patch from source package
[10:59] pieter_hintjens how to submit patch from git repository clone
[10:59] guido_g ok, then i'll write down what *I* did
[10:59] sustrik thanks
[10:59] pieter_hintjens how to make topic branches and then submit them as patches
[10:59] pieter_hintjens can you create a page on the wiki, minimal, and we can take it from there?
[10:59] pieter_hintjens you should be able to create pages in the docs: category
[11:00] pieter_hintjens "we" = everyone who cares about this
[11:01] guido_g this cross-domain cookie thing sucks
[11:02] sustrik ah, yes
[11:02] sustrik pieter, any idea what's the problem?
[11:02] guido_g ok, where's the how-to for the wiki?
[11:03] guido_g sustrik: it's a wikidot thing
[11:03] sustrik guido_g: just post the command lines here
[11:03] guido_g ok
[11:11] guido_g http://gist.github.com/565948
[11:11] guido_g that's the whole magic
[11:12] guido_g "git format-patch -1" is really nice
[11:13] sustrik thanks!
[11:16] jond hi
[11:16] sustrik jon: hi
[11:16] jond good to see a policy round the patches
[11:17] sustrik ack
[11:17] sustrik will you be able to make it for the meetup on tuesday?
[11:17] jond I've had trouble squahsing commits in the past, which are meant to be easier for maintainer
[11:17] sustrik i would like to have a talk with you about the subscription forwarding
[11:18] jond tues: should be ok, though had to re-arrange a family thing
[11:18] sustrik ah, sorry about that
[11:19] sustrik discussing development via emails takes much more time than having a private talk
[11:19] jond also; i can't see irc in the dayjob, or twitter etc so if there's any last minute change mailing list is best
[11:19] sustrik jond: ok, will do
[11:21] jond on the subject of git, the chacon book seems to recommend git pull requests and format patch. it's all very powerful but seems easy to hang yrself
[11:22] sustrik we'll refine the process as we get better in using git
[11:22] guido_g format-patch itself is quite nice, as shown above
[11:22] jond am I meant to bring my local patch branches upto date with master/maint before running the diff/format-patch?
[11:23] sustrik jond: that would be nice
[11:23] sustrik otherwise it's up to me to resolve the conflicts
[11:23] sustrik which can in turn break your code
[11:24] sustrik well, it depends
[11:24] sustrik if you are submitting a single-line patch
[11:24] sustrik and there have been a lot of change on master
[11:24] sustrik it's probably easier for me to merge it then for you
[11:24] jond I also think we should recommend squashing the commits, as it makes the patch simpler to read (if git rebase -i works that is)
[11:25] sustrik squash = rebase to the actual head?
[11:25] guido_g already going way to far
[11:26] jond i am trying to keep in line with 2.0.9 and maint at the moment. what mato seems to have put in place is similar to waht we do with clearcase in the day job.
[11:26] guido_g pull -> [branch or not] -> changes -> commit [may be repeated] -> format-patch -> mailing list
[11:27] jond well in the past, I used the rebase to make all the individual commits I did on my branch appear as 1 for delivery. It's really annoying to have seperate commits for typo's etc
[11:28] guido_g but it's great if you have different changes made in a row
[11:28] guido_g so the maintainer can the ones he likes
[11:29] guido_g anyhow, i think i'll stick with this easy way
[11:29] jond i'd use a seperate branch for each change.
[11:30] guido_g if you do large changes, that might be ok
[11:30] guido_g for smaller ones it's sounds like a nightmare
[11:32] sustrik jond: i wouldn't bother with maint branch for the development work
[11:32] sustrik it's meant rather for small fixes
[11:34] sustrik guido_g: yes, the smaller the patch the simpler the process should be
[11:34] sustrik that's why i'm saying i would accept even plain old diffs
[11:34] guido_g great
[11:35] guido_g lowers the entry barrier
[11:35] sustrik exactly
[11:35] sustrik however, if you ar woking on new functionality that'll take 1/2 yesr to complete
[11:35] sustrik you definitely want your own branch
[11:35] guido_g and i'd demand payment :)
[11:35] sustrik good point :)
[11:37] guido_g so, now that i burned my eggs for brunch while writing git examples, i'm in the right mood for writing some more tests...
[11:37] jond the many branches thing pretty much also depends on the turnaround of the patch getting to master etc
[11:38] jond guido_g : i should send you the prefix_tree test ( was on the mailing list, i think we need tests now)
[11:39] jond sustrik: hopefully get chance to discuss that pub filtering on tuesday
[11:41] sustrik jond: sorry, i've seen it, but i was busy since :(
[11:43] jond yes me too; 0MQ stuff very intermittent , as dayjob is not involving it all.
[11:44] guido_g *sigh* at least, I'm not alone...
[11:44] jond incidently a SUB socket can bind can't it and then mutiple PUB's connect
[11:46] sustrik jond: yes
[11:47] sustrik thus the subscription should be forwarded to _all_ connected publishers
[11:47] jond ok, then maybe that's one for the docs, someone asked that last night on here, they wanted fan-in to single server, I don't think we cover that in the docs.
[11:48] sustrik it kind of follows from the orthogonality of socket types and binds/connects
[11:48] sustrik but yes, it would be nice to mention it somewhere
[11:49] sustrik maybe in the guide?
[11:49] jond sustrik: sorry I meant the guide
[11:50] sustrik pieter_hintjens: see above, suggestion for the guide
[11:51] sustrik btw, the fact that binds/connects are independent of socket types is something that most people don't get immediately
[11:55] jond yes, i think that is the problem. there's that big book enterprise messaging or whatever be nice to lift some scenarios from there and put in guide in 0MQ terms, which is pretty much what ph has been doing very well.
[11:56] sustrik yes, it seems to work
[11:57] sustrik btw, i've already seen some job ads requiring 0MQ knowledge so hopefully we'll get into the phase where people are actually paid for working on 0MQ
[11:57] sustrik we'll see
[11:58] jond that's interesting news. I can see it being difficult to get 0MQ in some places with established code (understandly so in some cases, like reliable multicast)
[12:01] sustrik well, i don't expect nasdaq to replace their infrastrucutre by 0mq any time soon :)
[12:02] sustrik but small companies are more flexible in adopting new technologies
[12:05] guido_g hopefully
[12:05] guido_g would be nice to have a ømq related day job
[12:09] sustrik my guess is that the early adopters are stock trading start-ups
[12:09] sustrik that would mean that places like NY, London or Chicago are most interesting
[12:09] sustrik i may be wrong though
[12:10] guido_g hmmm... with mongrel2 there might be even ordinary web-work available
[12:10] sustrik interesting point
[12:11] sustrik i haven't thought of that as i never done anything with web myself
[12:11] guido_g i left it in early 2005 and never looked back :)
[12:11] sustrik :)
[12:11] guido_g never did fronted things anyway
[12:12] lestrrat I doubt mongrel2 would really generate that much 0mq jobs... not at least from the web developer's standpoint.
[12:12] lestrrat zeromq is completely abstracted away in the web framework layer, so the web developers won't even know ;)
[12:12] sustrik right -- unless you want to scale
[12:13] sustrik what if you want N web servers
[12:13] lestrrat of course, some of the more curious guys will do interesting web stuff with 0mq
[12:13] sustrik and load-balanced services
[12:13] sustrik persistence in the middle, etc.
[12:13] sustrik that are google-like scenarios afaics
[12:14] guido_g would also be nice to have it for mor ebay like things
[12:14] guido_g *more
[12:14] lestrrat well, you're talking about the smart guys, they'll do anything. your average joe web devs won't touch it with a 40ft pole
[12:14] lestrrat ;)
[12:14] guido_g lestrrat: you're wrong
[12:14] guido_g they do touch it
[12:14] guido_g which is the problem...
[12:15] lestrrat I knew where that was going ;)
[12:15] sustrik :)
[12:15] guido_g hehe
[12:31] guido_g q: where do i find the declaration of int64_t?
[12:33] sustrik guido_g: stdint.h
[12:33] guido_g thx
[13:15] jond sustrik: yes, for the large orgs change more likely thru acquisition/merger so small is best for more leading stuff
[13:16] jond re : mongrel2 does anyone understand how the 0MQ integration works? It uses that c based co-routine library?
[13:37] sustrik jond: i think so, but i haven't checked
[17:25] cremes Tasser: i'm not on irc very much during the weekends (Chicago, IL, US); if you have questions i recommend you send them to the mailing list
[18:05] pieter_hintjens sustrik: re, was out with the family in the woods...
[18:08] pieter_hintjens i will cover publishers connecting to subscriber when i find a sensible use case
[18:08] pieter_hintjens "this is what you can do" is not useful but "this is how you solve problem X" is useful
[18:09] pieter_hintjens fan-in to single server... maybe for collecting statistics
[18:09] pieter_hintjens though why not use PUSH rather than PUB...?
[18:30] cremes pieter_hintjens: fyi, there are some comments on that igvita article that probably need a response
[18:30] pieter_hintjens cremes: am actually doing that right now
[18:30] cremes i would respond if i knew anything about boost::asio or sc-whatever
[18:31] cremes cool
[18:31] pieter_hintjens though I doubt they "need" a response except "download it and try it"
[18:31] cremes ha
[18:31] pieter_hintjens i'm reminded of the virgin declaring why sex is so pointless and messy
[18:31] cremes it's just a good idea to make sure comments like that don't go unanswered especially when google turns up that article in 6 months
[18:32] pieter_hintjens ack
[18:32] cremes l8r
[18:32] pieter_hintjens if you have specific questions you'd like me to pontificate on, send me the URLs
[18:32] cremes ack :)
[18:35] sigmonsays does zeroMQ implement AMQP?
[18:35] DasIch sigmonsays: no
[18:37] sigmonsays [newb] is there a doc that describes what protocols it uses? I mean i'm sure the same concepts exist that AMQP implements..
[18:41] pieter_hintjens sigmonsays: rfc.zeromq.org but it's out of date
[18:42] pieter_hintjens this is one thing i have wanted to make for a while, a correct spec of the protocol
[18:42] pieter_hintjens but it is very different to AMQP
[18:42] pieter_hintjens AMQP has stupid clients talking to a smart broker
[18:43] pieter_hintjens 0MQ has smart clients talking to each other
[18:43] pieter_hintjens it means the wire level protocol can be a lot simpler
[18:50] zedas jond: i do. it uses the coroutines and zmq_poll
[18:52] sigmonsays So, if persistence were required, would that be something you'd implement on the sending side? How does the "broker" play a role if clients talk directly to each other? I'm having a hard time finding a doc that demonstrates a traditional setup
[18:53] pieter_hintjens sigmonsays: see if this helps at all: http://www.zeromq.org/whitepapers:switch-or-broker
[18:55] sigmonsays Thanks, will do
[18:55] sustrik in short, there are so called "devices" which are basically "mini-brokers"; it you need a full fledged broker, there's a project that allows you to plug rabbitmq into 0MQ network
[18:56] sigmonsays Yah, I was scanning the amq C API to see what it implements. Quite sexy API =P
[19:02] sigmonsays Is SCTP a supported transport?
[19:03] sustrik it would be nice if it was, but no, it isn't yet
[19:04] sustrik actually, it used to be implemented in old versions of 0MQ but people weren't much interested :|
[19:05] sustrik it seems there's still some kind of fear of using anything other than TCP
[19:05] pieter_hintjens I proposed to the IETF that they use SCTP as the basis for WebSockets
[19:05] pieter_hintjens the response was interesting
[19:06] pieter_hintjens something along the lines of "it's not present on most boxes and therefore will never be used by anyone at all"
[19:06] sustrik hm, right, it not on win boxes
[19:06] pieter_hintjens yes indeed
[19:06] pieter_hintjens end of story, pretty much
[19:06] Zao Nor on most FreeBSD boxen, as it conflicts with things like VIMAGE, if you're insane enough to want that.
[19:07] pieter_hintjens If anyone wants to comment on http://www.reddit.com/r/programming/comments/d9jrm/zeromq_networking_stack_sending_millions_of_small/#c0ylyla
[19:07] pieter_hintjens i can't login for some weird reason
[19:09] sustrik hm, win64 comment
[19:09] sustrik does anyone have a win64 installation here?
[19:09] sustrik it should be easy to fix
[19:09] sustrik if it doesn't work
[19:10] pieter_hintjens sorry, my virtual box is 32-bit...
[19:12] jond ph: the fan-in question was asked by twcc on irc about 20:00-21:00 last night. they wanted multiple publisher to single server.
[19:13] pieter_hintjens jond: do you recall the use case?
[19:13] pieter_hintjens "anyone have a drill for me?"
[19:14] pieter_hintjens "why?"
[19:14] pieter_hintjens "i have a headache that won't go away"
[19:14] jond ph: shall i mail the irc, I didnt answer as I wasnt around; steve-o was
[19:14] Zao sustrik: Does he refer to PGM being hard/impossible to build?
[19:14] jond ph: you considering trepanation then?
[19:14] pieter_hintjens well, exactly
[19:14] pieter_hintjens before answering people's questions about "how to do X", ask them "why"...
[19:14] sustrik zao: who are you referring to?
[19:15] pieter_hintjens i can see the irc log, will check
[19:15] Zao sustrik: I thought that by "win64" comment you referred to the lad talking about 64-bit windows in the reddit thread?
[19:16] pieter_hintjens jond: bah, simple PUSH->PULL pattern would do it IMO
[19:16] sustrik ah, yes, maybe he was referring to pgm rather than 0mq as such
[19:16] sustrik i don't have win32 box, cannot check
[19:16] sustrik win64*
[19:16] jond ph: IIRC correctly he wanted intermittent/transient clients sending no-ack-needed messages to single server. it was pretty vague
[19:16] Zao sustrik: I've got a 64-bit Windows box, but I don't trust ømq on it at all.
[19:17] sustrik Zao: why so?
[19:17] sustrik does it fail?
[19:17] pieter_hintjens jond: i just read it twice, he wants a push/pull fanin afaics
[19:17] pieter_hintjens it's covered in chapter 1 of the guide even
[19:18] pieter_hintjens should be a standard answer to people who ask for help
[19:18] pieter_hintjens "if you've not read ch1 & ch2 of the guide, do so and then come back"
[19:18] Zao sustrik: I set up a PUB and bind. I set up another thread with a SUB and connect. I subscribe to all topics. I then send lots of short messages which the SUB gets, until after a few dozen, it just blocks in zmq::poll.
[19:18] Zao sustrik: I've got to boil it down to a test case before I report it on the issue tracker.
[19:18] Zao As well as testing it with another compiler.
[19:18] sustrik Zao: great, thanks
[19:19] Zao This is 32-bit VC10, by the way.
[19:19] sustrik ack
[19:21] jond ph: yes, but you can also do a sub that binds and a pub that connects. There's more than one way at present, which is good.
[19:21] sustrik ph: centralised logging scanrio?
[19:25] sigmonsays Is the concept of headers up to the user to implement or does something like "routing-key" exist? ideally I wouldn't want to subscribe based on the serialized data, but the headers accompanying it.
[19:26] sigmonsays ie, ZMQ_SUBSCRIBE example from the user guide
[19:27] Zao As far as I understand it, the topic is the prefix part of the data.
[19:28] sigmonsays Well I wouldn't want to unpack a serialized message to inspect it is where i'm getting
[19:28] sigmonsays I suspect I would prefix the compress data with HTTP like headers, but i thought i'm missing something..
[19:29] sustrik you can place your topic into one message part, data to another
[19:29] sustrik the matching is always done on the first message part
[19:29] sigmonsays that's what I was missing. Thanks
[19:31] jond sustrik: i think logging would be scenario for that use of pub/sub
[19:32] sustrik jond: that's what ph asked for, no?
[19:33] sigmonsays Is there anything to facilitate message persistence? I am thinking of a WAN setup where aggregation points exist in each pop. Is that where a queue would fit in?
[19:34] DasIch fyi logbook (a python library for logging) does now provide support for a zeromq handler/subscriber
[19:34] DasIch http://packages.python.org/Logbook/api/queues.html#zeromq
[19:35] sustrik you want messages just to be offloaded to disk, or do you want to survive crash/shutdown?
[19:36] sigmonsays sustrik, good question, You mean HA replication versus disk persistence? I just want the message to be delivered and not lost/corrupted/etc.
[19:37] sustrik i mean swapping vs. storage
[19:37] sigmonsays Well swap dies when the box does. So that's not ideal
[19:38] sustrik then you have to put something that persists messages into the network
[19:38] jond sustrik: i think so. you could probably do a really nice demo with tokyo tyrant which is the reverse index lookup thing of tokyo cabinet. especially if log messages are text
[19:39] sustrik i've laready meantioned that there's a project that allows you to use RabbitMQ broker as a node in 0mq network
[19:39] pieter_hintjens jond: sorry, was writing a post on reddit
[19:39] pieter_hintjens sustrik: centralized logging... yes, perhaps
[19:39] pieter_hintjens why is that not a push/pull then?
[19:40] pieter_hintjens i mean, pubsub is a natural pattern with subscribers that select data
[19:40] pieter_hintjens centralized logging is a sink pattern
[19:40] pieter_hintjens i don't like using knives as screwdrivers, though it works
[19:40] sustrik think of the case you would add another application listening to log feed
[19:40] sigmonsays sustrik, I see, rabbitmq broker seems like overkill for persistence. I could see using tyrant/cabinet -- Was wondering if any facilities were already provided. Still newb here.
[19:40] sustrik would you want the logs to be load balanced?
[19:41] sustrik i don't think so
[19:41] pieter_hintjens sustrik: then every single app needs to connect to TWO sinks
[19:41] pieter_hintjens that's insane
[19:41] pieter_hintjens or you use a broker/forwarder
[19:41] sustrik but not streamer
[19:41] pieter_hintjens but then you don't connect pubs to subs
[19:41] pieter_hintjens no, not streamer
[19:41] pieter_hintjens broker, in that case
[19:41] sustrik => it's pub/sub
[19:42] pieter_hintjens why not push=>pull?
[19:42] sustrik because push/pull load balances
[19:42] sustrik rather than distributes
[19:42] pieter_hintjens yeah, if you want multiple sinks
[19:42] sustrik right
[19:42] pieter_hintjens you'd do a push-pull to the broker and then pub-sub to all sinks
[19:42] pieter_hintjens still not a use case for pub->sub connections
[19:42] sustrik you can do that, but why?
[19:43] pieter_hintjens because it's a sink
[19:43] pieter_hintjens not a subscriber
[19:43] pieter_hintjens ... message queuing semantics work better
[19:43] pieter_hintjens i don't want to lose log messages if broker isn't there
[19:43] pieter_hintjens i don't want subscriptions
[19:43] pieter_hintjens i want everything from all my upstream nodes
[19:43] pieter_hintjens it's a sink
[19:43] pieter_hintjens pubsub is just not accurate...
[19:44] pieter_hintjens we do IMO need a new socket type
[19:44] pieter_hintjens which is the bus
[19:44] sustrik shrug, it depends on you, if you believe that pipelining is the right pattern for message logs go for it
[19:44] pieter_hintjens ... as we discussed before wrt error messages internally
[19:44] pieter_hintjens it's not the right pattern, it's part of the pattern
[19:44] sustrik pub/sub is a bus
[19:44] jond ph+sustrik: what if we said 'audit' rather than logging. centralised audit of all messages?
[19:45] pieter_hintjens audit, yes, nothing can be lost
[19:45] pieter_hintjens pubsub is a stream with no guaranteed start
[19:45] sustrik then yes, you want something reliable, like req/rep or pipeline
[19:45] sustrik that's what most logs are
[19:45] pieter_hintjens yes
[19:45] sustrik audit is different
[19:46] pieter_hintjens logs are audits
[19:46] Zao Oh joy. Even more fun behaviour sneaks up. zmq::poll(NULL, 0, t) asserts as winsock select craps itself on maxfd=~0.
[19:46] pieter_hintjens losing a single message can be fatal
[19:46] pieter_hintjens Zao: FD_SETSIZE?
[19:47] pieter_hintjens what is ~0...?
[19:47] sustrik nope, zero-sized pollset
[19:47] sustrik win32 select obviously don't handle it well
[19:47] Zao pieter_hintjens: maxfd is initialized to zmq::retired_fd and never gets assigned to anything else, as itemcount is 0.
[19:47] sustrik Zao: can you report it please?
[19:47] Zao winsock whines about invalid parameter, and the only one looking shady is the maxfd.
[19:48] Zao sustrik: Yeah.
[19:48] sustrik thanks
[19:51] pieter_hintjens it looks to me that logging is a good usecase for a broker in fact
[19:52] pieter_hintjens i'll add this to the list of examples for the guide
[19:55] pieter_hintjens sustrik: would it be fair to generalize that any N-to-N architecture needs a device in the middle?
[19:55] sustrik definitely
[19:55] pieter_hintjens except multicast, perhaps
[19:55] pieter_hintjens ok
[19:56] sustrik even with multicast you need a network switch in the middle
[19:56] pieter_hintjens yes!
[19:56] sustrik which is actaully a HW forwarder device
[19:56] pieter_hintjens this is worth stating explicitly, it's not clear initially why we need devices
[19:56] pieter_hintjens i shall do this
[19:56] sustrik ok
[19:57] Zao Interesting. My testcase code doesn't break.
[19:57] Zao That's annoying :D
[19:57] sustrik you mean the poll bug?
[19:58] sustrik it's obvious what's going on there, no need for test case
[19:58] sustrik just report it
[20:00] pieter_hintjens cremes: ok, here's my reply on Reddit: http://www.reddit.com/r/programming/comments/d9jrm/zeromq_networking_stack_sending_millions_of_small/c0ymlox
[20:00] Zao sustrik: No, my other bug where poll on a SUB socket blocks even though the PUB peer keeps spamming it.
[20:00] Zao sustrik: The one I'm trying to drum up a test case for, which doesn't depend on D3D11.
[20:01] sustrik ah
[20:02] pieter_hintjens sustrik: you realize we're totally screwed?
[20:02] sustrik why so?
[20:02] pieter_hintjens lol :-)
[20:02] pieter_hintjens 10pm on a Sunday night and this is the most fun place to be
[20:03] sustrik good god
[20:03] pieter_hintjens yeah
[20:03] pieter_hintjens and 0MQ is still under the radar...
[20:03] sustrik i've been at a party, but returned to where fun is
[20:03] pieter_hintjens splutter...!
[20:04] sustrik :)
[20:04] pieter_hintjens btw i added two concepts in the latest guide
[20:05] pieter_hintjens 1. "0MQ string" -> length-delimited string with no trailing null
[20:05] pieter_hintjens all the examples now correctly read/write strings in that format when they use strings
[20:06] sustrik ok
[20:08] Zao Hrm. Docs say that it should completely ignore the maxfd parameter.
[20:12] sustrik Zao: i think it does
[20:12] sustrik Win32 fd_set is not a real fd_set
[20:12] sustrik instead it's a vector of SOCKETs
[20:12] sustrik it's vector of size 0 which causes trouble imo
[20:18] sigmonsays How does using (e)pgm allows multicast to be used reliably?
[20:19] sustrik check RFC3208
[20:19] sustrik there's a resend mechanism
[20:19] sigmonsays rgr

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy