Sunday June 12, 2011

[Time] NameMessage
[02:51] ssi hi folks... I'm doing a pipeline pattern using inproc sockets, and I have multiple nodes of the same class in an object pool all trying to bind their pull socket to the same address, but I'm getting "address already in use". Am I thinking about this wrong?
[03:18] ssi I think I see now how I need to do it
[04:28] gdan what does zmq use as a socket buffer size. i'm sending a lot of 512K blocks and was wondering if i could optimize for that (often bloks are sent and wait for an an ack)
[04:45] ssi SWEET, got it working
[04:50] Toba got what working
[05:14] ssi This ridiculous multithreaded object pooled pipeline workflow engine nonsense
[05:14] ssi with signaling
[05:15] ssi now I'm trying to decide if I ought to make my signaling bidirectional
[05:16] ssi or rather, I need to make it bidirectional, and I'm not sure the best topology to use
[05:18] ssi hrm... right now master binds to a PUB, and workers are connecting to it via SUB, and the master sends out a message indicating that the context has fully loaded and processing can start
[05:18] ssi problem is, slow worker initialization could mean that some actions' workers are not fully initialized when that message goes out
[05:18] ssi and I'd like to be able to guarantee that everything everywhere has completed initialization before any processing occurs
[05:19] ssi the more I think about that, the more complex it gets
[08:20] CIA-76 libzmq: 03Martin Sustrik 07bidi-pipes * re080e3e 10/ (src/dist.cpp src/dist.hpp src/xpub.cpp src/xpub.hpp): Publisher-side filtering for multi-part messages fixed ...
[13:25] CIA-76 libzmq: 03Martin Sustrik 07bidi-pipes * rff93f54 10/ (6 files in 3 dirs): ZMQ_FILTER socket option added ...
[17:16] CIA-76 libzmq: 03Martin Sustrik 07master * rff93f54 10/ (6 files in 3 dirs): ZMQ_FILTER socket option added ...
[17:38] CIA-76 libzmq: 03Steven McCoy 07master * rb164023 10/ (src/ctx.cpp src/select.cpp src/select.hpp src/windows.hpp): Fix scope on Windows includes. ...
[19:42] jond sustrik: good holiday?
[19:43] pieterh jond: he seems to have gone offline
[19:44] jond pieterh: oh, hopefully fixing the breakage on master - no rest for the wicked....
[19:45] pieterh how're you doing?
[19:45] jond pieterh: yr reports from stateside where interesting
[19:45] jond pieterh: i've moved to a new position
[19:45] pieterh zeromq-related?
[19:45] jond pieterh: sadly no
[19:46] pieterh that's sad
[19:46] pieterh but change is always something to work with
[19:46] pieterh I'm getting UDP integrated into 0MQ...
[19:48] jond pieterh: in the job before the last i wrote a nice omq/tokyocab prototype and at the last gig the situation was difficult for reasons i can't discuss here
[19:48] sustrik jond: hi
[19:48] pieterh hey, the man himself :)
[19:48] sustrik hi :)
[19:48] jond hi all!
[19:49] jond i'm going to have a look at the sub stuff now it's in master.
[19:49] sustrik jond: great
[19:49] sustrik it's a massive change, so it's going to be unstable for some time
[19:50] jond pieterh: udp -> destroy any network slogan can be reused!
[19:50] pieterh hehe
[19:50] pieterh i'm not sure it'll be superfast
[19:50] pieterh its more a proof of concept
[19:53] jond sustrik: I was looking at the engine stuff the other day. it's not edge-triggered is it and it is true it kind of relies on it being level-triggered?
[19:53] sustrik you mean zmq_engine_t?
[19:54] jond yep, and the tcp socket and the 8k buffer
[19:54] sustrik if so, yes, it uses level-triggred polling
[19:54] taotetek o/
[19:55] taotetek pieterh: got to meet with Ken in person again last week, was out in San Mateo for the week
[19:55] sustrik see select.cpp/poll.cpp/epoll.cpp etc.
[19:55] pieterh where are you usually based?
[19:55] taotetek pieterh: virginia
[19:55] pieterh ah
[19:55] pieterh Ken is an impressive character
[19:55] taotetek pieterh: I go to California for a week every two months or so
[19:55] pieterh sucks
[19:55] pieterh all that rain :-)
[19:56] taotetek pieterh: he is! I love working with him
[19:56] taotetek pieterh: the zeromq / rsyslog plugins is the second project I've worked with him on, it's always great.
[19:56] pieterh zeromq tends to connect interesting people...
[19:57] pieterh "5 of your friends Liked zmq_poll..."
[19:57] jond pieterh: so based on input from the companies stateside do you see the target area of omq changing? Originally it seemed financial arena but that has changed somewhat as far as i can see...
[19:57] taotetek pieterh: we did an internal presentation on rsyslog + zeromq for the company last week (part of what I was out there for), was good fun.
[19:57] pieterh I'm sure the financial market remains a huge market for 0MQ
[19:58] pieterh but the interesting projects seem to be open source and/or new cloud ventures
[19:58] taotetek jond: we're using it for distributed data analysis
[19:58] pieterh working in the financial arena is like having your soul sucked out through a straw
[19:59] taotetek jond: we use zeromq + some internal things we've written to do what others might use hadoop to do, etc
[19:59] jond pieterh: yes but someof the places already have company wide tibco, websphere or in house stuff. The resistance to change is massive...
[19:59] pieterh for me, at least, the vision statement at the start of the Guide explains it
[20:00] sustrik what's stateside?
[20:00] jond sustrik: the usa
[20:00] sustrik aha
[20:00] pieterh sustrik: I was explaining how the meetups in Portland and SFO went
[20:00] sustrik i missed that being on holiday
[20:00] sustrik anything interesting?
[20:01] pieterh I guess so...
[20:01] pieterh - a lot more users, fewer contributors than in Europe
[20:01] pieterh - heavy focus on using 0MQ for cloud computing
[20:01] jond taotetek: that sounds interesting.
[20:01] pieterh - lots of data distribution to clients across the Internet (which sucks, still)
[20:01] pieterh main requests were:
[20:02] pieterh 1. secure transport across HTTP or suchlike
[20:02] pieterh 2. service-named subports (i.e. within 1 process, N ports, each named, not numbered)
[20:03] pieterh 3. resistance to futzing
[20:03] pieterh that's more or less it
[20:03] jond taotetek: there was a guy at a london meetup who's company where pushing data over omq from all the twitter, facebook etc stuff. it was sort of marketing related as far as i could tell....
[20:03] sustrik ack
[20:03] sustrik i had a similar impression from the previous sfo meetup
[20:03] pieterh jond: could it be ViralHeat?
[20:04] jond pieterh: hang on i'll go and see if still have the card. it might be a few minutes
[20:04] pieterh it's not important
[20:05] taotetek hm.
[20:05] sustrik jond: subscription forwarding is a killer feature for financial services afaics, i would expect to see more users from that area once it gets stable
[20:06] pieterh sustrik: you have about... 6 months IMO before the AMQP refugees start arriving in volume
[20:06] taotetek pieterh: not at all really the same thing though is it?
[20:06] sustrik pieterh: have i missed something?
[20:06] pieterh yes, a major function in AMQP is the topic exchange, which does this
[20:06] taotetek pieterh: I've never actually used AMQP myself
[20:06] sustrik implosion of amqp wg?
[20:07] pieterh sustrik: well, you've not been following... AMQP/1.0 introduces a totally incompatible new set of semantics
[20:07] pieterh no exchanges, no queues, no bindings
[20:07] sustrik yep, i seen that
[20:08] pieterh I blogged about it:
[20:08] sustrik i had a look some 6 months ago
[20:08] pieterh the point being there's a massive investment in AMQP/0-8 and AMQP/0.9.1 semantics
[20:08] sustrik 1-0 was unusable at that time
[20:08] pieterh really huge
[20:08] sustrik basically an empty syntactic shell with no semantics attached
[20:09] pieterh well, the protocol is more concrete today
[20:09] pieterh I assume it will become usable at some stage, that's just a question of time and money
[20:10] pieterh I'm pretty sure people haven't realized what's coming in 1.0, and will be seriously annoyed when they find out
[20:10] pieterh it will probably take more than 6 months, since RabbitMQ will hide the coming change
[20:10] sustrik let's see
[20:10] pieterh imagine 0MQ/3.0 threw out all the socket patterns and started again with a totally new set...
[20:11] pieterh :-)
[20:11] taotetek pieterh: that would make me cry.
[20:11] pieterh yes, exactly
[20:11] pieterh me too
[20:11] Seta00 haha
[20:11] jond pieterh: i can't find the card but I'm pretty sure it was
[20:11] pieterh so AMQP/1.0 is going to make a lot of people cry
[20:11] sustrik it was a silly move back then
[20:11] pieterh I think AMQP was the first real open source messaging ever
[20:11] sustrik one wonders whether it was sheer incompetence
[20:12] pieterh silly, yes :)
[20:12] sustrik or whether there were other motives involved
[20:12] pieterh yes, incompetence, because there's no plausible benefit
[20:12] pieterh obviously RHAT wanted to undermine all other vendors but that's still incompetence
[20:13] pieterh point is, many of these crying users will be looking for a more reliable alternative
[20:13] pieterh 0MQ lacks the topic pubsub functionality that 80% of these folk use
[20:14] taotetek pieterh: that's part of what we use rsyslog for w/ the 0mq integration we're working on
[20:15] sustrik lacks pubsub?
[20:15] taotetek rsyslog doesn't have pubsub, correct
[20:15] pieterh sustrik: you know what I mean, filtering at the publisher side
[20:15] taotetek but what it does have is the ability to write chains of rules that are bound to ports
[20:15] sustrik ah
[20:15] sustrik i merged the branch today
[20:16] pieterh :-) yes, I know
[20:16] taotetek with filtering rules based on various parts of the message
[20:16] sustrik it'll take some time to mature
[20:16] taotetek we use rsyslog as our backbone for carrying high volume data and then zmq for distributing data from an output port to distributed processing systems
[20:16] sustrik the nice thing is that subscriptions are forwarded over multiple hops
[20:16] pieterh sustrik: do you have a test case for the subscription forwarding?
[20:16] taotetek we approached zmq much more as a tool for distributed processing than a tool for messaging to begin with
[20:17] sustrik which solves all kinds of hq/branch office stuff
[20:17] pieterh sustrik: yes, federation
[20:17] sustrik pieterh: just a silly publisher/device/subscriber test
[20:18] pieterh sustrik: if you want people to test it, we need to promote it a little
[20:18] sustrik funny thing is that if there's no subscriber the messages are not even send from the publisher to the broker :)
[20:18] shykes pieterh: I was among the chorus asking for service-based subports at the last meetup at dotCloud HQ
[20:18] pieterh shykes__: :-) hi
[20:18] sustrik which beats AMQP et al.
[20:18] shykes I'm happy to say I found a pretty sweet alternative!
[20:18] shykes
[20:18] shykes We're currently testing it at dotcloud
[20:18] pieterh shykes__: heh, that's Guido's MDP implementation...
[20:19] shykes exactly what I was looking for :)
[20:19] pieterh MDP is ok, but needs some polishing wrt heartbeats
[20:19] shykes at least, on paper
[20:19] shykes (and in my naive tests)
[20:19] guido_g did i hear my name? :)
[20:19] pieterh guido_g: :-) you have a user for
[20:20] guido_g not only a user, but also a contributor
[20:20] shykes guido_g hey! I was just told about mdp this week
[20:20] guido_g shykes__: did you see the comment to the pull-req?
[20:20] shykes it's exactly what we needed - I tried to implement the same thing but couldn't figure out how to selectively route on an XREQ/XREP
[20:21] shykes I guess the answer is - prepend the identity to the message? :)
[20:21] pieterh shykes__: it's explained in detail in the Guide, where this originated
[20:21] guido_g hehe, right
[20:21] sustrik pieterh: there's no change for the user other than you have to use XPUB/XSUB for devices instead of PUB/SUB
[20:21] shykes guido_g: yes, thanks - I'll push that later today
[20:21] sustrik the rest is transparent
[20:21] guido_g shykes__: cool, will then merge it
[20:21] sustrik examples from the guide should do
[20:21] pieterh sustrik: any semantic difference from PUB/SUB?
[20:21] sustrik no
[20:22] shykes pieterh : so it's fully supported by the specs? Not just a happy coincidence in the zmq implementation? :)
[20:22] pieterh I'm not keen on the 'X' names, as you know...
[20:22] pieterh shykes__: it's a fully supported formal spec with two independent implementations (Python and C)
[20:22] pieterh it's one of the unprotocols at
[20:23] pieterh shykes__: you really need to read the Guide in detail, otherwise it won't make sense
[20:23] guido_g shykes__: if you have questions about the implementation, don't hesitate to ask
[20:23] shykes pieterh: I did - I'm fairly familiar with all the topologies. I guess I missed the part about custom routing topologies
[20:23] taotetek I've officially incorporated "urban protocol" into one of our zeromq projects at work
[20:23] pieterh sustrik: a while back you were arguing that Xsomething meant "internal plumbing only"... :)
[20:24] pieterh so I'd recommend using PUB/SUB with socket options
[20:24] sustrik yes, these are socket types to be used in devices
[20:24] pieterh actually only on the SUB side
[20:24] sustrik intermediate nodes
[20:24] taotetek the guys from the dev group looked a little confused when I told them to send "NOM"s intead of "ACK"s
[20:24] pieterh taotetek: lol
[20:25] pieterh taotetek: I'm not 100% sure it's a serious tool, but it's fun anyhow
[20:26] pieterh sustrik: so you can't do subscription forwarding without a broker?
[20:26] sustrik you can
[20:26] sustrik it's hop-agnostic
[20:26] pieterh so then you use XSUB in the client?
[20:26] sustrik nope
[20:26] pieterh so if SUB talks to XPUB, that does the magic?
[20:26] sustrik how it works is that there are two layers in the stack
[20:27] pieterh and SUB to PUB does forwarding?
[20:27] sustrik hop-by-hop layer (the X- sockets)
[20:27] sustrik and end-to-end layer (non-X sockets)
[20:27] sustrik so you have a chain of hops
[20:27] pieterh sustrik: if you explain that I have to take the Guide examples and use XSUB/XPUB
[20:27] pieterh realize that the Guide examples do not do hops
[20:27] sustrik then use PUB/SUB
[20:27] sustrik it'll work ok
[20:28] sustrik XPUB/XSUB is only for deviceas
[20:28] pieterh OK, makes sense
[20:37] sustrik one more use case for subscription forwarding is logging
[20:37] sustrik many PUBs, one SUB
[20:38] sustrik if the logger is not interested in certain class of messages
[20:38] sustrik these are not even passed over the network
[20:40] mikko howdy
[20:40] sustrik hi
[20:44] pieterh hi mikko
[20:45] guido_g cu
[20:48] guido_g night
[20:56] iFire sustrik is it a common thing to just have a enum listing the message type or a string?
[20:56] iFire how else do you identify a message
[20:57] iFire oops
[20:57] iFire hmm
[20:57] iFire I mean in the encoding
[20:57] sustrik with relation to PUB/SUB you mean?
[20:57] iFire yeah
[20:57] iFire well messages in general
[20:57] sustrik the matching is done on the beginning of he message
[20:58] sustrik so you can place an integer there
[20:58] sustrik or a string
[20:58] sustrik it's up to you
[20:59] sustrik the nice thing with strings is that you can form a hierarchy of message types
[20:59] sustrik say, animals.mammals.dogs
[21:00] iFire so I can do it within zero and the not the message encoding
[21:00] sustrik sorry?
[21:00] iFire sorry
[21:00] iFire :(
[21:01] iFire I mean within the zeromq say pub sub message envelopes
[21:01] iFire and not say in protobuf
[21:01] sustrik it's up to you
[21:02] sustrik you can match messages on whatever is at the beginning of the message
[21:02] iFire is that the envelopes thing?
[21:03] iFire uh multipart messages for pub sub
[21:03] sustrik the matching is done on the first message part
[21:03] iFire yeah. I mean protobuf is in binary
[21:03] sustrik so you can place the "message type" to the first message part
[21:04] sustrik you can match on binary data as well
[21:04] iFire zeromq won't have any idea without decoding it
[21:04] iFire hmm?
[21:04] sustrik but you have to know how does it looks like
[21:04] sustrik so, if protobufs hide the actual binary layout, you'll have a hard time doing matching
[21:06] iFire well I'd assume there'll be a binary string to message thing
[21:07] iFire I'll have to assume that the first message is something I have hard coded to store metadata about the full message
[21:09] sustrik yes, it's called "topic"
[21:09] sustrik you can think of it as of "message type"
[21:11] sustrik cyl
[22:05] acts_as Hi, I've got some relatively basic questions about how to make use of ZeroMQ in the real world, so apologies before hand. I've read the guide and looked at some sample code.
[22:06] acts_as How do people setup a 0MQ server? I mean, it looks like you have X language with 0MQ bindings and you just write the server using that. So if your code chokes for some reason, you're SOL?
[22:08] acts_as 0MQ is really interesting to me *especially* for its REQ/REP... But I'm just wondering how I would design a 0MQ service in a volatile environment. Or how would I distribute load? Just have N 0MQ services and use a traditional load balancer?
[22:08] taotetek acts_as: REQ / REP has built in load balancing to answer your load balancing question specifically.
[22:44] acts_as taotetek: You mean, you say how many connections you can handle and zeromq will queue them up for you, etc?
[22:47] taotetek acts_as: if you have 1 REQ and 5 REP, then your requests will be round robin load balanced across the REP sockets
[22:48] taotetek acts_as: and if you have 5 REQ and 1 REP, then your requests will be fair queued
[22:48] taotetek acts_as: PUSH / PULL work the same way
[22:49] acts_as taotetek: Oh, so... If I write a zmq-based server and run 5 instances of it, pointing to the same port, zmq will handle that?
[23:10] acts_as nm to that