[Time] Name | Message |
[02:51] ssi
|
hi folks... I'm doing a pipeline pattern using inproc sockets, and I have multiple nodes of the same class in an object pool all trying to bind their pull socket to the same address, but I'm getting "address already in use". Am I thinking about this wrong?
|
[03:18] ssi
|
I think I see now how I need to do it
|
[04:28] gdan
|
what does zmq use as a socket buffer size. i'm sending a lot of 512K blocks and was wondering if i could optimize for that (often bloks are sent and wait for an an ack)
|
[04:45] ssi
|
SWEET, got it working
|
[04:50] Toba
|
got what working
|
[05:14] ssi
|
This ridiculous multithreaded object pooled pipeline workflow engine nonsense
|
[05:14] ssi
|
with signaling
|
[05:15] ssi
|
now I'm trying to decide if I ought to make my signaling bidirectional
|
[05:16] ssi
|
or rather, I need to make it bidirectional, and I'm not sure the best topology to use
|
[05:18] ssi
|
hrm... right now master binds to a PUB, and workers are connecting to it via SUB, and the master sends out a message indicating that the context has fully loaded and processing can start
|
[05:18] ssi
|
problem is, slow worker initialization could mean that some actions' workers are not fully initialized when that message goes out
|
[05:18] ssi
|
and I'd like to be able to guarantee that everything everywhere has completed initialization before any processing occurs
|
[05:19] ssi
|
the more I think about that, the more complex it gets
|
[08:20] CIA-76
|
libzmq: 03Martin Sustrik 07bidi-pipes * re080e3e 10/ (src/dist.cpp src/dist.hpp src/xpub.cpp src/xpub.hpp): Publisher-side filtering for multi-part messages fixed ...
|
[13:25] CIA-76
|
libzmq: 03Martin Sustrik 07bidi-pipes * rff93f54 10/ (6 files in 3 dirs): ZMQ_FILTER socket option added ...
|
[17:16] CIA-76
|
libzmq: 03Martin Sustrik 07master * rff93f54 10/ (6 files in 3 dirs): ZMQ_FILTER socket option added ...
|
[17:38] CIA-76
|
libzmq: 03Steven McCoy 07master * rb164023 10/ (src/ctx.cpp src/select.cpp src/select.hpp src/windows.hpp): Fix scope on Windows includes. ...
|
[19:42] jond
|
sustrik: good holiday?
|
[19:43] pieterh
|
jond: he seems to have gone offline
|
[19:44] jond
|
pieterh: oh, hopefully fixing the breakage on master - no rest for the wicked....
|
[19:45] pieterh
|
how're you doing?
|
[19:45] jond
|
pieterh: yr reports from stateside where interesting
|
[19:45] jond
|
pieterh: i've moved to a new position
|
[19:45] pieterh
|
zeromq-related?
|
[19:45] jond
|
pieterh: sadly no
|
[19:46] pieterh
|
that's sad
|
[19:46] pieterh
|
but change is always something to work with
|
[19:46] pieterh
|
I'm getting UDP integrated into 0MQ...
|
[19:48] jond
|
pieterh: in the job before the last i wrote a nice omq/tokyocab prototype and at the last gig the situation was difficult for reasons i can't discuss here
|
[19:48] sustrik
|
jond: hi
|
[19:48] pieterh
|
hey, the man himself :)
|
[19:48] sustrik
|
hi :)
|
[19:48] jond
|
hi all!
|
[19:49] jond
|
i'm going to have a look at the sub stuff now it's in master.
|
[19:49] sustrik
|
jond: great
|
[19:49] sustrik
|
it's a massive change, so it's going to be unstable for some time
|
[19:50] jond
|
pieterh: udp -> destroy any network slogan can be reused!
|
[19:50] pieterh
|
hehe
|
[19:50] pieterh
|
i'm not sure it'll be superfast
|
[19:50] pieterh
|
its more a proof of concept
|
[19:53] jond
|
sustrik: I was looking at the engine stuff the other day. it's not edge-triggered is it and it is true it kind of relies on it being level-triggered?
|
[19:53] sustrik
|
you mean zmq_engine_t?
|
[19:54] jond
|
yep, and the tcp socket and the 8k buffer
|
[19:54] sustrik
|
if so, yes, it uses level-triggred polling
|
[19:54] taotetek
|
o/
|
[19:55] taotetek
|
pieterh: got to meet with Ken in person again last week, was out in San Mateo for the week
|
[19:55] sustrik
|
see select.cpp/poll.cpp/epoll.cpp etc.
|
[19:55] pieterh
|
where are you usually based?
|
[19:55] taotetek
|
pieterh: virginia
|
[19:55] pieterh
|
ah
|
[19:55] pieterh
|
Ken is an impressive character
|
[19:55] taotetek
|
pieterh: I go to California for a week every two months or so
|
[19:55] pieterh
|
sucks
|
[19:55] pieterh
|
all that rain :-)
|
[19:56] taotetek
|
pieterh: he is! I love working with him
|
[19:56] taotetek
|
pieterh: the zeromq / rsyslog plugins is the second project I've worked with him on, it's always great.
|
[19:56] pieterh
|
zeromq tends to connect interesting people...
|
[19:57] pieterh
|
"5 of your friends Liked zmq_poll..."
|
[19:57] jond
|
pieterh: so based on input from the companies stateside do you see the target area of omq changing? Originally it seemed financial arena but that has changed somewhat as far as i can see...
|
[19:57] taotetek
|
pieterh: we did an internal presentation on rsyslog + zeromq for the company last week (part of what I was out there for), was good fun.
|
[19:57] pieterh
|
I'm sure the financial market remains a huge market for 0MQ
|
[19:58] pieterh
|
but the interesting projects seem to be open source and/or new cloud ventures
|
[19:58] taotetek
|
jond: we're using it for distributed data analysis
|
[19:58] pieterh
|
working in the financial arena is like having your soul sucked out through a straw
|
[19:59] taotetek
|
jond: we use zeromq + some internal things we've written to do what others might use hadoop to do, etc
|
[19:59] jond
|
pieterh: yes but someof the places already have company wide tibco, websphere or in house stuff. The resistance to change is massive...
|
[19:59] pieterh
|
for me, at least, the vision statement at the start of the Guide explains it
|
[20:00] sustrik
|
what's stateside?
|
[20:00] jond
|
sustrik: the usa
|
[20:00] sustrik
|
aha
|
[20:00] pieterh
|
sustrik: I was explaining how the meetups in Portland and SFO went
|
[20:00] sustrik
|
i missed that being on holiday
|
[20:00] sustrik
|
anything interesting?
|
[20:01] pieterh
|
I guess so...
|
[20:01] pieterh
|
- a lot more users, fewer contributors than in Europe
|
[20:01] pieterh
|
- heavy focus on using 0MQ for cloud computing
|
[20:01] jond
|
taotetek: that sounds interesting.
|
[20:01] pieterh
|
- lots of data distribution to clients across the Internet (which sucks, still)
|
[20:01] pieterh
|
main requests were:
|
[20:02] pieterh
|
1. secure transport across HTTP or suchlike
|
[20:02] pieterh
|
2. service-named subports (i.e. within 1 process, N ports, each named, not numbered)
|
[20:03] pieterh
|
3. resistance to futzing
|
[20:03] pieterh
|
that's more or less it
|
[20:03] jond
|
taotetek: there was a guy at a london meetup who's company where pushing data over omq from all the twitter, facebook etc stuff. it was sort of marketing related as far as i could tell....
|
[20:03] sustrik
|
ack
|
[20:03] sustrik
|
i had a similar impression from the previous sfo meetup
|
[20:03] pieterh
|
jond: could it be ViralHeat?
|
[20:04] jond
|
pieterh: hang on i'll go and see if still have the card. it might be a few minutes
|
[20:04] pieterh
|
it's not important
|
[20:05] taotetek
|
hm.
|
[20:05] sustrik
|
jond: subscription forwarding is a killer feature for financial services afaics, i would expect to see more users from that area once it gets stable
|
[20:06] pieterh
|
sustrik: you have about... 6 months IMO before the AMQP refugees start arriving in volume
|
[20:06] taotetek
|
pieterh: not at all really the same thing though is it?
|
[20:06] sustrik
|
pieterh: have i missed something?
|
[20:06] pieterh
|
yes, a major function in AMQP is the topic exchange, which does this
|
[20:06] taotetek
|
pieterh: I've never actually used AMQP myself
|
[20:06] sustrik
|
implosion of amqp wg?
|
[20:07] pieterh
|
sustrik: well, you've not been following... AMQP/1.0 introduces a totally incompatible new set of semantics
|
[20:07] pieterh
|
no exchanges, no queues, no bindings
|
[20:07] sustrik
|
yep, i seen that
|
[20:08] pieterh
|
I blogged about it: http://www.ipocracy.com/blog:amqp-lulz
|
[20:08] sustrik
|
i had a look some 6 months ago
|
[20:08] pieterh
|
the point being there's a massive investment in AMQP/0-8 and AMQP/0.9.1 semantics
|
[20:08] sustrik
|
1-0 was unusable at that time
|
[20:08] pieterh
|
really huge
|
[20:08] sustrik
|
basically an empty syntactic shell with no semantics attached
|
[20:09] pieterh
|
well, the protocol is more concrete today
|
[20:09] pieterh
|
I assume it will become usable at some stage, that's just a question of time and money
|
[20:10] pieterh
|
I'm pretty sure people haven't realized what's coming in 1.0, and will be seriously annoyed when they find out
|
[20:10] pieterh
|
it will probably take more than 6 months, since RabbitMQ will hide the coming change
|
[20:10] sustrik
|
let's see
|
[20:10] pieterh
|
imagine 0MQ/3.0 threw out all the socket patterns and started again with a totally new set...
|
[20:11] pieterh
|
:-)
|
[20:11] taotetek
|
pieterh: that would make me cry.
|
[20:11] pieterh
|
yes, exactly
|
[20:11] pieterh
|
me too
|
[20:11] Seta00
|
haha
|
[20:11] jond
|
pieterh: i can't find the card but I'm pretty sure it was http://datasift.net
|
[20:11] pieterh
|
so AMQP/1.0 is going to make a lot of people cry
|
[20:11] sustrik
|
it was a silly move back then
|
[20:11] pieterh
|
I think AMQP was the first real open source messaging ever
|
[20:11] sustrik
|
one wonders whether it was sheer incompetence
|
[20:12] pieterh
|
silly, yes :)
|
[20:12] sustrik
|
or whether there were other motives involved
|
[20:12] pieterh
|
yes, incompetence, because there's no plausible benefit
|
[20:12] pieterh
|
obviously RHAT wanted to undermine all other vendors but that's still incompetence
|
[20:13] pieterh
|
point is, many of these crying users will be looking for a more reliable alternative
|
[20:13] pieterh
|
0MQ lacks the topic pubsub functionality that 80% of these folk use
|
[20:14] taotetek
|
pieterh: that's part of what we use rsyslog for w/ the 0mq integration we're working on
|
[20:15] sustrik
|
lacks pubsub?
|
[20:15] taotetek
|
rsyslog doesn't have pubsub, correct
|
[20:15] pieterh
|
sustrik: you know what I mean, filtering at the publisher side
|
[20:15] taotetek
|
but what it does have is the ability to write chains of rules that are bound to ports
|
[20:15] sustrik
|
ah
|
[20:15] sustrik
|
i merged the branch today
|
[20:16] pieterh
|
:-) yes, I know
|
[20:16] taotetek
|
with filtering rules based on various parts of the message
|
[20:16] sustrik
|
it'll take some time to mature
|
[20:16] taotetek
|
we use rsyslog as our backbone for carrying high volume data and then zmq for distributing data from an output port to distributed processing systems
|
[20:16] sustrik
|
the nice thing is that subscriptions are forwarded over multiple hops
|
[20:16] pieterh
|
sustrik: do you have a test case for the subscription forwarding?
|
[20:16] taotetek
|
we approached zmq much more as a tool for distributed processing than a tool for messaging to begin with
|
[20:17] sustrik
|
which solves all kinds of hq/branch office stuff
|
[20:17] pieterh
|
sustrik: yes, federation
|
[20:17] sustrik
|
pieterh: just a silly publisher/device/subscriber test
|
[20:18] pieterh
|
sustrik: if you want people to test it, we need to promote it a little
|
[20:18] sustrik
|
funny thing is that if there's no subscriber the messages are not even send from the publisher to the broker :)
|
[20:18] shykes
|
pieterh: I was among the chorus asking for service-based subports at the last meetup at dotCloud HQ
|
[20:18] pieterh
|
shykes__: :-) hi
|
[20:18] sustrik
|
which beats AMQP et al.
|
[20:18] shykes
|
I'm happy to say I found a pretty sweet alternative!
|
[20:18] shykes
|
https://github.com/guidog/pyzmq-mdp
|
[20:18] shykes
|
We're currently testing it at dotcloud
|
[20:18] pieterh
|
shykes__: heh, that's Guido's MDP implementation...
|
[20:19] shykes
|
exactly what I was looking for :)
|
[20:19] pieterh
|
MDP is ok, but needs some polishing wrt heartbeats
|
[20:19] shykes
|
at least, on paper
|
[20:19] shykes
|
(and in my naive tests)
|
[20:19] guido_g
|
did i hear my name? :)
|
[20:19] pieterh
|
guido_g: :-) you have a user for https://github.com/guidog/pyzmq-mdp
|
[20:20] guido_g
|
not only a user, but also a contributor
|
[20:20] shykes
|
guido_g hey! I was just told about mdp this week
|
[20:20] guido_g
|
shykes__: did you see the comment to the pull-req?
|
[20:20] shykes
|
it's exactly what we needed - I tried to implement the same thing but couldn't figure out how to selectively route on an XREQ/XREP
|
[20:21] shykes
|
I guess the answer is - prepend the identity to the message? :)
|
[20:21] pieterh
|
shykes__: it's explained in detail in the Guide, where this originated
|
[20:21] guido_g
|
hehe, right
|
[20:21] sustrik
|
pieterh: there's no change for the user other than you have to use XPUB/XSUB for devices instead of PUB/SUB
|
[20:21] shykes
|
guido_g: yes, thanks - I'll push that later today
|
[20:21] sustrik
|
the rest is transparent
|
[20:21] guido_g
|
shykes__: cool, will then merge it
|
[20:21] sustrik
|
examples from the guide should do
|
[20:21] pieterh
|
sustrik: any semantic difference from PUB/SUB?
|
[20:21] sustrik
|
no
|
[20:22] shykes
|
pieterh : so it's fully supported by the specs? Not just a happy coincidence in the zmq implementation? :)
|
[20:22] pieterh
|
I'm not keen on the 'X' names, as you know...
|
[20:22] pieterh
|
shykes__: it's a fully supported formal spec with two independent implementations (Python and C)
|
[20:22] pieterh
|
it's one of the unprotocols at http://rfc.zeromq.org
|
[20:23] pieterh
|
shykes__: you really need to read the Guide in detail, otherwise it won't make sense
|
[20:23] guido_g
|
shykes__: if you have questions about the implementation, don't hesitate to ask
|
[20:23] shykes
|
pieterh: I did - I'm fairly familiar with all the topologies. I guess I missed the part about custom routing topologies
|
[20:23] taotetek
|
I've officially incorporated "urban protocol" into one of our zeromq projects at work
|
[20:23] pieterh
|
sustrik: a while back you were arguing that Xsomething meant "internal plumbing only"... :)
|
[20:24] pieterh
|
so I'd recommend using PUB/SUB with socket options
|
[20:24] sustrik
|
yes, these are socket types to be used in devices
|
[20:24] pieterh
|
actually only on the SUB side
|
[20:24] sustrik
|
intermediate nodes
|
[20:24] taotetek
|
the guys from the dev group looked a little confused when I told them to send "NOM"s intead of "ACK"s
|
[20:24] pieterh
|
taotetek: lol
|
[20:25] pieterh
|
taotetek: I'm not 100% sure it's a serious tool, but it's fun anyhow
|
[20:26] pieterh
|
sustrik: so you can't do subscription forwarding without a broker?
|
[20:26] sustrik
|
you can
|
[20:26] sustrik
|
it's hop-agnostic
|
[20:26] pieterh
|
so then you use XSUB in the client?
|
[20:26] sustrik
|
nope
|
[20:26] pieterh
|
so if SUB talks to XPUB, that does the magic?
|
[20:26] sustrik
|
how it works is that there are two layers in the stack
|
[20:27] pieterh
|
and SUB to PUB does forwarding?
|
[20:27] sustrik
|
hop-by-hop layer (the X- sockets)
|
[20:27] sustrik
|
and end-to-end layer (non-X sockets)
|
[20:27] sustrik
|
so you have a chain of hops
|
[20:27] pieterh
|
sustrik: if you explain that I have to take the Guide examples and use XSUB/XPUB
|
[20:27] pieterh
|
realize that the Guide examples do not do hops
|
[20:27] sustrik
|
then use PUB/SUB
|
[20:27] sustrik
|
it'll work ok
|
[20:28] sustrik
|
XPUB/XSUB is only for deviceas
|
[20:28] pieterh
|
OK, makes sense
|
[20:37] sustrik
|
one more use case for subscription forwarding is logging
|
[20:37] sustrik
|
many PUBs, one SUB
|
[20:38] sustrik
|
if the logger is not interested in certain class of messages
|
[20:38] sustrik
|
these are not even passed over the network
|
[20:40] mikko
|
howdy
|
[20:40] sustrik
|
hi
|
[20:44] pieterh
|
hi mikko
|
[20:45] guido_g
|
cu
|
[20:48] guido_g
|
night
|
[20:56] iFire
|
sustrik is it a common thing to just have a enum listing the message type or a string?
|
[20:56] iFire
|
how else do you identify a message
|
[20:57] iFire
|
oops
|
[20:57] iFire
|
hmm
|
[20:57] iFire
|
I mean in the encoding
|
[20:57] sustrik
|
with relation to PUB/SUB you mean?
|
[20:57] iFire
|
yeah
|
[20:57] iFire
|
well messages in general
|
[20:57] sustrik
|
the matching is done on the beginning of he message
|
[20:58] sustrik
|
so you can place an integer there
|
[20:58] sustrik
|
or a string
|
[20:58] sustrik
|
it's up to you
|
[20:59] sustrik
|
the nice thing with strings is that you can form a hierarchy of message types
|
[20:59] sustrik
|
say, animals.mammals.dogs
|
[21:00] iFire
|
so I can do it within zero and the not the message encoding
|
[21:00] sustrik
|
sorry?
|
[21:00] iFire
|
sorry
|
[21:00] iFire
|
:(
|
[21:01] iFire
|
I mean within the zeromq say pub sub message envelopes
|
[21:01] iFire
|
and not say in protobuf
|
[21:01] sustrik
|
it's up to you
|
[21:02] sustrik
|
you can match messages on whatever is at the beginning of the message
|
[21:02] iFire
|
is that the envelopes thing?
|
[21:03] iFire
|
uh multipart messages for pub sub
|
[21:03] sustrik
|
the matching is done on the first message part
|
[21:03] iFire
|
yeah. I mean protobuf is in binary
|
[21:03] sustrik
|
so you can place the "message type" to the first message part
|
[21:04] sustrik
|
you can match on binary data as well
|
[21:04] iFire
|
zeromq won't have any idea without decoding it
|
[21:04] iFire
|
hmm?
|
[21:04] sustrik
|
but you have to know how does it looks like
|
[21:04] sustrik
|
so, if protobufs hide the actual binary layout, you'll have a hard time doing matching
|
[21:06] iFire
|
well I'd assume there'll be a binary string to message thing
|
[21:07] iFire
|
I'll have to assume that the first message is something I have hard coded to store metadata about the full message
|
[21:09] sustrik
|
yes, it's called "topic"
|
[21:09] sustrik
|
you can think of it as of "message type"
|
[21:11] sustrik
|
cyl
|
[22:05] acts_as
|
Hi, I've got some relatively basic questions about how to make use of ZeroMQ in the real world, so apologies before hand. I've read the guide and looked at some sample code.
|
[22:06] acts_as
|
How do people setup a 0MQ server? I mean, it looks like you have X language with 0MQ bindings and you just write the server using that. So if your code chokes for some reason, you're SOL?
|
[22:08] acts_as
|
0MQ is really interesting to me *especially* for its REQ/REP... But I'm just wondering how I would design a 0MQ service in a volatile environment. Or how would I distribute load? Just have N 0MQ services and use a traditional load balancer?
|
[22:08] taotetek
|
acts_as: REQ / REP has built in load balancing to answer your load balancing question specifically.
|
[22:44] acts_as
|
taotetek: You mean, you say how many connections you can handle and zeromq will queue them up for you, etc?
|
[22:47] taotetek
|
acts_as: if you have 1 REQ and 5 REP, then your requests will be round robin load balanced across the REP sockets
|
[22:48] taotetek
|
acts_as: and if you have 5 REQ and 1 REP, then your requests will be fair queued
|
[22:48] taotetek
|
acts_as: PUSH / PULL work the same way
|
[22:49] acts_as
|
taotetek: Oh, so... If I write a zmq-based server and run 5 instances of it, pointing to the same port, zmq will handle that?
|
[23:10] acts_as
|
nm to that
|