ZeroMq IRC Log

Sunday June 12, 2011

[Time] Name	Message
[02:51] ssi	hi folks... I'm doing a pipeline pattern using inproc sockets, and I have multiple nodes of the same class in an object pool all trying to bind their pull socket to the same address, but I'm getting "address already in use". Am I thinking about this wrong?
[03:18] ssi	I think I see now how I need to do it
[04:28] gdan	what does zmq use as a socket buffer size. i'm sending a lot of 512K blocks and was wondering if i could optimize for that (often bloks are sent and wait for an an ack)
[04:45] ssi	SWEET, got it working
[04:50] Toba	got what working
[05:14] ssi	This ridiculous multithreaded object pooled pipeline workflow engine nonsense
[05:14] ssi	with signaling
[05:15] ssi	now I'm trying to decide if I ought to make my signaling bidirectional
[05:16] ssi	or rather, I need to make it bidirectional, and I'm not sure the best topology to use
[05:18] ssi	hrm... right now master binds to a PUB, and workers are connecting to it via SUB, and the master sends out a message indicating that the context has fully loaded and processing can start
[05:18] ssi	problem is, slow worker initialization could mean that some actions' workers are not fully initialized when that message goes out
[05:18] ssi	and I'd like to be able to guarantee that everything everywhere has completed initialization before any processing occurs
[05:19] ssi	the more I think about that, the more complex it gets
[08:20] CIA-76	libzmq: 03Martin Sustrik 07bidi-pipes * re080e3e 10/ (src/dist.cpp src/dist.hpp src/xpub.cpp src/xpub.hpp): Publisher-side filtering for multi-part messages fixed ...
[13:25] CIA-76	libzmq: 03Martin Sustrik 07bidi-pipes * rff93f54 10/ (6 files in 3 dirs): ZMQ_FILTER socket option added ...
[17:16] CIA-76	libzmq: 03Martin Sustrik 07master * rff93f54 10/ (6 files in 3 dirs): ZMQ_FILTER socket option added ...
[17:38] CIA-76	libzmq: 03Steven McCoy 07master * rb164023 10/ (src/ctx.cpp src/select.cpp src/select.hpp src/windows.hpp): Fix scope on Windows includes. ...
[19:42] jond	sustrik: good holiday?
[19:43] pieterh	jond: he seems to have gone offline
[19:44] jond	pieterh: oh, hopefully fixing the breakage on master - no rest for the wicked....
[19:45] pieterh	how're you doing?
[19:45] jond	pieterh: yr reports from stateside where interesting
[19:45] jond	pieterh: i've moved to a new position
[19:45] pieterh	zeromq-related?
[19:45] jond	pieterh: sadly no
[19:46] pieterh	that's sad
[19:46] pieterh	but change is always something to work with
[19:46] pieterh	I'm getting UDP integrated into 0MQ...
[19:48] jond	pieterh: in the job before the last i wrote a nice omq/tokyocab prototype and at the last gig the situation was difficult for reasons i can't discuss here
[19:48] sustrik	jond: hi
[19:48] pieterh	hey, the man himself :)
[19:48] sustrik	hi :)
[19:48] jond	hi all!
[19:49] jond	i'm going to have a look at the sub stuff now it's in master.
[19:49] sustrik	jond: great
[19:49] sustrik	it's a massive change, so it's going to be unstable for some time
[19:50] jond	pieterh: udp -> destroy any network slogan can be reused!
[19:50] pieterh	hehe
[19:50] pieterh	i'm not sure it'll be superfast
[19:50] pieterh	its more a proof of concept
[19:53] jond	sustrik: I was looking at the engine stuff the other day. it's not edge-triggered is it and it is true it kind of relies on it being level-triggered?
[19:53] sustrik	you mean zmq_engine_t?
[19:54] jond	yep, and the tcp socket and the 8k buffer
[19:54] sustrik	if so, yes, it uses level-triggred polling
[19:54] taotetek	o/
[19:55] taotetek	pieterh: got to meet with Ken in person again last week, was out in San Mateo for the week
[19:55] sustrik	see select.cpp/poll.cpp/epoll.cpp etc.
[19:55] pieterh	where are you usually based?
[19:55] taotetek	pieterh: virginia
[19:55] pieterh	ah
[19:55] pieterh	Ken is an impressive character
[19:55] taotetek	pieterh: I go to California for a week every two months or so
[19:55] pieterh	sucks
[19:55] pieterh	all that rain :-)
[19:56] taotetek	pieterh: he is! I love working with him
[19:56] taotetek	pieterh: the zeromq / rsyslog plugins is the second project I've worked with him on, it's always great.
[19:56] pieterh	zeromq tends to connect interesting people...
[19:57] pieterh	"5 of your friends Liked zmq_poll..."
[19:57] jond	pieterh: so based on input from the companies stateside do you see the target area of omq changing? Originally it seemed financial arena but that has changed somewhat as far as i can see...
[19:57] taotetek	pieterh: we did an internal presentation on rsyslog + zeromq for the company last week (part of what I was out there for), was good fun.
[19:57] pieterh	I'm sure the financial market remains a huge market for 0MQ
[19:58] pieterh	but the interesting projects seem to be open source and/or new cloud ventures
[19:58] taotetek	jond: we're using it for distributed data analysis
[19:58] pieterh	working in the financial arena is like having your soul sucked out through a straw
[19:59] taotetek	jond: we use zeromq + some internal things we've written to do what others might use hadoop to do, etc
[19:59] jond	pieterh: yes but someof the places already have company wide tibco, websphere or in house stuff. The resistance to change is massive...
[19:59] pieterh	for me, at least, the vision statement at the start of the Guide explains it
[20:00] sustrik	what's stateside?
[20:00] jond	sustrik: the usa
[20:00] sustrik	aha
[20:00] pieterh	sustrik: I was explaining how the meetups in Portland and SFO went
[20:00] sustrik	i missed that being on holiday
[20:00] sustrik	anything interesting?
[20:01] pieterh	I guess so...
[20:01] pieterh	- a lot more users, fewer contributors than in Europe
[20:01] pieterh	- heavy focus on using 0MQ for cloud computing
[20:01] jond	taotetek: that sounds interesting.
[20:01] pieterh	- lots of data distribution to clients across the Internet (which sucks, still)
[20:01] pieterh	main requests were:
[20:02] pieterh	1. secure transport across HTTP or suchlike
[20:02] pieterh	2. service-named subports (i.e. within 1 process, N ports, each named, not numbered)
[20:03] pieterh	3. resistance to futzing
[20:03] pieterh	that's more or less it
[20:03] jond	taotetek: there was a guy at a london meetup who's company where pushing data over omq from all the twitter, facebook etc stuff. it was sort of marketing related as far as i could tell....
[20:03] sustrik	ack
[20:03] sustrik	i had a similar impression from the previous sfo meetup
[20:03] pieterh	jond: could it be ViralHeat?
[20:04] jond	pieterh: hang on i'll go and see if still have the card. it might be a few minutes
[20:04] pieterh	it's not important
[20:05] taotetek	hm.
[20:05] sustrik	jond: subscription forwarding is a killer feature for financial services afaics, i would expect to see more users from that area once it gets stable
[20:06] pieterh	sustrik: you have about... 6 months IMO before the AMQP refugees start arriving in volume
[20:06] taotetek	pieterh: not at all really the same thing though is it?
[20:06] sustrik	pieterh: have i missed something?
[20:06] pieterh	yes, a major function in AMQP is the topic exchange, which does this
[20:06] taotetek	pieterh: I've never actually used AMQP myself
[20:06] sustrik	implosion of amqp wg?
[20:07] pieterh	sustrik: well, you've not been following... AMQP/1.0 introduces a totally incompatible new set of semantics
[20:07] pieterh	no exchanges, no queues, no bindings
[20:07] sustrik	yep, i seen that
[20:08] pieterh	I blogged about it: http://www.ipocracy.com/blog:amqp-lulz
[20:08] sustrik	i had a look some 6 months ago
[20:08] pieterh	the point being there's a massive investment in AMQP/0-8 and AMQP/0.9.1 semantics
[20:08] sustrik	1-0 was unusable at that time
[20:08] pieterh	really huge
[20:08] sustrik	basically an empty syntactic shell with no semantics attached
[20:09] pieterh	well, the protocol is more concrete today
[20:09] pieterh	I assume it will become usable at some stage, that's just a question of time and money
[20:10] pieterh	I'm pretty sure people haven't realized what's coming in 1.0, and will be seriously annoyed when they find out
[20:10] pieterh	it will probably take more than 6 months, since RabbitMQ will hide the coming change
[20:10] sustrik	let's see
[20:10] pieterh	imagine 0MQ/3.0 threw out all the socket patterns and started again with a totally new set...
[20:11] pieterh	:-)
[20:11] taotetek	pieterh: that would make me cry.
[20:11] pieterh	yes, exactly
[20:11] pieterh	me too
[20:11] Seta00	haha
[20:11] jond	pieterh: i can't find the card but I'm pretty sure it was http://datasift.net
[20:11] pieterh	so AMQP/1.0 is going to make a lot of people cry
[20:11] sustrik	it was a silly move back then
[20:11] pieterh	I think AMQP was the first real open source messaging ever
[20:11] sustrik	one wonders whether it was sheer incompetence
[20:12] pieterh	silly, yes :)
[20:12] sustrik	or whether there were other motives involved
[20:12] pieterh	yes, incompetence, because there's no plausible benefit
[20:12] pieterh	obviously RHAT wanted to undermine all other vendors but that's still incompetence
[20:13] pieterh	point is, many of these crying users will be looking for a more reliable alternative
[20:13] pieterh	0MQ lacks the topic pubsub functionality that 80% of these folk use
[20:14] taotetek	pieterh: that's part of what we use rsyslog for w/ the 0mq integration we're working on
[20:15] sustrik	lacks pubsub?
[20:15] taotetek	rsyslog doesn't have pubsub, correct
[20:15] pieterh	sustrik: you know what I mean, filtering at the publisher side
[20:15] taotetek	but what it does have is the ability to write chains of rules that are bound to ports
[20:15] sustrik	ah
[20:15] sustrik	i merged the branch today
[20:16] pieterh	:-) yes, I know
[20:16] taotetek	with filtering rules based on various parts of the message
[20:16] sustrik	it'll take some time to mature
[20:16] taotetek	we use rsyslog as our backbone for carrying high volume data and then zmq for distributing data from an output port to distributed processing systems
[20:16] sustrik	the nice thing is that subscriptions are forwarded over multiple hops
[20:16] pieterh	sustrik: do you have a test case for the subscription forwarding?
[20:16] taotetek	we approached zmq much more as a tool for distributed processing than a tool for messaging to begin with
[20:17] sustrik	which solves all kinds of hq/branch office stuff
[20:17] pieterh	sustrik: yes, federation
[20:17] sustrik	pieterh: just a silly publisher/device/subscriber test
[20:18] pieterh	sustrik: if you want people to test it, we need to promote it a little
[20:18] sustrik	funny thing is that if there's no subscriber the messages are not even send from the publisher to the broker :)
[20:18] shykes	pieterh: I was among the chorus asking for service-based subports at the last meetup at dotCloud HQ
[20:18] pieterh	shykes__: :-) hi
[20:18] sustrik	which beats AMQP et al.
[20:18] shykes	I'm happy to say I found a pretty sweet alternative!
[20:18] shykes	https://github.com/guidog/pyzmq-mdp
[20:18] shykes	We're currently testing it at dotcloud
[20:18] pieterh	shykes__: heh, that's Guido's MDP implementation...
[20:19] shykes	exactly what I was looking for :)
[20:19] pieterh	MDP is ok, but needs some polishing wrt heartbeats
[20:19] shykes	at least, on paper
[20:19] shykes	(and in my naive tests)
[20:19] guido_g	did i hear my name? :)
[20:19] pieterh	guido_g: :-) you have a user for https://github.com/guidog/pyzmq-mdp
[20:20] guido_g	not only a user, but also a contributor
[20:20] shykes	guido_g hey! I was just told about mdp this week
[20:20] guido_g	shykes__: did you see the comment to the pull-req?
[20:20] shykes	it's exactly what we needed - I tried to implement the same thing but couldn't figure out how to selectively route on an XREQ/XREP
[20:21] shykes	I guess the answer is - prepend the identity to the message? :)
[20:21] pieterh	shykes__: it's explained in detail in the Guide, where this originated
[20:21] guido_g	hehe, right
[20:21] sustrik	pieterh: there's no change for the user other than you have to use XPUB/XSUB for devices instead of PUB/SUB
[20:21] shykes	guido_g: yes, thanks - I'll push that later today
[20:21] sustrik	the rest is transparent
[20:21] guido_g	shykes__: cool, will then merge it
[20:21] sustrik	examples from the guide should do
[20:21] pieterh	sustrik: any semantic difference from PUB/SUB?
[20:21] sustrik	no
[20:22] shykes	pieterh : so it's fully supported by the specs? Not just a happy coincidence in the zmq implementation? :)
[20:22] pieterh	I'm not keen on the 'X' names, as you know...
[20:22] pieterh	shykes__: it's a fully supported formal spec with two independent implementations (Python and C)
[20:22] pieterh	it's one of the unprotocols at http://rfc.zeromq.org
[20:23] pieterh	shykes__: you really need to read the Guide in detail, otherwise it won't make sense
[20:23] guido_g	shykes__: if you have questions about the implementation, don't hesitate to ask
[20:23] shykes	pieterh: I did - I'm fairly familiar with all the topologies. I guess I missed the part about custom routing topologies
[20:23] taotetek	I've officially incorporated "urban protocol" into one of our zeromq projects at work
[20:23] pieterh	sustrik: a while back you were arguing that Xsomething meant "internal plumbing only"... :)
[20:24] pieterh	so I'd recommend using PUB/SUB with socket options
[20:24] sustrik	yes, these are socket types to be used in devices
[20:24] pieterh	actually only on the SUB side
[20:24] sustrik	intermediate nodes
[20:24] taotetek	the guys from the dev group looked a little confused when I told them to send "NOM"s intead of "ACK"s
[20:24] pieterh	taotetek: lol
[20:25] pieterh	taotetek: I'm not 100% sure it's a serious tool, but it's fun anyhow
[20:26] pieterh	sustrik: so you can't do subscription forwarding without a broker?
[20:26] sustrik	you can
[20:26] sustrik	it's hop-agnostic
[20:26] pieterh	so then you use XSUB in the client?
[20:26] sustrik	nope
[20:26] pieterh	so if SUB talks to XPUB, that does the magic?
[20:26] sustrik	how it works is that there are two layers in the stack
[20:27] pieterh	and SUB to PUB does forwarding?
[20:27] sustrik	hop-by-hop layer (the X- sockets)
[20:27] sustrik	and end-to-end layer (non-X sockets)
[20:27] sustrik	so you have a chain of hops
[20:27] pieterh	sustrik: if you explain that I have to take the Guide examples and use XSUB/XPUB
[20:27] pieterh	realize that the Guide examples do not do hops
[20:27] sustrik	then use PUB/SUB
[20:27] sustrik	it'll work ok
[20:28] sustrik	XPUB/XSUB is only for deviceas
[20:28] pieterh	OK, makes sense
[20:37] sustrik	one more use case for subscription forwarding is logging
[20:37] sustrik	many PUBs, one SUB
[20:38] sustrik	if the logger is not interested in certain class of messages
[20:38] sustrik	these are not even passed over the network
[20:40] mikko	howdy
[20:40] sustrik	hi
[20:44] pieterh	hi mikko
[20:45] guido_g	cu
[20:48] guido_g	night
[20:56] iFire	sustrik is it a common thing to just have a enum listing the message type or a string?
[20:56] iFire	how else do you identify a message
[20:57] iFire	oops
[20:57] iFire	hmm
[20:57] iFire	I mean in the encoding
[20:57] sustrik	with relation to PUB/SUB you mean?
[20:57] iFire	yeah
[20:57] iFire	well messages in general
[20:57] sustrik	the matching is done on the beginning of he message
[20:58] sustrik	so you can place an integer there
[20:58] sustrik	or a string
[20:58] sustrik	it's up to you
[20:59] sustrik	the nice thing with strings is that you can form a hierarchy of message types
[20:59] sustrik	say, animals.mammals.dogs
[21:00] iFire	so I can do it within zero and the not the message encoding
[21:00] sustrik	sorry?
[21:00] iFire	sorry
[21:00] iFire	:(
[21:01] iFire	I mean within the zeromq say pub sub message envelopes
[21:01] iFire	and not say in protobuf
[21:01] sustrik	it's up to you
[21:02] sustrik	you can match messages on whatever is at the beginning of the message
[21:02] iFire	is that the envelopes thing?
[21:03] iFire	uh multipart messages for pub sub
[21:03] sustrik	the matching is done on the first message part
[21:03] iFire	yeah. I mean protobuf is in binary
[21:03] sustrik	so you can place the "message type" to the first message part
[21:04] sustrik	you can match on binary data as well
[21:04] iFire	zeromq won't have any idea without decoding it
[21:04] iFire	hmm?
[21:04] sustrik	but you have to know how does it looks like
[21:04] sustrik	so, if protobufs hide the actual binary layout, you'll have a hard time doing matching
[21:06] iFire	well I'd assume there'll be a binary string to message thing
[21:07] iFire	I'll have to assume that the first message is something I have hard coded to store metadata about the full message
[21:09] sustrik	yes, it's called "topic"
[21:09] sustrik	you can think of it as of "message type"
[21:11] sustrik	cyl
[22:05] acts_as	Hi, I've got some relatively basic questions about how to make use of ZeroMQ in the real world, so apologies before hand. I've read the guide and looked at some sample code.
[22:06] acts_as	How do people setup a 0MQ server? I mean, it looks like you have X language with 0MQ bindings and you just write the server using that. So if your code chokes for some reason, you're SOL?
[22:08] acts_as	0MQ is really interesting to me especially for its REQ/REP... But I'm just wondering how I would design a 0MQ service in a volatile environment. Or how would I distribute load? Just have N 0MQ services and use a traditional load balancer?
[22:08] taotetek	acts_as: REQ / REP has built in load balancing to answer your load balancing question specifically.
[22:44] acts_as	taotetek: You mean, you say how many connections you can handle and zeromq will queue them up for you, etc?
[22:47] taotetek	acts_as: if you have 1 REQ and 5 REP, then your requests will be round robin load balanced across the REP sockets
[22:48] taotetek	acts_as: and if you have 5 REQ and 1 REP, then your requests will be fair queued
[22:48] taotetek	acts_as: PUSH / PULL work the same way
[22:49] acts_as	taotetek: Oh, so... If I write a zmq-based server and run 5 instances of it, pointing to the same port, zmq will handle that?
[23:10] acts_as	nm to that