ZeroMq IRC Log

Monday June 14, 2010

[Time] Name	Message
[04:16] Gambit-	evenin' chaps
[04:17] Gambit-	I'm trying to make the architectural decision between zeromq and rabbitmq. Anyone care to discuss, or have references to cite?
[04:18] sustrik	hi
[04:18] sustrik	depends on your scenario
[04:18] Gambit-	evenin' sustrik
[04:18] Gambit-	indeed it does
[04:18] sustrik	what's your use case?
[04:18] Gambit-	Would you like a description?
[04:18] sustrik	in broad terms?
[04:21] Gambit-	Topology: a cluster of widely distributed devices with no mutual connectivity contracts. Each device has several different active entities that may need to communiciate with a varying subset of other device in the cluster.
[04:21] sustrik	SOA-style?
[04:22] Gambit-	You'll have to explain what you mean by that.
[04:22] sustrik	ok, other way round: what industry?
[04:22] sustrik	financials/embedded/supercomputing etc.
[04:23] Gambit-	technically speaking, we're a major traffic manipulation company and we're supporting clusters of our devices spread across customer installations communicating and coordinating with eachother in heirarchies of active-active groups.
[04:24] Gambit-	I can totally be more specific, but I'm still learning how to phrase the question, if you understand :)
[04:25] sustrik	yep
[04:25] sustrik	so the whole system is out of scope of a single entrerprise
[04:25] Gambit-	it may all be owned by one enterprise.
[04:25] Gambit-	but it might be spread across many divisions
[04:26] sustrik	can there be a singe central point?
[04:26] Gambit-	our customers tend to do crazy things we don't expect, so any combination around there is likely.
[04:26] sustrik	messaging "broker"?
[04:26] sustrik	single*
[04:26] Gambit-	It's possible - though more likely there will be an active-active type point where there are multiple boxes that all act as failover points for eachother.
[04:27] sustrik	but in overall, all is centralised
[04:27] sustrik	all the failover boxes are at the same place, doing same thing
[04:27] Gambit-	in that case, yes.
[04:27] sustrik	well, in such case i would prefer rabbitmq
[04:28] Gambit-	in other cases, there may be, say, 5 boxes in an active-active-active-active-active configuration without any particular head end.
[04:29] Gambit-	What is your logic? Also, what is your experience? :)
[04:29] sustrik	as far as I understand your scenario is a typical corporate scenario with centralised control and no need for strange topologies
[04:29] sustrik	ah, one more thing:
[04:29] sustrik	expected performance?
[04:29] Gambit-	well we may have boxes taht will have no connectivity for long periods of time, and then connect in.
[04:30] Gambit-	also, strange topologies will be mitigated most likely by having an overlay network.
[04:30] Gambit-	Performance is something of a elephant in the room at the moment.
[04:30] sustrik	any estimates?
[04:30] Gambit-	Can you provide some buckets?
[04:30] sustrik	in orders of magnitude?
[04:30] sustrik	10 msgs./sec?
[04:31] sustrik	1000000 msgs./sec?
[04:31] Gambit-	:)
[04:31] sustrik	sub millisecond latency?
[04:31] Gambit-	I'm going to SWAG and say in the 100-1000/s range.
[04:31] sustrik	etc.
[04:31] Gambit-	but that's probably awefully high.
[04:32] Gambit-	I don't think there's any expectation for latency less than 100ms.
[04:32] sustrik	ok, rabbitmq looks to fit your case
[04:32] Gambit-	though our usecases are so fuzzy that it's hard to say for certain.
[04:32] Gambit-	we know we need some infrastructure, but we only know todays requirements.
[04:32] sustrik	the rationale is that rabbitmq is a typical corporate middleware
[04:33] sustrik	you'll have central server to control
[04:33] sustrik	management tools
[04:33] sustrik	decent performance etc.
[04:33] Gambit-	Well there may be no central server, there may just be 5-10 boxes, all peers. But they could all be 'central' servers, I would expect.
[04:33] sustrik	0mq is more adequate for extreme use cases
[04:34] Gambit-	can you establish your credentials and authority, for a moment?
[04:34] sustrik	i'm 0mq's author :)
[04:34] Gambit-	god I love irc :D
[04:35] sustrik	the point is that the two products are quite different in what they offer
[04:35] sustrik	0mq:
[04:35] sustrik	- it's a library
[04:35] sustrik	- fully automatic, no management, like TCP
[04:35] sustrik	- very fast
[04:35] sustrik	- less feature complete
[04:35] sustrik	rabbitmq:
[04:35] sustrik	- it's a server
[04:36] sustrik	- you manage it as any other server
[04:36] sustrik	- performance is lower but pretty decent
[04:36] sustrik	- more feature complete
[04:37] Gambit-	Whereabouts would you draw the performance line?
[04:37] sustrik	50000 msgs/sec
[04:38] sustrik	ah, it also depends on the size of messages
[04:38] sustrik	1kb vs. 100GB
[04:39] sustrik	for later i would go with 0mq
[04:39] Gambit-	I'm going to say we're in the 5-50k range
[04:39] sustrik	yes, so you have no performance problem
[04:39] sustrik	both solutions would be OK
[04:39] sustrik	in performance terms
[04:40] Gambit-	Does 0mq support actual multicast?
[04:40] sustrik	yes
[04:41] sustrik	PGM in particular
[04:42] Gambit-	That's something we'd have to cobble together with rabbitmq, I believe. I don't see any evidence of it being directly supported, though there are several plugins that could be used to bootstrap it.
[04:42] sustrik	no, rabbitmq doesn't support it
[04:42] sustrik	why do you need it?
[04:42] Gambit-	there's some things like 'shovels', which apparently lets you move messages from one queue to another, that could be used to fake it.
[04:43] Gambit-	Some nodes may be at the end of long thin mostly broken straws.
[04:43] Gambit-	if they have to do a broadcast, it's infeasible to expect them to send messages to everyone else.
[04:43] sustrik	that's solved by having broker in the middle, no?
[04:43] sustrik	send to broker, broker resends to everyone else
[04:44] Gambit-	The way I'd been thinking about it, though, is that you'd need to have a broker on each device.
[04:44] Gambit-	That may, obviously, be wrong.
[04:44] Gambit-	there are multiple agents on each device that would need to communicate to some subset of the rest of the cluster, after all.
[04:45] Gambit-	some orthogonal subset, that is.
[04:45] sustrik	device = hw device?
[04:45] Gambit-	A device is a linux box, for the sake of this conversation.
[04:45] Gambit-	s/sake/intent/
[04:45] sustrik	ok
[04:46] sustrik	well, you can even use both products if needed
[04:46] sustrik	there's nothing to stop you
[04:46] sustrik	in your case i would try to play with both for a while to get some feeling
[04:47] Gambit-	How's the intercompatability?
[04:47] sustrik	none at the moment
[04:48] Gambit-	So given what I'd describe, wouldn't I need to run a broker on each device? Especially since they may not have connectivity to any other device?
[04:48] sustrik	there are several services on a device, right?
[04:48] Gambit-	Correct.
[04:48] sustrik	those are running 24/7?
[04:48] Gambit-	Yup.
[04:49] sustrik	are the messages expected to survive service failure?
[04:50] sustrik	or device failure (like power outage)
[04:50] sustrik	it's pretty important question
[04:50] Gambit-	It's a massively important question.
[04:50] sustrik	if you need it, you need a broker on each device
[04:51] Gambit-	We already have, essentially, a process that manages all of our configuration and related things.
[04:51] Gambit-	Currently we aren't designing for any additional repositories.
[04:51] Gambit-	which means we're not expecting anything to survive device failure.
[04:52] Gambit-	however, if two agents require a broker to communicate between themselves, then it's absolutely necessary to run a broker on each device, since each device may be running in a standalone configuration.
[04:53] sustrik	if you need a brokerless design you have to use 0mq
[04:53] sustrik	you have to have some hands-on experience to do a correct decision
[04:53] Gambit-	That said, there's no reason we can't run a broker on each box.
[04:53] sustrik	but think of 0mq as something like TCP
[04:53] sustrik	and rabbitmq of something like SQL server
[04:54] Gambit-	That crosses my eyes a bit :)
[04:54] Gambit-	but I think I catch your intent.
[04:54] sustrik	:)
[04:55] Gambit-	How does 0mq handle membership?
[04:55] sustrik	membership in what?
[04:55] Gambit-	For the use case where, all the way across the cluster, a new device comes up and requests to receive messages of a certain topic.
[04:55] sustrik	ah
[04:55] Gambit-	Apologies, as always, for incorrect terminology usage.
[04:56] sustrik	pseudocode:
[04:56] sustrik	s = zmq_socket (SUBSCRIBER);
[04:56] sustrik	s.connect ("tcp://myserver:5555");
[04:56] sustrik	s.setsockopt (SUBSCRIBE, "ab.cd.");
[04:57] sustrik	that's it
[04:57] Gambit-	who is 'myserver'?
[04:57] sustrik	the peer you want to get messages from
[04:57] Gambit-	ah
[04:57] Gambit-	so I would have to connect to each peer I wanted to get messages from, and request the subscription?
[04:57] sustrik	yes
[04:57] Gambit-	How does the multicast fit into that?
[04:58] sustrik	s.connect ("pgm://eth0;224.0.0.1:5555");
[04:58] Gambit-	ah
[04:58] Gambit-	interesting.
[04:59] Gambit-	so you can use the multicast to get away from having to know specifically who is managing what.
[04:59] sustrik	yes
[04:59] sustrik	you have to have a node in the middle if you want to do everyone-to-everyone messaging
[05:00] sustrik	with IP multicast it's your network switch
[05:00] Gambit-	yeah
[05:00] sustrik	with tcp, you need the broker
[05:00] Gambit-	that ip switch is usually me.
[05:00] Gambit-	so we'd have to support that.
[05:00] sustrik	it's all on the local box?
[05:01] sustrik	then definitely use TCP
[05:01] Gambit-	No, sorry, that was poorly said.
[05:01] Gambit-	Our product is often used as a load balancer.
[05:01] Gambit-	how similar is the wire protocol between 0mq and rabbitmq, btw?
[05:01] Gambit-	are you guys expecting to converge at somepoint?
[05:02] sustrik	well, rabitmq is a multiprotocol server
[05:02] sustrik	so it's shouldn't be hard to plug 0mq into it as an underlying transport
[05:03] sustrik	especially that there's an Erlang binding for 0mq recently
[05:03] Gambit-	Again with the TCP/SQL Server comparisons :)
[05:04] Gambit-	I think I've been thinking of 0mq as another implementation of AMQP, when it's really not.
[05:04] sustrik	no, it's not
[05:04] sustrik	it's a layer in networking stack that allows you to speak with serveral peers at the same time
[05:04] sustrik	that's more or less it
[05:04] Gambit-	So... A better PGM?
[05:05] sustrik	PGM is only about data distribution
[05:05] Gambit-	Not about messages, eh?
[05:05] sustrik	it's more about scenarios:
[05:05] sustrik	PGM: send this message/data to everyone
[05:06] sustrik	0MQ: 1. dtto
[05:06] sustrik	2. request/reply
[05:06] sustrik	3. load balancing
[05:06] sustrik	etc.
[05:06] Gambit-	Right.
[05:06] Gambit-	Okay, many many thanks for your time. Any other words of wisdom here?
[05:07] sustrik	try it yourself, then decide :)
[05:07] Gambit-	Right then, thanks :)
[05:16] Gambit-	Well, practically speaking I'm likely to delegate a large chunk of this.
[05:16] Gambit-	but yeah.
[05:16] Gambit-	okay, that's that report put together and sent off.
[05:16] Gambit-	Thanks again sustrik, strongly appreciate the dialog. Next time you're in the Seattle area, give me a shout :)
[05:17] sustrik	never been there, but i'll should once i get there :)
[05:17] Gambit-	it's a nice town :)
[05:17] Gambit-	righto, then, ciao!
[05:20] umesh	is there any way by which server notifies client while going down in zmq ...
[05:20] umesh	?
[05:24] umesh	there is no such way of sending the notification ?
[05:34] sustrik	no, there's not, you would need a sync API for that
[05:34] sustrik	you can of course send messages by hand over TCP if needed
[05:45] umesh	ok
[05:45] umesh	so not possible in zermq
[05:46] umesh	hand over tcp means ... while going down ...i can send message to peer notifying of it
[05:47] umesh	right ?
[06:42] umesh	is it safe if I create a seperate thread for recv call ... in server ...
[06:42] umesh	means instead of waiting for the recv in hte main process ... I will create thread and wait there ...
[06:45] sustrik	hand over tcp = open raw BSD socket, push the data through it
[06:46] sustrik	yes, you can recv in a worker thread
[06:46] sustrik	the only requirement is that each 0MQ socket is used from exactly one OS thread
[06:47] umesh	ok
[07:36] mikko	good morning
[07:36] sustrik	morning
[09:03] mikko	i wrote a http publisher for 0MQ last night
[09:04] mikko	but if anyone happens to have need for such a thing it's here http://valokuva.org/~mikko/code/httpublish/
[09:04] mikko	it takes a http post and publishes it over 0MQ
[09:07] sustrik	mikko: i think we should somehow centralise links for useful stuff of this kind
[09:08] sustrik	tools section on the website?
[09:08] sustrik	at least mention it on the mailing list so that link won't get lost
[09:31] mikko	sustrik: "Misc" maybe
[09:32] mikko	i'll push it to github later so it won't get lost
[09:32] sustrik	great
[09:32] mikko	seems like i can push 6k messages a second with that
[09:32] mikko	(testing with ab)
[09:33] sustrik	is that good or bad result?
[09:34] mikko	it's fairly good
[09:34] mikko	doing the same thing over php i might get 800 - 1.2k
[09:34] sustrik	ok, i see
[09:34] mikko	i don't really know what to compare to
[09:34] sustrik	ack
[09:34] mikko	maybe even http device for 0MQ might make sense
[09:34] sustrik	ok, when you have the gihub repo we'll find some place for the link
[09:35] mikko	would allow pretty integration
[09:35] sustrik	mikko: i thought about that actually
[09:35] sustrik	but i am too lazy to write it :)
[09:36] sustrik	wouldn't it make sense to turn your code into a device?
[09:36] mikko	i remember seeing a tiny http parsing library somewhere
[09:36] mikko	yes, the code currently uses libevent http abstraction
[09:36] mikko	moving over to a device would mean leaving that abstraction
[09:36] mikko	and parsing the http
[09:36] sustrik	so now it sends the http message over 0mq?
[09:37] mikko	yep
[09:37] mikko	but all http protocol parsing is done by libevent
[09:37] mikko	i remember the node.js had a tiny C library for parsing http in async fashion
[09:37] sustrik	so the http header is stripped and only payload is passed over 0mq?
[09:38] mikko	yes
[09:38] sustrik	ok, got it
[09:38] mikko	it responds to /publish
[09:38] mikko	POST /publish HTTP/1.0
[09:38] mikko	Content-Length: 5
[09:38] mikko
[09:38] mikko	aaaaa
[09:38] mikko	would send aaaaa over to 0mq
[09:39] sustrik	can you have a look at the node.js library?
[09:39] mikko	yeah, trying to find
[09:41] mikko	http://github.com/ry/http-parser
[09:41] mikko	i think it's that one
[09:41] mikko	the readme looks promising
[09:44] sustrik	it looks viable
[09:45] mikko	i could take a poke to it
[09:45] mikko	later today
[09:45] mikko	good introduction to devices if nothing else :)
[09:46] sustrik	devices are dead simple
[10:30] rgl	mikko, have you seen zed mongrel2.org ?
[10:33] mikko	rgl: yeah, i saw the posting on mailing-lists
[10:33] mikko	mongrel however reminds me of Ruby for some reason
[10:36] rgl	yes. it's an http server for ruby.
[10:36] rgl	thou... mongrel2 is no longer that.
[10:37] rgl	mainly mongrel has a nice http parser. which is used in mongrel2.
[10:41] mikko	mongrel2 is C ?
[10:43] mikko	"Since Mongrel2 will be written in C it should do files pretty well too, but easy application architectures and operations support will come first. "
[11:07] rgl	yup.
[12:10] mikko	sustrik: forwarder.cpp
[12:10] sustrik	yes?
[12:10] mikko	sustrik: should that contain zmq_msg_close for the message?
[12:10] sustrik	let me see
[12:11] sustrik	there's no zmq_msg_close afaics
[12:11] mikko	i mean should there be?
[12:11] sustrik	it's an infinite loop
[12:12] mikko	ah ok
[12:12] mikko	btw
[12:12] mikko	how do you normally handle errors in constructing the objects inside zeromq2?
[12:12] mikko	throw an exception?
[12:12] sustrik	out of memory you mean?
[12:12] sustrik	assert
[12:13] mikko	ok
[12:13] sustrik	as for the infinite loop, it should be fixed
[12:13] sustrik	i forgot about it
[12:13] sustrik	now with 2.0.7 the loop can be exited by terminating the associated context
[13:11] willvarfar	so does zeromq handle persistance of messages, mailbox-like?
[13:29] sustrik	willvarfar: no, it does not
[13:37] willvarfar	what message queue system would you recommend if I needed offline delivery?
[13:53] sustrik	rabbitmq
[15:08] rgl	sustrik, even the queue device? I mean, doesn't it queue messages until someone can pick them up?
[15:17] sustrik	rgl: it does
[15:17] sustrik	but it does so in memory
[15:17] sustrik	so in case of power failure the messages are loast
[15:17] sustrik	lost*
[15:18] rgl	ah, non persistence there? ok.
[15:20] mikko	sustrik: are there any plans for persistence?
[15:23] sustrik	mikko: not near-term plans
[15:24] sustrik	on-disk offload (swap) should be added shortly, but no real persistence
[15:24] mikko	the queue device looks pretty simple
[15:24] sustrik	it is
[15:24] mikko	the http device looks like to be more complicated
[15:25] sustrik	the http part maybe
[15:25] sustrik	the 0mq part would be the same as in queue device afaiu
[15:26] mikko	should i inherit http_socket from socket_base?
[15:26] mikko	im still not got quite into the code
[15:26] sustrik	well, there are two options
[15:26] sustrik	either you add http as a new transport mechanism
[15:26] sustrik	aside of tcp, inproc et al.
[15:27] sustrik	that's pretty complex to implement, but most generic solution
[15:27] sustrik	the other option is create a device that would be http<->0mq bridge
[15:28] sustrik	http connections are on one side, 0mq connections on the other
[15:28] mikko	i was thinking about starting with porting the publisher over to 0mq device
[15:28] sustrik	sorry, what publisher?
[15:28] mikko	httpublish.c i pasted earlier
[15:29] mikko	and then maybe extend to have a req/rep support as well
[15:29] mikko	so you could do a http call -> 0mq req -> 0mq rep -> http rep
[15:30] sustrik	right, it's a standalone application
[15:30] sustrik	so it's the latter solution (the bridge)
[15:49] zedas	mikko, rgl: yep, mongrel2 is C.
[15:54] CIA-17	rbzmq: 03Brian Buchanan 07master * r3d19085 10/ rbzmq.cpp : remove support for libzmq < 2.0.7 - http://bit.ly/d26OdF
[15:54] CIA-17	rbzmq: 03Brian Buchanan 07master * r92d3af8 10/ (.gitignore zmq.gemspec rbzmq.c rbzmq.cpp): add rdoc documentation - http://bit.ly/cZzye1
[15:54] rgl	zedas, is there a reason to call it mongrel2? I mean, why use the mongrel moniker?
[16:03] zedas	rgl: 1. at the core is the mongrel http parser, which is the main thing. 2. i've wanted to reprise the mongrel server for a long time. 3. the current maintainers of mongrel let it die, which sucks. 4. the name still means "good web server" to a lot of people. 5. i don't believe everything in the universe should fit neatly into a single moniker like a library catalog.
[16:06] rgl	zedas, I see. that perfectly acceptable :D
[16:22] mikko	zedas: it is plain C?
[16:24] mikko	good
[16:24] mikko	i'll keep my eye on that
[16:26] zedas	mikko: so far.