Monday June 14, 2010

[Time] NameMessage
[04:16] Gambit- evenin' chaps
[04:17] Gambit- I'm trying to make the architectural decision between zeromq and rabbitmq. Anyone care to discuss, or have references to cite?
[04:18] sustrik hi
[04:18] sustrik depends on your scenario
[04:18] Gambit- evenin' sustrik
[04:18] Gambit- indeed it does
[04:18] sustrik what's your use case?
[04:18] Gambit- Would you like a description?
[04:18] sustrik in broad terms?
[04:21] Gambit- Topology: a cluster of widely distributed devices with no mutual connectivity contracts. Each device has several different active entities that may need to communiciate with a varying subset of other device in the cluster.
[04:21] sustrik SOA-style?
[04:22] Gambit- You'll have to explain what you mean by that.
[04:22] sustrik ok, other way round: what industry?
[04:22] sustrik financials/embedded/supercomputing etc.
[04:23] Gambit- technically speaking, we're a major traffic manipulation company and we're supporting clusters of our devices spread across customer installations communicating and coordinating with eachother in heirarchies of active-active groups.
[04:24] Gambit- I can totally be more specific, but I'm still learning how to phrase the question, if you understand :)
[04:25] sustrik yep
[04:25] sustrik so the whole system is out of scope of a single entrerprise
[04:25] Gambit- it may all be owned by one enterprise.
[04:25] Gambit- but it might be spread across many divisions
[04:26] sustrik can there be a singe central point?
[04:26] Gambit- our customers tend to do crazy things we don't expect, so any combination around there is likely.
[04:26] sustrik messaging "broker"?
[04:26] sustrik single*
[04:26] Gambit- It's possible - though more likely there will be an active-active type point where there are multiple boxes that all act as failover points for eachother.
[04:27] sustrik but in overall, all is centralised
[04:27] sustrik all the failover boxes are at the same place, doing same thing
[04:27] Gambit- in that case, yes.
[04:27] sustrik well, in such case i would prefer rabbitmq
[04:28] Gambit- in other cases, there may be, say, 5 boxes in an active-active-active-active-active configuration without any particular head end.
[04:29] Gambit- What is your logic? Also, what is your experience? :)
[04:29] sustrik as far as I understand your scenario is a typical corporate scenario with centralised control and no need for strange topologies
[04:29] sustrik ah, one more thing:
[04:29] sustrik expected performance?
[04:29] Gambit- well we may have boxes taht will have no connectivity for long periods of time, and then connect in.
[04:30] Gambit- also, strange topologies will be mitigated most likely by having an overlay network.
[04:30] Gambit- Performance is something of a elephant in the room at the moment.
[04:30] sustrik any estimates?
[04:30] Gambit- Can you provide some buckets?
[04:30] sustrik in orders of magnitude?
[04:30] sustrik 10 msgs./sec?
[04:31] sustrik 1000000 msgs./sec?
[04:31] Gambit- :)
[04:31] sustrik sub millisecond latency?
[04:31] Gambit- I'm going to SWAG and say in the 100-1000/s range.
[04:31] sustrik etc.
[04:31] Gambit- but that's probably awefully high.
[04:32] Gambit- I don't think there's any expectation for latency less than 100ms.
[04:32] sustrik ok, rabbitmq looks to fit your case
[04:32] Gambit- though our usecases are so fuzzy that it's hard to say for certain.
[04:32] Gambit- we know we need some infrastructure, but we only know todays requirements.
[04:32] sustrik the rationale is that rabbitmq is a typical corporate middleware
[04:33] sustrik you'll have central server to control
[04:33] sustrik management tools
[04:33] sustrik decent performance etc.
[04:33] Gambit- Well there may be no central server, there may just be 5-10 boxes, all peers. But they could all be 'central' servers, I would expect.
[04:33] sustrik 0mq is more adequate for extreme use cases
[04:34] Gambit- can you establish your credentials and authority, for a moment?
[04:34] sustrik i'm 0mq's author :)
[04:34] Gambit- god I love irc :D
[04:35] sustrik the point is that the two products are quite different in what they offer
[04:35] sustrik 0mq:
[04:35] sustrik - it's a library
[04:35] sustrik - fully automatic, no management, like TCP
[04:35] sustrik - very fast
[04:35] sustrik - less feature complete
[04:35] sustrik rabbitmq:
[04:35] sustrik - it's a server
[04:36] sustrik - you manage it as any other server
[04:36] sustrik - performance is lower but pretty decent
[04:36] sustrik - more feature complete
[04:37] Gambit- Whereabouts would you draw the performance line?
[04:37] sustrik 50000 msgs/sec
[04:38] sustrik ah, it also depends on the size of messages
[04:38] sustrik 1kb vs. 100GB
[04:39] sustrik for later i would go with 0mq
[04:39] Gambit- I'm going to say we're in the 5-50k range
[04:39] sustrik yes, so you have no performance problem
[04:39] sustrik both solutions would be OK
[04:39] sustrik in performance terms
[04:40] Gambit- Does 0mq support actual multicast?
[04:40] sustrik yes
[04:41] sustrik PGM in particular
[04:42] Gambit- That's something we'd have to cobble together with rabbitmq, I believe. I don't see any evidence of it being directly supported, though there are several plugins that could be used to bootstrap it.
[04:42] sustrik no, rabbitmq doesn't support it
[04:42] sustrik why do you need it?
[04:42] Gambit- there's some things like 'shovels', which apparently lets you move messages from one queue to another, that could be used to fake it.
[04:43] Gambit- Some nodes may be at the end of long thin mostly broken straws.
[04:43] Gambit- if they have to do a broadcast, it's infeasible to expect them to send messages to everyone else.
[04:43] sustrik that's solved by having broker in the middle, no?
[04:43] sustrik send to broker, broker resends to everyone else
[04:44] Gambit- The way I'd been thinking about it, though, is that you'd need to have a broker on each device.
[04:44] Gambit- That may, obviously, be wrong.
[04:44] Gambit- there are multiple agents on each device that would need to communicate to some subset of the rest of the cluster, after all.
[04:45] Gambit- some orthogonal subset, that is.
[04:45] sustrik device = hw device?
[04:45] Gambit- A device is a linux box, for the sake of this conversation.
[04:45] Gambit- s/sake/intent/
[04:45] sustrik ok
[04:46] sustrik well, you can even use both products if needed
[04:46] sustrik there's nothing to stop you
[04:46] sustrik in your case i would try to play with both for a while to get some feeling
[04:47] Gambit- How's the intercompatability?
[04:47] sustrik none at the moment
[04:48] Gambit- So given what I'd describe, wouldn't I need to run a broker on each device? Especially since they may not have connectivity to any other device?
[04:48] sustrik there are several services on a device, right?
[04:48] Gambit- Correct.
[04:48] sustrik those are running 24/7?
[04:48] Gambit- Yup.
[04:49] sustrik are the messages expected to survive service failure?
[04:50] sustrik or device failure (like power outage)
[04:50] sustrik it's pretty important question
[04:50] Gambit- It's a massively important question.
[04:50] sustrik if you need it, you need a broker on each device
[04:51] Gambit- We already have, essentially, a process that manages all of our configuration and related things.
[04:51] Gambit- Currently we aren't designing for any additional repositories.
[04:51] Gambit- which means we're not expecting anything to survive device failure.
[04:52] Gambit- however, if two agents require a broker to communicate between themselves, then it's absolutely necessary to run a broker on each device, since each device may be running in a standalone configuration.
[04:53] sustrik if you need a brokerless design you have to use 0mq
[04:53] sustrik you have to have some hands-on experience to do a correct decision
[04:53] Gambit- That said, there's no reason we can't run a broker on each box.
[04:53] sustrik but think of 0mq as something like TCP
[04:53] sustrik and rabbitmq of something like SQL server
[04:54] Gambit- That crosses my eyes a bit :)
[04:54] Gambit- but I think I catch your intent.
[04:54] sustrik :)
[04:55] Gambit- How does 0mq handle membership?
[04:55] sustrik membership in what?
[04:55] Gambit- For the use case where, all the way across the cluster, a new device comes up and requests to receive messages of a certain topic.
[04:55] sustrik ah
[04:55] Gambit- Apologies, as always, for incorrect terminology usage.
[04:56] sustrik pseudocode:
[04:56] sustrik s = zmq_socket (SUBSCRIBER);
[04:56] sustrik s.connect ("tcp://myserver:5555");
[04:56] sustrik s.setsockopt (SUBSCRIBE, "");
[04:57] sustrik that's it
[04:57] Gambit- who is 'myserver'?
[04:57] sustrik the peer you want to get messages from
[04:57] Gambit- ah
[04:57] Gambit- so I would have to connect to each peer I wanted to get messages from, and request the subscription?
[04:57] sustrik yes
[04:57] Gambit- How does the multicast fit into that?
[04:58] sustrik s.connect ("pgm://eth0;");
[04:58] Gambit- ah
[04:58] Gambit- interesting.
[04:59] Gambit- so you can use the multicast to get away from having to know specifically who is managing what.
[04:59] sustrik yes
[04:59] sustrik you have to have a node in the middle if you want to do everyone-to-everyone messaging
[05:00] sustrik with IP multicast it's your network switch
[05:00] Gambit- yeah
[05:00] sustrik with tcp, you need the broker
[05:00] Gambit- that ip switch is usually me.
[05:00] Gambit- so we'd have to support that.
[05:00] sustrik it's all on the local box?
[05:01] sustrik then definitely use TCP
[05:01] Gambit- No, sorry, that was poorly said.
[05:01] Gambit- Our product is often used as a load balancer.
[05:01] Gambit- how similar is the wire protocol between 0mq and rabbitmq, btw?
[05:01] Gambit- are you guys expecting to converge at somepoint?
[05:02] sustrik well, rabitmq is a multiprotocol server
[05:02] sustrik so it's shouldn't be hard to plug 0mq into it as an underlying transport
[05:03] sustrik especially that there's an Erlang binding for 0mq recently
[05:03] Gambit- Again with the TCP/SQL Server comparisons :)
[05:04] Gambit- I think I've been thinking of 0mq as another implementation of AMQP, when it's really not.
[05:04] sustrik no, it's not
[05:04] sustrik it's a layer in networking stack that allows you to speak with serveral peers at the same time
[05:04] sustrik that's more or less it
[05:04] Gambit- So... A better PGM?
[05:05] sustrik PGM is only about data distribution
[05:05] Gambit- Not about messages, eh?
[05:05] sustrik it's more about scenarios:
[05:05] sustrik PGM: send this message/data to everyone
[05:06] sustrik 0MQ: 1. dtto
[05:06] sustrik 2. request/reply
[05:06] sustrik 3. load balancing
[05:06] sustrik etc.
[05:06] Gambit- Right.
[05:06] Gambit- Okay, many many thanks for your time. Any other words of wisdom here?
[05:07] sustrik try it yourself, then decide :)
[05:07] Gambit- Right then, thanks :)
[05:16] Gambit- Well, practically speaking I'm likely to delegate a large chunk of this.
[05:16] Gambit- but yeah.
[05:16] Gambit- okay, that's that report put together and sent off.
[05:16] Gambit- Thanks again sustrik, strongly appreciate the dialog. Next time you're in the Seattle area, give me a shout :)
[05:17] sustrik never been there, but i'll should once i get there :)
[05:17] Gambit- it's a nice town :)
[05:17] Gambit- righto, then, ciao!
[05:20] umesh is there any way by which server notifies client while going down in zmq ...
[05:20] umesh ?
[05:24] umesh there is no such way of sending the notification ?
[05:34] sustrik no, there's not, you would need a sync API for that
[05:34] sustrik you can of course send messages by hand over TCP if needed
[05:45] umesh ok
[05:45] umesh so not possible in zermq
[05:46] umesh hand over tcp means ... while going down ...i can send message to peer notifying of it
[05:47] umesh right ?
[06:42] umesh is it safe if I create a seperate thread for recv call ... in server ...
[06:42] umesh means instead of waiting for the recv in hte main process ... I will create thread and wait there ...
[06:45] sustrik hand over tcp = open raw BSD socket, push the data through it
[06:46] sustrik yes, you can recv in a worker thread
[06:46] sustrik the only requirement is that each 0MQ socket is used from exactly one OS thread
[06:47] umesh ok
[07:36] mikko good morning
[07:36] sustrik morning
[09:03] mikko i wrote a http publisher for 0MQ last night
[09:04] mikko but if anyone happens to have need for such a thing it's here
[09:04] mikko it takes a http post and publishes it over 0MQ
[09:07] sustrik mikko: i think we should somehow centralise links for useful stuff of this kind
[09:08] sustrik tools section on the website?
[09:08] sustrik at least mention it on the mailing list so that link won't get lost
[09:31] mikko sustrik: "Misc" maybe
[09:32] mikko i'll push it to github later so it won't get lost
[09:32] sustrik great
[09:32] mikko seems like i can push 6k messages a second with that
[09:32] mikko (testing with ab)
[09:33] sustrik is that good or bad result?
[09:34] mikko it's fairly good
[09:34] mikko doing the same thing over php i might get 800 - 1.2k
[09:34] sustrik ok, i see
[09:34] mikko i don't really know what to compare to
[09:34] sustrik ack
[09:34] mikko maybe even http device for 0MQ might make sense
[09:34] sustrik ok, when you have the gihub repo we'll find some place for the link
[09:35] mikko would allow pretty integration
[09:35] sustrik mikko: i thought about that actually
[09:35] sustrik but i am too lazy to write it :)
[09:36] sustrik wouldn't it make sense to turn your code into a device?
[09:36] mikko i remember seeing a tiny http parsing library somewhere
[09:36] mikko yes, the code currently uses libevent http abstraction
[09:36] mikko moving over to a device would mean leaving that abstraction
[09:36] mikko and parsing the http
[09:36] sustrik so now it sends the http message over 0mq?
[09:37] mikko yep
[09:37] mikko but all http protocol parsing is done by libevent
[09:37] mikko i remember the node.js had a tiny C library for parsing http in async fashion
[09:37] sustrik so the http header is stripped and only payload is passed over 0mq?
[09:38] mikko yes
[09:38] sustrik ok, got it
[09:38] mikko it responds to /publish
[09:38] mikko POST /publish HTTP/1.0
[09:38] mikko Content-Length: 5
[09:38] mikko
[09:38] mikko aaaaa
[09:38] mikko would send aaaaa over to 0mq
[09:39] sustrik can you have a look at the node.js library?
[09:39] mikko yeah, trying to find
[09:41] mikko
[09:41] mikko i think it's that one
[09:41] mikko the readme looks promising
[09:44] sustrik it looks viable
[09:45] mikko i could take a poke to it
[09:45] mikko later today
[09:45] mikko good introduction to devices if nothing else :)
[09:46] sustrik devices are dead simple
[10:30] rgl mikko, have you seen zed ?
[10:33] mikko rgl: yeah, i saw the posting on mailing-lists
[10:33] mikko mongrel however reminds me of Ruby for some reason
[10:36] rgl yes. it's an http server for ruby.
[10:36] rgl thou... mongrel2 is no longer that.
[10:37] rgl mainly mongrel has a nice http parser. which is used in mongrel2.
[10:41] mikko mongrel2 is C ?
[10:43] mikko "Since Mongrel2 will be written in C it should do files pretty well too, but easy application architectures and operations support will come first. "
[11:07] rgl yup.
[12:10] mikko sustrik: forwarder.cpp
[12:10] sustrik yes?
[12:10] mikko sustrik: should that contain zmq_msg_close for the message?
[12:10] sustrik let me see
[12:11] sustrik there's no zmq_msg_close afaics
[12:11] mikko i mean should there be?
[12:11] sustrik it's an infinite loop
[12:12] mikko ah ok
[12:12] mikko btw
[12:12] mikko how do you normally handle errors in constructing the objects inside zeromq2?
[12:12] mikko throw an exception?
[12:12] sustrik out of memory you mean?
[12:12] sustrik assert
[12:13] mikko ok
[12:13] sustrik as for the infinite loop, it should be fixed
[12:13] sustrik i forgot about it
[12:13] sustrik now with 2.0.7 the loop can be exited by terminating the associated context
[13:11] willvarfar so does zeromq handle persistance of messages, mailbox-like?
[13:29] sustrik willvarfar: no, it does not
[13:37] willvarfar what message queue system would you recommend if I needed offline delivery?
[13:53] sustrik rabbitmq
[15:08] rgl sustrik, even the queue device? I mean, doesn't it queue messages until someone can pick them up?
[15:17] sustrik rgl: it does
[15:17] sustrik but it does so in memory
[15:17] sustrik so in case of power failure the messages are loast
[15:17] sustrik lost*
[15:18] rgl ah, non persistence there? ok.
[15:20] mikko sustrik: are there any plans for persistence?
[15:23] sustrik mikko: not near-term plans
[15:24] sustrik on-disk offload (swap) should be added shortly, but no real persistence
[15:24] mikko the queue device looks pretty simple
[15:24] sustrik it is
[15:24] mikko the http device looks like to be more complicated
[15:25] sustrik the http part maybe
[15:25] sustrik the 0mq part would be the same as in queue device afaiu
[15:26] mikko should i inherit http_socket from socket_base?
[15:26] mikko im still not got quite into the code
[15:26] sustrik well, there are two options
[15:26] sustrik either you add http as a new transport mechanism
[15:26] sustrik aside of tcp, inproc et al.
[15:27] sustrik that's pretty complex to implement, but most generic solution
[15:27] sustrik the other option is create a device that would be http<->0mq bridge
[15:28] sustrik http connections are on one side, 0mq connections on the other
[15:28] mikko i was thinking about starting with porting the publisher over to 0mq device
[15:28] sustrik sorry, what publisher?
[15:28] mikko httpublish.c i pasted earlier
[15:29] mikko and then maybe extend to have a req/rep support as well
[15:29] mikko so you could do a http call -> 0mq req -> 0mq rep -> http rep
[15:30] sustrik right, it's a standalone application
[15:30] sustrik so it's the latter solution (the bridge)
[15:49] zedas mikko, rgl: yep, mongrel2 is C.
[15:54] CIA-17 rbzmq: 03Brian Buchanan 07master * r3d19085 10/ rbzmq.cpp : remove support for libzmq < 2.0.7 -
[15:54] CIA-17 rbzmq: 03Brian Buchanan 07master * r92d3af8 10/ (.gitignore zmq.gemspec rbzmq.c rbzmq.cpp): add rdoc documentation -
[15:54] rgl zedas, is there a reason to call it mongrel2? I mean, why use the mongrel moniker?
[16:03] zedas rgl: 1. at the core is the mongrel http parser, which is the main thing. 2. i've wanted to reprise the mongrel server for a long time. 3. the current maintainers of mongrel let it die, which sucks. 4. the name still means "good web server" to a lot of people. 5. i don't believe everything in the universe should fit neatly into a single moniker like a library catalog.
[16:06] rgl zedas, I see. that perfectly acceptable :D
[16:22] mikko zedas: it is plain C?
[16:24] mikko good
[16:24] mikko i'll keep my eye on that
[16:26] zedas mikko: so far.