ZeroMq IRC Log

Friday July 23, 2010

[Time] Name	Message
[13:52] jared	Can anyone tell me what guarantees 0MQ provides for its queues? For example, with pub-sub, are all messages guaranteed to be delivered, and if so, will they be in order?
[13:52] guido_g	i can guarantee that there're none
[13:52] guido_g	pub/sub is like radio broadcast
[13:56] jared	do any of the models provide a guarantee that once I hand the message to 0MQ that it will be delivered........eventually
[13:56] jared	?
[13:57] jared	I am determining whether or not an additional 'message tracking' layer would need to be built on top of 0MQ, should I decide to use it.
[13:58] drbobbeaty	I was under the impression that the pgm and epgm protocols were using reliable multicast from OpenPGM. If so, then you're going to have receipt, but not guaranteed delivery. If the listener isn't on when the message is sent, then it's going ot miss it.
[13:59] drbobbeaty	If you need really "once-and-only-once" delivery, then you need to have a queue where it tracks persistent subscriptions and persistent publishers and that's a lot more work.
[14:00] drbobbeaty	So, from the sounds of it, I'd say the answer to your question is "Yes, you need to write that message tracking layer."
[14:01] drbobbeaty	Then again, I've only been in this codebase for about 3 days now, but having been around the block a few times with other messaging systems, the pieces become very familiar.
[14:01] jared	What I really need is a layer that can guarantee that once the application hands it a message to this layer, destined for a given endpoint, that the other endpoint will see them in order, and eventually.
[14:01] jared	I figured this was a guarantee not provided, since its pretty specific, but just wanted to make sure.
[14:02] drbobbeaty	OK, I'm going to ask a few "thought experiment" questions that will help me understand your requirements.
[14:03] jared	ok, sure.
[14:03] drbobbeaty	Say you have a server and ten clients. When the server sends the first message, only five of those clients are up and listening. An hour later, the other five start up. When the second set start up, do they need to see that message that was sent an hour ago?
[14:04] jared	I am more interested in a pipe between two endpoints only, but the request-reply model seems like it has low performance.
[14:04] drbobbeaty	OK, then let's look at a single server and a single client.
[14:05] guido_g	jared___: any numbers?
[14:05] drbobbeaty	Say the server is sending to the client. The client bounces, but while it's down, the server has sent three messages. Do you need those three messages to be read by the client when it comes up?
[14:05] jared	But for your question, yes, I would expect the down servers to receive those messages. But realistically in my case the connection would have been severed within 10 seconds.
[14:07] jared	Sorry, what I mean by that is that if the other end was truly disconnected, other mechanisms would have detected it, and this machine would then stop caring what messages made it through.
[14:07] jared	guido_g: what do you mean by numbers?
[14:07] guido_g	jared___: benchmarks for example
[14:07] drbobbeaty	OK, if you need to see messages "in the past" then you need a queue that's going to keep track of who has pulled off what message. Basically, a "message broker". These are in a lot of systems, and don't have to be a separate process. If you look at http://www.spread.org/ you'll see something that uses a messaging system that might be more what you're looking for.
[14:08] drbobbeaty	Spread has the delivery guarantee. It's also open source.
[14:09] guido_g	whereas spread also is not a _traditional_ broker based queueing system
[14:09] guido_g	but the process has to be member of the group when the message is sent, afair
[14:09] drbobbeaty	Agreed. Totally. And there are limitations to Spread. Which is why I'm here with ZeroMQ, but it does have the guaranteed delivery.
[14:10] jared	Great, I'll check that out right now. Thanks for your help! I'll stick around here in case I have more 0MQ questions.
[14:11] guido_g	there should be a book about brokerless messaging
[14:12] drbobbeaty	I've wished many times that there were more docs (books, sites, etc.) on this stuff. It's hard to find all this stuff and see the pros and cons easily. I'll ask Santa... he's been good to me in the past.
[14:12] guido_g	great idea :)
[14:34] jared	Bummer, the overhead of spread seems pretty high. I am need _very_ good performance so the group management overhead is probably more than I am willing to pay. I'll keep probing around though.
[14:39] guido_g	performance and delivery gurantees are contradictory in most cases
[14:43] drbobbeaty	Agreed... if you need guarantees, then you need persistence of some kind. That costs time. Regardless of anything else. You're not going to find a fast, reliable persistence. Memory is fast, but doesn't survive reboots. Disk is reliable, but not fast. You have a fundamental issue, here. No easy solution. You're going to have to make some kind of trade-off.
[14:44] guido_g	well phrased
[14:53] drbobbeaty	Thanks :)
[14:54] guido_g	you're welcome
[22:01] tupshin	i'm trying to understand whether zeromq would be a good general purpose message bus for publish/subscribe type exchanges in an environment where there are 1000 publishers and 100 subscribers. any pointers to good info?