Friday July 23, 2010

[Time] NameMessage
[13:52] jared Can anyone tell me what guarantees 0MQ provides for its queues? For example, with pub-sub, are all messages guaranteed to be delivered, and if so, will they be in order?
[13:52] guido_g i can guarantee that there're none
[13:52] guido_g pub/sub is like radio broadcast
[13:56] jared do any of the models provide a guarantee that once I hand the message to 0MQ that it will be delivered........eventually
[13:56] jared ?
[13:57] jared I am determining whether or not an additional 'message tracking' layer would need to be built on top of 0MQ, should I decide to use it.
[13:58] drbobbeaty I was under the impression that the pgm and epgm protocols were using reliable multicast from OpenPGM. If so, then you're going to have receipt, but not guaranteed delivery. If the listener isn't on when the message is sent, then it's going ot miss it.
[13:59] drbobbeaty If you need really "once-and-only-once" delivery, then you need to have a queue where it tracks persistent subscriptions and persistent publishers and that's a lot more work.
[14:00] drbobbeaty So, from the sounds of it, I'd say the answer to your question is "Yes, you need to write that message tracking layer."
[14:01] drbobbeaty Then again, I've only been in this codebase for about 3 days now, but having been around the block a few times with other messaging systems, the pieces become very familiar.
[14:01] jared What I really need is a layer that can guarantee that once the application hands it a message to this layer, destined for a given endpoint, that the other endpoint will see them in order, and eventually.
[14:01] jared I figured this was a guarantee not provided, since its pretty specific, but just wanted to make sure.
[14:02] drbobbeaty OK, I'm going to ask a few "thought experiment" questions that will help me understand your requirements.
[14:03] jared ok, sure.
[14:03] drbobbeaty Say you have a server and ten clients. When the server sends the first message, only five of those clients are up and listening. An hour later, the other five start up. When the second set start up, do they need to see that message that was sent an hour ago?
[14:04] jared I am more interested in a pipe between two endpoints only, but the request-reply model seems like it has low performance.
[14:04] drbobbeaty OK, then let's look at a single server and a single client.
[14:05] guido_g jared___: any numbers?
[14:05] drbobbeaty Say the server is sending to the client. The client bounces, but while it's down, the server has sent three messages. Do you need those three messages to be read by the client when it comes up?
[14:05] jared But for your question, yes, I would expect the down servers to receive those messages. But realistically in my case the connection would have been severed within 10 seconds.
[14:07] jared Sorry, what I mean by that is that if the other end was truly disconnected, other mechanisms would have detected it, and this machine would then stop caring what messages made it through.
[14:07] jared guido_g: what do you mean by numbers?
[14:07] guido_g jared___: benchmarks for example
[14:07] drbobbeaty OK, if you need to see messages "in the past" then you need a queue that's going to keep track of who has pulled off what message. Basically, a "message broker". These are in a lot of systems, and don't have to be a separate process. If you look at you'll see something that uses a messaging system that might be more what you're looking for.
[14:08] drbobbeaty Spread has the delivery guarantee. It's also open source.
[14:09] guido_g whereas spread also is not a _traditional_ broker based queueing system
[14:09] guido_g but the process has to be member of the group when the message is sent, afair
[14:09] drbobbeaty Agreed. Totally. And there are limitations to Spread. Which is why I'm here with ZeroMQ, but it does have the guaranteed delivery.
[14:10] jared Great, I'll check that out right now. Thanks for your help! I'll stick around here in case I have more 0MQ questions.
[14:11] guido_g there should be a book about brokerless messaging
[14:12] drbobbeaty I've wished many times that there were more docs (books, sites, etc.) on this stuff. It's hard to find all this stuff and see the pros and cons easily. I'll ask Santa... he's been good to me in the past.
[14:12] guido_g great idea :)
[14:34] jared Bummer, the overhead of spread seems pretty high. I am need _very_ good performance so the group management overhead is probably more than I am willing to pay. I'll keep probing around though.
[14:39] guido_g performance and delivery gurantees are contradictory in most cases
[14:43] drbobbeaty Agreed... if you need guarantees, then you need persistence of some kind. That costs time. Regardless of anything else. You're not going to find a fast, reliable persistence. Memory is fast, but doesn't survive reboots. Disk is reliable, but not fast. You have a fundamental issue, here. No easy solution. You're going to have to make some kind of trade-off.
[14:44] guido_g well phrased
[14:53] drbobbeaty Thanks :)
[14:54] guido_g you're welcome
[22:01] tupshin i'm trying to understand whether zeromq would be a good general purpose message bus for publish/subscribe type exchanges in an environment where there are 1000 publishers and 100 subscribers. any pointers to good info?