Friday April 15, 2011

[Time] NameMessage
[05:33] sustrik lestrrat: i wouldn't move to 3.0 yet
[05:33] sustrik it's not fully stable yet
[05:48] lestrrat well, not so much for really moving, but I want to silence the test failures (at least for now)
[05:54] sustrik ok, i see
[05:54] sustrik as for devices the plan is to move them to a separate project
[06:05] lestrrat k, I guess I'd need to create a separate binding for it, then
[06:05] sustrik ok, no haste needed, though
[06:06] lestrrat yeah, I'll just skip the code that uses it.
[06:48] lestrrat mikko: can you please point the jenkins tests to look at "libzmq3" branch for the perl binding + 0MQ 3.x? master is currently for 2.1.x
[06:48] lestrrat I started changing stuff for zmq3 on libzmq3 branch.
[07:04] mikko lestrrat: will do
[07:04] mikko give me a sec
[07:05] mikko lestrrat: building now
[07:05] lestrrat mikko: it will fail :)
[07:05] mikko it did
[07:06] mikko but the build should be now configured correctly
[07:06] mikko so it's a good start
[07:06] lestrrat cool :)
[07:25] pieterh lestrrat: we should backport the fixes to the tests to 2.x
[07:26] pieterh mikko: did you get a chance to look at the rhat packaging for 2.x?
[07:55] mikko i to the b
[13:46] yawniek hi, from the examples i try to run but the code where the ClientTask receives the answers is never executed. any hints?
[13:53] yawniek talking about
[13:57] pieterh yawniek: hi
[13:57] pieterh how far does the request get?
[14:00] yawniek the worker sends the replies
[14:00] pieterh you see them in the server?
[14:00] yawniek yes
[14:00] pieterh so the normal problem here is with the XREP/ROUTER socket
[14:00] pieterh which silently drops messages if the address is wrong
[14:01] pieterh you want to print/trace the first frame (message part) that the server sends back to the client
[14:01] pieterh make sure it's a valid identity
[14:02] pieterh see
[14:03] yawniek ok thanks
[14:08] yawniek pieterh: to trace i need to receive multipart and print all parts?
[14:08] pieterh yawniek: no idea how this works in Python, sorry... but that sounds right
[14:08] mikko hello
[14:08] pieterh in any case you need to receive / send multipart if you're using a XREP socket
[14:09] pieterh hi mikko!
[14:09] mikko been pretty exciting time in .nl
[14:10] pieterh conference?
[14:10] mikko im stalking people
[14:10] pieterh lol
[14:10] mikko using kinect sensor
[14:10] mikko
[15:23] Lemmih Hiya, say I want a bunch of nodes to agree on the ordering of messages. Should I use N publishers and N subscribers together with some logic for ordering the messages or is there an easier way?
[15:24] pieterh messages produced by multiple nodes?
[15:24] Lemmih Yes.
[15:24] pieterh only simple way is to push them all through a single point
[15:24] Lemmih That would give me a single point of failure, no?
[15:25] pieterh you have another way of ordering messages across multiple nodes?
[15:27] Lemmih There are a couple of different ways to do it but I don't know enough about publisher/subscriber semantics to know how to go about it.
[15:28] pieterh how would you do it?
[15:29] pieterh Lemmih: you need to explain what you mean by "agree on the ordering"
[15:31] pieterh However note that if you have multiple publishers and multiple subscribers talking N to N
[15:31] pieterh then each subscriber will get messages from different publishers in an arbitrary order
[15:31] Lemmih I want all nodes to see the messages in the same order. The exact order doesn't matter as long as it is consistent across each node.
[15:32] pieterh if you want a single stream of messages, simplest is a central node to create that ordering
[15:32] locklace Lemmih: sequence numbers
[15:32] pieterh you might plausibly reorder within each client but it's still not robust
[15:33] pieterh locklace: sequence numbers won't be unique if there are multiple publishers
[15:33] locklace timestamps then
[15:33] pieterh timestamps won't be accurate across a network
[15:33] pieterh not at high message rates
[15:33] Lemmih They don't have to be.
[15:33] locklace they will if you're doing proper time sync, as you should be
[15:34] pieterh not accurate down to the microsecond afaik
[15:34] pieterh Lemmih: what 0MQ will do for you is e.g. fair queuing from all publishers
[15:34] locklace depends on the time distribution method
[15:35] locklace but it's not clear what the precision requirement is for this application
[15:37] pieterh Lemmih: can you provide more background on your use case?
[15:38] Lemmih pieterh: Alright. I'm fine with ordering the messages myself. Just wanted to make sure I didn't reinvent the wheel.
[15:39] Lemmih I have a transaction queue which has to be identical on each node. And each node should be able to push data to the transaction queue.
[15:40] pieterh Hmm, nice problem, and you want to work without a central device
[15:41] pieterh If your transaction rate is not too high you can use locklace's timestamp method
[15:42] cremes ntp is usually good for +/- 1 millisecond on a LAN
[15:42] cremes i wouldn't count on it for microsecond or nanosecond granularity
[15:42] pieterh so it also depends on the speed of processing of the transaction queue
[15:42] pieterh if it's fast you won't be able to reorder properly
[15:42] Lemmih I can do the ordering using timestamps (still works when the clocks differs, just adds latency) or non-local sequence numbers. However, since some of the other messaging libraries I've used in the path provided this functionality by default, I wanted to make sure I didn't rewrite something that's already in 0MQ. Those other messaging systems wheren't brokerless, though.
[15:43] pieterh Lemmih: 0MQ is a rocket engine but without a chassis
[15:43] Lemmih Oops, meant to write 'non-unique', not 'non-local'.
[15:44] pieterh how do you reorder when you've already processed a transaction?
[15:44] pieterh i.e. two clients get B, A, and A, B
[15:45] Lemmih I don't. I keep the messages in a queue until they can be ordered.
[15:46] Lemmih If A and B have timestamps and I know that the publishers can't/won't send any messages younger than the oldest of A and B, then A and B has been assigned order globally.
[15:47] Lemmih This can also be done using ticks instead of timestamps.
[15:47] pieterh so you either timeout per publisher, or wait for another message to arrive
[15:49] pieterh in fact you can do it without timestamps at all
[15:49] pieterh as long as you have unique publisher ids
[15:49] Lemmih Right.
[15:49] Lemmih And I do have that.
[15:49] pieterh but it's going to add latency... and still be breakable
[15:49] Lemmih Breakable how?
[15:50] pieterh if a message gets stuck somewhere, for longer than your timeout, the order can become bogus
[15:50] Lemmih Is it possible for a publisher to send two messages and have the second arrive before the first?
[15:51] pieterh no, but two messages from two publishers will arrive in arbitrary order
[15:51] pieterh obviously
[15:51] pieterh from one publisher, it's a TCP stream
[15:51] pieterh also you have no way to handle a crashing publisher
[15:51] pieterh i.e. which has sent to some clients but not to others
[15:53] Lemmih I do, actually, but that's due to the way I use the transaction log.
[15:54] pieterh it'd be interesting to see your solution in print, once you get it working with 0MQ
[15:54] pieterh it sounds like it'll work easily
[15:54] Lemmih Just receiving a message isn't enough to consider it commited. It also has to be confirmed durable on a supermajority of the nodes.
[15:55] pieterh I still have my doubts about the design but that's just because I don't see enough of it
[15:55] pieterh you're basically doing atomic multicast iiuc
[15:56] guido_g did somone say lamport? :)
[15:56] Lemmih Well, I'll get back to you when it works (: Btw, I'm writing this code in Haskell.
[15:57] pieterh guido_g: sounds like it :)
[15:57] pieterh Lemmih: I assume whatever pattern you build will work in any language over 0MQ
[15:59] guido_g using vector clocks?
[16:00] guido_g ah ok
[16:00] guido_g will wait then
[16:00] pieterh Lemmih: I'm interested in your work, will it be open source?
[16:02] Lemmih pieterh: Oh, you wrote "The Guide"? Damn that thing is long. I just spent most of my day reading it. (:
[16:02] Lemmih pieterh: Yes, it will be open source.
[16:02] pieterh Lemmih: lol, it's my work, yes, with the help of lots of people translating the examples
[16:02] pieterh there is a lot of detail in some patterns
[16:02] Lemmih I might contribute a few Haskell examples along the way.
[16:02] pieterh I'm doing Clone, and have six models to show the evolution
[16:03] pieterh Haskell examples would be great, the language isn't well represented today
[16:10] Lemmih FYI, I'm working on adding multimaster replication to a database system written in Haskell.
[16:12] pieterh Lemmih: there's a database written in Haskell?
[16:12] pieterh neat
[16:13] pieterh BTW, I'm really happy to hear you're using 0MQ for this, it should be a perfect fit
[16:14] Lemmih Calling it a database is probably a bit generous. It's a system for adding ACID guarantees to native-language data-types. Like Prevaylor if you've heard of it.
[16:15] Lemmih Yeah, I really like 0MQ for this. Feels much better than the other message systems I've tried in the past.
[16:16] pieterh yeah
[16:23] iAmTheDave Not sure this is the forum, but we just released a Django message queue app using 0MQ.
[16:24] iAmTheDave It was so nice to work with 0MQ, it was so light-weight...
[16:24] pieterh iAmTheDave: nice!
[16:25] iAmTheDave pieterh: it totally is. we were using celery and all the cruft that goes along with it (not knocking it) but we wanted something WAY smaller.
[16:25] pieterh iAmTheDave: would you send me a short story of how this happened?
[16:25] iAmTheDave so like, thanks for 0MQ!
[16:25] pieterh I'd like to publish that on our wiki somewhere
[16:26] pieterh a testimonial...
[16:26] iAmTheDave yeah you bet. that'd be awesome!
[16:26] pieterh :)
[16:26] pieterh email it to
[16:26] iAmTheDave can do
[16:26] pieterh I'll make sure your project gets coverage
[16:26] pieterh have already twitted about it
[16:27] pieterh next step is to update the zeromq myspace page!
[16:27] iAmTheDave oh wow. thanks :) didn't expect all that, just wanted to let you know we're really happy with it
[16:27] locklace it's just like 2003
[16:28] locklace extra points for picking the right twitter verb too
[16:29] pieterh locklace: I tried to write twitted but my spelling checker changed it to twitted
[16:29] pieterh :)
[17:11] pdhborges s
[17:26] iAmTheDave pieterh: sent something your way, email-style
[17:27] pieterh iAmTheDave: read it, lovely
[17:27] pieterh I'll post it to the wiki shortly, am just going for pizza with the family
[17:29] iAmTheDave whoa whoa whoa - what's more important here??
[17:29] iAmTheDave ;)
[17:29] iAmTheDave also, now i'm hungry for pizza.
[17:32] pieterh :) OK, posted the story to, now it's really pizza time!
[17:32] iAmTheDave dude i was kidding
[19:29] pdhborges Dany
[19:29] pdhborges o leão nnao tá cá ainda
[19:30] pdhborges ups wrong window
[23:33] danolj good evening. I have a question on a log file collection scenario.
[23:34] danolj I'd like to take log files from n-servers and place the contents into a zmq bus on which there may be zero or more listeners
[23:34] danolj is there a pattern that someone here would recommend?
[23:35] danolj at the moment I can certainly have each server PUB to an endpoint, but the listeners would need to know all the endpoints in order to SUB
[23:35] danolj any thoughts? thanks