ZeroMq IRC Log

Friday April 15, 2011

[Time] Name	Message
[05:33] sustrik	lestrrat: i wouldn't move to 3.0 yet
[05:33] sustrik	it's not fully stable yet
[05:48] lestrrat	well, not so much for really moving, but I want to silence the test failures (at least for now)
[05:54] sustrik	ok, i see
[05:54] sustrik	as for devices the plan is to move them to a separate project
[06:05] lestrrat	k, I guess I'd need to create a separate binding for it, then
[06:05] sustrik	ok, no haste needed, though
[06:06] lestrrat	yeah, I'll just skip the code that uses it.
[06:48] lestrrat	mikko: can you please point the jenkins tests to look at "libzmq3" branch for the perl binding + 0MQ 3.x? master is currently for 2.1.x
[06:48] lestrrat	I started changing stuff for zmq3 on libzmq3 branch.
[07:04] mikko	lestrrat: will do
[07:04] mikko	give me a sec
[07:05] mikko	lestrrat: building now
[07:05] lestrrat	mikko: it will fail :)
[07:05] mikko	it did
[07:06] mikko	but the build should be now configured correctly
[07:06] mikko	so it's a good start
[07:06] lestrrat	cool :)
[07:25] pieterh	lestrrat: we should backport the fixes to the tests to 2.x
[07:26] pieterh	mikko: did you get a chance to look at the rhat packaging for 2.x?
[07:55] mikko	i to the b
[13:46] yawniek	hi, from the examples i try to run asyncsrv.py. but the code where the ClientTask receives the answers is never executed. any hints?
[13:53] yawniek	talking about https://gist.github.com/883983/1940c7aff75871e1a1dcad4f8d340cf3a9a26ae1
[13:57] pieterh	yawniek: hi
[13:57] pieterh	how far does the request get?
[14:00] yawniek	the worker sends the replies
[14:00] pieterh	you see them in the server?
[14:00] yawniek	yes
[14:00] pieterh	so the normal problem here is with the XREP/ROUTER socket
[14:00] pieterh	which silently drops messages if the address is wrong
[14:01] pieterh	you want to print/trace the first frame (message part) that the server sends back to the client
[14:01] pieterh	make sure it's a valid identity
[14:02] pieterh	see http://zguide.zeromq.org/page:all#Missing-Message-Problem-Solver
[14:03] yawniek	ok thanks
[14:08] yawniek	pieterh: to trace i need to receive multipart and print all parts?
[14:08] pieterh	yawniek: no idea how this works in Python, sorry... but that sounds right
[14:08] mikko	hello
[14:08] pieterh	in any case you need to receive / send multipart if you're using a XREP socket
[14:09] pieterh	hi mikko!
[14:09] mikko	been pretty exciting time in .nl
[14:10] pieterh	conference?
[14:10] mikko	im stalking people
[14:10] pieterh	lol
[14:10] mikko	using kinect sensor
[14:10] mikko	http://www.vimeo.com/22386826
[15:23] Lemmih	Hiya, say I want a bunch of nodes to agree on the ordering of messages. Should I use N publishers and N subscribers together with some logic for ordering the messages or is there an easier way?
[15:24] pieterh	messages produced by multiple nodes?
[15:24] Lemmih	Yes.
[15:24] pieterh	only simple way is to push them all through a single point
[15:24] Lemmih	That would give me a single point of failure, no?
[15:25] pieterh	you have another way of ordering messages across multiple nodes?
[15:27] Lemmih	There are a couple of different ways to do it but I don't know enough about publisher/subscriber semantics to know how to go about it.
[15:28] pieterh	how would you do it?
[15:29] pieterh	Lemmih: you need to explain what you mean by "agree on the ordering"
[15:31] pieterh	However note that if you have multiple publishers and multiple subscribers talking N to N
[15:31] pieterh	then each subscriber will get messages from different publishers in an arbitrary order
[15:31] Lemmih	I want all nodes to see the messages in the same order. The exact order doesn't matter as long as it is consistent across each node.
[15:32] pieterh	if you want a single stream of messages, simplest is a central node to create that ordering
[15:32] locklace	Lemmih: sequence numbers
[15:32] pieterh	you might plausibly reorder within each client but it's still not robust
[15:33] pieterh	locklace: sequence numbers won't be unique if there are multiple publishers
[15:33] locklace	timestamps then
[15:33] pieterh	timestamps won't be accurate across a network
[15:33] pieterh	not at high message rates
[15:33] Lemmih	They don't have to be.
[15:33] locklace	they will if you're doing proper time sync, as you should be
[15:34] pieterh	not accurate down to the microsecond afaik
[15:34] pieterh	Lemmih: what 0MQ will do for you is e.g. fair queuing from all publishers
[15:34] locklace	depends on the time distribution method
[15:35] locklace	but it's not clear what the precision requirement is for this application
[15:37] pieterh	Lemmih: can you provide more background on your use case?
[15:38] Lemmih	pieterh: Alright. I'm fine with ordering the messages myself. Just wanted to make sure I didn't reinvent the wheel.
[15:39] Lemmih	I have a transaction queue which has to be identical on each node. And each node should be able to push data to the transaction queue.
[15:40] pieterh	Hmm, nice problem, and you want to work without a central device
[15:41] pieterh	If your transaction rate is not too high you can use locklace's timestamp method
[15:42] cremes	ntp is usually good for +/- 1 millisecond on a LAN
[15:42] cremes	i wouldn't count on it for microsecond or nanosecond granularity
[15:42] pieterh	so it also depends on the speed of processing of the transaction queue
[15:42] pieterh	if it's fast you won't be able to reorder properly
[15:42] Lemmih	I can do the ordering using timestamps (still works when the clocks differs, just adds latency) or non-local sequence numbers. However, since some of the other messaging libraries I've used in the path provided this functionality by default, I wanted to make sure I didn't rewrite something that's already in 0MQ. Those other messaging systems wheren't brokerless, though.
[15:43] pieterh	Lemmih: 0MQ is a rocket engine but without a chassis
[15:43] Lemmih	Oops, meant to write 'non-unique', not 'non-local'.
[15:44] pieterh	how do you reorder when you've already processed a transaction?
[15:44] pieterh	i.e. two clients get B, A, and A, B
[15:45] Lemmih	I don't. I keep the messages in a queue until they can be ordered.
[15:46] Lemmih	If A and B have timestamps and I know that the publishers can't/won't send any messages younger than the oldest of A and B, then A and B has been assigned order globally.
[15:47] Lemmih	This can also be done using ticks instead of timestamps.
[15:47] pieterh	so you either timeout per publisher, or wait for another message to arrive
[15:49] pieterh	in fact you can do it without timestamps at all
[15:49] pieterh	as long as you have unique publisher ids
[15:49] Lemmih	Right.
[15:49] Lemmih	And I do have that.
[15:49] pieterh	but it's going to add latency... and still be breakable
[15:49] Lemmih	Breakable how?
[15:50] pieterh	if a message gets stuck somewhere, for longer than your timeout, the order can become bogus
[15:50] Lemmih	Is it possible for a publisher to send two messages and have the second arrive before the first?
[15:51] pieterh	no, but two messages from two publishers will arrive in arbitrary order
[15:51] pieterh	obviously
[15:51] pieterh	from one publisher, it's a TCP stream
[15:51] pieterh	also you have no way to handle a crashing publisher
[15:51] pieterh	i.e. which has sent to some clients but not to others
[15:53] Lemmih	I do, actually, but that's due to the way I use the transaction log.
[15:54] pieterh	it'd be interesting to see your solution in print, once you get it working with 0MQ
[15:54] pieterh	it sounds like it'll work easily
[15:54] Lemmih	Just receiving a message isn't enough to consider it commited. It also has to be confirmed durable on a supermajority of the nodes.
[15:55] pieterh	I still have my doubts about the design but that's just because I don't see enough of it
[15:55] pieterh	you're basically doing atomic multicast iiuc
[15:56] guido_g	did somone say lamport? :)
[15:56] Lemmih	Well, I'll get back to you when it works (: Btw, I'm writing this code in Haskell.
[15:57] pieterh	guido_g: sounds like it :)
[15:57] pieterh	Lemmih: I assume whatever pattern you build will work in any language over 0MQ
[15:59] guido_g	using vector clocks?
[16:00] guido_g	ah ok
[16:00] guido_g	will wait then
[16:00] pieterh	Lemmih: I'm interested in your work, will it be open source?
[16:02] Lemmih	pieterh: Oh, you wrote "The Guide"? Damn that thing is long. I just spent most of my day reading it. (:
[16:02] Lemmih	pieterh: Yes, it will be open source.
[16:02] pieterh	Lemmih: lol, it's my work, yes, with the help of lots of people translating the examples
[16:02] pieterh	there is a lot of detail in some patterns
[16:02] Lemmih	I might contribute a few Haskell examples along the way.
[16:02] pieterh	I'm doing Clone, and have six models to show the evolution
[16:03] pieterh	Haskell examples would be great, the language isn't well represented today
[16:10] Lemmih	FYI, I'm working on adding multimaster replication to a database system written in Haskell.
[16:12] pieterh	Lemmih: there's a database written in Haskell?
[16:12] pieterh	neat
[16:13] pieterh	BTW, I'm really happy to hear you're using 0MQ for this, it should be a perfect fit
[16:14] Lemmih	Calling it a database is probably a bit generous. It's a system for adding ACID guarantees to native-language data-types. Like Prevaylor if you've heard of it.
[16:15] Lemmih	Yeah, I really like 0MQ for this. Feels much better than the other message systems I've tried in the past.
[16:16] pieterh	yeah
[16:23] iAmTheDave	Not sure this is the forum, but we just released a Django message queue app using 0MQ. https://github.com/dmgctrl/django-ztask
[16:24] iAmTheDave	It was so nice to work with 0MQ, it was so light-weight...
[16:24] pieterh	iAmTheDave: nice!
[16:25] iAmTheDave	pieterh: it totally is. we were using celery and all the cruft that goes along with it (not knocking it) but we wanted something WAY smaller.
[16:25] pieterh	iAmTheDave: would you send me a short story of how this happened?
[16:25] iAmTheDave	so like, thanks for 0MQ!
[16:25] pieterh	I'd like to publish that on our wiki somewhere
[16:26] pieterh	a testimonial...
[16:26] iAmTheDave	yeah you bet. that'd be awesome!
[16:26] pieterh	:)
[16:26] pieterh	email it to ph@imatix.com
[16:26] iAmTheDave	can do
[16:26] pieterh	I'll make sure your project gets coverage
[16:26] pieterh	have already twitted about it
[16:27] pieterh	next step is to update the zeromq myspace page!
[16:27] iAmTheDave	oh wow. thanks :) didn't expect all that, just wanted to let you know we're really happy with it
[16:27] locklace	it's just like 2003
[16:28] locklace	extra points for picking the right twitter verb too
[16:29] pieterh	locklace: I tried to write twitted but my spelling checker changed it to twitted
[16:29] pieterh	:)
[17:11] pdhborges	s
[17:26] iAmTheDave	pieterh: sent something your way, email-style
[17:27] pieterh	iAmTheDave: read it, lovely
[17:27] pieterh	I'll post it to the wiki shortly, am just going for pizza with the family
[17:29] iAmTheDave	whoa whoa whoa - what's more important here??
[17:29] iAmTheDave	;)
[17:29] iAmTheDave	also, now i'm hungry for pizza.
[17:32] pieterh	:) OK, posted the story to http://www.zeromq.org/story:3, now it's really pizza time!
[17:32] iAmTheDave	dude i was kidding
[19:29] pdhborges	Dany
[19:29] pdhborges	o leÃ£o nnao tÃ¡ cÃ¡ ainda
[19:30] pdhborges	ups wrong window
[23:33] danolj	good evening. I have a question on a log file collection scenario.
[23:34] danolj	I'd like to take log files from n-servers and place the contents into a zmq bus on which there may be zero or more listeners
[23:34] danolj	is there a pattern that someone here would recommend?
[23:35] danolj	at the moment I can certainly have each server PUB to an endpoint, but the listeners would need to know all the endpoints in order to SUB
[23:35] danolj	any thoughts? thanks