ZeroMq IRC Log

Monday March 28, 2011

[Time] Name	Message
[01:35] jdroid-	These slides are awesome: http://www.slideshare.net/IanBarber/zeromq-is-the-answer
[01:41] xet	hi
[01:42] xet	im doing some tests in a small pub/sub app
[01:42] xet	need some help
[07:29] guido_g	pieterh: what the hell did you do to the guide? firefox uses 100% cpu and opera takes ages to load the page.
[10:59] pieterh	guido_g: hi
[13:43] nicolas	hi
[13:56] pieterh	nicolas: hi
[14:07] nicolas	i'm having some trouble with zmq pub/sub in c++
[14:08] nicolas	memory starts to be eaten by my application
[14:08] nicolas	and it has no memory leaks
[14:09] nicolas	its just a simple client/server pair to do performance tests
[14:15] nicolas	it seems to be that the client is receiving or processing data faster than the server, and then the server starts to eat memory up
[14:21] pieterh	nicolas: yes, this is normal
[14:22] pieterh	the client is presumably slower than the server
[14:23] pieterh	you should set a high-water mark, on the PUB socket
[14:24] traviscline	nicolas: with 0mq it's up to you in some sense how you want to deal with that mismatch, covered well by the guide
[14:25] CIA-22	jzmq: 03Gonzalo Diethelm 07master * re3dc6dc 10/ (2 files): Got rid of calls to sleep() in all examples / tests. - http://bit.ly/fnyW8U
[14:45] nicolas	I see
[14:46] nicolas	pieterh: thanks
[14:46] pieterh	nicolas: np
[14:46] nicolas	traviscline: any particular seciton of the guide that might be helpful for my problem ?
[14:47] traviscline	nicolas: absolutely read the whole thing, but in particular look for HWM references
[14:47] pieterh	nicolas: this is explained in Ch2, http://zguide.zeromq.org/page:all#toc43
[14:47] nicolas	thanks!
[15:17] private_meta	ahahaha... binary star... good name
[15:24] private_meta	pieterh: there?
[16:21] rbancroft	l
[16:21] rbancroft	im sorry, mischan
[16:22] mikko	hi
[16:53] private_meta	pieterh: a short question about zmsg. You are using unsigned characters as storage option for message parts, but whatever input or output you do or get from zmsg is signed char. You told me that void * instead of char * should be used, but still I find it somewhat inconsistent that you store in unsigned char and input/output in the zmsg as signed char.
[16:53] private_meta	pieterh: maybe some consistency update would be good, like making it signed char as the entire existing api of zmsg is signed char only
[17:40] pieterh	private_meta: re
[17:41] pieterh	zmsg in the Guide is string-based but in libzapi I moved to a proper binary API
[19:03] pieterh	mikko: hi!
[19:08] jobytaffey	Hi, I'm having a problem with libev and REP sockets. I get the fd from ZMQ_FD, I watch for EV_READ and then zmq_recv(). I then watch for EV_WRITE and zmq_send(). But, after the first pair of messages, I never see EV_READ again. Oddly, if I ctrl-c my REQ test app (infinite loop doing zmq_send + zmq_recv) my REP app receives a packet. Any ideas?
[19:09] pieterh	jobytaffey: are you checking for errors on all 0MQ calls?
[19:10] jobytaffey	I believe so, yes.
[19:11] pieterh	Sounds like you're stuck in a blocking call which exits when you press Ctrl-C, and then your test app does a send on the REQ socket
[19:11] pieterh	without a minimal example to look at, it's hard to help...
[19:12] jobytaffey	I thought you'd say that, I'll get to work on one :-)
[19:14] mikko	pieterh: hi
[19:14] pieterh	mikko: excellent latency this evening :-)
[19:40] Guthur	Describing ZeroMQ in five minutes, is it possible? hehe
[19:41] Guthur	there is space for a 5 min lightening talk at the ELS, and I'm tempted to give it a go
[19:41] mikko	whats ELS?
[19:41] Guthur	European Lisp Symposium
[19:42] Guthur	it's on this week in Hamburg
[19:43] jond	Guthur : which lisp are you using out of interest?
[19:44] pieterh	Guthur: 5 minutes is kind of tight :-)
[19:44] pieterh	you can say exactly one thing, and repeat it three times
[19:44] jdroid-	Guthur: I think you can get the gist of it quickly.
[19:44] Guthur	jond, I don't get to use it nearly as much as I like, but it would be usually Common Lisp, SBCL would be my implementation of choice
[19:44] jdroid-	I wrote this on HN yesterday and realized you actually can describe zmq fast. You just need to start with saying it's a sockets system.
[19:44] jdroid-	http://news.ycombinator.com/item?id=2377224
[19:45] jdroid-	Put messaging in to the idea once they understand how the socket's side of it works. I think Zed takes this approach too, but I'm inferring that from having seen him speak
[19:45] pieterh	"That's correct that ZeroMQ isn't a server"...
[19:45] jond	Guthur: didnt zed shaw have a "sockets done right" article somewhere which might have been appropriated into the guide somewhere?
[19:45] pieterh	jond: I had used part of Zed's text, originally, but took it out again
[19:45] jdroid-	pieterh: do you take issue with that line?
[19:45] Guthur	I'll have a look
[19:46] pieterh	jdroid-: nope, it's accurate
[19:46] jdroid-	sweet
[19:46] jdroid-	here's the whole thread: http://news.ycombinator.com/item?id=2376822
[19:46] pieterh	I'm just lolling because I finished writing Binary Star, a pattern for a high-availability server pair
[19:46] pieterh	0MQ is a high-level language for constructing asynchronous messaging architectures
[19:47] pieterh	it has implementations in 20+ programming languages
[19:47] Guthur	I suppose it will be a good challenge to try a lightening talk, if nothing else
[19:47] pieterh	it can construct brokers, clients, services, data distribution, workload distribution
[19:47] jdroid-	Guthur: I put together these examples with that in mind: https://github.com/j2labs/zmq_examples
[19:48] jond	pieterh : but actually via zfl zapi it becomes less about 0mq
[19:48] pieterh	jond: it's the same high level language
[19:48] jdroid-	You can simulate OneMQ, using ZeroMQ. And with a OneMQ, you can simulate InfinityMQ.
[19:48] jdroid-	Maybe that' not quite it... there's a ZOI analogy in there somewhere
[19:49] pieterh	well, all messaging systems are turing equivalent I guess
[19:49] pieterh	but they're not the same
[19:51] pieterh	IMO it's inaccurate to describe XREP/XREQ as non-blocking versions of REP/REQ
[19:51] jond	has anyone got a link to the Ian Barber presentation?
[19:52] mpales	pieterh: the guide describes them in the same way
[19:52] jobytaffey	I'd really appreciate it if someone could take a look at my libev/REQ:REP problem. It's probably something silly, but I can't see it. There's a minimal example (files and tarball) here http://www.hodgepig.org/testcase/
[19:52] pieterh	mpales: then I need to fix that, I'm going to review that whole explanation
[19:52] Guthur	jond, http://vimeo.com/20605470
[19:54] pieterh	jobytaffey: maybe the client should connect to localhost, not *?
[19:54] mpales	pieterh: in ch2: A Request-Reply Broker, there is this sentence: Luckily there are non-blocking versions of these two sockets, called XREQ and XREP.
[19:54] pieterh	mpales: ugh...
[19:54] jobytaffey	pieterh: Would that make a difference? It does connect ok.
[19:55] jond	Guthur: cheers. also sbcl is pretty cool, though havent used lisp for quite a while (though did manage to have a job where i did it for three years!)
[19:55] pieterh	jobytaffey: you connect to a name or IP address, not an interface
[19:56] pieterh	mpales: thanks for pointing this out, I'm going to try to make the explanations more accurate
[19:56] Guthur	jond, not many get the opportunity do such employment
[19:56] jobytaffey	Ah, I see what you mean. * means bind to any i/f, but you need to be explicit for the connect.
[19:56] pieterh	jobytaffey: indeed, fix this and see if your code works better, the rest looks fine though I've not run it
[19:56] jobytaffey	No, that didn't help.
[19:57] Guthur	jond, are you still in software development?
[19:57] mpales	pieterh: there is another thing I'd like to ask you
[19:57] jond	Guthur: yes. http://xkcd.com/297/
[19:57] pieterh	mpales: sure...
[19:58] pieterh	jond: "elegant weapons..." lol
[19:59] mpales	pieterh: the guide explains how to send a big file using multipart messages
[19:59] Guthur	jond, i love how when 'new' languages get new features which users trumpet the features are often ones that Lisps have had for over 20 years
[19:59] pieterh	mpales: well, multipart messages are not really meant for files
[20:00] mpales	pieterh: but also explains that until last part is sent/recvd, everything is kept in memory
[20:00] pieterh	rather, it's a simple way to marshal data
[20:00] pieterh	indeed, all in memory
[20:00] pieterh	if you want to send files you need to send chunks as individual messages
[20:01] pieterh	if you want to do restartable file transfer, then
[20:01] pieterh	handshake with the recipient, ask how much of the file they have, then send the rest
[20:02] pieterh	if you send large chunks, e.g. 1MB, you can destroy any network
[20:02] jond	Guthur. it's all true but http://xkcd.com/224/ and there's rarely any paying work in it
[20:02] mpales	pieterh: i see, my mistake, multipart messages are not mentioned in this context
[20:03] pieterh	mpales: I'll make a file transfer example later on
[20:03] jdroid-	pieterh: agreed, the guide gave me that text. In fact, the italicized text is literally copied from the guide so grep for that
[20:04] pieterh	jdroid-: lol, that's funny, I was about to rant about how inaccurate that text was :-)
[20:04] jdroid-	in yo face!
[20:04] jdroid-	:D
[20:04] mpales	pieterh: great!
[20:05] pieterh	:-)
[20:05] mpales	pieterh: i really like the style of the guide, thanks for the amazing work
[20:05] pieterh	mpales: np, it's fun to write, though it's getting quite ... large...
[20:07] Guthur	jond: I recently seen a job in London looking for a functional programmer with knowledge of clojure, very well paid
[20:07] Guthur	though that's relative to your current pay grade of course
[20:07] Guthur	mine is most certainly on the lower end, hehe
[20:08] mpales	pieterh: yes, it is
[20:10] mpales	pieterh: btw. I used to read the guide with a larger font size, but since last week when I increase the font size, the browser goes mad... I use google chrome
[20:11] pieterh	mpales: yeah, the HTML got significantly fatter, since I added color markup for the code examples
[20:11] pieterh	oh, I just tried a larger font in Chrome, it gets very unhappy...
[20:13] mpales	pieterh: you can get some cash from google if you report it :-)
[20:14] i2pi	Hi. I have a PUSH/PULL queue working with multiple PULL'ers. I'm using NOBLOCK on both ends. I'd like the PUSH'er to not block if no PULLers are available, but for messages to be queued up for when the PULLers come back online. Is this possible? If so, how?
[20:14] pieterh	mpales: they'll just tell me to stop writing insane HTML... :-)
[20:15] pieterh	i2pi: you can use zmq_poll
[20:16] i2pi	pieterh: reading the docs now. Thanks.
[20:16] Guthur	pieterh, your guide has actually only 3 validation errors
[20:16] Guthur	which is a lot better than the vast majority of the internet
[20:16] pieterh	Guthur: ah, that's because in fact I don't write any of the HTML at all
[20:17] pieterh	mpales: firefox has no trouble resizing the font, interestingly
[20:17] Guthur	probably the best way, hehe
[20:17] i2pi	pieterh: Hmm. I'm not sure if i understand this correctly. Can you give me a hint?
[20:18] pieterh	i2pi: you poll on the PUSH socket using ZMQ_POLLOUT
[20:18] mpales	pieterh: yes, looks definitely like a chrome bug
[20:18] pieterh	I've not tried that, to be honest, but it's the proper way IMO
[20:18] pieterh	mpales: yes, I'm looking for a place to file it
[20:19] mpales	pieterh: settings / tools / report...
[20:20] i2pi	pieterh: Is the correct pattern to try doing a zmq_send() and if that fails with EAGAIN, then add that item to a list to poll. And then keep polling until the recvr returns?
[20:20] pieterh	i2pi: look at the zmq_poll examples in the Guide, do the same except writing rather than reading
[20:21] pieterh	you loop: poll / write / poll / write...
[20:21] pieterh	doing a write only when poll tells you it's OK
[20:21] pieterh	s/write/send/
[20:21] i2pi	so, its up to my app to maintain the queue of unsent items?
[20:21] i2pi	(I'm coming from AMQP, where I expect the queue to do queuing for me)
[20:23] jobytaffey	The plot thickens on my problem - tcpdump shows that despite my evserver.c calling zmq_send() and it returning 0, the packet is never transmitted (I assume, because it thinks the socket is blocked).
[20:24] pieterh	i2pi: you need to read the Guide
[20:24] i2pi	ok :)
[20:24] pieterh	jobytaffey: have you already written some examples without using evserver?
[20:25] jobytaffey	Yes, they're in the URL I posted. There's a REQ/REP client and server without libev.
[20:25] i2pi	pieterh: I started on the guide, and please don't take offense to this, but it is a little daunting that it starts very philosophically. I'd rather get straight into the meat.
[20:25] pieterh	i2pi: there's only about 1 page of philosophy, then it gets straight into code...
[22:22] private_meta	pieterh: yes, you hinted on that before, but right now, the zguide version is still inconsistent
[22:44] i2pi	pieterh: I'm a few hours into the guide and I'm still not seeing it in the guide.