Monday March 28, 2011

[Time] NameMessage
[01:35] jdroid- These slides are awesome:
[01:41] xet hi
[01:42] xet im doing some tests in a small pub/sub app
[01:42] xet need some help
[07:29] guido_g pieterh: what the hell did you do to the guide? firefox uses 100% cpu and opera takes ages to load the page.
[10:59] pieterh guido_g: hi
[13:43] nicolas hi
[13:56] pieterh nicolas: hi
[14:07] nicolas i'm having some trouble with zmq pub/sub in c++
[14:08] nicolas memory starts to be eaten by my application
[14:08] nicolas and it has no memory leaks
[14:09] nicolas its just a simple client/server pair to do performance tests
[14:15] nicolas it seems to be that the client is receiving or processing data faster than the server, and then the server starts to eat memory up
[14:21] pieterh nicolas: yes, this is normal
[14:22] pieterh the client is presumably slower than the server
[14:23] pieterh you should set a high-water mark, on the PUB socket
[14:24] traviscline nicolas: with 0mq it's up to you in some sense how you want to deal with that mismatch, covered well by the guide
[14:25] CIA-22 jzmq: 03Gonzalo Diethelm 07master * re3dc6dc 10/ (2 files): Got rid of calls to sleep() in all examples / tests. -
[14:45] nicolas I see
[14:46] nicolas pieterh: thanks
[14:46] pieterh nicolas: np
[14:46] nicolas traviscline: any particular seciton of the guide that might be helpful for my problem ?
[14:47] traviscline nicolas: absolutely read the whole thing, but in particular look for HWM references
[14:47] pieterh nicolas: this is explained in Ch2,
[14:47] nicolas thanks!
[15:17] private_meta ahahaha... binary star... good name
[15:24] private_meta pieterh: there?
[16:21] rbancroft l
[16:21] rbancroft im sorry, mischan
[16:22] mikko hi
[16:53] private_meta pieterh: a short question about zmsg. You are using unsigned characters as storage option for message parts, but whatever input or output you do or get from zmsg is signed char. You told me that void * instead of char * should be used, but still I find it somewhat inconsistent that you store in unsigned char and input/output in the zmsg as signed char.
[16:53] private_meta pieterh: maybe some consistency update would be good, like making it signed char as the entire existing api of zmsg is signed char only
[17:40] pieterh private_meta: re
[17:41] pieterh zmsg in the Guide is string-based but in libzapi I moved to a proper binary API
[19:03] pieterh mikko: hi!
[19:08] jobytaffey Hi, I'm having a problem with libev and REP sockets. I get the fd from ZMQ_FD, I watch for EV_READ and then zmq_recv(). I then watch for EV_WRITE and zmq_send(). But, after the first pair of messages, I never see EV_READ again. Oddly, if I ctrl-c my REQ test app (infinite loop doing zmq_send + zmq_recv) my REP app receives a packet. Any ideas?
[19:09] pieterh jobytaffey: are you checking for errors on all 0MQ calls?
[19:10] jobytaffey I believe so, yes.
[19:11] pieterh Sounds like you're stuck in a blocking call which exits when you press Ctrl-C, and then your test app does a send on the REQ socket
[19:11] pieterh without a minimal example to look at, it's hard to help...
[19:12] jobytaffey I thought you'd say that, I'll get to work on one :-)
[19:14] mikko pieterh: hi
[19:14] pieterh mikko: excellent latency this evening :-)
[19:40] Guthur Describing ZeroMQ in five minutes, is it possible? hehe
[19:41] Guthur there is space for a 5 min lightening talk at the ELS, and I'm tempted to give it a go
[19:41] mikko whats ELS?
[19:41] Guthur European Lisp Symposium
[19:42] Guthur it's on this week in Hamburg
[19:43] jond Guthur : which lisp are you using out of interest?
[19:44] pieterh Guthur: 5 minutes is kind of tight :-)
[19:44] pieterh you can say exactly one thing, and repeat it three times
[19:44] jdroid- Guthur: I think you can get the gist of it quickly.
[19:44] Guthur jond, I don't get to use it nearly as much as I like, but it would be usually Common Lisp, SBCL would be my implementation of choice
[19:44] jdroid- I wrote this on HN yesterday and realized you actually can describe zmq fast. You just need to start with saying it's a sockets system.
[19:44] jdroid-
[19:45] jdroid- Put messaging in to the idea once they understand how the socket's side of it works. I think Zed takes this approach too, but I'm inferring that from having seen him speak
[19:45] pieterh "That's correct that ZeroMQ isn't a server"...
[19:45] jond Guthur: didnt zed shaw have a "sockets done right" article somewhere which might have been appropriated into the guide somewhere?
[19:45] pieterh jond: I had used part of Zed's text, originally, but took it out again
[19:45] jdroid- pieterh: do you take issue with that line?
[19:45] Guthur I'll have a look
[19:46] pieterh jdroid-: nope, it's accurate
[19:46] jdroid- sweet
[19:46] jdroid- here's the whole thread:
[19:46] pieterh I'm just lolling because I finished writing Binary Star, a pattern for a high-availability server pair
[19:46] pieterh 0MQ is a high-level language for constructing asynchronous messaging architectures
[19:47] pieterh it has implementations in 20+ programming languages
[19:47] Guthur I suppose it will be a good challenge to try a lightening talk, if nothing else
[19:47] pieterh it can construct brokers, clients, services, data distribution, workload distribution
[19:47] jdroid- Guthur: I put together these examples with that in mind:
[19:48] jond pieterh : but actually via zfl zapi it becomes less about 0mq
[19:48] pieterh jond: it's the same high level language
[19:48] jdroid- You can simulate OneMQ, using ZeroMQ. And with a OneMQ, you can simulate InfinityMQ.
[19:48] jdroid- Maybe that' not quite it... there's a ZOI analogy in there somewhere
[19:49] pieterh well, all messaging systems are turing equivalent I guess
[19:49] pieterh but they're not the same
[19:51] pieterh IMO it's inaccurate to describe XREP/XREQ as non-blocking versions of REP/REQ
[19:51] jond has anyone got a link to the Ian Barber presentation?
[19:52] mpales pieterh: the guide describes them in the same way
[19:52] jobytaffey I'd really appreciate it if someone could take a look at my libev/REQ:REP problem. It's probably something silly, but I can't see it. There's a minimal example (files and tarball) here
[19:52] pieterh mpales: then I need to fix that, I'm going to review that whole explanation
[19:52] Guthur jond,
[19:54] pieterh jobytaffey: maybe the client should connect to localhost, not *?
[19:54] mpales pieterh: in ch2: A Request-Reply Broker, there is this sentence: Luckily there are non-blocking versions of these two sockets, called XREQ and XREP.
[19:54] pieterh mpales: ugh...
[19:54] jobytaffey pieterh: Would that make a difference? It does connect ok.
[19:55] jond Guthur: cheers. also sbcl is pretty cool, though havent used lisp for quite a while (though did manage to have a job where i did it for three years!)
[19:55] pieterh jobytaffey: you connect to a name or IP address, not an interface
[19:56] pieterh mpales: thanks for pointing this out, I'm going to try to make the explanations more accurate
[19:56] Guthur jond, not many get the opportunity do such employment
[19:56] jobytaffey Ah, I see what you mean. * means bind to any i/f, but you need to be explicit for the connect.
[19:56] pieterh jobytaffey: indeed, fix this and see if your code works better, the rest looks fine though I've not run it
[19:56] jobytaffey No, that didn't help.
[19:57] Guthur jond, are you still in software development?
[19:57] mpales pieterh: there is another thing I'd like to ask you
[19:57] jond Guthur: yes.
[19:57] pieterh mpales: sure...
[19:58] pieterh jond: "elegant weapons..." lol
[19:59] mpales pieterh: the guide explains how to send a big file using multipart messages
[19:59] Guthur jond, i love how when 'new' languages get new features which users trumpet the features are often ones that Lisps have had for over 20 years
[19:59] pieterh mpales: well, multipart messages are not really meant for files
[20:00] mpales pieterh: but also explains that until last part is sent/recvd, everything is kept in memory
[20:00] pieterh rather, it's a simple way to marshal data
[20:00] pieterh indeed, all in memory
[20:00] pieterh if you want to send files you need to send chunks as individual messages
[20:01] pieterh if you want to do restartable file transfer, then
[20:01] pieterh handshake with the recipient, ask how much of the file they have, then send the rest
[20:02] pieterh if you send large chunks, e.g. 1MB, you can destroy any network
[20:02] jond Guthur. it's all true but and there's rarely any paying work in it
[20:02] mpales pieterh: i see, my mistake, multipart messages are not mentioned in this context
[20:03] pieterh mpales: I'll make a file transfer example later on
[20:03] jdroid- pieterh: agreed, the guide gave me that text. In fact, the italicized text is literally copied from the guide so grep for that
[20:04] pieterh jdroid-: lol, that's funny, I was about to rant about how inaccurate that text was :-)
[20:04] jdroid- in yo face!
[20:04] jdroid- :D
[20:04] mpales pieterh: great!
[20:05] pieterh :-)
[20:05] mpales pieterh: i really like the style of the guide, thanks for the amazing work
[20:05] pieterh mpales: np, it's fun to write, though it's getting quite ... large...
[20:07] Guthur jond: I recently seen a job in London looking for a functional programmer with knowledge of clojure, very well paid
[20:07] Guthur though that's relative to your current pay grade of course
[20:07] Guthur mine is most certainly on the lower end, hehe
[20:08] mpales pieterh: yes, it is
[20:10] mpales pieterh: btw. I used to read the guide with a larger font size, but since last week when I increase the font size, the browser goes mad... I use google chrome
[20:11] pieterh mpales: yeah, the HTML got significantly fatter, since I added color markup for the code examples
[20:11] pieterh oh, I just tried a larger font in Chrome, it gets very unhappy...
[20:13] mpales pieterh: you can get some cash from google if you report it :-)
[20:14] i2pi Hi. I have a PUSH/PULL queue working with multiple PULL'ers. I'm using NOBLOCK on both ends. I'd like the PUSH'er to not block if no PULLers are available, but for messages to be queued up for when the PULLers come back online. Is this possible? If so, how?
[20:14] pieterh mpales: they'll just tell me to stop writing insane HTML... :-)
[20:15] pieterh i2pi: you can use zmq_poll
[20:16] i2pi pieterh: reading the docs now. Thanks.
[20:16] Guthur pieterh, your guide has actually only 3 validation errors
[20:16] Guthur which is a lot better than the vast majority of the internet
[20:16] pieterh Guthur: ah, that's because in fact I don't write any of the HTML at all
[20:17] pieterh mpales: firefox has no trouble resizing the font, interestingly
[20:17] Guthur probably the best way, hehe
[20:17] i2pi pieterh: Hmm. I'm not sure if i understand this correctly. Can you give me a hint?
[20:18] pieterh i2pi: you poll on the PUSH socket using ZMQ_POLLOUT
[20:18] mpales pieterh: yes, looks definitely like a chrome bug
[20:18] pieterh I've not tried that, to be honest, but it's the proper way IMO
[20:18] pieterh mpales: yes, I'm looking for a place to file it
[20:19] mpales pieterh: settings / tools / report...
[20:20] i2pi pieterh: Is the correct pattern to try doing a zmq_send() and if that fails with EAGAIN, then add that item to a list to poll. And then keep polling until the recvr returns?
[20:20] pieterh i2pi: look at the zmq_poll examples in the Guide, do the same except writing rather than reading
[20:21] pieterh you loop: poll / write / poll / write...
[20:21] pieterh doing a write only when poll tells you it's OK
[20:21] pieterh s/write/send/
[20:21] i2pi so, its up to my app to maintain the queue of unsent items?
[20:21] i2pi (I'm coming from AMQP, where I expect the queue to do queuing for me)
[20:23] jobytaffey The plot thickens on my problem - tcpdump shows that despite my evserver.c calling zmq_send() and it returning 0, the packet is never transmitted (I assume, because it thinks the socket is blocked).
[20:24] pieterh i2pi: you need to read the Guide
[20:24] i2pi ok :)
[20:24] pieterh jobytaffey: have you already written some examples without using evserver?
[20:25] jobytaffey Yes, they're in the URL I posted. There's a REQ/REP client and server without libev.
[20:25] i2pi pieterh: I started on the guide, and please don't take offense to this, but it is a little daunting that it starts very philosophically. I'd rather get straight into the meat.
[20:25] pieterh i2pi: there's only about 1 page of philosophy, then it gets straight into code...
[22:22] private_meta pieterh: yes, you hinted on that before, but right now, the zguide version is still inconsistent
[22:44] i2pi pieterh: I'm a few hours into the guide and I'm still not seeing it in the guide.