Thursday May 26, 2011

[Time] NameMessage
[01:54] xet hi
[01:54] xet I got a small question
[01:57] xet any idea of what would be an optimal packet size to use for a real-time application?
[01:57] xet using req/rep
[02:11] ASY i don't believe there is such thing as an optimal packet size. I don't think there is a real answer to this. but if I was to guess, i would imagine it to be dependent on the transport layer. if tcp, you probably want to keep it under MTU size (< 1500) to avoid tcp fragmentation. If inproc, then I would imagine it to be under 4096 bytes. but the best thing to do in these cases it to measure
[02:11] ASY performance yourself and determine what is suitable for you.
[02:13] xet thanks for the answer
[02:13] xet i am doing some measurements and found tcp rather slow using ~500 packets
[02:14] xet 500 byte
[02:16] ASY the problem with tcp and 500 byte packets is usually ngale, which has to be disabled in real-time applications. don't know if zmq allows for this in setsockopt(). don't think so.
[02:16] xet i will check setsockopt() reference
[02:16] ASY actually this is a good question for someone. can ngale be disabled/enabled for the underlying tcp socket in zmq?
[02:17] ASY ngale can be avoided by oversizing the packet to about 4k. but that is a very very lame way of doing so.
[02:17] ASY but if 4k packet is faster then 500 bytes, then it is ngale.
[02:23] xet is that an option?
[02:25] ASY in tcp, donno about zmq
[04:55] cremes xet, ASY: 0mq disables nagle by default
[05:37] sustrik xet: why would you want small packets?
[05:38] sustrik the bigger the packet limit the better
[05:38] sustrik if there's not enough data to send the packets will be small anyway
[05:38] sustrik if latency is an issue, turn off packet coallescing in your NIC
[09:34] CIA-31 libzmq: 03Martin Sustrik 07bidi-pipes * r718885f 10/ (4 files): Pending messages are delivered even if connection doesn't exist yet ...
[12:12] mile evax, I have a problem with making erlzmq on debian lenny
[12:12] mile interested to take a look?
[12:26] Guthur has anyone compared ZeroMQ to netty for java
[13:25] ukandrewtaylor hi, I have a quick question. I saw a 0mq talk by Ian Barber at #dpc11 and it seemed pretty neat. What I'd like to do is create a queue that I can dump messages in and process without blocking on the initial request. What type of 'device' am I trying to create, 'queue' seems to be blocking by default?
[13:42] asmodai notukandrewtaylo: Be patient btw, people are travelling and whatnot, answers might be slow in coming, but they do come. :)
[13:43] asmodai notukandrewtaylo: I'm too much of a newbie myself to give an answer to that though.
[13:43] notukandrewtaylo yup no worries. I'm on a flaky connection, so, I keep getting disconnected.
[13:44] notukandrewtaylo I get the principle, I'm just not a software developer so I'm struggling to get my head around the practice. In a traditional message queue system, the sender doesn't really care about the response, just sends the request and moves on. Here, in a non-threaded application, the receive is always going to be blocking whilst it completes its task. I'm not sure if I'm right and trying to use zeromq wrong, or, I'm just looking in the wrong p
[13:44] notukandrewtaylo for the answer :)
[13:45] notukandrewtaylo i basically don't care how long the server process runs for or how much memory it uses, i just want it to keep processing requests and keep acting on them without blocking for more requests. If I lose data, or, messages get lost, I can handle that another way.
[13:49] asmodai sounds like one of the scenarios zmq handles from what I understand of it
[13:50] notukandrewtaylo yeah, i'm sure it does; i got that impression. I just can't work out how to approach it
[15:50] cremes notukandrewtaylo: check out the 0mq guide for the ROUTER/DEALER socket types
[15:50] cremes the REQ/REP sockets are built on top of those other sockets
[15:50] notukandrewtaylo ROUTER/DEALER - thanks very much.
[15:50] sustrik sounds more like push/pull afics
[15:50] cremes ROUTER/DEALER do not block while waiting for a response
[15:50] sustrik depends on whether you need responses from the workers
[15:50] cremes notukandrewtaylo: definitely read the guide if you haven't yet... it may take a few read throughs to "click"
[15:51] notukandrewtaylo yeah, i'm just trying to put together a proof-of-concept to explain a theory; I'm not a software developer and the guide is pretty hard going
[15:51] cremes notukandrewtaylo: okay, but oftentimes the answers we give will be from the guide
[15:52] zedas sustrik: hey did you change the way identities are done in 2.1.6 such that PUB socket identities work differently?
[15:52] cremes if you don't understand the guide, you likely won't understand our answers :)
[15:52] notukandrewtaylo yup, I just needed some way of zoning in my reading
[15:52] zedas sustrik: previously we could start 10 mongrel2 handlers, and they could all PUB using the same identity and the mongrel2 server would get it, but not only one of the handler's responses is going through.
[15:54] sustrik zedas: i haven't deliberately changed anything
[15:54] sustrik there were few identity-related bug fixes though
[15:55] sustrik having same identity with multiple peers doesn't seem ok
[15:55] sustrik what's the point?
[15:57] zedas when i read the docs during the 2.0.x series they said that each sending handler had to use the same identity in order to publish so that the sub side would handle them.
[15:58] zedas and i just confirmed this used to work fine in the 2.0.x zmq but now doesn't work in 2.1.x.
[15:58] sustrik that's weird
[15:58] sustrik do you have any idea where the text was located?
[15:58] zedas yeah, i wouldn't have done it if it didn't work.
[15:59] sustrik afaics you can just not set the identities and everything should work ok
[15:59] zedas ok, so to fix this, each handler PUB instance needs a totally new identity, or don't set the identity.
[15:59] sustrik yes
[15:59] zedas oh, but then does the PUB side get durability? if you don't set an identity then it's not durable correct?
[15:59] sustrik the whole point in identities is to let the peer know that you exist
[16:00] sustrik and have specific name
[16:00] zedas well let the peer know you exist and have durability, you guys have sort of conflated the two
[16:00] sustrik the peer has to know who you are when you reconnect
[16:01] sustrik so that it can send you messages queued for you in the meantime
[16:01] sustrik do you need durable messages?
[16:01] zedas i understand that now, so i'll change that, but how do i also get durability in the sender without setting an identity.
[16:01] zedas yes, people like knowing that their web requests will go through.
[16:01] sustrik you can't
[16:01] sustrik because the sender has no idea it's you when you reconnect
[16:02] sustrik so it has no idea whether it should send you old messages or not
[16:03] zedas ok, so you fixed identity bugs which changed the logic of how PUB socket identities work so that you can't have N handlers acting as one, they each have to have a different identity. so, another way to say this is that zmq identities have replaced port numbers for identifying clients.
[16:03] sustrik yes
[16:04] zedas and it also looks like most of you use the XREQ/XREP for this kind of thing, side stepping the whole problem.
[16:04] sustrik think of identity as an permanent identifier of application instance
[16:04] sustrik it stays the same even if application is restarted
[16:05] zedas yep that's what we were doing, but i was reading it as application, NOT process
[16:05] zedas so 10 processes could share the identity and they would get them round-robin and so on.
[16:05] sustrik ah
[16:05] zedas which is how it worked previously.
[16:06] zedas in other words you changed the semantics of identity from "application abstract identity" to "exact per process identity, no duplicates allowed"
[16:06] sustrik i wonder how the durability worked in such a scenario
[16:06] zedas as far as i can tell it worked great.
[16:06] zedas i guess we were just abusing a "feature" :-)
[16:07] sustrik i would say i didn't work, but you never stumbled over the problem
[16:08] sustrik anyway, if you use identity per instance is should work ok
[16:09] sustrik btw, if you can identify the text that you mentioned, i'll fix it as it seems to be pretty misleading
[16:11] zedas stutter: oh it worked, i have an application running on 2.0.x still that's doing it right now.
[16:12] zedas sustrik: ok thanks i'll work up a fix for this in mongrel2.
[16:12] sustrik ok
[17:14] ptrb A ZMQ_PUSH can connect to multiple ZMQ_PULL sockets, but can a single ZMQ_PULL connect to multiple ZMQ_PUSH sockets?
[17:18] taotetek ptrb: yes
[17:18] ptrb taotetek: ah, neat. cool.
[17:19] taotetek ptrb: it will fair queue all the incoming messages from all the push sockets
[23:27] zedas pieterh: hey so suggestion: this is confusing because you only show one side of the communication, and don't really say what the identity identifies.
[23:27] zedas pieterh: i think a full example, with a client/server, both setting an identity, and then making it clear that the identity identifies a process that is making a connection, would help quite a lot.
[23:36] zedas pieterh: also, you say this, "What durable sockets give you is the promise that the ØMQ transmit buffer is kept alive as long as the sender exists." But, in the examples that make a Durable pub/sub setup, the *receiver* is setting the identity so that the *sender* will use it to make a durable send socket.
[23:37] zedas what you have is kind of saying "If the *sender* sets an identity, then the transmit buffer will be durable." but it's actualy, "If the *receiver* sets an identity, then the *sender* will use a durable transmit buffer."
[23:37] zedas that means, is there a point for a receiver to be setting an identity? by reading this and the code it looks like that won't to anything.
[23:55] zedas pieterh: ok i think I got a replacement for that sentence which clears it up, let me know if this works: "If a *receiver* (SUB, PULL, REQ) side of a socket sets an identity the the *sending* (PUB, PUSH, PULL) side will buffer messages when they aren't connected up to the HWM. The *sending* side does not need to set an identity for this to work.
[23:56] zedas pieterh: then, you could explain the mechanism as, "It's kind of like a session cookie in an HTTP web application, except the client/sender is picking the cookie it will use."
[23:57] zedas s/PUB, PUSH, PULL/PUB, PUSH, REP/g