ZeroMq IRC Log

Thursday May 26, 2011

[Time] Name	Message
[01:54] xet	hi
[01:54] xet	I got a small question
[01:57] xet	any idea of what would be an optimal packet size to use for a real-time application?
[01:57] xet	using req/rep
[02:11] ASY	i don't believe there is such thing as an optimal packet size. I don't think there is a real answer to this. but if I was to guess, i would imagine it to be dependent on the transport layer. if tcp, you probably want to keep it under MTU size (< 1500) to avoid tcp fragmentation. If inproc, then I would imagine it to be under 4096 bytes. but the best thing to do in these cases it to measure
[02:11] ASY	performance yourself and determine what is suitable for you.
[02:13] xet	thanks for the answer
[02:13] xet	i am doing some measurements and found tcp rather slow using ~500 packets
[02:14] xet	500 byte
[02:16] ASY	the problem with tcp and 500 byte packets is usually ngale, which has to be disabled in real-time applications. don't know if zmq allows for this in setsockopt(). don't think so.
[02:16] xet	i will check setsockopt() reference
[02:16] ASY	actually this is a good question for someone. can ngale be disabled/enabled for the underlying tcp socket in zmq?
[02:17] ASY	ngale can be avoided by oversizing the packet to about 4k. but that is a very very lame way of doing so.
[02:17] ASY	but if 4k packet is faster then 500 bytes, then it is ngale.
[02:17] ASY	TCP_NODELAY
[02:23] xet	is that an option?
[02:25] ASY	in tcp, donno about zmq
[04:55] cremes	xet, ASY: 0mq disables nagle by default
[05:37] sustrik	xet: why would you want small packets?
[05:38] sustrik	the bigger the packet limit the better
[05:38] sustrik	if there's not enough data to send the packets will be small anyway
[05:38] sustrik	if latency is an issue, turn off packet coallescing in your NIC
[09:34] CIA-31	libzmq: 03Martin Sustrik 07bidi-pipes * r718885f 10/ (4 files): Pending messages are delivered even if connection doesn't exist yet ...
[12:12] mile	evax, I have a problem with making erlzmq on debian lenny
[12:12] mile	interested to take a look?
[12:26] Guthur	has anyone compared ZeroMQ to netty for java
[13:25] ukandrewtaylor	hi, I have a quick question. I saw a 0mq talk by Ian Barber at #dpc11 and it seemed pretty neat. What I'd like to do is create a queue that I can dump messages in and process without blocking on the initial request. What type of 'device' am I trying to create, 'queue' seems to be blocking by default?
[13:42] asmodai	notukandrewtaylo: Be patient btw, people are travelling and whatnot, answers might be slow in coming, but they do come. :)
[13:43] asmodai	notukandrewtaylo: I'm too much of a newbie myself to give an answer to that though.
[13:43] notukandrewtaylo	yup no worries. I'm on a flaky connection, so, I keep getting disconnected.
[13:44] notukandrewtaylo	I get the principle, I'm just not a software developer so I'm struggling to get my head around the practice. In a traditional message queue system, the sender doesn't really care about the response, just sends the request and moves on. Here, in a non-threaded application, the receive is always going to be blocking whilst it completes its task. I'm not sure if I'm right and trying to use zeromq wrong, or, I'm just looking in the wrong p
[13:44] notukandrewtaylo	for the answer :)
[13:45] notukandrewtaylo	i basically don't care how long the server process runs for or how much memory it uses, i just want it to keep processing requests and keep acting on them without blocking for more requests. If I lose data, or, messages get lost, I can handle that another way.
[13:49] asmodai	sounds like one of the scenarios zmq handles from what I understand of it
[13:50] notukandrewtaylo	yeah, i'm sure it does; i got that impression. I just can't work out how to approach it
[15:50] cremes	notukandrewtaylo: check out the 0mq guide for the ROUTER/DEALER socket types
[15:50] cremes	the REQ/REP sockets are built on top of those other sockets
[15:50] notukandrewtaylo	ROUTER/DEALER - thanks very much.
[15:50] sustrik	sounds more like push/pull afics
[15:50] cremes	ROUTER/DEALER do not block while waiting for a response
[15:50] sustrik	depends on whether you need responses from the workers
[15:50] cremes	notukandrewtaylo: definitely read the guide if you haven't yet... it may take a few read throughs to "click"
[15:51] notukandrewtaylo	yeah, i'm just trying to put together a proof-of-concept to explain a theory; I'm not a software developer and the guide is pretty hard going
[15:51] cremes	notukandrewtaylo: okay, but oftentimes the answers we give will be from the guide
[15:52] zedas	sustrik: hey did you change the way identities are done in 2.1.6 such that PUB socket identities work differently?
[15:52] cremes	if you don't understand the guide, you likely won't understand our answers :)
[15:52] notukandrewtaylo	yup, I just needed some way of zoning in my reading
[15:52] zedas	sustrik: previously we could start 10 mongrel2 handlers, and they could all PUB using the same identity and the mongrel2 server would get it, but not only one of the handler's responses is going through.
[15:54] sustrik	zedas: i haven't deliberately changed anything
[15:54] sustrik	there were few identity-related bug fixes though
[15:55] sustrik	having same identity with multiple peers doesn't seem ok
[15:55] sustrik	what's the point?
[15:57] zedas	when i read the docs during the 2.0.x series they said that each sending handler had to use the same identity in order to publish so that the sub side would handle them.
[15:58] zedas	and i just confirmed this used to work fine in the 2.0.x zmq but now doesn't work in 2.1.x.
[15:58] sustrik	that's weird
[15:58] sustrik	do you have any idea where the text was located?
[15:58] zedas	yeah, i wouldn't have done it if it didn't work.
[15:59] sustrik	afaics you can just not set the identities and everything should work ok
[15:59] zedas	ok, so to fix this, each handler PUB instance needs a totally new identity, or don't set the identity.
[15:59] sustrik	yes
[15:59] zedas	oh, but then does the PUB side get durability? if you don't set an identity then it's not durable correct?
[15:59] sustrik	the whole point in identities is to let the peer know that you exist
[16:00] sustrik	and have specific name
[16:00] zedas	well let the peer know you exist and have durability, you guys have sort of conflated the two
[16:00] sustrik	the peer has to know who you are when you reconnect
[16:01] sustrik	so that it can send you messages queued for you in the meantime
[16:01] sustrik	do you need durable messages?
[16:01] zedas	i understand that now, so i'll change that, but how do i also get durability in the sender without setting an identity.
[16:01] zedas	yes, people like knowing that their web requests will go through.
[16:01] sustrik	you can't
[16:01] sustrik	because the sender has no idea it's you when you reconnect
[16:02] sustrik	so it has no idea whether it should send you old messages or not
[16:03] zedas	ok, so you fixed identity bugs which changed the logic of how PUB socket identities work so that you can't have N handlers acting as one, they each have to have a different identity. so, another way to say this is that zmq identities have replaced port numbers for identifying clients.
[16:03] sustrik	yes
[16:04] zedas	and it also looks like most of you use the XREQ/XREP for this kind of thing, side stepping the whole problem.
[16:04] sustrik	think of identity as an permanent identifier of application instance
[16:04] sustrik	it stays the same even if application is restarted
[16:05] zedas	yep that's what we were doing, but i was reading it as application, NOT process
[16:05] zedas	so 10 processes could share the identity and they would get them round-robin and so on.
[16:05] sustrik	ah
[16:05] zedas	which is how it worked previously.
[16:06] zedas	in other words you changed the semantics of identity from "application abstract identity" to "exact per process identity, no duplicates allowed"
[16:06] sustrik	i wonder how the durability worked in such a scenario
[16:06] zedas	as far as i can tell it worked great.
[16:06] zedas	i guess we were just abusing a "feature" :-)
[16:07] sustrik	i would say i didn't work, but you never stumbled over the problem
[16:08] sustrik	anyway, if you use identity per instance is should work ok
[16:09] sustrik	btw, if you can identify the text that you mentioned, i'll fix it as it seems to be pretty misleading
[16:11] zedas	stutter: oh it worked, i have an application running on 2.0.x still that's doing it right now.
[16:12] zedas	sustrik: ok thanks i'll work up a fix for this in mongrel2.
[16:12] sustrik	ok
[17:14] ptrb	A ZMQ_PUSH can connect to multiple ZMQ_PULL sockets, but can a single ZMQ_PULL connect to multiple ZMQ_PUSH sockets?
[17:18] taotetek	ptrb: yes
[17:18] ptrb	taotetek: ah, neat. cool.
[17:19] taotetek	ptrb: it will fair queue all the incoming messages from all the push sockets
[23:27] zedas	pieterh: hey so suggestion: http://zguide.zeromq.org/page:all#Transient-vs-Durable-Sockets this is confusing because you only show one side of the communication, and don't really say what the identity identifies.
[23:27] zedas	pieterh: i think a full example, with a client/server, both setting an identity, and then making it clear that the identity identifies a process that is making a connection, would help quite a lot.
[23:36] zedas	pieterh: also, you say this, "What durable sockets give you is the promise that the ÃMQ transmit buffer is kept alive as long as the sender exists." But, in the examples that make a Durable pub/sub setup, the receiver is setting the identity so that the sender will use it to make a durable send socket.
[23:37] zedas	what you have is kind of saying "If the sender sets an identity, then the transmit buffer will be durable." but it's actualy, "If the receiver sets an identity, then the sender will use a durable transmit buffer."
[23:37] zedas	that means, is there a point for a receiver to be setting an identity? by reading this and the code it looks like that won't to anything.
[23:55] zedas	pieterh: ok i think I got a replacement for that sentence which clears it up, let me know if this works: "If a receiver (SUB, PULL, REQ) side of a socket sets an identity the the sending (PUB, PUSH, PULL) side will buffer messages when they aren't connected up to the HWM. The sending side does not need to set an identity for this to work.
[23:56] zedas	pieterh: then, you could explain the mechanism as, "It's kind of like a session cookie in an HTTP web application, except the client/sender is picking the cookie it will use."
[23:57] zedas	s/PUB, PUSH, PULL/PUB, PUSH, REP/g