ZeroMq IRC Log

Wednesday June 23, 2010

[Time] Name	Message
[12:27] Umesh	is the man page of zmq_outdated ...
[12:27] Umesh	is the man page of zmq_tcp (7) outdated
[12:28] Umesh	because on tcp socket i am not geeting the messages compatible with the messages mentioned on this page ...
[12:31] sustrik	Umesh: what are you getting exactly?
[12:32] Umesh	i first get 2 bytes 1 1
[12:32] Umesh	sorry ... forget first part now typed ...
[12:33] sustrik	that's an empty message
[12:33] sustrik	you are right, it's not documented
[12:33] Umesh	i get 2 reads ...
[12:33] sustrik	but peers use it to identify themselves when a connection is established
[12:33] Umesh	1st read 1 0 ... means empty message ...
[12:34] sustrik	so when connection is established
[12:34] sustrik	peer A says "i'm A"
[12:34] sustrik	and peer B says "i'm B"
[12:34] Umesh	and then 2nd read 1 1 + frame described in man zmq_tcp(7)
[12:35] sustrik	what's your topology?
[12:35] sustrik	req/rep with queue in the middle?
[12:35] Umesh	req/res socket ...
[12:35] Umesh	nope ... no que in the middle
[12:35] sustrik	ah, right
[12:35] sustrik	so the first (empty) message is an identification
[12:35] sustrik	that's say for all socket types
[12:36] sustrik	now, req/rep has more complex protocol
[12:36] Umesh	actually i wanted to use the sync client with zmq server ...
[12:36] sustrik	what it has to do is to attach a way back to the requester to each request
[12:36] Umesh	therefore i am trying to send /recv message mentioned in the format ...
[12:36] Umesh	of man 7 zmq_tcp
[12:37] sustrik	so that reply can be sent back to the right requester
[12:37] Umesh	using the normal tcp/ip socket apis
[12:37] sustrik	Umesh: yes
[12:37] sustrik	the format described is OK
[12:37] sustrik	however, sockets add one layer on top of it
[12:38] Umesh	ok
[12:38] sustrik	so some of the messages (message parts) are used by 0MQ infrastrucuture
[12:38] sustrik	namely: first message on TCP connection is peer's identity
[12:38] sustrik	then
[12:39] sustrik	each REQ/REP message carries a stack of identities that identify the way back to the requester
[12:39] sustrik	so for example
[12:39] sustrik	requester A says "ABC"
[12:40] sustrik	what you see on the wire is that message part ABC is preceded by an empty message part \|\|ABC\|
[12:40] sustrik	the empty message part serves to delimit the actual message from the routing info
[12:41] Umesh	ok
[12:41] sustrik	when the message is received the identity of the sending peer is attached to it
[12:41] sustrik	\|A\|\|ABC\|
[12:41] sustrik	if it's sent further (as when there's a zmq_queue in the middle)
[12:42] sustrik	there's another identity added to the request
[12:42] sustrik	\|my_queue\|A\|\|ABC\|
[12:42] sustrik	there can be multiple hops
[12:42] Umesh	ok
[12:42] sustrik	and thus multiple identities in the traceroute stack
[12:42] sustrik	what REP socket does
[12:42] Umesh	is there not any way to not to have these identities ?
[12:43] Umesh	for normal working with bsd tcp/ip sockets
[12:43] sustrik	you would need 0MQ equivalent of DNS then
[12:43] sustrik	still you would need a single identity in the request
[12:43] sustrik	so that REP would know where to send the reply
[12:44] Umesh	one identity is fine ... that i can decode easily ...
[12:44] sustrik	you can easily decode mulitple identities as well
[12:44] Umesh	one way can be never use hops/queue in between
[12:44] sustrik	the point is that the stack is terminated by empty message part
[12:45] sustrik	no need for such drastic measures
[12:45] sustrik	what you need to do is:
[12:45] sustrik	on the REQ side
[12:45] sustrik	1. when sending message prepend it by empty message part (1 1)
[12:45] sustrik	2. when receiving the message chop off the empty message part (1 1)
[12:45] sustrik	on the REP side
[12:46] sustrik	1. when receiving a message store all the identities in the stack until you encounter empty message part (stack delimiter)
[12:46] sustrik	2. when sending a reply prepend all the stored identities to the reply
[12:46] sustrik	that's it
[12:46] Umesh	ok
[12:47] Umesh	does i nned to send 1 0 first message always ?
[12:47] Umesh	need*
[12:48] sustrik	that's only when you open the connection
[12:48] sustrik	1. create TCP connection
[12:48] sustrik	2. send 1 0
[12:48] sustrik	3. receive one message (peer's identity)
[12:50] Umesh	ok ...
[13:37] Umesh	sustrik: one more doubt ... this multihop support is added in the release 2.0.7 ... right ?
[13:37] sustrik	yes
[14:01] Umesh	ok
[15:34] jldupont	FYI: http://nichol.as/zeromq-an-introduction
[15:35] sustrik	jldupont: yes, it's already linked from zeromq.org (see blogs section)
[15:35] jldupont	ok
[15:36] jldupont	so, if I got this right, the first string sent on a pub/sub constitutes the "message type", right?
[15:37] sustrik	it's a bit different
[15:37] jldupont	i.e. ?
[15:37] sustrik	any message beginning with particular sequence of bytes matches the subscription
[15:37] jldupont	ok
[15:38] jldupont	btw, I never was able to build the Python documentation on my system...
[15:38] jldupont	problem with matplotlib-sphinx-ext
[15:38] jldupont	linux karmic
[15:39] sustrik	dunno, i have nothing to do with python binding... try complaining on the mailing list
[15:39] jldupont	;-)
[15:40] sustrik	or there's an issue tracker on pyzmq github project
[15:41] jldupont	from what I can tell, the "wire" format might be brittle to "out-of-sync" cases... the delineation mechanisms appears weak... but of course, I might not have seen all documentation ;-)
[15:43] jldupont	anyhow, 0mq is definitely more friendly to work with than rabbitmq.
[15:44] sustrik	jldupont: can you elaborate on the delineation?
[15:50] jldupont	I mean there must be "symbols" reserved for the framing and "symbols" valid for the data.
[15:50] jldupont	I only see "payload length" which can cause problems when it comes to "out-of-sync" conditions i.e. where is the next boundary?
[15:51] sustrik	it can't go out-of-sync as it's layered on top of TCP
[15:51] sustrik	i would like to have first two bytes in each TCP packet reserved fro "offset of the first 0mq message in the packet"
[15:51] sustrik	but
[15:52] sustrik	1. BSD sockets have no API for sending data delineated with TCP packets
[15:52] sustrik	2. networking hardware does all kind of black magic with TCP packets (packet coallescing etc.)
[15:53] jldupont	I just mean that if something goes wrong on the software stack side, recovering the connection might not be possible... only tear-down, bring up will do.
[15:54] jldupont	but that might be a good compromise.
[15:54] jldupont	I mean, it's good enough.
[15:54] jldupont	since that sort of condition is rare-ish.
[15:56] jldupont	the trouble is detecting this situation....
[15:56] jldupont	interesting failure mode... can be mitigated with appropriate timeouts, drop/bring up
[16:02] sustrik	jldupont: reusing the failed connection?
[16:02] sustrik	that would be pretty dangerous imo
[16:02] jldupont	no. drop & reconnect
[16:02] sustrik	ah
[16:02] jldupont	( aka tear-down/bring-up )
[16:02] sustrik	that has no implication on wire protocol imo
[16:03] sustrik	as you start anew each time
[16:03] sztanpet	doesnt 0mq automatically reconnect you btw?
[16:03] sustrik	yes
[16:03] jldupont	... but the trouble is mid-flight
[16:03] sztanpet	what of it
[16:03] sztanpet	its the same with tcp, i cant quite see your problem
[16:05] jldupont	like I said, if a problem occurs mid-flight, then the connection can be detected faulty and tear/down/bring-up would occur.
[16:05] jldupont	if that failure mode is taken care of in 0mq, then let's finish this discussion.
[16:12] sustrik	yes, it is
[16:12] jldupont	ok
[18:41] chris2	hi. i'm trying to use the zmq2 gem for pub/sub
[18:42] chris2	and i can pub, but nothing appears at sub
[18:43] cremes	chris2: when you say "gem" i assume you are using ruby; is that correct?
[18:43] chris2	erm, yes
[18:43] chris2	right
[18:43] chris2	i can paste my code too
[18:43] cremes	please do (use pastie or a gist)
[18:44] chris2	http://pastie.org/1016242
[18:45] chris2	i ported that from the py at http://nichol.as/zeromq-an-introduction
[18:45] cremes	ok; is the pub loop printing out each message that it sends?
[18:45] chris2	the pub sends nicely
[18:46] chris2	the sub doesnt show anything
[18:46] cremes	let me try and run it myself... give me a couple...
[18:46] chris2	i ran strace on the sub, and it shows
[18:46] chris2	recvfrom(8, 0x7fff8c0810f0, 48, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
[18:47] chris2	roughly pulsed like the pub
[18:47] chris2	so i guess there is some kind of transmission
[18:51] chris2	hah, fixed it
[18:51] cremes	chris2: i'm having some trouble installing the zmq2 gem; it installed fine but now complains about a missing file (zmq_native)
[18:51] cremes	what was it?
[18:51] chris2	s.setsockopt(ZMQ::SUBSCRIBE, "")
[18:51] chris2	i need to subscribe to everything
[18:51] cremes	ah!
[18:51] chris2	the py api seems to do that by default
[18:52] cremes	good catch
[18:52] chris2	py api seems a bit nicer in general
[18:52] chris2	but im not sure if zmq2 is even official
[18:52] cremes	what does official mean anyway? use what works best.
[18:53] chris2	sure
[18:53] chris2	but then the api should be similar i think
[18:53] chris2	ok, multiple sub works too
[18:53] cremes	the zeromq guys don't anything about ruby (or python or php or lua) so they are hoping community folks take over the bindings
[18:54] chris2	http://www.zeromq.org/bindings:ruby ?
[18:54] cremes	yes; also http://www.zeromq.org/bindings:ruby-ffi for jruby, rubinius and others
[18:54] cremes	(i wrote those)
[18:55] cremes	er, i wrote the ffi bindings for ruby
[18:55] chris2	ok
[18:58] chris2	also, is there a thing i can place inbetween to add persistence of the messages?
[19:00] cremes	chris2: there isn't anything in the 0mq library that provides persistence; you would need to create that yourself
[19:00] chris2	just figured someone already wrote a daemon or something to do that in general
[19:00] cremes	not yet; you could be the first
[19:00] chris2	its probably not entirely trivial then :)
[19:01] cremes	nope, not trivial; messages can be in the 0mq queue, a system buffer, a driver buffer, on the wire, or in those same buffers on the receiving end
[19:01] chris2	yeah
[19:01] cremes	you would need an explicit ack that a message was persisted
[19:02] cremes	it's a thorny issue
[19:02] chris2	true that
[19:26] CIA-17	rbzmq: 03Brian Buchanan 07master * r5be8033 10/ rbzmq.c : Make the flags argument to ZMQ::Socket.send() and ZMQ::Socket.recv() optional. Resolves issue #1. - http://bit.ly/aeY6wZ
[19:26] chris2	awesome quick :D
[19:28] chris2	is pub/sub filtering really implemented on the pub side?
[19:28] chris2	or does sub get everything and then filters
[19:29] CIA-17	rbzmq: 03Brian Buchanan 07master * r7043d45 10/ rbzmq.c : Update method signatures for socket.send and socket.recv in the documentation. - http://bit.ly/aG9JWs
[19:33] sustrik	chris2: unfortunately, yes
[19:33] chris2	kinda lame?
[19:33] sustrik	passing the subscriptions upstream is on the roadmap for a long time
[19:33] sustrik	but it's not trivial and so far there was noone to sponsor the development
[19:34] chris2	okay
[19:34] chris2	zmq is still cool
[19:34] sustrik	:)
[19:34] chris2	i guess i have some use for it
[20:12] mml	is there a way to buffer unconsumed messages in a pub/sub config?
[20:12] mml	i.e. all consumers are down, but i'm using a forwarder process..
[20:12] mml	i want the forwarder to que msgs until the consumers return
[20:13] mml	like ::SWAP et. al.
[20:13] mml	or is that to 'brokery'
[20:13] mml	s/to/too
[20:17] mml	and, is the zmq forwarder config syntax documented somewhere?
[20:20] cremes	mml: the forwarder "device" included with 0mq doesn't do any buffering
[20:20] cremes	the forwarder also has no idea if anyone is connected to it; it would need to "heartbeat" or otherwise communicate with downstream clients
[20:20] cremes	all of this is possible, but you would need to write the logic yourself
[20:21] cremes	an no, the configuration for forwarder isn't documented anywhere; the code is short enough to figure it out though
[20:21] mml	cremes: tx
[20:22] cremes	you're welcome
[20:31] mml	the ruby bindings worked great fwiw
[20:32] bj_990	are you guys as slick as ice http://www.zeroc.com/ ?
[20:53] mivert	bj_990: yes? pretty different though. Ice does RPC and serialization, ÃMQ does messaging.
[20:54] bj_990	i just saw that
[20:54] bj_990	:D
[20:54] bj_990	thanks mivert
[20:54] mivert	np
[21:22] bj_990	this looks great
[21:30] bj_990	be back in the near future...