ZeroMq IRC Log

Monday May 17, 2010

[Time] Name	Message
[07:24] mike8901	what's the prefered way to typeset zeromq in LaTex? :) \varnothing?
[07:38] sustrik	:)
[07:39] sustrik	i personally prefer \varnothing
[07:39] sustrik	others may prefer 0MQ or ZeroMQ
[07:45] foob	hey
[07:45] foob	is there a zeromq latex macro?
[08:10] sustrik	foob: what exactly do you mean?
[08:26] mike8901	joshua, was just talking about this with sustrik :)
[08:26] mike8901	\varnothing is the way to go
[13:42] cremes	sustrik: question on cppbutterfly example regarding bind versus connect
[13:43] cremes	why does the send_requests program bind to a DOWNSTREAM socket? i thought transmission implied a "connect"
[13:45] cremes	similar question for the component program; it connects to an UPSTREAM socket though it only receives data
[13:45] cremes	i thought reception implied "bind"
[13:45] cremes	ping me if you get a chance
[14:26] sustrik	cremes: ping
[14:26] sustrik	let me see
[14:27] sustrik	(it was gonzalo diethelm who wrote the example)
[14:27] cremes	sustrik: pong
[14:27] sustrik	looking at butterfly...
[14:28] sustrik	ok, i see
[14:28] guido_g	ahhh... pipelines...
[14:29] guido_g	what is the best way of delivering error messages back to the originator of a message?
[14:29] sustrik	cremes: have a look here: http://www.zeromq.org/tutorials:butterfly#toc7
[14:29] sustrik	guido_g: req/rep
[14:29] sustrik	or you mean in piplined model?
[14:30] guido_g	in pipelined model
[14:30] sustrik	the errors are usually logged rather than returned
[14:30] cremes	yes, i stepped through that example; what should i look at
[14:30] guido_g	let's say than one of the processes in stage 3 of a pipeline decides that there is an error
[14:30] sustrik	the diagram
[14:30] sustrik	note that there are N component1's
[14:30] sustrik	each of them _connects_ to send_requests application
[14:31] sustrik	thus send_requests has to bind
[14:31] sustrik	guido_g: yes?
[14:31] cremes	ah...
[14:31] guido_g	so given the picture we're looking at
[14:32] guido_g	every intermediate needs to have an error exit back to the process which created the message in question
[14:32] guido_g	or something along the lines
[14:32] sustrik	guido_g: does it?
[14:32] cremes	since send_requests has no knowledge of how many component1s there might be, it binds to a socket;
[14:32] guido_g	how would you do it?
[14:33] sustrik	the point with pipelined model is that messages are passed downstream an forgotten
[14:33] cremes	then the component1s can connect to it and maybe in some cases be transient (disappear after a while) without send_requests caring
[14:33] sustrik	thus returning the error to particular point of the pipeline makes little sense
[14:33] cremes	it just pumps data to the socket and the lib handles the round-robin to each component1
[14:33] guido_g	ahh no
[14:33] cremes	i think i get it
[14:34] guido_g	sustrik: i'll try to explain the (imagined) problem better
[14:34] sustrik	cremes: yes, that's the point
[14:34] sustrik	components are transient
[14:34] sustrik	guid_g: go on
[14:35] sustrik	cremes: that way you can add more components when the load is big
[14:35] cremes	sustrik: next question; would a REQ socket bind to an address if it might have N possible components connecting to it via a REP socket?
[14:35] sustrik	yes
[14:35] cremes	ok
[14:37] cremes	sustrik: thanks for helping me out; i need to think about this some more...
[14:38] sustrik	cremes: no problem
[14:39] sustrik	we are about to make a video about the basic messaging patters so questions from people actually struggling to understand it are pretty valueable
[14:40] guido_g	better description, i hope: http://paste.pocoo.org/show/214874/
[14:40] cremes	no one struggles more than me! ;)
[14:41] sustrik	:)
[14:41] sustrik	guido_g: all the errors are sent to a single originating node?
[14:41] guido_g	to the originating node of this message
[14:41] sustrik	are there many of them?
[14:42] guido_g	yes, probably
[14:42] sustrik	what about the replies?
[14:42] sustrik	ain't there any?
[14:42] guido_g	who needs replies? :)
[14:43] sustrik	error is kind of reply, i would assume that if there are errors passed back the replie should be passed back as well
[14:43] guido_g	in the pure pipeline the last stage would generate the replies, right?
[14:43] sustrik	right
[14:43] sustrik	i just don't get why the errors are to be delivered back to the originating node is the replies are not
[14:43] guido_g	both are
[14:44] guido_g	but
[14:44] guido_g	the error can occur in every stage, where the _normal_ reply comes out of the last stage of the pipeline
[14:44] sustrik	sure, but why not use the same mechanism for delivering both of them?
[14:45] guido_g	so one would send the errors downstream and every stage has to hand down further?
[14:45] guido_g	hmm hmm hmm
[14:45] sustrik	that's one option
[14:45] guido_g	ok
[14:45] guido_g	what would be an alternative?
[14:45] sustrik	how you get the replies back to sender?
[14:46] guido_g	like in the butterfly diagram
[14:46] guido_g	last stage reports back
[14:46] sustrik	how does it know who's the originator?
[14:46] guido_g	my idea was a short-cut for errors detected early in the pipeline
[14:46] sustrik	in butterfly there's only one requester
[14:47] guido_g	in this example yes
[14:47] sustrik	guido_g: i am not sure of what you are doing, but let me guess:
[14:48] sustrik	you want a "server" that clients connect to and send messages to
[14:48] sustrik	they get replies from the "server"
[14:48] sustrik	the "server" itself is a pipeline
[14:48] sustrik	right?
[14:49] guido_g	yes, the pipeline processes requests originating from -- let's say -- lots of web-servers
[14:49] sustrik	ok, i see
[14:50] sustrik	so each client has a REQ socket to send request to the pipeline and receive a reply from it
[14:51] sustrik	pipeline itself has a central node, that gets the requests and sends the replies
[14:51] guido_g	i thought more of a downstream socket
[14:51] sustrik	you want to get replies back?
[14:51] sustrik	that's req/rep
[14:51] sustrik	the internal working of the server may be "parallelised pipeline"
[14:51] guido_g	yes, but not neccessarily the same way i sent the request
[14:51] sustrik	hm
[14:51] sustrik	why so?
[14:52] guido_g	i wanted to circumvent the synchronous nature of the req/rep and simply feed the req down to the pipeline
[14:53] sustrik	ah, i am getting to that
[14:53] sustrik	there is XREP/XREQ which are async
[14:53] guido_g	otherwiese i would need lots of connections on the "back side" of the web-servers, which wouldn't help them
[14:53] sustrik	yup, there should be only 1 connection
[14:54] sustrik	so the webserver is the client
[14:54] sustrik	if the request are to be sync it owns REQ socket
[14:54] guido_g	spoken in terms of the messaging patterns, i seem to need something like a n -> 1 thing
[14:54] sustrik	if it is to be async it owns XREQ socket
[14:55] guido_g	ok
[14:55] sustrik	it speaks to a single node that coordinates the pipeline
[14:55] sustrik	let's call it "coordinator"
[14:55] sustrik	coordinator is both beginning and end of the pipeline
[14:55] guido_g	then i need an intermediate node holding the rep sockets and pushing the messages downstream, right?
[14:55] sustrik	yes, that's the coordinator
[14:56] guido_g	cool
[14:56] guido_g	thanks
[14:56] sustrik	it both sends the messages downstream
[14:56] sustrik	and passes the replies back to the clients
[14:56] sustrik	however, you should use trunk version to get the correct behaviour
[14:57] sustrik	moreover, you should expect that each request will be composed of 2 parts
[14:57] guido_g	at the moment i'll try to wrap my head around the patterns and possibillities
[14:58] sustrik	ok, let's leave the details now
[14:58] sustrik	in overall i would say you need a pipeline
[14:58] guido_g	yes, that's definite
[14:58] sustrik	i.e. the same thing as in butterfly example
[14:58] sustrik	but it's endpoint should translate req/rep messages into pipeline messages
[14:59] guido_g	that's the point where i got the problems
[14:59] sustrik	yes
[14:59] sustrik	but it's not complex
[14:59] guido_g	how to translate the sync behvaiour into async and collect all information w/o generating uneccessary traffic
[15:00] sustrik	XREP socket in the "pipeline coordinator"
[15:00] guido_g	which also is the endpoint for the results
[15:00] sustrik	reight
[15:00] guido_g	i think i got it :)
[15:00] sustrik	the pipeline is a circle
[15:01] sustrik	the clients connect to the coordinator node of the circle
[15:01] guido_g	hmmm... yes, it's a circle
[15:01] guido_g	how simple...
[15:02] guido_g	the reduction of the whole messaging into four "patterns" is one of the best things since sliced bread :)
[15:03] sustrik	yeah, i think that's the main "invention" with 0mq
[15:03] sustrik	i should patent it :)
[15:04] guido_g	oh no...
[15:04] sustrik	no worries
[15:04] guido_g	then we'll call you bill or steve
[15:04] guido_g	:)
[15:04] sustrik	actually i rather would like to get it through IETF to make it a prior art
[15:05] sustrik	no exact idea how though...
[15:05] guido_g	why is that needed?
[15:06] guido_g	isn't it enough to have a publication with this idea?
[15:06] guido_g	like an article in the acm queue or something
[15:06] sustrik	well, that's the point, if you publish it as IETF I-D it's
[15:07] sustrik	a) published
[15:07] sustrik	b) dated
[15:07] sustrik	c) IETF is a trusted source
[15:07] guido_g	ok, i didn't knew that
[15:08] sustrik	just my impression :)
[15:08] guido_g	ahhh i see
[15:08] sustrik	they published TCP/IP specs
[15:08] sustrik	so they have to be trusted in a way
[15:09] sustrik	otherwise the internet as a whole could be attacked by a troll
[15:09] guido_g	i'd say that a tracable publication would be enough to have prior art working
[15:09] sustrik	sure
[15:10] sustrik	i'll try to get that done somehow
[15:11] guido_g	the video? should be enough :)
[15:13] sustrik	nor sure about the dating
[15:13] sustrik	the data can be easily faked in a video
[15:13] guido_g	right
[15:13] sustrik	either a scientific journal
[15:13] guido_g	best would be an article in a (paper) magazine
[15:14] sustrik	yup, there's one at LWN
[15:14] sustrik	still not sure that's enough
[15:14] guido_g	these are collected by the libraries etc. and need to have a date/issu-no. etc.
[15:14] sustrik	right
[15:14] guido_g	if this is not enough, nothing would be
[15:14] sustrik	one reason i like the IETF way is that's easy to publish
[15:14] sustrik	the magazine may be more complex
[15:15] guido_g	but for an article in a magazine you do get extra money :)
[15:15] sustrik	!
[15:15] guido_g	just an idea though...
[15:16] guido_g	ahh... my food is arriving
[15:17] sustrik	bot appettit!
[15:17] sustrik	bon*
[15:23] bgranger	sustrik: can a single zmq msg be sent to multiple socket without called zmq_msg_copy?
[15:25] cremes	is a XREQ/XREP socket the same as REQ/REP passed with ZMQ_NOBLOCK? or are the X* sockets a new beast?
[15:25] sustrik	bgranger: no, each send "empties" the message
[15:26] sustrik	so you have to copy it to be able to send it twice
[15:26] sustrik	cremes: no, it's kind of different
[15:26] sustrik	XREQ/XREP have to route the replies to correct requesters
[15:26] bgranger	sustrik: OK, but zmq_msg_copy doesn't actually copy the data, it simply makes a reference?
[15:26] sustrik	right
[15:27] bgranger	OK, we are implementing a Python Message object that will allow us to do non-copying recv's and multiple sends.
[15:28] sustrik	yes, that sound useful
[15:29] cremes	bgranger: looking forward to seeing that; i can't seem to figure out how to do it in ruby
[15:29] bgranger	When I get a message from a recv, it has a ref count of 1, so it can be sent once correct?
[15:29] cremes	sustrik: how does XREQ/XREP differ?
[15:29] sustrik	branger: right
[15:29] bgranger	cremes: it has been super subtle to figure out, but I think we have it. Will post on the list when it is done.
[15:30] cremes	bgranger: excellent </burns>
[15:30] sustrik	cremes: how it works is that XREP socket of receiving a request tags it with identity of the sender, when reply is to be sent it takes the tag and routes the reply to the correct requester
[15:30] cremes	oh, you're building on the multi-part message stuff?
[15:31] sustrik	yes, that's it
[15:31] sustrik	imagine a REQ sending a request
[15:31] cremes	ok, so i can have multiple requests "in flight" with XREQ
[15:31] sustrik	you receive it using a XREP on the other side
[15:31] sustrik	it'll look like this"
[15:32] sustrik	<part1>ApplicationA<part1><part2><part2><part3>my message data<part3>
[15:32] sustrik	part3 is the data you've sent
[15:32] sustrik	part1 is the name of the sending application
[15:32] cremes	ok
[15:32] sustrik	part2 is empty, it's a delimiter splitting the address and your data
[15:33] sustrik	and yes, you can have many messages in flight
[15:33] cremes	looking forward to seeing a code example with 2.0.7 :)
[15:39] bgranger	cremes: we have some examples of using XREQ in pyzmq in the examples/kernel directory. It is not a simple example, but shows what can be done.
[15:40] cremes	bgranger: i'll take a look; thanks
[16:20] cremes	does it make any sense for a socket to both bind and connect to different addresses? if so, what is an example?
[16:21] guido_g	a socket is either bound or connected, not both afaik
[16:35] sustrik	cremes: actually, you can do that
[16:35] sustrik	not sure about use case
[16:35] sustrik	let me think,,,
[16:36] guido_g	what would happen?
[16:36] sustrik	well, you would initiate some connections via connect
[16:37] sustrik	and peers may initiate other connections via you being bound to a specific port
[16:37] sustrik	both types of connections would be handled equally
[16:38] sustrik	ok, use cases are not really natural
[16:38] sustrik	but let's imagine this:
[16:38] sustrik	i am writing a service
[16:38] sustrik	i'll bind to a port
[16:39] sustrik	to that clients can connect and ask me to do work
[16:39] sustrik	additionally, imagine there's a big hub for dispatching work on the Internet somewhere
[16:40] sustrik	i may want to use free cycles on my box to execute foreign requests (for money of course)
[16:40] sustrik	so i _connect_ my service to the hub
[16:40] sustrik	now i have a socket that's both connected and bound
[16:41] guido_g	and if one really does that, he might see a lot of exploded heads :)
[16:42] sustrik	:)
[17:04] cremes	more bind / connect questions...
[17:05] cremes	is there anything wrong about having a REQ socket and REP socket both bind to the same address and send/recv to/from each other?
[17:05] cremes	based on our earlier discussions, this appears to be valid since this setup allows for multiple REQ and REP sockets to communicate in a load balanced manner
[17:06] cremes	s/address/endpoint/
[17:08] cremes	when i just tried it (using the local_lat/remote_lat samples) i got an EADDRINUSE error from remote_lat
[17:08] sustrik	you cannot bind two sockets to the same NIC and port
[17:09] sustrik	TCP doesn't allow for that kind of thing
[17:10] cremes	sustrik: what about ipc or inproc transports?
[17:10] sustrik	basically the same thing
[17:10] cremes	ok
[17:11] cremes	probably more questions later/tomorrow...
[17:11] cremes	i hope this is helping for your video!
[17:11] sustrik	surely it does :)
[20:31] bgranger	sustrik: what is the logic for zmq_msg_copy calling zmq_msg_close on the src msg?
[20:32] bgranger	NEvermind I see now.
[21:41] mikko	what a day.. can't do the win builds today
[21:41] mikko	retrying tomorrow
[21:41] mikko	nn