ZeroMq IRC Log

Sunday March 6, 2011

[Time] Name	Message
[06:55] guido_g	hello world!
[09:39] pieter_hintjens	hi all, I just made the 2.1.2 rc2 release
[09:40] guido_g	and lots of old tags on zeromq2-1
[09:44] pieter_hintjens	yeah, I was wondering about those... OK, will delete them...
[09:44] guido_g	pieter_hintjens: can we change the woker side framing of the mdp? would like to have an empty frame before the version, so it's the same as for the client
[09:45] guido_g	makes the code much easier and would be more consistent
[09:45] pieter_hintjens	guido_g: an extra frame for nothing...
[09:45] pieter_hintjens	I'd thought of it but figured it was pointless
[09:45] guido_g	it's not
[09:45] pieter_hintjens	ok, let me think about it a sec
[09:46] guido_g	if all goes over one zmq socket, you need to figure out where the message came from
[09:46] guido_g	if we could strip the return path with a generic function, the first part of the remaining message would be the sender
[09:47] pieter_hintjens	consistency? hmm, a novel idea...
[09:47] guido_g	sigh
[09:47] guido_g	,)
[09:48] pieter_hintjens	so the reason there's an empty frame before the version is purely to emulate a REQ socket
[09:49] pieter_hintjens	there's no consistent return path afaics
[09:50] pieter_hintjens	tell me precisely what commands you'd want to change, guido_g
[09:50] pieter_hintjens	ready, heartbeat, request, reply, disconnect?
[09:51] guido_g	it would change all of them
[09:51] pieter_hintjens	it would not make these consistent with client commands
[09:51] pieter_hintjens	because the worker commands all have that frame with the command number
[09:52] pieter_hintjens	i'm unsure what you mean by 'return path'
[09:52] pieter_hintjens	are you talking about the broker or the APIs?
[09:52] guido_g	it make the overall structure more consistent
[09:52] pieter_hintjens	overall structure of what?
[09:53] guido_g	i do talk about the sequence of message frames send to the broker
[09:53] pieter_hintjens	ok, so there is a reason the client and worker protocols are quite different
[09:53] guido_g	sigh
[09:54] guido_g	it makes the more complex, when using a shared socket in the broker
[09:54] pieter_hintjens	so use two sockets, that's fine
[09:54] pieter_hintjens	you've not actually explained what you mean, except "more consistent" and "sigh"
[09:54] pieter_hintjens	:-)
[09:54] pieter_hintjens	please be precise
[09:54] guido_g	to me it's obvious
[09:54] pieter_hintjens	obviously
[09:55] guido_g	the client sends id followed by delimiter (the empty frame)
[09:55] pieter_hintjens	wait, please
[09:55] pieter_hintjens	the client does not send ID
[09:55] guido_g	the worker send id and then immediately the 6 byte proto version
[09:56] guido_g	the req does
[09:56] pieter_hintjens	nope, the client does not send its id
[09:56] pieter_hintjens	if you believe it does, we have a communications issue
[09:56] pieter_hintjens	the xrep socket prepends the ID of the originator
[09:56] guido_g	ok so that is what the app sees
[09:57] pieter_hintjens	the client sends (if it's a REQ) the protocol version followed by the service name followed by the message
[09:57] guido_g	ok, again
[09:57] pieter_hintjens	the client sends (if it's an XREQ) this, prepended by an empty frame
[09:57] pieter_hintjens	the 'app' is what?
[09:57] pieter_hintjens	you need to be precise, we have three nodes here, each seeing different things
[09:57] guido_g	the program getting the messages from the xreq sees a leading id
[09:58] guido_g	followed by an empty frame
[09:58] guido_g	this was a recv gives
[09:58] pieter_hintjens	ok
[09:58] guido_g	this should be the same for both sides
[09:58] guido_g	so one could use the same function for splitting this information and the following part reliably
[09:58] pieter_hintjens	well, it's 2 lines of code more or less in the broker
[09:59] pieter_hintjens	and it's an extra frame for the worker protocol for absolutely no advantage
[09:59] pieter_hintjens	i will not make the protocol more complex to create minimal advantage to implementations
[09:59] pieter_hintjens	sorry
[10:00] pieter_hintjens	the frames need to have a functional purpose
[10:00] guido_g	says the guy who needs six byte to distinguish 2 protocols and their versions
[10:00] pieter_hintjens	"I can reuse the same code" is not a functional purpose
[10:00] guido_g	it does
[10:00] guido_g	it delimits the routing part from the mdp protocol information
[10:00] pieter_hintjens	here is how my code does it:
[10:00] pieter_hintjens	char *sender = zmsg_unwrap (msg);
[10:00] pieter_hintjens	char *header = zmsg_pop (msg);
[10:01] pieter_hintjens	totally generic, works for both client and worker messages
[10:01] pieter_hintjens	if (strcmp (header, MDPC_CLIENT) == 0)
[10:01] pieter_hintjens	s_client_process (self, sender, msg);
[10:01] pieter_hintjens	else
[10:01] pieter_hintjens	if (strcmp (header, MDPW_WORKER) == 0)
[10:01] pieter_hintjens	s_worker_process (self, sender, msg);
[10:01] pieter_hintjens	I'm surprised it's easier in C than in Python
[10:02] guido_g	i don't know what these function do
[10:02] pieter_hintjens	unwrap removes an address plus optional empty part
[10:02] guido_g	but the client header start w/ an emty frame per spec, can't see that handled here
[10:02] guido_g	ahhh
[10:02] pieter_hintjens	since this question of empty parts occurs in many places
[10:03] guido_g	only one address
[10:03] pieter_hintjens	you want to manage an address stack?
[10:03] guido_g	there is nothing to manage
[10:03] pieter_hintjens	there is no stack
[10:03] guido_g	it's just a list of ids followed by an empty frame
[10:03] pieter_hintjens	there is no list
[10:03] pieter_hintjens	it's exactly one
[10:04] pieter_hintjens	the only way you can get a list is to chain multiple brokers
[10:04] pieter_hintjens	and that won't work without other changes
[10:04] pieter_hintjens	this, at least, is the broker's view of the world
[10:05] pieter_hintjens	workers, as terminators, can manage a stack without trouble
[10:05] pieter_hintjens	so the request command emulates a REP socket (stack of addresses + empty part + body)
[10:06] pieter_hintjens	guido_g: ok, here's a compromise
[10:06] pieter_hintjens	your point is noted, I had the same idea but it didn't stick
[10:07] pieter_hintjens	we'll probably want to make more radical changes to MDP anyhow, later
[10:08] guido_g	and where's the compromise?
[10:09] guido_g	i mean we might do it later doesn't sound like one
[10:09] guido_g	but, ok i'll implement the way it is
[10:11] pieter_hintjens	Meaning, the issue remains open, it's not rejected, but needs more thought
[10:11] pieter_hintjens	I'm not sure the 'many frames' approach is ideal at all
[10:12] guido_g	no, it's significantly slower, but ra performance isn't important here i was told
[10:12] guido_g	s/ra/raw/
[10:13] Guthur	pieter_hintjens, was that second archive i sent yesterday ok?
[10:13] pieter_hintjens	Guthur: yup, I already pushed those translations, thanks
[10:14] pieter_hintjens	guido_g: indeed, I want the semantics of routing to be clear, this is educational
[10:14] Guthur	pieter_hintjens, cool, cheers
[10:14] pieter_hintjens	Guthur: only a dozen or so more, and you can get the Guide in C# :-)
[10:15] Guthur	I hope to have ch3 finished soon enough
[10:15] Guthur	1/2 half through peering2
[10:16] Guthur	I'm going with some friends to airsoft today, so there's likely to little progress today
[10:16] pieter_hintjens	that blasted hintjens guy keeps moving the goal posts...
[10:17] Guthur	is ch4 still in quite a bit of flux
[10:17] pieter_hintjens	What's there now is stable, it's coming to a close
[10:18] pieter_hintjens	two more large patterns and it's done
[10:18] Guthur	nice
[10:18] Guthur	is this the final chapter
[10:18] pieter_hintjens	lol, no :-)
[10:18] pieter_hintjens	there'll be a few more chapters
[10:23] Guthur	ok, got to go, later all
[11:09] mikko	good morning
[11:16] pieter_hintjens	hi mikko!
[11:22] mikko	solaris isn't playing ball :/
[11:23] mikko	i tried to catch up with you on friday i think
[11:23] mikko	i did some build refactoring and would've warned that there might be some emails coming to your way
[11:23] mikko	i think you get failure emails at least for some builds
[11:23] pieter_hintjens	I saw them, np
[11:24] pieter_hintjens	I assumed you were changing something
[11:25] mikko	i am exploring using amazon ec2 to add some capacity
[11:25] mikko	win64 / linux64
[11:25] pieter_hintjens	interesting...
[11:26] mikko	start build slaves on demand and shutdown after they are done
[11:26] mikko	jenkins has a plugin for this so it should be fairly easy
[11:27] pieter_hintjens	and amazon charges only for what is actually used?
[11:27] pieter_hintjens	I've never tried ec2
[11:29] mikko	yeah
[11:30] mikko	linux is $0.025 per hour
[11:30] mikko	and windows is $0.035 per hour
[11:30] mikko	for the smallest instances
[11:56] guido_g	ha! python mdp broker just routed its first requests
[11:58] mikko	nice!
[12:01] guido_g	jepp, write it in ca. 4 hours :)
[12:01] guido_g	s/write/wrote/
[12:31] guido_g	pieter_hintjens: error while compling https://gist.github.com/857252
[12:55] guido_g	tiem for a break
[12:55] guido_g	*time
[15:35] pieter_hintjens	guido_g: 4 hours for a messaging broker, sweet
[15:35] pieter_hintjens	I fixed that error in mdbroker and also added the empty frame as you wanted
[15:38] guido_g	great thanks!
[15:49] apexi200sx	Hi,does this channel get logged to anywhere ?
[16:30] travlr	apexi200sx: IRC log is located at http://travlr.github.com/zmqirclog/
[17:13] guido_g	pieter_hintjens: mdbroker seems to be broken
[17:14] pieter_hintjens	yeah, probably, I didn't retest it
[17:14] pieter_hintjens	sigh
[17:14] pieter_hintjens	am distracted today, sorry
[17:14] guido_g	np
[17:14] guido_g	my broker works :)
[17:14] pieter_hintjens	you win 9,000 internets, sir!
[17:14] pieter_hintjens	:-)
[17:15] guido_g	shit, just bought a new one last week
[17:15] pieter_hintjens	np, keep them in reserve
[17:15] guido_g	right
[17:15] guido_g	as as i can tell from mdclient2, the broker should queue incomming client requests?
[17:16] pieter_hintjens	yes, it should
[17:16] guido_g	ok
[17:17] pieter_hintjens	clearly if N clients send requests at once, the broker has to queue them until workers are available
[17:18] guido_g	"Workers will handle at most one request a time, and will issue exactly one reply for each successful request." <- from the spec
[17:18] pieter_hintjens	workers, yes
[17:18] pieter_hintjens	but clients no
[17:18] guido_g	mdclient2 violates that restriction
[17:18] pieter_hintjens	clients != workers, right
[17:18] guido_g	note to self: read carefully
[17:20] guido_g	what about requests for a service that was never registered?
[17:21] pieter_hintjens	indeed, that is not covered by the spec
[17:21] pieter_hintjens	I think the best semantics are that the broker accepts the client requests anyhow
[17:21] pieter_hintjens	and tries to deliver them at some future time
[17:22] guido_g	on the risk of filling all memory with useless requests
[17:22] pieter_hintjens	however it's open... if the broker does this, it risks accumulating undeliverable messages for non-existent services
[17:22] pieter_hintjens	and if it does not do this, it breaks the asynchronous model where nodes come and go randomly
[17:23] guido_g	so there is room for a version 2 of the spec :)
[17:23] pieter_hintjens	for sure, like I said, this is a learning process...
[17:23] guido_g	i'll implement a queue per service now
[17:23] pieter_hintjens	yes, that's what I've done
[17:24] pieter_hintjens	that lets you dispatch messages to services when workers appear
[17:25] guido_g	done in a few minutes
[17:30] pieter_hintjens	ok, fixed the mdworker api, it didn't add the empty part. broker seemed to be fine
[17:30] pieter_hintjens	pushed to master
[17:32] guido_g	will check
[17:33] guido_g	no output from mdworker or mdbroker
[17:36] pieter_hintjens	set VERBOSE to 1 in mdbroker head, rebuild...
[17:36] pieter_hintjens	I've fixed the spec to clarify how the broker should handle client requests for non-existent services
[17:37] guido_g	getting assertion in worker when using my broker:
[17:37] guido_g	mdworker: mdwrkapi.c:190: mdwrk_recv: Assertion `zmsg_parts (msg) >= 3' failed.
[17:37] guido_g	Aborted (core dumped)
[17:37] guido_g	ok, will do
[17:37] pieter_hintjens	did you add the empty part?
[17:37] guido_g	yes
[17:37] pieter_hintjens	empty + protocol + command = 3
[17:38] pieter_hintjens	you can also enable message debugging in mdworker, pass 1 instead of 0 to API constructor
[17:39] guido_g	ahhh ok
[17:46] guido_g	found it, forgot to adapt the hb message
[17:47] pieter_hintjens	you should use a single method to send to a particular peer, maybe
[17:47] guido_g	now the md* suite works fine
[17:47] pieter_hintjens	nice...
[17:47] pieter_hintjens	we have interoperability!
[17:47] guido_g	yes
[17:48] pieter_hintjens	that was fast
[17:49] pieter_hintjens	where's your git?
[17:49] guido_g	wait, still some glitches
[17:50] guido_g	and i need to test the queuing
[17:51] guido_g	where is the hb interval set in the broker?
[17:51] guido_g	need to sync mine w/ the md* values
[17:52] guido_g	ahh got it
[18:16] pieter_hintjens	g'night all...
[18:17] guido_g	oh
[18:17] guido_g	night
[18:51] guido_g	full interop betwenn c and python via mdp. goal achieved!
[22:22] Guthur	pieter_hintjens, The C# link in the language list for new examples I sent is not currently bold
[23:08] Skaag	I have an assertion while running the tests under /jzmq/perf
[23:09] Skaag	Assertion failed: msg_->flags & ZMQ_MSG_MORE (req.cpp:86)