Sunday March 6, 2011

[Time] NameMessage
[06:55] guido_g hello world!
[09:39] pieter_hintjens hi all, I just made the 2.1.2 rc2 release
[09:40] guido_g and lots of old tags on zeromq2-1
[09:44] pieter_hintjens yeah, I was wondering about those... OK, will delete them...
[09:44] guido_g pieter_hintjens: can we change the woker side framing of the mdp? would like to have an empty frame before the version, so it's the same as for the client
[09:45] guido_g makes the code much easier and would be more consistent
[09:45] pieter_hintjens guido_g: an extra frame for nothing...
[09:45] pieter_hintjens I'd thought of it but figured it was pointless
[09:45] guido_g it's not
[09:45] pieter_hintjens ok, let me think about it a sec
[09:46] guido_g if all goes over one zmq socket, you need to figure out where the message came from
[09:46] guido_g if we could strip the return path with a generic function, the first part of the remaining message would be the sender
[09:47] pieter_hintjens consistency? hmm, a novel idea...
[09:47] guido_g *sigh*
[09:47] guido_g ,)
[09:48] pieter_hintjens so the reason there's an empty frame before the version is purely to emulate a REQ socket
[09:49] pieter_hintjens there's no consistent return path afaics
[09:50] pieter_hintjens tell me precisely what commands you'd want to change, guido_g
[09:50] pieter_hintjens ready, heartbeat, request, reply, disconnect?
[09:51] guido_g it would change all of them
[09:51] pieter_hintjens it would not make these consistent with client commands
[09:51] pieter_hintjens because the worker commands all have that frame with the command number
[09:52] pieter_hintjens i'm unsure what you mean by 'return path'
[09:52] pieter_hintjens are you talking about the broker or the APIs?
[09:52] guido_g it make the overall structure more consistent
[09:52] pieter_hintjens overall structure of *what*?
[09:53] guido_g i do talk about the sequence of message frames send to the broker
[09:53] pieter_hintjens ok, so there is a reason the client and worker protocols are quite different
[09:53] guido_g *sigh*
[09:54] guido_g it makes the more complex, when using a shared socket in the broker
[09:54] pieter_hintjens so use two sockets, that's fine
[09:54] pieter_hintjens you've not actually explained what you mean, except "more consistent" and "*sigh*"
[09:54] pieter_hintjens :-)
[09:54] pieter_hintjens please be precise
[09:54] guido_g to me it's obvious
[09:54] pieter_hintjens obviously
[09:55] guido_g the client sends id followed by delimiter (the empty frame)
[09:55] pieter_hintjens wait, please
[09:55] pieter_hintjens the client does not send ID
[09:55] guido_g the worker send id and then immediately the 6 byte proto version
[09:56] guido_g the req does
[09:56] pieter_hintjens nope, the client does not send its id
[09:56] pieter_hintjens if you believe it does, we have a communications issue
[09:56] pieter_hintjens the xrep socket prepends the ID of the originator
[09:56] guido_g ok so that is what the app sees
[09:57] pieter_hintjens the client sends (if it's a REQ) the protocol version followed by the service name followed by the message
[09:57] guido_g ok, again
[09:57] pieter_hintjens the client sends (if it's an XREQ) this, prepended by an empty frame
[09:57] pieter_hintjens the 'app' is *what*?
[09:57] pieter_hintjens you need to be precise, we have three nodes here, each seeing different things
[09:57] guido_g the program getting the messages from the xreq sees a leading id
[09:58] guido_g followed by an empty frame
[09:58] guido_g this was a recv gives
[09:58] pieter_hintjens ok
[09:58] guido_g this should be the same for both sides
[09:58] guido_g so one could use the same function for splitting this information and the following part reliably
[09:58] pieter_hintjens well, it's 2 lines of code more or less in the broker
[09:59] pieter_hintjens and it's an extra frame for the worker protocol for absolutely no advantage
[09:59] pieter_hintjens i will not make the protocol more complex to create minimal advantage to implementations
[09:59] pieter_hintjens sorry
[10:00] pieter_hintjens the frames need to have a *functional* purpose
[10:00] guido_g says the guy who needs six byte to distinguish 2 protocols and their versions
[10:00] pieter_hintjens "I can reuse the same code" is not a functional purpose
[10:00] guido_g it does
[10:00] guido_g it delimits the routing part from the mdp protocol information
[10:00] pieter_hintjens here is how my code does it:
[10:00] pieter_hintjens char *sender = zmsg_unwrap (msg);
[10:00] pieter_hintjens char *header = zmsg_pop (msg);
[10:01] pieter_hintjens totally generic, works for both client and worker messages
[10:01] pieter_hintjens if (strcmp (header, MDPC_CLIENT) == 0)
[10:01] pieter_hintjens s_client_process (self, sender, msg);
[10:01] pieter_hintjens else
[10:01] pieter_hintjens if (strcmp (header, MDPW_WORKER) == 0)
[10:01] pieter_hintjens s_worker_process (self, sender, msg);
[10:01] pieter_hintjens I'm surprised it's easier in C than in Python
[10:02] guido_g i don't know what these function do
[10:02] pieter_hintjens unwrap removes an address plus optional empty part
[10:02] guido_g but the client header start w/ an emty frame per spec, can't see that handled here
[10:02] guido_g ahhh
[10:02] pieter_hintjens since this question of empty parts occurs in many places
[10:03] guido_g only one address
[10:03] pieter_hintjens you want to manage an address stack?
[10:03] guido_g there is nothing to manage
[10:03] pieter_hintjens there is no stack
[10:03] guido_g it's just a list of ids followed by an empty frame
[10:03] pieter_hintjens there is no list
[10:03] pieter_hintjens it's exactly one
[10:04] pieter_hintjens the only way you can get a list is to chain multiple brokers
[10:04] pieter_hintjens and that won't work without other changes
[10:04] pieter_hintjens this, at least, is the broker's view of the world
[10:05] pieter_hintjens workers, as terminators, can manage a stack without trouble
[10:05] pieter_hintjens so the request command emulates a REP socket (stack of addresses + empty part + body)
[10:06] pieter_hintjens guido_g: ok, here's a compromise
[10:06] pieter_hintjens your point is noted, I had the same idea but it didn't stick
[10:07] pieter_hintjens we'll probably want to make more radical changes to MDP anyhow, later
[10:08] guido_g and where's the compromise?
[10:09] guido_g i mean we *might* do it later doesn't sound like one
[10:09] guido_g but, ok i'll implement the way it is
[10:11] pieter_hintjens Meaning, the issue remains open, it's not rejected, but needs more thought
[10:11] pieter_hintjens I'm not sure the 'many frames' approach is ideal at all
[10:12] guido_g no, it's significantly slower, but ra performance isn't important here i was told
[10:12] guido_g s/ra/raw/
[10:13] Guthur pieter_hintjens, was that second archive i sent yesterday ok?
[10:13] pieter_hintjens Guthur: yup, I already pushed those translations, thanks
[10:14] pieter_hintjens guido_g: indeed, I want the semantics of routing to be clear, this is educational
[10:14] Guthur pieter_hintjens, cool, cheers
[10:14] pieter_hintjens Guthur: only a dozen or so more, and you can get the Guide in C# :-)
[10:15] Guthur I hope to have ch3 finished soon enough
[10:15] Guthur 1/2 half through peering2
[10:16] Guthur I'm going with some friends to airsoft today, so there's likely to little progress today
[10:16] pieter_hintjens that blasted hintjens guy keeps moving the goal posts...
[10:17] Guthur is ch4 still in quite a bit of flux
[10:17] pieter_hintjens What's there now is stable, it's coming to a close
[10:18] pieter_hintjens two more large patterns and it's done
[10:18] Guthur nice
[10:18] Guthur is this the final chapter
[10:18] pieter_hintjens lol, no :-)
[10:18] pieter_hintjens there'll be a few more chapters
[10:23] Guthur ok, got to go, later all
[11:09] mikko good morning
[11:16] pieter_hintjens hi mikko!
[11:22] mikko solaris isn't playing ball :/
[11:23] mikko i tried to catch up with you on friday i think
[11:23] mikko i did some build refactoring and would've warned that there might be some emails coming to your way
[11:23] mikko i think you get failure emails at least for some builds
[11:23] pieter_hintjens I saw them, np
[11:24] pieter_hintjens I assumed you were changing something
[11:25] mikko i am exploring using amazon ec2 to add some capacity
[11:25] mikko win64 / linux64
[11:25] pieter_hintjens interesting...
[11:26] mikko start build slaves on demand and shutdown after they are done
[11:26] mikko jenkins has a plugin for this so it should be fairly easy
[11:27] pieter_hintjens and amazon charges only for what is actually used?
[11:27] pieter_hintjens I've never tried ec2
[11:29] mikko yeah
[11:30] mikko linux is $0.025 per hour
[11:30] mikko and windows is $0.035 per hour
[11:30] mikko for the smallest instances
[11:56] guido_g ha! python mdp broker just routed its first requests
[11:58] mikko nice!
[12:01] guido_g jepp, write it in ca. 4 hours :)
[12:01] guido_g s/write/wrote/
[12:31] guido_g pieter_hintjens: error while compling
[12:55] guido_g tiem for a break
[12:55] guido_g *time
[15:35] pieter_hintjens guido_g: 4 hours for a messaging broker, sweet
[15:35] pieter_hintjens I fixed that error in mdbroker and also added the empty frame as you wanted
[15:38] guido_g great thanks!
[15:49] apexi200sx Hi,does this channel get logged to anywhere ?
[16:30] travlr apexi200sx: IRC log is located at
[17:13] guido_g pieter_hintjens: mdbroker seems to be broken
[17:14] pieter_hintjens yeah, probably, I didn't retest it
[17:14] pieter_hintjens sigh
[17:14] pieter_hintjens am distracted today, sorry
[17:14] guido_g np
[17:14] guido_g my broker works :)
[17:14] pieter_hintjens you win 9,000 internets, sir!
[17:14] pieter_hintjens :-)
[17:15] guido_g shit, just bought a new one last week
[17:15] pieter_hintjens np, keep them in reserve
[17:15] guido_g right
[17:15] guido_g as as i can tell from mdclient2, the broker should queue incomming client requests?
[17:16] pieter_hintjens yes, it should
[17:16] guido_g ok
[17:17] pieter_hintjens clearly if N clients send requests at once, the broker has to queue them until workers are available
[17:18] guido_g "Workers will handle at most one request a time, and will issue exactly one reply for each successful request." <- from the spec
[17:18] pieter_hintjens workers, yes
[17:18] pieter_hintjens but clients no
[17:18] guido_g mdclient2 violates that restriction
[17:18] pieter_hintjens clients != workers, right
[17:18] guido_g note to self: read carefully
[17:20] guido_g what about requests for a service that was never registered?
[17:21] pieter_hintjens indeed, that is not covered by the spec
[17:21] pieter_hintjens I think the best semantics are that the broker accepts the client requests anyhow
[17:21] pieter_hintjens and tries to deliver them at some future time
[17:22] guido_g on the risk of filling all memory with useless requests
[17:22] pieter_hintjens however it's open... if the broker does this, it risks accumulating undeliverable messages for non-existent services
[17:22] pieter_hintjens and if it does not do this, it breaks the asynchronous model where nodes come and go randomly
[17:23] guido_g so there is room for a version 2 of the spec :)
[17:23] pieter_hintjens for sure, like I said, this is a learning process...
[17:23] guido_g i'll implement a queue per service now
[17:23] pieter_hintjens yes, that's what I've done
[17:24] pieter_hintjens that lets you dispatch messages to services when workers appear
[17:25] guido_g done in a few minutes
[17:30] pieter_hintjens ok, fixed the mdworker api, it didn't add the empty part. broker seemed to be fine
[17:30] pieter_hintjens pushed to master
[17:32] guido_g will check
[17:33] guido_g no output from mdworker or mdbroker
[17:36] pieter_hintjens set VERBOSE to 1 in mdbroker head, rebuild...
[17:36] pieter_hintjens I've fixed the spec to clarify how the broker should handle client requests for non-existent services
[17:37] guido_g getting assertion in worker when using my broker:
[17:37] guido_g mdworker: mdwrkapi.c:190: mdwrk_recv: Assertion `zmsg_parts (msg) >= 3' failed.
[17:37] guido_g Aborted (core dumped)
[17:37] guido_g ok, will do
[17:37] pieter_hintjens did you add the empty part?
[17:37] guido_g yes
[17:37] pieter_hintjens empty + protocol + command = 3
[17:38] pieter_hintjens you can also enable message debugging in mdworker, pass 1 instead of 0 to API constructor
[17:39] guido_g ahhh ok
[17:46] guido_g found it, forgot to adapt the hb message
[17:47] pieter_hintjens you should use a single method to send to a particular peer, maybe
[17:47] guido_g now the md* suite works fine
[17:47] pieter_hintjens nice...
[17:47] pieter_hintjens we have interoperability!
[17:47] guido_g yes
[17:48] pieter_hintjens that was fast
[17:49] pieter_hintjens where's your git?
[17:49] guido_g wait, still some glitches
[17:50] guido_g and i need to test the queuing
[17:51] guido_g where is the hb interval set in the broker?
[17:51] guido_g need to sync mine w/ the md* values
[17:52] guido_g ahh got it
[18:16] pieter_hintjens g'night all...
[18:17] guido_g oh
[18:17] guido_g night
[18:51] guido_g full interop betwenn c and python via mdp. goal achieved!
[22:22] Guthur pieter_hintjens, The C# link in the language list for new examples I sent is not currently bold
[23:08] Skaag I have an assertion while running the tests under /jzmq/perf
[23:09] Skaag Assertion failed: msg_->flags & ZMQ_MSG_MORE (req.cpp:86)