Tuesday August 10, 2010

[Time] NameMessage
[07:00] mikko sustrik: there?
[07:01] sustrik mikko: hi
[07:01] mikko i am having a presentation at company evening today
[07:01] mikko suprisingly
[07:02] mikko how would you put "Internet Scale Messaging" in one short sentence?
[07:02] sustrik i am having a problem with that one myself
[07:02] sustrik maybe "decentralised" rather than Internet-scale
[07:03] sustrik no central authority
[07:03] sustrik same way as IP or Web
[07:03] mikko no guarantees on link speeds
[07:03] mikko i guess
[07:03] mikko unreliable networks
[07:03] sustrik no trust
[07:03] sustrik yes, that kind of things
[07:04] sustrik actually, comparison with ip/web can be an eye-opener imo
[07:04] sustrik people know how those work
[07:04] sustrik so "the same thing for distributed services"
[07:07] mikko I think I can explain it to the audience
[07:07] mikko luckily just a small audience
[07:11] sustrik well, good luck
[07:12] sustrik if there's any interesting feedback let us know
[08:51] pieter_hintjens mikko: perhaps something like "connecting nodes over an infinitely scalable, totally untrusted, and highly redundant network"
[08:57] aleator I need to make parallel requests for my worker nodes. Should I use XReq or device or something else entirely?
[08:58] pieterh parallel requests meaning the same requests sent to all workers, or requests distributed among workers?
[08:59] aleator different request for each worker, all at the same time
[09:00] guido_g you might want to use DOWNSTREAM/PUSH vs UPSTREAM/PULL sockets
[09:00] pieterh you could use the pipeline pattern, so ZMQ_DOWNSTREAM on client side, ZMQ_UPSTREAM on worker side, and then ZMQ_DOWNSTREAM from worker, back to ZMQ_UPSTREAM back on client
[09:00] pieterh DOWNSTREAM is now called PUSH and UPSTREAM is PULL btw
[09:01] pieterh you could also use a queue device, see figure 11 here:
[09:01] aleator Ok. So in essence I need to supply my own machinery for matching requests and replies with push/pull?
[09:02] pieterh aleator: yes, if you use push/pull you need to do the matching yourself
[09:02] pieterh if you use req-xrep-xreq-rep then it's done for you
[09:03] aleator I had originally just xreq/xrep and it seems to randomly utilise only some workers.
[09:03] aleator (I'm rather new at this)
[09:03] pieterh but the request-reply chain is hard to use if you try to start with xreq
[09:04] pieterh how many clients do you have?
[09:04] aleator In range 10-160, depending.
[09:04] aleator Tasks take few minutes each
[09:05] pieterh and each client sends one request at a time?
[09:06] aleator Ah. sorry, my bad. I meant workers. For clients there is at this point just one.
[09:06] pieterh so the client sends out jobs in parallel?
[09:06] aleator Thats my goal.
[09:06] pieterh then i'd suggest push/pull, and match the results up
[09:07] pieterh you can connect a pipeline in loops
[09:07] aleator Thank you.
[09:07] pieterh i don't have diagrams for that in the user guide yet, sorry
[09:07] aleator I just thought to ask since this seems rather basic operation that might have had support already.
[09:08] aleator But it is not difficult to do by hand either.
[09:08] pieterh you can do it by using the request-reply pattern as drawn in the diagram BUT then
[09:08] pieterh you need to use an XREQ socket at the client side and manually construct the frames
[09:09] pieterh since that is not documented, it's kind of hard to do
[09:10] aleator I did something like that with the result that just few workers got anything to do. Are there known pitfalls with this that I've might have stumbled upon?
[09:11] guido_g pieterh: is there a piece of code or a document that shows how req -> xrep is done?
[09:11] aleator Anyways, thank you for your help!
[09:12] pieterh guido_g: not in detail yet. The way I'm explaining XREP and XREQ in the user guide (basic section) is that they "extend" the request-reply pattern over intermediaries
[09:12] pieterh so you connect req-rep, or req-xrep-(code)-xreq-rep
[09:13] pieterh that is sufficient to use them in devices and stuff like multithreaded servers
[09:13] pieterh it's not sufficiently documented to do other stuff like aleator wants to do, e.g. client sending requests in parallel
[09:14] pieterh you can read the 0MQ code, people do, and it is helpful
[09:14] guido_g what is see is the (common?) pattern of simple clients -- using req -- connecting to a dispatcher (using x*)
[09:14] guido_g ok
[09:15] pieterh yes, that's probably the most common use case for X*
[09:15] pieterh mato: can i commit a man page change to git or will that provoke panic again?
[09:15] guido_g built something like that saturday morning while re-learning c++
[09:16] guido_g ØMQ is great for this kind of stuff
[09:22] aleator pieterh: Anyhow, I think my problem is so common that it might make good addition to zmg or tutorial of.
[09:23] pieterh aleator: i think so, yes. it's just a matter of documenting the request-reply chain fully
[09:23] pieterh i'll get to that in the user guide
[09:23] pieterh it's going to be huge...
[09:26] aleator Hope you do :)
[10:15] mato pieterh: if you want, the man pages won't conflict with anything
[10:15] mato pieterh: what change, by the way?
[10:16] pieterh promote use of "*" in binding, reduce emphasis on non-portable interface names
[10:16] pieterh it's a small change to zmq_tcp(7)
[10:16] mato pieterh: can i review that please? thanks?
[10:16] pieterh sure
[10:18] pieterh sent to list
[10:18] pieterh in the user guide i'm treating non-portable interfaces as 'advanced'
[10:18] mato that's fine
[10:18] mato now that there is actually a user guide
[10:19] pieterh further, it's more useful IMO to describe "*" in terms of positive outcome ("binds to all available interfaces") rather than negative semantics ("does not specify interface")
[10:19] pieterh "does not make coffee"... the list of negatives is long
[10:19] mato hmm, i will complain first about your mail client, as usual :-)
[10:20] pieterh after you convince github to fix the issue tracker, i've got Google's address for you too...
[10:20] pieterh i'm sure they would like the feedback on gmale
[10:20] pieterh freudian slip?
[10:20] pieterh gmail
[10:20] pieterh :-)
[10:20] mato pieterh: ah, gmail
[10:21] mato pieterh: suggestion...
[10:21] pieterh ja?
[10:22] mato pieterh: either include the patch inline in email, or call it something.diff.txt (gmail is stupid)
[10:22] mato all i see is:
[10:22] mato [-- Attachment #2: zmq_tcp.txt.diff --]
[10:22] mato [-- Type: application/octet-stream, Encoding: base64, Size: 3.0K --]
[10:22] mato [-- application/octet-stream is unsupported (use 'v' to view this part) --]
[10:23] pieterh ah, it takes two (email clients) to get confusion over attachments
[10:23] pieterh why don't you just upgrade to gmail like everyone else? :-)
[10:23] pieterh i'll resend it
[10:23] mato no need
[10:23] mato pieterh: i can view the attachment, that's not a problem
[10:23] mato just a suggestion
[10:23] mato that's all
[10:23] pieterh usually i just commit directly to master
[10:24] pieterh and allow CIA19 to announce the good news
[10:32] sjampoo No! The zeromq is my primary source for my fix of L0lcatZ
[10:33] sjampoo s/zeromq/0mq list
[10:33] mato :-)
[10:34] mato ok, attachments with application/octet-stream which is useless anyway
[10:34] mato or we can make an exception for image/x-lolcat :-)
[10:34] sjampoo heh :)
[10:40] CIA-20 zeromq2: 03Pieter Hintjens 07master * rb6cdd36 10/ (12 files in 2 dirs):
[10:40] CIA-20 zeromq2: Added error checking (EFAULT) for null arguments
[10:40] CIA-20 zeromq2: * Fixed zmq_term, zmq_socket, zmq_close, zmq_setsockopt,
[10:40] CIA-20 zeromq2: * zmq_getsockopt, zmq_bind, zmq_connect, zmq_send,
[10:40] CIA-20 zeromq2: * zmq_recv, zmq_poll, zmq_device, zmq_stopwatch_stop
[10:40] CIA-20 zeromq2: * Updated Reference Manual for these methods -
[10:40] CIA-20 zeromq2: 03Pieter Hintjens 07master * ra12f446 10/ doc/zmq_tcp.txt : Modified zmq_tcp(7) to emphasize wildcard interfaces -
[10:45] mato pieterh: not meaning to argue, but you trolled there a bit
[10:45] mato pieterh: there really is nothing stopping you from organising your repo branches any way you want...
[10:45] mato pieterh: that's part of the power of Git AFAICS
[10:46] mato pieterh: e.g. you can have an 'unstable' branch that pushes into zeromq2 'master', you can have local topic branches, and so on.
[10:49] pieterh mato: its only a part troll
[10:49] mato :)
[10:49] pieterh you said it yourself, we follow documented procedures
[10:49] pieterh few of us are git gurus
[10:49] mato yes, but those are being hashed out right now, and that will take some time
[10:50] pieterh i don't want to have to organize my branches any way I want
[10:50] pieterh i want to follow some clear procedure that stops me making mistakes
[10:50] pieterh if i have to invent it, i'll get it wrong
[10:50] pieterh thus, and i assume you work like that too, you avoid using stuff you are likely to get wrong
[10:51] mato yes, of course
[10:51] pieterh which was the point of my email
[10:51] mato but there's nothing wrong with experimenting with a procedure in your local git
[10:51] pieterh i do not want to experiment
[10:51] pieterh i have "real work" to do
[10:51] mato and then saying, hey guys, i have this all working nicely in my git, here's how it works
[10:52] pieterh no, because someone (you) will come along with a huge argument about why it's NFG
[10:52] pieterh i did in fact do that, and wrote it down, and that happened
[10:52] mato sigh
[10:53] pieterh i just point out that discarding rather than improving / fixing is kind of wasteful
[10:54] pieterh and certainly discouraging
[10:54] mato i saw several things with your proposed procedure that did things "differently" from what i percieve as the best practice being used by people out there
[10:54] pieterh it is a wiki
[10:54] pieterh edit / change / save
[10:54] mato if i edit/change/save without explaning why, that doesn't help, does it?
[10:54] pieterh simply pointing to another document and saying "i like that better" is not helpful
[10:54] pieterh yes, it helps
[10:55] pieterh you make the changes you feel necessary, then you explain why
[10:55] pieterh if someone else feels strongly, they take the effort to make more changes
[10:55] pieterh etc.
[10:55] pieterh it's a good process
[10:56] mato pieterh: note that i was not the only person who immediately pointed out that "unstable" was "different from everyone else" in your proposal
[10:56] mato anyhow
[10:56] pieterh it seems incredibly minor: you edit one sentence and that's it
[10:57] pieterh what i really needed was input on the git aspects themselves
[10:57] mato i can't think about this right now, and no, it is not just a question of naming, it's also a question of how commits flow between branches which you are completely ignoring out of i presume ignorance
[10:57] pieterh indeed
[10:58] pieterh that is the part which matters
[10:58] mato and i would like to write that down, but that needs time
[10:58] pieterh actually i did describe how commits flow between branches
[10:58] pieterh at least one way of doing it
[10:59] pieterh but that's the part which needs expert input
[10:59] pieterh not... sigh... whether work branches are called "issue" or "topic" fgs
[10:59] pieterh well
[10:59] mato pieterh: would you like me to drop everything and spend today working out the Git process?
[10:59] mato pieterh: yes?
[10:59] pieterh mato: no
[11:00] mato then let's just leave the damn thing alone for a while
[11:00] pieterh i'd like to thrash it out high bandwidth next week
[11:00] mato ok?
[11:00] pieterh ano ano
[11:00] pieterh in any case we need sustrik to mindmeld on the process
[11:00] mato so, topic "Git" is closed for now
[11:05] sustrik i would say let's adopt the linux kernel process
[13:56] pieterh sustrik: random question... PUB drops messages if it hits HWM. Does it drop oldest or newest messages?
[13:56] sustrik newest
[13:57] pieterh is that deliberate policy? (usually for pubsub you'd drop oldest messages)
[13:58] sustrik the reason is that what you are working with are basically pipes
[13:58] sustrik pipe can be a network connection, inter-thread connection
[13:58] sustrik a HW device
[13:58] sustrik whatever
[13:58] sustrik if you want to drop oldest message
[13:59] sustrik the publisher end of the pipe has to notify the receiver end of the pipe to drop a message
[13:59] sustrik the question is how to pass the notification
[13:59] sustrik so you have to have another pipe
[13:59] sustrik what if that one gets full
[13:59] sustrik etc.
[13:59] pieterh the HWM is not known to the receiver end
[13:59] pieterh right