ZeroMq IRC Log

Tuesday August 10, 2010

[Time] Name	Message
[07:00] mikko	sustrik: there?
[07:01] sustrik	mikko: hi
[07:01] mikko	i am having a presentation at company evening today
[07:01] mikko	suprisingly
[07:02] mikko	how would you put "Internet Scale Messaging" in one short sentence?
[07:02] sustrik	i am having a problem with that one myself
[07:02] sustrik	maybe "decentralised" rather than Internet-scale
[07:03] sustrik	no central authority
[07:03] sustrik	same way as IP or Web
[07:03] mikko	no guarantees on link speeds
[07:03] mikko	i guess
[07:03] mikko	unreliable networks
[07:03] sustrik	no trust
[07:03] sustrik	yes, that kind of things
[07:04] sustrik	actually, comparison with ip/web can be an eye-opener imo
[07:04] sustrik	people know how those work
[07:04] sustrik	so "the same thing for distributed services"
[07:07] mikko	I think I can explain it to the audience
[07:07] mikko	luckily just a small audience
[07:11] sustrik	well, good luck
[07:12] sustrik	if there's any interesting feedback let us know
[08:51] pieter_hintjens	mikko: perhaps something like "connecting nodes over an infinitely scalable, totally untrusted, and highly redundant network"
[08:57] aleator	I need to make parallel requests for my worker nodes. Should I use XReq or device or something else entirely?
[08:58] pieterh	parallel requests meaning the same requests sent to all workers, or requests distributed among workers?
[08:59] aleator	different request for each worker, all at the same time
[09:00] guido_g	you might want to use DOWNSTREAM/PUSH vs UPSTREAM/PULL sockets
[09:00] pieterh	you could use the pipeline pattern, so ZMQ_DOWNSTREAM on client side, ZMQ_UPSTREAM on worker side, and then ZMQ_DOWNSTREAM from worker, back to ZMQ_UPSTREAM back on client
[09:00] pieterh	DOWNSTREAM is now called PUSH and UPSTREAM is PULL btw
[09:01] pieterh	you could also use a queue device, see figure 11 here: http://www.zeromq.org/docs:user-guide#toc14
[09:01] aleator	Ok. So in essence I need to supply my own machinery for matching requests and replies with push/pull?
[09:02] pieterh	aleator: yes, if you use push/pull you need to do the matching yourself
[09:02] pieterh	if you use req-xrep-xreq-rep then it's done for you
[09:03] aleator	I had originally just xreq/xrep and it seems to randomly utilise only some workers.
[09:03] aleator	(I'm rather new at this)
[09:03] pieterh	but the request-reply chain is hard to use if you try to start with xreq
[09:04] pieterh	how many clients do you have?
[09:04] aleator	In range 10-160, depending.
[09:04] aleator	Tasks take few minutes each
[09:05] pieterh	and each client sends one request at a time?
[09:06] aleator	Ah. sorry, my bad. I meant workers. For clients there is at this point just one.
[09:06] pieterh	so the client sends out jobs in parallel?
[09:06] aleator	Thats my goal.
[09:06] pieterh	then i'd suggest push/pull, and match the results up
[09:07] pieterh	you can connect a pipeline in loops
[09:07] aleator	Thank you.
[09:07] pieterh	i don't have diagrams for that in the user guide yet, sorry
[09:07] aleator	I just thought to ask since this seems rather basic operation that might have had support already.
[09:08] aleator	But it is not difficult to do by hand either.
[09:08] pieterh	you can do it by using the request-reply pattern as drawn in the diagram BUT then
[09:08] pieterh	you need to use an XREQ socket at the client side and manually construct the frames
[09:09] pieterh	since that is not documented, it's kind of hard to do
[09:10] aleator	I did something like that with the result that just few workers got anything to do. Are there known pitfalls with this that I've might have stumbled upon?
[09:11] guido_g	pieterh: is there a piece of code or a document that shows how req -> xrep is done?
[09:11] aleator	Anyways, thank you for your help!
[09:12] pieterh	guido_g: not in detail yet. The way I'm explaining XREP and XREQ in the user guide (basic section) is that they "extend" the request-reply pattern over intermediaries
[09:12] pieterh	so you connect req-rep, or req-xrep-(code)-xreq-rep
[09:13] pieterh	that is sufficient to use them in devices and stuff like multithreaded servers
[09:13] pieterh	it's not sufficiently documented to do other stuff like aleator wants to do, e.g. client sending requests in parallel
[09:14] pieterh	you can read the 0MQ code, people do, and it is helpful
[09:14] guido_g	what is see is the (common?) pattern of simple clients -- using req -- connecting to a dispatcher (using x*)
[09:14] guido_g	ok
[09:15] pieterh	yes, that's probably the most common use case for X*
[09:15] pieterh	mato: can i commit a man page change to git or will that provoke panic again?
[09:15] guido_g	built something like that saturday morning while re-learning c++
[09:16] guido_g	ÃMQ is great for this kind of stuff
[09:22] aleator	pieterh: Anyhow, I think my problem is so common that it might make good addition to zmg or tutorial of.
[09:23] pieterh	aleator: i think so, yes. it's just a matter of documenting the request-reply chain fully
[09:23] pieterh	i'll get to that in the user guide
[09:23] pieterh	it's going to be huge...
[09:26] aleator	Hope you do :)
[10:15] mato	pieterh: if you want, the man pages won't conflict with anything
[10:15] mato	pieterh: what change, by the way?
[10:16] pieterh	promote use of "*" in binding, reduce emphasis on non-portable interface names
[10:16] pieterh	it's a small change to zmq_tcp(7)
[10:16] mato	pieterh: can i review that please? thanks?
[10:16] pieterh	sure
[10:18] pieterh	sent to list
[10:18] pieterh	in the user guide i'm treating non-portable interfaces as 'advanced'
[10:18] mato	that's fine
[10:18] mato	now that there is actually a user guide
[10:19] pieterh	further, it's more useful IMO to describe "*" in terms of positive outcome ("binds to all available interfaces") rather than negative semantics ("does not specify interface")
[10:19] pieterh	"does not make coffee"... the list of negatives is long
[10:19] mato	hmm, i will complain first about your mail client, as usual :-)
[10:20] pieterh	after you convince github to fix the issue tracker, i've got Google's address for you too...
[10:20] pieterh	i'm sure they would like the feedback on gmale
[10:20] pieterh	freudian slip?
[10:20] pieterh	gmail
[10:20] pieterh	:-)
[10:20] mato	pieterh: ah, gmail
[10:21] mato	pieterh: suggestion...
[10:21] pieterh	ja?
[10:22] mato	pieterh: either include the patch inline in email, or call it something.diff.txt (gmail is stupid)
[10:22] mato	all i see is:
[10:22] mato	[-- Attachment #2: zmq_tcp.txt.diff --]
[10:22] mato	[-- Type: application/octet-stream, Encoding: base64, Size: 3.0K --]
[10:22] mato	[-- application/octet-stream is unsupported (use 'v' to view this part) --]
[10:23] pieterh	ah, it takes two (email clients) to get confusion over attachments
[10:23] pieterh	why don't you just upgrade to gmail like everyone else? :-)
[10:23] pieterh	i'll resend it
[10:23] mato	no need
[10:23] mato	pieterh: i can view the attachment, that's not a problem
[10:23] mato	just a suggestion
[10:23] mato	that's all
[10:23] pieterh	usually i just commit directly to master
[10:24] pieterh	and allow CIA19 to announce the good news
[10:32] sjampoo	No! The zeromq is my primary source for my fix of L0lcatZ
[10:33] sjampoo	s/zeromq/0mq list
[10:33] mato	:-)
[10:34] mato	ok, attachments with application/octet-stream which is useless anyway
[10:34] mato	or we can make an exception for image/x-lolcat :-)
[10:34] sjampoo	heh :)
[10:40] CIA-20	zeromq2: 03Pieter Hintjens 07master * rb6cdd36 10/ (12 files in 2 dirs):
[10:40] CIA-20	zeromq2: Added error checking (EFAULT) for null arguments
[10:40] CIA-20	zeromq2: * Fixed zmq_term, zmq_socket, zmq_close, zmq_setsockopt,
[10:40] CIA-20	zeromq2: * zmq_getsockopt, zmq_bind, zmq_connect, zmq_send,
[10:40] CIA-20	zeromq2: * zmq_recv, zmq_poll, zmq_device, zmq_stopwatch_stop
[10:40] CIA-20	zeromq2: * Updated Reference Manual for these methods - http://bit.ly/a3QF1N
[10:40] CIA-20	zeromq2: 03Pieter Hintjens 07master * ra12f446 10/ doc/zmq_tcp.txt : Modified zmq_tcp(7) to emphasize wildcard interfaces - http://bit.ly/cWYMOK
[10:45] mato	pieterh: not meaning to argue, but you trolled there a bit
[10:45] mato	pieterh: there really is nothing stopping you from organising your repo branches any way you want...
[10:45] mato	pieterh: that's part of the power of Git AFAICS
[10:46] mato	pieterh: e.g. you can have an 'unstable' branch that pushes into zeromq2 'master', you can have local topic branches, and so on.
[10:49] pieterh	mato: its only a part troll
[10:49] mato	:)
[10:49] pieterh	you said it yourself, we follow documented procedures
[10:49] pieterh	few of us are git gurus
[10:49] mato	yes, but those are being hashed out right now, and that will take some time
[10:50] pieterh	i don't want to have to organize my branches any way I want
[10:50] pieterh	i want to follow some clear procedure that stops me making mistakes
[10:50] pieterh	if i have to invent it, i'll get it wrong
[10:50] pieterh	thus, and i assume you work like that too, you avoid using stuff you are likely to get wrong
[10:51] mato	yes, of course
[10:51] pieterh	which was the point of my email
[10:51] mato	but there's nothing wrong with experimenting with a procedure in your local git
[10:51] pieterh	i do not want to experiment
[10:51] pieterh	i have "real work" to do
[10:51] mato	and then saying, hey guys, i have this all working nicely in my git, here's how it works
[10:52] pieterh	no, because someone (you) will come along with a huge argument about why it's NFG
[10:52] pieterh	i did in fact do that, and wrote it down, and that happened
[10:52] mato	sigh
[10:53] pieterh	i just point out that discarding rather than improving / fixing is kind of wasteful
[10:54] pieterh	and certainly discouraging
[10:54] mato	i saw several things with your proposed procedure that did things "differently" from what i percieve as the best practice being used by people out there
[10:54] pieterh	it is a wiki
[10:54] pieterh	edit / change / save
[10:54] mato	if i edit/change/save without explaning why, that doesn't help, does it?
[10:54] pieterh	simply pointing to another document and saying "i like that better" is not helpful
[10:54] pieterh	yes, it helps
[10:55] pieterh	you make the changes you feel necessary, then you explain why
[10:55] pieterh	if someone else feels strongly, they take the effort to make more changes
[10:55] pieterh	etc.
[10:55] pieterh	it's a good process
[10:56] mato	pieterh: note that i was not the only person who immediately pointed out that "unstable" was "different from everyone else" in your proposal
[10:56] mato	anyhow
[10:56] pieterh	it seems incredibly minor: you edit one sentence and that's it
[10:57] pieterh	what i really needed was input on the git aspects themselves
[10:57] mato	i can't think about this right now, and no, it is not just a question of naming, it's also a question of how commits flow between branches which you are completely ignoring out of i presume ignorance
[10:57] pieterh	indeed
[10:58] pieterh	that is the part which matters
[10:58] mato	and i would like to write that down, but that needs time
[10:58] pieterh	actually i did describe how commits flow between branches
[10:58] pieterh	at least one way of doing it
[10:59] pieterh	but that's the part which needs expert input
[10:59] pieterh	not... sigh... whether work branches are called "issue" or "topic" fgs
[10:59] pieterh	well
[10:59] mato	pieterh: would you like me to drop everything and spend today working out the Git process?
[10:59] mato	pieterh: yes?
[10:59] pieterh	mato: no
[11:00] mato	then let's just leave the damn thing alone for a while
[11:00] pieterh	i'd like to thrash it out high bandwidth next week
[11:00] mato	ok?
[11:00] pieterh	ano ano
[11:00] pieterh	in any case we need sustrik to mindmeld on the process
[11:00] mato	so, topic "Git" is closed for now
[11:05] sustrik	i would say let's adopt the linux kernel process
[13:56] pieterh	sustrik: random question... PUB drops messages if it hits HWM. Does it drop oldest or newest messages?
[13:56] sustrik	newest
[13:57] pieterh	is that deliberate policy? (usually for pubsub you'd drop oldest messages)
[13:58] sustrik	the reason is that what you are working with are basically pipes
[13:58] sustrik	pipe can be a network connection, inter-thread connection
[13:58] sustrik	a HW device
[13:58] sustrik	whatever
[13:58] sustrik	if you want to drop oldest message
[13:59] sustrik	the publisher end of the pipe has to notify the receiver end of the pipe to drop a message
[13:59] sustrik	the question is how to pass the notification
[13:59] sustrik	so you have to have another pipe
[13:59] sustrik	what if that one gets full
[13:59] sustrik	etc.
[13:59] pieterh	the HWM is not known to the receiver end
[13:59] pieterh	right