ZeroMq IRC Log

Sunday August 8, 2010

[Time] Name	Message
[11:17] pieter_hintjens	Have written proposal for git workflow: http://www.zeromq.org/docs:procedures
[11:17] pieter_hintjens	what we're doing now is not really scalable
[11:21] sustrik	pieterh: ok, i'll give it a read in ~1hour
[11:21] pieterh	no hurry
[11:22] pieterh	it really needs a git guru to give it a quick review
[11:22] pieterh	but most of it is based on experience from wikidot.com
[11:22] sustrik	martin hurton seems to know git quite well
[11:23] pieterh	ah, great
[11:54] mato	pieterh: i'll take a look at it also in some time
[11:55] mato	pieterh: please note that could you maybe prominently mark your various strawmen with "* PROPOSAL *" or something?
[11:56] mato	pieterh: so that it's obvious that is not a final decision and is still being discussed, and may not reflect reality
[11:56] mato	pieterh: otherwise it's a bit confusing for people (think also of random people browsing the site)
[11:56] mato	pieterh: oh, and the page title is "Source" for some reason...
[11:59] mato	pieterh: I see you changed the C binding to push the age-old iMatix 'c' script, this is also somewhat counter-productive... the documentation should be environment-agnostic
[12:00] pieterh	mato: proposal, indeed.
[12:00] pieterh	i wrote that but not in huge letters.
[12:00] mato	pieterh: i'm almost thinking of something damn-bleedin-obvious for proposals
[12:00] pieterh	i find the 'c' script useful and felt it was worth sharing but feel free to remove it
[12:00] pieterh	mato: ?
[12:00] mato	pieterh: big header in colorful background at the start of the page
[12:01] mato	pieterh: i'm not against the 'c' script, i'm against the C binding docs having a single example about compiling which only mentions the 'c' script
[12:02] pieterh	i'll put something in large bold for the proposal, you can adjust to taste
[12:02] mato	pieterh: one proposal...
[12:02] pieterh	shoot
[12:02] mato	pieterh: I find it hard to see what's changed on the Wiki (notification emails are useless)... can we do one or both of the following?
[12:02] mato	pieterh: 1. Put a Recent Wiki Changes link on the sidebar or at least the home page
[12:03] pieterh	the emails are useless, they don't show diffs, i know
[12:03] mato	pieterh: 2. Maybe better, make that part of a box on the home page with a feed of the last X changes
[12:03] pieterh	recent changes would be good, yes
[12:03] mato	pieterh: ah, i see you took the ML summary out of the home page, why?
[12:03] mato	pieterh: Having the home page at least partly as a dashboard for all recent activity seemed useful
[12:04] mato	pieterh: also reassures newcomers that lots of stuff is going on
[12:04] pieterh	i was experimenting with the twitter box
[12:04] mato	forget twitter, it's just pass-along value
[12:04] pieterh	maybe but i get a lot of stuff off that
[12:04] mato	having a ML and recent changes feeds on the home page seems much more useful to me
[12:04] pieterh	sure
[12:04] pieterh	i'm pretty agnostic about this, it's good to try stuff and see what sticks
[12:05] pieterh	the ML stuff is there, commented
[12:05] mato	besides, i don't see the twitter box anyway :)
[12:05] pieterh	yeah, that explains something
[12:05] mato	what's that? :-)
[12:06] pieterh	that you ask where the ML history went to
[12:06] pieterh	otherwise it'd be obvious
[12:06] mato	yeah :)
[12:06] pieterh	i'll kill it, it's kind of annoying anyhow but it'd be nice to get one 'hate/like' response from a newcomer
[12:06] pieterh	we're both way too familiar with the project to know what visitors like
[12:07] mato	yeah, but the home page is also about being useful to all members of the community, not just impressing naive noobs
[12:07] mato	so there's a compromise there somewhere :-)
[12:07] mato	oh and you still have "Thanks to Matt Weinstein for the picture." which appears to have also gone
[12:07] pieterh	ah
[12:08] pieterh	btw, it's a wiki :-)
[12:08] mato	true
[12:08] pieterh	i mean, feel free to edit when you feel it's necessary
[12:08] pieterh	shoot first, argue later
[12:08] pieterh	that's the whole fun :-)
[12:09] pieterh	shall i restore the lolcat?
[12:09] pieterh	i figured that or twitter but not both
[12:11] mato	well, it's kind of cute...
[12:13] pieterh	futurama did it
[12:14] mato	makes me want to add in the ZMQ_IWANTBITES and ZMQ_ICANHAZBITES
[12:14] mato	software should not be too serious :-)
[12:17] pieterh	well, it got my vote
[12:17] pieterh	i think those would make
[12:17] pieterh	*would make good aliases for PULL/PUSH
[12:31] pieterh	mato: recent site changes added to front page
[12:33] mato	pieterh: nice
[12:33] pieterh	changed to show page size instead of editor
[12:33] pieterh	only shows pages from docs/whitepapers/areas
[12:33] mato	editor is not bad actually
[12:34] mato	page size is kind of useless
[12:34] mato	who cares if page is XXXX chars
[12:34] pieterh	well, this editor does :-)
[12:34] pieterh	anyhow, reverted
[12:34] sustrik	if you want all three (mailing list, wiki, blog) on the front page you should limit number of items in each
[12:35] sustrik	say 5
[12:35] mato	yeah. it'd also be really nice to get the date in the mailing list posts, but i remember trying to make that work and failing miserably :-(
[12:36] mato	due to the rss feed being kind of broken
[12:36] pieterh	yup
[12:37] mato	pieterh: i see you're editing it (ha, it said locked) can you change the line-height in the recent changes to match the others?
[12:38] pieterh	mato: ack
[12:40] pieterh	mato: it's kind of hacky, but now the right column is a lot more compact and IMO readable
[12:40] mato	sustrik: i'm trying to enumerate those cases we discussed yesterday. for (x)req/(x)rep/push it all makes sense
[12:40] mato	sustrik: PUB is the only "funny" one
[12:41] mato	sustrik: For PUB sockets, all connections are basically transient, right?
[12:41] pieterh	sustrik: i had a thought about identities... could they not always be permanent but with a timeout at the server side?
[12:42] pieterh	this would solve (a) resource death when using explicit identities and (b) weirdness due to permanent/transient differences
[12:42] mato	actually, the cases I have (except PUB) are all quite systematic:
[12:42] pieterh	by 'server side' i obviously mean 'node that binds endpoint'
[12:42] mato	â¢ Permanent, strong: Retry
[12:42] mato	â¢ Permanent, weak: Retry
[12:42] mato	â¢ Transient, strong: Retry
[12:42] mato	â¢ Transient, weak: Ignore
[12:43] pieterh	4 kinds of connection?
[12:43] mato	yup
[12:43] pieterh	times X types of socket policy?
[12:43] pieterh	seriously?
[12:44] pieterh	how about PAIR sockets?
[12:44] mato	in my doc it's all the same actually for all sockets except PUB(to be discussed) and the sockets which don't do send() where the policy is N/A
[12:44] sustrik	i thouht of it as well
[12:44] sustrik	the point is that in this case we are not trying to ensure consistent semantics
[12:44] sustrik	instead, we want to improve reliability a little
[12:45] sustrik	in a non-systematic way
[12:45] pieterh	reliability is semantics
[12:45] mato	yes, but the consistent semantics actually seem to make sense when you write them down
[12:45] sustrik	consider this case:
[12:45] pieterh	mato: if pub sockets are always transient then it's also consistent, no?
[12:45] mato	pieterh: precisely
[12:45] sustrik	pub/sub used for notifying all the componetns on the network to shut down
[12:45] pieterh	fair enough
[12:46] sustrik	if the component is disconnected, it's ok to loose the message
[12:46] pieterh	i suspect we want a new kind of socket that does reliable fanout
[12:46] pieterh	eventually
[12:46] sustrik	the admin should then shut it down by hand
[12:46] mato	sustrik: can you answer my question please? PUB sockets have only transient connections, right?
[12:46] mato	sustrik: that's the nature of a PUB socket..., right?
[12:46] sustrik	kind of
[12:46] mato	please explain "kind of" :-)
[12:46] sustrik	currently it allows you to set an identuity
[12:47] sustrik	but probably it should not
[12:47] mato	sustrik: how can i propose or discuss these cases with you?
[12:47] pieterh	sustrik: imo you are reinventing UDP and TCP to some extent
[12:48] mato	sustrik: i think the 1st naive consistent policy will work "well enough" for a 1st release
[12:48] pieterh	mato: gets my vote, yes
[12:48] mato	that's the point of my policy
[12:49] mato	try something "good enough" as a first cut, and then deal with the corners later
[12:49] mato	unless there's something terribly wrong with my "good enough" proposal
[12:49] pieterh	mato: what is 'strong' vs. 'weak'?
[12:49] pieterh	connect vs. bind?
[12:49] mato	pieterh: strong is where an identity has been set
[12:49] pieterh	and permanent vs. transient?
[12:50] mato	pieterh: weak is where the identity is auto-generated (i.e. won't persist against restarts of the context)
[12:50] pieterh	right
[12:50] mato	pieterh: permanent is basically outbound/connect and transient is inbound/bind
[12:50] pieterh	does receiver know if identity is generated (by sender) or not?
[12:50] mato	afaik yes
[12:50] pieterh	it's in the WLP?
[12:50] pieterh	ok. noted.
[12:50] mato	s/afaik/definitely/
[12:50] mato	what's the WLP?
[12:51] pieterh	it's the secret wire level protocol
[12:51] mato	pieterh: ah, yes
[12:51] pieterh	there _is_ a single 0MQ WLP there somewhere :-)
[12:51] pieterh	i'll extract it and document it
[12:51] mato	sustrik: hello?
[12:51] pieterh	so the one thing that's unpleasant about your model is that it's no longer orthogonal to connect/bind direction
[12:52] pieterh	but i don't agree with the thesis that connect/bind direction is irrelevant, i think it matters very mucjh
[12:52] pieterh	*much
[12:52] pieterh	so i endorse your model 100%
[12:52] pieterh	basically transient connections to dynamic nodes are not reliable
[12:52] pieterh	and all other ones are kind of reliable
[12:53] mato	yup
[12:53] pieterh	if you want reliable pub
[12:53] pieterh	then pub has to connect to all subs
[12:53] pieterh	nice use case in fact
[12:53] mato	well, no, actually if you want reliable pub you set identity on your subs
[12:53] mato	that's the idea
[12:53] mato	pub doesn't really have the concept of "connecting" anywhere
[12:53] pieterh	no, because then pub will crash
[12:54] mato	?
[12:54] pieterh	sustrik: currently it [PUB] allows you to set an identuity
[12:54] pieterh	sustrik: but probably it should not
[12:54] pieterh	accumulation of identities will kill server
[12:55] mato	pieterh: i'm talking about identity being set on the SUB side, not on the PUB side
[12:55] pieterh	ok, maybe i'm confused
[12:55] pieterh	but still... you don't need identities for this
[12:55] mato	pieterh: that way the SUB connection to PUB becomes transient, strong, which is "reliable"
[12:55] pieterh	ok, fair enough
[12:56] pieterh	sustrik is coding, IMO :_)
[12:56] pieterh	:~)
[12:56] pieterh	:=)
[12:56] pieterh	damn, can't get it working
[12:56] mato	yeah, but then i'm not sure what he's coding, and why he asked for opinions in the 1st place :-)
[12:57] pieterh	mato: your model looks right, from my perspective
[13:01] mato	let's see what sustrik comes up with, there's nothing else i can do right now
[13:23] sustrik	re for a second
[13:23] sustrik	mato: your proposal doesn't solve the case when connection is not yet established when zmq_close is called
[13:25] mato	sustrik: hmm, i'm thinking about it
[13:26] mato	sustrik: why does it not solve the case
[13:27] mato	sustrik: Permanent, strong: Retry, Permanent, weak: Retry
[13:27] mato	sustrik: seems to work to me...
[13:27] mato	sustrik: it makes PUB behave funny, but that's because I have a problem understanding PUB semantics, which is why I asked you about it
[13:29] sustrik	that's what i was trying to explain
[13:29] mato	sustrik: however, it does give you a case where you can make PUB behave "reliably", namely by setting your identity on the SUB side
[13:30] sustrik	as already noted, that shouldn't be allowed
[13:30] mato	sustrik: if PUB/SUB should not allow identities then the point is moot./
[13:30] sustrik	consider the shutdown example abover
[13:31] sustrik	publication is not reliable
[13:31] sustrik	message can not be delivered in certain cases
[13:31] sustrik	it's ok as admin can shut down the component by hand
[13:32] sustrik	however, systematic consequence of such behaviour => drop all messages-in-flight at zmq_close
[13:32] sustrik	doesn't really work
[13:32] sustrik	foreach (components)
[13:32] sustrik	send (component, shutdown)
[13:32] sustrik	close()
[13:32] sustrik	would result in no message being delivered
[13:33] sustrik	even to boxes that are online and connected
[13:33] sustrik	what we need is kind of IP's "best effort" policy
[13:33] sustrik	i.e. send if possible, drop otherwise
[13:33] sustrik	that's not a theoretically coherent behaviour
[13:34] mato	the problem is the "send if possible" policy collides with connect being completely asynchronous
[13:34] sustrik	it's just making it work in least-surprise way
[13:34] sustrik	just devise a heuristic that would handle all the cases is not surprising manner
[13:34] sustrik	that's it
[13:35] sustrik	no need for real coherency
[13:35] mato	sustrik: i think that heuristic has something to do with "first connection showing up on socket"
[13:35] sustrik	yes, somethig like that
[13:35] mato	and timeouts...
[13:35] sustrik	possibly
[13:36] mato	but timeouts are step 2
[13:36] mato	in step 1 (what we're doing now)...
[13:36] sustrik	sure
[13:36] sustrik	step 0, is composing i/o objects into bundles
[13:36] mato	is it not sufficient to make the heuristic be that we wait forever for a connection to show up iff none have at the point close was called?
[13:37] sustrik	so that i/o object A can postpone termination of i/o object B
[13:37] sustrik	that's what i am working on now
[13:37] sustrik	mato: we can do it that way for the starters
[13:37] mato	sustrik: that would seem to be the right thing to do now
[13:38] sustrik	on zmq_connect side of course
[13:38] mato	sustrik: it seems to me the only problem is PUB, actually
[13:39] mato	sustrik: can we talk about this face to face when you finish your step 0 ?
[13:39] sustrik	sure
[13:39] mato	sustrik: good. maybe this evening?
[13:40] sustrik	step 0 is a lot of work
[13:40] sustrik	will be working till night i suppose
[13:40] mato	sustrik: well, whenever you want
[13:40] mato	sustrik: so this step 0 is mandatory to make everything else work?
[13:41] sustrik	it's mandatory to wait for first connection
[13:41] mato	right, makes sense, and we need that in any case
[13:41] sustrik	connecter object cannot be terminated while it's session is still alive wanting to send data
[13:41] mato	right
[13:42] mato	sustrik: ok well, i'll leave you to it. am happy to discuss when you have time.
[13:42] sustrik	ok
[13:42] mato	sustrik: i don't think there's anything else i can do on the core side right now
[13:42] sustrik	hm
[13:42] mato	sustrik: no point testing until things are at least half-working, no?
[13:42] sustrik	right
[13:42] mato	right now they're all on the floor
[13:43] sustrik	you are right, it's up to me right now
[13:43] mato	sustrik: what i don't understand is the current (pre-refactoring) codebase had to handle this already, no? (wait for a connection to show up)
[13:43] mato	sustrik: so why is it so complex all of a sudden?
[13:43] mato	(naive question maybe, but worth asking just so i understand what's going on)
[13:44] sustrik	because back then when you called zmq_close it just terminated both session and connecter
[13:44] sustrik	easy
[13:44] mato	ah, ok
[13:44] mato	i get it
[13:44] mato	sustrik: ok, i'll leave you to it
[13:44] mato	sustrik: do call if you want a break or something, marathon coding is not a mandatory requirement :-)
[13:45] sustrik	i have to deliver the article to linux congress till 16th
[13:45] sustrik	so i'm in a hurry
[13:46] mato	being in a hurry while doing this sort of stuff is not a good sign
[13:47] mato	anyway, if there's some way i can help you, shout
[17:55] dos000	can someone tell me if you can have zmq_poll while other sockets are happening ? is is mutually exclusive ?
[17:58] travlr	you can use standard sockets in zmq_poll() along with zmq sockets. is that what you are asking? if not i don't understand the question.