Sunday August 8, 2010

[Time] NameMessage
[11:17] pieter_hintjens Have written proposal for git workflow:
[11:17] pieter_hintjens what we're doing now is not really scalable
[11:21] sustrik pieterh: ok, i'll give it a read in ~1hour
[11:21] pieterh no hurry
[11:22] pieterh it really needs a git guru to give it a quick review
[11:22] pieterh but most of it is based on experience from
[11:22] sustrik martin hurton seems to know git quite well
[11:23] pieterh ah, great
[11:54] mato pieterh: i'll take a look at it also in some time
[11:55] mato pieterh: please note that could you maybe prominently mark your various strawmen with "*** PROPOSAL ***" or something?
[11:56] mato pieterh: so that it's obvious that is not a final decision and is still being discussed, and may not reflect reality
[11:56] mato pieterh: otherwise it's a bit confusing for people (think also of random people browsing the site)
[11:56] mato pieterh: oh, and the page title is "Source" for some reason...
[11:59] mato pieterh: I see you changed the C binding to push the age-old iMatix 'c' script, this is also somewhat counter-productive... the documentation should be environment-agnostic
[12:00] pieterh mato: proposal, indeed.
[12:00] pieterh i wrote that but not in huge letters.
[12:00] mato pieterh: i'm almost thinking of something damn-bleedin-obvious for proposals
[12:00] pieterh i find the 'c' script useful and felt it was worth sharing but feel free to remove it
[12:00] pieterh mato: ?
[12:00] mato pieterh: big header in colorful background at the start of the page
[12:01] mato pieterh: i'm not against the 'c' script, i'm against the C binding docs having a single example about compiling which only mentions the 'c' script
[12:02] pieterh i'll put something in large bold for the proposal, you can adjust to taste
[12:02] mato pieterh: one proposal...
[12:02] pieterh shoot
[12:02] mato pieterh: I find it hard to see what's changed on the Wiki (notification emails are useless)... can we do one or both of the following?
[12:02] mato pieterh: 1. Put a Recent Wiki Changes link on the sidebar or at least the home page
[12:03] pieterh the emails are useless, they don't show diffs, i know
[12:03] mato pieterh: 2. Maybe better, make that part of a box on the home page with a feed of the last X changes
[12:03] pieterh recent changes would be good, yes
[12:03] mato pieterh: ah, i see you took the ML summary out of the home page, why?
[12:03] mato pieterh: Having the home page at least partly as a dashboard for all recent activity seemed useful
[12:04] mato pieterh: also reassures newcomers that lots of stuff is going on
[12:04] pieterh i was experimenting with the twitter box
[12:04] mato forget twitter, it's just pass-along value
[12:04] pieterh maybe but i get a lot of stuff off that
[12:04] mato having a ML and recent changes feeds on the home page seems much more useful to me
[12:04] pieterh sure
[12:04] pieterh i'm pretty agnostic about this, it's good to try stuff and see what sticks
[12:05] pieterh the ML stuff is there, commented
[12:05] mato besides, i don't see the twitter box anyway :)
[12:05] pieterh yeah, that explains something
[12:05] mato what's that? :-)
[12:06] pieterh that you ask where the ML history went to
[12:06] pieterh otherwise it'd be obvious
[12:06] mato yeah :)
[12:06] pieterh i'll kill it, it's kind of annoying anyhow but it'd be nice to get one 'hate/like' response from a newcomer
[12:06] pieterh we're both way too familiar with the project to know what visitors like
[12:07] mato yeah, but the home page is also about being useful to all members of the community, not just impressing naive noobs
[12:07] mato so there's a compromise there somewhere :-)
[12:07] mato oh and you still have "Thanks to Matt Weinstein for the picture." which appears to have also gone
[12:07] pieterh ah
[12:08] pieterh btw, it's a wiki :-)
[12:08] mato true
[12:08] pieterh i mean, feel free to edit when you feel it's necessary
[12:08] pieterh shoot first, argue later
[12:08] pieterh that's the whole fun :-)
[12:09] pieterh shall i restore the lolcat?
[12:09] pieterh i figured that or twitter but not both
[12:11] mato well, it's kind of cute...
[12:13] pieterh futurama did it
[12:14] mato makes me want to add in the ZMQ_IWANTBITES and ZMQ_ICANHAZBITES
[12:14] mato software should not be too serious :-)
[12:17] pieterh well, it got my vote
[12:17] pieterh i think those would make
[12:17] pieterh *would make good aliases for PULL/PUSH
[12:31] pieterh mato: recent site changes added to front page
[12:33] mato pieterh: nice
[12:33] pieterh changed to show page size instead of editor
[12:33] pieterh only shows pages from docs/whitepapers/areas
[12:33] mato editor is not bad actually
[12:34] mato page size is kind of useless
[12:34] mato who cares if page is XXXX chars
[12:34] pieterh well, this editor does :-)
[12:34] pieterh anyhow, reverted
[12:34] sustrik if you want all three (mailing list, wiki, blog) on the front page you should limit number of items in each
[12:35] sustrik say 5
[12:35] mato yeah. it'd also be really nice to get the date in the mailing list posts, but i remember trying to make that work and failing miserably :-(
[12:36] mato due to the rss feed being kind of broken
[12:36] pieterh yup
[12:37] mato pieterh: i see you're editing it (ha, it said locked) can you change the line-height in the recent changes to match the others?
[12:38] pieterh mato: ack
[12:40] pieterh mato: it's kind of hacky, but now the right column is a lot more compact and IMO readable
[12:40] mato sustrik: i'm trying to enumerate those cases we discussed yesterday. for (x)req/(x)rep/push it all makes sense
[12:40] mato sustrik: PUB is the only "funny" one
[12:41] mato sustrik: For PUB sockets, all connections are basically transient, right?
[12:41] pieterh sustrik: i had a thought about identities... could they not always be permanent but with a timeout at the server side?
[12:42] pieterh this would solve (a) resource death when using explicit identities and (b) weirdness due to permanent/transient differences
[12:42] mato actually, the cases I have (except PUB) are all quite systematic:
[12:42] pieterh by 'server side' i obviously mean 'node that binds endpoint'
[12:42] mato • Permanent, strong: Retry
[12:42] mato • Permanent, weak: Retry
[12:42] mato • Transient, strong: Retry
[12:42] mato • Transient, weak: Ignore
[12:43] pieterh 4 kinds of connection?
[12:43] mato yup
[12:43] pieterh times X types of socket policy?
[12:43] pieterh seriously?
[12:44] pieterh how about PAIR sockets?
[12:44] mato in my doc it's all the same actually for all sockets except PUB(to be discussed) and the sockets which don't do send() where the policy is N/A
[12:44] sustrik i thouht of it as well
[12:44] sustrik the point is that in this case we are not trying to ensure consistent semantics
[12:44] sustrik instead, we want to improve reliability a little
[12:45] sustrik in a non-systematic way
[12:45] pieterh reliability is semantics
[12:45] mato yes, but the consistent semantics actually seem to make sense when you write them down
[12:45] sustrik consider this case:
[12:45] pieterh mato: if pub sockets are always transient then it's also consistent, no?
[12:45] mato pieterh: precisely
[12:45] sustrik pub/sub used for notifying all the componetns on the network to shut down
[12:45] pieterh fair enough
[12:46] sustrik if the component is disconnected, it's ok to loose the message
[12:46] pieterh i suspect we want a new kind of socket that does reliable fanout
[12:46] pieterh eventually
[12:46] sustrik the admin should then shut it down by hand
[12:46] mato sustrik: can you answer my question please? PUB sockets have only transient connections, right?
[12:46] mato sustrik: that's the nature of a PUB socket..., right?
[12:46] sustrik kind of
[12:46] mato please explain "kind of" :-)
[12:46] sustrik currently it allows you to set an identuity
[12:47] sustrik but probably it should not
[12:47] mato sustrik: how can i propose or discuss these cases with you?
[12:47] pieterh sustrik: imo you are reinventing UDP and TCP to some extent
[12:48] mato sustrik: i think the 1st naive consistent policy will work "well enough" for a 1st release
[12:48] pieterh mato: gets my vote, yes
[12:48] mato that's the point of my policy
[12:49] mato try something "good enough" as a first cut, and then deal with the corners later
[12:49] mato unless there's something terribly wrong with my "good enough" proposal
[12:49] pieterh mato: what is 'strong' vs. 'weak'?
[12:49] pieterh connect vs. bind?
[12:49] mato pieterh: strong is where an identity has been set
[12:49] pieterh and permanent vs. transient?
[12:50] mato pieterh: weak is where the identity is auto-generated (i.e. won't persist against restarts of the context)
[12:50] pieterh right
[12:50] mato pieterh: permanent is basically outbound/connect and transient is inbound/bind
[12:50] pieterh does receiver know if identity is generated (by sender) or not?
[12:50] mato afaik yes
[12:50] pieterh it's in the WLP?
[12:50] pieterh ok. noted.
[12:50] mato s/afaik/definitely/
[12:50] mato what's the WLP?
[12:51] pieterh it's the secret wire level protocol
[12:51] mato pieterh: ah, yes
[12:51] pieterh there _is_ a single 0MQ WLP there somewhere :-)
[12:51] pieterh i'll extract it and document it
[12:51] mato sustrik: hello?
[12:51] pieterh so the one thing that's unpleasant about your model is that it's no longer orthogonal to connect/bind direction
[12:52] pieterh but i don't agree with the thesis that connect/bind direction is irrelevant, i think it matters very mucjh
[12:52] pieterh *much
[12:52] pieterh so i endorse your model 100%
[12:52] pieterh basically transient connections to dynamic nodes are not reliable
[12:52] pieterh and all other ones are kind of reliable
[12:53] mato yup
[12:53] pieterh if you want reliable pub
[12:53] pieterh then pub has to connect to all subs
[12:53] pieterh nice use case in fact
[12:53] mato well, no, actually if you want reliable pub you set identity on your subs
[12:53] mato that's the idea
[12:53] mato pub doesn't really have the concept of "connecting" anywhere
[12:53] pieterh no, because then pub will crash
[12:54] mato ?
[12:54] pieterh sustrik: currently it [PUB] allows you to set an identuity
[12:54] pieterh sustrik: but probably it should not
[12:54] pieterh accumulation of identities will kill server
[12:55] mato pieterh: i'm talking about identity being set on the SUB side, not on the PUB side
[12:55] pieterh ok, maybe i'm confused
[12:55] pieterh but still... you don't need identities for this
[12:55] mato pieterh: that way the SUB connection to PUB becomes transient, strong, which is "reliable"
[12:55] pieterh ok, fair enough
[12:56] pieterh sustrik is coding, IMO :_)
[12:56] pieterh :~)
[12:56] pieterh :=)
[12:56] pieterh damn, can't get it working
[12:56] mato yeah, but then i'm not sure what he's coding, and why he asked for opinions in the 1st place :-)
[12:57] pieterh mato: your model looks right, from my perspective
[13:01] mato let's see what sustrik comes up with, there's nothing else i can do right now
[13:23] sustrik re for a second
[13:23] sustrik mato: your proposal doesn't solve the case when connection is not yet established when zmq_close is called
[13:25] mato sustrik: hmm, i'm thinking about it
[13:26] mato sustrik: why does it not solve the case
[13:27] mato sustrik: Permanent, strong: Retry, Permanent, weak: Retry
[13:27] mato sustrik: seems to work to me...
[13:27] mato sustrik: it makes PUB behave funny, but that's because I have a problem understanding PUB semantics, which is why I asked you about it
[13:29] sustrik that's what i was trying to explain
[13:29] mato sustrik: however, it does give you a case where you can make PUB behave "reliably", namely by setting your identity on the SUB side
[13:30] sustrik as already noted, that shouldn't be allowed
[13:30] mato sustrik: if PUB/SUB should not allow identities then the point is moot./
[13:30] sustrik consider the shutdown example abover
[13:31] sustrik publication is not reliable
[13:31] sustrik message can not be delivered in certain cases
[13:31] sustrik it's ok as admin can shut down the component by hand
[13:32] sustrik however, systematic consequence of such behaviour => drop all messages-in-flight at zmq_close
[13:32] sustrik doesn't really work
[13:32] sustrik foreach (components)
[13:32] sustrik send (component, shutdown)
[13:32] sustrik close()
[13:32] sustrik would result in no message being delivered
[13:33] sustrik even to boxes that are online and connected
[13:33] sustrik what we need is kind of IP's "best effort" policy
[13:33] sustrik i.e. send if possible, drop otherwise
[13:33] sustrik that's not a theoretically coherent behaviour
[13:34] mato the problem is the "send if possible" policy collides with connect being completely asynchronous
[13:34] sustrik it's just making it work in least-surprise way
[13:34] sustrik just devise a heuristic that would handle all the cases is not surprising manner
[13:34] sustrik that's it
[13:35] sustrik no need for real coherency
[13:35] mato sustrik: i think that heuristic has something to do with "first connection showing up on socket"
[13:35] sustrik yes, somethig like that
[13:35] mato and timeouts...
[13:35] sustrik possibly
[13:36] mato but timeouts are step 2
[13:36] mato in step 1 (what we're doing now)...
[13:36] sustrik sure
[13:36] sustrik step 0, is composing i/o objects into bundles
[13:36] mato is it not sufficient to make the heuristic be that we wait forever for a connection to show up *iff* none have at the point close was called?
[13:37] sustrik so that i/o object A can postpone termination of i/o object B
[13:37] sustrik that's what i am working on now
[13:37] sustrik mato: we can do it that way for the starters
[13:37] mato sustrik: that would seem to be the right thing to do now
[13:38] sustrik on zmq_connect side of course
[13:38] mato sustrik: it seems to me the only problem is PUB, actually
[13:39] mato sustrik: can we talk about this face to face when you finish your step 0 ?
[13:39] sustrik sure
[13:39] mato sustrik: good. maybe this evening?
[13:40] sustrik step 0 is a lot of work
[13:40] sustrik will be working till night i suppose
[13:40] mato sustrik: well, whenever you want
[13:40] mato sustrik: so this step 0 is mandatory to make everything else work?
[13:41] sustrik it's mandatory to wait for first connection
[13:41] mato right, makes sense, and we need that in any case
[13:41] sustrik connecter object cannot be terminated while it's session is still alive wanting to send data
[13:41] mato right
[13:42] mato sustrik: ok well, i'll leave you to it. am happy to discuss when you have time.
[13:42] sustrik ok
[13:42] mato sustrik: i don't think there's anything else i can do on the core side right now
[13:42] sustrik hm
[13:42] mato sustrik: no point testing until things are at least half-working, no?
[13:42] sustrik right
[13:42] mato right now they're all on the floor
[13:43] sustrik you are right, it's up to me right now
[13:43] mato sustrik: what i don't understand is the current (pre-refactoring) codebase had to handle this already, no? (wait for a connection to show up)
[13:43] mato sustrik: so why is it so complex all of a sudden?
[13:43] mato (naive question maybe, but worth asking just so i understand what's going on)
[13:44] sustrik because back then when you called zmq_close it just terminated both session and connecter
[13:44] sustrik easy
[13:44] mato ah, ok
[13:44] mato i get it
[13:44] mato sustrik: ok, i'll leave you to it
[13:44] mato sustrik: do call if you want a break or something, marathon coding is not a mandatory requirement :-)
[13:45] sustrik i have to deliver the article to linux congress till 16th
[13:45] sustrik so i'm in a hurry
[13:46] mato being in a hurry while doing this sort of stuff is not a good sign
[13:47] mato anyway, if there's some way i can help you, shout
[17:55] dos000 can someone tell me if you can have zmq_poll while other sockets are happening ? is is mutually exclusive ?
[17:58] travlr you can use standard sockets in zmq_poll() along with zmq sockets. is that what you are asking? if not i don't understand the question.