Sunday March 27, 2011

[Time] NameMessage
[04:17] EduardoFonseca Hi Guys. I'm trying to figure out how to implement connection dettection on top of zeroMQ. Any tips?
[04:17] EduardoFonseca *detection, sorry.
[04:37] EduardoFonseca great
[04:38] EduardoFonseca but what about detecting when this connection has gone away?
[04:38] EduardoFonseca I tried implementing heartbeats, but the code started to get really complex.
[04:40] EduardoFonseca I'm using XREQ and XREP... but the loadbalancing part got complex.
[04:40] EduardoFonseca I was planning to reimplement the Majordomo pattern... but, why use zeroMQ if you need a broker? :)
[17:09] mikko howdy guys
[17:57] Guthur hey mikko
[17:57] Guthur do you know if Ian Barbers PHP presentation slides are available anywhere for download, besides slideshare
[17:58] mikko Guthur: let me check
[17:59] mikko his blog seems to be down
[18:06] Guthur ok, I'll check it later
[18:06] Guthur cheers mikko
[18:14] mikko i can ask him for a pdf
[18:25] Guthur mikko, that would great, cheers
[18:36] pieter_hintjens Guthur: slideshare should offer downloads, doesn't it?
[18:36] Guthur pieter_hintjens, It does. i just didn't want to sign up
[18:36] pieter_hintjens yeah
[18:41] mikko finally got some more build capacity online
[18:41] mikko x86_64 centos
[18:41] mikko rpms are now built every night
[18:43] mikko pieter_hintjens: i'll try to add libzapi today
[18:43] pieter_hintjens mikko: nice!
[18:43] pieter_hintjens I also renamed ZFL to libzfl, you saw that
[18:43] mikko i did
[18:43] Guthur I was actually going to draw some inspiration from Ian's presentation for a presentation at work
[18:43] mikko i was away yesterday and today
[18:43] mikko spa weekend
[18:43] pieter_hintjens :-) it was excellent weather here in Belgium
[18:43] mikko been doing nothing but soaking in warm water
[18:44] Guthur I've been wanting to ask him if it would be ok to borrow some stuff, just haven't been able to catch him in here
[18:49] pieterh all code blocks highlighted, all collapsible (except a few)
[18:49] pieterh
[19:45] Guthur it loads for me
[19:45] Guthur but there is issues with the language translation link
[19:45] Guthur links*
[19:45] pieterh yes, reload in a few heartbeats...
[19:45] pieterh the problem is the marked up code blocks are huge
[19:46] pieterh including 320 of them in the page kind of does not work
[19:46] Guthur hehe, I can imagine
[19:47] pieterh it should work better now, code examples appear in a new tab
[19:47] pieterh shame, it was kind of nice to have them inline...
[19:50] Guthur oh new tabs, umm, I tend to curse pages that do that
[19:51] Guthur is the C version still inline?
[19:53] pieterh well, nope, but I'm going to make a further round of improvements
[19:53] pieterh will inline the C version
[19:53] pieterh and ditto for the PHP version
[19:56] mikko pieterh:
[19:56] mikko ROUTER and DEALER are not in the main repo?
[19:56] mikko the one that sustrik maintains
[19:56] pieterh nope
[19:57] pieterh libzapi needs 2.1.latest
[19:57] pieterh I'm hoping sustrik will eventually make the ROUTER/DEALER changes but can't force him to
[19:57] pieterh mikko: hang on, I'
[19:57] mikko this situation is very confusing
[19:57] pieterh I'll make a patch to libzapi
[19:57] pieterh it'll become less confusing over time
[19:57] pieterh a lot of change happening in a lot of places...
[19:58] pieterh gimme 2 minutes...
[19:58] mikko no hurry
[20:00] pieterh mikko: ok, done, new version pushed
[20:00] mikko kicking off a new build
[20:00] mikko #4 on the way
[20:00] pieterh :-)
[20:02] mikko builds
[20:02] mikko
[20:02] mikko here is the code coverage
[20:04] mikko i will refactor the builds as soon as the repository rename happens
[20:04] Guthur pieterh, are you using ROUTER/DEALER in the guide
[20:04] mikko so that i can do the changes in one go
[20:04] pieterh Guthur: not yet, but that's the next step
[20:04] pieterh then, move the C code over to using libzapi
[20:04] pieterh mikko: sweet!
[20:05] mikko zeromq2-2, is this a new branch as well?
[20:06] pieterh mikko: yes, that's the packaging project for 2.2
[20:06] pieterh the packaging projects will include more than libzmq, hopefully
[20:06] mikko did my patch get through to the list ?
[20:06] pieterh but they'll track the libzmq major/minor versions
[20:06] pieterh yes, your patch made it
[20:06] mikko cool, not very good connectivity out in the sticks
[20:06] pieterh from the hot spas... :-)
[20:08] mikko so, what is the role of zeromq2, zeromq2-2 and zeromq2-1 repos?
[20:08] pieterh just for consistency, really
[20:08] pieterh zeromq2 will be deprecated, become a map for visitors who use the old URI
[20:08] mikko do we in the future push changes to upstream first?
[20:08] pieterh 2-1 and 2-2 are packaging projects, 2-2 will be where we add new projects like libzapi
[20:08] mikko because that would be ideal
[20:09] pieterh yes, if possible
[20:09] pieterh sometimes it won't be possible
[20:09] pieterh the 3.0 libzmq core will eventually change too much
[20:10] pieterh we already had this case for one patch to 2.0, which had to be hand-made for that version
[20:10] mikko something in this model feels very wrong but i really can't put my finger on it
[20:11] mikko it feels like duplicating the effort in some ways
[20:11] pieterh in fact it's not
[20:11] mikko i made the openpgm build patches twice
[20:11] mikko 2-1 and upstream
[20:11] pieterh yes, because you used pull requests to send to me
[20:11] pieterh and sustrik does not accept anything except patches to ML
[20:12] pieterh it would not change if we had one repo, two branches
[20:12] pieterh the advantage is you were able to get rapid feedback that your changes worked
[20:12] mikko so, we are still doing the patches on ML?
[20:12] pieterh if you send upstream, you have no control over when they are released
[20:12] pieterh i guess so, yes
[20:13] pieterh I'd rather use the pull model explained on the contributing page
[20:13] pieterh especially since that's how Sustriks sends me changes... :-/
[20:14] pieterh basically, any contributor has to convince the project owner to accept his patch
[20:14] pieterh project owner can define how patches are submitted
[20:15] pieterh and contributor is the one who stresses if project A and project B have different codebases
[20:15] Guthur but is this not sort of fracturing zeromq too much
[20:15] pieterh nah
[20:15] mikko yes, it is
[20:15] Guthur projects with in a project
[20:15] pieterh libzmq != ZeroMQ
[20:15] mikko this puts more pressure on contributors
[20:15] pieterh yes, mikko, it does
[20:15] pieterh but that's necessary if we're not to have bottlenecks
[20:16] mikko we already have massive bottlenecks
[20:16] pieterh yes, but this is one big one removed
[20:16] mikko we have three separate branches as three separate projects
[20:16] pieterh that's not a bottleneck...
[20:16] mikko every contributor has to mix and match their patches
[20:16] mikko and only one person can accept that patch
[20:17] pieterh sorry, this isn't accurate, mikko
[20:17] mikko if that person is on hols your patch is sitting and waiting somewhere
[20:17] pieterh this is standard with Git
[20:17] pieterh you send patches or pull requests to a project maintainer
[20:17] mikko no, it's not
[20:17] pieterh you want everyone to commit to the repository?
[20:17] mikko this is one possible workflow
[20:18] mikko who do you mean by everyone?
[20:18] pieterh it's the one that scales to many projects, because each one is smallish
[20:18] pieterh and this only affects libzmq because it's got several incompatible versions
[20:18] pieterh that is only one project out of 20 or 30
[20:20] pieterh mikko: do you have an alternative proposal that lets us do what we want to do?
[20:21] mikko currently we are doing everything very differently from what people contributing to open source are used to
[20:21] mikko and that alienates contributors
[20:21] pieterh that's true if you look at the core project but it's not true for the overall scheme
[20:22] pieterh each project accepts contributors and contributions in its own way
[20:22] pieterh this is consistent with how open source communities work
[20:22] pieterh the only difference here is we have separate projects for *packaging* ZeroMQ
[20:22] pieterh but that is necessary because libzmq != ZeroMQ
[20:23] pieterh which will become much clearer once the core repo is renamed
[20:23] mikko a contributor should not need to understand this amazing complex grand schema
[20:23] pieterh well, we do have a lot of projects, how do you propose to map this?
[20:23] mikko it should be as straigh-forward as: i run 2.1, find the git repo, make a patch
[20:24] pieterh the packaging projects are largely hidden
[20:24] pieterh make your patches against the libzmq git
[20:24] mikko this should not be followed by having to understand that actually if my patch goes to libzmq i need to get it accepted to zeromq2-2 and zeromq2-1
[20:24] mikko and possibly need to rework the patch for each version separately
[20:24] pieterh well, fair enough, but understand that...
[20:25] pieterh these repositories are for stabilization
[20:25] mikko but they aren't
[20:25] pieterh that means they only in theory get bug fixes or very safe functional changes
[20:25] mikko we pushed openpgm build fixes straight to 2-1
[20:25] mikko not through 'libzmq' repo
[20:25] pieterh indeed, let's not do that again
[20:25] pieterh however, realize that it's how you got those changes *tested* rapidly
[20:26] pieterh next time, send to the ML
[20:26] Guthur i've not really seen various version of a library being seen as projects
[20:26] pieterh RHEL x.x?
[20:26] Guthur if that happens they usually fork off and rename,
[20:26] Guthur not familiar with RH to be honest
[20:27] pieterh the linux kernel is repackaged in many distributions, each is a project
[20:27] Guthur yes but are we not talking about the 'kernel' at the moment
[20:27] pieterh well, libzmq is like the kernel for ZeroMQ
[20:28] pieterh mikko: next time you send me a pull request for 2-1, I'll reject it, OK?
[20:28] Guthur so 2, 2-1, 2-2 are distros?
[20:28] pieterh yes, thank you
[20:28] pieterh 2 is not
[20:28] pieterh 2 is dead
[20:28] pieterh will be dead once it's been renamed to libzmq
[20:29] Guthur seems I a bit crazy to me, but i'm not overly experienced with these sort of things
[20:29] pieterh there is no other way to create the freedom to make real packaging projects
[20:29] pieterh it seems pretty clear to me
[20:29] pieterh alternative is to all work in one repo, and have no packaging projects
[20:30] mikko hahaha, you designed the schema
[20:30] mikko of course it's clear to you
[20:30] pieterh sure
[20:30] mikko but from contributor's point of view the situation is not very good (imho)
[20:30] pieterh well, it'll be clearer IMO when libzmq is properly named
[20:30] pieterh "send your patch to the libzmq project"
[20:30] pieterh and that can downstream to the various distros
[20:30] mikko what happens when sustrik is on holidays?
[20:31] pieterh ask him, I don't know
[20:31] pieterh mikko: this is not a command-and-control model
[20:31] mikko see, are we not creating massive bottlenecks?
[20:31] pieterh propose an alternative to "person who knows the code real well"
[20:31] pieterh we're not *creating* that bottleneck, it exists
[20:31] pieterh seriously, your alternative is what, clone sustrik?
[20:32] pieterh there are other people who know the core structures pretty well
[20:32] pieterh if sustrik is unable to act as maintainer, I'm sure he'll delegate
[20:32] pieterh but it is *his* responsibility
[20:32] pieterh you and me can debate that for years, it matters not at all
[20:32] mikko yes, what i don't get is this cycle:
[20:32] Guthur could we not have the situation where the 'distros' start deviating quite far from the main dev branch
[20:33] pieterh Guthur: unlikely, why would they?
[20:33] mikko fork libzmq, create patch, commit, create signed patch, reset, wait for a merge, then pull down your own changes
[20:33] pieterh the very point is that the people maintaining distros don't know the code
[20:33] Guthur just curious really, they seemed to have become more than just previous versions
[20:33] pieterh mikko: ack, it's complex and I don't like it
[20:33] Guthur they seemed to have become their own projects
[20:34] mikko forgot the "send mail to mailing-list" from that
[20:34] pieterh Guthur: 'seem to' is mild, the goal was to create dependent projects
[20:34] pieterh mikko: yes, I know
[20:34] pieterh I've lobbied sustrik on this several times, here and on the list
[20:34] pieterh next time, chime in, on the ML thread ... :-/
[20:35] mikko i had forgotten about the ML patches already
[20:35] pieterh I spent a lot of effort documenting alternative flows on the contributing page
[20:39] Guthur pieterh, dependent or independent?
[20:40] pieterh both :-)
[20:40] pieterh look, 2-1 is not a good example, it's too young
[20:40] Guthur though it already as deviation
[20:40] pieterh 2-2 will include, in my plans, libzmq, libzapi, the new C++ binding
[20:40] Guthur where something needs forward ported
[20:40] pieterh obviously it depends on those N projects
[20:41] Guthur admittedly only a minor #define
[20:41] pieterh yes
[20:41] pieterh one way would be to use submodules and branches in the source repo
[20:41] pieterh that would require that the branches are maintained in that repo
[20:42] pieterh experience is that this wasn't happening, due to complexity of moving changes between branches (or somesuch)
[20:42] pieterh lack of interest from main developers in maintaining branches
[20:42] mikko it's the same amount of complexity as between different repos
[20:42] pieterh yes
[20:43] pieterh exactly, but with different repos, you can slice the work over several people much more easily
[20:43] pieterh this is the key insight
[20:43] pieterh collaborating inside one repo is really difficult
[20:43] pieterh it demands agreement
[20:43] pieterh clearly we don't get that easily
[20:43] Guthur I thought git made it easier
[20:43] pieterh it doesn't change people and make them less stubborn
[20:43] Guthur certainly easier than the likes of SVN
[20:44] pieterh git makes it easier to have many repos
[20:44] pieterh indeed
[20:44] pieterh git lets us solve the problem by slicing projects up, like I said on the ML, per ego
[20:44] pieterh it's actually a really pleasant way to work
[20:44] Guthur yeah, that discussion got a little out of hand
[20:45] Guthur well it appeared so from an onlookers perspective
[20:45] pieterh exactly, and that was the same last time we discussed multiple branches in the main repo
[20:46] pieterh that was sometime in July iir, and my conclusion was "don't work in the core repo, ever"
[20:47] mikko as the community grows an ideal process to me would be to have multiple people with commit access
[20:47] mikko and each set of changes is mailed automatically to mailing-list
[20:47] pieterh mikko: yes, probably a good model
[20:48] mikko this also gives more visibility to changes going into repo and allows easily start conversation on mailing-list
[20:48] pieterh however, I'm not going to raise this discussion for libzmq
[20:49] pieterh you already have committer access to libzapi and libzfl
[20:50] Guthur libzmq is the heart of it all though, and so the most important
[20:50] mikko yes
[20:50] mikko for example now i would be keen on getting coverage into libzmq
[20:50] pieterh yes
[20:50] mikko but it's futile to work on the stuff before openpgm patch has been merged
[20:51] pieterh mikko: my experience contributing to core has been quite negative, I have to admit
[20:51] mikko i don't even know if rebase works well on that cycle
[20:51] mikko as in, when i pull back the changes merged by sustrik does that conflict with my local changes?
[20:51] mikko or is it the same commit?
[20:52] pieterh depends how you do it IMO
[20:58] pieterh Guthur: OK, I'm able to inline the main language, the page is slow but loads
[21:00] Seta00 pieterh: the libzapi project file is broken
[21:00] pieterh Seta00: yes, I've not tested it yet on MSVC
[21:01] pieterh do you want to try fixing it and sending a patch?
[21:01] Seta00 yes
[21:01] pieterh it's a copy of the libzfl project file, but obviously refers to the wrong sources & includes
[21:01] pieterh thanks! :-)
[21:06] pieterh g'night folks, I need to crash, early morning tomorrow
[21:06] Seta00 pieterh: just a small question before you go, if you have to include non-standard headers to the project, where should they go? on the standard include directory?
[21:06] Seta00 (MSVC doesn't have inttypes.h)
[21:07] Seta00 but there's
[21:07] pieterh Seta00: hmm, edit zapi_prelude.h to include the right files and move the existing include to one of the #if UNIX blocks
[21:07] Seta00 okay thanks
[21:07] pieterh there's a section in the doc on Porting libzapi, may be helpful
[21:39] Seta00 apparently msvc has no implementation of sigaction
[21:39] Seta00 :(
[21:42] JusticeFries what are the differences in typical use cases for 0mq vs something like rabbitmq?
[23:05] klestes hello, is anyone talking today ?
[23:19] klestes if anyone does see this, what is the best way to interoperate with channels in basic (non 0MQ) tcp ? do you have to write a custom gateway, TCP <--> 0MQ, or is there a better solution inherent in 0MQ that I've missed ?
[23:20] jdroid- i've been using zmq with python and am trying to compile some stuff with c now, but I keep getting errors.
[23:20] jdroid- does this make sense to anyone?
[23:21] jdroid- here's the code:
[23:21] klestes you have to program a lot lower with C than with python - how would you say the experience has been using python with 0MQ ?
[23:22] jdroid- The experience with python has been awesome so far. Very easy.
[23:22] jdroid- I built a Mongrel2 handler with eventlet and zeromq recently.
[23:23] jdroid- I've been posting a few pyzmq examples to github as it comes up in conversation, but I want to add a multilang example AND get back in C doing it.
[23:23] jdroid-
[23:23] klestes after learning Perl, my goal is never go back to C! Evah !
[23:24] jdroid- After learning Perl, I vowed never to go back to Perl. So right back atcha buddy!
[23:24] jdroid- :)
[23:24] klestes also you might want to check out the new zapi. Hintjen is very excited about it, so it could be big.
[23:25] klestes well, I'm not going to stir up a fight. I have a lot of respect for Python, and might be doing some work in it soon myself.
[23:26] klestes my point was to be, programming 0MQ in either python or perl is much easier than C, IMHO.
[23:26] jdroid- Well I'm only kidding anyway. Perl is a fine language if you're working with people who respect others. :)
[23:28] klestes It has its warts, and its uhm, non warty really ok type features.
[23:28] klestes I get sick of all those $ signs all the time, esp. since its been a while since I've seen $$$$ !
[23:29] jdroid- You'll dig python then I'm sure. The code is generally as easy to read as people say.
[23:29] jdroid- But anyway... I'm trying to get some c compiled!
[23:29] klestes I looked at your code, but not being in C for at least a decade, I couildn't catch anything right off the bat. I know its either a linking issue, or maybe forgetting to include a header (but you seem to have done that right)
[23:30] klestes what compiler are you using, Gcc ?
[23:31] jdroid- oh great question actually... i'm using the gcc xcode gave me
[23:31] jdroid- seems to compile mongrel2 just fine, so i figure i've got my args wrong somehow
[23:31] jdroid- gcc 4.2.1 specifically tho
[23:31] klestes uhm, xcode - are you on OS/X ?
[23:31] jdroid- yep
[23:31] klestes lucky guy !
[23:32] jdroid- i guess..? macs are cheap these days
[23:32] klestes someday I'm gonna go mac...and never come back.
[23:32] klestes well, depends on what kinda mac you want. the one I want is 2.5 - 3k.
[23:34] klestes try compiling and linking one of the simple examples from the guide. do you get the same errors ?
[23:35] jdroid- i seem to get the same errrors
[23:35] jdroid- hmm
[23:36] jdroid- i tried this:
[23:38] jdroid- ah got it
[23:39] jdroid- i forgot -lzmq at the end of the gcc cmd
[23:39] JusticeFries what is the difference in typical use case for 0mq versus something in the realm of rabbitmq?
[23:40] jdroid- JusticeFries: I think the mistake there is assuming zmq is a replacement for rabbitmq
[23:40] JusticeFries nope
[23:40] JusticeFries not assuming replacement.
[23:40] JusticeFries i'm interested in how they complement each other, too. :)
[23:40] JusticeFries and what each one excels at.
[23:40] jdroid- :)
[23:41] jdroid- So ZMQ is more like sockets with messaging semantics. You'd still build your broker, which might be dropping stuff from one zmq socket in a db, while sending stuff out of the db across another socket
[23:41] JusticeFries okay.
[23:42] JusticeFries unix sockets, or something more specific to ZMQ?
[23:42] jdroid- Zmq offers multiple types of sockets.
[23:42] JusticeFries fair enough.
[23:42] jdroid- You could use unix sockets if you want. You could use tcp just as easily
[23:43] JusticeFries so then - I'm looking at doing something that settles somewhere near rabbit's topic exchanges. basically, using it for some group messaging functions for end users that doesn't hit the DB.
[23:43] klestes I've heard that rabbitMQ is somewhat synergetic with ZMQ, it can be made into a broker.
[23:44] JusticeFries i've heard that too.
[23:44] klestes we musta read the same article ;)_
[23:44] JusticeFries just not sure where to fit it in, or if it is even useful in this current context.
[23:44] jdroid- what you heard is probably that zeromq has amqp support:
[23:44] JusticeFries either that or ilya's blog.
[23:45] JusticeFries igvita
[23:45] JusticeFries heh I like the gross generalizations.
[23:45] jdroid- thanks. i'm the James Dennis that's quoted!
[23:45] klestes yeah, he does that well :)
[23:45] JusticeFries ha nice.
[23:46] klestes its a small 0world after all.
[23:46] JusticeFries well i may assault you with questions later - though the guide is pretty extensive
[23:46] klestes 0mq used to - didn't they chunk the amqp thang ?
[23:46] klestes yeah, and it seems to be getting biggeer and betta' all the time...