ZeroMq IRC Log

Sunday March 27, 2011

[Time] Name	Message
[04:17] EduardoFonseca	Hi Guys. I'm trying to figure out how to implement connection dettection on top of zeroMQ. Any tips?
[04:17] EduardoFonseca	*detection, sorry.
[04:37] EduardoFonseca	great
[04:38] EduardoFonseca	but what about detecting when this connection has gone away?
[04:38] EduardoFonseca	I tried implementing heartbeats, but the code started to get really complex.
[04:40] EduardoFonseca	I'm using XREQ and XREP... but the loadbalancing part got complex.
[04:40] EduardoFonseca	I was planning to reimplement the Majordomo pattern... but, why use zeroMQ if you need a broker? :)
[17:09] mikko	howdy guys
[17:57] Guthur	hey mikko
[17:57] Guthur	do you know if Ian Barbers PHP presentation slides are available anywhere for download, besides slideshare
[17:58] mikko	Guthur: let me check
[17:59] mikko	his blog seems to be down
[18:06] Guthur	ok, I'll check it later
[18:06] Guthur	cheers mikko
[18:14] mikko	i can ask him for a pdf
[18:25] Guthur	mikko, that would great, cheers
[18:36] pieter_hintjens	Guthur: slideshare should offer downloads, doesn't it?
[18:36] Guthur	pieter_hintjens, It does. i just didn't want to sign up
[18:36] pieter_hintjens	yeah
[18:41] mikko	finally got some more build capacity online
[18:41] mikko	x86_64 centos
[18:41] mikko	rpms are now built every night
[18:43] mikko	pieter_hintjens: i'll try to add libzapi today
[18:43] pieter_hintjens	mikko: nice!
[18:43] pieter_hintjens	I also renamed ZFL to libzfl, you saw that
[18:43] mikko	i did
[18:43] Guthur	I was actually going to draw some inspiration from Ian's presentation for a presentation at work
[18:43] mikko	i was away yesterday and today
[18:43] mikko	spa weekend
[18:43] pieter_hintjens	:-) it was excellent weather here in Belgium
[18:43] mikko	been doing nothing but soaking in warm water
[18:44] Guthur	I've been wanting to ask him if it would be ok to borrow some stuff, just haven't been able to catch him in here
[18:49] pieterh	all code blocks highlighted, all collapsible (except a few)
[18:49] pieterh	http://zguide.zeromq.org/test:ch1
[19:45] Guthur	it loads for me
[19:45] Guthur	but there is issues with the language translation link
[19:45] Guthur	links*
[19:45] pieterh	yes, reload in a few heartbeats...
[19:45] pieterh	the problem is the marked up code blocks are huge
[19:46] pieterh	including 320 of them in the page kind of does not work
[19:46] Guthur	hehe, I can imagine
[19:47] pieterh	it should work better now, code examples appear in a new tab
[19:47] pieterh	shame, it was kind of nice to have them inline...
[19:50] Guthur	oh new tabs, umm, I tend to curse pages that do that
[19:51] Guthur	is the C version still inline?
[19:53] pieterh	well, nope, but I'm going to make a further round of improvements
[19:53] pieterh	will inline the C version
[19:53] pieterh	and ditto for the PHP version
[19:56] mikko	pieterh: http://build.zero.mq/job/libzapi/3/console
[19:56] mikko	ROUTER and DEALER are not in the main repo?
[19:56] mikko	the one that sustrik maintains
[19:56] pieterh	nope
[19:57] pieterh	libzapi needs 2.1.latest
[19:57] pieterh	I'm hoping sustrik will eventually make the ROUTER/DEALER changes but can't force him to
[19:57] pieterh	mikko: hang on, I'
[19:57] mikko	this situation is very confusing
[19:57] pieterh	I'll make a patch to libzapi
[19:57] pieterh	it'll become less confusing over time
[19:57] pieterh	a lot of change happening in a lot of places...
[19:58] pieterh	gimme 2 minutes...
[19:58] mikko	no hurry
[20:00] pieterh	mikko: ok, done, new version pushed
[20:00] mikko	kicking off a new build
[20:00] mikko	#4 on the way
[20:00] pieterh	:-)
[20:02] mikko	builds
[20:02] mikko	http://build.zero.mq/job/libzapi/4/cobertura/_default_/
[20:02] mikko	here is the code coverage
[20:04] mikko	i will refactor the builds as soon as the repository rename happens
[20:04] Guthur	pieterh, are you using ROUTER/DEALER in the guide
[20:04] mikko	so that i can do the changes in one go
[20:04] pieterh	Guthur: not yet, but that's the next step
[20:04] pieterh	then, move the C code over to using libzapi
[20:04] pieterh	mikko: sweet!
[20:05] mikko	zeromq2-2, is this a new branch as well?
[20:06] pieterh	mikko: yes, that's the packaging project for 2.2
[20:06] pieterh	the packaging projects will include more than libzmq, hopefully
[20:06] mikko	did my patch get through to the list ?
[20:06] pieterh	but they'll track the libzmq major/minor versions
[20:06] pieterh	yes, your patch made it
[20:06] mikko	cool, not very good connectivity out in the sticks
[20:06] pieterh	from the hot spas... :-)
[20:08] mikko	so, what is the role of zeromq2, zeromq2-2 and zeromq2-1 repos?
[20:08] pieterh	just for consistency, really
[20:08] pieterh	zeromq2 will be deprecated, become a map for visitors who use the old URI
[20:08] mikko	do we in the future push changes to upstream first?
[20:08] pieterh	2-1 and 2-2 are packaging projects, 2-2 will be where we add new projects like libzapi
[20:08] mikko	because that would be ideal
[20:09] pieterh	yes, if possible
[20:09] pieterh	sometimes it won't be possible
[20:09] pieterh	the 3.0 libzmq core will eventually change too much
[20:10] pieterh	we already had this case for one patch to 2.0, which had to be hand-made for that version
[20:10] mikko	something in this model feels very wrong but i really can't put my finger on it
[20:11] mikko	it feels like duplicating the effort in some ways
[20:11] pieterh	in fact it's not
[20:11] mikko	i made the openpgm build patches twice
[20:11] mikko	2-1 and upstream
[20:11] pieterh	yes, because you used pull requests to send to me
[20:11] pieterh	and sustrik does not accept anything except patches to ML
[20:12] pieterh	it would not change if we had one repo, two branches
[20:12] pieterh	the advantage is you were able to get rapid feedback that your changes worked
[20:12] mikko	so, we are still doing the patches on ML?
[20:12] pieterh	if you send upstream, you have no control over when they are released
[20:12] pieterh	i guess so, yes
[20:13] pieterh	I'd rather use the pull model explained on the contributing page
[20:13] pieterh	especially since that's how Sustriks sends me changes... :-/
[20:14] pieterh	basically, any contributor has to convince the project owner to accept his patch
[20:14] pieterh	project owner can define how patches are submitted
[20:15] pieterh	and contributor is the one who stresses if project A and project B have different codebases
[20:15] Guthur	but is this not sort of fracturing zeromq too much
[20:15] pieterh	nah
[20:15] mikko	yes, it is
[20:15] Guthur	projects with in a project
[20:15] pieterh	libzmq != ZeroMQ
[20:15] mikko	this puts more pressure on contributors
[20:15] pieterh	yes, mikko, it does
[20:15] pieterh	but that's necessary if we're not to have bottlenecks
[20:16] mikko	we already have massive bottlenecks
[20:16] pieterh	yes, but this is one big one removed
[20:16] mikko	we have three separate branches as three separate projects
[20:16] pieterh	that's not a bottleneck...
[20:16] mikko	every contributor has to mix and match their patches
[20:16] mikko	and only one person can accept that patch
[20:17] pieterh	sorry, this isn't accurate, mikko
[20:17] mikko	if that person is on hols your patch is sitting and waiting somewhere
[20:17] pieterh	this is standard with Git
[20:17] pieterh	you send patches or pull requests to a project maintainer
[20:17] mikko	no, it's not
[20:17] pieterh	you want everyone to commit to the repository?
[20:17] mikko	this is one possible workflow
[20:18] mikko	who do you mean by everyone?
[20:18] pieterh	it's the one that scales to many projects, because each one is smallish
[20:18] pieterh	and this only affects libzmq because it's got several incompatible versions
[20:18] pieterh	that is only one project out of 20 or 30
[20:20] pieterh	mikko: do you have an alternative proposal that lets us do what we want to do?
[20:21] mikko	currently we are doing everything very differently from what people contributing to open source are used to
[20:21] mikko	and that alienates contributors
[20:21] pieterh	that's true if you look at the core project but it's not true for the overall scheme
[20:22] pieterh	each project accepts contributors and contributions in its own way
[20:22] pieterh	this is consistent with how open source communities work
[20:22] pieterh	the only difference here is we have separate projects for packaging ZeroMQ
[20:22] pieterh	but that is necessary because libzmq != ZeroMQ
[20:23] pieterh	which will become much clearer once the core repo is renamed
[20:23] mikko	a contributor should not need to understand this amazing complex grand schema
[20:23] pieterh	well, we do have a lot of projects, how do you propose to map this?
[20:23] mikko	it should be as straigh-forward as: i run 2.1, find the git repo, make a patch
[20:24] pieterh	the packaging projects are largely hidden
[20:24] pieterh	make your patches against the libzmq git
[20:24] mikko	this should not be followed by having to understand that actually if my patch goes to libzmq i need to get it accepted to zeromq2-2 and zeromq2-1
[20:24] mikko	and possibly need to rework the patch for each version separately
[20:24] pieterh	well, fair enough, but understand that...
[20:25] pieterh	these repositories are for stabilization
[20:25] mikko	but they aren't
[20:25] pieterh	that means they only in theory get bug fixes or very safe functional changes
[20:25] mikko	we pushed openpgm build fixes straight to 2-1
[20:25] mikko	not through 'libzmq' repo
[20:25] pieterh	indeed, let's not do that again
[20:25] pieterh	however, realize that it's how you got those changes tested rapidly
[20:26] pieterh	next time, send to the ML
[20:26] Guthur	i've not really seen various version of a library being seen as projects
[20:26] pieterh	RHEL x.x?
[20:26] Guthur	if that happens they usually fork off and rename,
[20:26] Guthur	not familiar with RH to be honest
[20:27] pieterh	the linux kernel is repackaged in many distributions, each is a project
[20:27] Guthur	yes but are we not talking about the 'kernel' at the moment
[20:27] pieterh	well, libzmq is like the kernel for ZeroMQ
[20:28] pieterh	mikko: next time you send me a pull request for 2-1, I'll reject it, OK?
[20:28] Guthur	so 2, 2-1, 2-2 are distros?
[20:28] pieterh	yes, thank you
[20:28] pieterh	2 is not
[20:28] pieterh	2 is dead
[20:28] pieterh	will be dead once it's been renamed to libzmq
[20:29] Guthur	seems I a bit crazy to me, but i'm not overly experienced with these sort of things
[20:29] pieterh	there is no other way to create the freedom to make real packaging projects
[20:29] pieterh	it seems pretty clear to me
[20:29] pieterh	alternative is to all work in one repo, and have no packaging projects
[20:30] mikko	hahaha, you designed the schema
[20:30] mikko	of course it's clear to you
[20:30] pieterh	sure
[20:30] mikko	but from contributor's point of view the situation is not very good (imho)
[20:30] pieterh	well, it'll be clearer IMO when libzmq is properly named
[20:30] pieterh	"send your patch to the libzmq project"
[20:30] pieterh	and that can downstream to the various distros
[20:30] mikko	what happens when sustrik is on holidays?
[20:31] pieterh	ask him, I don't know
[20:31] pieterh	mikko: this is not a command-and-control model
[20:31] mikko	see, are we not creating massive bottlenecks?
[20:31] pieterh	propose an alternative to "person who knows the code real well"
[20:31] pieterh	we're not creating that bottleneck, it exists
[20:31] pieterh	seriously, your alternative is what, clone sustrik?
[20:32] pieterh	there are other people who know the core structures pretty well
[20:32] pieterh	if sustrik is unable to act as maintainer, I'm sure he'll delegate
[20:32] pieterh	but it is his responsibility
[20:32] pieterh	you and me can debate that for years, it matters not at all
[20:32] mikko	yes, what i don't get is this cycle:
[20:32] Guthur	could we not have the situation where the 'distros' start deviating quite far from the main dev branch
[20:33] pieterh	Guthur: unlikely, why would they?
[20:33] mikko	fork libzmq, create patch, commit, create signed patch, reset, wait for a merge, then pull down your own changes
[20:33] pieterh	the very point is that the people maintaining distros don't know the code
[20:33] Guthur	just curious really, they seemed to have become more than just previous versions
[20:33] pieterh	mikko: ack, it's complex and I don't like it
[20:33] Guthur	they seemed to have become their own projects
[20:34] mikko	forgot the "send mail to mailing-list" from that
[20:34] pieterh	Guthur: 'seem to' is mild, the goal was to create dependent projects
[20:34] pieterh	mikko: yes, I know
[20:34] pieterh	I've lobbied sustrik on this several times, here and on the list
[20:34] pieterh	next time, chime in, on the ML thread ... :-/
[20:35] mikko	i had forgotten about the ML patches already
[20:35] pieterh	I spent a lot of effort documenting alternative flows on the contributing page
[20:39] Guthur	pieterh, dependent or independent?
[20:40] pieterh	both :-)
[20:40] pieterh	look, 2-1 is not a good example, it's too young
[20:40] Guthur	though it already as deviation
[20:40] pieterh	2-2 will include, in my plans, libzmq, libzapi, the new C++ binding
[20:40] Guthur	where something needs forward ported
[20:40] pieterh	obviously it depends on those N projects
[20:41] Guthur	admittedly only a minor #define
[20:41] pieterh	yes
[20:41] pieterh	one way would be to use submodules and branches in the source repo
[20:41] pieterh	that would require that the branches are maintained in that repo
[20:42] pieterh	experience is that this wasn't happening, due to complexity of moving changes between branches (or somesuch)
[20:42] pieterh	lack of interest from main developers in maintaining branches
[20:42] mikko	it's the same amount of complexity as between different repos
[20:42] pieterh	yes
[20:43] pieterh	exactly, but with different repos, you can slice the work over several people much more easily
[20:43] pieterh	this is the key insight
[20:43] pieterh	collaborating inside one repo is really difficult
[20:43] pieterh	it demands agreement
[20:43] pieterh	clearly we don't get that easily
[20:43] Guthur	I thought git made it easier
[20:43] pieterh	it doesn't change people and make them less stubborn
[20:43] Guthur	certainly easier than the likes of SVN
[20:44] pieterh	git makes it easier to have many repos
[20:44] pieterh	indeed
[20:44] pieterh	git lets us solve the problem by slicing projects up, like I said on the ML, per ego
[20:44] pieterh	it's actually a really pleasant way to work
[20:44] Guthur	yeah, that discussion got a little out of hand
[20:45] Guthur	well it appeared so from an onlookers perspective
[20:45] pieterh	exactly, and that was the same last time we discussed multiple branches in the main repo
[20:46] pieterh	that was sometime in July iir, and my conclusion was "don't work in the core repo, ever"
[20:47] mikko	as the community grows an ideal process to me would be to have multiple people with commit access
[20:47] mikko	and each set of changes is mailed automatically to mailing-list
[20:47] pieterh	mikko: yes, probably a good model
[20:48] mikko	this also gives more visibility to changes going into repo and allows easily start conversation on mailing-list
[20:48] pieterh	however, I'm not going to raise this discussion for libzmq
[20:49] pieterh	you already have committer access to libzapi and libzfl
[20:50] Guthur	libzmq is the heart of it all though, and so the most important
[20:50] mikko	yes
[20:50] mikko	for example now i would be keen on getting coverage into libzmq
[20:50] pieterh	yes
[20:50] mikko	but it's futile to work on the stuff before openpgm patch has been merged
[20:51] pieterh	mikko: my experience contributing to core has been quite negative, I have to admit
[20:51] mikko	i don't even know if rebase works well on that cycle
[20:51] mikko	as in, when i pull back the changes merged by sustrik does that conflict with my local changes?
[20:51] mikko	or is it the same commit?
[20:52] pieterh	depends how you do it IMO
[20:58] pieterh	Guthur: OK, I'm able to inline the main language, the page is slow but loads
[21:00] Seta00	pieterh: the libzapi project file is broken
[21:00] pieterh	Seta00: yes, I've not tested it yet on MSVC
[21:01] pieterh	do you want to try fixing it and sending a patch?
[21:01] Seta00	yes
[21:01] pieterh	it's a copy of the libzfl project file, but obviously refers to the wrong sources & includes
[21:01] pieterh	thanks! :-)
[21:06] pieterh	g'night folks, I need to crash, early morning tomorrow
[21:06] Seta00	pieterh: just a small question before you go, if you have to include non-standard headers to the project, where should they go? on the standard include directory?
[21:06] Seta00	(MSVC doesn't have inttypes.h)
[21:07] Seta00	but there's https://code.google.com/p/msinttypes/
[21:07] pieterh	Seta00: hmm, edit zapi_prelude.h to include the right files and move the existing include to one of the #if UNIX blocks
[21:07] Seta00	okay thanks
[21:07] pieterh	there's a section in the doc on Porting libzapi, may be helpful
[21:39] Seta00	apparently msvc has no implementation of sigaction
[21:39] Seta00	:(
[21:42] JusticeFries	what are the differences in typical use cases for 0mq vs something like rabbitmq?
[23:05] klestes	hello, is anyone talking today ?
[23:19] klestes	if anyone does see this, what is the best way to interoperate with channels in basic (non 0MQ) tcp ? do you have to write a custom gateway, TCP <--> 0MQ, or is there a better solution inherent in 0MQ that I've missed ?
[23:20] jdroid-	i've been using zmq with python and am trying to compile some stuff with c now, but I keep getting errors.
[23:20] jdroid-	does this make sense to anyone? http://pastebin.com/7Mj5KSSH
[23:21] jdroid-	here's the code: http://pastebin.com/efKgRJiK
[23:21] klestes	you have to program a lot lower with C than with python - how would you say the experience has been using python with 0MQ ?
[23:22] jdroid-	The experience with python has been awesome so far. Very easy.
[23:22] jdroid-	I built a Mongrel2 handler with eventlet and zeromq recently. github.com/j2labs/brubeck
[23:23] jdroid-	I've been posting a few pyzmq examples to github as it comes up in conversation, but I want to add a multilang example AND get back in C doing it.
[23:23] jdroid-	https://github.com/j2labs/zmq_examples
[23:23] klestes	after learning Perl, my goal is never go back to C! Evah !
[23:24] jdroid-	After learning Perl, I vowed never to go back to Perl. So right back atcha buddy!
[23:24] jdroid-	:)
[23:24] klestes	also you might want to check out the new zapi. Hintjen is very excited about it, so it could be big.
[23:25] klestes	well, I'm not going to stir up a fight. I have a lot of respect for Python, and might be doing some work in it soon myself.
[23:26] klestes	my point was to be, programming 0MQ in either python or perl is much easier than C, IMHO.
[23:26] jdroid-	Well I'm only kidding anyway. Perl is a fine language if you're working with people who respect others. :)
[23:28] klestes	It has its warts, and its uhm, non warty really ok type features.
[23:28] klestes	I get sick of all those $ signs all the time, esp. since its been a while since I've seen $$$$ !
[23:29] jdroid-	You'll dig python then I'm sure. The code is generally as easy to read as people say.
[23:29] jdroid-	But anyway... I'm trying to get some c compiled!
[23:29] klestes	I looked at your code, but not being in C for at least a decade, I couildn't catch anything right off the bat. I know its either a linking issue, or maybe forgetting to include a header (but you seem to have done that right)
[23:30] klestes	what compiler are you using, Gcc ?
[23:31] jdroid-	oh great question actually... i'm using the gcc xcode gave me
[23:31] jdroid-	seems to compile mongrel2 just fine, so i figure i've got my args wrong somehow
[23:31] jdroid-	gcc 4.2.1 specifically tho
[23:31] klestes	uhm, xcode - are you on OS/X ?
[23:31] jdroid-	yep
[23:31] klestes	lucky guy !
[23:32] jdroid-	i guess..? macs are cheap these days
[23:32] klestes	someday I'm gonna go mac...and never come back.
[23:32] klestes	well, depends on what kinda mac you want. the one I want is 2.5 - 3k.
[23:34] klestes	try compiling and linking one of the simple examples from the guide. do you get the same errors ?
[23:35] jdroid-	i seem to get the same errrors
[23:35] jdroid-	hmm
[23:36] jdroid-	i tried this: http://zguide.zeromq.org/c:hwclient
[23:38] jdroid-	ah got it
[23:39] jdroid-	i forgot -lzmq at the end of the gcc cmd
[23:39] JusticeFries	what is the difference in typical use case for 0mq versus something in the realm of rabbitmq?
[23:40] jdroid-	JusticeFries: I think the mistake there is assuming zmq is a replacement for rabbitmq
[23:40] JusticeFries	nope
[23:40] JusticeFries	not assuming replacement.
[23:40] JusticeFries	i'm interested in how they complement each other, too. :)
[23:40] JusticeFries	and what each one excels at.
[23:40] jdroid-	:)
[23:41] jdroid-	So ZMQ is more like sockets with messaging semantics. You'd still build your broker, which might be dropping stuff from one zmq socket in a db, while sending stuff out of the db across another socket
[23:41] JusticeFries	okay.
[23:42] JusticeFries	unix sockets, or something more specific to ZMQ?
[23:42] jdroid-	Zmq offers multiple types of sockets.
[23:42] JusticeFries	fair enough.
[23:42] jdroid-	You could use unix sockets if you want. You could use tcp just as easily
[23:43] JusticeFries	so then - I'm looking at doing something that settles somewhere near rabbit's topic exchanges. basically, using it for some group messaging functions for end users that doesn't hit the DB.
[23:43] klestes	I've heard that rabbitMQ is somewhat synergetic with ZMQ, it can be made into a broker.
[23:44] JusticeFries	i've heard that too.
[23:44] klestes	we musta read the same article ;)_
[23:44] JusticeFries	just not sure where to fit it in, or if it is even useful in this current context.
[23:44] jdroid-	what you heard is probably that zeromq has amqp support: http://www.zeromq.org/docs:welcome-from-amqp
[23:44] JusticeFries	either that or ilya's blog.
[23:45] JusticeFries	igvita
[23:45] JusticeFries	heh I like the gross generalizations.
[23:45] jdroid-	thanks. i'm the James Dennis that's quoted!
[23:45] klestes	yeah, he does that well :)
[23:45] JusticeFries	ha nice.
[23:46] klestes	its a small 0world after all.
[23:46] JusticeFries	well i may assault you with questions later - though the guide is pretty extensive
[23:46] klestes	0mq used to - didn't they chunk the amqp thang ?
[23:46] klestes	yeah, and it seems to be getting biggeer and betta' all the time...