IRC Log

Saturday March 12, 2011

[Time] NameMessage
[08:14] pieterh hi dermoth|home... who are the chanserv admins then?
[08:53] guido_g good morning all
[08:55] pieterh hi guido_g!
[08:55] guido_g howdy pieterh
[08:56] guido_g but in general in works :)
[08:57] pieterh nice
[08:58] guido_g did my first steps with oprofile last night
[08:59] pieterh sounds like a profiler... any good?
[08:59] guido_g oh yes
[08:59] guido_g one need to knwo al lot in advance
[08:59] guido_g omg
[08:59] guido_g more coffee
[08:59] pieterh heh
[09:00] guido_g this thing does a overall profiling of the whole system
[09:00] guido_g so you can see how your app is interfering w/ the kernel
[09:00] guido_g amazing
[09:00] guido_g as well in complexity as in usefulness
[09:00] pieterh hmm, do keep notes, this sounds like useful stuff
[09:01] guido_g and i only discovered it because i got crashes when i tried to use the -pg option (gprof)
[09:02] guido_g as i said, it's a beast http://oprofile.sourceforge.net/docs/ <- just for short look, if you dare
[09:02] pieterh nope, I don't dare
[09:03] pieterh but do keep notes and perhaps make a page explaining how you used it to profile your 0MQ app...
[09:03] guido_g ok
[09:04] guido_g will try, but i'm not good at keeping notes :/
[09:04] pieterh just copy the commands you used, with one line of 'why' each time
[09:04] pieterh if you like I'll help turn that into a tutorial
[09:04] pieterh if it's useful and I can understand it...
[09:04] guido_g ok
[09:05] guido_g i'm sure i'll have a deeper look at it next week
[09:05] pieterh I was going to use -pg/gprof but didn't have time to yet
[09:05] guido_g as i said, -pg crashes w/ the normal libzmq build
[12:21] Guthur pieterh, I think this signal catching will working as a solution for clrzmq2 on MONO
[12:21] Guthur it's made the binding code a little bit messier though
[12:21] Guthur too many precompiler directives flying around for my liking now
[12:38] Guthur welcome back sustrik
[12:38] Guthur hope you had a nice time in the USA
[12:39] sustrik hi
[12:39] sustrik yeah, it was quite ok
[12:40] Guthur sounds like the meetup was productive, at least
[12:40] sustrik yeah, seem the minutes?
[12:42] Guthur seen the messages on the mailing list
[12:43] sustrik meeting in office space seems to be more productive than meeting in a pub :)
[12:43] sustrik where are you located, btw?
[12:43] Guthur Northern Ireland
[12:44] Guthur I would be hard pushed to take time off for the london meetup this month, even if it wasn't a pub
[12:44] Guthur I'd already arranged to go to the European Lisp Symposium this month in Hamburg
[12:45] sustrik you can meet guido_g there
[12:45] sustrik he's from hamburg iirc
[12:45] Guthur oh, i'd know that
[12:45] Guthur i didn't*
[12:45] guido_g hmm what?
[12:45] guido_g oh yes, i'm in hamburg
[12:46] Guthur I'm there for a Lisp conference at the end of the month
[12:46] guido_g Guthur: cool, drop me a note if you see time to meet
[12:46] Guthur sure
[12:47] Guthur I'm there from 30th march to 2n april
[12:47] sustrik i would suggest asking on the mailing list, there may be more 0mq users in that location
[12:47] guido_g Guthur: that's nice
[12:47] Guthur super Idea
[12:50] Guthur It's a shame I never thought of producing a presentation for the ELS of Common Lisp and 0MQ
[12:50] Guthur the main topic this year is Concurrent Computing
[12:51] Guthur well one of the topics at least
[12:52] sustrik it would fit it
[12:52] sustrik in
[12:53] mikko sustrik: welcome back
[12:53] sustrik hi
[12:53] sustrik how it's going?
[12:53] Guthur I'll definitely be mentioning 0MQ to anyone I speak to
[12:53] mikko same old same old
[12:53] Guthur spread the word and all... hehe
[12:53] mikko how was US?
[12:53] sustrik good
[12:53] sustrik met a lot of people there
[12:54] sustrik i've met the original author of Tibco RV :)
[12:54] Guthur oh cool
[12:54] sustrik btw, i've wanted to update openpgm in master but got some compilation errors
[12:54] Guthur There has been some interest in work at comparing 0MQ to Tibco RV
[12:54] sustrik Guthur: yes, seen it
[12:55] sustrik mikko: pieter mentioned you've already experimented with that in stable
[12:56] guido_g Guthur: your attending the welcome at freiheit.com on march 30?
[12:57] guido_g Guthur: if this is the right event at all http://www.european-lisp-symposium.org/content-programme-full.html
[12:58] Guthur guido_g, I unfortunately booked the flights before I knew about the welcome event
[12:58] Guthur that is the correct event
[12:59] guido_g ok
[12:59] Guthur they only added that date later, a shame
[12:59] guido_g Guthur: the event is quite a bit outside of hamburg city
[12:59] Guthur yeah, it's proofing bothersome on the accomodation front
[12:59] guido_g Guthur: so planning in advance is unavoidable for a good overall experience
[13:00] Guthur guido_g, is the public transport good in hamburg
[13:01] Guthur I've only been to Berlin, top rate there
[13:01] guido_g Guthur: it's ok
[13:01] guido_g Guthur: and friday night counts as weekend, so no problem
[13:01] guido_g Guthur: but harburg <-> city takes some time
[13:06] guido_g Guthur: for your bookmarks: http://www.hvv.de/en/index.php public transport in and around hamburg
[13:06] guido_g Guthur: if you've questions, feel free to ask
[13:06] Guthur cool, cheers guido_g
[13:14] pieterh sustrik: welcome back!
[13:14] pieterh Guthur: glad you got the signal handling working
[13:15] sustrik hi!
[13:15] sustrik Guthur: yust reading the signal thread
[13:15] sustrik have you solved the problem?
[13:16] pieterh sustrik: wrt multicast loop
[13:16] sustrik yes?
[13:16] pieterh do you really consider this part of the 0MQ API?
[13:17] sustrik it would definitely break the applications using it
[13:17] pieterh do we know of any using it?
[13:17] sustrik people ask about it once a while on the ml
[13:17] guido_g and here too
[13:17] pieterh ok, I'll restore it in 2.1 then
[13:18] sustrik ok
[13:18] sustrik as for the rate limit, i am not sure
[13:18] pieterh btw, we've tested Mikko's patch for OpenPGM 5.x integration, it appears to work perfectly
[13:18] sustrik great, send it to the ml
[13:18] pieterh so mikko should be able to send you a patch for 2.2
[13:18] pieterh yes
[13:19] sustrik i'll port it to master
[13:19] pieterh also btw, the contribution-to-release process is starting to work
[13:19] pieterh you saw I rewrote the contributing page because it wasn't fully accurate
[13:20] sustrik yes, nice
[13:21] pieterh it's based on walking people through the actual process...
[13:21] pieterh so we'll have releases for 2.0 and 2.1 in a few days
[13:22] sustrik goodo
[13:22] sustrik i though of actually moving to 3.0 afterwards
[13:22] sustrik the sooner it's done the less pain it will involve
[13:22] pieterh we should close 2.2 functionality first, no?
[13:22] pieterh or do we jump from 2.1 to 3.x?
[13:22] sustrik the latter
[13:22] pieterh I like it.
[13:23] pieterh however, note that people will be using 2.1.x for a long time
[13:23] pieterh maybe even several months... :-)
[13:23] sustrik that's why we have stable
[13:24] sustrik it's less contentious to break the backward compatibility now
[13:24] pieterh what I mean is, it's quite hard to backport fixes from 2.2 to 2.0, we should be careful about big internal changes for 3.0
[13:24] pieterh external API changes is perfectly fine
[13:24] sustrik the 2->3 shift is about API/ABI
[13:24] sustrik it would involve a lot of pain in bindings
[13:24] sustrik and a slighly less pain for end users
[13:24] pieterh shrug, we've been through that, it's moderate
[13:25] sustrik but it has to be done
[13:25] pieterh I mean, the pain seems to be tolerable
[13:25] pieterh so indeed the sooner the better
[13:25] sustrik exactly
[13:25] sustrik so i'll wait for stable 2.1 release
[13:25] sustrik and move on afterwards
[13:26] pieterh ... hmm, so...
[13:26] pieterh there'll be no 2.2 release then
[13:26] sustrik yes
[13:26] pieterh and the source repository should be called zeromq3 in that case
[13:26] pieterh rename/fork it's all the same
[13:27] pieterh pure rename means old links break, which you don't really want
[13:27] pieterh I'd keep the zeromq2, start a new zeromq3
[13:27] sustrik what with zeromq2 then?
[13:27] sustrik it won't be used at all
[13:27] pieterh well, true, it becomes irrelevant
[13:27] pieterh we have zeromq2-1, zeromq2-0, zeromq3
[13:27] pieterh nice enough
[13:28] pieterh I'd just like to keep the naming consistent with version numbers
[13:28] sustrik it will break build tools :(
[13:28] pieterh so later we can make zeromq3-0 for releases, etc.
[13:28] pieterh the build tools... mikko will take care of it
[13:28] sustrik but it's up to mikko, i assume
[13:29] pieterh mikko uses git submodules to pull in the zeromq repos for building
[13:29] pieterh quite neat
[13:29] pieterh hmm, if we stop using zeromq2 for development, I can change that README to act as an index
[13:29] pieterh that'll work nicely for people still using the old URL
[13:30] pieterh sustrik: I only have one concern with the stuff you discussed in London...
[13:31] pieterh having spent several weeks making reliability designs *on top* of 0MQ, for the Guide, I'm going to gnash my teeth when it all goes into the core and my work is for nothing...
[13:31] pieterh :-)
[13:32] sustrik index?
[13:32] sustrik i don't get it
[13:32] pieterh index = web page that sends people off to the correct gits
[13:33] pieterh table of contents
[13:33] sustrik don't make it part of any particular git
[13:33] pieterh dermoth|home: yesterday I asked to register the channel too
[13:34] pieterh but I don't know who the admins are...
[13:34] pieterh hello admins, are you there?
[13:34] sustrik admins of what?
[13:34] pieterh this channel
[13:34] sustrik dermoth: you've done the setting up iirc
[13:34] sustrik i think you said i am an admin back then
[13:34] pieterh sustrik, try /kick pieterh
[13:35] sustrik i have to log in first
[13:35] pieterh :-)
[13:35] sustrik a sec
[13:35] pieterh dermoth|home: anyhow, it'd be useful if anyone can change the topic IMO
[13:35] pieterh if and when we find spammers coming here we can lock it down
[13:37] pieterh two things
[13:37] pieterh a welcome message to new joiners explaining that the IRC channel is logged
[13:37] yrashk "You are under surveillance. Beware!" :)
[13:37] sustrik hm, "You're not channel operator"
[13:37] pieterh and secondly, either in the topic or the welcome message, something like "Before asking for advice here, Please Read the Guide"
[13:38] sustrik dermoth|home: any idea how can i log in as a channel operator?
[13:38] pieterh perhaps also a third warnings, "And ps., don't share sockets between threads, even if it seems excellent design, seriously, just don't do it!"
[13:38] sustrik :)
[13:38] yrashk sustrik: /msg chanserv op #zeromq
[13:38] yrashk may be add your name too
[13:38] yrashk dont remember
[13:38] pieterh yay!
[13:39] yrashk power trip!
[13:39] yrashk :D
[13:39] sustrik :D
[13:39] pieterh all bow before our sustrik overlords
[13:39] sustrik ok
[13:39] pieterh aight
[13:39] sustrik what now
[13:39] sustrik ?
[13:39] pieterh dermoth|home: http://travlr.github.com/zmqirclog/
[13:40] pieterh sustrik: shrug, it's enough for now we know dermoth|home and you are chanops
[13:40] sustrik ok
[13:40] pieterh heh :-) magic
[13:41] pieterh BTW if anyone wants short URLs for 0MQ projects, give me a shout
[13:41] pieterh The Guide is e.g. also at http://zero.mq/zg
[13:42] pieterh and I've just created http://zero.mq/irc -> http://travlr.github.com/zmqirclog/
[13:42] pieterh :-)
[13:44] pieterh sustrik: leaping to 3.0 is a great idea, I see San Francisco inspired you
[13:44] sustrik it calmed down my fear wrt the api breakage
[13:45] sustrik the people haven't seem to mind it that much
[13:45] sustrik although there were no binding maintainers there
[13:45] pieterh I think a key thing is to prove we have a healthy 2.1
[13:45] sustrik ack
[13:46] sustrik when there's a stable branch, the changes to master become somehow less threatening
[13:46] pieterh indeed
[13:47] sustrik Guthur: you there?
[13:47] pieterh btw, there's a really good article by Zed Shaw on coders vs. sysadmins: http://sheddingbikes.com/posts/1299555462.html
[13:47] pieterh very relevant to this discussion
[13:52] yrashk interesting to know what his ideas are that he has mentioned in the very end
[13:53] yrashk with Zed you can never predict what he comes up with next
[13:53] pieterh with 0MQ we split the traffic right up front, at the zeromq.org main page
[13:53] pieterh ah, yrashk, yes, you can... Zed will always create a new project :-)
[13:54] yrashk pieterh: well that, yes
[13:54] yrashk but what will be there...
[13:54] Guthur sustrik, back
[13:54] Guthur sorry was AFK there
[13:54] sustrik i've just read the EINTR thread
[13:54] Guthur As I binding author I'm actually in a small way looking forward to the jump to ZMQ3
[13:54] sustrik have you solved that already?
[13:55] Guthur it'll let me make some changes
[13:55] yrashk when zmq3 is due?
[13:55] sustrik when, 2.1 is stable
[13:55] yrashk I as a binding author is interested in it, too
[13:55] Guthur sustrik, For clrzmq2, yes, I just catch SYSINT and SYSTERM, and exit the program
[13:55] sustrik so, the bump is actually just an API/ABI change, so it can be done quickly
[13:55] sustrik basically in 1 go
[13:55] yrashk hey my binding ends with 2, too!
[13:56] pieterh Guthur: do you re-enter the poll call?
[13:56] Guthur pieterh, yeah
[13:56] Guthur I have a timer to track how long has elapsed
[13:56] pieterh so that worked... excellent...
[13:56] Guthur yep, all good
[13:56] Guthur pieterh, cheers for the suggestion
[13:56] pieterh np, happy to help
[13:57] sustrik yrashk: what's that, a collision in binding names?
[13:57] yrashk sustrik: nope
[13:57] sustrik ok, good
[13:57] yrashk sustrik: clrzmq2 and erlzmq2
[13:57] yrashk just both are -twos
[13:57] yrashk :D
[13:57] pieterh sustrik: just happy naming conventions
[13:57] sustrik as for 3.0 bump
[13:58] sustrik anybody who's interested in the bump, check the wiki page
[13:58] pieterh Guthur: yrashk: please do *not* bump that to x3 when 0MQ/3 comes out...
[13:58] Guthur oh
[13:58] yrashk is 3.0 api gonna be significantly different btw?
[13:58] sustrik not really
[13:58] pieterh just cleaner
[13:58] sustrik there are some shortcoming though
[13:58] sustrik yes
[13:58] Guthur pieterh, my rational is that I would like to clean up some API issues I perceive
[13:58] yrashk pieterh: sure. erlzmq2 is v2 to erlzmq only
[13:58] Guthur I can't remove them for backwards compatibility reasons
[13:59] pieterh Guthur: you can in fact make your own version jump any time you like
[13:59] sustrik i guess the C and Erlang APIs are not tightly connected
[13:59] sustrik exactly
[13:59] pieterh that's a contract between you and your users
[13:59] pieterh feel free to refer to the 0MQ contract as your guide
[13:59] pieterh 'release policy' is the page
[14:00] yrashk sustrik: they are not, but they are very close
[14:00] Guthur pieterh, I suppose I was being slightly sneaky and was going to spin it as a move to ZMQ3
[14:00] Guthur sorry, hehe
[14:00] pieterh sustrik: I found another case where one *has* to set linger on a socket or term freezes
[14:00] yrashk sustrik: I effectively imitate 0mq api more or less
[14:00] pieterh that's when there are pending outgoing connects
[14:00] yrashk sustrik: http://zeromq.github.com/erlzmq2/
[14:01] sustrik pieterh: yes
[14:01] pieterh that doesn't seem accurate semantics
[14:01] yrashk sustrik: they only addition is brecv which is a vm blocking receive (if somebody will ever need that)
[14:01] sustrik pieterh: 0mq has to connect first to be able to send pending messages
[14:02] pieterh hmm, true but IMO at termination time, pending connections should be killed
[14:02] pieterh well
[14:02] pieterh sigh
[14:02] pieterh indeed
[14:02] pieterh *sigh*
[14:02] pieterh it's all logical but my fear is lots and lots of people are going to stumble on this
[14:02] sustrik yrashk: however, the changes between 2 and 3 likely have no counterpart in Erlang API
[14:02] yrashk btw with erlang api, you can terminate the context and then start closing sockets in parallel :)
[14:02] pieterh sustrik: any chance we can revert zmq_term() to acting like 2.0, in 2.1 stable?
[14:03] sustrik pieterh: why would you want that?
[14:03] pieterh recall, we discussed it
[14:03] yrashk our zmq_term even has an optional timeout value
[14:03] pieterh two zmq_term methods, one blocking and one non-blocking
[14:03] pieterh blocking with timeout
[14:03] sustrik yes?
[14:04] pieterh old name, old semantics, compatible
[14:04] pieterh new name, new semantics
[14:04] pieterh in fact the 2.1 change broke the API IMO
[14:04] sustrik pieterh: ah, backward compatibility
[14:04] pieterh I know it was for good reasons but it broke the API
[14:04] pieterh and people are going to bitch when they move from 2.0 and stuff stops working
[14:04] sustrik yes, it would make sense to make linger default to 0
[14:04] pieterh yes
[14:05] sustrik but at the moment most people are using 2.1 with default of "infinite" :(
[14:05] pieterh 2.1 is not stable yet, it's rc2
[14:05] sustrik it was a fault not to preserve backward compatibility when moving to 2.1
[14:05] pieterh well, I did argue for it quite loudly iirc
[14:05] sustrik well, it's more about how many people you are going to hurt
[14:06] sustrik you did
[14:06] pieterh to be honest there are marginal areas we *should* improve but don't because of the API contract
[14:06] pieterh and here we have something that breaks a majority of code...
[14:06] sustrik yes
[14:06] sustrik that's why 3.0 should be done asap
[14:07] sustrik it was a fault back then
[14:07] pieterh that would help, but doesn't make 2.1 less dangerous IMO
[14:07] sustrik i am sorry for it
[14:07] pieterh sorry shmorry, can we fix this before it's too lare?
[14:07] pieterh *late
[14:07] sustrik it's already to late
[14:07] pieterh why?
[14:07] pieterh 2.1 is not stable
[14:08] sustrik majority has already migrated
[14:08] pieterh you have figures?
[14:08] pieterh to be honest I don't believe that
[14:08] pieterh besides, it's not the point
[14:08] pieterh we have a release policy, which 2.1 broke, and should unbreak
[14:08] sustrik that means second breaking
[14:09] sustrik one was enough
[14:09] pieterh otherwise the release policy is not credible and when you refer to it, I'll mock you bitterly
[14:09] pieterh "oh, it's fine when YOU want to break it..."...
[14:09] pieterh seriously... :-/
[14:09] sustrik the problem is that the contract was written after 2.1
[14:09] pieterh that is untrue
[14:09] pieterh you modified it afterwards, but it's from 2.0
[14:09] sustrik it was discussed because of 2.1
[14:09] pieterh I'll provide you the date it was written... hang on
[14:09] sustrik and introduced
[14:10] pieterh I wrote it on 4th August 2010
[14:10] sustrik yep
[14:10] pieterh it was written for 2.0 to be precise
[14:10] pieterh I flatly disagree that it did not apply, does not *now* apply to 2.1
[14:10] pieterh that is completely wrong
[14:11] sustrik i recall explicitly addressing 2.1 compatibility issues when writing it
[14:11] pieterh We wrote it in August
[14:11] pieterh your memory is ... curious
[14:11] sustrik ah, yes, i think thr 2.1 work started back then
[14:11] pieterh Check the history
[14:11] sustrik remember the great heats in BTS?
[14:12] pieterh please, sustrik, check the dates when you introduced the zmq_term blocking
[14:12] pieterh anyhow, this is irrelevant
[14:12] sustrik yes, it was at the same time
[14:12] pieterh you have a release policy and we are about to release a stable 2.1
[14:12] pieterh which breaks the release policy
[14:12] pieterh I'm formally against that
[14:12] pieterh and formally asking you to respect the documented policy
[14:13] sustrik i am against repeated breaking of API
[14:13] sustrik 2.1.1->2.1.2
[14:13] pieterh the API includes the semantics of zmq_term
[14:13] sustrik well, let's do a poll then
[14:13] pieterh blocking or non-blocking?
[14:13] pieterh yes, let's do a poll
[14:14] sustrik default linger of 0 vs. infinite
[14:14] pieterh indeed
[14:14] pieterh note, you can add zmq_term_blocking () which does infinite blocking with a timeout
[14:15] sustrik sure, but that's not the point
[14:15] pieterh yes it is
[14:15] pieterh if people absolutely need the semantics, they are present
[14:15] sustrik we are discussing backward comaptibility not new apis
[14:15] pieterh ok, setting LINGER gives the same results
[14:15] sustrik anyway, let's ask people
[14:15] yrashk I vote for keeping it as is
[14:15] yrashk :)
[14:15] yrashk if anybody's interested :)
[14:15] sustrik i am against :)
[14:15] pieterh yrashk: 'as is' depends where you are today
[14:15] yrashk master
[14:15] sustrik let me ask on ml
[14:16] pieterh could you please be specific: infinite blocking, or not
[14:16] yrashk infinite
[14:16] sustrik yes
[14:16] sustrik there are two sides to the problem
[14:16] sustrik i'll try to explain on the ml
[14:17] pieterh sustrik: please, do point out this is for the stable 2.1 to be released shortly
[14:17] pieterh and do explain that infinite blocking means all existing apps are likely to break
[14:17] pieterh when they migrate from 2.0 to 2.1
[14:17] sustrik 2.0 apps
[14:17] sustrik in the opposite case
[14:17] sustrik the 2.1 apps are going to break
[14:17] sustrik i'll explain that
[14:17] pieterh all the ones that don't explicitly set LINGER to zero, which is loads and loads
[14:18] pieterh (loads and loads take the effort to set LINGER to zero)
[14:18] pieterh yes, a poll will be interesting...
[14:19] pieterh BTW there's a line in the Release Policies page I am going to remove
[14:19] pieterh "In general we have a stable and an unstable releases of the product, e.g. 2.0.x and 2.1.x"
[14:19] pieterh that is just wrong
[14:21] pieterh bleh, the contract is vague for minor releases
[14:21] pieterh "It may even run"
[14:21] pieterh some lawyer has been editing that page...
[14:30] Guthur is the reactor pattern still ear marked for 3.0
[14:30] Guthur just thinking that if it was to replace Poll it might be a good opportunity to work IOCP into the windows version
[14:31] Guthur which would obviously immediately open up IPC
[14:32] pieterh Guthur: I don't think that's on the 3.0 roadmap
[14:32] Guthur it is according to the website
[14:32] Guthur http://www.zeromq.org/docs:3-0#toc3
[14:33] Guthur it does say replacing OR supplementing though
[14:33] pieterh indeed, but I don't think it's on the roadmap
[14:33] Guthur ok, is there a more up to date doc
[14:34] pieterh sorry to be unhelpful but the only accurate doc is the NEWS when a feature's been released
[14:34] pieterh and then the man page...
[15:29] zedas pieterh: or write a book :-)
[15:29] pieterh :-) yeah, that's always a good option
[15:31] sustrik 3.0 road map is out of date
[15:31] sustrik :(
[15:32] sustrik i would rather use the page to collect the API changes
[15:32] pieterh sustrik: ctrl-e (stuff) ctrl-s
[15:32] sustrik new features, otoh, will anyway be only those that people contribute
[15:32] sustrik right
[15:36] Skaag sustrik: you heard about Sector and UDT?
[15:36] sustrik i know UDT
[15:36] sustrik don't know about Sector
[15:37] Skaag that's the funny thing
[15:37] Skaag I thought about implementing a filesystem based on ZMQ
[15:37] Skaag and while looking for prior work, I found Sector...
[15:37] Skaag http://sector.sourceforge.net/
[15:37] Skaag I'm currently installing it and about to try it out
[15:37] Skaag on paper, it looks very interesting
[15:38] sustrik there are several distributed databases out there
[15:38] sustrik i am not an expert in the topic though
[15:38] Skaag this one offers a fuse client which makes it interesting
[15:38] Skaag the reason I mentioned it is UDT actually
[15:39] sustrik like in "uses UDT" fro communication between nodes, right?
[15:39] Skaag yes
[15:39] sustrik nice
[15:40] sustrik could work well for large-scale synchronisations over a WAN
[15:40] Skaag that's what I need to establish now
[15:40] Skaag I have at the moment machines in 7 ~ 8 different data centers
[15:41] Skaag by the way, zmq is working wonders for us
[15:41] Skaag we are graphing in real-time the data from 20+ nodes
[15:41] Skaag without the illnesses we were experiencing while using rabbitmq
[15:43] pieterh Skaag: we'd welcome a short story that we could publish on the wiki...
[15:43] pieterh about your experiences in moving to 0MQ
[15:43] Skaag ok
[15:44] sustrik Skaag: what's the project?
[15:45] Skaag it's a control panel for a CDN
[15:45] sustrik is there a link?
[15:45] Skaag I'm still making the site for it :-\
[15:45] sustrik i see
[15:45] sustrik when you have it, share the link
[15:45] Skaag will do
[15:45] sustrik it would be interesting to have a look
[16:17] CIA-103 zeromq2: 03Guido Goldstein 07master * rd0c8edd 10/ include/zmq.hpp :
[16:17] CIA-103 zeromq2: Added missing close method w/ check if socket is already closed.
[16:17] CIA-103 zeromq2: Signed-off-by: Guido Goldstein <github@a-nugget.de> - http://bit.ly/ggZZI6
[17:04] guido_g sustrik: thx for mentioning the sign-off thingie
[17:04] guido_g didn't move the .gitconfig to the new machine
[22:48] nadime any zmq gurus around to help me figure out a performance bottleneck issue?
[22:49] nadime i think it's kind of interesting!
[22:53] mikko nadime: what's the issue?
[22:54] nadime i have the following setup
[22:54] nadime push socket fans out from a thread into N (defaults to 4) IO threads using push/pull to distribute the work to the io threads
[22:55] nadime the io threads then talk 1 to 1 with N io threads on a client
[22:55] nadime the io threads use pub/sub since it's 1 to 1
[22:55] nadime each io thread has 1 zmq io thread
[22:55] nadime and i set affinities
[22:55] nadime to make it as quick as possible
[22:55] nadime the issue i'm having
[22:56] nadime is that i see about 500k msgs/sec go into my push socket
[22:56] nadime but i fall way behind on my 4 io sockets
[22:56] nadime sorry the io sockets are tcp
[22:56] nadime so i didn't expec tthem to keep up, but i figured with 4 and a pretty solid machine (12 cores, about 1 thread per core)
[22:56] nadime they'd be ok
[22:57] mikko are they tcp over local loopback?
[22:57] mikko or over network?
[22:57] nadime local 1 GB network
[22:57] mikko what is your message size?
[22:57] nadime avg prolly 50b
[22:57] nadime sorry B
[22:57] nadime bytes
[22:59] mikko so, 50b messages x 500k would be roughly 238 megabytes per second
[22:59] nadime yeah but i fall WAY behind
[22:59] nadime like the io threads barely do 80kmsg/s
[22:59] mikko one gigabit per second is 128 megabytes per second
[23:00] nadime 50b is probably wrong, i don't see anywhere near full bandwidth utilization
[23:01] nadime the point though is that what the io threads do is nowhere near even 20% capacity
[23:02] mikko are you seeing high utilisation anywhere?
[23:02] mikko or just falling rates?
[23:03] nadime no high network utilization usually around 15%
[23:03] mikko what os is this?
[23:03] nadime win7
[23:03] nadime 64b
[23:04] mikko hmm, can't help you much there
[23:04] mikko i have got very little experience with windows and performance
[23:04] nadime i'm not missing anything though in terms of things you'd do with zmq?
[23:05] mikko sounds ok
[23:05] mikko i would maybe use zmq_pair socket for 1:1
[23:05] nadime k, i can try that
[23:05] nadime is the push/pull fair queuing slow?
[23:05] mikko shouldn't be
[23:06] nadime k
[23:06] mikko it's very simple algorithm
[23:06] nadime i would think
[23:06] mikko if you have a profiler available that might help
[23:06] mikko to see where time is spent
[23:06] nadime yeah i do
[23:06] nadime i will use it
[23:06] nadime i just thught i might be missing something obvious that would help
[23:08] mikko so, do i understand your architecture correctly:
[23:09] mikko you have push/pull socket, which balances work to four threads
[23:09] mikko each one of the threads has pub socket which publishes messages over tcp to workers
[23:09] mikko ?
[23:14] mikko i gotta head to bed
[23:15] mikko nadime: the channel is usually more lively during the day time in europe (large amount of devs from eu)
[23:39] cremes nadime: i have a few suggestions for things to look at
[23:39] cremes 1. if your PUSH socket is in the same process as your "io thread PULL sockets" you should be using inproc transport instead of tcp
[23:40] cremes 2. take a look at your message serialization; sometimes deserialization is more costly than serialization
[23:40] cremes however, i assume you are sending packed structures so you are probably just casting a pointer to get to your data
[23:41] cremes 3. instrument your messages so that you can measure how many nano/micro/milliseconds it takes for the PUSH to deliver to the PULL socket
[23:42] cremes 4. instrument the messages that go from the "server io threads" to the "client io threads" via PUB/SUB to measure latency

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy