ZeroMq IRC Log

Wednesday December 22, 2010

[Time] Name	Message
[07:14] ytang	hi, anyone online?
[07:33] sustrik	hi
[07:38] Steve-o	good morning
[08:05] mindful	anybody in
[08:08] sustrik	yup
[10:29] mikko	good morning
[10:29] mikko	i made it
[10:39] Skaag	made what?
[10:39] Skaag	to work? :)
[12:29] mikko	Skaag: made it home for christmas
[12:29] mikko	Skaag: i was flying from london to finland
[12:30] mikko	took five days
[12:31] Skaag	wow!
[12:31] Skaag	why 5 days?!
[12:31] Skaag	I thought 5 days is with a ferry
[12:31] Skaag	London to Stockholm, then Stockholm to Helsinki
[12:32] Skaag	brb
[12:34] mikko	Skaag: tons of canceled flights
[12:34] mikko	i was supposed to fly last friday and finally managed to fly tuesday evening
[12:35] Skaag	sounds like a nightmare
[12:40] Skaag	back.
[12:40] Skaag	I really should bring my bouncer into freenode as well...
[15:55] dos000	howdy
[15:55] dos000	can someone tell me if concurrent lock free circular buffers are used in zmq ? i read something recently that was talking about the perf improvements
[16:43] cremes	dos000: 0mq definitely uses lock-free algos; i don't know if they protect a circular buffer though
[16:43] cremes	the code answers all!
[16:43] Skaag	lol
[16:43] Skaag	snicker typical developer answer ;)
[16:44] Skaag	what documentation? look at the comments in the code... ;)
[16:44] cremes	yep ;)
[16:45] cremes	docs usually don't cover those kinds of implementation details anyway
[16:45] cremes	they are intended to remain hidden under the covers; anyone using the api shouldn't care what the algo is
[16:46] cremes	since dos000 was asking specifically about it, the only way to get the real answer is to 1) ask sustrik
[16:46] cremes	or 2) check the source
[16:46] dos000	ah!
[16:48] dos000	i was also looking for some experience in other libs offering lock-free circular buffers
[16:48] dos000	this stuff seems like black magic
[16:48] cremes	yes, very dark :)
[16:49] dos000	no one is doing it or talking about their implementation
[16:49] dos000	and no central de facto to go fo this .. evryone rolls their own
[16:49] cremes	true
[16:49] cremes	the main idea behind lock-free is to leverage a platform's atomic compare-and-swap operation
[16:50] cremes	if your hw supports that, a lock-free impl is fairly "easy" to create
[16:50] dos000	yes .. and i am hoping (please) support it
[16:50] cremes	the hard part then becomes dealing with cache coherency and its perf implications
[16:50] dos000	i am hoping 0mq has it for sparc and x86 and/or x86_64
[16:51] cremes	it definitely has it for x86 and x86_64; not sure on sparc
[16:51] dos000	cremes: zmq does not deal with threads correct ? i was thinking a way of "forcing" a thread to run on a particular core
[16:52] cremes	dos000: not correct; you can force cpu affinity for sockets
[16:52] cremes	check the setsockopt docs on the web site
[16:52] dos000	ok
[16:52] cremes	(i haven't used it myself otherwise i'd be more helpful...)
[16:52] dos000	here is the (interesting) talk http://www.infoq.com/presentations/LMAX
[16:53] cremes	ah, i just saw that this morning! it's in my queue to watch a bit later
[16:53] dos000	ah!
[16:53] cremes	i do trading stuff but i'm not in the super high-freq space
[16:54] dos000	they mention someway of sending a packet straight to user space bypassing (virtually) the kernel ... but they dont explain how or which os supports it
[16:55] dos000	very interestng talk indeed
[16:55] dos000	cremes, which blog/sites do you follow for this types of talks ?
[16:56] dos000	how did you stumble on the LMAX presentation ?
[16:56] cremes	i subscribe to reddit rss feeds as well as the infoq rss stuff
[16:58] dos000	the solution they propose starts from 36:00 if you want to skip through .. a picture is worth a thousand words
[16:58] cremes	cool, thanks for the heads up
[16:58] cremes	are you involved with trading systems or similar?
[16:58] dos000	yes very similar .. payment systems
[16:59] Skaag	dos000: don't the boost libraries do this type of locking?
[17:00] dos000	i asked already in #C++ it might be only circular buffers .. not lock free implementations
[17:01] Skaag	weren't all such kernel modules that broke the core -> user space membrane been kicked from the kernel project by Linus?
[17:02] dos000	no idea!
[17:02] Skaag	they are doing this without a custom kernel module?
[17:02] dos000	not sure how .. its just a blimp in the video
[17:40] dos000	ok ... now i found this http://www.puppetmastertrading.com/blog/2010/02/16/lock-free/
[17:52] mikko	dos000: http://www.zeromq.org/blog:multithreading-magic
[17:52] mikko	this might be interesting read as well
[17:52] mikko	not directly related
[17:58] dos000	fastflow seems to be up there with 0mq .. lots of similarities
[18:03] cremes	dos000: they diss ruby in that blog so now i need to go on a rant! ;)
[18:08] Skaag	lol!!!
[18:08] dos000	lol
[20:11] Skaag	the developer I work with tells me he managed to publish and consume a message on a topic with rabbitmq, but it's still unclear to him how to cluster the thing ;)
[21:41] mikko	Skaag: you can publish and consume a message with zeromq within minutes
[21:41] mikko	:)
[21:42] Skaag	I know
[21:42] Skaag	anyway, what I am thinking to answer him, is that Zookeeper will be in charge of managing a list of IP addresses of available hosts
[21:42] Skaag	and he will just connect to all of them and subscribe
[21:43] mikko	did you test zookeeper
[21:43] mikko	?
[21:43] Skaag	not yet
[21:43] Skaag	I'm installing it as we speak
[21:44] mikko	the only annoyance i found was that zookeeper didnt have rhel packaging
[21:44] mikko	i think i made spec-files back then
[21:45] Skaag	well I am running on Ubuntu myself. I can't stand rhel.
[21:45] Skaag	I used to be very redhat oriented
[21:45] Skaag	but once I was exposed to debian, and then to ubuntu, I knew my destiny was sealed ;)
[21:46] mikko	i use debian on my own stuff
[21:46] Skaag	I know it may sound stupid, and rhel is considered a kind of 'standard'.
[21:46] mikko	but many companies i work with prefer rhel
[21:46] dos000	i hate rhel .. anything non debian (based) for that matter
[21:47] dos000	apt-get is sweet
[21:47] mikko	rhel packaging is far more pleasant to make imho
[21:47] mikko	compared to debs
[21:47] Skaag	i disagree
[21:47] dos000	to use however isa different thing
[21:47] Skaag	I make a deb in 3 minutes.
[21:47] Skaag	less...
[21:48] Skaag	well I have a ready skeleton, from the first 5 minutes I spent trying to make a deb
[21:48] dos000	there are some utilities that will create the pkgs for you .. based on make && make installl
[21:48] Skaag	so now to make a deb is a matter of copying the files to the right places, specifying dependencies, and perhaps adding some install / uninstall scripts...
[21:48] Skaag	not to mention how easy it is to create a repository
[21:49] mikko	Skaag: it's easy for yum as well
[21:49] mikko	createrepo /path/to
[21:49] dos000	i did not find companies that do support for debian so far
[21:50] dos000	something that passes the CEO test
[21:51] Skaag	the CEO test is different in every company...
[21:51] Skaag	in some, the money talks
[21:51] Skaag	in some others, it's how large the support company is
[21:51] Skaag	and in some, it's how technically knowledgable :-)
[21:51] dos000	in some companies the CEO (and evryone around him) is dumb !
[21:51] dos000	thats the problem
[21:51] Skaag	and here in Russia! It's about how much money you slip into the CEO's pocket, behind the back of the board of directors...! ;-)
[21:52] dos000	heh!
[21:53] mikko	im not far from russia atm
[21:53] mikko	a few hundred km if even that
[21:54] mikko	Skaag: https://github.com/mkoppanen/myservices
[21:54] mikko	it's not very well tested but should give some ideas for zk integration
[21:54] Skaag	mikko: I read your page already ;)
[21:54] Skaag	ah, that's another page
[21:54] Skaag	sorry
[21:54] Skaag	Where in Finland are you?
[21:55] Skaag	I took the bus from Helsinki to St. Petersburg, just 6 month ago :-)
[21:55] Skaag	17 euros and I was in Russia!
[21:56] mikko	they got a new fast train as well
[21:56] mikko	three hours to st petersburg from helsinki
[21:56] Skaag	amazing
[21:56] dos000	lets say i had 3 packets coming from 3 different sources ... and all packets are sequentially numbered.
[21:56] Skaag	but I doubt in such a weather condition?
[21:57] dos000	what would you do to wait for three packets with the same seq numbers ?
[21:57] mikko	Skaag: even in such conditions
[21:57] Skaag	amazing!
[21:57] mikko	things dont stop in .fi even when its cold
[21:57] Skaag	I know
[21:57] Skaag	but on the russian side...?
[21:57] Skaag	:-)
[21:57] Skaag	I lived in Helsinki for 5 month
[21:57] Skaag	I absolutely love Finland
[21:57] Skaag	awesome country.
[21:57] Skaag	I loved it since childhood.
[21:58] Skaag	and recently, after living there, I understood why.
[21:58] mikko	i dislike the winter. thats why i moved to london
[21:59] Skaag	maybe because you grew up with it.
[21:59] Skaag	this is my second REAL winter here in Moscow, first time in Helsinki, where it was almost constantly under -10Ëc
[21:59] Skaag	and I don't mind it much.
[22:01] mikko	im in middle .fi now
[22:01] mikko	-30 outside
[22:02] mikko	does your company operate in russia?
[22:03] Skaag	yes
[22:03] Skaag	among other places
[22:03] Skaag	middle = near SodankylÃ¤?
[22:03] mikko	thats the northest place in finland
[22:04] mikko	close to the sea up north
[22:04] mikko	pretty close to JyvÃ¤skylÃ¤
[22:05] Skaag	cool
[22:05] Skaag	I have a good friend in Mikkeli
[22:07] Skaag	I actually passed through JyvÃ¤skylÃ¤ one time
[22:09] Skaag	they just prepared OlutglÃ¶gi
[22:09] Skaag	http://mika.letonsaari.net/foobar/olutglÃ¶gi/14.jpg
[22:10] Skaag	some part of me is glad I can not taste it.
[22:11] mikko	warm beer and spices?
[22:11] Skaag	actually it looks interesting
[22:13] Skaag	yes
[22:59] ryandmonk	hello... curious if someone can help me out with at pyzmq issue
[22:59] ryandmonk	i followed the instructions on site
[22:59] ryandmonk	and ran python setup.py install
[23:00] ryandmonk	its spits out some stuff.. but eventually gives an error saying "zmq/core/constants.c:1: error: #error Do not use this file, it is the result of a failed Cython compilation."
[23:01] ryandmonk	im on ubuntu 10.10.. using zeromq-2.0.10
[23:10] cremes	ryandmonk: try searching the ML archives; i have seen discussions on python binding problems there before
[23:11] cremes	perhaps the solution is buried in the archives
[23:12] ryandmonk	where are these archives?
[23:13] Skaag	cool - I have a working 16 node zookeeper cluster :)
[23:18] ryandmonk	found em.. thanks cremes!