ZeroMq IRC Log

Sunday February 6, 2011

[Time] Name	Message
[10:00] mikko	good morning
[10:04] sustrik	morning
[10:10] mikko	neale is doing pretty good job
[10:11] mikko	we have s390x soon in build cluster
[10:16] mikko	im going to visit chinese new year and get back to hacking
[10:33] sustrik	great!
[11:51] Guthur	pieterh, Nice talk, the one you sent on the list
[11:51] pieterh	Hi Guthur :-)
[11:51] pieterh	It was very short, next time we'll try to get a longer slot
[11:51] Guthur	I'm going to try and get some of my colleagues to watch that, might help me convince them that OMQ fits our needs
[11:52] Guthur	a very good introduction though, even with the short time slot
[11:52] Guthur	I liked the E=mc2, hehe
[11:52] pieterh	It's a perspective I've used a few times in presentations, multithreading done wrong, and right
[11:53] Guthur	My team leader is pushing me to take that approach, the wrong way
[11:53] pieterh	It can be very hard to fight established ways of working...
[11:54] Guthur	I'm trying to convince him that multiple threads and global state without message passing is a problem waiting to happen
[11:54] pieterh	Well, it's a lot of pain and cost waiting to happen...
[11:54] pieterh	if the organization is large, it doesn't really care
[11:55] pieterh	But perhaps the argument that you can't stretch the results across multiple boxes may help
[11:55] pieterh	The best argument is usually running code
[11:56] Guthur	That's the thing I have it running with ZMQ, he's trying to argue against the dependency now
[11:56] pieterh	Hmm, what OS are you on?
[11:57] Guthur	windows, it's a .NET app
[11:57] Guthur	so he would rather it was nothing but .NET
[11:57] pieterh	Oh boy :-)
[11:57] pieterh	You can't argue logic with people who develop on Windows
[11:57] Guthur	hehe
[11:58] pieterh	I'm serious, they've swallowed the magic potion and believe in what they learn from MSDN
[11:58] Guthur	yeah it seems like that sometimes
[11:58] pieterh	And your organization has _no_ Linux?
[11:59] Guthur	oh yeah it has plenty
[11:59] Guthur	it's a very large American Bank
[11:59] pieterh	Oh boy
[11:59] pieterh	Well, performance maybe
[11:59] Guthur	Don't ask me the rationale for moving this app to windows, I still haven't got an answer to that
[12:00] pieterh	Banks don't use rationales
[12:00] pieterh	In my experience, anyhow, they often make IT choices based on egos and opinions
[12:00] Guthur	We are moving most client apps to windows (.NET), I can kind of see that as windows boxes are prevalent, but this app I am working on has windows through the whole stack
[12:01] Guthur	server included
[12:01] pieterh	So you can't use arguments like cost, portability, longeivity
[12:02] Guthur	no, but we do want scalability in our app, and there is many 'services' to talk to each other
[12:03] pieterh	So performance is one thing big egos are sensitive too
[12:03] pieterh	Especially if you can compare a pure .NET solution to other projects in the company
[12:03] pieterh	*sensitive to
[12:03] pieterh	If you can get a visible performance kick by using 0MQ, that's an argument that will fly IMO
[12:04] Guthur	I am hoping I can show that
[12:04] Guthur	and also with less code
[12:05] pieterh	Well, good luck and it'd be interesting to know how this plays out
[12:05] Guthur	cheers, yeah I hope I can win through
[12:06] Guthur	it's tiring fighting this mentality though
[12:06] pieterh	Personally, I'd look to change projects :-)
[12:06] Guthur	for the first time this week I started wanting to get a new job
[12:07] Guthur	hehe, yeah agree
[12:07] Guthur	this/last
[12:07] pieterh	It's not worth working with people who are mediocre, not worth working on projects that are mediocre
[12:07] pieterh	A terrible, tragic project... at least you learn a lot
[12:08] Guthur	maybe I will have more success next week
[12:08] Guthur	I have still to show the full working app
[12:08] Guthur	only got it ready on friday
[12:08] pieterh	I'd really focus purely on speed... nothing else
[12:09] sustrik	pieterh: the presentation should be definitely linked from zeromq.org
[12:09] pieterh	sustrik: the video or the slideshow?
[12:09] sustrik	video
[12:09] Guthur	I'd second that, its a good video
[12:10] sustrik	do they leave it on the site for a long time?
[12:10] pieterh	Guthur: especially if you can show an architecture that's hard to make with the pure .NET toolkit
[12:10] pieterh	sustrik: IMO it's there forever
[12:10] sustrik	then it should be easy to link...
[12:10] pieterh	I think it would be better to convert to youtube but I'm not sure how, will try on a Linux box
[12:11] sustrik	yes, it behaved kind of capricious
[12:11] sustrik	i was able to get it running after ~10 atemps
[12:11] pieterh	the HTML5/theora format is kind of strange
[12:12] pieterh	I'll give it a shot
[12:12] pieterh	I guess we already have a place on the wiki for this, i.e. the right hand side where we list the events
[12:12] Guthur	I had trouble accessing it with Chrome
[12:12] Guthur	worked on firefox though
[12:12] sustrik	i would say it will be valueable even after a year or so
[12:12] pieterh	Heh, from Slideshare, " "FOSDEM 2011 - 0MQ" is being tweeted more than anything else on SlideShare right now. So we've put it on the homepage of SlideShare.net (in the "Hot on Twitter" section)."
[12:13] sustrik	and the events list is going to rollover
[12:13] sustrik	something like "links" section, or "articles" or "media"
[12:13] pieterh	we can leave the events list to grow
[12:14] pieterh	eventually move it to its own page
[12:14] pieterh	makes sense that people use that list to find corresponding videos / slideshows
[12:15] sustrik	well, the problem is broader that that
[12:15] sustrik	for example, there are useful blogs out there
[12:15] Guthur	actually talking about performance of .NET vs ZMQ, I was testing a REQ/REP vs .NET remoting, both using TCP, and got some interesting results...
[12:15] sustrik	it's not clear where to link the from
[12:15] pieterh	sustrik:... Google kind of solves this problem, eventually
[12:16] Guthur	Up to about 100 requests ZMQ was faster or at least as performant, after that it started to fall behind remoting
[12:16] pieterh	I'm not sure collecting hundreds of links to articles and blogs makes sense
[12:16] pieterh	people want reusable material
[12:16] Guthur	I haven't investigated much further to identify the reasoning though, and my knowledge to inner works of remoting is limited
[12:16] pieterh	Guthur: interesting
[12:17] pieterh	0MQ on Windows is not optimized in any way
[12:17] Guthur	though I think remoting is now being replace by MS's new 'standard', WCF
[12:17] pieterh	aka WTF
[12:17] Guthur	are you familiar with it?
[12:17] pieterh	Nope... :-)
[12:18] Guthur	hehe
[12:18] pieterh	I maintain high levels of ignorance concerning anything coming from MSFT
[12:18] Guthur	nice for some, hehe
[12:19] pieterh	You know, what you might just do... is abandon the attempt to convert your team, which is most likely hopeless and will stress you
[12:19] pieterh	and instead take the opportunity to learn a lot about how WCF works, and then extend 0MQ to use that
[12:20] sustrik	pieterh: yes, i meant reusable links: your and oliver's video, nicolases' and ilya's articles etc.
[12:20] Guthur	unfortunately WCF only solves the interprocess connection
[12:21] pieterh	sustrik: we did collect the most valuable ones on the intro doc page, I think
[12:21] Guthur	there is nothing in .NET to solve inter thread, besides traditional thread mechanisms
[12:21] pieterh	well... i'm not against collecting resources but it does need upkeep
[12:22] pieterh	Guthur: right, and this is missing in 0MQ for Windows too
[12:23] Guthur	I wouldn't be surprised if WCF is .NET only, but I'm not 100% sure on that
[12:23] Guthur	Remoting was only intended for .NET endpoints
[12:27] pieterh	Let us know when you know :-)
[12:31] sustrik	why not link your talk from intro page then?
[12:40] pieterh	sustrik: could do... for now I've linked to it from the events list
[12:40] pieterh	let me try to convert that video first
[12:42] sustrik	sure
[12:42] sustrik	Guthur: getting slower after 100 messages -- isn't is just the queueing effect?
[12:43] sustrik	ie. publisher publishes faster than consumer consumes, thus making latencies gwo?
[12:43] sustrik	grow
[13:10] Guthur	sustrik, Possibly
[13:10] Guthur	if i remember right the majority of the time was in the Recv method
[13:11] Guthur	I have the code lying around somewhere, let me check
[13:38] pieterh	sustrik: I've uploaded to Youtube here: http://www.youtube.com/watch?v=CCBYzKfmQ4U
[13:38] pieterh	can you check if that works better?
[13:38] sustrik	yes
[13:38] sustrik	works with no problem
[13:40] Guthur	oh interesting, the tables turn drastically when I reuse the context/connection
[13:40] Guthur	OMQ starts to win hands down
[13:40] Guthur	0MQ
[13:42] pieterh	Guthur: you were opening a new connection each time?
[13:42] Guthur	I though it was fairer to remoting
[13:42] pieterh	huh
[13:42] Guthur	actually, there was another more important reason...
[13:43] Guthur	I was simulating a web application, which may not be able to reuse the connection
[13:43] pieterh	huh
[13:43] pieterh	you could add 'sleep(1)' into the main loop to properly simulate WCF
[13:44] pieterh	opening a socket each time is really slow
[13:45] Guthur	yep, I think remoting was probably keeping it's open in the background
[13:46] Guthur	but the target application was an IIS based web app, which would connect to a non IIS based process, and I was sure how well it would cope with persisting socket connections to external apps
[13:47] pieterh	is the target application performance critical?
[13:47] Guthur	we are hoping to ramp it up
[13:47] Guthur	its a FX Options trading app
[13:48] Guthur	so there could be lots of high frequency price movements
[13:48] pieterh	sure, so then you IMO want to make your 0MQ app as _fast_ as possible and use that as carrot
[13:48] pieterh	the stick would be to find a competing internal project and feed them juicy low latency technology
[13:48] Guthur	Well, reusing the context/connection, 0MQ is twice as fast
[13:49] pieterh	the code is in c#?
[13:49] Guthur	yeah
[13:49] pieterh	i'd probably make a C/Linux version just for sanity checking
[13:49] Guthur	though remoting does offer type safety
[13:50] pieterh	"safety" as in "ensure your types will never work on anything except our OS"
[13:50] pieterh	sustrik: OK, I've added the video to http://www.zeromq.org/intro:read-the-manual
[13:52] Guthur	getting them to accept 0MQ as the solution will be tough, getting them to dump .NET would be nigh on impossible
[13:52] pieterh	Guthur: oh, you'd just get blamed for the eventual failure
[13:52] pieterh	you have to leave technology evangelism to people outside the team
[13:54] sustrik	pieterh: thx
[13:54] pieterh	probably the best you can do is insert 0MQ as a "double the speed of this critical bus" solution
[13:54] pieterh	you would probably want to compare with another option too, to make it look fair
[13:55] pieterh	hmm, bank, hmm, for example RabbitMQ
[13:55] pieterh	and of course you need to make the architect think it was his idea all along
[13:55] Guthur	That's the thing the Team Lead is not really the architect, he is just too hands on
[13:56] Guthur	which is ok up to a point
[13:56] pieterh	well, s/architect/team lead/, it applies just the same
[13:56] pieterh	ego is ego
[14:49] Guthur	oops, i broke clrzmq2 build
[14:52] Guthur	I'll be pushing a fix promptly
[14:57] Guthur	is there no notification email when a build goes wrong?
[15:01] sustrik	Guthur: it's definitely possible
[15:02] sustrik	you have to ask mikko to add you to the list
[15:02] Guthur	I am one of the users registered, should I be receiving emails?
[15:02] sustrik	the auto builds are done once each 12hrs rather than immediately after commit btw
[15:02] sustrik	sorry, i am not a hudson expert
[15:03] sustrik	ask mikko
[15:03] Guthur	It is possible my over zealous email server is bouncing them
[15:03] Guthur	sustrik, do you receive mails?
[15:04] sustrik	nope, but i am not in the list afaik
[15:04] Guthur	ok
[15:04] Guthur	I can just monitor it after making a commit though
[15:04] sustrik	definitely
[15:05] sustrik	you have to wait till 5am or 5pm GMT though
[15:05] sustrik	that's when autoguilds are run
[15:05] sustrik	autobuilds*
[15:05] Guthur	ok, cheers
[15:57] Guthur	is Peter's video on the website yet?
[15:57] Guthur	oh sorry, found it
[15:58] Guthur	right in front of my eyes
[15:59] Guthur	would be really nice if it was on youtube or something though
[16:18] Guthur	I have multiple requests coming in, which will then make a further request to a service, this service will then reply back async on a separate thread, what is the appropriate 0MQ pattern to match the async response to the appropriate request thread?
[16:23] sustrik	req/rep
[16:24] Guthur	and match them using identity?
[16:25] sustrik	you don't have to care, rep socket does that for you
[16:28] Guthur	Either my understanding is wrong or it's bad problem description, because I can't see how that would work
[16:28] sustrik	rep sends the reply to the original requester
[16:29] Guthur	oh that part, yes
[16:29] Guthur	but its the async request from the service, it's not 0MQ
[16:29] Guthur	it's quickFIX to be precise
[16:29] sustrik	right
[16:30] sustrik	is there a reply-to field in the FIX message
[16:30] sustrik	?
[16:30] Guthur	and the reply will come back on a completely separate thread
[16:30] Guthur	there is reqID
[16:30] Guthur	that's it
[16:31] sustrik	who's behind the fix connection?
[16:31] sustrik	requester?
[16:31] sustrik	or replier?
[16:31] Guthur	replier
[16:31] Guthur	we are making Quote Requests
[16:31] sustrik	sure
[16:32] sustrik	reqID is an opaque correlation ID?
[16:32] Guthur	sorry, opaque correlation?
[16:33] sustrik	i.e. is it an ID used to pair the request and the reply?
[16:33] sustrik	opaque = can you put anything into ReqID?
[16:33] sustrik	or is it restricted to UUIDs or somesuch
[16:33] Guthur	we can place anything
[16:34] Guthur	and yes it matches request and reply
[16:34] sustrik	then what you have to do at FIX/0MQ boundary
[16:34] sustrik	is to get all the address message parts from 0mq request
[16:35] sustrik	and put them into FIX ReqID
[16:35] sustrik	when the reply arrives
[16:35] sustrik	you get all the message parts from thr ReqID
[16:35] sustrik	that would guarantee that the reply gets to the original requester
[16:36] Guthur	yeah that sounds like a nice way
[16:38] Guthur	cheers, my initial vision had an intermediary hop
[16:38] mikey_w	As a newbee is there a document that explains zeromq?
[16:38] Guthur	I never thought about trying to reply directly to the Req Socket
[16:38] Guthur	cheers sustrik
[16:38] Guthur	mikey_w, the guide, http://zguide.zeromq.org/chapter:all
[16:38] sustrik	what it is actually, is a 0mq-FIX bridge
[16:38] mikey_w	TU
[16:39] sustrik	so it's actually pretty generic and useful
[16:39] sustrik	think of making it an opensource
[16:40] Guthur	sustrik, Sure
[16:41] Guthur	that solution just might make all the complexity completely disappear, awesome
[17:17] mikko	good evening
[17:19] mikko	pieterh: how was fosdem?
[17:19] pieterh	hi mikko
[17:19] pieterh	short and sweet
[17:19] pieterh	there's a thread on hackernews: http://news.ycombinator.com/item?id=2185418
[17:20] pieterh	There's a video here: http://www.youtube.com/watch?v=CCBYzKfmQ4U
[17:20] pieterh	I think I successfully annoyed and intrigued various people :-)
[17:21] pieterh	anyone here interested in a command-line API for 0MQ?
[17:22] mikko	what do you mean by command line api?
[17:23] pieterh	export CONTEXT=`zmq context`
[17:24] pieterh	export SUBSOCK=`zmq socket $CONTEXT`
[17:24] pieterh	zmq connect $SUBSOCK randomaddress
[17:24] pieterh	etc.
[17:24] pieterh	I just had a rough idea of how to make it
[17:24] pieterh	could be fun for demos, especially
[17:25] mikko	i saw one guy making a small utility that reads from stdin and publishes that
[17:25] mikko	apart from that haven't seen much direct command line interaction
[17:25] pieterh	could it be useful?
[17:25] mikko	personally i dont see the immediate use-case but might be for some people
[17:26] pieterh	hmm, a resounding shrug, then :-)
[17:26] Guthur	hehe, of course Lisp would make such demos easy
[17:27] Guthur	the joys of having a REPL
[17:27] pieterh	problem with _any_ language is as soon as you go there you've lost 70% of your general audience
[17:28] Guthur	a REPL is such a cool feature I'm amazed more languages/environments don't offer it
[17:28] Guthur	i so miss having it
[17:29] pieterh	Read-Eval-Print loop, ok, right
[17:29] pieterh	Perl has it, and indeed it's useful
[17:29] Guthur	in Perl case the language completely negates any usefulness, hehe
[17:30] pieterh	Spoken like a true language bigot :-)
[17:30] Guthur	indeed, hehe
[17:33] Guthur	Interesting aside,
[17:33] Guthur	oops
[17:33] Guthur	what i meant to say was: an interesting side note is that Unix initially was suppose to come with a Lisp
[17:34] Guthur	C for the Kernel, Lisp for some sort of shell, I believe
[17:34] pieterh	hmm
[17:35] Guthur	Not Unix actually, GNU OS
[17:35] pieterh	ah, for sure, GNU OS had lots of weird and wonderful ideas :-)
[17:35] Guthur	Stallman like lisp obvious, look at Emacs
[17:37] pieterh	Lisp comes from MIT, where Stallman learned to hack
[22:42] devon_hillard	Is it correct to describe zeroMQ as a message-passing architecture?
[23:16] Guthur	I think i would go more with 'allows for a message passing architecture'