Sunday February 6, 2011

[Time] NameMessage
[10:00] mikko good morning
[10:04] sustrik morning
[10:10] mikko neale is doing pretty good job
[10:11] mikko we have s390x soon in build cluster
[10:16] mikko im going to visit chinese new year and get back to hacking
[10:33] sustrik great!
[11:51] Guthur pieterh, Nice talk, the one you sent on the list
[11:51] pieterh Hi Guthur :-)
[11:51] pieterh It was very short, next time we'll try to get a longer slot
[11:51] Guthur I'm going to try and get some of my colleagues to watch that, might help me convince them that OMQ fits our needs
[11:52] Guthur a very good introduction though, even with the short time slot
[11:52] Guthur I liked the E=mc2, hehe
[11:52] pieterh It's a perspective I've used a few times in presentations, multithreading done wrong, and right
[11:53] Guthur My team leader is pushing me to take that approach, the wrong way
[11:53] pieterh It can be very hard to fight established ways of working...
[11:54] Guthur I'm trying to convince him that multiple threads and global state without message passing is a problem waiting to happen
[11:54] pieterh Well, it's a lot of pain and cost waiting to happen...
[11:54] pieterh if the organization is large, it doesn't really care
[11:55] pieterh But perhaps the argument that you can't stretch the results across multiple boxes may help
[11:55] pieterh The best argument is usually running code
[11:56] Guthur That's the thing I have it running with ZMQ, he's trying to argue against the dependency now
[11:56] pieterh Hmm, what OS are you on?
[11:57] Guthur windows, it's a .NET app
[11:57] Guthur so he would rather it was nothing but .NET
[11:57] pieterh Oh boy :-)
[11:57] pieterh You can't argue logic with people who develop on Windows
[11:57] Guthur hehe
[11:58] pieterh I'm serious, they've swallowed the magic potion and believe in what they learn from MSDN
[11:58] Guthur yeah it seems like that sometimes
[11:58] pieterh And your organization has _no_ Linux?
[11:59] Guthur oh yeah it has plenty
[11:59] Guthur it's a very large American Bank
[11:59] pieterh Oh boy
[11:59] pieterh Well, performance maybe
[11:59] Guthur Don't ask me the rationale for moving this app to windows, I still haven't got an answer to that
[12:00] pieterh Banks don't use rationales
[12:00] pieterh In my experience, anyhow, they often make IT choices based on egos and opinions
[12:00] Guthur We are moving most client apps to windows (.NET), I can kind of see that as windows boxes are prevalent, but this app I am working on has windows through the whole stack
[12:01] Guthur server included
[12:01] pieterh So you can't use arguments like cost, portability, longeivity
[12:02] Guthur no, but we do want scalability in our app, and there is many 'services' to talk to each other
[12:03] pieterh So performance is one thing big egos are sensitive too
[12:03] pieterh Especially if you can compare a pure .NET solution to other projects in the company
[12:03] pieterh *sensitive to
[12:03] pieterh If you can get a visible performance kick by using 0MQ, that's an argument that will fly IMO
[12:04] Guthur I am hoping I can show that
[12:04] Guthur and also with less code
[12:05] pieterh Well, good luck and it'd be interesting to know how this plays out
[12:05] Guthur cheers, yeah I hope I can win through
[12:06] Guthur it's tiring fighting this mentality though
[12:06] pieterh Personally, I'd look to change projects :-)
[12:06] Guthur for the first time this week I started wanting to get a new job
[12:07] Guthur hehe, yeah agree
[12:07] Guthur this/last
[12:07] pieterh It's not worth working with people who are mediocre, not worth working on projects that are mediocre
[12:07] pieterh A terrible, tragic project... at least you learn a lot
[12:08] Guthur maybe I will have more success next week
[12:08] Guthur I have still to show the full working app
[12:08] Guthur only got it ready on friday
[12:08] pieterh I'd really focus purely on speed... nothing else
[12:09] sustrik pieterh: the presentation should be definitely linked from
[12:09] pieterh sustrik: the video or the slideshow?
[12:09] sustrik video
[12:09] Guthur I'd second that, its a good video
[12:10] sustrik do they leave it on the site for a long time?
[12:10] pieterh Guthur: especially if you can show an architecture that's hard to make with the pure .NET toolkit
[12:10] pieterh sustrik: IMO it's there forever
[12:10] sustrik then it should be easy to link...
[12:10] pieterh I think it would be better to convert to youtube but I'm not sure how, will try on a Linux box
[12:11] sustrik yes, it behaved kind of capricious
[12:11] sustrik i was able to get it running after ~10 atemps
[12:11] pieterh the HTML5/theora format is kind of strange
[12:12] pieterh I'll give it a shot
[12:12] pieterh I guess we already have a place on the wiki for this, i.e. the right hand side where we list the events
[12:12] Guthur I had trouble accessing it with Chrome
[12:12] Guthur worked on firefox though
[12:12] sustrik i would say it will be valueable even after a year or so
[12:12] pieterh Heh, from Slideshare, " "FOSDEM 2011 - 0MQ" is being tweeted more than anything else on SlideShare right now. So we've put it on the homepage of (in the "Hot on Twitter" section)."
[12:13] sustrik and the events list is going to rollover
[12:13] sustrik something like "links" section, or "articles" or "media"
[12:13] pieterh we can leave the events list to grow
[12:14] pieterh eventually move it to its own page
[12:14] pieterh makes sense that people use that list to find corresponding videos / slideshows
[12:15] sustrik well, the problem is broader that that
[12:15] sustrik for example, there are useful blogs out there
[12:15] Guthur actually talking about performance of .NET vs ZMQ, I was testing a REQ/REP vs .NET remoting, both using TCP, and got some interesting results...
[12:15] sustrik it's not clear where to link the from
[12:15] pieterh sustrik:... Google kind of solves this problem, eventually
[12:16] Guthur Up to about 100 requests ZMQ was faster or at least as performant, after that it started to fall behind remoting
[12:16] pieterh I'm not sure collecting hundreds of links to articles and blogs makes sense
[12:16] pieterh people want reusable material
[12:16] Guthur I haven't investigated much further to identify the reasoning though, and my knowledge to inner works of remoting is limited
[12:16] pieterh Guthur: interesting
[12:17] pieterh 0MQ on Windows is not optimized in any way
[12:17] Guthur though I think remoting is now being replace by MS's new 'standard', WCF
[12:17] pieterh aka WTF
[12:17] Guthur are you familiar with it?
[12:17] pieterh Nope... :-)
[12:18] Guthur hehe
[12:18] pieterh I maintain high levels of ignorance concerning anything coming from MSFT
[12:18] Guthur nice for some, hehe
[12:19] pieterh You know, what you might just do... is abandon the attempt to convert your team, which is most likely hopeless and will stress you
[12:19] pieterh and instead take the opportunity to learn a lot about how WCF works, and then extend 0MQ to use that
[12:20] sustrik pieterh: yes, i meant reusable links: your and oliver's video, nicolases' and ilya's articles etc.
[12:20] Guthur unfortunately WCF only solves the interprocess connection
[12:21] pieterh sustrik: we did collect the most valuable ones on the intro doc page, I think
[12:21] Guthur there is nothing in .NET to solve inter thread, besides traditional thread mechanisms
[12:21] pieterh well... i'm not against collecting resources but it does need upkeep
[12:22] pieterh Guthur: right, and this is missing in 0MQ for Windows too
[12:23] Guthur I wouldn't be surprised if WCF is .NET only, but I'm not 100% sure on that
[12:23] Guthur Remoting was only intended for .NET endpoints
[12:27] pieterh Let us know when you know :-)
[12:31] sustrik why not link your talk from intro page then?
[12:40] pieterh sustrik: could do... for now I've linked to it from the events list
[12:40] pieterh let me try to convert that video first
[12:42] sustrik sure
[12:42] sustrik Guthur: getting slower after 100 messages -- isn't is just the queueing effect?
[12:43] sustrik ie. publisher publishes faster than consumer consumes, thus making latencies gwo?
[12:43] sustrik grow
[13:10] Guthur sustrik, Possibly
[13:10] Guthur if i remember right the majority of the time was in the Recv method
[13:11] Guthur I have the code lying around somewhere, let me check
[13:38] pieterh sustrik: I've uploaded to Youtube here:
[13:38] pieterh can you check if that works better?
[13:38] sustrik yes
[13:38] sustrik works with no problem
[13:40] Guthur oh interesting, the tables turn drastically when I reuse the context/connection
[13:40] Guthur OMQ starts to win hands down
[13:40] Guthur 0MQ
[13:42] pieterh Guthur: you were opening a new connection each time?
[13:42] Guthur I though it was fairer to remoting
[13:42] pieterh huh
[13:42] Guthur actually, there was another more important reason...
[13:43] Guthur I was simulating a web application, which may not be able to reuse the connection
[13:43] pieterh huh
[13:43] pieterh you could add 'sleep(1)' into the main loop to properly simulate WCF
[13:44] pieterh opening a socket each time is really slow
[13:45] Guthur yep, I think remoting was probably keeping it's open in the background
[13:46] Guthur but the target application was an IIS based web app, which would connect to a non IIS based process, and I was sure how well it would cope with persisting socket connections to external apps
[13:47] pieterh is the target application performance critical?
[13:47] Guthur we are hoping to ramp it up
[13:47] Guthur its a FX Options trading app
[13:48] Guthur so there could be lots of high frequency price movements
[13:48] pieterh sure, so then you IMO want to make your 0MQ app as _fast_ as possible and use that as carrot
[13:48] pieterh the stick would be to find a competing internal project and feed them juicy low latency technology
[13:48] Guthur Well, reusing the context/connection, 0MQ is twice as fast
[13:49] pieterh the code is in c#?
[13:49] Guthur yeah
[13:49] pieterh i'd probably make a C/Linux version just for sanity checking
[13:49] Guthur though remoting does offer type safety
[13:50] pieterh "safety" as in "ensure your types will never work on anything except our OS"
[13:50] pieterh sustrik: OK, I've added the video to
[13:52] Guthur getting them to accept 0MQ as the solution will be tough, getting them to dump .NET would be nigh on impossible
[13:52] pieterh Guthur: oh, you'd just get blamed for the eventual failure
[13:52] pieterh you have to leave technology evangelism to people outside the team
[13:54] sustrik pieterh: thx
[13:54] pieterh probably the best you can do is insert 0MQ as a "double the speed of this critical bus" solution
[13:54] pieterh you would probably want to compare with another option too, to make it look fair
[13:55] pieterh hmm, bank, hmm, for example RabbitMQ
[13:55] pieterh and of course you need to make the architect think it was his idea all along
[13:55] Guthur That's the thing the Team Lead is not really the architect, he is just too hands on
[13:56] Guthur which is ok up to a point
[13:56] pieterh well, s/architect/team lead/, it applies just the same
[13:56] pieterh ego is ego
[14:49] Guthur oops, i broke clrzmq2 build
[14:52] Guthur I'll be pushing a fix promptly
[14:57] Guthur is there no notification email when a build goes wrong?
[15:01] sustrik Guthur: it's definitely possible
[15:02] sustrik you have to ask mikko to add you to the list
[15:02] Guthur I am one of the users registered, should I be receiving emails?
[15:02] sustrik the auto builds are done once each 12hrs rather than immediately after commit btw
[15:02] sustrik sorry, i am not a hudson expert
[15:03] sustrik ask mikko
[15:03] Guthur It is possible my over zealous email server is bouncing them
[15:03] Guthur sustrik, do you receive mails?
[15:04] sustrik nope, but i am not in the list afaik
[15:04] Guthur ok
[15:04] Guthur I can just monitor it after making a commit though
[15:04] sustrik definitely
[15:05] sustrik you have to wait till 5am or 5pm GMT though
[15:05] sustrik that's when autoguilds are run
[15:05] sustrik autobuilds*
[15:05] Guthur ok, cheers
[15:57] Guthur is Peter's video on the website yet?
[15:57] Guthur oh sorry, found it
[15:58] Guthur right in front of my eyes
[15:59] Guthur would be really nice if it was on youtube or something though
[16:18] Guthur I have multiple requests coming in, which will then make a further request to a service, this service will then reply back async on a separate thread, what is the appropriate 0MQ pattern to match the async response to the appropriate request thread?
[16:23] sustrik req/rep
[16:24] Guthur and match them using identity?
[16:25] sustrik you don't have to care, rep socket does that for you
[16:28] Guthur Either my understanding is wrong or it's bad problem description, because I can't see how that would work
[16:28] sustrik rep sends the reply to the original requester
[16:29] Guthur oh that part, yes
[16:29] Guthur but its the async request from the service, it's not 0MQ
[16:29] Guthur it's quickFIX to be precise
[16:29] sustrik right
[16:30] sustrik is there a reply-to field in the FIX message
[16:30] sustrik ?
[16:30] Guthur and the reply will come back on a completely separate thread
[16:30] Guthur there is reqID
[16:30] Guthur that's it
[16:31] sustrik who's behind the fix connection?
[16:31] sustrik requester?
[16:31] sustrik or replier?
[16:31] Guthur replier
[16:31] Guthur we are making Quote Requests
[16:31] sustrik sure
[16:32] sustrik reqID is an opaque correlation ID?
[16:32] Guthur sorry, opaque correlation?
[16:33] sustrik i.e. is it an ID used to pair the request and the reply?
[16:33] sustrik opaque = can you put anything into ReqID?
[16:33] sustrik or is it restricted to UUIDs or somesuch
[16:33] Guthur we can place anything
[16:34] Guthur and yes it matches request and reply
[16:34] sustrik then what you have to do at FIX/0MQ boundary
[16:34] sustrik is to get all the address message parts from 0mq request
[16:35] sustrik and put them into FIX ReqID
[16:35] sustrik when the reply arrives
[16:35] sustrik you get all the message parts from thr ReqID
[16:35] sustrik that would guarantee that the reply gets to the original requester
[16:36] Guthur yeah that sounds like a nice way
[16:38] Guthur cheers, my initial vision had an intermediary hop
[16:38] mikey_w As a newbee is there a document that explains zeromq?
[16:38] Guthur I never thought about trying to reply directly to the Req Socket
[16:38] Guthur cheers sustrik
[16:38] Guthur mikey_w, the guide,
[16:38] sustrik what it is actually, is a 0mq-FIX bridge
[16:38] mikey_w TU
[16:39] sustrik so it's actually pretty generic and useful
[16:39] sustrik think of making it an opensource
[16:40] Guthur sustrik, Sure
[16:41] Guthur that solution just might make all the complexity completely disappear, awesome
[17:17] mikko good evening
[17:19] mikko pieterh: how was fosdem?
[17:19] pieterh hi mikko
[17:19] pieterh short and sweet
[17:19] pieterh there's a thread on hackernews:
[17:20] pieterh There's a video here:
[17:20] pieterh I think I successfully annoyed and intrigued various people :-)
[17:21] pieterh anyone here interested in a command-line API for 0MQ?
[17:22] mikko what do you mean by command line api?
[17:23] pieterh export CONTEXT=`zmq context`
[17:24] pieterh export SUBSOCK=`zmq socket $CONTEXT`
[17:24] pieterh zmq connect $SUBSOCK randomaddress
[17:24] pieterh etc.
[17:24] pieterh I just had a rough idea of how to make it
[17:24] pieterh could be fun for demos, especially
[17:25] mikko i saw one guy making a small utility that reads from stdin and publishes that
[17:25] mikko apart from that haven't seen much direct command line interaction
[17:25] pieterh could it be useful?
[17:25] mikko personally i dont see the immediate use-case but might be for some people
[17:26] pieterh hmm, a resounding shrug, then :-)
[17:26] Guthur hehe, of course Lisp would make such demos easy
[17:27] Guthur the joys of having a REPL
[17:27] pieterh problem with _any_ language is as soon as you go there you've lost 70% of your general audience
[17:28] Guthur a REPL is such a cool feature I'm amazed more languages/environments don't offer it
[17:28] Guthur i so miss having it
[17:29] pieterh Read-Eval-Print loop, ok, right
[17:29] pieterh Perl has it, and indeed it's useful
[17:29] Guthur in Perl case the language completely negates any usefulness, hehe
[17:30] pieterh Spoken like a true language bigot :-)
[17:30] Guthur indeed, hehe
[17:33] Guthur Interesting aside,
[17:33] Guthur oops
[17:33] Guthur what i meant to say was: an interesting side note is that Unix initially was suppose to come with a Lisp
[17:34] Guthur C for the Kernel, Lisp for some sort of shell, I believe
[17:34] pieterh hmm
[17:35] Guthur Not Unix actually, GNU OS
[17:35] pieterh ah, for sure, GNU OS had lots of weird and wonderful ideas :-)
[17:35] Guthur Stallman like lisp obvious, look at Emacs
[17:37] pieterh Lisp comes from MIT, where Stallman learned to hack
[22:42] devon_hillard Is it correct to describe zeroMQ as a message-passing architecture?
[23:16] Guthur I think i would go more with 'allows for a message passing architecture'