Friday October 1, 2010

[Time] NameMessage
[04:20] twittard can I use zeromq to talk to applications with a proprietary message structure
[04:20] twittard ?
[04:23] guido_g what does that mean?
[04:34] guido_g *shrug*
[04:38] Pe_Ell I'm hoping from the nickname that it was a joke question....
[04:38] twittard heh
[04:38] twittard I just used the wrong lingo. Networking is not my forte.
[04:38] Pe_Ell you want to embed the 0mq socket into your application?
[04:39] twittard Basically, pretend you went "TCPSocket.connect" and it magically opened a connection for you. Then you call ".send" and send it some series of bytes that you cobbled together.
[04:39] twittard Pe_Ell: well, yeah. I'm currently bargaining with the notion that I might have to write my own networking code.
[04:40] Pe_Ell you need your application traffic to be distributed?
[04:41] Pe_Ell or you just need to make a connection between two application processes on different machines?
[04:41] twittard Yes; however, to someone else's application that has its own way of structuring messages
[04:42] twittard Pe_Ell: The later, more than the former.
[04:42] Pe_Ell gonna guess no, but hopefully someone with more knowledge will answer..... :)
[04:43] twittard my knowledge of 0mq is nil. my education process generally starts with some stupid questions.
[04:44] guido_g ØMQ can't talk to arbitrary endpoints, it needs to be a ØMQ aware endpoint due to the fact that there is "invisible" communication going on on the wire
[04:44] Pe_Ell fair enough, I do the same thing after I read a few webpages... :)
[05:16] rntz Is there some way to wait for a socket to send all its queued messages to the network?
[05:17] rntz analogous in some sense to fflush()
[05:17] guido_g nope
[05:17] rntz ... really? There's no way to do that? That seems like a terrible oversight.
[05:19] guido_g ømq is async in nature, so waiting was not built in
[05:21] sustrik guido_g, entz: it's a actually a new feature in 2.1
[05:21] guido_g but only when you close the socket, right?
[05:21] sustrik sure
[05:22] guido_g flush on close, so to speak
[05:22] sustrik otherwise zmq_term hangs :)
[05:22] rntz "flush when you close the socket" actually covers precisely what I was trying to do
[05:22] guido_g not a practical way of flushing messages out
[05:22] sustrik use the master branch from the git repo
[05:22] sustrik guido_g: yes, right
[05:23] guido_g but there more importnat things to improve
[05:23] sustrik ?
[05:23] guido_g like how do i "reset" a req socket if the peer died before an answer was received
[05:24] sustrik yes, that has to be done
[05:24] guido_g full ack
[05:24] sustrik i simply have no resources to do all the should be done :|
[05:24] guido_g otherwise the req/rep thingy is not usable the way it should be used
[05:24] guido_g there might be light
[05:25] sustrik we also need subscription forwarding etc.
[05:25] guido_g i took some time of, so it might happen that i look into some of these things
[05:25] sustrik wow
[05:25] guido_g of course i'd need a guide
[05:25] sustrik that would be great
[05:25] sustrik i'll be happy to help you
[05:25] sustrik actually
[05:25] sustrik it may be a good intention to write a "how it works" guide
[05:26] sustrik which in turn would allow other people to jump in
[05:26] sustrik incentive*
[05:26] guido_g uhm... i'd let pieter do the writing for humans :)
[05:26] Pe_Ell heh
[05:26] sustrik the problem is he doesn't know how it works internally
[05:27] Pe_Ell I find his guide amusing
[05:27] guido_g *sigh*
[05:27] sustrik but maybe we can do that together
[05:27] sustrik it is
[05:27] guido_g yes, but some parts are really irritating
[05:27] sustrik i'll give the details, pieter writes the text...
[05:27] sustrik like?
[05:27] guido_g like this wholwe mama, papa thingy, i needed a second browser tab so i could look up the definitions
[05:28] sustrik what's mama, papa?
[05:28] guido_g see
[05:28] guido_g ch 3 of the guide
[05:28] sustrik let me see
[05:28] guido_g don't... too late...
[05:28] guido_g :)
[05:29] sustrik have the quality of the guide deteriorated?
[05:29] sustrik or is it just that new chapters should not be there?
[05:30] guido_g the quality is good, imho
[05:30] Pe_Ell heh
[05:30] sustrik chapter 1. looks much the same like it did before
[05:30] Pe_Ell the pages are rather long
[05:31] Pe_Ell but there's a lot of ideas packed into them
[05:31] guido_g right
[05:31] Pe_Ell you added more to page2 and expanded page3 recently
[05:31] sustrik it was pieter not myself :)
[05:31] guido_g but given the very short time pieter managed to that written, it's a good thing
[05:31] sustrik yes, he's actually very good at writing
[05:31] guido_g *to get
[05:32] sustrik guido_g: so your prolbme is that it introduces new terminology, right?
[05:33] guido_g yes
[05:33] guido_g but no-one else complained so far, so wouldn't take it too serious
[05:33] Pe_Ell I thought that part was funny, but not sure of a better way to put that that isn't rather technical....
[05:34] sustrik well, if there's a well-coined term instead of mama and papa
[05:34] sustrik we should rather use that
[05:34] guido_g full ack
[05:34] guido_g but i guess this is the "pieter the sales man" part that's shining through :)
[05:35] sustrik :)
[05:36] sustrik but thst doesn't have to affect usability of the guide...
[05:36] guido_g CH3 "Custom Request-Reply Routing" there it is
[05:37] sustrik finally scrolled that far :)
[05:38] guido_g you could have used the table of content at the top :)
[05:38] sustrik i see
[05:38] Pe_Ell he's making fun of the old stereo types where the at the dinner table the mom will just talk and the dad only talks when spoken to directly
[05:39] guido_g might be
[05:39] sustrik just reading it
[05:39] Pe_Ell that was how I took it
[05:39] guido_g but he's using the names all after, so it's confusing for people who are used to req/rep and friends
[05:39] sustrik can you propose better naming?
[05:40] sustrik ah
[05:40] guido_g i knew this was comming...
[05:40] sustrik they are synonyms for existing socket types
[05:40] sustrik now i realised that
[05:40] guido_g i'm actually happy that i'm not the only one who was/is confused
[05:41] sustrik ok, i'll ask pieter mama & papa as an allegory but stick with official socket type names
[05:41] sustrik use...
[05:41] guido_g thx
[05:42] Pe_Ell he explains the names btw...
[05:42] Pe_Ell you read that right?
[05:42] sustrik yes, that's fun
[05:42] sustrik what i am saying is, let's keep that part in
[05:42] guido_g yes, but if you're used to the socket names you have to learn "additional" names w/o any real meaning in this context
[05:42] sustrik REQ socket it mama. blah-blah
[05:43] sustrik but use REQ later on, non mama
[05:43] Pe_Ell true
[05:43] Pe_Ell course if you already know the socket names and types... would you be reading that?
[05:43] sustrik possibly not
[05:43] Pe_Ell I only read it because I knew next to nothing about how 0mq worked... :)
[05:43] guido_g i tried to read that part last week after work, but failed because i forgot the menaing of mama and papa and whatnot the next day
[05:44] guido_g oh, don't get me wrong
[05:44] guido_g the concepts tought in there are important
[05:44] sustrik yup, i've got that
[05:44] Pe_Ell yeah my issues was there was so much to read and I read it over a week and a half after work I'd have to go re-read previous parts since I'd already forgotten the concepts...
[05:44] guido_g these things will be *the* bulding blocks to use for apps
[05:45] sustrik ack
[05:45] Pe_Ell and I was almost done when he updated the docs again so I had to start over one chapter 2... :)
[05:45] sustrik i've gave up already :)
[05:45] guido_g hehe
[05:45] sustrik wait till it stabilises
[05:45] Pe_Ell wasn't a big deal for me though... I tend to work on more than one project at a time.. so I'm used to having to crossreference things from memory leakage..
[05:45] guido_g it's been stable the last week! :)
[05:46] Pe_Ell hehe
[05:56] guido_g hmmm... amsterdam
[05:58] sustrik guido_g: where are you based?
[05:59] guido_g hamburg, germany
[05:59] sustrik that's not that far,no?
[05:59] guido_g my ip is a bit missleading :)
[05:59] Pe_Ell geoip tends to be very misleading..
[05:59] guido_g 1h flight or 5.5h train
[05:59] sustrik ah, not that close
[05:59] Pe_Ell I need to go to Germany... Holand sounds like fun too....
[05:59] guido_g Pe_Ell: not really, you'll hit the right town quite often
[06:00] sustrik looks much closer when looking from afar
[06:00] guido_g hehe
[06:00] Pe_Ell really? I've found that depending on how dense the area is it won't be very accurate beyond the region for large portions of europe...
[06:01] Pe_Ell same as the US when you're in places like LA or the bay area in California
[06:01] guido_g true
[06:01] Pe_Ell the ISPs all route the traffic out of two or three nodes and those aren't anywhere near the uers
[06:01] guido_g but in my case it's due to a vpn
[06:01] Pe_Ell ah
[06:02] sustrik on the other hand, if there's you, gonzalo, mikko, pieter, myself + 1-2 more guys it may be worth of coming
[06:02] sustrik let's see whether anyone else joins
[06:02] guido_g would be cool
[06:03] Pe_Ell err, less
[06:03] guido_g i could book it as a "conference" :)
[06:03] sustrik i haven't been in amsterdam for 20 years or so :(
[06:03] sustrik definitely, a mini-conference
[08:11] mikko sustrik: re: jzmq 10
[08:12] mikko sustrik: i wonder if imatix has capabilities to provide machines to run tests on for language bindings builders?
[08:12] mikko little and big endians etc
[08:18] pieterh mikko: it could be good to create some shared virtual machines for this
[08:18] mikko pieterh: usually they don't have things such as SPARC available
[08:18] pieterh true
[08:19] pieterh we do, in fact, have a rather large sparc machine here
[08:19] pieterh but it would not run all the time
[08:19] mikko i was wondering if something like hudson would work?
[08:19] mikko setup master instance and use different architectures as slaves
[08:20] pieterh looks nice
[08:21] mikko i can provide x86 (32bit)
[08:21] mikko i got ~20 DL380s lying around
[08:21] mikko old ones
[08:21] pieterh i think boxes is not the real problem
[08:22] pieterh rather, who would make and maintain such a project
[08:24] mikko i could make a pilot on some evening
[08:24] mikko using a couple of virtual machines to see how feasible the idea is
[08:24] mikko testing with different bingdings and main libzmq
[08:46] sustrik mikko: what boxes are you actually interested in?
[08:46] sustrik SPARCs?
[08:47] pieterh mikko: that would be a good start, then we can extend that over random boxes
[08:48] sustrik SPARC is good for testing as it's a RISC, pretty different from x86
[08:50] mikko pieterh, sustrik ill send an email to the list to see what different language bindings would need in terms of testing platform
[08:50] sustrik good idea
[08:50] pieterh mikko: sounds good
[08:55] mikko my experience on anything else than x86(_64?) is very limited so I would assume this kind of setup would help on finding the possible porting issues
[08:55] sustrik ack
[08:55] mikko and also zeromq developers get feedback on how changes affect the language bindings
[08:55] mikko etc
[08:55] mikko a lot of benefits
[08:58] sustrik sure, the only problem i see is running such a cluster
[08:58] sustrik the servers are noisy and power-hungry
[08:59] sustrik maybe switching them on once a week to do automated builds/tests would work
[09:02] mikko if the setup is distributed then maybe people with old sparcs or other weird hardware lying around can chip in
[09:08] sustrik yeah, but i assume the problem is not having the hardware as such, but running it 24/7
[09:08] sustrik if the setup is distributed you would have then to chase all the participants to turn it on at the same time
[09:08] sustrik etc.
[09:09] mikko i mean, there are people who already run hardware 24/7
[09:09] mikko such as myself
[09:10] mikko i got a some hardware in a datacentre mainly running idle
[09:10] mikko hosting my blog and irc client
[09:10] sustrik right
[09:10] sustrik iirc mato has an itanium box running in datacenter
[09:10] guido_g but most people don't have access to a data center
[09:10] mikko i could just setup new vm instance on the esxi and that would participate in the test system
[09:11] guido_g so having a box differs from beeing able to run in 24/7
[09:11] mikko so it would need to be pull instead of push
[09:11] mikko so that people who turn on the machines occasionally could run the thing and shutdown again
[09:12] guido_g yep
[09:13] sustrik which actually makes the design pretty simple
[09:13] sustrik write the tests
[09:14] sustrik then the last command in the test stores the results on some 24/7 available box
[09:14] sustrik thus anyone can run the tests if they feel like it
[09:16] sustrik i haven't meant just your tests in the core
[09:16] sustrik the discussion was mainly about testing bindings
[09:16] guido_g i know
[09:16] sustrik anyway, the first step: write the tests
[09:16] guido_g but that doesn't mean that they don't need a lot of love...
[09:17] sustrik yeah
[09:17] guido_g yes, then automate the build
[09:17] mikko i got 21 tests for php bindings this far
[09:17] sustrik nice
[09:17] mikko i dont know if other language binding writers do testing
[09:17] sustrik maybe brian?
[09:18] mikko
[09:18] mikko yep
[11:38] CIA-20 zeromq2: 03Martin Sustrik 07master * r0bb76b6 10/ src/xrep.cpp : assert when xrep socket gets reconnected in the middle of the shutdown -- fixed -
[11:38] CIA-20 zeromq2: 03Martin Sustrik 07master * r2a85cce 10/ src/zmq.cpp : Merge branch 'master' of -
[11:55] jasong_at_apache so right now I'm messing around with zmq to see how it can fit my needs, and I've read a few spots on the mlist about having native secure communication
[11:56] jasong_at_apache which brings me to that topic, security. right now, where does zmq stand with regard to "in the wild" communication?
[11:56] jasong_at_apache sorry, thats " secure and in the wild" communication
[12:10] sustrik it is not ready for that
[12:17] jasong_at_apache thx
[14:03] CIA-20 zeromq2: 03Martin Sustrik 07master * r1a6cd59 10/ (tests/ tests/test_shutdown_stress.cpp): stress test for shutdown process added -
[14:23] mato sustrik: ping
[14:24] sustrik pong
[14:24] mato sustrik: i saw you committed a fix to the xrep assert thing
[14:24] sustrik yes?
[14:24] mato sustrik: it's the same as the patch you gave me yesterday, right?
[14:24] sustrik yes
[14:24] mato ok, good, just checking
[14:25] sustrik ok
[16:46] CIA-20 zeromq2: 03Martin Sustrik 07master * r2142b89 10/ src/pair.cpp : issue 92 -- Assertion failed: inpipe && outpipe (pair.cpp:86) -- fixed -
[17:27] mikko got hudson running and building libzmq for the first time
[17:31] omarkj Hey guys. I'm publishing data to a zeromq socket with the erlang libs. After a few thousand messages the whole process comes down with "Assertion failed: nbytes == sizeof (command_t) (signaler.cpp:284)"
[17:31] omarkj Any idea what that could be?
[17:34] omarkj HWM is at 1.
[18:11] sustrik mikko: !
[18:12] sustrik omarkj: what system are you running on?
[18:13] omarkj sustrik: As in, what OS? In that case, Ubuntu server at the moment. R13B.
[18:13] sustrik omarkj: can you possibly check what the nbytes value is?
[18:14] omarkj Hm. I'll try.
[18:14] omarkj Not quite sure how I'd do that.
[18:14] sustrik hust modify the code to print it
[18:15] sustrik if (nbytes != sizeof (command_t) {
[18:15] omarkj ah, ok
[18:15] sustrik printf ("%d\n", (int) nbytes);
[18:15] sustrik zmq_assert (false);
[18:15] sustrik }
[18:17] omarkj Hm, I'll have to do this Monday. The test market at the exchange is not running at the moment so I have no data to feed into the process.
[18:18] omarkj Well, not the levels I need to make it crash.
[18:18] omarkj Sorry for causing the hassle, but I'll get this value.
[18:18] sustrik ack, just send the result to the mailing list then
[18:18] omarkj Okay, thanks a lot.
[18:18] sustrik np