IRC Log

Thursday February 10, 2011

[Time] NameMessage
[00:07] Seta00 forgot to install g++-multilib
[00:07] Seta00 :P
[07:37] suzan_shakya hi all, can anybody explain me why a single context should be used in a process ?
[07:42] Guthur suzan_shakya, you can use multiple contexts in a process
[07:42] Guthur with you want to use inproc transport the sockets need to be created by the same context though
[07:43] Guthur with/if
[07:46] suzan_shakya Guthur: thanks
[07:46] guido_g but is not the canonical way to have more then one context per process
[09:54] mikko sustrik: http://build.valokuva.org/job/ZeroMQ2-core-master_mingw32-win7/79/console
[09:55] mikko started failing after reaper changes
[10:20] mikko man you are fast pieter_hintjens
[10:21] pieter_hintjens mikko: huh?
[10:21] mikko pieter_hintjens: the "What happens when push socket is closed"
[10:21] pieterh :-)
[10:21] mikko i had barely read the message and already gmail indicated response from you
[10:22] pieterh 4 minutes is still pretty high latency
[10:22] pieterh I need to find a way to get alerts on 0MQ list traffic...
[14:46] mikko pieterh: https://build.valokuva.org/job/zfl-master_ZeroMQ2-master_mingw32/9/console
[14:46] mikko zfl mingw32 running now
[14:47] pieterh mikko, nice... there are still a few glitches but most of it builds
[14:48] mikko fails
[15:02] pieterh mikko: I've committed further fixes, send me a log of the errors and I'll check them out
[15:05] mikko pieterh: you should be able to init build on hudson
[15:05] mikko if you still have your username and pass
[15:05] pieterh mikko: I have too many pots boiling :-)
[15:06] mikko https://gist.github.com/73dae202027903631905
[15:06] mikko looks like include issue
[15:07] mikko path missing or something
[15:07] mikko https://github.com/mkoppanen/zfl/commit/b54ea4c22956818f435ee085d01847bb5f6c9335
[15:07] mikko this is some old fix i did
[15:07] mikko not sure how relevant that still is
[15:10] pieterh ah... the errors from zfl_prelude.h are because that file thinks it's building on a Unix box and is looking for Unix include files
[15:11] pieterh I've not used ming32 for a really long time, so don't recall what's needed here
[15:11] mikko not fully true
[15:11] mikko you wouldn't include using '\' on unix ?
[15:11] pieterh indeed
[15:11] pieterh :-)
[15:11] mikko zfl_prelude.h:255 is inside windows defs
[15:12] pieterh it thinks it's both __MSDOS__ and __UNIX__
[15:13] mikko regarding the build issue on freebsd someone mailed about
[15:13] mikko i think a switch in configure would be nice "--with-libzmq=" or so
[15:13] pieterh well, zfl is by definition dependent on zmq
[15:13] mikko that could use pkg-config to check the prefix and required flags
[15:14] mikko it also currently depending on user to specify CFLAGS and LDFLAGS for non-standard location
[15:14] pieterh right
[15:14] pieterh mikko, i'm going to leave this to settle for a while and come back to it later
[15:14] mikko --with-libzmq=/opt/i/installed/libzmq/here style
[15:14] mikko ok
[15:14] pieterh may install mingw32 if needed
[15:15] mikko i can look into patching this
[15:15] pieterh have gotten the thing to compile on MSVC and almost link successfully...
[15:15] mikko i'll send you a mail if i got time to look into it today
[15:15] pieterh that earns me lunch, in any timezone :-)
[15:15] mikko true
[15:15] pieterh cyl
[15:16] mikko later
[16:07] andrewvc sustrik: If you're around I have an strace that shows where that nbytes error is I believe
[16:07] andrewvc https://gist.github.com/820779
[16:07] andrewvc basically, it looks like close is called on FD 22
[16:11] cremes andrewvc: fyi, i found (and fixed) a bug in ffi-rzmq
[16:11] andrewvc oh?
[16:12] cremes sometimes when a recv failed with rc = -1, eagain is *false* but it was still acting like it was a successful read
[16:12] cremes i'll push the patch in a minute
[16:12] andrewvc oh, good catch
[16:12] andrewvc cool, I'll test it, I wonder if that was related, it sounds like it could have been
[16:12] cremes i need to do a code clean up... some of my logic is too hard to follow
[16:12] cremes simpler is better
[16:13] andrewvc agreed, but don't bash your code, I've found it pretty clear ):
[16:13] andrewvc :)
[16:14] cremes andrewvc: you are too kind!
[16:14] cremes ok, pushed
[16:15] andrewvc cool, I've got my test suite all up already
[16:15] cremes give it a try and see if it makes a difference
[16:15] cremes btw, at the same time i found an error in zmqmachine related to the same thing
[16:15] cremes i'll push that too
[16:16] andrewvc sadly, not related
[16:17] andrewvc i mean, that patch didn't fix this bug
[16:17] cremes bummer
[16:17] cremes it's likely a 0mq flaw
[16:17] andrewvc yeah, I was just about to post to the list
[16:17] cremes do you have a gist you can share? i have an hour or so today where i can poke at this
[16:18] andrewvc yep
[16:18] andrewvc well, so this is interesting in and of itself I guess
[16:18] andrewvc https://gist.github.com/820779
[16:18] andrewvc that's an strace to start with, I'll post the ruby code in a sec
[16:18] andrewvc FD 22 is closed
[16:18] andrewvc then recvfrom is called on it again later on
[16:19] andrewvc I think that's the bug, unless I'm missing something, I'm a little shaky on low level syscall semantics
[16:20] andrewvc FD22 being a unix domain socket, but one that was opened internally, as the ZMQ sockets are all tcp
[16:20] andrewvc lemme upload the ruby code actually, right now it needs RBX or 1.9.2 to run. I'm working on jruby support (kicking Thin out)
[16:20] cremes i wonder if FD 22 has some special meaning? i don't have a lot of knowledge about this lower level stuff
[16:20] cremes cool, i use both
[16:21] mikko no, it shouldn't have
[16:22] andrewvc It's opened higher up with 116809 [pid 11284] socketpair(PF_FILE, SOCK_STREAM, 0, [21, 22]) = 0
[16:23] andrewvc if you want to give the ruby code a shot cremes I can set you up pretty quick, lemme just push a couple things
[16:24] cremes sure
[16:26] andrewvc ok, so just a few things, you need either rbx or 1.9.2. then....
[16:26] andrewvc gem install dripdrop
[16:26] andrewvc then clone this somewhere git://github.com/andrewvc/socket-motor.git
[16:27] andrewvc then, inside the socket-motor dir: ruby bin/socket-motor
[16:28] andrewvc oh you also need: gem install sinatra haml
[16:28] andrewvc inside socket-motor's example dir, run ./run.sh
[16:29] andrewvc finally visit localhost:3000 and try putting a message in the chat entry box and hitting send. that'll cause the bin/socket-motor process to die
[16:29] andrewvc phew
[16:29] andrewvc yeah, I've been trying to communicate this via strace because the setup is a bit cumbersome
[16:34] cremes needs hashie/mash too
[16:35] andrewvc ah crap
[16:35] cremes andrewvc: another require error, but this one doesn't look like a gem
[16:35] cremes <internal:lib/rubygems/custom_require>:29:in `require': no such file to load -- socket-motor/messages/req_rep_message (LoadError)
[16:35] andrewvc gem install hashie
[16:35] andrewvc oh, forgot to check that in damnit
[16:36] andrewvc wait, hmm it is there
[16:37] cremes probably a path issue
[16:37] andrewvc I've been running it with -I lib/ in the socket motor dir
[16:37] andrewvc yeah
[16:38] andrewvc this is bin/socket-motor yeah
[16:39] cremes so, what's the right command? ruby -l lib bin/socket-motor doesn't work
[16:39] andrewvc I start it with ruby -I lib/ bin/socket-motor
[16:39] cremes ruby -I lib bin/socket-motor works (where -I is an "eye")
[16:40] andrewvc yep
[16:40] cremes now i'm getting an in6_addr error (invalid address)
[16:40] cremes do you have some addresses hard coded?
[16:40] cremes or ipv6 stuff?
[16:40] andrewvc nope
[16:41] andrewvc I disabled it actually
[16:41] cremes https://gist.github.com/820848
[16:41] cremes i'm on osx... don't know if that should matter
[16:41] andrewvc oh, it uses resolv to do a hostname lookup, weird lemme disable that whole block
[16:42] andrewvc right now it doesn't do anything, it just looks for the first ipv4 addr
[16:42] andrewvc for a given host
[16:43] andrewvc prolly an OSX vs linux thing
[16:44] cremes how can i disable it locally in your code?
[16:44] andrewvc gem install dripdrop should fix it
[16:44] andrewvc well hold on one sec
[16:44] andrewvc pushing...
[16:44] cremes heh
[16:44] andrewvc ok pushed, dripdrop 0.9.6 should do it
[16:46] cremes success
[16:46] andrewvc nice!
[16:47] andrewvc basically, this is a websocket <-> ZMQ <-> HTTP <-> Rack bridge
[16:47] cremes andrewvc: new error: https://gist.github.com/820863
[16:48] andrewvc hmm, that's a new one
[16:48] cremes i'm looking at config.ru... do i need some /swf and /js dirs somewhere?
[16:48] andrewvc yeah, they're in public
[16:49] cremes ah, so they are
[16:49] andrewvc not sure about thin, hmmm
[16:49] cremes is thin 1.2.7 the right one?
[16:50] andrewvc oh, I just got it as well, it's a 1.9.2 vs rbx issue hold on lemme pin it down
[16:51] andrewvc ah, here we go
[16:51] andrewvc thin -R config.ru start
[16:51] andrewvc I was using old style syntax, apparently RBX doesn't mind weirdly
[16:52] cremes okay, i get the crash
[16:52] cremes there are a *lot* of moving parts here... this is going to be tough
[16:53] andrewvc yeah, that's the thing
[16:53] cremes do you also see the EM crash from run.sh?
[16:53] andrewvc ohh yeah, now I do, this is the first time i've run it in 1.9.2 in a while, I guess rbx has been shielding me
[16:53] andrewvc interesting
[16:54] cremes i'm going to let you chase this a little bit more... :)
[16:54] andrewvc yeah
[16:54] andrewvc good find
[16:54] cremes i find that i fix a *lot* of bugs by running my code across all 3 of the major runtimes
[16:54] andrewvc i mean, if they're synchronized, that opens up a whole new world
[16:54] cremes some are more forgiving than others
[16:55] cremes you might also try MRI 1.8.7p-<whatever is latest> just for giggles
[16:55] andrewvc hehe
[16:55] andrewvc yeah, well here's the crazy thing, I've been running socket-motor in rbx and 1.9.2
[16:55] andrewvc just never thought to run the rack app in it
[16:55] cremes i know the bindings don't really have support for it, but it might shed some light
[16:56] cremes heh
[16:56] andrewvc I'm looking forward to jruby support in dripdrop, that's around the corner, thin wasn't a great choice
[16:56] andrewvc thanks for shedding light on this though
[16:57] andrewvc I guess I should still submit the strace zmq dev, prolly is a zmq bug, shouldn't be possible anyway right?
[16:57] cremes well, it depends
[16:57] cremes is your rack app sending a 0mq packet in the right wire format?
[16:58] cremes i.e. are you using a 0mq socket from rack or something else?
[16:58] andrewvc nope, it doesn't speak 0mq, it speaks http. Dripdrop lets you drop HTTP in for a req/rep
[16:58] andrewvc so, I'm guessing EM is going crazy, and terminating strangely
[16:59] cremes ok, then i don't know
[16:59] cremes don't you just love bug hunts?
[16:59] andrewvc yeah, well, the HTTP message gets turned into ZMQ xreq/xrep, then comes back as HTTP
[16:59] andrewvc hah
[17:00] andrewvc Ummm, I woke up at 4am yesterday with an idea to fix it, went back to sleep at 4:30. grumpy
[17:00] cremes so funny... i was just telling my partner that i went to bed at midnight, laid awake for 2 hours and started coding again at 2 until 3
[17:01] cremes i'm a bit grumpy today myself :)
[17:01] cremes sometimes it's impossible to turn off my brain
[17:01] andrewvc lol
[17:01] andrewvc same
[17:01] andrewvc btw, apparently that error was unrelated
[17:02] andrewvc I wasn't handling an HTTP response, so EM died, handled it, now it lives but zmq still dies, I'm gonna send this off to the list
[17:03] andrewvc my hunch is there's a bug in my app code, and EM is handling an exception strangely, and ZMQ is dying. I have no proof of this, but that's my feeling
[17:04] cremes andrewvc: any idea what line of code in dripdrop that this dies on?
[17:05] cremes have you narrowed that piece down yet?
[17:05] andrewvc I think that's a red herring, I traced all the way through to ffi-rzmq
[17:05] andrewvc to right before libzmq send
[17:05] andrewvc but it's dying trying to recv from a socket
[17:05] andrewvc according to strace
[17:05] andrewvc a tried changing up the message being sent, allocating a new one
[17:06] cremes is the socket that it is sending on the same as the one libzmq is recv on?
[17:06] andrewvc I tried checking ZMQ::EVENTS in a debugger right before the call as well, making sure it was writable
[17:06] andrewvc nah, it's a pub socket
[17:06] cremes ok
[17:06] andrewvc tcp vs inprocess vs ipc makes no diff either
[17:06] cremes i bet this is some internal FD that the lib uses for talking amongst itself
[17:07] andrewvc yeah, it uses a unix socket for that weird FD no matter which protocols I use
[17:07] cremes you need to catch this in gdb
[17:08] cremes run it like; gdb --args ruby -I lib bin/socket-motor
[17:08] cremes when it dies, you can print the zmq backtrace
[17:08] andrewvc awesome, I know nothing about GDB
[17:09] cremes i know enough to be dangerous
[17:09] andrewvc lol
[17:09] cremes i have it in gdb... now i need sustrik or mato to tell me what they need to figure this out
[17:10] andrewvc btw, I fixed the rack client
[17:10] cremes https://gist.github.com/820902 the backtrace in gdb
[17:10] andrewvc if you pull socket motor thin won't keep dying
[17:11] cremes it didn't die when i caught the error in gdb
[17:11] andrewvc that's good and bad
[17:11] cremes it died when i exited gdb though
[17:11] andrewvc interesting
[17:12] andrewvc zmq acts weird in general when you ctrl-C
[17:12] cremes andrewvc: which line in ffi-rzmq does it execute before choking?
[17:12] andrewvc lemme dig it up
[17:12] cremes yeah, the new reaper thread stuff is supposed to solve that
[17:13] andrewvc socket.rb line 260
[17:13] andrewvc yeah, I just pulled this morning
[17:13] andrewvc no luck there, as far as the new reaper stuff
[17:13] andrewvc well, it used to die there
[17:14] andrewvc maybe not anymore
[17:14] cremes any flags?
[17:14] cremes NOBLOCK?
[17:14] andrewvc yeah, I tried messing with those as well, it's a two part PUB messages, topic + body
[17:14] andrewvc yeah, noblock
[17:14] cremes ok, so NOBLOCK & SNDMORE
[17:14] andrewvc i tried it with no flags as well
[17:14] cremes ok
[17:15] cremes how are you constructing the message? is message.address valid?
[17:15] cremes i.e. maybe there is garbage in the message and 0mq is choking on that somehow
[17:16] andrewvc https://github.com/andrewvc/dripdrop/blob/master/lib/dripdrop/handlers/zeromq.rb#L44
[17:16] cremes is the message freshly allocated with Message.new? or is it sourced from elsewhere?
[17:16] andrewvc is how it's constructed
[17:16] andrewvc I tried firing up a debugger, allocating a new message, and sending that, right before it died
[17:16] andrewvc so I used to be able to time it, like, it'd send 4 messages, and die on # 4
[17:17] andrewvc so I stepped through the first 3, and inserted a new message right before zmq_send to test that out
[17:17] andrewvc no luck there either
[17:17] cremes are you sure this line is correct?
[17:17] cremes https://github.com/andrewvc/dripdrop/blob/master/lib/dripdrop/handlers/zeromq.rb#L51
[17:17] cremes generally you OR flags together, not add them
[17:18] cremes e.g. flags = (i + 1 < num_parts ? ZMQ::SNDMORE : 0) | ZMQ::NOBLOCK
[17:18] andrewvc oye, crap good catch
[17:18] andrewvc *pacepalm
[17:18] andrewvc *face palm
[17:18] andrewvc lemme try that
[17:18] cremes still crashes
[17:20] andrewvc yep
[17:20] andrewvc I think this is in libzmq, I mean, it shouldn't ever reach there anyway
[17:20] andrewvc no matter how inept I am :)
[17:21] andrewvc that socket is used for a remote logger, so it's not critical to the app. If I disable it, it runs fairly well
[17:21] cremes still looking...
[17:21] andrewvc well, aside from all the other bugs in it, that I haven't had time to deal with as I've tracked down this.
[17:22] andrewvc brb
[17:26] cremes andrewvc: found something
[17:30] cremes andrewvc: nope, false alarm
[17:33] andrewvc back
[17:33] andrewvc the amount of time I've dumped into this is a bit ridiculous at this point, hopefully the mailing list will have an answer
[17:33] Rich_Morin Zed Shaw gave a very nice talk last night at the Bay Area Python group, talking about how he used ZeroMQ on assorted projects.
[17:33] andrewvc my last hope is finishing jruby dripdrop support
[17:34] andrewvc and seeing if jruby gives a better error
[17:36] andrewvc that, or running git bisect on libzmq on a lazy saturday
[17:37] andrewvc course, I guess I need a known good rev for that
[17:40] alexbOrsova im having an issue with zeromq on ubuntu 10.04.1. I downloaded the library for posix, did "./configure", "make", "sudo make install". Then I complied one of the examples and tried to run it and I got "error while loading shared libraries: libzmq.so.0"
[17:40] alexbOrsova any suggestions?
[17:42] alexbOrsova nevermind, it was a bug in the makefile, "/sbin/ldconfig" should probably be called as part of the "make install" process...
[18:03] Rich_Morin Hi, guys. Should we try for a SUSS session this Sunday?
[18:03] mikko what is a SUSS session ?
[18:05] Rich_Morin Sorry, wrong channel :-/
[18:05] Rich_Morin SketchUp Sunday Social
[19:11] bitweiler is ZMQ_REQ and ZMQ_REP still use in library or should one use ZMQ_PULL & ZMQ_PUSH?
[19:17] Seta00 bitweiler, PUSH and PULL aren't replacements to REQ and REP
[19:19] andrewvc cremes: I think I figured it out
[19:20] andrewvc thanks for all the help, I mean it's still most likely a ZMQ bug I think, but it looks like ZMQ::EVENTS is the cause
[19:20] cremes andrewvc: good deal
[19:20] andrewvc so, thanks for GDB, that narrowed it down to getsockopt
[19:20] cremes bitweiler: REQ/REP and PUSH/PULL are different kinds of sockets
[19:21] cremes bitweiler: PUSH/PULL used to have different names that were deprecated; i forget what they were
[19:21] andrewvc and ZMQ::EVENTS would get checked by my em-zeromq library, and it'd work for a while, then kaput
[19:21] cremes andrewvc: race condition in getsockopt you think?
[19:21] andrewvc I think what may be happening still is, ruby is generating some kind of exception but ZMQ is dying first.
[19:22] andrewvc the ZMQ::EVENTS calls aren't necessary strictly, as far as I can tell, commenting them out in em-zeromq just results in recv or send being called, and those can fail and be handled
[19:23] andrewvc so, I can't speak to it being a race condition, but that sounds right to me
[19:26] cremes interesting
[19:26] cremes you should post your observations to the ML
[19:26] andrewvc I'm going to for sure
[19:29] andrewvc if you have some time chuck: https://github.com/andrewvc/em-zeromq/blob/master/lib/em-zeromq/connection.rb
[19:29] andrewvc I just removed the calls to ZMQ::EVENTS
[19:30] andrewvc I mean, if I just attempt to send or recv and it fails, that works, I can see how it could be slower depending on the ZMQ internals, since those FDs are kind of blunt instruments
[19:30] andrewvc could use a second pair of more experienced eyes
[19:31] cremes looking...
[19:33] andrewvc I think it may just be calling ZMQ::EVENTS on writable sockets
[19:34] andrewvc it may even just be PUB sockets, since that's what it is in this case, leaving ZMQ::EVENTS enabled for readables doesn't trigger it
[19:34] andrewvc though, this is obviously all new, and perhaps further testing will reveal problems there as well
[19:35] cremes andrewvc: not sure about the purpose of #deregister_writable; it just sets a flag that isn't used anywhere
[19:36] andrewvc oh, that's used by eventmachine
[19:36] andrewvc so, it tells EM to regard or disregard activity on the FD
[19:37] andrewvc yeah, it's a bit funky looking, could use a comment
[19:37] andrewvc same thing in notify_readable
[19:37] cremes ok
[19:37] cremes nothing major jumps out at me; looks good
[19:38] andrewvc cool. Well, I've got to get back to other stuff, but I'll be posting gdb traces and other goodness on the list tonight
[19:38] cremes ok
[19:38] andrewvc thanks a bunch, if you hadn't brought out GDB I think I'd still be stuck
[19:38] sysgears hi guys, i'm trying to extend zmq termination, so that messages at the receiving pipe were all fetched before ZMQ terminates
[19:38] cremes it's nice to have multiple tools in your toolbag
[19:39] cremes sysgears: take a look at ZMQ_LINGER
[19:41] sysgears cremes, what for? I'm trying to not loose any messages at receiver end, ZMQ_LINGER is to set TTL for messages, its a different thing I think
[19:43] cremes ZMQ_LINGER prevents the socket from closing until it has flushed all messages OR until the timeout has expired
[19:43] cremes there is no TTL in 0mq
[19:43] cremes and it isn't possible to not lose messages at receiver end unless you do an explicit ACK/reply
[19:43] cremes for each message
[19:43] sysgears Cremes, this is true for sending end
[19:44] sysgears I know it is not possible now, thats why I'm patching ZMQ
[19:44] cremes ok
[19:44] cremes are you creating a new socket type?
[19:45] sysgears nope, I want that recv fetched all the messages it has before it will return ETERM
[19:46] sysgears currently it returns ETERM right after it detects ctx_terminated is on
[19:46] cremes do you plan to make that a socket option?
[19:47] Seta00 I'm having difficulties to track down the reason I'm hitting this assertion: nbytes == sizeof (command_t) (mailbox.cpp:244)
[19:47] cremes Seta00: what os?
[19:47] Seta00 Linux
[19:47] Seta00 Ubuntu 10.10
[19:48] cremes you might want to look at modifying some sysctls; the command_t's use some of those buffers for ipc/tcp/udp
[19:48] sysgears I plan to extend zmq_term semantics - it will not only flush all messages to be sent before termination it will also recv all messages that are currently in its buffers before termination.
[19:48] cremes search the ML for sysctl and see what comes up
[19:49] Seta00 the ML?
[19:49] cremes mailing list
[19:49] Seta00 oh
[19:49] Seta00 lol
[19:49] Seta00 don't use ANU's
[19:49] Seta00 (Acronyms Nobody Understand)
[19:50] cremes yessir
[19:51] andrewvc ANU's are fine so long as they're TLA's
[19:51] cremes andrewvc: hey, get back to work!
[19:51] andrewvc lol
[20:00] Guthur i have the scenario where requests come in on individual threads the reply to each comes back async on other threads, my solution is to use 0MQ, is it overkill and normal thread constructs more suited?
[20:00] Seta00 tried this: http://www.mail-archive.com/zeromq-dev@lists.zeromq.org/msg06901.html
[20:00] Seta00 didn't work
[20:03] cremes Guthur: that's a great use-case for the inproc transport
[20:03] kaib hi all. are there any other native implementation of zeromq than the reference c++ implementation?
[20:03] cremes kaib: not that i know of
[20:04] cremes Seta00: hmm... are there are sysctl's for ipc?
[20:04] cremes i'm pretty sure the command mailboxes are using ipc or something similar internal to the library
[20:04] kaib are they actively discouraged or just haven't happened yet? the protocol seems like a great little language interoperability system.
[20:05] cremes (though i could definitely be wrong)
[20:05] cremes kaib: no one has come along to take a whack at it yet; you could be the first
[20:05] cremes i've never seen the imatix guys discourage anybody from hacking on the lib or attempting a port
[20:05] Guthur cremes, that's my thought as well
[20:06] Guthur trying to convince the team lead at work is proofing a challenge
[20:06] Guthur he seems to think normal threading constructs and a global store are an 'easy' solution
[20:07] Guthur it's funny though, this easy solution still does work right yet
[20:07] cremes well, one way to discuss it is in terms of future scalability
[20:07] Guthur does/doesn't
[20:07] cremes what might be a thread today could easily be a separate process (or on a separate box) tomorrow
[20:07] cremes and the only thing you need to change is the transport string: inproc -> tcp
[20:08] Guthur yeah, the flexibility is a real seller for me personally
[20:08] Guthur that and the whole message passing concept
[20:09] kaib cremes: so the transports add some framing, do the sockets add any framing of their own?
[20:09] Guthur I just have to keep plugging a way at work to try and convince others
[20:09] kaib cremes: i've been trying to figure out the exact wire format.
[20:09] kaib Guthur: what is the problem you are trying to solve, i logged on just past your description..
[20:09] Guthur kaib i have the scenario where requests come in on individual threads the reply to each comes back async on other threads, my solution is to use 0MQ, is it overkill and normal thread constructs more suited?
[20:11] kaib ok, so are the original threads that received the request the ones that transmit out the response?
[20:12] kaib Guthur: ie. Client -> Thread0 -> async backend -> Thread1 -> Thread0 -> Client
[20:13] Seta00 hmm, it has something to do with the way I'm killing the socket on disconnect
[20:17] Guthur sorry i disco'd
[20:17] Guthur kaib yeah that's the scenario
[20:17] Guthur multiple client requests as well
[20:17] Guthur concurrent requests
[20:17] cremes kaib: wire format documented here: http://api.zeromq.org/zmq_tcp.html
[20:18] Guthur I actually have it solved with ZMQ
[20:18] Guthur but just can't get the buy in from my team lead to include the dependency
[20:19] Guthur very frustrating
[20:19] cremes Guthur: show your team lead how you can now break it up into multiple processes without changing any of the message queueing code
[20:19] cremes that's a pretty big win in my opinion
[20:19] kaib cremes: that's the wire protocol for tcp, i'm reading the code and it seems like there is some other framing going on as well. there is talk about "data batches"
[20:20] cremes kaib: not sure then; i haven't heard the term "data batches" come up before
[20:23] kaib Guthur: when you fire off the request to the async backend do you have any chance of deciding which thread gets the response? or even which pool of threads?
[20:24] Guthur kaib:nope its pretty opaque, the scenario is a request to a process running a quickFIX app
[20:24] Guthur the requesting thread send off a, QuoteRequest FIX message, this will then return a Quote async on another thread
[20:27] kaib Guthur: ok, your team lead might be trying to avoid extra dependencies on external projects... ie. if you guys can get it to work with a simpler approach you will be good for a while.
[20:27] cremes s/simpler/more familiar/
[20:28] kaib Guthur: i would benchmark your implementation. that gives you data to go on when the discussion resurfaces later..
[20:28] kaib cremes: that's true. much can be said for believing in the familiar.. :-)
[20:28] cremes kaib: don't i know it! :)
[20:29] kaib cremes: in zmq_encoder.cpp, talking about batches.
[20:29] cremes kaib: you need to chat with sustrik (primary dev) or drop a note to the mailing list
[20:30] cremes i only dig into the internals when i have a showstopper bug
[20:30] kaib cremes: thanks.
[20:30] cremes so far, i have somehow avoided the encoder stuff
[20:30] kaib sustrik: ping?
[20:30] cremes you are welcome
[20:30] cremes kaib: i think most of the devs are GMT+3
[20:30] Guthur I think it is telling how no one has actually implemented the a solution using the 'easy' global data approach
[20:32] kaib Guthur: heh.. well, you need to come well armed. working code and good political connections go a long way. :-)
[20:32] kaib cremes: oh, well i'm gmt+2, they seem to drop off awfully early.. :-)
[20:32] Guthur well I have the first,
[20:33] Guthur The half working solution they currently have takes the same time to do one quote as mine does to 60
[20:33] Guthur to do 60*
[20:34] Guthur That performance measurement has only became available to me this evening
[20:34] Guthur though they are using HTTP for the remoting
[20:35] Guthur so it will improve some when they can move a fast protocol
[20:35] Guthur faster*
[20:36] Guthur though it will need to be considerably faster than that, 5 secs per quote is just not going to cut it at all
[20:43] cremes Guthur: i do *way* faster than that in *ruby*
[21:21] chmod700 so I just grabbed 0mq+python bindings for v2.0.10 and am trying to get https://github.com/imatix/zguide/blob/master/examples/Python/mtserver.py to work, can someone tell me what I should be sending to get the "World" response back?
[21:24] chmod700 all I see on my connection to port 5555 is '\x01\x00'
[21:25] ianbarber are you connecting with 0mq?
[21:26] ianbarber on both sides
[21:28] chmod700 ianbarber: no I was just using telnet to connect
[21:29] chmod700 I guess I missed something in the docs :), how does one go about layering their wire-format stuff on top of that XREP device?
[21:30] ianbarber 0mq pretty much does all the wire level stuff, you send your own format in the messages
[21:31] ianbarber its very little overhead
[21:32] chmod700 ianbarber: so why doesn't telnet work (or wget, or a browser) when hitting the XREP port?
[22:11] ianbarber there is still a wire protocol, so you need to be able to understand that
[22:11] ianbarber in general zmq is designed for people using the library at both ends
[22:14] chmod700 ianbarber: thanks, so for HTTP for instance, do I need to stand up my own HTTP server and then forward requests through a zmq socket to workers?
[22:15] chmod700 or is there some magic http zmq device?
[22:15] Skaag the first.
[22:16] Skaag there is no direct relation between http and zmq
[22:16] Skaag if with java, you can use something like jetty to expose some kind of rest/json/xml interface, then pass the messages to a thread that publishes messages
[22:18] Guthur are select_t, poll_t, epoll_t, devpoll_t and kqueue_t all essentially doing the same thing except with different raw connections
[22:19] ianbarber chmod700: mongrel2 is quite good for doing a webserver with 0MQ transport to the backend
[22:19] ianbarber but in general, yeah, that's the kind of thing
[22:21] Skaag ianbarber: any samples of such a setup with mongrel2?
[22:21] Skaag would love to play around with it
[22:21] ianbarber Skaag: look at the mongrel2 book, 0MQ is the only way you can do dynamic content with it
[22:22] Skaag they have that as a sample? :)
[22:22] ianbarber basically it binds a push socket, and sends the headers, path, and some other bits and pieces over it
[22:22] ianbarber yeah, download mongrel2 and look in the examples dir
[22:23] Skaag awesome
[22:23] Skaag will do that now
[22:24] ianbarber if you just need one way comms, take a look at mikko's httpush https://github.com/mkoppanen/httpush
[22:24] ianbarber that's great if you want to do network logging over http or similar
[22:41] Skaag that's one way, towards the client side, I assume?
[22:42] Skaag damn -17˚ outside... not daring to go out.
[22:42] Skaag celcius.
[23:58] eut hello
[23:59] eut when using zmq_xrep sockets how are the three message pieces (identity, delimiter, and body) supposed to be separated?

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy