Monday September 26, 2011

[Time] NameMessage
[11:21] mikko sustrik: Assertion failed: ok (mailbox.cpp:79)
[11:21] mikko is this a race to shutdown ?
[11:22] sustrik mikko: hi
[11:22] mikko 2.1
[11:22] sustrik let me see
[11:24] sustrik yes, looks like a race condition
[12:03] CIA-79 libzmq: 03Martin Sustrik 07master * rd726120 10/ src/mtrie.cpp : Bug in matching algorithm fixed ...
[12:41] mikko sustrik: did you see the another ticket i opened?
[12:42] sustrik mikko: which one is that?
[12:44] mikko it's in jira
[12:44] mikko assertion in kevent
[12:44] mikko LIBZMQ-261
[12:44] mikko ah
[12:44] mikko you answered
[12:45] mikko odd that i didn't get an email
[12:45] mikko you want access to freebsd?
[12:51] calvin hi sustrik - i just responded to your question on jira
[12:52] calvin both of us ended up at nearly the same solution
[12:52] calvin so whichever you like you can use
[12:56] sustrik mikko: if it's reproducible, it would definitely help
[12:56] sustrik calvin_: hi
[12:57] sustrik calvin_: so your patch consists of 2 patches?
[13:02] calvin well originally i had that idea with the parent_trie_
[13:02] calvin but then I realized it was much simpler so i reverted that and just did something similar to you
[13:02] sustrik ah
[13:02] sustrik it seems that pieter have mrged the first version to 3.0
[13:03] calvin yeah i put in a pull request last night
[13:03] calvin with the additional changes
[13:03] sustrik ok, i see
[13:03] sustrik we'll sort that out somehow
[13:03] sustrik thanks for giving it a look btw
[13:03] calvin np thanks for helping as well, i'm glad we came to the same conclusion
[14:27] pieterh yo
[14:27] pieterh hi guys
[14:36] cremes good morning
[14:37] cremes so, it appears you liked my ZMQ_LEGO suggestion :)
[14:38] sundbp hi, has anyone had success using zeromq on jruby using the ffi-rzmq gem on a windows machine? I'm having issues with zmq_term hanging forever.
[14:40] cremes sundbp: i am just looking at it now
[14:40] cremes i use ffi-rzmq with jruby every day
[14:40] cremes but let me try and repro
[14:40] sundbp cremes: cool. on windows?
[14:40] cremes yes
[14:40] cremes win7 x64
[14:40] sundbp cremes: oh - great. there's hope
[14:41] sundbp cremes: i'm on a shitty win xp machine 32bit
[14:41] cremes hell, windows is the *reason* i wrote the gem!
[14:41] cremes :)
[14:41] sundbp cremes: thanks a lot for the effort so far :)
[14:42] sundbp cremes: i doubt it has anything to do with the acutal libzmq.dll - i used the one from the pyzmq MSI. also built it using the mingw devkit and had same hang on zmq_term
[14:43] sundbp cremes: if you happen to have a 2.1.9 dll that works for you around i'd be happy to give it a go with that to eliminate one possible point of failure. could email to
[14:43] mikko sundbp: i got 32bit dlls
[14:43] cremes i don't... i run with 2.1.7
[14:43] cremes 32-bit
[14:43] mikko
[14:43] sundbp cremes: i can give that a spin
[14:44] sundbp mikko: and that one :)
[14:44] mikko also, if zmq_term hangs make sure that all sockets are closed
[14:44] mikko and that you have set linger on the sockets
[14:45] mikko otherwise zmq_term will hang trying to flush messages
[14:45] pieterh cremes: I was thinking ZMQ_RAW
[14:46] pieterh LEGO is trademarked IMO
[14:46] pieterh we'd be banned in Germany
[14:46] cremes sundbp: it doesn't hang for me
[14:46] cremes pieterh_: too bad
[14:46] mikko ZMQ_PLAY_DOH
[14:46] cremes ZMQ_RAW isn't bad
[14:47] pieterh ZMQ_PUTTY
[14:47] cremes maybe you don't want the germans using 0mq... ?
[14:47] pieterh "raw 0MQ sockets" is a nice way of putting it but inaccurate
[14:47] sundbp mikko: this is the local_lat / remote_lat test
[14:48] cremes sundbp: in the examples i ship with the gem, it doesn't close the socket or try
[14:48] cremes to terminate the context
[14:48] sundbp cremes: would you mind emailing me your 2.1.7 dll known to behave correctly on your box?
[14:48] cremes sure
[14:48] sundbp cremes: the finalizer does
[14:50] cremes sundbp: finalizers are not guaranteed to run, particularly when the jvm is exiting
[14:50] cremes and i don't see it happen on mine
[14:50] cremes sundbp: dll is on its way
[14:50] sundbp cremes: ahh! i see. that's why it randomly works every once in ~10 times.
[14:50] sundbp cremes: will try adding some explicit closing
[14:51] cremes mine exits cleanly every time
[14:51] cremes never hangs
[14:56] sundbp cremes: adding explicit close and terminate not run by finalizer and it works as expected
[14:56] cremes sundbp: does it hang with 2.1.7?
[14:56] sundbp cremes: so it's probably that my machine is so slow and crappy that the finalizer timinigs are different from yours
[14:56] cremes could be
[14:56] sundbp cremes: will try the 2.1.7 as well without explicit close/terminate. 1min
[14:57] cremes i will patch the examples to explicitly close & terminate
[14:59] sundbp cremes: no cigar on 2.1.7 without explicit close/terminate
[14:59] sundbp cremes: same behavior of it "randomly" exiting cleanly via the finalizer and not
[15:00] cremes ok
[15:00] cremes i'll patch the examples
[15:00] cremes please close that jira for jruby
[15:00] sundbp cremes: will do
[15:00] sundbp cremes: thanks for hte finalizer tip. i knew that in the back of my mind but had forgotten
[15:05] sundbp cremes: excellent - then i'm back on track and zeromq is possible to use for my purposes. thanks for the gem
[15:05] cremes you're welcome
[15:05] cremes sundbp: new release coming soon that supports 3.0 api...
[15:06] sundbp cremes: yep, saw that. using it from github
[15:06] cremes sundbp: i wouldn't do that if i were you... use the last one i pushed to rubygems 0.8.2
[15:07] sundbp cremes: for my deployment purposes creating a version of the gem including the windows dll in ext that i put in my internal geminabox server. seems easiest way to deploy incl dll
[15:07] cremes ok, your call
[15:07] cremes the api for send_string and recv_string is going to be changing...
[15:07] sundbp cremes: sure, aware it's not a release. early stages on my side as well so not an issue for the moment.
[15:07] cremes so if you track the github release, it will break your stuff
[15:08] cremes ok
[15:08] sundbp cremes: btw, adding the ext library to the PATH when requiring the lib seems like a good idea. needed on windows, right? (at least for me)
[15:09] cremes the gem explicitly checks for the dll inside of itself
[15:09] cremes so i think that obviates the need for it to be on the PATH for windows
[15:09] cremes but i haven't tested it on windows yet so i don't know for sure
[15:09] sundbp cremes: nope, it still fails loading it if not on PATH. it does find it but fails loading
[15:09] cremes interesting
[15:10] cremes well, i'm accepting patches so if you figure out how to add it to the windows path, i'll take the code!
[15:10] sundbp cremes: it's a windows thing, something to do with the dynamic library loader
[15:10] sundbp cremes: sure, can make a patch
[15:10] cremes cool... make sure it *only* adds the gem path to PATH *if* the dll is found inside the gem
[15:11] cremes otherwise we don't want to be cluttering up people's PATHs with superfluous stuff
[15:11] sundbp cremes: agreed
[15:11] sundbp cremes: only on windows, and only if it exists
[15:11] cremes yes, please
[15:39] sundbp cremes: sorry, i'm wrong. it seems to load it properly without adding it to path. it didn't when i used the mingw compiled one since that depends also on libstdc_+_6.dll and libgcc-someting.dll. the clean msvc8 compiled one workd fine without referencing explicitly
[15:39] cremes sundbp: good to know!
[16:12] MattH I want to multiplex in across a few sockets but prioritize the first socket, anyone know an elegant way to handle this?
[16:13] Steve-o "multiplex in" ?
[16:14] Steve-o just use zmq_poll and prefer the ZMQ_POLLIN from one socket.
[16:15] MattH ok, i guess i can just check the ID of the socket on the poll in right?
[16:15] Steve-o yup
[16:15] MattH ok , that sounds easy
[16:15] Steve-o because you need to iterate through the poll set you can apply weighting
[16:15] MattH ok thanks
[16:23] MattH steve-o: Is there a way to check the socket to see if there is a message waiting to be read?
[16:24] Steve-o zmq_poll
[16:24] Steve-o >>
[16:25] Steve-o or are you looking for something diffferent?
[16:27] cremes MattH: careful with zmq_poll()... it's edge-triggered so it won't fire again for a specific socket until all messages in queue
[16:27] cremes have been read
[16:28] MattH I see, how would i prioritize a certain socket then? If i am stuck reading say Priority 3 and there is a backlog, how can i get back to reading Priority 1?
[16:29] Steve-o cremes: the man page says level triggered
[16:30] cremes MattH: Steve-o already described how... since you control how you iterate over poll items, you can prioritize
[16:30] cremes that socket
[16:31] cremes Steve-o: sustrik says "edge-triggered" all over the mailing list
[16:31] cremes and that's been my experience too, so the man page must be wrong
[16:31] cremes weird that such an error would have remained for so long
[16:35] Steve-o I always read edge-triggered and it should be pretty easy to work with too
[16:35] Steve-o just store a flag with each socket indicating the level
[16:50] Steve-o cremes: just checking the zmq_poll impl, the poll() version is level based
[16:51] Steve-o need to check further in
[16:51] cremes Steve-o: huh...
[16:52] cremes you know, it might be a misreading on my part where zmq_poll() is level-triggered but ZMQ_FD is edge-triggered
[16:52] cremes yep, that's what the docs say
[16:53] cremes so i was wrong about zmq_poll()... i was remembering the ZMQ_FD behavior
[16:54] MattH I guess i am stuck on one part: Lets say i have three sockets, Priority1, 2, and 3. Lets say there are messages to read on all sockets.
[16:55] Steve-o priority is a bit confusing though,
[16:55] MattH If i am currently reading from socket 3, how can i stop reading and move to socket 1?
[16:55] Steve-o do you want to read #1 twice for every #2 and #3 for example
[16:55] Steve-o or just always read #1 first when multiple events arrive
[16:56] MattH I want to read on socket 1 until there are no messages, move to socket 2 and read there (meanwhile checking socket 1), move to socket 3 and read there (meanwhile checking 1 and 2)
[16:57] MattH In other words, if there is a high priority socket with messages, then i want to move to read the highest priority ones
[16:57] Steve-o one example is you can use ZMQ_FD for each socket, poll on the set, when #1 triggers read until -1 returns, then continue onto lower priorities
[16:58] Steve-o otherwise call zmq_poll, check the set and read #1 until empty
[16:58] Steve-o or even interleave calls to zmq_poll and zmq_recv
[16:59] MattH ok, ill try those suggestions, thanks
[16:59] Steve-o I don't recall seeing performance numbers for investigating each route though
[17:00] Steve-o edge is supposed to be faster but it might not even be measurable for some cases
[17:05] minrk A possible scheme:
[17:05] minrk while True:
[17:05] minrk evts = poll(1,2,3)
[17:05] minrk if 1:
[17:05] minrk # flush all messages
[17:05] minrk recv_all(1)
[17:05] minrk # poll again, before moving on to 2,3
[17:05] minrk # to ensure 2 has priority over 3
[17:05] minrk break
[17:05] minrk if 2:
[17:05] minrk # recv one (multipart) message
[17:05] minrk recv(2)
[17:05] minrk # poll again, in case a message came to #1
[17:06] minrk which ensures that 1. can starve 2,3, and 2. can starve 3, but can't starve 1., etc.
[17:10] Steve-o minrk: I think that meets my option (2), technically (1) would be fastest as less syscalls included, (3) worst.
[17:11] Steve-o but I think zmq_fd is 0mq generated? that implies additional latency, perversely making the others better in some circumstances
[17:11] cremes MattH: you need a multilevel feedback priority queue so that the lower levels don't get starved
[23:06] CIA-79 jzmq: 03Jason Chown 07master * rfe0e870 10/ src/org/zeromq/ : Corrected poll() comments, timeouts are in milliseconds ...
[23:06] CIA-79 jzmq: 03Gonzalo Diethelm 07master * r9228406 10/ src/org/zeromq/ : Merge pull request #76 from jchown/master ...