ZeroMq IRC Log

Monday September 26, 2011

[Time] Name	Message
[11:21] mikko	sustrik: Assertion failed: ok (mailbox.cpp:79)
[11:21] mikko	is this a race to shutdown ?
[11:22] sustrik	mikko: hi
[11:22] mikko	2.1
[11:22] sustrik	let me see
[11:24] sustrik	yes, looks like a race condition
[12:03] CIA-79	libzmq: 03Martin Sustrik 07master * rd726120 10/ src/mtrie.cpp : Bug in matching algorithm fixed ...
[12:41] mikko	sustrik: did you see the another ticket i opened?
[12:42] sustrik	mikko: which one is that?
[12:44] mikko	it's in jira
[12:44] mikko	assertion in kevent
[12:44] mikko	LIBZMQ-261
[12:44] mikko	ah
[12:44] mikko	you answered
[12:45] mikko	odd that i didn't get an email
[12:45] mikko	you want access to freebsd?
[12:51] calvin	hi sustrik - i just responded to your question on jira
[12:52] calvin	both of us ended up at nearly the same solution
[12:52] calvin	so whichever you like you can use
[12:56] sustrik	mikko: if it's reproducible, it would definitely help
[12:56] sustrik	calvin_: hi
[12:57] sustrik	calvin_: so your patch consists of 2 patches?
[13:02] calvin	well originally i had that idea with the parent_trie_
[13:02] calvin	but then I realized it was much simpler so i reverted that and just did something similar to you
[13:02] sustrik	ah
[13:02] sustrik	it seems that pieter have mrged the first version to 3.0
[13:03] calvin	yeah i put in a pull request last night
[13:03] calvin	with the additional changes
[13:03] sustrik	ok, i see
[13:03] sustrik	we'll sort that out somehow
[13:03] sustrik	thanks for giving it a look btw
[13:03] calvin	np thanks for helping as well, i'm glad we came to the same conclusion
[14:27] pieterh	yo
[14:27] pieterh	hi guys
[14:36] cremes	good morning
[14:37] cremes	so, it appears you liked my ZMQ_LEGO suggestion :)
[14:38] sundbp	hi, has anyone had success using zeromq on jruby using the ffi-rzmq gem on a windows machine? I'm having issues with zmq_term hanging forever.
[14:40] cremes	sundbp: i am just looking at it now
[14:40] cremes	i use ffi-rzmq with jruby every day
[14:40] cremes	but let me try and repro
[14:40] sundbp	cremes: cool. on windows?
[14:40] cremes	yes
[14:40] cremes	win7 x64
[14:40] sundbp	cremes: oh - great. there's hope
[14:41] sundbp	cremes: i'm on a shitty win xp machine 32bit
[14:41] cremes	hell, windows is the reason i wrote the gem!
[14:41] cremes	:)
[14:41] sundbp	cremes: thanks a lot for the effort so far :)
[14:42] sundbp	cremes: i doubt it has anything to do with the acutal libzmq.dll - i used the one from the pyzmq MSI. also built it using the mingw devkit and had same hang on zmq_term
[14:43] sundbp	cremes: if you happen to have a 2.1.9 dll that works for you around i'd be happy to give it a go with that to eliminate one possible point of failure. could email to patrik.sundberg@gmail.com
[14:43] mikko	sundbp: i got 32bit dlls
[14:43] cremes	i don't... i run with 2.1.7
[14:43] cremes	32-bit
[14:43] mikko	http://snapshot.zero.mq/msvc2008/2011-09-26_05-19-59/
[14:43] sundbp	cremes: i can give that a spin
[14:44] sundbp	mikko: and that one :)
[14:44] mikko	also, if zmq_term hangs make sure that all sockets are closed
[14:44] mikko	and that you have set linger on the sockets
[14:45] mikko	otherwise zmq_term will hang trying to flush messages
[14:45] pieterh	cremes: I was thinking ZMQ_RAW
[14:46] pieterh	LEGO is trademarked IMO
[14:46] pieterh	we'd be banned in Germany
[14:46] cremes	sundbp: it doesn't hang for me
[14:46] cremes	pieterh_: too bad
[14:46] mikko	ZMQ_PLAY_DOH
[14:46] cremes	ZMQ_RAW isn't bad
[14:47] pieterh	ZMQ_PUTTY
[14:47] cremes	maybe you don't want the germans using 0mq... ?
[14:47] pieterh	"raw 0MQ sockets" is a nice way of putting it but inaccurate
[14:47] sundbp	mikko: this is the local_lat / remote_lat test
[14:48] cremes	sundbp: in the examples i ship with the gem, it doesn't close the socket or try
[14:48] cremes	to terminate the context
[14:48] sundbp	cremes: would you mind emailing me your 2.1.7 dll known to behave correctly on your box?
[14:48] cremes	sure
[14:48] sundbp	cremes: the finalizer does
[14:50] cremes	sundbp: finalizers are not guaranteed to run, particularly when the jvm is exiting
[14:50] cremes	and i don't see it happen on mine
[14:50] cremes	sundbp: dll is on its way
[14:50] sundbp	cremes: ahh! i see. that's why it randomly works every once in ~10 times.
[14:50] sundbp	cremes: will try adding some explicit closing
[14:51] cremes	mine exits cleanly every time
[14:51] cremes	never hangs
[14:56] sundbp	cremes: adding explicit close and terminate not run by finalizer and it works as expected
[14:56] cremes	sundbp: does it hang with 2.1.7?
[14:56] sundbp	cremes: so it's probably that my machine is so slow and crappy that the finalizer timinigs are different from yours
[14:56] cremes	could be
[14:56] sundbp	cremes: will try the 2.1.7 as well without explicit close/terminate. 1min
[14:57] cremes	i will patch the examples to explicitly close & terminate
[14:59] sundbp	cremes: no cigar on 2.1.7 without explicit close/terminate
[14:59] sundbp	cremes: same behavior of it "randomly" exiting cleanly via the finalizer and not
[15:00] cremes	ok
[15:00] cremes	i'll patch the examples
[15:00] cremes	please close that jira for jruby
[15:00] sundbp	cremes: will do
[15:00] sundbp	cremes: thanks for hte finalizer tip. i knew that in the back of my mind but had forgotten
[15:05] sundbp	cremes: excellent - then i'm back on track and zeromq is possible to use for my purposes. thanks for the gem
[15:05] cremes	you're welcome
[15:05] cremes	sundbp: new release coming soon that supports 3.0 api...
[15:06] sundbp	cremes: yep, saw that. using it from github
[15:06] cremes	sundbp: i wouldn't do that if i were you... use the last one i pushed to rubygems 0.8.2
[15:07] sundbp	cremes: for my deployment purposes creating a version of the gem including the windows dll in ext that i put in my internal geminabox server. seems easiest way to deploy incl dll
[15:07] cremes	ok, your call
[15:07] cremes	the api for send_string and recv_string is going to be changing...
[15:07] sundbp	cremes: sure, aware it's not a release. early stages on my side as well so not an issue for the moment.
[15:07] cremes	so if you track the github release, it will break your stuff
[15:08] cremes	ok
[15:08] sundbp	cremes: btw, adding the ext library to the PATH when requiring the lib seems like a good idea. needed on windows, right? (at least for me)
[15:09] cremes	the gem explicitly checks for the dll inside of itself
[15:09] cremes	so i think that obviates the need for it to be on the PATH for windows
[15:09] cremes	but i haven't tested it on windows yet so i don't know for sure
[15:09] sundbp	cremes: nope, it still fails loading it if not on PATH. it does find it but fails loading
[15:09] cremes	interesting
[15:10] cremes	well, i'm accepting patches so if you figure out how to add it to the windows path, i'll take the code!
[15:10] sundbp	cremes: it's a windows thing, something to do with the dynamic library loader
[15:10] sundbp	cremes: sure, can make a patch
[15:10] cremes	cool... make sure it only adds the gem path to PATH if the dll is found inside the gem
[15:11] cremes	otherwise we don't want to be cluttering up people's PATHs with superfluous stuff
[15:11] sundbp	cremes: agreed
[15:11] sundbp	cremes: only on windows, and only if it exists
[15:11] cremes	yes, please
[15:39] sundbp	cremes: sorry, i'm wrong. it seems to load it properly without adding it to path. it didn't when i used the mingw compiled one since that depends also on libstdc_+_6.dll and libgcc-someting.dll. the clean msvc8 compiled one workd fine without referencing explicitly
[15:39] cremes	sundbp: good to know!
[16:12] MattH	I want to multiplex in across a few sockets but prioritize the first socket, anyone know an elegant way to handle this?
[16:13] Steve-o	"multiplex in" ?
[16:14] Steve-o	just use zmq_poll and prefer the ZMQ_POLLIN from one socket.
[16:15] MattH	ok, i guess i can just check the ID of the socket on the poll in right?
[16:15] Steve-o	yup
[16:15] MattH	ok , that sounds easy
[16:15] Steve-o	because you need to iterate through the poll set you can apply weighting
[16:15] MattH	ok thanks
[16:23] MattH	steve-o: Is there a way to check the socket to see if there is a message waiting to be read?
[16:24] Steve-o	zmq_poll
[16:24] Steve-o	>> http://api.zeromq.org/2-1:zmq-poll
[16:25] Steve-o	or are you looking for something diffferent?
[16:27] cremes	MattH: careful with zmq_poll()... it's edge-triggered so it won't fire again for a specific socket until all messages in queue
[16:27] cremes	have been read
[16:28] MattH	I see, how would i prioritize a certain socket then? If i am stuck reading say Priority 3 and there is a backlog, how can i get back to reading Priority 1?
[16:29] Steve-o	cremes: the man page says level triggered
[16:30] cremes	MattH: Steve-o already described how... since you control how you iterate over poll items, you can prioritize
[16:30] cremes	that socket
[16:31] cremes	Steve-o: sustrik says "edge-triggered" all over the mailing list
[16:31] cremes	and that's been my experience too, so the man page must be wrong
[16:31] cremes	weird that such an error would have remained for so long
[16:35] Steve-o	I always read edge-triggered and it should be pretty easy to work with too
[16:35] Steve-o	just store a flag with each socket indicating the level
[16:50] Steve-o	cremes: just checking the zmq_poll impl, the poll() version is level based
[16:51] Steve-o	need to check further in
[16:51] cremes	Steve-o: huh...
[16:52] cremes	you know, it might be a misreading on my part where zmq_poll() is level-triggered but ZMQ_FD is edge-triggered
[16:52] cremes	yep, that's what the docs say
[16:53] cremes	so i was wrong about zmq_poll()... i was remembering the ZMQ_FD behavior
[16:54] MattH	I guess i am stuck on one part: Lets say i have three sockets, Priority1, 2, and 3. Lets say there are messages to read on all sockets.
[16:55] Steve-o	priority is a bit confusing though,
[16:55] MattH	If i am currently reading from socket 3, how can i stop reading and move to socket 1?
[16:55] Steve-o	do you want to read #1 twice for every #2 and #3 for example
[16:55] Steve-o	or just always read #1 first when multiple events arrive
[16:56] MattH	I want to read on socket 1 until there are no messages, move to socket 2 and read there (meanwhile checking socket 1), move to socket 3 and read there (meanwhile checking 1 and 2)
[16:57] MattH	In other words, if there is a high priority socket with messages, then i want to move to read the highest priority ones
[16:57] Steve-o	one example is you can use ZMQ_FD for each socket, poll on the set, when #1 triggers read until -1 returns, then continue onto lower priorities
[16:58] Steve-o	otherwise call zmq_poll, check the set and read #1 until empty
[16:58] Steve-o	or even interleave calls to zmq_poll and zmq_recv
[16:59] MattH	ok, ill try those suggestions, thanks
[16:59] Steve-o	I don't recall seeing performance numbers for investigating each route though
[17:00] Steve-o	edge is supposed to be faster but it might not even be measurable for some cases
[17:05] minrk	A possible scheme:
[17:05] minrk	while True:
[17:05] minrk	evts = poll(1,2,3)
[17:05] minrk	if 1:
[17:05] minrk	# flush all messages
[17:05] minrk	recv_all(1)
[17:05] minrk	# poll again, before moving on to 2,3
[17:05] minrk	# to ensure 2 has priority over 3
[17:05] minrk	break
[17:05] minrk	if 2:
[17:05] minrk	# recv one (multipart) message
[17:05] minrk	recv(2)
[17:05] minrk	# poll again, in case a message came to #1
[17:06] minrk	which ensures that 1. can starve 2,3, and 2. can starve 3, but can't starve 1., etc.
[17:10] Steve-o	minrk: I think that meets my option (2), technically (1) would be fastest as less syscalls included, (3) worst.
[17:11] Steve-o	but I think zmq_fd is 0mq generated? that implies additional latency, perversely making the others better in some circumstances
[17:11] cremes	MattH: you need a multilevel feedback priority queue so that the lower levels don't get starved
[23:06] CIA-79	jzmq: 03Jason Chown 07master * rfe0e870 10/ src/org/zeromq/ZMQ.java : Corrected poll() comments, timeouts are in milliseconds ...
[23:06] CIA-79	jzmq: 03Gonzalo Diethelm 07master * r9228406 10/ src/org/zeromq/ZMQ.java : Merge pull request #76 from jchown/master ...

ZeroMq Home

Monday September 26, 2011