ZeroMq IRC Log

Thursday January 13, 2011

[Time] Name	Message
[00:45] zchrish	Can one create a zmq::pollitem_t object dynamically? I am considering adding and subtracting PUSH patterns as subscribers come onboard.
[00:49] zchrish	For example, in my publisher, I have a PULL pattern that syncs my subscribers. I want to add dedicated PUSH patterns that can connect to each of those subscribers. I would get the connection information from a message from the subscriber to the publisher.
[00:51] zchrish	According to sustrik, using a vector for the pattern is supported (I think this is the result based on an earlier xChat post.
[02:41] jugg	sustrik, if the offer for access to the zeromq/erlzmq repo is still on the table, please ping me.
[07:39] benoitc	mikko: i'm not awer of one, except something i launched in the past but aborted cause erlzmq wasn't working so well
[07:39] benoitc	sustrik: did you see my pull request for rebar ?
[07:41] benoitc	how would you validate suscriptions of a client ?
[07:58] yrashk	another bizzare random zeromq segfault: https://gist.github.com/b5495c3f5dd29e193e9b I must be definitely doing something wrong
[09:02] sustrik	benoitc: i am not maintaining the erlzmq binding
[09:02] sustrik	try contacting serge aleynikov
[09:03] sustrik	serge@aleynikov.org
[09:03] sustrik	yrashk: what's the error?
[09:08] yrashk	sustrik: the one in the gist i mentioned
[09:08] sustrik	there's no error there
[09:10] jugg	benoitc, you may want to look at http://github.com/csrl/erlzmq for what I believe is a usable erlang binding
[09:10] sustrik	jugg: any answer from serge?
[09:11] yrashk	sustrik: its a segfault point inspected in gdb
[09:11] sustrik	ah, segfault
[09:11] jugg	sustrik, ? I think I may have missed something.
[09:12] jugg	I take it I should contact him. :)
[09:12] sustrik	doyrashk: can you provide a minimal test case?
[09:12] sustrik	jugg: well, try to ask him to pull the changes
[09:12] jugg	does he handle pull requests on zeromq/erlzmq?
[09:13] yrashk	sustrik: im trying but this segfault is only randomly reproducible :(
[09:13] sustrik	he did so a week ago
[09:13] jugg	k
[09:13] sustrik	so he's probably still maintaining the project
[09:13] sustrik	anyway, if he doesn't answer, ping me
[09:13] sustrik	i'll grant you commit access to the project
[09:13] jugg	ok
[09:14] sustrik	yrashk: maybe be using a loop in your test program?
[09:15] yrashk	sustrik: may be, i just have to narrow it down somehow
[09:15] sustrik	goodo
[09:15] yrashk	i have a pushing thread and a pulling thread
[09:15] yrashk	not sure how this can affect it
[09:15] yrashk	given i use ffn to deallocate stuff
[09:16] benoitc	sustrik: k
[09:19] benoitc	jugg: does it change the use of erlzmq ?
[09:20] benoitc	ah yes so you create one context / socket
[09:20] jugg	sustrik, to close the loop on an old discussion from a few weeks ago. The problems I was seeing with bogus blank messages seems to have gone away with the updated erlang bindings, and I believe I have an explanation as to what was causing. As such I don't believe the issue exists but is masked, but that the issue was in the bindings.
[09:21] jugg	benoitc, no, there are multiple sockets per context.
[09:21] jugg	There is one erlang process per socket.
[09:21] sustrik	jugg: great
[09:21] benoitc	right parsing pursub server ex
[09:22] jugg	the main difference is that zmq:start_link no longer exists (which used to initialize a zmq context). Now you must call zmq:init/1 instead to establish a context. And you must pass the context to zmq:socket/2. Also, the returned socket from zmq:socket/2 must be passed around.
[09:23] benoitc	mmm ok
[09:23] benoitc	would be good if the change happen soon, so i could test it on couch_zmq
[09:23] jugg	basically you can treat the erlang bindings very much like the C bindings. See: http://csrl.github.com/erlzmq/ for documented differences.
[09:24] jugg	(under architecture)
[09:25] benoitc	jugg: i will provide a branch using rebar with that in coming hour
[09:25] jugg	however, if you want low level control, you can utilize the zmq_drv:xxx functions directly, however there are certain constraints to respect. I don't have those documented yet.
[09:26] benoitc	cool
[09:26] jugg	benoitc, great, I saw your rebar update and was waiting to see if my fork would get merged before integrating.
[09:26] benoitc	i'm trying to only se rebar these days (hopefully it will be integrated in erlang)
[09:26] benoitc	so I need it
[09:31] sustrik	jugg: btw, i like the idea of erlzmq looking more alike to C API and other bindings
[09:32] jugg	things started coming together better once I took that approach.
[09:33] sustrik	nice
[09:36] jugg	ok, pull request sent.
[09:36] yrashk	jugg: how stable erlzmq now? I have another project in the pipeline and I am again trying to choose between zeromq and rabbitmq (I used both, although more of rabbitmq)
[09:36] yrashk	is now*
[09:37] jugg	fairly. I haven't tested all of the get/setsockopt however, so I might have a bug in there - hopefully not. And it'll be going into production use within two weeks here.
[09:37] yrashk	oh nice
[09:37] mikko	sustrik: im wondering that do we currently provide a way to remove subscription?
[09:38] mikko	sustrik: im thinking about internet like usage scenario where subscribers with identities might come and go
[09:50] sustrik	mikko: ZMQ_UNSUBSCRIBE
[10:08] benoitc	jugg: i've an error while building your version :
[10:08] benoitc	c_src/zmq_drv.cpp: In function âvoid zmqdrv_ready_input(_erl_drv_data, _erl_drv_event)â:
[10:08] benoitc	c_src/zmq_drv.cpp:939: error: cast from â_erl_drv_event*â to âintâ loses precision
[10:08] benoitc	mmm maybe I miss a flag
[10:11] mikko	sustrik: i meant from server side
[10:12] mikko	i've never used PUB/SUB with identities
[10:12] mikko	but i understand if subscriber with identity goes away the messages are buffered
[10:12] mikko	are they buffered up to HWM?
[10:12] sustrik	yes
[10:12] sustrik	basically, identity defines a "permanent connection"
[10:13] sustrik	one that remains even though reconnect happens, one application is shut down or alike
[10:13] sustrik	then there's orthogonal issue of how many massages can be buffered
[10:13] sustrik	that's HWM and SWAP
[10:13] mikko	but it's good that HWM limits it
[10:14] sustrik	yes, it's orthogonal
[10:14] mikko	i was thinking about untrusted subscribers creating a resource exchaustion
[10:14] sustrik	it can still happen
[10:14] sustrik	it there's a lot of them
[10:14] mikko	yes, i guess so
[10:14] sustrik	is it wise to ude identities?
[10:15] sustrik	use
[10:15] sustrik	seb': kind of
[10:15] mikko	i guess a sockopt disabling permanent connections even if client requests might be good
[10:15] mikko	if clients are untrusted it might prove to be useful
[10:15] sustrik	seb': but the connection is not persistent in the sense the resources are deallocated once the connection breaks
[10:44] CIA-21	zeromq2: 03Martin Sustrik 07master * r18f29de 10/ (52 files):
[10:44] CIA-21	zeromq2: Make cppcheck not complain about "'operator=' should return something"
[10:44] CIA-21	zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/hUiXv8
[10:45] mikko	heh, i wonder other static analysis tools are there
[10:45] mikko	currently it runs cppcheck, duplicate code check and soon sloccount
[10:50] sustrik	let me first fix these :)
[10:51] sustrik	mikko: btw, is it possible to switch off certain types of errors?
[10:51] mikko	sustrik: i can create a suppressions file
[10:51] mikko	which ones specifically?
[10:51] sustrik	Redundant condition. It is safe to deallocate a NULL pointer
[10:51] sustrik	The code is redundant but it improves readability
[10:52] sustrik	so I would prefer to leave it in
[10:52] sustrik	cppCheckId= redundantIfDelete0
[10:53] mikko	ok, i'll try adding suppressions file
[10:53] mikko	just a sec
[11:19] jugg	benoitc, 32bit or 64bit?
[11:22] jugg	benoitc, looks like for some reason I changed the cast from a long to an int. I'll take a look and see if I can figure out why.
[11:29] mikko	sustrik: http://build.valokuva.org/job/ZeroMQ2-core-master_static-analysis/12/cppcheckResult/?
[11:30] mikko	suppressed now
[11:31] mikko	still a couple of redundant ones but fixing later
[11:34] sustrik	mikko: strange that it is analysing the system header files
[11:34] sustrik	like /usr/include/stdlib.h
[11:36] mikko	yeah, i removed those
[11:36] mikko	http://build.valokuva.org/job/ZeroMQ2-core-master_static-analysis/14/cppcheckResult/?
[11:46] sustrik	great
[11:46] sustrik	i have a look at remaining problems later on
[11:46] mikko	let me know if more suppressions are needed
[11:47] jugg	benoitc, It is because zeromq returns an 'int' for that value, so the conversion is harmless, but the warning should be suppressed.
[11:53] benoitc	jugg: sound like there are other error
[11:53] benoitc	yes 64 bits
[11:53] benoitc	jugg: there is also a problem l 446 and 820
[11:53] benoitc	int e = (uintptr_t)event;
[11:53] benoitc	solve the first one
[11:54] benoitc	on 446 i've to assert((sizeof(uint8_t)+ n+1) == size);
[11:54] benoitc	but my C is really rosten
[11:54] benoitc	rusty
[12:00] jugg	hmm, that 446 error is strange. I don't yet understand why your change would be correct.
[12:01] jugg	size should be equal to n + 1
[12:01] jugg	sizeof(uint8_t) == 1
[12:01] jugg	so, sizeof(uint8_t) + n should be correct...
[12:04] zchrish	I have a use case and am wondering whether zeromq is the right implementation method :
[12:04] jugg	benoitc, what is the error on 820?
[12:04] zchrish	Actually will send to the discussion group; sorry.
[12:07] jugg	benoitc, I believe I've understood the problem at 446. Verifying...
[12:13] jugg	yes, 'n' is not byte size, it is the number of options. So what is being asserted itself is bogus. I'm making a fix and will push it shortly.
[12:38] jugg	benoitc, I have the fixes in place for the first two issues. Is there anything to fix for line 820?
[12:57] benoitc	jugg: testing
[13:00] benoitc	jugg: new error c_src/zmq_drv.cpp: In function âvoid wrap_zmq_setsockopt(zmq_drv_t, const uint8_t, size_t)â:
[13:00] benoitc	c_src/zmq_drv.cpp:492: error: âoptvalenâ was not declared in this scope
[13:00] jugg	eh
[13:01] jugg	wow, how is my compiler not giving me an error there?
[13:02] benoitc	-Wl isn't an option on osx
[13:02] benoitc	ah yes a typo
[13:03] benoitc	and now have a new warning c_src/zmq_drv.cpp:492: warning: comparison between signed and unsigned integer expressions
[13:04] jugg	ah NDEBUG is defined in my build environment... no wonder I'm not finding these problems.
[13:04] benoitc	heh
[13:05] jugg	sorry... too many things going at once; missing these types of details.
[13:05] benoitc	np
[13:05] benoitc	that' where opensource helps :)
[13:05] jugg	quite so :)
[13:08] jugg	there we go... now I'm getting the errors.
[13:09] benoitc	i'm not sure it is, but cool :-)
[13:20] jugg	fix pushed
[13:22] benoitc	one last sorry :(
[13:22] benoitc	10> Assertion failed: (sizeof(uint32_t) == size), function wrap_zmq_poll, file c_src/zmq_drv.cpp, line 821. Abort trap
[13:22] jugg	I guess I should re-run my tests now that the asserts are actually
[13:22] benoitc	the one I had with my fixes
[13:23] benoitc	jugg: btw rebared version https://github.com/benoitc/erlzmq/tree/rebar-socketprocess
[13:24] benoitc	the pubtest is ok here
[13:24] benoitc	only repserver hangs
[13:29] jugg	yes, I have another bug there. I'll have to rerun my tests now that asserts are actually being checked.
[13:30] benoitc	k i will concentrate on app level for now
[13:34] efhache	mikko: hi do you have seen how i can configure zmq 2.0.7 to work under linux 2.4 without epooll ?
[13:35] mikko	efhache: take a look at src/poller.h
[13:35] mikko	i reckon you could change ZMQ_HAVE_LINUX condition to use poll instead of epoll
[13:35] mikko	not sure if that's the only thing you need to do
[13:35] mikko	maybe we could add a configure flag in the future to choose poller implementation
[13:37] efhache	so the define become : elif defined ZMQ_HAVE_LINUX typedef epoll_t poller_t; // fhn modification 20110113 epoll_t poller_t;
[13:37] efhache	are you ok with me?
[13:37] mikko	no
[13:38] efhache	sorry typedef : poll_t poller_t;
[13:38] mikko	yes
[13:38] mikko	try that
[13:38] efhache	yes sorry wrong copy past ;-)
[13:39] sustrik	efhache: you can use ZMQ_FORCE_POLL or somesuch
[13:40] mikko	did we have that in 2.0.7 ?
[13:40] mikko	oh
[13:41] mikko	it is there
[13:41] mikko	so no modification should be needed
[13:41] mikko	export CPPFLAGS="-DZMQ_FORCE_POLL" ./configure
[13:45] efhache	ah sorry I've modified source... it recompile it, I'll test and come back... if ok I'll retry with the flag
[13:45] efhache	thanks a lot and SYL ;-)
[13:46] mikko	you should really look into making the software work with 2.1.0 as well
[13:46] mikko	there aren't that many modifications needed usually
[13:52] efhache	mmhh I''ve begun with 2.0.6 and then update to 2.0.7 thath normally "no many modification" but in result I've rework all the code... So now I fix it and later I'll update the library if need it
[14:02] efhache	mikko: it does not work more, i've test with poller.h modified and obtain another problem : "Assertion failed: !(it->revents & POLLNVAL) (poll.cpp:171)"
[14:02] efhache	with the export of the FORCE POLL FLAG I reobtain the assetion with epoll.cpp as yesterday
[14:02] mikko	im not sure if anyone has really tested extensively with 2.4 kernel
[14:11] jugg	is there still plans for a zeromq conference in Budapest coming up?
[14:12] sustrik	pieter was organising it
[14:12] sustrik	but he got ill
[14:12] sustrik	he's out of hospital for only a few days now
[14:13] sustrik	so, hard to say
[14:13] sustrik	depends on when he gets ok
[14:14] jugg	He's recovering well?
[14:16] benoitc	jugg: i'm not sure why assertion is here line 821. on 64 bits it's always false.
[14:16] benoitc	removing it, pubrep works of course ..
[14:16] sustrik	i believe so
[14:17] jugg	I know why. I'll have a fix soon. You can modify src/zmq_drv.erl line: 190 to be: <<(?ZMQ_POLL):8, (events_to_int(Events)):32>>
[14:20] jugg	actually, I might change poll/send/recv flags to all be a byte size instead.
[14:21] jugg	yes, that makes more sense
[14:21] benoitc	k
[14:21] benoitc	need to refresh my C
[14:22] jugg	so instead, change the assert in poll to: assert(sizeof(uint8_t) == size);
[14:23] benoitc	works
[14:25] efhache	mikko: do you know if for zmq there is a use of "EPOLLET" (triggered mode) ??
[14:27] yawn	is there a way to determine the underlying zmq version in jzmq?
[14:42] benoitc	jugg: just did https://github.com/benoitc/erlzmq/commit/463d727dd90d7da94047e5cdb5b231672829d116 but anyway will wait your final fix
[14:42] benoitc	rebar is working well on erlzmq btw
[14:43] jugg	ok, almost ready.
[14:45] benoitc	cool :)
[14:48] mikko	efhache: i dont know what EPOLLET means
[14:48] mikko	yawn: not sure if it's exposed
[14:49] mikko	yawn: ZMQ::version_full() ?
[14:49] yawn	mikko: not in my version - maybe an old build ...
[14:50] mikko	https://github.com/zeromq/jzmq/blob/master/src/ZMQ.cpp#L30
[14:50] mikko	at least that's what the source says
[14:50] sustrik	mikko: it's edge-triggered
[14:51] sustrik	efhache: the underlying file descriptor (ZMQ_FD) is edge-triggered
[14:51] mikko	sustrik: EPOLLET stands for edge-triggered?
[14:51] sustrik	i think so
[14:51] sustrik	let me check
[14:52] sustrik	yes
[14:54] mikko	ok
[14:59] efhache	mikko: I've resolved the problem, I've used the export CPPFLAGS="-DZMQ_FORCE_SELECT"
[15:03] mikko	efhache: cool
[15:04] mikko	sustrik: i guess that makes sense if it standa for EPOLL Egde-Triggered
[15:04] yawn	mikko: protected statics
[15:04] yawn	mikko: one needs to do the usual dance & sing to make java return the values
[15:25] Hybelkanin	Anyone with experience on zmq on osx 10.6 with the C# bindings?
[15:27] mikko	Hybelkanin: unusual combination
[15:27] mikko	can't say i've heard that combo before
[15:27] Hybelkanin	:P
[15:27] mikko	i have built and tested the c# bindings on linux
[15:28] Hybelkanin	I think my problem may be monodevelop
[15:29] Hybelkanin	So this might be the wrong channel
[15:43] amacleod	My Python 0MQ REP server doesn't stop when I interrupt it with ^C. How do I make it respect ^C?
[15:49] s0undt3ch	ppl, pub/sub forward devices should connect_in and bind_out?
[15:49] s0undt3ch	or should they bind_in and bind_out?
[16:01] mikko	amacleod: i would i think it might be blocking on zmq_term
[16:02] mikko	amacleod: i dont really know how things are implemented in python so its just guessing
[16:02] amacleod	mikko, I wrote it like the examples with a "while True" loop.
[16:05] amacleod	I can stop it from within just fine--if I set up a conditional where my message handler calls self.context.term () when it receives "EXIT" or something, it's okay.
[16:05] mikko	amacleod: can you see where it's stuck?
[16:05] amacleod	As far as I know, it's not stuck. It's just completely ignoring ^C or SIGTERM.
[16:06] cremes	amacleod: if it's blocked in zmq_term(), it won't let python's signal handler run to catch the signal
[16:06] cremes	this has been reported a few times; i think sustrik is planning a fix
[16:06] cremes	same thing happens with the ruby runtime
[16:07] amacleod	Oh, interesting.
[16:07] amacleod	If I hit ^C and then try sending something to it, it does get the KeyboardInterrupt exception.
[16:07] amacleod	So it's actually probably blocking in recv()
[16:07] cremes	huh, that might be a new problem
[16:07] cremes	if you can reduce it, file an issue
[16:08] mikko	recv should return on ctrl+c
[16:20] s0undt3ch	sustrik, guido_g: I found out the issue about my forwarder problem
[16:21] s0undt3ch	sustrik, guido_g: publishers were binding instead of connecting
[16:21] s0undt3ch	sustrik, guido_g: the forwarder device was the one binding so, no one else should bind
[16:29] jugg	benoitc, updates pushed. All get/setsockopt options are tested now.
[16:38] benoitc	jugg: works
[16:38] benoitc	thanks :)
[16:39] benoitc	smth weird though when you launch pubsub test
[16:39] benoitc	10 is sent but none of subclient get it
[16:40] benoitc	jugg: adding a timer after the bind fix that but ...
[16:42] benoitc	anyway https://github.com/benoitc/erlzmq/tree/rebar-socketprocess
[16:44] jugg	yes, that is normal - not a binding issue.
[16:44] benoitc	ok
[16:45] benoitc	didn't check but did you add PUSH and PULL
[16:45] jugg	yes
[16:45] benoitc	or DOWNSTREAM, UPSTREAM are current names ?
[16:45] benoitc	cool
[16:45] jugg	using your rebar, I always get errors: fatal: No tags can describe '6944b664b20b525e210f1eaebea00bbcd21ec217'
[16:45] benoitc	ok time to code this app then. more in the night
[16:45] jugg	when doing $ make
[16:46] benoitc	jugg: you have to tag
[16:46] jugg	just to build ??
[16:46] benoitc	i can remove it though
[16:46] benoitc	jugg: this is for archive creation
[16:46] jugg	yes, I build much more often than I tag
[16:46] jugg	no, that happens when doing $ make
[16:46] benoitc	let me edit this Makefile :)
[16:46] jugg	k :)
[16:46] benoitc	you could just do ./rebar compile anyway
[16:52] amacleod	Should I file my test case as an attachment to https://github.com/zeromq/zeromq2/issues/#issue/127 , or start a new issue?
[16:52] mikko	amacleod: is it python?
[16:53] amacleod	mikko, yes, Python, not Ruby.
[16:53] mikko	open it under https://github.com/zeromq/pyzmq
[16:58] benoitc	jugg: fixed and pushed
[17:07] jugg	benoitc, $git rebase -i can be your friend... :)
[17:14] benoitc	mmm ?
[17:15] benoitc	I've used rebase -i
[17:15] benoitc	just cherry-picking last update
[17:15] jugg	why not just rebase your commits ontop of the last update?
[17:16] benoitc	in rebar-socketprocess
[17:16] benoitc	dunno
[17:16] benoitc	rebasing now
[17:21] benoitc	jugg: done
[17:25] benoitc	win 6
[18:14] amacleod	Bleh. Didn't realize pyzmq master won't build against zmq 2.1.0.
[18:19] benoitc	mmm it is
[18:19] benoitc	at least it does
[18:20] amacleod	Do I need to delete my previous installation of pyzmq in order to make use of the development version?
[18:20] benoitc	i've installed it in my virtualenv
[18:22] amacleod	I must be doing something wrong. I built and installed pyzmq master, and when I try to import zmq, it gets an error, 'cause it can't import initthreads from zmq.util
[18:23] benoitc	did you change paths in setup.cfg?
[18:26] amacleod	Ah, nope. That must be it.
[18:26] amacleod	Does it not use /usr/local/lib and /usr/local/include by default?
[18:33] benoitc	not sure
[18:45] amacleod	The weird thing is that everything seems to get installed just fine. It's only when I go to import zmq that the error occurs.
[19:50] mikko	sustrik: https://github.com/danmar/cppcheck/commit/36c18072280d71e49d4d2a38d011666f792d6283
[19:50] mikko	wrong one
[19:51] mikko	maybe not, not sure if that affects the one you reported
[19:52] sustrik	looks like
[19:53] mikko	testing now
[19:54] mikko	nope, still see those reported
[22:56] yrashk	can push socket be shared across threads?
[22:56] yrashk	or is it unsafe to do so?
[23:29] yrashk	if zmq_send with ZMQ_NOBLOCK returns EAGAIN, do I have to reinitialize the message?