Thursday January 13, 2011

[Time] NameMessage
[00:45] zchrish Can one create a zmq::pollitem_t object dynamically? I am considering adding and subtracting PUSH patterns as subscribers come onboard.
[00:49] zchrish For example, in my publisher, I have a PULL pattern that syncs my subscribers. I want to add dedicated PUSH patterns that can connect to each of those subscribers. I would get the connection information from a message from the subscriber to the publisher.
[00:51] zchrish According to sustrik, using a vector for the pattern is supported (I think this is the result based on an earlier xChat post.
[02:41] jugg sustrik, if the offer for access to the zeromq/erlzmq repo is still on the table, please ping me.
[07:39] benoitc mikko: i'm not awer of one, except something i launched in the past but aborted cause erlzmq wasn't working so well
[07:39] benoitc sustrik: did you see my pull request for rebar ?
[07:41] benoitc how would you validate suscriptions of a client ?
[07:58] yrashk another bizzare random zeromq segfault: I must be definitely doing something wrong
[09:02] sustrik benoitc: i am not maintaining the erlzmq binding
[09:02] sustrik try contacting serge aleynikov
[09:03] sustrik
[09:03] sustrik yrashk: what's the error?
[09:08] yrashk sustrik: the one in the gist i mentioned
[09:08] sustrik there's no error there
[09:10] jugg benoitc, you may want to look at for what I believe is a usable erlang binding
[09:10] sustrik jugg: any answer from serge?
[09:11] yrashk sustrik: its a segfault point inspected in gdb
[09:11] sustrik ah, segfault
[09:11] jugg sustrik, ? I think I may have missed something.
[09:12] jugg I take it I should contact him. :)
[09:12] sustrik doyrashk: can you provide a minimal test case?
[09:12] sustrik jugg: well, try to ask him to pull the changes
[09:12] jugg does he handle pull requests on zeromq/erlzmq?
[09:13] yrashk sustrik: im trying but this segfault is only randomly reproducible :(
[09:13] sustrik he did so a week ago
[09:13] jugg k
[09:13] sustrik so he's probably still maintaining the project
[09:13] sustrik anyway, if he doesn't answer, ping me
[09:13] sustrik i'll grant you commit access to the project
[09:13] jugg ok
[09:14] sustrik yrashk: maybe be using a loop in your test program?
[09:15] yrashk sustrik: may be, i just have to narrow it down somehow
[09:15] sustrik goodo
[09:15] yrashk i have a pushing thread and a pulling thread
[09:15] yrashk not sure how this can affect it
[09:15] yrashk given i use ffn to deallocate stuff
[09:16] benoitc sustrik: k
[09:19] benoitc jugg: does it change the use of erlzmq ?
[09:20] benoitc ah yes so you create one context / socket
[09:20] jugg sustrik, to close the loop on an old discussion from a few weeks ago. The problems I was seeing with bogus blank messages seems to have gone away with the updated erlang bindings, and I believe I have an explanation as to what was causing. As such I don't believe the issue exists but is masked, but that the issue was in the bindings.
[09:21] jugg benoitc, no, there are multiple sockets per context.
[09:21] jugg There is one erlang process per socket.
[09:21] sustrik jugg: great
[09:21] benoitc right parsing pursub server ex
[09:22] jugg the main difference is that zmq:start_link no longer exists (which used to initialize a zmq context). Now you must call zmq:init/1 instead to establish a context. And you must pass the context to zmq:socket/2. Also, the returned socket from zmq:socket/2 must be passed around.
[09:23] benoitc mmm ok
[09:23] benoitc would be good if the change happen soon, so i could test it on couch_zmq
[09:23] jugg basically you can treat the erlang bindings very much like the C bindings. See: for documented differences.
[09:24] jugg (under architecture)
[09:25] benoitc jugg: i will provide a branch using rebar with that in coming hour
[09:25] jugg however, if you want low level control, you can utilize the zmq_drv:xxx functions directly, however there are certain constraints to respect. I don't have those documented yet.
[09:26] benoitc cool
[09:26] jugg benoitc, great, I saw your rebar update and was waiting to see if my fork would get merged before integrating.
[09:26] benoitc i'm trying to only se rebar these days (hopefully it will be integrated in erlang)
[09:26] benoitc so I need it
[09:31] sustrik jugg: btw, i like the idea of erlzmq looking more alike to C API and other bindings
[09:32] jugg things started coming together better once I took that approach.
[09:33] sustrik nice
[09:36] jugg ok, pull request sent.
[09:36] yrashk jugg: how stable erlzmq now? I have another project in the pipeline and I am again trying to choose between zeromq and rabbitmq (I used both, although more of rabbitmq)
[09:36] yrashk is now*
[09:37] jugg fairly. I haven't tested all of the get/setsockopt however, so I might have a bug in there - hopefully not. And it'll be going into production use within two weeks here.
[09:37] yrashk oh nice
[09:37] mikko sustrik: im wondering that do we currently provide a way to remove subscription?
[09:38] mikko sustrik: im thinking about internet like usage scenario where subscribers with identities might come and go
[09:50] sustrik mikko: ZMQ_UNSUBSCRIBE
[10:08] benoitc jugg: i've an error while building your version :
[10:08] benoitc c_src/zmq_drv.cpp: In function ‘void zmqdrv_ready_input(_erl_drv_data*, _erl_drv_event*)’:
[10:08] benoitc c_src/zmq_drv.cpp:939: error: cast from ‘_erl_drv_event*’ to ‘int’ loses precision
[10:08] benoitc mmm maybe I miss a flag
[10:11] mikko sustrik: i meant from server side
[10:12] mikko i've never used PUB/SUB with identities
[10:12] mikko but i understand if subscriber with identity goes away the messages are buffered
[10:12] mikko are they buffered up to HWM?
[10:12] sustrik yes
[10:12] sustrik basically, identity defines a "permanent connection"
[10:13] sustrik one that remains even though reconnect happens, one application is shut down or alike
[10:13] sustrik then there's orthogonal issue of how many massages can be buffered
[10:13] sustrik that's HWM and SWAP
[10:13] mikko but it's good that HWM limits it
[10:14] sustrik yes, it's orthogonal
[10:14] mikko i was thinking about untrusted subscribers creating a resource exchaustion
[10:14] sustrik it can still happen
[10:14] sustrik it there's a lot of them
[10:14] mikko yes, i guess so
[10:14] sustrik is it wise to ude identities?
[10:15] sustrik use
[10:15] sustrik seb': kind of
[10:15] mikko i guess a sockopt disabling permanent connections even if client requests might be good
[10:15] mikko if clients are untrusted it might prove to be useful
[10:15] sustrik seb': but the connection is not persistent in the sense the resources are deallocated once the connection breaks
[10:44] CIA-21 zeromq2: 03Martin Sustrik 07master * r18f29de 10/ (52 files):
[10:44] CIA-21 zeromq2: Make cppcheck not complain about "'operator=' should return something"
[10:44] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[10:45] mikko heh, i wonder other static analysis tools are there
[10:45] mikko currently it runs cppcheck, duplicate code check and soon sloccount
[10:50] sustrik let me first fix these :)
[10:51] sustrik mikko: btw, is it possible to switch off certain types of errors?
[10:51] mikko sustrik: i can create a suppressions file
[10:51] mikko which ones specifically?
[10:51] sustrik Redundant condition. It is safe to deallocate a NULL pointer
[10:51] sustrik The code is redundant but it improves readability
[10:52] sustrik so I would prefer to leave it in
[10:52] sustrik cppCheckId= redundantIfDelete0
[10:53] mikko ok, i'll try adding suppressions file
[10:53] mikko just a sec
[11:19] jugg benoitc, 32bit or 64bit?
[11:22] jugg benoitc, looks like for some reason I changed the cast from a long to an int. I'll take a look and see if I can figure out why.
[11:29] mikko sustrik:
[11:30] mikko suppressed now
[11:31] mikko still a couple of redundant ones but fixing later
[11:34] sustrik mikko: strange that it is analysing the system header files
[11:34] sustrik like /usr/include/stdlib.h
[11:36] mikko yeah, i removed those
[11:36] mikko
[11:46] sustrik great
[11:46] sustrik i have a look at remaining problems later on
[11:46] mikko let me know if more suppressions are needed
[11:47] jugg benoitc, It is because zeromq returns an 'int' for that value, so the conversion is harmless, but the warning should be suppressed.
[11:53] benoitc jugg: sound like there are other error
[11:53] benoitc yes 64 bits
[11:53] benoitc jugg: there is also a problem l 446 and 820
[11:53] benoitc int e = (uintptr_t)event;
[11:53] benoitc solve the first one
[11:54] benoitc on 446 i've to assert((sizeof(uint8_t)+ n+1) == size);
[11:54] benoitc but my C is really rosten
[11:54] benoitc rusty
[12:00] jugg hmm, that 446 error is strange. I don't yet understand why your change would be correct.
[12:01] jugg size should be equal to n + 1
[12:01] jugg sizeof(uint8_t) == 1
[12:01] jugg so, sizeof(uint8_t) + n should be correct...
[12:04] zchrish I have a use case and am wondering whether zeromq is the right implementation method :
[12:04] jugg benoitc, what is the error on 820?
[12:04] zchrish Actually will send to the discussion group; sorry.
[12:07] jugg benoitc, I believe I've understood the problem at 446. Verifying...
[12:13] jugg yes, 'n' is not byte size, it is the number of options. So what is being asserted itself is bogus. I'm making a fix and will push it shortly.
[12:38] jugg benoitc, I have the fixes in place for the first two issues. Is there anything to fix for line 820?
[12:57] benoitc jugg: testing
[13:00] benoitc jugg: new error c_src/zmq_drv.cpp: In function ‘void wrap_zmq_setsockopt(zmq_drv_t*, const uint8_t*, size_t)’:
[13:00] benoitc c_src/zmq_drv.cpp:492: error: ‘optvalen’ was not declared in this scope
[13:00] jugg eh
[13:01] jugg wow, how is my compiler not giving me an error there?
[13:02] benoitc -Wl isn't an option on osx
[13:02] benoitc ah yes a typo
[13:03] benoitc and now have a new warning c_src/zmq_drv.cpp:492: warning: comparison between signed and unsigned integer expressions
[13:04] jugg ah NDEBUG is defined in my build environment... no wonder I'm not finding these problems.
[13:04] benoitc heh
[13:05] jugg sorry... too many things going at once; missing these types of details.
[13:05] benoitc np
[13:05] benoitc that' where opensource helps :)
[13:05] jugg quite so :)
[13:08] jugg there we go... now I'm getting the errors.
[13:09] benoitc i'm not sure it is, but cool :-)
[13:20] jugg fix pushed
[13:22] benoitc one last sorry :(
[13:22] benoitc 10> Assertion failed: (sizeof(uint32_t) == size), function wrap_zmq_poll, file c_src/zmq_drv.cpp, line 821. Abort trap
[13:22] jugg I guess I should re-run my tests now that the asserts are actually
[13:22] benoitc the one I had with my fixes
[13:23] benoitc jugg: btw rebared version
[13:24] benoitc the pubtest is ok here
[13:24] benoitc only repserver hangs
[13:29] jugg yes, I have another bug there. I'll have to rerun my tests now that asserts are actually being checked.
[13:30] benoitc k i will concentrate on app level for now
[13:34] efhache mikko: hi do you have seen how i can configure zmq 2.0.7 to work under linux 2.4 without epooll ?
[13:35] mikko efhache: take a look at src/poller.h
[13:35] mikko i reckon you could change ZMQ_HAVE_LINUX condition to use poll instead of epoll
[13:35] mikko not sure if that's the only thing you need to do
[13:35] mikko maybe we could add a configure flag in the future to choose poller implementation
[13:37] efhache so the define become : elif defined ZMQ_HAVE_LINUX typedef epoll_t poller_t; // fhn modification 20110113 epoll_t poller_t;
[13:37] efhache are you ok with me?
[13:37] mikko no
[13:38] efhache sorry typedef : poll_t poller_t;
[13:38] mikko yes
[13:38] mikko try that
[13:38] efhache yes sorry wrong copy past ;-)
[13:39] sustrik efhache: you can use ZMQ_FORCE_POLL or somesuch
[13:40] mikko did we have that in 2.0.7 ?
[13:40] mikko oh
[13:41] mikko it is there
[13:41] mikko so no modification should be needed
[13:41] mikko export CPPFLAGS="-DZMQ_FORCE_POLL" ./configure
[13:45] efhache ah sorry I've modified source... it recompile it, I'll test and come back... if ok I'll retry with the flag
[13:45] efhache thanks a lot and SYL ;-)
[13:46] mikko you should really look into making the software work with 2.1.0 as well
[13:46] mikko there aren't that many modifications needed usually
[13:52] efhache mmhh I''ve begun with 2.0.6 and then update to 2.0.7 thath normally "no many modification" but in result I've rework all the code... So now I fix it and later I'll update the library if need it
[14:02] efhache mikko: it does not work more, i've test with poller.h modified and obtain another problem : "Assertion failed: !(it->revents & POLLNVAL) (poll.cpp:171)"
[14:02] efhache with the export of the FORCE POLL FLAG I reobtain the assetion with epoll.cpp as yesterday
[14:02] mikko im not sure if anyone has really tested extensively with 2.4 kernel
[14:11] jugg is there still plans for a zeromq conference in Budapest coming up?
[14:12] sustrik pieter was organising it
[14:12] sustrik but he got ill
[14:12] sustrik he's out of hospital for only a few days now
[14:13] sustrik so, hard to say
[14:13] sustrik depends on when he gets ok
[14:14] jugg He's recovering well?
[14:16] benoitc jugg: i'm not sure why assertion is here line 821. on 64 bits it's always false.
[14:16] benoitc removing it, pubrep works of course ..
[14:16] sustrik i believe so
[14:17] jugg I know why. I'll have a fix soon. You can modify src/zmq_drv.erl line: 190 to be: <<(?ZMQ_POLL):8, (events_to_int(Events)):32>>
[14:20] jugg actually, I might change poll/send/recv flags to all be a byte size instead.
[14:21] jugg yes, that makes more sense
[14:21] benoitc k
[14:21] benoitc need to refresh my C
[14:22] jugg so instead, change the assert in poll to: assert(sizeof(uint8_t) == size);
[14:23] benoitc works
[14:25] efhache mikko: do you know if for zmq there is a use of "EPOLLET" (triggered mode) ??
[14:27] yawn is there a way to determine the underlying zmq version in jzmq?
[14:42] benoitc jugg: just did but anyway will wait your final fix
[14:42] benoitc rebar is working well on erlzmq btw
[14:43] jugg ok, almost ready.
[14:45] benoitc cool :)
[14:48] mikko efhache: i dont know what EPOLLET means
[14:48] mikko yawn: not sure if it's exposed
[14:49] mikko yawn: ZMQ::version_full() ?
[14:49] yawn mikko: not in my version - maybe an old build ...
[14:50] mikko
[14:50] mikko at least that's what the source says
[14:50] sustrik mikko: it's edge-triggered
[14:51] sustrik efhache: the underlying file descriptor (ZMQ_FD) is edge-triggered
[14:51] mikko sustrik: EPOLLET stands for edge-triggered?
[14:51] sustrik i think so
[14:51] sustrik let me check
[14:52] sustrik yes
[14:54] mikko ok
[14:59] efhache mikko: I've resolved the problem, I've used the export CPPFLAGS="-DZMQ_FORCE_SELECT"
[15:03] mikko efhache: cool
[15:04] mikko sustrik: i guess that makes sense if it standa for EPOLL Egde-Triggered
[15:04] yawn mikko: protected statics
[15:04] yawn mikko: one needs to do the usual dance & sing to make java return the values
[15:25] Hybelkanin Anyone with experience on zmq on osx 10.6 with the C# bindings?
[15:27] mikko Hybelkanin: unusual combination
[15:27] mikko can't say i've heard that combo before
[15:27] Hybelkanin :P
[15:27] mikko i have built and tested the c# bindings on linux
[15:28] Hybelkanin I think my problem may be monodevelop
[15:29] Hybelkanin So this might be the wrong channel
[15:43] amacleod My Python 0MQ REP server doesn't stop when I interrupt it with ^C. How do I make it respect ^C?
[15:49] s0undt3ch ppl, pub/sub forward devices should connect_in and bind_out?
[15:49] s0undt3ch or should they bind_in and bind_out?
[16:01] mikko amacleod: i would i think it might be blocking on zmq_term
[16:02] mikko amacleod: i dont really know how things are implemented in python so its just guessing
[16:02] amacleod mikko, I wrote it like the examples with a "while True" loop.
[16:05] amacleod I can stop it from within just fine--if I set up a conditional where my message handler calls self.context.term () when it receives "EXIT" or something, it's okay.
[16:05] mikko amacleod: can you see where it's stuck?
[16:05] amacleod As far as I know, it's not stuck. It's just completely ignoring ^C or SIGTERM.
[16:06] cremes amacleod: if it's blocked in zmq_term(), it won't let python's signal handler run to catch the signal
[16:06] cremes this has been reported a few times; i think sustrik is planning a fix
[16:06] cremes same thing happens with the ruby runtime
[16:07] amacleod Oh, interesting.
[16:07] amacleod If I hit ^C and then try sending something to it, it does get the KeyboardInterrupt exception.
[16:07] amacleod So it's actually probably blocking in recv()
[16:07] cremes huh, that might be a new problem
[16:07] cremes if you can reduce it, file an issue
[16:08] mikko recv should return on ctrl+c
[16:20] s0undt3ch sustrik, guido_g: I found out the issue about my forwarder problem
[16:21] s0undt3ch sustrik, guido_g: publishers were binding instead of connecting
[16:21] s0undt3ch sustrik, guido_g: the forwarder device was the one binding so, no one else should bind
[16:29] jugg benoitc, updates pushed. All get/setsockopt options are tested now.
[16:38] benoitc jugg: works
[16:38] benoitc thanks :)
[16:39] benoitc smth weird though when you launch pubsub test
[16:39] benoitc 10 is sent but none of subclient get it
[16:40] benoitc jugg: adding a timer after the bind fix that but ...
[16:42] benoitc anyway
[16:44] jugg yes, that is normal - not a binding issue.
[16:44] benoitc ok
[16:45] benoitc didn't check but did you add PUSH and PULL
[16:45] jugg yes
[16:45] benoitc or DOWNSTREAM, UPSTREAM are current names ?
[16:45] benoitc cool
[16:45] jugg using your rebar, I always get errors: fatal: No tags can describe '6944b664b20b525e210f1eaebea00bbcd21ec217'
[16:45] benoitc ok time to code this app then. more in the night
[16:45] jugg when doing $ make
[16:46] benoitc jugg: you have to tag
[16:46] jugg just to build ??
[16:46] benoitc i can remove it though
[16:46] benoitc jugg: this is for archive creation
[16:46] jugg yes, I build much more often than I tag
[16:46] jugg no, that happens when doing $ make
[16:46] benoitc let me edit this Makefile :)
[16:46] jugg k :)
[16:46] benoitc you could just do ./rebar compile anyway
[16:52] amacleod Should I file my test case as an attachment to , or start a new issue?
[16:52] mikko amacleod: is it python?
[16:53] amacleod mikko, yes, Python, not Ruby.
[16:53] mikko open it under
[16:58] benoitc jugg: fixed and pushed
[17:07] jugg benoitc, $git rebase -i can be your friend... :)
[17:14] benoitc mmm ?
[17:15] benoitc I've used rebase -i
[17:15] benoitc just cherry-picking last update
[17:15] jugg why not just rebase your commits ontop of the last update?
[17:16] benoitc in rebar-socketprocess
[17:16] benoitc dunno
[17:16] benoitc rebasing now
[17:21] benoitc jugg: done
[17:25] benoitc win 6
[18:14] amacleod Bleh. Didn't realize pyzmq master won't build against zmq 2.1.0.
[18:19] benoitc mmm it is
[18:19] benoitc at least it does
[18:20] amacleod Do I need to delete my previous installation of pyzmq in order to make use of the development version?
[18:20] benoitc i've installed it in my virtualenv
[18:22] amacleod I must be doing something wrong. I built and installed pyzmq master, and when I try to import zmq, it gets an error, 'cause it can't import initthreads from zmq.util
[18:23] benoitc did you change paths in setup.cfg?
[18:26] amacleod Ah, nope. That must be it.
[18:26] amacleod Does it not use /usr/local/lib and /usr/local/include by default?
[18:33] benoitc not sure
[18:45] amacleod The weird thing is that everything seems to get installed just fine. It's only when I go to import zmq that the error occurs.
[19:50] mikko sustrik:
[19:50] mikko wrong one
[19:51] mikko maybe not, not sure if that affects the one you reported
[19:52] sustrik looks like
[19:53] mikko testing now
[19:54] mikko nope, still see those reported
[22:56] yrashk can push socket be shared across threads?
[22:56] yrashk or is it unsafe to do so?
[23:29] yrashk if zmq_send with ZMQ_NOBLOCK returns EAGAIN, do I have to reinitialize the message?