Friday February 4, 2011

[Time] NameMessage
[00:10] cyball hi ... is there a way to read a message and after that to send it to a message queue ?
[00:11] cyball or to switch between message types ?
[00:58] mikko cyball: what do you mean?
[01:00] hero hi
[01:00] mikko hello
[01:01] nathanmarz i'm getting some strange errors from zeromq
[01:01] stodge Is there a performance hit using the Java bindings for zeromq? I assume it uses jini?
[01:02] mikko stodge: yes, it uses jni
[01:02] stodge So I presume there's a performance hit?
[01:02] mikko stodge: performance hit compared to what?
[01:02] stodge If it were a native java app with no jini calls
[01:02] stodge Significant
[01:03] stodge Maybe that's not a reaonable question
[01:03] mikko i dont really know the amount of overhead jni adds
[01:04] stodge Me either
[01:04] stodge :)
[01:05] mikko but having native java app would mean replicating all zeromq logic in java
[01:05] mikko so that doesnt seem appealing
[01:06] stodge true
[01:06] stodge I don't meant this in a negative sense, but I understand that rabbitmq has a native java library?
[01:07] stodge I could be wrong
[01:08] mikko probably almost any amqp client could communicate with it
[01:08] stodge true
[01:09] stodge zeromq has reliable multicast? Still reading through that long doc
[01:10] stodge Ah pgm
[01:17] stodge Hmm anyone know of a python/zeromq example of pgm
[01:26] mikko stodge: its not really much different from using any other transport
[01:26] mikko just different dsn
[01:26] mikko anyways, off to bed
[01:26] mikko nn
[01:38] stodge thanks
[05:31] kabs As per my question above "Hello, in pub-sub model , if sub starts later and pub started earlier all messages by pub will be dropped. Is there a way to buffer those messages ( with some buffer limit) so that they can sent to subscriber joining later. Can I get some link to go through it or if it is there in zeroMQ guide can someone point out which section it is ??"
[05:32] kabs sustrik:Then how can be sync pub-sub there in one way in the zeroMQ guide , in chapter 2 in Node Coordination section, but it expect us pub to know the number of subscriber which is not a real world scenerio , how can be sync up pub-sub
[06:36] cyball mikko: i mean is there a way to have a look inside the message to define the type of the message an base o that to rout the request to a queue that can handle the message type ?
[06:40] guido_g cyball: see the guide, there is a lot about routing
[06:40] guido_g and: you're defining what's in the message, ømq doesn't do anything w/ the your payload
[06:40] guido_g s/your//
[06:40] cyball guido_g: yes i know ... it is some times much easier to aks first if it is there or not :)
[06:43] guido_g cyball: sure, why invest your own time when you can waste the time of others
[06:43] guido_g cyball: it's a very fast and safe way to be ignored
[06:45] cyball guido_g: it was not my intention to waste the time of others ... and i do not think that the question was about that but by the way discussions like the one we have now is time wasting :)
[06:46] guido_g cyball: sure, from your point of view
[06:48] guido_g cyball: but given that you showed nearly no effort in understanding ømq since you're here, it is not a waste of time -- from my point of view
[06:50] cyball guido_g: i would not say that ... it is sometimes hard to understand written words ... and in some cases it is more efficient to have an discussion about a think to proof also if the idea of the system you want to write is really good or only bullsh*t :)
[06:51] guido_g unfortunately does a "discussion" require both sides to something about the topic
[06:51] guido_g *to know
[06:52] cyball guido_g: i would say the conversation in alwas the key to success but you're right about the fact that i do not read all the documentation but if there are such great people like you it is much easier to get involved that is in my point of view the main fact of a community or not ?
[06:53] guido_g no
[06:53] guido_g experience shows that less than 10% of people like you ever come back and contribute
[06:56] cyball you're right but i guess that i belong to the 10% ... i'm not a scripting kiddy who makes a lot of fun trying something out and through it away after playing with it ... there is a lot more behind my intention than only have a look on zeromq :)
[06:56] cyball but also this information is something you can ask .... as i mentioned i will answer to it and i will contribute if i know enough about it ... this system as i mentioned yesterday is really really great
[06:58] cyball i was looking forward to write myself such a system for communication for my project i'm developing at the moment so that was the reason checking out some of them ... and 0mq is the only one fits to my needs :)
[07:06] guido_g guess how often i must have read things like these...
[07:07] guido_g in short agin: show your effort and you'll receive much more answers and insight
[07:07] cyball probably every time of the 90% from the of the people do not belong to the 10% you mentioned ? :)
[07:07] guido_g what?
[07:07] cyball i guess you read this very often
[07:07] guido_g right
[07:08] guido_g for decades
[07:13] cyball ok i will have a look on the manual now ... you have a lot of experinces and you are still patient ... thx for your time i will ask if there is something not clear in the manual :)
[07:13] sustrik :)
[08:57] Guthur is there a limit of how many subscribers can simultaneously connect to tcp publisher?
[08:59] mikko Guthur: no, not really
[08:59] mikko apart from the obvious limits of hardware / operating system etc
[09:01] Guthur i seem to be getting an unhandled exception when I try to connect multiple times, if I put in a wait it is ok
[09:01] Guthur I am using the CLR binding
[09:01] mikko Guthur: which exception is it?
[09:02] Guthur that's the horrible thing, it passes through the .NET app unhandled, so it just says unhandle exception in win32 code
[09:03] mikko can you attach a debugger and check ?
[09:03] Guthur I probably have implement a global unhandled exception handler to catch it
[09:03] mikko have you tried to reproduce it on other machines?
[09:03] mikko if it's reproducable you could open an issue with reproduce code
[09:04] sustrik Guthur: are you saying that clrzmq throws a generic exception?
[09:04] sustrik then it's a bug in clrzmq
[09:04] mikko sustrik: i think it might be C++ exception from unmanaged code
[09:04] mikko which bubbles up and terminates the program
[09:04] mikko (i assume)
[09:04] sustrik what does that mean?
[09:04] sustrik ah
[09:04] sustrik got it
[09:05] sustrik a C++ exception that gets to C# code
[09:05] mikko thats how i understood it
[09:05] Guthur yeah, clr does provide a mechanism to catch these though, I'll try adding it
[09:05] sustrik wait a sec
[09:05] sustrik clrzmq is using C API
[09:05] sustrik not C++ API
[09:05] sustrik underneath
[09:05] sustrik so there should be no C++ exceptions
[09:06] sustrik C API returns errnos...
[09:07] Guthur I should be catching any exceptions in the clrzmq code
[09:07] Guthur it is very strange
[09:07] sustrik what i'm saying is that there are no exceptions from 0mq
[09:07] sustrik so if there's an exception is rather generated by clrzmq
[09:08] sustrik what's the win32 err code you are getting btw?
[09:09] Guthur 1 min
[09:09] Guthur i get the JIT debugger saying unhandled win32 exception
[09:10] sustrik no further info?
[09:11] sustrik mikko: btw have you seen the email from neale?
[09:11] sustrik the one about system z
[09:12] Guthur sustrik: No, it just asks to connect the debugger, but the problem is a debugger is already attached, I am running the server app in visual studio in debug mode
[09:12] sustrik yuck :(
[09:12] Guthur yeah tell me about it
[09:14] Guthur I'll add the global handler and see if I get anything more meaninful
[09:14] sustrik ok
[09:18] mikko sustrik: i was just replying it
[09:18] mikko replying to it
[09:18] mikko oddly enough on windows the openpgm installation just disappears occasionally
[09:18] mikko havent really figured out yet what happens
[09:19] mikko but in any case we have a release build with ZeroMQ+OpenPGM for windows
[09:19] mikko is built from WithOpenPGM configuration
[09:31] Guthur the debugger seems to say libzmq is C++
[09:32] Guthur I got a debugger attached, all I have is the stack trace though
[09:33] Guthur
[09:37] sustrik mikko: congrats!
[09:37] sustrik Guthur: it's in C++
[09:37] sustrik but the API is pure C
[09:38] sustrik so no exceptions should pass out of libzmq
[09:38] Guthur sustrik: does the stack trace give an clues?
[09:38] Guthur an,any
[09:38] sustrik sure, let's have a look
[09:38] mikko abort()
[09:38] mikko seems to be an assertion
[09:41] sustrik Guthur: what version of 0mq?
[09:42] Guthur 2.0.10
[09:43] sustrik thx
[09:44] sustrik Guthur: i think this is the patch fixed by dhammika in 2.1 some time ago
[09:44] sustrik can you check with latest master?
[09:45] Guthur just trying that now, hehe
[09:45] Guthur sorry it never occurred to me to try before
[10:07] Guthur I am getting the same with 2.1
[10:07] Guthur same trace as well
[10:11] Guthur oh wait I think this is my fault
[10:11] Guthur what is suppose to happen if two sockets try to connect with the same identity
[10:12] sustrik you shouldn't do that
[10:12] sustrik but in 2.1 it shouldn't fail
[10:13] sustrik the bind side should silently drop the second connection iirc
[10:13] mikko it still asserts
[10:13] Guthur Yeah, I don't really want to be doing that, but this prototype code, and I haven't generated very unique uid
[10:13] sustrik ah, ok
[10:13] Guthur will when I remove the set Identity it works
[10:13] Guthur well*
[10:14] Guthur ok, so mystery solved then?
[10:14] sustrik i think so
[10:15] Guthur is there any plans to make this scenario a little less fatal
[10:18] sustrik that would be nice
[10:18] cyball is there a way to tell the XREP(router) depending on the MessageBody to which XREQ(dealer) to route the message .... i could not figure that out the only way to work on specific messages i found was to use the PubSub scenario but this is only unidirectional so no response will be routed back
[10:19] sustrik nbody volunteered yet though
[10:20] sustrik REP socket routes the reply back to original requester automagically
[10:21] Guthur unfortunately I seem to fighting a losing battle to allow the use of ZeroMQ in our app, the team lead is adamant I should use .Net threads with global state instead
[10:21] Guthur he is not listening to reason
[10:22] cyball yes i know but can i tell the REP(papa) to get only specific messages for example if i have some calculation and some data get from DB i want to make 2 different REP(papas) one for calcucation and one for data
[10:22] ianbarber sustrik - i filed a bug about the dual identity issue by the way. it certainly looks like it should drop, but doesn't
[10:23] ianbarber cyball: the XREP is going to route based on the identity, and based on the first message segment
[10:24] ianbarber so if you had a 'calc' and a 'data' identity, you'd just need to prepend the message with that identity and a blank one before sending it to the xrep.
[10:25] ianbarber that can be in the message body any way you want, just write a bit of code to pull it out and stick it in as a message segment before sending on to the xrep
[10:27] sustrik Guthur: that's pretty common occurence
[10:28] sustrik cyball: you have to use two different sockets
[10:29] sustrik so, say calculations are passed on port 5555
[10:29] sustrik and DB data on 5556
[10:31] cyball sustrik: is that the only way ?
[10:31] sustrik it's the only sensible way
[10:32] sustrik that way your network knows about your feeds
[10:32] sustrik and thus, for example, large DB dataset being passed
[10:32] sustrik doesn't block little calc operations
[10:33] cyball sustrik: thx
[10:33] sustrik np
[13:54] mikko sustrik: there?
[13:54] sustrik mikko: hi
[13:54] mikko i would like to discuss openpgm situation at some point
[13:54] sustrik sure
[13:55] mikko have you got a long-term vision for it?
[13:55] sustrik i have no idea what's going on
[13:55] mikko i am now looking into using the autoconf steven added to openpgm trunk
[13:55] mikko i can get it to build
[13:55] sustrik well, from 30,000 feet
[13:55] mikko but i'm afraid the way i am doing it at the moment is not portable
[13:56] sustrik the pgm is needed for stock traders, banks etc.
[13:56] mikko so im wondering is the longer-term plan to bundle openpgm with zeromq
[13:56] mikko or is it feasible to ask users to install openpgm if they want to use it with zeromq
[13:56] mikko that would seem like the normal way "if you compile --with-pgm make sure you have installed openpgm"
[13:56] mikko brb
[13:57] Silly huh
[13:58] sustrik well, i am not sure openpgm is moving towards becoming a stand-alone package
[13:58] sustrik it would be ideal if it was so
[13:58] sustrik but we have to be pragmatic at the moment
[14:03] mikko sustrik: i'm beginning to doubt whether bundling is the pragmatic approach
[14:04] mikko as the problem with the autotools build is that some platforms dont support linking static libraries into shared libraries
[14:05] mikko at least thats my understand
[14:05] mikko ing
[14:05] sustrik i see
[14:05] mikko then again, i dont see why openpgm couldnt be installed as shared library
[14:05] mikko and let libtool worry about the best way
[14:05] mikko but that might cause confusion as well
[14:05] mikko because some people might have openpgm already installed and 0mq would write over that
[14:05] sustrik i have no specific opinion on that
[14:06] sustrik the only goal imo is to make it as easy to build/use as possible
[14:12] sustrik mikko: what are your thoughts atm?
[14:37] mikko sustrik: my current thinking is that those two builds make more sense as separate ones
[14:37] sustrik how would people install it then?
[14:38] mikko execute their build
[14:38] mikko and install shared library
[14:38] sustrik which build?
[14:39] sustrik openpgm?
[14:39] mikko execute openpgm build, install it
[14:39] mikko execute zeromq build with --with-pgm
[14:40] sustrik well, it makes the whole process 2x as difficult
[14:40] sustrik 1. install scons
[14:40] sustrik 2. download openpgm (the right version!)
[14:40] sustrik 4. build & install
[14:40] sustrik 4. install autotools
[14:40] sustrik 5. download 0mq
[14:40] sustrik 6. build & install
[14:41] sustrik but as i say, i don't have a strong opinion
[14:42] sustrik pgm is used by stock traders, which have enough money to actually pay someone to that for them
[14:42] mikko thats not 100% correct now that openpgm has autotools build
[14:42] sustrik or even sponsor binary builds & installers
[14:42] mikko execute ./configure on openpgm, make install
[14:43] mikko execute ./configure --with-pgm on zeromq, make install
[14:43] sustrik yeah, that's better
[14:43] mikko if you use release packages. autotools are needed for trunk version
[14:43] mikko and even that is mitigated with snapshots which have configure script in them
[14:43] sustrik ack
[14:44] sustrik the hardest part, i assume, is getting right versions of 0mq and openpgm
[14:44] sustrik anyway, you are the build system maintainer, so it's up to you
[14:45] sustrik i would consult the mailing list first though
[14:47] drbobbeaty I know I'm only a user, and this is your choice, but as for me, ZeroMQ "contains" OpenPGM. There's no utility of OpenPGM for me outside of ZeroMQ, and a large part of ZeroMQ (for me) is the epgm transport. So if you unbundle these, it's going to make maintenance harder for me. I'll have to make sure that no machine in the organization has the "wrong" OpenPGM for ZeroMQ, etc.
[14:48] drbobbeaty I know it seems simple to unbundle them, and I can certainly see the advantages, but just one vote from your user base is that in an organization with many machines, this is something that I'd rather not have to deal with.
[14:49] sustrik that was the point i was trying to make but you are more eloquent than i am :)
[15:18] edw The clojure glue has a destory-socket call which calls the destroy method of a socket. There is no destroy method. There is a close method. Did this ever work (ie was destroy renamed to close) or has it been broken forever?
[15:20] sustrik there's no close method in C API
[15:21] sustrik there's zmq_close() though
[15:21] sustrik what are you referring to?
[15:21] edw I'm looking at he jzmq Java bindings. Those have a Socket::close method.
[15:22] sustrik that seems right
[15:22] sustrik i am not aware there was a destroy method originally, but maybe
[15:22] sustrik why not just use close instad of destroy?
[15:24] edw ...which calls the finalizer which calls zmq_close. Mystery solved. I think that the Clojure glue is a piece of fairly shoddy workmanship. Time to fork or re-write it.
[15:25] edw It's not idiomatic Clojure e.g. constants look like +this+. And it calls methods that don't exist.
[15:27] sustrik sure, go on
[15:27] edw Oh, and it (inaccurately) redefines all the enums in the org.zeromq.ZMQ class. Ugh...
[15:28] sustrik well, it's up to you to fix it :)
[15:30] edw Actually, it should probably just be deprecated, as it's a molecule-thick layer of sewing machine oil over the JZMQ classes.
[15:30] edw E.g. "(defn make-context [io-threads] (ZMQ/context io-threads))
[15:33] sustrik maybe the intent was to make the API more clojure-like?
[15:33] edw For sure, but there's a better way to do it... I'm typing away right now...
[15:45] CIA-21 zeromq2: 03Thijs Terlouw 07master * r042e34a 10/ include/zmq.hpp :
[15:45] CIA-21 zeromq2: operator void* () added to context_t.
[15:45] CIA-21 zeromq2: Makes it possible to share the context from C++ to C.
[15:45] CIA-21 zeromq2: Signed-off-by: Thijs Terlouw <> -
[18:04] Esmil Hmm.. suppose I have a select()/epoll()/what-have-you mainloop running, is there a way I can get events on a filedescripter whenever a socket has a new message?
[18:05] Esmil A zeromq socket that is
[18:12] Steve-o try zmq_poll
[18:14] Esmil Yeah, but that means I'd have to rewrite the mainloop to use zmq_poll. I'm quite happy with libev
[18:18] Steve-o check the mailling list archive, it certainly has come up before
[18:27] Esmil Damn, once again I'm ahead of time :P
[18:33] mikko good evening
[18:34] mikko Esmil: yes there is
[18:34] mikko Esmil: with 2.1.x you can use zmq_getsockopt( ZMQ_FD
[18:35] Esmil mikko: Cool! What version is the reference manual on refering to?
[18:35] mikko 2.0.x i think
[18:35] Esmil Ahh, I see. Thanks
[18:41] Esmil 2.0.x can speak with 2.1.x right? Or is there a reason my distribution hasn't updated to 2.1.x yet?
[18:42] Esmil Ohh, it's considered beta
[18:43] mikko can speak with yes
[18:43] Esmil Great
[18:43] mikko there are slight changes in the api so code written for 2.0.x might require modifications
[18:44] mikko the biggest changes are in zmq_term semantics and that zmq_context takes one param instead of many
[18:45] Esmil I see
[18:47] mikko drbobbeaty: i am not looking to necessarily uncouple
[18:47] mikko but what i am looking at is possible linking shared library rather than static
[18:47] mikko Steve-o: there?
[18:48] mikko so, i was building OpenPGM autoconf builds and ran into following thing which im not sure how to solve the cleanest way:
[18:48] mikko a) running the autoconf build generates static libraries that are non PIC by default
[18:48] mikko b) on 64bit linux you need to have pic static libraries to link them into shared libraries
[18:49] mikko c) this is probably not portable
[18:49] drbobbeaty mikko: I'm all for linking shared libs - even preferred. But it's the ability to have it built all at once, so that I can build RPMs and Ubuntu deb packages and have it be treated as a single, atomic, deliverable.
[18:50] Steve-o k
[18:51] Steve-o I normally make a -pic lib in scons
[18:51] Steve-o is there an easy way to do this with libtool & automake?
[18:52] Steve-o I thought the .la lib was supposed to be PIC
[18:53] Steve-o otherwise libtool couldn't build the shared library?
[19:06] mikko .la can contain shared and static
[19:07] mikko shared -pic and static non pic
[19:07] mikko at least on my system
[19:07] mikko i could link against the libtool archive in which case libtool can decide what to link against
[19:24] zchrish Question: What happens to the server, in a XREQ/XREP pattern, when a client dies after sending a request but before it receives the response from its query?
[19:25] cremes zchrish: when the server tries to send the reply, it should note that the peer is gone and drop the packet
[19:26] zchrish cremes: I assume this failure mechanism is modeled in zeromq and, therefore, handles the situation ok. Will a reconnect from the same client pose any issue I should be aware of?
[19:27] zchrish I.E. the client program dies and it is restarted from the OS.
[19:28] zchrish In a different situation, suppose a server dies, what should the client do to restart w/o resorting to a complete program restart?
[19:29] zchrish Thank you.
[19:52] Leolo_3 zchrish : depends on weither you gave the socket an identity
[20:33] zchrish Leolo_3: No, I didn't explicity give an identity.
[20:35] Leolo_3 then the messages get dropped
[20:35] Leolo_3 for your server dies senario : zeromq does automatic reconnect
[20:37] zchrish I am not worried about dropped messages. I just I am trying to learn the proper reconnection strategy. For example, do I need to shut down the zmq::context and start another or do I simply close and reopen?
[20:38] zchrish My current situation appears that clients hang during a server crash and must be shut down (ctrl+c) and restarted.
[20:39] zchrish The docs don't discuss this area much. Clients don't have an issue if the server shuts down gracefully.
[20:43] Steve-o so the clients think the TCP socket is still open
[20:44] Steve-o I think this was discovered before as reasons for implementing heartbeats
[20:44] Steve-o in the list somewhere
[20:50] Steve-o >>
[21:50] sustrik zchris: to be resilient in case of server failure
[21:50] sustrik you have to timeout waiting for the reply
[21:50] sustrik and resend the request
[21:51] sustrik in theory, 0mq should do that for you, but it's not yet implemented :(
[23:31] zchrish Yes, using heartbeats is what I figured I needed to do. Thank you.
[23:33] zchrish How best to set a timeout for : string *string = s_recv (socket) ?
[23:55] Steve-o synchronous or asynchronous?
[23:55] zchrish In my case, I am using XREP/XREQ