ZeroMq IRC Log

Friday February 4, 2011

[Time] Name	Message
[00:10] cyball	hi ... is there a way to read a message and after that to send it to a message queue ?
[00:11] cyball	or to switch between message types ?
[00:58] mikko	cyball: what do you mean?
[01:00] hero	hi
[01:00] mikko	hello
[01:01] nathanmarz	i'm getting some strange errors from zeromq
[01:01] stodge	Is there a performance hit using the Java bindings for zeromq? I assume it uses jini?
[01:02] mikko	stodge: yes, it uses jni
[01:02] stodge	So I presume there's a performance hit?
[01:02] mikko	stodge: performance hit compared to what?
[01:02] stodge	If it were a native java app with no jini calls
[01:02] stodge	Significant
[01:03] stodge	Maybe that's not a reaonable question
[01:03] mikko	i dont really know the amount of overhead jni adds
[01:04] stodge	Me either
[01:04] stodge	:)
[01:05] mikko	but having native java app would mean replicating all zeromq logic in java
[01:05] mikko	so that doesnt seem appealing
[01:06] stodge	true
[01:06] stodge	I don't meant this in a negative sense, but I understand that rabbitmq has a native java library?
[01:07] stodge	I could be wrong
[01:08] mikko	probably almost any amqp client could communicate with it
[01:08] stodge	true
[01:09] stodge	zeromq has reliable multicast? Still reading through that long doc
[01:10] stodge	Ah pgm
[01:17] stodge	Hmm anyone know of a python/zeromq example of pgm
[01:26] mikko	stodge: its not really much different from using any other transport
[01:26] mikko	just different dsn
[01:26] mikko	anyways, off to bed
[01:26] mikko	nn
[01:38] stodge	thanks
[05:31] kabs	As per my question above "Hello, in pub-sub model , if sub starts later and pub started earlier all messages by pub will be dropped. Is there a way to buffer those messages ( with some buffer limit) so that they can sent to subscriber joining later. Can I get some link to go through it or if it is there in zeroMQ guide can someone point out which section it is ??"
[05:32] kabs	sustrik:Then how can be sync pub-sub there in one way in the zeroMQ guide , in chapter 2 in Node Coordination section, but it expect us pub to know the number of subscriber which is not a real world scenerio , how can be sync up pub-sub
[06:36] cyball	mikko: i mean is there a way to have a look inside the message to define the type of the message an base o that to rout the request to a queue that can handle the message type ?
[06:40] guido_g	cyball: see the guide, there is a lot about routing
[06:40] guido_g	and: you're defining what's in the message, Ã¸mq doesn't do anything w/ the your payload
[06:40] guido_g	s/your//
[06:40] cyball	guido_g: yes i know ... it is some times much easier to aks first if it is there or not :)
[06:43] guido_g	cyball: sure, why invest your own time when you can waste the time of others
[06:43] guido_g	cyball: it's a very fast and safe way to be ignored
[06:45] cyball	guido_g: it was not my intention to waste the time of others ... and i do not think that the question was about that but by the way discussions like the one we have now is time wasting :)
[06:46] guido_g	cyball: sure, from your point of view
[06:48] guido_g	cyball: but given that you showed nearly no effort in understanding Ã¸mq since you're here, it is not a waste of time -- from my point of view
[06:50] cyball	guido_g: i would not say that ... it is sometimes hard to understand written words ... and in some cases it is more efficient to have an discussion about a think to proof also if the idea of the system you want to write is really good or only bullsh*t :)
[06:51] guido_g	unfortunately does a "discussion" require both sides to something about the topic
[06:51] guido_g	*to know
[06:52] cyball	guido_g: i would say the conversation in alwas the key to success but you're right about the fact that i do not read all the documentation but if there are such great people like you it is much easier to get involved that is in my point of view the main fact of a community or not ?
[06:53] guido_g	no
[06:53] guido_g	experience shows that less than 10% of people like you ever come back and contribute
[06:56] cyball	you're right but i guess that i belong to the 10% ... i'm not a scripting kiddy who makes a lot of fun trying something out and through it away after playing with it ... there is a lot more behind my intention than only have a look on zeromq :)
[06:56] cyball	but also this information is something you can ask .... as i mentioned i will answer to it and i will contribute if i know enough about it ... this system as i mentioned yesterday is really really great
[06:58] cyball	i was looking forward to write myself such a system for communication for my project i'm developing at the moment so that was the reason checking out some of them ... and 0mq is the only one fits to my needs :)
[07:06] guido_g	guess how often i must have read things like these...
[07:07] guido_g	in short agin: show your effort and you'll receive much more answers and insight
[07:07] cyball	probably every time of the 90% from the of the people do not belong to the 10% you mentioned ? :)
[07:07] guido_g	what?
[07:07] cyball	i guess you read this very often
[07:07] guido_g	right
[07:08] guido_g	for decades
[07:13] cyball	ok i will have a look on the manual now ... you have a lot of experinces and you are still patient ... thx for your time i will ask if there is something not clear in the manual :)
[07:13] sustrik	:)
[08:57] Guthur	is there a limit of how many subscribers can simultaneously connect to tcp publisher?
[08:59] mikko	Guthur: no, not really
[08:59] mikko	apart from the obvious limits of hardware / operating system etc
[09:01] Guthur	i seem to be getting an unhandled exception when I try to connect multiple times, if I put in a wait it is ok
[09:01] Guthur	I am using the CLR binding
[09:01] mikko	Guthur: which exception is it?
[09:02] Guthur	that's the horrible thing, it passes through the .NET app unhandled, so it just says unhandle exception in win32 code
[09:03] mikko	can you attach a debugger and check ?
[09:03] Guthur	I probably have implement a global unhandled exception handler to catch it
[09:03] mikko	have you tried to reproduce it on other machines?
[09:03] mikko	if it's reproducable you could open an issue with reproduce code
[09:04] sustrik	Guthur: are you saying that clrzmq throws a generic exception?
[09:04] sustrik	then it's a bug in clrzmq
[09:04] mikko	sustrik: i think it might be C++ exception from unmanaged code
[09:04] mikko	which bubbles up and terminates the program
[09:04] mikko	(i assume)
[09:04] sustrik	what does that mean?
[09:04] sustrik	ah
[09:04] sustrik	got it
[09:05] sustrik	a C++ exception that gets to C# code
[09:05] mikko	thats how i understood it
[09:05] Guthur	yeah, clr does provide a mechanism to catch these though, I'll try adding it
[09:05] sustrik	wait a sec
[09:05] sustrik	clrzmq is using C API
[09:05] sustrik	not C++ API
[09:05] sustrik	underneath
[09:05] sustrik	so there should be no C++ exceptions
[09:06] sustrik	C API returns errnos...
[09:07] Guthur	I should be catching any exceptions in the clrzmq code
[09:07] Guthur	it is very strange
[09:07] sustrik	what i'm saying is that there are no exceptions from 0mq
[09:07] sustrik	so if there's an exception is rather generated by clrzmq
[09:08] sustrik	what's the win32 err code you are getting btw?
[09:09] Guthur	1 min
[09:09] Guthur	i get the JIT debugger saying unhandled win32 exception
[09:10] sustrik	no further info?
[09:11] sustrik	mikko: btw have you seen the email from neale?
[09:11] sustrik	the one about system z
[09:12] Guthur	sustrik: No, it just asks to connect the debugger, but the problem is a debugger is already attached, I am running the server app in visual studio in debug mode
[09:12] sustrik	yuck :(
[09:12] Guthur	yeah tell me about it
[09:14] Guthur	I'll add the global handler and see if I get anything more meaninful
[09:14] sustrik	ok
[09:18] mikko	sustrik: i was just replying it
[09:18] mikko	replying to it
[09:18] mikko	oddly enough on windows the openpgm installation just disappears occasionally
[09:18] mikko	havent really figured out yet what happens
[09:19] mikko	but in any case we have a release build with ZeroMQ+OpenPGM for windows
[09:19] mikko	http://snapshot.valokuva.org/msvc2008/2011-02-04_09-09-05/ is built from WithOpenPGM configuration
[09:31] Guthur	the debugger seems to say libzmq is C++
[09:32] Guthur	I got a debugger attached, all I have is the stack trace though
[09:33] Guthur	http://paste.lisp.org/display/119368
[09:37] sustrik	mikko: congrats!
[09:37] sustrik	Guthur: it's in C++
[09:37] sustrik	but the API is pure C
[09:38] sustrik	so no exceptions should pass out of libzmq
[09:38] Guthur	sustrik: does the stack trace give an clues?
[09:38] Guthur	an,any
[09:38] sustrik	sure, let's have a look
[09:38] mikko	abort()
[09:38] mikko	seems to be an assertion
[09:41] sustrik	Guthur: what version of 0mq?
[09:42] Guthur	2.0.10
[09:43] sustrik	thx
[09:44] sustrik	Guthur: i think this is the patch fixed by dhammika in 2.1 some time ago
[09:44] sustrik	can you check with latest master?
[09:45] Guthur	just trying that now, hehe
[09:45] Guthur	sorry it never occurred to me to try before
[10:07] Guthur	I am getting the same with 2.1
[10:07] Guthur	same trace as well
[10:11] Guthur	oh wait I think this is my fault
[10:11] Guthur	what is suppose to happen if two sockets try to connect with the same identity
[10:12] sustrik	you shouldn't do that
[10:12] sustrik	but in 2.1 it shouldn't fail
[10:13] sustrik	the bind side should silently drop the second connection iirc
[10:13] mikko	it still asserts
[10:13] Guthur	Yeah, I don't really want to be doing that, but this prototype code, and I haven't generated very unique uid
[10:13] sustrik	ah, ok
[10:13] Guthur	will when I remove the set Identity it works
[10:13] Guthur	well*
[10:14] Guthur	ok, so mystery solved then?
[10:14] sustrik	i think so
[10:15] Guthur	is there any plans to make this scenario a little less fatal
[10:18] sustrik	that would be nice
[10:18] cyball	is there a way to tell the XREP(router) depending on the MessageBody to which XREQ(dealer) to route the message .... i could not figure that out the only way to work on specific messages i found was to use the PubSub scenario but this is only unidirectional so no response will be routed back
[10:19] sustrik	nbody volunteered yet though
[10:20] sustrik	REP socket routes the reply back to original requester automagically
[10:21] Guthur	unfortunately I seem to fighting a losing battle to allow the use of ZeroMQ in our app, the team lead is adamant I should use .Net threads with global state instead
[10:21] Guthur	he is not listening to reason
[10:22] cyball	yes i know but can i tell the REP(papa) to get only specific messages for example if i have some calculation and some data get from DB i want to make 2 different REP(papas) one for calcucation and one for data
[10:22] ianbarber	sustrik - i filed a bug about the dual identity issue by the way. it certainly looks like it should drop, but doesn't
[10:23] ianbarber	cyball: the XREP is going to route based on the identity, and based on the first message segment
[10:24] ianbarber	so if you had a 'calc' and a 'data' identity, you'd just need to prepend the message with that identity and a blank one before sending it to the xrep.
[10:25] ianbarber	that can be in the message body any way you want, just write a bit of code to pull it out and stick it in as a message segment before sending on to the xrep
[10:27] sustrik	Guthur: that's pretty common occurence
[10:28] sustrik	cyball: you have to use two different sockets
[10:29] sustrik	so, say calculations are passed on port 5555
[10:29] sustrik	and DB data on 5556
[10:31] cyball	sustrik: is that the only way ?
[10:31] sustrik	it's the only sensible way
[10:32] sustrik	that way your network knows about your feeds
[10:32] sustrik	and thus, for example, large DB dataset being passed
[10:32] sustrik	doesn't block little calc operations
[10:33] cyball	sustrik: thx
[10:33] sustrik	np
[13:54] mikko	sustrik: there?
[13:54] sustrik	mikko: hi
[13:54] mikko	i would like to discuss openpgm situation at some point
[13:54] sustrik	sure
[13:55] mikko	have you got a long-term vision for it?
[13:55] sustrik	i have no idea what's going on
[13:55] mikko	i am now looking into using the autoconf steven added to openpgm trunk
[13:55] mikko	i can get it to build
[13:55] sustrik	well, from 30,000 feet
[13:55] mikko	but i'm afraid the way i am doing it at the moment is not portable
[13:56] sustrik	the pgm is needed for stock traders, banks etc.
[13:56] mikko	so im wondering is the longer-term plan to bundle openpgm with zeromq
[13:56] mikko	or is it feasible to ask users to install openpgm if they want to use it with zeromq
[13:56] mikko	that would seem like the normal way "if you compile --with-pgm make sure you have installed openpgm"
[13:56] mikko	brb
[13:57] Silly	huh
[13:58] sustrik	well, i am not sure openpgm is moving towards becoming a stand-alone package
[13:58] sustrik	it would be ideal if it was so
[13:58] sustrik	but we have to be pragmatic at the moment
[14:03] mikko	sustrik: i'm beginning to doubt whether bundling is the pragmatic approach
[14:04] mikko	as the problem with the autotools build is that some platforms dont support linking static libraries into shared libraries
[14:05] mikko	at least thats my understand
[14:05] mikko	ing
[14:05] sustrik	i see
[14:05] mikko	then again, i dont see why openpgm couldnt be installed as shared library
[14:05] mikko	and let libtool worry about the best way
[14:05] mikko	but that might cause confusion as well
[14:05] mikko	because some people might have openpgm already installed and 0mq would write over that
[14:05] sustrik	i have no specific opinion on that
[14:06] sustrik	the only goal imo is to make it as easy to build/use as possible
[14:12] sustrik	mikko: what are your thoughts atm?
[14:37] mikko	sustrik: my current thinking is that those two builds make more sense as separate ones
[14:37] sustrik	how would people install it then?
[14:38] mikko	execute their build
[14:38] mikko	and install shared library
[14:38] sustrik	which build?
[14:39] sustrik	openpgm?
[14:39] mikko	execute openpgm build, install it
[14:39] mikko	execute zeromq build with --with-pgm
[14:40] sustrik	well, it makes the whole process 2x as difficult
[14:40] sustrik	1. install scons
[14:40] sustrik	2. download openpgm (the right version!)
[14:40] sustrik	4. build & install
[14:40] sustrik	4. install autotools
[14:40] sustrik	5. download 0mq
[14:40] sustrik	6. build & install
[14:41] sustrik	but as i say, i don't have a strong opinion
[14:42] sustrik	pgm is used by stock traders, which have enough money to actually pay someone to that for them
[14:42] mikko	thats not 100% correct now that openpgm has autotools build
[14:42] sustrik	or even sponsor binary builds & installers
[14:42] mikko	execute ./configure on openpgm, make install
[14:43] mikko	execute ./configure --with-pgm on zeromq, make install
[14:43] sustrik	yeah, that's better
[14:43] mikko	if you use release packages. autotools are needed for trunk version
[14:43] mikko	and even that is mitigated with snapshots which have configure script in them
[14:43] sustrik	ack
[14:44] sustrik	the hardest part, i assume, is getting right versions of 0mq and openpgm
[14:44] sustrik	anyway, you are the build system maintainer, so it's up to you
[14:45] sustrik	i would consult the mailing list first though
[14:47] drbobbeaty	I know I'm only a user, and this is your choice, but as for me, ZeroMQ "contains" OpenPGM. There's no utility of OpenPGM for me outside of ZeroMQ, and a large part of ZeroMQ (for me) is the epgm transport. So if you unbundle these, it's going to make maintenance harder for me. I'll have to make sure that no machine in the organization has the "wrong" OpenPGM for ZeroMQ, etc.
[14:48] drbobbeaty	I know it seems simple to unbundle them, and I can certainly see the advantages, but just one vote from your user base is that in an organization with many machines, this is something that I'd rather not have to deal with.
[14:49] sustrik	that was the point i was trying to make but you are more eloquent than i am :)
[15:18] edw	The clojure glue has a destory-socket call which calls the destroy method of a socket. There is no destroy method. There is a close method. Did this ever work (ie was destroy renamed to close) or has it been broken forever?
[15:20] sustrik	there's no close method in C API
[15:21] sustrik	there's zmq_close() though
[15:21] sustrik	what are you referring to?
[15:21] edw	I'm looking at he jzmq Java bindings. Those have a Socket::close method.
[15:22] sustrik	that seems right
[15:22] sustrik	i am not aware there was a destroy method originally, but maybe
[15:22] sustrik	why not just use close instad of destroy?
[15:24] edw	...which calls the finalizer which calls zmq_close. Mystery solved. I think that the Clojure glue is a piece of fairly shoddy workmanship. Time to fork or re-write it.
[15:25] edw	It's not idiomatic Clojure e.g. constants look like +this+. And it calls methods that don't exist.
[15:27] sustrik	sure, go on
[15:27] edw	Oh, and it (inaccurately) redefines all the enums in the org.zeromq.ZMQ class. Ugh...
[15:28] sustrik	well, it's up to you to fix it :)
[15:30] edw	Actually, it should probably just be deprecated, as it's a molecule-thick layer of sewing machine oil over the JZMQ classes.
[15:30] edw	E.g. "(defn make-context [io-threads] (ZMQ/context io-threads))
[15:33] sustrik	maybe the intent was to make the API more clojure-like?
[15:33] edw	For sure, but there's a better way to do it... I'm typing away right now...
[15:45] CIA-21	zeromq2: 03Thijs Terlouw 07master * r042e34a 10/ include/zmq.hpp :
[15:45] CIA-21	zeromq2: operator void* () added to context_t.
[15:45] CIA-21	zeromq2: Makes it possible to share the context from C++ to C.
[15:45] CIA-21	zeromq2: Signed-off-by: Thijs Terlouw <thijsterlouw@gmail.com> - http://bit.ly/ew59JP
[18:04] Esmil	Hmm.. suppose I have a select()/epoll()/what-have-you mainloop running, is there a way I can get events on a filedescripter whenever a socket has a new message?
[18:05] Esmil	A zeromq socket that is
[18:12] Steve-o	try zmq_poll
[18:14] Esmil	Yeah, but that means I'd have to rewrite the mainloop to use zmq_poll. I'm quite happy with libev
[18:18] Steve-o	check the mailling list archive, it certainly has come up before
[18:27] Esmil	Damn, once again I'm ahead of time :P http://article.gmane.org/gmane.network.zeromq.devel/1628
[18:33] mikko	good evening
[18:34] mikko	Esmil: yes there is
[18:34] mikko	Esmil: with 2.1.x you can use zmq_getsockopt( ZMQ_FD
[18:35] Esmil	mikko: Cool! What version is the reference manual on zeromq.org refering to?
[18:35] mikko	2.0.x i think
[18:35] Esmil	Ahh, I see. Thanks
[18:41] Esmil	2.0.x can speak with 2.1.x right? Or is there a reason my distribution hasn't updated to 2.1.x yet?
[18:42] Esmil	Ohh, it's considered beta
[18:43] mikko	can speak with yes
[18:43] Esmil	Great
[18:43] mikko	there are slight changes in the api so code written for 2.0.x might require modifications
[18:44] mikko	the biggest changes are in zmq_term semantics and that zmq_context takes one param instead of many
[18:45] Esmil	I see
[18:47] mikko	drbobbeaty: i am not looking to necessarily uncouple
[18:47] mikko	but what i am looking at is possible linking shared library rather than static
[18:47] mikko	Steve-o: there?
[18:48] mikko	so, i was building OpenPGM autoconf builds and ran into following thing which im not sure how to solve the cleanest way:
[18:48] mikko	a) running the autoconf build generates static libraries that are non PIC by default
[18:48] mikko	b) on 64bit linux you need to have pic static libraries to link them into shared libraries
[18:49] mikko	c) this is probably not portable
[18:49] drbobbeaty	mikko: I'm all for linking shared libs - even preferred. But it's the ability to have it built all at once, so that I can build RPMs and Ubuntu deb packages and have it be treated as a single, atomic, deliverable.
[18:50] Steve-o	k
[18:51] Steve-o	I normally make a -pic lib in scons
[18:51] Steve-o	is there an easy way to do this with libtool & automake?
[18:52] Steve-o	I thought the .la lib was supposed to be PIC
[18:53] Steve-o	otherwise libtool couldn't build the shared library?
[19:06] mikko	.la can contain shared and static
[19:07] mikko	shared -pic and static non pic
[19:07] mikko	at least on my system
[19:07] mikko	i could link against the libtool archive in which case libtool can decide what to link against
[19:24] zchrish	Question: What happens to the server, in a XREQ/XREP pattern, when a client dies after sending a request but before it receives the response from its query?
[19:25] cremes	zchrish: when the server tries to send the reply, it should note that the peer is gone and drop the packet
[19:26] zchrish	cremes: I assume this failure mechanism is modeled in zeromq and, therefore, handles the situation ok. Will a reconnect from the same client pose any issue I should be aware of?
[19:27] zchrish	I.E. the client program dies and it is restarted from the OS.
[19:28] zchrish	In a different situation, suppose a server dies, what should the client do to restart w/o resorting to a complete program restart?
[19:29] zchrish	Thank you.
[19:52] Leolo_3	zchrish : depends on weither you gave the socket an identity
[20:33] zchrish	Leolo_3: No, I didn't explicity give an identity.
[20:35] Leolo_3	then the messages get dropped
[20:35] Leolo_3	for your server dies senario : zeromq does automatic reconnect
[20:37] zchrish	I am not worried about dropped messages. I just I am trying to learn the proper reconnection strategy. For example, do I need to shut down the zmq::context and start another or do I simply close and reopen?
[20:38] zchrish	My current situation appears that clients hang during a server crash and must be shut down (ctrl+c) and restarted.
[20:39] zchrish	The docs don't discuss this area much. Clients don't have an issue if the server shuts down gracefully.
[20:43] Steve-o	so the clients think the TCP socket is still open
[20:44] Steve-o	I think this was discovered before as reasons for implementing heartbeats
[20:44] Steve-o	in the list somewhere
[20:50] Steve-o	>> http://lists.zeromq.org/pipermail/zeromq-dev/2010-October/007011.html
[21:50] sustrik	zchris: to be resilient in case of server failure
[21:50] sustrik	you have to timeout waiting for the reply
[21:50] sustrik	and resend the request
[21:51] sustrik	in theory, 0mq should do that for you, but it's not yet implemented :(
[23:31] zchrish	Yes, using heartbeats is what I figured I needed to do. Thank you.
[23:33] zchrish	How best to set a timeout for : string *string = s_recv (socket) ?
[23:55] Steve-o	synchronous or asynchronous?
[23:55] zchrish	In my case, I am using XREP/XREQ