Tuesday June 8, 2010

[Time] NameMessage
[00:59] jugg I got this assert in my app when rebooting a peer computer: tcp_connector.cpp:296: zmq_assert (err == ECONNREFUSED || err == ETIMEDOUT); Is there a reason that zmq dies if the remote port is unavailable?
[03:30] jugg sustrik: github is showing that you committed a comment change on that very assert 9 hours ago... ??? that is very wierd.
[04:58] umesh I am wondering if ZeroMQ supports a many to many delivery model.
[04:58] umesh The documentation for publish and subscribe mention that it is used to implement a one to many model:
[04:58] umesh "The publish-subscribe pattern is used for one-to-many distribution of data from a single publisher to multiple subscribers in a fanout fashion."
[04:58] umesh Can multiple entities publish on the same channel and thus make it a many to many model ?
[05:01] sustrik jugg: the assert tries to capture possible netwrok-related error codes
[05:02] sustrik can you have a look which error are you exactly experiencing?
[05:02] sustrik as for the commit it just fixed the comment there, it haven't changed the actual code
[05:03] sustrik umesh: you have to have zmq_forwarder device in the middle
[05:09] alefka hi
[05:10] sustrik hi
[05:11] umesh ok
[05:11] alefka I have some questions about 0mq. Could u help me?
[05:11] umesh Sustrik: can you explain with very simple example ...
[05:12] sustrik umesh: have a look here:
[05:12] sustrik aalefka: yes
[05:12] umesh is there are sample code given in the cookbook or anywahere where we have zmq_forwarder in between
[05:12] umesh ok
[05:13] alefka I created publisher socket (ZMQ_PUB) and bind it. Can I wait a connection from client on this socket?
[05:14] sustrik yes
[05:14] sustrik well, you mean get notified?
[05:14] sustrik then no
[05:17] alefka sustrik: How I can wait? After binding I begin send messages, but because there is no connected subscribers I send this messages in the nowhere
[05:17] sustrik that's the nature of the thing
[05:18] sustrik PUB is like a radio transmission
[05:18] sustrik if noone is listening, the data go to nowhere
[05:18] sustrik what are you trying to do?
[05:23] alefka I need some client/server app which can dispatch about million messages per sec. In my REQ/REP socket tests there is not enough throughput, therefore I trying implement this logic on the PUB/SUB sockets.
[05:27] sustrik sure
[05:27] sustrik but why do you need not to drop messages when there's no client?
[05:28] sustrik how many listerenrs there are in your scenario?
[05:28] sustrik just a single one?
[05:29] alefka sustrik: >> sure. I executed latency tests on a 100 000 messages (16 bytes) and it spend about 5 secs
[05:30] alefka sustrik: >> just a single one? : yes
[05:30] sustrik so all you need is one-to-one communication?
[05:30] sustrik why aren't you using standard sockets then?
[05:31] sustrik (there's a PAIR socket for this use case, but it's still experimental and misses features such as reconnection)
[05:35] alefka yes. by command of my boss "just test it" :)
[05:36] alefka >> (there's a PAIR socket... I will try this feature
[05:38] sustrik alefka: actually, it won't work
[05:39] sustrik the problem you are trying to solve is still unresolved in PAIR
[05:39] sustrik anyway, what's the scenario 0mq is intended to be used in?
[05:39] sustrik one-to-many data distribution?
[05:40] alefka one-to-one
[05:40] sustrik ok, there's one option
[05:41] sustrik there are DOWNSTREAM and UPSTREAM sockets
[05:41] sustrik these are meant for pipelining
[05:41] sustrik but let's say your one-to-one scenario is a simple pipeline
[05:41] sustrik use DOWNSTREAM on the publisher side
[05:41] sustrik and UPSTREAM on the receiver side
[05:42] umesh
[05:42] sustrik the messages should not be dropped
[05:42] umesh where can i get this ?
[05:42] sustrik ?
[05:42] umesh its not wirh zeromq sources which I dodnloaded ...
[05:42] umesh with*
[05:42] umesh the example mentioned in this is not with zeromq sources ..
[05:42] sustrik it's old outdated tutorial
[05:42] umesh ok
[05:43] sustrik use the github project i've pointed you to
[05:43] umesh ok
[05:43] sustrik see the README file
[05:43] alefka How I understand they are contains some buffer inside which stores output/input messages
[05:43] alefka is it so?
[05:43] umesh I thought this is the more sopisticated example of the link you provided ... therefore just wanted to have a look at it ...
[05:44] sustrik alefka: yes there are buffers inside sockets
[05:44] sustrik umesh: it was the old version of the same thing
[05:44] sustrik some 2 years old :)
[05:45] umesh ok
[05:45] umesh sustrik: one question ..
[05:45] sustrik go on
[05:45] alefka sustrik: ok
[05:45] umesh zmq does not support many to many directly ... right ...
[05:46] sustrik you mean without device in a middle?
[05:46] umesh but this way of doing broker achitecture or many to many will add more latency/delay ...
[05:46] sustrik you can do that but it's annoying
[05:46] umesh yup ... without device in the middle ...
[05:46] sustrik you have to bind each publsiher
[05:47] sustrik and connect to each publisher from each subscriber
[05:47] sustrik that means subscriber should be aware of all publishers
[05:47] umesh which will be better performance wise ...
[05:47] sustrik yes, it'll be better
[05:47] sustrik but it's management nightmare :)
[05:47] sustrik it's up to you
[05:48] umesh and when you say annoying ... what is annoying nature in it ...
[05:48] umesh ok
[05:48] sustrik you have to maintail list of publishers at each subscriber
[05:48] umesh annoying is subsriber should know publishers ...
[05:48] sustrik yes
[05:49] umesh ok
[05:49] umesh thanx ...
[05:49] sustrik np
[05:50] umesh also adding more publishers will be diffcult in that case ...
[05:51] sustrik yes
[05:52] sustrik it can be done in opposite way also: publishers have to know subscribers
[05:52] sustrik but that's even worse
[05:52] umesh hmmm ...
[05:52] sustrik you can still use multicast
[05:53] sustrik all the publishers publish to the same multicast group
[05:53] sustrik all the subscribers subscribe to the multicast group
[05:53] sustrik but it works on LAN only
[05:54] umesh yup
[05:54] umesh everything has some procs and cons ... better to me looks like is device in the middle ...
[05:54] sustrik ack
[05:55] umesh ack ?
[05:55] sustrik yes (acknowledged) :)
[05:55] umesh ok
[05:55] umesh you believe the same ... :)
[05:57] umesh one more question : not related to this discussio...
[05:57] umesh zmq does not have zmq_select ... why ?
[05:57] umesh it does have zmq_poll ... but not zmq_select ?
[05:58] sustrik pollsets used in select are in fact bitmaps
[05:59] sustrik you can use that approach when you fds are integers
[05:59] sustrik sockets in 0mq are void*
[05:59] sustrik that doesn't fit to a bitmap
[05:59] umesh ok
[06:00] umesh is there any way of using standard select with zmq_sockets ...
[06:00] umesh ?
[06:01] umesh i think no ...
[06:01] umesh but still thought asking you ...
[06:09] sustrik umesh: no way at the moment
[06:09] sustrik zmq socket is not a file descriptor
[06:09] sustrik so you cannot use it for polling via *any* standard mechanism
[06:10] sustrik we'll be moving that way in the future though
[06:12] umesh ok
[06:24] umesh where is the source code for zmq_server ?
[06:24] umesh I did not find any ...
[06:24] sustrik there's no zmq_server
[06:24] sustrik where are you looking at?
[06:25] umesh ..
[06:26] umesh I also looked at teh sources
[06:26] umesh the*
[06:26] sustrik it's outdated
[06:26] umesh it mentions about zmq_server ...
[06:26] sustrik i should delete these two papers
[06:26] umesh
[06:26] umesh even this as well
[06:26] umesh why ?
[06:27] sustrik it's old documentation
[06:27] umesh ok
[06:27] sustrik it should have been deleted long ago
[06:27] umesh is there any new example of this ...
[06:27] sustrik the one i pointed you to
[06:27] sustrik is there any problem with it?
[06:28] umesh nope ...
[06:29] umesh i was just looking at the different code examples ...
[06:29] umesh to understand zmq in much better way ...
[06:30] sustrik dunno, have a look at the blogs
[06:30] sustrik there's "multithreaded server in 10mins." might be helpful
[06:30] umesh also in that chat example you pointed out me ... is there any way to listen to specific events only ...
[06:30] umesh we ca do this by selective subscribe ... right ?
[06:31] umesh can*
[06:31] sustrik yes
[06:31] umesh instead of subscribing to all messages ...
[06:31] umesh ok
[08:28] alefka anybody here?
[08:29] guido_g sure
[08:33] alefka guido_g: it seems I found some bug or I'm mistaken.
[08:36] alefka guido_g: why some packets not received when I send them from publisher socket (server side, socket listening) to the client (whos connecting)
[08:38] alefka anybody...
[08:49] alefka :(
[08:52] sustrik alfka: i'm here now :)
[08:53] alefka sustrik: Hi :)
[08:53] alefka it seems I found some bug or I'm mistaken.
[08:53] alefka why some packets not received when I send them from publisher socket (server side, socket listening) to the client (whos connecting)
[08:53] sustrik while the client is _connecting_ and not yet _connected_ the messages are dropped
[08:54] sustrik as i said, it's like radio transmission
[08:54] alefka but waiting a client connection
[08:54] sustrik ?
[08:54] alefka just reading from keyboard char
[08:54] alefka but I'm waiting a client connection
[08:54] sustrik but it's not yet connected, right?
[08:55] sustrik it's PUB/SUB?
[08:55] sustrik can you show your test program?
[08:55] alefka yes
[08:56] alefka how I can send you it?
[08:57] sustrik it doesn't seem to get through
[08:57] sustrik paste is to some public board
[08:58] sustrik or send it be email
[08:59] alefka what is your email?
[08:59] sustrik send it to the mailing list
[08:59] sustrik or is it private?
[08:59] sustrik confidential i mean?
[09:00] alefka I don't see.
[09:01] alefka ????
[09:02] sustrik see here:
[09:11] alefka files attach to the message or insert into the body?
[09:11] alefka *as text
[09:12] sustrik doesn't matter
[09:20] alefka sended
[10:31] versificateur hello
[10:32] versificateur could someone good in JNI have a look at this?
[10:33] versificateur
[10:34] versificateur I tried to run a java code on it but it crash
[10:34] versificateur sustrik: your help is welcome
[10:36] sustrik versificateur: you have to convrt jobject representing the 0MQ socket to handle to the socket
[10:36] sustrik iirc the jobject contains the void* stored as jint
[10:36] sustrik or something like that
[11:02] versificateur i correct it
[11:02] versificateur thanks
[11:02] versificateur it's seems that it works
[11:03] versificateur i'll test with some REQ/REP code
[11:06] umesh subscribe to specific topic not working ...
[11:06] umesh I am using the simple chat program using forwarder ...
[11:07] umesh I want to display the specic messages ...
[11:07] umesh so what I did was I subscribed to the specific messages ... using s.setsockopt (ZMQ_SUBSCRIBE, "umesh\x00", 0);
[11:08] umesh and i haven't change the code for prompt ...
[11:08] sustrik the third argument is the size
[11:08] umesh ohhh ... i missed it ...
[11:08] umesh sorry ...
[11:08] sustrik :)
[11:08] umesh I knew it but ... i forgot ...
[11:10] CIA-17 zeromq-examples: 03Martin Lucina 07master * r8cee253 10/ zmq-camera.c : Update to 0MQ 2.0.7 API -
[11:12] versificateur sustrik: if i bind a socket will it be opened or only when i started send/recv data??
[11:14] versificateur because i bind a PUB socket on a TCP port but executing a netstat command do show it
[11:14] sustrik bind translates to standard TCP bind
[11:15] sustrik connect translates to TCP connect
[11:26] CIA-17 zeromq-examples: 03Martin Lucina 07turbo * r267de4a 10/ zmq-camera.c : Update to 0MQ 2.0.7 API -
[11:33] umesh in setsocket_opt like s.setsockopt (ZMQ_SUBSCRIBE, "xy.z\x00", 5);
[11:33] umesh why it is required to give \x00 ...
[11:33] umesh if i just give "xy.z" will also be null terminated ...
[11:33] umesh ?
[11:34] guido_g the \000 is part of the subscription
[11:35] sustrik it's used as delimiter in the example
[11:35] sustrik if it's confusing use some other character instead
[11:35] sustrik like |
[11:35] sustrik or something
[11:36] umesh ok
[11:37] umesh its really very confusing ...
[11:37] sustrik let me see...
[11:37] umesh i think in cookbook you can update this ...
[11:37] umesh also strlen gives difficulty if we use delimiter as \x00 ..
[11:38] sustrik umesh: want to do that?
[11:40] umesh yup I can also do that ...
[11:40] sustrik great, thanks!
[11:41] umesh but do I have rights to change cookbook text ...
[11:41] sustrik you have to join the wiki to be able to edit
[11:41] sustrik see left below
[11:41] umesh ok
[11:41] umesh I have already joined it ...
[11:41] sustrik ah, then just edit it
[11:46] umesh does password update takes time ...
[11:46] umesh i am not able to login ... it says login and password does not match
[11:46] sustrik :)
[11:46] sustrik dunno
[11:47] sustrik maybe you've made type when entering it?
[11:47] sustrik typo
[11:47] umesh confirmed 2 times ...
[11:48] sustrik ignore it and create a new account
[11:55] umesh done
[11:56] umesh check the page once ...
[11:56] umesh
[11:56] umesh one more question : When I susbscribe to a topic ...
[11:56] umesh I get complete message with topic prefix ...
[11:56] umesh is it correct behaviour ?
[11:57] umesh because I have assumed that it will only give me the message without topic ...
[12:01] guido_g where does this assumption come from?
[12:01] guido_g the topic is part of the (multipart-)message
[12:13] umesh it was just my thinking that topic will be removed before giving it to user ... not from any reading ...
[12:16] versificateur how can i read ONLY a part of a multipart message with java binding
[12:17] versificateur for example with a socket like this
[12:17] versificateur jsonSubscriber.setsockopt(ZMQ.SUBSCRIBE, "servers.smpp");
[12:17] sustrik umesh: look good
[12:17] sustrik but you forgot this piece: "Messages that receivers are going to subscribe to have to start with a NULL-terminated string (the topic)"
[12:17] sustrik it's |-terminated now
[12:18] versificateur actually i got two messages
[12:18] sustrik if you send two messages
[12:18] sustrik you receive two messages
[12:19] sustrik there's nothing to do about that
[12:20] umesh yup
[12:20] umesh I will change that ...
[12:22] umesh done the change ...
[12:24] versificateur i thought that if subscribe with the channel name i'll only receive one message
[12:25] versificateur the message alone, not with the channel name as another message
[12:27] sustrik it's fifo
[12:27] sustrik what goes in, get out
[12:28] sustrik umesh: thx
[12:31] versificateur yes I knew... but when i use the python bindings i only receive the message itself not the channel
[12:33] versificateur so what <socket>.setsockopt(ZMQ.SUBSCRIBE, <some channel>); does in java bindings?
[12:35] sustrik see zmq_setsockopt(3) man page
[12:39] umesh in zmq how we can do content based publish/subscrive instead of topic based ...
[12:40] sustrik you want to add your own matching algorithm?
[12:42] umesh yup ...
[12:43] umesh how can i do that if I need to do ...
[12:43] umesh it not decided yet ... but I am trying to evalute that option as well ...
[12:54] versificateur sustrik: so SUBSCRIBE setsockopt is only a filter ? i'll receive the channel submessage as well as the data submessage?
[12:57] versificateur like i started with the python bindings which has recv_multipart that handle each part sent I thought it was like that in java also
[12:57] versificateur but i'll have to do my recv multipart myself
[12:58] versificateur sustrik: I need more information on device behavior like it's not documented
[13:00] versificateur it's seems that it run an infinite loop in background
[13:01] versificateur because code after it don't run
[13:03] sustrik umesh: you have to plug in your own algorithm
[13:03] sustrik the current algorithm is placed in src/prefix_tree.hpp and .cpp
[13:04] sustrik versificatuer: there's no such thing as channel and data submessages
[13:04] sustrik there are just messages
[13:04] sustrik some of them are single-part
[13:04] sustrik some of them are multipart
[13:05] sustrik subscription means 0MQ looks at the message and checks whether it's beginning matches the subscription
[13:05] sustrik if not so it drops it
[13:05] sustrik that's it
[13:05] versificateur well
[13:05] versificateur i understand
[13:05] versificateur it's a filter
[13:06] sustrik versificateur: wrt device, yes it uses your thread to loop
[13:07] versificateur ok it's why you put it at the end of your multithreaded example
[13:08] sustrik yes
[13:08] versificateur thanks!
[13:08] sustrik np
[13:09] versificateur i finished test of Device implemetation in java
[13:09] versificateur i'll put it on giithub and ask you for a pull equest
[13:14] sustrik grat, thanks!
[13:21] versificateur is SNDMORE in java bindings implemented to send mutipart message?
[13:24] sustrik not sure
[13:24] sustrik let me see
[13:25] sustrik a-ha it misses
[13:26] sustrik should go to
[13:26] sustrik like this:
[13:26] sustrik public static final int NOBLOCK = 1;
[13:26] sustrik public static final int SNDMORE = 2;
[13:27] sustrik will you add that to your fix?
[13:27] versificateur yes
[15:36] sustrik versificateur: there's a question for you on the mailing list
[15:36] versificateur susrik: okay
[15:37] versificateur i send the pull request to you only?? or to mato too?
[15:39] versificateur sustrik:???
[15:40] sustrik send it to me
[15:41] sustrik thx!
[15:42] versificateur :)
[15:42] versificateur the link to check the mailing list question
[15:42] versificateur i don't find it
[16:51] mikko sustrik:
[16:52] mikko happens when you dont send/recv/bind/connect
[16:52] mikko just create context and socket
[19:32] sustrik mikko: i think it's OK
[19:33] sustrik command_t structure that is being sent
[19:33] sustrik is actually a union
[19:33] sustrik for this particular command type there is less actual data than sizeof (command_t)
[19:34] sustrik that's why valgring complains
[20:28] dermoth Hey there... It's been a while I wantes to ask this; I'm curious about the high water mark
[20:28] dermoth /swap options, specifically how they work and where the data is stored...
[20:29] sustrik dermoth: hi
[20:29] dermoth hy sustrik
[20:29] dermoth hi
[20:29] sustrik hwm limits number of messages in the memory
[20:29] dermoth I understand the concept
[20:30] sustrik swap is not ported from 1.0 yet
[20:30] sustrik but should arrive soon
[20:30] dermoth oh so it's just a no-op right now?
[20:30] sustrik yes
[20:30] sustrik it should simply offload the data on the HD when memory limits are hit
[20:33] dermoth Ok, assuming it was working as expected, lets say I have web server workers that connects using an upstream/downstream scheme... could I just set HWM to zero (or nearly zero) to swap everything to disk when my forwarder device goes down? If the server crash and gets back up, will zeromq try to resume the transferts as soon as it is initialized? connected to the same device again?
[20:34] sustrik no, it's not a persistence device
[20:34] dermoth obviously with ZMQ_SWAP too ;)
[20:34] sustrik it's swap
[20:35] dermoth ok, so it's just to avoid using up all memory
[20:35] sustrik used just to handle the situation when the data don't fit into memory
[20:35] sustrik exactly
[20:35] dermoth I see
[20:43] travisbrady Where can I find the stock exchange example? ./configure --with-exchange isn't recognized anymore.
[20:52] sustrik it haven't been ported to 0MQ/2.0 yet
[20:53] sustrik you can give it a try
[20:56] travisbrady Ahh, ok, I'll take a look in the old repo on github
[22:45] jugg ZMQ_PAIR - for inproc:// endpoints, is the implementation usable? The docs say its still experiemental (missing auto-reconnection), but I think that may not be so applicable to inproc?
[22:47] dermoth jugg, I guess inproc reconnection migth be needed if you have threads that may restart and re-bind... I have no idea how stable it is though :)
[22:51] jugg For this implementation, that situation won't(!) occur.
[22:53] jugg hmm, "auto-reconnection" I hope has different meaning than one side of the endpoint closing the socket and re-opening though.
[22:55] jugg sockets are per thread, so it would be necessary to close the socket on thread exit, and create the socket on thread start. I'd expect that use case *not* to fall under the missing "auto-reconnection" feature.
[22:56] jugg But my expectations don't always line up with reality :)
[23:05] dermoth jugg, I could be wrong, but my understanding is that auto-reconnection means if the side that bind shuts down and restart, the socket will re-connect.
[23:06] dermoth jugg, this is how it works with TCP...
[23:08] dermoth note that unless you use req/rep sockets there is no guarantee the data will be delivered - however as long as your socket stays alive it will keep trying to deliver the data (that is with TCP)
[23:14] jugg Yah, if the bind socket goes away, I could see that "auto-reconnect" could be applicable to connected sockets. thanks.