Thursday June 24, 2010

[Time] NameMessage
[03:37] alisdair is zeromq a protocol at all, or is it mostly an alternate interface to sockets?
[06:07] travlr alisdair: zmq transports a "blob" of data so the descriptor of the blob is provided by the app programmer.
[06:08] alisdair travlr: i get that, but does it wrap that at all? can i use zeromq at only one end of a connection, or does it need to be used at both?
[06:09] travlr alisdair: "only one end" ... do you mean using standard-sockets in the system... if so see zmq_poll() doc
[06:09] alisdair well
[06:09] alisdair if i write a client using zeromq
[06:09] alisdair does the server need to be aware of that?
[06:10] travlr is the server using a standard-socket?
[06:10] travlr bsd socket?
[06:11] travlr look at the zmq_poll() doc I think that will help answer your question.
[06:13] alisdair standard sockets
[06:13] travlr also, see the mato's zmq-camera example of integrating other system aspects:
[06:14] alisdair i just want to know, does zeromq add any wrapping to a packet? if i send a packet from a client written using zeromq, does the receiving system need to know anything about it to process it? is it just a regular udp/tcp packet?
[06:16] travlr zmq_recv() just recieves the "blob". no un-wrapping is necessary. it does wrap the blob with possible addressing and delineation protocol aspects.
[06:17] travlr see: zmq_tcp(7), zmq_pgm(7), zmq_ipc(7), and zmq_inproc(7) for details.
[06:17] travlr
[06:33] Umesh
[06:33] Umesh is this white paper outdated ?
[06:36] travlr Umesh: In a sense yes, but not totally. zmq2 is a departure from zmq1 in that is entirely brokerless and there is no message protocol. I don't know yet myself where all the similarities and differences lie.
[06:39] sustrik Umesh: in general, yes. what exactly are you interested in?
[06:41] Umesh I am intrested in load balancing for client server
[06:41] Umesh there it was how we can do load balancing in the zmq is described in that white paper
[06:42] Umesh while reading that paper i felt it is outdated ...
[06:42] Umesh so we can't do load balancing inbuilt in the zmq ... right ?
[06:42] Umesh for REQ/REP socket
[06:45] sustrik yes, there's load balancing in-built
[06:45] sustrik when there are several peers to REQ socket
[06:45] sustrik the requests are load-balanced among them
[06:45] Umesh ok
[06:46] Umesh then how we can choose the endpoint for that peer ...
[06:46] sustrik ?
[06:46] Umesh I haven't seen any such apis
[06:46] sustrik there's no need for specific apis
[06:46] Umesh means say i have a name server on m1, m2 and m3 ...
[06:47] sustrik s =socket (REQ);
[06:47] sustrik s.connect (m1);
[06:47] sustrik s.connect (m2);
[06:47] sustrik s.connect (m3);
[06:47] sustrik that's what you meant?
[06:47] Umesh then I create a REQ socket and then connect ...
[06:48] Umesh then send data ...
[06:48] sustrik yes
[06:48] Umesh so data will go to machine where i connect ...
[06:48] Umesh right ...
[06:48] sustrik or you can bind and let name servers connect to your app
[06:48] Umesh ok so you mean i should connect to all name server ... right ?
[06:48] sustrik depends on what you are doing
[06:48] sustrik right
[06:48] Umesh to enable laod balancing ...
[06:49] Umesh is there any way to know all machines on the port using the zmq ... something like getaddribfo
[06:50] Umesh just a summary of our discussion : To enable load balansing in the zmq ... REQ socket should connect to all REP sockets it know ... right ?
[06:50] sustrik REQ sockets should have multiple peers
[06:51] Umesh yup ...
[06:51] sustrik whether they are connected to are they connect themselves
[06:51] sustrik as for locating the machines... it's pretty hard when you think of Intetnet
[06:51] sustrik there may be a machine running somewhere in Ougadougou
[06:52] sustrik no, no way to locate all the instances
[06:52] Umesh ok
[06:52] sustrik what you may want is a shared queue
[06:52] sustrik i.e. all clients connect to the shared queue
[06:52] Umesh but for lan ... I can still use the getaddrinfo call ... to find all the machines to connect to ...
[06:52] sustrik all servers connect to the shared queue
[06:53] sustrik the share queue load balances the requests among the servers
[06:53] Umesh means use a queue device in between ... create one hop ..
[06:53] sustrik yes
[06:53] sustrik ad getaddrinfo: i am afraid it doesn't work that way
[06:53] Umesh but for lan ... I can still use the getaddrinfo call ... to find all the machines to connect to ...
[06:54] sustrik how would you do that?
[06:54] Umesh i will just not specify node in that call ... only specify port ... get all the machines listening on that port ...
[06:55] sustrik i doubt getaddrinfo works that way
[06:55] Umesh getaddrinfo(NULL, PORT, &hints, &servinfo))
[06:55] sustrik does it work?
[06:55] Umesh here PORT will be the port of the name server ...
[06:56] Umesh it works for normal tcp ip ...
[06:56] sustrik wow
[06:56] Umesh so i think it will work for zmq ...
[06:56] Umesh isn't it ...
[06:56] Umesh ?
[06:56] sustrik if it works then it would surely work
[06:56] sustrik even for 0mq
[06:57] Umesh ok
[06:57] Umesh i will give the try ...
[06:58] Umesh one more doubt ... which load balancing we does ?
[06:58] Umesh means round robin ... etc ?
[07:15] sustrik round robin
[07:16] Umesh ok
[07:27] Umesh hi sustrik ...
[07:27] Umesh i think i was wrong for getaddrinfo ...
[07:27] Umesh it will not work for normal tcp/ip as well
[07:39] sustrik :)
[07:55] Efhache Hi everybody
[07:56] Efhache I come back to you because I need some help... I'm learning how to use zmq... In this goal I test some example application.
[07:56] Efhache Right now I test the tcp client/server application example
[07:57] Efhache but I've a little problem... when I try to send my message I've an "unknown error" (returned by strerror) and the client of course segfault when I try to receive
[07:58] Efhache The error is "ERREUR -1 send: Unknown error 156384763 "
[07:58] Efhache Could you help me ???
[07:58] Efhache here is the line code used to send my message : rc = zmq_send (s, &query, 0);
[08:03] travlr If you post the code piece somewhere, I'll try to help.
[08:08] Efhache Can I post it here?
[08:08] Efhache (piece by piece) ?
[08:09] travlr use dpaste or gist or some similar web service so I may be able to see what might be incorrect
[08:12] travlr or or some other
[08:12] Efhache
[08:14] sustrik Efhache: 0mq defines few additional errors
[08:14] sustrik use zmq_strerror instead of strerror to decode them
[08:14] Efhache ok
[08:17] Efhache ERREUR -1 send: Operation cannot be accomplished in current state
[08:17] Efhache no more information for me...
[08:17] Efhache :-/
[08:21] travlr has the server been executed before the client?
[08:22] Efhache yes of course
[08:22] Efhache and the server is running
[08:22] sustrik REQ/REP sockets reuquire alternate sequence of sends & recvs
[08:22] sustrik so with REQ send a request, the receveive a reply
[08:23] sustrik with REP receiver a request, then send the reply
[08:23] sustrik if you fail to do so you'll get the above error
[08:24] travlr yeah i see it now too. you need to use ZMQ_REQ in your client.
[08:24] Efhache ok
[08:26] Efhache I replace REP by REQ... now I've not error on the send() function. But a segfault on the recv. I must rechange the context before received???
[08:26] travlr no but did you use ZMQ_REP in the server?
[08:27] Efhache Yes
[08:28] Efhache This code comes from the cookbook examples
[08:28] travlr yeah, it should work out of the box. do you want to post the server code to dpaste
[08:29] Efhache ok
[08:29] Efhache
[08:30] Efhache May be I'll study the code from the latency example
[08:32] travlr i don't know because it looks ok, but i may be over looking something
[08:33] Efhache It's ok now
[08:33] travlr what was it?
[08:33] Efhache I've foget the zmq_msg_init of my resultset variable
[08:34] travlr yes i see.. ok cool
[08:34] Efhache thank's a lot for your help ;-)
[08:35] travlr sure
[08:35] Efhache Just to be sure... with this example, I've just a P2P connection between processes ?
[08:35] Efhache If I'd like to make a communication between a process and two (or more) others
[08:36] Efhache I need to create each time a endpoint ?
[08:36] travlr REQ/REP you can use multiple peers.
[08:37] Efhache ok
[08:37] travlr just use connect to the socket multiple times for each client.. etc.
[08:37] travlr then it will load-balance in a round robin fashion.. i don't know if that's what you want
[08:38] travlr you may want PUB/SUB maybe..
[08:38] travlr or other
[08:43] Efhache I've make a rapid test with two client and it seems that it works... but I'd be sure... how can I make in the server to have a specific message for each client
[08:43] Efhache I can find in the server an id of the sender ?
[08:44] travlr i don't think so. try returning a specific message for each client.
[08:44] travlr for testing
[08:45] Efhache ok
[08:48] Efhache thank you for your help
[08:48] travlr k
[08:53] Efhache no chance... the server do not make automatically a difference between the senders... clients received all replies...
[09:00] travlr Efhache: From the zmq_socket doc: "Each request sent is load-balanced among all services, and each reply received is matched with the last issued request."
[09:52] sustrik Efhache: what do you want to do?
[09:52] sustrik if you want to reply to the requester use REQ/REP
[09:52] sustrik if you want to send to all client use PUB/SUB
[11:00] bavluo While compiling/install the PyZMQ extension I am getting "zmq/_zmq.c: In function ‘init_zmq’: ‘EMTHREAD’ undeclared (first use in this function)". I have cloned the latest ZMQ and PyZMQ for github
[12:02] Umesh Is there a way to write distributed REP server in zeromq ...
[12:12] sustrik yes, place a queue device in the middle
[12:26] Umesh ok
[12:27] Umesh but it will send a request to all the servers ... write ?
[12:27] Umesh right ?
[12:28] Umesh or it will load balance the traffic ?
[12:29] sustrik load balance
[12:29] sustrik REQ/REP always load balances
[12:30] sustrik PUB/SUB always distributes the messages to everyone
[12:31] Umesh just one question : I don't see any lb object in req.hpp
[12:31] Umesh its there in xreq.hpp
[12:31] Umesh so it is like xreq only have laod balancing ...
[12:34] Efhache travlr : I see only now your response... sorry... what happens if multi client send request, each of the clients received all the replies? right?
[12:35] Efhache sustrik : I'just want to make a lighted broker : when it received a message from one process it re-sent this message to all connected processes.
[12:36] sustrik Umesh: it's there but it's written by hand
[12:36] sustrik the code should be polished a bit so that REQ uses lb_t as well...
[12:37] sustrik Efhache: such a thing already exists
[12:37] sustrik use zmq_forwarder to do this
[12:38] sustrik have a look at an example here:
[12:38] Efhache but the example code that I've used seems to do it...
[12:38] sustrik sure, if you want to implement it yourself, why not
[12:40] Efhache witouht implemented it myself... the code from 2 hours ago seems do it... I've tested the "server" with two clients and the both have received the reply from servers
[12:40] Efhache may be a "bug" of my code or is it normal?
[12:40] sustrik what code? sorry, i missed it
[12:41] Umesh sustrik : code present in the devices/zmq_queue/zmq_queue.cpp ... can be straight away used as a queue in the middle right ?
[12:41] sustrik Umesh: yes
[12:41] Umesh ok
[12:41] sustrik there's an executable generated for convenience sake: devices/zmq_queue
[12:42] Umesh but it uses XREQ/XREP sockets .... and my client server uses REQ/REP ...
[12:42] Umesh then will it work with them ?
[12:42] sustrik yes
[12:42] Umesh ok
[12:54] Efhache sustrik : it was a problem in my code... with a remanence in a string used for reply...
[12:54] Efhache sustrik : have you an exemple of "broadcasting" ?
[12:56] sustrik what about this one:
[12:56] sustrik or this one: (Java vs. Python)
[12:57] sustrik or this one: (Python)
[13:11] Efhache sustrik : thank you... if I goof understand to make a sort of lighted broker. I will must creating two context. One for receiving and one tu publish (server part) and for the clients one for sending and one for subscribing
[13:11] Efhache is it right?
[13:12] sustrik what are you trying to do, i still don't get it
[13:12] sustrik can you describe the scenario?
[13:14] Efhache OK... one process who received messagre from two (or more) other process and re-sent the receive message to all connected processes
[13:36] Efhache in C language
[13:54] sustrik what you need is a single context
[13:54] sustrik one sub socket
[13:54] sustrik and one pub
[13:54] sustrik socket
[13:55] sustrik get messages from sub socket
[13:55] sustrik and send them to the pub socket
[13:55] sustrik that's exactly what zmq_forwarder does
[14:01] Efhache ok
[14:02] Efhache sorry but it's a new concept for me...
[14:03] Efhache have you an example in C ?
[14:05] Efhache I can not find information about zmq_forwarder in the doc (man page) just "comming soon"
[14:10] Efhache But if I read the chat example ( message do not come from the client... is it true? And the client can only receive from prompt...
[14:10] sustrik
[14:11] Efhache so I need to context no? one for sending message (from a client to prompt) and an other for SUB
[14:11] sustrik you can merge display and prompt into a single application if you want
[14:11] Efhache no ?
[14:11] sustrik nope you can open both sockets from the same context
[14:12] sustrik forget about multiple context, you're never going to need it
[14:12] sustrik unless you are doing very advanced stuff
[14:13] Efhache I think so I do not understand the concept of "context"... I'll read more documentation... sorry
[14:14] sustrik context is a global state of the library
[14:14] sustrik you need only one context
[14:15] Efhache ok
[14:16] Efhache absolute not link with a protocol or a specific socket