[Time] Name | Message |
[04:56] CIA-17
|
zeromq2: 03Martin Sustrik 07master * r27877d7 10/ src/tcp_connecter.cpp : EHOSTUNREACH is acceptable outcome from connect - http://bit.ly/dzcG79
|
[05:03] CIA-17
|
zeromq2: 03Martin Sustrik 07master * r5ee355d 10/ src/tcp_connecter.cpp : if connect asserts, exact error is reported - http://bit.ly/b0P5Qg
|
[05:56] umesh
|
can I use XREP and REQ socket pairs ...
|
[05:56] umesh
|
means in server I will use XREP and client will use REQ socket ...
|
[05:56] umesh
|
I want to do this because I want to send multiple responses to a single request ...
|
[05:57] sustrik
|
umesh that won't work
|
[05:57] sustrik
|
use XREQ/XREP
|
[05:57] umesh
|
ok
|
[05:58] sustrik
|
be aware that you have to copy routeback info from request to reply
|
[05:58] sustrik
|
by hand
|
[05:58] umesh
|
will I be able to send multiple responses to the single request ...
|
[05:58] umesh
|
?
|
[05:59] sustrik
|
when you get the request on the XREP side
|
[05:59] sustrik
|
it'll be composed of routeback info and your message
|
[05:59] sustrik
|
routeback info says how to route the reply back to the requester
|
[05:59] sustrik
|
you have to copy it to the reply
|
[05:59] sustrik
|
so that 0MQ knows where to send the message
|
[06:00] sustrik
|
REQ/REP does this for you; with XREQ/XREP you have to handle it by hand
|
[06:00] sustrik
|
btw, sending a multi-part reply won't do?
|
[06:04] CIA-17
|
zeromq2: 03Brian Buchanan 07master * rac90b7e 10/ (AUTHORS src/socket_base.cpp): issue 35 - ZMQ_RCVMORE sometimes erroneously returns false - http://bit.ly/dvw31S
|
[06:05] umesh
|
sustrik: I want to avoid parsing in client ...
|
[06:05] umesh
|
of multipart reply ...
|
[06:05] sustrik
|
then use multi-part reply
|
[06:05] sustrik
|
?
|
[06:05] sustrik
|
what parsing
|
[06:06] sustrik
|
in any case the point is that multi-part message is guaranteed to be atomic
|
[06:06] sustrik
|
you either get all parts or none
|
[06:06] umesh
|
if I send multipart reply then in the client when I recv ... I will get one message and then need to parse that multipart message
|
[06:07] umesh
|
ok
|
[06:07] umesh
|
means I will get multple recv ...
|
[06:07] sustrik
|
yes
|
[06:07] umesh
|
ok
|
[06:07] sustrik
|
see ZMQ_SNDMORE and ZMQ_RCVMORE
|
[06:09] umesh
|
no documention of this in zmq_setsocketopt manaul
|
[06:09] umesh
|
therefore I was not aware of this multipart reply ...
|
[06:11] umesh
|
not even in send and recv api
|
[12:31] sustrik
|
why ephemeral ports then?
|
[12:31] brickhazel
|
yes the nameservice locations are fixed, everything else is random
|
[12:33] brickhazel
|
we have a resource manager that is capable of dynamically starting a service anywhere in the network depending on loading
|
[12:33] brickhazel
|
we then want to be able to discover the service
|
[12:33] sustrik
|
if it's accessible :)
|
[12:33] sustrik
|
well, this may work in corporate environment, but won't work in Internet scenarios
|
[12:33] brickhazel
|
if what is accessible?
|
[12:34] brickhazel
|
true.. this is a closed system
|
[12:34] sustrik
|
the location where you want to start the service
|
[12:34] sustrik
|
yes, but 0mq is meant also for Internet environments
|
[12:34] sustrik
|
and thus ephemeral ports are not an option
|
[12:34] sustrik
|
you can fork the project and add the code if needed...
|
[12:35] brickhazel
|
hmm... not a big fan of forking
|
[12:38] brickhazel
|
ok, thanks for your help.. i need to think about this some more
|
[12:45] sustrik
|
brickhazel: think of services, http is on port 80, ssh is on port 22, why should your service be different
|
[12:46] brickhazel
|
sustrik: these are more like micro-services sprinkled throughout our closed network
|
[12:47] brickhazel
|
failover is a common occurrence
|
[12:47] brickhazel
|
it is a very dynamic system and service discovery is how we have been connecting things so far
|
[12:48] sustrik
|
sure, it's up to you
|
[12:48] brickhazel
|
i think there are too many endpoints to try to manage by hand (aka assigning fixed ports to services)
|
[12:48] sustrik
|
too many service types?
|
[12:48] brickhazel
|
yes
|
[12:48] sustrik
|
the person who implements the service should choose a port number for it
|
[12:49] sustrik
|
it's not that much work
|
[12:50] brickhazel
|
true... but this is a system of subsystems
|
[12:50] brickhazel
|
someone has to manage a master list of all ports
|
[12:50] brickhazel
|
across many different projects
|
[12:51] sustrik
|
IANA? :)
|
[12:51] brickhazel
|
:) yes but this is not internet scale
|
[12:51] sustrik
|
having a port number registered at IANA is good for your ego :)
|
[12:51] brickhazel
|
more like a distributed/parallel processing system
|
[12:52] brickhazel
|
:) i bet
|
[12:52] sustrik
|
i understand
|
[12:52] sustrik
|
the whole point of 0mq, however, is to make app you write infinitely scalable
|
[12:53] sustrik
|
unfortunately, local hacks like ephemeral port numbers interfere with that
|
[12:54] sustrik
|
that's why i don't want it in the trunk
|
[12:54] brickhazel
|
i understand
|
[12:59] umesh
|
I am getting the extra appended string on the server in REQ/REP sockets
|
[12:59] sustrik
|
?
|
[12:59] umesh
|
I send string "ns_set app app1" from client
|
[13:00] umesh
|
and I recieved "ns_set app app1fifo_fast" on server
|
[13:00] umesh
|
I checked pcap capture ... it shows correctly "ns_set app app1"
|
[13:00] sustrik
|
i assume there's no terminating zero in the string
|
[13:01] sustrik
|
thus printf prints also the bogus memory after the end of the string
|
[13:02] umesh
|
but original string does have ...
|
[13:03] sustrik
|
i assume you don't send it in the message
|
[13:18] umesh
|
now working ...
|
[13:19] umesh
|
it was the issue of null char ... i saw pcap as well ... end 0 was not there
|
[13:19] umesh
|
one more point ... how is multipart messages are implemented ...
|
[13:19] umesh
|
means only one message is send or multiple messges clubbed together ...
|
[13:20] umesh
|
and then send ...
|
[13:20] umesh
|
i assume later
|
[13:21] sustrik
|
the latter
|
[13:23] umesh
|
ok
|
[18:22] cremes
|
is the "request identity" prepended to messages sent on XREQ sockets guaranteed to remain the same for the life of the socket?
|
[18:24] sustrik
|
cremes: yes
|
[18:24] sustrik
|
hm, well, unless reconnection happens
|
[18:25] sustrik
|
if you set the identity by hand it'll never change
|
[18:25] sustrik
|
if you don't set it, it is generated by the peer each time TCP connection is established
|
[18:26] cremes
|
oh, i see
|
[18:26] cremes
|
i'll set it myself then
|
[18:26] cremes
|
i don't want it to ever change
|
[18:27] sustrik
|
this is to be done in the binding or in your app?
|
[18:27] cremes
|
in my app
|
[18:27] sustrik
|
ok
|
[18:27] sustrik
|
makes semse
|
[18:27] sustrik
|
sense
|
[18:27] cremes
|
i'm leaving the binding as "vanilla" as possible so it matches the C api closely
|
[18:27] sustrik
|
yes, that's good
|
[18:28] cremes
|
sustrik: on a related note, i'd like to confirm that it doesn't make any sense to set an identity on a SUB socket since it can't transmit
|
[18:28] sustrik
|
it does
|
[18:28] sustrik
|
the point with identities
|
[18:28] cremes
|
rephrase please
|
[18:29] sustrik
|
is that they allow to track resources assigned to particular connection
|
[18:29] sustrik
|
example:
|
[18:29] sustrik
|
when SUB socket connect to PUB socket
|
[18:29] versificateur
|
hello
|
[18:29] sustrik
|
PUB socket creates a message queue
|
[18:29] sustrik
|
for that particular connection
|
[18:30] sustrik
|
if the connection breaks and gets reestablished
|
[18:30] sustrik
|
SUB's identity allows the PUB to pair the new connection with the old queue
|
[18:30] cremes
|
ah, it maps the queue by identity? i think i get it now.
|
[18:30] sustrik
|
yes
|
[18:30] versificateur
|
sustrik: did you had a look to my jzmq modifications?
|
[18:31] cremes
|
thank you for the clarification
|
[18:31] sustrik
|
np
|
[18:31] sustrik
|
versificateur: yes i did
|
[18:31] sustrik
|
i replied on github
|
[18:31] sustrik
|
you've probably missed the message
|
[18:31] sustrik
|
anyway the problem is that every single line of zmq is touched in the patch
|
[18:31] sustrik
|
it's impossible to review it
|
[18:32] sustrik
|
maybe the line endings are different?
|
[18:34] versificateur
|
hum...
|
[18:34] versificateur
|
i can tell you files I added
|
[18:34] sustrik
|
you have written it on windows?
|
[18:34] versificateur
|
and files i modified...
|
[18:35] versificateur
|
no no not on windows
|
[18:35] versificateur
|
i currently work on Fedora 12 and Ubuntu 10.04
|
[18:35] sustrik
|
i recall there were changes to MSVC project
|
[18:35] sustrik
|
let me see...
|
[18:36] versificateur
|
i made them manually
|
[18:36] versificateur
|
then tested them on windows
|
[18:36] sustrik
|
ah
|
[18:37] sustrik
|
have a look here:
|
[18:37] sustrik
|
http://github.com/bluig/jzmq/commit/20c2ead95d48039950d52c929daf3de2b8d05e4f
|
[18:38] sustrik
|
how come everything is marked as changed?
|
[18:39] versificateur
|
probably because I'm not good with git
|
[18:39] sustrik
|
neither am i
|
[18:39] sustrik
|
hallo! any git experts around???
|
[18:41] sustrik
|
hm, maybe the diff itself failed in some strange way
|
[18:41] versificateur
|
perhaps i should remove the repository and restart from scratch???
|
[18:41] sustrik
|
there are 2 unmodified lines there...
|
[18:42] sustrik
|
well, you are testing it on linux and windows, right?
|
[18:42] versificateur
|
yeah
|
[18:42] sustrik
|
that's the same i would do
|
[18:42] sustrik
|
so let's do it this way
|
[18:43] sustrik
|
i'll add you as contributor to the project
|
[18:43] versificateur
|
:)
|
[18:43] sustrik
|
that'll allow you to commit yourself
|
[18:43] versificateur
|
thanks
|
[18:43] sustrik
|
you make sure that it builds/works on the two platforms
|
[18:43] sustrik
|
wait a sec
|
[18:45] sustrik
|
versificateur: done
|
[18:45] versificateur
|
:)
|
[18:55] travisbrady
|
versificateur: which language are you writing a binding for?
|
[19:00] versificateur
|
java
|
[20:20] sg
|
anybody on?
|
[21:06] soren
|
I'm hoping someone can help me explain this: http://pastebin.com/qsWYHhdK
|
[21:07] soren
|
When I run those two, the consumer only sees every 256th message.
|
[21:08] soren
|
I thought the producer might be going to fast, so I added the sleep statement.. Didn't help at all.
|
[21:08] soren
|
I'm sure I'm doing something silly, but I can't see what it is.
|
[21:11] soren
|
err... but now it works.
|
[21:12] soren
|
I can't imagine what I changed..
|
[21:12] sustrik
|
soren: hi
|
[21:13] sustrik
|
you are subsribed only for messages that start with zero byte :)
|
[21:13] sustrik
|
zmq_setsockopt (socket, ZMQ_SUBSCRIBE, "", 1);
|
[21:13] soren
|
Yeah, I fixed that.
|
[21:14] soren
|
...and I wasn't half an hour ago when it was acting strangely.
|
[21:14] soren
|
So, I read in the docs that zmq_send returning doesn't mean it's been sent.
|
[21:14] sustrik
|
nope, it's async
|
[21:15] soren
|
What I swear I was experiencing was that the consumer would only see anything happen once for every 256 messages sent by the producer, and it would only receive one message.
|
[21:15] soren
|
It was very odd.
|
[21:15] sustrik
|
it looks exactly as if the subscription for zero byte was in place
|
[21:16] soren
|
Why would anything go through that way?
|
[21:16] soren
|
I mean... Why would I see every 256th message?
|
[21:16] sustrik
|
you are putting sequence of integers into the message
|
[21:16] soren
|
Oh.
|
[21:16] sustrik
|
so the messages look like this
|
[21:16] soren
|
Oh!
|
[21:16] sustrik
|
00000000
|
[21:16] soren
|
Right, right.
|
[21:16] sustrik
|
01000000
|
[21:16] soren
|
I see where you're going.
|
[21:16] soren
|
That answers my next question.
|
[21:17] soren
|
...which was going to be "how does the sender set the "topic" of the message (i.e. what the consumer subscribes to)", but I get that now.
|
[21:17] soren
|
...but you're probably quite right.
|
[21:17] soren
|
zeromq was just too clever for me. :)
|
[21:17] sustrik
|
:)
|
[21:18] soren
|
Ok, another question...
|
[21:19] soren
|
I currently have an app that uses amqp to send messages around using a topic exchange. All nodes connect to the same broker and pass msgs around.
|
[21:20] soren
|
To achieve something similar with zeromq, I should use multicast, right?
|
[21:20] sustrik
|
multicast is quite hard to configure
|
[21:20] soren
|
Yeah, that's the impression I get.
|
[21:20] sustrik
|
if you don't care about having a single process in the middle
|
[21:20] sustrik
|
use zmq_forwarder
|
[21:21] sustrik
|
see example here: http://github.com/sustrik/zeromq-chat
|
[21:21] soren
|
Well, I do that now, but I'd like to not have to have that.
|
[21:21] sustrik
|
other option is to connect each subscriber to every publisher
|
[21:22] soren
|
Yeah, that would suck.
|
[21:22] sustrik
|
then multicast
|
[21:22] soren
|
I'd like to be able to add and remove both publishers and subscribers on the fly.
|
[21:22] soren
|
I've never wrapped my head around multicast addressing.
|
[21:22] sustrik
|
the point is that for all-to-all communication you need a central node
|
[21:22] sustrik
|
it can be either zmq_forwarder
|
[21:23] sustrik
|
of your network switch (in case of multicast)
|
[21:23] sustrik
|
so these are the only options:
|
[21:23] sustrik
|
1. connect each-to-each by hand
|
[21:23] sustrik
|
2. have zmq_forwarder in the middle
|
[21:23] sustrik
|
3. use multicast
|
[21:24] soren
|
Ok. I'm trying the multicast example from the cookbook..
|
[21:24] soren
|
I've replaced my zmq_bind and zmq_connects from the example in the pastebin with:
|
[21:24] soren
|
zmq_connect(socket, "epgm://eth0;239.192.1.1:5555");
|
[21:25] soren
|
and nothing is getting through. Does this not work locally?
|
[21:25] soren
|
On the same box, I mean.
|
[21:31] sustrik
|
it should
|
[21:32] sustrik
|
multicast loopback is set to on by default
|
[21:32] sustrik
|
not sure about eth0 though
|
[21:32] sustrik
|
try lo instead
|
[21:33] soren
|
eth0 has the MULTICAST flag set.
|
[21:34] sustrik
|
i mean, i am not sure that you can loopback via eth0
|
[21:34] sustrik
|
thy lo (127.0.0.1)
|
[21:34] sustrik
|
try*
|
[21:34] soren
|
Yeah, lo works.
|
[21:35] soren
|
The multicast addresss... Is that just something everyone needs to agree on?
|
[21:35] sustrik
|
there's specific interval of addresses assigned to multicast
|
[21:35] sustrik
|
within that interval your apps must agree on specific address
|
[21:36] soren
|
Right, right, sure.
|
[21:36] soren
|
Right, it was just that last bit I wasn't sure about.
|
[21:36] soren
|
How far does multicast reach? Broadcast domain?
|
[21:36] soren
|
Or can it go further somehow?
|
[21:37] sustrik
|
depends on how your routers are set
|
[21:37] soren
|
Ok.
|
[21:37] sustrik
|
but normally multicast is disabled on routers
|
[21:38] soren
|
Gotcha.
|
[21:38] soren
|
Ok, so not only must the interface used be UP and have MULTICAST set, it must also have an address.
|
[21:38] soren
|
eth0 only passed two of those requirements :)
|
[21:40] soren
|
sustrik: Thanks a lot! This has been very enlightening.
|
[21:40] sustrik
|
you are welcome
|