[Time] Name | Message |
[00:27] andrewvc
|
cremes: ping
|
[07:05] maxpn
|
how can I handle timeouts in pyzmq ?
|
[07:05] maxpn
|
for PUSH socket
|
[07:24] maxpn
|
hmm... looks like I should use NOBLOCK in a loop
|
[10:26] s0undt3ch
|
hello ppl
|
[10:27] s0undt3ch
|
xreq_socket.recv(zmq.NOBLOCK) should never, ever block right? seems it's blocking for at least 1.5 secs for me :\
|
[10:32] mikko
|
maxpn: you can use poll
|
[10:32] mikko
|
s0undt3ch: it shouldn't block
|
[10:33] s0undt3ch
|
mikko: I'm using evenlet with zmq, it has block detector, I'm setting it to 1.5 secs, and it's triggering :|
|
[10:34] mikko
|
are you doing more than "xreq_socket.recv(zmq.NOBLOCK)" in that block of code?
|
[10:35] s0undt3ch
|
mikko: involved blocks of code -> http://paste.pocoo.org/show/317519/
|
[10:35] maxpn
|
is there an option like recv(NOBLOCK, timeout= ... )
|
[10:37] maxpn
|
would be very usable
|
[10:38] mikko
|
maxpn: noblock returns immediately
|
[10:39] mikko
|
maxpn: it's very simple to do with poll
|
[10:39] maxpn
|
mikko: yes, but in that case I have to sleep
|
[10:39] maxpn
|
for some amount of time
|
[10:40] mikko
|
maxpn: https://github.com/mkoppanen/httpush/blob/master/src/helpers.c#L110
|
[10:41] mikko
|
there is similar thing in C
|
[10:41] mikko
|
it uses poll to handle the timeout
|
[10:42] mikko
|
s0undt3ch: are you getting eagain?
|
[10:42] s0undt3ch
|
mikko: nope
|
[10:42] mikko
|
s0undt3ch: you are sure that it actually blocks inside recv call?
|
[10:43] maxpn
|
ok I see, is such an option for Python?
|
[10:43] mikko
|
maxpn: yes, poll
|
[10:43] s0undt3ch
|
mikko: aparently is, I'm trying to refactor it a bit to try and confirm it
|
[10:43] mikko
|
maxpn: https://github.com/zeromq/pyzmq/blob/master/examples/poll/pair.py
|
[10:48] maxpn
|
as I understand poller.poll() returns a list of paris (socket,state) , right?
|
[10:53] mikko
|
maxpn: i don't really know how it's implemented in python
|
[10:55] maxpn
|
poller.poll() will return data when complete ZMQ message received in POLLIN socket?
|
[10:55] mikko
|
it will retur when there is a state change in one or more of the sockets in the poll set
|
[10:57] s0undt3ch
|
mikko: how can I know if a message is complete?
|
[10:57] mikko
|
s0undt3ch: you mean with x(rep|req) sockets?
|
[10:57] s0undt3ch
|
mikko: yes
|
[10:58] mikko
|
the first parts will contain RCVMORE flag
|
[10:58] mikko
|
the part that doesn't contain it terminates the message
|
[10:59] s0undt3ch
|
mikko: so I can append to a buffer untill I get RCVMORE at wich time I can process the message right?
|
[11:00] mikko
|
the other way around
|
[11:00] s0undt3ch
|
ah yes
|
[11:01] mikko
|
RCVMORE indicates that there is at least one more part coming
|
[11:01] maxpn
|
is recv_json() handles that situation correctly?
|
[11:01] s0undt3ch
|
mikko: xrep.getsockopt(zmq.RCVMORE) right?
|
[11:01] mikko
|
s0undt3ch: yes
|
[11:02] mikko
|
maxpn: i don't really know, that sounds like a python specific method
|
[11:02] s0undt3ch
|
mikko: RCVMORE is socket option right?
|
[11:02] s0undt3ch
|
ie, I test it in getsockopt
|
[11:03] mikko
|
yes
|
[11:03] s0undt3ch
|
mikko: Hurray! Thanks! Now working as suposed
|
[11:04] s0undt3ch
|
was implementing a heartbeater and each of the hearts, if more than one, would fail in turns, which let me to think that my could must have been wrong
|
[11:06] s0undt3ch
|
but now one of the processors is at 100% :)
|
[11:06] s0undt3ch
|
need to refactor again
|
[11:12] mikko
|
s0undt3ch: spinning on something?
|
[11:13] s0undt3ch
|
mikko: I think I'm checking socket state too often, unfotunately, eventlet does not allow listening to fd changes
|
[11:13] s0undt3ch
|
ok, now dropped to 8-10%
|
[11:13] s0undt3ch
|
:)
|
[11:23] s0undt3ch
|
handling 50 heartbeats at 7-9% ;)
|
[12:05] s0undt3ch
|
a REQ socket can be connected to multiple "endpoints" right? can it send a message to a specific endpoint?
|
[12:06] s0undt3ch
|
or sending will send to all and the receiving endpoint cannot know for who the message is?
|
[12:56] zchrish
|
That's my question. I am reading 3.0 roadmap and it appears that TCP and PUB/SUB are distributed across all receivers.
|
[12:57] zchrish
|
I think in this reference TCP=REQ.
|
[17:01] sustrik
|
s0undt3ch: you send a request to a "cloud"
|
[17:02] sustrik
|
so it eventually gets to someone
|
[18:02] s0undt3ch
|
sustrik: I'm trying to do some aync message handling. I have gobject listening for the socket's FD and when there's a change on it, a function is called. For REQ/REP, this is ok, I handle incoming message aync, and outgoing messages sync, however, for XREP, I seem to be unable to do async out messages
|
[18:03] s0undt3ch
|
especially since I'm calling socket.send_pyobj(data)
|
[18:03] s0undt3ch
|
I think that makes the socket go into zmq.POLLOUT
|
[18:03] s0undt3ch
|
but how do I handle zmq.POLLOUT?
|
[18:06] s0undt3ch
|
intead of directly calling send() should I put messages in a queue and then handle them in the queue?
|
[18:07] s0undt3ch
|
am I making any sense?
|
[18:15] sustrik
|
have a look at zmq_getsockopt(3), the ZMQ_FD and ZMQ_EVENTS section
|
[18:15] sustrik
|
ZMQ_FD signals that "something happened"
|
[18:15] sustrik
|
you have to get ZMQ_EVENTS to find out what actually happened
|
[18:17] s0undt3ch
|
sustrik: that part I know, that's how I'm handling incoming messges, I'm just failing to get own to handle outgoing messages with the same resources, ie, EVENTS and FD
|
[18:19] s0undt3ch
|
I already catched the (events & POOLOUT), just don't know what and from where to actualy send, since I've already done .send()
|
[18:23] s0undt3ch
|
s/own/how/
|
[18:24] s0undt3ch
|
is there a way to get the contents of the outgoing XREP buffer?
|
[18:25] s0undt3ch
|
perhaps better
|
[18:25] s0undt3ch
|
if I try to send NOBLOCK, how do I handle it?
|
[18:28] sustrik
|
send with a non-block
|
[18:28] sustrik
|
if it cannot be sent it returns EAGAIN
|
[18:28] sustrik
|
then you can poll on ZMQ_FD
|
[18:28] sustrik
|
once it signals POLLIN
|
[18:29] sustrik
|
you getsockopt(ZMQ_EVENTS)
|
[18:29] sustrik
|
if ZMQ_POLLOUT is set, you can send the message
|
[19:33] zchrish
|
Is using a vector a supported method to implement socket_t? I want to service 10,000 different subscription channels so that each channel has independent information. According to the roadmap, all data goes to all subscribers even if a subscriber is only interested in certain data.
|
[19:42] mikko
|
zchrish: currently the filtering is done at subscriber side, correct
|
[19:42] zchrish
|
mikko: Yes, that is what I have learned. I don't want to do that; too much information flying across the publisher.
|
[19:54] sustrik
|
zchrish: why not use a single pub socket with subscriptions instead of handling 10,000 sockets
|
[19:55] sustrik
|
and rather focus on implementing subscription forwarding inside 0mq
|
[19:56] zchrish
|
sustrik: I'd like to but as far as I can tell, if my subscriber only wants data to travel across the wire for only 1 of the 10,000 channels, that isn't currently supported. The roadmap says that is a 3.0 feature possibly.
|
[19:57] sustrik
|
it can go even to 2.x if someone actually implements it
|
[19:57] sustrik
|
there are several people trying at the moment
|
[19:57] zchrish
|
So what to do now?
|
[19:58] sustrik
|
it's up to you
|
[19:58] sustrik
|
you can either write your code to use 10,000 sockets
|
[19:58] sustrik
|
or instead try to implement the subscription forwarding inside 0mq
|
[19:58] zchrish
|
So my idea was to create a vector of publishers and bind them to different ports. Sort of crude but was wondering if it was even supported.
|
[19:59] sustrik
|
yes, you can do that
|
[19:59] zchrish
|
can you estimate the work difference?
|
[19:59] zchrish
|
between the 2 methods?
|
[20:00] sustrik
|
well, the developer has to estimate the work
|
[20:00] sustrik
|
have a look at the mailing list
|
[20:00] zchrish
|
understood; thank you.
|
[20:00] sustrik
|
the subscription forwarding was discussed several times there
|
[20:01] sustrik
|
you'll get some idea
|
[20:02] zchrish
|
At this point, I'd like to remain a user of zeromq in my application and focus on that. Seems like great software but this feature seems really important as I guess people already know.
|
[20:03] sustrik
|
sure, it's up to you
|
[20:03] zchrish
|
But maybe I could be beta people's work if that'd help. I agree subscription forwarding is the way to go.
|
[20:11] sustrik
|
well, as i said, there are people trying, let's see what emerges
|