IRC Log


Sunday January 9, 2011

[Time] NameMessage
[00:27] andrewvc cremes: ping
[07:05] maxpn how can I handle timeouts in pyzmq ?
[07:05] maxpn for PUSH socket
[07:24] maxpn hmm... looks like I should use NOBLOCK in a loop
[10:26] s0undt3ch hello ppl
[10:27] s0undt3ch xreq_socket.recv(zmq.NOBLOCK) should never, ever block right? seems it's blocking for at least 1.5 secs for me :\
[10:32] mikko maxpn: you can use poll
[10:32] mikko s0undt3ch: it shouldn't block
[10:33] s0undt3ch mikko: I'm using evenlet with zmq, it has block detector, I'm setting it to 1.5 secs, and it's triggering :|
[10:34] mikko are you doing more than "xreq_socket.recv(zmq.NOBLOCK)" in that block of code?
[10:35] s0undt3ch mikko: involved blocks of code -> http://paste.pocoo.org/show/317519/
[10:35] maxpn is there an option like recv(NOBLOCK, timeout= ... )
[10:37] maxpn would be very usable
[10:38] mikko maxpn: noblock returns immediately
[10:39] mikko maxpn: it's very simple to do with poll
[10:39] maxpn mikko: yes, but in that case I have to sleep
[10:39] maxpn for some amount of time
[10:40] mikko maxpn: https://github.com/mkoppanen/httpush/blob/master/src/helpers.c#L110
[10:41] mikko there is similar thing in C
[10:41] mikko it uses poll to handle the timeout
[10:42] mikko s0undt3ch: are you getting eagain?
[10:42] s0undt3ch mikko: nope
[10:42] mikko s0undt3ch: you are sure that it actually blocks inside recv call?
[10:43] maxpn ok I see, is such an option for Python?
[10:43] mikko maxpn: yes, poll
[10:43] s0undt3ch mikko: aparently is, I'm trying to refactor it a bit to try and confirm it
[10:43] mikko maxpn: https://github.com/zeromq/pyzmq/blob/master/examples/poll/pair.py
[10:48] maxpn as I understand poller.poll() returns a list of paris (socket,state) , right?
[10:53] mikko maxpn: i don't really know how it's implemented in python
[10:55] maxpn poller.poll() will return data when complete ZMQ message received in POLLIN socket?
[10:55] mikko it will retur when there is a state change in one or more of the sockets in the poll set
[10:57] s0undt3ch mikko: how can I know if a message is complete?
[10:57] mikko s0undt3ch: you mean with x(rep|req) sockets?
[10:57] s0undt3ch mikko: yes
[10:58] mikko the first parts will contain RCVMORE flag
[10:58] mikko the part that doesn't contain it terminates the message
[10:59] s0undt3ch mikko: so I can append to a buffer untill I get RCVMORE at wich time I can process the message right?
[11:00] mikko the other way around
[11:00] s0undt3ch ah yes
[11:01] mikko RCVMORE indicates that there is at least one more part coming
[11:01] maxpn is recv_json() handles that situation correctly?
[11:01] s0undt3ch mikko: xrep.getsockopt(zmq.RCVMORE) right?
[11:01] mikko s0undt3ch: yes
[11:02] mikko maxpn: i don't really know, that sounds like a python specific method
[11:02] s0undt3ch mikko: RCVMORE is socket option right?
[11:02] s0undt3ch ie, I test it in getsockopt
[11:03] mikko yes
[11:03] s0undt3ch mikko: Hurray! Thanks! Now working as suposed
[11:04] s0undt3ch was implementing a heartbeater and each of the hearts, if more than one, would fail in turns, which let me to think that my could must have been wrong
[11:06] s0undt3ch but now one of the processors is at 100% :)
[11:06] s0undt3ch need to refactor again
[11:12] mikko s0undt3ch: spinning on something?
[11:13] s0undt3ch mikko: I think I'm checking socket state too often, unfotunately, eventlet does not allow listening to fd changes
[11:13] s0undt3ch ok, now dropped to 8-10%
[11:13] s0undt3ch :)
[11:23] s0undt3ch handling 50 heartbeats at 7-9% ;)
[12:05] s0undt3ch a REQ socket can be connected to multiple "endpoints" right? can it send a message to a specific endpoint?
[12:06] s0undt3ch or sending will send to all and the receiving endpoint cannot know for who the message is?
[12:56] zchrish That's my question. I am reading 3.0 roadmap and it appears that TCP and PUB/SUB are distributed across all receivers.
[12:57] zchrish I think in this reference TCP=REQ.
[17:01] sustrik s0undt3ch: you send a request to a "cloud"
[17:02] sustrik so it eventually gets to someone
[18:02] s0undt3ch sustrik: I'm trying to do some aync message handling. I have gobject listening for the socket's FD and when there's a change on it, a function is called. For REQ/REP, this is ok, I handle incoming message aync, and outgoing messages sync, however, for XREP, I seem to be unable to do async out messages
[18:03] s0undt3ch especially since I'm calling socket.send_pyobj(data)
[18:03] s0undt3ch I think that makes the socket go into zmq.POLLOUT
[18:03] s0undt3ch but how do I handle zmq.POLLOUT?
[18:06] s0undt3ch intead of directly calling send() should I put messages in a queue and then handle them in the queue?
[18:07] s0undt3ch am I making any sense?
[18:15] sustrik have a look at zmq_getsockopt(3), the ZMQ_FD and ZMQ_EVENTS section
[18:15] sustrik ZMQ_FD signals that "something happened"
[18:15] sustrik you have to get ZMQ_EVENTS to find out what actually happened
[18:17] s0undt3ch sustrik: that part I know, that's how I'm handling incoming messges, I'm just failing to get own to handle outgoing messages with the same resources, ie, EVENTS and FD
[18:19] s0undt3ch I already catched the (events & POOLOUT), just don't know what and from where to actualy send, since I've already done .send()
[18:23] s0undt3ch s/own/how/
[18:24] s0undt3ch is there a way to get the contents of the outgoing XREP buffer?
[18:25] s0undt3ch perhaps better
[18:25] s0undt3ch if I try to send NOBLOCK, how do I handle it?
[18:28] sustrik send with a non-block
[18:28] sustrik if it cannot be sent it returns EAGAIN
[18:28] sustrik then you can poll on ZMQ_FD
[18:28] sustrik once it signals POLLIN
[18:29] sustrik you getsockopt(ZMQ_EVENTS)
[18:29] sustrik if ZMQ_POLLOUT is set, you can send the message
[19:33] zchrish Is using a vector a supported method to implement socket_t? I want to service 10,000 different subscription channels so that each channel has independent information. According to the roadmap, all data goes to all subscribers even if a subscriber is only interested in certain data.
[19:42] mikko zchrish: currently the filtering is done at subscriber side, correct
[19:42] zchrish mikko: Yes, that is what I have learned. I don't want to do that; too much information flying across the publisher.
[19:54] sustrik zchrish: why not use a single pub socket with subscriptions instead of handling 10,000 sockets
[19:55] sustrik and rather focus on implementing subscription forwarding inside 0mq
[19:56] zchrish sustrik: I'd like to but as far as I can tell, if my subscriber only wants data to travel across the wire for only 1 of the 10,000 channels, that isn't currently supported. The roadmap says that is a 3.0 feature possibly.
[19:57] sustrik it can go even to 2.x if someone actually implements it
[19:57] sustrik there are several people trying at the moment
[19:57] zchrish So what to do now?
[19:58] sustrik it's up to you
[19:58] sustrik you can either write your code to use 10,000 sockets
[19:58] sustrik or instead try to implement the subscription forwarding inside 0mq
[19:58] zchrish So my idea was to create a vector of publishers and bind them to different ports. Sort of crude but was wondering if it was even supported.
[19:59] sustrik yes, you can do that
[19:59] zchrish can you estimate the work difference?
[19:59] zchrish between the 2 methods?
[20:00] sustrik well, the developer has to estimate the work
[20:00] sustrik have a look at the mailing list
[20:00] zchrish understood; thank you.
[20:00] sustrik the subscription forwarding was discussed several times there
[20:01] sustrik you'll get some idea
[20:02] zchrish At this point, I'd like to remain a user of zeromq in my application and focus on that. Seems like great software but this feature seems really important as I guess people already know.
[20:03] sustrik sure, it's up to you
[20:03] zchrish But maybe I could be beta people's work if that'd help. I agree subscription forwarding is the way to go.
[20:11] sustrik well, as i said, there are people trying, let's see what emerges