Wednesday April 21, 2010

[Time] NameMessage
[10:22] sjampoo awesome, got PyZMQ working without message copying, even with acquiring and releasing the GIL it isn't slower on small messages and about twice as fast on messages larger than 512K
[10:22] sustrik sjampoo: is that a patch to pyzmq?
[10:23] sjampoo yes
[10:23] sjampoo haven't committed it yet.
[10:23] sjampoo I want to have a chat with Brian first changed a few things
[10:24] sustrik can it in be made switching from copy to non-copy depending on message size?
[10:24] sjampoo I don't see any reason too, it is even faster on smaller messages!
[10:25] sjampoo So there hardly seems to be any locking overhead
[10:25] sustrik ah, i see
[10:25] sustrik anyway, nice work!
[10:26] sjampoo it works now with a message object instead of a string
[10:26] sjampoo so you could also pass it directly a buffer
[10:26] sustrik how does the buffer get deallocated?
[10:26] sjampoo I think this will allow for neat stuff such as Tee-ing from a regular TCP socket buffer and than feeding it to ZMQ
[10:27] sjampoo I use the callback function to decrease the reference count
[10:27] sjampoo (the python ref. count)
[10:27] sustrik python refcount?
[10:27] sustrik :)
[10:28] sustrik doesn't python object about decreasing the ref count from a different thread?
[10:28] sjampoo and i artificially increase the python ref. count prior to setting the call back
[10:28] sjampoo Well the callback acquires the GIL
[10:28] sjampoo (The Python Global Interpreter Lock)
[10:29] sjampoo so there only runs one Python Thread at all time
[10:29] sustrik the lock is held all the time from invoking send to message being deallocated?
[10:30] sustrik ... to free function being called?
[10:30] sjampoo the lock gets released on blocking calls
[10:31] sjampoo and on calls to zmq_send / zmq_recv etc..
[10:31] sjampoo not sure if that answers your question
[10:32] sustrik what i meant is 'how long it the lock held'
[10:32] sjampoo Oh right you are worrying about me overwriting the buffer right prior to the message being dequeued by ZMQ ?
[10:32] sustrik say if the peer is offline and the message cannot be sent for a prolonged time
[10:33] sustrik i meant, what if message stays in a queue for an hour
[10:33] sjampoo Then the message object will stay alive
[10:33] sustrik does that block all python threads for 1 hour?
[10:33] sjampoo No no
[10:37] sjampoo Python will just keep on running and doing its stuff in its own separate thread, meanwhile all Python references set by the user might be gone, but Python will not garbage collect it.
[10:37] sjampoo the message object itself is immutable
[10:40] sustrik great
[10:42] sjampoo The only thing that could mess it up is when you have an external blocking call which doesn't release the GIL.
[10:42] sjampoo This would of course block the Python thread, but I think it could also block the ZMQ thread as the callback will hang.
[10:47] sustrik ack
[10:59] guido_g whooo... sounds great!
[15:32] sjampoo i've plotted the latency results btw
[15:34] guido_g looks good
[15:48] sjampoo right now i have an api something like this: m = Message('blabla')
[15:48] sjampoo s.send(m)
[15:48] sjampoo I wonder if Message is the right name
[15:49] sjampoo On the one hand, if you send a message you do not really have it anymore
[15:50] sjampoo just as in real life, but it might be a little bit counter intuitive if you come across it for the first time in the object sense