ZeroMq IRC Log

Wednesday April 21, 2010

[Time] Name	Message
[10:22] sjampoo	awesome, got PyZMQ working without message copying, even with acquiring and releasing the GIL it isn't slower on small messages and about twice as fast on messages larger than 512K
[10:22] sustrik	sjampoo: is that a patch to pyzmq?
[10:23] sjampoo	yes
[10:23] sjampoo	haven't committed it yet.
[10:23] sjampoo	I want to have a chat with Brian first changed a few things
[10:24] sustrik	can it in be made switching from copy to non-copy depending on message size?
[10:24] sjampoo	I don't see any reason too, it is even faster on smaller messages!
[10:25] sjampoo	So there hardly seems to be any locking overhead
[10:25] sustrik	ah, i see
[10:25] sustrik	anyway, nice work!
[10:26] sjampoo	it works now with a message object instead of a string
[10:26] sjampoo	so you could also pass it directly a buffer
[10:26] sustrik	how does the buffer get deallocated?
[10:26] sjampoo	I think this will allow for neat stuff such as Tee-ing from a regular TCP socket buffer and than feeding it to ZMQ
[10:27] sjampoo	I use the callback function to decrease the reference count
[10:27] sjampoo	(the python ref. count)
[10:27] sustrik	python refcount?
[10:27] sustrik	:)
[10:28] sustrik	doesn't python object about decreasing the ref count from a different thread?
[10:28] sjampoo	and i artificially increase the python ref. count prior to setting the call back
[10:28] sjampoo	Well the callback acquires the GIL
[10:28] sjampoo	(The Python Global Interpreter Lock)
[10:29] sjampoo	so there only runs one Python Thread at all time
[10:29] sustrik	the lock is held all the time from invoking send to message being deallocated?
[10:30] sustrik	... to free function being called?
[10:30] sjampoo	the lock gets released on blocking calls
[10:31] sjampoo	and on calls to zmq_send / zmq_recv etc..
[10:31] sjampoo	not sure if that answers your question
[10:32] sustrik	what i meant is 'how long it the lock held'
[10:32] sjampoo	Oh right you are worrying about me overwriting the buffer right prior to the message being dequeued by ZMQ ?
[10:32] sustrik	say if the peer is offline and the message cannot be sent for a prolonged time
[10:33] sustrik	i meant, what if message stays in a queue for an hour
[10:33] sjampoo	Then the message object will stay alive
[10:33] sustrik	does that block all python threads for 1 hour?
[10:33] sjampoo	No no
[10:37] sjampoo	Python will just keep on running and doing its stuff in its own separate thread, meanwhile all Python references set by the user might be gone, but Python will not garbage collect it.
[10:37] sjampoo	the message object itself is immutable
[10:40] sustrik	great
[10:42] sjampoo	The only thing that could mess it up is when you have an external blocking call which doesn't release the GIL.
[10:42] sjampoo	This would of course block the Python thread, but I think it could also block the ZMQ thread as the callback will hang.
[10:47] sustrik	ack
[10:59] guido_g	whooo... sounds great!
[15:32] sjampoo	i've plotted the latency results btw http://nichol.as/wp-content/latency.png
[15:34] guido_g	looks good
[15:48] sjampoo	right now i have an api something like this: m = Message('blabla')
[15:48] sjampoo	s.send(m)
[15:48] sjampoo	I wonder if Message is the right name
[15:49] sjampoo	On the one hand, if you send a message you do not really have it anymore
[15:50] sjampoo	just as in real life, but it might be a little bit counter intuitive if you come across it for the first time in the object sense