ZeroMq IRC Log

Sunday February 13, 2011

[Time] Name	Message
[00:08] rbitcon	Howdy, I am kooking to be pointed in the right direction for help
[00:09] rbitcon	I am using zeromq with Python in a REQ/RESP model with one server and many "workers". I see that the zmq library take up hude amounts of RAM
[00:11] rbitcon	I read the docs about socket.setsockopt(zmq.HWM, 125), but it is not that clear
[00:11] rbitcon	any pointers?
[00:52] rbitcon	Solved my problem with memory usage.
[00:53] rbitcon	I downloaded pyzmq 2.0.10.1 and dis a plan python.exe setup.py install, previously i did a development build and install or 2.0.10. This is all under windows7 and using python2.6
[01:10] tjgillies	if im behind NAT can i use zeromq to do udp hole punching?
[03:35] Seta00	anyone from the pyzmq team here?
[03:35] Seta00	in fact, anyone with python 2.7 pyzmq builds for windows here?
[03:35] Seta00	it just won't install
[08:50] sustrik	zedas: just set ZMQ_LINGER option to zero for all the sockets and everything should work as with 2.0.x
[08:52] sustrik	Guthut: wow!
[09:06] CIA-21	zeromq2: 03Martin Sustrik 07master * rba26cc9 10/ src/zmq.cpp :
[09:06] CIA-21	zeromq2: Build problem with OpenPGM and MSVC fixed
[09:06] CIA-21	zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/guRl75
[11:30] Guthur	Sustrik: ping
[11:33] DeadZen	whats zeromq?
[11:38] mikko	DeadZen: http://www.zeromq.org/
[11:44] Guthur	in like the '0MQ in 100 words' in the guide as a description
[11:44] Guthur	http://zguide.zeromq.org/chapter:all#toc2
[11:49] Guthur	mikko, would be possible to have tests run as part of a bindings build, on the build server?
[11:51] mikko	sure
[11:53] Guthur	cool, I don't have the tests at the moment, but it is something I would like to add in the future for clrzmq2
[12:04] mikko	it should be a fairly painless process to add them to builds
[12:20] mikko	i to the b
[12:45] Guthur	I've been looking into IOCP for named pipes connection under windows, the main issue I have found is differentiating between the read and write operations. It is possible to pass in a handle to an event object to writefile function (as part of the overlapped struct), which will allow differentiation.
[12:46] Guthur	The only issue with that is that both the poller and and the writer need the handle to that event object.
[12:47] Guthur	or at least for the poller to know it is a write event object, it will be matched to the right pipe as part of the return from IOCP
[15:25] sustrik	Guthur: pong
[15:30] Guthur	sustrik, did that stuff i posted earlier make much sense?
[15:30] sustrik	i have no idea how IOCP works :\|
[15:30] sustrik	anyway, what you need for integration with 0mq
[15:31] sustrik	is a polling mechanism that triggers an event when particular "socket" is readable and/or writeable
[15:31] sustrik	it has to be able to handle multiple "sockets" in parallel
[15:31] Guthur	oh yeah IOCP does multiple sockets very well
[15:31] sustrik	and all the event handlers have to be invoked within a single thread
[15:32] Guthur	that's no problem as well
[15:32] sustrik	then it should be easy, no?
[15:32] Guthur	the only complication as far as I can see is differentiating between what event is fired
[15:33] sustrik	can't you use 2 different events?
[15:33] sustrik	event objects i mean
[15:34] Guthur	if we can pass a different event object when writing to the pipe then yes
[15:35] Guthur	2 secs, just going to ask someone about it now
[15:36] Guthur	when one creates the name pipe one passes in an overlapped struct with an event object, this is set active by all events as far as I can see
[15:39] sustrik	aha
[15:40] sustrik	so how you are supposed to know whether the socket is readable or writeable?
[15:40] Guthur	you mean like a connected event?
[15:41] sustrik	hm
[15:41] sustrik	CreateNamedPipe doesn't seem to accept OVERLAPPED struct
[15:42] Guthur	writefile does
[15:43] Guthur	i'm talking to someone more knowledgeable now
[15:43] Guthur	2 secs
[15:48] mikko	Guthur: steve-o might be able to help you with IOCP
[15:48] mikko	he did some tests with it some time ago
[15:48] Guthur	mikko, did he rule it out for any technical reason?
[15:49] mikko	Guthur: i think it wasnt as fast as he expected
[15:50] sustrik	the problem with 0mq is a bit different
[15:50] Guthur	oh, interesting
[15:50] sustrik	namely, select() function that's currently used for polling
[15:50] sustrik	is limited to POSIX-like TCP sockets (winsock)
[15:50] sustrik	it doesn't work with anything else afaik
[15:51] sustrik	ie. named pipes etc.
[15:51] mikko	and kelly brock has done some work
[15:51] mikko	"For those interested, I highly
[15:51] mikko	suggest reading the change lists from V1 to V2rc of libevent, most changes
[15:51] mikko	were required because IOCP is nearly the opposite of all other async
[15:51] mikko	solutions."
[15:52] sustrik	the first question imo is whether IOCP is the only option to poll on TCP sockets and named pipes
[15:52] sustrik	maybe there's a different (and maybe less complex) way
[15:53] mikko	or whether named pipes could be emulated on windows using shared memory
[15:53] mikko	not sure if that makes sense as you would busy wait
[15:53] sustrik	you still to have poll in I/O thread
[15:53] sustrik	if you emulate the named pipes
[15:53] sustrik	you still have to event the I/O thread
[15:53] sustrik	which can be done via TCP socket (slow)
[15:54] sustrik	other option would be something like Event object
[15:54] sustrik	but once again, you can't poll on Event object using select()
[15:54] mikko	thinking out loud: optional dependency to libevent?
[15:55] mikko	new poller type which uses libevent or libev
[15:55] mikko	disabled by default but can be enabled during build
[15:55] sustrik	that would be more work than supporting IOCP natively IMO
[15:56] mikko	i really don't know whats involved in adding support for IOCP
[15:56] Guthur	does steve-o come on irc
[15:56] mikko	yes
[15:57] mikko	he is here usually during the week
[15:57] mikko	he's on NYC timezone
[15:57] mikko	EST?
[15:57] sustrik	still in NY?
[15:57] mikko	i thought so
[15:57] sustrik	he's from hong kong
[15:57] mikko	yeah, i thought he was going to be in NYC for a longer period
[15:57] mikko	maybe i misunderstood
[15:57] sustrik	ah, no idea
[15:58] mikko	he was in NYC and referred to "buying a house" during that time. i thought he might be there for a while
[15:58] mikko	anyway, he does come here occasionally during the week
[16:00] Guthur	there is a way to getting this IOCP working I'm sure
[16:01] Guthur	I'm missing something regarding the identification of the various ops, I can know that an something is happening on a pipe, just not what that something is
[16:01] Guthur	there has to be a way
[16:02] sustrik	Guthur: my impression so far is that event object is associated with particular I/O operation (Read/Write)
[16:03] sustrik	rather than Pipe as such
[16:03] Guthur	yeah, that's correct enough
[16:03] Guthur	sorry if I drew you to a different conclusion before
[16:04] sustrik	ok, maybe i misunderstood what the problem is then
[16:05] Guthur	`Orum on #winapi is posting some IOCP code, it might shed some light
[16:21] zchrish	I am performing a test to perform repeated zmq_init and zmq_term w/o performing any bind; is that permissible? I am noticing a memory leak after a continued run of this test.
[16:22] sustrik	zchrish: yes, it's permissible
[16:22] sustrik	what's the leak?
[16:23] zchrish	I am performing a "cout" of the context pointer and it is giving me a different value over time. I am watching "top" and the amount of memory required by my process is slowing growing. I am performing a zmq_init/zmq_term avery 1-2 seconds.
[16:24] mikko	zchrish: have you checked in valgrind?
[16:24] zchrish	Let me check; thanks.
[16:35] zchrish	Not sure whether the output of valgrind is related to the way I am using my program. I am creating threads which zmq_init and then zmq_term using a pthread_cleanup_push concept. After 20 iterations, I close the master thread which closes the main program. But I don't know whether this closing affects the slave threads which perform the zmq_init/zmq_term.
[16:36] zchrish	==22197== LEAK SUMMARY:
[16:36] zchrish	==22197== definitely lost: 0 bytes in 0 blocks
[16:36] zchrish	==22197== indirectly lost: 0 bytes in 0 blocks
[16:36] zchrish	==22197== possibly lost: 12,584 bytes in 43 blocks
[16:36] zchrish	==22197== still reachable: 4,552 bytes in 5 blocks
[16:36] zchrish	==22197== suppressed: 0 bytes in 0 blocks
[16:37] zchrish	I have 3 threads: (A) master and (B),(C) slave threads. In (B), (C), I peform zmq_init and zmq_term based on a simulated incident which indicates a problem.
[16:38] sustrik	run the valgrind with --leak-check=full
[16:38] sustrik	to find out what's leaking
[16:39] zchrish	Anyway, it seems like the amount lost isn't that much, if it is from 0mq, so I won't worry about it at this point. I don't expect to perform a zmq_term and zmq_init very often.
[16:43] zchrish	Hmm. I did this but I am not proficient enough with valgrind to properly interpret the results. The program is simple enough but I am not sure whether someone is interested in re-creating the situation. If so, I will upload somewhere.
[16:47] sustrik	yes. do so. i'll have a look
[16:53] zchrish	I remember watching discussions about uploading to a website but I don't remember; would you please let me know. Thank you.
[16:54] zchrish	I will upload it.
[16:55] sustrik	gist.github.com
[16:56] sustrik	place it there
[16:56] sustrik	post the link here
[16:57] Guthur	sustrik, re: IOCP, what we can do is pass a custom overlapped struct to Connect, Read, Write etc, and this struct will be returned by IOCP
[16:58] Guthur	so the custom struct can have a member containing and enum of the op type, for example
[16:58] sustrik	how would you physically perform the polling?
[16:59] sustrik	WaitForMultiple objects or what?
[16:59] Guthur	GetQueuedCompletionStatus
[16:59] Guthur	GetQueuedCompletionStatusEx
[16:59] zchrish	Not sure if this is the output result; this is my first gist upload.... gist: 824833
[16:59] sustrik	can you prpvide the link?
[17:00] zchrish	https://gist.github.com/824833
[17:00] zchrish	Is that right?
[17:00] zchrish	I think so.
[17:01] sustrik	ah
[17:01] sustrik	i meant, can you provide the valgrind output?
[17:01] zchrish	Oh, sorry. Let me do another.
[17:03] sustrik	Guthur: I see
[17:03] sustrik	so, a completion port
[17:03] sustrik	is actually a queue of pointers to OVERLAPPED structures
[17:05] zchrish	https://gist.github.com/824840
[17:05] zchrish	is the valgrind output
[17:07] sustrik	thanks
[17:08] Guthur	sustrik, essentially yes
[17:08] Guthur	there is also a completion key
[17:08] Guthur	it's quite common to place the handle of the socket in that
[17:08] Guthur	or some other context struct
[17:09] sustrik	ack
[17:09] sustrik	that seems reasonable
[17:09] Guthur	sustrik, does it sound like it will fullfil 0MQs needs?
[17:09] sustrik	definitely
[17:09] Guthur	sweet
[17:10] sustrik	you'll simply have a I/O thread loop with GetIOCompletionStatus
[17:10] sustrik	should be relativelty easy
[17:14] Guthur	so, do you want me put some code together for an implementation of epoll et al. using IOCP
[17:15] sustrik	definitely
[17:15] sustrik	try to hack something
[17:15] sustrik	feel free to ask about 0mq details in the process
[17:16] Guthur	will do, cheers to all for the help so far
[17:16] Guthur	I need to do something else tonight, for MSc thesis, but I'll continue with the IOCP this week
[17:18] sustrik	great
[17:18] Guthur	just as an aside: according to wiki IOCP is also available on Solaris 10+
[17:19] Guthur	is ZMQ on solaris?
[17:19] sustrik	sure
[17:20] sustrik	but i think solaris I/O completion ports are a different beast
[17:20] Guthur	yeah probably
[17:20] Guthur	not like I have a solaris box hanging around anyway, hehe
[17:21] sustrik	i recall we've investigated polling mechanisms in solris quite a long time ago
[17:21] sustrik	and devided to go with /dev/poll rather than completion ports
[17:23] Guthur	if I can help bring IOCP on windows to 0MQ i'll be happy enough
[18:16] sustrik	zchrish: it looks like you are not terminating your threads cleanly
[18:44] ianbarber	m to the k (delayed response goodness)
[18:51] zchrish	I see; any pointer is appreciated. Thanks.
[19:28] sustrik	zchrish: pthread_cancel basically means you kill the thread
[19:28] sustrik	what you should do instead is to pthread_join it
[21:54] eut	cremes, i think that the delay or whatever in nonblocking packet delivery is manageable. if things on my end are designed properly then it shouldn't be a problem.
[21:55] bitweiler	hey using a publisher, socket should the publisher receive some request?
[21:55] eut	bitweiler, no
[21:55] bitweiler	eut: why is that?
[21:56] eut	http://api.zeromq.org/zmq_socket.html see summary of ZMQ_PUB characteristics. it is unidirectional and send only
[21:56] bitweiler	okay thanks
[22:01] bitweiler	what is meant by: The zmq_send() function is not implemented for this socket type. where socket type is ZMQ_SUB
[22:04] eut	did you bind or connect to this socket?
[22:04] bitweiler	yes, I bind in the server side and connect on the client side
[22:05] eut	whoops
[22:05] bitweiler	but I'm getting nothing on the client side in response
[22:05] mikko	bitweiler: are you subscribing
[22:05] eut	see that page i linked to earlier for description of ZMQ_SUB
[22:05] mikko	thats the most common thing causing messages not to be received
[22:06] eut	ZMQ_SUB is unidirectional receive only... you cannot send...
[22:07] bitweiler	mikko: yes I'm subscribing
[22:08] eut	i have a question about the xrep/xreq sockets. on the xreq side i have set the socket identity to some 4 byte long string ("111\0"). when i send a message, on the xrep side, i'm getting a 17 byte long response in the first received part.
[22:08] mikko	eut: you get three messages
[22:09] mikko	one with identity, empty message and the actual message
[22:09] mikko	you should send in xreq side in that fashion as well
[22:10] mikko	identity (ZMQ_SNDMORE set), empty message (ZMQ_SNDMORE set), message
[22:10] eut	i thought that zmq takes care of prepending the socket identity and empty message when sending from a xreq socket
[22:11] mikko	let me check the code, it's been a while since i used xreq/xreq
[22:14] eut	i hope it doesnt matter... but the xrep side is in lua and the xreq side is c
[22:18] mikko	yes, client side prepends
[22:18] mikko	on xrep side you should receive multiple messages
[22:18] mikko	check for ZMQ_RCVMORE
[22:21] eut	yes
[22:21] mikko	does the lua binding expose that?
[22:21] eut	it exposes getsockopt
[22:22] eut	i'm able to see that more messages parts are there
[22:22] mikko	how are you replying with xreq?
[22:22] eut	i'm not replying at all yet, i'm just trying to interpret the received messages correctly.
[22:23] mikko	can i see the xrep code?
[22:23] mikko	trying to focus on too many things at once
[22:24] eut	sure
[22:26] eut	http://codepad.org/TqP35vOT
[22:27] mikko	what about xreq?
[22:27] eut	whoops, this is more correct xrep code: http://codepad.org/YfZepWdv
[22:27] eut	a moment for xreq
[22:35] eut	xreq code: http://codepad.org/3SFTxCwh
[22:37] Guthur	are you not suppose to set the identity before connecting
[22:38] eut	this is the output i'm getting and what i expect should happen: http://codepad.org/PSnx9yqX
[22:38] eut	the first part of the multipart message is 17 bytes for some reason...
[22:39] eut	the identity was set to "222" so i dont understand what is going on
[22:40] Guthur	from guide: If you want to set an identity you must do it before connecting or binding the socket.
[22:40] Guthur	you connect then set the identity in that code
[22:40] Guthur	the one for the xreq
[22:41] eut	:D
[22:41] eut	it works. thanks very much
[22:41] Guthur	you're welcome
[22:41] Guthur	it actually should mention that in the manual
[22:42] Guthur	i'm surprised it doesn't actually
[22:42] eut	it would make a nice addition to the zmq_socket api page
[22:42] eut	err, setsockopt
[22:42] Guthur	yeah I agree
[22:43] eut	do all options need to be set prior to connecting? or are some more flexible?
[22:43] Guthur	no identity is special
[22:44] Guthur	it makes sense that it needs to be before connecting, in a way
[22:44] Guthur	though it might be nice to be able to change an identity
[22:45] Guthur	after connecting, but things could get confusing then, and packet delivery might go terribly wrong
[22:45] eut	right
[22:45] eut	i think its fine the way it is, but it would help to have the api docs updated
[22:46] Guthur	yeah, I'll submit a patch
[22:46] Guthur	there was another thing I wanted to submit for the docs anyway
[22:46] eut	ok sure, thanks for the help
[22:47] Guthur	sure, anytime
[22:48] eut	is the development branch github.com/zeromq/zeromq2?
[22:49] mikko	eut: yes
[22:49] mikko	sorry, had to jet for a minute
[22:50] mikko	17 part response is UUID most likely, should've catched that
[22:50] mikko	17 char part*
[22:50] eut	ah, and of course its binary data so it wont display as normal text
[22:54] Guthur	mikko, do you think the api documentation for setsockopt identity should mention that it has to be set before bind/connect?
[22:54] Guthur	and if so should it be a caution note?
[22:54] mikko	Guthur: i think majority of them should mention it
[22:54] mikko	like HWM affects only subsequent binds/connects
[22:55] sustrik	all of them, actually
[22:55] Guthur	oh, I thought some didn't matter
[22:55] eut	is the delimiter message necessary if only going directly from xrep <-> xreq? it seems like it is only useful for chains longer than two nodes
[22:55] Guthur	good to know
[22:55] sustrik	it should be mentioned in the docs, you are right
[22:56] mikko	i ended up doing a patch for jenkins after running an issue with build slaves
[22:56] mikko	hopefully they'll accept it
[22:56] sustrik	congrats!
[22:56] sustrik	what was wrong there
[22:56] sustrik	?
[22:56] mikko	it adds ssl between slave and master
[22:56] Guthur	is there any opts it doesn't matter for?
[22:56] mikko	rather than having to use stunnel
[22:57] mikko	good for WAN connected slaves
[22:57] sustrik	ack
[22:57] sustrik	Guthur: the system applies to all the options
[22:57] mikko	what are the big todo items for 2.1.x stable ?
[22:57] Guthur	sustrik, ack
[22:58] mikko	i think it might be useful at some point to look into the todo items and sort them by value and effort
[22:58] Guthur	except for subscription string though, correct?
[22:58] Guthur	strings*
[22:59] mikko	that way people needing those features would have a place to check "rough estimation" and possibly chip in
[22:59] mikko	currently todo-list is github tickets mainly (?)
[23:01] mikko	http://sourceforge.net/projects/udt/forums/forum/393036/topic/4061830
[23:01] mikko	interesting, as this seems like the feature that would allow integrating udt with minimal pain
[23:09] Guthur	I updated the docs on a my fork, should I just submit a pull request?
[23:09] mikko	i think currently things go as signed patches over the mailing lists
[23:10] Guthur	ah ok
[23:36] Guthur	I have two minor doc updates, should they be separate patches?