IRC Log


Wednesday March 16, 2011

[Time] NameMessage
[00:00] Guthur mikko, That Visual studio fix worked
[00:00] Guthur I successfully built the jzmq
[00:00] Guthur cheers
[00:06] sed set up an ipc connection and still see the memory increasing
[00:18] Guthur sed: I think that would be worth putting on the mail list
[00:19] Guthur to get wider exposure to the community
[00:24] cremes sed: try inproc; ipc still uses sockets
[00:24] sed k will give that a shot
[01:15] sed thanks for answering questions guys. Will be looking at this tomorow so will reach out further to the community.
[02:43] Honeyman Hello. I am using PyZMQ, and trying to do the REQ-REP communication, polling for the replies with the zmq.core.poll.select(), using timeout = 5.0 (seconds). But most of the times, this call returns immediately, without waiting for 5 seconds, returning empty lists, like if the socket is not available.
[02:47] Honeyman What could be the reason of this problem? A bug in the PyZMQ? If pyzmq's select() is using zmq_poll(), maybe it is zmq_poll who returns prematurely, not complying to the timeouts?
[02:50] Honeyman It is interesting that if I read from this socket no matter that it is absent from the rlist, the read always succeeds...
[03:13] Honeyman Uh, found myself, that's that known bug in ZMQ 2.0
[08:14] pieterh g'morning
[08:15] pieterh mikko: i'll be happy to explain the sanity behind the version numbering and release gits, this evening
[08:18] mikko pieterh_: sent you my mobile number
[08:19] mikko to work ->
[08:19] pieterh mikko: ok, will send you mine too
[08:54] CIA-103 zeromq2: 03Jon Dyte 07master * rb79d07b 10/ src/xrep.cpp :
[08:54] CIA-103 zeromq2: reset socket state when identity message cannot be written to xrep
[08:54] CIA-103 zeromq2: Signed-off-by: Jon Dyte <jon@totient.co.uk> - http://bit.ly/g67Cnf
[09:32] Steve-o mikko: I'll add autoconf output into new tarballs
[10:16] Steve-o mikko__
[10:34] mikko hi steve
[11:15] Guthur can anyone download the tarball from github?
[11:15] mikko Guthur: probably
[11:15] Guthur I can't at the moment
[11:15] mikko Guthur: you can use snapshots if you want the master
[11:15] Guthur it could be my company firewall though
[11:15] mikko snapshot.zero.mq
[11:16] mikko latest is 16-Mar-2011 05:19
[11:16] Guthur mikko__: is that from github?
[11:16] mikko sorry
[11:17] mikko i keep killing cgi irc
[11:17] Guthur np
[11:17] mikko yes, it's master
[11:17] Guthur It's more github i'm interested in
[11:17] Guthur I actually want clrzmq2
[11:17] Guthur but it keeps giving me a bad gateway wrror
[11:17] Guthur error*
[11:18] Guthur I suspect the company has started blocking the traffic, worked yesterday though
[11:18] mikko times out for me
[11:18] Guthur oh
[11:18] Guthur maybe it's not just me then
[12:22] stimpie works for me
[12:24] stimpie try: http://www.klaproos.net/zeromq-clrzmq2-f420936.tar.gz
[12:26] CIA-103 zeromq2: 03Martin Sustrik 07master * r32ded2b 10/ (9 files):
[12:26] CIA-103 zeromq2: Duplicate identities now checked with zmq_connect
[12:26] CIA-103 zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/fgzmQX
[12:49] CIA-103 zeromq2: 03Martin Sustrik 07master * rf5015f4 10/ src/tcp_listener.cpp :
[12:49] CIA-103 zeromq2: Incorrect errno reported from tcp_listener_t::set_address
[12:49] CIA-103 zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/hvsEGl
[13:31] Guthur stimpie: cheers
[13:31] Guthur It's actually working for me now as well
[13:31] Guthur must have been an issue at the github end
[14:30] sustrik cremes: hi, are you here?
[15:26] cremes sustrik: i am here; how can i help?
[15:26] sustrik i had a look at your issue
[15:27] sustrik the one with leaks
[15:27] sustrik trying to reproduce it i hit a max fd limit
[15:27] sustrik how can i increase it
[15:27] sustrik on linux?
[15:27] cremes let me check...
[15:27] cremes :)
[15:28] cremes http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
[15:28] cremes that gives all of the details
[15:28] cremes but i would just do "sudo sysctl -w fs.file-max=100000"
[15:29] sustrik let me try
[15:30] cremes fyi, this change won't survive a reboot
[15:30] sustrik "Too many open files"
[15:31] sustrik doesn't seem to help
[15:31] mato sustrik: ulimit
[15:31] cremes what is the output of ulimit ?
[15:31] sustrik 1024
[15:32] mato sustrik: you need to change ulimit -n to some higher value, as root
[15:32] mikko hi mato
[15:32] mikko welcome back
[15:32] mato hi mikko
[15:32] mato thx
[15:32] mikko a lot of changes in the builds
[15:33] mato sustrik: and then from that shell where you changed it, start another shell as yourself
[15:33] mato sustrik: so 1) sudo bash 2) ulimit -n 8192 3) su - sustrik 4) run what you need
[15:33] mato mikko__: yes, i noticed, lots of work
[15:33] sustrik le t me try
[15:34] mato mikko__: also lots of release changes, etc. i'm completely out of the loop :-)
[15:34] mato mikko__: good work taking over the autobuilds :-)
[15:34] CIA-103 zeromq2: 03Martin Sustrik 07master * rfac9c2d 10/ (doc/zmq_setsockopt.txt doc/zmq_socket.txt):
[15:34] CIA-103 zeromq2: zmq_socket(3) and zmq_setsockopt(3) man pages improved
[15:34] CIA-103 zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/i8ZYxS
[15:34] cremes sustrik: mato is correct; the change i gave you only changed the system-wide setting
[15:34] cremes you now need ulimit to change your user setting
[15:36] mikko mato: we are currently integrating with openpgm autotools build
[15:36] mikko mato: to remove duplication of the build logic
[15:37] sustrik mato: works! thanks!
[16:49] cremes sustrik: any luck with reproducing that issue?
[16:50] sustrik yup
[16:50] sustrik i've just posted a short test case
[16:51] cremes great!
[16:52] sustrik cremes: can you test with that program
[16:52] sustrik and let me know whether you are seeing the same behaviour?
[16:52] cremes yes; i'll report back in a few
[17:02] cremes sustrik: can't compile it on my linux box
[17:02] cremes i get this error: /usr/local/include/zmq.hpp:26:19: fatal error: cassert: No such file or directory
[17:03] sustrik interesting
[17:03] sustrik it should be a standard header file
[17:04] sustrik are you compiling it with g++?
[17:04] sustrik g++ -pthread -luuid -o leaker leaker.cpp .libs/libzmq.a
[17:04] cremes i was using pieterh's 'build' script
[17:04] cremes let me try your line
[17:05] sustrik well, libzmq.a should be on library path
[17:05] cremes compiled
[17:05] sustrik good
[17:05] sustrik does it leak?
[17:06] cremes yes; heap jumped from 4MB to 24MB and stayed there
[17:06] sustrik osx?
[17:06] cremes and CPU is at 400%
[17:06] cremes linux
[17:06] cremes CPU dropped to 0, heap is now at 18MB
[17:07] cremes 10554 cremes 20 0 399m 18m 1148 S 0 0.2 1:44.09 leaker9
[17:07] sustrik same as with the original test case?
[17:07] cremes pretty much the same
[17:07] cremes my original case ran longer and therefore created a larger heap
[17:07] cremes but the idea is the same
[17:08] sustrik well, i checked the malloc's you've reported to be leaking
[17:08] sustrik (in yqueue_t)
[17:08] sustrik the number of chunks get to something like 2600
[17:08] sustrik den it gradually falls down to 0
[17:09] cremes so this program does *not* leak on your system?
[17:09] sustrik hard to say as it never finishes
[17:09] sustrik let me look at mem usage
[17:09] cremes i don't think this is a genuine leak where the mallocs are lost
[17:10] cremes i think some structure is holding on to their references instead of freeing them up like it should
[17:10] cremes that's why the conventional leak detectors aren't finding it
[17:10] cremes the mallocs aren't getting orphaned; something in the lib is holding the references
[17:11] sustrik the number of allocated chunks in yqueue_t drops to 0
[17:11] sustrik but memory usage stays the same
[17:11] sustrik that means that either some other object is leaking
[17:11] sustrik or that OS simply doesn't return the allocated memory to the global pool
[17:12] cremes i don't think it's an OS issue
[17:12] sustrik how did you figured out that yqueue_t is leaking?
[17:13] cremes on OSX i ran the "Instruments" application; it can track every malloc in the program and give you a trace of where it's happening
[17:13] cremes i can send you a screenshot if you would like
[17:13] sustrik no need
[17:13] sustrik what i done was increment and print out a global variable after each malloc in yqueue_t
[17:14] sustrik and decrement it and print out after each free
[17:14] sustrik the program ends at zero :|
[17:14] sustrik i can try with the other object you pointed out...
[17:15] cremes i just sent you the screen shot; it can't hurt to look at it
[17:15] sustrik ok
[17:16] cremes convince pieterh that you need your own OSX box to test some of this stuff
[17:16] cremes it has some really good dev tools
[17:17] sustrik i am working on my own on this
[17:17] sustrik anyway
[17:18] sustrik so the leak either happens on osx and not on linux
[17:18] mikko ok, time to head out to the pubmeet
[17:18] sustrik or the osx tools are incorrectly reporting the leak
[17:18] sustrik mikko__: have a nice evening
[17:20] cremes i see the leak on both osx and linux; the memory is never freed on the linux side
[17:21] cremes i thought you saw the same
[17:35] Guthur it's a wonder this was never noticed before
[17:35] Guthur was the leak introduced recently
[17:36] sustrik cremes: i did (using the top tool)
[17:37] sustrik however, as for mallocs/free's in yqueue_t, those match perfectly
[17:37] sustrik so either:
[17:37] sustrik 1. the leak is somewhere else and osx tools are reporting false positives
[17:37] sustrik 2. the leaks on linux and osx are different problems
[17:38] sustrik 3. processes don't return memory to os
[17:38] sustrik (3) is know to be true in many cases
[17:38] sustrik but contradicsts the report from the osx tool
[17:45] cremes it's a head scratcher, that's for sure
[21:04] jobytaffey Other than the data itself, is there any space in a zmq_msg_t for some user data? I have a buffer full of messages which I need to attach TTL values to. I could wrap each zmq_msg_t in a struct containing a TTL, but I'm wondering if there's a way of hiding it inside. Could I abuse the msg_content_t.hint pointer safely?
[21:17] Seta00 what's the value of EAGAIN/
[21:17] Seta00 ? *
[21:18] Seta00 er.. nevermind
[21:20] Seta00 clrzmq2 doesn't let me detect EAGAIN's >_>
[21:43] sustrik jobytaffey: check multi-part messages
[21:43] sustrik ZMQ_SNDMORE, ZMQ_RCVMORE
[21:48] jobytaffey I'm using multi-part msgs for some thing already. I suspect that my approach here is not very 0mq-ish. I'm generating data for clients, then storing it until they poll. Each message needs a TTL and there's a garbage collector to destroy when TTL reaches 0. It's not strictly a queue, so I'm storing zmq_msgs in RAM until they're polled for. To be honest, I've just wrapped them in a struct with a ttl...
[22:03] Guthur sustrik, did you get any further with the mem leak investigation?