IRC Log

Friday September 9, 2011

[Time] NameMessage
[01:24] strich WOndering if anyone could help me - I'm programming zmq in C++. Had it working fine all in a single class with methods. I refactored it into a more OO and classful system. But I'm getting crashes now that I've tried to define socket and pollitems as pointers and initialize them in the constructor. Whenever poll() is called it crashes
[01:25] strich Its got to be that I'm passing a bad pointer around or not defining and initializing properly. I've been on this for half a day now
[01:26] strich Is there are documentation or code examples that show it being done in a class?
[01:32] strich My code: http://pastebin.com/avpqM2JG
[01:33] strich Can anyone point out whats wrong?
[02:17] strich Can anyone help?
[03:03] strich WOndering if anyone could help me - I'm programming zmq in C++. Had it working fine all in a single class with methods. I refactored it into a more OO and classful system. But I'm getting crashes now that I've tried to define socket and pollitems as pointers and initialize them in the constructor. Whenever poll() is called it crashes
[03:03] strich Its got to be that I'm passing a bad pointer around or not defining and initializing properly. I've been on this for half a day now
[03:03] strich Is there are documentation or code examples that show it being done in a class?
[03:03] strich My code: http://pastebin.com/avpqM2JG
[03:03] strich Can anyone point out whats wrong?
[03:49] neopallium strich: line 63: pollitems = pts; /* taking the pointer to a stack variable and assigning it to a member variable. Dont' do that. */
[03:54] neopallium strich: also the way the code handles the zmq::context_t and zmq::socket_t values looks wrong too. You create a context object but don't store it anywhere except in a stack variable. When creating the socket object you should be storing it directly in a member variable.
[03:56] strich neopallium - Thanks for responding. How would you suggest I keep a pointer on the pollitems array then?
[03:57] neopallium strich: you need to allocate the array using "new"
[03:57] neopallium so that it will be in the heap instead of on the stack.
[03:58] strich pollitems = new zmq::pollitem_t[1]; ?
[04:00] neopallium Yeah that should work for the pollitems. It has been a few years since I last worked on C++ code, I mainly write code in C now, so I am a little rusty with C++.
[04:00] strich Doing this with socket now: socket = new zmq::socket_t(context, ZMQ_ROUTER);
[04:00] neopallium also I think line 64 is wrong, you are taking a pointer to another stack variable.
[04:01] neopallium that looks right too.
[04:01] strich Yeah thats commented out now
[04:01] strich Lastly, for context, I'm not assigning a global member variable because none of the class methods use it directly.
[04:02] neopallium are you creating one context per socket?
[04:02] strich Yes
[04:02] neopallium you shouldn't do that.
[04:02] strich Only the one socket too, really. The class is only instantiated once
[04:02] strich Oh?
[04:03] neopallium should only use one context for the whole process/program and create sockets from that one instance.
[04:03] neopallium also when your program close it should cleanup that context.
[04:04] strich Pretty sure zmq does that as part of its C++ codebase when the zmq destructor is called
[04:04] neopallium sending messages is async. in zmq. You can't be sure that all messages are sent until you terminate the zmq context.
[04:05] neopallium I haven't worked with the C++ bindings for zmq, so maybe that will not be an issue.
[04:06] strich Either way, it still seems to be doing weird things
[04:06] neopallium is it still crashing?
[04:08] strich No, but something is blocking. I basically have al oop that runs, and it appears something it blocking during instantiation of Game
[04:09] strich mmm, its that try catch I think. Weird
[04:09] neopallium ah, I know why
[04:09] neopallium the try catch is the scope for the zmq context
[04:09] strich ohhhh
[04:09] neopallium it is trying to destroy the context
[04:09] strich and its being deleted
[04:09] neopallium yup
[04:09] strich hah
[04:10] neopallium another reason to make it a member variable and not a stack variable.
[04:10] strich Yep.
[04:10] strich I'll try that now
[04:11] strich Yep thats fixed it
[04:11] strich You're a champion Neo
[04:11] neopallium np
[04:12] strich I think I would have been pulling my hair out :P
[04:12] strich Scope always gets me
[06:16] strich neopallium - Hrm, doesn't look like its fully fixed. Doing this causes a crash: zmq::poll(pollitems, 1, 0);
[06:17] strich Its got to be that pollitems pointer is bad
[06:26] strich Fixed it
[06:26] strich pollitems[0].socket = *socket; // Must derefence pointer. Still compiles but will crash
[10:12] loxs folks, when I use the REQ/RES pattern, does it imply that I must always wait for the previous RES to arrive before I fire the next REQ?
[10:15] jcase Yes
[10:16] loxs can't I emulate "multiple clients" from a single application?
[10:18] loxs oh, just saw it says "too many open files"
[10:18] jcase You can. Have a look here: http://zguide.zeromq.org/page:all#A-Request-Reply-Broker. It describes the zmq_router and zmq_dealer sockets.
[10:19] jcase Ah, I see :)
[10:21] loxs I'm trying something like http://friendpaste.com/1z1hl45e5lbBqRzPu2bKWw
[10:21] loxs (which spawns 1000 erlang processes, each of them tring to open a connection to the server)
[10:21] loxs so why do I get too many files?
[10:43] jcase iirc you want to create one context
[10:45] jcase Also, what OS are you on?
[10:46] loxs linux
[10:58] guido_g see ulimnit
[10:58] guido_g ulimit even
[10:59] ntelford anyone here use the jzmq bindings?
[11:00] ntelford I've been seeing some strange behviour when manually calling socket.close and context.destroy
[11:00] ntelford socket.close *ocassionally* causes an "Assertion: ok (mailbox.cpp:78)"
[11:01] ntelford and context.destroy() seems to block and never return :-\
[11:01] ntelford both of these are bound to the object destructors, so (in theory) are called by the JVM when the object's go out of scope (on shutdown, in the case of my application) - and when this happens they both seem fine
[13:21] CIA-121 jzmq: 03Nicholas Telford 07master * rf84eb8d 10/ src/Socket.cpp : Fixed setLongSockOpt() not permitting LINGER for ZMQ 3.0. - http://git.io/Mup03A
[13:21] CIA-121 jzmq: 03Gonzalo Diethelm 07master * r8658b84 10/ src/Socket.cpp : Merge pull request #69 from nicktelford/upstream-fixes ...
[13:30] ntelford using jzmq, we're seeing occasional problems on a SUB socket - sometimes the Thread segfaults, sometimes we get this: "*** glibc detected *** /usr/java/jdk1.6.0_23/bin/java: *** glibc detected *** /usr/java/jdk1.6.0_23/bin/java: double free or corruption (!prev): 0x00000000570dc5f0 ***"
[13:33] ntelford basically, I'm looking for any help as to how to debug this
[13:42] mikko valgrind / gdb
[13:42] mikko compile with --enable-debug
[13:42] mikko and jzmq with CPPFLAGS='-g -O0'
[13:43] mikko that way you get symbols
[13:44] mikko you can also set /proc/sys/kernel/core_pattern to something sensible and ulimit -c to get core dump
[13:53] pouete Hi all 0/ : question : i have a program that launch a thread to listen to a TCP port and a main program that read all the connected clients with a select(). it is working great. I add a ZMQ socket to send the data i receive from the TCP port to other modules, but the program "crash". Here is the code : http://pastebin.com/0fsCeC6R it is just a PoC , so, there maybe is some ugly hacks
[13:54] pouete ( the code as it is on the pastebin does not actually works.
[13:57] ntelford mikko, the error above, is that indicative of an issue in the JVM itself, or could it also have been caused by a problem in zmq/jzmq?
[13:57] ntelford (given that the JVM loads jzmq.so which in turn is linked to zmq.so)
[14:01] mikko ntelford: it could be in the jni
[14:01] ninjas Hi, I'm a n00b and trying to understand ZMQ_DEALER sockets a little bit better. I have 2 ZMQ_DEALER sockets which work pretty much one-way : A sends to B. In what circumstances could A's internal message queue build up? My assumption is that the queue would be building up on the incoming side, but the behavior I am seeing suggests the buildup is on the outgoing side.
[14:02] ninjas no HWM is set, btw.
[14:02] ntelford mikko, that's my thought, seems to be the area with the least amount of testing certainly
[14:23] ntelford zmq_poll is interrupted by delivery of signals, right?
[14:38] ntelford scratch that question
[14:38] ntelford turns out in Java, there's no way to interrupt a syscall (or other blocking IO call like zmq_poll)
[18:06] mickem Is there a "c++" version czmq?
[18:09] roadrunner is this the correct channel to as czmq questions on?
[18:16] cremes yes, it's the correct channel
[18:23] roadrunner cool - is czmq intended for production? It's not just a reference implementation, is it?
[18:41] Fade hum.. i'm getting this error starting a just checked out mongrel2 : http://librelist.com/browser//mongrel2/2011/4/25/mongrel2-trap-divide-error-linux-kernel-reports-no-time-stamp-counter-tsc/#33fceb84d325363ae7e1ebd9ca8203f6
[18:42] Fade the system is running the ubuntu oneric beta, with sqlite and omq installed from the ubuntu pool.
[18:45] Fade this is with just the test config as documented in section 3.2 of the manual
[18:45] Fade so everything should be pretty vanilla.
[19:09] roadrunner Do most people run czmq using the 'safe_malloc()' version of the allocation macro? Seems like that could be painful on systems that actually return NULL if too much memory is asked for...
[19:25] cremes roadrunner: czmq is intended for production
[19:26] cremes Fade: i don't know how much mongrel2 help you'll be able to get in here... it's not often
[19:26] cremes i see people discussing it
[19:29] Fade cremes: yeah. this is indirectly related to mongrel. the problem is in zeromq.
[19:30] Fade I got it past the problem by commenting out the --with-pgm call in the zeromq.spec file.
[19:31] cremes pgm support was poor in the earlier 2.1.x releases
[19:31] cremes however, i don't think you can easily upgrade to later ones without breaking mongrel2
[19:32] cremes i have heard/seen that it is pretty tied to specific 0mq releases
[19:33] Fade i guess the zmq packages in the ubuntu pool (which are in the 2.1.x line) use the standard spec file.
[20:17] Matt Hi, I am stuck on a problem and maybe someone can help me come up with a solution.
[20:17] Matt I have web services facade that is receiving requests and I want to perform a REQ/REP within the service call.
[20:18] Matt My initial attempt was to just open a new REQ socket for each request, but under any load the whole thing crashed
[20:18] Matt I assume this is either a bug, or my strategy is flawed
[20:18] Matt Anyone care to comment?
[20:23] michelp Matt__, are you cleaning up the sockets when you're done with them?
[20:23] Matt Not sure, it seems like the CLRZMQ library takes care of that, how would i explicitly clean them up?
[20:24] michelp not sure about clr, but in python you do socket.close()
[20:24] cremes Matt__: that's the wrong pattern anyway
[20:24] cremes it doesn't make sense to create a socket for each request
[20:24] Matt ok
[20:24] Matt What would be the best strategy?
[20:24] cremes Matt__: can the web service process multiple simultaneous requests or is it one at a time?
[20:25] cremes is the processing synch or async?
[20:25] Matt Multi requests - each request is handled on a new thread
[20:25] Matt sync
[20:25] cremes ok
[20:25] cremes are you creating the threads when the request arrives or are they from a pool?
[20:26] Matt they are from a pool
[20:26] cremes great
[20:26] Matt i dont have control over the threads
[20:26] cremes there is a whole discussion on how to handle this in the guide but i'll give you the short version
[20:26] cremes 1. 1 context
[20:26] cremes 2. when you create the thread, pass it the context and create a socket inside the thread
[20:27] cremes 3. when your request comes in, the next guy in the pool should be passed the request
[20:27] cremes make sense?
[20:27] cremes the 0mq restriction is that sockets are not thread-safe so you can't read/write to them from many
[20:27] Matt problem is, i dont have control over the incoming threads
[20:27] cremes threads simultaneously unless you surround them with a mutex
[20:28] Matt the services are sitting in IIS and a new thread is created automatically
[20:28] cremes you don't control the thread pool?
[20:28] cremes hmmm, ok
[20:28] cremes do you have a max number of threads that can be created?
[20:29] cremes regardless, here's the new short version
[20:29] cremes 1. create 1 context
[20:29] cremes 2. populate a socket pool with REQ sockets
[20:29] cremes 3. from each request thread, so a thread-safe checkout of a socket from the pool
[20:30] cremes as long as the socket is only ever read/write in one thread at a time, you're safe
[20:30] Matt I see
[20:30] cremes but check the guide for other patterns... i know this is covered in one of the chapters
[20:30] Matt is it bad practice to create/destroy the sockets?
[20:30] cremes yes
[20:30] roadrunner cremes: my initial scan of some of the czmq code made me nervous about what will happen in OOM situations on systems that aren't quite as lenient as linux about allocations
[20:30] cremes they are cheap
[20:31] cremes roadrunner: you should post your concerns to the ML... i'm a ruby guy so i don't use that lib :)
[20:31] roadrunner cremes: okie dokie :) thanks!
[20:31] Matt why is that bad practice to create/destroy?
[20:31] Matt if you say they are "cheap"...
[20:32] cremes cheap in terms of memory
[20:32] cremes so keeping them around doesn't hurt (a few K)
[20:32] Matt right, so expensive to create/destroy?
[20:32] cremes but each time you create one and you connect/bind to a socket, that process takes upwards of 100ms
[20:32] Matt I see
[20:33] cremes ok, everyone have a nice weekend!
[20:33] Matt Would there be anyway to have some type of asynchronous proxy that would receive the requests and respond to the requester?
[20:33] Matt Instead of having a pool of sockets?
[20:33] cremes Matt__: read the entire guide, please. i'm taking off now.
[20:33] Matt ok, thank you
[20:44] michelp Matt__, can you create thread local variables?
[20:45] michelp you could try that approach if you can. if the variable exists, then use it, if not, create a new thread local initialized with a socket
[20:47] replicator 1. Is that a good idea to just use ØMQ as an in-memory queue (inproc://)? Rather than for example to use various Queue data structures in many languages..
[20:50] replicator I guess.. is there any immediate advantage in this case?
[20:59] zedas so, i get this Assertion failed: zmq_msg_size (msg_) == 0 (req.cpp:88)
[20:59] zedas yet another assert, and i'm just pointing a plain REQ at an XREP
[20:59] zedas and this is in pyzmq, so, since this is a hard abort and not a real exception, it can't get caught and tell me what line of python is doing it
[21:00] zedas this is why your asserts suck. they should be exceptions or error codes, not aborts, there should be less of them, and they should clearly state what went wrong and how to fix it.
[21:01] zedas for example, in this assert, i have no idea what could possibly be sending a 0 length message or even what that means.
[21:54] bluesmoon hi, I was wondering if there was any way to tell zeromq where it should create its swap files
[21:55] bluesmoon at the moment it writes to the current directory, but I have my app set up such that its working directory is not writable, but an alternate "running state" directory is
[22:00] mikko i think the swap is a bit bad
[22:01] mikko you can chdir at the beginning of the app
[22:01] mikko usually the swap setting shouldn't be needed imho
[22:02] mikko zedas: i think i raised this issue earlier
[22:02] mikko zedas: i was trying to make a reproducable test case but couldn't reproduce it with a simple case
[22:06] bluesmoon Ideally I could do that and then chdir to wherever I need to be, but swap files are created on demand, when a client with an identity connects
[22:06] bluesmoon and this might happen at any time in the process's lifecycle
[22:39] ninjas hi I have a question. Given a one-way ZMQ TCP socket between two points A and B, where B is not consuming the messages fast enough to keep up with A (such that there is a ZMQ queue buildup occurring somewhere), what determines where the messages build up? Will it build up on A or B?
[23:02] mikko zedas: i got a test case done and hopefully a valid fix
[23:14] mikko zedas: https://zeromq.jira.com/browse/LIBZMQ-252
[23:14] mikko ninjas: you can set HWM on each side
[23:21] ninjas mikko, thank you, but I'm interested in knowing what the behavior is, assuming HWM isn't set on either end
[23:22] ninjas granted, I may have missed it in the documentation
[23:49] mikko ninjas: iirc on receivers side
[23:49] mikko assuming they can be sent over the network