Friday September 16, 2011

[Time] NameMessage
[00:13] rb Anyone there?
[00:19] cremes rb_: have you read the 0mq guide? it covers lots and lots of use cases
[00:40] narobi any gurus around?
[00:48] narobi ah well, i'll try anyway, i have a pretty basic question
[00:49] narobi have worker thread and non worker thread, worker thread sits on a blocked receive on a pull socket waiting for instructions
[00:49] narobi i have a main push socket that pushes to workers
[00:49] narobi thenw hen i am shutting down, i create another push socket and connect it to the workers and send them a message to tell them to quit
[00:49] narobi on windows, the code works fine
[00:50] narobi on linux, using zeromq 2.1.9, the worker never gets the message from the secondary push socket
[00:50] narobi the push socket only connects to that one pull socket
[00:50] narobi so it's not a fair queuing issue
[00:51] narobi i have tried sleeping for 5 seconds after send and making sure the push socket does not close until the worker has quit
[00:51] narobi no dice
[00:51] narobi nothing is working
[00:51] narobi if anyone has any ideas, iw ould be greatful
[01:21] cremes narobi: i don't understand your setup
[01:21] cremes so you have a PUSH socket that connects to your PULL sockets in the non-worker thread
[01:22] cremes if blocks on recv and sends data out as it comes in
[01:22] cremes when you want to shut down, you create a second PUSH socket and connect it to your PULL sockets
[01:22] cremes and then send a termination message
[01:22] cremes is that correct?
[01:23] cremes if so, is the transport tcp or inproc?
[06:01] MickeM Whats the best way to store "information" with a connection in req/rep (for instance autenticated as and that kind of thing)? (one option would be to have a hashmap with subscriber id and the relevant information I guess, but is there anything fancy?)
[07:32] CIA-121 libzmq: 03Martin Sustrik 07master * r06bdf2c 10/ (8 files): Check message syntax in REQ asynchronously ...
[07:42] mikko MickeM: what language?
[08:46] MickeM mikko: C++
[08:52] mikko MickeM: i guess it depends on what information you store and how you want to access it
[08:54] MickeM mikko: well, a struct with some stuff and as I said I want to have clients "login once" and then allow them to be logged in... problem with "subscriber" is ofcourse that someone could fake it, hence some form of "session state" would be preferable (as zeromq could reset that when a connection is reset)
[08:58] mikko MickeM: zeromq doesn't really expose the connections to the application level
[08:59] mikko MickeM: there are several ways to tackle this, i guess
[08:59] mikko MickeM: you could use PKI i guess
[09:00] mikko MickeM: or after login give a trancient secret to the client and encrypt messages using that
[09:00] mikko i remember there being some talk about more client oriented library that would communicate with zeromq servers
[09:00] mikko more of a one connection concept
[09:02] MickeM mikko: humm.. ok... I guess the poor mans version would then be to just send a cookie to the clients then... signing messages will be used at some point but I am not there yet...
[13:03] CIA-121 libzmq: 03Martin Sustrik 07master * r5936379 10/ src/mtrie.cpp : Bug in mtrie fixed ...
[14:16] taghawi Hello I am looking for binaries of zeromq for a Mac OS X Server v10.5 Leopard
[14:16] taghawi with a One 2.8GHz Quad-Core Intel Xeon (quad-core)
[14:32] cremes taghawi: no one is distributing binaries for osx; you can use macports (zmq) or homebrew (zeromq) to handle
[14:32] cremes the compilation and installation for you
[14:32] cremes or, download the posix tarball and build & install it yourself
[14:32] cremes we can provide help if you are unfamiliar with how to do that
[14:35] CIA-121 libzmq: 03Martin Sustrik 07master * re170136 10/ (src/mtrie.cpp src/mtrie.hpp): More bugs in mtrie fixed ...
[14:50] errordeveloper hm .. I'm not getting much response on #msgpack ..
[14:50] errordeveloper can anyone have a look here
[14:50] errordeveloper msgpack_pack_int (&mpkg, i);
[14:50] errordeveloper this function results in packing 0 instead of incr value
[14:51] errordeveloper
[16:01] cremes sustrik: is 3.x and 4.x going to set the RCVMORE flag on the socket even for label parts?
[16:01] cremes sustrik: according to a thread on the ML from a few weeks ago, that seemed to be the concensus...
[16:13] cremes sustrik: i also notice that printing the contents of the label message does not return the identity data
[16:13] cremes that i have previously set
[16:13] cremes is this a bug (on 3.0 master) or is the custom identity no longer sent through?
[16:46] pieterh cremes: nice catch... I've not actually tested multipart messages in CZMQ over 0MQ/3.x
[16:47] cremes pieterh_: i don't know if it's a catch or not... i'm thoroughly baffled over what *should* happen! :)
[16:47] pieterh check the design documents, it's all there
[16:47] pieterh oh wait...
[16:47] cremes ;)
[16:48] cremes btw, john murphy says hi
[16:48] cremes i see him all of the time now...
[16:48] pieterh hey!
[16:48] pieterh awesome
[16:48] pieterh give him my highest five
[16:48] cremes will do
[16:49] pieterh so wrt the labels IMO we need to just insist on the behavior we want
[16:49] pieterh (a) patch 3.x until it does what we want and (b) lobby for this to go into 4.x
[16:49] cremes ok
[16:50] pieterh wrt identities, I'd suggest treating this as a bug: test case, issue, etc.
[16:50] cremes i'll write one up and attach some C
[16:50] pieterh yup
[16:50] cremes that will be an *easy* one (i like easy ones)
[16:51] pieterh I'll take a quick look if there's a simple way to make RCVMORE work on labels
[16:52] pieterh cremes: would you also file an issue for the RCVMORE change? we will treat this as a bug too
[16:52] pieterh don't need a test case for that
[16:52] cremes sure
[16:53] cremes let me do that first
[16:53] cremes i won't get to the identity repro until tomorrow
[16:53] pieterh I'll change 3-0 master and we can then work on how that'd go into 4.0
[16:55] pieterh I've made the change, it's trivial
[16:57] cremes great
[16:57] pieterh you have an issue number for me?
[16:57] cremes LIBZMQ-255 tracks it
[16:57] pieterh 255 I assume
[16:58] pieterh sweet
[17:00] errordeveloper I have this strange issue with msgpack
[17:00] errordeveloper
[17:00] errordeveloper when I try to do 'msgpack_pack_int (&mpkg, i);',
[17:00] pieterh cremes: ok, you can grab it from 3-0 master
[17:01] errordeveloper it doesn't pick-up the right value of 'i'
[17:01] errordeveloper I don't use any otimization flags on my code
[17:02] errordeveloper is one who is using msgpack around ?
[17:03] cremes errordeveloper: is there a msgpack channel?
[17:03] cremes or a mailing list?
[17:03] cremes i've played with it lately but using the ruby bindings
[17:03] cremes it works okay for me
[17:05] cremes pieterh_: did you confirm it worked? because my test still shows false
[17:06] pieterh you mean like *test* the code?
[17:06] cremes heh
[17:06] pieterh for that I'd need a test case, if you want to make me one I'll use it NP
[17:06] cremes i'll take that as a no
[17:06] cremes ok
[17:06] pieterh lol
[17:06] pieterh :)
[17:06] cremes i'll do that tomorrow too
[17:07] pieterh there is probably something evil in the code that I missed
[17:07] cremes probably... sustrik likes evil code
[17:07] cremes :)
[17:08] pieterh evil slavic code...
[17:08] pieterh wouldn't trust my business to anything less
[17:08] cremes nor would i
[17:08] cremes btw, are you still in texas?
[17:09] pieterh yup, Dallas
[17:09] pieterh it's surprisingly fun tbh
[17:09] cremes i believe it... texans are good folks
[17:09] cremes gotta run... back in a bit
[17:10] pieterh cyl
[17:15] errordeveloper cremes: #msgpack has a few people there :)
[17:25] errordeveloper cremes: ok, I see where the problem is
[17:25] errordeveloper cremes: msgpack_pack_<type> are macros
[17:25] errordeveloper that's what I thought they were :))
[17:26] errordeveloper well, they are macros expanding to inline functions
[17:34] errordeveloper it's nuts, even using a pointer to the counter varieble still does the same
[18:21] errordeveloper ok, i'll see if I get an answer to this on SO :)
[18:22] errordeveloper
[18:46] rb Any Gurus?
[18:49] michelp rb_, there's 95 people in here. If you have a question, go ahead and ask it
[18:50] benbangert is there any way to see how many undelivered messages are in memory for a zmq socket?
[18:51] socksandsandals benbangert: AFAIK no
[18:51] benbangert bummer
[18:51] socksandsandals I too wish for that ;-)
[18:52] benbangert the closest I saw was the ability to see whether or not adding a message would cause blocking
[18:52] socksandsandals yeah, but that just means the next message will hit the disk queue
[18:52] benbangert right, it just means, yea, there's stuff that isn't delivered yet
[18:52] socksandsandals you can infer from that whether you're at the HWM and then you'd know at least that much
[18:52] benbangert but not the 'how'
[18:52] benbangert well, assuming I didn't have the disk based messages
[18:53] socksandsandals pretty ghetto, though
[18:53] benbangert which reminds me, I didn't see anything indicating whether the disk based messages will resume upon an app restart
[18:53] socksandsandals ah right, yeah
[18:53] socksandsandals good question
[18:53] socksandsandals idk about that
[18:54] benbangert easy enough to try it I guess :)
[18:54] socksandsandals actually, I'm pretty sure they won't
[18:54] socksandsandals since otherwise, nothing I've written with 0MQ would be working ;-)
[18:54] benbangert since there's nothing explicit about naming the file to use, I'd guess not
[18:55] rb I am looking for a solution for this : 1, there is a message bus; 2, any number of processes can attach themselves to the bus; 3, each process attached to the bus behave as both a client and server; 4, each process will send requests to any other process and will get reply from it, at the same time, it will answer requests from other process that are addressed to it. Thankx
[18:56] benbangert rb_: xmpp? :)
[18:56] rb benbangert, What is xmpp?
[18:57] benbangert jabber protocol, just ahve them all go to the same 'channel'
[18:57] benbangert presence handles indicating what processes are available, they can all send/recieve messages on the channel, and directly
[18:57] benbangert (assuming you only care about real-time)
[18:58] socksandsandals rb_: the "bus" would be a message broker of some kind
[18:58] rb How is it related to 0mq?
[18:59] benbangert rb_: it isn't, its just a soution to the problem you mentioned, cause that sounds like a better match
[18:59] benbangert zmq has no such central broker, you'd be implementing it yourself
[18:59] benbangert especially if you want new processes that join this 'bus' to see messages that occured prior to them joining
[18:59] socksandsandals rb_: you could do that by having each node have a strong name and one pub socket and one sub socket
[19:00] socksandsandals the broker then filters on strong names, similar to what a switch would do
[19:00] benbangert socksandsandals: they'd have to all find each other and link up...
[19:00] socksandsandals indeed
[19:01] shales rb_: I wouldn't call it a bus, but I do something similar using one process as a "router" that binds an XREP socket and all the other processes connect to that socket using XREP or XREQ sockets
[19:04] rb shales, I haven't finished reading the GUIDE , do you mean ROUTER when you say XREQ?
[19:05] rb socksandsandals, I need request/reply kind of communication so PUB/SUB is not an option
[19:06] socksandsandals rb_: XREP/XREQ then
[19:06] shales oh, XREP=ROUTER and XREQ=DEALER
[19:12] rb I'm thinking to create a customized device that have ROUTER on both ends, or shoud I just use the normal QUEUE device and have each process connect to both ends of the QUEUE device?
[19:13] socksandsandals you'd need two ROUTERS and two DEALERS, I'd think
[19:13] socksandsandals one going one way and the other the other
[19:14] socksandsandals the question in my mind is still: how are the nodes on either side going to find each other?
[19:14] socksandsandals how will they know which others to address?
[19:15] shales why does there need to be two "sides" to this central thing? Why can't it bind a single ROUTER socket at a well-known address that all the other processes connect to?
[19:20] shales as for the addressing problem, in my system all of the processes have fixed identities and my router process knows to route a particular kind of request to identity "foo". If the number of roles of the processes aren't fixed in your system then I guess they'd each have to generate unique identities and send a registration message after connecting to the router.
[19:21] shales Up to you how processes would discover eachother.
[19:31] CIA-121 pyzmq: 03Min RK 07master * rad1fa9f 10/ (11 files in 5 dirs): Merge pull request #135 from minrk/3.0 ...
[20:02] calvin in zmq3, can the same XSUB socket send multiple messages to subscribe for multiple filters?
[20:46] rb shales. I'll try out your single router solution. Thx a lot for your help!
[20:54] rb socksandsandals, you are right. If I use 2 queues 1 for each direction, the DEALER's load banlancing behavour is still problematic, what I need is broadcasting on the dealer end with which each node can filter out messages destinated to it.