Wednesday June 15, 2011

[Time] NameMessage
[00:01] dhawth nevermind, the fix was to set DYLD_LIBRARY_PATH to the location of the shared libs. I shall send mail to jzmq suggesting the add a note about this to the readme in case anyone else runs into it.
[10:04] PaowZ hello there ! got a question about multiple socket connection, when a client connect to two servers, for instance..
[10:05] PaowZ it concerns this piece of python code..
[10:05] PaowZ "Selecting a message pattern" section.
[10:06] PaowZ my question is: for 10 send(), why servers both receive 5 requests and not 10 ?
[10:07] PaowZ how is request handling done by multiple servers ?
[10:07] PaowZ thanks :)
[10:08] guido_g hint: load balancing
[10:10] PaowZ load balancing ? okay.. and I guess that if we have an odd number of sends, one server shall have to handle one more request..
[10:11] guido_g this kind of things is explained in the guide, btw
[13:06] drbobbeaty Has anyone heard of any kind of ZMQ-over-RDMA (on 10GbE)? I'm being asked for some kind of RDMA/RoCE and wanted to know if there is anyone that has done something like this in the ZeroMQ-space.
[13:15] pieterh drbobbeaty: what message size, any idea?
[13:16] drbobbeaty pieterh: The messages are all in the sub-1Kb size... all are exchange feed messages. So there are a lot of them, but none are all that big. Probably 80% are below 128bytes.
[13:17] pieterh so the only benefit of RDMA would be latency, not throughput
[13:18] drbobbeaty That, and the exchange feed socket reading and message decoding could be done on another box where the jumps to kernel space would be paid - leaving the other box just to "use" the data.
[13:19] pieterh so to use as a superfast interconnect between two tasks on different boxes
[13:20] pieterh it would mean changing the libzmq TCP transport to use RDMA
[13:20] pieterh no-one's done that to my knowledge
[13:20] pieterh or they haven't shared...
[13:21] pieterh the VTX layer I'm building could be another option
[13:22] pieterh you basically build an inprocess bridge that does weird stuff at one edge, and inproc at the other
[13:22] pieterh that then emulates a 'normal' socket
[13:22] pieterh would be fairly doable to get SUB socket emulation over VTX and RDMA, I reckon
[13:31] drbobbeaty OK, can you please keep me up to date (or just announce it) when you have something like this I can look at and benchmark? I'd really appreciate it.
[13:40] pieterh drbobbeaty: highly unlikely anyone will make this unless there's a pulling client
[13:42] drbobbeaty pieterh: Seems reasonable. The NYSE came by selling something like this, and it wasn't going to be cheap. I didn't see anything in a quick open source scan that would do this. So I think it's got to be demand-driven.
[13:43] pieterh what we have is a plausible framework for integrating an eventual RDMA driver with 0MQ apps
[13:43] drbobbeaty pieterh: I'll see if they are interested in being that demand.
[13:43] pieterh sure
[13:43] pieterh the VTX work is happening at if you want details
[14:01] john I am having trouble understanding how to divide up my program using zmq. Should I make a separate thread for every function that takes any period of time to execute?
[14:02] john Is there any documentation that provides a guideline for when/how to spearate out each function of a complex system using zmq?
[14:05] john Is there a guide for best coding practices for distributed systems using 0mq?
[14:07] drbobbeaty john___: Good questions, all, but the point of ZMQ is that it's the transport layer to do all kinds of messaging things. Figuring out how to working messaging into an app is something you sort-of need to come to the party with. It's a far more general, and therefore, far more open-ended and complex topic than specifics about ZMQ.
[14:08] drbobbeaty john___: It's like using threads, in general. You have to have some basis of understanding why you'd want to use them in the first place to *then* talk about the differences in the boost::thread library or the native pthreads, etc. See what I mean?
[14:08] travlr john___: Reading the guide may inspire certain architectures though...
[14:08] michelp john___, i found that reading some erlang books helped me understand how to design applications in a message passing environment. it's not perfect but it's a good start
[14:09] drbobbeaty michelp: very good point. Erlang forces you to think in those terms.
[14:09] michelp lately i've been selling 0mq to my colleagues by telling them "it's like erlang for all languages"
[14:11] john Yeah I'll have to check out some erlang books. It's just a whole nother paradigm that I haven't yet got my head around.
[14:15] john The other issue I'm having is since zmq doesn't force you to use any particular messaging protocol, is there any common one's out there?
[14:15] john I guess it kind of depends on what kind of data is being passed.
[14:20] michelp well i'm using python pickles :)
[14:20] michelp but there's protocol buffers, and others like that
[14:21] john yeah I've been using pickles too.
[14:22] john I suppose as long as everything is programmed in python, pickling things is okay.
[14:22] michelp if both ends of your project are pickles then it's a very flexible approach
[14:22] michelp er... s/pickles/python
[14:22] michelp cuz protobuffs and stuf like that need to be declared and maintained
[14:23] john yeah.
[14:25] pieterh john___: here's my under-intellectualized opinion on serialization:
[14:32] mrmermaid Hello all, I'm doing projects with ZMQ on Java. Can someone please tell me if there is anyway to limit a number of connection to a PULL socket, and notify when that limit is reached?
[14:33] john so Cheap would be used to trigger a process or send small bits of information? And nasty would be for sending complex data structures in a serialized fashion like pickle?
[14:34] john to pieterh ^
[14:34] pieterh john___: Cheap would be for low-volume complex messages, Nasty for high-volume simple messages
[14:34] pieterh in most distributed architectures, messages fall into these two categories
[14:35] pieterh for example, you typically have control and data flows
[14:35] pieterh where control is low volume and complex (e.g. logging statistics)
[14:36] pieterh and data is high-volume and simple (e.g. market data)
[14:36] john ok so something like streaming video would be high volume
[14:36] pieterh yes
[14:36] pieterh and often you'll have multiple levels of encoding
[14:39] nicolas I am currently using C structs for pub/sub fast data flow
[14:39] nicolas and protocol buffers for commands req/rep
[14:40] pieterh makes sense... formats like JSON are a real pain to work with in C
[14:45] john Protocol buffers seem interesting, and very simple
[14:47] ASY pieterh: are you around for the next 5 min? I would like to run few questions by you. trying to figure out the best way to structure ZMQ use in my alg.
[14:47] pieterh ASY: I was actually about to leave but shoot...
[14:47] ASY thanks. 1 sec.
[14:48] ASY i have a very performance sensetive application that relays data from multiple threads to multiple ZMQ clients. it needs to be stateful, i.e. each client uses dedicated push/pull pair.
[14:50] pieterh ok...
[14:51] ASY 1st question: i receive data from multiple threads. each thread has associated zmq push socket. I am looking at doing a blocking send() from within this thread, which means I am looking at creating a dedicated context for each thread. Alternative solution is to channel everything down the single poll loop. polls are difficult because I have SENDING socket but I can not signal a poll to
[14:51] ASY pick up data and send it the second it is ready. I could use another internal socket to cause poll loop to cycle. it would be like a "data ready" event notifier. that would cause poll to examine senders and send out data immediately.
[14:52] pieterh what is "performance" in your case?
[14:52] pieterh how many clients, how many msg/sec
[14:53] ASY the last one sounds like a correct approach. just use another socket for messaging. but that channels everything down to a single thread (that does polling) and I am wondering that this may endup being not very efficient. although efficiency is always a questionable factor. I need to ensure that in my sending of data from multiple providers there is less then 1ms delay. --- I have a
[14:53] ASY millisecond clock(s) going out on the screen. there can be let's say 10-20 data providers but one of them may be feeding clock updates every millisecond...
[14:54] pieterh just use poll
[14:54] pieterh it sounds like you're making it much too complex
[14:55] pieterh to solve performance problems you don't have
[14:55] ASY so whenever data is ready on the thread, send it to an event socket, that causes poll rotation and upon rotation if will see that data is ready and send it when POLL_OUT state is available.
[14:55] pieterh also, it'd help a lot if you start by explaining your use case
[14:55] pieterh rather than the solution
[14:55] pieterh since your solution is most likely wrong
[14:55] ASY lol
[14:55] pieterh and I can't read your mind about what your design process was :)
[14:56] ASY well, i have different data providers
[14:56] ASY that send property updates from multiple directions/devices/sockets/pipes/threads you name it.
[14:56] pieterh that's it?
[14:56] ASY all of this goes into a single in-memory hash.
[14:56] pieterh ah
[14:57] ASY and needs to be distributed to clients
[14:57] pieterh have you read the Guideclone pattern?
[14:57] pieterh *Guide clone pattern
[14:58] ASY i am sure i did at some point.
[14:58] pieterh it covers exactly this, property distribution to multiple clients
[14:58] ASY my problem is state
[14:58] pieterh ASY, I have to go, so please send an email to the list, with background and maybe a diagram
[14:58] ASY when client connects, he and only he because he connected needs a full state flush so that he can establish the state since further messages are only deltas
[14:58] pieterh cyl
[14:59] ASY k
[14:59] ASY cheers
[15:05] john is there a simple way of keeping track of how many subscribers are currently connected?
[15:11] mrmermaid I lost the connection, but can anyone please point me to the right direction to limit the number of connections to a PULL socket ?
[15:15] mrmermaid What I'm trying to do is having a server with a Pull socket, and client with push socket
[15:18] PaowZ hello there! just entered the word of 0MQ and saw that the archive contains what one needs to build from MSVC under windows.. but what if I wish to build the lib using mingw32 ?
[16:41] john I am getting a weird error when I try to start 100 threads each with it's own zmq.SUB connection. It says "Too many open files"
[16:41] john any ideas?
[16:41] ssi ulimit -n
[16:41] john ssi, is that to allow unlimited number of threads?
[16:41] ssi no, set a limit
[16:42] ssi ulimit -n 256, or something
[16:42] ssi is number of open files allowed
[16:47] john I bumped it up to 512 and I still have the same problem. :(
[16:48] john I did ulimit -a and it says it is set to 512 now.
[16:49] john ah, I have some permission problems it looks like I will have to add my user to a group or something.
[17:45] cremes PaowZ: take a look at the FAQ:
[17:45] cremes PaowZ: there are instructions for building 0mq under mingw there
[17:49] magarwal how do i filter a multipart message at the subscriber end
[17:59] cremes magarwal: the filtering happens on the bytes of the first message part
[18:00] cremes use setsockopt and ZM_FILTER to set the byte string to use as the filter
[18:00] magarwal cremes, i am sending a multipart message, how can i filter the message on one of the parts of the message
[18:01] cremes magarwal: if you want to use the built-in filtering, then you can *only* filter on the first message part
[18:01] cremes if you want to filter on some other part, then that's just plain 'ol application logic and has
[18:01] magarwal thanks
[18:01] cremes nothing to do with 0mq
[18:02] magarwal is it possible to extract a part of the multipart message instead of getting the compete data/message
[18:03] cremes magarwal: no, all parts are delivered atomically
[18:04] magarwal once i received the complete the data message, is it possible to look/extract a part of the message
[18:04] magarwal without parsing the whole message string
[18:04] cremes magarwal: yes
[18:04] cremes if your message has 3 parts
[18:05] cremes and the 3rd part is all you want
[18:05] cremes your app can skip decoding the first two (make sure to dealloc them) and just decode the final one
[18:05] cremes this is all application logic though and has little to do with 0mq
[18:05] cremes 0mq just ships bytes around
[18:05] cremes what you do with them on the receiving end is up to you
[18:07] magarwal cremes, got it thanks
[18:07] magarwal i am doing this to subscribe only to SPY subscriber.setsockopt( ZMQ_SUBSCRIBE,"SPY" , 3);
[18:08] magarwal but this would also match "SPY1" for example... rt?
[18:08] magarwal how do i ONLY subscribe to "SPY"
[18:09] cremes magarwal: not possible
[18:10] cremes not possible at the 0mq level
[18:10] magarwal hmm.. so I have to do a string comparison at the app level
[18:10] cremes yes
[18:11] cremes unless you want to patch the SUB socket type to have more intelligent filters
[18:11] cremes there has been some discussion about that; look at the mailing list archives
[18:11] magarwal ok.. i'll prbably break it
[18:11] magarwal let the professionals implement it
[18:12] cremes if you need it, you could probably get some professional services to have it implemented
[18:12] magarwal cremes, i was looking at the document but couldnt find a function to skip decoding the first two (as suggested by you above) in a multi-part
[18:13] cremes magarwal: have you read the guide yet? (see the topic)
[18:13] cremes there are *lots* of code examples
[18:14] john Is there a good way to separate a GUI frontend from the backend.
[18:15] cremes magarwal:
[18:15] cremes john__: that's a pretty general design question... any answer would have little to do with 0mq itself
[18:15] nishttal2 we have been using ActiveMQ as our message middleware and looking at ZeroMQ to replace ActiveMQ. is there a web interface for ZeroMQ where we can monitor messages/publishers/subscribers etc
[18:16] cremes nishttal2: not at this time; there has been some discussion about baking that into 0mq
[18:16] cremes nishttal2: but for now it's something you would have to build yourself
[18:16] nishttal2 cremes, i see.. how do we monitor message traffic then?
[18:17] cremes nishttal2: what do you mean by monitor? measure latency, measure total packets, ??
[18:18] nishttal2 cremes, that and to debug situation where something is not working.. in activemq we first see what is connected and what is not and where is the message waiting
[18:18] nishttal2 really helps a lot of debugging
[18:18] cremes nishttal2: yeah... i'm not the best person to answer that
[18:18] cremes you might want to ping sustrik or pieterh
[18:19] cremes or ask on the mailing list
[18:19] nishttal2 ok will do thanks
[18:20] CIA-20 jzmq: 03Brandon Ehle 07master * r6982e0f 10/ (src/Socket.cpp test/src/org/zeromq/ Fix unittest failures with libzmq 3.x -
[18:20] CIA-20 jzmq: 03Gonzalo Diethelm 07master * r9454aca 10/ (src/Socket.cpp test/src/org/zeromq/ Merge pull request #52 from azverkan/master ...
[18:20] cremes nishttal2: i also highly recommend you read the guide (
[18:24] nishttal2 cremes, i read the guide (didnt read the code)
[18:24] nishttal2 cremes, i dont remember any mention of the UI
[18:25] cremes there isn't any UI but i thought the guide talked about some "system" sockets that were used for logging
[18:25] cremes internal behavior
[18:26] nishttal2 ok will look into that
[18:26] nishttal2 thanks
[18:30] CIA-20 jzmq: 03Gonzalo Diethelm 07master * rdd3327d 10/ README : Added more information to README file. -
[18:50] john is there a way to send / recv without blocking?
[18:57] john I want to be able to update a gui from another process but without blocking the gui. I suppose I could pickle the entire gui, send it to the other process to be modified then send it back.
[18:58] linuxfood
[18:58] linuxfood it's got an extra parameter flags
[18:58] linuxfood ZMQ_NOBLOCK
[18:58] linuxfood as does send()
[18:59] john thanks