IRC Log


Friday May 7, 2010

[Time] NameMessage
[06:36] sjampoo sustrik, got your github comment and closed the issue. I obviously made an error while translating the Python code, will dig deeper as to understand why the test suite is failing.
[06:42] sustrik ok
[06:50] muffinpeddler Hey guys, does zeromq have priority messaging support, or some kind of priority queue feature?
[06:57] muffinpeddler Or would it be effective to just have my own priority queue after receiving each message
[06:59] sustrik muffinpeddler: there's no inbuild one
[06:59] sustrik you can implement it yourself
[07:00] sustrik but that still doesn't solve the head of line blocking issue (large low priority messages blocking high priority messages)
[07:00] sustrik the only systematic solution is to have serveral underlying connections
[07:00] sustrik one for each priority level
[07:14] sjampoo sustrik, when dealing with PAIR sockets, ZMQ_POLL should always report them as POLLOUT no matter if they have been bind() or connect()ed, right?
[07:15] sjampoo right now i have this: http://pastebin.com/cWVJZ3nu
[07:15] sjampoo i would expect 2/2 but the PAIR socket that binds returns 0
[07:15] sustrik yup, a known issue
[07:16] sjampoo oh
[07:16] sustrik PAIR sockets are still unfinished
[07:16] sustrik i should discard them from the documentation
[07:16] sustrik so that people don't get congfused
[07:18] sjampoo that would be better i think yes, i really thought that P2P sockets just worked and that PAIR was just a renaming
[07:21] CIA-15 zeromq2: 03Martin Sustrik 07master * r36b044a 10/ doc/zmq_socket.txt : ZMQ_PAIR socket removed from the documentation as it is unfinished yet - http://bit.ly/aXiRxv
[07:21] sustrik :)
[07:22] sjampoo heh
[08:30] sjampoo hmm i keep stumbling against these issues where i don't know if it is my lack of understanding or just a bug.
[08:30] sjampoo Is there some unwritten rule that a ZMQ_PUB socket has to connect and a SUB socket has to bind
[08:31] sjampoo I have example code at http://pastebin.com/bdPa5ra1
[08:31] sjampoo it works as it is, but imho i should be able to have the publisher bind and the subscriber connect as well
[08:32] sustrik yes, that should be possible
[08:32] sustrik what's the problem?
[08:32] sjampoo if i do that the recv() hangs
[08:32] sjampoo as if there is nothing to be received
[08:33] sustrik that's possibly because you send the message before subscribed connects
[08:33] sjampoo but shouldn't the sleep solve that?
[08:34] sustrik presumably, but there's no guarantee
[08:34] sjampoo ie: http://pastebin.com/KACJudRr
[08:35] sjampoo but isn't a 3 second sleep time more than enough?
[08:35] sustrik yeah, even few microseconds should be enough
[08:36] sustrik i can have a look at what's causing the behaviour
[08:36] sjampoo that would be great
[08:36] sustrik but the point is that you don't have guarantee anyway
[08:36] sjampoo not even with sleep?
[08:36] sustrik nope, pub/sub works like a radio transmission
[08:37] sustrik publisher pusblishes continuous stream of messages
[08:37] sustrik subscribed joins at certain point
[08:37] sustrik from that point on it'll get all the messages
[08:37] sustrik then it leaves
[08:38] sustrik but there's no guarantee on what will be the first message you'll get
[08:38] sjampoo but in this example i have the subscriber 'tune in to the broadcast'
[08:39] sjampoo wait 3 seconds as in not to miss any important message
[08:39] sustrik yup, try to publish a stream of messages
[08:39] sustrik well, if you want the delivery of each message to be guaranteed you cannot use pub/sub
[08:40] sustrik it's not possible to combine one-to-many distribution with guaranteed delivery
[08:41] sustrik without running into global deadlocks
[08:41] sustrik caused by slow consumers
[08:41] sjampoo well in this example it looks more like a bug imho as the other way around works perfectly
[08:42] sustrik yep, i'll have a look
[08:49] sjampoo thanks, i do understand the radio analogy of pub/sub.
[08:49] sjampoo I think a nice analogy of bind/connect in this sense would be having the publisher transmit at a certain radio frequency (bind) or have it announce its messages through a megaphone directly at the listeners (connect)
[08:57] sustrik yes, nicely said
[08:57] sustrik unfortunately, there's something missing in 0mq documentation
[08:57] sustrik a text that would explain individual messaging patterns in such simple words
[09:09] CIA-15 zeromq2: 03Martin Sustrik 07master * r4a3b857 10/ src/app_thread.cpp : commands not processed immediatelly in some scenarios; fixed - http://bit.ly/aSMRIF
[09:09] sustrik sjampoo: try now
[09:15] sjampoo cool! works :)
[09:18] sjampoo I am planning to write a little bit about 0MQ in the future, might bother you with some extra questions by then
[09:20] sustrik sjampoo: sure, no problem
[09:20] sustrik also, if it's possible i'll link your text from the website
[09:22] sustrik there's 0mq blog on the site btw, so you can even publish there...
[09:23] sjampoo ok! i'll keep that in mind
[09:28] sjampoo btw, that last commit also fixed the zmq_pair issue
[09:47] sustrik :)
[15:21] muffinpeddler sustrik: thanks. So you can keep calling socket.bind() to bind to as many ports/interfaces as you like with the zmq::socket_t?
[15:22] sustrik yes
[15:39] muffinpeddler So how will recv help me figure out which "priority queue" my message is from? Or will it iterate through each connection and if each queue has messages available to consume, will it grab one from the first one and then the second?
[15:43] muffinpeddler So even if I'm getting tons of messages on one port, the other port will still be given fair consideration?
[15:52] sustrik yes, exactly
[15:52] sustrik the algorithm it's known as fair queueing
[15:52] sustrik as for the prioritisation, as I said it's not yet in the product
[15:53] sustrik so you'll have to use several sockets
[15:53] sustrik or fix 0mq scheduling algorithm to use priorities
[15:54] sustrik i.e. instead of fair queueing, pick the message from queue with highest priority
[16:27] cremes sustrik: how do i get a wikidot account? i'd like my ffi bindings to be listed on the 0mq home page / wiki
[16:27] sustrik open www.zeromq.org
[16:28] sustrik right top corner
[16:28] sustrik create account
[16:28] cremes ah, i see it
[16:28] sustrik yeah, let me know your login afterwards
[16:28] cremes it's cremes
[16:29] sustrik cremes: you've been invited
[16:29] sustrik let me create a template page for you
[16:29] cremes cool, thanks
[16:30] sustrik how's should the binding be called?
[16:30] sustrik JRuby?
[16:30] cremes for now JRuby is the only ruby runtime that has proper FFI support, so yes
[16:30] cremes but eventually it will work with rubinius and ironruby too
[16:31] sustrik probably no generic name for all of those?
[16:31] cremes nope; how about we call it ruby-ffi?
[16:31] cremes then the page itself can list the currently supported runtimes
[16:31] sustrik ffi is rather generic
[16:31] sustrik which ffi you have in mind?
[16:32] cremes it's ruby specific (ruby-ffi)
[16:32] cremes i.e. http://github.com/ffi/ffi
[16:33] sustrik term 'ffi' is used in different languages
[16:33] sustrik see here"
[16:33] sustrik http://en.wikipedia.org/wiki/Foreign_function_interface
[16:33] cremes yeah, i know
[16:33] sustrik so maybe Ruby-FFI?
[16:33] cremes but these bindings are written in ruby and will only work with ruby
[16:33] cremes yes
[16:33] sustrik or Ruby (FFI)
[16:33] cremes bingo
[16:34] sustrik :)
[16:34] sustrik ok, a sec
[16:34] cremes btw, everything is working pretty well with these bindings except they are leaking every message
[16:34] cremes i need to find & fix the memory leak :(
[16:35] sustrik http://www.zeromq.org/bindings:ruby-ffi
[16:35] sustrik i've pasted the content of the python bindings wiki page there to give you a template
[16:35] sustrik but feel free to modify it in any way
[16:35] cremes great, i'll do that
[16:36] sustrik once you are ready, let me know and I'll link the page from the left pane
[16:37] cremes okay, probably later today i'll get it updated
[16:38] sustrik ok
[17:39] muffinpeddler sustrik: thanks. Perhaps I'll look into the 0mq algorithm
[17:41] sustrik muffinpeddler: it's here: http://github.com/sustrik/zeromq2/blob/master/src/fq.cpp
[17:42] sustrik it's simple array of pipes
[17:42] sustrik when recv is called you round-robin among them to get the message
[17:44] cremes sustrik: i want to confirm that when zmq_msg_close is called, the lib handles freeing the data buffer given it via zmq_msg_init_data
[17:44] cremes and it isn't necessary to pass a "free" function
[17:44] sustrik it's necessary
[17:44] sustrik how would it otherwise know how to deallocate the buffer?
[17:45] cremes the call to zmq_close should free it
[17:45] cremes the doc for zmq_msg_init_data says that 0mq takes ownership of the data buffer
[17:45] sustrik well, but you supply the buffer you allocated yourself, right?
[17:45] cremes i supply it but 0mq owns it once i hand it off
[17:46] sustrik yes, it does, but you have to let it know how to deallocate it:
[17:46] sustrik free?
[17:46] sustrik delete?
[17:46] sustrik HGlobalDealloc?
[17:46] sustrik whatever
[17:46] cremes ok
[17:46] cremes so if now free function is passed to zmq_msg_init_data, then those buffers will all leak, yes?
[17:46] sustrik yes
[17:47] cremes then i think i found my memory leak; thanks
[17:47] sustrik not passing deallocation function makes sense only for static data
[17:47] sustrik such as"
[17:47] sustrik zmq_msg_init_data (&msg, "ABC", 3, NULL);
[17:47] sustrik the data in this case are part of the executable, so there's no need to deallocate it
[17:48] cremes ok
[17:48] cremes so what is the "hint" argument for?
[17:49] sustrik imagine more complex allocation mechanism
[17:49] sustrik such as a garbage collector
[17:49] sustrik you have to pass pointer to the gc object in the hint
[17:50] sustrik so that free function can do something like this:
[17:50] sustrik ((gc*) hint)->dealloc (data);
[17:50] cremes oh, that makes sense too
[17:57] muffinpeddler thanks sustrik, I've been meaning to make an open source contribution
[17:58] sustrik that would be great!
[18:00] muffinpeddler no guarantees though, I have little experience with 0mq :)
[18:01] sustrik the functionality in question is pretty dumb, so no worries :)
[18:02] sustrik there are some design issues though:
[18:03] sustrik for example: how many priority levels do we want to support?
[18:08] muffinpeddler Right. And what would you set the priority on? The pipe itself?
[18:09] sustrik i would say the priority has to associated with the particular transport
[18:09] sustrik say port 5555
[18:09] sustrik that would allow you to configure nwtworking hardware to treat it with specific QoS
[18:10] sustrik for example, you can ask the router to drop packets on port 5556 rather than on port 5555
[18:11] muffinpeddler interesting
[18:11] sustrik makes sense, no?
[18:12] sustrik in practical terms it means that prioritised messages get through even over congested network
[18:12] muffinpeddler Yea it does
[18:12] muffinpeddler anything else I should take in to consideration?
[18:13] sustrik well, there are two parts to it
[18:13] sustrik 1. how to get priority value to the scheduler
[18:13] sustrik 2. how to do scheduling itself
[18:14] sustrik pt. 1 requires more knowledge of the codebase, so I volunteer to implement it
[18:15] sustrik pt. 2 doesn't require much knowlegde so that's where you can start
[18:15] cornbread sounds good
[18:18] cornbread So which class would the priority value be placed in? I see a reader_t has a reference to an i_endpoint
[18:19] sustrik let me see...
[18:19] sustrik have a look at this function: void zmq::fq_t::attach (reader_t *pipe_)
[18:19] sustrik that's when new pipe is attached to the scheduler
[18:20] sustrik what if it was modified like this:
[18:20] sustrik zmq::fq_t::attach (reader_t *pipe_, int priority_)
[18:20] sustrik ?
[18:21] cornbread Sure. So you think the fq_t should be modified, or a new scheduler be created? What if you just want to use the fair queue
[18:21] sustrik it should be modified imp
[18:22] sustrik imo
[18:22] sustrik just allow for several pipe arrays instead of a single one
[18:22] sustrik each array for a specific priority level
[18:23] sustrik the scheduler would try to get a message from the highest priority array
[18:23] sustrik if there's none, try the second one etc.
[18:23] cornbread What is different about that compared to how the round robin scheduler already works?
[18:24] sustrik with RR you get a message from a pipe then shift to another pipe
[18:24] sustrik with priorities you do the same thing on each priotity level
[18:24] sustrik but if there's no message on that level you fail-over to the lower level
[18:25] sustrik so as long as there are messages on highest priority level
[18:25] sustrik you'll never retrieve messages from lower level
[18:26] cornbread okay, well I'll take a look at this here and there today. I've got some exams to study for
[18:26] sustrik sure, no haste
[19:03] cremes sustrik: if you are still around, another api question...
[19:04] cremes does zmq_msg_init_size allocate a data buffer of "size" bytes, and if so, is a later call to zmq_msg_init_data redundant?
[19:05] cremes that is, a later call to zmq_msg_init_data using the same zmq_msg_t
[19:06] cremes nevermind... i just proved it to myself via a code test
[19:06] cremes btw, that was my memory leak
[19:06] cremes i called zmq_msg_init_size on a structure and then called zmq_msg_init_data on the same structure
[19:07] cremes i *thought* it was necessary to call them in that order; 1) allocate "size" space and 2) hand the buffer over
[19:07] cremes now when i look at it i clearly see that is wrong and unnecessary
[19:09] cremes i do recommend adding a warning to the docs for those calls saying that calling them both *will* result in a memory leak of "size" bytes
[19:23] sustrik cremes: feel free to submit a documentation patch
[19:23] sustrik or maybe saying that you should call onle *one* of the init functions for a single messagw
[19:23] cremes will do
[19:41] sustrik cremes: the same text should be in documentation for zmq_msg_init
[19:41] cremes okay, i'll add it
[19:41] sustrik it's kind of strange having it in three different places...
[19:42] sustrik isn't there some common place to place the warning?
[19:42] sustrik hm...
[19:42] sustrik cremes: what about zmq(7)?
[19:42] cremes i don't know
[19:42] sustrik Initialise a message
[19:42] sustrik zmq_msg_init(3) zmq_msg_init_size(3) zmq_msg_init_data(3)
[19:42] cremes no opinion
[19:43] sustrik that can be possibly changed to something like
[19:43] sustrik To initialise message use one of the following:
[19:43] cremes i think ONE needs to be emphasized
[19:44] cremes how about adding the caution paragraph right under that grouping?
[19:44] cremes hmmm... i think this is not a good area to DRY up the documentation
[19:44] cremes i think all 3 docs should contain the caution
[19:44] cremes otherwise it's too easy for it to be missed
[19:45] sustrik right
[19:45] sustrik but let's make the text more concise
[19:46] sustrik "never initialise same message twice" ?
[19:47] sustrik "never initialise same zmq_msg_t twice" ?
[19:47] sustrik "never initialise same zmq_msg_t more than once" ?
[19:47] cremes how about this:
[19:47] cremes CAUTION: The functions _zmq_msg_init()_, _zmq_msg_init_data()_ and
[19:47] cremes _zmq_msg_init_size()_ all initialize the 'zmq_msg_t' struct. Never initialize
[19:47] cremes the same 'zmq_msg_t' twice.
[19:48] sustrik not bad
[19:48] sustrik CAUTION: The functions _zmq_msg_init()_, _zmq_msg_init_data()_ and
[19:48] cremes and add "or it will leak memory" to the end
[19:48] sustrik _zmq_msg_init_size()_ are mutually exclusive. Never initialize
[19:48] sustrik ...
[19:48] cremes ah, that's good
[19:49] sustrik i don't think the memory leak has to be mentioned
[19:49] sustrik "never do it" means anything can happen if you do
[19:49] cremes sure
[19:49] sustrik oki, let me fix the docs
[19:50] cremes ok
[19:54] CIA-15 zeromq2: 03Martin Sustrik 07master * r4d33c43 10/ (3 files): caution about zmq_msg_init_* functions added to the docs - http://bit.ly/cvOC5r