[Time] Name | Message |
[06:36] sjampoo
|
sustrik, got your github comment and closed the issue. I obviously made an error while translating the Python code, will dig deeper as to understand why the test suite is failing.
|
[06:42] sustrik
|
ok
|
[06:50] muffinpeddler
|
Hey guys, does zeromq have priority messaging support, or some kind of priority queue feature?
|
[06:57] muffinpeddler
|
Or would it be effective to just have my own priority queue after receiving each message
|
[06:59] sustrik
|
muffinpeddler: there's no inbuild one
|
[06:59] sustrik
|
you can implement it yourself
|
[07:00] sustrik
|
but that still doesn't solve the head of line blocking issue (large low priority messages blocking high priority messages)
|
[07:00] sustrik
|
the only systematic solution is to have serveral underlying connections
|
[07:00] sustrik
|
one for each priority level
|
[07:14] sjampoo
|
sustrik, when dealing with PAIR sockets, ZMQ_POLL should always report them as POLLOUT no matter if they have been bind() or connect()ed, right?
|
[07:15] sjampoo
|
right now i have this: http://pastebin.com/cWVJZ3nu
|
[07:15] sjampoo
|
i would expect 2/2 but the PAIR socket that binds returns 0
|
[07:15] sustrik
|
yup, a known issue
|
[07:16] sjampoo
|
oh
|
[07:16] sustrik
|
PAIR sockets are still unfinished
|
[07:16] sustrik
|
i should discard them from the documentation
|
[07:16] sustrik
|
so that people don't get congfused
|
[07:18] sjampoo
|
that would be better i think yes, i really thought that P2P sockets just worked and that PAIR was just a renaming
|
[07:21] CIA-15
|
zeromq2: 03Martin Sustrik 07master * r36b044a 10/ doc/zmq_socket.txt : ZMQ_PAIR socket removed from the documentation as it is unfinished yet - http://bit.ly/aXiRxv
|
[07:21] sustrik
|
:)
|
[07:22] sjampoo
|
heh
|
[08:30] sjampoo
|
hmm i keep stumbling against these issues where i don't know if it is my lack of understanding or just a bug.
|
[08:30] sjampoo
|
Is there some unwritten rule that a ZMQ_PUB socket has to connect and a SUB socket has to bind
|
[08:31] sjampoo
|
I have example code at http://pastebin.com/bdPa5ra1
|
[08:31] sjampoo
|
it works as it is, but imho i should be able to have the publisher bind and the subscriber connect as well
|
[08:32] sustrik
|
yes, that should be possible
|
[08:32] sustrik
|
what's the problem?
|
[08:32] sjampoo
|
if i do that the recv() hangs
|
[08:32] sjampoo
|
as if there is nothing to be received
|
[08:33] sustrik
|
that's possibly because you send the message before subscribed connects
|
[08:33] sjampoo
|
but shouldn't the sleep solve that?
|
[08:34] sustrik
|
presumably, but there's no guarantee
|
[08:34] sjampoo
|
ie: http://pastebin.com/KACJudRr
|
[08:35] sjampoo
|
but isn't a 3 second sleep time more than enough?
|
[08:35] sustrik
|
yeah, even few microseconds should be enough
|
[08:36] sustrik
|
i can have a look at what's causing the behaviour
|
[08:36] sjampoo
|
that would be great
|
[08:36] sustrik
|
but the point is that you don't have guarantee anyway
|
[08:36] sjampoo
|
not even with sleep?
|
[08:36] sustrik
|
nope, pub/sub works like a radio transmission
|
[08:37] sustrik
|
publisher pusblishes continuous stream of messages
|
[08:37] sustrik
|
subscribed joins at certain point
|
[08:37] sustrik
|
from that point on it'll get all the messages
|
[08:37] sustrik
|
then it leaves
|
[08:38] sustrik
|
but there's no guarantee on what will be the first message you'll get
|
[08:38] sjampoo
|
but in this example i have the subscriber 'tune in to the broadcast'
|
[08:39] sjampoo
|
wait 3 seconds as in not to miss any important message
|
[08:39] sustrik
|
yup, try to publish a stream of messages
|
[08:39] sustrik
|
well, if you want the delivery of each message to be guaranteed you cannot use pub/sub
|
[08:40] sustrik
|
it's not possible to combine one-to-many distribution with guaranteed delivery
|
[08:41] sustrik
|
without running into global deadlocks
|
[08:41] sustrik
|
caused by slow consumers
|
[08:41] sjampoo
|
well in this example it looks more like a bug imho as the other way around works perfectly
|
[08:42] sustrik
|
yep, i'll have a look
|
[08:49] sjampoo
|
thanks, i do understand the radio analogy of pub/sub.
|
[08:49] sjampoo
|
I think a nice analogy of bind/connect in this sense would be having the publisher transmit at a certain radio frequency (bind) or have it announce its messages through a megaphone directly at the listeners (connect)
|
[08:57] sustrik
|
yes, nicely said
|
[08:57] sustrik
|
unfortunately, there's something missing in 0mq documentation
|
[08:57] sustrik
|
a text that would explain individual messaging patterns in such simple words
|
[09:09] CIA-15
|
zeromq2: 03Martin Sustrik 07master * r4a3b857 10/ src/app_thread.cpp : commands not processed immediatelly in some scenarios; fixed - http://bit.ly/aSMRIF
|
[09:09] sustrik
|
sjampoo: try now
|
[09:15] sjampoo
|
cool! works :)
|
[09:18] sjampoo
|
I am planning to write a little bit about 0MQ in the future, might bother you with some extra questions by then
|
[09:20] sustrik
|
sjampoo: sure, no problem
|
[09:20] sustrik
|
also, if it's possible i'll link your text from the website
|
[09:22] sustrik
|
there's 0mq blog on the site btw, so you can even publish there...
|
[09:23] sjampoo
|
ok! i'll keep that in mind
|
[09:28] sjampoo
|
btw, that last commit also fixed the zmq_pair issue
|
[09:47] sustrik
|
:)
|
[15:21] muffinpeddler
|
sustrik: thanks. So you can keep calling socket.bind() to bind to as many ports/interfaces as you like with the zmq::socket_t?
|
[15:22] sustrik
|
yes
|
[15:39] muffinpeddler
|
So how will recv help me figure out which "priority queue" my message is from? Or will it iterate through each connection and if each queue has messages available to consume, will it grab one from the first one and then the second?
|
[15:43] muffinpeddler
|
So even if I'm getting tons of messages on one port, the other port will still be given fair consideration?
|
[15:52] sustrik
|
yes, exactly
|
[15:52] sustrik
|
the algorithm it's known as fair queueing
|
[15:52] sustrik
|
as for the prioritisation, as I said it's not yet in the product
|
[15:53] sustrik
|
so you'll have to use several sockets
|
[15:53] sustrik
|
or fix 0mq scheduling algorithm to use priorities
|
[15:54] sustrik
|
i.e. instead of fair queueing, pick the message from queue with highest priority
|
[16:27] cremes
|
sustrik: how do i get a wikidot account? i'd like my ffi bindings to be listed on the 0mq home page / wiki
|
[16:27] sustrik
|
open www.zeromq.org
|
[16:28] sustrik
|
right top corner
|
[16:28] sustrik
|
create account
|
[16:28] cremes
|
ah, i see it
|
[16:28] sustrik
|
yeah, let me know your login afterwards
|
[16:28] cremes
|
it's cremes
|
[16:29] sustrik
|
cremes: you've been invited
|
[16:29] sustrik
|
let me create a template page for you
|
[16:29] cremes
|
cool, thanks
|
[16:30] sustrik
|
how's should the binding be called?
|
[16:30] sustrik
|
JRuby?
|
[16:30] cremes
|
for now JRuby is the only ruby runtime that has proper FFI support, so yes
|
[16:30] cremes
|
but eventually it will work with rubinius and ironruby too
|
[16:31] sustrik
|
probably no generic name for all of those?
|
[16:31] cremes
|
nope; how about we call it ruby-ffi?
|
[16:31] cremes
|
then the page itself can list the currently supported runtimes
|
[16:31] sustrik
|
ffi is rather generic
|
[16:31] sustrik
|
which ffi you have in mind?
|
[16:32] cremes
|
it's ruby specific (ruby-ffi)
|
[16:32] cremes
|
i.e. http://github.com/ffi/ffi
|
[16:33] sustrik
|
term 'ffi' is used in different languages
|
[16:33] sustrik
|
see here"
|
[16:33] sustrik
|
http://en.wikipedia.org/wiki/Foreign_function_interface
|
[16:33] cremes
|
yeah, i know
|
[16:33] sustrik
|
so maybe Ruby-FFI?
|
[16:33] cremes
|
but these bindings are written in ruby and will only work with ruby
|
[16:33] cremes
|
yes
|
[16:33] sustrik
|
or Ruby (FFI)
|
[16:33] cremes
|
bingo
|
[16:34] sustrik
|
:)
|
[16:34] sustrik
|
ok, a sec
|
[16:34] cremes
|
btw, everything is working pretty well with these bindings except they are leaking every message
|
[16:34] cremes
|
i need to find & fix the memory leak :(
|
[16:35] sustrik
|
http://www.zeromq.org/bindings:ruby-ffi
|
[16:35] sustrik
|
i've pasted the content of the python bindings wiki page there to give you a template
|
[16:35] sustrik
|
but feel free to modify it in any way
|
[16:35] cremes
|
great, i'll do that
|
[16:36] sustrik
|
once you are ready, let me know and I'll link the page from the left pane
|
[16:37] cremes
|
okay, probably later today i'll get it updated
|
[16:38] sustrik
|
ok
|
[17:39] muffinpeddler
|
sustrik: thanks. Perhaps I'll look into the 0mq algorithm
|
[17:41] sustrik
|
muffinpeddler: it's here: http://github.com/sustrik/zeromq2/blob/master/src/fq.cpp
|
[17:42] sustrik
|
it's simple array of pipes
|
[17:42] sustrik
|
when recv is called you round-robin among them to get the message
|
[17:44] cremes
|
sustrik: i want to confirm that when zmq_msg_close is called, the lib handles freeing the data buffer given it via zmq_msg_init_data
|
[17:44] cremes
|
and it isn't necessary to pass a "free" function
|
[17:44] sustrik
|
it's necessary
|
[17:44] sustrik
|
how would it otherwise know how to deallocate the buffer?
|
[17:45] cremes
|
the call to zmq_close should free it
|
[17:45] cremes
|
the doc for zmq_msg_init_data says that 0mq takes ownership of the data buffer
|
[17:45] sustrik
|
well, but you supply the buffer you allocated yourself, right?
|
[17:45] cremes
|
i supply it but 0mq owns it once i hand it off
|
[17:46] sustrik
|
yes, it does, but you have to let it know how to deallocate it:
|
[17:46] sustrik
|
free?
|
[17:46] sustrik
|
delete?
|
[17:46] sustrik
|
HGlobalDealloc?
|
[17:46] sustrik
|
whatever
|
[17:46] cremes
|
ok
|
[17:46] cremes
|
so if now free function is passed to zmq_msg_init_data, then those buffers will all leak, yes?
|
[17:46] sustrik
|
yes
|
[17:47] cremes
|
then i think i found my memory leak; thanks
|
[17:47] sustrik
|
not passing deallocation function makes sense only for static data
|
[17:47] sustrik
|
such as"
|
[17:47] sustrik
|
zmq_msg_init_data (&msg, "ABC", 3, NULL);
|
[17:47] sustrik
|
the data in this case are part of the executable, so there's no need to deallocate it
|
[17:48] cremes
|
ok
|
[17:48] cremes
|
so what is the "hint" argument for?
|
[17:49] sustrik
|
imagine more complex allocation mechanism
|
[17:49] sustrik
|
such as a garbage collector
|
[17:49] sustrik
|
you have to pass pointer to the gc object in the hint
|
[17:50] sustrik
|
so that free function can do something like this:
|
[17:50] sustrik
|
((gc*) hint)->dealloc (data);
|
[17:50] cremes
|
oh, that makes sense too
|
[17:57] muffinpeddler
|
thanks sustrik, I've been meaning to make an open source contribution
|
[17:58] sustrik
|
that would be great!
|
[18:00] muffinpeddler
|
no guarantees though, I have little experience with 0mq :)
|
[18:01] sustrik
|
the functionality in question is pretty dumb, so no worries :)
|
[18:02] sustrik
|
there are some design issues though:
|
[18:03] sustrik
|
for example: how many priority levels do we want to support?
|
[18:08] muffinpeddler
|
Right. And what would you set the priority on? The pipe itself?
|
[18:09] sustrik
|
i would say the priority has to associated with the particular transport
|
[18:09] sustrik
|
say port 5555
|
[18:09] sustrik
|
that would allow you to configure nwtworking hardware to treat it with specific QoS
|
[18:10] sustrik
|
for example, you can ask the router to drop packets on port 5556 rather than on port 5555
|
[18:11] muffinpeddler
|
interesting
|
[18:11] sustrik
|
makes sense, no?
|
[18:12] sustrik
|
in practical terms it means that prioritised messages get through even over congested network
|
[18:12] muffinpeddler
|
Yea it does
|
[18:12] muffinpeddler
|
anything else I should take in to consideration?
|
[18:13] sustrik
|
well, there are two parts to it
|
[18:13] sustrik
|
1. how to get priority value to the scheduler
|
[18:13] sustrik
|
2. how to do scheduling itself
|
[18:14] sustrik
|
pt. 1 requires more knowledge of the codebase, so I volunteer to implement it
|
[18:15] sustrik
|
pt. 2 doesn't require much knowlegde so that's where you can start
|
[18:15] cornbread
|
sounds good
|
[18:18] cornbread
|
So which class would the priority value be placed in? I see a reader_t has a reference to an i_endpoint
|
[18:19] sustrik
|
let me see...
|
[18:19] sustrik
|
have a look at this function: void zmq::fq_t::attach (reader_t *pipe_)
|
[18:19] sustrik
|
that's when new pipe is attached to the scheduler
|
[18:20] sustrik
|
what if it was modified like this:
|
[18:20] sustrik
|
zmq::fq_t::attach (reader_t *pipe_, int priority_)
|
[18:20] sustrik
|
?
|
[18:21] cornbread
|
Sure. So you think the fq_t should be modified, or a new scheduler be created? What if you just want to use the fair queue
|
[18:21] sustrik
|
it should be modified imp
|
[18:22] sustrik
|
imo
|
[18:22] sustrik
|
just allow for several pipe arrays instead of a single one
|
[18:22] sustrik
|
each array for a specific priority level
|
[18:23] sustrik
|
the scheduler would try to get a message from the highest priority array
|
[18:23] sustrik
|
if there's none, try the second one etc.
|
[18:23] cornbread
|
What is different about that compared to how the round robin scheduler already works?
|
[18:24] sustrik
|
with RR you get a message from a pipe then shift to another pipe
|
[18:24] sustrik
|
with priorities you do the same thing on each priotity level
|
[18:24] sustrik
|
but if there's no message on that level you fail-over to the lower level
|
[18:25] sustrik
|
so as long as there are messages on highest priority level
|
[18:25] sustrik
|
you'll never retrieve messages from lower level
|
[18:26] cornbread
|
okay, well I'll take a look at this here and there today. I've got some exams to study for
|
[18:26] sustrik
|
sure, no haste
|
[19:03] cremes
|
sustrik: if you are still around, another api question...
|
[19:04] cremes
|
does zmq_msg_init_size allocate a data buffer of "size" bytes, and if so, is a later call to zmq_msg_init_data redundant?
|
[19:05] cremes
|
that is, a later call to zmq_msg_init_data using the same zmq_msg_t
|
[19:06] cremes
|
nevermind... i just proved it to myself via a code test
|
[19:06] cremes
|
btw, that was my memory leak
|
[19:06] cremes
|
i called zmq_msg_init_size on a structure and then called zmq_msg_init_data on the same structure
|
[19:07] cremes
|
i *thought* it was necessary to call them in that order; 1) allocate "size" space and 2) hand the buffer over
|
[19:07] cremes
|
now when i look at it i clearly see that is wrong and unnecessary
|
[19:09] cremes
|
i do recommend adding a warning to the docs for those calls saying that calling them both *will* result in a memory leak of "size" bytes
|
[19:23] sustrik
|
cremes: feel free to submit a documentation patch
|
[19:23] sustrik
|
or maybe saying that you should call onle *one* of the init functions for a single messagw
|
[19:23] cremes
|
will do
|
[19:41] sustrik
|
cremes: the same text should be in documentation for zmq_msg_init
|
[19:41] cremes
|
okay, i'll add it
|
[19:41] sustrik
|
it's kind of strange having it in three different places...
|
[19:42] sustrik
|
isn't there some common place to place the warning?
|
[19:42] sustrik
|
hm...
|
[19:42] sustrik
|
cremes: what about zmq(7)?
|
[19:42] cremes
|
i don't know
|
[19:42] sustrik
|
Initialise a message
|
[19:42] sustrik
|
zmq_msg_init(3) zmq_msg_init_size(3) zmq_msg_init_data(3)
|
[19:42] cremes
|
no opinion
|
[19:43] sustrik
|
that can be possibly changed to something like
|
[19:43] sustrik
|
To initialise message use one of the following:
|
[19:43] cremes
|
i think ONE needs to be emphasized
|
[19:44] cremes
|
how about adding the caution paragraph right under that grouping?
|
[19:44] cremes
|
hmmm... i think this is not a good area to DRY up the documentation
|
[19:44] cremes
|
i think all 3 docs should contain the caution
|
[19:44] cremes
|
otherwise it's too easy for it to be missed
|
[19:45] sustrik
|
right
|
[19:45] sustrik
|
but let's make the text more concise
|
[19:46] sustrik
|
"never initialise same message twice" ?
|
[19:47] sustrik
|
"never initialise same zmq_msg_t twice" ?
|
[19:47] sustrik
|
"never initialise same zmq_msg_t more than once" ?
|
[19:47] cremes
|
how about this:
|
[19:47] cremes
|
CAUTION: The functions _zmq_msg_init()_, _zmq_msg_init_data()_ and
|
[19:47] cremes
|
_zmq_msg_init_size()_ all initialize the 'zmq_msg_t' struct. Never initialize
|
[19:47] cremes
|
the same 'zmq_msg_t' twice.
|
[19:48] sustrik
|
not bad
|
[19:48] sustrik
|
CAUTION: The functions _zmq_msg_init()_, _zmq_msg_init_data()_ and
|
[19:48] cremes
|
and add "or it will leak memory" to the end
|
[19:48] sustrik
|
_zmq_msg_init_size()_ are mutually exclusive. Never initialize
|
[19:48] sustrik
|
...
|
[19:48] cremes
|
ah, that's good
|
[19:49] sustrik
|
i don't think the memory leak has to be mentioned
|
[19:49] sustrik
|
"never do it" means anything can happen if you do
|
[19:49] cremes
|
sure
|
[19:49] sustrik
|
oki, let me fix the docs
|
[19:50] cremes
|
ok
|
[19:54] CIA-15
|
zeromq2: 03Martin Sustrik 07master * r4d33c43 10/ (3 files): caution about zmq_msg_init_* functions added to the docs - http://bit.ly/cvOC5r
|