[Time] Name | Message |
[07:47] vrc
|
hi
|
[07:47] vrc
|
had a quick question about the pub-sub model and reliability
|
[07:48] vrc
|
i set up a pub-sub model..and if my subscribers receive packets (in a loop) -> and spend a lot of time processing them, will they lose packets
|
[07:49] vrc
|
or is the pub-sub model guaranteed to provide reliability i.e, all packets sent by the pub are received by each sub..i do realize that some packets in the beginning could be lost due to connection delays, etc
|
[07:49] vrc
|
thanks - i'm new to zeromq
|
[07:51] cremes
|
vrc: think of a PUB socket as a radio broadcast
|
[07:51] cremes
|
if your subscriber isn't around to "hear" it, there is no guarantee it will get all packets
|
[07:52] cremes
|
by default, PUB will queue packets for all connected subscribers but a slow subscriber can cause a huge backlog
|
[07:52] cremes
|
and memory pressure on the publisher if they can't keep up
|
[07:52] cremes
|
so, there is a mechanism called ZM_HWM (high water mark)
|
[07:53] cremes
|
if there are more than X outstanding packets (where X is the HWM) then the pub socket drops them to /dev/null
|
[07:53] cremes
|
if you need reliability, then you need to work out some kind of 'ack' mechanism and a pub/sub pattern is no
|
[07:53] cremes
|
longer appropriate
|
[07:53] cremes
|
be sure to read all 4 chapters of the guide... of lot of these use-cases are discussed in there
|
[07:54] cremes
|
vrc: did that help?
|
[07:57] vrc
|
cremes - thank you very much. that was very useful - in fact, i was reading your comments somewhere else on the internet
|
[07:57] cremes
|
heh
|
[07:58] cremes
|
well, it's bed time; good luck with your investigations
|
[07:58] vrc
|
even if you use PUB/SUB over TCP-IP, there's no guaranteed delivery ?
|
[07:58] cremes
|
this channel is usually pretty well covered during the weekdays
|
[07:58] cremes
|
no
|
[07:59] cremes
|
read the guide... this is almost faq :)
|
[07:59] vrc
|
ok. thanks - won't keep you up
|
[07:59] cremes
|
cool
|
[08:16] guido_g
|
good morning all
|
[09:43] pieter_hintjens
|
guido_g: good morning
|
[09:43] guido_g
|
howdy pieter_hintjens
|
[09:43] pieter_hintjens
|
how's it going... ?
|
[09:44] guido_g
|
didn't do anything yesterday
|
[09:44] guido_g
|
except finding a glitch in the spec, but i need to verify that first
|
[09:45] pieter_hintjens
|
a glitch, interesting
|
[09:45] guido_g
|
yep
|
[09:46] guido_g
|
in client <-> broker there is no client address mentioned
|
[09:46] guido_g
|
but it appears magically (as full list even) in worker <-> broker
|
[09:46] pieter_hintjens
|
right, it's the same as for broker-worker
|
[09:47] pieter_hintjens
|
that note should be made to cover both sides
|
[09:47] guido_g
|
i mean where does the client address in frame 2 come from? that should be explaind
|
[09:47] guido_g
|
*ed
|
[09:47] pieter_hintjens
|
hmm, one assumes the implementor knows his XREP/ROUTER stuff
|
[09:48] guido_g
|
but still it's not clear where the address in frame 2 comes from, itr should be mentioned in the spec -- imho
|
[09:50] pieter_hintjens
|
indeed... it assumes too much, I'll fix that
|
[09:50] guido_g
|
oh and eventually move the XREQ note to the front of the spec
|
[09:51] guido_g
|
so one can see it before its needed
|
[09:52] pieter_hintjens
|
indeed
|
[09:53] guido_g
|
i think i'll start on some broker parts this afternoon
|
[09:54] guido_g
|
if we could get that running, all these "but i need reliability" questions could be answered w/ a link :)
|
[09:55] pieter_hintjens
|
ah, almost :-)
|
[09:56] pieter_hintjens
|
after this, it's the Titanic pattern
|
[09:56] mikko
|
looks like steve got openpgm out
|
[09:56] pieter_hintjens
|
"but I need reliability on spinning rust"
|
[09:56] pieter_hintjens
|
mikko: saw that...
|
[09:57] guido_g
|
rust?
|
[09:57] pieter_hintjens
|
guido_g: iron oxide? magnetised brown stuff used to hold data
|
[09:57] guido_g
|
ahhh...
|
[09:58] guido_g
|
sorry, thought you just got a ssd, so it'll be "built on sand"
|
[09:59] pieter_hintjens
|
guido_g: I replaced the HD in my notebook with an SSD that Sustrik gave me a while back
|
[09:59] pieter_hintjens
|
quite neat
|
[09:59] djc
|
in the guide's fig9, shouldn't the second right-hand box be "Start the SUB sockets before the PUB to avoid loss" (i.e. the other way around)?
|
[09:59] pieter_hintjens
|
djc: let me check...
|
[10:00] guido_g
|
pieter_hintjens: spec:7 "Clients will issue at most one request at a time, and will wait for a reply in realtime." <- can this be lifted, i can't see why
|
[10:01] djc
|
pieter_hintjens: in the bottom box it also says 9zeromq, I think it means #zeromq?
|
[10:01] guido_g
|
oops... time for furniture shopping
|
[10:01] guido_g
|
l8tr
|
[10:02] pieter_hintjens
|
guido_g: it can't be lifted without other changes
|
[10:02] pieter_hintjens
|
djc: fixed both errors, thanks
|
[10:02] djc
|
awesome, thanks
|
[10:02] djc
|
I could write up a list of language refinements if you're interested
|
[10:02] djc
|
I presume you're not a native speaker, either? :)
|
[10:03] pieter_hintjens
|
djc: of what language?
|
[10:03] djc
|
the refinements, or a native speaker?
|
[10:03] djc
|
if the former, some things in the guide read awkwardly
|
[10:04] pieter_hintjens
|
uhm, English is my native language, yes
|
[10:04] pieter_hintjens
|
feel free to point out the awkward parts
|
[10:04] djc
|
oh, ok, I assumed from your name
|
[10:04] djc
|
sorry about that!
|
[10:04] pieter_hintjens
|
np
|
[10:05] pieter_hintjens
|
there's an issue tracker on the Guide github
|
[10:05] djc
|
ok
|
[10:06] djc
|
another question: what's the reason I have to memcpy() my data to the zmq_message, for example in the guide's pub/sub example's publisher
|
[10:06] djc
|
it seems like it should be okay to reference it as long as I make sure it stays around until the zmq_msg_close()?
|
[10:07] pieter_hintjens
|
sure
|
[10:07] mikko
|
djc: with inproc that is ok
|
[10:07] pieter_hintjens
|
the examples don't try to optimise
|
[10:08] pieter_hintjens
|
mikko: fixed issue #3 on ztools
|
[10:09] mikko
|
thanks
|
[10:09] mikko
|
im trying to figure out a good build order for zeromq2 in the daily builds
|
[10:09] mikko
|
tests use the same ports so concurrent builds have intermittent test fails
|
[10:10] pieter_hintjens
|
how so?
|
[10:10] mikko
|
well, multiple builds run test that tries to bind to 5555
|
[10:11] mikko
|
which throws EADDRINUSE
|
[10:11] mikko
|
btw, did you see this: https://github.com/mkoppanen/zeromq2-docs
|
[10:11] mikko
|
that uses submobules to bring everything in to same 'workspace'
|
[10:12] pieter_hintjens
|
ah, right... tests should use unique ports
|
[10:12] mikko
|
you mean randomise the port?
|
[10:12] pieter_hintjens
|
not randomize, but allocate sequentially or something
|
[10:12] pieter_hintjens
|
number the tests 1 up
|
[10:13] mikko
|
that's all fine but i build 4 zeromq builds on the same machine at the same time
|
[10:13] pieter_hintjens
|
oh, hang on, it's the same test running in parallel, right
|
[10:13] pieter_hintjens
|
bleh
|
[10:13] mikko
|
so sequence would be the same
|
[10:13] pieter_hintjens
|
no answer except "serialize the tests"
|
[10:14] pieter_hintjens
|
btw, we forked off zeromq2-0 as well
|
[10:14] djc
|
hmmm, so what does it mean for zmq_msg_init_data() to "take ownership of the supplied buffer"? for example, I want to send a part of a stack-allocated buffer, what would be the optimal way to put this into a message?
|
[10:14] djc
|
given that there are no weird threading things going on in my app
|
[10:15] pieter_hintjens
|
djc: 'optimal' as in 'save the last CPU cycle' or as in 'don't write code that confuses me'?
|
[10:15] pieter_hintjens
|
if you aren't really comfortable with the API you should not be trying to optimize, period
|
[10:15] pieter_hintjens
|
and even when you are, you should still not be optimizing
|
[10:16] djc
|
this is me trying to become comfortable with the API :)
|
[10:16] djc
|
I know, I know, I should only optimize after stress-testing
|
[10:16] djc
|
I was just wondering about the underlying model
|
[10:17] pieter_hintjens
|
use mempcy and focus on more interesting aspects of the model
|
[10:17] djc
|
ok
|
[10:38] mikko
|
pieter_hintjens: i updated zfl README.txt last night
|
[10:38] mikko
|
but didnt have time to investigate into generating README.md
|
[10:38] mikko
|
can you update that?
|
[10:39] pieter_hintjens
|
sure, it uses gitdown
|
[10:39] pieter_hintjens
|
will do now
|
[10:39] pieter_hintjens
|
done
|
[10:53] mikko
|
thanks
|
[14:05] larhat
|
Hi, folks! If i set ZMQ_HWM=1 and ZMQ_SWP=N (N>0) on socket, process will store all, that cannot be processed immediately, on disk. But it process fails, can it restore swapped messages and retry sending? I wonder, can i use this for some kind of persistence queue?
|
[14:13] Guthur
|
larhat, I have a feeling it cannot
|
[14:51] mikko
|
larhat: no, it cannot be done at the moment
|
[14:51] mikko
|
larhat: there has been discussion about creating persistence devices
|
[14:52] mikko
|
where you could 'plug in' persistence on sockets you need
|
[14:54] larhat
|
ok, thanks, guys
|
[21:45] surikator
|
hi! anyone out there who tried ZeroMQ for OCaml?
|
[22:15] Guthur
|
is there any reason why rtrouter from the guide should not worker with inproc transport
|
[22:32] Guthur
|
surikator, there does not yet seem to be a binding for OCaml
|
[22:32] surikator
|
yes, there is...
|
[22:32] Guthur
|
oh ok, it's not mentioned on the site
|
[22:32] surikator
|
https://github.com/pdhborges/ocaml-zmq
|
[22:33] surikator
|
=)
|
[22:35] surikator
|
but regarding zeromq in general, can we pass whole data structures and objects through the sockets?
|
[22:38] Guthur
|
surikator, you'd need to serialize them
|
[22:39] Guthur
|
I'm not familiar with that binding, it may or may not have that built in
|
[22:39] Guthur
|
ZeroMQ does not provide it though
|
[22:39] Guthur
|
there is some pretty decent serialization solutions out there though
|
[22:43] surikator
|
it doesn't have that built in. it mimics ZeroMQ only.
|
[22:51] surikator
|
ok, so I'd probably have to use Ocaml Marshal library. which solutions do you mean?
|
[23:08] Guthur
|
surikator, protocol buffers, thrift, or maybe even JSON
|
[23:08] Guthur
|
i'm not familiar with OCaml though
|