IRC Log


Saturday March 5, 2011

[Time] NameMessage
[07:47] vrc hi
[07:47] vrc had a quick question about the pub-sub model and reliability
[07:48] vrc i set up a pub-sub model..and if my subscribers receive packets (in a loop) -> and spend a lot of time processing them, will they lose packets
[07:49] vrc or is the pub-sub model guaranteed to provide reliability i.e, all packets sent by the pub are received by each sub..i do realize that some packets in the beginning could be lost due to connection delays, etc
[07:49] vrc thanks - i'm new to zeromq
[07:51] cremes vrc: think of a PUB socket as a radio broadcast
[07:51] cremes if your subscriber isn't around to "hear" it, there is no guarantee it will get all packets
[07:52] cremes by default, PUB will queue packets for all connected subscribers but a slow subscriber can cause a huge backlog
[07:52] cremes and memory pressure on the publisher if they can't keep up
[07:52] cremes so, there is a mechanism called ZM_HWM (high water mark)
[07:53] cremes if there are more than X outstanding packets (where X is the HWM) then the pub socket drops them to /dev/null
[07:53] cremes if you need reliability, then you need to work out some kind of 'ack' mechanism and a pub/sub pattern is no
[07:53] cremes longer appropriate
[07:53] cremes be sure to read all 4 chapters of the guide... of lot of these use-cases are discussed in there
[07:54] cremes vrc: did that help?
[07:57] vrc cremes - thank you very much. that was very useful - in fact, i was reading your comments somewhere else on the internet
[07:57] cremes heh
[07:58] cremes well, it's bed time; good luck with your investigations
[07:58] vrc even if you use PUB/SUB over TCP-IP, there's no guaranteed delivery ?
[07:58] cremes this channel is usually pretty well covered during the weekdays
[07:58] cremes no
[07:59] cremes read the guide... this is almost faq :)
[07:59] vrc ok. thanks - won't keep you up
[07:59] cremes cool
[08:16] guido_g good morning all
[09:43] pieter_hintjens guido_g: good morning
[09:43] guido_g howdy pieter_hintjens
[09:43] pieter_hintjens how's it going... ?
[09:44] guido_g didn't do anything yesterday
[09:44] guido_g except finding a glitch in the spec, but i need to verify that first
[09:45] pieter_hintjens a glitch, interesting
[09:45] guido_g yep
[09:46] guido_g in client <-> broker there is no client address mentioned
[09:46] guido_g but it appears magically (as full list even) in worker <-> broker
[09:46] pieter_hintjens right, it's the same as for broker-worker
[09:47] pieter_hintjens that note should be made to cover both sides
[09:47] guido_g i mean where does the client address in frame 2 come from? that should be explaind
[09:47] guido_g *ed
[09:47] pieter_hintjens hmm, one assumes the implementor knows his XREP/ROUTER stuff
[09:48] guido_g but still it's not clear where the address in frame 2 comes from, itr should be mentioned in the spec -- imho
[09:50] pieter_hintjens indeed... it assumes too much, I'll fix that
[09:50] guido_g oh and eventually move the XREQ note to the front of the spec
[09:51] guido_g so one can see it before its needed
[09:52] pieter_hintjens indeed
[09:53] guido_g i think i'll start on some broker parts this afternoon
[09:54] guido_g if we could get that running, all these "but i need reliability" questions could be answered w/ a link :)
[09:55] pieter_hintjens ah, almost :-)
[09:56] pieter_hintjens after this, it's the Titanic pattern
[09:56] mikko looks like steve got openpgm out
[09:56] pieter_hintjens "but I need reliability on spinning rust"
[09:56] pieter_hintjens mikko: saw that...
[09:57] guido_g rust?
[09:57] pieter_hintjens guido_g: iron oxide? magnetised brown stuff used to hold data
[09:57] guido_g ahhh...
[09:58] guido_g sorry, thought you just got a ssd, so it'll be "built on sand"
[09:59] pieter_hintjens guido_g: I replaced the HD in my notebook with an SSD that Sustrik gave me a while back
[09:59] pieter_hintjens quite neat
[09:59] djc in the guide's fig9, shouldn't the second right-hand box be "Start the SUB sockets before the PUB to avoid loss" (i.e. the other way around)?
[09:59] pieter_hintjens djc: let me check...
[10:00] guido_g pieter_hintjens: spec:7 "Clients will issue at most one request at a time, and will wait for a reply in realtime." <- can this be lifted, i can't see why
[10:01] djc pieter_hintjens: in the bottom box it also says 9zeromq, I think it means #zeromq?
[10:01] guido_g oops... time for furniture shopping
[10:01] guido_g l8tr
[10:02] pieter_hintjens guido_g: it can't be lifted without other changes
[10:02] pieter_hintjens djc: fixed both errors, thanks
[10:02] djc awesome, thanks
[10:02] djc I could write up a list of language refinements if you're interested
[10:02] djc I presume you're not a native speaker, either? :)
[10:03] pieter_hintjens djc: of what language?
[10:03] djc the refinements, or a native speaker?
[10:03] djc if the former, some things in the guide read awkwardly
[10:04] pieter_hintjens uhm, English is my native language, yes
[10:04] pieter_hintjens feel free to point out the awkward parts
[10:04] djc oh, ok, I assumed from your name
[10:04] djc sorry about that!
[10:04] pieter_hintjens np
[10:05] pieter_hintjens there's an issue tracker on the Guide github
[10:05] djc ok
[10:06] djc another question: what's the reason I have to memcpy() my data to the zmq_message, for example in the guide's pub/sub example's publisher
[10:06] djc it seems like it should be okay to reference it as long as I make sure it stays around until the zmq_msg_close()?
[10:07] pieter_hintjens sure
[10:07] mikko djc: with inproc that is ok
[10:07] pieter_hintjens the examples don't try to optimise
[10:08] pieter_hintjens mikko: fixed issue #3 on ztools
[10:09] mikko thanks
[10:09] mikko im trying to figure out a good build order for zeromq2 in the daily builds
[10:09] mikko tests use the same ports so concurrent builds have intermittent test fails
[10:10] pieter_hintjens how so?
[10:10] mikko well, multiple builds run test that tries to bind to 5555
[10:11] mikko which throws EADDRINUSE
[10:11] mikko btw, did you see this: https://github.com/mkoppanen/zeromq2-docs
[10:11] mikko that uses submobules to bring everything in to same 'workspace'
[10:12] pieter_hintjens ah, right... tests should use unique ports
[10:12] mikko you mean randomise the port?
[10:12] pieter_hintjens not randomize, but allocate sequentially or something
[10:12] pieter_hintjens number the tests 1 up
[10:13] mikko that's all fine but i build 4 zeromq builds on the same machine at the same time
[10:13] pieter_hintjens oh, hang on, it's the same test running in parallel, right
[10:13] pieter_hintjens bleh
[10:13] mikko so sequence would be the same
[10:13] pieter_hintjens no answer except "serialize the tests"
[10:14] pieter_hintjens btw, we forked off zeromq2-0 as well
[10:14] djc hmmm, so what does it mean for zmq_msg_init_data() to "take ownership of the supplied buffer"? for example, I want to send a part of a stack-allocated buffer, what would be the optimal way to put this into a message?
[10:14] djc given that there are no weird threading things going on in my app
[10:15] pieter_hintjens djc: 'optimal' as in 'save the last CPU cycle' or as in 'don't write code that confuses me'?
[10:15] pieter_hintjens if you aren't really comfortable with the API you should not be trying to optimize, period
[10:15] pieter_hintjens and even when you are, you should still not be optimizing
[10:16] djc this is me trying to become comfortable with the API :)
[10:16] djc I know, I know, I should only optimize after stress-testing
[10:16] djc I was just wondering about the underlying model
[10:17] pieter_hintjens use mempcy and focus on more interesting aspects of the model
[10:17] djc ok
[10:38] mikko pieter_hintjens: i updated zfl README.txt last night
[10:38] mikko but didnt have time to investigate into generating README.md
[10:38] mikko can you update that?
[10:39] pieter_hintjens sure, it uses gitdown
[10:39] pieter_hintjens will do now
[10:39] pieter_hintjens done
[10:53] mikko thanks
[14:05] larhat Hi, folks! If i set ZMQ_HWM=1 and ZMQ_SWP=N (N>0) on socket, process will store all, that cannot be processed immediately, on disk. But it process fails, can it restore swapped messages and retry sending? I wonder, can i use this for some kind of persistence queue?
[14:13] Guthur larhat, I have a feeling it cannot
[14:51] mikko larhat: no, it cannot be done at the moment
[14:51] mikko larhat: there has been discussion about creating persistence devices
[14:52] mikko where you could 'plug in' persistence on sockets you need
[14:54] larhat ok, thanks, guys
[21:45] surikator hi! anyone out there who tried ZeroMQ for OCaml?
[22:15] Guthur is there any reason why rtrouter from the guide should not worker with inproc transport
[22:32] Guthur surikator, there does not yet seem to be a binding for OCaml
[22:32] surikator yes, there is...
[22:32] Guthur oh ok, it's not mentioned on the site
[22:32] surikator https://github.com/pdhborges/ocaml-zmq
[22:33] surikator =)
[22:35] surikator but regarding zeromq in general, can we pass whole data structures and objects through the sockets?
[22:38] Guthur surikator, you'd need to serialize them
[22:39] Guthur I'm not familiar with that binding, it may or may not have that built in
[22:39] Guthur ZeroMQ does not provide it though
[22:39] Guthur there is some pretty decent serialization solutions out there though
[22:43] surikator it doesn't have that built in. it mimics ZeroMQ only.
[22:51] surikator ok, so I'd probably have to use Ocaml Marshal library. which solutions do you mean?
[23:08] Guthur surikator, protocol buffers, thrift, or maybe even JSON
[23:08] Guthur i'm not familiar with OCaml though