Sunday April 4, 2010

[Time] NameMessage
[12:24] mikko good morning
[12:34] Skaag good morning mikko
[12:34] Skaag :-)
[12:41] mikko i added a couple of more doc pages
[12:42] mikko
[12:42] mikko and updated this a bit
[12:55] Skaag awesome
[12:56] mikko now, time to head out to see tiergarten and brandenburg tor
[12:56] mikko laters
[13:45] Skaag tiergarten or biergarten?! :-)
[16:31] jomofo If I want to create a new language binding for 0MQ, should I use the LGPL? The language is friendlier toward the Eclipse license, but I think either would work.
[18:10] desrt sustrik: hey
[18:12] sustrik desrt: hi
[18:13] desrt sustrik: are you a zmq developer?
[18:13] desrt (just out of curiosity)
[18:13] sustrik you mean whether i wrote the 0mq code?
[18:13] sustrik yes, most of it
[18:13] desrt cool
[18:14] desrt a few questions
[18:14] desrt 1) is there any chance of some convenience additions/cleanups to some of the APIs?
[18:14] sustrik jomofo_: you can use any license that suits you
[18:15] desrt like maybe using opaque structure types zmq_socket_t, zmq_context_t instead of void*
[18:15] desrt and having all methods on sockets start like zmq_socket_ {send, recv, whatever}
[18:15] desrt (note: zmq_msg follows this convention, but not socket and context)
[18:15] sustrik desrt: the method names are based on POSIX socket API
[18:15] desrt ah. interesting.
[18:15] sustrik usign zmq_type_function would break the 1:1 correspondence
[18:16] desrt well.. the correspondence is already fairly loose...
[18:16] desrt they take pointers intaed of fds
[18:16] desrt and strings instead of struct sockaddr and size
[18:16] sustrik that's because POSIX doesn't allow for user-space implementation of fds
[18:17] sustrik if the codebase ever moves to kernel space the void* would of course be replaced by fd
[18:17] desrt hm.
[18:17] sustrik context will drop out entirely
[18:17] desrt interesting.
[18:18] desrt ok. higher level question:
[18:18] desrt say i have a publisher of a stream of events and many subscribers to that stream
[18:18] desrt and some devices in between storing the events
[18:19] desrt how does a subscriber tell a device about which events it has already seen?
[18:20] desrt is this part of the ZMQ_SUBSCRIBE sockopt?
[18:20] sustrik it's like a radio transmission, you start getting messages at the point when you create the connection
[18:20] sustrik you'll never get older messages
[18:20] desrt oh.
[18:21] desrt i was under the impressiont that devices could queue up events
[18:21] desrt so that implies that it's quite possible to miss events in the case of shaky connectivity, then?
[18:21] sustrik they can, but in PUB/SUB messaging pattern it simply doesn't make sense
[18:22] sustrik how long would you cache the events?
[18:22] sustrik a day?
[18:22] sustrik month?
[18:22] sustrik year?
[18:22] desrt i have two possible answers
[18:22] desrt the easy one is "forever"
[18:22] sustrik out of memory results
[18:22] desrt the less easy one goes something like this:
[18:22] desrt you maintain a list of active subscribers
[18:23] desrt each subscriber must check in at least once a month (say) to keep their subscription valid
[18:23] desrt and you keep events so long as you have active subscribers who the server knows are not up-to-date
[18:23] sustrik how do you know there'll be a subscriber before it evn connects?
[18:23] desrt i'm saying that a subscriber isn't a subscriber until they first subscribe
[18:24] sustrik ah, sure, it works that way
[18:24] desrt ah. nice.
[18:24] desrt so i can subscribe
[18:24] sustrik the only thing you need to do is set the identity of the subscriber
[18:24] desrt go away for a week
[18:24] desrt come back and say "i'm back!"
[18:24] desrt and get a list of everything that happened meanwhile?
[18:24] sustrik so that the peer knows it's the same subscriber
[18:24] sustrik when reconnection happens
[18:24] desrt yes. that's exactly what i'm getting at
[18:25] desrt awesome :)
[18:25] sustrik you just get it
[18:25] sustrik no need to do anything special
[18:25] desrt so just popping a uuid in there is a reasonable thing to do
[18:26] sustrik well, the uuid is generated automatically anyway
[18:26] sustrik what you have to ensure
[18:26] desrt ya. but i need it to be the same oen across invocations
[18:26] sustrik if you want the subscription to last after subscriber's restart
[18:27] sustrik is that the identity is always the same
[18:27] desrt so how does that work with respect to subscriptions timing out?
[18:27] desrt surely the server won't save messages for an identity forever...
[18:27] sustrik no timeouts
[18:27] sustrik it would now :)
[18:27] desrt heh
[18:27] desrt so it has an on-disk cache, or....?
[18:27] sustrik feel free to make a patch
[18:27] sustrik in-memory
[18:27] desrt oh
[18:28] desrt so it only saves it until reboot...
[18:28] sustrik yes
[18:28] desrt hrmph
[18:28] sustrik sure, there's a lot of messaging solutions out there
[18:28] sustrik rabbitma, activemq etc.
[18:29] sustrik try one of those
[18:29] desrt ya. i've had a look around
[18:29] desrt what attracted me to zmq was its simplicity
[18:29] sustrik yup
[18:29] desrt my original introduction to this stuff was via a talk about amqp at fudcon
[18:29] sustrik however, it's not as feature complete as other projects
[18:30] sustrik they've mentioned 0mq?
[18:30] desrt i walked out of the room thinking "neat. sounds useful. way too bloated/complex."
[18:30] sustrik interesting
[18:30] desrt no
[18:30] desrt a few days ago i saw a post on reddit about 0mq being by the original makers of amqp because amqp had gotten too bloated/complex
[18:30] sustrik ok, i see
[18:30] desrt and i was like "hm. i agree. it must be nice!"
[18:31] sustrik well, persistence is still missing
[18:31] desrt i'm writing the configuration system for GNOME 3
[18:31] desrt currently figuring out how to do network support
[18:31] sustrik why do you need persistence?
[18:32] desrt it has support for databases on your computer that are controlled by someone else
[18:32] desrt "site settings"
[18:32] desrt the way it works, when you are online with the network the updates come right away
[18:32] desrt and you keep a local copy
[18:32] desrt so when you go offline you still have the settings
[18:32] desrt and when you connect again, it syncs back up
[18:33] sustrik ok, understood
[18:33] desrt i currently have a flaky hand-written tcp protocol to do this
[18:33] desrt was looking for something a bit less crappy
[18:34] desrt also: i'm sort of drawn to the idea that people could do cool stuff with zeromq that has nothing to do with my particular use case that would result in new ways for people to use my stuff
[18:34] desrt like add new queue or distribution logic, etc...
[18:35] sustrik desrt: right
[18:35] sustrik still, at the moment there's no persistence
[18:35] sustrik so you have 3 options:
[18:35] sustrik 1. build persistence on top of 0mq
[18:35] sustrik 2. patch 0mq
[18:35] sustrik 3. move to a different solution
[18:36] desrt 4. use it anyway :)
[18:36] sustrik 5. hack the persistence in
[18:37] sustrik persistence is really a very complex problem
[18:37] sustrik to do it properly, lot of resources is needed
[18:37] sustrik however, to server a particulat use case, it's normally pretty easy to do
[18:37] desrt i'm thinking along these lines: if i can detect that i missed events then i can just redownload the whole DB
[18:38] sustrik so you can place sequence numbers into messages
[18:38] desrt that's what i do now
[18:38] desrt each change to the DB has a sequence number
[18:38] desrt the connecting client tells of the last change it saw
[18:38] desrt and the server replays what happened since
[18:38] sustrik you can use req/rep sockets for that
[18:39] desrt sequence numbers are 64bit, so i assume they never wrap
[18:39] desrt right.
[18:39] sustrik it's much more natural than pub/sub anyway
[18:39] desrt that gets me RPC-over-TCP(ish), right?
[18:39] sustrik pub/sub is for things like stock quotes
[18:39] sustrik media
[18:39] sustrik etc.
[18:39] sustrik content that lookses value quickly
[18:39] sustrik looses*
[18:39] sustrik desrt: yes
[18:39] sustrik do you mind a roundtrip?
[18:39] desrt might be appropriate
[18:40] desrt well
[18:40] desrt my protocol works sort of like imap now
[18:40] desrt it's very much not structured RPC-like
[18:40] desrt the client understands that the server sends it a stream of events
[18:41] desrt those events are because things happened [some time]
[18:41] sustrik still, the subscriber must acknowledge update receival
[18:41] desrt so when the client goes online it just says "last time you saw me was _____"
[18:41] desrt and the server sends events.....
[18:41] desrt and the client just sees those as events....
[18:41] desrt no. they need not.
[18:41] desrt since the client keeps track of what it knows -- not the server
[18:41] sustrik how do you know how long you keep the updates in the database?
[18:42] desrt so if the client gets the new events then it will update its serial number and ask for new events next time
[18:42] sustrik hm, forever, i guess?
[18:42] desrt yes and no
[18:42] desrt events will eventually drop from the queue if they are overridden by newer events
[18:42] desrt like if someone writes to /some/setting with sequence number 5
[18:42] desrt then laster /some/setting again with sequence number 8
[18:42] desrt 5 will never be sent ever again
[18:42] sustrik sure, understood
[18:43] desrt so the number of events is bound by the natural size of the database
[18:44] sustrik well, i don't believe abt queueing system has functionality like that
[18:44] sustrik if you send X to the queue
[18:44] desrt k
[18:45] sustrik it will be delivered
[18:45] desrt i might just stick with my homebrew then
[18:45] sustrik not depending on whether you send X+1 later on
[18:45] sustrik yes, that's an option
[18:45] sustrik anyway, the scenario is interesting
[18:46] sustrik i'll keep it in mind
[18:46] sustrik it's somehow similar to a thing called "last value cache"
[18:46] sustrik it's almost the same as with your scenario
[18:47] sustrik however, after client connect it get _whole_ set of settings
[18:47] sustrik not just those that have been changed in the meantime
[18:47] desrt hmm
[18:47] desrt well, my situation is slightly more complicated
[18:48] desrt since writes may contain more than one value
[18:48] desrt and there could be non-simple overlaps between different write events
[18:48] sustrik then you definitely have to perform the logic on application level
[18:48] sustrik however, you can still use 0mq instead of raw sockets
[18:49] sustrik just to transport the updates
[18:49] desrt i'm using GNIO anyway (glib's socket library)
[18:49] desrt which has the benefit of glib mainloop integration
[18:49] desrt (which is another question i wanted to ask about 0mq)
[18:50] sustrik what question exactly?
[18:50] desrt i notice you have your own poll abstraction
[18:50] desrt but most non-headless apps will already have their own mainloop
[18:50] desrt and i didn't see a way to integrate 0mq with another mainloop (or even get the fd...)
[18:50] sustrik well, there's no way out of that
[18:51] sustrik POSIX doesn't allow for creating your own fd implementation in user space
[18:51] sustrik lot of political pressure would be needed to amend POSIX
[18:52] sustrik still, it'll take lot of time till all OSes would implement the new functionality
[18:52] desrt many libraries provide mainloop integration hooks
[18:52] desrt dbus comes to mind
[18:52] sustrik how does it achieve it?
[18:52] desrt you give it a set of functions to call
[18:52] desrt to start watching an fd for read/write
[18:53] sustrik ah, sure that's easy
[18:53] desrt i guess your case is hard since you're essentially using a synchronous interface
[18:53] sustrik what i mean is that you cannot create your own "fd" in user space
[18:53] sustrik on Linux you have eventfd
[18:53] sustrik which is almost it
[18:54] sustrik (a little bit is missing that can be theoretically added in Linux)
[18:54] sustrik however, it's not POSIX
[18:54] sustrik so other OSes have no equivalent
[21:45] mikko sustrik: hi!
[21:54] mikko too late