ZeroMq IRC Log

Sunday April 4, 2010

[Time] Name	Message
[12:24] mikko	good morning
[12:34] Skaag	good morning mikko
[12:34] Skaag	:-)
[12:41] mikko	i added a couple of more doc pages
[12:42] mikko	http://wiki.github.com/mkoppanen/php-zeromq/persistent-context-socket
[12:42] mikko	and updated this a bit http://wiki.github.com/mkoppanen/php-zeromq/api-documentation
[12:55] Skaag	awesome
[12:56] mikko	now, time to head out to see tiergarten and brandenburg tor
[12:56] mikko	laters
[13:45] Skaag	tiergarten or biergarten?! :-)
[16:31] jomofo	If I want to create a new language binding for 0MQ, should I use the LGPL? The language is friendlier toward the Eclipse license, but I think either would work.
[18:10] desrt	sustrik: hey
[18:12] sustrik	desrt: hi
[18:13] desrt	sustrik: are you a zmq developer?
[18:13] desrt	(just out of curiosity)
[18:13] sustrik	you mean whether i wrote the 0mq code?
[18:13] sustrik	yes, most of it
[18:13] desrt	cool
[18:14] desrt	a few questions
[18:14] desrt	1) is there any chance of some convenience additions/cleanups to some of the APIs?
[18:14] sustrik	jomofo_: you can use any license that suits you
[18:15] desrt	like maybe using opaque structure types zmq_socket_t, zmq_context_t instead of void*
[18:15] desrt	and having all methods on sockets start like zmq_socket_ {send, recv, whatever}
[18:15] desrt	(note: zmq_msg follows this convention, but not socket and context)
[18:15] sustrik	desrt: the method names are based on POSIX socket API
[18:15] desrt	ah. interesting.
[18:15] sustrik	usign zmq_type_function would break the 1:1 correspondence
[18:16] desrt	well.. the correspondence is already fairly loose...
[18:16] desrt	they take pointers intaed of fds
[18:16] desrt	and strings instead of struct sockaddr and size
[18:16] sustrik	that's because POSIX doesn't allow for user-space implementation of fds
[18:17] sustrik	if the codebase ever moves to kernel space the void* would of course be replaced by fd
[18:17] desrt	hm.
[18:17] sustrik	context will drop out entirely
[18:17] desrt	interesting.
[18:18] desrt	ok. higher level question:
[18:18] desrt	say i have a publisher of a stream of events and many subscribers to that stream
[18:18] desrt	and some devices in between storing the events
[18:19] desrt	how does a subscriber tell a device about which events it has already seen?
[18:20] desrt	is this part of the ZMQ_SUBSCRIBE sockopt?
[18:20] sustrik	it's like a radio transmission, you start getting messages at the point when you create the connection
[18:20] sustrik	you'll never get older messages
[18:20] desrt	oh.
[18:21] desrt	i was under the impressiont that devices could queue up events
[18:21] desrt	so that implies that it's quite possible to miss events in the case of shaky connectivity, then?
[18:21] sustrik	they can, but in PUB/SUB messaging pattern it simply doesn't make sense
[18:22] sustrik	how long would you cache the events?
[18:22] sustrik	a day?
[18:22] sustrik	month?
[18:22] sustrik	year?
[18:22] desrt	i have two possible answers
[18:22] desrt	the easy one is "forever"
[18:22] sustrik	out of memory results
[18:22] desrt	the less easy one goes something like this:
[18:22] desrt	you maintain a list of active subscribers
[18:23] desrt	each subscriber must check in at least once a month (say) to keep their subscription valid
[18:23] desrt	and you keep events so long as you have active subscribers who the server knows are not up-to-date
[18:23] sustrik	how do you know there'll be a subscriber before it evn connects?
[18:23] desrt	i'm saying that a subscriber isn't a subscriber until they first subscribe
[18:24] sustrik	ah, sure, it works that way
[18:24] desrt	ah. nice.
[18:24] desrt	so i can subscribe
[18:24] sustrik	the only thing you need to do is set the identity of the subscriber
[18:24] desrt	go away for a week
[18:24] desrt	come back and say "i'm back!"
[18:24] desrt	and get a list of everything that happened meanwhile?
[18:24] sustrik	so that the peer knows it's the same subscriber
[18:24] sustrik	when reconnection happens
[18:24] desrt	yes. that's exactly what i'm getting at
[18:25] desrt	awesome :)
[18:25] sustrik	you just get it
[18:25] sustrik	no need to do anything special
[18:25] desrt	so just popping a uuid in there is a reasonable thing to do
[18:26] sustrik	well, the uuid is generated automatically anyway
[18:26] sustrik	what you have to ensure
[18:26] desrt	ya. but i need it to be the same oen across invocations
[18:26] sustrik	if you want the subscription to last after subscriber's restart
[18:27] sustrik	is that the identity is always the same
[18:27] desrt	so how does that work with respect to subscriptions timing out?
[18:27] desrt	surely the server won't save messages for an identity forever...
[18:27] sustrik	no timeouts
[18:27] sustrik	it would now :)
[18:27] desrt	heh
[18:27] desrt	so it has an on-disk cache, or....?
[18:27] sustrik	feel free to make a patch
[18:27] sustrik	in-memory
[18:27] desrt	oh
[18:28] desrt	so it only saves it until reboot...
[18:28] sustrik	yes
[18:28] desrt	hrmph
[18:28] sustrik	sure, there's a lot of messaging solutions out there
[18:28] sustrik	rabbitma, activemq etc.
[18:29] sustrik	try one of those
[18:29] desrt	ya. i've had a look around
[18:29] desrt	what attracted me to zmq was its simplicity
[18:29] sustrik	yup
[18:29] desrt	my original introduction to this stuff was via a talk about amqp at fudcon
[18:29] sustrik	however, it's not as feature complete as other projects
[18:30] sustrik	they've mentioned 0mq?
[18:30] desrt	i walked out of the room thinking "neat. sounds useful. way too bloated/complex."
[18:30] sustrik	interesting
[18:30] desrt	no
[18:30] desrt	a few days ago i saw a post on reddit about 0mq being by the original makers of amqp because amqp had gotten too bloated/complex
[18:30] sustrik	ok, i see
[18:30] desrt	and i was like "hm. i agree. it must be nice!"
[18:31] sustrik	well, persistence is still missing
[18:31] desrt	i'm writing the configuration system for GNOME 3
[18:31] desrt	currently figuring out how to do network support
[18:31] sustrik	why do you need persistence?
[18:32] desrt	it has support for databases on your computer that are controlled by someone else
[18:32] desrt	"site settings"
[18:32] desrt	the way it works, when you are online with the network the updates come right away
[18:32] desrt	and you keep a local copy
[18:32] desrt	so when you go offline you still have the settings
[18:32] desrt	and when you connect again, it syncs back up
[18:33] sustrik	ok, understood
[18:33] desrt	i currently have a flaky hand-written tcp protocol to do this
[18:33] desrt	was looking for something a bit less crappy
[18:34] desrt	also: i'm sort of drawn to the idea that people could do cool stuff with zeromq that has nothing to do with my particular use case that would result in new ways for people to use my stuff
[18:34] desrt	like add new queue or distribution logic, etc...
[18:35] sustrik	desrt: right
[18:35] sustrik	still, at the moment there's no persistence
[18:35] sustrik	so you have 3 options:
[18:35] sustrik	1. build persistence on top of 0mq
[18:35] sustrik	2. patch 0mq
[18:35] sustrik	3. move to a different solution
[18:36] desrt	4. use it anyway :)
[18:36] sustrik	5. hack the persistence in
[18:37] sustrik	persistence is really a very complex problem
[18:37] sustrik	to do it properly, lot of resources is needed
[18:37] sustrik	however, to server a particulat use case, it's normally pretty easy to do
[18:37] desrt	i'm thinking along these lines: if i can detect that i missed events then i can just redownload the whole DB
[18:38] sustrik	so you can place sequence numbers into messages
[18:38] desrt	that's what i do now
[18:38] desrt	each change to the DB has a sequence number
[18:38] desrt	the connecting client tells of the last change it saw
[18:38] desrt	and the server replays what happened since
[18:38] sustrik	you can use req/rep sockets for that
[18:39] desrt	sequence numbers are 64bit, so i assume they never wrap
[18:39] desrt	right.
[18:39] sustrik	it's much more natural than pub/sub anyway
[18:39] desrt	that gets me RPC-over-TCP(ish), right?
[18:39] sustrik	pub/sub is for things like stock quotes
[18:39] sustrik	media
[18:39] sustrik	etc.
[18:39] sustrik	content that lookses value quickly
[18:39] sustrik	looses*
[18:39] sustrik	desrt: yes
[18:39] sustrik	do you mind a roundtrip?
[18:39] desrt	might be appropriate
[18:40] desrt	well
[18:40] desrt	my protocol works sort of like imap now
[18:40] desrt	it's very much not structured RPC-like
[18:40] desrt	the client understands that the server sends it a stream of events
[18:41] desrt	those events are because things happened [some time]
[18:41] sustrik	still, the subscriber must acknowledge update receival
[18:41] desrt	so when the client goes online it just says "last time you saw me was _____"
[18:41] desrt	and the server sends events.....
[18:41] desrt	and the client just sees those as events....
[18:41] desrt	no. they need not.
[18:41] desrt	since the client keeps track of what it knows -- not the server
[18:41] sustrik	how do you know how long you keep the updates in the database?
[18:42] desrt	so if the client gets the new events then it will update its serial number and ask for new events next time
[18:42] sustrik	hm, forever, i guess?
[18:42] desrt	yes and no
[18:42] desrt	events will eventually drop from the queue if they are overridden by newer events
[18:42] desrt	like if someone writes to /some/setting with sequence number 5
[18:42] desrt	then laster /some/setting again with sequence number 8
[18:42] desrt	5 will never be sent ever again
[18:42] sustrik	sure, understood
[18:43] desrt	so the number of events is bound by the natural size of the database
[18:44] sustrik	well, i don't believe abt queueing system has functionality like that
[18:44] sustrik	if you send X to the queue
[18:44] desrt	k
[18:45] sustrik	it will be delivered
[18:45] desrt	i might just stick with my homebrew then
[18:45] sustrik	not depending on whether you send X+1 later on
[18:45] sustrik	yes, that's an option
[18:45] sustrik	anyway, the scenario is interesting
[18:46] sustrik	i'll keep it in mind
[18:46] sustrik	it's somehow similar to a thing called "last value cache"
[18:46] sustrik	it's almost the same as with your scenario
[18:47] sustrik	however, after client connect it get _whole_ set of settings
[18:47] sustrik	not just those that have been changed in the meantime
[18:47] desrt	hmm
[18:47] desrt	well, my situation is slightly more complicated
[18:48] desrt	since writes may contain more than one value
[18:48] desrt	and there could be non-simple overlaps between different write events
[18:48] sustrik	then you definitely have to perform the logic on application level
[18:48] sustrik	however, you can still use 0mq instead of raw sockets
[18:49] sustrik	just to transport the updates
[18:49] desrt	i'm using GNIO anyway (glib's socket library)
[18:49] desrt	which has the benefit of glib mainloop integration
[18:49] desrt	(which is another question i wanted to ask about 0mq)
[18:50] sustrik	what question exactly?
[18:50] desrt	i notice you have your own poll abstraction
[18:50] desrt	but most non-headless apps will already have their own mainloop
[18:50] desrt	and i didn't see a way to integrate 0mq with another mainloop (or even get the fd...)
[18:50] sustrik	well, there's no way out of that
[18:51] sustrik	POSIX doesn't allow for creating your own fd implementation in user space
[18:51] sustrik	lot of political pressure would be needed to amend POSIX
[18:52] sustrik	still, it'll take lot of time till all OSes would implement the new functionality
[18:52] desrt	many libraries provide mainloop integration hooks
[18:52] desrt	dbus comes to mind
[18:52] sustrik	how does it achieve it?
[18:52] desrt	you give it a set of functions to call
[18:52] desrt	to start watching an fd for read/write
[18:53] sustrik	ah, sure that's easy
[18:53] desrt	i guess your case is hard since you're essentially using a synchronous interface
[18:53] sustrik	what i mean is that you cannot create your own "fd" in user space
[18:53] sustrik	on Linux you have eventfd
[18:53] sustrik	which is almost it
[18:54] sustrik	(a little bit is missing that can be theoretically added in Linux)
[18:54] sustrik	however, it's not POSIX
[18:54] sustrik	so other OSes have no equivalent
[21:45] mikko	sustrik: hi!
[21:54] mikko	too late