ZeroMq IRC Log

Monday May 24, 2010

[Time] Name	Message
[01:34] postmodern	so when i read "A single socket may be connected to an arbitrary number of peer addresses using zmq_connect(), while also having an arbitrary number of local addresses assigned to it using zmq_bind()."
[01:34] postmodern	im a bit confused how zmq routes messages through sockets with multiple end-points?
[01:34] postmodern	does it just fan-out?
[01:34] postmodern	or is it load-balanced?
[03:25] cremes	postmodern: it depends on the socket type; i don't know for certain, but in some cases it uses round-robin and in other fair-balance queueing meaning...
[03:25] postmodern	ah interesting
[03:25] cremes	that it sends messages based on its internal knowledge of queue lengths
[03:25] cremes	i suggest asking on the list for a breakdown of queueing algorithms by socket
[05:46] sustrik	hassox: http://lwn.net/Articles/370307/
[05:47] sustrik	postmodern: each socket type does what you would expect it to do
[05:47] sustrik	requests are load balanced between services
[05:47] sustrik	while publishes are distributed to every subscriber
[05:48] postmodern	sustrik_, also there is no restriction on which direction the connection was made from, for pub/sub?
[05:49] postmodern	sustrik_, for instance i could connect into each subscriber socket, and publish to them
[05:49] postmodern	sustrik_, instead of binding to a port, and waiting for the subscribers to connect in
[05:56] hassox	sustrik_: churs
[05:57] hassox	sustrik_: is 0mq able to work with threadless evented systems like ruby EventMachine and Node.js?
[06:05] sustrik	postmodern: no restriction
[06:05] postmodern	excellent
[06:05] sustrik	hassox: no idea, depends of what kind of requirements the two systems have
[06:06] hassox	sustrik_: they're not threaded
[06:06] sustrik	single-thread?
[06:06] hassox	event loop
[06:06] hassox	yeah single thread, but they live in an event loop
[06:06] hassox	so async style
[06:06] sustrik	what do you have to conform to register to the event loop?
[06:07] hassox	sorry my train has just arrived I'll be back on later
[06:07] hassox	not sure what that means but I'll jump on when I get back
[09:30] hassox	sustrik_: still here mate?
[09:30] sustrik	yup
[09:31] sustrik	how do you add event sources to your event loop?
[09:31] sustrik	what does the event source have to conform to?
[09:49] hassox	so
[09:49] hassox	with event machine for eg
[09:49] hassox	you create a socket, and when an event occurs, i.e. some data is received
[09:49] hassox	a method on the handler is called
[09:50] hassox	you can then add callbacks, timers etc
[09:50] hassox	inside the method
[09:50] hassox	there's no threads, you instead gain concurrency by splitting execution up and using callbacks
[09:51] hassox	there is an event queue, where events are queued up, and then the event loop consumes each event from the queue
[09:51] mikko	can you add custom event sources?
[09:51] mikko	other than fds
[09:51] hassox	what do you mean
[09:51] hassox	manually put things onto the event queue?
[09:52] sustrik	no, you put a socket into the event loop, right? can you put something else there?
[09:52] hassox	the whole application lives in the event loop
[09:53] hassox	so you can have many sockets, tcp connections database connections, or just have your application setting things up to do and executing them
[09:53] sustrik	how does the loop know what sockets to inspect?
[09:54] hassox	I'm not sure of the underlying system calls that are made...
[09:54] hassox	I usually live a little higher up the stack
[09:54] sustrik	you create a connection to database
[09:54] mikko	like for example libev's case http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#Examples-8
[09:54] sustrik	how do you integrate it with the loop?
[09:55] hassox	lemme see if I can find someone who will be able to talk at the right level
[09:56] hassox	I just grab a socket
[09:57] hassox	but not sure underneath at the c level what it's actually doing
[09:57] hassox	they don't look like they're about
[09:58] sustrik	example:
[09:58] sustrik	s = create_socket (...);
[09:58] sustrik	loop.register (s);
[09:59] sustrik	does you use it in a similar way?
[09:59] hassox	http://github.com/eventmachine/eventmachine/blob/650ff5e19ee87c0bceba46e5961c415d64e6e519/lib/em/protocols/socks4.rb
[09:59] hassox	there's an example
[10:00] hassox	here's the underlying class
[10:00] hassox	http://github.com/eventmachine/eventmachine/blob/master/lib/em/connection.rb
[10:02] hassox	all the c is in http://github.com/eventmachine/eventmachine/tree/master/ext/
[10:02] hassox	i'll brb
[10:02] sustrik	i don't speak ruby, sorry, can you point me to the line where you open the socket?
[10:32] hassox	sustrik_: an http connection http://github.com/eventmachine/eventmachine/blob/fb11a41671b5bc1b15372d319280cecf086c9104/lib/eventmachine.rb#L578
[10:33] hassox	not sure what c is going on underneath
[10:36] sustrik	hassox: forget about c
[10:36] sustrik	what are you doing in ruby?
[10:36] hassox	I use the higer level stuff
[10:37] sustrik	and that's?
[10:37] hassox	I just fire up an http server, or smtp server
[10:37] hassox	or both
[10:37] sustrik	and?
[10:37] hassox	I'm trying to find someone who can talk at the right level
[10:37] hassox	well
[10:37] hassox	when ythe connection receives data
[10:37] hassox	EM calls receive_data(data)
[10:37] hassox	so you react to the event
[10:38] hassox	I'm not describing this well :(
[10:38] sustrik	that's inside HTTP server?
[10:38] hassox	maybe the concept will be easier to talk over node
[10:38] mikko	sustrik_: its probably like libevent http
[10:38] mikko	sustrik_: similar concept to http://3.rdrail.net/blog/libevent-webserver-in-40-lines-of-c/
[10:38] mikko	unless i am wrong
[10:38] mikko	underneath it uses normal event loop
[10:39] hassox	the docs are better for node
[10:39] mikko	which usually polls a socket
[10:39] hassox	http://nodejs.org/
[10:39] mikko	node.js is a lot more complicated
[10:39] hassox	oh
[10:39] hassox	:'(
[10:39] hassox	damn I suck at talking about this
[10:39] hassox	I'm sorry lads
[10:39] hassox	I'm not familiar with territory this low down
[10:40] sustrik	np, can you check the mikko's link?
[10:40] sustrik	are you doing something similar?
[10:41] hassox	I don't know C but the concept seems about right
[10:42] sustrik	so you launch a http server in your process
[10:42] mikko	it's a higher level abstraction where you talk 'http concepts' rather than raw sockets
[10:42] sustrik	and supply it a callback function
[10:42] mikko	but in the end its just an event loop
[10:42] hassox	yeah
[10:42] hassox	EventMachine and Node.js are both event loops at their hearts
[10:43] hassox	but all the 0mq stuff I've seen talks about requiring 2 or more threads
[10:43] sustrik	hassox: ok, then you need to check your documentation whether there are extension points in the library you are using
[10:43] sustrik	i.e. ways to plug-in non-HTTP entities to the even loop
[10:44] mikko	hassox: http://www.igvita.com/2008/05/27/ruby-eventmachine-the-speed-demon/
[10:44] mikko	sometghing like that might be relevant to you
[10:44] hassox	yeah that's just doing http servers
[10:45] mikko	hassox: why don't you use 0MQ's event loop?
[10:45] hassox	what I would like to know is can I get a 0mq service running, or is it a must to use threads
[10:45] hassox	there is one?
[10:45] mikko	or do you need the higher level abstraction?
[10:45] hassox	I'd like to be able to use 0mq from my ruby or node programs
[10:47] hassox	wondering if it's possible, or if I _need_ the threads
[10:47] mikko	ideally your event loop such as event machine would support "polling" a callback for events
[10:47] mikko	i would imagine it would be possible to create a struct with necessary io abstraction that would work with any data provider including zeromq
[10:48] mikko	whether event machine does that i dont know
[10:49] mikko	sustrik_: isn't XREQ / XREP non-blocking?
[10:50] mikko	(sorry if im confusing the issue more)
[10:50] hassox	np
[10:50] hassox	I'm just in really unfamiliar territory :(
[10:53] sustrik	mikko: any 0mq socket can be either blocking or non-blocking
[10:53] sustrik	s.recv (&msg);
[10:53] sustrik	vs.
[10:53] sustrik	s.recv (&msg, ZMQ_NOBLOCK);
[10:53] mikko	so the end result would be: you want to embed zmq into node.js/eventmachine loop and receive events and have the events fire the callbacks
[10:53] hassox	that sounds like what I want
[10:53] mikko	this functionality depends on whether your event loop allows you to register custom resources with the loop
[10:53] sustrik	the XREP/XREQ allows you to have multiple request in the air at the same time
[10:54] mikko	as 0MQ does not expose the underlying sockets directly
[10:55] hassox	mikko: but you connect to the 0MQ service via a socket yeah?
[10:55] mikko	the socket 0MQ gives you is an abstraction
[10:55] mikko	not the raw underlying posix socket
[10:56] mikko	http://pastebin.com/zmaZe9QR
[10:56] mikko	as a very elementary pseudo-code
[10:56] mikko	i would imagine soemthing like that would be needed from the event loops side
[10:56] mikko	but i'm not 100% sure
[10:56] mikko	naturally you would need more abstraction but that should show the principle
[10:57] hassox	kk
[11:00] hassox	thanx lads
[11:00] hassox	I'll talk to a guy I know who knows this stuff much better
[11:00] hassox	it looks like it could work though :)
[11:02] mikko	what kind of messaging are you doing?
[11:02] mikko	are you trying to write a zmq server?
[11:02] mikko	or use 0mq within your existing server?
[11:03] hassox	I'd like to use 0mq to talk between a bunch of http servers and supporting scripts
[11:03] mikko	http://github.com/mkoppanen/php-zmq/blob/master/examples/poll-server.php
[11:03] hassox	atm I'm setting things up to use firewall settings and api keys for authentication, but a 0mq setup would make like a _lot_ better
[11:04] mikko	there is an example of php polling server
[11:45] cremes	hassox: i am working on a "zmq-machine" in ruby; i should have something worthy of a push to github in another week or so
[11:45] cremes	but for now, EM and 0mq can't interact
[11:45] hassox	cremes: :D and :(
[11:45] cremes	EM uses POSIX sockets while 0mq is an abstraction on top of POSIX sockets
[11:46] cremes	e.g. you couldn't have an EM handler do a "connect" to a 0mq REQ socket
[11:46] hassox	can you do it in normal ruby?
[11:46] cremes	if you use the rbzmq or ffi-rzmq gems, then yes
[11:46] cremes	you need something that "speaks" 0mq
[11:47] hassox	right
[11:47] hassox	does the zmq-machine use threads or does it assimilate into the event loop?
[11:48] cremes	hassox: good question...
[11:48] cremes	it will have a similar setup to EM
[11:48] cremes	in EM we have the EM.run {} configuration
[11:49] cremes	in zmq-machine it will be almost the same; something like ZM::Context.run {}
[11:49] cremes	each Context will be its own thread; any sockets created within that context will all be non-blocking and will fire callbacks for events
[11:49] cremes	but you can create multiple contexts each with its own thread
[11:49] cremes	think of it as firing up a separate reactor loop for each thread
[11:50] cremes	if EM allowed more than one reactor, we could already do this
[11:50] cremes	make sense?
[11:53] hassox	kinda
[11:53] hassox	so 0mq won't be available as a callback inside the em
[11:53] hassox	but rather it's a parallell event loop in the same process
[11:56] cremes	hassox: yes, it is completely independent of EM; they can't work together
[11:57] hassox	:(