Monday May 24, 2010

[Time] NameMessage
[01:34] postmodern so when i read "A single socket may be connected to an arbitrary number of peer addresses using zmq_connect(), while also having an arbitrary number of local addresses assigned to it using zmq_bind()."
[01:34] postmodern im a bit confused how zmq routes messages through sockets with multiple end-points?
[01:34] postmodern does it just fan-out?
[01:34] postmodern or is it load-balanced?
[03:25] cremes postmodern: it depends on the socket type; i don't know for certain, but in some cases it uses round-robin and in other fair-balance queueing meaning...
[03:25] postmodern ah interesting
[03:25] cremes that it sends messages based on its internal knowledge of queue lengths
[03:25] cremes i suggest asking on the list for a breakdown of queueing algorithms by socket
[05:46] sustrik hassox:
[05:47] sustrik postmodern: each socket type does what you would expect it to do
[05:47] sustrik requests are load balanced between services
[05:47] sustrik while publishes are distributed to every subscriber
[05:48] postmodern sustrik_, also there is no restriction on which direction the connection was made from, for pub/sub?
[05:49] postmodern sustrik_, for instance i could connect into each subscriber socket, and publish to them
[05:49] postmodern sustrik_, instead of binding to a port, and waiting for the subscribers to connect in
[05:56] hassox sustrik_: churs
[05:57] hassox sustrik_: is 0mq able to work with threadless evented systems like ruby EventMachine and Node.js?
[06:05] sustrik postmodern: no restriction
[06:05] postmodern excellent
[06:05] sustrik hassox: no idea, depends of what kind of requirements the two systems have
[06:06] hassox sustrik_: they're not threaded
[06:06] sustrik single-thread?
[06:06] hassox event loop
[06:06] hassox yeah single thread, but they live in an event loop
[06:06] hassox so async style
[06:06] sustrik what do you have to conform to register to the event loop?
[06:07] hassox sorry my train has just arrived I'll be back on later
[06:07] hassox not sure what that means but I'll jump on when I get back
[09:30] hassox sustrik_: still here mate?
[09:30] sustrik yup
[09:31] sustrik how do you add event sources to your event loop?
[09:31] sustrik what does the event source have to conform to?
[09:49] hassox so
[09:49] hassox with event machine for eg
[09:49] hassox you create a socket, and when an event occurs, i.e. some data is received
[09:49] hassox a method on the handler is called
[09:50] hassox you can then add callbacks, timers etc
[09:50] hassox inside the method
[09:50] hassox there's no threads, you instead gain concurrency by splitting execution up and using callbacks
[09:51] hassox there is an event queue, where events are queued up, and then the event loop consumes each event from the queue
[09:51] mikko can you add custom event sources?
[09:51] mikko other than fds
[09:51] hassox what do you mean
[09:51] hassox manually put things onto the event queue?
[09:52] sustrik no, you put a socket into the event loop, right? can you put something else there?
[09:52] hassox the whole application lives in the event loop
[09:53] hassox so you can have many sockets, tcp connections database connections, or just have your application setting things up to do and executing them
[09:53] sustrik how does the loop know what sockets to inspect?
[09:54] hassox I'm not sure of the underlying system calls that are made...
[09:54] hassox I usually live a little higher up the stack
[09:54] sustrik you create a connection to database
[09:54] mikko like for example libev's case
[09:54] sustrik how do you integrate it with the loop?
[09:55] hassox lemme see if I can find someone who will be able to talk at the right level
[09:56] hassox I just grab a socket
[09:57] hassox but not sure underneath at the c level what it's actually doing
[09:57] hassox they don't look like they're about
[09:58] sustrik example:
[09:58] sustrik s = create_socket (...);
[09:58] sustrik loop.register (s);
[09:59] sustrik does you use it in a similar way?
[09:59] hassox
[09:59] hassox there's an example
[10:00] hassox here's the underlying class
[10:00] hassox
[10:02] hassox all the c is in
[10:02] hassox i'll brb
[10:02] sustrik i don't speak ruby, sorry, can you point me to the line where you open the socket?
[10:32] hassox sustrik_: an http connection
[10:33] hassox not sure what c is going on underneath
[10:36] sustrik hassox: forget about c
[10:36] sustrik what are you doing in ruby?
[10:36] hassox I use the higer level stuff
[10:37] sustrik and that's?
[10:37] hassox I just fire up an http server, or smtp server
[10:37] hassox or both
[10:37] sustrik and?
[10:37] hassox I'm trying to find someone who can talk at the right level
[10:37] hassox well
[10:37] hassox when ythe connection receives data
[10:37] hassox EM calls receive_data(data)
[10:37] hassox so you react to the event
[10:38] hassox I'm not describing this well :(
[10:38] sustrik that's inside HTTP server?
[10:38] hassox maybe the concept will be easier to talk over node
[10:38] mikko sustrik_: its probably like libevent http
[10:38] mikko sustrik_: similar concept to
[10:38] mikko unless i am wrong
[10:38] mikko underneath it uses normal event loop
[10:39] hassox the docs are better for node
[10:39] mikko which usually polls a socket
[10:39] hassox
[10:39] mikko node.js is a lot more complicated
[10:39] hassox oh
[10:39] hassox :'(
[10:39] hassox damn I suck at talking about this
[10:39] hassox I'm sorry lads
[10:39] hassox I'm not familiar with territory this low down
[10:40] sustrik np, can you check the mikko's link?
[10:40] sustrik are you doing something similar?
[10:41] hassox I don't know C but the concept seems about right
[10:42] sustrik so you launch a http server in your process
[10:42] mikko it's a higher level abstraction where you talk 'http concepts' rather than raw sockets
[10:42] sustrik and supply it a callback function
[10:42] mikko but in the end its just an event loop
[10:42] hassox yeah
[10:42] hassox EventMachine and Node.js are both event loops at their hearts
[10:43] hassox but all the 0mq stuff I've seen talks about requiring 2 or more threads
[10:43] sustrik hassox: ok, then you need to check your documentation whether there are extension points in the library you are using
[10:43] sustrik i.e. ways to plug-in non-HTTP entities to the even loop
[10:44] mikko hassox:
[10:44] mikko sometghing like that might be relevant to you
[10:44] hassox yeah that's just doing http servers
[10:45] mikko hassox: why don't you use 0MQ's event loop?
[10:45] hassox what I would like to know is can I get a 0mq service running, or is it a must to use threads
[10:45] hassox there is one?
[10:45] mikko or do you need the higher level abstraction?
[10:45] hassox I'd like to be able to use 0mq from my ruby or node programs
[10:47] hassox wondering if it's possible, or if I _need_ the threads
[10:47] mikko ideally your event loop such as event machine would support "polling" a callback for events
[10:47] mikko i would imagine it would be possible to create a struct with necessary io abstraction that would work with any data provider including zeromq
[10:48] mikko whether event machine does that i dont know
[10:49] mikko sustrik_: isn't XREQ / XREP non-blocking?
[10:50] mikko (sorry if im confusing the issue more)
[10:50] hassox np
[10:50] hassox I'm just in really unfamiliar territory :(
[10:53] sustrik mikko: any 0mq socket can be either blocking or non-blocking
[10:53] sustrik s.recv (&msg);
[10:53] sustrik vs.
[10:53] sustrik s.recv (&msg, ZMQ_NOBLOCK);
[10:53] mikko so the end result would be: you want to embed zmq into node.js/eventmachine loop and receive events and have the events fire the callbacks
[10:53] hassox that sounds like what I want
[10:53] mikko this functionality depends on whether your event loop allows you to register custom resources with the loop
[10:53] sustrik the XREP/XREQ allows you to have multiple request in the air at the same time
[10:54] mikko as 0MQ does not expose the underlying sockets directly
[10:55] hassox mikko: but you connect to the 0MQ service via a socket yeah?
[10:55] mikko the socket 0MQ gives you is an abstraction
[10:55] mikko not the raw underlying posix socket
[10:56] mikko
[10:56] mikko as a very elementary pseudo-code
[10:56] mikko i would imagine soemthing like that would be needed from the event loops side
[10:56] mikko but i'm not 100% sure
[10:56] mikko naturally you would need more abstraction but that should show the principle
[10:57] hassox kk
[11:00] hassox thanx lads
[11:00] hassox I'll talk to a guy I know who knows this stuff much better
[11:00] hassox it looks like it could work though :)
[11:02] mikko what kind of messaging are you doing?
[11:02] mikko are you trying to write a zmq server?
[11:02] mikko or use 0mq within your existing server?
[11:03] hassox I'd like to use 0mq to talk between a bunch of http servers and supporting scripts
[11:03] mikko
[11:03] hassox atm I'm setting things up to use firewall settings and api keys for authentication, but a 0mq setup would make like a _lot_ better
[11:04] mikko there is an example of php polling server
[11:45] cremes hassox: i am working on a "zmq-machine" in ruby; i should have something worthy of a push to github in another week or so
[11:45] cremes but for now, EM and 0mq can't interact
[11:45] hassox cremes: :D and :(
[11:45] cremes EM uses POSIX sockets while 0mq is an abstraction on top of POSIX sockets
[11:46] cremes e.g. you couldn't have an EM handler do a "connect" to a 0mq REQ socket
[11:46] hassox can you do it in normal ruby?
[11:46] cremes if you use the rbzmq or ffi-rzmq gems, then yes
[11:46] cremes you need something that "speaks" 0mq
[11:47] hassox right
[11:47] hassox does the zmq-machine use threads or does it assimilate into the event loop?
[11:48] cremes hassox: good question...
[11:48] cremes it will have a similar setup to EM
[11:48] cremes in EM we have the {} configuration
[11:49] cremes in zmq-machine it will be almost the same; something like {}
[11:49] cremes each Context will be its own thread; any sockets created within that context will all be non-blocking and will fire callbacks for events
[11:49] cremes *but* you can create multiple contexts each with its own thread
[11:49] cremes think of it as firing up a separate reactor loop for each thread
[11:50] cremes if EM allowed more than one reactor, we could already do this
[11:50] cremes make sense?
[11:53] hassox kinda
[11:53] hassox so 0mq won't be available as a callback inside the em
[11:53] hassox but rather it's a parallell event loop in the same process
[11:56] cremes hassox: yes, it is completely independent of EM; they can't work together
[11:57] hassox :(