Tuesday December 21, 2010

[Time] NameMessage
[01:25] falconair hi all, a few newbie questions: can zeromq api be used to do raw tcp/ip programming?
[02:21] xpromache1 hello guys, I'm considering zmq for replacing corba in an application
[02:21] xpromache1 the typical rpc stuff I think I know how to implement
[02:22] xpromache1 but at some point a client can send a request to the server that will have to send some large amounts of data back to the clinet
[02:22] xpromache1 how is this best done?
[02:22] xpromache1 I guess I need to answer the request with "OK, here it comes", and then to open another socket for pushing the data?
[02:22] xpromache1 what sort of socket?
[04:53] cremes falconair: no
[04:53] cremes it's a higher level api than that with a few abstractions
[04:54] cremes make sure to read the guide linked off of the main page
[04:54] cremes xpromache1: open a socket for each kind of messaging pattern you plan to use
[04:54] cremes if you are waiting for a request/reply to start a large data download, use the REQ/REP sockets for that purpose
[04:55] cremes for the big data download, consider PUB/SUB or PUSH/PULL depending on what you need
[04:55] cremes you may even want to look at PAIR; it really depends on your app requirement
[04:56] cremes make sure to read the guide linked off of the main page; it may help you pick the right pattern and its associated sockets
[04:56] xpromache1 ok, thanks for the answer
[04:56] xpromache1 is there a way to use all these over a limited number of ports
[04:57] xpromache1 like one for req/rep
[04:57] xpromache1 and another one for the data tansfer
[05:00] cremes absolutely; each socket should get its own transport (tcp/pgm/ipc/inproc) and port (not necessary for transports other than tcp)
[05:01] cremes 0mq doesn't use "random" ports; you specifically connect or bind to whatever you need
[05:02] xpromache1 yes but I want to use the same port for all clients
[05:02] xpromache1 or maybe this doesn't make sense
[05:02] xpromache1 I mean it only makes sense when there is some firewalls in between
[05:03] xpromache1 but this is not the use case where zeromq should be used, right?
[05:03] cremes xpromache1: not really sure what you are trying to d
[05:04] cremes if you have a complex use-case, you may consider posting it to the ML for comment
[05:04] xpromache1 not really complex
[05:05] xpromache1 it's like a database driver
[05:05] xpromache1 you execute some small queries from time to time
[05:05] xpromache1 but ocasionally you want to retrieve a lot of data
[05:05] xpromache1 and I prefer not to have a request for each row
[05:07] cremes then you would probably use the pattern i suggested above; REQ/REP for the query, PUB/SUB for the data transmission
[05:08] cremes but if thisis over tcp,then the reqp/rep would get its own port as would the pub/sub
[05:08] cremes they cannot share the same port simultaneously
[05:08] xpromache1 so if I have 10 clients connected simultaneously, how many ports do I need?
[05:09] cremes 2, one for the req/rep and 1 for the pub/sub
[05:10] xpromache1 but then all the clients will get all the data even from the requests of the other clients?
[05:10] cremes they could ignore it if the published topic is not something they are interested in; altnerately, use the XREQ/XREP socket types
[05:10] cremes have you read the guide yet?
[05:11] xpromache1 I did partly but this is not clear to me
[05:11] xpromache1 exactly what you can multiplex on a server socket
[05:12] cremes i may not be able to clarify it completely for you :)
[05:12] cremes a 0mq socket is a smart socket; it can connect/bind to multiple endpoints simultaneously
[05:13] cremes so in your situation the server would likely bind a REP or XREP socket
[05:13] xpromache1 yes, that's clear for the response reply pattern
[05:13] cremes the clients would come in with REQ (or XREQ) sockets via "connect" call; 0mq handles the multiplexing for you
[05:13] xpromache1 what is not clear is for the data transfer
[05:14] cremes well, i suggested pub/sub but maybe that's not appropriate
[05:14] xpromache1 so a client connects and sends a request for the data
[05:14] cremes you could use the xreq/xrep sockets for the data transfer
[05:14] cremes go on
[05:14] xpromache1 I can send a reply saying connect here to get the data
[05:14] cremes sure
[05:14] xpromache1 but what does it mean here
[05:15] cremes "here" where? explain.
[05:15] xpromache1 that's my question
[05:15] xpromache1 what do I answer to the client
[05:15] xpromache1 where to connect to get its data
[05:15] xpromache1 do I have to start a new thread, create a new socket on a random port, and start pushing data once a client is connected
[05:16] xpromache1 or there is a smarter way
[05:16] cremes you could send in your reply something like: :transport => :tcp, :address => '', :port => 1800
[05:16] cremes then the client would know where to go for getting the data
[05:16] cremes you can't do random ports without informing the client what it will be
[05:17] xpromache1 yes, I realize that
[05:17] xpromache1 I was hoping that it is somehow possible to do without random ports
[05:17] cremes you *could* use another thread or use the poll api to do it all within one
[05:17] xpromache1 just one fixed port
[05:17] xpromache1 like I have port x for request/reply
[05:17] xpromache1 and port y for data transfer
[05:17] cremes sure, you could use 1 fixed port but you could run into issues if you have lots of concurrent requests/replies
[05:18] cremes you would probably want to use xreq/xrep sockets so your data transfer could target the right client
[05:18] cremes take a look at the "advanced routing" section of the guide; i'm pretty sure they wrote about that
[05:19] xpromache1 but xreq/xrep needs one message in both ways
[05:19] cremes not true
[05:19] xpromache1 I would need something like xpush xpull
[05:19] cremes not true
[05:19] cremes reqp/rep enforces a strict request/reply/request/reply pattern
[05:19] cremes the xreq/xrep sockets do *not* enforce that
[05:20] cremes so, using xreq/xrep, you could send a single request and get a multiple message reply
[05:21] xpromache1 ok, I didn't know that
[05:22] xpromache1 I will look further into it, thanks for the hints
[05:22] cremes no prob...
[05:22] cremes time for bed!
[05:26] xpromache1 me too, I have to go to work in 2 hours :(
[06:15] bkad anyone know how to get the zmq_server binary? cant find it in the installation instructions
[06:22] guido_g what server?
[06:23] bkad zmq_server?
[06:23] bkad there's lots of references to it in the zmq docs
[06:24] bkad but the ./configure, make, make install didn't build that binary
[06:24] guido_g the docs are a bit outdated i fear
[06:25] bkad in
[06:25] bkad under Running It
[06:25] guido_g there is a devices directory w/ some executables
[06:25] guido_g ouch
[06:25] guido_g this example is way outdated
[06:25] sustrik see the top of the page
[06:25] sustrik WARNING: This text is deprecated and refers to an old version of ØMQ. It remains here for historical interest. DO NOT USE THIS TO LEARN ØMQ.
[06:25] guido_g it'S even pre zeromq2 afair
[06:26] bkad ahhh, oops
[08:34] sustrik mikko: there's a build patch on the mailing list
[08:34] sustrik can you give it a look?
[08:35] sustrik mato is on his way to new zealand for next few days
[08:56] mikko sustrik: sure
[08:56] mikko it looks like my flight is departing today
[08:56] mikko causiously excited
[08:57] sustrik good luck :)
[09:28] mikko reviewed
[09:29] sustrik thanks
[09:56] mikko Steve-o: hi
[09:57] mikko what does the scons build produce?
[09:57] mikko Makefiles?
[09:57] Steve-o No it runs commands in Python directly
[09:58] Steve-o CMake is the Make wrapper
[09:58] mikko ok
[09:58] Steve-o you are supposed to be able to see the commands by "scons -Q" but I haven't seen that working on my Scons files in a long time
[09:59] mikko would've been nice if it produced something that could be directly used
[09:59] Steve-o well, all the other build systems kinda suck though :P
[10:00] Steve-o Although I think I had to patch Scons again for Intel C on Linux, urgh
[10:00] Steve-o CMake just doesn't support anything interesting
[10:02] mikko interesting development, my airline sends twitter DMs to inform about flights
[10:04] Steve-o Cathay likes to send emails after the flight has left
[10:04] mikko im wondering if we should split OS specific builds to separate automake files in zmq
[10:04] mikko it might be more maintainable than conditionals in one file
[10:04] Steve-o Makes patching a bit cleaner
[10:06] Steve-o unless you want to write several new autoconf rules
[10:06] mikko yeah
[10:08] Steve-o only ~24 tests required :P
[10:09] Steve-o many should already have autoconf macros available
[10:11] mikko it feels like duplicating work
[10:11] mikko hmm
[10:11] mikko would be nicer to launch scons from zmq build
[10:11] mikko and then just link
[10:12] Steve-o I've looked for a long time for building Debian packages
[10:13] Steve-o there isn't much out there despite all the big packages using SCons (Blender, Quake 3, etc)
[10:14] Steve-o as in, for nice hooks for the deb-buildpackager
[10:14] Steve-o you can always build a basic scons command based off uname otherwise
[10:15] mikko autoconf might be slightly dreadful but its fairly well supported
[10:19] Steve-o I don't think you can easily get multiple build environments in autoconf/automake though
[10:19] Steve-o Scons is killing me on crap compiler dependencies for unit tests
[10:29] Steve-o autoconf is a bit too cumbersome for testing different compilers on one platform
[10:29] Steve-o I'm sure there has to be a better way \:D/
[10:30] mikko it is slightly cumbersome yes
[14:15] ptrb correct me if I'm wrong, but if I'm sitting in a blocking recv() on a socket in one thread, and I zmq_term() the context in another thread, the recv() should return (in C++, should throw) -- right?
[14:22] sustrik ptrb: right
[14:22] sustrik ETERM
[14:26] ptrb hmm... the zmq_term() goes through OK, but I'm still blocking in the other thread... time for a minimal test case I guess :\
[14:28] sustrik yes, please
[15:48] mikko sustrik: zmq_term(ZMQ_NOBLOCK) would be useful
[15:48] mikko or another function call
[15:51] cremes mikko: something similar has been discussed on the ML; sustrik says it isn't possible
[15:51] sustrik you mean, instead setting all the sockets to ZMQ_LINGER=0
[15:51] sustrik right?
[15:54] mikko sustrik: no
[15:54] mikko sustrik: i mean instead of manually closing all sockets
[15:54] mikko because currently it's not possible to force closing of sockets
[15:55] sustrik hm, how would you migrate all the sockets to the thread calling zmq_term()?
[15:55] mikko do they need to be migrated to obtain information whether zmq_term would block?
[15:56] mikko i meant that zmq_term() would return EGAIN if given a flag
[15:56] mikko EAGAIN*
[15:56] sustrik how would the main thread know that they are not used in the moment otherwise?
[15:56] sustrik at the moment*
[15:57] sustrik you cannot grab a socket from underneath the feer of another thread
[15:58] mikko sockets is removed from sockets array in context when it's closed?
[15:59] sustrik yes
[15:59] mikko slightly laggy connection at the moment
[15:59] mikko sockets.size() > 0 would block, yes?
[15:59] sustrik yes
[16:00] mikko sozmq_term_nb woudl return EAGAIN if sockets.size() > 0
[16:00] sustrik aha
[16:00] mikko no need to migrate anything (?)
[16:00] sustrik i though you want zmq_term to close the sockets
[16:01] mikko im after info whether call to zmq_term will block
[16:01] mikko not to close sockets
[16:01] sustrik yes, that should be doable
[16:02] sustrik if you have a look atc ctx.cpp
[16:02] sustrik ::terminate()
[16:02] mikko around line 118?
[16:02] sustrik the function begins at 107
[16:03] sustrik you have 2 waits there
[16:03] sustrik line 130:
[16:03] sustrik no_sockets_sync.wait ();
[16:03] sustrik and line 152:
[16:03] sustrik usleep (1000);
[16:04] sustrik the first wait waits for all sockets being closed (zmq_close)
[16:04] sustrik the second one wait for all pending data to be sent
[16:04] mikko ok
[16:08] cremes i like this EAGAIN idea for zmq_term()
[16:27] Skaag Hello!
[16:27] Skaag (again)
[16:27] mikko i
[16:27] mikko hi
[16:27] Skaag hi mikko :)
[16:28] Skaag we're about to perform a POC for ZMQ, and if all goes well, convert a system to ZMQ that uses RabbitMQ at the moment.
[16:29] Skaag but I wanted to ask, in general, about the theoretical potential of ZMQ to scale to thousands of servers...
[16:29] mikko Skaag: can you describe your specific use-case?
[16:29] mikko it would give some context
[16:29] Skaag And also how practical it is to use the multicast functionality - does it require special network equipment? is it IP Multicast? Or is it just a name really, and the multicast is just synonimous for sending the message at once to many hosts?
[16:29] Skaag sure!
[16:30] Skaag I have at the moment only 16 nodes, distributed around the world
[16:30] Skaag they send once per second a small block of data, statistical data about themselves
[16:30] Skaag less than 512 bytes
[16:31] Skaag so it's "all to all", and all of them need to collect this data, and produce historical graphs, etc.
[16:31] Skaag so i'm asking myself what will happen when this scales, to 1000 nodes
[16:32] mikko so every node needs to know about all others?
[16:35] cremes sounds like a perfect use-case for pub/sub with a forwarder device
[16:35] cremes Skaag: need more info; do you have particular latency requirements? how frequent is the data generated?
[16:36] mikko yeah, if latency is not a problem you could batch updates
[16:36] Skaag every 1 second, the nodes send to all other nodes a less than 512 byte data structure (json or binary, I don't mind)
[16:36] Skaag could also be every 5 seconds. still fine.
[16:36] Skaag but while the cluster is small, it would be cool to have fast updates.
[16:36] mikko sure
[16:37] mikko you could have quick local updates
[16:37] Skaag the latency between the nodes is a maximum of 100ms.
[16:37] mikko and batch the packets to remote locations
[16:37] Skaag there are at the moment 6 locations around the world
[16:37] Skaag with a bunch of machines in each location
[16:37] mikko update every second locally and send out every 5 seconds
[16:37] mikko so every five seconds you would send five updates to remote locations
[16:37] cremes multicast/pgm transport won't help you if this is over a public network (internet)
[16:38] Skaag ok so I was right to assume it is IP Multicast...
[16:38] Skaag and not some internal terminology
[16:38] mikko i would probably look into something like zookeeper for service discovery
[16:38] Skaag sounds interesting!
[16:38] mikko create ephimeral nodes of available endpoints
[16:38] mikko i used zookeeper ages ago for services discovery
[16:39] mikko wrote a small daemon called 'myservices' and each node on cluster had a configuration file of services it has
[16:39] mikko so as long as node was up the ephimeral node existed in zookeeper
[16:39] mikko when a node died the ephimeral node is automatically removed by zookeeper
[16:40] mikko allows 'automatic' graceful failure handling
[16:40] Skaag that's absolutely awesome
[16:41] mikko sadly the project never went live so i cant really tell how well it would've worked in production
[16:44] mikko im gonna go get some coffee
[16:44] mikko bbl
[16:45] ngerakines is there a list of large companies that use zmq on the site?
[16:52] Skaag mikko: Would love to take a look at your 'myservices' code ;)
[16:54] cremes ngerakines: i don't think so
[16:54] ngerakines cremes: ok, thanks
[16:58] sustrik Skaag: one thing to understand is that "all to all" model is inherently unscalable
[16:59] Skaag I thought about creating 'bubbles'
[16:59] Skaag so machines in the same DC would update quickly, internally
[16:59] sustrik at some point the number of nodes would be so high, that amount of messages will overload any receiver
[16:59] Skaag with less frequent updates being sent between such bubbles, aggregated
[16:59] sustrik yes, aggregation solves the problem
[17:00] Skaag we should probably do that from the start
[17:00] sustrik depends on the requirements
[17:00] sustrik it the goal is to have 1000 nodes
[17:00] sustrik with 1 msg per 5 secs
[17:00] sustrik you probably don't need it
[17:13] Skaag yes that's more or less the goal.
[17:13] Skaag it will take some time to get there, too...
[17:14] Skaag for now we have just 25 machines.
[17:14] Skaag I expect in 6 month it will maybe grow to 150.
[17:14] Skaag so there is time to ameliorate the system.
[17:15] Skaag uh, to improve the system.
[17:15] Skaag (I think maybe I used a french word?)
[17:25] mikko Skaag: what is this system doing?
[17:25] mikko what kinda of data are you pushing through?
[17:25] mikko just out of interest
[17:28] Skaag I can tell you exactly..!
[17:28] Skaag it is a bunch of media streamers
[17:28] Skaag the important information is cpu load, bandwidth usage, and number of connections
[17:28] Skaag and in the future, cost per traffic (actual cost, in dollars)
[17:29] Skaag or some combination of cost, and remaining quota
[17:29] Skaag but that's "phase 2" stuff :)
[17:29] Skaag and this will be used to make smart decisions about where to send new traffic
[17:29] Skaag and indirectly, will also be used to generate graphs for me and other admins, to view what's going on in the network
[17:31] mikko how do you do the routing on global level?
[18:05] Skaag Level3 fibers, connect all the DC's
[18:06] Skaag you can see our map here:
[18:07] Skaag right now, I am near :-)
[18:07] Skaag (physically speaking)
[18:11] Skaag and this means that this year, I will celebrate xmas in moscow \o/ yey!