ZeroMq IRC Log

Thursday July 29, 2010

[Time] Name	Message
[05:29] dos000	howdy
[05:30] dos000	how would you do an http protocol listener in zmq ?
[05:37] dos000	or how do you listen for (normal) sockets asynchronously in zmq ? is the library godd for that level of (low level) cross platform programming ?
[05:46] bgranger	What language are you working in?
[05:46] dos000	i can shoose between java and c/c++
[05:46] dos000	but i would like to compare the java version against the c/c++ one
[05:46] bgranger	OK
[05:47] dos000	i need something that listens for incoming http connections
[05:47] bgranger	The general idea is that you would use zmq_poll which works for both zmq sockets and native file descriptor based sockets. Then you can build an event loop that does both http and zmq
[05:47] bgranger	But there will be a lot of work to do I think.
[05:48] dos000	i need something that works based on callbacks like the java nio
[05:48] bgranger	The Python bindings have an eventloop that already integrates zmq with the Tornado web server.
[05:49] bgranger	But I don't work in Java, so I haven't followed that as much.
[05:50] dos000	ok .. i just read the zmq_poll which looks cool !
[05:50] dos000	now one thread would be enough to do the zmq_poll ?
[05:50] bgranger	Yes
[05:50] bgranger	One application thread that is
[05:51] dos000	if you have say 10000 (10k problem) connections coming in ?
[05:51] bgranger	Yes
[05:51] dos000	nice !
[05:51] bgranger	If your eventloop implementation is good enough.
[05:51] dos000	and zmq_poll is cross platform ? i only care about linux and solaris
[05:51] bgranger	I shouldn't promise that, but zmq_poll is just going epoll or kqueue undermeath
[05:52] bgranger	Linux for sure, I am not sure about solaris
[05:52] bgranger	If zmq runs on solaris (which it should), then zmq_poll should work.
[05:53] bgranger	zmq_poll might add a small amount of overhead to epoll/kqueue though, and that might limit the number of connections you can handle. But, it is fast C++, so you may be OK
[05:53] dos000	where is the overhead coming from ?
[05:54] dos000	because i dont want to do multiplexing or message checking at that level
[05:54] bgranger	In the fact that zmq_poll handles both native file descriptors and ZMQ sockets and has to do extra logic to manage all of that.
[05:54] dos000	ah!
[05:55] bgranger	So you want a single server that does both HTTP and OMQ sockets?
[05:55] dos000	so the idea i have is to get something zmq_poll based listening for incoming connections ..
[05:56] dos000	bgranger, nop . i want something that listens for normall sockets in async way. but i dont want to use another lib if zmq already does it.
[05:56] bgranger	Yes, you would pass the TCP socket that is bound and listening (in non-blocking mode) to the zmq_poll and it would tell you when the socket is ready for reading/writing/error
[05:56] bgranger	How are you wanting to use 0MQ sockets though?
[05:57] dos000	after connections arrives on the listener they are sent to a pub/sub and to handle messages
[05:57] dos000	sorry ...
[05:57] bgranger	OK, so traffic comes in over HTTP and then is sent out for processing using OMQ messages?
[05:57] bgranger	So I think that oMQ should run on solaris...
[05:57] dos000	exactly !
[05:58] bgranger	Yes, then you will register both TCP socket and OMQ sockets with zmq_poll and it will tel you which TCP socket are ready for send/recv of bytes and which 0MQ sockets are ready for send/recv of messsages
[05:59] bgranger	But you will have to write the eventloop and HTTP handler yourself.
[05:59] dos000	wait .. i dont want to mix both sockets on the incoming traffic
[06:00] dos000	i want to only poll for nor mal sockets ..
[06:00] bgranger	Why not 0MQ sockets?
[06:00] dos000	bgranger, i did that in java already
[06:01] bgranger	I am not quite following you on this part of it.
[06:01] dos000	http-traffic -> listener send to zmq socket
[06:01] bgranger	Also, this might be of interest to you:
[06:01] bgranger	http://mongrel2.org/home
[06:01] bgranger	What type of ZMQ socket?
[06:01] dos000	i briefly looked at that ..
[06:02] dos000	on the right hand i have pub/subs so i can have many resources responding to http traffic
[06:02] dos000	bgranger, are you behing mongrel2 ?
[06:02] dos000	oops s/behing/behind ?
[06:03] bgranger	No, I develop the Python bindings to 0MQ (pyzmq) which are used in mongrel2
[06:03] bgranger	Zed Shaw is behind it
[06:03] dos000	but i am wondering why i would want to mix different traffic on the same socket .. there is so many ports available !
[06:04] sjampoo	dos000, firewall issues is one. But another reason is that for example FLASH (which can be used as a websocket fallback) needs a policy server which listens on port 80.
[06:05] dos000	i am a speed freak .. so even the extra processing required to differentiate between the two should not be occured .. if there is no need to
[06:05] bgranger	I think we are missing, I am not saying you would mix traffic on a single socket.
[06:05] sjampoo	So if you want to have the fallback and a regular server you will need 2 different IP adresses.
[06:05] bgranger	You would have a TCP socket that would handle HTTP requests and replies
[06:05] dos000	ok .. ok i get the firewall point
[06:05] bgranger	And then a separate 0MQ socket (or multiple of them) that would handle the talking to the other parts of your system.
[06:06] dos000	yes exactly
[06:06] bgranger	All of those sockets will be passed to zmq_poll
[06:06] dos000	http-traffic -> listener -> subscribers to zmq messages
[06:07] dos000	but i want to poll for the http traffic only on the left . the rest i am (candidly) suuming i dont have to poll for it
[06:08] dos000	s/suuming/assuming
[06:08] bgranger	But if you want your system to handle lots of connections, you should also poll for the OMQ sockets as well (both send/recv)
[06:09] bgranger	Otherwise, your 0MQ send/recv calls will block.
[06:09] dos000	ah!
[06:09] dos000	i did not notice that !
[06:09] dos000	mmmmmm
[06:10] dos000	maybe i want that afterall
[06:10] dos000	or maybe not. i mean the blocking calls on the right
[06:10] bgranger	You definitely don't want that
[06:11] dos000	ah!
[06:11] bgranger	Also, I am not sure you can use the blocking 0MQ calls and zmq_poll at the same time.
[06:11] bgranger	In general, it is usually one or the other
[06:11] dos000	the listener part will get very complicated with all the states machines it has to track
[06:12] bgranger	Yes, definitely, that is why I keep saying you will have to write your own eventloop and that part is not trivial
[06:12] dos000	bgranger, ok .. good to know ... where is the definite answer to that question ?
[06:12] bgranger	I would email the list and ask the core 0MQ devs
[06:13] dos000	i will
[06:13] dos000	thanks for your kind help !
[06:13] bgranger	No problem
[06:14] dos000	one more question ... i wanted to put some counters for 0mq messages. like how many incoming/ outgoing if i can do all of that using the zmq_poll then this would be cool
[06:15] bgranger	You just want to count incoming/outgoing messages?
[06:15] dos000	i wanted to track the state of each message and put counters
[06:15] dos000	per messages .. plus errors/timouts ..
[06:15] bgranger	Yes, I think zmq_poll can do that
[06:16] dos000	this is quite nice
[06:59] zedas	dos000_: mongrel2 does that. i'm behind it.
[07:00] zedas	dos000_: there's no processing overhead in handling multiple protocols on the same port. they're just different parts of the same parser, so the math makes them the same.
[07:00] zedas	dos000_: if you wrote it by hand then it'd be different, but with mongrel2 it's not any slower.
[07:01] zedas	dos000_: otherwise, all the stuff you want to do is already done in mongrel2.
[07:01] zedas	at least look at how it's done to see how to solve lots of the questions you have, even if you don't use it.
[07:25] dos000	sure !
[07:25] dos000	i promess i will
[07:57] dos000	can i ask which ide you guys use ? i am coming from eclipse after a long absence in c/c++
[08:07] dos000	zedas, what do you use to compile the state_machine files ?
[08:07] dos000	i checked the makefile but it is not there unless i missed it
[08:11] dos000	got it !
[08:12] dos000	i never heard of ragel until now
[17:18] Nwallins	how would zeromq overlap or integrate with something like BEEP (http://www.ibm.com/developerworks/webservices/library/x-beep/) ?
[18:38] dos000	anyone did perf comparisons between solaris and linux or even *bsd yet ? i want to see the difference between epoll and devpoll
[18:42] dos000	i found some papers on the net. if you can share i would appreciate
[18:47] dos000	bgranger, hey
[18:53] bgranger	dos000_: hi
[18:56] dos000	have you seen anyone doing comparisons between the solaris or linux implementations of zmq from performance prespective ?
[18:56] dos000	i am searching now the epoll and delpoll comparisions to get a better idea
[18:59] dos000	the first question is what would you use to drive the test . if there are tools out there to create 10000 simultaneous sockets then i dont have to write them
[19:17] bgranger	I am sure there are HTTP testing programs, but I am not sure what they are...
[19:18] bgranger	http://www.hpl.hp.com/research/linux/httperf/
[19:18] bgranger	I know of people who have used this...
[19:19] dos000	i found this so far .. which i can use http://libev.schmorp.de/bench.html
[19:20] dos000	this will be nice if i can use the client against zmq
[19:20] bgranger	I think httpperf will only test HTTP TCP sockets.
[19:20] bgranger	But those could be managed by a zmq_poll loop
[19:21] dos000	but the above client can test any protocol i think
[19:27] Nwallins	dos000_: I wonder what was used for http://www.zeromq.org/results:ib-tests-v206
[19:29] dos000	Nwallins, i assume the perf stuff under zmq was used
[19:30] Nwallins	dos000_: can it be used for your purpose?
[19:30] dos000	those tests are not connections oreinted they are message and payload oriented
[19:30] dos000	i need something that tests 10k simultaneous connections
[19:30] dos000	witha variable number of them active
[19:33] dos000	these guys also cover timeouts which is very important from a client perspective
[19:35] dos000	this is the latest thread on this http://lists.schmorp.de/pipermail/libev/2010q2/001044.html