ZeroMq IRC Log

Tuesday August 3, 2010

[Time] Name	Message
[12:47] cremes	interesting poll analysis from zed; may influence 0mq polling architecture
[12:47] cremes	http://sheddingbikes.com/posts/1280829388.html
[13:10] sustrik	yes, seen it
[13:11] sustrik	if anyone wants to experiment with different polling algorithms 0mq offers a compile time option to enforce usage of a particular one
[13:11] sustrik	ZMQ_FORCE_POLL
[13:11] sustrik	ZMQ_FORCE_EPOLL
[13:11] sustrik	etc.
[13:13] CIA-19	jzmq: 03Gonzalo Diethelm 07master * r362370e 10/ src/org/zeromq/ZMQ.java :
[13:13] CIA-19	jzmq: Added in Socket a reference to Context, so that Java's GC
[13:13] CIA-19	jzmq: won't deem the Context as unused because there are no references
[13:13] CIA-19	jzmq: to it. - http://bit.ly/d2uhuU
[14:46] cremes	any reason zed's suggestion for dynamically shuttling FDs between poll & epoll couldn't be wrapped up in 0mq?
[15:15] sustrik	cremes: no technical reason
[15:15] sustrik	however, i am not sure it would help much
[15:16] sustrik	what 0mq does in its I/O thread is opportunistic polling
[15:16] sustrik	if it has no data to send
[15:16] sustrik	it never polls the socket for OUT
[15:16] cremes	it's probably wise to let zed experiment with the idea (and implementation) a bit before changing the library :)
[15:16] sustrik	if it does have data to send it polls the socket once
[15:16] sustrik	then writes the data
[15:17] cremes	sustrik: that seems smart
[15:17] sustrik	if the write succeeds it assumes the socket is still available for writing
[15:17] sustrik	so it does not poll again
[15:17] sustrik	until send fails
[15:17] sustrik	same thing on the recv side
[15:18] sustrik	this kind of algorithm would heavily distort any perf results we have
[15:18] sustrik	however, if you want to go experimenting with different poll types
[15:18] sustrik	there's the compile time setting (ZMQ_FORCE_EPOLL etc.)
[15:18] sustrik	so you may do some benchmarking yourself
[15:19] cremes	sure
[15:19] cremes	assuming zed's results are trustworthy, i do think it could help 0mq
[15:20] cremes	let's assume that there are half a dozen 0mq sockets fronting thousands of tcp sockets
[15:20] sustrik	yes?
[15:20] cremes	further, assume they are XREQ/XREP pairs
[15:21] cremes	as the XREQ sends data round-robin to the XREP sockets, the majority of those sockets are idle
[15:22] cremes	hmmm... as i'm writing this out i see your point...
[15:22] sustrik	:)
[15:22] cremes	you wouldn't poll any of the other sockets for writability until their "turn" came up in the round-robin scheme
[15:22] cremes	never mind!
[15:22] cremes	;)
[15:22] sustrik	right
[15:24] cremes	so if zed is building mongrel2 on top of 0mq, i'm wondering if he needs to split poll/epoll like he wrote in his latest posting
[15:24] sjampoo	for the other side (http)
[15:25] cremes	sjampoo: ah... yes, that makes sense; it would be necessary for the non-0mq sockets
[15:25] cremes	but then why not make "superpoll" into "opportunisticpoll" and use the same concept as 0mq for polling regular FDs
[15:26] cremes	perhaps that would result in too many syscalls?
[15:27] sustrik	sjampoo: as far as i understand the superpoll thing is intended for HTTP sockets
[15:27] sustrik	not the 0mq sockets
[15:38] sjampoo	sustrik: yah i believe so too ;)
[15:41] sjampoo	cremes, I am not sure if he does or doesn't do that. Maybe it has to with how he uses/wants to use libtask, which currently uses poll.
[15:43] sjampoo	It could be that such an opportunistic poll is difficult to implement in his FSM approach, but then again it could be that i am just talking out of my ass.
[15:44] cremes	heh
[15:44] cremes	perhaps zedas will enlighten us in channel
[16:02] Utopiah	hi #zeromq
[16:04] Utopiah	does ÃMQ beats NodeJS and APE Project hands down?
[16:04] sjampoo	heh. it is something completely different
[16:04] Utopiah	I take it as a yes
[16:04] sjampoo	depends on what you want to do ;)
[16:04] sjampoo	0MQ is a messaging serevr
[16:05] Utopiah	right
[16:05] sjampoo	the other two you mentioned are web servers that allow you to write javascript
[16:05] sjampoo	0MQ has nothing to do with javascript
[16:05] Utopiah	hmmm they describe themselves as event frameworks with efficient messaging capabilities though AFAIK
[16:05] Utopiah	JS is just used because it's a fast event loop
[16:05] guido_g	sjampoo: it's not a server
[16:05] Utopiah	(thanks to Google V8 for NodeJS at least)
[16:05] sjampoo	guido, correct
[16:06] sjampoo	0MQ is a library even.
[16:06] guido_g	http://www.zeromq.org/docs:user-guide <- nice read
[16:06] Utopiah	"Node's goal is to provide an easy way to build scalable network programs. "
[16:07] Utopiah	(note that Im not a JS/NodeJS/APE advoce, I jsut try to compare with what I kind of know)
[16:07] Utopiah	s/advoce/advocate/
[16:07] guido_g	apples and oranges
[16:07] guido_g	you know both but can't really compare them
[16:09] sjampoo	Ape and Node.JS allows you to write web application in javascript, they might be able to scale over multiple CPU's but I doubt if they can scale over multiple machines. 0MQ is a messaging library which doesn't understand HTTP but can provide you with the messaging to scale over multiple machine.
[16:10] Utopiah	right AFAIK NodeJS doesn't aim at being scaled over different machines
[16:12] Utopiah	hmm Ill watch the video and scan through the docs, thanks
[16:12] Utopiah	(and grab another coffee ;)
[16:13] sjampoo	:)
[16:14] Utopiah	quick question before, are there torrent/virtual file system/freenode-like projects? EC2 projects?
[17:07] cremes	Utopiah: 0mq is at a lower level than node.js; for a better comparison of 0mq to node.js take a look at a few projects built on top of 0mq that mimic some node.js features...
[17:07] cremes	git://github.com/mjw9100/zmq_reactor.git
[17:07] cremes	http://github.com/chuckremes/zmqmachine
[17:08] cremes	both of those projects implement the reactor pattern using 0mq sockets
[17:08] Utopiah	because for example NodeJS can't handle UDP?
[17:08] cremes	the first is in c while the second is in ruby
[17:08] cremes	i don't know what udp has to do with it
[17:09] Utopiah	well if you can't handle UDP you're higher level
[17:09] Utopiah	on the network stack I mean
[17:10] cremes	i see... yeah, you need to read a bit more about 0mq to understand its purpose
[17:10] cremes	did you read the link guido_g posted?
[17:12] Utopiah	yes Im starting to see the point and where it's called
[17:13] Utopiah	ironically maybe it's easier to read than watch a video to discover it
[17:13] Utopiah	(at least for me)
[17:14] CIA-19	pyzmq: 03Brian Granger 07master * rf98ee71 10/ zmq/eventloop/ioloop.py : Minor changes to ioloop.py. - http://bit.ly/d2dwCg
[17:15] cremes	Utopiah: it's probably worthwhile for you to read through the blog postings and available docs
[17:15] cremes	we're a pretty friendly bunch in here but it's hard to explain 0mq from A to Z when there are already lots of words available on the website ;)
[17:16] Utopiah	don't worry I won't ask for an over the shoulder tutorial
[17:17] Utopiah	but it gives me a quick boost to situate sth new against sth related I know
[17:18] cremes	right; feel free to ask questions as you peruse the docs
[17:18] cremes	i'll try to answer if i'm around/free or there are about 5 or 6 other folks who are usually in channel around this time who may speak up
[20:40] MrEvil	if more messages are being sent faster than the subscribers can process them are the messages queued by the sender or the receiver?
[20:43] MrEvil	and is this queue backed by a the filesystem? I'm not looking for anything complicated. If the items scheduled to be sent or items received but not yet processed were backed by a memory mapped file I would be happy.
[20:47] sustrik	MrEvil: yes, they are queued
[20:48] sustrik	you can set ZMQ_HWM socket option to limit the size of the queue in the memory
[20:48] sustrik	when the size is reached messages start to be dropped
[20:48] sustrik	however, you can set ZMQ_SWAP socket option
[20:49] sustrik	if set, messages start to be offloaded on disk in case HWM is reached
[20:49] sustrik	instead of being dropped
[20:53] MrEvil	what happens if a process crashes? are those messages lost?