Tuesday August 3, 2010

[Time] NameMessage
[12:47] cremes interesting poll analysis from zed; may influence 0mq polling architecture
[12:47] cremes
[13:10] sustrik yes, seen it
[13:11] sustrik if anyone wants to experiment with different polling algorithms 0mq offers a compile time option to enforce usage of a particular one
[13:11] sustrik ZMQ_FORCE_POLL
[13:11] sustrik ZMQ_FORCE_EPOLL
[13:11] sustrik etc.
[13:13] CIA-19 jzmq: 03Gonzalo Diethelm 07master * r362370e 10/ src/org/zeromq/ :
[13:13] CIA-19 jzmq: Added in Socket a reference to Context, so that Java's GC
[13:13] CIA-19 jzmq: won't deem the Context as unused because there are no references
[13:13] CIA-19 jzmq: to it. -
[14:46] cremes any reason zed's suggestion for dynamically shuttling FDs between poll & epoll couldn't be wrapped up in 0mq?
[15:15] sustrik cremes: no technical reason
[15:15] sustrik however, i am not sure it would help much
[15:16] sustrik what 0mq does in its I/O thread is opportunistic polling
[15:16] sustrik if it has no data to send
[15:16] sustrik it never polls the socket for OUT
[15:16] cremes it's probably wise to let zed experiment with the idea (and implementation) a bit before changing the library :)
[15:16] sustrik if it does have data to send it polls the socket *once*
[15:16] sustrik then writes the data
[15:17] cremes sustrik: that seems smart
[15:17] sustrik if the write succeeds it assumes the socket is still available for writing
[15:17] sustrik so it does not poll again
[15:17] sustrik until send fails
[15:17] sustrik same thing on the recv side
[15:18] sustrik this kind of algorithm would heavily distort any perf results we have
[15:18] sustrik however, if you want to go experimenting with different poll types
[15:18] sustrik there's the compile time setting (ZMQ_FORCE_EPOLL etc.)
[15:18] sustrik so you may do some benchmarking yourself
[15:19] cremes sure
[15:19] cremes assuming zed's results are trustworthy, i do think it could help 0mq
[15:20] cremes let's assume that there are half a dozen 0mq sockets fronting thousands of tcp sockets
[15:20] sustrik yes?
[15:20] cremes further, assume they are XREQ/XREP pairs
[15:21] cremes as the XREQ sends data round-robin to the XREP sockets, the majority of those sockets are idle
[15:22] cremes hmmm... as i'm writing this out i see your point...
[15:22] sustrik :)
[15:22] cremes you wouldn't poll any of the other sockets for writability until their "turn" came up in the round-robin scheme
[15:22] cremes never mind!
[15:22] cremes ;)
[15:22] sustrik right
[15:24] cremes so if zed is building mongrel2 on top of 0mq, i'm wondering if he needs to split poll/epoll like he wrote in his latest posting
[15:24] sjampoo for the other side (http)
[15:25] cremes sjampoo: ah... yes, that makes sense; it would be necessary for the non-0mq sockets
[15:25] cremes but then why not make "superpoll" into "opportunisticpoll" and use the same concept as 0mq for polling regular FDs
[15:26] cremes perhaps that would result in too many syscalls?
[15:27] sustrik sjampoo: as far as i understand the superpoll thing is intended for HTTP sockets
[15:27] sustrik not the 0mq sockets
[15:38] sjampoo sustrik: yah i believe so too ;)
[15:41] sjampoo cremes, I am not sure if he does or doesn't do that. Maybe it has to with how he uses/wants to use libtask, which currently uses poll.
[15:43] sjampoo It could be that such an opportunistic poll is difficult to implement in his FSM approach, but then again it could be that i am just talking out of my ass.
[15:44] cremes heh
[15:44] cremes perhaps zedas will enlighten us in channel
[16:02] Utopiah hi #zeromq
[16:04] Utopiah does ØMQ beats NodeJS and APE Project hands down?
[16:04] sjampoo heh. it is something completely different
[16:04] Utopiah I take it as a yes
[16:04] sjampoo depends on what you want to do ;)
[16:04] sjampoo 0MQ is a messaging serevr
[16:05] Utopiah right
[16:05] sjampoo the other two you mentioned are web servers that allow you to write javascript
[16:05] sjampoo 0MQ has nothing to do with javascript
[16:05] Utopiah hmmm they describe themselves as event frameworks with efficient messaging capabilities though AFAIK
[16:05] Utopiah JS is just used because it's a fast event loop
[16:05] guido_g sjampoo: it's not a server
[16:05] Utopiah (thanks to Google V8 for NodeJS at least)
[16:05] sjampoo guido, correct
[16:06] sjampoo 0MQ is a library even.
[16:06] guido_g <- nice read
[16:06] Utopiah "Node's goal is to provide an easy way to build scalable network programs. "
[16:07] Utopiah (note that Im *not* a JS/NodeJS/APE advoce, I jsut try to compare with what I kind of know)
[16:07] Utopiah s/advoce/advocate/
[16:07] guido_g apples and oranges
[16:07] guido_g you know both but can't really compare them
[16:09] sjampoo Ape and Node.JS allows you to write web application in javascript, they might be able to scale over multiple CPU's but I doubt if they can scale over multiple machines. 0MQ is a messaging library which doesn't understand HTTP but can provide you with the messaging to scale over multiple machine.
[16:10] Utopiah right AFAIK NodeJS doesn't aim at being scaled over different machines
[16:12] Utopiah hmm Ill watch the video and scan through the docs, thanks
[16:12] Utopiah (and grab another coffee ;)
[16:13] sjampoo :)
[16:14] Utopiah quick question before, are there torrent/virtual file system/freenode-like projects? EC2 projects?
[17:07] cremes Utopiah: 0mq is at a lower level than node.js; for a better comparison of 0mq to node.js take a look at a few projects built on top of 0mq that mimic some node.js features...
[17:07] cremes git://
[17:07] cremes
[17:08] cremes both of those projects implement the reactor pattern using 0mq sockets
[17:08] Utopiah because for example NodeJS can't handle UDP?
[17:08] cremes the first is in c while the second is in ruby
[17:08] cremes i don't know what udp has to do with it
[17:09] Utopiah well if you can't handle UDP you're higher level
[17:09] Utopiah on the network stack I mean
[17:10] cremes i see... yeah, you need to read a bit more about 0mq to understand its purpose
[17:10] cremes did you read the link guido_g posted?
[17:12] Utopiah yes Im starting to see the point and where it's called
[17:13] Utopiah ironically maybe it's easier to read than watch a video to discover it
[17:13] Utopiah (at least for me)
[17:14] CIA-19 pyzmq: 03Brian Granger 07master * rf98ee71 10/ zmq/eventloop/ : Minor changes to -
[17:15] cremes Utopiah: it's probably worthwhile for you to read through the blog postings and available docs
[17:15] cremes we're a pretty friendly bunch in here but it's hard to explain 0mq from A to Z when there are already lots of words available on the website ;)
[17:16] Utopiah don't worry I won't ask for an over the shoulder tutorial
[17:17] Utopiah but it gives me a quick boost to situate sth new against sth related I know
[17:18] cremes right; feel free to ask questions as you peruse the docs
[17:18] cremes i'll try to answer if i'm around/free or there are about 5 or 6 other folks who are usually in channel around this time who may speak up
[20:40] MrEvil if more messages are being sent faster than the subscribers can process them are the messages queued by the sender or the receiver?
[20:43] MrEvil and is this queue backed by a the filesystem? I'm not looking for anything complicated. If the items scheduled to be sent or items received but not yet processed were backed by a memory mapped file I would be happy.
[20:47] sustrik MrEvil: yes, they are queued
[20:48] sustrik you can set ZMQ_HWM socket option to limit the size of the queue in the memory
[20:48] sustrik when the size is reached messages start to be dropped
[20:48] sustrik however, you can set ZMQ_SWAP socket option
[20:49] sustrik if set, messages start to be offloaded on disk in case HWM is reached
[20:49] sustrik instead of being dropped
[20:53] MrEvil what happens if a process crashes? are those messages lost?