Wednesday September 21, 2011

[Time] NameMessage
[00:01] Guest4929 Got a question: I am using a C# zmq binding. I have a thread that blocks on recv(). At some point I want to call abort() on the thread - catch the exception, and terminate. But Abort() seems to have no effect while thread is in recv(). Also, cant seem to find zmq_term() in C#? Any help is much appreciated!
[02:03] dcolish i wish pgm did req/reply. I cant really think of another pattern where i can do broadcast style req/reply. does such a pattern exist?
[02:03] dcolish i'm thinking server broadcasts, clients all reply
[02:03] dcolish or is it simple enough that you just use a req/reply socket where the clients connect and the server reqs on *:5555?
[02:53] CIA-79 pyzmq: 03MinRK 07master * ra53f687 10/ zmq/core/socket.pyx : move self.close() to __del__ from __dealloc -
[02:55] CIA-79 pyzmq: 03MinRK 07master * ra41a9e9 10/ zmq/core/socket.pyx : move socket.close() to __del__ from __dealloc__ ...
[09:48] someone-noone hello! If zmq_connect doesn't return real connection state (for tcp) how can it be resolved? I mean how can i know that connection successfully established?
[12:50] mikko someone-noone: the connection happens async
[12:51] mikko there is really no concept of connection state
[12:51] someone-noone mikko: it's bad, i need to handle unsuccesful connection somehow
[12:51] mikko what sort of pattern are you using?
[12:51] someone-noone req rep
[12:52] mikko you can poll the connection whether it becomes writable
[12:52] mikko and assume failure if timeout exceeds
[12:52] someone-noone thanks
[13:34] enleth Hi there. Anyone knows which git commit for php-zmq actually builds with PHP 5.3.8? I'm getting undefined PHP_ZMQ_TIMEOUT and no wonder, it's commented out in php_zmq_private.h
[13:36] mikko enleth: have you tried the current master?
[13:36] enleth mikko: I think it is current master - but maybe I should just rm -rf the tree and clone it from scratch, just a moment
[13:37] enleth It's the same 0a963..., configuring right now
[13:39] enleth zmq.c:1068:54: error: ‘PHP_ZMQ_TIMEOUT’ undeclared
[13:39] enleth And I just noticed it complains about undefined ZMQ_DEALER at zmq.c:1712:2
[13:40] mikko what version of zeromq?
[13:40] mikko i assume 2.0.x ?
[13:40] enleth 2.1.9, just installed
[13:40] mikko that can't be right
[13:40] mikko let me check
[13:40] enleth [enleth@heroes] [15:40:25] [php-zmq]$ grep ZMQ_DEALER /usr/include/zmq*
[13:40] enleth /usr/include/zmq.h:#define ZMQ_DEALER 5
[13:40] mikko yes
[13:41] enleth This looks very wrong
[13:41] mikko have you got /usr/local/include/zmq.h ?
[13:41] mikko can check what is reported prefix and version for zeromq
[13:41] enleth No, nothing in local
[13:41] mikko you should see those in configure output
[13:41] enleth You mean the zeromq itself?
[13:41] mikko no, php-zmq
[13:42] mikko when you run configure it should report the version and i think prefix as well
[13:42] enleth No, just PHP prefixes
[13:43] enleth Those are PHP includes only
[13:43] mikko checking libzmq installation... found version 2.1.10
[13:43] mikko it should tell you something like that
[13:44] enleth Uh. That's funny. checking libzmq installation... found version 4.0.0
[13:45] mikko yes
[13:45] mikko that explains
[13:45] enleth I mean, WTF?
[13:45] mikko do pkg-config libzmq --prefix
[13:45] mikko did you install libzmq master?
[13:45] mikko that is not 2.1 series
[13:45] mikko
[13:46] enleth That was most certainly built from the 2.1.9 tarball
[13:46] enleth I've never even looked at the zmq git repo
[13:47] enleth I'm building an RPM from a spec file I wrote, but it worked just fine for 2.1.4
[13:47] enleth And there weren't any unpackaged files for that matter
[13:47] mikko what does the spec file do?
[13:48] mikko can you update your php-zmq and rerun phpize && ./configure
[13:48] mikko it should now report the location where it finds libzmq
[13:48] mikko my assumption is that your spec file clones the git repo (?)
[13:49] enleth Nope, it's using the tarball
[13:49] enleth I mean, the spec file for libzmq - php-zmq I'm building manually
[13:50] mikko did you download the tarball?
[13:50] mikko for some reason it looks like you have 4.0.0
[13:50] enleth It's looking for libzmq in /usr/local
[13:50] mikko let me check the tarballs
[13:50] mikko maybe you need --with-zmq=/usr
[13:51] mikko you seem to have 4.x installed under /usr/local
[13:51] enleth My libzmq is definitely built from
[13:51] enleth And /usr/include/zmq.h does contain 2.1.9 version defines
[13:51] enleth I'm trying the --with-zmq
[13:52] enleth libzmq installation... found version 2.1.9, under /usr
[13:52] mikko so you have another installation under /usr/local ?
[13:54] enleth My goodness, the other admin probably did that. It is there, yes. Bah, can't leave the server for a week without anything getting screwed up. But the build still fails, I'll clean up this mess and try again.
[13:56] enleth It appears that neither /usr/include nor /usr/local/include is passed to cc as -I
[13:57] tohava I am using zeromq with erlang, I had an erlang process with zeromq terminate, when it restarted it failed to listen on the same socket due to EADDRINUSE
[13:57] tohava Supposedly there is some way to tell it to reuse address in sockets api, how do I do it in zeromq?
[13:58] mikko i think we already use SO_REUSEADDR
[13:58] mikko but the socket might be in TIME_WAIT state (?)
[13:59] tohava if the socket entered TIME_WAIT, reuse won't work?
[13:59] enleth Ugh, it does build now. Thanks mikko, sorry for bothering.
[14:01] mikko tohava: not 100% sure
[14:02] mikko SO_REUSEADDR allows your server to bind to an address which is in a
[14:02] mikko TIME_WAIT state.
[14:02] mikko so yes, it should be able to bind
[14:02] enleth By the way, what's the status of zmq3 support for PHP and Python?
[14:02] mikko tohava: if that happens again can you check what state the socket is in?
[14:02] mikko enleth: i did zmq3 support for php
[14:02] tohava mikko, I will take a look
[14:02] mikko the current master supports 2.1.x and 3.x
[14:03] enleth Is it documented yet?
[14:03] mikko it should be
[14:03] mikko not sure if the doc generation script works
[14:04] mikko i got it running somewhere
[14:04] mikko the changelog should be accurate
[14:07] tohava mikko, state is "LISTEN"
[14:07] tohava mikko, I wonder if this could be a leak in the erlang binding for zeromq
[14:09] mikko tohava: is the process gone?
[14:11] tohava mikko, the erlang process is gone, however all erlang processes are ran through a single unix process (virtual machine)
[14:12] tohava mikko, I think the problem is on our side, thanks anyway, if I find anything of interest on zeromq I'll let you know
[14:12] mikko cool, thanks
[14:13] mikko i've been using zeromq so long that bsd socket api is slowly getting erased from my head
[14:24] enleth Looks like zmq is better than brain bleach
[15:23] taotetek pieterh_: around?
[16:36] bitcycle Hey all. Any ideas as to why I'm getting a segfault using pyzmq?
[16:39] Steve-o can you trace in gdb?
[16:40] bitcycle I'm using the python bindings. The segfault doesn't occur when the pub & sub are in the same process.
[16:42] Steve-o but you still should be able to catch the dump, gdb python ...
[16:44] bitcycle is not a core dump: File format not recognized
[16:59] Steve-o you probably want "gdb `which python`" then "run", difficult to remember off the top of my head
[17:06] mikko gdb --args python
[17:07] mikko and hes gone
[17:08] Steve-o doh, missed by 9 minutes too, :-)
[17:19] cremes hi, i have a question but i *need* it answered within the next 30 seconds or else i'm leaving
[17:19] cremes hurry up
[17:19] cremes anyone?
[17:20] cremes really? no one can help?
[17:20] mikko i've seen a few reports of segfaults with pyzmq lately
[17:20] mikko more or less informal
[17:20] mikko i wonder if they have a bug
[17:49] CIA-79 pyzmq: 03MinRK 07master * re4c3be6 10/ zmq/core/context.pyx : move ctx.term() to __del__ from __dealloc__ ...
[17:52] dcolish I'm running into a "resource not available" issue with pub/sub sockets and i'm not too sure how to fix it. I've never had this issue before. Here's a paste of the two simultaneous sessions
[17:54] Steve-o out of fd's? love vague libc errors
[17:55] dcolish man i hope not, i guess i could be
[17:55] dcolish i'm not running anything else really, should be fine
[17:57] Steve-o can you strace / truss it and capture the error code?
[17:58] dcolish lsof says i'm only using 4094 fds
[17:58] dcolish i can try stracing it
[18:00] dcolish oh thats right, osx doesnt have strace :/
[18:00] Steve-o awesome
[18:01] Steve-o dtruss
[18:01] Steve-o they've ported Solaris' truss it seems
[18:01] Steve-o >>
[18:02] dcolish ah nice
[18:02] dcolish
[18:02] dcolish didnt know about that one
[18:18] Steve-o so that's the connect failure?
[18:18] Steve-o is that edeadlock I don't have a reliable list to hand
[18:19] dcolish yeah, its EINPROGRESS
[18:22] Steve-o that appears to be handled correctly in the code, converted to EAGAIN
[18:23] dcolish right, i guess i'm not sure why its not connecting at all
[18:23] dcolish its over localhost so i'd expect it to connect
[18:23] Steve-o the code means it is connecting though
[18:24] dcolish hmm, ok but even when i poll i'm not receiving messages
[18:24] Steve-o but it hasn't finished connecting, so a bit strange
[18:24] Steve-o it should be a very quick transient state
[18:25] dcolish right
[18:27] Steve-o but it is caught somewhere, is it pyzmq?
[18:27] dcolish nowhere i can tell, maybe there is something internal to pyzmq
[18:27] dcolish ?
[18:27] Steve-o are you on the latest from upstream?
[18:27] dcolish nope, 2.1.7
[18:28] dcolish this has worked for me in the past
[18:28] Steve-o it might have been fixed already :/
[18:28] dcolish i'll look
[18:29] Steve-o but maybe not:
[18:30] Steve-o it does not handle -1/EAGAIN on connect
[18:30] Steve-o file a bug in github and ping the list
[18:30] Steve-o s/ping/prod/ uggh, I shouldn't use that word
[18:30] dcolish no ping?
[18:31] Steve-o prod the list, cf. ping the list
[18:33] dcolish poke the list?
[18:33] dcolish *sigh*
[18:35] Steve-o needs to be "if rc != 0 or zmq_errno() != EAGAIN:"
[18:36] Steve-o should be easy to test
[18:50] dcolish hmm so updated to pyzmq HEAD and I get "No such file or directory" when calling connect
[18:51] dcolish did the zmq api change much between 2.1.7 and 2.1.9, or at least enough to break pyzmq?
[18:51] pieterh dcolish: what's the connect string?
[18:51] dcolish "tcp://"
[18:51] dcolish i've also tried "tcp://localhost:7004"
[18:52] pieterh so there was a change that pre-validated the connect string
[18:52] dcolish i'm binding "tcp://*:7004" on the same host
[18:52] pieterh allowing zmq_connect to return an error immediately
[18:52] pieterh does "ping" work?
[18:52] dcolish yes
[18:52] dcolish so does "ping localhost"
[18:52] pieterh can you try a simple C test?
[18:52] dcolish sure
[18:54] dcolish just to reiterate, PUSH/PULL sockets seemed to work on the same host, its just PUB/SUB which is giving me trouble
[18:59] pieterh dcolish: that's weird
[18:59] dcolish aye, i'm stumped
[19:00] pieterh well, usual problem solving technique applies
[19:00] pieterh 1. disregard the cases where it works, we don't care about those
[19:00] pieterh 2. reproduce the case where it fails using simplest possible code
[19:00] Steve-o probably easiest to test plain C version, but that return value does look suspect
[19:00] pieterh 3. file issue with test case
[19:01] pieterh if it's returning EINVAL that's the pre-validation complaining about the connect hostname
[19:01] Steve-o but "resource not available" is EAGAIN
[19:02] pieterh Steve-o: hmm
[19:03] Steve-o unless that is a red herring
[19:03] dcolish i think it might be
[19:04] dcolish the EAGAIN behavior looks as i'd expect
[19:04] jond dcolish: can you strace the binary?
[19:04] Steve-o output listed above
[19:05] Steve-o >>
[19:05] Steve-o it's truss or rather dtruss on OSX
[19:06] Steve-o which as Pieter suggests might be all bogus as the error could come from 0mq's validation routine
[19:07] dcolish i gotta run out and grab some food, i'll run the C test cases when i get back
[19:07] pieterh ack
[19:11] jond in the link posted here the ports are different
[20:01] dcolish jond: yeah that was a mistake, the output is the same when they're the same
[20:02] jond dcolish: i was just checking.... ;-)
[20:55] dcolish well here's a C test case for my issue Doesnt seem to be receiving messages at all, but I am not getting errors while calling connect
[20:57] jond dcolish: you need to subscribe to a pattern
[20:59] dcolish jond: shoot, you're right. i dont know why I thought it would always subscribe to all by default
[20:59] dcolish thanks
[21:00] dcolish although, still doesnt explain those errors in pyzmq
[21:01] jamescarr how come it's not possible to have a pure java client?
[21:02] jond dcolish: no prob, get the c example working, then contact the pyzmq people if problem still there.
[21:06] jond dcolish: actually in the pyzmq does sock.recv(1) mean non-blocking recv? you hadnt subscribed there either in which case EAGAIN might be ok
[21:06] dcolish no i would expect it to time out, but i'm not sure its reporting the right errno
[21:10] jond dcolish: pyzmq src doc "If NOBLOCK is set, this method will raise a ZMQError with EAGAIN if a message is not ready"
[21:11] dcolish right, but when its not set what should happen?
[21:11] dcolish normally i just use a poller but i wasnt seeing the messages so i ended up trying recv with a timeout
[21:11] dcolish thats how i got on this path
[21:12] dcolish turned out that all i needed was a subscribe pattern
[21:12] GregBowyer I have a zeromq application that is currently in two parts, the top of the application is called "capture-head" and is a program that listens to and reassembles packets from net_ring or libpcap off the network wire. Once he has reassembled a packet (typically http) the http packet is put out onto zero-mq for worker-processes to do with as they will.
[21:12] GregBowyer currently these worker processes take the (in this case) http request, and replay it against a different server
[21:13] GregBowyer the upshot of this is I can consume our production traffic, and use that to 24x7 load test new software waiting release in staging
[21:13] GregBowyer this is done with PUSH/PULL sockets, there is one capture head and many (identical) workers
[21:14] GregBowyer now I have been reading the documentation, is there a way I could have two sets of workers see these messages and act on the independently
[21:15] GregBowyer effectivly I would like to copy the request as sent on the PUSH socket and have worker group 1 consume it *and* worker group 2 consume it
[21:15] Steve-o GregBowyer: so zeromq as a network thread pool, you want pub functionality?
[21:16] GregBowyer Steve-o not really I guess I should draw a picture as its a little hard to explain (or I am not getting your answer)
[21:16] Steve-o the obvious seems a broker in each group that has SUB and sends out on PUSH
[21:18] GregBowyer so the workers connect to a PULL socket in the broker, which connects to the PUB in the capture head
[21:19] Steve-o yup
[21:20] GregBowyer .... yeah that could work
[21:20] Steve-o depends on reliability requirements if you need a more advanced pattern
[21:21] GregBowyer its quite nice the reliability requirements for this software stack are to just drop things if anything goes wrong
[21:23] jond dcolish: I don't think that calling pyzmq.recv(1) is passing a timeout, it's a flag meaning don't block. glad yr sorted anyhow.
[21:24] dcolish crap, you're right
[21:24] dcolish i probably got mixed up with poll(timeout)
[21:25] GregBowyer Steve-o, that does mean I end up with an extra process in each worker group, but I guess thats ok since I can abuse the broker to scale up and down a number of subordinates
[21:26] GregBowyer Steve-o, .... which makes me ask is there a way I can see the size of the message queue for PUSH/PULL sockets
[21:26] Steve-o GregBowyer: actually I was wondering if you could multi-push
[21:26] Steve-o like this:
[21:29] GregBowyer similer but I dont have to worry about lost messages, in fact I could open multiple PULL sockets and have different worker groups read from different PULL sockets, I do need the load balancing within a worker group
[21:34] Steve-o two sockets is the cleaner method I guess
[21:34] Steve-o send twice
[21:34] GregBowyer ok I will try that
[21:43] Steve-o or something elaborate like pub-sub ipc to two devices that forward on push, so only one send and scalable
[21:46] Steve-o you can then dynamically increase or decrease the groups
[21:46] Steve-o pretty nifty
[21:50] Steve-o GregBowyer: I think many users prefer using devices to manually hacking distribution with multiple sockets and sends
[21:53] jond steve-o: yep it's fowarder and streamer you are talking about, but worth noting these are gone in 3.0 onwards from the core library
[21:53] GregBowyer that seems like a nice approach, the only thing that put me off was the concept of having yet another process on the box fwiw zero-mq works really well, but the box has some quite funny networking requirements (e.g. I have to be careful what I send since I listen to the network at the same time) and its definatly using the CPU in the box (the entire thing cruses at GB's of vmem and fairly constant CPU 100% burn
[21:54] Steve-o device you can spawn off in another thread, makes it easy
[21:54] GregBowyer my PUSH/PULL is over unix sockets at the moment, that basically avoids the networking stack and means I dont end up hearing things I send (which was really fun last time I had that happen :D)
[21:55] Steve-o I think we still don't have ITC sockets (?)
[21:56] GregBowyer you must have ipc sockets I am using them !
[21:56] GregBowyer no wait
[21:56] Steve-o between threads, not processes
[21:56] GregBowyer sorry my bad parse error
[21:56] Steve-o so disruptor basically
[21:56] GregBowyer I read that as IPC
[21:56] Steve-o stick it on my todo list
[21:57] GregBowyer you meaning the java distruptor LMAX ?
[21:57] Steve-o yup
[21:57] Steve-o really needs proactor pattern which is my, I guess, present outstanding task
[21:58] Steve-o trying to improve Win32/64 performance by targeting IOCP
[21:58] GregBowyer I was under the impression that was what zero-mq used under the hood no ?\
[21:58] Steve-o a big long list of dependencies
[21:58] Steve-o Win32 is all select() reactor based
[21:59] Steve-o and IPC is TCP socket based
[21:59] Steve-o not so great unfortunately
[21:59] Steve-o but still pretty good numbers
[21:59] GregBowyer Steve-o, are unix versions picking between kqueue/ epoll and friends
[21:59] Steve-o yes
[22:00] Steve-o you have an option at configure time to override
[22:00] GregBowyer I care less about windows as I could never get the packet capture rate on windows that I get on linux / freebsd, pcap on windows is way to slow, and the native option is no fun
[22:01] Steve-o which is why we are in the state today, Windows is second citizen
[22:01] GregBowyer (not that I am saying that windows is not important for zero-mq its just never going to be usedful for this app)
[22:01] Steve-o I've spent the last few years re-architecting and developing OpenPGM for Windows, headache at times
[22:01] GregBowyer its interesting on windows you do IPC with TCP, I would have thought namedpipes would have been better (my windows API knowledge is super rusty so I might be way off here)
[22:02] Steve-o you cannot poll a named pipe
[22:02] Steve-o so you need the proactor model
[22:02] Steve-o which I actually have on a local branch
[22:03] Steve-o and IOCP on another, they all need to integrate a bit cleaner
[22:03] Steve-o last project was IPv6 though so distracted *a lot*
[22:03] GregBowyer ah check :D and hence we are reduced to your earlier statements
[22:03] jhawk28 whats a proactor?
[22:03] Steve-o completion ports
[22:04] Steve-o basically, notification when something has finished,
[22:04] GregBowyer out of sheer curiosity what would be your take on sctp and similer protocols
[22:04] Steve-o because you start the job before you can run it, you can queue up several receives for better performance, and usually zero-copy
[22:04] Steve-o sctp is in 0mq 1.0
[22:05] GregBowyer jhawk28, there is a fairly nice (albiet slighty dated and biased) description of it here
[22:05] Steve-o I think Martin Sustrik or someone commented on how it looks well on paper but doesn't implement well in practice
[22:06] Steve-o the framing you see in SCTP is effectively improved with something like SPDY by Google too, to a certain extent
[22:06] GregBowyer makes sense
[22:07] GregBowyer just curious because I saw the wiki page but didnt see anything about it anywhere else
[22:08] Steve-o new protocols have a large mountain to climb to get acceptance anywhere
[22:09] dcolish wrt my no resource found errors earlier, i think i may have actually run out of fds
[22:09] dcolish *to
[22:09] GregBowyer Steve-o, heh well I said zero-mq for this application and that was a big enuff battle, so a brand new layer 4 protocol ......
[22:09] Steve-o which is why 0mq has UDP/PGM in addition to IP/PGM
[22:10] Steve-o gigabit wire speed Cisco routers are expensive for IP/PGM
[22:10] GregBowyer Steve-o, ? do you offer similer atomic guarantees on UDP/PGM ?
[22:11] Steve-o actually UDP tends to run a bit faster too
[22:11] GregBowyer (again curious more than intending to use this at the moment, adding endpoints is pretty involved for me)
[22:11] Steve-o PGM is available for PUB/SUB only
[22:12] Steve-o makes things easier
[22:12] Steve-o you could stretch it like TIBCO RV for PUSH/PULL and REQ/REP but it can get overly complicated and slow
[22:13] Steve-o unfortunately 0mq has no bus socket either, but no so much need so far
[22:13] GregBowyer I think I would end up using it to spin out multiple brokers on seperate boxes and use PUSH/PULL on those boxes rather than try to warp it
[22:14] Steve-o it is nice to have the options
[22:15] GregBowyer so far its been seriously neat stuff I am digging zero-mq for how fast its working although I will have to see if it puts up with the pull production capture one day rather than just a slice of the traffic :D
[22:16] Steve-o it has a good record so far, so give it a spin
[22:17] GregBowyer for now my biggest bottle neck is in libpcap, packet reassembly and damnable memory leaks occuring with broken tcp conversations that dont quite reassmble nicly (the same ones that would appear to be FIN_CLOSE forever)
[22:17] GregBowyer but thats not zero-mq's fault thats my own shoddy tcp state machine
[22:17] Steve-o you need hardware acceleration?
[22:17] Steve-o oodles of cards available
[22:17] GregBowyer what it did was take out all of the bzyantine queuing instantly, I think I dropped about 3000 lines of code outright
[22:18] GregBowyer Steve-o, has to be commodity hardware only unfortunatly, but I can still get a damn good rate, just screwed up latency between requests
[22:19] GregBowyer Steve-o, I am more than tempted to get an FPGA and try this on it mind
[22:19] Steve-o does libpcap work with user-space mellanox and solarflare drivers?
[22:20] GregBowyer no idea
[22:20] Steve-o otherwise endace, corvil, tsassociates all have capture hardware
[22:21] GregBowyer the next version may be using PF_RING from here it seems to work out pretty nice in testing
[22:22] Steve-o I didn't fancy the hassle of writing to that
[22:22] Steve-o libpcap can be hardware accelerated transparently which is why it is nice
[22:22] Steve-o I think openonload with solarflare would be worth a check before the effort required for pf_ring
[22:23] GregBowyer I had not seen openonload before thanks !
[22:25] GregBowyer anyhow thanks for the pointers towards PUB/SUB and brokers, I think I will go with that long term
[22:30] Steve-o mellanox have a similar technology now
[22:30] Steve-o I think it was 2-3 months ago, mentioned on
[22:30] Steve-o pass on the name
[22:32] Steve-o it might be VMA:
[22:32] Steve-o yup, that's right:
[22:34] jond pieterh_: are you attending this PubSub thing in london....
[22:34] pieterh jond: nope, I'm in the States right now
[22:35] pieterh Steve-o: do you think VMA would work well with 0MQ?
[22:35] jond pieterh_: I'm not going either, but i've emailed sustrik/ian in case they are around later.
[22:35] GregBowyer Steve-o, that looks quite nice, albeit it seems to be a faster network card right rather than a dedicated capture card
[22:35] Steve-o should be pretty good, on par with IP-over-IB
[22:37] Steve-o GregBowyer> libpcap isn't my forte but this would offer performance improvement without code change and no overly custom hardware
[22:38] Steve-o otherwise ask libpcap or ts-associates for their (recommended) capture hardware
[22:40] Steve-o I know the ts-associates dev team are written their own capture drivers so, not very flexible from resourcing point of view
[22:41] GregBowyer I will put it on the list to investigate, for now I get big improvements in small changes so I am not too unhappy at the moment
[22:41] GregBowyer and the current use of this software it more than meets its requirements, my one itty bitty box turns over a full production workload that is serviced by hundreads of machines in a search engine cluster
[22:42] GregBowyer but yeah, more people I work with ask to play with this stuff so the requirements and performance is going to become harder, so I might just say fuckit and look at custom hardware
[22:42] Steve-o the libpcap hardware is popular with analysing mdds networks, so a bit niche
[22:43] GregBowyer the biggest lift recently was going to linux 3.0 and having native compiled BPF filters
[22:43] Steve-o linux 3.0 introduces sendmmsg() which is quite interesting
[22:44] Steve-o IBM posted speed improvements with that and recvmmsg() but I have difficulty reproducing on IBM hardware
[22:44] Steve-o that's pretty much UDP only concerns though, you can skip all that with pcap
[22:46] GregBowyer BPF jit here
[22:46] GregBowyer I was just about to mention sendmmsg
[22:47] GregBowyer I think I can bench comfortably at 1Gbe 10Gbe is a challange, but I dont see that much traffic yet
[22:48] GregBowyer just thought you might like to know that zero-mq IPC is not a barrier in there :D
[22:48] Steve-o :-)
[22:52] GregBowyer anyhow thanks again for the interesting conversation, I have to leave it at that and run off to catch my train
[22:53] Steve-o no worries, laters