Thursday October 13, 2011

[Time] NameMessage
[01:49] CIA-79 pyzmq: 03MinRK 07 * r4d74a9c 10/ : exclude mailmap from manifest -
[01:49] CIA-79 pyzmq: 03MinRK 07 * r530edba 10/ zmq/eventloop/ : small bugfixes in ZMQStream.flush ...
[03:06] streamer is it possible to use 0mq for streaming messages, i.e. req -> rep1, rep2, rep3, ... repN ?
[04:14] cactuss is there a way to have zeromq buffer publisher messages onto disk during times when it has no subscribers, and then guarantee that subsequent scribers will get that data eventually given enough disk space?
[04:22] cactuss i'm wondering if zeromq can save me from having to use S3 as an intermediary data placeholder
[13:54] cremes cactuss: no, 0mq doesn't have that capability built-in; you would need to build it on top
[14:09] CIA-79 libzmq: 03Martin Sustrik 07 * r8ec3840 10/ (src/session_base.cpp src/socket_base.cpp src/socket_base.hpp): Dummy dns:// transport added. ...
[16:25] przemeklach Is there a difference between Java classpath and java.library.path
[16:25] przemeklach I'm just going through the jzmq installation tutorial
[16:26] przemeklach and I'm trying to compile one of the examples and I'm getting an error which I assume is due to the fact that my java path is not set correctly
[16:50] Steve-o java.library.path is the JNI
[16:50] Steve-o classpath is the JAR/class files
[16:52] przemek ok, so i set the classpath to be where the code is set CLASSPATH=/home/camera/source/zguide/examples/Java/
[16:52] przemek when i try to javac I still get the error
[16:53] przemek I know i have to do the -Djava.library etc but I need to compile first don't i?
[16:54] Steve-o you need a dll or so file yes
[16:55] przemek right but I'm getting an error when I try to compile
[16:55] przemek javac
[16:55] przemek I'm not even trying to run it yet
[16:56] przemek when I type java -Djava.library etc that is to run, right?
[16:56] Steve-o there should be a libzmq.jar or similar and a I think before you start building apps
[16:57] Steve-o 1) build 0mq, 2) build jzmq, 3) build java examples
[16:58] przemek Ya I followed the instructions for installing 0mq and jzmq
[16:58] przemek libzmq.jar dne on my system
[16:58] przemek does
[16:58] Steve-o I'll have a look now as I've never tried the Java bindings
[16:58] przemek ok tks for your help
[16:59] Steve-o Debian or RH?
[16:59] przemek RH
[16:59] przemek I assume RH = red hat
[17:00] Steve-o yup or Fedora
[17:01] przemek whoops, i mean Scientific Linux
[17:01] przemek sry
[17:01] Steve-o same thing pretty much
[17:01] przemek k
[17:01] Steve-o RHEL = CentOS = Scientific Linux
[17:01] przemek k
[17:02] przemek and I'm using zmq v2.1 btw
[17:02] Steve-o I'm trying 2.1.10 now
[17:02] przemek k
[17:06] Steve-o ok 0mq built, one bug with the make system found, next step jzmq
[17:06] przemek k
[17:07] Steve-o are you using master or a download?
[17:10] przemek i git cloned from github
[17:11] Steve-o ok, I'll try openjdk 7
[17:12] przemek hm i think i'm using oracle java
[17:12] przemek not openjdk
[17:12] Steve-o I'm on a Debian VM first, not much choice
[17:12] przemek k
[17:16] Steve-o ok that's built
[17:17] Steve-o I now have x64/share/java/zmq.jar & x64/lib/
[17:18] przemek what the heck; i just searched for zmq.jar and it found it this time
[17:18] przemek blh
[17:18] przemek ok
[17:18] przemek so they are both in different dir's than yours but they do exist on my system; the so and the jar
[17:19] przemek '/usr/local/share/java/zmq.jar'
[17:19] przemek and
[17:20] przemek '/usr/local/lib/'
[17:20] Steve-o I'm using a custom prefix for install
[17:20] przemek oh ok
[17:26] przemek so I also downloaded the zguide
[17:26] przemek and what i've been trying to get to work is to compile and run the example
[17:26] przemek and that is where I'm having the issues
[17:27] przemek the example is in zguide/examples/Java
[17:28] Steve-o what an ugly command line:
[17:28] Steve-o LD_LIBRARY_PATH=/home/local/zeromq/x64/lib java "-Djava.library.path=/home/local/zeromq/jzmq/x64/lib" -classpath /home/local/zeromq/jzmq/x64/share/java/zmq.jar:. local_lat tcp:// 1 100
[17:28] przemek omg
[17:28] przemek ok
[17:28] Steve-o three libraries required
[17:29] Steve-o zmq.jar, and
[17:29] Steve-o classpath, java.library.path, and ld_library_path
[17:29] przemek ok
[17:29] przemek so if I have a file
[17:30] Steve-o javac -cp /home/local/zeromq/jzmq/x64/share/java/zmq.jar:. hwserver for the build
[17:30] Steve-o you don't need JNI or native libraries for that step
[17:30] przemek ok tks for ur help
[17:35] Steve-o I think the Maven build is supposed to make it easier
[17:35] Steve-o I'lm trying that now
[17:49] przemek k
[17:50] Steve-o lol maven is a mess
[17:51] przemek lol
[17:53] Steve-o with "mvn jar:jar" you can get a jar with bundled jni lib
[17:54] Steve-o so the command line shortens a little, only a little:
[17:54] Steve-o LD_LIBRARY_PATH=/home/local/zeromq/x64/lib java -classpath /me/local/zeromq/jzmq/jzmq/target/jzmq-2.1.0-SNAPSHOT.jar:. local_lat tcp:// 1
[17:55] Steve-o If you install the 0mq package then you don't need the library path, just the class path
[17:56] przemek but if I set the classpath
[17:56] przemek then I should be able to just run my class via
[17:56] przemek java local_lat
[17:56] przemek ?
[17:56] Steve-o looks like it
[17:57] przemek what is the command to see what my class path is
[17:58] przemek echo ...
[17:59] Steve-o echo $CLASSPATH
[17:59] przemek ya i tried that
[17:59] przemek and it returns nothing
[18:00] Steve-o because no default classpath
[18:00] Steve-o export CLASSPATH=..../target/jzmq-2.1.0-SNAPSHOT.jar:.
[18:00] Steve-o then, java local_lat
[18:01] Steve-o yup, works great.
[18:02] Steve-o I'm doing:
[18:02] Steve-o export CLASSPATH=/home/local/zeromq/jzmq/jzmq/target/jzmq-2.1.0-SNAPSHOT.jar:.
[18:02] Steve-o export LD_LIBRARY_PATH=/home/local/zeromq/x64/lib
[18:02] Steve-o java local_lat
[18:04] przemek having a hard time finding these paths
[18:04] przemek like
[18:04] przemek I should be poiting my paths based on where zeromq and jzmq where installed
[18:04] Steve-o that classpath is pointing to the output of maven2
[18:04] przemek not to the original source folder I downloaded right
[18:05] Steve-o the ld path to where 0mq lib is living
[18:05] Steve-o but you stated: '/usr/local/lib/'
[18:05] przemek yes
[18:05] Steve-o so you should not need to specify ld-library-path
[18:05] Steve-o only classpath
[18:05] Steve-o have you built via maven?
[18:05] przemek no
[18:05] przemek i have no idea how to do that
[18:06] przemek i just followed the instrutions for installing 0mq and jzmq
[18:06] Steve-o if you have maven installed: sudo yum install maven2 I guess
[18:06] Steve-o then run: mvn jar:jar
[18:06] Steve-o then you get the jni+jar file bundle you can specify in the classpath
[18:06] przemek and from within which dir do I run that comand
[18:07] Steve-o in the jzmq directory with pom.xml
[18:08] Steve-o ...
[18:08] Steve-o why not try this:
[18:08] Steve-o export CLASSPATH=/usr/local/share/java/zmq.jar:.
[18:08] przemek fighting to get maven installed
[18:08] Steve-o javac hwserver
[18:08] Steve-o java hwserver
[18:10] Steve-o java should be picking up both native libraries automatically
[18:10] przemek ok i'll let you know once i get maven installed
[18:10] przemek just doing yum install is not finding it
[18:11] Steve-o skip that for a bit as I think you don't need it
[18:11] przemek ok
[18:12] Steve-o you have JNI & native library in /usr/local/lib the files and
[18:12] Steve-o these shouldn't need any effort to use
[18:13] Steve-o the only one needed attention is /usr/local/share/java/zmq.jar
[18:13] Steve-o so find your example:
[18:13] przemek ok
[18:13] Steve-o try to build it:
[18:13] Steve-o export CLASSPATH=/usr/local/share/java/zmq.jar:.
[18:13] Steve-o javac hwserver
[18:13] przemek do i need the ':.' at the end?
[18:14] przemek and what does that mean?
[18:14] Steve-o it means also search the current directory, i.e. for
[18:14] Steve-o Java is a bit stupid, but I think they've fixed it a bit
[18:15] Steve-o classpath specifies jar files and directories containing .class files
[18:15] przemek ok so compiling works awesome now
[18:15] Steve-o now try running
[18:15] przemek however
[18:15] Steve-o java hwserver
[18:15] przemek java hwserver
[18:15] przemek gives errors
[18:15] przemek Exception in thread "main" java.lang.UnsatisfiedLinkError: no jzmq in java.library.path
[18:15] przemek at java.lang.ClassLoader.loadLibrary(
[18:15] przemek at java.lang.Runtime.loadLibrary0(
[18:15] przemek at java.lang.System.loadLibrary(
[18:15] przemek at org.zeromq.ZMQ.<clinit>(
[18:15] przemek at hwserver.main(
[18:16] Steve-o try this then:
[18:16] Steve-o export LD_LIBRARY_PATH=/usr/local/lib
[18:17] queuenoob Hi all, wondering if anyone can point me in the right direction: with 0MQ, is there a way to have a subset of message within a single queue be "correlated" in that for a specific message to be served, any prior correlated queue message must have been served and received an ack indicating it was fully processed. If it hasn't then have the queue serve the next message.. does this concept actually exist in any queuing systems?
[18:17] Steve-o przemek_: actually just running "ldconfig" should bring in the library
[18:17] przemek that did it!
[18:17] Steve-o on debian /etc/ contains /usr/local/lib
[18:17] Steve-o so the library database just needs updating
[18:18] Steve-o so, run "sudo ldconfig"
[18:18] przemek ya just did
[18:18] przemek but i already ran the LD_Library one
[18:18] Steve-o then wipe the env variable: export LD_LIBRARY_PATH=
[18:18] Steve-o and it should also work
[18:19] przemek no
[18:19] przemek doesn't work if i remove LD
[18:19] przemek even after running ldconfig
[18:19] przemek well that's fine
[18:19] Steve-o it might be a weird default setting in /etc/
[18:20] Steve-o usually it should pull in /usr/local/lib
[18:20] przemek so I just need to add the classpath and the LD path and I'm good
[18:20] przemek so there is only one line in that file
[18:20] Steve-o let me check Fedora settings
[18:20] przemek include*.conf
[18:20] przemek k
[18:23] Steve-o queuenoob: that sounds pretty much limited to websphere mq
[18:24] Steve-o queuenoob: but also not much difference from a req/rep socket
[18:25] Steve-o przemek_: so Fedora is empty, just create a new file:
[18:25] przemek wel my isn't empty
[18:25] przemek include*.conf
[18:25] Steve-o echo /usr/local/lib > /etc/
[18:25] przemek it has the line in it
[18:26] Steve-o empty as in no specification of /usr/local, only llvm & xulrunner
[18:26] queuenoob Steve-o: Thanks for the reply :) I'll check into that
[18:26] Steve-o queuenoob: the critical point is reliability requirement
[18:28] przemek it's telling me permission denied
[18:28] przemek even when i run as sudo
[18:28] Steve-o przemek_: so create a new file, say local.conf, then re-run ldconfig and will be good to go
[18:28] Steve-o su first
[18:28] Steve-o redirect on sudo will lose the permissions for writing
[18:28] Steve-o or sudo bash without su
[18:30] przemek so i created that file as su
[18:30] calvin I'm seeing a very strange error with PUB/SUB that I can recreate in a simple test case... It basically just involves one subscriber socket subscribing to a single publisher socket 5 times with 5 different strings of 3-5 characters
[18:30] przemek so i guess i need to run the echo command as su as well?
[18:31] Steve-o try this:
[18:31] Steve-o sudo bash
[18:31] Steve-o echo /usr/local/lib > /etc/
[18:31] Steve-o ldconfig
[18:31] Steve-o exit
[18:31] przemek k that worked
[18:31] przemek ok so now I shouldn't need the LD path correct?
[18:33] przemek no, when I run java hwserver I still get an error
[18:39] przemek ok so the hwserver, hwclient ran fine
[18:39] przemek after I added the LD path again
[18:40] przemek hm
[18:40] przemek the weird thing is that when I opened a new terminal and ssh'd into the server
[18:40] przemek and tried to javac and java again i had to redo the exports
[18:40] przemek I thought exports where permanent?
[18:43] guido_g omg
[18:44] guido_g they handed down to sub-processes
[18:44] guido_g so if you export something in a shell, it's visible in the environment of a process started from that shell
[18:45] przemek ok tks
[18:50] Steve-o you could probably put the CLASSPATH in /etc/environment
[18:50] Steve-o I think Debian has an alternative method available but I don't see it in Fedora
[18:53] Steve-o I'm trying Fedora now to see what happens
[19:17] Steve-o wow, that's messed up, java doesn't support settings
[19:20] Steve-o >>
[19:42] streamer REQ-REP socket pairs require lockstep pairs of send-recv. Is there some way to hold open a socket after a request to stream back a series of separate messages? REQ -> REP1, REP2, REP3, ... REPn?
[19:44] streamer And I'm not talking about an inproc Pair socket. I need this to work inter-node.
[19:45] mikko streamer: req/rep isnt suitable for that
[19:45] mikko either DEALER or ROUTER
[19:46] streamer So I should set up a DEALER/DEALER pair?
[19:47] mikko streamer: most likely
[19:47] mikko as req/rep is strict pattern
[19:49] streamer mikko: cool, that seems to work. I would not have expected DEALER/DEALER to work like that. what's the intuition?
[19:50] mikko streamer: not sure i understand the question
[19:51] streamer mikko: why DEALER/DEALER, and not ROUTER/ROUTER, for example. If I read the man page for both ROUTER and DEALER, it says it's bidirectional and send/recv order doesn't matter.
[19:51] streamer so why does ROUTER/ROUTER not work?
[19:51] mikko replace REQ with DEALER
[19:51] mikko and REP with ROUTER
[19:52] mikko the REP needs to be able to route the reply to the peer
[19:52] mikko if you got multiple requesters that is
[19:52] mikko DEALER is fair-queued so the reply might end up to any peer
[19:54] streamer is there any use for the zmq_streamer device in this scenario? (one request responded to with multiple replies)
[20:04] mikko streamer: probably not
[20:05] mikko depends on the scenario
[20:09] CIA-79 pyzmq: 03MinRK 07 * rfb534ce 10/ (3 files in 3 dirs): Merge branch 'version_info' -
[20:09] CIA-79 pyzmq: 03MinRK 07 * r1188668 10/ zmq/eventloop/platform/ : fix 2.5-3.x except syntax in tornado's -
[20:09] CIA-79 pyzmq: 03MinRK 07 * r97c21ac 10/ (README.rst docs/source/changelog.rst docs/source/index.rst): update README/docs ahead of 2.1.10 release -
[21:15] pieterh streamer: DEALER and ROUTER are really different beasts
[21:15] pieterh think of DEALER as PUSH+PULL combined
[21:16] pieterh and ROUTER as the only socket type you can use to address specific connections
[21:28] jhawk28 is the salt dev on?
[21:30] pieterh jhawk28: he'll be on in a second or two...
[21:31] mikko evenin'
[21:32] thatch45 someone looking for me? Thomas Hatch?
[21:32] jhawk28 yep
[21:32] jhawk28 been looking at salt - looks really nice
[21:32] thatch45 thanks!
[21:33] jhawk28 have you thought about a Windows friendly impl?
[21:33] thatch45 yes
[21:33] thatch45 it is not on the top of my list right now
[21:33] thatch45 but it would be possible
[21:33] thatch45 I don't think there is much to change
[21:33] thatch45 although some paths may need to be scrubbed
[21:34] thatch45 do you have something specific in mind?
[21:34] jhawk28 I need a bare minimum version that can remotely execute scripts
[21:34] pieterh mikko: hi!
[21:34] thatch45 ahh
[21:35] pieterh jhawk28: building a botfarm? :-)
[21:35] jhawk28 :)
[21:35] thatch45 it should not need too much modification to run on windows
[21:35] jhawk28 no, we have too many demo servers
[21:35] thatch45 the minion would be easy, the master would be a little harder I think
[21:35] thatch45 but you would really only need to modify the minion
[21:36] jhawk28 I can run the master on linux
[21:36] thatch45 awesome
[21:36] jhawk28 as we have a heterogenous env
[21:36] thatch45 this is acctually the first request for windows so far :)
[21:37] jhawk28 the biggest setup problem was in the m2crypto
[21:37] thatch45 oh?
[21:37] jhawk28 doesnt seem to compile well on windows
[21:37] thatch45 honestly I think it would run just fine so long as the deps are in place and you are willing to clear up a few stack traces
[21:37] thatch45 ouch
[21:38] jhawk28 there is a precompiled version, but its only on 32bit
[21:38] thatch45 heh, figures :)
[21:39] jhawk28 need openssl installed also - so thats a missing dependency
[21:39] jhawk28 Ive done a little python dev so Im not an expert, but I was able to get it compiled
[21:40] jhawk28 but when I ran the salt script, it couldn't import the cli for some reason
[21:41] thatch45 ? import the cli? I don't follow
[21:41] jhawk28 the salt folder was in the site-packages, but it wouldn't work
[21:41] thatch45 the cli module?
[21:41] jhawk28 yes
[21:41] thatch45 ahh
[21:41] thatch45 did it give you a trace?
[21:41] jhawk28 yes, but that was it
[21:41] jhawk28 I then switched to the c: drive and it would import in the repl
[21:42] jhawk28 but not from "python"
[21:42] thatch45 hmm, this is something I would need to see, or at least see more of the python output
[21:43] thatch45 Been a while since I have worked on Windows
[21:43] jhawk28 when I ran the repl from the d: drive, it wouldn't import either
[21:44] jhawk28 I'll try setting it up tonight on my mac and pc
[21:44] jhawk28 the last thing I was working on was to get it installed into a virtualenv so that I could just drop it on the different servers and not install the dependency each time
[21:44] thatch45 if you can get a full stack trace and send it to me that will help a lot
[21:45] jhawk28 I'll also fork the git repo and maybe contribute fixes
[21:45] thatch45 here are the best ways to contact me:
[21:45] thatch45 and fixes are always welcome of course!
[21:46] thatch45 I will have to get a windows vm going so I can test them
[21:48] thatch45 what exact version of windows are you using?
[21:48] jhawk28 win 7
[21:48] jhawk28 64bit
[21:48] thatch45 ok, I will get a copy
[21:49] thatch45 although, I must admit that most of my time right now is going to the salt state system, but I am excited at the prospect of porting Salt to windows!
[21:50] jhawk28 the only other nastiness is if we wanted to use the file server to deploy out zip
[21:51] jhawk28 since things seem to be in the 70mb range
[21:51] thatch45 you think that would be an issue?
[21:51] thatch45 what do you mean?
[21:51] thatch45 the 70 MB range
[21:51] thatch45 I have run it wil DVD iso images
[21:51] jhawk28 so you do chunking?
[21:51] thatch45 yes
[21:52] jhawk28 or just use up a bunch of memory
[21:52] jhawk28 oh, good - I didn't see it
[21:52] jhawk28 but as I said before - havent done much python
[21:52] thatch45
[21:53] thatch45 explains the core of the file server
[21:53] thatch45 the one inside Salt is a little more powerful
[21:53] thatch45 and fully encrypted of course :)
[21:55] jhawk28 yeah
[21:55] jhawk28 if I can get it running under windows, the main issue will be the dependencies
[21:56] jhawk28 but if I can get it working under virtualenv, it will make that much easier
[21:56] thatch45 sounds good
[21:56] jhawk28 which is the main issue with puppet and chef
[21:56] thatch45 keep me posted!
[21:56] thatch45 the windows support?
[21:57] jhawk28 puppet is a little better on the dependencies with their enterprise version
[21:57] jhawk28 neither have windows support yet - they are getting closer
[21:57] thatch45 yes, I want to bundle the deps for platforms
[21:57] jhawk28 but they still are much more complicated
[21:57] jhawk28 salt feels much cleaner
[21:58] thatch45 yes, Salt can do almost everything puppet can do now but it is much less complicated
[21:58] jhawk28 and python is better than ruby
[21:58] jhawk28 ;)
[21:58] thatch45 porting it to windows should be much less of an issue
[21:58] thatch45 heh, yes, I like python much more than Ruby
[21:58] thatch45 although Ruby is a great language :)
[21:58] jhawk28 thanks for stopping in, Ill keep you posted
[21:59] jhawk28 if the interfaces were documented, I could also make a Java minion
[22:00] jhawk28 but first things first
[22:00] thatch45 yes, the problem is that it uses python pickles in the serialization
[22:00] thatch45 2.0 will switch to json
[22:00] jhawk28 bson may be a little more efficient
[22:01] thatch45 and I will find time to better document the interface :)
[22:01] jhawk28 or you could go protobufs
[22:01] thatch45 you think so?
[22:01] thatch45 I like bson a little more than protobufs
[22:01] jhawk28 bson is more efficient than json
[22:01] thatch45 I will have to evaluate them for the purpose at hand
[22:01] thatch45 efficentcy, size and speed are the primary concerns
[22:01] jhawk28 I seem to prefer protobuf because it forces you to define your messages
[22:02] jhawk28 protobuf is more efficient because it just sends the data, the schema stays in the protobuf descriptor
[22:02] jhawk28 whereas json and bson send the schema as part of the payload
[22:03] jhawk28 thift is the same as protobuf
[22:03] jhawk28 avro is kind of a merger
[22:04] tjholowaychuk "ØMQ ensures atomic delivery of messages; peers shall receive either all message parts of a message or none at all."
[22:04] tjholowaychuk wont that make streaming a little difficult?
[22:04] thatch45 good info
[22:05] jhawk28 tj: yes
[22:05] tjholowaychuk :(
[22:05] jhawk28 tj: but zmq is not trying to do streaming
[22:05] jhawk28 if you want, you send multiple messages
[22:05] jhawk28 and that will "stream"
[22:05] tjholowaychuk right
[22:05] tjholowaychuk good call
[22:06] tjholowaychuk too obvious
[22:06] jhawk28 but if you need to send super large payloads, you need to chunk them
[22:06] tjholowaychuk loving it so far, fantastic piece of work, thanks for the tip
[22:06] jhawk28 like thatch45 found out:
[22:07] tjholowaychuk so SNDMORE is really just so you dont have to manually buffer
[22:07] tjholowaychuk and send a large message?
[22:08] thatch45 I don't really understand SNDMORE, since it seems to still send in a single message
[22:08] thatch45 I need to refresh my memory on why that was not the right way to do it
[22:08] tjholowaychuk im just curious for use-cases, im sure i'll find out the more i use zmq
[22:19] tjholowaychuk a pair is the one most like a tcp connection right?
[22:27] jhawk28 SNDMORE is just a flag that is used to separate parts of a message.
[22:27] jhawk28 think if a message as an array of bytes
[22:29] jhawk28 the message isn't sent until you use a 0 flag
[22:29] jhawk28 if I remember correctly
[22:38] jhawk28 nope, it looks like it sends each separate parts if I am reading the source right. its the reaper that makes the receipt atomic
[22:38] jhawk28 so it fills up memory on the receiving side
[22:45] tjholowaychuk for chunks of an http request you would have to tag it right?
[22:46] jhawk28 yes, so that you can keep track of them to put them back together
[22:46] tjholowaychuk k, just making sure im not mad
[22:46] tjholowaychuk haha
[22:46] jhawk28 well, you may still be
[22:47] tjholowaychuk perhaps
[22:55] streamer Should the following setup work?? Dealer client tcp <-> [tcp |Queue| inprocess] <-> inprocess Dealer worker
[22:58] cactuss there isn't a way to have zeromq buffer publisher messages onto disk during times when it has no subscribers, and then guarantee that subsequent scribers will get that data eventually given enough disk space is there?
[22:59] tjholowaychuk i should probably read first but does zmq have any load introspection capabilities available through the public api?
[23:00] jhawk28 cactuss: no - 0mq sees the pub/sub pattern like a radio
[23:01] jhawk28 tj: not much - you have the HWM, but you are not notified if it drops messages
[23:01] cactuss are there any known projects that buffer to disk? rabbit mq etc? thanks jhawk28
[23:02] jhawk28 kestrel, rabbit, kafka, etc
[23:03] jhawk28 kafka will be the most complicated, but it was designed with scalable persistence in mind
[23:04] jhawk28 kestrel is probably the simplest with Rabbit in the middle
[23:04] jhawk28 also, be sure to look at the titanic pattern from the guide
[23:05] Guest75658 is a single socket threadsafe when it comes to polling?
[23:06] jhawk28 sockets are not threadsafe
[23:06] Guest75658 as in, it isn't safe to send/recv in different threads, but is it safe to poll/send?
[23:06] jhawk28 probably not
[23:07] Guest75658 ok cool, thanks
[23:07] jhawk28 its not safe, but you can always try it
[23:07] jhawk28 it usually core dumps if it gets out of state
[23:07] Guest75658 yeah, tracking down an assertion failure that looks like a thread safety issue
[23:08] jhawk28 you may want to use inproc sockets for each thread and then use a reactor to put the outgoing data on external one
[23:09] jhawk28 or you can cheat and just use a mutex
[23:16] cactuss oh neat going to read up jhawk28
[23:18] jhawk28 the easiest way to get persistence is to do it yourself. if you want to guarantee no message loss, use a sequence number on the data.
[23:18] jhawk28 the client then can keep track of the position
[23:19] jhawk28 a second XREP would be needed so that the clients can request the missing messages
[23:19] jhawk28 its basically what kafka does. they have a nice design doc that explains what they do
[23:20] jhawk28 you would then just need to store the data on the disk with the ability to clean it later
[23:20] jhawk28 titanic uses a file per message approach
[23:21] cactuss 1 file per message would be devestating for me
[23:21] jhawk28 if you want a "persistance library", you can look at leveldb or bitcask
[23:21] cactuss yeah familiar with those
[23:21] jhawk28 then you just persist the data in a folder
[23:22] cactuss we are using s3 as a persistence middle layer at the moment
[23:22] cactuss well we're still designing this part of the system
[23:22] cactuss so we're not really doing anything yet
[23:22] cactuss but we have ids and many (millions) messages per id, and 1-100 ids per day
[23:22] jhawk28 bitcask is faster, but it would need to store all keys in memory. leveldb doesnt have that limitation
[23:22] cactuss and we want workers to be able to look at 'all' of the messages from each id
[23:23] cactuss so we're processing data into files by id, sending to s3 and anticipating having a worker pull from there
[23:23] cactuss but the synchronization is a little messy / unclear
[23:23] mikko cactuss: do you actually need a queue or a database?
[23:23] cactuss we probably just need map reduce
[23:24] cactuss we're collecting data, and then eventually generating analytics on the data
[23:24] jhawk28 is it a batch approach?
[23:24] cactuss yeah
[23:24] jhawk28 or do want realtime?
[23:24] cactuss it's a matter of coordinating the batches
[23:24] mikko hbase might work for you
[23:24] cactuss we don't really need message level granularity
[23:24] jhawk28 batch == mapreduce
[23:24] cactuss we have like 100-1000 bytes per message so message level granularity is mostly out of the picture
[23:24] cactuss and we're doing everything we can to avoid needing random / arbitrary access
[23:25] jhawk28 if you want to process it in realtime, look into storm
[23:25] cactuss batch is fine, i think we simply want map reduce, it's more of an implementation issue
[23:25] cactuss for hadoop we want to chunk our files to 64MBs
[23:26] cactuss and we want to be able to keep track of all of our files, and not need a standing persistent hadoop cluster
[23:59] CIA-79 pyzmq: 03MinRK 07 * r2e5519e 10/ zmq/eventloop/ : upstream tornado fix for PeriodicCallback ...
[23:59] CIA-79 pyzmq: 03MinRK 07 * r503b9e7 10/ zmq/core/version.pyx : version bump to 2.1.10 -