| [Time] Name | Message |
|
[01:49] CIA-79
|
pyzmq: 03MinRK 07 * r4d74a9c 10/ MANIFEST.in : exclude mailmap from manifest - http://git.io/qslltQ
|
|
[01:49] CIA-79
|
pyzmq: 03MinRK 07 * r530edba 10/ zmq/eventloop/zmqstream.py : small bugfixes in ZMQStream.flush ...
|
|
[03:06] streamer
|
is it possible to use 0mq for streaming messages, i.e. req -> rep1, rep2, rep3, ... repN ?
|
|
[04:14] cactuss
|
is there a way to have zeromq buffer publisher messages onto disk during times when it has no subscribers, and then guarantee that subsequent scribers will get that data eventually given enough disk space?
|
|
[04:22] cactuss
|
i'm wondering if zeromq can save me from having to use S3 as an intermediary data placeholder
|
|
[13:54] cremes
|
cactuss: no, 0mq doesn't have that capability built-in; you would need to build it on top
|
|
[14:09] CIA-79
|
libzmq: 03Martin Sustrik 07 * r8ec3840 10/ (src/session_base.cpp src/socket_base.cpp src/socket_base.hpp): Dummy dns:// transport added. ...
|
|
[16:25] przemeklach
|
Is there a difference between Java classpath and java.library.path
|
|
[16:25] przemeklach
|
I'm just going through the jzmq installation tutorial
|
|
[16:26] przemeklach
|
and I'm trying to compile one of the examples and I'm getting an error which I assume is due to the fact that my java path is not set correctly
|
|
[16:50] Steve-o
|
java.library.path is the JNI
|
|
[16:50] Steve-o
|
classpath is the JAR/class files
|
|
[16:52] przemek
|
ok, so i set the classpath to be where the code is set CLASSPATH=/home/camera/source/zguide/examples/Java/
|
|
[16:52] przemek
|
when i try to javac hwserver.java I still get the error
|
|
[16:53] przemek
|
I know i have to do the -Djava.library etc but I need to compile first don't i?
|
|
[16:54] Steve-o
|
you need a dll or so file yes
|
|
[16:55] przemek
|
right but I'm getting an error when I try to compile
|
|
[16:55] przemek
|
javac hwserver.java
|
|
[16:55] przemek
|
I'm not even trying to run it yet
|
|
[16:56] przemek
|
when I type java -Djava.library etc that is to run, right?
|
|
[16:56] Steve-o
|
there should be a libzmq.jar or similar and a libjzmq.so I think before you start building apps
|
|
[16:57] Steve-o
|
1) build 0mq, 2) build jzmq, 3) build java examples
|
|
[16:58] przemek
|
Ya I followed the instructions for installing 0mq and jzmq
|
|
[16:58] przemek
|
libzmq.jar dne on my system
|
|
[16:58] przemek
|
libjzmq.so does
|
|
[16:58] Steve-o
|
I'll have a look now as I've never tried the Java bindings
|
|
[16:58] przemek
|
ok tks for your help
|
|
[16:59] Steve-o
|
Debian or RH?
|
|
[16:59] przemek
|
RH
|
|
[16:59] przemek
|
I assume RH = red hat
|
|
[17:00] Steve-o
|
yup or Fedora
|
|
[17:01] przemek
|
whoops, i mean Scientific Linux
|
|
[17:01] przemek
|
sry
|
|
[17:01] Steve-o
|
same thing pretty much
|
|
[17:01] przemek
|
k
|
|
[17:01] Steve-o
|
RHEL = CentOS = Scientific Linux
|
|
[17:01] przemek
|
k
|
|
[17:02] przemek
|
and I'm using zmq v2.1 btw
|
|
[17:02] Steve-o
|
I'm trying 2.1.10 now
|
|
[17:02] przemek
|
k
|
|
[17:06] Steve-o
|
ok 0mq built, one bug with the make system found, next step jzmq
|
|
[17:06] przemek
|
k
|
|
[17:07] Steve-o
|
are you using master or a download?
|
|
[17:10] przemek
|
i git cloned from github
|
|
[17:11] Steve-o
|
ok, I'll try openjdk 7
|
|
[17:12] przemek
|
hm i think i'm using oracle java
|
|
[17:12] przemek
|
not openjdk
|
|
[17:12] Steve-o
|
I'm on a Debian VM first, not much choice
|
|
[17:12] przemek
|
k
|
|
[17:16] Steve-o
|
ok that's built
|
|
[17:17] Steve-o
|
I now have x64/share/java/zmq.jar & x64/lib/libjzmq.so.0.0.0
|
|
[17:18] przemek
|
what the heck; i just searched for zmq.jar and it found it this time
|
|
[17:18] przemek
|
blh
|
|
[17:18] przemek
|
ok
|
|
[17:18] przemek
|
so they are both in different dir's than yours but they do exist on my system; the so and the jar
|
|
[17:19] przemek
|
'/usr/local/share/java/zmq.jar'
|
|
[17:19] przemek
|
and
|
|
[17:20] przemek
|
'/usr/local/lib/libjzmq.so'
|
|
[17:20] Steve-o
|
I'm using a custom prefix for install
|
|
[17:20] przemek
|
oh ok
|
|
[17:26] przemek
|
so I also downloaded the zguide
|
|
[17:26] przemek
|
and what i've been trying to get to work is to compile and run the hwserver.java example
|
|
[17:26] przemek
|
and that is where I'm having the issues
|
|
[17:27] przemek
|
the example is in zguide/examples/Java
|
|
[17:28] Steve-o
|
what an ugly command line:
|
|
[17:28] Steve-o
|
LD_LIBRARY_PATH=/home/local/zeromq/x64/lib java "-Djava.library.path=/home/local/zeromq/jzmq/x64/lib" -classpath /home/local/zeromq/jzmq/x64/share/java/zmq.jar:. local_lat tcp://127.0.0.1:5555 1 100
|
|
[17:28] przemek
|
omg
|
|
[17:28] przemek
|
ok
|
|
[17:28] Steve-o
|
three libraries required
|
|
[17:29] Steve-o
|
zmq.jar, libjzmq.so and libzmq.so
|
|
[17:29] Steve-o
|
classpath, java.library.path, and ld_library_path
|
|
[17:29] przemek
|
ok
|
|
[17:29] przemek
|
so if I have a hwserver.java file
|
|
[17:30] Steve-o
|
javac -cp /home/local/zeromq/jzmq/x64/share/java/zmq.jar:. hwserver for the build
|
|
[17:30] Steve-o
|
you don't need JNI or native libraries for that step
|
|
[17:30] przemek
|
ok tks for ur help
|
|
[17:35] Steve-o
|
I think the Maven build is supposed to make it easier
|
|
[17:35] Steve-o
|
I'lm trying that now
|
|
[17:49] przemek
|
k
|
|
[17:50] Steve-o
|
lol maven is a mess
|
|
[17:51] przemek
|
lol
|
|
[17:53] Steve-o
|
with "mvn jar:jar" you can get a jar with bundled jni lib
|
|
[17:54] Steve-o
|
so the command line shortens a little, only a little:
|
|
[17:54] Steve-o
|
LD_LIBRARY_PATH=/home/local/zeromq/x64/lib java -classpath /me/local/zeromq/jzmq/jzmq/target/jzmq-2.1.0-SNAPSHOT.jar:. local_lat tcp://127.0.0.1:5555 1
|
|
[17:55] Steve-o
|
If you install the 0mq package then you don't need the library path, just the class path
|
|
[17:56] przemek
|
but if I set the classpath
|
|
[17:56] przemek
|
then I should be able to just run my class via
|
|
[17:56] przemek
|
java local_lat
|
|
[17:56] przemek
|
?
|
|
[17:56] Steve-o
|
looks like it
|
|
[17:57] przemek
|
what is the command to see what my class path is
|
|
[17:58] przemek
|
echo ...
|
|
[17:59] Steve-o
|
echo $CLASSPATH
|
|
[17:59] przemek
|
ya i tried that
|
|
[17:59] przemek
|
and it returns nothing
|
|
[18:00] Steve-o
|
because no default classpath
|
|
[18:00] Steve-o
|
export CLASSPATH=..../target/jzmq-2.1.0-SNAPSHOT.jar:.
|
|
[18:00] Steve-o
|
then, java local_lat
|
|
[18:01] Steve-o
|
yup, works great.
|
|
[18:02] Steve-o
|
I'm doing:
|
|
[18:02] Steve-o
|
export CLASSPATH=/home/local/zeromq/jzmq/jzmq/target/jzmq-2.1.0-SNAPSHOT.jar:.
|
|
[18:02] Steve-o
|
export LD_LIBRARY_PATH=/home/local/zeromq/x64/lib
|
|
[18:02] Steve-o
|
java local_lat
|
|
[18:04] przemek
|
having a hard time finding these paths
|
|
[18:04] przemek
|
like
|
|
[18:04] przemek
|
I should be poiting my paths based on where zeromq and jzmq where installed
|
|
[18:04] Steve-o
|
that classpath is pointing to the output of maven2
|
|
[18:04] przemek
|
not to the original source folder I downloaded right
|
|
[18:05] Steve-o
|
the ld path to where 0mq lib is living
|
|
[18:05] Steve-o
|
but you stated: '/usr/local/lib/libjzmq.so'
|
|
[18:05] przemek
|
yes
|
|
[18:05] Steve-o
|
so you should not need to specify ld-library-path
|
|
[18:05] Steve-o
|
only classpath
|
|
[18:05] Steve-o
|
have you built via maven?
|
|
[18:05] przemek
|
no
|
|
[18:05] przemek
|
i have no idea how to do that
|
|
[18:06] przemek
|
i just followed the instrutions for installing 0mq and jzmq
|
|
[18:06] Steve-o
|
if you have maven installed: sudo yum install maven2 I guess
|
|
[18:06] Steve-o
|
then run: mvn jar:jar
|
|
[18:06] Steve-o
|
then you get the jni+jar file bundle you can specify in the classpath
|
|
[18:06] przemek
|
and from within which dir do I run that comand
|
|
[18:07] Steve-o
|
in the jzmq directory with pom.xml
|
|
[18:08] Steve-o
|
...
|
|
[18:08] Steve-o
|
why not try this:
|
|
[18:08] Steve-o
|
export CLASSPATH=/usr/local/share/java/zmq.jar:.
|
|
[18:08] przemek
|
fighting to get maven installed
|
|
[18:08] Steve-o
|
javac hwserver
|
|
[18:08] Steve-o
|
java hwserver
|
|
[18:10] Steve-o
|
java should be picking up both native libraries automatically
|
|
[18:10] przemek
|
ok i'll let you know once i get maven installed
|
|
[18:10] przemek
|
just doing yum install is not finding it
|
|
[18:11] Steve-o
|
skip that for a bit as I think you don't need it
|
|
[18:11] przemek
|
ok
|
|
[18:12] Steve-o
|
you have JNI & native library in /usr/local/lib the files libzmq.so and libjzmq.so
|
|
[18:12] Steve-o
|
these shouldn't need any effort to use
|
|
[18:13] Steve-o
|
the only one needed attention is /usr/local/share/java/zmq.jar
|
|
[18:13] Steve-o
|
so find your example: hwserver.java
|
|
[18:13] przemek
|
ok
|
|
[18:13] Steve-o
|
try to build it:
|
|
[18:13] Steve-o
|
export CLASSPATH=/usr/local/share/java/zmq.jar:.
|
|
[18:13] Steve-o
|
javac hwserver
|
|
[18:13] przemek
|
do i need the ':.' at the end?
|
|
[18:14] przemek
|
and what does that mean?
|
|
[18:14] Steve-o
|
it means also search the current directory, i.e. for hwserver.java
|
|
[18:14] Steve-o
|
Java is a bit stupid, but I think they've fixed it a bit
|
|
[18:15] Steve-o
|
classpath specifies jar files and directories containing .class files
|
|
[18:15] przemek
|
ok so compiling works awesome now
|
|
[18:15] Steve-o
|
now try running
|
|
[18:15] przemek
|
however
|
|
[18:15] Steve-o
|
java hwserver
|
|
[18:15] przemek
|
java hwserver
|
|
[18:15] przemek
|
gives errors
|
|
[18:15] przemek
|
Exception in thread "main" java.lang.UnsatisfiedLinkError: no jzmq in java.library.path
|
|
[18:15] przemek
|
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
|
|
[18:15] przemek
|
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
|
|
[18:15] przemek
|
at java.lang.System.loadLibrary(System.java:1028)
|
|
[18:15] przemek
|
at org.zeromq.ZMQ.<clinit>(ZMQ.java:34)
|
|
[18:15] przemek
|
at hwserver.main(hwserver.java:13)
|
|
[18:16] Steve-o
|
try this then:
|
|
[18:16] Steve-o
|
export LD_LIBRARY_PATH=/usr/local/lib
|
|
[18:17] queuenoob
|
Hi all, wondering if anyone can point me in the right direction: with 0MQ, is there a way to have a subset of message within a single queue be "correlated" in that for a specific message to be served, any prior correlated queue message must have been served and received an ack indicating it was fully processed. If it hasn't then have the queue serve the next message.. does this concept actually exist in any queuing systems?
|
|
[18:17] Steve-o
|
przemek_: actually just running "ldconfig" should bring in the library
|
|
[18:17] przemek
|
that did it!
|
|
[18:17] Steve-o
|
on debian /etc/ld.so.conf.d/libc.conf contains /usr/local/lib
|
|
[18:17] Steve-o
|
so the library database just needs updating
|
|
[18:18] Steve-o
|
so, run "sudo ldconfig"
|
|
[18:18] przemek
|
ya just did
|
|
[18:18] przemek
|
but i already ran the LD_Library one
|
|
[18:18] Steve-o
|
then wipe the env variable: export LD_LIBRARY_PATH=
|
|
[18:18] Steve-o
|
and it should also work
|
|
[18:19] przemek
|
no
|
|
[18:19] przemek
|
doesn't work if i remove LD
|
|
[18:19] przemek
|
even after running ldconfig
|
|
[18:19] przemek
|
well that's fine
|
|
[18:19] Steve-o
|
it might be a weird default setting in /etc/ld.so.conf
|
|
[18:20] Steve-o
|
usually it should pull in /usr/local/lib
|
|
[18:20] przemek
|
so I just need to add the classpath and the LD path and I'm good
|
|
[18:20] przemek
|
so there is only one line in that file
|
|
[18:20] Steve-o
|
let me check Fedora settings
|
|
[18:20] przemek
|
include ld.so.conf.d/*.conf
|
|
[18:20] przemek
|
k
|
|
[18:23] Steve-o
|
queuenoob: that sounds pretty much limited to websphere mq
|
|
[18:24] Steve-o
|
queuenoob: but also not much difference from a req/rep socket
|
|
[18:25] Steve-o
|
przemek_: so Fedora is empty, just create a new file:
|
|
[18:25] przemek
|
wel my isn't empty
|
|
[18:25] przemek
|
include ld.so.conf.d/*.conf
|
|
[18:25] Steve-o
|
echo /usr/local/lib > /etc/ld.so.conf.d/local.conf
|
|
[18:25] przemek
|
it has the line in it
|
|
[18:26] Steve-o
|
empty as in no specification of /usr/local, only llvm & xulrunner
|
|
[18:26] queuenoob
|
Steve-o: Thanks for the reply :) I'll check into that
|
|
[18:26] Steve-o
|
queuenoob: the critical point is reliability requirement
|
|
[18:28] przemek
|
it's telling me permission denied
|
|
[18:28] przemek
|
even when i run as sudo
|
|
[18:28] Steve-o
|
przemek_: so create a new file, say local.conf, then re-run ldconfig and will be good to go
|
|
[18:28] Steve-o
|
su first
|
|
[18:28] Steve-o
|
redirect on sudo will lose the permissions for writing
|
|
[18:28] Steve-o
|
or sudo bash without su
|
|
[18:30] przemek
|
so i created that file as su
|
|
[18:30] calvin
|
I'm seeing a very strange error with PUB/SUB that I can recreate in a simple test case... It basically just involves one subscriber socket subscribing to a single publisher socket 5 times with 5 different strings of 3-5 characters
|
|
[18:30] przemek
|
so i guess i need to run the echo command as su as well?
|
|
[18:31] Steve-o
|
try this:
|
|
[18:31] Steve-o
|
sudo bash
|
|
[18:31] Steve-o
|
echo /usr/local/lib > /etc/ld.so.conf.d/local.conf
|
|
[18:31] Steve-o
|
ldconfig
|
|
[18:31] Steve-o
|
exit
|
|
[18:31] przemek
|
k that worked
|
|
[18:31] przemek
|
ok so now I shouldn't need the LD path correct?
|
|
[18:33] przemek
|
no, when I run java hwserver I still get an error
|
|
[18:39] przemek
|
ok so the hwserver, hwclient ran fine
|
|
[18:39] przemek
|
after I added the LD path again
|
|
[18:40] przemek
|
hm
|
|
[18:40] przemek
|
the weird thing is that when I opened a new terminal and ssh'd into the server
|
|
[18:40] przemek
|
and tried to javac and java again i had to redo the exports
|
|
[18:40] przemek
|
I thought exports where permanent?
|
|
[18:43] guido_g
|
omg
|
|
[18:44] guido_g
|
they handed down to sub-processes
|
|
[18:44] guido_g
|
so if you export something in a shell, it's visible in the environment of a process started from that shell
|
|
[18:45] przemek
|
ok tks
|
|
[18:50] Steve-o
|
you could probably put the CLASSPATH in /etc/environment
|
|
[18:50] Steve-o
|
I think Debian has an alternative method available but I don't see it in Fedora
|
|
[18:53] Steve-o
|
I'm trying Fedora now to see what happens
|
|
[19:17] Steve-o
|
wow, that's messed up, java doesn't support ld.so.conf settings
|
|
[19:20] Steve-o
|
>> http://lists.fedoraproject.org/pipermail/java-devel/2011-March/004153.html
|
|
[19:42] streamer
|
REQ-REP socket pairs require lockstep pairs of send-recv. Is there some way to hold open a socket after a request to stream back a series of separate messages? REQ -> REP1, REP2, REP3, ... REPn?
|
|
[19:44] streamer
|
And I'm not talking about an inproc Pair socket. I need this to work inter-node.
|
|
[19:45] mikko
|
streamer: req/rep isnt suitable for that
|
|
[19:45] mikko
|
either DEALER or ROUTER
|
|
[19:46] streamer
|
So I should set up a DEALER/DEALER pair?
|
|
[19:47] mikko
|
streamer: most likely
|
|
[19:47] mikko
|
as req/rep is strict pattern
|
|
[19:49] streamer
|
mikko: cool, that seems to work. I would not have expected DEALER/DEALER to work like that. what's the intuition?
|
|
[19:50] mikko
|
streamer: not sure i understand the question
|
|
[19:51] streamer
|
mikko: why DEALER/DEALER, and not ROUTER/ROUTER, for example. If I read the man page for both ROUTER and DEALER, it says it's bidirectional and send/recv order doesn't matter.
|
|
[19:51] streamer
|
so why does ROUTER/ROUTER not work?
|
|
[19:51] mikko
|
replace REQ with DEALER
|
|
[19:51] mikko
|
and REP with ROUTER
|
|
[19:52] mikko
|
the REP needs to be able to route the reply to the peer
|
|
[19:52] mikko
|
if you got multiple requesters that is
|
|
[19:52] mikko
|
DEALER is fair-queued so the reply might end up to any peer
|
|
[19:54] streamer
|
is there any use for the zmq_streamer device in this scenario? (one request responded to with multiple replies)
|
|
[20:04] mikko
|
streamer: probably not
|
|
[20:05] mikko
|
depends on the scenario
|
|
[20:09] CIA-79
|
pyzmq: 03MinRK 07 * rfb534ce 10/ (3 files in 3 dirs): Merge branch 'version_info' - http://git.io/RwtUvw
|
|
[20:09] CIA-79
|
pyzmq: 03MinRK 07 * r1188668 10/ zmq/eventloop/platform/windows.py : fix 2.5-3.x except syntax in tornado's platform.windows - http://git.io/fIMlCQ
|
|
[20:09] CIA-79
|
pyzmq: 03MinRK 07 * r97c21ac 10/ (README.rst docs/source/changelog.rst docs/source/index.rst): update README/docs ahead of 2.1.10 release - http://git.io/UHRbbg
|
|
[21:15] pieterh
|
streamer: DEALER and ROUTER are really different beasts
|
|
[21:15] pieterh
|
think of DEALER as PUSH+PULL combined
|
|
[21:16] pieterh
|
and ROUTER as the only socket type you can use to address specific connections
|
|
[21:28] jhawk28
|
is the salt dev on?
|
|
[21:30] pieterh
|
jhawk28: he'll be on in a second or two...
|
|
[21:31] mikko
|
evenin'
|
|
[21:32] thatch45
|
someone looking for me? Thomas Hatch?
|
|
[21:32] jhawk28
|
yep
|
|
[21:32] jhawk28
|
been looking at salt - looks really nice
|
|
[21:32] thatch45
|
thanks!
|
|
[21:33] jhawk28
|
have you thought about a Windows friendly impl?
|
|
[21:33] thatch45
|
yes
|
|
[21:33] thatch45
|
it is not on the top of my list right now
|
|
[21:33] thatch45
|
but it would be possible
|
|
[21:33] thatch45
|
I don't think there is much to change
|
|
[21:33] thatch45
|
although some paths may need to be scrubbed
|
|
[21:34] thatch45
|
do you have something specific in mind?
|
|
[21:34] jhawk28
|
I need a bare minimum version that can remotely execute scripts
|
|
[21:34] pieterh
|
mikko: hi!
|
|
[21:34] thatch45
|
ahh
|
|
[21:35] pieterh
|
jhawk28: building a botfarm? :-)
|
|
[21:35] jhawk28
|
:)
|
|
[21:35] thatch45
|
it should not need too much modification to run on windows
|
|
[21:35] jhawk28
|
no, we have too many demo servers
|
|
[21:35] thatch45
|
the minion would be easy, the master would be a little harder I think
|
|
[21:35] thatch45
|
but you would really only need to modify the minion
|
|
[21:36] jhawk28
|
I can run the master on linux
|
|
[21:36] thatch45
|
awesome
|
|
[21:36] jhawk28
|
as we have a heterogenous env
|
|
[21:36] thatch45
|
this is acctually the first request for windows so far :)
|
|
[21:37] jhawk28
|
the biggest setup problem was in the m2crypto
|
|
[21:37] thatch45
|
oh?
|
|
[21:37] jhawk28
|
doesnt seem to compile well on windows
|
|
[21:37] thatch45
|
honestly I think it would run just fine so long as the deps are in place and you are willing to clear up a few stack traces
|
|
[21:37] thatch45
|
ouch
|
|
[21:38] jhawk28
|
there is a precompiled version, but its only on 32bit
|
|
[21:38] thatch45
|
heh, figures :)
|
|
[21:39] jhawk28
|
need openssl installed also - so thats a missing dependency
|
|
[21:39] jhawk28
|
Ive done a little python dev so Im not an expert, but I was able to get it compiled
|
|
[21:40] jhawk28
|
but when I ran the salt script, it couldn't import the cli for some reason
|
|
[21:41] thatch45
|
? import the cli? I don't follow
|
|
[21:41] jhawk28
|
the salt folder was in the site-packages, but it wouldn't work
|
|
[21:41] thatch45
|
the cli module?
|
|
[21:41] jhawk28
|
yes
|
|
[21:41] thatch45
|
ahh
|
|
[21:41] thatch45
|
did it give you a trace?
|
|
[21:41] jhawk28
|
yes, but that was it
|
|
[21:41] jhawk28
|
I then switched to the c: drive and it would import in the repl
|
|
[21:42] jhawk28
|
but not from "python salt.py"
|
|
[21:42] thatch45
|
hmm, this is something I would need to see, or at least see more of the python output
|
|
[21:43] thatch45
|
Been a while since I have worked on Windows
|
|
[21:43] jhawk28
|
when I ran the repl from the d: drive, it wouldn't import either
|
|
[21:44] jhawk28
|
I'll try setting it up tonight on my mac and pc
|
|
[21:44] jhawk28
|
the last thing I was working on was to get it installed into a virtualenv so that I could just drop it on the different servers and not install the dependency each time
|
|
[21:44] thatch45
|
if you can get a full stack trace and send it to me that will help a lot
|
|
[21:45] jhawk28
|
I'll also fork the git repo and maybe contribute fixes
|
|
[21:45] thatch45
|
here are the best ways to contact me: http://saltstack.org/topics/community.html
|
|
[21:45] thatch45
|
and fixes are always welcome of course!
|
|
[21:46] thatch45
|
I will have to get a windows vm going so I can test them
|
|
[21:48] thatch45
|
what exact version of windows are you using?
|
|
[21:48] jhawk28
|
win 7
|
|
[21:48] jhawk28
|
64bit
|
|
[21:48] thatch45
|
ok, I will get a copy
|
|
[21:49] thatch45
|
although, I must admit that most of my time right now is going to the salt state system, but I am excited at the prospect of porting Salt to windows!
|
|
[21:50] jhawk28
|
the only other nastiness is if we wanted to use the file server to deploy out zip
|
|
[21:51] jhawk28
|
since things seem to be in the 70mb range
|
|
[21:51] thatch45
|
you think that would be an issue?
|
|
[21:51] thatch45
|
what do you mean?
|
|
[21:51] thatch45
|
the 70 MB range
|
|
[21:51] thatch45
|
I have run it wil DVD iso images
|
|
[21:51] jhawk28
|
so you do chunking?
|
|
[21:51] thatch45
|
yes
|
|
[21:52] jhawk28
|
or just use up a bunch of memory
|
|
[21:52] jhawk28
|
oh, good - I didn't see it
|
|
[21:52] jhawk28
|
but as I said before - havent done much python
|
|
[21:52] thatch45
|
http://red45.wordpress.com/2011/05/19/zeromq-file-server-better/
|
|
[21:53] thatch45
|
explains the core of the file server
|
|
[21:53] thatch45
|
the one inside Salt is a little more powerful
|
|
[21:53] thatch45
|
and fully encrypted of course :)
|
|
[21:55] jhawk28
|
yeah
|
|
[21:55] jhawk28
|
if I can get it running under windows, the main issue will be the dependencies
|
|
[21:56] jhawk28
|
but if I can get it working under virtualenv, it will make that much easier
|
|
[21:56] thatch45
|
sounds good
|
|
[21:56] jhawk28
|
which is the main issue with puppet and chef
|
|
[21:56] thatch45
|
keep me posted!
|
|
[21:56] thatch45
|
the windows support?
|
|
[21:57] jhawk28
|
puppet is a little better on the dependencies with their enterprise version
|
|
[21:57] jhawk28
|
neither have windows support yet - they are getting closer
|
|
[21:57] thatch45
|
yes, I want to bundle the deps for platforms
|
|
[21:57] jhawk28
|
but they still are much more complicated
|
|
[21:57] jhawk28
|
salt feels much cleaner
|
|
[21:58] thatch45
|
yes, Salt can do almost everything puppet can do now but it is much less complicated
|
|
[21:58] jhawk28
|
and python is better than ruby
|
|
[21:58] jhawk28
|
;)
|
|
[21:58] thatch45
|
porting it to windows should be much less of an issue
|
|
[21:58] thatch45
|
heh, yes, I like python much more than Ruby
|
|
[21:58] thatch45
|
although Ruby is a great language :)
|
|
[21:58] jhawk28
|
thanks for stopping in, Ill keep you posted
|
|
[21:59] jhawk28
|
if the interfaces were documented, I could also make a Java minion
|
|
[22:00] jhawk28
|
but first things first
|
|
[22:00] thatch45
|
yes, the problem is that it uses python pickles in the serialization
|
|
[22:00] thatch45
|
2.0 will switch to json
|
|
[22:00] jhawk28
|
bson may be a little more efficient
|
|
[22:01] thatch45
|
and I will find time to better document the interface :)
|
|
[22:01] jhawk28
|
or you could go protobufs
|
|
[22:01] thatch45
|
you think so?
|
|
[22:01] thatch45
|
I like bson a little more than protobufs
|
|
[22:01] jhawk28
|
bson is more efficient than json
|
|
[22:01] thatch45
|
I will have to evaluate them for the purpose at hand
|
|
[22:01] thatch45
|
efficentcy, size and speed are the primary concerns
|
|
[22:01] jhawk28
|
I seem to prefer protobuf because it forces you to define your messages
|
|
[22:02] jhawk28
|
protobuf is more efficient because it just sends the data, the schema stays in the protobuf descriptor
|
|
[22:02] jhawk28
|
whereas json and bson send the schema as part of the payload
|
|
[22:03] jhawk28
|
thift is the same as protobuf
|
|
[22:03] jhawk28
|
avro is kind of a merger
|
|
[22:04] tjholowaychuk
|
"ÃMQ ensures atomic delivery of messages; peers shall receive either all message parts of a message or none at all."
|
|
[22:04] tjholowaychuk
|
wont that make streaming a little difficult?
|
|
[22:04] thatch45
|
good info
|
|
[22:05] jhawk28
|
tj: yes
|
|
[22:05] tjholowaychuk
|
:(
|
|
[22:05] jhawk28
|
tj: but zmq is not trying to do streaming
|
|
[22:05] jhawk28
|
if you want, you send multiple messages
|
|
[22:05] jhawk28
|
and that will "stream"
|
|
[22:05] tjholowaychuk
|
right
|
|
[22:05] tjholowaychuk
|
good call
|
|
[22:06] tjholowaychuk
|
too obvious
|
|
[22:06] jhawk28
|
but if you need to send super large payloads, you need to chunk them
|
|
[22:06] tjholowaychuk
|
loving it so far, fantastic piece of work, thanks for the tip
|
|
[22:06] jhawk28
|
like thatch45 found out: http://red45.wordpress.com/2011/05/19/zeromq-file-server-better/
|
|
[22:07] tjholowaychuk
|
so SNDMORE is really just so you dont have to manually buffer
|
|
[22:07] tjholowaychuk
|
and send a large message?
|
|
[22:08] thatch45
|
I don't really understand SNDMORE, since it seems to still send in a single message
|
|
[22:08] thatch45
|
I need to refresh my memory on why that was not the right way to do it
|
|
[22:08] tjholowaychuk
|
im just curious for use-cases, im sure i'll find out the more i use zmq
|
|
[22:19] tjholowaychuk
|
a pair is the one most like a tcp connection right?
|
|
[22:27] jhawk28
|
SNDMORE is just a flag that is used to separate parts of a message.
|
|
[22:27] jhawk28
|
think if a message as an array of bytes
|
|
[22:29] jhawk28
|
the message isn't sent until you use a 0 flag
|
|
[22:29] jhawk28
|
if I remember correctly
|
|
[22:38] jhawk28
|
nope, it looks like it sends each separate parts if I am reading the source right. its the reaper that makes the receipt atomic
|
|
[22:38] jhawk28
|
so it fills up memory on the receiving side
|
|
[22:45] tjholowaychuk
|
for chunks of an http request you would have to tag it right?
|
|
[22:46] jhawk28
|
yes, so that you can keep track of them to put them back together
|
|
[22:46] tjholowaychuk
|
k, just making sure im not mad
|
|
[22:46] tjholowaychuk
|
haha
|
|
[22:46] jhawk28
|
well, you may still be
|
|
[22:47] tjholowaychuk
|
perhaps
|
|
[22:55] streamer
|
Should the following setup work?? Dealer client tcp <-> [tcp |Queue| inprocess] <-> inprocess Dealer worker
|
|
[22:58] cactuss
|
there isn't a way to have zeromq buffer publisher messages onto disk during times when it has no subscribers, and then guarantee that subsequent scribers will get that data eventually given enough disk space is there?
|
|
[22:59] tjholowaychuk
|
i should probably read first but does zmq have any load introspection capabilities available through the public api?
|
|
[23:00] jhawk28
|
cactuss: no - 0mq sees the pub/sub pattern like a radio
|
|
[23:01] jhawk28
|
tj: not much - you have the HWM, but you are not notified if it drops messages
|
|
[23:01] cactuss
|
are there any known projects that buffer to disk? rabbit mq etc? thanks jhawk28
|
|
[23:02] jhawk28
|
kestrel, rabbit, kafka, etc
|
|
[23:03] jhawk28
|
kafka will be the most complicated, but it was designed with scalable persistence in mind
|
|
[23:04] jhawk28
|
kestrel is probably the simplest with Rabbit in the middle
|
|
[23:04] jhawk28
|
also, be sure to look at the titanic pattern from the guide
|
|
[23:05] Guest75658
|
is a single socket threadsafe when it comes to polling?
|
|
[23:06] jhawk28
|
sockets are not threadsafe
|
|
[23:06] Guest75658
|
as in, it isn't safe to send/recv in different threads, but is it safe to poll/send?
|
|
[23:06] jhawk28
|
probably not
|
|
[23:07] Guest75658
|
ok cool, thanks
|
|
[23:07] jhawk28
|
its not safe, but you can always try it
|
|
[23:07] jhawk28
|
it usually core dumps if it gets out of state
|
|
[23:07] Guest75658
|
yeah, tracking down an assertion failure that looks like a thread safety issue
|
|
[23:08] jhawk28
|
you may want to use inproc sockets for each thread and then use a reactor to put the outgoing data on external one
|
|
[23:09] jhawk28
|
or you can cheat and just use a mutex
|
|
[23:16] cactuss
|
oh neat going to read up jhawk28
|
|
[23:18] jhawk28
|
the easiest way to get persistence is to do it yourself. if you want to guarantee no message loss, use a sequence number on the data.
|
|
[23:18] jhawk28
|
the client then can keep track of the position
|
|
[23:19] jhawk28
|
a second XREP would be needed so that the clients can request the missing messages
|
|
[23:19] jhawk28
|
its basically what kafka does. they have a nice design doc that explains what they do
|
|
[23:20] jhawk28
|
you would then just need to store the data on the disk with the ability to clean it later
|
|
[23:20] jhawk28
|
titanic uses a file per message approach
|
|
[23:21] cactuss
|
1 file per message would be devestating for me
|
|
[23:21] jhawk28
|
if you want a "persistance library", you can look at leveldb or bitcask
|
|
[23:21] cactuss
|
yeah familiar with those
|
|
[23:21] jhawk28
|
then you just persist the data in a folder
|
|
[23:22] cactuss
|
we are using s3 as a persistence middle layer at the moment
|
|
[23:22] cactuss
|
well we're still designing this part of the system
|
|
[23:22] cactuss
|
so we're not really doing anything yet
|
|
[23:22] cactuss
|
but we have ids and many (millions) messages per id, and 1-100 ids per day
|
|
[23:22] jhawk28
|
bitcask is faster, but it would need to store all keys in memory. leveldb doesnt have that limitation
|
|
[23:22] cactuss
|
and we want workers to be able to look at 'all' of the messages from each id
|
|
[23:23] cactuss
|
so we're processing data into files by id, sending to s3 and anticipating having a worker pull from there
|
|
[23:23] cactuss
|
but the synchronization is a little messy / unclear
|
|
[23:23] mikko
|
cactuss: do you actually need a queue or a database?
|
|
[23:23] cactuss
|
we probably just need map reduce
|
|
[23:24] cactuss
|
we're collecting data, and then eventually generating analytics on the data
|
|
[23:24] jhawk28
|
is it a batch approach?
|
|
[23:24] cactuss
|
yeah
|
|
[23:24] jhawk28
|
or do want realtime?
|
|
[23:24] cactuss
|
it's a matter of coordinating the batches
|
|
[23:24] mikko
|
hbase might work for you
|
|
[23:24] cactuss
|
we don't really need message level granularity
|
|
[23:24] jhawk28
|
batch == mapreduce
|
|
[23:24] cactuss
|
we have like 100-1000 bytes per message so message level granularity is mostly out of the picture
|
|
[23:24] cactuss
|
and we're doing everything we can to avoid needing random / arbitrary access
|
|
[23:25] jhawk28
|
if you want to process it in realtime, look into storm
|
|
[23:25] cactuss
|
batch is fine, i think we simply want map reduce, it's more of an implementation issue
|
|
[23:25] cactuss
|
for hadoop we want to chunk our files to 64MBs
|
|
[23:26] cactuss
|
and we want to be able to keep track of all of our files, and not need a standing persistent hadoop cluster
|
|
[23:59] CIA-79
|
pyzmq: 03MinRK 07 * r2e5519e 10/ zmq/eventloop/ioloop.py : upstream tornado fix for PeriodicCallback ...
|
|
[23:59] CIA-79
|
pyzmq: 03MinRK 07 * r503b9e7 10/ zmq/core/version.pyx : version bump to 2.1.10 - http://git.io/QrOvXw
|
