IRC Log

Tuesday January 11, 2011

[Time] NameMessage
[03:13] potatodemon Howdy, how many queues can 0mq handle on a machine with a gig of ram?
[12:20] mikko hi steve
[12:20] mikko is it possible to have the scons to build to a specific subdirectory rather than platform specific one?
[12:21] mikko i played around during weekend with invoking scons build from zeromq build
[12:26] Steve-o just use autoconf & automake?
[12:26] Steve-o check the new trunk, committed support ~90 mins ago
[12:27] Steve-o please test on old platforms as I've hard coded the minimum versions to Ubuntu 10.10 :-)
[12:27] Steve-o "silent building" is limited to automake 1.11 I think
[12:28] Steve-o I'll have a look at CMake for Windows later
[12:28] mikko yes, automake would make things easier
[12:28] mikko the current way doesn't seem sustainable to me in the longer run
[12:28] mikko another question was about openpgm ABI/API
[12:29] mikko is there a guarantee in openpgm related to those?
[12:29] Steve-o The version numbers tie to the ABI
[12:30] Steve-o minor version on ABI changes, micro version on fixes
[12:30] Steve-o I've tried setting release/version-info with automake
[12:30] mikko because if we invoke openpgm build from 0mq we could support specifying arbitrary openpgm location as well
[12:30] mikko so that we wouldn't force certain version of openpgm
[12:31] Steve-o with autoconf distros are more likely to package it :D
[12:31] mikko brb, lunch
[13:42] sustrik mikko: have you seen the s360 patch?
[13:42] sustrik oh, it wasnt't sent to the mailing list
[13:42] sustrik wait a sec
[13:45] sustrik sent to the mailing list
[15:18] Skaag I want to subscribe to the mailing list
[15:21] mikko Skaag: http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[15:23] Skaag thank you :)
[15:23] Skaag my internet is painfully slow :-(
[15:23] Skaag So even subscribing to this will take a while
[15:24] Skaag there, done, thankfully that page is very minimalistic and not loaded with images
[16:18] jugg sustrik, my implementation of erlzmq can be found here: https://github.com/csrl/erlzmq If you want me to push to the zeromq/erlzmq repo, let me know.
[17:54] sustrik jugg: try to ask serge aleynikov first
[17:54] sustrik he've been maintaining the erlzmq
[17:54] sustrik serge@aleynikov.org
[18:03] s0undt3ch on a PUB socket, it can only be binded by a single instance? ie, can serveral unrelated parts of my code be bind to the same PUB socket?
[18:03] s0undt3ch from what I'm seeing from my code, it apears that it works like that :\
[18:04] s0undt3ch do I need to create a FORWARDER device of some kind?
[18:06] s0undt3ch sustrik: any idea?
[18:07] sustrik no. several sockets can't bind to the same endpoint
[18:07] sustrik if there are N pubs and M subs you need a forwarding device of some kind
[18:08] sustrik 0MQ socket is always 1:N
[18:08] sustrik N:M topologies are created using devices in the middle
[18:09] s0undt3ch sustrik: hmm, any examples of that anywhere?
[18:09] guido_g http://paste.pocoo.org/show/318987/ <- minimal forwarder config
[18:10] guido_g the in address is where the pubs connect to
[18:10] guido_g the out where the subs connect to
[18:10] guido_g simply start zmq_forwarder <config-file>
[18:10] s0undt3ch hmm, ok
[18:10] s0undt3ch Thanks!
[18:11] guido_g you can have multiple bind addresses per section
[18:12] sustrik the config file looks something like this:
[18:12] sustrik <forwarder>
[18:12] sustrik <in>
[18:12] sustrik <bind addr = "tcp://eth0:5555&quot;/>
[18:12] sustrik </in>
[18:12] sustrik <out>
[18:12] sustrik <bind addr = "tcp://eth0:5556&quot;/>
[18:12] sustrik </out>
[18:12] sustrik </forwarder>
[18:12] s0undt3ch guido_g: the forwarder device will then allow multiple binds and connects
[18:12] s0undt3ch I thinks that's what I'm really after
[18:12] s0undt3ch *think
[18:12] sustrik oops, there should be double quoute instead of &quot;
[18:12] s0undt3ch yeah
[18:13] s0undt3ch I also saw guido_g's paste
[18:13] guido_g you PUB sockets all connect to the in addresses
[18:13] sustrik ah, i missed that
[18:13] s0undt3ch yeah, I think I got it
[18:13] guido_g you SUB sockets connect to the out addresses
[18:45] s0undt3ch do I need to set zmq.SUBSCRIBE on the device?
[18:47] guido_g no
[18:47] guido_g just what i pasted
[18:51] s0undt3ch guido_g: I'm not using zmq_formwarder, I'm using the python bindings :\
[18:52] guido_g writing you own device? why?
[18:52] guido_g btw, i'm also using python
[18:52] s0undt3ch guido_g: zmq_forwarder is python?
[18:53] guido_g no
[18:53] guido_g its a program that comes with ømq
[18:53] s0undt3ch guido_g: http://paste.pocoo.org/show/319013/ my forwarding device
[18:54] guido_g seems you like it complicated
[18:54] s0undt3ch not realy, I'd love to make it simple
[18:54] s0undt3ch guido_g: what have I made complicate?
[18:54] guido_g by not using what's already there
[18:55] s0undt3ch I'm sorry... I don't know all of pyzmq's source code
[18:55] guido_g me neither
[18:55] s0undt3ch so what's there, where can I read it?
[18:56] guido_g but if you install ønmq, zmq_forwarder is also installed
[18:56] s0undt3ch guido_g: I'm launching my own app, I'm not gonna use subprocess to lauch zmq_forwarder!?
[18:56] s0undt3ch espcially since pyzmq also provides irt
[18:56] s0undt3ch *it
[18:56] guido_g just start it in backround
[18:57] yawn regarding jzmq - is there any advice on error handling? i regularly experience the jvm dying due to zmq errors.
[18:57] s0undt3ch ok, as a last resort. Can you please, since you also code in python, walk me through setting up the threaddevice?
[18:58] guido_g what threaddevice?
[18:58] s0undt3ch guido_g: http://zeromq.github.com/pyzmq/api/generated/zmq.devices.basedevice.html#threaddevice
[18:59] guido_g where is the problem?
[19:00] s0undt3ch guido_g: my code is aparently not working
[19:00] guido_g its nearly the same as the config i pasted
[19:00] guido_g s0undt3ch: i can't see that from here, to many walls
[19:01] guido_g s0undt3ch: iow, could show the error message and the corresponding code?
[19:01] s0undt3ch it' s probably the subscriber
[19:01] s0undt3ch no error
[19:01] s0undt3ch I'm just not getting anything on the M subscribers
[19:01] guido_g show code
[19:01] s0undt3ch although the N publishers are publishing
[19:03] s0undt3ch guido_g: test subscriber
[19:04] guido_g s0undt3ch: huh? what does that mean?
[19:04] s0undt3ch guido_g: sorry
[19:04] s0undt3ch guido_g: test subscriber http://bpaste.net/show/12837/
[19:04] s0undt3ch forgot the paste
[19:06] guido_g does the device publish to the unix socket the subscriber expects?
[19:06] sustrik yawn: what errors are you getting?
[19:06] sustrik can you send them to the mailing list?
[19:07] s0undt3ch guido_g: http://bpaste.net/show/12838/ test publisher
[19:07] s0undt3ch sustrik: no errors at all
[19:07] sustrik it's question for yawn
[19:07] sustrik he's complaining about jzmq craching
[19:07] s0undt3ch sustrik: oh, sorry
[19:08] sustrik np
[19:08] s0undt3ch guido_g: the forwarder seems to not be forwarding nothing at all
[19:09] guido_g s0undt3ch: there is no forwarder involved as far as i can see
[19:09] s0undt3ch guido_g: the forwarder was the first paste, the one you said I was complicating
[19:09] guido_g *sigh*
[19:10] guido_g s0undt3ch: see which endpoints your code is using
[19:10] guido_g s0undt3ch: pub and sub are using the unix socket
[19:10] s0undt3ch yes
[19:10] s0undt3ch can't be unix sockets?
[19:11] s0undt3ch guido_g: diferent sockets though
[19:11] s0undt3ch xxx-in and xxx-out
[19:15] yawn sustrik: Bad file descriptorrc != -1 (kqueue.cpp:79)
[19:15] yawn sustrik: "somewhere", no stacktrace available when dealing with jni i suppose
[19:16] sustrik is it deterministic?
[19:17] s0undt3ch guido_g: test code -> http://bpaste.net/show/12840/
[19:17] s0undt3ch guido_g: what's wrong there?
[19:18] guido_g ipc endpoints are file names, http://api.zeromq.org/zmq_ipc.html
[19:18] s0undt3ch yes
[19:19] s0undt3ch that will create a file on the curent dir
[19:19] yawn sustrik: no really. only in the sense that it happens from time to time.
[19:19] s0undt3ch the diferences in paths on the previous code mean nothing, my app changes dir
[19:21] snidely If I have a high volume of stock/option quotes coming into a data center and I want clients connected over a low speed WAN link to subscribe to quotes from that dc. Should I use 0MQ? Or do I need some broker based solution so that the whole multicast doesnt have to go over the WAN for key based filtering on each client?
[19:23] sustrik snidely, are you looking for solution for both WAN and LAN?
[19:24] sustrik in that case the best way is to place forwarder device at the edge of your LAN
[19:24] sustrik then use TCP for WAN
[19:24] sustrik and multicast only on LAN part
[19:24] s0undt3ch guido_g: forwarders can't use unix sockets?
[19:24] guido_g don't know, ask sustric
[19:24] s0undt3ch sustrik: forwarders can't use unix sockets?
[19:25] guido_g s/c$/k/
[19:25] yawn sustrik: but is there a sensible way to handle zmq errors when using it in a jvm?
[19:25] sustrik s0oundt3ch: they use the transport you specify
[19:25] sustrik it you are using ipc:// then they use unix domain sockets
[19:25] s0undt3ch sustrik: what's wrong with http://bpaste.net/show/12840/?
[19:26] sustrik if you use tcp:// they'll use TCP socekts
[19:26] s0undt3ch http://bpaste.net/show/12840/
[19:26] s0undt3ch yeah
[19:26] s0undt3ch but my forwarder isn't forwarding
[19:27] snidely yeah I want to be able to have client apps at the datacenter subscribe to quotes (in large volume) and also have apps connected via WAN subscribe in small quantities.
[19:28] snidely Where can I get more info about forwarders? This is unfinished: http://api.zeromq.org/zmq_forwarder.html
[19:29] sustrik it's a simple executable, zmq_forwarder
[19:29] sustrik the argument is a config file
[19:29] sustrik http://paste.pocoo.org/show/318987/
[19:30] sustrik s0oundt3ch: hm, it's a pyzmq implementation of forwarder
[19:30] guido_g s0undt3ch: can't get your python forwarder to work even w/ tcp
[19:30] sustrik no idea how it works
[19:30] s0undt3ch hmmm
[19:30] s0undt3ch dam
[19:30] sustrik try asking on the mailing list
[19:30] s0undt3ch guido_g: something must be terrably wrong
[19:30] sustrik pyzmq devs should be able to reply
[19:31] guido_g with the original zmq forwarder it works ove rtcp
[19:33] s0undt3ch guido_g: the original zmq_forwarder?
[19:33] guido_g yes
[19:33] s0undt3ch hmm
[19:33] guido_g the one that comes with ømq
[19:33] guido_g the one i pasted the config for
[19:34] s0undt3ch yeah
[19:36] guido_g just verified that that zmq_forwarder works w/ ipc
[19:37] guido_g did you try a zmq.SUBSCRIBE on the in socket of you device?
[19:37] guido_g *your
[19:40] s0undt3ch guido_g: just tried, it works
[19:40] guido_g what a crap
[19:41] s0undt3ch guido_g: http://paste.pocoo.org/show/319036/ <- this is what I tried, it works
[19:41] s0undt3ch bypasses the device
[19:43] guido_g that's what you had before the devices were mentioned, right?
[19:44] s0undt3ch guido_g: yes, but I need multiple publishers
[19:44] s0undt3ch for multiple subscribers
[19:44] s0undt3ch thats where devices came in
[19:44] guido_g i know
[19:46] guido_g forwarder.setsockopt_in(zmq.SUBSCRIBE, '') <- that's the trick
[19:47] guido_g at least it works w/ tcp
[19:48] s0undt3ch guido_g: can you paste your code?
[19:48] guido_g just add the line shown
[19:49] s0undt3ch guido_g: to which paste?
[19:49] guido_g omg
[19:49] s0undt3ch because I had tried that
[19:50] s0undt3ch guido_g: this one?
[19:50] s0undt3ch http://bpaste.net/show/12840/
[19:50] guido_g http://bpaste.net/show/12841/
[19:53] s0undt3ch great, bpast is throwing 500's
[19:53] s0undt3ch *bpaste
[19:54] guido_g works here
[19:55] s0undt3ch dunno then
[19:57] s0undt3ch heh, tpc, address already in use....
[19:57] s0undt3ch *tcp
[19:57] s0undt3ch guido_g: yeah, tried that too
[19:58] s0undt3ch guido_g: are you using 2.1.0?
[19:58] guido_g no
[19:58] s0undt3ch hmm, I am
[19:58] s0undt3ch hmm, I am
[19:59] s0undt3ch sorry for doubling
[19:59] s0undt3ch quassel wasn't responsive
[20:01] s0undt3ch ok
[20:01] s0undt3ch seems to be a 2.1.0 issue
[20:02] s0undt3ch on the subscribers endpoint
[20:04] s0undt3ch I think I found a bug
[20:04] s0undt3ch just recreating a minimal example to create a ticket
[20:11] s0undt3ch dam, with pyzmq's ioloop it works correctly
[20:11] s0undt3ch even on 2.1.0
[20:13] CIA-21 zeromq2: 03Martin Sustrik 07master * r725ebce 10/ include/zmq.h :
[20:13] CIA-21 zeromq2: Version bumped to 2.1.1
[20:13] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/ihsCvD
[20:41] mikko sustrik: will look into the build patch now
[20:43] sustrik mikko: i've seen there are some additional functions being checked for
[20:43] sustrik for OpenPGM's sake
[20:43] sustrik they probably should not be checked for if pgm is disabled though
[20:44] mikko yes
[20:44] mikko steve has been working on openpgm autoconf as well
[20:44] mikko which is very good progress
[20:44] mikko we could invoke the openpgm build from zeromq build without additional deps
[20:45] sustrik that would be fantastic
[20:45] sustrik getting rid of the foreign build system...
[20:46] mikko and not maintain a copy of the openpgm build system
[20:46] sustrik exactly
[20:46] mikko i think those changes are in openpgm trunk now
[20:46] mikko i might take a poke on that side as well and test freebsd/solaris at some point
[20:46] sustrik ack
[21:25] mikko sustrik: getting build error on linux after the patch
[21:25] mikko will investigate a bit
[21:26] mikko hmm
[22:29] amacleod With the Python bindings for zmq, is it okay to just let the Context expire with the running script? Or should I explicitly close it?
[22:39] sustrik amacleod: closing of the context ensures the all the pending messages are actually pushed to the network before the application extis
[22:39] sustrik exits
[22:47] codebeaker hi all - anyone know how to build Patrick Chen's fork properly, it says "See the INSTAL file", but that file is absent, and my typical cmake routine "$ mkdir build; $ cd build; $ cmake ../; ......" doesn't seem to want to work
[22:58] codebeaker I got as far as setting CMAKE_MODULE_PATH and ZMQ_DIR to the ./src/zmq directory, to which I cloned pchen's repository, now I get the error "include could not find load file: src/zmq/ZMQConfig.cmake" - which makes sense, because it only exists as ./src/zmq/ZMQConfig.cmake.in
[22:59] codebeaker how to make sure my top-level project configures the ./src/zmq correctly ? and preprocesses the .in files
[23:12] zchrish I made a test case whereby I created a publisher -> forwarder -> subscriber and feed it a 500MB file with 7500847 lines and the time it took to process the file was 102.39 seconds. This equates to about 72,350 messages per second. This was done on localhost. Do these results seem reasonable?
[23:19] zchrish Pleased to see that files pass "diff" !
[23:20] codebeaker zchrish: that does seem to tally with the numbers quoted in the various docs
[23:20] codebeaker … you can check the Introduction - into it a little way are some comparisons of throuput and message sizes
[23:35] snidely I'm told that 0MQ has missing libraries which prevent it from running on windows. Anyone know about that?

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy