Thursday February 3, 2011

[Time] NameMessage
[00:00] mikko yeah, i guess it's partially convenience
[00:00] mikko being able to build with distro-provided tools without having to first update the autoconf toolchain
[00:00] Steve-o the new silent build is nice though, brings it upto building Linux and how CMake runs
[00:01] Steve-o s/upto/upto date/
[00:01] mikko and i would think that at least the parts of the autoconf stuff i've written follow documentation or tutorials found online
[00:01] mikko not always necessarily up to date
[00:01] Steve-o that's the biggest problem,
[00:02] Steve-o every blog and walkthough refers to an older upstream version
[00:03] mikko yeah, i learned to appreciate the autoconf documentation at later stages
[00:03] Steve-o doesn't stop autoheader from being broken
[00:03] Steve-o gettext doesn't work out of the box
[00:03] Steve-o I didn't spend too long looking at it though
[00:06] Steve-o similarly detecting files is a bit odd
[00:06] Steve-o AC_CHECK_FILE seems to check for being able to read with the users permissions
[00:07] Steve-o so /dev/hpet always fails for example
[00:08] Steve-o oh and it looks like I didn't check for Sun Pro's hidden attribute
[00:08] mikko test -r ?
[00:08] mikko or -x ?
[00:09] Steve-o not too important, only /dev/rtc and /dev/hpet which have major problems anyway
[00:11] Steve-o overall not too bad, I'm certainly not interested in copying the autoconf to SCons or CMake, they look a lot more verbose
[00:12] Steve-o I should get around to checking cygwin/mingw though, I guess autoconf doesn't work with msvc?
[00:13] mikko no, you would need MSBuild or visual studio templates
[00:13] mikko autoconf works on cygwin / mingw
[00:13] Steve-o that's why I have CMake :D
[00:16] Steve-o a bit limiting that you need a ssl certificate to get your name in the publisher field of Windows installers
[00:16] mikko i guess they have a reason for that
[00:17] Steve-o still need to look at getting an installer for 0mq though
[00:18] mikko is it much more than copying the dll in place?
[00:19] Steve-o I call NSIS through CPack as part of CMake
[00:19] Steve-o you setup the target directory structure and simply fill in a few fields unless you need something special
[00:21] mikko hmm
[00:21] Steve-o the main job of the installer is to prepare the uninstaller
[00:22] mikko for zeromq it might be possible to add an installer project into the solution
[00:22] mikko msvc
[00:22] Steve-o I'm guessing thats how it was done before
[00:24] Steve-o possibly pertinent to bundle the msvc 2010 redistributable too
[00:24] Steve-o Microsoft released an opensource installer thing a while back no, WiX?
[00:25] Steve-o
[00:26] mikko cool
[00:29] Steve-o I don't have any templates in visual studio for an installer, maybe I missed something
[00:32] mikko are you using express edition?
[00:32] Steve-o yup
[00:33] mikko i dont think express includes the installer technology
[00:33] mikko so wix might be a better solution
[00:45] mikko openpgm autoconf was actually pretty easy to fit into zeromq build
[00:45] mikko also it's using libtool to link so it should continue working if libpgm provides shared libraries later on
[00:47] mikko hmm
[00:47] mikko hitting an assertion with epgm transport
[00:47] mikko i was getting this before as well
[00:49] mikko Assertion failed: rc == 0 (connect_session.cpp:82)
[00:50] Steve-o common on connect()
[00:50] Steve-o usually means resolving the network parameter failed
[00:51] Steve-o fyi: libpm in autoconf is already set to make static & shared libraries
[00:52] Steve-o ideally the pgm error object should be routed through the zeromq logging service
[00:53] Steve-o the zeromq error detail is incredibly brief
[00:54] Steve-o I'm currently pondering over chasing the libc defect with /etc/hosts resolution on multiple entries
[00:54] Steve-o I can't help think it's just NetworkManager being shit
[00:54] mikko what is the issue?
[00:54] mikko order of resolution?
[00:54] Steve-o two issues
[00:55] Steve-o one it seems to pickup ::1 as
[00:55] mikko my connection string is epgm://eth0;
[00:55] Steve-o and secondly it seems to read bottom to top
[00:55] Steve-o I'd try using the IP address but that looks fine
[00:56] mikko still asserts with ip
[00:56] mikko hmm
[00:56] Steve-o anything if you run with PGM_MIN_LOG_LEVEL=TRACE ?
[00:57] mikko
[00:57] mikko it outputs that
[00:59] Steve-o odd, I'm guessing setsockopt on IP_PKTINFO is failing
[01:01] Steve-o this is with my autoconf?
[01:02] Steve-o or the stuff in master?
[01:02] mikko it seems to happen with stuff on master and autoconf
[01:03] Steve-o but not with openpgm tools?
[01:04] mikko havent tested
[01:04] Steve-o have to prod Sustrik when he is online about PGM logging
[01:04] mikko trying to find the tools among the openpgm source
[01:05] mikko hmmm.. the autoconf doenst seem to build them
[01:05] Steve-o nope :P
[01:06] mikko sconssing now
[01:06] Steve-o autoconf & cmake only build libraries, easiest method for distribution
[01:07] mikko what is good test program?
[01:08] Steve-o purinrecv or purinsend are the easiest to use
[01:08] mikko hmm
[01:08] mikko nothing happens
[01:08] mikko but no errors
[01:09] Steve-o you can always build them with something like gcc -o purinrecv examples/purinrecv.c `PKG_CONFIG_PATH=. pkg-config --cflags --libs`
[01:09] mikko i got purinrecv running
[01:09] mikko and executing purinsend
[01:09] mikko should i see something in the receiver?
[01:10] Steve-o usually if you don't see anything it means it's picking up the loopback device
[01:10] mikko Trace: Multicast send interface set to index 0
[01:10] mikko correct
[01:11] Steve-o that's the /etc/hosts problem correct
[01:11] Steve-o is the default ip for dhcp linux boxes
[01:11] Steve-o you should have a second entry with the dhcp address
[01:12] Steve-o but getaddrinfo('hostname') is returning a different order to what NetworkManager thinks it should
[01:12] Steve-o so annoying
[01:12] mikko ok, changed to the eth0 ip but still not seeing anything
[01:12] mikko odd
[01:12] mikko Trace: Multicast send interface set to index 0
[01:13] Steve-o oh, add '-l' for loopback
[01:13] Steve-o it's disabled by default
[01:14] Steve-o doh
[01:14] mikko ok now seeing something: Trace: Recv again on not-full
[01:14] mikko in any case, it gets further than with zeromq
[01:14] Steve-o yup
[01:15] mikko comparing the results, inside zeromq i dont see this: Trace: Assuming IP header size of 20 bytes
[01:16] mikko Trace: Request IP headers. is the last line
[01:16] Steve-o i can try debian in a vm
[01:18] mikko
[01:18] mikko that should be enough to reproduce
[01:22] Steve-o ok, I'll be a bit, I need to reboot into Ubuntu and download & install everything ...
[01:22] mikko hehe
[01:23] mikko i think im gonna hit the sack
[01:23] mikko getting a bit late here
[01:23] mikko good night
[01:23] Steve-o I'll shoot you a mail
[01:23] Steve-o night
[01:23] mikko thanks
[07:48] CIA-21 zeromq2: 03Martin Sustrik 07master * rca1acc3 10/ (src/poll.cpp src/poll.hpp):
[07:48] CIA-21 zeromq2: RLIMIT_NOFILE not used in poll_t anymore
[07:48] CIA-21 zeromq2: The problem was that RLIMIT_NOFILE can be set to RLIM_INIFINITY
[07:48] CIA-21 zeromq2: (and that appears to be default on AIX) which caused 0MQ to fail.
[07:48] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[07:59] mikko good morning
[08:06] sustrik morning
[08:11] sustrik mikko: i've realised you are running solaris builds
[08:11] sustrik is there a chance to test a patch on solaris?
[08:11] mikko sustrik: sure
[08:12] sustrik great
[08:12] mikko i can do it over lunch if its more involved
[08:12] mikko have you got it in github?
[08:12] sustrik nope
[08:12] sustrik the problem is i have no access to solaris box
[08:12] sustrik and i don't want to commit something i haven't even been able to compile
[08:13] sustrik let me create the patch
[08:13] sustrik we can check whether it builds on solaris
[08:13] sustrik and if so, i'll commit it
[08:13] mikko i noticed that default ZMQ_RATE seems to be too small for debian
[08:13] mikko Assertion failed: rc == 0 (connect_session.cpp:82)
[08:13] mikko i can add a separate build on hudson which builds from a specific github branch so you can build when you want
[08:13] mikko would that be useufl?
[08:14] sustrik dunno
[08:14] sustrik in most cases i can build at home
[08:14] sustrik this time the problem is that the code is solaris-specific
[08:14] mikko ok, i can give it a spin
[08:14] sustrik ok, i'll send you the patch
[08:15] sustrik no haste though
[08:17] mikko steven suggested adding 1024 to both rxw_sqns and txw_sqns in pgm_socket.cpp
[08:17] mikko but that still seems to be too low
[08:17] mikko sustrik: have you got ubuntu / debian at hand?
[08:18] mikko
[08:18] mikko this code should hit assertion
[08:18] mikko it does on debian 5.0
[08:18] sustrik yup, i'm on ubuntu right now
[08:19] mikko increasing ZMQ_RATE to 100000 solves it for me
[08:20] sustrik let me check the problem
[08:36] sustrik hm
[08:36] sustrik for me it fails here:
[08:36] sustrik 153 if (!pgm_setsockopt (sock, IPPROTO_PGM, PGM_UDP_ENCAP_UCAST_PORT,
[08:36] sustrik 154 &encapsulation_port, sizeof (encapsulation_port)) ||
[08:36] sustrik 155 !pgm_setsockopt (sock, IPPROTO_PGM, PGM_UDP_ENCAP_MCAST_PORT,
[08:36] sustrik 156 &encapsulation_port, sizeof (encapsulation_port)))
[08:36] sustrik which has nothing to do with the rate
[08:37] mikko do you run as root?
[08:37] sustrik i've changed the protocol to epgm
[08:37] sustrik no need to be root in such case
[08:43] sustrik mikko: patch sent
[08:44] sustrik when you have few mins free, try it with solaris
[08:44] sustrik thanks
[08:59] mikko sustrik: All 8 tests passed
[09:00] sustrik great
[09:00] mikko solaris studio on solaris
[09:00] sustrik i'll apply the patch
[09:02] CIA-21 zeromq2: 03Martin Sustrik 07master * r3f758ab 10/ (src/devpoll.cpp src/devpoll.hpp):
[09:02] CIA-21 zeromq2: Don't use RLIMIT_NOFILES in devpoll_t.
[09:02] CIA-21 zeromq2: The patch allows for running 0MQ on Solaris and HP-UX
[09:02] CIA-21 zeromq2: even though ulimit for max number of file descriptors
[09:02] CIA-21 zeromq2: is set to unlimited.
[09:02] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[10:23] mikko hmm
[10:23] mikko on platforms that dont support pic objects is it possible to link static library inside a shared library?
[10:37] sustrik no idea, sorry
[10:48] mikko [equilibrium]: gelato con leche, dos bolas. uno fresa y uno cafe
[10:48] mikko thats what i learned last night
[10:48] mikko hamburguesa con queso y patatas fritas
[10:50] sustrik ha, the original post was in italian :)
[10:51] sustrik how's barcelona btw?
[10:53] sustrik or are you in san diego? /me is not quite sure
[12:08] kabs Hello, in pub-sub model , if sub starts later and pub started earlier all messages by pub will be dropped. Is there a way to buffer those messages ( with some buffer limit) so that they can sent to subscriber joining later. Can I get some link to go through it or if it is there in zeroMQ guide can someone point out which section it is ??
[12:14] sustrik how could the PUB socket possibly know that there will be subscriber in the future?
[12:32] mikko sustrik: barcelona at the moment
[12:32] mikko going to san diego end of may
[15:02] udoprog Hey, I'm currently developing a game which will require REQ-REP but also sporadic pushing of information to the client, I've got a picture of how to implement this in twisted, but with additional coding overhead (for framing), I would really like to give zeromq a shot.
[15:08] mikko sure
[15:09] mikko you can even use zeromq with twisted
[15:17] udoprog okey, care to elaborate? do you know of any references I can use?
[15:42] sustrik udoprog: no idea myself, try asking on the mailing list
[16:26] sustrik mikko: what do you think about the system z proposal?
[18:11] cyball hi ... does somebody have some samples of implementing ZMQ_XREP service with ZMQ_REQ clients i could not found any ?
[18:45] starkdg cyball, what doyou mean ? I think XREP and REQ work fine together.
[18:46] starkdg i mean, ive done it just fine.
[18:49] cyball starkdg: i never said that it does not work i'm new to ZeroMQ and i'm a little bit confused because of the description :
[18:49] cyball When a ZMQ_REQ socket is connected to a ZMQ_XREP socket, in addition to the identity of the originating peer each message received shall contain an empty delimiter message part. Hence, the entire structure of each received message as seen by the application becomes: one or more identity parts, delimiter part, one or more body parts. When sending replies to a ZMQ_REQ socket the application must include the delimiter part.
[18:50] cyball what exactly does delimiter part or identity part means ?
[18:50] starkdg i think that's just the protocol of the message buffer
[18:50] cyball how should that look like ?
[18:50] starkdg it should work seemlessly for the user
[18:51] cyball ok so i have only to define the constant ZMQ_XREQ in the zqm_socket function ?
[18:51] starkdg i dont know how it actually looks, maybe you can ask martin or pieter
[18:51] starkdg yeah, when creating the socket, just use ZMQ_REQ in the client app
[18:51] cyball ok i should mention that i'm using the c language
[18:51] starkdg and ZMQ_XREP in the server part
[18:51] cyball ok and that is enough ?
[18:51] starkdg yeah, i use c
[18:51] starkdg yes, that's enough
[18:52] cyball starkdg: that's great
[18:52] starkdg yeah, it works pretty well
[18:53] cyball ok i will give it a try :) thx a lot you saved my brain from burning :)
[18:55] cyball one more question ... if i understood this stuff well ... the ZMQ_REQ waits until the response from the ZMQ_XREP returnes before sending a new request ? is that right ?
[19:12] cyball starkdg: do you have an hello world sample for that ?
[19:12] starkdg no i dont
[19:12] starkdg let me see if i can find it
[19:14] starkdg wow, they've been busy , its all different now !
[19:14] starkdg I can give you a link to my project which uses zeromq , that might takes some sorting through though
[19:16] cyball starkdg: that would be great ... i try to find out how to use it :)
[19:16] starkdg cyball: you can pick through the source here:
[19:16] starkdg that's my project i'm working on , the source you can browse through, or download the project
[19:18] cyball ok thx a lot i've found already the client code with ZMQ_REQ :-)
[19:18] starkdg yeah, the code with XREP you will see in auscout.c
[19:18] guido_g for examples and explanations always see
[19:19] guido_g advanced routing stuff is in chapter three
[19:20] starkdg guido_g: he was looking for an example of using REQ socket connecting to XREP , i couldnt find that specific example.
[19:20] guido_g it also explains the parts of req/rep messages and shows some tricks
[19:20] starkdg ive noticed things have been busy on the site, alot more additions.
[19:24] cyball guido_g: thx a lot i've found now the samples for ZMQ_REQ and ZMQ_XREP
[19:25] cyball i think it should be enoug information for now -.... :-) thx for helping
[19:25] cyball by the way ... ZeroMQ is really a great think :)
[19:31] Slurpy hello
[19:38] Slurpy I have a bit of a high-level question regarding the use of 0mq. Ultimately I think I understand the examples in the documentation. I have a business need where I can have logical queues that are serviced by more than 1 process (or server). I'm wondering if there is a pattern that makes sense with 0mq. I know a client can connect to multiple sockets, so I can have more than one server serving a logical connection. Seems a bit mess
[19:45] sustrik Slurpy: what you want afaiu is having a device in the middle
[19:47] guido_g and, of course, reading the guide :)
[20:12] cremes cyball:
[20:13] cremes for some reason that doc doesn't show up on any of the community pages
[20:13] cremes you can only find it by searching for it
[20:21] cyball cremes: thx
[20:22] sustrik looks like there are some inaccessible pages on the website
[20:22] sustrik presumably, some reorg is needed
[20:22] cremes yep, all of the tutorials are orphaned
[20:24] sustrik should we introduce a "tutorial" section?
[20:24] sustrik or does it fit somewhere else?
[20:24] sustrik whitepapers maybe?
[20:24] sustrik rename whitepaper->articles
[20:24] sustrik and place tutorials there?
[20:27] cremes i think we should add back a tutorials page
[20:27] guido_g imho articles section would be a good idea
[20:30] cyball i have found this really nice multithreaded hello world server is it possible to get some subscriber do the work of the threads ?
[20:30] guido_g anyhow, a cleanup would be nice, some links lead to non-existing pages
[20:31] guido_g like
[20:36] sustrik where's the link?
[20:38] guido_g <- Traffic monitoring
[20:40] cyball there is also no working link to the zqm_device function
[20:41] cyball
[20:42] sustrik guido_g: ok, removed
[20:42] sustrik cyball: where's the link?
[20:43] guido_g sustrik: great, thanks!
[20:43] cyball #Built in Devices under Figure 20
[20:45] sustrik actually, it turns out there's in no zmq_device man page
[20:45] sustrik i wonder how that link have got into the guide
[20:47] cyball sustrik: that is act of nature behind control :)
[20:47] sustrik :)
[20:49] cyball is it possible to replace the workers in the multithreaded hello world sample with subscribers ? that was the reason checking out zmq_device there seems to be a FORWARDER flag ... could that be a solution ? :)
[20:50] sustrik you need a PUB/SUB device, right?
[20:50] sustrik if so, yes, use forwarder device
[20:52] cyball sustrik: but which parameter does it need .... for the clients workes it is fine rmq_device(FORWARDER, clients, ????) but i do not have subscribers or somethink similar ...
[20:52] sustrik ?
[20:53] cyball ohhh sorry i got it now
[20:53] cyball it was a little bit misunderstood because of the words clients and workers
[20:53] guido_g btw, it's all in the guide
[20:54] cyball yes sorry i realized to late that this are sockets
[20:54] cyball i guess i need a break
[20:54] cyball :)
[20:58] _jud hey sustrik: what is the current status of xpub/xsub w/ pub side filtering?
[21:05] sustrik _jud: the ongoing work is on sub-forward branch
[21:06] sustrik it propagates the subscriptions upstream
[21:06] sustrik what's missing is actual filtering on pub side
[21:07] _jud I have been trying to keep up on the mailing list
[21:08] jhawk28 silly question: what will it be called when the xpub gets extended?
[21:09] jhawk28 :)
[21:09] _jud I've been wanting to use 0mq in a ticker plant
[21:10] sustrik _jud: yeah, subscription forwarding is what's missing for 0mq to be a sane option for market data distribution
[21:11] sustrik jhawk28: yes, the names are lame
[21:11] sustrik it's actually two layers of the stack
[21:11] sustrik X sockets are like IP
[21:11] sustrik non-X sockets are like TCP
[21:11] sustrik no idea how to formalise that decently
[21:12] _jud sustrik: when you say sub forwarding - I assume that encompasses pub filtering also?
[21:13] sustrik yes
[21:13] sustrik the whole thing consist from two parts
[21:13] sustrik 1. pass the subsription upstream
[21:13] sustrik 2. do the filtering on the most upstream node
[21:14] _jud in regular pub/sub, all nodes just rebroadcast, correct?
[21:15] sustrik right
[21:15] sustrik that's the way it work now
[21:15] _jud just trying to clear up my surface knowledge of the 0mq layout
[21:16] _jud I will be monitoring the sub-forward branch
[21:16] sustrik ah, you are asking about distinction beween X and non-X sockets, right?
[21:16] _jud well, yes
[21:17] sustrik X sockets are to be used by intermediate nodes
[21:17] sustrik you can recv() a subsctiption on XPUB socket
[21:17] sustrik and send() a subscription to XSUB socket
[21:17] sustrik PUB and SUB are for endpoints
[21:18] sustrik the subsctiption are mananged automatically for you
[21:18] sustrik no need to explicitly send/recv them
[21:19] _jud so when pub filtering is implimented
[21:19] _jud it won't require the use of X sockets unless their are intermediaries?
[21:20] sustrik yes
[21:20] sustrik it will be opaque for you
[21:21] _jud ahhh
[21:21] sustrik X sockets are meant only for implemeting devices
[21:21] sustrik it's similar to IP
[21:21] sustrik normally, you don't use IP directly
[21:21] _jud ah, thanks for clearing that up
[21:22] sustrik unless you want to implement a router or somesuch
[21:22] _jud thanks for that sustrik