Tuesday November 1, 2011

[Time] NameMessage
[06:37] sand hello everybody
[06:38] sand i don't understand If you're using TCP, and a subscriber is slow, messages will queue up on the publisher
[07:45] sustrik sand_: up until HWM is reached
[07:46] sustrik then the sender blocks or messages are dropped, depending on the pattern
[07:46] sustrik see zmq_socket(3)
[07:52] sand i use pub-sub pattern.
[08:03] sand i had write example use pub-sub.publisher send 10000 message , subscriber receive one message and sleep(1000) and go on.
[08:04] sand i can't see that messages queue up on the publisher
[08:04] sand infact messages queue up on the subscriber
[08:05] sand something wrong with me?
[08:06] sustrik nope, messages queue up as for downstream as they are able to get
[08:06] sustrik do you want them to queue on the publisher?
[08:12] sand i read the guide,and code to comfirm that,but it differen, so i puzzled
[08:15] sand i using TCP, and a subscriber is slow(force to sleep 1s).is thart right?
[09:13] sanduu yeah
[10:29] sustrik mikko: hi
[10:32] mikko hi
[10:32] mikko whatcha need?
[10:43] sustrik mikko: hi
[10:44] sustrik a question: is it possible for make check to print the name of the test before running it?
[10:44] sustrik right now it prints nothing
[10:44] sustrik and if the test hangs up
[10:44] sustrik it's not clear which one is hanging
[10:45] mikko sustrik_: let me check
[10:58] mikko sustrik_: can't see an easy way
[10:58] sustrik mikko: ok
[10:58] sustrik np
[10:59] mikko apart from adding to test
[10:59] mikko std::cerr or something in each test
[10:59] sustrik yeah, i can do that
[16:09] k0p hi folks
[16:09] k0p the following page is not working for me:
[16:14] sustrik it works ok for me
[16:15] k0p oh, now works. but it waits for wikidot :S
[16:18] mikko k0p: looks fine from here
[16:20] k0p weird :S
[18:36] solvik hi, i would like as a server, to know every clients (subscribers) who are connected to me. it seems easy but i realy can't find the method which permits that. any hints ?
[18:52] RandalSchwartz so devices are broken in the Perl bindings?
[18:52] RandalSchwartz this seems to be my conclusion so far. :(
[18:52] RandalSchwartz is there a way I could write a SUB-to-PUB relay without devices?
[18:53] RandalSchwartz I'm not sure what each side of it would look like. a SUB subscribing to ""?, and then a tight loop that ->recv and just sends it on the pub?
[19:28] sustrik RandalSchwarz: universal device polls on both sockets and forwards messages from the first on to the second one and vice versa
[19:30] RandalSchwartz but for PUB sockets, there'd never be any need to poll for reading, right?
[19:30] RandalSchwartz so if I want a SUB-to-PUB proxy, a simple while (1) { $m = $sub->recv; $pub->send($m) } will work, yes?
[19:30] RandalSchwartz or do I need to do something for $pub somwhere?
[19:31] sustrik depends on the 0mq version you are using
[19:32] sustrik from version 3.0 the subscriptions are propagated upstream
[19:32] sustrik ie. from SUB to PUB
[19:32] sustrik thus you need both directions
[19:32] RandalSchwartz the C-level device handles that?
[19:32] RandalSchwartz I'd have to copy that to the Perl code by hand then
[19:33] sustrik what version are you using?
[19:33] RandalSchwartz or figure out why ZeroMQ::Raw::zmq_device is broken
[19:33] RandalSchwartz 0.18
[19:33] RandalSchwartz err.. 0.17
[19:33] RandalSchwartz can't type
[19:33] sustrik version of 0mq i mean
[19:33] RandalSchwartz that's the latest in the CPAN
[19:34] RandalSchwartz zmq is 2.10
[19:34] sustrik i see
[19:34] RandalSchwartz 2.1.10
[19:34] sustrik in that case while (1) { $m = $sub->recv; $pub->send($m) } will work
[19:34] RandalSchwartz ok
[19:34] RandalSchwartz but I hope someone fixes the device() call :)
[19:35] RandalSchwartz it blows an error for me when I try to call it. wrong type for socket
[19:35] RandalSchwartz even though I've copied the example directly
[19:35] sustrik fill in a ticket in perl binding issue tracker
[19:35] RandalSchwartz there are no tests to see that it broke though :(
[19:36] RandalSchwartz I thought setting HWM to 1 would ensure only one message in an unconnected PUB socket, but that seems not to be the case
[19:36] RandalSchwartz looks like they keep queeing until there's a reader
[19:36] RandalSchwartz when I finally connect, I get a big burst
[19:37] RandalSchwartz that'd be ok for this app, but I'm wondering what I did wrong
[19:37] sustrik setting HWM *after* connecting, maybe?
[19:38] RandalSchwartz well, that's transparent.
[19:38] RandalSchwartz I call ->connect
[19:38] RandalSchwartz and then setsockopt
[19:38] RandalSchwartz but then I start sendign things
[19:38] RandalSchwartz at first, the SUB isn't there
[19:38] sustrik setsockopt() is applied only to subsequent connects
[19:38] RandalSchwartz but when it does show up, I finally get a burst
[19:38] RandalSchwartz oh!
[19:38] RandalSchwartz swap those two
[19:38] RandalSchwartz ok
[19:40] RandalSchwartz interesting. it seems to stash the first message, and toss the later ones
[19:40] sustrik yeah, that's the case now
[19:41] RandalSchwartz if I leave it at 0, will it just fill memory eventually?
[19:41] sustrik yes
[19:41] RandalSchwartz it'd be nice to have some sort of flush
[19:41] RandalSchwartz since "1" doesn't quite do it
[19:41] sustrik ?
[19:41] RandalSchwartz imagine your "current weather" publish
[19:42] RandalSchwartz you don't want the old weather to be the one in the first connect
[19:42] RandalSchwartz you want always the latest
[19:42] sustrik sure, that would work
[19:42] RandalSchwartz so how does the weather publisher say "here's the newest value"
[19:42] RandalSchwartz no - it doesn't work now
[19:42] sustrik because PUB binds
[19:42] sustrik and SUB connects
[19:42] RandalSchwartz I'm seeing PUB save the first ->send
[19:42] RandalSchwartz not the recent one
[19:42] RandalSchwartz when HWM = 1
[19:42] sustrik it's because it connects
[19:43] RandalSchwartz so that's broken with respect to the weather example
[19:43] sustrik so it creates the queue in advance
[19:43] RandalSchwartz how do I get it to do what I want then?
[19:43] sustrik on the bind sind the queues are created on demand as peer connect
[19:43] sustrik bind PUB
[19:43] sustrik connect SUB
[19:44] RandalSchwartz can't do that. the SUB is a relay at a well-known point
[19:44] RandalSchwartz so the SUB must bind
[19:44] RandalSchwartz and when it does, it gets stale data
[19:44] sustrik true
[19:44] RandalSchwartz I suppose I could re-connect on each loop
[19:45] RandalSchwartz but that seems wasteful
[19:45] RandalSchwartz anyway, minor issue
[19:45] RandalSchwartz doesn't matter much for this app. just thinking ahead
[19:45] sustrik what's needed is more sophisticated algorithm for dropping messages in pub/sub
[19:45] sustrik presumable a rate based one
[19:45] sustrik not trivial to implement
[19:45] RandalSchwartz I'm building a git commit hook that notifies jenkins
[19:46] RandalSchwartz and maybe anyone else interested
[19:46] sustrik try speaking to mikko
[19:46] RandalSchwartz so I've got a tiny SUB-to-PUB relay going from 40000 to 40001 on this box
[19:46] sustrik he's managing the daily builds
[19:46] RandalSchwartz and my git hooks will PUB to 40000
[19:46] RandalSchwartz and everyone of interest (jenkins, monitors) will SUB 40001
[19:46] sustrik ack
[19:47] sustrik where's is the device going to run?
[19:47] RandalSchwartz on the git box
[19:47] RandalSchwartz so that's the well-known rendevous for everyone
[19:48] RandalSchwartz it also happens that the initial PUB will also come from the same box, for now
[19:48] RandalSchwartz short-lived, and ok if it fails
[19:48] RandalSchwartz actually, if I make the front-end REQ/REP, then I can notify the git-hook if anyone listened
[19:49] RandalSchwartz as long as I had some sort of timeout
[19:49] RandalSchwartz hmm
[19:51] RandalSchwartz I think I'll start with the simplest thing that could possibly work, first. :)
[19:54] alh Might I ask about a FreeBSD/ZeroMQ/Perl issue here?
[19:57] RandalSchwartz heh
[19:57] RandalSchwartz that's what I'm working on too
[19:57] mikko alh: it's better to just ask
[19:57] mikko rather than ask to ask
[19:58] alh mikko: I agree entirely, I hang out in a few #-help channels, wasn't sure that was this ones' purpose, sorry.
[19:59] alh Basically the problem I have is on FreeBSD, my Perl is compiled without threads, zeroMQ is compiled with threads, and when I try to use ZeroMQ, I get "/libexec/ /usr/local/lib/ Undefined symbol "pthread_create""
[20:00] mikko what does ldd /usr/local/lib/ give you?
[20:00] alh If I define LD_PRELOAD=/usr/local/, everything works fine
[20:00] mikko that should not be necessary
[20:01] alh
[20:01] alh mikko: I agree :)
[20:01] _iamarobot I found the same thing building pyzmq on freebsd
[20:01] alh enumerates the same problem I am having with Perl, but has no solutions
[20:02] _iamarobot pthread had to be linked explicitly by pyzmq on the platform
[20:02] alh Ports should not link threading libraries directly, but should use -pthread (which seems wrong). If I try to build it with -lpthread, it hangs when I try creating a ZeroMQ context
[20:03] alh ( )
[20:04] mikko hmm
[20:04] mikko on linux we (libzmq) link against pthreads
[20:04] mikko is this zeromq from ports or vanilla?
[20:04] alh From ports, I get the pthread issue
[20:05] mikko we have this AC_CHECK_LIB([pthread], [pthread_create])
[20:05] mikko in
[20:05] mikko i wonder if ports removes this
[20:05] mikko sec, let me check freebsd
[20:06] alh Ports does -pthread, vanilla does -lpthread
[20:06] mikko if i look at vanilla, we have => /lib/ (0x800c00000)
[20:06] mikko so do the ports make the assumption that all users of the library link against pthreads?
[20:06] mikko separately
[20:07] alh It seems that way - that the user of the library is free to dynamically choose a threading library
[20:07] alh So this may be a FreeBSD question :/
[20:07] mikko is the perl coming from ports as well?
[20:08] alh Yes
[20:08] mikko looks like freebsd issue in that case
[20:08] alh And if I build it vanilla, it links against -lpthread, and context creation freezes
[20:09] alh Alright, it was worth a try. Thanks mikko
[20:09] mikko hmm
[20:09] mikko build what vanilla?
[20:10] alh Sorry, if I download zeromq, untar, configure, make install
[20:10] mikko do you run a test script with just context creation?
[20:10] alh I try to run the Perl ZeroMQ tests and they just halt
[20:11] alh I believe statically linking is wrong, I think the FreeBSD port is doing it right honestly.
[20:12] mikko it's not linking statically
[20:13] alh Sorry, not linking in a specific threading library, but letting the application decide
[20:23] mikko;a=blob_plain;f=m4/ax_pthread.m4
[20:23] mikko something like this might make sense for checking the threading library