ZeroMq IRC Log

Tuesday November 1, 2011

[Time] Name	Message
[06:37] sand	hello everybody
[06:38] sand	i don't understand If you're using TCP, and a subscriber is slow, messages will queue up on the publisher
[07:45] sustrik	sand_: up until HWM is reached
[07:46] sustrik	then the sender blocks or messages are dropped, depending on the pattern
[07:46] sustrik	see zmq_socket(3)
[07:52] sand	i use pub-sub pattern.
[08:03] sand	i had write example use pub-sub.publisher send 10000 message , subscriber receive one message and sleep(1000) and go on.
[08:04] sand	i can't see that messages queue up on the publisher
[08:04] sand	infact messages queue up on the subscriber
[08:05] sand	something wrong with me?
[08:06] sustrik	nope, messages queue up as for downstream as they are able to get
[08:06] sustrik	do you want them to queue on the publisher?
[08:12] sand	i read the guide,and code to comfirm that,but it differen, so i puzzled
[08:15] sand	i using TCP, and a subscriber is slow(force to sleep 1s).is thart right?
[09:13] sanduu	yeah
[10:29] sustrik	mikko: hi
[10:32] mikko	hi
[10:32] mikko	whatcha need?
[10:43] sustrik	mikko: hi
[10:44] sustrik	a question: is it possible for make check to print the name of the test before running it?
[10:44] sustrik	right now it prints nothing
[10:44] sustrik	and if the test hangs up
[10:44] sustrik	it's not clear which one is hanging
[10:45] mikko	sustrik_: let me check
[10:58] mikko	sustrik_: can't see an easy way
[10:58] sustrik	mikko: ok
[10:58] sustrik	np
[10:59] mikko	apart from adding to test
[10:59] mikko	std::cerr or something in each test
[10:59] sustrik	yeah, i can do that
[16:09] k0p	hi folks
[16:09] k0p	the following page is not working for me: http://www.zeromq.org/bindings:java
[16:14] sustrik	it works ok for me
[16:15] k0p	oh, now works. but it waits for wikidot :S
[16:18] mikko	k0p: looks fine from here
[16:20] k0p	weird :S
[18:36] solvik	hi, i would like as a server, to know every clients (subscribers) who are connected to me. it seems easy but i realy can't find the method which permits that. any hints ?
[18:52] RandalSchwartz	so devices are broken in the Perl bindings?
[18:52] RandalSchwartz	this seems to be my conclusion so far. :(
[18:52] RandalSchwartz	is there a way I could write a SUB-to-PUB relay without devices?
[18:53] RandalSchwartz	I'm not sure what each side of it would look like. a SUB subscribing to ""?, and then a tight loop that ->recv and just sends it on the pub?
[19:28] sustrik	RandalSchwarz: universal device polls on both sockets and forwards messages from the first on to the second one and vice versa
[19:30] RandalSchwartz	but for PUB sockets, there'd never be any need to poll for reading, right?
[19:30] RandalSchwartz	so if I want a SUB-to-PUB proxy, a simple while (1) { $m = $sub->recv; $pub->send($m) } will work, yes?
[19:30] RandalSchwartz	or do I need to do something for $pub somwhere?
[19:31] sustrik	depends on the 0mq version you are using
[19:32] sustrik	from version 3.0 the subscriptions are propagated upstream
[19:32] sustrik	ie. from SUB to PUB
[19:32] sustrik	thus you need both directions
[19:32] RandalSchwartz	the C-level device handles that?
[19:32] RandalSchwartz	I'd have to copy that to the Perl code by hand then
[19:33] sustrik	what version are you using?
[19:33] RandalSchwartz	or figure out why ZeroMQ::Raw::zmq_device is broken
[19:33] RandalSchwartz	0.18
[19:33] RandalSchwartz	err.. 0.17
[19:33] RandalSchwartz	can't type
[19:33] sustrik	version of 0mq i mean
[19:33] RandalSchwartz	that's the latest in the CPAN
[19:34] RandalSchwartz	zmq is 2.10
[19:34] sustrik	i see
[19:34] RandalSchwartz	2.1.10
[19:34] sustrik	in that case while (1) { $m = $sub->recv; $pub->send($m) } will work
[19:34] RandalSchwartz	ok
[19:34] RandalSchwartz	but I hope someone fixes the device() call :)
[19:35] RandalSchwartz	it blows an error for me when I try to call it. wrong type for socket
[19:35] RandalSchwartz	even though I've copied the example directly
[19:35] sustrik	fill in a ticket in perl binding issue tracker
[19:35] RandalSchwartz	there are no tests to see that it broke though :(
[19:36] RandalSchwartz	I thought setting HWM to 1 would ensure only one message in an unconnected PUB socket, but that seems not to be the case
[19:36] RandalSchwartz	looks like they keep queeing until there's a reader
[19:36] RandalSchwartz	when I finally connect, I get a big burst
[19:37] RandalSchwartz	that'd be ok for this app, but I'm wondering what I did wrong
[19:37] sustrik	setting HWM after connecting, maybe?
[19:38] RandalSchwartz	well, that's transparent.
[19:38] RandalSchwartz	I call ->connect
[19:38] RandalSchwartz	and then setsockopt
[19:38] RandalSchwartz	but then I start sendign things
[19:38] RandalSchwartz	at first, the SUB isn't there
[19:38] sustrik	setsockopt() is applied only to subsequent connects
[19:38] RandalSchwartz	but when it does show up, I finally get a burst
[19:38] RandalSchwartz	oh!
[19:38] RandalSchwartz	swap those two
[19:38] RandalSchwartz	ok
[19:40] RandalSchwartz	interesting. it seems to stash the first message, and toss the later ones
[19:40] sustrik	yeah, that's the case now
[19:41] RandalSchwartz	if I leave it at 0, will it just fill memory eventually?
[19:41] sustrik	yes
[19:41] RandalSchwartz	it'd be nice to have some sort of flush
[19:41] RandalSchwartz	since "1" doesn't quite do it
[19:41] sustrik	?
[19:41] RandalSchwartz	imagine your "current weather" publish
[19:42] RandalSchwartz	you don't want the old weather to be the one in the first connect
[19:42] RandalSchwartz	you want always the latest
[19:42] sustrik	sure, that would work
[19:42] RandalSchwartz	so how does the weather publisher say "here's the newest value"
[19:42] RandalSchwartz	no - it doesn't work now
[19:42] sustrik	because PUB binds
[19:42] sustrik	and SUB connects
[19:42] RandalSchwartz	I'm seeing PUB save the first ->send
[19:42] RandalSchwartz	not the recent one
[19:42] RandalSchwartz	when HWM = 1
[19:42] sustrik	it's because it connects
[19:43] RandalSchwartz	so that's broken with respect to the weather example
[19:43] sustrik	so it creates the queue in advance
[19:43] RandalSchwartz	how do I get it to do what I want then?
[19:43] sustrik	on the bind sind the queues are created on demand as peer connect
[19:43] sustrik	bind PUB
[19:43] sustrik	connect SUB
[19:44] RandalSchwartz	can't do that. the SUB is a relay at a well-known point
[19:44] RandalSchwartz	so the SUB must bind
[19:44] RandalSchwartz	and when it does, it gets stale data
[19:44] sustrik	true
[19:44] RandalSchwartz	I suppose I could re-connect on each loop
[19:45] RandalSchwartz	but that seems wasteful
[19:45] RandalSchwartz	anyway, minor issue
[19:45] RandalSchwartz	doesn't matter much for this app. just thinking ahead
[19:45] sustrik	what's needed is more sophisticated algorithm for dropping messages in pub/sub
[19:45] sustrik	presumable a rate based one
[19:45] sustrik	not trivial to implement
[19:45] RandalSchwartz	I'm building a git commit hook that notifies jenkins
[19:46] RandalSchwartz	and maybe anyone else interested
[19:46] sustrik	try speaking to mikko
[19:46] RandalSchwartz	so I've got a tiny SUB-to-PUB relay going from 40000 to 40001 on this box
[19:46] sustrik	he's managing the daily builds
[19:46] RandalSchwartz	and my git hooks will PUB to 40000
[19:46] RandalSchwartz	and everyone of interest (jenkins, monitors) will SUB 40001
[19:46] sustrik	ack
[19:47] sustrik	where's is the device going to run?
[19:47] RandalSchwartz	on the git box
[19:47] RandalSchwartz	so that's the well-known rendevous for everyone
[19:48] RandalSchwartz	it also happens that the initial PUB will also come from the same box, for now
[19:48] RandalSchwartz	short-lived, and ok if it fails
[19:48] RandalSchwartz	actually, if I make the front-end REQ/REP, then I can notify the git-hook if anyone listened
[19:49] RandalSchwartz	as long as I had some sort of timeout
[19:49] RandalSchwartz	hmm
[19:51] RandalSchwartz	I think I'll start with the simplest thing that could possibly work, first. :)
[19:54] alh	Might I ask about a FreeBSD/ZeroMQ/Perl issue here?
[19:57] RandalSchwartz	heh
[19:57] RandalSchwartz	that's what I'm working on too
[19:57] mikko	alh: it's better to just ask
[19:57] mikko	rather than ask to ask
[19:58] alh	mikko: I agree entirely, I hang out in a few #-help channels, wasn't sure that was this ones' purpose, sorry.
[19:59] alh	Basically the problem I have is on FreeBSD, my Perl is compiled without threads, zeroMQ is compiled with threads, and when I try to use ZeroMQ, I get "/libexec/ld-elf.so.1: /usr/local/lib/libzmq.so.1: Undefined symbol "pthread_create""
[20:00] mikko	what does ldd /usr/local/lib/libzmq.so.1 give you?
[20:00] alh	If I define LD_PRELOAD=/usr/local/libpthread.so, everything works fine
[20:00] mikko	that should not be necessary
[20:01] alh	http://paste.scsys.co.uk/155794
[20:01] alh	mikko: I agree :)
[20:01] _iamarobot	I found the same thing building pyzmq on freebsd
[20:01] alh	https://github.com/lestrrat/ZeroMQ-Perl/issues/18 enumerates the same problem I am having with Perl, but has no solutions
[20:02] _iamarobot	pthread had to be linked explicitly by pyzmq on the platform
[20:02] alh	Ports should not link threading libraries directly, but should use -pthread (which seems wrong). If I try to build it with -lpthread, it hangs when I try creating a ZeroMQ context
[20:03] alh	( http://www.freebsd.org/doc/en/books/porters-handbook/dads-pthread.html )
[20:04] mikko	hmm
[20:04] mikko	on linux we (libzmq) link against pthreads
[20:04] mikko	is this zeromq from ports or vanilla?
[20:04] alh	From ports, I get the pthread issue
[20:05] mikko	we have this AC_CHECK_LIB([pthread], [pthread_create])
[20:05] mikko	in configure.in
[20:05] mikko	i wonder if ports removes this
[20:05] mikko	sec, let me check freebsd
[20:06] alh	Ports does -pthread, vanilla does -lpthread
[20:06] mikko	if i look at vanilla, we have libthr.so.3 => /lib/libthr.so.3 (0x800c00000)
[20:06] mikko	so do the ports make the assumption that all users of the library link against pthreads?
[20:06] mikko	separately
[20:07] alh	It seems that way - that the user of the library is free to dynamically choose a threading library
[20:07] alh	So this may be a FreeBSD question :/
[20:07] mikko	is the perl coming from ports as well?
[20:08] alh	Yes
[20:08] mikko	looks like freebsd issue in that case
[20:08] alh	And if I build it vanilla, it links against -lpthread, and context creation freezes
[20:09] alh	Alright, it was worth a try. Thanks mikko
[20:09] mikko	hmm
[20:09] mikko	build what vanilla?
[20:10] alh	Sorry, if I download zeromq, untar, configure, make install
[20:10] mikko	do you run a test script with just context creation?
[20:10] alh	I try to run the Perl ZeroMQ tests and they just halt
[20:11] alh	I believe statically linking is wrong, I think the FreeBSD port is doing it right honestly.
[20:12] mikko	it's not linking statically
[20:13] alh	Sorry, not linking in a specific threading library, but letting the application decide
[20:23] mikko	http://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_pthread.m4
[20:23] mikko	something like this might make sense for checking the threading library

ZeroMq Home

Tuesday November 1, 2011