ZeroMq IRC Log

Thursday March 24, 2011

[Time] Name	Message
[00:05] mikko	pieterh: did you push the latest make dist fixes?
[00:12] Guthur	jeckstein, any success getting a minimal test case?
[00:14] jeckstein	no, its going to take me a little while to do that, i just compiled and loaded the newest clrzmq2 (2.1.0) and am testing it on my application now
[00:18] Guthur	ok, cool
[00:37] jaten	hey guys, I'm getting a core dump in /lib/libpthread.so in the head of the git repos for zeromq, running the first server.cpp example in the Guide. This is on ubuntu 10.10 with /lib/libpthread.so.0 from libc 2.12.1-0ubuntu10.2. It doesn't happen with with the earlier 10.04 with libc 2.11.1-0ubuntu7.8 or 7.7, which has /lib/libpthread.so.0 version 2.11.1. Is this a known issue? If not where would I post for assistance/bug report?
[00:43] Guthur	jaten, you should probably raise a ticket on github, and maybe fire a description on to the mailing list
[01:51] jaten	Guthur: okay, thanks. I'll do that.
[06:16] ferrouswheel	is there any support for message priorities or QoS? saw various discussions on the mailing list, but not sure if it eventuated into anything?
[06:26] sustrik	ad message priorities: no, you have to use different message feeds to get different priorities
[06:26] sustrik	ad QoS: it can be set on network level
[06:27] sustrik	http://www.zeromq.org/whitepapers:traffic-monitoring-v20
[06:30] ferrouswheel	sustrik, thanks - that's what i suspected
[06:39] ferrouswheel	i want to use zeromq for an (initially) inprocess rpc, but would like to decrease the priority of clients that make more requests. I can see various ways to do this, but am curious if a particular design makes more logical sense than others when using zeromq...
[06:46] sustrik	ferrouswheel: create several REP sockets, one for each priority
[06:46] sustrik	read requests from high priority sockets first
[06:47] sustrik	alternatively, you can add priority scheduling to 0mq core directly, which is something that's on the roadmap already
[06:50] ferrouswheel	sustrik, okay - the async server (fig 47) in the guide also seems appropriate... deciding on which worker to send a client to, depending on the request rate
[06:50] ferrouswheel	but i think i'll leave 0mq core modifications until i've actually used and understand 0mq better ;-)
[06:50] sustrik	ok
[08:28] ianbarber	fully confused by git. i pulled in changes from the upstream, but it seems to have committed it as a commit, so now if I do a pull req it includes those
[08:28] ianbarber	bunch of zguide examples and stuff
[08:42] Steve-o	hence it's called "git"
[08:42] Steve-o	I think it's git-rebase for the equivalent of svn update
[08:43] ianbarber	ah cool, will have a look
[09:55] private_meta	I've got a tcp connection, my client is stopping, but it won't quit the program, it hangs at "return" in main in mailbox_t::recv(), any idea why?
[09:56] private_meta	It hangs there at the termination of the context, although the connection should have been set not to linger (zmq_linger with a value of 0)
[10:01] pieterh	private_meta: you have a socket open somewhere
[10:02] pieterh	you have to close every single socket and you have to first set linger to zero on any socket which may still be doing some work (sending, or connecting)
[10:03] pieterh	this is the rather painful way 2.1 works but it can be fixed in the language binding
[10:03] private_meta	pieterh: linger is on zero on EVERY socket that is open, and how can I close the sockets in C++ apart from calling their destructor (which I'm already doing)?
[10:04] pieterh	I see two possibilities... either you have forgotten a socket somewhere, or the C++ binding is buggy
[10:05] private_meta	let me try something
[10:05] pieterh	private_meta: I've just checked the C++ binding code and there is an explicit close method
[10:06] pieterh	which the destructor just wraps
[10:06] private_meta	Ah ok... the socket was a pointer in there, and destructor of a pointer isn't called implicitly
[10:06] pieterh	right
[10:06] private_meta	only if it's a non-pointer object (which I use normally)
[10:07] pieterh	in zapi, I'm delingering/closing sockets automatically when the context is destroyed
[10:07] pieterh	it makes life so much simpler
[10:08] private_meta	zapi?
[10:08] pieterh	the new high-level C API
[10:08] pieterh	http://zero.mq/c
[10:09] private_meta	Well, if the socket object were destroyed that would have been the case, but it wasn't
[10:10] pieterh	Sure, you still need to go set linger to zero on sockets
[10:11] private_meta	I did some other stuff for a week so I'm still at translating the guide examples
[10:11] pieterh	into C++?
[10:11] private_meta	yeah
[10:11] pieterh	was it you who asked about modifying the C++ API?
[10:12] private_meta	Well, I asked about using pointers in the C++ binding
[10:12] private_meta	and the zmq::version
[10:12] pieterh	yeah, and that led to thinking about splitting the C++ binding into its own project
[10:12] private_meta	Yes
[10:13] private_meta	As far as I remember, it did
[10:13] pieterh	you already translated zhelpers, zmsg, etc. right?
[10:13] private_meta	Yes
[10:13] pieterh	so... I have a small suggestion
[10:13] pieterh	let's make zapi++
[10:13] private_meta	I didN't get to continue until now, so I'm at ppworker/ppqueue now
[10:13] pieterh	zapi basically refactors the zhelpers/zmsg APIs into something neater
[10:15] private_meta	Sounds ok, but I'd recommend not using pointers in function parameters
[10:15] pieterh	indeed
[10:16] pieterh	so the thing here is that I'm going to migrate the C examples onto zapi
[10:16] pieterh	maybe not all, but the more complex ones, in any case
[10:17] pieterh	since you know the zhelpers/zmsg API well, would you take a look at zapi?
[10:18] private_meta	sure
[10:18] pieterh	it'd be really useful to get your design of a C++ binding at the same level
[10:18] private_meta	but right now I need to fix this (somewhat) small ppworker problem I have, it doesn't get a connection
[10:19] pieterh	if you need a hand, shout, I'm happy to help debug code
[10:19] pieterh	paranoid pirates :-)
[10:20] private_meta	Apparently you dislike curly brackets in example code
[10:20] pieterh	I use them only where needed
[10:20] pieterh	code minimalism is kind of a style with me
[10:20] private_meta	Every programming teacher tells you to not do that ;)
[10:20] pieterh	that's why they're teachers
[10:20] pieterh	not programmers
[10:20] private_meta	It destroys readability
[10:21] pieterh	there's only one way to learn how to program
[10:21] pieterh	to quote Zed Shaw, by "programming, motherfucker"
[10:21] private_meta	Oh, I tend to agree with them, and many programming teachers we have here had huge programming projects already
[10:21] pieterh	so you mean for example an if statement with only one line, no braces?
[10:22] private_meta	You add one line to an if statement, tend to forget you didn't have brackets
[10:22] pieterh	sure
[10:23] pieterh	then this is one of my bad habits, removing braces when they're not needed
[10:23] private_meta	Code shouldn't be hard to read, especially if adding brackets doesn't change the compiled results
[10:23] pieterh	readability comes only from consistency, adding or removing brackets doesn't affect it IMO
[10:23] pieterh	however, if you want, go through all my examples and add brackets
[10:23] private_meta	Consistency also means you use brackets, so use them everywhere :)
[10:23] pieterh	my only rule is 100% consistency, really
[10:24] pieterh	I consistently don't use them when I can avoid them
[10:24] pieterh	it creates extra lines that do nothing, grouping a single line is pointless IMO
[10:24] private_meta	sure, it seems pointless if you like minimalism
[10:24] pieterh	indeed
[10:24] pieterh	which I really do
[10:25] pieterh	functional minimalism, of course, so using clear names, blank space, etc.
[10:25] pieterh	but it is probably just a bad habit... :-)
[10:25] private_meta	I mean, using "cond ? statem1 : statem2" would be minimalistic
[10:25] pieterh	yes, but unreadable in practice
[10:25] private_meta	still you should only do it in very special cases, 'cause mostly it's unreadable
[10:26] pieterh	I do that in some cases when it's obvious
[10:26] private_meta	yes
[10:26] pieterh	the test over time is whether you can easily reuse code
[10:26] pieterh	I was able to put together 3K lines of code in zapi, fully tested and documented, in 1 day
[10:26] private_meta	by the way, is it your style to put the opening bracket on functions in the next line, but the opening brackt of ifs and elses on the same?
[10:27] pieterh	yes, this is historic, we reused the GNU style for functions but our own style for statements
[10:27] private_meta	yeah, you used that argument already with a simple text editor and printf debugging, but that still didn't make me use that
[10:27] pieterh	I debug zapi using printfs and valgrind
[10:27] pieterh	it's extraordinarily effective
[10:28] private_meta	not for me :)
[10:28] pieterh	I'm actually shocked at being able to write masses of C code and then hammer it into shape with brute force, and it works as designed
[10:29] pieterh	trick is to start with the API, and tests, and then implement & test small pieces at a time
[10:29] pieterh	presumably C++ eliminates a lot of the pointer errors we make in C
[10:30] pieterh	anyhow, if zapi++ takes off, you can impose your style on that :-)
[10:30] private_meta	hmm
[10:31] private_meta	I've got a weird error in ppqueue that doesn't let lpclient connect
[10:31] pieterh	can you paste the code somewhere?
[10:31] private_meta	sec
[10:34] private_meta	Well, I don't quite see where the error is exactly... If I use the c version of ppqueue and ppworker, lpclient works fine
[10:34] private_meta	https://gist.github.com/5f6a4b0a41981a66d99f https://gist.github.com/0d7456f199b546ffe5e9 https://gist.github.com/253bc2038df7d9b3ba00
[10:34] pieterh	you can put multiple files into one gist BTW
[10:35] private_meta	files, yeah, but I wasn't logged in at my virtual machine in gist
[10:35] private_meta	so I didn't have my files here
[10:35] pieterh	so ppqueue doesn't poll the client at all, unless there are workers, that's the first thing
[10:36] pieterh	code looks nice, very neat...
[10:37] private_meta	My test setup is starting ppqueue, 1 ppworker, 1 lpclient
[10:37] pieterh	have you already traced activity by putting in msg.dump() all over the place?
[10:37] private_meta	ppworker uses heartbeats
[10:37] private_meta	but lpclient can't connect
[10:37] pieterh	'can't connect' means 'does not get a reply within X seconds'
[10:37] private_meta	sec
[10:37] pieterh	so you need to trace activity in the queue
[10:38] private_meta	I: connecting to server...
[10:38] private_meta	I: connecting to server...
[10:38] private_meta	W: no response from server, retrying...
[10:38] private_meta	I: connecting to server...
[10:38] pieterh	sure
[10:38] pieterh	that can mean anything
[10:38] pieterh	it's not really a connection failure
[10:38] Guthur	that could be seem behaviour I experience with teh default timeout
[10:38] Guthur	It was too aggressive in it's timeout
[10:39] Guthur	relaxing the intervals helped
[10:39] pieterh	Guthur: good memory...!
[10:39] private_meta	if I'm using ppworker and ppqueue in the c version, lpclient doesn't receive timeouts or anything
[10:39] pieterh	I fixed this in the C code, hang on, let me check...
[10:39] Guthur	pieterh: it had me scratching my head long enough that I have deep grove to remind me, hehe
[10:40] Guthur	groove*
[10:40] private_meta	by the way, using the c++ client takes way more time than the c client
[10:41] pieterh	Guthur: it was in the ppworker when it simulated a CPU overload, that was always more than the client's full timeout
[10:41] private_meta	meaning, using c queue and worker normally, then alternating between lpclient for c and c++, the c version is way faster
[10:41] pieterh	private_meta: weird, that's not expected
[10:42] pieterh	sounds like a timer problem of some kind
[10:42] pieterh	there should be no measurable (let alone visible) difference between the two
[10:43] pieterh	private_meta: so please add msg.dump on everything the broker recvs and sends, that's step 1
[10:45] private_meta	k, receiving is always done in the zmsg constructor, so I'll add the dump there. should I paste the output of ppqueue?
[10:45] pieterh	sounds good
[10:45] pieterh	that will show us what the client is sending
[10:46] private_meta	Should I also dump messages in worker and client or just queue for now?
[10:46] pieterh	then we also want to see what ppqueue is sending
[10:46] pieterh	start with queue
[10:46] private_meta	k
[10:46] pieterh	presumably there's a problem between queue and worker
[10:46] pieterh	so next, worker
[10:51] private_meta	argh
[10:51] private_meta	how do I redirect output AND stderror to file?
[10:51] pieterh	myprog > err.log 2>&1
[10:52] private_meta	"Aborted"
[10:53] private_meta	ah ok now
[10:53] private_meta	argh
[10:54] pieterh	sounds positive... :-)
[10:54] pieterh	or else you're at the dentist... "open wider, please!"
[10:54] ianbarber	pieterh: sent a pull req for the zguide, noticed that it was notifying sustrik though, which seems probably incorrect
[10:55] private_meta	no, that didn't redirect everything
[10:55] pieterh	ianbarber: weird, but presumably he's a watcher of that repo?
[10:55] private_meta	ok, screw it, I'll just copy from console if that doesn't work
[10:57] private_meta	pieterh: https://gist.github.com/2a9823bb9163202ec026
[10:57] private_meta	pieterh: that's the log from starting ppworker till disconnecting (killing) ppworker
[10:58] private_meta	in the middle, I started lpclient and it terminated by itself
[10:58] pieterh	ianbarber: got them, thanks!
[10:58] pieterh	private_meta: ok, reading that...
[10:59] pieterh	private_meta: I assume it should say 'after recv' not 'before recv'...
[10:59] private_meta	ah yes
[10:59] pieterh	so you have a bug in the worker
[10:59] pieterh	try running your queue, your client, my worker
[11:00] pieterh	I'll let you figure out what the error is, it's clear from the trace
[11:00] pieterh	:-)
[11:00] private_meta	Doesn't work
[11:00] private_meta	same error
[11:01] pieterh	that's unpossible! :-)
[11:01] pieterh	ok, try my worker, my client, your queue...
[11:02] pieterh	hang on, that's just the same, sorry...
[11:02] pieterh	your worker, my queue, your client...
[11:03] pieterh	the error shows on lines 32-41
[11:03] private_meta	should be the same, let me check (i think i tried that already)
[11:03] private_meta	No, doesn't work
[11:04] private_meta	sec, gotta fetch a delivery, bb in 5 minutes or so
[11:04] pieterh	then you have a problem in BOTH your worker and your queue, perhaps because it's in the msg class
[11:04] pieterh	cyl, then
[11:08] private_meta	i hope this time the tablet pc is alright
[11:08] private_meta	pieterh: ok, then what might be the problem in the zmsg class? any idea?
[11:08] pieterh	private_meta: look at lines 32-41 of your trace
[11:08] pieterh	the worker sends back (or the queue wrongly receives) the worker address twice
[11:09] pieterh	when it should be worker address, then original client address
[11:09] pieterh	thus the reply to the client gets sent back to the worker instead
[11:09] private_meta	hmm
[11:09] pieterh	you see the wrong addresses, right?
[11:10] private_meta	yes
[11:10] pieterh	this is kind of standard when using envelopes, they are tricky to get right
[11:11] pieterh	you can now add printfs to msg where it's recving and sending frames
[11:11] private_meta	So this might be a problem the selftest didn't catch?
[11:11] pieterh	for sure
[11:12] pieterh	hmm, there is a unit test for this but it's not double-checking every frame
[11:12] pieterh	could improve that
[11:13] private_meta	ah goddammit, i need to fetch a usb dvd drive again...
[11:13] pieterh	:-)
[11:13] private_meta	well, hp guys didn't think it would be a good idea to install a 64bit operating system on their latest notebooks
[11:13] pieterh	32 bits should be enough for anyone
[11:13] pieterh	maybe it's a subtle hint to install Ubuntu...
[11:14] private_meta	No, I can't use Linux in everyday life, creates too many problems
[11:14] private_meta	(Murphies Law and everything)
[11:14] pieterh	my sympathies...
[11:15] private_meta	Everytime I tried to use something on Linux for personal use, it failed in a way that wasn't fixable
[11:15] pieterh	weird, I've been using Linux for 10+ years and it's irreplaceable for me
[11:15] private_meta	I've been using Linux for that long as well, but I can't use it as a primary system
[11:16] private_meta	apart from that, as a gamer I need Windows anyway
[11:16] pieterh	10+ years for primary personal use, since early 1990's for work
[11:16] pieterh	ah, games
[11:16] private_meta	Well, some like it, others don't, I tend to be someone fond of games :)
[11:17] private_meta	But it's not like playing games is the only reason ;)
[11:17] pieterh	are any game engines using 0MQ yet, I wonder...
[11:18] pieterh	"shoots 4M bullets per second!!!"
[11:27] mikko-	mato
[11:27] mikko-	there?
[11:27] mikko-	pieterh: you remember ben from zeromq london meetup?
[11:27] pieterh	sure
[11:27] mikko-	he works in gaming industry
[11:28] pieterh	right, he was "putting 0MQ everywhere"
[11:28] mikko-	you could pry from him on which games he works on
[11:28] pieterh	nah, I don't really care :-)
[11:28] ianbarber	his latest, total war shogun 2 is out now! i'm sure he'd support your purchasing :)
[11:29] pieterh	I used to write games but don't play them
[11:29] Guthur	I really liked the total war series
[11:29] Guthur	don't game much anymore though
[11:30] ianbarber	shogun 2 is actually quite a lot of fun. i'm all mac everywhere now though, and they're windows only though
[11:30] Guthur	no time these days unfortunately
[11:31] ianbarber	i think it would be very beneficial for engines though, everyone has to deal with many cores these days
[11:32] ianbarber	x360 is what, 6? ps3 7. and most decent desktops are at least 4
[11:32] pieterh	I recall people asking about this ages ago, when 0MQ was 0.something
[11:45] pieterh	mikko-_: you got a second?
[11:48] mikko-	pieterh: yes
[11:49] pieterh	I'd like to start on a new phase of the zmq packaging
[11:49] pieterh	that is, start including other parts of the community into "ZeroMQ/x.y"
[11:49] pieterh	2.1 is libzmq and libpgm
[11:50] pieterh	I'm thinking 2.2 will be libzmq, libpgm, libzapi, and maybe one or two other language bindings
[11:51] pieterh	any thoughts on that? libzapi uses autotools, should be simple
[11:53] sustrik	given that it's exactly 1 day old, it sound a bit premature to include it into stable
[11:53] pieterh	sustrik: that's why I said 2.2...
[11:53] sustrik	ah, ok
[11:53] pieterh	:-) indeed, it's not ready for stable
[11:54] pieterh	it strikes me that libpgm integration doesn't have to be a special case
[11:55] pieterh	sustrik: btw, I used inproc pipes to create a lockfree object, as you do inside core
[11:55] pieterh	it's tricky because the object is actually the context wrapper, but it's really neat otherwise
[11:58] mikko-	hmm
[11:58] mikko-	pieter: you mean into same package?
[11:59] pieterh	yes, I mean expand the scope of the download
[11:59] mikko-	distros like if you keep things separate
[11:59] pieterh	different issue
[11:59] pieterh	distros can repackage individual libraries
[11:59] pieterh	users like getting one package
[11:59] pieterh	imagine having to download & build libpgm separately
[11:59] mikko-	thats what i would normally expect to do
[12:00] mikko-	:)
[12:00] mikko-	install dependencies before the software
[12:00] pieterh	yes, but you're a developer not a sysadmin
[12:00] mikko-	from sysadmin pov i would package rpm and install the packages that are defined in "BuildRequires" stanza
[12:00] mikko-	assuming everything is available as rpm
[12:00] pieterh	we can still publish minimal packages for distros
[12:01] mikko-	do you want these to be built together as well?
[12:01] mikko-	or just packaged together?
[12:01] pieterh	well, there are two main reasons for packaging together
[12:01] pieterh	a. convenience for installing
[12:01] pieterh	b. tested as a single unit
[12:02] pieterh	so for b, yes, we'd also build/test the collection
[12:02] pieterh	as we would do for libpgm, it's exactly the same
[12:02] pieterh	otherwise 0MQ cannot scale
[12:03] mikko-	where do you draw the line what should be packaged?
[12:03] mikko-	should all language bindings be bundled?
[12:03] pieterh	well, I don't draw a line as such
[12:03] mikko-	what about devices?
[12:03] pieterh	what I'd do is do this step by step and stop when it's no longer worth going further
[12:03] pieterh	imagine there's a separate project with overall tests
[12:03] pieterh	I'd probably package that
[12:04] pieterh	I'd probably package all available documentation
[12:04] pieterh	but gradually, otherwise it's indigestable
[12:05] pieterh	this also makes it easier to e.g. make a Windows kit, it's the same set of packages with a MSI frontend
[12:06] pieterh	so the release repositories are where this happens, presumably using submodules
[12:07] pieterh	this also gives project developers something to aim at
[12:07] Guthur	I have to admit that I would take a similar point of view as mikko
[12:07] pieterh	many people will, for sure
[12:07] Guthur	I have a similar decision to make with clrzmq2
[12:08] pieterh	if you actually look at the process of getting 0MQ working with language XYZ, it's often a real pain
[12:08] Guthur	prepackaging libzmq.dll would allow the binding to seamlessly support 32/64 bit environment
[12:08] pieterh	indeed
[12:08] Guthur	but I would rather have the users get the libzmq straight from the 'golden' source
[12:09] pieterh	If I download 0MQ for Windows I'd expect to get clrzmq2 included
[12:09] Guthur	so bug fixes etc can be propagated
[12:09] pieterh	well, as an option
[12:09] pieterh	but mostly, it's asking users to do extra work
[12:09] pieterh	that always eliminates a large chunk of users
[12:10] Guthur	yeah, I'm aware I maybe be falling into the worse is better mind set
[12:10] pieterh	people who don't (yet) have an emotional connection to a project simply won't tolerate pain to start using it
[12:11] pieterh	it has to be really, really simple to "get" 0MQ
[12:11] pieterh	after that, it can become more and more complex, that's fine
[12:11] pieterh	but the first taste must not be complex or difficult
[12:12] pieterh	IMHO
[12:12] Guthur	sure, someone is actually looking to contribute something to clrzmq2 to improve the 32/64 bit issue
[12:12] Guthur	that issue would even more simply removed by prepackaging
[12:13] pieterh	so let's look at making a Windows package?
[12:13] pieterh	libzmq+clrzmq2+libzapi
[12:13] pieterh	+libpgm
[12:14] Guthur	I'll put looking at a WiX project for it on my ToDo list
[12:14] pieterh	WiX = windows installer?
[12:14] Guthur	WiX is a windows installer declarative language
[12:14] pieterh	nice
[12:15] Guthur	yeah only down side is it uses XML based grammar
[12:15] pieterh	XML grammars can be neat if done right but usually they're not
[12:15] Guthur	the worlds irrational love of XML pains me
[12:16] pieterh	Well, I've written maybe 100k lines of declarative XML and it works fine if you're minimalist
[12:16] Guthur	there is some many attributes and elements that it quickly becomes messy
[12:16] Guthur	...for WiX
[12:17] pieterh	http://download.imatix.com/mop/beginning.html for example
[12:17] pieterh	anyhow, WiX...
[12:17] pieterh	if you decide to start a packaging project, let's make a separate repo for that
[12:17] pieterh	not inside clrzmq2
[12:17] Guthur	on the other hand I like sexp, which a lot of people don't like
[12:18] Guthur	pieterh: ok
[12:30] mikko-	pieter
[12:30] mikko-	the doc generation is broken
[12:30] mikko-	in mnay ways
[12:31] pieterh	which project?
[12:31] mikko-	:)
[12:31] pieterh	rfc?
[12:31] mikko-	zeromq
[12:31] mikko-	the wikidot api thingy
[12:31] pieterh	hang on a sec...
[12:31] mikko-	1) the call to wikidot fails with "Unauthorised"
[12:31] mikko-	2) it prints out your API key in the error message
[12:31] pieterh	yes, I pinged you about that, you missed that
[12:31] pieterh	Wikidot upgraded their API for security reasons
[12:32] mikko-	i miss so many things using this web irc client
[12:32] pieterh	you need to be a member of the rfc site to publish to it, that's one thing
[12:32] pieterh	afair I did make you member, let me check
[12:32] pieterh	s/rfc/api/
[12:33] mikko-	it uses pieterh user
[12:33] pieterh	you should be using your own API key by now :-)
[12:33] pieterh	you were authorized ages ago
[12:33] mikko-	oh, ok
[12:33] pieterh	mine changed, old one is no longer valid
[12:33] mikko-	will check that this evening
[12:33] mikko-	good
[12:34] pieterh	what's your wikidot user id?
[12:34] pieterh	i need to make you member of the site
[12:34] pieterh	mkoppanen, I assume, you're invited
[12:34] pieterh	you were already invited, mikko :-)
[12:35] pieterh	accept the invite, grab your own API key, fix your environment, and it'll work
[12:36] pieterh	meanwhile I'll update that site for 2.1.3
[12:53] Guthur	pieterh: actually in relation to clrzmq2 packaging, there is something called NuGet
[12:53] Guthur	which I think is something similar to Maven
[12:53] pieterh	Guthur tries to confuse pieterh by throwing random words at him
[12:53] Guthur	similiar to ANT with dependency repo management
[12:53] private_meta	This is great
[12:54] Guthur	hehe I was just trying to confuse you more with that last one
[12:54] private_meta	I have VirtualBox Virtual Machines running on Windows
[12:54] pieterh	private_meta: VirtualBox is really useful, indeed
[12:54] private_meta	As soon as I pull up the Microsoft VirtualPC Settings dialog, every VirtualBox Machine crashes
[12:54] Guthur	private_meta: I go the other way round VM windows on Linux, heeh
[12:54] pieterh	oh, lol
[12:54] pieterh	VirtualPC has the VirtualBox killer option enabled!
[12:54] private_meta	great, isn't it?
[12:55] Guthur	Virtual Box is pretty neat
[12:55] pieterh	it is, indeed... I used it to install a Windows iTunes to reformat my old iPod so I could use that under Ubuntu...
[12:55] private_meta	It has it's disadvantages, but there is only so much choice you have if you go the gratis way
[12:55] Guthur	oracle is not renowned for their generosity
[12:55] private_meta	Not really, no
[12:56] pieterh	Guthur: the GPL doesn't depend on generosity...
[12:56] private_meta	That's not the problem imho
[12:57] private_meta	Oracle owns the majority of the developers working on VirtualBox
[12:57] private_meta	That means, whatever they want in, they can put in
[12:57] private_meta	Same with Java or OpenOffice
[12:57] private_meta	Now, as seen with OpenOffice, the only way to go against that would be a fork
[12:57] private_meta	and that's never an ideal solution
[12:57] pieterh	I doubt Oracle is worse than, say, IBM, who have lots of people working on Apache projects
[12:58] pieterh	Oracle will happily do what IBM does, use free software to hurt competitors
[12:59] private_meta	The good/bad thing about doubt is that it's not the same thing as proof
[13:00] pieterh	well, Sun weren't particularly great with FOSS either
[13:00] pieterh	shrug, there is no certainty in life, and that's probably a good thing
[13:01] private_meta	Well, seeing it in a different way, when you analyze evolution, and you think about mutation, it's not always the "better" solution that survives
[13:01] private_meta	it's the stronger one...
[13:01] private_meta	Oracle might be stronger, doesn't mean it's better for everyone
[13:01] pieterh	private_meta: actually, by the very definition of selection, it is always the better one... :-)
[13:02] pieterh	but business is not evolutionary, really
[13:02] private_meta	in some way it is
[13:02] pieterh	well, actually yes, it is
[13:02] pieterh	and large firms are like large predators
[13:02] private_meta	it's just a redefinition of "better"
[13:02] pieterh	yes, precisely
[13:03] pieterh	large firms can be extraordinarily greedy, damaging, etc. but they are also slow, stupid, cautious
[13:03] Guthur	0MQ is better, so we win, hehe
[13:03] pieterh	100 genius brains, sitting in a corporate office, cannot compete with 100 happy brains, organized freely
[13:04] pieterh	we're better, therefore 0MQ is better
[13:04] stimpie	if we find out why my application gets slow when using inproc to communicate between java threads
[13:05] pieterh	stimpie: how are you measuring it?
[13:06] stimpie	systems with network: p-> b ->s where b uses a thread to receivce tcp and an inproc socket to sent to another thread
[13:06] stimpie	when b is a single thread I get 200k m/s
[13:07] stimpie	adding a dispatcher to another thread drops speed to 70k m/s
[13:07] stimpie	I would expect the performance penalty in througput would be almost zero
[13:08] pieterh	stimpie: how many cores are you running on?
[13:08] pieterh	private_meta: http://www.devilswiki.com/wiki:collective-intelligence
[13:08] stimpie	8 available 3 are used
[13:09] pieterh	do you see a CPU usage increase?
[13:09] pieterh	there are two possible causes, IMO
[13:09] pieterh	a. actually doing extra work, e.g. recoding messages
[13:10] pieterh	b. round-tripping, i.e. doing req/rep on each message
[13:12] stimpie	extra work is minimal, all sockets are XREP
[13:12] pieterh	no acknowledgments?
[13:12] stimpie	none
[13:12] stimpie	iam sending and receiving on a single socket
[13:13] stimpie	could that cause some race condition?
[13:13] pieterh	hmm, interesting. So do you see CPU usage or memory usage going up?
[13:14] pieterh	presumably messages queues will back up and memory use will increase
[13:14] pieterh	sounds like something in the Java binding... I've not played much with that
[13:15] sustrik	if you are sending and receiving on the same socket from different threads, that can result in race condition
[13:17] stimpie	cpu usage is ~230% memory usages appears to keep increasing
[13:17] pieterh	stimpie: are you using the same socket from more than one thread?
[13:18] stimpie	one sockets sends to multiple threads
[13:18] stimpie	each thread has his own socket
[13:18] pieterh	something is doing a lot of work... how large are these messages?
[13:19] stimpie	message size is about 40 bytes, I will try using 2 sockets, one for sending one for receiving
[13:19] pieterh	that won't help IMO
[13:19] pieterh	stimpie: even your single thread case is abnormally slow
[13:19] pieterh	200K per second is not what you'd expect
[13:20] pieterh	best approach IMO would be to verify Java binding performance with a simple test case
[13:21] pieterh	two threads, send messages from A to B over inproc
[13:21] stimpie	oke let me write something, just a sec
[13:22] pieterh	problem can be in 3 layers, libzmq, jzmq, or your code
[13:22] pieterh	this test should tell us what layer is guilty
[14:02] stimpie	while(true) send(); //is not nice on inproc
[14:02] stimpie	duration: 1961.39 ms, messages:1000000, speed: 509842.88 m/s
[14:02] stimpie	xrep inproc->xrep inproc
[14:07] stimpie	ps. this is on my 'slow' laptop
[14:13] stimpie	my test for 2way messages appears to deadlock, see http://pastebin.com/K20P6YhZ
[14:14] stimpie	aha my mistake
[14:16] stimpie	performance on with a 2 way ping-pong
[14:16] stimpie	duration: 3871.77 ms, messages:100000, speed: 25827.98 m/s
[14:21] sustrik	what's wrong with that?
[14:22] stimpie	nothing
[14:22] stimpie	2 threads both sending and receiving 1M messages (not interleaved)
[14:22] stimpie	duration: 531.86 ms, messages:100000, speed: 188020.73 m/s
[14:22] sustrik	is it 25k rountrips/sec
[14:22] sustrik	?
[14:22] stimpie	yes roundtrips
[14:22] sustrik	that's 40us per roundrip
[14:22] sustrik	i.e. 20us single-way
[14:22] pieterh	stimpie: you said you weren't doing round trips in your application
[14:22] sustrik	that sounds reasonable, given that JNI and java is on the stack
[14:23] stimpie	pieterh, its just a test case
[14:24] pieterh	sure, but we want a test case that simulates the basic flow you're using
[14:24] stimpie	so far everything appears fine with jzmq+zmq
[14:24] pieterh	i.e. xrep to xrep over inproc, one-way
[14:24] pieterh	hang on, you did that, 509K/sec...
[14:25] pieterh	and 188k send/recv...
[14:25] pieterh	IMO there's something with the Java threading
[14:25] stimpie	the send/recv should be double its only measured on 1 side
[14:28] stimpie	I do believe my setup with a client -> dispatcher -> thread -> dispatcher -> client should not use one socket to connect thread and dispatcher
[14:29] stimpie	probably I should run each dispatcher in his own thread
[14:30] stimpie	that should give a flow of ~500K m/s
[14:32] pieterh	looks like that's the jzmq limit per thread, on your 'slow' machine
[14:38] mato	mikko-_: hi
[14:39] mato	actually, more a question for pieterh
[14:39] pieterh	mato: shoot
[14:39] mato	pieterh: how soon until 2.1.4? i can either arm-twist the debian build to work with 2.1.3 or wait for 2.1.4 (re the openpgm thread)
[14:40] pieterh	I can make 2.1.4 this weekend, I'd like to wait a day or two for more feedback on use
[14:40] mato	ok, fine
[14:40] pieterh	if you are blocked by this, I can make it sooner, or give you a preview
[14:40] pieterh	but you need an official package, don't you
[14:41] mato	pretty much
[14:41] mato	don't worry about it, it can wait till the weekend
[14:41] pieterh	ok, that'll work then
[14:41] mato	but, i need to test the fixes. so, if i want "what will become 2.1.4", that is in zeromq2-1.git ?
[14:42] pieterh	yes, the 2-1 git is the to-be-released 2.1.4
[14:42] pieterh	master branch, no surprises
[14:42] mato	good,m so i can do make dist myself from that to test things
[14:42] pieterh	exactly
[14:42] pieterh	I noticed dist also makes zip files, that's nice
[14:42] pieterh	I'd like to do that in zapi and zfl too...
[14:43] mato	just copy what libzmq does
[14:43] pieterh	yup, will do
[14:47] stimpie	my 'faster' machine gets the following perfomance on inproc duration: 12681.44 ms, messages:10000000, speed: 788554.25 m/s
[14:48] stimpie	considering I got over 2M with tcp its still quite slow
[14:57] private_meta	788km/sec... that's almost twice as fast as the earths rotational speed!
[14:57] private_meta	oh wait
[14:57] private_meta	way more
[14:57] private_meta	spinning out of control
[14:58] stimpie	it would save nasa some money ;-)
[15:02] sustrik	stimpie: iirc JNI locks a mutex for each call
[15:02] sustrik	which could account for limiting number of calls per sec to below 1M
[15:05] SurfaceTension	Does anybody have an approximate object code size for ARM Thumb?
[15:12] private_meta	hmpf... it's difficult to find the problem with the paranoid pirate/zmsg problem
[15:13] mato	pieterh: ok, 2.1.4 git fixes my problem but still has a couple of problems with openpgm integration, have reported those on the ML
[15:15] sustrik	mato: also notice that pre-compiled devices were removed
[15:15] mato	sustrik: yes, i have already noticed
[15:15] mato	sustrik: thx
[15:15] sustrik	which means some functionality will actually disappear after installing new package
[15:15] mato	yes, i know
[15:15] mato	not up to me
[15:16] sustrik	no idea what process debian uses for that kind of thing
[15:16] mato	actually, it won't disappear in debian
[15:16] sustrik	heh
[15:16] mato	you'll just get the 2.0.x devices staying installed if you installed them in the 1st place
[15:16] sustrik	ok
[15:17] mato	i.e. after upgrading to 2.1.x you will get libzmq0 (2.0.10) zeromq-bin (2.0.10) libzmq1 (2.1.x) and libzmq-dev (2.1.x) installed
[15:18] mato	which may confuse some people, in which case i will cheerfully pass on said people to the upstream maintainer :-)
[15:19] mato	hmm.
[15:20] mato	sustrik: i'm just thinking, about those devices
[15:20] mato	sustrik: d'you think i should force their removal?
[15:20] sustrik	mato: no idea
[15:21] sustrik	i would say the minor upgrade shouldn't break backward compatibility
[15:21] sustrik	but it's too late for that
[15:22] sustrik	hm, presumably, leaving the old devices there would be better
[15:22] sustrik	at least the things won't stop working after upgrade
[15:22] mato	yes, except that they won't talk to 2.1 applications anyway
[15:22] mato	(wire protocol change)
[15:23] mato	i'll ask the debian people i'm working with
[15:23] sustrik	ah, the last debian release was 2.0, right?
[15:23] mato	yup.
[15:23] mikko-	mato: i will look at the two issues you mentioned
[15:23] sustrik	then delete the devices
[15:23] sustrik	missing executable is pretty obvious
[15:24] sustrik	incompatible wire format is hard to detect
[15:24] mato	sustrik: precisely
[15:24] mato	sustrik: ok i'll see if there's a way to do that
[15:24] mato	mikko-_: thanks!
[15:25] mikko-	mato: the linking issue im not sure how that will work
[15:25] mikko-	might not have time today as i got leaving drinks tonight
[15:25] mato	mikko-_: well, i used to do it by building openpgm with -fvisibility=hidden
[15:26] mato	mikko-_: we should be able to do that with the current system, no?
[15:26] mikko-	mato: i thought that openpgm builds with visibility=hidden
[15:27] mikko-	but i'll get into this at some point today or tomorrow
[15:27] mato	mikko-_: IIRC what it does is use __attribute__ internally in the code
[15:27] mato	mikko-_: but what we need is for it to hide the API symbols as well
[15:28] mato	mikko-_: and I managed to override that by forcing it to build with -fvisibility=hidden
[15:28] mato	mikko-_: back in the day...
[15:29] mikko-	roughly what happens at the moment:
[15:29] mikko-	zeromq build invokes openpgm build and they build a libtool convenience library
[15:30] mikko-	which contains the built objects
[15:30] mikko-	called libpgm_noinsta.la
[15:30] mikko-	libpgm_noinst.la
[15:30] mikko-	and that libtool archive is added with libzmq_la_LIBADD
[15:31] mato	ignoring the libtool bit, that libpgm_noinst.la is essentially a static library archive, right?
[15:31] mikko-	essentially yes
[15:32] mato	and AC_CONIFG_SUBDIRS(...) is what invokes the OpenPGM configure?
[15:33] mikko-	it's collection of the built objects
[15:33] mikko-	yes
[15:34] mato	ok, problem is i see no way to pass e.g. CFLAGS to AC_CONFIG_SUBDIRS
[15:34] mikko-	mato: if you look at http://openpgm.googlecode.com/svn/trunk/openpgm/pgm/configure.ac
[15:34] mato	or options, or anything
[15:34] mikko-	they are detecting visibility
[15:35] mikko-	CFLAGS are passed down from parent
[15:35] mikko-	CFLAGS='-xyz' ./configure in zeromq build passes those down to openpgm build
[15:35] mato	mikko-_: yes, but openpgm deals with visibility differently than zmq
[15:36] mato	mikko-_: zmq uses the "*everything is hidden" approach
[15:36] mikko-	CFLAGS should be safe to modify even within zeromq build as that should not be used anywhere
[15:36] mikko-	CXXFLAGS and CPPFLAGS for zeromq
[15:36] mikko-	as the source is c++
[15:36] sustrik	mato: btw, how do i remove an outdated topic branch from github?
[15:36] mato	pgm uses default visibility, and defines PGM_GNUC_INTERNAL to use __attribute__ ((visibility("hidden"))
[15:37] mato	which means if the default is set to hidden with -fvisibility=hidden
[15:37] mato	then the pgm api symbols will disappear from the library
[15:37] mato	sustrik: git push origin :<branch name> i think
[15:38] sustrik	the colon deletes the branch?
[15:38] sustrik	strange
[15:38] sustrik	thanks
[15:38] mato	yeah, it pushes an empty ref to it or something
[15:38] mato	deleting the remote branch via push is not the same as deleting a local branch...
[15:39] sustrik	ok, np
[15:42] mato	mikko-_: in theory all that needs to be done is CFLAGS=-fvisibility=hidden for the PGM build
[15:42] mato	mikko-_: let me know if you can't make it work
[15:42] mato	i have to go, am on a train and battery dying
[15:42] mikko-	mato: i will test as soon as i get a chance
[15:42] mikko-	wil let you know
[15:44] mato	ok, cyl
[15:58] CIA-22	zeromq2: 03Martin Sustrik 07pre30 * rbc4a1ce 10/ (8 files in 4 dirs):
[15:58] CIA-22	zeromq2: ZMQ_HWM split into ZMQ_SNDHWM and ZMQ_RCVHWM
[15:58] CIA-22	zeromq2: These new options allow to control the maximum size of the
[15:58] CIA-22	zeromq2: inbound and outbound message pipe separately.
[15:58] CIA-22	zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> (+17 more commits...) - http://bit.ly/gIsQa8
[17:11] pieterh	sustrik: random question
[17:11] sustrik	yes?
[17:11] pieterh	you're working both on master and on the pre30 branch?
[17:11] sustrik	it's a topic branch
[17:11] pieterh	is that a yes or a no?
[17:11] sustrik	it's like a transaction
[17:12] sustrik	not to break the master
[17:12] pieterh	e.g. ZMQ_HWM split, is on master or on pre30
[17:12] pieterh	?
[17:12] sustrik	the work is done on a branch
[17:12] sustrik	then it can be pulled to master in one go
[17:12] sustrik	yes
[17:13] sustrik	on pre30
[17:13] pieterh	I'll rephrase: are you still making improvements to the master branch provisionally marked as 2.2.0?
[17:13] pieterh	or is that frozen?
[17:13] sustrik	2.2 is frozen
[17:13] sustrik	tou are free to use it
[17:13] sustrik	you
[17:13] pieterh	yes, that was my idea, to pull specific changes into that
[17:13] sustrik	note that first patch to be pulled to master is version bump to 3.0
[17:14] pieterh	hmm
[17:14] pieterh	ok, let me create a 2.2 release git, then we can do the switch for 3.0
[17:14] pieterh	there's not much point using a topic branch for the 3.0 stuff IMO
[17:15] ptrb	mato: you around?
[17:15] pieterh	I'd like to cherry pick some changes into 2.2, such as the HWM change
[17:15] sustrik	sure, do so
[17:15] pieterh	ack, going for it now
[17:16] sustrik	i would wait for reviews at least
[17:16] sustrik	otherwise you risk having conflictins with actual master
[17:16] pieterh	how so?
[17:17] sustrik	well, say someone complains about something in pre30
[17:17] sustrik	it will be fixed
[17:17] sustrik	and the fixed patch will be pulled to master
[17:17] sustrik	then you have an incompatible version in 2.2
[17:17] sustrik	and you have to reconcile the problem by hand
[17:18] pieterh	sure... as long as you make atomic commits, it'll be doable IMO
[17:18] sustrik	ok, it's up to you
[17:18] pieterh	at some point the structure of 3.0 will change too far
[17:18] pieterh	but we're not going to make big changes in 2.2, just tweaks
[17:19] pieterh	we may not even need the 2.2 release but I'd rather be ready than have to try to make it later
[17:19] sustrik	are you aware that the patch in question is backward incompatible?
[17:19] sustrik	actually all the patches in pre30 are
[17:19] pieterh	yes, indeed
[17:20] sustrik	and that it breaks all the language bindings?
[17:20] pieterh	my immediate reaction is, this patch adds value, is worth it
[17:20] sustrik	ok
[17:20] pieterh	I think each time we've asked binding authors this question, they've answered the same
[17:21] pieterh	something like, "we'd rather get new useful functionality now, even if it means fixing up stuff"
[17:21] pieterh	ok
[17:22] cremes	as a bindings author, i agree with that quote
[17:22] pieterh	cremes: thanks, I don't like to speak for others
[17:24] sustrik	cremes: it's up to you obviously
[17:24] sustrik	my point was that if bacward incompatible changes are to happen anytime
[17:24] sustrik	the bindings end up as a mess of ifdefs
[17:24] cremes	since the library version is exposed via the zmq_* api, it's pretty easy to turn things on/off at runtime
[17:25] cremes	i agree that with too many minor differences things could get ugly
[17:25] cremes	as a bindings author, i reserve the right to require minimum versions of the library
[17:25] cremes	so that i can keep those ifdefs to a manageable size
[17:26] cremes	btw, this weekend i plan to do a small test using cxxtest for writing unit tests for yqueue
[17:27] sustrik	great
[17:27] cremes	i'm going to put that into a fork of the current master; i'll fork it under my username chuckremes
[17:27] cremes	and point you guys to it when i get something working
[17:27] cremes	i want to just stick with ruby but i have an itch that can only be scratched by c++ apparently... ;)
[17:28] sustrik	i think the tests unit tests have to be incorporated into the build system somehow ultimately
[17:28] pieterh	sustrik: in zfl & zapi we do this with 'make check', which runs a single main which runs all unit tests
[17:29] cremes	sustrik: that will be part of my experiment
[17:29] sustrik	yes, something like tha
[17:29] cremes	the cxxtest readme has examples on how to include that in the build process
[17:29] sustrik	that
[17:29] pieterh	sustrik: ok, 2.2 is safely forked off, we can now play with the core repo...
[17:30] pieterh	no hurry but I'd suggest we make and announce the rename when you're comfortable with it
[17:31] sustrik	have you done the test?
[17:31] sustrik	does rename retain watchers
[17:31] sustrik	?
[17:32] sustrik	do the forks map nicely to the renamed repo?
[17:32] pieterh	let me run some tests in my own profile, then
[17:32] pieterh	(good questions)
[17:32] sustrik	ok
[17:33] sustrik	if you need a watcher, ping me
[17:34] pieterh	ok, can you watch https://github.com/pieterh/test ?
[17:34] pieterh	and also fork it
[17:36] sustrik	pieterh: done
[17:36] pieterh	ok, I've also made a couple of forks...
[17:36] pieterh	now renaming it...
[17:37] pieterh	you watched it as 'zeromq'?
[17:37] sustrik	no idea
[17:37] pieterh	oh snap, sorry
[17:37] pieterh	I forked it, and then renamed the fork... :-)
[17:39] pieterh	so it works, the repo is now called https://github.com/pieterh/libtest
[17:39] pieterh	watchers and forks shaken but still attached
[17:40] sustrik	the 'network' graph seems to be broken
[17:40] sustrik	i don't see you there
[17:41] pieterh	it seemed broken before the rename, too
[17:41] pieterh	only showed a single entry
[17:41] sustrik	that's ok as there was only a single entry
[17:41] sustrik	what i'm seeing
[17:41] sustrik	is that the fist commit you've made is under 'sustrik'
[17:42] pieterh	what repo?
[17:42] sustrik	sustrik/test
[17:42] pieterh	on my box the whole network viewer is buggy, shows no commits at all
[17:42] pieterh	but the commit list is accurate
[17:43] sustrik	yes, commits look ok
[17:43] pieterh	I've no idea what this means, a github bug?
[17:43] sustrik	no idea
[17:44] pieterh	well, I'd suggest we go ahead with the rename, if the network appears broken afterwards we ask github to fix it
[17:44] pieterh	but let's wait a day or two until the new name is obvious
[17:44] pieterh	libzmq is my preference right now
[17:45] pieterh	but it's your project, your call :-)
[17:45] sustrik	i'll give it a thought
[17:45] sustrik	zeromq otoh would require less renaming
[17:45] sustrik	ie. stable repos would stay as they are now
[17:46] pieterh	indeed, it's more consistent with that
[17:46] pieterh	but if we start to make the release repos into distribution containers, that changes
[17:46] sustrik	yes, i know
[17:52] klestes	hullo, is anyone actively awake ?
[17:53] pieterh	klestes: passively awake, here...
[17:53] klestes	that'll do :)
[17:53] pieterh	what's up?
[17:53] klestes	sorry about my first patch. I could have sworn I read somewhere that a 2.2 version wasn't being pursued.
[17:54] pieterh	we changed our minds...
[17:54] klestes	D'oh!
[17:54] klestes	oh well, back to the hacking boarrd =)
[17:55] klestes	uhm, would minor spelling corrections to header files be a worthwile contribution ?
[17:57] pieterh	depends how perfectionist the maintainer is, they may just find it annoying
[17:57] klestes	reasonable.
[17:57] klestes	also, on to the matter of LOAD BALANCING.
[17:58] klestes	now, suppose I have a client talking to a server, a PUSH/PULL kinda relationship.
[17:58] klestes	once I have these two chatting, I introduce a second client PULLing from the server as well.
[17:59] klestes	my question is, upon introduction of the second client, will the load be balanced again ?
[18:03] klestes	(oooh! FF6 is out !)
[18:03] klestes	(sorry, bug in my FPU, thats FF4)
[18:08] sustrik	patches to the spelling are acceptable
[18:09] sustrik	the load already assigned to the first client won't be re-balanced
[18:09] sustrik	but new messages will be
[18:10] klestes	ok, thats what I had thought the proper behavior should be.
[18:10] klestes	where in the source is the load balancing mechanism ?
[18:14] cremes	sustrik: running a test using jemalloc and LD_PRELOAD on linux
[18:14] cremes	results look promising so far
[18:23] pieterh	sustrik: actually there are some nice improvements in the 2.2 branch over 2.1...
[18:23] pieterh	e.g. checking &logging of duplicate identities...
[18:23] pieterh	I'll check them all
[18:25] cremes	pieterh: i have an open ticket on that duplicate identity commit; i get crashes with it
[18:26] pieterh	cremes: ah, right...
[18:26] pieterh	well, 2.2 is not going to be stable for a while, that's OK
[18:26] cremes	true; hopefully i'll be able to come up with a small repro
[18:27] Guthur	is 2.2 available?
[18:27] Guthur	is it the pgm stuff?
[18:28] pieterh	Guthur: not yet, soon enough
[18:28] pieterh	the pgm stuff is already in 2.1
[18:28] klestes	can you not get 2.2 with git ?
[18:28] sustrik	still missing in master :(
[18:28] pieterh	sustrik: what's missing in master?
[18:28] pieterh	ah, yes
[18:29] sustrik	jemalloc: let me know whether it helps
[18:29] pieterh	sustrik, mikko is still working on making a patch, maybe he's stuck, I'll ask him... I know he's real busy right now
[18:29] pieterh	I'm applying those openpgm changes to 2.2, I'll see if it works
[18:29] pieterh	configure.in, Makefile.am, libpgm tarball, that's it...
[18:38] Guthur	would it be possible to expose the subscription mechanism on SUB sockets, so that people could maybe add other models to replace the prefix one
[18:39] Guthur	this is in relation to 3.0
[18:47] pieterh	Guthur: it's a neat idea but probably quite difficult to make happen
[18:56] sustrik	Guthur: can be doable on top of 0mq, with XPUB/XSUB sockets
[18:56] sustrik	when those are finished :)
[18:58] pieterh	sustrik: upstream arbitrary subscription keys?
[19:02] sustrik	possibly
[19:02] sustrik	we'll see
[19:02] sustrik	it's not done yet
[19:02] pieterh	could be fun to turn xpub sockets into mini exchanges
[19:03] pieterh	sustrik: if you want a patch for the openpgm stuff, I can make it, have tested that on 2.2
[19:04] pieterh	but I think you want the real author's patch... sigh
[19:04] pieterh	well, it's LGPL
[19:04] pieterh	already signed off
[19:04] pieterh	this whole free software concept sometimes confuses me... :-)
[19:29] Guthur	cool, I look forward to seeing XSUB and XPUB
[19:29] Guthur	will they need to be renamed after they have been released for a while, hehe
[19:31] cremes	Guthur: yes, they are going to be renamed SUCK and BLOW
[19:31] cremes	no one raised any objection on the mailing list, so that's going into 2.1.4
[19:31] cremes	:)
[19:41] jond	cremes: what's the conclusion on jemalloc
[19:42] cremes	jond: superior to glibc malloc on linux for the way i use 0mq
[19:43] cremes	the memory footprint still grows, but now the 'fragmentation leak' is slower
[19:43] jond	cremes: does jemalloc just replace malloc/free et al or the C++ operator's as well
[19:44] cremes	jond: it's my understanding that under the covers C++ #new, et al. are calling malloc/free
[19:44] cremes	so jemalloc replaces them too
[19:44] pieterh	cremes: sometimes you're evil...
[19:44] cremes	pieterh: ah, you like the new pub/sub names then?
[19:45] pieterh	I think the old names were fine (PUB/SUB)
[19:45] jond	cremes: ok ; think I asked the tcmalloc people a while back and technically new etc don't have to use malloc free ....
[19:46] cremes	jond: that's true; from comments i have read while researching this stuff, no one was able to find a case where c++ did not
[19:46] jond	pieterh: HOOKER and PIMP?
[19:46] cremes	use malloc/free
[19:46] pieterh	HOOKER would be XSUB then?
[19:46] cremes	jond: ooh, good replacement names for PUSH/PULL
[19:47] cremes	:)
[19:47] pieterh	where does JOHN fit in? that's my question
[19:47] cremes	pieterh: PAIR -> JOHN
[19:47] pieterh	surely PAIR-> FABIO
[19:47] cremes	surely
[19:48] pieterh	let's start a Naming Committee
[19:48] pieterh	with its own repository
[19:48] cremes	heh
[19:49] jond	cremes: c++ new et al; that's what I've read too, but there's probably some obscure games console compiler somewhere
[19:49] cremes	jond: probably
[19:50] cremes	but since i don't use it, i don't care ;)
[19:51] jond	so you just LD_PRELOAD with jemalloc on the path and that's it? I'm sure google recommend recompile with tcmalloc though LD_PRELOAD can work
[19:52] cremes	yes
[19:52] cremes	LD_PRELOAD=/usr/local/lib/libjemalloc.so a.out
[19:53] jond	cremes does it improve ruby performance at all apart from 0mq aspect?
[19:53] cremes	jond: if it does, i can't see it... there may be a small % difference but i can't tell
[19:54] jond	cremes: I'll go and have a look at jemalloc now
[19:55] cremes	jond: when i ran sustrik's leak repro (ticket 174) the resident memory would jump to 25MB and stay there
[19:55] cremes	with jemalloc it would drop back down to 15MB
[19:55] cremes	so while it isn't perfect, it was able to help out a bit
[19:58] jond	cremes: i may give jemalloc a test against an app at work, non zeromq though sadly
[20:04] cremes	jond: please share your results (0mq or non-0mq) with me; i'd love to hear your results
[20:16] jond	pieterh: you ever thought of putting a 'jobs' section on the website?
[20:16] pieterh	jond: yes
[20:17] pieterh	but rather, I'd like to make a 'marketplace'
[20:17] jond	pieterh: yes better idea
[20:18] pieterh	the idea would be that proven contributors can promote their business
[20:18] pieterh	this would happen on the main website (not the community site) and would be run by iMatix
[20:19] pieterh	'run' = vetted, more than anything else
[20:20] sustrik	btw, i've created a zeromq group on linked-in long time ago
[20:21] sustrik	i don't use it much
[20:21] sustrik	i don't even recall why i've created the group
[20:21] sustrik	someone asked me to, i suppose
[20:21] pieterh	the idea of promoting partners able to help in specific areas, or specific regions, seems win-win
[20:21] pieterh	:-)
[20:21] sustrik	intersting
[20:21] sustrik	28 members
[20:22] sustrik	looks like some people are actually using linkedin
[20:22] pieterh	some people do seem to like collecting contacts, indeed...
[20:22] Moltar	I'm trying to limit the outstanding messages in python using setsockopt(zmq.HWM, 1000) (for example) and it seems that this has no effect. Any suggestions to what I might be doing wrong?
[20:24] hexa	Hi I just went tought the guide.. and wow :) .. but it got me many ideas one of which is to use it to do some worker/task management inprocess .. but I'm struggling to find a way to pass for exemple function pointers to a generic worker thread.. I checked google protobufs .. things like that but i'm not sure about it when it comes to pointers etc.. any suggestions ideas on this ?
[20:25] pieterh	hexa: the way I send pointers between threads is printf ("%p", pointer) and sscanf (message, "%p", &pointer)
[20:25] sustrik	Moltar: the messages are presumably queued in TCP buffers
[20:25] pieterh	seems to work
[20:26] pieterh	hexa: there's an example in the zapi, zctx class, somewhere in http://zero.mq/c
[20:26] hexa	pieterh, heheh guess that would work.. but I would like something a bit more versatile...
[20:27] pieterh	well, just shove the pointer in a message of the right size
[20:27] Moltar	sustrik: I'm using unix-domain socket with zSocket.bind('ipc:///tmp/sendalias'), so probably queued in unix socket buffer
[20:27] Moltar	sustrik: But I have quite a lot of data to push to my worker processes, and I don't think having gigabytes of data in this buffer is going to be a good idea.
[20:28] hexa	pieterh, heh I guess I could just do void* func , void* marshalling_func, void* data , pieterh thx for the link I hadn't see that yet
[20:33] jond	sustrik: what's the status of subscription forwarding in master?
[20:50] SurfaceTension	still nobody with a (rough) number for the binary code size on arm thumb?
[21:11] sustrik	Moltar: you can set the buffer size using ZMQ_SNDBUF and ZMQ_RCVBUF options
[21:11] sustrik	jond: no progress
[21:11] sustrik	SurfaceTension: try asking on the mailing list
[21:12] Moltar	sustrik: Thanks, I'll give that a go
[21:12] sustrik	there are few people using 0mq on ARM iirc
[21:18] jond	sustrik: does that mean it's all in master but incomplete or on that branch?
[21:19] sustrik	it's incomplete and located in sub-forward branch
[21:20] jond	sustrik: ok, thanks. I'm abandoning xrep you see from my proof of concept
[21:56] SurfaceTension	sustrik: will do, thanks
[22:47] Guthur	thrift provides a means for RPC, which I suppose could be used instead passing func ptrs
[22:47] Guthur	but hexa has left...