ZeroMq IRC Log

Monday May 2, 2011

[Time] Name	Message
[01:00] neopallium	iFire: I am here.
[01:01] neopallium	iFire: yes, build it like any C Lua module. Make sure to link it against libzmq (-lzmq).
[01:10] iFire	neopallium
[01:10] iFire	> require "lua-zmq"
[01:10] iFire	error loading module 'lua-zmq' from file '.\lua-zmq.dll':
[01:10] iFire	The specified procedure could not be found.
[01:10] iFire	I get that
[01:10] iFire	where pre_generated-zmq.nobj.c is compiled into lua-zmq.dll
[01:11] iFire	on windows
[01:11] iFire	using lua-jit
[02:12] iFire	neopallium how was dinner :P
[02:58] iFire	neopallium I can't seem to load the library from a lua interpreter
[02:58] iFire	what's a short lua thing to test if it works
[03:02] private_meta	AH GODDAMMIT!
[03:07] private_meta	I spent so much time to implement our communications library with ZeroMQ and now it seems like it's got the same problem as ASIO
[03:07] private_meta	OMG this pisses me off so much
[03:08] iFire	private_meta same problem?
[03:08] private_meta	Messages are lost/garbled
[03:08] private_meta	when I compile it with OpenMPI
[03:10] iFire	I thought you weren't supposed to use zeromq with locks and stuff
[03:10] iFire	each thread is supposed to be its own unit
[03:10] iFire	and you communicate through inproc/ipc/tcp/etc.
[03:11] private_meta	I don't know what you get at with that
[03:13] private_meta	iFire: not sure what that has to do with my problem
[03:13] iFire	hmm
[03:13] iFire	sorry
[03:13] iFire	well mpi and zeromq do the same things
[03:14] iFire	unless you're using it to connect two systems
[03:14] iFire	http://zguide.zeromq.org/page:all#Multithreading-with-MQ
[03:16] private_meta	I'm using MPI to create parallel processes and to control them, I wanted to use zeromq for backend communication
[03:16] private_meta	each process being a client
[03:16] private_meta	these two are independent
[03:16] jer	why use MPI for that and not zeromq for both?
[03:18] private_meta	Because zmq doesn't supply most of the mpi functions like scattering, gathering, ... out of the box like mpi
[03:29] neopallium	iFire: back, you need to use: require"zmq"
[03:30] iFire	hmm
[03:30] iFire	what about the dll
[03:30] iFire	I'm doing a custom packaging
[03:30] iFire	so everything is in the wrong place
[03:30] neopallium	the dll, I think you will need to call: zmq.dll
[03:31] iFire	> require "zmq"
[03:31] iFire	error loading module 'zmq' from file '.\zmq.dll':
[03:31] iFire	The specified procedure could not be found.
[03:31] iFire	I renamed lua-zmq.dll to zmq.dll
[03:31] iFire	hmm
[03:31] iFire	I don't actually NEED poller.lua [for testing if the dll loads right?]
[03:31] neopallium	can you check that luaopen_zmq() is a public/exported symbol in the .dll?
[03:32] neopallium	for testing, no, you don't need poller.lua or threads.lua
[03:32] neopallium	this page has instructions on compiling a Lua module on Windows: http://lua-users.org/wiki/BuildingModules
[03:33] iFire	neopallium will depdency walk in windows show me it
[03:33] iFire	dependency walker
[03:34] neopallium	I am not a windows dev., so I am not sure.
[03:35] iFire	int __declspec(dllexport) MyModuleName (lua_State* L) { ... } ?
[03:35] iFire	note : make sure your main init function is exported with __declspec(dllexport) like this :
[03:35] iFire	int __declspec(dllexport) MyModuleName (lua_State* L) { ... }
[03:35] iFire	neopallium is that needeD?
[03:36] neopallium	no, the symbol that needs to be exported for this module is: int luaopen_zmq(lua_State *L);
[03:37] neopallium	in the pre_generated-zmq.nobj.c file you will see it as: LUALIB_API int luaopen_zmq(lua_State *L) {
[03:38] neopallium	very close to the end of the .c file
[03:38] iFire	yes
[03:38] iFire	hmm
[03:38] iFire	somehow add int __declspec(dllexport) <-?
[03:39] neopallium	LUALIB_API will do that on windows.
[03:39] iFire	it doesn't :/
[03:40] iFire	neopallium ok I did __declspec(dllexport) LUALIB_API int luaopen_zmq(lua_State *L) {
[03:40] iFire	and it seems to work
[03:41] iFire	what's a way of printing the zmq table
[03:41] neopallium	look in your luaconf.h file, that is where LUALIB_API & LUA_API are defined.
[03:41] iFire	using luajit
[03:41] neopallium	for k,v in pairs(zmq) do print(k,v) end
[03:43] neopallium	ah, you need to define LUA_BUILD_AS_DLL on windows when compiling a Lua C module.
[03:43] iFire	http://pastebin.ca/2052715 <- this is a working table print right?
[03:43] neopallium	yup, that looks right.
[03:45] iFire	ha
[03:45] iFire	well using that define makes the build fail
[03:46] iFire	>..\..\..\deps\lua-zmq\src\pre_generated-zmq.nobj.c(3121): error C2491: 'luaopen_zmq' : definition of dllimport function not allowed
[03:46] iFire	neopallium you probably mean LUA_LIB
[03:47] iFire	#if defined(LUA_CORE) \|\| defined(LUA_LIB)
[03:47] iFire	#define LUA_API __declspec(dllexport)
[03:47] iFire	#else
[03:47] iFire	#define LUA_API __declspec(dllimport)
[03:47] iFire	#endif
[03:47] iFire	actually both
[03:47] neopallium	hmm, maybe I shouldn't be using LUALIB_API or LUA_API for exporting the module's luaopen_*() function.
[03:47] iFire	http://www.lua.org/source/5.1/luaconf.h.html [search for LUA_LIB]
[03:48] neopallium	I will look for a better solution, but for now just replace LUALIB_API with __declspec(dllexport)
[03:48] neopallium	yeah, I think those defines are more for the Lua core not modules.
[03:49] neopallium	ok, I need to go now, I will be back in about an hour or two.
[03:49] neopallium	but, I will take a closer look at the problem later tonight.
[03:50] iFire	neopallium pm
[03:55] iFire	I added two defines and it worked "LUA_BUILD_AS_DLL", "LUA_LIB"
[06:03] neopallium	iFire: I committed a fix to lua-zmq that should make it easier to compile on windows. You don't need to define LUA_BUILD_AS_DLL, LUA_LIB, or LUALIB_API.
[06:48] staylor	is there a way to get the connection status of a socket to a given endpoint?
[06:49] guido_g	no
[07:06] jer	i've got a couple of workers creating push sockets sending some results out to a single node binding a pull socket. communication works fine, as i'd expect. however, after i send a first message off, my workers cpu usage goes up over 30% and none of the workers are processing any jobs, sockets are idle. not sure wtf is going on. anyone have any suggestions on what i might look for first?
[07:09] staylor	guido_g so would the best way to know if a connection can be made to open a separate standard connection to the host, but not exchange any data?
[07:09] guido_g	you still don't know if the Ã¸mq endpoint is connected and alive
[07:09] guido_g	only Ã¸mq could tell you that
[07:10] guido_g	and it doesn't
[07:11] staylor	well ideally it would, but I guess at least a standard socket would tell me that at least that much works. I'd like to know if a connection is refused basically, as it tells me most likely the other end failed to map a upnp port or the service isn't running.
[09:43] edwin	hi, the helloworld zeromq example is not working properly for me with 0MQ 2.1.6 on Debian x86-64, and here's a testcase: https://gist.github.com/951368
[09:43] edwin	output from testcase( zmq_send fails with Bad address): https://gist.github.com/951369
[09:43] edwin	the server keeps giving those errors, and the client hangs
[09:43] edwin	same with hwserver/hwclient
[09:46] pieterh	edwin: could you run hwserver & hwclient (not your test case, which is rather more code) and post the messages that gives? thanks
[09:47] edwin	https://gist.github.com/951374
[09:47] edwin	pieterh: the client hangs after the first send
[09:48] edwin	sometimes it receives too, but doesn't do a 2nd send
[09:48] pieterh	so it hangs, presumably the server isn't sending a reply?
[09:48] edwin	interesting thing is that it worked better yesterday
[09:48] edwin	yes the server gives an error on zmq_send
[09:48] edwin	see output from my code: https://gist.github.com/951369
[09:49] edwin	ZMQ error: Bad address
[09:49] pieterh	what does the server say on zmq_send (the hwserver, not yours)
[09:49] edwin	let me add a printf hold on
[09:50] edwin	hwserver zmq_send failed: Bad address
[09:50] edwin	https://gist.github.com/951380
[09:50] edwin	that is the change I made
[09:50] edwin	to print the error
[09:51] pieterh	ok, let me just retest that
[09:52] pieterh	it works for me, but that's not helpful...
[09:52] pieterh	I'm not sure what 'bad address' means for 0MQ, let me dig a little
[09:53] pieterh	EFAULT = invalid memory access, I'd assume
[09:54] pieterh	ah, edwin....
[09:54] edwin	ah yes version 2.0.x crashes with SIGSEGV with invalid access
[09:54] pieterh	can you please take the actual current version of hwserver
[09:55] pieterh	I'd posted a version with a bug in it, that's what's hitting you
[09:55] pieterh	you can see it's putting "World" into the request, not the reply... :-/
[09:55] edwin	yeah that works
[09:56] edwin	ok let me fix my latency measurement tool then :)
[09:56] pieterh	sorry about that, someone caught it but it was there for a day or two
[09:57] edwin	Time: 1.521894s, 6570.759856 Req/s, Latency: 152.189400 us
[09:57] edwin	yep works now
[09:57] edwin	thanks :)
[09:58] edwin	I was going to try and improve the latency by using a pipe instead of socketpair for the mailbox
[09:58] edwin	but then the helloworld started failing...
[10:00] edwin	(unix socket latency is ~28us, and pipe is 3-4x faster than the socket according to hackbench)
[10:03] mikko	good morning
[10:04] pieterh	mikko: hi, how're you doing?
[10:04] mikko	i am excellent
[10:04] mikko	extra bank holiday today
[10:04] pieterh	:) yeah
[10:04] mikko	very bad time for freelancers
[10:04] mikko	:)
[10:05] mikko	you?
[10:06] pieterh	working on a new android app
[10:06] pieterh	quite fun
[10:07] mikko	are you using zeromq?
[10:07] pieterh	it's fun to see that stuff I was designing in 1999 is still better than web frameworks today
[10:07] pieterh	nope, this is a different project
[10:07] pieterh	well, the web frameworks are pretty, and easy to use
[10:08] pieterh	I'd not use 0MQ across the Internet anyhow
[10:08] pieterh	we're making connectivity for this app across RestTL, and we'll use 0MQ in the backend
[10:15] edwin	hmm, latency with pipes for mailbox is 43 - 44us (vs 49-50 us with unix socket for mailbox). If I turn on CPU freq scaling then latency is 135 - 150 us
[10:16] edwin	however using just a unix socketpair (without zeromq) latency is 10us
[10:18] edwin	why is a socket used for zeromq's mailbox, instead of lets say a ypipe and a pthread condition?
[10:27] sustrik	edwin: even with ypipe you have to synchronise the two sides of the pipe somehow
[10:27] sustrik	but yes, ypipe + using socketpair just for signaling would be superior solution
[10:27] mikko	sustrik: did you notice that several compilers fail now with master?
[10:27] edwin	not sure if socketpair is faster than a pthread condition for signaling
[10:28] edwin	will have to test
[10:28] mikko	well, ICC and sun studio
[10:28] sustrik	the problem is that the signal has to interrupt a polling loop
[10:28] sustrik	you can't do that with condition variable afaik
[10:29] edwin	ah because you also use it to poll network sockets
[10:29] sustrik	exactly
[10:29] edwin	won't a signal() interrupt poll with EINTR?
[10:30] sustrik	it's a nightmare to combine signals with threads
[10:30] edwin	the signal handler doesn't have to actually do anything, just wake the poller
[10:31] edwin	of course there is a problem which signal to use
[10:31] edwin	since libzmq is a library..
[10:31] sustrik	there's a problem which thread the signal will be delivered to
[10:32] edwin	yeah that too, ok bad idea. using pipe to wake poll() sounds fine, we can avoid a recv() on the mailbox
[10:32] sustrik	you've just said it's slower
[10:32] edwin	its slower than pthread, but pipe is faster than unix sockets
[10:33] edwin	(which is used now)
[10:33] sustrik	ok, we can do that
[10:33] sustrik	if you have a patch, send it to the mailing list
[10:33] edwin	sure, I'll be testing some variants
[10:33] sustrik	ok
[10:33] edwin	patch aganist 2.1.6 is fine?
[10:33] edwin	or the 3.0 master?
[10:33] sustrik	master is better
[10:34] sustrik	but the mailbox code haven't changed iirc
[10:34] sustrik	so, it'll be the same anyway
[10:34] edwin	ok
[10:34] sustrik	btw, if you want to implement ypipe+pipe thing
[10:34] sustrik	that would solve one serious problem 0mq is experiencing
[10:35] sustrik	namely that the mailbox can overflow in some special cases
[10:35] sustrik	ypipe+pipe would solve that as there's no limit on number of items in yqueue
[10:36] edwin	won't send() block if mailbox is full?
[10:36] edwin	or zeromq spins on send with errno=EAGAIN?
[10:37] sustrik	it have blocked originally, but that leads to deadlocks
[10:37] sustrik	so now it tries to increase the mailbox size
[10:37] sustrik	which doesn't work well on different OSes
[10:38] sustrik	btw, note that if pipe is used only for signaling
[10:38] sustrik	it can be optimised on some OSes
[10:38] sustrik	eg. on Linux we can use eventfd which is much faster than socketpair or pipe
[10:42] edwin	eventfd looks cool, I should use that in clamd as well (I use a pipe with 1 byte write for signaling)
[11:20] th	sustrik: did you have a look at #199 yet? thats a 3-0 master issue..
[11:22] sustrik	th: yes, i've reproduced it this morning
[11:22] th	sustrik: thats a great start :)
[12:22] djc	guido_g: turned out my test box had something else listening on a port pyzmq was using for testing...
[13:05] private_meta	pieterh: eh... remember that I went to zeromq to fix a problem I had with asio? got the same problem now with zeromq
[13:05] pieterh	private_meta: lol...
[13:05] pieterh	what is happening?
[13:06] private_meta	I send normal strings with the messaging library, all is ok, I send (boost)serialized classes, all is ok. Once I send boost serialized classes when the stuff is compiled and used with MPI, some of these messages are garbled beyond recognicion
[13:06] private_meta	*recognition
[13:07] pieterh	hmm, so it's a problem in your app, not in the transport
[13:07] private_meta	Figured that out last night, but not sure if it's really the fault of boost serialization in combination with mpi
[13:07] private_meta	well, yes and no
[13:07] pieterh	sounds like a memory management issue
[13:07] private_meta	the message i send, and the serialize looks ok
[13:07] private_meta	but it's broken when it arriveso n the other side
[13:08] pieterh	you sure you're not deallocating the memory before the message is really sent?
[13:08] private_meta	but only when i compile it with mpi
[13:08] private_meta	pretty sure
[13:08] pieterh	and of course a minimal test case requires MPI... so I can't really help much... :/
[13:09] private_meta	That was one of the problems. It seems the minimal test case is quite complex
[13:09] private_meta	Well, I need to write a paper now, but as soon as I'm finished, I'm going back to the matter, in a couple of days I hope
[13:09] pieterh	well, try sending the data via a device, log it there and see if it's ok or corrupt
[13:09] private_meta	I DID log it
[13:10] pieterh	and it was corrupt?
[13:10] private_meta	Yes
[13:10] pieterh	so the sender is getting it wrong...
[13:10] pieterh	and you're logging at the zmq_send() call?
[13:10] private_meta	in one case, out of a 300-symbol long string I got a 4-digit number, in another case I got a 2-byte-garbled string
[13:10] pieterh	did you log at the zmq_send() call?
[13:12] private_meta	I logged at the zmq method I think. I wanted to pry into it deeper, but I'm out of time right now, I'll get to the deepest send part and log there once I'm finished with the matter at hand
[13:12] pieterh	ack
[13:12] pieterh	I'm not always on IRC so if you get useful output, send to zeromq-dev...
[13:50] martin	Trying to build 0MQ with python bindings on OS X 10.6 - 0MQ installed into /usr/local/ no problems, building python extension gives me an exception at zmq/core/constants.c:2574: fatal error: error writing to -: Broken pipe
[13:50] martin	Any ideas?
[14:00] pieterh	martin_____: I'd check disk full, permissions on directory you're writing to
[14:02] martin	pieterh: Definitely not disk full, and I'm running the install as root (it's my own laptop)
[14:06] martin	here's the compilation command: gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -I/usr/local/include -Izmq/utils -Izmq/core -Izmq/devices -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c zmq/core/constants.c -o build/temp.macosx-10.6-universal-2.6/zmq/core/constants.o -Wno-unused-function -Wn
[15:37] jhawk28	pieterh: was looking at the specs, does the GPL license for the spec means that any impl of the specs need to be GPL?
[15:37] pieterh	jhawk28: nope
[15:37] pieterh	since you ask, I'll clarify this in the text
[15:37] jhawk28	it just means the spec itself
[15:38] pieterh	indeed
[15:38] jhawk28	good, I was hoping to use them to create a native Java impl
[15:39] jhawk28	in the future
[15:39] jhawk28	and I prefer BSD/MIT/Apache 2.0
[15:41] pieterh	jhawk28: can you check http://rfc.zeromq.org/spec:13 and tell me if it's clear?
[15:41] pieterh	see right hand panel...
[15:43] jhawk28	yes, that is clear. wasn't expecting it to be on the right hand panel, but ok
[15:44] pieterh	well, that makes it easier to apply to all specs, otherwise I have to edit each one individually
[15:44] pieterh	also, it's not really part of the spec but an explanatory note
[15:46] jhawk28	yes, thank you for the clarification
[15:51] pieterh	jhawk28: I'd be really interested in your experiences making a native implementation
[15:55] sustrik	th, pieterh: patch for issue 199 is at the ML
[15:55] pieterh	sustrik: great! I'll test that immediately
[15:58] pieterh	sustrik: it works, confirmed on 3.0
[15:58] sustrik	:)
[16:30] th	sustrik: oh really? fantastic!
[16:36] pieterh	sustrik: should I try to backport this to 2.1?
[16:36] sustrik	pieterh: yes
[16:36] pieterh	ok
[16:38] NikolaVeber	does anyone else have problems by copy/paste from the guide?
[16:38] __alex	erm, yes with the python-snippets
[16:39] NikolaVeber	I had the same with php and c...
[16:39] __alex	chars like "..." where one thing, but there's more which i couldn't figure out
[16:39] NikolaVeber	I guess it's the sodify's fault, or whatever this code formatting thing is calles
[16:40] NikolaVeber	codify*
[16:40] NikolaVeber	seen it in gedit and eclipse...
[16:41] __alex	was strange, took it to vim with utf8 encoding(also the file) and a valid python-encoding header
[16:41] __alex	it did boom
[16:44] th	sustrik: yea - confirmed again for 3-0
[16:44] pieterh	NikolaVeber: copy/paste from the Guide may do strange things...
[16:45] NikolaVeber	could be the formatting library for the code...
[16:46] pieterh	it should work but a 'smart' editor may trip over the formatting
[16:46] pieterh	I'm using Kate and can copy/paste without trouble
[16:46] pieterh	do you have a specific example that's failing?
[16:46] NikolaVeber	hm, my colleague just went home
[16:46] NikolaVeber	it was the php binding code
[16:47] pieterh	ok, raise an issue on the guide tracker
[16:47] NikolaVeber	req resp example
[16:47] pieterh	there are some characters that might need escaping
[16:47] NikolaVeber	I'll look at it tomorrow, with exact problem
[16:47] NikolaVeber	I had the same in c++
[16:48] pieterh	ack
[16:48] NikolaVeber	not a big deal, but spoils the fun sometimes :)
[16:48] pieterh	well, let me know the specific problem and we'll fix it... "it doesn't work" isn't useful data :)
[16:49] NikolaVeber	I know :)
[16:50] NikolaVeber	I tracked it last time it failed with c++ examples to stack overflow describing the bug in some code-colouring plugin
[16:51] NikolaVeber	ok, I have it
[16:51] NikolaVeber	:)
[16:52] NikolaVeber	I have copied the first c++ example in gedit
[16:52] NikolaVeber	http://pastebin.com/zKhHBnr3
[16:52] pieterh	ok, let me try that
[16:52] pieterh	which is the 'first c++ example'?
[16:52] pieterh	hwserver?
[16:53] NikolaVeber	/ââHelloâWorldâserverâinâC++
[16:53] NikolaVeber	//ââHelloâWorldâserverâinâC++
[16:53] NikolaVeber	right beneath figure one
[16:53] pieterh	I copied/pasted into gedit, it works fine
[16:54] NikolaVeber	gives a bunch of crap in my case...
[16:54] pieterh	works fine in Kate too
[16:54] pieterh	vim shows nothing, I think because of pasteboard confusion
[16:54] NikolaVeber	could be the firefox' fault
[16:54] pieterh	did you enable the "Paste as crap" option in gedit?
[16:54] NikolaVeber	hm, not that I know
[16:54] pieterh	it needs to paste as plain text, not HTML
[16:56] pieterh	NikolaVeber: perhaps your default language setting affects it
[16:56] pieterh	I'm using US English
[16:56] NikolaVeber	hm, my UI language is english
[16:56] pieterh	hmm, so the error isn't in gedit
[16:56] pieterh	it's in gcc when you save the file
[16:57] NikolaVeber	whell, if I "clean up" the code
[16:57] NikolaVeber	that is, do a bunch of backspace/enter moves
[16:57] NikolaVeber	it works
[16:57] NikolaVeber	so it has to be some invisible characters
[16:58] pieterh	can you save, put the source into a zip, send me that?
[16:58] NikolaVeber	shure
[16:58] pieterh	ah, I know...
[16:58] pieterh	hehe :)
[16:59] NikolaVeber	?
[16:59] NikolaVeber	are you getting the file?
[16:59] pieterh	yeah, but it's not necessary any more
[16:59] NikolaVeber	ok :)
[16:59] pieterh	I use unicode whitespace characters to force the layout to work
[17:00] th	Assertion failed: new_sndbuf > old_sndbuf (mailbox.cpp:183)
[17:00] th	whats that?
[17:00] pieterh	it was... difficult to get the coloring to work otherwise
[17:00] NikolaVeber	hehe, that could be it :)
[17:00] pieterh	th: looks like a resource issue
[17:00] th	pieterh: after running 199 testcase with 3-0 for a while
[17:00] pieterh	th: read http://zero.mq/tips
[17:00] CIA-75	libzmq: 03Pieter Hintjens 07master * re78cc47 10/ (3 files): Moved tests off 5555 (conflict with Eclipse) ...
[17:01] pieterh	NikolaVeber: there is no way to fix this afaik, but I could be wrong
[17:01] NikolaVeber	do you have the sources for all examples somewhere?
[17:01] pieterh	NikolaVeber: please log an issue in the tracker, I'll think about it and find a solution
[17:01] NikolaVeber	it might be a good idea to have plain-text files linked
[17:01] pieterh	if you click on the "GET THE EXAMPLES" button on the main guide page ...
[17:02] sustrik	th: it's socketpair buffers filled up
[17:02] pieterh	NikolaVeber: there is also a section at http://zguide.zeromq.org/page:all#Getting-the-Examples
[17:02] NikolaVeber	lets see
[17:02] NikolaVeber	:)
[17:03] th	sustrik: what resource would that be? vmsize?
[17:04] th	sustrik: ahh data segment probably.
[17:04] th	ah HWM would probably help
[17:04] NikolaVeber	pieterh, maybe linking to sources in the guide? I don't mind getting them all personally, just speculating on the general user experience :)
[17:05] sustrik	th: no, it's socketpair buffer size limit
[17:05] pieterh	NikolaVeber: the Guide already links to the sources, I'm not sure what you mean
[17:05] pieterh	you mean a link in each example to the source code?
[17:05] sustrik	th: what OS are you on?
[17:05] NikolaVeber	pieterh, I mean links to the plaintext-source
[17:06] pieterh	yes, that part I understood but since there are already two such links I am asking what more links you want, and where exactly
[17:06] NikolaVeber	but if they are "pimped" with utf8 whitespaces, it's not much of a help
[17:06] NikolaVeber	under each example, a link to the appropriate plain-text version
[17:07] pieterh	it's better just to make copy/paste work, no?
[17:07] NikolaVeber	that would be optimal :)
[17:07] pieterh	please enter an issue in the tracker, I'll get to it later
[17:07] pieterh	thanks
[17:07] NikolaVeber	np, thank you :)
[17:09] NikolaVeber	done
[17:09] th	sustrik: ubuntu10.04 64bit
[17:26] CIA-75	libzmq: 03Martin Sustrik 07master * re5d4cd3 10/ src/dist.cpp : Yet one more fix related to PUB socket and multipart messages ...
[17:28] CIA-75	libzmq: 03Martin Sustrik 07master * r6ecec9b 10/ src/xrep.cpp : Current inpipe remains unchaged in XREP when other pipe terminates ...
[17:38] iFire	neopallium hmm lthreads uses pthreads which is unix only [windows is lgpl] :/
[17:39] iFire	I wonder if it's possible to rewrite it using tbb or boost threads
[17:41] pieterh	iFire: what's lthreads?
[17:42] iFire	https://github.com/Neopallium/lua-llthreads
[17:42] iFire	apparently it's a lua pthreads wrapper
[17:42] pieterh	I wrote a similar class for czmq, zthreads, which has the win32 code
[17:42] pieterh	doesn't wrap all pthreads, just thread creation
[17:43] iFire	well I'm seeing how far I can go doing it in lua
[17:43] iFire	Using send_msg/recv_msg functions running under LuaJIT2 (git HEAD):
[17:43] iFire	mean throughput: 6160911 [msg/s]
[17:43] iFire	mean throughput: 1478.619 [Mb/s]
[17:43] iFire	C++ code:
[17:43] iFire	mean throughput: 6241452 [msg/s]
[17:43] iFire	mean throughput: 1497.948 [Mb/s]
[17:43] iFire	from the lua-zmq repo
[17:43] pieterh	that's pretty nice
[17:44] iFire	apparently I have to compile lua sockets if I want to run those benchmarks though
[17:45] iFire	that's what I've been trying to do
[18:02] th	pieterh: patch works for 2.1 as well. it even applies with a small offset
[18:03] pieterh	th: great! I've not had time to test it yet
[18:03] pieterh	I'll do this right now
[18:10] pieterh	th: ack, confirmed on 2.1
[18:10] pieterh	it'll go into 2.1.7
[18:12] pieterh	th: if you want to get the 2.1 master, it's at git://github.com/zeromq/zeromq2-1
[18:41] iFire	pieterh ok that was weird
[18:41] iFire	first windows sucks
[18:41] iFire	and luajit has a higher performance than c O.o
[18:42] pieterh	performance figures are quite variable, +- 10% afaik
[18:42] pieterh	more, depending on your configuration
[18:42] iFire	pieterh what throughput do you usually get for windowS?
[18:42] pieterh	I don't use Windows, personally
[18:43] pieterh	but reports are 50-20% of Linux
[18:43] iFire	http://pastebin.ca/2052970
[18:43] iFire	this is what I got
[18:43] iFire	for some reason c++ is slower than luajit
[18:44] pieterh	iFire: without specifying what setup you're on, the figures are pretty meaningless, sorry
[18:44] iFire	ok
[18:44] iFire	well I'm trying to figure out what's the difference between windows c++ and lua
[18:44] pieterh	also, you need to run the tests quite a few times
[18:44] iFire	hmm
[18:44] pieterh	and take the average of the N fastest runs
[18:44] iFire	it's quite variable?
[18:44] pieterh	one run proves very little
[18:45] pieterh	well, we know that lua can't be faster than the C api it's calling :)
[18:45] pieterh	so you have at least the variance shown here
[18:48] sustrik	throughput is silly metric, it's inherently unstable
[18:49] iFire	the question I want to answer is how much of a difference is between lua and c++
[18:50] sustrik	check latencies
[18:50] sustrik	latency is much better metric
[18:51] sustrik	it should be fairly stable and give you an sane idea what the difference between the two is
[18:59] iFire	something is wrong on my system
[19:00] iFire	http://pastebin.ca/2052978
[19:01] iFire	also missed a zero on the inproc. but I'm not comparing that
[19:03] iFire	bah
[19:03] iFire	I should do this correctly
[19:03] iFire	and have multiple tests
[19:13] sustrik	latency tests are stable, you'll see similar figures all the time
[19:13] iFire	I wonder where the extra 10 us comes from
[19:14] iFire	sustrik actually it fluctuates too
[19:14] iFire	on a desktop
[19:14] sustrik	more than 2-3 us?
[19:14] sustrik	well, if there's lot of stuff running on the box, it can blur the results
[19:16] pieterh	iFire: if you don't have 4 cores, and a quiet box, any test will be quite flaky
[19:16] pieterh	flakey
[19:16] iFire	hmm
[19:17] iFire	I guess I can try doing the tests on a vps where it's not a desktop
[19:17] pieterh	ideal is two boxes connected over a private network
[19:17] pieterh	how many cores on your desktop?
[19:17] iFire	4
[19:17] pieterh	and you're running other software?
[19:17] iFire	yes
[19:18] iFire	windows says 3-5% CPU use though
[19:18] pieterh	you're on Windows?
[19:18] iFire	windows is probably a terrible platform
[19:18] pieterh	forget it, you're not going to get serious metrics
[19:19] pieterh	but if you really must use that, then kill anything you don't absolutely need
[19:19] pieterh	and run the tests multiple times, taking average of the 3 lowest runs (run each 10-20 times)
[19:20] iFire	well it's not serious but it's just nagging me to see c++ how slower than lua on thr and lat
[19:20] pieterh	also, make sure you send/recv enough messages, e.g. 10M or 20M
[19:20] iFire	hmm
[19:20] iFire	ok I'll try 10 million
[19:20] sustrik	don't do that with latency tests, it'll take forever
[19:21] sustrik	10 million for throughput
[19:21] sustrik	100,000 for latency
[19:23] sustrik	the lua being faster is kind of strange
[19:23] sustrik	how does it compute the result?
[19:24] iFire	https://github.com/Neopallium/lua-zmq/tree/master/perf
[19:24] iFire	not sure how faithful are the ports
[19:25] iFire	bah
[19:25] iFire	that might be why
[19:26] sustrik	ah, it uses lua's time function
[19:26] sustrik	if you want to have it consistent with C tests
[19:27] sustrik	you should use "stopwatch" provided with libzmq (see zmq_utils.h_
[19:35] iFire	I guess it doesn't really matter :) it's fast enough. especially if I'm using inproc
[20:04] zerosanity	Is it possible to use epgm or pgm on EC2?
[20:29] jhawk28	zerosanity: it doesnt look like EC2 supports multicast
[21:44] dan	I have a really basic forwarder question... I'm using pubsub and when I connect directly, it seems to work just fine. When I put a forwarder in the middle, I don't get any messages. I suspect I'm doing something wrong, but can't figure it out.
[21:46] dan	My minimal code looks like: publishers = context.socket(zmq.SUB) publishers.bind("tcp://127.0.0.1:5555"); subscribers = context.socket(zmq.PUB) subscribers.bind('tcp://127.0.0.1:5556')zmq.device(zmq.FORWARDER, publishers, subscribers)
[22:12] jhawk28	dan__: are you subscribing to all?
[22:16] jhawk28	dan__: you need to call publishers.subscribe("".getBytes, 0);
[22:19] neopallium	what version of zeromq was the zmq_stopwatch_*() functions introduced?
[22:25] jhawk28	neopallium: https://github.com/zeromq/libzmq/commit/3b636d7d185cd1a1fa300b4d9ca78d2587cb4bf5#diff-1
[22:25] jhawk28	thats when it was moved to zmq_utils.cpp
[22:27] neopallium	jhawk28: thanks, I am going to add support for those functions to my lua bindings and wanted to make sure it wasn't a new feature.
[22:28] jhawk28	I'm seeing them at least back to Aug 7
[22:28] jhawk28	2010
[22:30] jhawk28	ah, found when it was added: 7f01e9970d211235fc8057de6dc41ba8ceafe795
[22:30] jhawk28	Jun 17, 2010
[22:32] neopallium	looks like that is zeromq 2.0.7
[22:33] jhawk28	based on the comment, it looks like it was removed then added
[22:34] neopallium	that is old enough, also those functions are not documented.
[22:34] jhawk28	and then moved on Apr 11, 2011
[22:35] neopallium	only the source was moved not the header prototypes.
[22:35] neopallium	so that is not a problem for bindings.
[22:36] neopallium	jhawk28: thanks for helping me track that down. I didn't think to use git blame to find out the answer.
[22:38] jhawk28	np