Monday May 2, 2011

[Time] NameMessage
[01:00] neopallium iFire: I am here.
[01:01] neopallium iFire: yes, build it like any C Lua module. Make sure to link it against libzmq (-lzmq).
[01:10] iFire neopallium
[01:10] iFire > require "lua-zmq"
[01:10] iFire error loading module 'lua-zmq' from file '.\lua-zmq.dll':
[01:10] iFire The specified procedure could not be found.
[01:10] iFire I get that
[01:10] iFire where pre_generated-zmq.nobj.c is compiled into lua-zmq.dll
[01:11] iFire on windows
[01:11] iFire using lua-jit
[02:12] iFire neopallium how was dinner :P
[02:58] iFire neopallium I can't seem to load the library from a lua interpreter
[02:58] iFire what's a short lua thing to test if it works
[03:02] private_meta AH GODDAMMIT!
[03:07] private_meta I spent so much time to implement our communications library with ZeroMQ and now it seems like it's got the same problem as ASIO
[03:07] private_meta OMG this pisses me off so much
[03:08] iFire private_meta same problem?
[03:08] private_meta Messages are lost/garbled
[03:08] private_meta when I compile it with OpenMPI
[03:10] iFire I thought you weren't supposed to use zeromq with locks and stuff
[03:10] iFire each thread is supposed to be its own unit
[03:10] iFire and you communicate through inproc/ipc/tcp/etc.
[03:11] private_meta I don't know what you get at with that
[03:13] private_meta iFire: not sure what that has to do with my problem
[03:13] iFire hmm
[03:13] iFire sorry
[03:13] iFire well mpi and zeromq do the same things
[03:14] iFire unless you're using it to connect two systems
[03:14] iFire
[03:16] private_meta I'm using MPI to create parallel processes and to control them, I wanted to use zeromq for backend communication
[03:16] private_meta each process being a client
[03:16] private_meta these two are independent
[03:16] jer why use MPI for that and not zeromq for both?
[03:18] private_meta Because zmq doesn't supply most of the mpi functions like scattering, gathering, ... out of the box like mpi
[03:29] neopallium iFire: back, you need to use: require"zmq"
[03:30] iFire hmm
[03:30] iFire what about the dll
[03:30] iFire I'm doing a custom packaging
[03:30] iFire so everything is in the wrong place
[03:30] neopallium the dll, I think you will need to call: zmq.dll
[03:31] iFire > require "zmq"
[03:31] iFire error loading module 'zmq' from file '.\zmq.dll':
[03:31] iFire The specified procedure could not be found.
[03:31] iFire I renamed lua-zmq.dll to zmq.dll
[03:31] iFire hmm
[03:31] iFire I don't actually NEED poller.lua [for testing if the dll loads right?]
[03:31] neopallium can you check that luaopen_zmq() is a public/exported symbol in the .dll?
[03:32] neopallium for testing, no, you don't need poller.lua or threads.lua
[03:32] neopallium this page has instructions on compiling a Lua module on Windows:
[03:33] iFire neopallium will depdency walk in windows show me it
[03:33] iFire dependency walker
[03:34] neopallium I am not a windows dev., so I am not sure.
[03:35] iFire int __declspec(dllexport) MyModuleName (lua_State* L) { ... } ?
[03:35] iFire note : make sure your main init function is exported with __declspec(dllexport) like this :
[03:35] iFire int __declspec(dllexport) MyModuleName (lua_State* L) { ... }
[03:35] iFire neopallium is that needeD?
[03:36] neopallium no, the symbol that needs to be exported for this module is: int luaopen_zmq(lua_State *L);
[03:37] neopallium in the pre_generated-zmq.nobj.c file you will see it as: LUALIB_API int luaopen_zmq(lua_State *L) {
[03:38] neopallium very close to the end of the .c file
[03:38] iFire yes
[03:38] iFire hmm
[03:38] iFire somehow add int __declspec(dllexport) <-?
[03:39] neopallium LUALIB_API will do that on windows.
[03:39] iFire it doesn't :/
[03:40] iFire neopallium ok I did __declspec(dllexport) LUALIB_API int luaopen_zmq(lua_State *L) {
[03:40] iFire and it seems to work
[03:41] iFire what's a way of printing the zmq table
[03:41] neopallium look in your luaconf.h file, that is where LUALIB_API & LUA_API are defined.
[03:41] iFire using luajit
[03:41] neopallium for k,v in pairs(zmq) do print(k,v) end
[03:43] neopallium ah, you need to define LUA_BUILD_AS_DLL on windows when compiling a Lua C module.
[03:43] iFire <- this is a working table print right?
[03:43] neopallium yup, that looks right.
[03:45] iFire ha
[03:45] iFire well using that define makes the build fail
[03:46] iFire >..\..\..\deps\lua-zmq\src\pre_generated-zmq.nobj.c(3121): error C2491: 'luaopen_zmq' : definition of dllimport function not allowed
[03:46] iFire neopallium you probably mean LUA_LIB
[03:47] iFire #if defined(LUA_CORE) || defined(LUA_LIB)
[03:47] iFire #define LUA_API __declspec(dllexport)
[03:47] iFire #else
[03:47] iFire #define LUA_API __declspec(dllimport)
[03:47] iFire #endif
[03:47] iFire actually both
[03:47] neopallium hmm, maybe I shouldn't be using LUALIB_API or LUA_API for exporting the module's luaopen_*() function.
[03:47] iFire [search for LUA_LIB]
[03:48] neopallium I will look for a better solution, but for now just replace LUALIB_API with __declspec(dllexport)
[03:48] neopallium yeah, I think those defines are more for the Lua core not modules.
[03:49] neopallium ok, I need to go now, I will be back in about an hour or two.
[03:49] neopallium but, I will take a closer look at the problem later tonight.
[03:50] iFire neopallium pm
[03:55] iFire I added two defines and it worked "LUA_BUILD_AS_DLL", "LUA_LIB"
[06:03] neopallium iFire: I committed a fix to lua-zmq that should make it easier to compile on windows. You don't need to define LUA_BUILD_AS_DLL, LUA_LIB, or LUALIB_API.
[06:48] staylor is there a way to get the connection status of a socket to a given endpoint?
[06:49] guido_g no
[07:06] jer i've got a couple of workers creating push sockets sending some results out to a single node binding a pull socket. communication works fine, as i'd expect. however, after i send a first message off, my workers cpu usage goes up over 30% and none of the workers are processing any jobs, sockets are idle. not sure wtf is going on. anyone have any suggestions on what i might look for first?
[07:09] staylor guido_g so would the best way to know if a connection can be made to open a separate standard connection to the host, but not exchange any data?
[07:09] guido_g you still don't know if the ømq endpoint is connected and alive
[07:09] guido_g only ømq could tell you that
[07:10] guido_g and it doesn't
[07:11] staylor well ideally it would, but I guess at least a standard socket would tell me that at least that much works. I'd like to know if a connection is refused basically, as it tells me most likely the other end failed to map a upnp port or the service isn't running.
[09:43] edwin hi, the helloworld zeromq example is not working properly for me with 0MQ 2.1.6 on Debian x86-64, and here's a testcase:
[09:43] edwin output from testcase( zmq_send fails with Bad address):
[09:43] edwin the server keeps giving those errors, and the client hangs
[09:43] edwin same with hwserver/hwclient
[09:46] pieterh edwin: could you run hwserver & hwclient (not your test case, which is rather more code) and post the messages that gives? thanks
[09:47] edwin
[09:47] edwin pieterh: the client hangs after the first send
[09:48] edwin sometimes it receives too, but doesn't do a 2nd send
[09:48] pieterh so it hangs, presumably the server isn't sending a reply?
[09:48] edwin interesting thing is that it worked better yesterday
[09:48] edwin yes the server gives an error on zmq_send
[09:48] edwin see output from my code:
[09:49] edwin ZMQ error: Bad address
[09:49] pieterh what does the server say on zmq_send (the hwserver, not yours)
[09:49] edwin let me add a printf hold on
[09:50] edwin hwserver zmq_send failed: Bad address
[09:50] edwin
[09:50] edwin that is the change I made
[09:50] edwin to print the error
[09:51] pieterh ok, let me just retest that
[09:52] pieterh it works for me, but that's not helpful...
[09:52] pieterh I'm not sure what 'bad address' means for 0MQ, let me dig a little
[09:53] pieterh EFAULT = invalid memory access, I'd assume
[09:54] pieterh ah, edwin....
[09:54] edwin ah yes version 2.0.x crashes with SIGSEGV with invalid access
[09:54] pieterh can you please take the actual current version of hwserver
[09:55] pieterh I'd posted a version with a bug in it, that's what's hitting you
[09:55] pieterh you can see it's putting "World" into the request, not the reply... :-/
[09:55] edwin yeah that works
[09:56] edwin ok let me fix my latency measurement tool then :)
[09:56] pieterh sorry about that, someone caught it but it was there for a day or two
[09:57] edwin Time: 1.521894s, 6570.759856 Req/s, Latency: 152.189400 us
[09:57] edwin yep works now
[09:57] edwin thanks :)
[09:58] edwin I was going to try and improve the latency by using a pipe instead of socketpair for the mailbox
[09:58] edwin but then the helloworld started failing...
[10:00] edwin (unix socket latency is ~28us, and pipe is 3-4x faster than the socket according to hackbench)
[10:03] mikko good morning
[10:04] pieterh mikko: hi, how're you doing?
[10:04] mikko i am excellent
[10:04] mikko extra bank holiday today
[10:04] pieterh :) yeah
[10:04] mikko very bad time for freelancers
[10:04] mikko :)
[10:05] mikko you?
[10:06] pieterh working on a new android app
[10:06] pieterh quite fun
[10:07] mikko are you using zeromq?
[10:07] pieterh it's fun to see that stuff I was designing in 1999 is still better than web frameworks today
[10:07] pieterh nope, this is a different project
[10:07] pieterh well, the web frameworks are pretty, and easy to use
[10:08] pieterh I'd not use 0MQ across the Internet anyhow
[10:08] pieterh we're making connectivity for this app across RestTL, and we'll use 0MQ in the backend
[10:15] edwin hmm, latency with pipes for mailbox is 43 - 44us (vs 49-50 us with unix socket for mailbox). If I turn on CPU freq scaling then latency is 135 - 150 us
[10:16] edwin however using just a unix socketpair (without zeromq) latency is 10us
[10:18] edwin why is a socket used for zeromq's mailbox, instead of lets say a ypipe and a pthread condition?
[10:27] sustrik edwin: even with ypipe you have to synchronise the two sides of the pipe somehow
[10:27] sustrik but yes, ypipe + using socketpair just for signaling would be superior solution
[10:27] mikko sustrik: did you notice that several compilers fail now with master?
[10:27] edwin not sure if socketpair is faster than a pthread condition for signaling
[10:28] edwin will have to test
[10:28] mikko well, ICC and sun studio
[10:28] sustrik the problem is that the signal has to interrupt a polling loop
[10:28] sustrik you can't do that with condition variable afaik
[10:29] edwin ah because you also use it to poll network sockets
[10:29] sustrik exactly
[10:29] edwin won't a signal() interrupt poll with EINTR?
[10:30] sustrik it's a nightmare to combine signals with threads
[10:30] edwin the signal handler doesn't have to actually do anything, just wake the poller
[10:31] edwin of course there is a problem which signal to use
[10:31] edwin since libzmq is a library..
[10:31] sustrik there's a problem which thread the signal will be delivered to
[10:32] edwin yeah that too, ok bad idea. using pipe to wake poll() sounds fine, we can avoid a recv() on the mailbox
[10:32] sustrik you've just said it's slower
[10:32] edwin its slower than pthread, but pipe is faster than unix sockets
[10:33] edwin (which is used now)
[10:33] sustrik ok, we can do that
[10:33] sustrik if you have a patch, send it to the mailing list
[10:33] edwin sure, I'll be testing some variants
[10:33] sustrik ok
[10:33] edwin patch aganist 2.1.6 is fine?
[10:33] edwin or the 3.0 master?
[10:33] sustrik master is better
[10:34] sustrik but the mailbox code haven't changed iirc
[10:34] sustrik so, it'll be the same anyway
[10:34] edwin ok
[10:34] sustrik btw, if you want to implement ypipe+pipe thing
[10:34] sustrik that would solve one serious problem 0mq is experiencing
[10:35] sustrik namely that the mailbox can overflow in some special cases
[10:35] sustrik ypipe+pipe would solve that as there's no limit on number of items in yqueue
[10:36] edwin won't send() block if mailbox is full?
[10:36] edwin or zeromq spins on send with errno=EAGAIN?
[10:37] sustrik it have blocked originally, but that leads to deadlocks
[10:37] sustrik so now it tries to increase the mailbox size
[10:37] sustrik which doesn't work well on different OSes
[10:38] sustrik btw, note that if pipe is used only for signaling
[10:38] sustrik it can be optimised on some OSes
[10:38] sustrik eg. on Linux we can use eventfd which is much faster than socketpair or pipe
[10:42] edwin eventfd looks cool, I should use that in clamd as well (I use a pipe with 1 byte write for signaling)
[11:20] th sustrik: did you have a look at #199 yet? thats a 3-0 master issue..
[11:22] sustrik th: yes, i've reproduced it this morning
[11:22] th sustrik: thats a great start :)
[12:22] djc guido_g: turned out my test box had something else listening on a port pyzmq was using for testing...
[13:05] private_meta pieterh: eh... remember that I went to zeromq to fix a problem I had with asio? got the same problem now with zeromq
[13:05] pieterh private_meta: lol...
[13:05] pieterh what is happening?
[13:06] private_meta I send normal strings with the messaging library, all is ok, I send (boost)serialized classes, all is ok. Once I send boost serialized classes when the stuff is compiled and used with MPI, some of these messages are garbled beyond recognicion
[13:06] private_meta *recognition
[13:07] pieterh hmm, so it's a problem in your app, not in the transport
[13:07] private_meta Figured that out last night, but not sure if it's really the fault of boost serialization in combination with mpi
[13:07] private_meta well, yes and no
[13:07] pieterh sounds like a memory management issue
[13:07] private_meta the message i send, and the serialize looks ok
[13:07] private_meta but it's broken when it arriveso n the other side
[13:08] pieterh you sure you're not deallocating the memory before the message is really sent?
[13:08] private_meta but only when i compile it with mpi
[13:08] private_meta pretty sure
[13:08] pieterh and of course a minimal test case requires MPI... so I can't really help much... :/
[13:09] private_meta That was one of the problems. It seems the minimal test case is quite complex
[13:09] private_meta Well, I need to write a paper now, but as soon as I'm finished, I'm going back to the matter, in a couple of days I hope
[13:09] pieterh well, try sending the data via a device, log it there and see if it's ok or corrupt
[13:09] private_meta I DID log it
[13:10] pieterh and it was corrupt?
[13:10] private_meta Yes
[13:10] pieterh so the sender is getting it wrong...
[13:10] pieterh and you're logging at the zmq_send() call?
[13:10] private_meta in one case, out of a 300-symbol long string I got a 4-digit number, in another case I got a 2-byte-garbled string
[13:10] pieterh did you log at the zmq_send() call?
[13:12] private_meta I logged at the zmq method I think. I wanted to pry into it deeper, but I'm out of time right now, I'll get to the deepest send part and log there once I'm finished with the matter at hand
[13:12] pieterh ack
[13:12] pieterh I'm not always on IRC so if you get useful output, send to zeromq-dev...
[13:50] martin Trying to build 0MQ with python bindings on OS X 10.6 - 0MQ installed into /usr/local/ no problems, building python extension gives me an exception at zmq/core/constants.c:2574: fatal error: error writing to -: Broken pipe
[13:50] martin Any ideas?
[14:00] pieterh martin_____: I'd check disk full, permissions on directory you're writing to
[14:02] martin pieterh: Definitely not disk full, and I'm running the install as root (it's my own laptop)
[14:06] martin here's the compilation command: gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -I/usr/local/include -Izmq/utils -Izmq/core -Izmq/devices -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c zmq/core/constants.c -o build/temp.macosx-10.6-universal-2.6/zmq/core/constants.o -Wno-unused-function -Wn
[15:37] jhawk28 pieterh: was looking at the specs, does the GPL license for the spec means that any impl of the specs need to be GPL?
[15:37] pieterh jhawk28: nope
[15:37] pieterh since you ask, I'll clarify this in the text
[15:37] jhawk28 it just means the spec itself
[15:38] pieterh indeed
[15:38] jhawk28 good, I was hoping to use them to create a native Java impl
[15:39] jhawk28 in the future
[15:39] jhawk28 and I prefer BSD/MIT/Apache 2.0
[15:41] pieterh jhawk28: can you check and tell me if it's clear?
[15:41] pieterh see right hand panel...
[15:43] jhawk28 yes, that is clear. wasn't expecting it to be on the right hand panel, but ok
[15:44] pieterh well, that makes it easier to apply to all specs, otherwise I have to edit each one individually
[15:44] pieterh also, it's not really part of the spec but an explanatory note
[15:46] jhawk28 yes, thank you for the clarification
[15:51] pieterh jhawk28: I'd be really interested in your experiences making a native implementation
[15:55] sustrik th, pieterh: patch for issue 199 is at the ML
[15:55] pieterh sustrik: great! I'll test that immediately
[15:58] pieterh sustrik: it works, confirmed on 3.0
[15:58] sustrik :)
[16:30] th sustrik: oh really? fantastic!
[16:36] pieterh sustrik: should I try to backport this to 2.1?
[16:36] sustrik pieterh: yes
[16:36] pieterh ok
[16:38] NikolaVeber does anyone else have problems by copy/paste from the guide?
[16:38] __alex erm, yes with the python-snippets
[16:39] NikolaVeber I had the same with php and c...
[16:39] __alex chars like "..." where one thing, but there's more which i couldn't figure out
[16:39] NikolaVeber I guess it's the sodify's fault, or whatever this code formatting thing is calles
[16:40] NikolaVeber codify*
[16:40] NikolaVeber seen it in gedit and eclipse...
[16:41] __alex was strange, took it to vim with utf8 encoding(also the file) and a valid python-encoding header
[16:41] __alex it did boom
[16:44] th sustrik: yea - confirmed again for 3-0
[16:44] pieterh NikolaVeber: copy/paste from the Guide may do strange things...
[16:45] NikolaVeber could be the formatting library for the code...
[16:46] pieterh it should work but a 'smart' editor may trip over the formatting
[16:46] pieterh I'm using Kate and can copy/paste without trouble
[16:46] pieterh do you have a specific example that's failing?
[16:46] NikolaVeber hm, my colleague just went home
[16:46] NikolaVeber it was the php binding code
[16:47] pieterh ok, raise an issue on the guide tracker
[16:47] NikolaVeber req resp example
[16:47] pieterh there are some characters that might need escaping
[16:47] NikolaVeber I'll look at it tomorrow, with exact problem
[16:47] NikolaVeber I had the same in c++
[16:48] pieterh ack
[16:48] NikolaVeber not a big deal, but spoils the fun sometimes :)
[16:48] pieterh well, let me know the specific problem and we'll fix it... "it doesn't work" isn't useful data :)
[16:49] NikolaVeber I know :)
[16:50] NikolaVeber I tracked it last time it failed with c++ examples to stack overflow describing the bug in some code-colouring plugin
[16:51] NikolaVeber ok, I have it
[16:51] NikolaVeber :)
[16:52] NikolaVeber I have copied the first c++ example in gedit
[16:52] NikolaVeber
[16:52] pieterh ok, let me try that
[16:52] pieterh which is the 'first c++ example'?
[16:52] pieterh hwserver?
[16:53] NikolaVeber /  Hello World server in C++
[16:53] NikolaVeber //  Hello World server in C++
[16:53] NikolaVeber right beneath figure one
[16:53] pieterh I copied/pasted into gedit, it works fine
[16:54] NikolaVeber gives a bunch of crap in my case...
[16:54] pieterh works fine in Kate too
[16:54] pieterh vim shows nothing, I think because of pasteboard confusion
[16:54] NikolaVeber could be the firefox' fault
[16:54] pieterh did you enable the "Paste as crap" option in gedit?
[16:54] NikolaVeber hm, not that I know
[16:54] pieterh it needs to paste as plain text, not HTML
[16:56] pieterh NikolaVeber: perhaps your default language setting affects it
[16:56] pieterh I'm using US English
[16:56] NikolaVeber hm, my UI language is english
[16:56] pieterh hmm, so the error isn't in gedit
[16:56] pieterh it's in gcc when you save the file
[16:57] NikolaVeber whell, if I "clean up" the code
[16:57] NikolaVeber that is, do a bunch of backspace/enter moves
[16:57] NikolaVeber it works
[16:57] NikolaVeber so it has to be some invisible characters
[16:58] pieterh can you save, put the source into a zip, send me that?
[16:58] NikolaVeber shure
[16:58] pieterh ah, I know...
[16:58] pieterh hehe :)
[16:59] NikolaVeber ?
[16:59] NikolaVeber are you getting the file?
[16:59] pieterh yeah, but it's not necessary any more
[16:59] NikolaVeber ok :)
[16:59] pieterh I use unicode whitespace characters to force the layout to work
[17:00] th Assertion failed: new_sndbuf > old_sndbuf (mailbox.cpp:183)
[17:00] th whats that?
[17:00] pieterh it was... difficult to get the coloring to work otherwise
[17:00] NikolaVeber hehe, that could be it :)
[17:00] pieterh th: looks like a resource issue
[17:00] th pieterh: after running 199 testcase with 3-0 for a while
[17:00] pieterh th: read
[17:00] CIA-75 libzmq: 03Pieter Hintjens 07master * re78cc47 10/ (3 files): Moved tests off 5555 (conflict with Eclipse) ...
[17:01] pieterh NikolaVeber: there is no way to fix this afaik, but I could be wrong
[17:01] NikolaVeber do you have the sources for all examples somewhere?
[17:01] pieterh NikolaVeber: please log an issue in the tracker, I'll think about it and find a solution
[17:01] NikolaVeber it might be a good idea to have plain-text files linked
[17:01] pieterh if you click on the "GET THE EXAMPLES" button on the main guide page ...
[17:02] sustrik th: it's socketpair buffers filled up
[17:02] pieterh NikolaVeber: there is also a section at
[17:02] NikolaVeber lets see
[17:02] NikolaVeber :)
[17:03] th sustrik: what resource would that be? vmsize?
[17:04] th sustrik: ahh data segment probably.
[17:04] th ah HWM would probably help
[17:04] NikolaVeber pieterh, maybe linking to sources in the guide? I don't mind getting them all personally, just speculating on the general user experience :)
[17:05] sustrik th: no, it's socketpair buffer size limit
[17:05] pieterh NikolaVeber: the Guide already links to the sources, I'm not sure what you mean
[17:05] pieterh you mean a link in each example to the source code?
[17:05] sustrik th: what OS are you on?
[17:05] NikolaVeber pieterh, I mean links to the plaintext-source
[17:06] pieterh yes, that part I understood but since there are already two such links I am asking what more links you want, and where exactly
[17:06] NikolaVeber but if they are "pimped" with utf8 whitespaces, it's not much of a help
[17:06] NikolaVeber under each example, a link to the appropriate plain-text version
[17:07] pieterh it's better just to make copy/paste work, no?
[17:07] NikolaVeber that would be optimal :)
[17:07] pieterh please enter an issue in the tracker, I'll get to it later
[17:07] pieterh thanks
[17:07] NikolaVeber np, thank you :)
[17:09] NikolaVeber done
[17:09] th sustrik: ubuntu10.04 64bit
[17:26] CIA-75 libzmq: 03Martin Sustrik 07master * re5d4cd3 10/ src/dist.cpp : Yet one more fix related to PUB socket and multipart messages ...
[17:28] CIA-75 libzmq: 03Martin Sustrik 07master * r6ecec9b 10/ src/xrep.cpp : Current inpipe remains unchaged in XREP when other pipe terminates ...
[17:38] iFire neopallium hmm lthreads uses pthreads which is unix only [windows is lgpl] :/
[17:39] iFire I wonder if it's possible to rewrite it using tbb or boost threads
[17:41] pieterh iFire: what's lthreads?
[17:42] iFire
[17:42] iFire apparently it's a lua pthreads wrapper
[17:42] pieterh I wrote a similar class for czmq, zthreads, which has the win32 code
[17:42] pieterh doesn't wrap all pthreads, just thread creation
[17:43] iFire well I'm seeing how far I can go doing it in lua
[17:43] iFire Using send_msg/recv_msg functions running under LuaJIT2 (git HEAD):
[17:43] iFire mean throughput: 6160911 [msg/s]
[17:43] iFire mean throughput: 1478.619 [Mb/s]
[17:43] iFire C++ code:
[17:43] iFire mean throughput: 6241452 [msg/s]
[17:43] iFire mean throughput: 1497.948 [Mb/s]
[17:43] iFire from the lua-zmq repo
[17:43] pieterh that's pretty nice
[17:44] iFire apparently I have to compile lua sockets if I want to run those benchmarks though
[17:45] iFire that's what I've been trying to do
[18:02] th pieterh: patch works for 2.1 as well. it even applies with a small offset
[18:03] pieterh th: great! I've not had time to test it yet
[18:03] pieterh I'll do this right now
[18:10] pieterh th: ack, confirmed on 2.1
[18:10] pieterh it'll go into 2.1.7
[18:12] pieterh th: if you want to get the 2.1 master, it's at git://
[18:41] iFire pieterh ok that was weird
[18:41] iFire first windows sucks
[18:41] iFire and luajit has a higher performance than c O.o
[18:42] pieterh performance figures are quite variable, +- 10% afaik
[18:42] pieterh more, depending on your configuration
[18:42] iFire pieterh what throughput do you usually get for windowS?
[18:42] pieterh I don't use Windows, personally
[18:43] pieterh but reports are 50-20% of Linux
[18:43] iFire
[18:43] iFire this is what I got
[18:43] iFire for some reason c++ is slower than luajit
[18:44] pieterh iFire: without specifying what setup you're on, the figures are pretty meaningless, sorry
[18:44] iFire ok
[18:44] iFire well I'm trying to figure out what's the difference between windows c++ and lua
[18:44] pieterh also, you need to run the tests quite a few times
[18:44] iFire hmm
[18:44] pieterh and take the average of the N fastest runs
[18:44] iFire it's quite variable?
[18:44] pieterh one run proves very little
[18:45] pieterh well, we *know* that lua can't be faster than the C api it's *calling* :)
[18:45] pieterh so you have at least the variance shown here
[18:48] sustrik throughput is silly metric, it's inherently unstable
[18:49] iFire the question I want to answer is how much of a difference is between lua and c++
[18:50] sustrik check latencies
[18:50] sustrik latency is much better metric
[18:51] sustrik it should be fairly stable and give you an sane idea what the difference between the two is
[18:59] iFire something is wrong on my system
[19:00] iFire
[19:01] iFire also missed a zero on the inproc. but I'm not comparing that
[19:03] iFire bah
[19:03] iFire I should do this correctly
[19:03] iFire and have multiple tests
[19:13] sustrik latency tests are stable, you'll see similar figures all the time
[19:13] iFire I wonder where the extra 10 us comes from
[19:14] iFire sustrik actually it fluctuates too
[19:14] iFire on a desktop
[19:14] sustrik more than 2-3 us?
[19:14] sustrik well, if there's lot of stuff running on the box, it can blur the results
[19:16] pieterh iFire: if you don't have 4 cores, and a quiet box, any test will be quite flaky
[19:16] pieterh flakey
[19:16] iFire hmm
[19:17] iFire I guess I can try doing the tests on a vps where it's not a desktop
[19:17] pieterh ideal is two boxes connected over a private network
[19:17] pieterh how many cores on your desktop?
[19:17] iFire 4
[19:17] pieterh and you're running other software?
[19:17] iFire yes
[19:18] iFire windows says 3-5% CPU use though
[19:18] pieterh you're on Windows?
[19:18] iFire windows is probably a terrible platform
[19:18] pieterh forget it, you're not going to get serious metrics
[19:19] pieterh but if you really must use that, then kill anything you don't absolutely need
[19:19] pieterh and run the tests multiple times, taking average of the 3 lowest runs (run each 10-20 times)
[19:20] iFire well it's not serious but it's just nagging me to see c++ how slower than lua on thr and lat
[19:20] pieterh also, make sure you send/recv enough messages, e.g. 10M or 20M
[19:20] iFire hmm
[19:20] iFire ok I'll try 10 million
[19:20] sustrik don't do that with latency tests, it'll take forever
[19:21] sustrik 10 million for throughput
[19:21] sustrik 100,000 for latency
[19:23] sustrik the lua being faster is kind of strange
[19:23] sustrik how does it compute the result?
[19:24] iFire
[19:24] iFire not sure how faithful are the ports
[19:25] iFire bah
[19:25] iFire that might be why
[19:26] sustrik ah, it uses lua's time function
[19:26] sustrik if you want to have it consistent with C tests
[19:27] sustrik you should use "stopwatch" provided with libzmq (see zmq_utils.h_
[19:35] iFire I guess it doesn't really matter :) it's fast enough. especially if I'm using inproc
[20:04] zerosanity Is it possible to use epgm or pgm on EC2?
[20:29] jhawk28 zerosanity: it doesnt look like EC2 supports multicast
[21:44] dan I have a really basic forwarder question... I'm using pubsub and when I connect directly, it seems to work just fine. When I put a forwarder in the middle, I don't get any messages. I suspect I'm doing something wrong, but can't figure it out.
[21:46] dan My minimal code looks like: publishers = context.socket(zmq.SUB) publishers.bind("tcp://"); subscribers = context.socket(zmq.PUB) subscribers.bind('tcp://')zmq.device(zmq.FORWARDER, publishers, subscribers)
[22:12] jhawk28 dan__: are you subscribing to all?
[22:16] jhawk28 dan__: you need to call publishers.subscribe("".getBytes, 0);
[22:19] neopallium what version of zeromq was the zmq_stopwatch_*() functions introduced?
[22:25] jhawk28 neopallium:
[22:25] jhawk28 thats when it was moved to zmq_utils.cpp
[22:27] neopallium jhawk28: thanks, I am going to add support for those functions to my lua bindings and wanted to make sure it wasn't a new feature.
[22:28] jhawk28 I'm seeing them at least back to Aug 7
[22:28] jhawk28 2010
[22:30] jhawk28 ah, found when it was added: 7f01e9970d211235fc8057de6dc41ba8ceafe795
[22:30] jhawk28 Jun 17, 2010
[22:32] neopallium looks like that is zeromq 2.0.7
[22:33] jhawk28 based on the comment, it looks like it was removed then added
[22:34] neopallium that is old enough, also those functions are not documented.
[22:34] jhawk28 and then moved on Apr 11, 2011
[22:35] neopallium only the source was moved not the header prototypes.
[22:35] neopallium so that is not a problem for bindings.
[22:36] neopallium jhawk28: thanks for helping me track that down. I didn't think to use git blame to find out the answer.
[22:38] jhawk28 np