Wednesday January 19, 2011

[Time] NameMessage
[01:41] lechon mikko, i hope you've been able to get to sleep by now :]
[01:41] lechon i will think about the proposal a bit more and will send out some ideas soon
[01:42] mikko no
[01:42] mikko still up
[01:42] mikko looked into udt sockets a bit
[01:43] lechon what do you think about them?
[01:43] mikko seems like a nice idea
[01:44] mikko the api is very friendly if you have done socket programming
[01:44] lechon
[01:45] mikko yes
[01:56] lechon yeah, just like sockets
[02:12] jhawk28 is another one
[05:55] traviscline i'm writing a compat layer with pyzmq and a greenlet lib (gevent) and want to do some performance testing of a few configurations: namely: pure c, vanilla pyzmq, my pure python greened version, and a cython verision of my greened code
[05:55] traviscline I initially set up a little harness to send 1mil random ints from a c producer and just saw how fast I consumed with the various configurations
[05:56] traviscline I want to know if this seems like a valid test to compare the basic overhead of the alternate python options
[05:56] traviscline I basically want to determine how much overhead is incurred by using the greened version of pyzmq (and do some work to minimize it)
[05:56] traviscline thoughts?
[06:05] Steve-o Similar to the pyzmq graphs previously published?
[09:14] mikko good morning
[09:21] guido_g morning
[10:17] kabs Hello, I tried writing hello-world application for pub-sub system in c, but when I complile it , I get these errors , what libs it is looking for , I have already installed zeromq-2.0.10 on my system
[10:19] mikko -lzmq
[10:19] mikko you are not linking against libs
[10:33] kabs Thanks you, it worked!!
[13:44] benoitc /j/win 18
[14:31] pythonirc1011 I'm trying to build zeromq python bindings on win 7 64-bit with 32-bit python installed and am getting in trouble. Can anyone help please?
[14:31] pythonirc1011 the compilation of libzmq.dll is fine. but "python build" looks for libzmq.lib ?
[14:32] mikko pythonirc1011: yes
[14:32] mikko as far as i understand windows is that you link against .libs
[14:33] mikko .dll for runtime and .lib for linking
[14:33] pythonirc1011 mikko: the problem is that when i try to create .lib files from zeromq code, it fails on VS2010
[14:33] mikko pythonirc1011: whats the issue?
[14:33] pythonirc1011 lots of unresolved symbols
[14:34] pythonirc1011 stuff like this --
[14:35] mikko how do you create .lib files?
[14:35] mikko shouldnt they be build by default when .dll is built?
[14:36] mikko and can you use 64bit zeromq with 32bit python?
[14:36] mikko pythonirc1011: that looks like you are building static libraries
[14:36] mikko rather than dlls
[14:37] pythonirc1011 mikko: i can build dlls, no problem
[14:37] mikko pythonirc1011: it seems that when i do the continous integration builds the dll and lib are built to same directory
[14:37] mikko without using any special options
[14:38] pythonirc1011 but the pyzeromq, wants a .lib file --LINK : fatal error LNK1181: cannot open input file 'libzmq.lib'
[14:38] mikko pythonirc1011: yes, the build should generate you .lib and .dll into the lib/ directory
[14:38] pythonirc1011 mikko: In my case i only see the .dll built, no lib there...I changed the configuration to Release. Thats all i changed.
[14:38] mikko hmm
[14:39] mikko pythonirc1011: and thats 64bit build?
[14:39] sustrik with msvc2008 the lib was generated alongside the dll
[14:39] mikko do you need something special with 2010?
[14:40] mikko im building with 2008 atm
[14:40] sustrik maybe some msvc2010 quirk?
[14:40] pythonirc1011 no idea
[14:40] mikko do we support 64bit windows?
[14:40] sustrik no idea
[14:40] sustrik some people complained
[14:40] sustrik some have noted that the 64bit build it ok
[14:40] mikko and i dont think 64bit zeromq works with 32bit python
[14:41] sustrik i don't habe win64
[14:41] sustrik so i can't check myself
[14:41] pythonirc1011 mikko: can i just build 32-bit zeromq on 64-bit machines?
[14:41] mikko i think i got win7 64bit image somewhere
[14:41] pythonirc1011 I think i did...
[14:41] mikko pythonirc1011: sure
[14:41] pythonirc1011 lemme try again
[14:42] pythonirc1011 building now
[14:42] pythonirc1011 only .dll file there
[14:42] pythonirc1011 win32 build + release -- no .lib there
[14:43] mikko pythonirc1011: what does it say under Properties->Configuration Properties->Linker->Advanced
[14:43] mikko and import library there
[14:44] mikko foudn a reference online that it might be msvc2008 -> 2010 project conversion issue
[14:44] pythonirc1011 Import library there is void
[14:44] pythonirc1011 i can add something there...if you need me to
[14:45] mikko libzmq.lib
[14:45] mikko is what it should output
[14:46] pythonirc1011 mikko: -- no .lib file still
[14:52] mikko pythonirc1011: see the error message
[14:52] pythonirc1011 ah! $(OutDir), $(TargetName) and $(TargetExt) match?
[14:52] mikko maybe $(TargetName).lib
[14:52] mikko as the import library name
[14:54] pythonirc1011 still trying to find where they are ...
[14:55] mikko i dont have msvc2010 at hand here
[14:56] pythonirc1011 got it
[14:57] pythonirc1011 General->Taget Ext -> .lib (from .dll)
[14:57] pythonirc1011 that works
[14:57] mikko ahmm
[14:57] mikko did you rename the .dll to lib?
[14:57] pythonirc1011 i tried that...that didnt work
[14:57] pythonirc1011 pyzmq complained
[14:58] mikko yes, because these are two different things
[14:58] mikko import library and dll
[14:58] pythonirc1011 >>> import zmq --> Traceback (most recent call last): File "<stdin>", line 1, in <module> File "zmq\", line 26, in <module> from zmq.utils import initthreads # initialize threads ImportError: cannot import name initthreads
[14:58] mikko pythonirc1011:
[14:58] pythonirc1011 could compile and install pyzmq, but now when i try to import, get this
[14:58] mikko hmm
[14:59] mikko did you download pyzmq matching 2.0.10 version?
[14:59] pythonirc1011 yes indeed
[14:59] pythonirc1011 zeromq-2.0.10 == pyzmq-2.0.10
[14:59] mikko it tries to import something called "initthreads" ?
[15:00] mikko from zmq.utils
[15:00] pythonirc1011 i did a python build_ext and python install...anything wrong there?
[15:00] mikko pythonirc1011: not that i know of. however i dont really use python or windows
[15:01] mikko the process we run for pyzmq on linux is
[15:01] mikko python cython
[15:01] mikko python build_ext
[15:02] mikko python test
[15:34] traviscline any instructions on running the pyzmq perf tests?
[15:36] mikko none
[15:40] traviscline but if anyone has a 'how to run' lemme know
[15:43] pythonirc1011 anyone using pyzeromq on win 7 + VS2010 + python 32-bit here?
[15:53] traviscline figured it out
[20:09] lt_schmidt_jr Hello
[20:10] lt_schmidt_jr Looking for someone to shoot holes in an approach to implement pub/sub for web clients
[20:11] s0undt3ch ppl, with 2.1.0, on the call socket.recv_pyobj() I'm getting:
[20:11] s0undt3ch File "socket.pyx", line 657, in zmq.core.socket.Socket.recv_pyobj (zmq/core/socket.c:5470)
[20:11] s0undt3ch File "socket.pyx", line 94, in zmq.core.socket.Socket.__cinit__ (zmq/core/socket.c:1036)
[20:11] s0undt3ch TypeError: __cinit__() takes exactly 2 positional arguments (0 given)
[20:11] s0undt3ch why is this!?
[20:15] traviscline s0undt3ch: up to date with cython?
[20:15] s0undt3ch traviscline: think so
[20:15] s0undt3ch traviscline: at least, up to date with my distro's cython
[20:15] traviscline built with 0.14 fine
[20:15] s0undt3ch traviscline: and on another piece of code, it works good
[20:19] Skaag lt_schmidt_jr: shoo
[20:19] Skaag shoot
[20:40] lt_schmidt_jr Thanks Skaag
[20:40] Skaag lt_schmidt_jr: We actually are looking to implement the same, sort of a clone of
[20:41] lt_schmidt_jr not familiar with that, but ours is domain specific to allow multiple clients to manipulate graph data
[20:44] lt_schmidt_jr but the basic idea is to use Jetty Websocket (with Flash Socket when necessary) and have the instance specify what it is working on throgh the socket, establishing a pub and sub zeromq socket for each client and hooking it up using devices depending on deployment
[20:44] lt_schmidt_jr single machine - no devices
[20:44] lt_schmidt_jr using ICP (if I remember correctly)
[20:44] lt_schmidt_jr sorry single machine - local device with interproc
[20:45] lt_schmidt_jr multiple machines - somehow put the devices together
[20:45] lt_schmidt_jr not sure how though
[20:45] lt_schmidt_jr does this make any sense?
[20:46] lt_schmidt_jr On my earlier comment about no devices - I don't think that is possible with many pubs and subs
[20:47] lt_schmidt_jr unless they can discover each other somehow
[20:48] lt_schmidt_jr I am thinking about using zookeeper perhaps at a later time
[20:48] Skaag ok, we've been down the same road
[20:48] Skaag at the moment, zookeeper is good for a fixed (unchanging) number of nodes
[20:49] Skaag where it can help you, is in the notification on services that go up/down on those nodes
[20:49] Skaag and in leader election, among the fixed set of nodes
[20:52] lt_schmidt_jr Our number of nodes is fairly low (in terms of what we want to subscribe to)
[20:53] lt_schmidt_jr we are thinking of it more like channels vs individual components
[20:54] lt_schmidt_jr otherwise the client has to mirror whole state to the server
[20:54] lt_schmidt_jr Skaag are you using a similar Java stack?
[20:55] Skaag I'm not sure I understand how you use the term 'component'
[20:56] lt_schmidt_jr leaf nodes , which are many - we are looking to route fairly close to the base of the tree - if that makes sense
[20:56] Skaag also I recommend you make a spreadsheet to calculate data packet sizes, to avoid bottlenecks, and to properly shape your data structures/sizes, it helps a lot to manage expectations
[20:56] Skaag (unrelated to the issue but important)
[20:56] Skaag yes understood
[20:57] lt_schmidt_jr I see
[20:57] lt_schmidt_jr so you are currently using zookeeper?
[20:57] Skaag not anymore
[20:57] Skaag I installed it, tried it, and found out it does not know how to add new nodes to the cluster
[20:58] Skaag the list of nodes is hardcoded in a config file
[20:58] Skaag quite annoying, and kinda defeating the purpose in my opinion
[20:58] lt_schmidt_jr you are talking about Zookeeper nodes?
[20:58] Skaag well in our case the idea was that machines would suddenly show up in a cluster
[20:59] Skaag now I have some idea about using some sort of gnutella variant :-)
[20:59] Skaag decentralized, etc.
[20:59] lt_schmidt_jr interesting
[21:00] lt_schmidt_jr so do you have a similar topology with many publishers and subscribers on the same channels?
[21:00] Skaag yes
[21:00] Skaag at the moment, we have a POC cluster of 16 nodes
[21:00] Skaag it's tiny
[21:00] Skaag we used Rabbitmq-server at first
[21:00] Skaag but I was told by one of their team members that 16 nodes is considered a large cluster for them
[21:00] lt_schmidt_jr I see
[21:00] Skaag and since we're looking to scale to thousands of nodes, we had to abandon it
[21:01] lt_schmidt_jr I was looking at RabbitMq and XMPP pubsub initially
[21:01] lt_schmidt_jr as well
[21:01] Skaag same here
[21:01] lt_schmidt_jr how are you bridging pubsub between nodes?
[21:01] Skaag this is at the moment done naively
[21:02] lt_schmidt_jr are you also using a forwarder on each machine
[21:02] lt_schmidt_jr ?
[21:02] Skaag all nodes are publishing to all nodes with zmq
[21:02] Skaag yes with forwarders
[21:02] lt_schmidt_jr this is too funny
[21:03] lt_schmidt_jr so 2 forwarders per machine?
[21:03] lt_schmidt_jr one in each direction?
[21:03] Skaag I need to check with my team member who implemented that part
[21:03] Skaag (we're a tiny team)
[21:04] lt_schmidt_jr I am a team of 1 right now, building a small POC
[21:04] lt_schmidt_jr are you also using jetty - or are you non-http system?
[21:07] Skaag yes we plan to use jetty
[21:08] Skaag since we are 99% java based
[21:08] lt_schmidt_jr Have you looked at Atmosphere?
[21:08] lt_schmidt_jr its attractive, but I am not sure that frameworks fits well with zmq
[21:29] Skaag no I haven't
[21:29] Skaag checking it out
[21:37] mikko good evening
[21:37] lt_schmidt_jr Hello
[21:37] mikko sustrik_: there?
[22:03] lt_schmidt_jr Skaag, what is your application doing, if you don't mind me asking?
[22:04] Skaag the system that uses zmq is intended to send/collect statistics information and aggregate them into a single body of statistical data
[22:04] Skaag then inject it back to all nodes via same route
[22:05] lt_schmidt_jr I see
[22:06] mikko Skaag: you went ahead with the POC?
[22:07] Skaag yes
[22:07] mikko how is it working?
[22:07] Skaag we are launching it tomorrow, hopefully
[22:07] Skaag so far so good!
[22:08] lt_schmidt_jr that is encouraging to hear
[22:11] lt_schmidt_jr Skaag, what versions are you running for zmq and jzmq?
[22:11] Skaag the very latest I guess
[22:25] pythonirc101 is it easy to use zeromq to communicate between machines behind nats?
[22:30] pythonirc101 anyone using pyzmq on win 7 here?
[22:40] traviscline \whois beppu
[22:40] traviscline whoops
[22:40] traviscline does anyone know if MinRK (benjamin) or Brian Granger hang out in here?
[22:40] traviscline looking for pyzmq authors
[22:41] mikko traviscline: they do occasionally
[22:42] traviscline cool, thanks
[22:42] mikko mr granger is bgranger
[22:42] mikko and MinRK is MinRK
[22:42] traviscline failing with some cython interop stuff and wanted to toss some Qs their way
[22:42] traviscline rgr
[23:10] traviscline pushed gevent compat with very basic cython support -- if any cython folks have input/critique please provide
[23:57] andrewvc cremes: around?