ZeroMq IRC Log

Wednesday January 19, 2011

[Time] Name	Message
[01:41] lechon	mikko, i hope you've been able to get to sleep by now :]
[01:41] lechon	i will think about the proposal a bit more and will send out some ideas soon
[01:42] mikko	no
[01:42] mikko	still up
[01:42] mikko	looked into udt sockets a bit
[01:43] lechon	what do you think about them?
[01:43] mikko	seems like a nice idea
[01:44] mikko	the api is very friendly if you have done socket programming
[01:44] lechon	udt.sf.net?
[01:45] mikko	yes
[01:56] lechon	yeah, just like sockets
[02:12] jhawk28	https://github.com/bittorrent/libutp is another one
[05:55] traviscline	i'm writing a compat layer with pyzmq and a greenlet lib (gevent) and want to do some performance testing of a few configurations: namely: pure c, vanilla pyzmq, my pure python greened version, and a cython verision of my greened code
[05:55] traviscline	I initially set up a little harness to send 1mil random ints from a c producer and just saw how fast I consumed with the various configurations
[05:56] traviscline	I want to know if this seems like a valid test to compare the basic overhead of the alternate python options
[05:56] traviscline	I basically want to determine how much overhead is incurred by using the greened version of pyzmq (and do some work to minimize it)
[05:56] traviscline	thoughts?
[06:05] Steve-o	Similar to the pyzmq graphs previously published?
[09:14] mikko	good morning
[09:21] guido_g	morning
[10:17] kabs	Hello, I tried writing hello-world application for pub-sub system in c, but when I complile it , I get these errors http://pastebin.com/n8F3CG8c , what libs it is looking for , I have already installed zeromq-2.0.10 on my system
[10:19] mikko	-lzmq
[10:19] mikko	you are not linking against libs
[10:33] kabs	Thanks you, it worked!!
[13:44] benoitc	/j/win 18
[14:31] pythonirc1011	I'm trying to build zeromq python bindings on win 7 64-bit with 32-bit python installed and am getting in trouble. Can anyone help please?
[14:31] pythonirc1011	the compilation of libzmq.dll is fine. but "python setup.py build" looks for libzmq.lib ?
[14:32] mikko	pythonirc1011: yes
[14:32] mikko	as far as i understand windows is that you link against .libs
[14:33] mikko	.dll for runtime and .lib for linking
[14:33] pythonirc1011	mikko: the problem is that when i try to create .lib files from zeromq code, it fails on VS2010
[14:33] mikko	pythonirc1011: whats the issue?
[14:33] pythonirc1011	lots of unresolved symbols
[14:34] pythonirc1011	stuff like this -- http://paste.pocoo.org/show/323265/
[14:35] mikko	how do you create .lib files?
[14:35] mikko	shouldnt they be build by default when .dll is built?
[14:36] mikko	and can you use 64bit zeromq with 32bit python?
[14:36] mikko	pythonirc1011: that looks like you are building static libraries
[14:36] mikko	rather than dlls
[14:37] pythonirc1011	mikko: i can build dlls, no problem
[14:37] mikko	pythonirc1011: it seems that when i do the continous integration builds the dll and lib are built to same directory
[14:37] mikko	without using any special options
[14:38] pythonirc1011	but the pyzeromq, wants a .lib file --LINK : fatal error LNK1181: cannot open input file 'libzmq.lib'
[14:38] mikko	pythonirc1011: yes, the build should generate you .lib and .dll into the lib/ directory
[14:38] pythonirc1011	mikko: In my case i only see the .dll built, no lib there...I changed the configuration to Release. Thats all i changed.
[14:38] mikko	hmm
[14:39] mikko	pythonirc1011: and thats 64bit build?
[14:39] sustrik	with msvc2008 the lib was generated alongside the dll
[14:39] mikko	do you need something special with 2010?
[14:40] mikko	im building with 2008 atm
[14:40] sustrik	maybe some msvc2010 quirk?
[14:40] pythonirc1011	no idea
[14:40] mikko	do we support 64bit windows?
[14:40] sustrik	no idea
[14:40] sustrik	some people complained
[14:40] sustrik	some have noted that the 64bit build it ok
[14:40] mikko	and i dont think 64bit zeromq works with 32bit python
[14:41] sustrik	i don't habe win64
[14:41] sustrik	so i can't check myself
[14:41] pythonirc1011	mikko: can i just build 32-bit zeromq on 64-bit machines?
[14:41] mikko	i think i got win7 64bit image somewhere
[14:41] pythonirc1011	I think i did...
[14:41] mikko	pythonirc1011: sure
[14:41] pythonirc1011	lemme try again
[14:42] pythonirc1011	building now
[14:42] pythonirc1011	only .dll file there
[14:42] pythonirc1011	win32 build + release -- no .lib there
[14:43] mikko	pythonirc1011: what does it say under Properties->Configuration Properties->Linker->Advanced
[14:43] mikko	and import library there
[14:44] mikko	foudn a reference online that it might be msvc2008 -> 2010 project conversion issue
[14:44] pythonirc1011	Import library there is void
[14:44] pythonirc1011	i can add something there...if you need me to
[14:45] mikko	libzmq.lib
[14:45] mikko	is what it should output
[14:46] pythonirc1011	mikko: http://paste.pocoo.org/show/323272/ -- no .lib file still
[14:52] mikko	pythonirc1011: see the error message
[14:52] pythonirc1011	ah! $(OutDir), $(TargetName) and $(TargetExt) match?
[14:52] mikko	maybe $(TargetName).lib
[14:52] mikko	as the import library name
[14:54] pythonirc1011	still trying to find where they are ...
[14:55] mikko	i dont have msvc2010 at hand here
[14:56] pythonirc1011	got it
[14:57] pythonirc1011	General->Taget Ext -> .lib (from .dll)
[14:57] pythonirc1011	that works
[14:57] mikko	ahmm
[14:57] mikko	did you rename the .dll to lib?
[14:57] pythonirc1011	i tried that...that didnt work
[14:57] pythonirc1011	pyzmq complained
[14:58] mikko	yes, because these are two different things
[14:58] mikko	import library and dll
[14:58] pythonirc1011	>>> import zmq --> Traceback (most recent call last): File "<stdin>", line 1, in <module> File "zmq\__init__.py", line 26, in <module> from zmq.utils import initthreads # initialize threads ImportError: cannot import name initthreads
[14:58] mikko	pythonirc1011: http://msdn.microsoft.com/en-us/library/9yd93633%28v=vs.80%29.aspx
[14:58] pythonirc1011	could compile and install pyzmq, but now when i try to import, get this
[14:58] mikko	hmm
[14:59] mikko	did you download pyzmq matching 2.0.10 version?
[14:59] pythonirc1011	yes indeed
[14:59] pythonirc1011	zeromq-2.0.10 == pyzmq-2.0.10
[14:59] mikko	it tries to import something called "initthreads" ?
[15:00] mikko	from zmq.utils
[15:00] pythonirc1011	i did a python setup.py build_ext and python setup.py install...anything wrong there?
[15:00] mikko	pythonirc1011: not that i know of. however i dont really use python or windows
[15:01] mikko	the process we run for pyzmq on linux is
[15:01] mikko	python setup.py cython
[15:01] mikko	python setup.py build_ext
[15:02] mikko	python setup.py test
[15:34] traviscline	any instructions on running the pyzmq perf tests?
[15:36] mikko	none
[15:40] traviscline	but if anyone has a 'how to run' lemme know
[15:43] pythonirc1011	anyone using pyzeromq on win 7 + VS2010 + python 32-bit here?
[15:53] traviscline	figured it out
[20:09] lt_schmidt_jr	Hello
[20:10] lt_schmidt_jr	Looking for someone to shoot holes in an approach to implement pub/sub for web clients
[20:11] s0undt3ch	ppl, with 2.1.0, on the call socket.recv_pyobj() I'm getting:
[20:11] s0undt3ch	File "socket.pyx", line 657, in zmq.core.socket.Socket.recv_pyobj (zmq/core/socket.c:5470)
[20:11] s0undt3ch	File "socket.pyx", line 94, in zmq.core.socket.Socket.__cinit__ (zmq/core/socket.c:1036)
[20:11] s0undt3ch	TypeError: __cinit__() takes exactly 2 positional arguments (0 given)
[20:11] s0undt3ch	why is this!?
[20:15] traviscline	s0undt3ch: up to date with cython?
[20:15] s0undt3ch	traviscline: think so
[20:15] s0undt3ch	traviscline: at least, up to date with my distro's cython
[20:15] traviscline	built with 0.14 fine
[20:15] s0undt3ch	traviscline: and on another piece of code, it works good
[20:19] Skaag	lt_schmidt_jr: shoo
[20:19] Skaag	shoot
[20:40] lt_schmidt_jr	Thanks Skaag
[20:40] Skaag	lt_schmidt_jr: We actually are looking to implement the same, sort of a clone of pubnub.com
[20:41] lt_schmidt_jr	not familiar with that, but ours is domain specific to allow multiple clients to manipulate graph data
[20:44] lt_schmidt_jr	but the basic idea is to use Jetty Websocket (with Flash Socket when necessary) and have the instance specify what it is working on throgh the socket, establishing a pub and sub zeromq socket for each client and hooking it up using devices depending on deployment
[20:44] lt_schmidt_jr	single machine - no devices
[20:44] lt_schmidt_jr	using ICP (if I remember correctly)
[20:44] lt_schmidt_jr	sorry single machine - local device with interproc
[20:45] lt_schmidt_jr	multiple machines - somehow put the devices together
[20:45] lt_schmidt_jr	not sure how though
[20:45] lt_schmidt_jr	does this make any sense?
[20:46] lt_schmidt_jr	On my earlier comment about no devices - I don't think that is possible with many pubs and subs
[20:47] lt_schmidt_jr	unless they can discover each other somehow
[20:48] lt_schmidt_jr	I am thinking about using zookeeper perhaps at a later time
[20:48] Skaag	ok, we've been down the same road
[20:48] Skaag	at the moment, zookeeper is good for a fixed (unchanging) number of nodes
[20:49] Skaag	where it can help you, is in the notification on services that go up/down on those nodes
[20:49] Skaag	and in leader election, among the fixed set of nodes
[20:52] lt_schmidt_jr	Our number of nodes is fairly low (in terms of what we want to subscribe to)
[20:53] lt_schmidt_jr	we are thinking of it more like channels vs individual components
[20:54] lt_schmidt_jr	otherwise the client has to mirror whole state to the server
[20:54] lt_schmidt_jr	Skaag are you using a similar Java stack?
[20:55] Skaag	I'm not sure I understand how you use the term 'component'
[20:56] lt_schmidt_jr	leaf nodes , which are many - we are looking to route fairly close to the base of the tree - if that makes sense
[20:56] Skaag	also I recommend you make a spreadsheet to calculate data packet sizes, to avoid bottlenecks, and to properly shape your data structures/sizes, it helps a lot to manage expectations
[20:56] Skaag	(unrelated to the issue but important)
[20:56] Skaag	yes understood
[20:57] lt_schmidt_jr	I see
[20:57] lt_schmidt_jr	so you are currently using zookeeper?
[20:57] Skaag	not anymore
[20:57] Skaag	I installed it, tried it, and found out it does not know how to add new nodes to the cluster
[20:58] Skaag	the list of nodes is hardcoded in a config file
[20:58] Skaag	quite annoying, and kinda defeating the purpose in my opinion
[20:58] lt_schmidt_jr	you are talking about Zookeeper nodes?
[20:58] Skaag	well in our case the idea was that machines would suddenly show up in a cluster
[20:59] Skaag	now I have some idea about using some sort of gnutella variant :-)
[20:59] Skaag	decentralized, etc.
[20:59] lt_schmidt_jr	interesting
[21:00] lt_schmidt_jr	so do you have a similar topology with many publishers and subscribers on the same channels?
[21:00] Skaag	yes
[21:00] Skaag	at the moment, we have a POC cluster of 16 nodes
[21:00] Skaag	it's tiny
[21:00] Skaag	we used Rabbitmq-server at first
[21:00] Skaag	but I was told by one of their team members that 16 nodes is considered a large cluster for them
[21:00] lt_schmidt_jr	I see
[21:00] Skaag	and since we're looking to scale to thousands of nodes, we had to abandon it
[21:01] lt_schmidt_jr	I was looking at RabbitMq and XMPP pubsub initially
[21:01] lt_schmidt_jr	as well
[21:01] Skaag	same here
[21:01] lt_schmidt_jr	how are you bridging pubsub between nodes?
[21:01] Skaag	this is at the moment done naively
[21:02] lt_schmidt_jr	are you also using a forwarder on each machine
[21:02] lt_schmidt_jr	?
[21:02] Skaag	all nodes are publishing to all nodes with zmq
[21:02] Skaag	yes with forwarders
[21:02] lt_schmidt_jr	this is too funny
[21:03] lt_schmidt_jr	so 2 forwarders per machine?
[21:03] lt_schmidt_jr	one in each direction?
[21:03] Skaag	I need to check with my team member who implemented that part
[21:03] Skaag	(we're a tiny team)
[21:04] lt_schmidt_jr	I am a team of 1 right now, building a small POC
[21:04] lt_schmidt_jr	are you also using jetty - or are you non-http system?
[21:07] Skaag	yes we plan to use jetty
[21:08] Skaag	since we are 99% java based
[21:08] lt_schmidt_jr	Have you looked at Atmosphere?
[21:08] lt_schmidt_jr	its attractive, but I am not sure that frameworks fits well with zmq
[21:29] Skaag	no I haven't
[21:29] Skaag	checking it out
[21:37] mikko	good evening
[21:37] lt_schmidt_jr	Hello
[21:37] mikko	sustrik_: there?
[22:03] lt_schmidt_jr	Skaag, what is your application doing, if you don't mind me asking?
[22:04] Skaag	the system that uses zmq is intended to send/collect statistics information and aggregate them into a single body of statistical data
[22:04] Skaag	then inject it back to all nodes via same route
[22:05] lt_schmidt_jr	I see
[22:06] mikko	Skaag: you went ahead with the POC?
[22:07] Skaag	yes
[22:07] mikko	how is it working?
[22:07] Skaag	we are launching it tomorrow, hopefully
[22:07] Skaag	so far so good!
[22:08] lt_schmidt_jr	that is encouraging to hear
[22:11] lt_schmidt_jr	Skaag, what versions are you running for zmq and jzmq?
[22:11] Skaag	the very latest I guess
[22:25] pythonirc101	is it easy to use zeromq to communicate between machines behind nats?
[22:30] pythonirc101	anyone using pyzmq on win 7 here?
[22:40] traviscline	\whois beppu
[22:40] traviscline	whoops
[22:40] traviscline	does anyone know if MinRK (benjamin) or Brian Granger hang out in here?
[22:40] traviscline	looking for pyzmq authors
[22:41] mikko	traviscline: they do occasionally
[22:42] traviscline	cool, thanks
[22:42] mikko	mr granger is bgranger
[22:42] mikko	and MinRK is MinRK
[22:42] traviscline	failing with some cython interop stuff and wanted to toss some Qs their way
[22:42] traviscline	rgr
[23:10] traviscline	pushed gevent compat with very basic cython support -- if any cython folks have input/critique please provide https://github.com/traviscline/gevent-zeromq
[23:57] andrewvc	cremes: around?