ZeroMq IRC Log

Friday November 19, 2010

[Time] Name	Message
[03:02] gancient	hi all, i am trying to install python bindings for zeromq and it installs fine, but while importing i get the following error
[03:03] gancient	ImportError: cannot import name initthreads
[03:03] gancient	what does , build "in place" mean ?
[03:06] nettok	gancient: you ran "ldconfig" after installing zeromq lib?
[03:14] gancient	nettok: yes
[03:14] gancient	nettok: what is "build in place" it is necessary for running the pyzeromq test suite
[03:17] fowlduck	ah, a crapton of people
[03:18] fowlduck	so, how are people using zeromq in the wild?
[03:18] fowlduck	i'm just trying to wrap my head around how i could use it
[03:18] fowlduck	particularly for using it to queue long-running jobs
[03:18] nettok	I might be wrong but I think that is running "python setup.py build"
[03:19] nettok	did you copied setup.cfg.template -> setup.cfg
[03:20] nettok	I don't know what else could be
[03:20] gancient	nettok: i just closed the terminal and started back, and the import error mysteriously goes away !
[03:21] gancient	nettok: now no error with "import zmq"
[03:21] nettok	gancient: :)
[03:22] gancient	nettok: any idea why this might have happened ?
[04:12] nettok	why there isn't a "disconnect" in zmq sockets?
[04:56] nettok	How do I disconnect a socket from a single endpoint without closing the socket?
[05:13] cremes	nettok_: you can't
[05:14] nettok	cremes: well, that sucks
[05:20] nettok	I guess i'll have to sacrifice auto load-balancing and have one socket per pusher
[07:53] travlr	sustrik: morning. are you around?
[07:55] sustrik	travlr: hi
[07:55] sustrik	morning
[07:55] travlr	well, it took me a couple more days. it's been a busy week.
[07:55] travlr	but
[07:56] travlr	heres the first go of it http://travlr.github.com/zmqirclog/
[07:57] sustrik	wow
[07:57] sustrik	great
[07:57] travlr	still needs some work
[07:57] sustrik	can i link it from the website?
[07:57] travlr	sure
[07:58] travlr	this is in raw form... should i remove some of the noise like 'joins' and 'quits'?
[07:59] sustrik	done
[08:00] sustrik	travlr: i would keep it in such a form as to minimise work on uploading new archives
[08:00] travlr	well its pretty easy actually. I'm already parsing lines.
[08:01] sustrik	it's up to you then
[08:01] travlr	ok then. cleaner is nicer... no?
[08:02] sustrik	sure :)
[08:02] travlr	next, when i find some time, is to rebuild the zmqircd
[08:03] travlr	its broken being single threaded
[08:03] sustrik	it tends to quit
[08:03] sustrik	when commits are done
[08:03] travlr	yup
[08:04] travlr	i'll get to sometime soon
[08:04] travlr	btw, this is a manual upload for now, because building with sphinx is in efficient... i may use something else... so...
[08:05] travlr	i'll upload on sporatic occasion for now.
[08:05] sustrik	sure
[08:05] sustrik	understood, manual upload sucks
[08:06] travlr	no.. its automated, but its not on a cron yet
[08:06] travlr	it takes 5 solid minutes to build each time and 90% proc power
[08:06] travlr	it's not smart enough to only build the new page
[08:08] travlr	ok, gotta go for now.. have a nice day
[08:08] sustrik	i see
[08:08] sustrik	see you
[08:11] CIA-20	zeromq2: 03Mikko Koppanen 07master * redf7c18 10/ (acinclude.m4 configure.in): (log message trimmed)
[08:11] CIA-20	zeromq2: Add a check that the compiler actually works.
[08:11] CIA-20	zeromq2: The patch tests that the C and C++ compilers actually exist
[08:11] CIA-20	zeromq2: and work. autoconf seems to default to 'g++' when C++ compiler is not
[08:11] CIA-20	zeromq2: found, which causes the following error (when the compiler isn't
[08:11] CIA-20	zeromq2: there):
[08:11] CIA-20	zeromq2: checking for uuid_generate in -luuid... no
[11:36] CIA-20	zeromq2: 03Martin Sustrik 07master * rac40680 10/ src/object.cpp :
[11:36] CIA-20	zeromq2: Problem with blob_t initialisation fixed.
[11:36] CIA-20	zeromq2: HP's version of STL doesn't allow for initialisation of basic_string
[11:36] CIA-20	zeromq2: (blob_t) using NULL pointer, while SGI's implementation is OK with
[11:36] CIA-20	zeromq2: that. Fixed.
[11:36] CIA-20	zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/d5OV4Y
[22:10] nettok	Is it possible to force dropping of the queued messages in an XREQ socket?
[22:12] TomHome	noob question : what's the difference between rabbitmq and zeromq ?
[22:15] cremes	nettok: yes, by closing the socket; all unsent messages will be dropped
[22:15] nettok	cremes: thank you
[22:15] cremes	TomHome: i think your best bet is to google for that answer; it's been covered a bunch by folks far more glib than i
[22:17] travlr	you might also want to scan the new zeromq irc logs i put up this morning...
[22:17] travlr	http://travlr.github.com/zmqirclog/
[22:18] travlr	cremes: i think that you mentioned that you might have some logs of this channel... do you still have them?
[22:22] cremes	travlr: i do; do you want them?
[22:22] cremes	if so, give me a date range
[22:23] travlr	at the moment, i know i need around the last few days of June to around July 14
[22:23] travlr	i'm not sure if that is the biggest gap i have or not.
[22:24] travlr	there are various days and hours missing though.... i have to go through it yet
[22:25] cremes	ok... the logs i have are via a mac app called colloquy and are in a weird xml format
[22:25] cremes	they'll likely need conversion to be useful
[22:25] travlr	that's ok.. can you upload them somewhere.. maybe github?
[22:27] cremes	travlr: sure
[22:27] travlr	thanks :)
[22:28] cremes	i'll aim to do it tomorrow... i'm about to cut out now for the day
[22:28] travlr	sure thing... no hurry..
[22:41] mikko	travlr: http://valokuva.org/~mikko/zeromq.log
[22:41] mikko	thats roughly from Wed Mar 31 20:29:51 2010
[22:41] mikko	there might be some gaps
[22:41] travlr	mikko, thank you
[22:44] doug	huh, how can i tell if a message i get on a zmq socket has come from someone i've seen before?
[22:46] travlr	pretty much by including something for that in your app's message layer
[22:47] doug	ouch
[22:47] doug	gonna make authentication a little difficult.
[22:47] travlr	there has been recent discussions about authentication here and on the mail list.
[22:48] doug	i read a little bit of that on the list, know when it was discussed here?
[22:48] travlr	the idea is to keep zmq fundamentally light and fast
[22:48] doug	always nice.
[22:48] doug	i gotta client to serve, tho.
[22:49] doug	figured zmq might be a great choice based on the plug that's it's used for financial messaging.
[22:49] travlr	i here ya... i use it for financial too... check the irc log search engine.. see the url above
[22:50] travlr	and the mail list archives are linked on the website
[22:50] doug	i've already gone through hits on "authentication" from the list archives
[22:51] travlr	not enough info?
[22:51] travlr	check for 'security' too.
[22:52] travlr	remember that zmq is not necessarily 'internet' capable in itself
[22:53] travlr	security wise...
[22:55] doug	yes, of course.
[22:55] doug	looks like xrep sockets would at least give me identity info.
[22:56] travlr	i guess. i still have to involve myself more with the intracacies
[22:57] travlr	i'm about to dig into zmq deeper now
[22:58] travlr	pieter did a really nice job with the guide, i guess there is enough now for me to dig in 'deeper'
[23:03] doug	yup. would be nice to have some history on the specific situations zmq was evolved to solve
[23:17] doug	it's sort of like a subset of mpi, without a lot of the support infrastructure
[23:17] doug	which i imagine makes it easier to deal with in a lot of ways
[23:17] doug	a little less capable tho
[23:19] Guthur	ZMQ seems to be more the foundation about which domain specific infrastructure solutions can be built
[23:20] Guthur	I feel it help avoid ZMQ trying to many things badly and not many well
[23:20] doug	i'm not sure what your first sentence actually means, guther.
[23:21] Guthur	i.e. some other project will build that missing infrastructure you mentioned on top of ZMQ
[23:22] Guthur	and so the vision of ZMQ will not get convoluted be some application specific requirements
[23:23] doug	i'd guess that zmq is already the product of application specific requirements
[23:25] Guthur	really, what application would that be
[23:25] doug	knowing more about its history would help be to determine if my own application will benefit greatly from what zmq can solve.
[23:26] Guthur	seems quite a generic message passing abstraction to me
[23:26] Guthur	allowing a decent variety of transport protocols as well
[23:26] doug	i don't know the specific application, but i did read about openamq not being quite what imatix wanted or needed
[23:26] doug	for some scalable financial application
[23:28] Guthur	But there is nothing in the API that ties it to the financial application domain
[23:29] doug	why would there be?
[23:29] doug	> From 2004-2007 iMatix built a new messaging protocol (AMQP) for JPMorganChase, implemented this as a software product (OpenAMQ), and helped JPMorganChase migrate their largest investment bank trading system off a legacy middleware and onto OpenAMQ.
[23:30] doug	that might be handy for me, as my client is also (part of) that same company.
[23:30] doug	unless they had some terrible experience with it.
[23:31] doug	this makes sense, 0mq seems geared toward homogeneous networks with large consumer counts and a need for minimal delay
[23:33] doug	and smaller messages.
[23:38] Guthur	I'm sort of a similar situation, part of a financial company (investment banking) and I'd like to work ZMQ into a refactor of our application
[23:39] Guthur	the applications architecture is a bit pants at the moment
[23:40] doug	why would zmq make sense?
[23:40] Guthur	well it has a lot of subsystems which at the minute are all using HTTP as the transport mechanism
[23:40] Guthur	for some bizarre reason
[23:41] doug	probably not so bizarre if you dig for the rationale.
[23:41] Guthur	I can understand the customer facing connection being HTTP
[23:41] Guthur	HTTPS actually
[23:42] Guthur	but internally I can't really see why the need for extra stack layer
[23:42] doug	there's lots of existing feature-rich libraries that use http for the transport.
[23:42] doug	and plenty of managers who are probably happy knowing that such a well-known and standard protocol is in use under the hood.
[23:43] Guthur	wrong people being the decision about implementation if you ask me
[23:43] Guthur	managers that is
[23:44] Guthur	HTTP get shoehorned into doing things it was never designed to do
[23:45] doug	eh, if it works well enough..
[23:47] Guthur	we do quite a bit of data streaming in this application, with long polling
[23:48] Guthur	and it goes through quite a few layers all using HTTP
[23:48] Guthur	it just not very scalable in my opinion
[23:49] doug	what's the bottleneck?
[23:50] Guthur	To be honest I haven't been on the team long enough to get a complete handle on the situation
[23:50] Guthur	but from discussions with more senior members it does sound like its the comms on the wire
[23:51] doug	i'd find it hard to make a decision without some serious understanding of how that plays out.
[23:51] Guthur	well none is being made to be honest
[23:51] Guthur	I have one part of it to port from a Java JMS to C#
[23:52] Guthur	and I'd like to use ZMQ there if possible
[23:52] doug	if i were you, i'd worry about having my decision be viewed as bizarre by the next guy who gets stuck with the result.
[23:53] Guthur	Not sure how ZMQ is bizarre
[23:53] Guthur	to me using HTTP is bizarre
[23:54] doug	any protocol, library, or tool choice is going to be bizarre for a large number of applications.
[23:54] doug	i'm pretty sure it's not going to be a great choice for what i'm doing.
[23:54] Guthur	well this particular part of the application will be getting alot of spot price data
[23:55] Guthur	and publishing it
[23:55] doug	that sounds closer suited to zmq than my stuff
[23:55] doug	which involves finding unknown network endpoints, authenticating them, and providing them 500K-5M chunks of data.
[23:56] doug	with distributed locking.
[23:56] doug	and may cross political boundaries.
[23:56] Guthur	You can poll plain sockets actually
[23:57] Guthur	with ZMQ
[23:57] travlr	doug, take a look at what i'm working on. this all started with the need for high frequency trading: http://prodatalab.wikidot.com/
[23:57] travlr	that's a tempory page. i'm starting to build a community for it now.
[23:57] doug	yeah, i can poll plain sockets without zmq as well.
[23:58] Guthur	doug, I think you missed my point
[23:58] Guthur	If you had to cross political boundaries, I assume you meant to places where there is no ZMQ, that might facilitate it, which is what I meant
[23:59] doug	sounds cool, travlr. i look forward to hearing about the first release.
[23:59] doug	there'd be zmq on both ends, but there's a strong need for authentication and authorization in that case