Friday November 19, 2010

[Time] NameMessage
[03:02] gancient hi all, i am trying to install python bindings for zeromq and it installs fine, but while importing i get the following error
[03:03] gancient ImportError: cannot import name initthreads
[03:03] gancient what does , build "in place" mean ?
[03:06] nettok gancient: you ran "ldconfig" after installing zeromq lib?
[03:14] gancient nettok: yes
[03:14] gancient nettok: what is "build in place" it is necessary for running the pyzeromq test suite
[03:17] fowlduck ah, a crapton of people
[03:18] fowlduck so, how are people using zeromq in the wild?
[03:18] fowlduck i'm just trying to wrap my head around how i could use it
[03:18] fowlduck particularly for using it to queue long-running jobs
[03:18] nettok I might be wrong but I think that is running "python build"
[03:19] nettok did you copied setup.cfg.template -> setup.cfg
[03:20] nettok I don't know what else could be
[03:20] gancient nettok: i just closed the terminal and started back, and the import error mysteriously goes away !
[03:21] gancient nettok: now no error with "import zmq"
[03:21] nettok gancient: :)
[03:22] gancient nettok: any idea why this might have happened ?
[04:12] nettok why there isn't a "disconnect" in zmq sockets?
[04:56] nettok How do I disconnect a socket from a single endpoint without closing the socket?
[05:13] cremes nettok_: you can't
[05:14] nettok cremes: well, that sucks
[05:20] nettok I guess i'll have to sacrifice auto load-balancing and have one socket per pusher
[07:53] travlr sustrik: morning. are you around?
[07:55] sustrik travlr: hi
[07:55] sustrik morning
[07:55] travlr well, it took me a couple more days. it's been a busy week.
[07:55] travlr but
[07:56] travlr heres the first go of it
[07:57] sustrik wow
[07:57] sustrik great
[07:57] travlr still needs some work
[07:57] sustrik can i link it from the website?
[07:57] travlr sure
[07:58] travlr this is in raw form... should i remove some of the noise like 'joins' and 'quits'?
[07:59] sustrik done
[08:00] sustrik travlr: i would keep it in such a form as to minimise work on uploading new archives
[08:00] travlr well its pretty easy actually. I'm already parsing lines.
[08:01] sustrik it's up to you then
[08:01] travlr ok then. cleaner is nicer... no?
[08:02] sustrik sure :)
[08:02] travlr next, when i find some time, is to rebuild the zmqircd
[08:03] travlr its broken being single threaded
[08:03] sustrik it tends to quit
[08:03] sustrik when commits are done
[08:03] travlr yup
[08:04] travlr i'll get to sometime soon
[08:04] travlr btw, this is a manual upload for now, because building with sphinx is in efficient... i may use something else... so...
[08:05] travlr i'll upload on sporatic occasion for now.
[08:05] sustrik sure
[08:05] sustrik understood, manual upload sucks
[08:06] travlr no.. its automated, but its not on a cron yet
[08:06] travlr it takes 5 solid minutes to build each time and 90% proc power
[08:06] travlr it's not smart enough to only build the new page
[08:08] travlr ok, gotta go for now.. have a nice day
[08:08] sustrik i see
[08:08] sustrik see you
[08:11] CIA-20 zeromq2: 03Mikko Koppanen 07master * redf7c18 10/ (acinclude.m4 (log message trimmed)
[08:11] CIA-20 zeromq2: Add a check that the compiler actually works.
[08:11] CIA-20 zeromq2: The patch tests that the C and C++ compilers actually exist
[08:11] CIA-20 zeromq2: and work. autoconf seems to default to 'g++' when C++ compiler is not
[08:11] CIA-20 zeromq2: found, which causes the following error (when the compiler isn't
[08:11] CIA-20 zeromq2: there):
[08:11] CIA-20 zeromq2: checking for uuid_generate in -luuid... no
[11:36] CIA-20 zeromq2: 03Martin Sustrik 07master * rac40680 10/ src/object.cpp :
[11:36] CIA-20 zeromq2: Problem with blob_t initialisation fixed.
[11:36] CIA-20 zeromq2: HP's version of STL doesn't allow for initialisation of basic_string
[11:36] CIA-20 zeromq2: (blob_t) using NULL pointer, while SGI's implementation is OK with
[11:36] CIA-20 zeromq2: that. Fixed.
[11:36] CIA-20 zeromq2: Signed-off-by: Martin Sustrik <> -
[22:10] nettok Is it possible to force dropping of the queued messages in an XREQ socket?
[22:12] TomHome noob question : what's the difference between rabbitmq and zeromq ?
[22:15] cremes nettok: yes, by closing the socket; all unsent messages will be dropped
[22:15] nettok cremes: thank you
[22:15] cremes TomHome: i think your best bet is to google for that answer; it's been covered a bunch by folks far more glib than i
[22:17] travlr you might also want to scan the new zeromq irc logs i put up this morning...
[22:17] travlr
[22:18] travlr cremes: i think that you mentioned that you might have some logs of this channel... do you still have them?
[22:22] cremes travlr: i do; do you want them?
[22:22] cremes if so, give me a date range
[22:23] travlr at the moment, i know i need around the last few days of June to around July 14
[22:23] travlr i'm not sure if that is the biggest gap i have or not.
[22:24] travlr there are various days and hours missing though.... i have to go through it yet
[22:25] cremes ok... the logs i have are via a mac app called colloquy and are in a weird xml format
[22:25] cremes they'll likely need conversion to be useful
[22:25] travlr that's ok.. can you upload them somewhere.. maybe github?
[22:27] cremes travlr: sure
[22:27] travlr thanks :)
[22:28] cremes i'll aim to do it tomorrow... i'm about to cut out now for the day
[22:28] travlr sure thing... no hurry..
[22:41] mikko travlr:
[22:41] mikko thats roughly from Wed Mar 31 20:29:51 2010
[22:41] mikko there might be some gaps
[22:41] travlr mikko, thank you
[22:44] doug huh, how can i tell if a message i get on a zmq socket has come from someone i've seen before?
[22:46] travlr pretty much by including something for that in your app's message layer
[22:47] doug ouch
[22:47] doug gonna make authentication a little difficult.
[22:47] travlr there has been recent discussions about authentication here and on the mail list.
[22:48] doug i read a little bit of that on the list, know when it was discussed here?
[22:48] travlr the idea is to keep zmq fundamentally light and fast
[22:48] doug always nice.
[22:48] doug i gotta client to serve, tho.
[22:49] doug figured zmq might be a great choice based on the plug that's it's used for financial messaging.
[22:49] travlr i here ya... i use it for financial too... check the irc log search engine.. see the url above
[22:50] travlr and the mail list archives are linked on the website
[22:50] doug i've already gone through hits on "authentication" from the list archives
[22:51] travlr not enough info?
[22:51] travlr check for 'security' too.
[22:52] travlr remember that zmq is not necessarily 'internet' capable in itself
[22:53] travlr security wise...
[22:55] doug yes, of course.
[22:55] doug looks like xrep sockets would at least give me identity info.
[22:56] travlr i guess. i still have to involve myself more with the intracacies
[22:57] travlr i'm about to dig into zmq deeper now
[22:58] travlr pieter did a really nice job with the guide, i guess there is enough now for me to dig in 'deeper'
[23:03] doug yup. would be nice to have some history on the specific situations zmq was evolved to solve
[23:17] doug it's sort of like a subset of mpi, without a lot of the support infrastructure
[23:17] doug which i imagine makes it easier to deal with in a lot of ways
[23:17] doug a little less capable tho
[23:19] Guthur ZMQ seems to be more the foundation about which domain specific infrastructure solutions can be built
[23:20] Guthur I feel it help avoid ZMQ trying to many things badly and not many well
[23:20] doug i'm not sure what your first sentence actually means, guther.
[23:21] Guthur i.e. some other project will build that missing infrastructure you mentioned on top of ZMQ
[23:22] Guthur and so the vision of ZMQ will not get convoluted be some application specific requirements
[23:23] doug i'd guess that zmq is already the product of application specific requirements
[23:25] Guthur really, what application would that be
[23:25] doug knowing more about its history would help be to determine if my own application will benefit greatly from what zmq can solve.
[23:26] Guthur seems quite a generic message passing abstraction to me
[23:26] Guthur allowing a decent variety of transport protocols as well
[23:26] doug i don't know the specific application, but i did read about openamq not being quite what imatix wanted or needed
[23:26] doug for some scalable financial application
[23:28] Guthur But there is nothing in the API that ties it to the financial application domain
[23:29] doug why would there be?
[23:29] doug > From 2004-2007 iMatix built a new messaging protocol (AMQP) for JPMorganChase, implemented this as a software product (OpenAMQ), and helped JPMorganChase migrate their largest investment bank trading system off a legacy middleware and onto OpenAMQ.
[23:30] doug that might be handy for me, as my client is also (part of) that same company.
[23:30] doug unless they had some terrible experience with it.
[23:31] doug this makes sense, 0mq seems geared toward homogeneous networks with large consumer counts and a need for minimal delay
[23:33] doug and smaller messages.
[23:38] Guthur I'm sort of a similar situation, part of a financial company (investment banking) and I'd like to work ZMQ into a refactor of our application
[23:39] Guthur the applications architecture is a bit pants at the moment
[23:40] doug why would zmq make sense?
[23:40] Guthur well it has a lot of subsystems which at the minute are all using HTTP as the transport mechanism
[23:40] Guthur for some bizarre reason
[23:41] doug probably not so bizarre if you dig for the rationale.
[23:41] Guthur I can understand the customer facing connection being HTTP
[23:41] Guthur HTTPS actually
[23:42] Guthur but internally I can't really see why the need for extra stack layer
[23:42] doug there's lots of existing feature-rich libraries that use http for the transport.
[23:42] doug and plenty of managers who are probably happy knowing that such a well-known and standard protocol is in use under the hood.
[23:43] Guthur wrong people being the decision about implementation if you ask me
[23:43] Guthur managers that is
[23:44] Guthur HTTP get shoehorned into doing things it was never designed to do
[23:45] doug eh, if it works well enough..
[23:47] Guthur we do quite a bit of data streaming in this application, with long polling
[23:48] Guthur and it goes through quite a few layers all using HTTP
[23:48] Guthur it just not very scalable in my opinion
[23:49] doug what's the bottleneck?
[23:50] Guthur To be honest I haven't been on the team long enough to get a complete handle on the situation
[23:50] Guthur but from discussions with more senior members it does sound like its the comms on the wire
[23:51] doug i'd find it hard to make a decision without some serious understanding of how that plays out.
[23:51] Guthur well none is being made to be honest
[23:51] Guthur I have one part of it to port from a Java JMS to C#
[23:52] Guthur and I'd like to use ZMQ there if possible
[23:52] doug if i were you, i'd worry about having my decision be viewed as bizarre by the next guy who gets stuck with the result.
[23:53] Guthur Not sure how ZMQ is bizarre
[23:53] Guthur to me using HTTP is bizarre
[23:54] doug any protocol, library, or tool choice is going to be bizarre for a large number of applications.
[23:54] doug i'm pretty sure it's not going to be a great choice for what i'm doing.
[23:54] Guthur well this particular part of the application will be getting alot of spot price data
[23:55] Guthur and publishing it
[23:55] doug that sounds closer suited to zmq than my stuff
[23:55] doug which involves finding unknown network endpoints, authenticating them, and providing them 500K-5M chunks of data.
[23:56] doug with distributed locking.
[23:56] doug and may cross political boundaries.
[23:56] Guthur You can poll plain sockets actually
[23:57] Guthur with ZMQ
[23:57] travlr doug, take a look at what i'm working on. this all started with the need for high frequency trading:
[23:57] travlr that's a tempory page. i'm starting to build a community for it now.
[23:57] doug yeah, i can poll plain sockets without zmq as well.
[23:58] Guthur doug, I think you missed my point
[23:58] Guthur If you had to cross political boundaries, I assume you meant to places where there is no ZMQ, that might facilitate it, which is what I meant
[23:59] doug sounds cool, travlr. i look forward to hearing about the first release.
[23:59] doug there'd be zmq on both ends, but there's a strong need for authentication and authorization in that case