Tuesday January 25, 2011

[Time] NameMessage
[02:05] oaror Hi there!
[02:05] oaror I am having trouble setting up pyzmq on windows 7
[02:06] oaror it fails with the error : "Setup script exited with the error: \Mozilla was unexpected at this time"
[02:06] oaror This happens with easy_install
[02:08] oaror Also downloaded pyzmq-2.0.10 and tried executing "python install" after building libzmq.dll with VS 2008
[02:08] oaror same result
[02:08] oaror any ideas how I can fix this?
[02:14] oaror In particular this procedure has not worked for me :-
[02:15] oaror oops.
[02:15] oaror that procedure did now work for me.
[02:16] oaror I have it running on my Ubuntu desktop no issues. However we use windows 7 at the office - sigh.
[05:29] stockMQ Hello!
[05:30] stockMQ Though this does not really come under the scope of zeromq but am sure most who have used it would have tackled it and i am looking for some suggestions
[05:30] stockMQ I have a PUB-SUB pattern
[05:31] stockMQ On the Subscriber end when the stock feed is received currently I write it as binary to a file
[08:43] stockMQ Hi
[08:43] stockMQ In a PUB-SUB pattern
[08:43] stockMQ i want to save the feed recvd at SUB side
[08:44] stockMQ what is advisable
[08:44] stockMQ saving it in file as binary ?
[08:46] stockMQ I am sure many have dealt with this
[08:46] stockMQ please share your experiences
[08:50] sustrik stockMQ: there's no single answer
[08:50] sustrik depends on what do you want to do with the file
[08:51] stockMQ basically there will be other analysis applications which will be reading from the file
[08:51] stockMQ this could happen during market open thus could create a race condition
[08:51] stockMQ and also after market close when the write is not happening
[08:52] stockMQ i want to save all the intra day data
[08:58] stockMQ any suggestions?
[09:20] mikko stockMQ: how many days are you storing?
[09:20] mikko and how many messages per second are you handling (roughly)
[09:20] stockMQ every day a new file will be created
[09:20] mikko what is the size of a single message?
[09:20] stockMQ I am handling around 1 message every 20ms
[09:21] stockMQ so 50messages per second
[09:21] mikko so not a huge amount but decent flow
[09:22] stockMQ and each message say about 256B
[09:23] mikko and each day is a separate file?
[09:23] stockMQ for now yes
[09:23] stockMQ but i also have a separate file for certain scrips
[09:24] stockMQ In that case each scrip will have its own file and each file could have data across days
[09:24] mikko maybe use some sort of database?
[09:24] stockMQ so i would like to design a solution which will be scalable as far as the size is concerned
[09:24] stockMQ actually the issue is
[09:25] stockMQ i cannot expect the Subscribers to setup and configure a DB
[09:25] stockMQ they are lay men brokers
[09:25] stockMQ not IT educated
[09:26] mikko stockMQ: it doesn't have to be a centralised db
[09:27] stockMQ it cannot be a centralized db..
[09:27] mikko bdb, kyoto cabinet, firebird etc
[09:27] stockMQ The publisher publishes the feed
[09:27] mikko there is a large amount of embedded databases available
[09:27] stockMQ okk
[09:28] stockMQ how are they in handling real time fast data
[09:28] stockMQ in your experience
[09:29] mikko different kind of databases have different models for handling concurrency and consistency
[09:29] mikko i am testing kyoto cabinet at the moment and it provides different levels of consistency
[09:30] stockMQ okk
[09:30] mikko you can execute transactions, force sync to disk etc
[09:30] stockMQ okk
[09:31] stockMQ apart from db do you think it is possible to handle race condition in file storage
[09:31] stockMQ The database is an option
[09:31] mikko what sort of race conditions?
[09:31] stockMQ The issue is
[09:31] stockMQ when the feed is being written into a file at SUB end
[09:32] stockMQ there could be Technical analysis tools reading data from the same file simultaneously
[09:32] stockMQ though the OS should handle synchronization
[09:32] stockMQ but i suspect there will be a case since the feed is being recvd so fast that
[09:33] stockMQ the write will be given more time slice and the read may have to wait
[09:33] stockMQ Database is an option..but i think the TA tools are designed to take .dat files as input
[09:34] mikko can't really comment on that side as i don't really know the tools you are talking about
[09:35] stockMQ That is right.. in fact just think that when the file is being written, there are another applications reading/wanting to read from it
[09:36] stockMQ A DB would handle concurrency.. But any experience with file storage
[10:36] sustrik you would have to ensure consistency yourself
[10:36] sustrik which is kind of like writing a little database engine of your own
[10:36] sustrik it's definitely better to use an existing product
[10:37] sustrik ha, Vint Cerf talks at LCA: "We currently have no equivalent to the Erlang distribution to describe usage at the edges of the network, making provisioning and scaling difficult."
[10:37] sustrik actually, we do :)
[10:38] benoitc erlang OTP for supervision and such + zeromq make a good couple
[11:34] pieterhintjens hi guys
[11:36] mikko pieterh!
[11:36] pieterh sustrik: where can I find the summary of changes between 2.0.10 and 2.1.0?
[11:36] mikko been a while
[11:36] pieterh hi mikko :-)
[11:36] pieterh yeah, I was a bit sick
[11:36] mikko i had a question about zfl
[11:36] pieterh shoot
[11:36] mikko is it supposed to work on windows?
[11:37] sustrik pieterh: check the ChangeLog
[11:37] pieterh yes, it's supposed to be a portability layer
[11:37] sustrik in the main directory
[11:37] mikko i noticed the use of pthreads while building on mingw32
[11:37] pieterh sustrik: thanks
[11:37] pieterh but like all such layers, until it's actually used on an OS it won't work on the OS
[11:38] pieterh the idea is to put stuff like #ifdef _WIN32_ into one layer rather than sprinkled through apps
[11:38] pieterh at least for the basic functionality a typical 0MQ app needs...
[11:39] mikko cool
[11:39] pieterh I should add something about porting ZFL to the docs, maybe
[11:39] pieterh we did this for years with SFL, an older C library, the approach works nicely
[11:39] mikko kinda the same that apache httpd does with apr
[11:40] mikko i'll take a look at providing a patch for mingw32 etc
[11:42] pieterh yes, APR is very similar in concept but (a) doesn't use a class model, which makes the code messy, and (b) is rather heavy
[11:42] pieterh patches will be welcome :-)
[11:48] drbobbeaty sustrik: you around? Did you get a look at the gist with the issue re: Ubuntu 10.04.1 and OpenPGM failing on the connect()?
[11:49] sustrik the gist was dead in the morning
[11:49] sustrik let me have a look now
[11:50] sustrik drbobbeaty: can you please try using different port numbers for individual connections?
[11:50] drbobbeaty sustrik: Sure, let me generate a new gist with the minimal code.
[11:51] sustrik i mean, isn't the problem caused by connecting twice to the same port number?
[11:51] drbobbeaty Nope, it's just plain connecting. On Ubuntu only. CentOS is fine.
[11:54] drbobbeaty This is the minimal code sample: -- it jsut creates the context, creates the socket, and then calls connect(). It works great on CentOS 5 but fails on Ubuntu 10.04.1.
[11:55] sustrik have you tried using different port numbers?
[11:55] sustrik ah, ok
[11:55] sustrik i'll check it out shortly
[11:55] drbobbeaty thanks so much - any help you can provide would be greatly appreciated.
[12:04] pieterh sustrik: there is no ChangeLog in the git repository
[12:05] pieterh however there is a summary of changes in NEWS
[12:05] sustrik right
[12:05] sustrik ChangeLog is auto generated
[12:06] sustrik you have to look into the package
[12:06] sustrik if you want complete changelog
[12:06] pieterh right, it's not in the git even after building
[12:06] sustrik generating ChangeLog is part of the release proces
[12:06] sustrik so have a look in 2.1.0 package
[12:07] pieterh it's not documented on the wiki
[12:07] pieterh there is a link to NEWS, which is rather more readable
[12:07] sustrik is it sufficient for you?
[12:08] pieterh I'll fix the wiki to explicitly answer the question, "what's changed in this version"
[12:08] mikko "a lot"
[12:08] sustrik if you need a comlete change log from the head version just use "git log"
[12:09] pieterh a lot changed, but most users just want the headlines
[12:09] pieterh a git log is close to useless for that
[12:09] pieterh my question, for example, is "what do I need to change in the Guide"
[12:09] pieterh other people will want to know, "what do I need to change in my apps, and what cool new stuff is there?"
[12:10] pieterh NEWS looks like the best source, hand written and clear
[12:14] sustrik yes, i think so
[12:31] kabs Hello, in most of the zeroMQ examples server binds to for example : tcp://*:5560 while client connects to for example tcp://localhost:5560, if I replace * with localhost for server it doesn't work , didn't understand why , any one??
[12:32] traviscline I'm working on a libevent zeromq integration and have a question (that might require libevent knowledge): libevent is optionally edge-trigger aware (EV_ET) on some backends (kqueue, epoll) -- is this feature desirable/useful in conjunction with a zmq socket? read events seem to be triggered fine without the EV_ET flag -- as of now i'm handling the "edge-triggered" nature of the zmq.FD by setting a readable and writable flag until respective EAGAIN
[12:35] kabs Forgot to mention, examples I am talking about is from the guide and my client servers are on same machine
[12:36] traviscline mikko: you had some great input for me before, ^ that rambling make any sense?
[12:36] traviscline i'm getting my feet wet with this stuff so I may be missing/misunderstanding concepts
[12:37] mikko traviscline: not sure i understand what the question is
[12:38] mikko you dont have to use edge-triggered event loop with zeromq but you need to handle the zeromq fd in edge-triggered fashion
[12:38] traviscline mikko: if libevent's support of edge-triggered events makes sense in the context of ZMQ_FD descriptors
[12:38] traviscline ok that's what I thought
[12:38] traviscline thank you
[12:38] traviscline I think my implementation is correct then
[12:39] mikko i haven't seen any problems with level-triggered loop this far
[12:39] mikko doing a loop in callback and reading until EAGAIN
[12:39] mikko or checking ZMQ_EVENTS
[12:41] traviscline what i'm doing is waiting on an internal read (and write) event and then when ZMQ_FD is readable checking EVENTS to see which internal events to set and unsetting them on EAGAIN and if after a read|write EVENTS no longer shows the read|write as available
[12:41] traviscline maybe I could simplify it a bit
[12:59] stockMQ Anyone used HDF5
[13:03] sustrik kabs: you are binding to a network interface
[13:04] kabs sustrik: yes
[13:04] sustrik so, on linux, the loopback interface is called "lo"
[13:04] sustrik "localhost" is a hostname
[13:04] kabs sustrik: even if I change both client and server to lo , it doesn't work
[13:04] sustrik connect connects to host
[13:04] kabs sustrik: I tried with ip address, it worked,
[13:05] sustrik so it's "lo" on bind side
[13:05] sustrik and "localhost" on connect side
[13:05] kabs sustrik: hmm ...
[13:06] kabs sustrik: got your point, thanks!
[13:06] sustrik np
[13:07] pythonirc101 mikko: I've two machines behind NATs and I want to do publish/subs for them (mutual). Is this something that can be done using zeromq?
[13:13] drbobbeaty sustrik: I think I have a lead to the problem with OpenPGM... It appears in socket_base.cpp (line 195) that in order to use the epgm protocol I need to NOT have options.requies_in and options.requires_out set to TRUE. Yet, in the constructor in sub.cpp (line 27), the superclass constructor - xsub_t (line 33 in xsub.cpp) is setting them BOTH to be TRUE. With the subclass, sub_t not chancing these values, it appears that they are going to be TRUE for all
[13:13] drbobbeaty sockets.
[13:13] sustrik yes, that's the recent change i was referring to
[13:14] drbobbeaty Ah... OK, perfect. So I'll just wait for your resolution. Thanks.
[13:25] spht pythonirc101: without a proxy in between you need to look at UPnP
[13:29] pythonirc101 spht: I could use a proxy for connection establishment, but not for data transfer...
[13:31] spht pythonirc101: Except for manual NAT config this is your only option:
[13:32] spht though note that it's not implemented by all vendors/devices
[15:32] traviscline ANN: much improved gevent-zeromq compatibility pushed at
[15:53] traviscline seb`: cool, more than happy to work out any kinks but feel much more confident with the recent changes
[15:53] traviscline seb`: I wonder if the zguide guys would take some python-gevent submissions
[15:54] traviscline thought part of the point is the ease in going multi-proc, it's nice seeing lots going on in one process smoothly
[15:58] pieterh traviscline: anything you want to submit to zguide, more than welcome!
[15:59] pieterh so you have a specific API for python-gevent?
[15:59] traviscline pieterh: trouble is the single threaded examples are the same sans a different import
[15:59] traviscline when you go multi-threaded it'd change
[15:59] traviscline pieterh: they may not be different enough to be that useful but i'll compare
[16:00] traviscline pieterh: it's meant to be the pyzmq api but it replaces parts that would block and gets the FD into libevent's loop
[16:00] pieterh do you expect this to become the standard pyzmq eventually?
[16:01] traviscline I've been talking to the guys while working on this but it's possible it'd end up in some sort of 'green' submodule somewhere
[16:01] traviscline haven't had that discussion in detail
[16:02] pieterh you consider this a fork / improvement of pyzmq, right?
[16:02] traviscline it's not uncommon in gevent land to provide compat libs which mirror and/or patch the apis of other libs
[16:03] traviscline i don't bundle pyzmq, i depend on it and wrap it
[16:03] traviscline but as I said, it could change, depends on what hte pyzmq guys think
[16:03] pieterh ok...
[16:03] pieterh what I'd suggest is translate the interesting examples to python-gevent-zmq
[16:04] pieterh package them at least with your code
[16:04] traviscline pieterh: if you're interested in the gevent bits: here's all the rr* stuff in one file
[16:04] pieterh and then we'll also package them with zguide
[16:04] traviscline seb`: agreed
[16:04] traviscline pieterh: cool, can do
[16:04] traviscline pieterh: whoops, link:
[16:04] pieterh send me a zip with everything when it's ready
[16:04] pieterh
[16:04] traviscline rgr
[16:04] pieterh or better, to zeromq-dev so it's public
[16:05] pieterh i'll add Python-gevent as a 'Language'
[16:06] pieterh we have variants of other language APIs too, and have solved this simply by adding the examples to the relevant directory
[16:06] traviscline hmm
[16:06] pieterh they don't show up when people read the guide online but they are in the git if they clone it
[16:06] traviscline or the like?
[16:07] pieterh yes
[16:07] pieterh at the least we archive and publish the examples
[16:07] pieterh i may add 'variants' to the languages if we get this happening more
[16:07] traviscline roger, pieterh one place the gevent compat is interesting is that you can do multi-thread/proc stuff in one file -- as I mentioned before that kind of defeats the purpose, any interest in examples like that or is that a little too off the reservation?
[16:08] pieterh what purpose does this defeat?
[16:08] traviscline defeats the purpose in terms of zmq scalability and whatnot
[16:08] pieterh ah
[16:08] pieterh i'd not say this defeats the purpose
[16:08] pieterh rather it lets you build properly MT apps
[16:09] pieterh which you can later scale out over other transports if needed
[16:09] traviscline alright, cool
[16:09] pieterh in the guide I switch between the two architectures (initial examples are multiple processes, later examples are MT in one process)
[16:09] pieterh being able to do real MT in one process is actually really valuable
[16:09] pieterh it lets you build realistic brokers, for example
[16:10] pieterh since this is one of the big limitations with pyzmq, the gevent stuff is very significant
[16:10] pieterh so yes, definite interest in such examples
[16:26] traviscline seb`: none, ano
[16:26] traviscline s/ano/no/
[16:28] traviscline seb`: pushed
[16:32] traviscline between cython and not? yes, for a straight tight loop it's about 50% faster with cython
[16:32] traviscline i'm working through a couple issues with the pyzmq guys on further cython intergration
[16:32] traviscline i should be able to eliminate virtually all python overhead
[16:33] traviscline i think libevents loop is still going to be slower than zmq's but I'll do some performance testing
[16:33] traviscline I have a small perf bench that I'll commit once it's a bit cleaner
[16:40] traviscline seb`: about to push a 0.13 compat fix
[16:43] traviscline good
[16:47] traviscline seb`: pushed compat
[16:57] traviscline seb`: works with pip
[18:09] benoitc mmm please don't :)
[18:09] benoitc you can add a requirements.txt for people that need to install gevent and other first
[18:25] benoitc i mean adding setuptools dependancy
[18:25] benoitc hopefully we will all use distutils2 soon :)
[20:11] traviscline getting this error when using pyzmq: Assertion failed: !more || pipes [current] != pipe_ (fq.cpp:61) traceback down into zmq:
[20:13] traviscline oh there's a todo above it
[20:15] traviscline ( )
[21:34] traviscline odd. goes away if i try again on EAGAIN without waiting for the fd to be marked as readable
[21:34] traviscline mikko: ^ that sound right?
[21:41] traviscline what I was/am doing is on first EAGAIN sleeping until the fd marks readable and EVENTS shows writable
[21:48] minrk It seems like you might not handle multipart messages properly
[21:50] traviscline details?
[21:50] minrk oh, maybe I'm wrong
[21:51] minrk I misinterpreted your loop, I think
[21:51] minrk I'm not familiar with gevent, so the event logic isn't obvious to me
[21:54] traviscline the wait suspends the current green thread until the file descriptor is marked readable (based on EVENTS)
[21:54] traviscline or writable obviously
[22:01] traviscline seb`: don't
[22:01] traviscline use an inproc or a gevent queue or something
[22:02] traviscline i'm not sure both greenlets would get triggered by libevent's picking up of the file descriptors read state
[22:02] traviscline that little problem could be solved, but i don't think you'd ever want to do that
[23:33] andrewvc cremes: around?