ZeroMq IRC Log

Tuesday January 25, 2011

[Time] Name	Message
[02:05] oaror	Hi there!
[02:05] oaror	I am having trouble setting up pyzmq on windows 7
[02:06] oaror	it fails with the error : "Setup script exited with the error: \Mozilla was unexpected at this time"
[02:06] oaror	This happens with easy_install
[02:08] oaror	Also downloaded pyzmq-2.0.10 and tried executing "python setup.py install" after building libzmq.dll with VS 2008
[02:08] oaror	same result
[02:08] oaror	any ideas how I can fix this?
[02:14] oaror	In particular this procedure has not worked for me :- http://www.zeromq.org/docs:windows-installations
[02:15] oaror	oops.
[02:15] oaror	that procedure did now work for me.
[02:16] oaror	I have it running on my Ubuntu desktop no issues. However we use windows 7 at the office - sigh.
[05:29] stockMQ	Hello!
[05:30] stockMQ	Though this does not really come under the scope of zeromq but am sure most who have used it would have tackled it and i am looking for some suggestions
[05:30] stockMQ	I have a PUB-SUB pattern
[05:31] stockMQ	On the Subscriber end when the stock feed is received currently I write it as binary to a file
[08:43] stockMQ	Hi
[08:43] stockMQ	In a PUB-SUB pattern
[08:43] stockMQ	i want to save the feed recvd at SUB side
[08:44] stockMQ	what is advisable
[08:44] stockMQ	saving it in file as binary ?
[08:46] stockMQ	I am sure many have dealt with this
[08:46] stockMQ	please share your experiences
[08:50] sustrik	stockMQ: there's no single answer
[08:50] sustrik	depends on what do you want to do with the file
[08:51] stockMQ	basically there will be other analysis applications which will be reading from the file
[08:51] stockMQ	this could happen during market open thus could create a race condition
[08:51] stockMQ	and also after market close when the write is not happening
[08:52] stockMQ	i want to save all the intra day data
[08:58] stockMQ	any suggestions?
[09:20] mikko	stockMQ: how many days are you storing?
[09:20] mikko	and how many messages per second are you handling (roughly)
[09:20] stockMQ	every day a new file will be created
[09:20] mikko	what is the size of a single message?
[09:20] stockMQ	I am handling around 1 message every 20ms
[09:21] stockMQ	so 50messages per second
[09:21] mikko	so not a huge amount but decent flow
[09:22] stockMQ	and each message say about 256B
[09:23] mikko	and each day is a separate file?
[09:23] stockMQ	for now yes
[09:23] stockMQ	but i also have a separate file for certain scrips
[09:24] stockMQ	In that case each scrip will have its own file and each file could have data across days
[09:24] mikko	maybe use some sort of database?
[09:24] stockMQ	so i would like to design a solution which will be scalable as far as the size is concerned
[09:24] stockMQ	actually the issue is
[09:25] stockMQ	i cannot expect the Subscribers to setup and configure a DB
[09:25] stockMQ	they are lay men brokers
[09:25] stockMQ	not IT educated
[09:26] mikko	stockMQ: it doesn't have to be a centralised db
[09:27] stockMQ	it cannot be a centralized db..
[09:27] mikko	bdb, kyoto cabinet, firebird etc
[09:27] stockMQ	The publisher publishes the feed
[09:27] mikko	there is a large amount of embedded databases available
[09:27] stockMQ	okk
[09:28] stockMQ	how are they in handling real time fast data
[09:28] stockMQ	in your experience
[09:29] mikko	different kind of databases have different models for handling concurrency and consistency
[09:29] mikko	i am testing kyoto cabinet at the moment and it provides different levels of consistency
[09:30] stockMQ	okk
[09:30] mikko	you can execute transactions, force sync to disk etc
[09:30] stockMQ	okk
[09:31] stockMQ	apart from db do you think it is possible to handle race condition in file storage
[09:31] stockMQ	The database is an option
[09:31] mikko	what sort of race conditions?
[09:31] stockMQ	The issue is
[09:31] stockMQ	when the feed is being written into a file at SUB end
[09:32] stockMQ	there could be Technical analysis tools reading data from the same file simultaneously
[09:32] stockMQ	though the OS should handle synchronization
[09:32] stockMQ	but i suspect there will be a case since the feed is being recvd so fast that
[09:33] stockMQ	the write will be given more time slice and the read may have to wait
[09:33] stockMQ	Database is an option..but i think the TA tools are designed to take .dat files as input
[09:34] mikko	can't really comment on that side as i don't really know the tools you are talking about
[09:35] stockMQ	That is right.. in fact just think that when the file is being written, there are another applications reading/wanting to read from it
[09:36] stockMQ	A DB would handle concurrency.. But any experience with file storage
[10:36] sustrik	you would have to ensure consistency yourself
[10:36] sustrik	which is kind of like writing a little database engine of your own
[10:36] sustrik	it's definitely better to use an existing product
[10:37] sustrik	ha, Vint Cerf talks at LCA: "We currently have no equivalent to the Erlang distribution to describe usage at the edges of the network, making provisioning and scaling difficult."
[10:37] sustrik	actually, we do :)
[10:38] benoitc	erlang OTP for supervision and such + zeromq make a good couple
[11:34] pieterhintjens	hi guys
[11:36] mikko	pieterh!
[11:36] pieterh	sustrik: where can I find the summary of changes between 2.0.10 and 2.1.0?
[11:36] mikko	been a while
[11:36] pieterh	hi mikko :-)
[11:36] pieterh	yeah, I was a bit sick
[11:36] mikko	i had a question about zfl
[11:36] pieterh	shoot
[11:36] mikko	is it supposed to work on windows?
[11:37] sustrik	pieterh: check the ChangeLog
[11:37] pieterh	yes, it's supposed to be a portability layer
[11:37] sustrik	in the main directory
[11:37] mikko	i noticed the use of pthreads while building on mingw32
[11:37] pieterh	sustrik: thanks
[11:37] pieterh	but like all such layers, until it's actually used on an OS it won't work on the OS
[11:38] pieterh	the idea is to put stuff like #ifdef _WIN32_ into one layer rather than sprinkled through apps
[11:38] pieterh	at least for the basic functionality a typical 0MQ app needs...
[11:39] mikko	cool
[11:39] pieterh	I should add something about porting ZFL to the docs, maybe
[11:39] pieterh	we did this for years with SFL, an older C library, the approach works nicely
[11:39] mikko	kinda the same that apache httpd does with apr
[11:40] mikko	i'll take a look at providing a patch for mingw32 etc
[11:42] pieterh	yes, APR is very similar in concept but (a) doesn't use a class model, which makes the code messy, and (b) is rather heavy
[11:42] pieterh	patches will be welcome :-)
[11:48] drbobbeaty	sustrik: you around? Did you get a look at the gist with the issue re: Ubuntu 10.04.1 and OpenPGM failing on the connect()?
[11:49] sustrik	the gist was dead in the morning
[11:49] sustrik	let me have a look now
[11:50] sustrik	drbobbeaty: can you please try using different port numbers for individual connections?
[11:50] drbobbeaty	sustrik: Sure, let me generate a new gist with the minimal code.
[11:51] sustrik	i mean, isn't the problem caused by connecting twice to the same port number?
[11:51] drbobbeaty	Nope, it's just plain connecting. On Ubuntu only. CentOS is fine.
[11:54] drbobbeaty	This is the minimal code sample: https://gist.github.com/794827 -- it jsut creates the context, creates the socket, and then calls connect(). It works great on CentOS 5 but fails on Ubuntu 10.04.1.
[11:55] sustrik	have you tried using different port numbers?
[11:55] sustrik	ah, ok
[11:55] sustrik	i'll check it out shortly
[11:55] drbobbeaty	thanks so much - any help you can provide would be greatly appreciated.
[12:04] pieterh	sustrik: there is no ChangeLog in the git repository
[12:05] pieterh	however there is a summary of changes in NEWS
[12:05] sustrik	right
[12:05] sustrik	ChangeLog is auto generated
[12:06] sustrik	you have to look into the package
[12:06] sustrik	if you want complete changelog
[12:06] pieterh	right, it's not in the git even after building
[12:06] sustrik	generating ChangeLog is part of the release proces
[12:06] sustrik	so have a look in 2.1.0 package
[12:07] pieterh	it's not documented on the wiki
[12:07] pieterh	there is a link to NEWS, which is rather more readable
[12:07] sustrik	is it sufficient for you?
[12:08] pieterh	I'll fix the wiki to explicitly answer the question, "what's changed in this version"
[12:08] mikko	"a lot"
[12:08] sustrik	if you need a comlete change log from the head version just use "git log"
[12:09] pieterh	a lot changed, but most users just want the headlines
[12:09] pieterh	a git log is close to useless for that
[12:09] pieterh	my question, for example, is "what do I need to change in the Guide"
[12:09] pieterh	other people will want to know, "what do I need to change in my apps, and what cool new stuff is there?"
[12:10] pieterh	NEWS looks like the best source, hand written and clear
[12:14] sustrik	yes, i think so
[12:31] kabs	Hello, in most of the zeroMQ examples server binds to for example : tcp://:5560 while client connects to for example tcp://localhost:5560, if I replace with localhost for server it doesn't work , didn't understand why , any one??
[12:32] traviscline	I'm working on a libevent zeromq integration and have a question (that might require libevent knowledge): libevent is optionally edge-trigger aware (EV_ET) on some backends (kqueue, epoll) -- is this feature desirable/useful in conjunction with a zmq socket? read events seem to be triggered fine without the EV_ET flag -- as of now i'm handling the "edge-triggered" nature of the zmq.FD by setting a readable and writable flag until respective EAGAIN
[12:35] kabs	Forgot to mention, examples I am talking about is from the guide http://zguide.zeromq.org/chapter:all#toc0 and my client servers are on same machine
[12:36] traviscline	mikko: you had some great input for me before, ^ that rambling make any sense?
[12:36] traviscline	i'm getting my feet wet with this stuff so I may be missing/misunderstanding concepts
[12:37] mikko	traviscline: not sure i understand what the question is
[12:38] mikko	you dont have to use edge-triggered event loop with zeromq but you need to handle the zeromq fd in edge-triggered fashion
[12:38] traviscline	mikko: if libevent's support of edge-triggered events makes sense in the context of ZMQ_FD descriptors
[12:38] traviscline	ok that's what I thought
[12:38] traviscline	thank you
[12:38] traviscline	I think my implementation is correct then
[12:39] mikko	i haven't seen any problems with level-triggered loop this far
[12:39] mikko	doing a loop in callback and reading until EAGAIN
[12:39] mikko	or checking ZMQ_EVENTS
[12:41] traviscline	what i'm doing is waiting on an internal read (and write) event and then when ZMQ_FD is readable checking EVENTS to see which internal events to set and unsetting them on EAGAIN and if after a read\|write EVENTS no longer shows the read\|write as available
[12:41] traviscline	maybe I could simplify it a bit
[12:59] stockMQ	Anyone used HDF5
[13:03] sustrik	kabs: you are binding to a network interface
[13:04] kabs	sustrik: yes
[13:04] sustrik	so, on linux, the loopback interface is called "lo"
[13:04] sustrik	"localhost" is a hostname
[13:04] kabs	sustrik: even if I change both client and server to lo , it doesn't work
[13:04] sustrik	connect connects to host
[13:04] kabs	sustrik: I tried with ip address, it worked,
[13:05] sustrik	so it's "lo" on bind side
[13:05] sustrik	and "localhost" on connect side
[13:05] kabs	sustrik: hmm ...
[13:06] kabs	sustrik: got your point, thanks!
[13:06] sustrik	np
[13:07] pythonirc101	mikko: I've two machines behind NATs and I want to do publish/subs for them (mutual). Is this something that can be done using zeromq?
[13:13] drbobbeaty	sustrik: I think I have a lead to the problem with OpenPGM... It appears in socket_base.cpp (line 195) that in order to use the epgm protocol I need to NOT have options.requies_in and options.requires_out set to TRUE. Yet, in the constructor in sub.cpp (line 27), the superclass constructor - xsub_t (line 33 in xsub.cpp) is setting them BOTH to be TRUE. With the subclass, sub_t not chancing these values, it appears that they are going to be TRUE for all
[13:13] drbobbeaty	sockets.
[13:13] sustrik	yes, that's the recent change i was referring to
[13:14] drbobbeaty	Ah... OK, perfect. So I'll just wait for your resolution. Thanks.
[13:25] spht	pythonirc101: without a proxy in between you need to look at UPnP
[13:29] pythonirc101	spht: I could use a proxy for connection establishment, but not for data transfer...
[13:31] spht	pythonirc101: Except for manual NAT config this is your only option: http://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol
[13:32] spht	though note that it's not implemented by all vendors/devices
[15:32] traviscline	ANN: much improved gevent-zeromq compatibility pushed at https://github.com/traviscline/gevent-zeromq
[15:53] traviscline	seb`: cool, more than happy to work out any kinks but feel much more confident with the recent changes
[15:53] traviscline	seb`: I wonder if the zguide guys would take some python-gevent submissions
[15:54] traviscline	thought part of the point is the ease in going multi-proc, it's nice seeing lots going on in one process smoothly
[15:58] pieterh	traviscline: anything you want to submit to zguide, more than welcome!
[15:59] pieterh	so you have a specific API for python-gevent?
[15:59] traviscline	pieterh: trouble is the single threaded examples are the same sans a different import
[15:59] traviscline	when you go multi-threaded it'd change
[15:59] traviscline	pieterh: they may not be different enough to be that useful but i'll compare
[16:00] traviscline	pieterh: it's meant to be the pyzmq api but it replaces parts that would block and gets the FD into libevent's loop
[16:00] pieterh	do you expect this to become the standard pyzmq eventually?
[16:01] traviscline	I've been talking to the guys while working on this but it's possible it'd end up in some sort of 'green' submodule somewhere
[16:01] traviscline	haven't had that discussion in detail
[16:02] pieterh	you consider this a fork / improvement of pyzmq, right?
[16:02] traviscline	it's not uncommon in gevent land to provide compat libs which mirror and/or patch the apis of other libs
[16:03] traviscline	i don't bundle pyzmq, i depend on it and wrap it
[16:03] traviscline	but as I said, it could change, depends on what hte pyzmq guys think
[16:03] pieterh	ok...
[16:03] pieterh	what I'd suggest is translate the interesting examples to python-gevent-zmq
[16:04] pieterh	package them at least with your code
[16:04] traviscline	pieterh: if you're interested in the gevent bits: here's all the rr* stuff in one file
[16:04] pieterh	and then we'll also package them with zguide
[16:04] traviscline	seb`: agreed
[16:04] traviscline	pieterh: cool, can do
[16:04] traviscline	pieterh: whoops, link: https://github.com/traviscline/gevent-zeromq/blob/master/examples/reqrep.py
[16:04] pieterh	send me a zip with everything when it's ready
[16:04] pieterh	ph@imatix.com
[16:04] traviscline	rgr
[16:04] pieterh	or better, to zeromq-dev so it's public
[16:05] pieterh	i'll add Python-gevent as a 'Language'
[16:06] pieterh	we have variants of other language APIs too, and have solved this simply by adding the examples to the relevant directory
[16:06] traviscline	hmm
[16:06] pieterh	they don't show up when people read the guide online but they are in the git if they clone it
[16:06] traviscline	taskvent-gevent.py or the like?
[16:07] pieterh	yes
[16:07] pieterh	at the least we archive and publish the examples
[16:07] pieterh	i may add 'variants' to the languages if we get this happening more
[16:07] traviscline	roger, pieterh one place the gevent compat is interesting is that you can do multi-thread/proc stuff in one file -- as I mentioned before that kind of defeats the purpose, any interest in examples like that or is that a little too off the reservation?
[16:08] pieterh	what purpose does this defeat?
[16:08] traviscline	defeats the purpose in terms of zmq scalability and whatnot
[16:08] pieterh	ah
[16:08] pieterh	i'd not say this defeats the purpose
[16:08] pieterh	rather it lets you build properly MT apps
[16:09] pieterh	which you can later scale out over other transports if needed
[16:09] traviscline	alright, cool
[16:09] pieterh	in the guide I switch between the two architectures (initial examples are multiple processes, later examples are MT in one process)
[16:09] pieterh	being able to do real MT in one process is actually really valuable
[16:09] pieterh	it lets you build realistic brokers, for example
[16:10] pieterh	since this is one of the big limitations with pyzmq, the gevent stuff is very significant
[16:10] pieterh	so yes, definite interest in such examples
[16:26] traviscline	seb`: none, ano
[16:26] traviscline	s/ano/no/
[16:28] traviscline	seb`: pushed
[16:32] traviscline	between cython and not? yes, for a straight tight loop it's about 50% faster with cython
[16:32] traviscline	i'm working through a couple issues with the pyzmq guys on further cython intergration
[16:32] traviscline	i should be able to eliminate virtually all python overhead
[16:33] traviscline	i think libevents loop is still going to be slower than zmq's but I'll do some performance testing
[16:33] traviscline	I have a small perf bench that I'll commit once it's a bit cleaner
[16:40] traviscline	seb`: about to push a 0.13 compat fix
[16:43] traviscline	good
[16:47] traviscline	seb`: pushed compat
[16:57] traviscline	seb`: works with pip
[18:09] benoitc	mmm please don't :)
[18:09] benoitc	you can add a requirements.txt for people that need to install gevent and other first
[18:25] benoitc	i mean adding setuptools dependancy
[18:25] benoitc	hopefully we will all use distutils2 soon :)
[20:11] traviscline	getting this error when using pyzmq: Assertion failed: !more \|\| pipes [current] != pipe_ (fq.cpp:61) traceback down into zmq: https://gist.github.com/4fa28274b1c08bb1c976
[20:13] traviscline	oh there's a todo above it
[20:15] traviscline	( https://github.com/zeromq/zeromq2/blob/master/src/fq.cpp#L61 )
[21:34] traviscline	odd. goes away if i try again on EAGAIN without waiting for the fd to be marked as readable
[21:34] traviscline	mikko: ^ that sound right?
[21:41] traviscline	what I was/am doing is on first EAGAIN sleeping until the fd marks readable and EVENTS shows writable
[21:48] minrk	It seems like you might not handle multipart messages properly
[21:50] traviscline	details?
[21:50] minrk	oh, maybe I'm wrong
[21:51] minrk	I misinterpreted your loop, I think
[21:51] minrk	I'm not familiar with gevent, so the event logic isn't obvious to me
[21:54] traviscline	the wait suspends the current green thread until the file descriptor is marked readable (based on EVENTS)
[21:54] traviscline	or writable obviously
[22:01] traviscline	seb`: don't
[22:01] traviscline	use an inproc or a gevent queue or something
[22:02] traviscline	i'm not sure both greenlets would get triggered by libevent's picking up of the file descriptors read state
[22:02] traviscline	that little problem could be solved, but i don't think you'd ever want to do that
[23:33] andrewvc	cremes: around?