ZeroMq IRC Log

Tuesday February 8, 2011

[Time] Name	Message
[01:50] Steve-o	mikko: the x1024 patch on PGM 'works' but isn't correct, I'll have to look at Ian's suggestion tomorrow
[01:51] Steve-o	all day at IKEA today, patience is a virtue as is said.
[01:57] d4de	Hi, any idea on when aprox. v2.1.0 will go stable?
[02:00] Steve-o	good question, probably dependent on the issue list on github getting cleaned up a bit
[02:01] d4de	link?
[02:02] d4de	meh nvm ... I think google can answer that one :-)
[02:02] Steve-o	https://github.com/zeromq/zeromq2/issues
[02:02] d4de	ooh well ... thanks :-D
[02:03] Steve-o	there are a couple of weird platform dependent / compiler optimisation issues with the PGM socket
[02:03] Steve-o	and there has been issues with zmq_term
[02:03] d4de	so it is just PGM that is putting it back ... I saw that you guys has already fixed some annoying bugs in 2.1 beta, so I keep wondering what is left
[02:03] Steve-o	you can always ask Martin again on the list
[02:04] d4de	hmm still no IPv6 support huh?
[02:04] Steve-o	well yes and no
[02:05] Steve-o	there probably needs to be extra flags or specification for using IPv4 or IPv6, or both
[02:05] Steve-o	similar to BSD's IPV6ONLY
[02:06] d4de	aha
[02:06] Steve-o	binding to eth0:* tends to be complicated in a multi-stack environment
[02:07] Steve-o	surprisingly PGM has better support for IPv6 but it depends on reasonable OS support
[02:08] Steve-o	there are some issues with the glibc in Ubuntu 10.10 there are definitely unexpected
[02:08] Steve-o	for example ::1 resolves as 127.0.0.1
[02:08] Steve-o	getaddrinfo (`hostname`) returns different address to ping `hostname` when using DHCP
[02:09] Steve-o	one easy path is to assume IPv4 only and follow SMTP's requirement to use a IPv6: prefix
[02:11] Steve-o	I have to concede I only follow the PGM parts of ZeroMQ, not so much the other parts
[02:12] Steve-o	There is certainly work on improving ZMQ_SWAP and XPUB/XSUB sockets
[06:18] stockMQ	Hi.. I have a binary char* which i want to send from one Thread to Another.. i am trying inproc.. sending the char* as it is as a message does not work as at the receiving end while decoding i get erroneous data..I understand i have to use protobuf..but then can i serialize to string
[06:42] sustrik	you have to copy the data to the message
[07:38] stockMQ	I want to send a Byte* over a socket.. What would be the right way to do it
[07:53] Guthur	stockMQ, do you mean send the pointer?
[08:54] Guthur	I am sorry, I've busted the clrzmq2 build again, this MSVS solution stuff is utter crap
[09:05] mikko	good morning
[09:52] sustrik	morning
[13:11] Skaag	BAHAHAHAHA.... IT'S ALIVE!!!
[13:11] Skaag	The horror!
[13:14] sustrik	run!
[13:14] sustrik	:)
[13:19] ianbarber	another epgm question: if I, in one thread, do a PUB send, then a SUB recv, using PGM, that should work right?
[13:21] sustrik	ianbarber: that's using loopback multicast
[13:21] Guthur	is the lack of IPC on windows more a lack of motivation issue as oppose to technical?
[13:21] sustrik	in reality, multicast doesn't work in loopback manner
[13:22] sustrik	so there's an emulation layer in the OS that simulates it
[13:22] ianbarber	oh right
[13:22] sustrik	however, as i am said, it's pretty crappy
[13:22] sustrik	so rather use ipc/tcp for loopback
[13:22] sustrik	Guthur: both in a way
[13:22] ianbarber	that makes sense, i mean it's not something that is a good idea other than for testing stuff
[13:23] sustrik	ack
[13:23] sustrik	Guthur: to implement IPC on windows you need NamedPipes
[13:23] sustrik	these are identified by handles rather than fds
[13:24] sustrik	so you can't use select() for polling
[13:24] sustrik	thus you have to implement IOCP polling mechanism
[13:24] Guthur	oh, I should have none it was a simple issue, hehe
[13:25] sustrik	well, it's doable
[13:25] Skaag	what happens when one client out of 20, is down for an hour?
[13:25] Skaag	it loses all data, right?
[13:25] sustrik	Skaag: if no identity is set, then yes
[13:25] Skaag	and if identity is set?
[13:26] sustrik	then only the data that were "on the wire" during the failure are lost
[13:26] sustrik	the subsequent messages are queued in the publisher
[13:26] Skaag	and are saved in memory?
[13:26] sustrik	yes
[13:26] Skaag	and if the publisher died too, they are lost?
[13:26] sustrik	sure thing
[13:26] Skaag	ok :)
[13:27] Skaag	i'm trying to figure out how to keep 20+ machines in sync with the same data
[13:27] Skaag	they all get small incremental updates via zmq
[13:27] Skaag	but it accumulates to quite a lot
[13:28] sustrik	ask for a snapshot after starting the client using req/rep
[13:28] sustrik	then subscribe to incremental upgrades using pub/sub
[13:31] Skaag	sounds simple enough
[13:32] Skaag	the remaining problem is knowing who to ask the snapshot from
[13:32] Skaag	who is the leader with the best snapshot
[13:32] Skaag	probably the one with the longest uptime
[13:32] mikko	are you using zookeeper in the setup?
[13:32] Skaag	not anymore
[13:32] Skaag	I can't use zookeeper because it requires a fixed number of nodes
[13:33] mikko	why is that?
[13:33] Skaag	when when nodes are added, a configuration file needs to be edited, and the whole thing restarted
[13:33] mikko	you mean to zookeeper itself?
[13:33] Skaag	I followed a thread on their list, and found a person started working on fixing this
[13:33] Skaag	yes to zookeeper itself
[13:33] Skaag	anyway his work has stalled
[13:34] Skaag	and he offered someone else to continue working on it, and the ETA is 3 ~ 5 month to implement
[13:34] Skaag	why, no idea..
[13:35] Guthur	sustrik: why not just use sockets for IPC
[13:36] Guthur	sockets can be polled with select, or am I missing something big here
[13:36] sustrik	Guthur: nope
[13:36] sustrik	you can use sockets
[13:36] sustrik	just use tcp://127.0.0.1:port
[13:38] sustrik	Skaag: choosing a leader in a distributed env is a complex problem...
[13:38] Guthur	are pipes more performant?
[13:38] sustrik	you should peek into some disttibuted algorithm textbook
[13:38] zchrish	mikko: You referred me to valokuva.org for a mingw dll for Windows build. Is there a library option also available?
[13:38] sustrik	Guthur: never measured it
[13:38] sustrik	but it would make sense if they were
[13:39] sustrik	Skaag: especially beware of split-brain syndrome
[13:40] Skaag	right, I heard about that
[13:40] Skaag	a problem that exists in the glusterfs project
[13:40] Skaag	(which I also use)
[13:40] sustrik	it's when the network is broken into two parts
[13:40] sustrik	each part selects its own leader
[13:40] sustrik	resulting in inconsistent state
[13:41] Skaag	can run a competition with set rules, and the winner gets its data replicated to everybody
[13:41] sustrik	to prevent the split-brain the competition could be won only by a node that gets more than 50% of the vote
[13:42] sustrik	to compute 50% you need to know the size of the cluster
[13:42] sustrik	it's complex...
[13:45] mikko	zchrish: what do you mean?
[13:45] mikko	zchrish: static library?
[13:46] zchrish	Yes, a static library.
[13:46] mikko	zchrish: no, we don't support building static libraries under mingw32
[13:46] zchrish	Oh, I see. Why not?
[13:46] mikko	zchrish: it's a massive hassle with the dllexport/dllimport stuff
[13:46] mikko	on windows
[13:47] zchrish	Sure, ok. Thank you.
[13:47] mikko	you can probably build libs by hacking the build scripts
[13:47] mikko	but to keep things somewhat sane for mingw32/visual studio we only support dlls
[13:48] CIA-21	zeromq2: 03Martin Sustrik 07master * r5dcbc34 10/ src/zmq.cpp :
[13:48] CIA-21	zeromq2: zmq_poll with no fds behaves decently
[13:48] CIA-21	zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/g191uU
[13:48] zchrish	But I see a .lib in the msvc2008 output.
[13:49] mikko	thats the import library
[13:49] zchrish	Oh, ok.
[14:00] zchrish	mikko: Is WIN7 required to use the DLL?
[14:02] ianbarber	sustrik/mikko: any thoughts on best way to debug if I see PGM packets from wireshark on my reciever host, but am not getting anything from 0MQ listening on that host
[14:07] sustrik	check whether the multicast addresses and ports match on publisher and receiver
[14:08] ianbarber	yep, definitely match
[14:08] ianbarber	subscribe is ""
[14:08] ianbarber	(as well)
[14:09] sustrik	then you can place breakpoint to pgm_socket_t::recv()
[14:09] sustrik	to see whether the data are getting from openPGM to 0MQ
[14:16] signalseeker	Is a pdf version of the zmq guide available?
[14:17] mikko	zchrish: i wouldn't think so
[14:17] zchrish	OK.
[14:18] mikko	might need the visual studio 2008 redist libs
[14:18] ianbarber	sustrik: it's not breaking in there, so it looks like 0MQ isn't seeing the packets
[14:33] sustrik	ianbarber: ask steve-o then
[14:33] ianbarber	steve-o: is there anything to check whether libpgm can see the packets on a host, for example? I've can see them in wireshark, but 0MQ doesn't seem to be getting them.
[14:33] ianbarber	was just about to, and did :)
[14:34] Steve-o	mmm
[14:34] Steve-o	You can try increasing the log level, i.e. PGM_MIN_LOG_LEVEL=TRACE
[14:35] ianbarber	where would the logging go?
[14:36] Steve-o	stdout
[14:36] Steve-o	is it completely silent in zeromq or a little bit of data then silence?
[14:36] ianbarber	ah
[14:36] ianbarber	Trace: Discarded packet for muted receiver
[14:37] ianbarber	awesome!
[14:37] ianbarber	now, why would it do that
[14:37] ianbarber	:)
[14:39] Steve-o	a sender would see that on data packets
[14:39] Steve-o	possible cause is multiple active senders
[14:40] Steve-o	it's not expected for a SUB socket
[14:41] Steve-o	only PUBs
[14:45] ianbarber	hmm, that was definitely on a sub
[14:45] ianbarber	but maybe there's something rogue running as well
[14:47] Steve-o	it is only emitted if the socket is not permitted to receive data
[14:48] Steve-o	so a SUB socket that cannot recv sounds a bit broken
[14:48] ianbarber	hmm, that's strange
[14:48] ianbarber	nothing else running
[14:48] ianbarber	definitely only triggers when I'm sending on the other host
[14:48] ianbarber	if i don't send, no output
[14:49] Guthur	If i was wanting to learn what interface a windows IPC implementation should have where is the best places to look in the source code?
[14:50] ianbarber	it does mention a send interface as well steve-o
[14:50] ianbarber	Trace: Create transmit window.
[14:50] ianbarber	Trace: Binding receive socket to INADDR_ANY
[14:50] ianbarber	Trace: Binding send socket to interface index 2
[14:50] ianbarber	Trace: Join multicast group 239.192.0.1 on interface index 2
[14:50] ianbarber	Trace: Multicast send interface set to 192.168.192.132 index 2
[14:50] ianbarber	is that normal?
[14:50] Steve-o	ianbarber: PGM requires a back-channel from recv to sender for reliability
[14:50] ianbarber	oh, of course
[14:50] ianbarber	silly me
[14:51] ianbarber	if you have a sec, could you look at https://gist.github.com/816533 and see if I'm doing something obviously dumb?
[14:51] Steve-o	on Linux or Windows?
[14:52] Steve-o	Linux flavour I'm guessing
[14:52] ianbarber	linux
[14:53] Steve-o	looks reasonable although you may hit this bug: https://github.com/zeromq/zeromq2/issues#issue/139
[14:53] sustrik	Guthur: you would have to start with implementing IOCP poller
[14:54] sustrik	see select_t, poll_t, epoll_t, devpoll_t and kqueue_t
[14:54] sustrik	you would have to make something similar
[14:54] sustrik	presumably called iocp_t
[14:54] Steve-o	ianbarber: sleeping before zmq_term can usually help diagnosis, basically keeping the socket open a bit longer.
[14:54] Steve-o	ianbarber: or sending more data.
[14:55] Guthur	sustrik: ok, thanks
[14:55] ianbarber	steve-o: I've got the sender sending then sleeping for 1 second, in a loop, so it's c onstant stream of data. At the moment, I just wait on recv in the test code
[14:56] ianbarber	each send triggers about 4-5 of those muted receiver notices
[14:56] Guthur	from MS docs local named pipes would be considerably faster than sockets
[14:56] Guthur	so maybe worth the look
[14:56] Steve-o	ianbarber: send me some code, steven.mccoy@miru.hk, I'm working on Windows this morning and I'll have a look later
[14:57] ianbarber	will do, thanks steve
[14:57] sustrik	Gurthur: yes, definitely
[15:17] neale1	sustrik: Have you seen this whitepaper: http://mnb.ociweb.com/mnb/MiddlewareNewsBrief-201004.html
[15:18] drbobbeaty	Question: Let's say I am using ZMQ/OpenPGM on the epgm: protocol... also let's assume that I have several multicast channels sending data. On the client, is it better to have a single zmq::context_t with 1 thread working for a series of zmq::socket_t connections, each ZMQ_SUB-ed to different multicast channels? Or would it be better to have a 1:1 mapping between the zmq::context_t and it's zmq::socket_t?
[15:21] CIA-21	zeromq2: 03Martin Sustrik 07master * r889424e 10/ src/ctx.cpp :
[15:21] CIA-21	zeromq2: max_sockets honoured precisely
[15:21] CIA-21	zeromq2: The internal log socket was subtracted from the number of
[15:21] CIA-21	zeromq2: available sockets. So, if max_sockets was set to 100,
[15:21] CIA-21	zeromq2: you could create only 99 sockets. Fixed.
[15:21] CIA-21	zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/g08FfG
[15:34] Guthur	what is the default max number of sockets per context?
[15:34] Guthur	and what happens if you try to exceed that
[15:35] Guthur	oh I've just answered my own question
[15:38] drbobbeaty	I believe the context is a "one per process space" entity, so I believe you have one context for any number of sockets in the process, but I just wanted a verification from the designers.
[16:02] sustrik	Guthur: 512
[16:03] Guthur	sustrik: is it possible to set it higher?
[16:03] sustrik	neale1: yep, it's an old one
[16:04] sustrik	i think there is some discussion with opendds guys on the mailing list or somewhere
[16:04] sustrik	Guthur: sure
[16:05] sustrik	drbobbeaty: confirmed
[16:06] drbobbeaty	sustrik: thank you very much for the confirmation. Appreciated.
[16:07] Guthur	sustrik: I did a search for max_sockets in the src and couldn't find it
[16:07] Guthur	is it a build variable
[16:08] sustrik	src/confing.hpp
[16:10] Guthur	all there is is max_app_threads = 512,
[16:11] Guthur	is that equivalent?
[16:12] mikko	Guthur: https://github.com/zeromq/zeromq2/blob/master/src/config.hpp#L31
[16:13] Guthur	umm thats called max_app_threads = 512, in my source
[16:13] mikko	2.0.x ?
[16:14] Guthur	ah, yeah
[16:28] sustrik	the theading has changed between 2.0 and 2.1
[16:29] sustrik	with 2.0 the limit is on threads using 0mq sockets
[16:29] sustrik	in 2.1 the limit is on the number of the sockets themselves
[16:30] Guthur	I think this may explain a mysterious app crash I was having
[16:30] mikko	Guthur: on windows?
[16:31] Guthur	yep
[16:32] Guthur	but it should have thrown an exception
[16:32] Guthur	might have been another cause
[16:34] mikko	running out of filehandles?
[16:35] mikko	i think there was some shenanigans with filedescriptor count on windows
[16:37] kristsk	like max 512 per process or so i've heard
[16:37] mikko	i think it was way less by default
[16:37] mikko	64 even
[16:38] kristsk	thats not much.
[16:38] mikko	https://github.com/zeromq/zeromq2/issues/issue/57
[16:39] sustrik	64 is the default max descriptors you can poll on on win32
[16:39] mikko	ah ok
[16:39] sustrik	i've overloaded the value to 1024 recently
[16:53] Guthur	I'm not sure what my issue was, the app just silently shutdown, not very helpful at all
[16:55] sustrik	not even an assertion?
[16:55] Guthur	nothing, it may not have been ZMQ
[16:55] Guthur	there was quickfix as well
[16:56] sustrik	i see
[16:59] Guthur	well, home time now
[17:09] stockMQ	Hi.. I am receiving protobuf messages from my zmq socket which i then want to serialize to a file as binary.. this file in turn should be readable by other processes simultaneously..
[17:09] stockMQ	Any pointers?
[17:10] mikko	one file per message or all messages in one file?
[17:11] stockMQ	all messages in one file.. the messages are sequentially timestamped
[17:12] stockMQ	basically the writer will just dump all incoming messages to the file
[17:12] stockMQ	but the reader processes should be able to read and de serialize them
[17:13] mikko	how does the reader know that message is not impartial?
[17:14] stockMQ	by impartial what do you mean?can you elaborate please
[17:15] mikko	if the writer is writing a message how do the readers know that the message has been fully written?
[17:15] mikko	are you planning to use locking?
[17:15] stockMQ	No..basically what i was doing till now is
[17:15] mikko	and if yes how are you going to sync readers/writers?
[17:15] stockMQ	intradayFile.Open(dataPath,CFile::modeWrite \| CFile::modeNoTruncate \| CFile::modeCreate \| CFile::shareDenyWrite \| CFile::osWriteThrough)
[17:16] mikko	because you could easily "DoS" yourself by having high rate of messages coming in
[17:16] mikko	in which case the lock wouldnt be released
[17:16] sustrik	i would just use a DB
[17:16] stockMQ	This basically is supposed to allow Shared read.. if the reader process opens the file with appropriate permission
[17:17] stockMQ	The synchronization is supposed to be handled by the OS
[17:17] stockMQ	I am working on VC++
[17:18] sustrik	the OS won't handle the sync for you
[17:18] sustrik	you have to implement it yourself
[17:18] sustrik	that's why it's better to use a DB
[17:18] stockMQ	ok
[17:19] mikko	i think posix will guarantee that write within a process is atomic
[17:19] mikko	but thats about it
[17:19] stockMQ	ok
[17:19] mikko	between processes
[17:19] mikko	but that doesn't really help you with readers at all
[17:20] mikko	and not sure how posix windows is
[17:20] mikko	this is what posix says:
[17:20] mikko	"Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. "
[17:21] stockMQ	My understanding was that CFile::ShareDenyWrite is a flag that tells the operating system not to permit any other process to write to your file while you have it open.
[17:21] mikko	stockMQ: it will probably but what about if you have a constant stream of messages coming in?
[17:22] stockMQ	and shareRead would allow it to be read simultaneously
[17:22] stockMQ	yes u r right.. i had that in mind.. but wanted to check that heuristically
[17:22] mikko	writer will block the readers
[17:23] mikko	usually
[17:23] mikko	if you have readers reading while the writer is writing you might end up with partial / incomplete data
[17:23] mikko	then you need to track if the message is fully written or not
[17:24] neale1	mikko: did you want to try enabling the clrzmq build for s390x to see how it goes?
[17:24] mikko	neale1: i can at some point, still at work for about an hour
[17:24] neale1	no worries
[17:25] stockMQ	yes u r right.. Hence i planned to first write it to a file..see how the readers behave and then switch to a DB.. to learn along the process as well
[17:26] stockMQ	I have received suggestions here like Kyoto cabinet,BDB etc.. More inputs/suggestions are welcome..Also it would be helpful if there is anyone who has written protobufs to a file can share his/her experience
[17:29] mikko	do you need to keep this data?
[17:29] stockMQ	yes
[17:29] stockMQ	basically these are stock trade data..
[17:29] stockMQ	they need to be kept for back analysis
[17:29] mikko	and this is on windows?
[17:30] stockMQ	and there could be more than one analysis tool accessing the data and hence the parallel readers
[17:30] stockMQ	correct
[17:31] mikko	do you want to keep the raw data or the results of the analysis?
[17:31] stockMQ	The raw data
[17:31] stockMQ	the analysis tools they basically read the data..do their stuff and display
[17:31] mikko	so you want to be able to get historic data for time periods etc?
[17:31] stockMQ	yess
[17:31] mikko	do you want to be able to query the data?
[17:31] stockMQ	till now i was copying the protobufs to struct and then writing the structs
[17:32] stockMQ	now i want to bypass the structs and directly write the protobufs
[17:33] mikko	the data you are receiveing, what kind of data is it?
[17:33] stockMQ	yes.. query for the time window..nothing complex
[17:33] stockMQ	as for nw
[17:33] mikko	numbers, text?
[17:33] stockMQ	both..mostly numbers.. text are just names of the Stock scrip
[17:34] mikko	what sort of consistency requirements do you have for the data?
[17:35] mikko	or maybe more accurate question would be resilience requirements
[17:36] mikko	for example is it ok to fsync to disk every 10 seconds and if your system crashes you lose 10 seconds of data currently in buffers?
[17:36] stockMQ	well.. if it is better not to lose any data :)
[17:36] stockMQ	the analysis are going to be sensitive to loss of data
[17:39] mikko	it's not an easy engineering task in that case
[17:41] mikko	does the data need to be consistent on all consumers all the time?
[17:41] stockMQ	I understand.. In fact to convince the management to go for DB.. i need to first implement a prototype writing it to a file and show them how the readers fair.
[17:41] kristsk	CAP is a bitch.
[17:42] mikko	yes, i was thinking this in terms of cap theorem
[17:42] stockMQ	yes. basically more than one instance of a tool would be accessing it
[17:42] stockMQ	so yes
[17:42] stockMQ	CAP? please enlighten
[17:43] mikko	http://en.wikipedia.org/wiki/CAP_theorem
[17:43] kristsk	consistency, availability, partitioning - choose any two.
[17:44] stockMQ	C and A
[17:44] ianbarber	stockMQ: you might have more success in writing straight out to a file, less disk seeks
[17:44] ianbarber	the kafka guys talk about that a bit on their site
[17:45] mikko	ianbarber: the problematic thing is that the processes reading the files are external
[17:45] mikko	unlikely with kafka
[17:45] kristsk	kafka ?
[17:45] mikko	http://sna-projects.com/kafka/
[17:46] ianbarber	mikko: oh, i see. I was thinking write file for backanalysis, 0MQ for distribution
[17:46] kristsk	oh.
[17:46] mikko	ianbarber: the process writing file is a subscriber as far as i understood
[17:46] mikko	and the analysis tools read the file and create reports and such
[17:47] mikko	so what you are effectively creating is a database
[17:47] stockMQ	that is right Mikko
[17:47] ianbarber	yeah, sounds like a database and replication
[17:48] mikko	so, without knowing the application / architecture very deeply somehow i would be inclined to say that it makes sense to investigate existing database technologies before going through all the hurdles
[17:49] mikko	maybe you have volatile database for quick access to data and push to another queue which warehouses the data
[17:49] mikko	very hard to say without knowing much about the application or the overall architecture
[17:50] stockMQ	In a nutshell.. The exchange multicasts feed which contains the Intra day data.. I have intercepted this feed and then implemented a PUB
[17:51] kristsk	uh, kafka is java-ish...
[17:52] stockMQ	The SUBs need to dump this data somewhere.. which should be accessible to the Analysis tools during market hours for intra day analysis and after market hours for back analysis
[17:52] stockMQ	After market hours, the write will not be happening..so it will be only reads.
[17:52] stockMQ	The hurdle is during market hours
[17:53] kristsk	umm, isnt pub/sub kinda "you weren't here, you missed, sorry" ?
[17:53] stockMQ	yes that is right
[17:53] stockMQ	hence the PUB also need to keep a copy of the data
[17:53] mikko	pub/sub is like a radio broadcast (a metaphore sustrik tends to use)
[17:54] mikko	when your receiver is on you will receive the broadcast
[17:54] stockMQ	If any SUB failed to connect on time..it should be able to borrow the missed data from PUB
[17:54] mikko	you can use identities for the subscribers
[17:55] kristsk	how many subscribers? maye you could make a queue per "subscriber"?
[17:56] stockMQ	Well hard to say..to start with internally it will be around 10.. but it can scale quickly
[17:57] kristsk	umyeah, ten-folding data probably is not a good idea.
[17:57] mikko	so effetively you want separate req/rep socket on the publisher where subscribers can ask for a delta
[17:58] mikko	is subscriber is offline for let's say 10 minutes it could ask the publisher through REQ for the last ten minutes
[17:58] mikko	to get the missing pieces
[17:59] stockMQ	yes .. that is going to be phase 2.. for now the market hours are known to all.. so the SUBs take up the responsiblity of listening before the market opens
[17:59] stockMQ	yes somewhat like that
[17:59] mikko	anyway, gotta commute home ->
[18:00] stockMQ	ohkk..can i catchu later?
[18:04] sustrik	stockMQ: you should also be concerned about HD performance
[18:04] sustrik	if the feed is fast, as it tends to be with market data
[18:04] sustrik	you may not be able to write the data to the HD fast enough
[18:05] stockMQ	ok
[18:06] stockMQ	So what would be the right approach in your opinion
[18:07] sustrik	depends on details
[18:07] sustrik	like, are you NASDAQ?
[18:07] sustrik	if so, a single HD won't be enough
[18:08] sustrik	if you are just a small trading firm getting modest amount of market data
[18:08] sustrik	it may suffice though
[18:08] kristsk	SSD then heh
[18:08] sustrik	SSD is better in terms of latencies
[18:09] sustrik	the throughput is similar to standard HD
[18:09] sustrik	what you need is either software load balancer
[18:09] sustrik	or a RAID
[18:09] sustrik	SAN maybe...
[18:09] stockMQ	ok
[18:10] sustrik	in any case i would not try to implement a database myself
[18:10] kristsk	netwrokish san would actually ensure no data gets lost
[18:10] sustrik	it takes tens of manyears to do it properly
[18:12] stockMQ	Actually the Interceptor aggregates the data and then Publishes it.. so the data received by the SUB will be lesser and slower than the actual feed received by the interceptor
[18:13] stockMQ	Any suggestions for DBs.. open source
[18:14] kristsk	pg ?
[18:14] sustrik	given you need very little functionality, i would check those new nosql projects
[18:14] kristsk	postgre
[18:15] stockMQ	like Kyoto cabinet?
[18:15] stockMQ	or
[18:15] stockMQ	http://en.wikipedia.org/wiki/Berkeley_DB
[18:15] stockMQ	any users here already?
[18:20] stockMQ	I think Mikko had used Kyoto
[18:28] stockMQ	so to summarize.. DB seems to be the only sane solution.. Is that right
[19:03] stockMQ	What would be the behaviour of the sockets if it is kept idle say overnight..
[19:06] Guthur	One would hope it is still there in the morning
[20:06] cremes	Guthur: max_sockets in src/config.hpp
[20:06] cremes	it defaults to 512; i usually override it to 10k for my own use
[20:33] Guthur	cremes, cheers, I was looking in the wrong src version, pre-2.1 it was just a thread count
[20:34] cremes	Guthur: ah, ok
[20:44] andrewvc	any idea what to make of nbytes != -1 (mailbox.cpp:241) errors in zmq?
[21:32] mikko	neale1: there?
[21:33] neale1	mikko: Yes
[21:33] neale1	mikko: BTW, where r u located?
[21:33] mikko	neale1: London
[21:33] mikko	London United Kingdom, not London Canada
[21:33] neale1	mikko: Better not mention the cricket - I'm an Australian living in the US
[21:34] mikko	I'm a Finn living in London and know nothing about cricket
[21:34] mikko	heh
[21:34] taotetek	gah. anyone here have some pointers on how one could make a pull socket appear as a file (stringio object) using python with pyzmq? I suppose that's more of a python question than a zeromq question.
[21:34] neale1	:-)
[21:34] Guthur	Australians, sure they are always winning the Ashes
[21:34] Guthur	...oh wait
[21:34] mikko	so anyway, i added a possibility for SSL tunneling the Hudson slave connection to add transport layer security to the connection
[21:34] Guthur	hehe, only teasing, I don't much care for cricket
[21:35] mikko	currently the master-slave connection goes over plain text which is not ideal as the master executes commands on the slave
[21:35] neale1	It's okay, the poms have waited 20 years to get back at us
[21:35] neale1	mikko: Do I need to do anything from my end?
[21:35] Guthur	neale1, Yeah, sure you're just lending it to them for a while
[21:36] bitweiler	hey what is this uuid-dev package & uuid/e2fsprogs?
[21:36] mikko	neale1: yes, unfortunately Jenkins doesn't support SSL natively with JNLP connections so it would require setting up stunnel
[21:36] mikko	i got a ready configuration for stunnel
[21:36] neale1	mikko: I'm not as paranoid about SSL etc on the Marist system
[21:37] mikko	neale1: ok
[21:37] mikko	neale1: i just want to provide options to people as they are contributing the slaves
[21:37] mikko	you don't need to add TLS but it's there if you want at some point
[21:37] bitweiler	are those packages needed to run zeromq
[21:38] mikko	bitweiler: libuuid from efs2progs is, yes
[21:38] bitweiler	okay
[21:38] neale1	mikko: Understood
[21:38] mikko	hopefully rpm specs and debian packages have a dependency on that
[21:39] bitweiler	slackware does have rpm specs or debian packages
[21:40] sustrik	andrwvc: that's interesting, are you able to reproduce it?
[21:40] neale1	bitweller: libuuid keeps changing packages. In Fedora 13 it's efs2progs but in 14 it's in its own libuuid-devel
[21:43] kristsk	it likes to travel.
[21:43] neale1	It's a pain the arse trying to keep a spec file that will work on all distributions
[21:47] mikko	anyway, time to sleep
[21:47] mikko	good night
[21:48] neale1	'night
[21:49] bitweiler	lucky me slackware has it pre-installed
[21:50] Guthur	I'm making devices with a start and stop, and the way I was going to do it was have poll that would timeout every interval, what would be the opinion of the appropriate balance of performance and response?
[21:50] Guthur	I am currently using a 750ms polling interval
[21:50] Guthur	so at worse case it would take 750ms to stop
[21:58] sustrik	what do you do on the timeout?
[22:03] Guthur	just check if the stop flag has been set
[22:03] Guthur	then go start polling again
[22:05] Guthur	I have a feeling it may actually be necessary when implementing the device in a managed code environment
[22:06] sustrik	who sets the stop flag?
[22:06] Guthur	either the user by calling Stop(), or the Dispose method
[22:06] Guthur	the device is nonblocking
[22:07] sustrik	who calls the Stop() methos?
[22:07] sustrik	method*
[22:07] sustrik	main thread?
[22:07] sustrik	a thread different to the device thread?
[22:07] Guthur	sustrik, you've just given me a bad threading feeling, hehe
[22:07] Guthur	I have a feeling I am doing something bad
[22:08] sustrik	well, if it's invoked from another thread
[22:08] sustrik	you have to synchronise it using a mutex
[22:08] sustrik	or something
[22:08] sustrik	which will kill the performance
[22:08] sustrik	so I would suggest simply calling zmq_term() from the main thread
[22:08] Guthur	I have a a second flag that actually indicates when it has stopped
[22:09] sustrik	which will cause next call to libzmq in the device to return ETERM error
[22:09] sustrik	you can exit the device loop then
[22:09] Guthur	I never considered that option
[22:10] sustrik	ETERM was introduced specifically for this scenario
[22:10] sustrik	shutting down multithreaded programs
[22:10] Guthur	oh, this could make the code a bit nicer
[22:10] sustrik	a much more performant
[22:11] sustrik	as you can avoid mutex that way
[22:37] tiglionabbit	I'm having a bit of trouble wading through the pretentious documentation. Is there a simple multiple-chatrooms example for zeromq?
[22:38] tiglionabbit	like, if I have a variety of incoming long-polling connections each listening for different criteria
[22:43] Steve-o	if someone is bored I have plenty of x86 atomic ops instructions to convert from GCC format to SunPro format, already done MSVC :D
[22:45] Steve-o	ending up almost two screen fulls of code to simply perform number++, lol
[23:01] Guthur	tiglionabbit, do you mean the Guide?
[23:01] tiglionabbit	yes
[23:02] Guthur	I actually thought it was quite reasonable, you could just jump straight to code examples from the guide
[23:02] Guthur	and then refer back to the guide if you get confused
[23:03] Guthur	there is so many patterns possible with 0MQ that it would be impossible to convey properly with just one code example
[23:03] Guthur	IMO
[23:07] Guthur	is there a function that returns the ZMQ_HAUSNUMERO
[23:11] kristsk	ZMQ_NOTAMUSED
[23:11] kristsk	sorry.
[23:14] bitweiler	does zeromq include the mongrel2 server?
[23:15] bitweiler	just trying to see if I need to install that as well
[23:21] Steve-o	bitweiler: no, the core zeromq package is just the sockets, not even the bindings any more
[23:23] bitweiler	oh, okay
[23:41] bitweiler	does zeromq have any other documentation besides the man pages?
[23:42] bitweiler	I ask cause I trying to make a package for slackware
[23:44] Guthur	bitweiler, I think the guide is the only other doc, http://zguide.zeromq.org/chapter:all
[23:44] Guthur	When connecting a Push to a Pull using TCP, does one need a delay before sending?
[23:45] bitweiler	Guthur: thanks
[23:46] bitweiler	things look real promising for this project, i my eyes
[23:47] Guthur	I think it's pretty cool, the guys have produced some cool tech
[23:49] bitweiler	hmm, hopeful everything works smoothly with scheme's c-ffi
[23:49] Guthur	I think there is a racket binding
[23:49] bitweiler	there's one for chicken as well
[23:49] Guthur	and someone was working on one for that chicken scheme i think
[23:49] Guthur	there is also a Common Lisp one
[23:50] bitweiler	I aim to test drive the chicken binding
[23:51] Guthur	On my issue...I seem to be having an issue with REQ<->XREP<-PULL<-PUSH, using TCP
[23:52] Guthur	it was working OK with inproc, now I have place a delay when connecting at the PUSH end