Tuesday February 8, 2011

[Time] NameMessage
[01:50] Steve-o mikko: the x1024 patch on PGM 'works' but isn't correct, I'll have to look at Ian's suggestion tomorrow
[01:51] Steve-o all day at IKEA today, patience is a virtue as is said.
[01:57] d4de Hi, any idea on when aprox. v2.1.0 will go stable?
[02:00] Steve-o good question, probably dependent on the issue list on github getting cleaned up a bit
[02:01] d4de link?
[02:02] d4de meh nvm ... I think google can answer that one :-)
[02:02] Steve-o
[02:02] d4de ooh well ... thanks :-D
[02:03] Steve-o there are a couple of weird platform dependent / compiler optimisation issues with the PGM socket
[02:03] Steve-o and there has been issues with zmq_term
[02:03] d4de so it is just PGM that is putting it back ... I saw that you guys has already fixed some annoying bugs in 2.1 beta, so I keep wondering what is left
[02:03] Steve-o you can always ask Martin again on the list
[02:04] d4de hmm still no IPv6 support huh?
[02:04] Steve-o well yes and no
[02:05] Steve-o there probably needs to be extra flags or specification for using IPv4 or IPv6, or both
[02:05] Steve-o similar to BSD's IPV6ONLY
[02:06] d4de aha
[02:06] Steve-o binding to eth0:* tends to be complicated in a multi-stack environment
[02:07] Steve-o surprisingly PGM has better support for IPv6 but it depends on reasonable OS support
[02:08] Steve-o there are some issues with the glibc in Ubuntu 10.10 there are definitely unexpected
[02:08] Steve-o for example ::1 resolves as
[02:08] Steve-o getaddrinfo (`hostname`) returns different address to ping `hostname` when using DHCP
[02:09] Steve-o one easy path is to assume IPv4 only and follow SMTP's requirement to use a IPv6: prefix
[02:11] Steve-o I have to concede I only follow the PGM parts of ZeroMQ, not so much the other parts
[02:12] Steve-o There is certainly work on improving ZMQ_SWAP and XPUB/XSUB sockets
[06:18] stockMQ Hi.. I have a binary char* which i want to send from one Thread to Another.. i am trying inproc.. sending the char* as it is as a message does not work as at the receiving end while decoding i get erroneous data..I understand i have to use protobuf..but then can i serialize to string
[06:42] sustrik you have to copy the data to the message
[07:38] stockMQ I want to send a Byte* over a socket.. What would be the right way to do it
[07:53] Guthur stockMQ, do you mean send the pointer?
[08:54] Guthur I am sorry, I've busted the clrzmq2 build again, this MSVS solution stuff is utter crap
[09:05] mikko good morning
[09:52] sustrik morning
[13:11] Skaag BAHAHAHAHA.... IT'S ALIVE!!!
[13:11] Skaag The horror!
[13:14] sustrik run!
[13:14] sustrik :)
[13:19] ianbarber another epgm question: if I, in one thread, do a PUB send, then a SUB recv, using PGM, that should work right?
[13:21] sustrik ianbarber: that's using loopback multicast
[13:21] Guthur is the lack of IPC on windows more a lack of motivation issue as oppose to technical?
[13:21] sustrik in reality, multicast doesn't work in loopback manner
[13:22] sustrik so there's an emulation layer in the OS that simulates it
[13:22] ianbarber oh right
[13:22] sustrik however, as i am said, it's pretty crappy
[13:22] sustrik so rather use ipc/tcp for loopback
[13:22] sustrik Guthur: both in a way
[13:22] ianbarber that makes sense, i mean it's not something that is a good idea other than for testing stuff
[13:23] sustrik ack
[13:23] sustrik Guthur: to implement IPC on windows you need NamedPipes
[13:23] sustrik these are identified by handles rather than fds
[13:24] sustrik so you can't use select() for polling
[13:24] sustrik thus you have to implement IOCP polling mechanism
[13:24] Guthur oh, I should have none it was a simple issue, hehe
[13:25] sustrik well, it's doable
[13:25] Skaag what happens when one client out of 20, is down for an hour?
[13:25] Skaag it loses all data, right?
[13:25] sustrik Skaag: if no identity is set, then yes
[13:25] Skaag and if identity is set?
[13:26] sustrik then only the data that were "on the wire" during the failure are lost
[13:26] sustrik the subsequent messages are queued in the publisher
[13:26] Skaag and are saved in memory?
[13:26] sustrik yes
[13:26] Skaag and if the publisher died too, they are lost?
[13:26] sustrik sure thing
[13:26] Skaag ok :)
[13:27] Skaag i'm trying to figure out how to keep 20+ machines in sync with the same data
[13:27] Skaag they all get small incremental updates via zmq
[13:27] Skaag but it accumulates to quite a lot
[13:28] sustrik ask for a snapshot after starting the client using req/rep
[13:28] sustrik then subscribe to incremental upgrades using pub/sub
[13:31] Skaag sounds simple enough
[13:32] Skaag the remaining problem is knowing who to ask the snapshot from
[13:32] Skaag who is the leader with the best snapshot
[13:32] Skaag probably the one with the longest uptime
[13:32] mikko are you using zookeeper in the setup?
[13:32] Skaag not anymore
[13:32] Skaag I can't use zookeeper because it requires a fixed number of nodes
[13:33] mikko why is that?
[13:33] Skaag when when nodes are added, a configuration file needs to be edited, and the whole thing restarted
[13:33] mikko you mean to zookeeper itself?
[13:33] Skaag I followed a thread on their list, and found a person started working on fixing this
[13:33] Skaag yes to zookeeper itself
[13:33] Skaag anyway his work has stalled
[13:34] Skaag and he offered someone else to continue working on it, and the ETA is 3 ~ 5 month to implement
[13:34] Skaag why, no idea..
[13:35] Guthur sustrik: why not just use sockets for IPC
[13:36] Guthur sockets can be polled with select, or am I missing something big here
[13:36] sustrik Guthur: nope
[13:36] sustrik you can use sockets
[13:36] sustrik just use tcp://
[13:38] sustrik Skaag: choosing a leader in a distributed env is a complex problem...
[13:38] Guthur are pipes more performant?
[13:38] sustrik you should peek into some disttibuted algorithm textbook
[13:38] zchrish mikko: You referred me to for a mingw dll for Windows build. Is there a library option also available?
[13:38] sustrik Guthur: never measured it
[13:38] sustrik but it would make sense if they were
[13:39] sustrik Skaag: especially beware of split-brain syndrome
[13:40] Skaag right, I heard about that
[13:40] Skaag a problem that exists in the glusterfs project
[13:40] Skaag (which I also use)
[13:40] sustrik it's when the network is broken into two parts
[13:40] sustrik each part selects its own leader
[13:40] sustrik resulting in inconsistent state
[13:41] Skaag can run a competition with set rules, and the winner gets its data replicated to everybody
[13:41] sustrik to prevent the split-brain the competition could be won only by a node that gets more than 50% of the vote
[13:42] sustrik to compute 50% you need to know the size of the cluster
[13:42] sustrik it's complex...
[13:45] mikko zchrish: what do you mean?
[13:45] mikko zchrish: static library?
[13:46] zchrish Yes, a static library.
[13:46] mikko zchrish: no, we don't support building static libraries under mingw32
[13:46] zchrish Oh, I see. Why not?
[13:46] mikko zchrish: it's a massive hassle with the dllexport/dllimport stuff
[13:46] mikko on windows
[13:47] zchrish Sure, ok. Thank you.
[13:47] mikko you can probably build libs by hacking the build scripts
[13:47] mikko but to keep things somewhat sane for mingw32/visual studio we only support dlls
[13:48] CIA-21 zeromq2: 03Martin Sustrik 07master * r5dcbc34 10/ src/zmq.cpp :
[13:48] CIA-21 zeromq2: zmq_poll with no fds behaves decently
[13:48] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[13:48] zchrish But I see a .lib in the msvc2008 output.
[13:49] mikko thats the import library
[13:49] zchrish Oh, ok.
[14:00] zchrish mikko: Is WIN7 required to use the DLL?
[14:02] ianbarber sustrik/mikko: any thoughts on best way to debug if I see PGM packets from wireshark on my reciever host, but am not getting anything from 0MQ listening on that host
[14:07] sustrik check whether the multicast addresses and ports match on publisher and receiver
[14:08] ianbarber yep, definitely match
[14:08] ianbarber subscribe is ""
[14:08] ianbarber (as well)
[14:09] sustrik then you can place breakpoint to pgm_socket_t::recv()
[14:09] sustrik to see whether the data are getting from openPGM to 0MQ
[14:16] signalseeker Is a pdf version of the zmq guide available?
[14:17] mikko zchrish: i wouldn't think so
[14:17] zchrish OK.
[14:18] mikko might need the visual studio 2008 redist libs
[14:18] ianbarber sustrik: it's not breaking in there, so it looks like 0MQ isn't seeing the packets
[14:33] sustrik ianbarber: ask steve-o then
[14:33] ianbarber steve-o: is there anything to check whether libpgm can see the packets on a host, for example? I've can see them in wireshark, but 0MQ doesn't seem to be getting them.
[14:33] ianbarber was just about to, and did :)
[14:34] Steve-o mmm
[14:34] Steve-o You can try increasing the log level, i.e. PGM_MIN_LOG_LEVEL=TRACE
[14:35] ianbarber where would the logging go?
[14:36] Steve-o stdout
[14:36] Steve-o is it completely silent in zeromq or a little bit of data then silence?
[14:36] ianbarber ah
[14:36] ianbarber Trace: Discarded packet for muted receiver
[14:37] ianbarber awesome!
[14:37] ianbarber now, why would it do that
[14:37] ianbarber :)
[14:39] Steve-o a sender would see that on data packets
[14:39] Steve-o possible cause is multiple active senders
[14:40] Steve-o it's not expected for a SUB socket
[14:41] Steve-o only PUBs
[14:45] ianbarber hmm, that was definitely on a sub
[14:45] ianbarber but maybe there's something rogue running as well
[14:47] Steve-o it is only emitted if the socket is not permitted to receive data
[14:48] Steve-o so a SUB socket that cannot recv sounds a bit broken
[14:48] ianbarber hmm, that's strange
[14:48] ianbarber nothing else running
[14:48] ianbarber definitely only triggers when I'm sending on the other host
[14:48] ianbarber if i don't send, no output
[14:49] Guthur If i was wanting to learn what interface a windows IPC implementation should have where is the best places to look in the source code?
[14:50] ianbarber it does mention a send interface as well steve-o
[14:50] ianbarber Trace: Create transmit window.
[14:50] ianbarber Trace: Binding receive socket to INADDR_ANY
[14:50] ianbarber Trace: Binding send socket to interface index 2
[14:50] ianbarber Trace: Join multicast group on interface index 2
[14:50] ianbarber Trace: Multicast send interface set to index 2
[14:50] ianbarber is that normal?
[14:50] Steve-o ianbarber: PGM requires a back-channel from recv to sender for reliability
[14:50] ianbarber oh, of course
[14:50] ianbarber silly me
[14:51] ianbarber if you have a sec, could you look at and see if I'm doing something obviously dumb?
[14:51] Steve-o on Linux or Windows?
[14:52] Steve-o Linux flavour I'm guessing
[14:52] ianbarber linux
[14:53] Steve-o looks reasonable although you may hit this bug:
[14:53] sustrik Guthur: you would have to start with implementing IOCP poller
[14:54] sustrik see select_t, poll_t, epoll_t, devpoll_t and kqueue_t
[14:54] sustrik you would have to make something similar
[14:54] sustrik presumably called iocp_t
[14:54] Steve-o ianbarber: sleeping before zmq_term can usually help diagnosis, basically keeping the socket open a bit longer.
[14:54] Steve-o ianbarber: or sending more data.
[14:55] Guthur sustrik: ok, thanks
[14:55] ianbarber steve-o: I've got the sender sending then sleeping for 1 second, in a loop, so it's c onstant stream of data. At the moment, I just wait on recv in the test code
[14:56] ianbarber each send triggers about 4-5 of those muted receiver notices
[14:56] Guthur from MS docs local named pipes would be considerably faster than sockets
[14:56] Guthur so maybe worth the look
[14:56] Steve-o ianbarber: send me some code,, I'm working on Windows this morning and I'll have a look later
[14:57] ianbarber will do, thanks steve
[14:57] sustrik Gurthur: yes, definitely
[15:17] neale1 sustrik: Have you seen this whitepaper:
[15:18] drbobbeaty Question: Let's say I am using ZMQ/OpenPGM on the epgm: protocol... also let's assume that I have several multicast channels sending data. On the client, is it better to have a single zmq::context_t with 1 thread working for a series of zmq::socket_t connections, each ZMQ_SUB-ed to different multicast channels? Or would it be better to have a 1:1 mapping between the zmq::context_t and it's zmq::socket_t?
[15:21] CIA-21 zeromq2: 03Martin Sustrik 07master * r889424e 10/ src/ctx.cpp :
[15:21] CIA-21 zeromq2: max_sockets honoured precisely
[15:21] CIA-21 zeromq2: The internal log socket was subtracted from the number of
[15:21] CIA-21 zeromq2: available sockets. So, if max_sockets was set to 100,
[15:21] CIA-21 zeromq2: you could create only 99 sockets. Fixed.
[15:21] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[15:34] Guthur what is the default max number of sockets per context?
[15:34] Guthur and what happens if you try to exceed that
[15:35] Guthur oh I've just answered my own question
[15:38] drbobbeaty I *believe* the context is a "one per process space" entity, so I *believe* you have one context for any number of sockets in the process, but I just wanted a verification from the designers.
[16:02] sustrik Guthur: 512
[16:03] Guthur sustrik: is it possible to set it higher?
[16:03] sustrik neale1: yep, it's an old one
[16:04] sustrik i think there is some discussion with opendds guys on the mailing list or somewhere
[16:04] sustrik Guthur: sure
[16:05] sustrik drbobbeaty: confirmed
[16:06] drbobbeaty sustrik: thank you very much for the confirmation. Appreciated.
[16:07] Guthur sustrik: I did a search for max_sockets in the src and couldn't find it
[16:07] Guthur is it a build variable
[16:08] sustrik src/confing.hpp
[16:10] Guthur all there is is max_app_threads = 512,
[16:11] Guthur is that equivalent?
[16:12] mikko Guthur:
[16:13] Guthur umm thats called max_app_threads = 512, in my source
[16:13] mikko 2.0.x ?
[16:14] Guthur ah, yeah
[16:28] sustrik the theading has changed between 2.0 and 2.1
[16:29] sustrik with 2.0 the limit is on *threads* using 0mq sockets
[16:29] sustrik in 2.1 the limit is on the number of the sockets themselves
[16:30] Guthur I think this may explain a mysterious app crash I was having
[16:30] mikko Guthur: on windows?
[16:31] Guthur yep
[16:32] Guthur but it should have thrown an exception
[16:32] Guthur might have been another cause
[16:34] mikko running out of filehandles?
[16:35] mikko i think there was some shenanigans with filedescriptor count on windows
[16:37] kristsk like max 512 per process or so i've heard
[16:37] mikko i think it was way less by default
[16:37] mikko 64 even
[16:38] kristsk thats not much.
[16:38] mikko
[16:39] sustrik 64 is the default max descriptors you can poll on on win32
[16:39] mikko ah ok
[16:39] sustrik i've overloaded the value to 1024 recently
[16:53] Guthur I'm not sure what my issue was, the app just silently shutdown, not very helpful at all
[16:55] sustrik not even an assertion?
[16:55] Guthur nothing, it may not have been ZMQ
[16:55] Guthur there was quickfix as well
[16:56] sustrik i see
[16:59] Guthur well, home time now
[17:09] stockMQ Hi.. I am receiving protobuf messages from my zmq socket which i then want to serialize to a file as binary.. this file in turn should be readable by other processes simultaneously..
[17:09] stockMQ Any pointers?
[17:10] mikko one file per message or all messages in one file?
[17:11] stockMQ all messages in one file.. the messages are sequentially timestamped
[17:12] stockMQ basically the writer will just dump all incoming messages to the file
[17:12] stockMQ but the reader processes should be able to read and de serialize them
[17:13] mikko how does the reader know that message is not impartial?
[17:14] stockMQ by impartial what do you mean?can you elaborate please
[17:15] mikko if the writer is writing a message how do the readers know that the message has been fully written?
[17:15] mikko are you planning to use locking?
[17:15] stockMQ No..basically what i was doing till now is
[17:15] mikko and if yes how are you going to sync readers/writers?
[17:15] stockMQ intradayFile.Open(dataPath,CFile::modeWrite | CFile::modeNoTruncate | CFile::modeCreate | CFile::shareDenyWrite | CFile::osWriteThrough)
[17:16] mikko because you could easily "DoS" yourself by having high rate of messages coming in
[17:16] mikko in which case the lock wouldnt be released
[17:16] sustrik i would just use a DB
[17:16] stockMQ This basically is supposed to allow Shared read.. if the reader process opens the file with appropriate permission
[17:17] stockMQ The synchronization is supposed to be handled by the OS
[17:17] stockMQ I am working on VC++
[17:18] sustrik the OS won't handle the sync for you
[17:18] sustrik you have to implement it yourself
[17:18] sustrik that's why it's better to use a DB
[17:18] stockMQ ok
[17:19] mikko i think posix will guarantee that write within a process is atomic
[17:19] mikko but thats about it
[17:19] stockMQ ok
[17:19] mikko between processes
[17:19] mikko but that doesn't really help you with readers at all
[17:20] mikko and not sure how posix windows is
[17:20] mikko this is what posix says:
[17:20] mikko "Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. "
[17:21] stockMQ My understanding was that CFile::ShareDenyWrite is a flag that tells the operating system not to permit any other process to write to your file while you have it open.
[17:21] mikko stockMQ: it will probably but what about if you have a constant stream of messages coming in?
[17:22] stockMQ and shareRead would allow it to be read simultaneously
[17:22] stockMQ yes u r right.. i had that in mind.. but wanted to check that heuristically
[17:22] mikko writer will block the readers
[17:23] mikko usually
[17:23] mikko if you have readers reading while the writer is writing you might end up with partial / incomplete data
[17:23] mikko then you need to track if the message is fully written or not
[17:24] neale1 mikko: did you want to try enabling the clrzmq build for s390x to see how it goes?
[17:24] mikko neale1: i can at some point, still at work for about an hour
[17:24] neale1 no worries
[17:25] stockMQ yes u r right.. Hence i planned to first write it to a file..see how the readers behave and then switch to a DB.. to learn along the process as well
[17:26] stockMQ I have received suggestions here like Kyoto cabinet,BDB etc.. More inputs/suggestions are welcome..Also it would be helpful if there is anyone who has written protobufs to a file can share his/her experience
[17:29] mikko do you need to keep this data?
[17:29] stockMQ yes
[17:29] stockMQ basically these are stock trade data..
[17:29] stockMQ they need to be kept for back analysis
[17:29] mikko and this is on windows?
[17:30] stockMQ and there could be more than one analysis tool accessing the data and hence the parallel readers
[17:30] stockMQ correct
[17:31] mikko do you want to keep the raw data or the results of the analysis?
[17:31] stockMQ The raw data
[17:31] stockMQ the analysis tools they basically read the their stuff and display
[17:31] mikko so you want to be able to get historic data for time periods etc?
[17:31] stockMQ yess
[17:31] mikko do you want to be able to query the data?
[17:31] stockMQ till now i was copying the protobufs to struct and then writing the structs
[17:32] stockMQ now i want to bypass the structs and directly write the protobufs
[17:33] mikko the data you are receiveing, what kind of data is it?
[17:33] stockMQ yes.. query for the time window..nothing complex
[17:33] stockMQ as for nw
[17:33] mikko numbers, text?
[17:33] stockMQ both..mostly numbers.. text are just names of the Stock scrip
[17:34] mikko what sort of consistency requirements do you have for the data?
[17:35] mikko or maybe more accurate question would be resilience requirements
[17:36] mikko for example is it ok to fsync to disk every 10 seconds and if your system crashes you lose 10 seconds of data currently in buffers?
[17:36] stockMQ well.. if it is better not to lose any data :)
[17:36] stockMQ the analysis are going to be sensitive to loss of data
[17:39] mikko it's not an easy engineering task in that case
[17:41] mikko does the data need to be consistent on all consumers all the time?
[17:41] stockMQ I understand.. In fact to convince the management to go for DB.. i need to first implement a prototype writing it to a file and show them how the readers fair.
[17:41] kristsk CAP is a bitch.
[17:42] mikko yes, i was thinking this in terms of cap theorem
[17:42] stockMQ yes. basically more than one instance of a tool would be accessing it
[17:42] stockMQ so yes
[17:42] stockMQ CAP? please enlighten
[17:43] mikko
[17:43] kristsk consistency, availability, partitioning - choose any two.
[17:44] stockMQ C and A
[17:44] ianbarber stockMQ: you might have more success in writing straight out to a file, less disk seeks
[17:44] ianbarber the kafka guys talk about that a bit on their site
[17:45] mikko ianbarber: the problematic thing is that the processes reading the files are external
[17:45] mikko unlikely with kafka
[17:45] kristsk kafka ?
[17:45] mikko
[17:46] ianbarber mikko: oh, i see. I was thinking write file for backanalysis, 0MQ for distribution
[17:46] kristsk oh.
[17:46] mikko ianbarber: the process writing file is a subscriber as far as i understood
[17:46] mikko and the analysis tools read the file and create reports and such
[17:47] mikko so what you are effectively creating is a database
[17:47] stockMQ that is right Mikko
[17:47] ianbarber yeah, sounds like a database and replication
[17:48] mikko so, without knowing the application / architecture very deeply somehow i would be inclined to say that it makes sense to investigate existing database technologies before going through all the hurdles
[17:49] mikko maybe you have volatile database for quick access to data and push to another queue which warehouses the data
[17:49] mikko very hard to say without knowing much about the application or the overall architecture
[17:50] stockMQ In a nutshell.. The exchange multicasts feed which contains the Intra day data.. I have intercepted this feed and then implemented a PUB
[17:51] kristsk uh, kafka is java-ish...
[17:52] stockMQ The SUBs need to dump this data somewhere.. which should be accessible to the Analysis tools during market hours for intra day analysis and after market hours for back analysis
[17:52] stockMQ After market hours, the write will not be it will be only reads.
[17:52] stockMQ The hurdle is during market hours
[17:53] kristsk umm, isnt pub/sub kinda "you weren't here, you missed, sorry" ?
[17:53] stockMQ yes that is right
[17:53] stockMQ hence the PUB also need to keep a copy of the data
[17:53] mikko pub/sub is like a radio broadcast (a metaphore sustrik tends to use)
[17:54] mikko when your receiver is on you will receive the broadcast
[17:54] stockMQ If any SUB failed to connect on should be able to borrow the missed data from PUB
[17:54] mikko you can use identities for the subscribers
[17:55] kristsk how many subscribers? maye you could make a queue per "subscriber"?
[17:56] stockMQ Well hard to start with internally it will be around 10.. but it can scale quickly
[17:57] kristsk umyeah, ten-folding data probably is not a good idea.
[17:57] mikko so effetively you want separate req/rep socket on the publisher where subscribers can ask for a delta
[17:58] mikko is subscriber is offline for let's say 10 minutes it could ask the publisher through REQ for the last ten minutes
[17:58] mikko to get the missing pieces
[17:59] stockMQ yes .. that is going to be phase 2.. for now the market hours are known to all.. so the SUBs take up the responsiblity of listening before the market opens
[17:59] stockMQ yes somewhat like that
[17:59] mikko anyway, gotta commute home ->
[18:00] stockMQ ohkk..can i catchu later?
[18:04] sustrik stockMQ: you should also be concerned about HD performance
[18:04] sustrik if the feed is fast, as it tends to be with market data
[18:04] sustrik you may not be able to write the data to the HD fast enough
[18:05] stockMQ ok
[18:06] stockMQ So what would be the right approach in your opinion
[18:07] sustrik depends on details
[18:07] sustrik like, are you NASDAQ?
[18:07] sustrik if so, a single HD won't be enough
[18:08] sustrik if you are just a small trading firm getting modest amount of market data
[18:08] sustrik it may suffice though
[18:08] kristsk SSD then heh
[18:08] sustrik SSD is better in terms of latencies
[18:09] sustrik the throughput is similar to standard HD
[18:09] sustrik what you need is either software load balancer
[18:09] sustrik or a RAID
[18:09] sustrik SAN maybe...
[18:09] stockMQ ok
[18:10] sustrik in any case i would not try to implement a database myself
[18:10] kristsk netwrokish san would actually ensure no data gets lost
[18:10] sustrik it takes tens of manyears to do it properly
[18:12] stockMQ Actually the Interceptor aggregates the data and then Publishes it.. so the data received by the SUB will be lesser and slower than the actual feed received by the interceptor
[18:13] stockMQ Any suggestions for DBs.. open source
[18:14] kristsk pg ?
[18:14] sustrik given you need very little functionality, i would check those new nosql projects
[18:14] kristsk postgre
[18:15] stockMQ like Kyoto cabinet?
[18:15] stockMQ or
[18:15] stockMQ
[18:15] stockMQ any users here already?
[18:20] stockMQ I think Mikko had used Kyoto
[18:28] stockMQ so to summarize.. DB seems to be the only sane solution.. Is that right
[19:03] stockMQ What would be the behaviour of the sockets if it is kept idle say overnight..
[19:06] Guthur One would hope it is still there in the morning
[20:06] cremes Guthur: max_sockets in src/config.hpp
[20:06] cremes it defaults to 512; i usually override it to 10k for my own use
[20:33] Guthur cremes, cheers, I was looking in the wrong src version, pre-2.1 it was just a thread count
[20:34] cremes Guthur: ah, ok
[20:44] andrewvc any idea what to make of nbytes != -1 (mailbox.cpp:241) errors in zmq?
[21:32] mikko neale1: there?
[21:33] neale1 mikko: Yes
[21:33] neale1 mikko: BTW, where r u located?
[21:33] mikko neale1: London
[21:33] mikko London United Kingdom, not London Canada
[21:33] neale1 mikko: Better not mention the cricket - I'm an Australian living in the US
[21:34] mikko I'm a Finn living in London and know nothing about cricket
[21:34] mikko heh
[21:34] taotetek gah. anyone here have some pointers on how one could make a pull socket appear as a file (stringio object) using python with pyzmq? I suppose that's more of a python question than a zeromq question.
[21:34] neale1 :-)
[21:34] Guthur Australians, sure they are always winning the Ashes
[21:34] Guthur ...oh wait
[21:34] mikko so anyway, i added a possibility for SSL tunneling the Hudson slave connection to add transport layer security to the connection
[21:34] Guthur hehe, only teasing, I don't much care for cricket
[21:35] mikko currently the master-slave connection goes over plain text which is not ideal as the master executes commands on the slave
[21:35] neale1 It's okay, the poms have waited 20 years to get back at us
[21:35] neale1 mikko: Do I need to do anything from my end?
[21:35] Guthur neale1, Yeah, sure you're just lending it to them for a while
[21:36] bitweiler hey what is this uuid-dev package & uuid/e2fsprogs?
[21:36] mikko neale1: yes, unfortunately Jenkins doesn't support SSL natively with JNLP connections so it would require setting up stunnel
[21:36] mikko i got a ready configuration for stunnel
[21:36] neale1 mikko: I'm not as paranoid about SSL etc on the Marist system
[21:37] mikko neale1: ok
[21:37] mikko neale1: i just want to provide options to people as they are contributing the slaves
[21:37] mikko you don't need to add TLS but it's there if you want at some point
[21:37] bitweiler are those packages needed to run zeromq
[21:38] mikko bitweiler: libuuid from efs2progs is, yes
[21:38] bitweiler okay
[21:38] neale1 mikko: Understood
[21:38] mikko hopefully rpm specs and debian packages have a dependency on that
[21:39] bitweiler slackware does have rpm specs or debian packages
[21:40] sustrik andrwvc: that's interesting, are you able to reproduce it?
[21:40] neale1 bitweller: libuuid keeps changing packages. In Fedora 13 it's efs2progs but in 14 it's in its own libuuid-devel
[21:43] kristsk it likes to travel.
[21:43] neale1 It's a pain the arse trying to keep a spec file that will work on all distributions
[21:47] mikko anyway, time to sleep
[21:47] mikko good night
[21:48] neale1 'night
[21:49] bitweiler lucky me slackware has it pre-installed
[21:50] Guthur I'm making devices with a start and stop, and the way I was going to do it was have poll that would timeout every interval, what would be the opinion of the appropriate balance of performance and response?
[21:50] Guthur I am currently using a 750ms polling interval
[21:50] Guthur so at worse case it would take 750ms to stop
[21:58] sustrik what do you do on the timeout?
[22:03] Guthur just check if the stop flag has been set
[22:03] Guthur then go start polling again
[22:05] Guthur I have a feeling it may actually be necessary when implementing the device in a managed code environment
[22:06] sustrik who sets the stop flag?
[22:06] Guthur either the user by calling Stop(), or the Dispose method
[22:06] Guthur the device is nonblocking
[22:07] sustrik who calls the Stop() methos?
[22:07] sustrik method*
[22:07] sustrik main thread?
[22:07] sustrik a thread different to the device thread?
[22:07] Guthur sustrik, you've just given me a bad threading feeling, hehe
[22:07] Guthur I have a feeling I am doing something bad
[22:08] sustrik well, if it's invoked from another thread
[22:08] sustrik you have to synchronise it using a mutex
[22:08] sustrik or something
[22:08] sustrik which will kill the performance
[22:08] sustrik so I would suggest simply calling zmq_term() from the main thread
[22:08] Guthur I have a a second flag that actually indicates when it has stopped
[22:09] sustrik which will cause next call to libzmq in the device to return ETERM error
[22:09] sustrik you can exit the device loop then
[22:09] Guthur I never considered that option
[22:10] sustrik ETERM was introduced specifically for this scenario
[22:10] sustrik shutting down multithreaded programs
[22:10] Guthur oh, this could make the code a bit nicer
[22:10] sustrik a much more performant
[22:11] sustrik as you can avoid mutex that way
[22:37] tiglionabbit I'm having a bit of trouble wading through the pretentious documentation. Is there a simple multiple-chatrooms example for zeromq?
[22:38] tiglionabbit like, if I have a variety of incoming long-polling connections each listening for different criteria
[22:43] Steve-o if someone is bored I have plenty of x86 atomic ops instructions to convert from GCC format to SunPro format, already done MSVC :D
[22:45] Steve-o ending up almost two screen fulls of code to simply perform number++, lol
[23:01] Guthur tiglionabbit, do you mean the Guide?
[23:01] tiglionabbit yes
[23:02] Guthur I actually thought it was quite reasonable, you could just jump straight to code examples from the guide
[23:02] Guthur and then refer back to the guide if you get confused
[23:03] Guthur there is so many patterns possible with 0MQ that it would be impossible to convey properly with just one code example
[23:03] Guthur IMO
[23:07] Guthur is there a function that returns the ZMQ_HAUSNUMERO
[23:11] kristsk ZMQ_NOTAMUSED
[23:11] kristsk sorry.
[23:14] bitweiler does zeromq include the mongrel2 server?
[23:15] bitweiler just trying to see if I need to install that as well
[23:21] Steve-o bitweiler: no, the core zeromq package is just the sockets, not even the bindings any more
[23:23] bitweiler oh, okay
[23:41] bitweiler does zeromq have any other documentation besides the man pages?
[23:42] bitweiler I ask cause I trying to make a package for slackware
[23:44] Guthur bitweiler, I think the guide is the only other doc,
[23:44] Guthur When connecting a Push to a Pull using TCP, does one need a delay before sending?
[23:45] bitweiler Guthur: thanks
[23:46] bitweiler things look real promising for this project, i my eyes
[23:47] Guthur I think it's pretty cool, the guys have produced some cool tech
[23:49] bitweiler hmm, hopeful everything works smoothly with scheme's c-ffi
[23:49] Guthur I think there is a racket binding
[23:49] bitweiler there's one for chicken as well
[23:49] Guthur and someone was working on one for that chicken scheme i think
[23:49] Guthur there is also a Common Lisp one
[23:50] bitweiler I aim to test drive the chicken binding
[23:51] Guthur On my issue...I seem to be having an issue with REQ<->XREP<-PULL<-PUSH, using TCP
[23:52] Guthur it was working OK with inproc, now I have place a delay when connecting at the PUSH end