IRC Log


Friday January 21, 2011

[Time] NameMessage
[01:29] nathanmarz hey all - is there any way to do a push socket that does fan-out instead of load-balancing?
[01:29] nathanmarz or should i just have the push node open a socket for every node it needs to send data to and do the send for each receiver
[01:29] nathanmarz ?
[05:01] stockMQ Hi
[05:01] stockMQ I am facing a problem
[05:02] stockMQ I am trying publish a struct
[05:03] stockMQ I do it as follows
[05:03] stockMQ ScripRecord* scrip = &scripStatus[token]; char message[sizeof(*scrip)]; memcpy(&message,scrip, sizeof(scripStatus[token])); s_send (*this->publisher,message);
[05:04] stockMQ where scrip* is the pointer to the strcut
[05:04] stockMQ But the first message i recv at receiver is of size 1
[05:04] stockMQ the struct is
[05:05] stockMQ struct ScripRecord { int test; int token; float open,high,low,close; double volume, volumeTradedToday, dateTime, buyVolume, sellVolume; float bid, ask, openInterest; float dayOpen, dayLow, dayHigh, previousClose; ScripRecord() { memset(this, 0, sizeof(ScripRecord)); } };
[05:05] stockMQ and its size is 92
[05:13] stockMQ Any help ??
[05:32] stockMQ I tried sending a sinple String "Hi"
[05:32] stockMQ and that works fine
[07:11] sustrik sam`: PUB/SUB
[07:11] sustrik nathanmarz: PUB/SUB
[07:13] sustrik stockMQ: s_send is only for sending stings AFAIU
[07:13] sustrik use standard 0mq API
[07:13] sustrik zmq_send()
[07:51] Evet have you ever used zeromq for layer 7 routing?
[08:02] mikko Evet: what do you mean by that?
[08:03] mikko and good morning
[08:04] guido_g 'morning all
[08:10] Evet mikko: zeromq between front-end loadbalancer and back-end servers
[08:10] mikko Evet: that is what mongrel2 does
[08:15] Evet mikko: sure. but i need to implement some other protocols
[08:16] mikko what is the question here actually?
[08:16] mikko is the message coming in from front a zeromq message
[08:17] mikko or are you proxying other protocols?
[08:20] Evet mikko: i want to do what mongrel2 does, but for other protocols. the actual question is
[08:20] Evet im afraid to run multiple event loop in a process
[08:20] Evet one for accepting real world connections, and other for zeromq
[08:21] Evet how can i implement real world protocols to zeromq's event-loop?
[08:21] mikko why?
[08:21] mikko just run them on separate threads?
[08:21] mikko or run a one bigger loop
[08:21] Evet yes, one bigger loop is what i need
[08:21] mikko you can also run two threads
[08:22] mikko and send messages between the threads using inproc://
[08:22] Evet i want to use all cores for that two jobs
[08:23] mikko well, then you need multiple threads in any case i guess
[08:24] mikko is there a large overhead to protocol parsing?
[08:26] Evet mikko: no, almost zero overhead for parsing
[08:26] mikko so you could accept front-end connections in a single thread i assume
[08:28] Evet single cpu core cant respond more than 8k requests per second
[08:28] mikko what does responding mean?
[08:29] Evet the request-reply pattern
[08:29] mikko that depends on what kind of processing needs to be done to generate reply
[08:29] mikko otherwise its just accept, read, write
[08:32] Evet yes, and i need more than one thread for a simple accept-read-write job, which is enough to serve dynamic content through embedded in-memory database
[08:33] mikko isn't it quite common to have one thread accepting and then pass down to worker threads?
[08:33] mikko Evet: if i remember correctly for example memcached performs a lot better when used single-threaded
[08:34] mikko and it sounds very much like memcached you are creating there
[08:37] mikko maybe thats different with UDP
[08:37] mikko (the performance)
[08:43] Evet well
[08:43] Evet inproc:// makes sense now
[08:44] mikko Evet: also, if you have single thread accessing the hash database it probably reduces lock congestion
[08:46] mikko assuming you write as well
[08:46] Evet oh wait, youre also using kyoto cabinet, right?
[08:47] mikko i've used tokyo cabinet/tyrant in the past
[08:47] mikko and now testing kyoto cabinet
[08:48] Evet for caching only?
[08:49] mikko no, persistence
[08:49] Evet do you store relationships?
[08:50] mikko no, not really
[08:53] Evet im building an SQL and SPARQL api on top of kyoto cabinet
[08:53] Evet and planning to use zeromq for backup-restore logic
[08:56] sustrik 8k 0MQ requests per second?
[08:56] sustrik doesn't seem likely unless you are doing it in lock-step manner
[08:57] Evet single core of a single desktop pc
[08:59] stockMQ I am typecasting stuct to char*
[08:59] stockMQ before sending
[08:59] mikko stockMQ: sure, but first null character would cut your char*
[08:59] mikko stockMQ: s_send uses strlen () probably
[09:00] stockMQ ohk.. will pack help?
[09:00] mikko stockMQ: can you show the code?
[09:00] mikko gist.github.com
[09:01] stockMQ i have pasted it before.. coz now i do not have access to it :(
[09:01] mikko are you running the same architecture on all your nodes?
[09:02] mikko as sending a struct is not portable
[09:02] mikko unless you serialize it for transport (protobufs, thrift etc)
[10:36] nathanmarz sustrik_: but my sender already knows who all the recievers are going to be
[10:36] nathanmarz sustrik_: seems simpler to not have to worry about synchronization of pub/sub (so that messages don't get lost)
[15:11] mikko i to the b
[17:59] Evet sustrik_: why do you think 8k request/second is very high for a desktop pc?
[18:05] mikko so finally things working properly
[18:05] mikko http://snapshot.valokuva.org/
[18:05] mikko mingw32 and msvc2008 snapshot
[18:05] Evet congrats
[19:52] mikko sustrik_: seen #153
[19:52] mikko ?
[20:05] sustrik Evet: i think it's too low
[20:06] sustrik the rate should go up to 100,000's or millions, depending on the hardware
[20:06] sustrik 8000 can result from lock-step processing
[20:07] sustrik if the roundtrip time on the network is kind of low (~120us)
[20:07] sustrik mikko: let me see
[20:09] Evet sustrik_: what about inproc and ipc performance?
[20:10] sustrik in the range of millions per sec
[20:10] Evet wow
[20:10] Evet does it use b+tree?
[20:11] sustrik for what?
[20:11] sustrik subscription matching?
[20:11] Evet storing messages in memory
[20:11] Evet and yes, subscription matching
[20:11] sustrik storage in memory is linear, plain fifo
[20:12] sustrik so no need for searching
[20:12] sustrik as for subscription, the algorithm is "trie"
[20:15] sustrik mikko: ok, #153 answered
[20:15] Evet hmm
[20:19] cleifer hi, was having a little bit of trouble with the hello example -- i'm not sure i'm compiling it right, but ("cc -o hello hello.c /usr/local/lib/libzmq.so") it runs and just hangs. connecting to it, sending data, aren't seeming to work
[20:19] Evet sustrik_: is it possible to implement a snapshot to disk mechanism for those fifo
[20:20] sustrik cleifer: are you sure the connection exists? can you check netstat?
[20:21] sustrik Evet: how would that work?
[20:21] cleifer let me do that, just as ec
[20:22] cleifer sustrik_, tcp 0 0 localhost:5555 localhost:48696 ESTABLISHED
[20:22] cleifer and another the other way
[20:22] sustrik looks ok
[20:22] cleifer when i recv from the other, all i get is \x01\x00
[20:23] cleifer and then nothing :-/ -- i see in the code that it'll print when a connection is established but that's not happening
[20:23] sustrik that's connection initialisation
[20:23] Evet sustrik_: it may need a broker
[20:23] cleifer sustrik_, i'm connecting to it in python using a raw socket, maybe that's the problem?
[20:24] sustrik you mean TCP socket?
[20:24] cleifer yah
[20:24] sustrik then you have to follow the 0mq protocol
[20:24] cleifer s.connect(('localhost', 5555))
[20:24] sustrik first, send \x01\x00
[20:25] sustrik that's connection initialisation
[20:25] sustrik then you can exchange messages
[20:25] sustrik the format of message is described in zmq_tcp(7)
[20:25] cleifer coolio
[20:26] cleifer thanks sustrik_ !
[21:02] drbobbeaty Got a question for the ZeroMQ TCP usage... is it possible to have a ZeroMQ server create a TCP connection on an ephermal port and then obtain that port number so that it can provide a good URL to a client? I'm thinking of a use-case where I want to fire up a TCP PUB server, on an arbitrary port and then get that port (and URL) and provide it to clients for their connection. Can this be done?
[21:02] drbobbeaty I'm thinking "No", but I wanted to ask to make sure.
[21:06] sustrik no
[21:24] iDude IOT/Abort trap in pthread_kill at 0xd005f734 ($t1) 0xd005f734 (pthread_kill+0x88) 80410014 lwz r2,0x14(r1)
[21:25] iDude I am using the 2.1.0 version of zmq on AIX 5.3... any ideas to get it working?
[21:28] sustrik hm, i've made AIX port work some time ago
[21:28] sustrik no idea of what version though
[21:28] sustrik can you provide a backtrace for the problem?
[21:31] iDude pthread_kill(??, ??) at 0xd005f734 _p_raise(??) at 0xd005f1a4 raise.raise(??) at 0xd02a38f0 abort() at 0xd0307778 myabort()() at 0xd0244aac terminate()() at 0xd0242df0 terminate()() at 0xd024424c __DoThrowV6() at 0xd02468f0 _Xlen__Q2_3std6vectorXTQ3_3zmq6poll_t10fd_entry_tTQ2_3std9allocatorXTQ3_3zmq6poll_t10fd_entry_t__CFv(0x2000b990) at 0xd6bccb10 insert__Q2_3std6vectorXTQ3_3zmq6poll_t10fd_entry_tTQ2_3std9allocatorXTQ3_3zmq6poll_t10fd
[21:31] iDude sorry... I am new to the whole IRC chatting thing... how can I format the text to show on multiple lines?
[21:37] sustrik paste the baktrace to gist or somesuch and provide the link
[21:47] iDude here you go: https://gist.github.com/790486
[21:48] iDude I have included the source code and the Makefile used