ZeroMq IRC Log

Friday January 21, 2011

[Time] Name	Message
[01:29] nathanmarz	hey all - is there any way to do a push socket that does fan-out instead of load-balancing?
[01:29] nathanmarz	or should i just have the push node open a socket for every node it needs to send data to and do the send for each receiver
[01:29] nathanmarz	?
[05:01] stockMQ	Hi
[05:01] stockMQ	I am facing a problem
[05:02] stockMQ	I am trying publish a struct
[05:03] stockMQ	I do it as follows
[05:03] stockMQ	ScripRecord* scrip = &scripStatus[token]; char message[sizeof(scrip)]; memcpy(&message,scrip, sizeof(scripStatus[token])); s_send (this->publisher,message);
[05:04] stockMQ	where scrip* is the pointer to the strcut
[05:04] stockMQ	But the first message i recv at receiver is of size 1
[05:04] stockMQ	the struct is
[05:05] stockMQ	struct ScripRecord { int test; int token; float open,high,low,close; double volume, volumeTradedToday, dateTime, buyVolume, sellVolume; float bid, ask, openInterest; float dayOpen, dayLow, dayHigh, previousClose; ScripRecord() { memset(this, 0, sizeof(ScripRecord)); } };
[05:05] stockMQ	and its size is 92
[05:13] stockMQ	Any help ??
[05:32] stockMQ	I tried sending a sinple String "Hi"
[05:32] stockMQ	and that works fine
[07:11] sustrik	sam`: PUB/SUB
[07:11] sustrik	nathanmarz: PUB/SUB
[07:13] sustrik	stockMQ: s_send is only for sending stings AFAIU
[07:13] sustrik	use standard 0mq API
[07:13] sustrik	zmq_send()
[07:51] Evet	have you ever used zeromq for layer 7 routing?
[08:02] mikko	Evet: what do you mean by that?
[08:03] mikko	and good morning
[08:04] guido_g	'morning all
[08:10] Evet	mikko: zeromq between front-end loadbalancer and back-end servers
[08:10] mikko	Evet: that is what mongrel2 does
[08:15] Evet	mikko: sure. but i need to implement some other protocols
[08:16] mikko	what is the question here actually?
[08:16] mikko	is the message coming in from front a zeromq message
[08:17] mikko	or are you proxying other protocols?
[08:20] Evet	mikko: i want to do what mongrel2 does, but for other protocols. the actual question is
[08:20] Evet	im afraid to run multiple event loop in a process
[08:20] Evet	one for accepting real world connections, and other for zeromq
[08:21] Evet	how can i implement real world protocols to zeromq's event-loop?
[08:21] mikko	why?
[08:21] mikko	just run them on separate threads?
[08:21] mikko	or run a one bigger loop
[08:21] Evet	yes, one bigger loop is what i need
[08:21] mikko	you can also run two threads
[08:22] mikko	and send messages between the threads using inproc://
[08:22] Evet	i want to use all cores for that two jobs
[08:23] mikko	well, then you need multiple threads in any case i guess
[08:24] mikko	is there a large overhead to protocol parsing?
[08:26] Evet	mikko: no, almost zero overhead for parsing
[08:26] mikko	so you could accept front-end connections in a single thread i assume
[08:28] Evet	single cpu core cant respond more than 8k requests per second
[08:28] mikko	what does responding mean?
[08:29] Evet	the request-reply pattern
[08:29] mikko	that depends on what kind of processing needs to be done to generate reply
[08:29] mikko	otherwise its just accept, read, write
[08:32] Evet	yes, and i need more than one thread for a simple accept-read-write job, which is enough to serve dynamic content through embedded in-memory database
[08:33] mikko	isn't it quite common to have one thread accepting and then pass down to worker threads?
[08:33] mikko	Evet: if i remember correctly for example memcached performs a lot better when used single-threaded
[08:34] mikko	and it sounds very much like memcached you are creating there
[08:37] mikko	maybe thats different with UDP
[08:37] mikko	(the performance)
[08:43] Evet	well
[08:43] Evet	inproc:// makes sense now
[08:44] mikko	Evet: also, if you have single thread accessing the hash database it probably reduces lock congestion
[08:46] mikko	assuming you write as well
[08:46] Evet	oh wait, youre also using kyoto cabinet, right?
[08:47] mikko	i've used tokyo cabinet/tyrant in the past
[08:47] mikko	and now testing kyoto cabinet
[08:48] Evet	for caching only?
[08:49] mikko	no, persistence
[08:49] Evet	do you store relationships?
[08:50] mikko	no, not really
[08:53] Evet	im building an SQL and SPARQL api on top of kyoto cabinet
[08:53] Evet	and planning to use zeromq for backup-restore logic
[08:56] sustrik	8k 0MQ requests per second?
[08:56] sustrik	doesn't seem likely unless you are doing it in lock-step manner
[08:57] Evet	single core of a single desktop pc
[08:59] stockMQ	I am typecasting stuct to char*
[08:59] stockMQ	before sending
[08:59] mikko	stockMQ: sure, but first null character would cut your char*
[08:59] mikko	stockMQ: s_send uses strlen () probably
[09:00] stockMQ	ohk.. will pack help?
[09:00] mikko	stockMQ: can you show the code?
[09:00] mikko	gist.github.com
[09:01] stockMQ	i have pasted it before.. coz now i do not have access to it :(
[09:01] mikko	are you running the same architecture on all your nodes?
[09:02] mikko	as sending a struct is not portable
[09:02] mikko	unless you serialize it for transport (protobufs, thrift etc)
[10:36] nathanmarz	sustrik_: but my sender already knows who all the recievers are going to be
[10:36] nathanmarz	sustrik_: seems simpler to not have to worry about synchronization of pub/sub (so that messages don't get lost)
[15:11] mikko	i to the b
[17:59] Evet	sustrik_: why do you think 8k request/second is very high for a desktop pc?
[18:05] mikko	so finally things working properly
[18:05] mikko	http://snapshot.valokuva.org/
[18:05] mikko	mingw32 and msvc2008 snapshot
[18:05] Evet	congrats
[19:52] mikko	sustrik_: seen #153
[19:52] mikko	?
[20:05] sustrik	Evet: i think it's too low
[20:06] sustrik	the rate should go up to 100,000's or millions, depending on the hardware
[20:06] sustrik	8000 can result from lock-step processing
[20:07] sustrik	if the roundtrip time on the network is kind of low (~120us)
[20:07] sustrik	mikko: let me see
[20:09] Evet	sustrik_: what about inproc and ipc performance?
[20:10] sustrik	in the range of millions per sec
[20:10] Evet	wow
[20:10] Evet	does it use b+tree?
[20:11] sustrik	for what?
[20:11] sustrik	subscription matching?
[20:11] Evet	storing messages in memory
[20:11] Evet	and yes, subscription matching
[20:11] sustrik	storage in memory is linear, plain fifo
[20:12] sustrik	so no need for searching
[20:12] sustrik	as for subscription, the algorithm is "trie"
[20:15] sustrik	mikko: ok, #153 answered
[20:15] Evet	hmm
[20:19] cleifer	hi, was having a little bit of trouble with the hello example -- i'm not sure i'm compiling it right, but ("cc -o hello hello.c /usr/local/lib/libzmq.so") it runs and just hangs. connecting to it, sending data, aren't seeming to work
[20:19] Evet	sustrik_: is it possible to implement a snapshot to disk mechanism for those fifo
[20:20] sustrik	cleifer: are you sure the connection exists? can you check netstat?
[20:21] sustrik	Evet: how would that work?
[20:21] cleifer	let me do that, just as ec
[20:22] cleifer	sustrik_, tcp 0 0 localhost:5555 localhost:48696 ESTABLISHED
[20:22] cleifer	and another the other way
[20:22] sustrik	looks ok
[20:22] cleifer	when i recv from the other, all i get is \x01\x00
[20:23] cleifer	and then nothing :-/ -- i see in the code that it'll print when a connection is established but that's not happening
[20:23] sustrik	that's connection initialisation
[20:23] Evet	sustrik_: it may need a broker
[20:23] cleifer	sustrik_, i'm connecting to it in python using a raw socket, maybe that's the problem?
[20:24] sustrik	you mean TCP socket?
[20:24] cleifer	yah
[20:24] sustrik	then you have to follow the 0mq protocol
[20:24] cleifer	s.connect(('localhost', 5555))
[20:24] sustrik	first, send \x01\x00
[20:25] sustrik	that's connection initialisation
[20:25] sustrik	then you can exchange messages
[20:25] sustrik	the format of message is described in zmq_tcp(7)
[20:25] cleifer	coolio
[20:26] cleifer	thanks sustrik_ !
[21:02] drbobbeaty	Got a question for the ZeroMQ TCP usage... is it possible to have a ZeroMQ server create a TCP connection on an ephermal port and then obtain that port number so that it can provide a good URL to a client? I'm thinking of a use-case where I want to fire up a TCP PUB server, on an arbitrary port and then get that port (and URL) and provide it to clients for their connection. Can this be done?
[21:02] drbobbeaty	I'm thinking "No", but I wanted to ask to make sure.
[21:06] sustrik	no
[21:24] iDude	IOT/Abort trap in pthread_kill at 0xd005f734 ($t1) 0xd005f734 (pthread_kill+0x88) 80410014 lwz r2,0x14(r1)
[21:25] iDude	I am using the 2.1.0 version of zmq on AIX 5.3... any ideas to get it working?
[21:28] sustrik	hm, i've made AIX port work some time ago
[21:28] sustrik	no idea of what version though
[21:28] sustrik	can you provide a backtrace for the problem?
[21:31] iDude	pthread_kill(??, ??) at 0xd005f734 _p_raise(??) at 0xd005f1a4 raise.raise(??) at 0xd02a38f0 abort() at 0xd0307778 myabort()() at 0xd0244aac terminate()() at 0xd0242df0 terminate()() at 0xd024424c __DoThrowV6() at 0xd02468f0 _Xlen__Q2_3std6vectorXTQ3_3zmq6poll_t10fd_entry_tTQ2_3std9allocatorXTQ3_3zmq6poll_t10fd_entry_t__CFv(0x2000b990) at 0xd6bccb10 insert__Q2_3std6vectorXTQ3_3zmq6poll_t10fd_entry_tTQ2_3std9allocatorXTQ3_3zmq6poll_t10fd
[21:31] iDude	sorry... I am new to the whole IRC chatting thing... how can I format the text to show on multiple lines?
[21:37] sustrik	paste the baktrace to gist or somesuch and provide the link
[21:47] iDude	here you go: https://gist.github.com/790486
[21:48] iDude	I have included the source code and the Makefile used