ZeroMq IRC Log

Sunday October 2, 2011

[Time] Name	Message
[00:31] mikko	bitcycle: yes
[00:31] mikko	bitcycle: DEALER just load-balances (deals out) the messages you send to all connected peers, and fair-queues (deals in) the messages it receives. It is exactly like a PUSH and PULL socket combined.
[00:32] bitcycle	mikko: So, what design would warrent the use of a dealer? I'm a noob when it comes to zmq.
[00:32] mikko	if you want to for example load balance messages to backends and get responses back
[00:32] mikko	i use dealer socket in pzq to distribute messages to clients and receive ACKs after clients are done with the messages
[00:39] bitcycle	mikko: What's a situation where you would use ROUTER then, in comparison?
[02:33] Trev	Can someone tell me if zeromq adds anything to messages I am trying to use zeromq as a basic file server but I am getting errors when saving the file
[07:23] AmunRa	As I understand, REQ/REP is set up so that the client and the server send messages in turn. Is there something similar but without such restriction? My use case is to have hundreds of (unstable) clients sending queries and for the server to respond with lots ticking updates. The client could then occasionally update its request. In other words: [REQ, REP, REP, REP, REP, ..., REP, REQ, REP, REP, REP, ...]
[07:25] sustrik	XREQ and XREP
[07:48] AmunRa	Thanks for pointing me in the right direction: http://zguide.zeromq.org/page:all#toc54 -- Figure 47 is precisely the recipe I was after :-)
[10:32] mikko	good morning
[10:37] AmunRa	As a learning excercise, I am trying to write a simple chat client and server. To publish I use pub/sub. From the server, I have a 'ChatMsg' protobuf object, which I serialize to a byte string.
[10:38] AmunRa	When I try and invoke publisher.send_multipart([topic,obj.SerializeToString()]) I get an error: TypeError: unicode not allowed, use send_unicode
[10:39] AmunRa	Would anyone be able to shed some light?
[10:48] AmunRa	found the culprit, send_multipart([str(topic), obj.SerializeToString()]) did the trick.
[11:47] mikko	sustrik: how should i go about debugging this further?
[11:47] sustrik	mikko: hi
[11:47] mikko	sustrik: i've now established that on network level i recv () less bytes than client sends
[11:47] sustrik	yup
[11:47] mikko	but i dont yet know whether that is because poll doesn't get fired or that the bytes are simply not there
[11:47] sustrik	i think it's up to me to follow on
[11:47] mikko	i tried increasing and decreasing tcp buffers
[11:47] mikko	no difference
[11:48] sustrik	however, i need something simple to work with
[11:48] mikko	but what i did not is when i added random garbage in to the ACK message
[11:48] mikko	like 1KB+
[11:48] mikko	then less messages get there
[11:48] sustrik	can we reduce the test to a single connection, if possible
[11:49] mikko	let me test quickly
[11:49] sustrik	ie. get rid of all the inproc stuff that would trigger my breakpoints etc.
[11:49] sustrik	chopping off the DB would be nice as well
[11:50] sustrik	removing the need for producer would be great
[11:53] mikko	cutting off the sender device is fairly easy
[11:55] sustrik	i means something like removing the sender and DB part entirely and simply generating the messages to send instead of reading them from DB
[11:56] mikko	pushed first things to debug branch
[11:56] mikko	removes devices completely
[11:56] sustrik	bug is still reproducible, right?
[11:56] mikko	yes
[11:56] mikko	looks like so
[11:57] mikko	now taking off database
[12:06] mikko	try now
[12:06] mikko	you should be able to just see how many ACKs come back
[12:06] mikko	hopefully i didnt break the reproducibility
[12:07] sustrik	great
[12:07] sustrik	what's the branch?
[12:10] mikko	debug
[12:10] mikko	i wonder why i even added the two devices
[12:10] mikko	i was solving some problem
[12:10] mikko	but cant really remember what
[12:10] mikko	i think they are unneeded complexity
[12:10] sustrik	ok
[12:18] sustrik	reproduced 834 acks
[12:19] sustrik	so, the only 0mq traffic now is sending of the messages from pzq to consumer and sending acks back, right?
[12:37] mikko	yes
[13:09] mikko	sustrik: geting anywhere?
[13:09] sustrik	minimising the test case...
[13:34] sustrik	mikko: there?
[13:34] sustrik	what handle_sender_out supposed to do?
[13:38] jond	sustrik: send all the messages you havent had before, as far as I could see yesterday
[13:40] sustrik	i'm wondering about the infinite loop there...
[13:45] mikko	sustrik: it's roughly what the old one did but doesnt quite easily translate
[13:45] mikko	it loops until the messages can be sent
[13:45] mikko	until = as long as
[13:46] sustrik	which may be a problem if it's less than 1000
[13:46] sustrik	let me change that to for(i=0; i!=1000; i++)
[13:47] mikko	sustrik: but it should come back there?
[13:48] mikko	it's invoked when socket is marked writable by poll
[13:48] sustrik	ah
[13:48] sustrik	right
[13:48] mikko	which should happen when the client reads (hwm)
[13:49] mikko	so client should now get as many messages as it reads
[13:50] sustrik	ack
[13:50] sustrik	unless there's a bug in POLLOUT signalling :)
[13:51] mikko	you can dump amount received in consumer
[13:51] mikko	but then again, consumer does a blocking read
[13:51] mikko	it shouldnt exit unless 1000 is received
[13:51] sustrik	ok, fair enough
[13:55] mikko	are you still as puzzled as i am?
[14:00] sustrik	mikko: still stripping it down
[14:00] sustrik	at the moment i have 79 lines remaining
[14:04] mikko	still being able to reproduce?
[14:04] sustrik	yup
[14:05] mikko	cool
[14:07] sustrik	mikko: are the timestamps in the messages important in any way?
[14:09] sustrik	doesn't seem so
[14:21] mikko	sustrik: no
[14:21] mikko	just follows the old structure
[14:21] mikko	they are important in the real app but not in this test
[14:23] sustrik	https://gist.github.com/1257494
[14:23] sustrik	still reproduces...
[14:24] mikko	you know what i think
[14:24] mikko	it seems to boil down to the pollout
[14:24] sustrik	actually, if you comment out the HWM
[14:24] sustrik	it works ok
[14:24] mikko	nope
[14:25] mikko	hmm
[14:25] mikko	interesting
[14:25] mikko	i wonder if this is hwm interacting with poll
[14:28] mikko	seems unlikely
[14:30] mikko	i need to run, i'll be back in the evening
[14:31] sustrik	cya
[14:47] jond	sustrik: what client are you using with that gist?
[14:48] sustrik	jond: here it is:
[14:48] sustrik	https://gist.github.com/1257512
[14:49] jond	i hacked a simplified example yesterday but not as stripped down, as yours... but mine just hangs after a while
[14:50] jond	thx
[14:50] sustrik	this one should print a number per message
[14:51] sustrik	it the last one less than 1000, it means it failed
[14:53] jond	i get 982. strange isnt it?
[14:54] mikko	its starting to look like a bug
[14:54] sustrik	yes
[14:55] jond	i'm glad you got the test case to fail without that extra thread running the pair socket
[14:55] mikko	i think ill remove those from pzq
[14:55] mikko	they seem unnecessary
[14:56] jond	mikko: i agree! took me a while to understand what was going on in yr app!
[14:56] mikko	not sure anymore what problem they were solving
[14:56] mikko	off again->
[17:31] mikko	backg
[18:28] mitsuhiko	hello everybody
[18:29] mitsuhiko	i know this might be a very odd question, but was there ever some indication that DICE uses zeromq for battlefield 3?
[18:38] mikko	mitsuhiko: haven't heard anything about that
[18:38] mitsuhiko	mikko: i just was forwarded a screenshot from leaked data and it showed a libzmq.dll so i was kinda wondering :)
[18:56] sustrik	:)
[18:57] sustrik	there were some guys from ea on the mailing list iirc
[19:04] mikko	sustrik: did you debug the issue furhter?
[19:04] sustrik	nope, haven't got much further
[19:04] sustrik	will check it out tomorrow
[19:05] mikko	cool
[19:06] mikko	it seems like a tricky one
[19:21] AmunRa	Hello, I have a thread that loops on a SUB socket. What's the right way to kill it?
[19:23] AmunRa	I.e.: Is there anything 0mq specific I should be aware of?
[19:31] sustrik	AmunRa: when you call zmq_term() all socket calls in all threads return ETERM error

ZeroMq Home

Sunday October 2, 2011