Sunday October 2, 2011

[Time] NameMessage
[00:31] mikko bitcycle: yes
[00:31] mikko bitcycle: DEALER just load-balances (deals out) the messages you send to all connected peers, and fair-queues (deals in) the messages it receives. It is exactly like a PUSH and PULL socket combined.
[00:32] bitcycle mikko: So, what design would warrent the use of a dealer? I'm a noob when it comes to zmq.
[00:32] mikko if you want to for example load balance messages to backends and get responses back
[00:32] mikko i use dealer socket in pzq to distribute messages to clients and receive ACKs after clients are done with the messages
[00:39] bitcycle mikko: What's a situation where you would use ROUTER then, in comparison?
[02:33] Trev Can someone tell me if zeromq adds anything to messages I am trying to use zeromq as a basic file server but I am getting errors when saving the file
[07:23] AmunRa As I understand, REQ/REP is set up so that the client and the server send messages in turn. Is there something similar but without such restriction? My use case is to have hundreds of (unstable) clients sending queries and for the server to respond with lots ticking updates. The client could then occasionally update its request. In other words: [REQ, REP, REP, REP, REP, ..., REP, REQ, REP, REP, REP, ...]
[07:25] sustrik XREQ and XREP
[07:48] AmunRa Thanks for pointing me in the right direction: -- Figure 47 is precisely the recipe I was after :-)
[10:32] mikko good morning
[10:37] AmunRa As a learning excercise, I am trying to write a simple chat client and server. To publish I use pub/sub. From the server, I have a 'ChatMsg' protobuf object, which I serialize to a byte string.
[10:38] AmunRa When I try and invoke publisher.send_multipart([topic,obj.SerializeToString()]) I get an error: TypeError: unicode not allowed, use send_unicode
[10:39] AmunRa Would anyone be able to shed some light?
[10:48] AmunRa found the culprit, send_multipart([str(topic), obj.SerializeToString()]) did the trick.
[11:47] mikko sustrik: how should i go about debugging this further?
[11:47] sustrik mikko: hi
[11:47] mikko sustrik: i've now established that on network level i recv () less bytes than client sends
[11:47] sustrik yup
[11:47] mikko but i dont yet know whether that is because poll doesn't get fired or that the bytes are simply not there
[11:47] sustrik i think it's up to me to follow on
[11:47] mikko i tried increasing and decreasing tcp buffers
[11:47] mikko no difference
[11:48] sustrik however, i need something simple to work with
[11:48] mikko but what i did not is when i added random garbage in to the ACK message
[11:48] mikko like 1KB+
[11:48] mikko then less messages get there
[11:48] sustrik can we reduce the test to a single connection, if possible
[11:49] mikko let me test quickly
[11:49] sustrik ie. get rid of all the inproc stuff that would trigger my breakpoints etc.
[11:49] sustrik chopping off the DB would be nice as well
[11:50] sustrik removing the need for producer would be great
[11:53] mikko cutting off the sender device is fairly easy
[11:55] sustrik i means something like removing the sender and DB part entirely and simply generating the messages to send instead of reading them from DB
[11:56] mikko pushed first things to debug branch
[11:56] mikko removes devices completely
[11:56] sustrik bug is still reproducible, right?
[11:56] mikko yes
[11:56] mikko looks like so
[11:57] mikko now taking off database
[12:06] mikko try now
[12:06] mikko you should be able to just see how many ACKs come back
[12:06] mikko hopefully i didnt break the reproducibility
[12:07] sustrik great
[12:07] sustrik what's the branch?
[12:10] mikko debug
[12:10] mikko i wonder why i even added the two devices
[12:10] mikko i was solving some problem
[12:10] mikko but cant really remember what
[12:10] mikko i think they are unneeded complexity
[12:10] sustrik ok
[12:18] sustrik reproduced 834 acks
[12:19] sustrik so, the only 0mq traffic now is sending of the messages from pzq to consumer and sending acks back, right?
[12:37] mikko yes
[13:09] mikko sustrik: geting anywhere?
[13:09] sustrik minimising the test case...
[13:34] sustrik mikko: there?
[13:34] sustrik what handle_sender_out supposed to do?
[13:38] jond sustrik: send all the messages you havent had before, as far as I could see yesterday
[13:40] sustrik i'm wondering about the infinite loop there...
[13:45] mikko sustrik: it's roughly what the old one did but doesnt quite easily translate
[13:45] mikko it loops until the messages can be sent
[13:45] mikko until = as long as
[13:46] sustrik which may be a problem if it's less than 1000
[13:46] sustrik let me change that to for(i=0; i!=1000; i++)
[13:47] mikko sustrik: but it should come back there?
[13:48] mikko it's invoked when socket is marked writable by poll
[13:48] sustrik ah
[13:48] sustrik right
[13:48] mikko which should happen when the client reads (hwm)
[13:49] mikko so client should now get as many messages as it reads
[13:50] sustrik ack
[13:50] sustrik unless there's a bug in POLLOUT signalling :)
[13:51] mikko you can dump amount received in consumer
[13:51] mikko but then again, consumer does a blocking read
[13:51] mikko it shouldnt exit unless 1000 is received
[13:51] sustrik ok, fair enough
[13:55] mikko are you still as puzzled as i am?
[14:00] sustrik mikko: still stripping it down
[14:00] sustrik at the moment i have 79 lines remaining
[14:04] mikko still being able to reproduce?
[14:04] sustrik yup
[14:05] mikko cool
[14:07] sustrik mikko: are the timestamps in the messages important in any way?
[14:09] sustrik doesn't seem so
[14:21] mikko sustrik: no
[14:21] mikko just follows the old structure
[14:21] mikko they are important in the real app but not in this test
[14:23] sustrik
[14:23] sustrik still reproduces...
[14:24] mikko you know what i think
[14:24] mikko it seems to boil down to the pollout
[14:24] sustrik actually, if you comment out the HWM
[14:24] sustrik it works ok
[14:24] mikko nope
[14:25] mikko hmm
[14:25] mikko interesting
[14:25] mikko i wonder if this is hwm interacting with poll
[14:28] mikko seems unlikely
[14:30] mikko i need to run, i'll be back in the evening
[14:31] sustrik cya
[14:47] jond sustrik: what client are you using with that gist?
[14:48] sustrik jond: here it is:
[14:48] sustrik
[14:49] jond i hacked a simplified example yesterday but not as stripped down, as yours... but mine just hangs after a while
[14:50] jond thx
[14:50] sustrik this one should print a number per message
[14:51] sustrik it the last one less than 1000, it means it failed
[14:53] jond i get 982. strange isnt it?
[14:54] mikko its starting to look like a bug
[14:54] sustrik yes
[14:55] jond i'm glad you got the test case to fail without that extra thread running the pair socket
[14:55] mikko i think ill remove those from pzq
[14:55] mikko they seem unnecessary
[14:56] jond mikko: i agree! took me a while to understand what was going on in yr app!
[14:56] mikko not sure anymore what problem they were solving
[14:56] mikko off again->
[17:31] mikko backg
[18:28] mitsuhiko hello everybody
[18:29] mitsuhiko i know this might be a very odd question, but was there ever some indication that DICE uses zeromq for battlefield 3?
[18:38] mikko mitsuhiko: haven't heard anything about that
[18:38] mitsuhiko mikko: i just was forwarded a screenshot from leaked data and it showed a libzmq.dll so i was kinda wondering :)
[18:56] sustrik :)
[18:57] sustrik there were some guys from ea on the mailing list iirc
[19:04] mikko sustrik: did you debug the issue furhter?
[19:04] sustrik nope, haven't got much further
[19:04] sustrik will check it out tomorrow
[19:05] mikko cool
[19:06] mikko it seems like a tricky one
[19:21] AmunRa Hello, I have a thread that loops on a SUB socket. What's the right way to kill it?
[19:23] AmunRa I.e.: Is there anything 0mq specific I should be aware of?
[19:31] sustrik AmunRa: when you call zmq_term() all socket calls in all threads return ETERM error