Tuesday October 12, 2010

[Time] NameMessage
[00:48] delaney okay, so i got all the python stuff working (thank you). i want to use zmq as the basis for a turn based game server with mutli games going on. going through the documentation it seems like that would need a pub-sub for 'your-turn' type information and req/rep for sending turns and get ack/nacks back. is this logical?
[02:12] delaney what's deprecated in ?
[04:17] delaney can someone go into a little more detail when to use XREP/XREQ?
[04:32] steveng I am trying to install the zeromq on my fedora machine, I have installed both e2fsprogs-devel and uuid-dev. However, the ./configure shows error "cannot link with -luuid, install uuid-dev". Any ideas?
[07:21] andrewvc just loops twice, increments requester_id
[07:21] andrewvc i mean , the first iteration it's 1, second it's 2
[07:22] delaney where are you setting the identifier for xrep?
[07:22] delaney ah, nm
[07:24] andrewvc yeah, I don't really have any great examples for xreq
[07:24] andrewvc I need to write one
[07:25] andrewvc $ cat 006_xreq_rep.rb
[07:25] andrewvc #TODO: Write this
[07:25] andrewvc clearly, I need to read my todos more often
[07:25] delaney heh
[07:25] delaney so are you a dev on zmq?
[07:25] andrewvc nope
[07:25] andrewvc but I work on some ruby stuff
[07:26] andrewvc
[07:26] andrewvc it's a more concise way of using zmq functionality
[07:26] andrewvc handles serialization, socket setup, runs in a reactor, etc.
[07:27] delaney zmqmachine?
[07:27] andrewvc yep
[07:27] andrewvc not sure if python has one, does it?
[07:27] delaney no idea
[07:28] delaney the pyzmq is all pyd magic... hard to go through the code and learn that way.
[07:28] andrewvc pyd?
[07:28] delaney a compiled python moudle
[07:28] andrewvc oh, python + D
[07:28] delaney *module... basically a *.dll/so
[07:28] andrewvc oh, gotcha
[07:29] andrewvc might be a little basic for where you're at, but you might find useful
[07:29] andrewvc I still forget stuff and use it as a reference
[07:30] andrewvc ruby though it is, the C API and ruby API are fairly similar, so I imagine the python one is as well
[07:31] delaney true
[07:32] delaney on the pub/sub you talked about messages getting lost... you meant if there is noone listening... right?
[07:32] andrewvc yes
[07:32] delaney but if 2+ people are listening, its fine?
[07:32] andrewvc yeah, both of them get it
[07:33] andrewvc now, actually
[07:33] andrewvc there's a little trick
[07:33] andrewvc I forgot about till just now
[07:34] andrewvc so, if explicitly define the identity of a subscriber
[07:34] andrewvc messages are guaranteed to arrive at that subscriber
[07:34] andrewvc the downside being, if the subscriber never reconnects, you may hit the HWM eventually due to a backlog
[07:35] andrewvc not sure where that info specifically is
[07:35] delaney and is that HWM on the publisher or per subscriber?
[07:35] andrewvc publisher
[07:35] delaney hmm
[07:35] andrewvc I'm off to sleep, but if there's anything else you need help with, I may be around in chat. The mailing list seems like a more responsive place than chat sometimes btw.
[07:35] delaney so PUB/SUB is similar to UDP
[07:36] delaney in the way i should be thinking of it
[07:36] andrewvc I guess, yeah
[07:36] delaney great, thanks so much
[07:36] andrewvc I mean, it is reliable, so long as the clients connect
[07:36] delaney right
[07:36] andrewvc and they reconnect if the conn dies
[07:36] delaney one last thing if you have a sec
[07:36] andrewvc sure
[07:36] delaney in
[07:36] delaney why are you using ZMQ::SNDMORE
[07:37] andrewvc ah
[07:37] andrewvc well, when you create a multipart message
[07:37] delaney instead of just doing the whole messagE?
[07:37] andrewvc oh
[07:37] andrewvc well, with pub/sub filtering right
[07:37] andrewvc you filter on the message prefix
[07:37] andrewvc so it makes life a bit easier because
[07:37] andrewvc if my filter is
[07:38] andrewvc andrewsfilter
[07:38] andrewvc well, lemme rephrase that
[07:38] andrewvc the filtering terminates after the first message part
[07:38] andrewvc it's convenient to use a separate part for the topic so the message filtering doesn't extend into the message body
[07:38] delaney ah... i thought it was on the first " "
[07:39] andrewvc it's not mandatory, you can use a single message
[07:39] andrewvc is it?
[07:39] delaney "id message"
[07:39] delaney probably missunderstood that
[07:39] andrewvc from the guide?
[07:40] andrewvc yeah, just tested it
[07:40] andrewvc topics can have spaces
[07:40] andrewvc there's other strategies
[07:40] andrewvc mongrel2 just uses a single message, and delineates the topic by terminating with a null byte I beleive
[07:41] delaney ah, but the multi-part message is a right way to do rep/xreq ids
[07:41] andrewvc well, that's different
[07:41] andrewvc with pub/sub the multi-part is for my convenience really
[07:41] delaney gotcha
[07:41] andrewvc with req/xreq it's mandatory, zeromq creates the ids
[07:41] andrewvc errr those extra parts
[07:42] andrewvc anyway, off to sleep. Let me know how the game goes, I'm curious to see what happens with it
[07:43] delaney thanks!
[10:26] CIA-14 zeromq2: 03Steven McCoy 07master * rc6d74e0 10/ src/pgm_socket.cpp :
[10:26] CIA-14 zeromq2: Drop PGM parameters to 100mb friendly range.
[10:26] CIA-14 zeromq2: Signed-off-by: Steven McCoy <> -
[13:03] BooTheHamster Hi. I have problem with call zmq::message->size() function after send.
[13:04] BooTheHamster before send a size of message is 4, after - 0
[13:07] mato BooTheHamster: Yes, send takes ownership of your message content
[13:07] mato BooTheHamster: Hence, after calling send you can't use that message any more.
[13:07] mato BooTheHamster: Use zmq_msg_copy() or the C++ ::copy() (AFAIK) to take a reference-counted copy if you need it.
[13:09] BooTheHamster thanx, mato
[13:11] BooTheHamster If I create zmq::message in thread, then make send in threadproc, and then exit from the thread - I recieve a message in another thread? I use PUSH/PULL sockets.
[13:16] mato BooTheHamster: I'm not sure what you're asking
[13:17] BooTheHamster Sorry for my English ((
[13:19] sustrik i think boo asks whether you get the message even though you exit the sending threads immediately after calling send
[13:19] sustrik yes, you will get it
[13:20] sustrik in version 2.1
[13:20] sustrik (master)
[13:27] BooTheHamster sustrik, yes
[13:27] BooTheHamster thanx
[13:30] BooTheHamster In version 2.0.9 I must wait when message is delivered?
[13:31] sustrik are you using inproc transport?
[13:31] sustrik if so, it should work ok even with 2.0.x versions
[13:31] BooTheHamster No. I use tcp transport.
[13:31] sustrik then yes, you have to wait till the message is delivered
[13:32] sustrik why not switch to 2.1 though
[13:32] sustrik ?
[13:32] BooTheHamster I got sources from
[13:32] BooTheHamster for Windows
[13:39] sustrik sure, but why not use the master from github?
[13:39] sustrik anyway, it's up to you
[13:40] BooTheHamster 2.1 is stable?
[13:47] sustrik it's getting stable
[13:47] sustrik it depends on what you want to do with it
[13:53] BooTheHamster Just send messages between many threads and many processes. I use tcp transport (localhost and remote address) and REP/REQ and PUSH/PULL sockets. Some threads and processes may frozee or die. I already have many bugs in my own application :)
[13:58] sustrik then go for 2.1
[13:59] BooTheHamster The main problem is when I recieve message from I terminated process. The content of the message broken and I get access violation.
[14:00] BooTheHamster terminated thread, sorry
[14:00] sustrik as i said, try 2.1, the whole shutdown mechanism was rewritten there
[14:01] sustrik so it's quite possible that the bug will just disappear when you switch
[14:02] BooTheHamster Ok. I try to get version 2.1 sources from github and rebuild. Thank you/
[14:07] sustrik mikko: are you there?
[14:08] mikko yes
[14:11] sustrik i was browsing thought the hudson results
[14:12] sustrik and i've noticed jzmq does not build because javac is missing
[14:12] sustrik have you installed java on the build box?
[14:12] mikko hudson is java so yes
[14:12] mikko might be missing javac
[14:12] sustrik jdk?
[14:12] mikko i recently moved it to a dedicated box
[14:12] mikko let me check
[14:13] sustrik thx
[14:13] mikko jre installed but not jdk
[14:13] mikko sec
[14:16] mikko
[14:18] sustrik :)
[14:18] mikko pyzmq master build seems to get stuck
[14:18] mikko probably close semantics i assume
[14:19] sustrik possible
[15:57] CIA-14 zeromq2: 03Martin Sustrik 07master * r954d7f7 10/ (src/session.cpp src/session.hpp): (log message trimmed)
[15:57] CIA-14 zeromq2: Avoid duplicate creation of pipes for a single session
[15:57] CIA-14 zeromq2: When a session is being closed down its inbound and outbound
[15:57] CIA-14 zeromq2: pipe pointers are set to null. If (re) connection happens at
[15:57] CIA-14 zeromq2: that time, session may try to reinistantiate the pipes which
[15:57] CIA-14 zeromq2: is wrong. This patch allows session to attach pipes only once
[15:57] CIA-14 zeromq2: in its lifetime.
[15:58] sustrik mikko: your assert should be solved in the master
[15:58] sustrik (Assertion failed: inpipe_ && outpipe_ (xreq.cpp:42))
[16:00] mikko sustrik: good stuff
[16:00] mikko now i can add more tests for xre(q|p)
[17:30] sustrik bgranger: hi
[17:34] starkdg is there any way to monitor the buffer lengths in zeromq ? how many messages in a queue at a given time ?
[17:44] bgranger sustrik: hi
[17:44] sustrik hi
[17:45] sustrik i've noticed that there's some problem with pyzmq vs. master branch of the code
[17:45] sustrik core
[17:45] sustrik ...
[17:45] sustrik test_create (zmq.tests.test_socket.TestSocket) ... ok
[17:45] sustrik test sending unicode objects ... ok
[17:45] sustrik test the MessageTracker object for tracking when zmq is done with a buffer ... ERROR
[17:45] sustrik Build was aborted
[17:45] sustrik any idea what's going on there?
[17:46] sustrik is it a bug in master?
[17:46] sustrik starkds: no, there is not
[17:46] bgranger We have not tested pyzmq with zeromq master yet.
[17:46] sustrik ah, ok
[17:46] sustrik i was just wondering
[17:46] bgranger We are getting close to finishing up work on 2.0.9 (it was a lot) and after that we will start looking at 2.1
[17:46] sustrik ack
[17:47] sustrik have you seen mikko's build system btw?
[17:47] bgranger Yes, it is very nice!
[17:47] sustrik i love it :)
[17:47] bgranger I can imagine that some bug have show up with 2.1 and pyzmq though. Although I would guess that at least some are in zeromq
[17:48] sustrik definitely
[17:48] bgranger But we will be able to tell pretty quickly once we start the 2.1 work.
[17:48] sustrik i am fixing them gradually
[17:48] sustrik sure, no problrm
[17:49] bgranger But we are using 2.0.9 heavily and it is working quite well.
[17:53] delaney has anyone tried pyzmq building against python 3.1?
[17:54] jdroid- i'm sorta new to messaging, so i might get terminology wrong...
[17:55] jdroid- anyway, i'm curious what the zmq approach is to handling handlers that went down
[17:55] jdroid- if it's in the docs, please link me there. trying to work my way through those
[17:58] delaney new too, but from what i can see the message would just keep queueing up for the disconnected client
[17:58] delaney so you'd have to have some kind fo ping timeout or relie on the 'high water mark' settings
[17:59] jdroid- hm.. ok.
[17:59] delaney i also would like a better explanation of it
[18:01] bgranger delaney: a few people have forks of pyzmq that semi-work with python 3, but they are not fully supported yet. There are some subtle issues about bytes and unicode that we are working out.
[18:02] delaney k, i'll hold off for now, don't feel comfortable enough with the lib to know what's 'right' behavior
[18:03] delaney so what does happen if say a REP/REQ connection if the REQ sends a request and the REP receives and before sending the REQ goes away? what's happens during the rep.socket.send?
[18:03] delaney maybe i missed that in the guide
[18:09] jdroid- so in the event of a time-out, what would be the 0mq way of handling that? does each box that would send a message to the failed handler just figure out that the box is down and pull it out of the round robin queue?
[18:11] jdroid- sorry... my question assumes a setup like shown in these mongrel2 slides are used:
[18:12] jdroid- ok. i think my question is based on a push/pull kind of situation. i guess it makes sense to hang until the handler comes up or time out.
[18:16] sustrik jdroid: yes, if there's no handler it hangs
[18:16] sustrik however, if the handler was processing the message while it crashed, it's lost
[18:16] jdroid- ok. i'm sorta familiar with how zeromq doesn't offer persistance. it looksl ike some projects offer it via redis and other things
[18:17] delaney so what happens in my example sustrik?
[18:18] sustrik the message gets lost
[18:18] sustrik so you have to timeout
[18:18] sustrik and resend it
[18:18] sustrik or something
[18:18] sustrik in the future it would be nice to make REQ/REP do this automatically
[18:18] sustrik but it's not there yet
[18:19] delaney ah send with a return of !0 is a failure
[18:19] sustrik send is async
[18:20] sustrik so if the remote application fails, you won't be notified about it
[18:20] sustrik you have to time-out
[18:20] delaney right but it would return an ETERM if the other side disconnected?
[18:20] sustrik no
[18:21] sustrik ETERM means that the context you are running in was terminated
[18:21] delaney ah
[18:22] delaney i don't see a timeout param in socket.send
[18:22] sustrik you have to use zmq_poll for that
[18:22] sustrik there's a timeout param there
[18:22] delaney gotcha, okay
[18:22] delaney sorry for the confusion
[18:23] sustrik you are welcome
[18:23] delaney i noticed in the pyzmq there is a poll.Poller class and a method. any reason to use the select instead of the Poller?
[18:24] sustrik that one is up to bgranger
[18:25] delaney gotcha, and is there any roadmap for having reliable udp?
[18:32] bgranger delaney: we provide both to be compatible with Python select and poll functions in the stdlib
[18:32] bgranger Both use zmq_poll though
[18:33] bgranger Both should be fully compatible with those in the stdlib
[18:33] delaney right, i wondered why they are part of the Poller class directly?
[18:34] delaney like
[18:34] sustrik which reliable udp do you mean?:)
[18:35] delaney i meant for games stuff sustrik. its by no means necessary but faster than tcp for high traffic game servers
[18:35] sustrik i mean, which spec?
[18:35] sustrik which rfc?
[18:35] delaney i don't think there is one
[18:36] sustrik there are dozens of them
[18:36] sustrik there's rfc3208 integrated with 0mq
[18:36] sustrik but that's meant to be used for multicast, not unicast
[18:36] delaney what about 768?
[18:37] sustrik that's raw udp
[18:38] sustrik it would be nice to have unreliable udp transport
[18:38] sustrik fancy writing it?
[18:38] delaney yeah i know but most games build on that to add reliable ack/nack stuff
[18:38] sustrik if you want to implement that then go on
[18:39] sustrik but designing a network protocol is a non-trivial task
[18:39] delaney well... A. i just got to using zmq so probably should get used to it. B. i'm a 3d artist guy, so yeah, can make you a pretty logo for it. :P
[18:39] Guthur Surely unreliable UDP isn't that hard considering it already fulfils the 'unreliable' portion
[18:39] sustrik guthur: yes
[18:39] sustrik the reliability piece is the problem
[18:39] sustrik also congestion control
[18:39] sustrik etc.
[18:40] sustrik delaney :)
[18:40] delaney yeah, the reliable its a small thing... but having access to UDP on the socket would help
[18:40] delaney depending on the game obviously
[18:41] andrewvc delaney, I thought you were building a turn-based game. is TCP latency going to be an issue?
[18:41] delaney no no, for my game its fine, but talking to some other indies about zmq cause it makes socket programming make sense, they had an rts in mind
[18:42] delaney which is usually a peer-to-peer udp type of thing
[18:43] delaney its in no means something i'd be using for my project, just a really good featue for other style games where availablity is more important than reliability
[18:44] sustrik well, it requires some coding
[18:44] sustrik so if there are couple of devs who really want the reliable udp thing
[18:45] sustrik they can join forces to write one
[18:45] sustrik plugging it into 0mq is a trifle
[18:45] Guthur I personally wish someone would improve the Java support, it's not very good
[18:46] sustrik Guthur: i've heard this multiple times
[18:46] sustrik but nobody really specified what's wrong
[18:46] Guthur its very lacking in most feature areas as far as I can see
[18:46] sustrik specifically?
[18:46] Guthur I only really has basic rep/req
[18:47] sustrik i think other socket types are supported as well
[18:47] Guthur It doesn't have devices as far as I can see
[18:48] Guthur And I don't think it has named sockets either
[18:48] sustrik ?
[18:48] sustrik ok, one by one
[18:48] Guthur My 0mq knowledge is quite limited though
[18:49] sustrik are you saying that pub/sub sockets are missing?
[18:49] Guthur I only stumbled upon it recently
[18:49] sustrik let me check
[18:49] sustrik disclaimer: i am not jzmq dev
[18:50] Guthur Though to be honest one had to use an earlier git commit than the most recent
[18:50] sustrik there are all socket types defined...
[18:50] Guthur due to a compile error
[18:50] sustrik yeah, that's a real problem
[18:50] sustrik we have to release 2.0.10
[18:50] sustrik that has version macros in
[18:50] sustrik then jzmq would be able to #ifdef the new stuff out
[18:51] sustrik so it would be possible to build it with stable versions
[18:51] delaney bgranger: is pyzmq your thing?
[18:51] sustrik it's being worked on
[18:52] Guthur sustrik, I'll keep an eye on it, cheers
[18:52] bgranger delaney: i started it and am maintaining it, but others are working on it now too.
[18:52] Guthur Java isn't my language of choice but a lot of business' are infatuated with it
[18:53] sustrik Guthur: I am aware of that
[18:53] sustrik the problem is that 0mq is a community project
[18:53] sustrik thus people hack it for fun
[18:53] sustrik which makes jun and .net a bit orphaned
[18:53] sustrik java
[18:54] sustrik no much people doing these two for fun
[18:54] bgranger because they are not fun? ;-)
[18:54] sustrik same witg windows platform :(
[18:54] Guthur yeah I can understand that
[18:54] sustrik well, it's something you get paid for to do
[18:55] Guthur There would be quite handful of languages I would rather use before I get to Java or C#
[18:56] delaney bgranger: i noticed that going through the guide the hwserver/ if you start the clients first then the server it just hangs. starting the server first does work fine.
[18:56] delaney does the example need to set the ZMQ_INDENTIFY?
[18:57] sustrik it's your choice, obviously, but people are doing a lot of C hacking for fun
[18:58] delaney i now have binaries versions of pyzmq for windows for both 2.6 and 2.7 if ya'll want to add to the download page.
[19:01] sustrik delaney: the problem is not supplying binaries, rather maintaining them in the future
[19:01] sustrik creating new versions etc.
[19:02] sustrik it would be even nice to have an installer
[19:02] sustrik but it's just too much work
[19:03] sustrik nobody volunteered for that so far :)
[19:13] bgranger delaney: how did you create the binaries?
[19:21] delaney with msvc 2010 and mingw
[19:22] delaney its not 'the python way' but if you dump my folders into /python27/Lib/site-packages it 'just works'
[19:23] delaney i actually avoided the library a while back cause the barrier to play with it was too high for a python specific dev
[19:23] delaney i figure since there is no one doing it if ya'll want them at least as a starting point then it helps traction for pyzmq, which is awesome
[19:29] delaney i see in the git master for pyzmq it using cython... does that mean there's going to a single dll to use?
[20:12] jhawk28 hello, is sustrik on?
[20:13] sustrik yup, i'm here
[20:13] jhawk28 the patch for 68 didnt seem to solve the problem
[20:14] sustrik yes, i've seen your comment
[20:14] jhawk28 Im not sure if I was doing something wrong
[20:14] sustrik how did you apply the patch?
[20:14] jhawk28 I manually applied it to the pub.cpp
[20:14] jhawk28 then I compiled using the mvsc
[20:15] sustrik ah, have you noticed the one line to be deleted?
[20:15] jhawk28 yes
[20:15] sustrik then it's probably a different bug
[20:15] jhawk28 I deleted the lib file just in case
[20:16] sustrik anyway, the original fix is now in both maint and master
[20:16] sustrik in case you want to try
[20:16] sustrik but as i said, it's probably a different bug
[20:16] jhawk28 the example on how to reproduce it was cut off by github, not sure why
[20:16] sustrik yuck
[20:17] jhawk28 you basically create a PUB socket, but use a connect. Set the HWM to 1 after the connect
[20:17] jhawk28 iterate forever producing data
[20:17] sustrik easy enough
[20:17] sustrik i'll check it tomorrow
[20:18] jhawk28 Create a subscriber that binds to the address and does a simple read to get some items
[20:18] jhawk28 it works correctly when the HWM is set before the connect
[20:18] jhawk28 or if the pub socket uses a bind
[20:19] sustrik :)
[20:19] sustrik ok, i see
[20:20] sustrik the point is that HWM is applied once, you cannot change it afterwards
[20:20] sustrik so if you do setsockopt(HWM);connect();
[20:20] sustrik the HWM applies
[20:20] jhawk28 yes
[20:20] sustrik if you do connect();setsockopt(HWM);
[20:21] sustrik is does not
[20:21] jhawk28 correct
[20:21] sustrik so you have default HWM
[20:21] sustrik which is infinite
[20:21] sustrik then, if you are sending faster than receiving
[20:22] sustrik the buffer will obviously grow ad infinitum
[20:22] jhawk28 yes
[20:22] jhawk28 and die of out of memory
[20:22] sustrik yes
[20:24] jhawk28 I just wanted to make sure you knew what the problem was before spending more time on it
[20:24] delaney bgranger: figure 17 of the user guide doesn't match the given python code
[20:24] bgranger Let me look
[20:25] sustrik jhawk28: yes, it works that way
[20:25] sustrik set the HWM before doing connect and it should work fine
[20:27] bgranger delaney: which part does not match?
[20:31] jhawk28 sustrik: ok if thats the way it works, ill document that in the issue and close it
[20:33] sustrik yes, thanks!
[20:34] sustrik jhawk28: btw, if it was man pages that confused you, feel free to propose a documentation patch
[20:47] mikko good evening
[20:52] delaney bgranger: the section above it is talking about setting up a XREQ->XREP/XREQ<-XREP setup but the rr_server/ are basically the same as the hwserver/
[20:52] delaney ah, wasn't looking at the broker.c part sorry
[20:52] delaney would be nice to have the broker in py as well :)
[22:29] delaney is there a good example of using polling with a REQ/XREP setup?
[22:30] delaney i'm getting a Assertion failed: msg_->flags & ZMQ_MSG_MORE (..\..\..\src\req.cpp:225) when trying to recv() on a socket with a status of POLLIN
[22:31] delaney also tried recv_multipart