ZeroMq IRC Log

Tuesday October 12, 2010

[Time] Name	Message
[00:48] delaney	okay, so i got all the python stuff working (thank you). i want to use zmq as the basis for a turn based game server with mutli games going on. going through the documentation it seems like that would need a pub-sub for 'your-turn' type information and req/rep for sending turns and get ack/nacks back. is this logical?
[02:12] delaney	what's deprecated in http://www.zeromq.org/code:examples-chat ?
[04:17] delaney	can someone go into a little more detail when to use XREP/XREQ?
[04:32] steveng	I am trying to install the zeromq on my fedora machine, I have installed both e2fsprogs-devel and uuid-dev. However, the ./configure shows error "cannot link with -luuid, install uuid-dev". Any ideas?
[07:21] andrewvc	just loops twice, increments requester_id
[07:21] andrewvc	i mean , the first iteration it's 1, second it's 2
[07:22] delaney	where are you setting the identifier for xrep?
[07:22] delaney	ah, nm
[07:24] andrewvc	yeah, I don't really have any great examples for xreq
[07:24] andrewvc	I need to write one
[07:25] andrewvc	$ cat 006_xreq_rep.rb
[07:25] andrewvc	#TODO: Write this
[07:25] andrewvc	clearly, I need to read my todos more often
[07:25] delaney	heh
[07:25] delaney	so are you a dev on zmq?
[07:25] andrewvc	nope
[07:25] andrewvc	but I work on some ruby stuff
[07:26] andrewvc	http://github.com/andrewvc/dripdrop
[07:26] andrewvc	it's a more concise way of using zmq functionality
[07:26] andrewvc	handles serialization, socket setup, runs in a reactor, etc.
[07:27] delaney	zmqmachine?
[07:27] andrewvc	yep
[07:27] andrewvc	not sure if python has one, does it?
[07:27] delaney	no idea
[07:28] delaney	the pyzmq is all pyd magic... hard to go through the code and learn that way.
[07:28] andrewvc	pyd?
[07:28] delaney	a compiled python moudle
[07:28] andrewvc	oh, python + D
[07:28] delaney	module... basically a .dll/so
[07:28] andrewvc	oh, gotcha
[07:29] andrewvc	might be a little basic for where you're at, but you might find http://github.com/andrewvc/learn-ruby-zeromq useful
[07:29] andrewvc	I still forget stuff and use it as a reference
[07:30] andrewvc	ruby though it is, the C API and ruby API are fairly similar, so I imagine the python one is as well
[07:31] delaney	true
[07:32] delaney	on the pub/sub you talked about messages getting lost... you meant if there is noone listening... right?
[07:32] andrewvc	yes
[07:32] delaney	but if 2+ people are listening, its fine?
[07:32] andrewvc	yeah, both of them get it
[07:33] andrewvc	now, actually
[07:33] andrewvc	there's a little trick
[07:33] andrewvc	I forgot about till just now
[07:34] andrewvc	so, if explicitly define the identity of a subscriber
[07:34] andrewvc	messages are guaranteed to arrive at that subscriber
[07:34] andrewvc	the downside being, if the subscriber never reconnects, you may hit the HWM eventually due to a backlog
[07:35] andrewvc	not sure where that info specifically is
[07:35] delaney	and is that HWM on the publisher or per subscriber?
[07:35] andrewvc	publisher
[07:35] delaney	hmm
[07:35] andrewvc	I'm off to sleep, but if there's anything else you need help with, I may be around in chat. The mailing list seems like a more responsive place than chat sometimes btw.
[07:35] delaney	so PUB/SUB is similar to UDP
[07:36] delaney	in the way i should be thinking of it
[07:36] andrewvc	I guess, yeah
[07:36] delaney	great, thanks so much
[07:36] andrewvc	I mean, it is reliable, so long as the clients connect
[07:36] delaney	right
[07:36] andrewvc	and they reconnect if the conn dies
[07:36] delaney	one last thing if you have a sec
[07:36] andrewvc	sure
[07:36] delaney	in http://github.com/andrewvc/learn-ruby-zeromq/blob/master/001_Socket_Types/002_publish_subscribe.rb
[07:36] delaney	why are you using ZMQ::SNDMORE
[07:37] andrewvc	ah
[07:37] andrewvc	well, when you create a multipart message
[07:37] delaney	instead of just doing the whole messagE?
[07:37] andrewvc	oh
[07:37] andrewvc	well, with pub/sub filtering right
[07:37] andrewvc	you filter on the message prefix
[07:37] andrewvc	so it makes life a bit easier because
[07:37] andrewvc	if my filter is
[07:38] andrewvc	andrewsfilter
[07:38] andrewvc	well, lemme rephrase that
[07:38] andrewvc	the filtering terminates after the first message part
[07:38] andrewvc	it's convenient to use a separate part for the topic so the message filtering doesn't extend into the message body
[07:38] delaney	ah... i thought it was on the first " "
[07:39] andrewvc	it's not mandatory, you can use a single message
[07:39] andrewvc	is it?
[07:39] delaney	"id message"
[07:39] delaney	probably missunderstood that
[07:39] andrewvc	from the guide?
[07:40] andrewvc	yeah, just tested it
[07:40] andrewvc	topics can have spaces
[07:40] andrewvc	there's other strategies
[07:40] andrewvc	mongrel2 just uses a single message, and delineates the topic by terminating with a null byte I beleive
[07:41] delaney	ah, but the multi-part message is a right way to do rep/xreq ids
[07:41] andrewvc	well, that's different
[07:41] andrewvc	with pub/sub the multi-part is for my convenience really
[07:41] delaney	gotcha
[07:41] andrewvc	with req/xreq it's mandatory, zeromq creates the ids
[07:41] andrewvc	errr those extra parts
[07:42] andrewvc	anyway, off to sleep. Let me know how the game goes, I'm curious to see what happens with it
[07:43] delaney	thanks!
[10:26] CIA-14	zeromq2: 03Steven McCoy 07master * rc6d74e0 10/ src/pgm_socket.cpp :
[10:26] CIA-14	zeromq2: Drop PGM parameters to 100mb friendly range.
[10:26] CIA-14	zeromq2: Signed-off-by: Steven McCoy <steven.mccoy@miru.hk> - http://bit.ly/d41QBm
[13:03] BooTheHamster	Hi. I have problem with call zmq::message->size() function after send.
[13:04] BooTheHamster	before send a size of message is 4, after - 0
[13:07] mato	BooTheHamster: Yes, send takes ownership of your message content
[13:07] mato	BooTheHamster: Hence, after calling send you can't use that message any more.
[13:07] mato	BooTheHamster: Use zmq_msg_copy() or the C++ ::copy() (AFAIK) to take a reference-counted copy if you need it.
[13:09] BooTheHamster	thanx, mato
[13:11] BooTheHamster	If I create zmq::message in thread, then make send in threadproc, and then exit from the thread - I recieve a message in another thread? I use PUSH/PULL sockets.
[13:16] mato	BooTheHamster: I'm not sure what you're asking
[13:17] BooTheHamster	Sorry for my English ((
[13:19] sustrik	i think boo asks whether you get the message even though you exit the sending threads immediately after calling send
[13:19] sustrik	yes, you will get it
[13:20] sustrik	in version 2.1
[13:20] sustrik	(master)
[13:27] BooTheHamster	sustrik, yes
[13:27] BooTheHamster	thanx
[13:30] BooTheHamster	In version 2.0.9 I must wait when message is delivered?
[13:31] sustrik	are you using inproc transport?
[13:31] sustrik	if so, it should work ok even with 2.0.x versions
[13:31] BooTheHamster	No. I use tcp transport.
[13:31] sustrik	then yes, you have to wait till the message is delivered
[13:32] sustrik	why not switch to 2.1 though
[13:32] sustrik	?
[13:32] BooTheHamster	I got sources from http://www.zeromq.org/local--files/area:download/zeromq-2.0.9.zip
[13:32] BooTheHamster	for Windows
[13:39] sustrik	sure, but why not use the master from github?
[13:39] sustrik	anyway, it's up to you
[13:40] BooTheHamster	2.1 is stable?
[13:47] sustrik	it's getting stable
[13:47] sustrik	it depends on what you want to do with it
[13:53] BooTheHamster	Just send messages between many threads and many processes. I use tcp transport (localhost and remote address) and REP/REQ and PUSH/PULL sockets. Some threads and processes may frozee or die. I already have many bugs in my own application :)
[13:58] sustrik	then go for 2.1
[13:59] BooTheHamster	The main problem is when I recieve message from I terminated process. The content of the message broken and I get access violation.
[14:00] BooTheHamster	terminated thread, sorry
[14:00] sustrik	as i said, try 2.1, the whole shutdown mechanism was rewritten there
[14:01] sustrik	so it's quite possible that the bug will just disappear when you switch
[14:02] BooTheHamster	Ok. I try to get version 2.1 sources from github and rebuild. Thank you/
[14:07] sustrik	mikko: are you there?
[14:08] mikko	yes
[14:11] sustrik	i was browsing thought the hudson results
[14:12] sustrik	and i've noticed jzmq does not build because javac is missing
[14:12] sustrik	have you installed java on the build box?
[14:12] mikko	hudson is java so yes
[14:12] mikko	might be missing javac
[14:12] sustrik	jdk?
[14:12] mikko	i recently moved it to a dedicated box
[14:12] mikko	let me check
[14:13] sustrik	thx
[14:13] mikko	jre installed but not jdk
[14:13] mikko	sec
[14:16] mikko	http://valokuva.org:8080/job/jzmq_master/23/console
[14:18] sustrik	:)
[14:18] mikko	pyzmq master build seems to get stuck
[14:18] mikko	probably close semantics i assume
[14:19] sustrik	possible
[15:57] CIA-14	zeromq2: 03Martin Sustrik 07master * r954d7f7 10/ (src/session.cpp src/session.hpp): (log message trimmed)
[15:57] CIA-14	zeromq2: Avoid duplicate creation of pipes for a single session
[15:57] CIA-14	zeromq2: When a session is being closed down its inbound and outbound
[15:57] CIA-14	zeromq2: pipe pointers are set to null. If (re) connection happens at
[15:57] CIA-14	zeromq2: that time, session may try to reinistantiate the pipes which
[15:57] CIA-14	zeromq2: is wrong. This patch allows session to attach pipes only once
[15:57] CIA-14	zeromq2: in its lifetime.
[15:58] sustrik	mikko: your assert should be solved in the master
[15:58] sustrik	(Assertion failed: inpipe_ && outpipe_ (xreq.cpp:42))
[16:00] mikko	sustrik: good stuff
[16:00] mikko	now i can add more tests for xre(q\|p)
[17:30] sustrik	bgranger: hi
[17:34] starkdg	is there any way to monitor the buffer lengths in zeromq ? how many messages in a queue at a given time ?
[17:44] bgranger	sustrik: hi
[17:44] sustrik	hi
[17:45] sustrik	i've noticed that there's some problem with pyzmq vs. master branch of the code
[17:45] sustrik	core
[17:45] sustrik	...
[17:45] sustrik	test_create (zmq.tests.test_socket.TestSocket) ... ok
[17:45] sustrik	test sending unicode objects ... ok
[17:45] sustrik	test the MessageTracker object for tracking when zmq is done with a buffer ... ERROR
[17:45] sustrik	Build was aborted
[17:45] sustrik	any idea what's going on there?
[17:46] sustrik	is it a bug in master?
[17:46] sustrik	starkds: no, there is not
[17:46] bgranger	We have not tested pyzmq with zeromq master yet.
[17:46] sustrik	ah, ok
[17:46] sustrik	i was just wondering
[17:46] bgranger	We are getting close to finishing up work on 2.0.9 (it was a lot) and after that we will start looking at 2.1
[17:46] sustrik	ack
[17:47] sustrik	have you seen mikko's build system btw?
[17:47] bgranger	Yes, it is very nice!
[17:47] sustrik	i love it :)
[17:47] bgranger	I can imagine that some bug have show up with 2.1 and pyzmq though. Although I would guess that at least some are in zeromq
[17:48] sustrik	definitely
[17:48] bgranger	But we will be able to tell pretty quickly once we start the 2.1 work.
[17:48] sustrik	i am fixing them gradually
[17:48] sustrik	sure, no problrm
[17:49] bgranger	But we are using 2.0.9 heavily and it is working quite well.
[17:53] delaney	has anyone tried pyzmq building against python 3.1?
[17:54] jdroid-	i'm sorta new to messaging, so i might get terminology wrong...
[17:55] jdroid-	anyway, i'm curious what the zmq approach is to handling handlers that went down
[17:55] jdroid-	if it's in the docs, please link me there. trying to work my way through those
[17:58] delaney	new too, but from what i can see the message would just keep queueing up for the disconnected client
[17:58] delaney	so you'd have to have some kind fo ping timeout or relie on the 'high water mark' settings
[17:59] jdroid-	hm.. ok.
[17:59] delaney	i also would like a better explanation of it
[18:01] bgranger	delaney: a few people have forks of pyzmq that semi-work with python 3, but they are not fully supported yet. There are some subtle issues about bytes and unicode that we are working out.
[18:02] delaney	k, i'll hold off for now, don't feel comfortable enough with the lib to know what's 'right' behavior
[18:03] delaney	so what does happen if say a REP/REQ connection if the REQ sends a request and the REP receives and before sending the REQ goes away? what's happens during the rep.socket.send?
[18:03] delaney	maybe i missed that in the guide
[18:09] jdroid-	so in the event of a time-out, what would be the 0mq way of handling that? does each box that would send a message to the failed handler just figure out that the box is down and pull it out of the round robin queue?
[18:11] jdroid-	sorry... my question assumes a setup like shown in these mongrel2 slides are used: http://www.slideshare.net/hungryblank/mongrel2-rugb
[18:12] jdroid-	ok. i think my question is based on a push/pull kind of situation. i guess it makes sense to hang until the handler comes up or time out.
[18:16] sustrik	jdroid: yes, if there's no handler it hangs
[18:16] sustrik	however, if the handler was processing the message while it crashed, it's lost
[18:16] jdroid-	ok. i'm sorta familiar with how zeromq doesn't offer persistance. it looksl ike some projects offer it via redis and other things
[18:17] delaney	so what happens in my example sustrik?
[18:18] sustrik	the message gets lost
[18:18] sustrik	so you have to timeout
[18:18] sustrik	and resend it
[18:18] sustrik	or something
[18:18] sustrik	in the future it would be nice to make REQ/REP do this automatically
[18:18] sustrik	but it's not there yet
[18:19] delaney	ah send with a return of !0 is a failure
[18:19] sustrik	send is async
[18:20] sustrik	so if the remote application fails, you won't be notified about it
[18:20] sustrik	you have to time-out
[18:20] delaney	right but it would return an ETERM if the other side disconnected?
[18:20] sustrik	no
[18:21] sustrik	ETERM means that the context you are running in was terminated
[18:21] delaney	ah
[18:22] delaney	i don't see a timeout param in socket.send
[18:22] sustrik	you have to use zmq_poll for that
[18:22] sustrik	there's a timeout param there
[18:22] delaney	gotcha, okay
[18:22] delaney	sorry for the confusion
[18:23] sustrik	you are welcome
[18:23] delaney	i noticed in the pyzmq there is a poll.Poller class and a poll.select method. any reason to use the select instead of the Poller?
[18:24] sustrik	that one is up to bgranger
[18:25] delaney	gotcha, and is there any roadmap for having reliable udp?
[18:32] bgranger	delaney: we provide both to be compatible with Python select and poll functions in the stdlib
[18:32] bgranger	Both use zmq_poll though
[18:33] bgranger	Both should be fully compatible with those in the stdlib
[18:33] delaney	right, i wondered why they are part of the Poller class directly?
[18:34] delaney	like Poller.select()
[18:34] sustrik	which reliable udp do you mean?:)
[18:35] delaney	i meant for games stuff sustrik. its by no means necessary but faster than tcp for high traffic game servers
[18:35] sustrik	i mean, which spec?
[18:35] sustrik	which rfc?
[18:35] delaney	i don't think there is one
[18:36] sustrik	there are dozens of them
[18:36] sustrik	there's rfc3208 integrated with 0mq
[18:36] sustrik	but that's meant to be used for multicast, not unicast
[18:36] delaney	what about 768?
[18:37] sustrik	that's raw udp
[18:38] sustrik	it would be nice to have unreliable udp transport
[18:38] sustrik	fancy writing it?
[18:38] delaney	yeah i know but most games build on that to add reliable ack/nack stuff
[18:38] sustrik	if you want to implement that then go on
[18:39] sustrik	but designing a network protocol is a non-trivial task
[18:39] delaney	well... A. i just got to using zmq so probably should get used to it. B. i'm a 3d artist guy, so yeah, can make you a pretty logo for it. :P
[18:39] Guthur	Surely unreliable UDP isn't that hard considering it already fulfils the 'unreliable' portion
[18:39] sustrik	guthur: yes
[18:39] sustrik	the reliability piece is the problem
[18:39] sustrik	also congestion control
[18:39] sustrik	etc.
[18:40] sustrik	delaney :)
[18:40] delaney	yeah, the reliable its a small thing... but having access to UDP on the socket would help
[18:40] delaney	depending on the game obviously
[18:41] andrewvc	delaney, I thought you were building a turn-based game. is TCP latency going to be an issue?
[18:41] delaney	no no, for my game its fine, but talking to some other indies about zmq cause it makes socket programming make sense, they had an rts in mind
[18:42] delaney	which is usually a peer-to-peer udp type of thing
[18:43] delaney	its in no means something i'd be using for my project, just a really good featue for other style games where availablity is more important than reliability
[18:44] sustrik	well, it requires some coding
[18:44] sustrik	so if there are couple of devs who really want the reliable udp thing
[18:45] sustrik	they can join forces to write one
[18:45] sustrik	plugging it into 0mq is a trifle
[18:45] Guthur	I personally wish someone would improve the Java support, it's not very good
[18:46] sustrik	Guthur: i've heard this multiple times
[18:46] sustrik	but nobody really specified what's wrong
[18:46] Guthur	its very lacking in most feature areas as far as I can see
[18:46] sustrik	specifically?
[18:46] Guthur	I only really has basic rep/req
[18:47] sustrik	i think other socket types are supported as well
[18:47] Guthur	It doesn't have devices as far as I can see
[18:48] Guthur	And I don't think it has named sockets either
[18:48] sustrik	?
[18:48] sustrik	ok, one by one
[18:48] Guthur	My 0mq knowledge is quite limited though
[18:49] sustrik	are you saying that pub/sub sockets are missing?
[18:49] Guthur	I only stumbled upon it recently
[18:49] sustrik	let me check
[18:49] sustrik	disclaimer: i am not jzmq dev
[18:50] Guthur	Though to be honest one had to use an earlier git commit than the most recent
[18:50] sustrik	there are all socket types defined...
[18:50] Guthur	due to a compile error
[18:50] sustrik	yeah, that's a real problem
[18:50] sustrik	we have to release 2.0.10
[18:50] sustrik	that has version macros in
[18:50] sustrik	then jzmq would be able to #ifdef the new stuff out
[18:51] sustrik	so it would be possible to build it with stable versions
[18:51] delaney	bgranger: is pyzmq your thing?
[18:51] sustrik	it's being worked on
[18:52] Guthur	sustrik, I'll keep an eye on it, cheers
[18:52] bgranger	delaney: i started it and am maintaining it, but others are working on it now too.
[18:52] Guthur	Java isn't my language of choice but a lot of business' are infatuated with it
[18:53] sustrik	Guthur: I am aware of that
[18:53] sustrik	the problem is that 0mq is a community project
[18:53] sustrik	thus people hack it for fun
[18:53] sustrik	which makes jun and .net a bit orphaned
[18:53] sustrik	java
[18:54] sustrik	no much people doing these two for fun
[18:54] bgranger	because they are not fun? ;-)
[18:54] sustrik	same witg windows platform :(
[18:54] Guthur	yeah I can understand that
[18:54] sustrik	well, it's something you get paid for to do
[18:55] Guthur	There would be quite handful of languages I would rather use before I get to Java or C#
[18:56] delaney	bgranger: i noticed that going through the guide the hwserver/client.py if you start the clients first then the server it just hangs. starting the server first does work fine.
[18:56] delaney	does the example need to set the ZMQ_INDENTIFY?
[18:57] sustrik	it's your choice, obviously, but people are doing a lot of C hacking for fun
[18:58] delaney	i now have binaries versions of pyzmq for windows for both 2.6 and 2.7 if ya'll want to add to the download page.
[19:01] sustrik	delaney: the problem is not supplying binaries, rather maintaining them in the future
[19:01] sustrik	creating new versions etc.
[19:02] sustrik	it would be even nice to have an installer
[19:02] sustrik	but it's just too much work
[19:03] sustrik	nobody volunteered for that so far :)
[19:13] bgranger	delaney: how did you create the binaries?
[19:21] delaney	with msvc 2010 and mingw
[19:22] delaney	its not 'the python way' but if you dump my folders into /python27/Lib/site-packages it 'just works'
[19:23] delaney	i actually avoided the library a while back cause the barrier to play with it was too high for a python specific dev
[19:23] delaney	i figure since there is no one doing it if ya'll want them at least as a starting point then it helps traction for pyzmq, which is awesome
[19:29] delaney	i see in the git master for pyzmq it using cython... does that mean there's going to a single dll to use?
[20:12] jhawk28	hello, is sustrik on?
[20:13] sustrik	yup, i'm here
[20:13] jhawk28	the patch for 68 didnt seem to solve the problem
[20:14] sustrik	yes, i've seen your comment
[20:14] jhawk28	Im not sure if I was doing something wrong
[20:14] sustrik	how did you apply the patch?
[20:14] jhawk28	I manually applied it to the pub.cpp
[20:14] jhawk28	then I compiled using the mvsc
[20:15] sustrik	ah, have you noticed the one line to be deleted?
[20:15] jhawk28	yes
[20:15] sustrik	then it's probably a different bug
[20:15] jhawk28	I deleted the lib file just in case
[20:16] sustrik	anyway, the original fix is now in both maint and master
[20:16] sustrik	in case you want to try
[20:16] sustrik	but as i said, it's probably a different bug
[20:16] jhawk28	the example on how to reproduce it was cut off by github, not sure why
[20:16] sustrik	yuck
[20:17] jhawk28	you basically create a PUB socket, but use a connect. Set the HWM to 1 after the connect
[20:17] jhawk28	iterate forever producing data
[20:17] sustrik	easy enough
[20:17] sustrik	i'll check it tomorrow
[20:18] jhawk28	Create a subscriber that binds to the address and does a simple read to get some items
[20:18] jhawk28	it works correctly when the HWM is set before the connect
[20:18] jhawk28	or if the pub socket uses a bind
[20:19] sustrik	:)
[20:19] sustrik	ok, i see
[20:20] sustrik	the point is that HWM is applied once, you cannot change it afterwards
[20:20] sustrik	so if you do setsockopt(HWM);connect();
[20:20] sustrik	the HWM applies
[20:20] jhawk28	yes
[20:20] sustrik	if you do connect();setsockopt(HWM);
[20:21] sustrik	is does not
[20:21] jhawk28	correct
[20:21] sustrik	so you have default HWM
[20:21] sustrik	which is infinite
[20:21] sustrik	then, if you are sending faster than receiving
[20:22] sustrik	the buffer will obviously grow ad infinitum
[20:22] jhawk28	yes
[20:22] jhawk28	and die of out of memory
[20:22] sustrik	yes
[20:24] jhawk28	I just wanted to make sure you knew what the problem was before spending more time on it
[20:24] delaney	bgranger: figure 17 of the user guide doesn't match the given python code
[20:24] bgranger	Let me look
[20:25] sustrik	jhawk28: yes, it works that way
[20:25] sustrik	set the HWM before doing connect and it should work fine
[20:27] bgranger	delaney: which part does not match?
[20:31] jhawk28	sustrik: ok if thats the way it works, ill document that in the issue and close it
[20:33] sustrik	yes, thanks!
[20:34] sustrik	jhawk28: btw, if it was man pages that confused you, feel free to propose a documentation patch
[20:47] mikko	good evening
[20:52] delaney	bgranger: the section above it is talking about setting up a XREQ->XREP/XREQ<-XREP setup but the rr_server/client.py are basically the same as the hwserver/client.py
[20:52] delaney	ah, wasn't looking at the broker.c part sorry
[20:52] delaney	would be nice to have the broker in py as well :)
[22:29] delaney	is there a good example of using polling with a REQ/XREP setup?
[22:30] delaney	i'm getting a Assertion failed: msg_->flags & ZMQ_MSG_MORE (..\..\..\src\req.cpp:225) when trying to recv() on a socket with a status of POLLIN
[22:31] delaney	also tried recv_multipart