ZeroMq IRC Log

Tuesday August 24, 2010

[Time] Name	Message
[02:00] zedas	sustrik: so i'm back on the trail of that bug with the 100% cpu in mongrel2.
[02:00] zedas	sustrik: so far, the EINTR in the zmq_poll is actually coming from one of the io threads. it's not a real signal.
[02:01] emacsen	I suspect sustrik is asleep :)
[02:01] zedas	sustrik: also, when it happens, the receiver gets duplicate messages. i've checked the mongrel2 code i'm only sending the message once. right after they get the double message, the handler dies and mongrel2 goes 100%
[02:01] emacsen	it's quite late there
[02:01] zedas	emacsen: ah, well he'll see them when he gets up
[04:49] shekispeaks	hi
[04:49] shekispeaks	an
[04:49] shekispeaks	any examples around using zmq_queue in python
[05:19] sustrik	zedas: morning
[05:19] sustrik	you still there?
[05:20] sustrik	shekispeaks: it's an executable, it is used the same in all languages
[05:21] zedas	sustrik: yep
[05:21] zedas	sustrik: so i've been running tests thrashing with IO threads set to 1 and no problems.
[05:22] zedas	definitely a problem with multiple IO threads, duplicate message gets sent, and then EINTR in zmq_poll and the receiving handler stalls.
[05:22] sustrik	ok, do you have a test program?
[05:23] zedas	i'll work something up. right now the test program is thrashing the hell out of mongrel2
[05:23] zedas	with threads > 1 it locks immediately
[05:23] zedas	so i should be able to reproduce
[05:23] sustrik	ok
[06:12] shekispeaks	what is the python binding for zmq-device
[06:13] sustrik	zmq_device is an executable
[06:14] sustrik	you can launch it from the command line
[06:14] sustrik	oops
[06:14] sustrik	sorry, i thought you meant zmq_queue
[06:15] sustrik	as for python binding for zmq_deivce, i don't know, sorry
[06:17] shekispeaks	the documentation is lacking for the bindings
[06:17] shekispeaks	:)
[06:18] sustrik	maybe it's not there
[06:18] sustrik	you can provide a patch then
[06:39] xrfang	hi, I get this error: Failed to send message: Operation cannot be accomplished in current state, the program must receive the reply from server before it can send second message, why?
[06:45] zedas	xrfang: different types of sockets have different usage pattersn. A REQ has to send, then receive, but a REP socket receives then sends. PUB can only send, SUB can only receive. and so on.
[06:45] zedas	xrfang: you probably tried to do something that the socket type didn't support.
[06:50] travlr	zedas: just got done reading mongrol2.org. i'll like what you've done with the concept, source as well as the docs.
[06:51] travlr	*mongrel2.org
[07:04] zedas	travlr: cool glad you like it. shooting for a minimalist 1.0 aug 31 and looks like we might do it
[07:04] travlr	i'm looking forward to using it..
[07:39] CIA-20	zeromq2: 03Martin Sustrik 07wip-shutdown * r2a52455 10/ src/session.cpp : sessions created by listerner are correctly shut down - http://bit.ly/aZSPJY
[07:57] CIA-20	zeromq2: 03Martin Sustrik 07wip-shutdown * r2022dbd 10/ (src/zmq_engine.cpp src/zmq_listener.cpp src/zmq_listener.hpp): listener object unregisters its fd correctly - http://bit.ly/dzDxI0
[08:08] rbraley	zedas, say, where in your code is your coroutine implementation? Do you have an M:N threading model?
[08:11] rbraley	hehe, libtask eh? I thought there might be something to do with plan9 in there. Is this derived from libthread?
[08:12] zedas	rbraley: not sure, i know russ cox wrote it years ago but probably just inspired by various other coroutine libs
[08:12] zedas	rbraley: i also don't do N:M threads explicitly. they're pretty hard to coordinate, so instead i just have a fast as hell I/O coroutine processing in the main thread, then 0mq gives me threading on the IO it works with.
[08:13] zedas	next up is to let you load handlers that are inproc and taken from .so files, so you can offload work into threads safely
[08:14] rbraley	zedas, I was thinking about sending coroutines over zeromq to different threads with fair queuing. That way M:N is easier.
[08:14] rbraley	The trick is getting coroutines to serialize
[08:15] zedas	oh good luck with that! :-)
[08:16] zedas	you do know that Lua can do that though right?
[08:16] zedas	you can have a Lua vm start a bit of code, have the code yield in a coroutine, then back in C land pull that coroutine out and make it into a new VM, then send that whole kit over to a separate thread and let it continue there
[08:17] rbraley	I suppose Lua is worth a try. I wish it's syntax was more like python but at least it doesn't have the GIL
[08:17] zedas	yep, lua is very threadsafe, as long as you don't share vms between threads you can go fairly crazy with it.
[08:18] zedas	and since it's so easy to spawn off new vms you just don't have a problem.
[08:18] zedas	but, as for getting something like libtask to swap around between threads, it'd be a trick and a half.
[08:18] zedas	i'm sure you could make it work, but coordinating the shuffle will be very error prone.
[08:19] rbraley	zedas, yeah I was looking at the Go code to see if I could gain anything. I found the G struct and it is pretty nifty
[08:19] rbraley	http://code.google.com/p/go/source/browse/src/pkg/runtime/runtime.h
[08:21] CIA-20	zeromq2: 03Martin Sustrik 07wip-shutdown * rc573c5c 10/ (src/zmq_connecter.cpp src/zmq_connecter.hpp): connecter object unregisters its fd correctly - http://bit.ly/cnKJLS
[08:21] zedas	uh why is Go not written in Go?
[08:21] rbraley	apparently they have Goroutines move to a different or new system thread if one of them blocks on a system call so they don't have to wait :D
[08:22] zedas	sigh. i need to write my book on C.
[08:22] zedas	i mean yeah, the guy who wrote C works on this code.
[08:22] rbraley	zedas, do tell
[08:22] zedas	but you think he'd stop naming shit M, G, cret, tls
[08:22] rbraley	hehe
[08:23] zedas	i mean are they missing fingers or something?
[08:23] rbraley	I kinda like the laconic naming conventions
[08:24] rbraley	as long as there are adequate comments
[08:24] zedas	nah, it's why people think C is hard. i mean, if the experts write it this way, obviously you can't write it clean and readably
[08:25] zedas	and then weird things like trying to use tabs to line stuff up, but then naming variables and functions so nobody can read them.
[08:25] rbraley	zedas, if you think this code is bad try the Erlang source!
[08:25] zedas	oh well, i'm sure if google's using it everyone will pick it up.
[08:26] zedas	oh i know, i've looked at it.
[08:26] rbraley	http://github.com/mfoemmel/erlang-otp/blob/master/erts/emulator/beam/erl_process.c
[08:26] rbraley	yuk
[08:26] zedas	it's like the inverse of javascript.
[08:26] zedas	when good programmers code javascript their sense of style gets turned off and they crank out unreadable barely working crap.
[08:27] zedas	with C, their sense of style turns off and they write unreadable very magic crap.
[08:27] rbraley	I wonder how big a Lua vm is? I wonder if I can spawn a bunch of em or if it is just too heavy.
[08:27] rbraley	zedas, haha
[08:28] zedas	lua's C code is pretty good last time i read it.
[08:29] zedas	yep good old erlang. 9000+ lines of ifdef convolution
[08:29] rbraley	I'm making a game engine where every entity is an actor (as in actor model of concurrency) which communicate by message passing with 0MQ.
[08:29] rbraley	I want an M:N threading model for optimal use of the hardware
[08:30] rbraley	it seems like you are after a similar architecture with mongrel2
[08:31] travlr	rbraley: btw -- another tidbit for your topical-vid viewing pleasure: http://www.ted.com/talks/tan_le_a_headset_that_reads_your_brainwaves.html
[08:31] keffo	rbraley, lua is very performant
[08:32] zedas	rbraley: you want lua
[08:32] zedas	especially if you're doing a game.
[08:33] rbraley	for those that don't know I have done some work on a Brain/Machine interface, that's why that's a topical vid
[08:34] rbraley	alright, I'll see if I can make a bunch of lua + zeromq actor coroutines and see how much memory they take up
[08:34] keffo	a coroutine is just a stack + ip
[08:35] rbraley	well in this case it would also be a Lua vm :P
[08:38] keffo	it mostly depends on what the vm does, starting it is very fast, loading libraries usually isnt
[08:40] rbraley	travlr, yeah, a dude at my hackerspace has a neural headset like that.
[08:41] rbraley	keffo, most actors will be just running functions and sending messages, the libraries will be loaded in the components which the entities will talk to.
[08:42] keffo	then I would just do all of them as normal coroutines in one vm
[08:43] rbraley	oh right :P that is smarter.
[08:43] keffo	starting a vm from scratch and loading all the standard libs will take time (and ram obviously), but starting a coroutine is near instant
[08:44] keffo	your cant use any blocking calls though :)
[08:44] rbraley	travlr, actually that's the exact same neural headset that he has. Same software and everything.
[08:44] travlr	yeah its cool chit
[08:44] rbraley	keffo, well, I can spawn another thread if I have to make a blocking call
[08:45] keffo	rbraley, why bother? Just dont block and be done with it :)
[08:46] keffo	unless you're aiming for linear gain in performance, spawning another thread is just a lazy way to not block..
[08:47] rbraley	;)
[08:47] rbraley	I don't want any blocking calls between frames ;)
[08:48] keffo	then dont impose such a design :)
[08:49] rbraley	I wouldn't dream of it :D
[08:54] jsimmons	Lua is pretty light on memory, rbraley. ~300k with the whole shebang.
[08:54] rbraley	nice!
[08:54] jsimmons	you can cut it down really easily too
[08:55] jsimmons	I have it down to 200k here just stripping out libraries and compiling for size.
[08:55] jsimmons	but you most likely don't want to do that
[08:58] rbraley	it'd be neat if Lua had pattern matching like Scala or Erlang
[08:59] rbraley	or Haskell for that matter
[08:59] jsimmons	lua has pro pattern matching
[08:59] rbraley	really?
[08:59] jsimmons	well, if by pro you mean really light, fast and covers most real use-cases.
[08:59] jsimmons	and there's lpeg if you need more.
[09:01] jsimmons	oh and if you need more performance, you can drop in LuaJIT 2 and get ~ the same speeds as the Java 6 vm
[09:02] rbraley	interesting
[09:03] jsimmons	it's a pretty cool platform, and for it's minimalism it's often used in the game/other embedded world.
[09:03] keffo	jsimmons, 200kb?
[09:03] keffo	on what platform?
[09:04] jsimmons	yeah keffo linux, that's the .a, though.
[09:04] keffo	oh
[09:06] keffo	furthest I've come is a 17.8kb executable(win32), and printing hello world in a loop sits the process at around 568kb .. Could probably squeeze out more though, but not without a lot of fuzz
[09:08] rbraley	woohoo it's a register-based VM
[09:08] rbraley	meaning string manipulation won't take 3-4 instructions per character
[09:09] jsimmons	the string manipulation is generally done in C anyway
[09:10] keffo	and the lua string hash algo is very fast
[09:13] jsimmons	and the c api is pretty fantastic
[09:14] keffo	as long as you grasp the stack stuff properly, the language wont matter :)
[09:15] rbraley	I like what I see. It may not be as pretty as python, but I already like two lua syntax decisions better than python ^ instead of ** and not needing to put self everywhere.
[09:16] keffo	you'll use self soon enough in lua too
[09:19] CIA-20	zeromq2: 03Martin Sustrik 07wip-shutdown * rbdc4adc 10/ (6 files): elementary fixes to the named session - http://bit.ly/9GVADQ
[11:39] shekispeaks	i am using the req/rep paradigm
[11:39] shekispeaks	it is a basic server client architecture
[11:40] shekispeaks	i know the requests are queued at the Server
[11:40] shekispeaks	any idea how long are the messages queued. Can I break the processing into a seperate queue
[13:59] CIA-20	zeromq2: 03Martin Sustrik 07wip-shutdown * r713fbdd 10/ (3 files in 2 dirs): MSVC build fixed - http://bit.ly/9supt0
[14:29] mato	sustrik: what does lb_t::active (number of active pipes) actually mean?
[14:29] mato	sustrik: i.e. what is an "active pipe"?
[14:29] sustrik	number of outbound pipes able to accept a message
[14:30] sustrik	"high watermark is not yet reached"
[14:30] mato	able, i.e. not full
[14:30] sustrik	yes
[14:31] mato	sustrik: so, you see, we had the "xreq drops messages" behaviour in the core for a while
[14:31] mato	sustrik: you may remember a thread where i took that out
[14:31] sustrik	yes
[14:32] mato	sustrik: now, i think that what i actually want is for xreq to drop message if and only if there are no pipes at all
[14:32] mato	sustrik: if they're all full, then it must block since otherwise HWM would be useless
[14:32] sustrik	or if all the pipes are full
[14:32] mato	no
[14:32] sustrik	HWM limits the memory used
[14:32] sustrik	that's it single purpose
[14:33] mato	huh? it limits the queue length
[14:33] sustrik	yes
[14:33] sustrik	so it's not useless
[14:33] sustrik	it limits the queue length
[14:34] sustrik	think of HWM as SO_SNDBUF
[14:34] mato	but SO_SNDBUF being full blocks your socket
[14:35] mato	which is what i want, short queues on one end which block if they are full
[14:35] mato	my problem is the case where there is "no queue"
[14:36] sustrik	what you are doing is a hack, so do as you please
[14:37] mato	yes, but i'm trying to understand the problem at the same time
[14:38] sustrik	ok, in that case the message should be dropped in pipes are full
[14:38] sustrik	this would make it behave as expected in devices
[14:38] sustrik	i.e. "never block the execution"
[14:39] mato	ok, and in the hypothetical case of a req/rep with resend, who's job would it then be to resend the request?
[14:40] mato	the original requester's ?
[14:40] sustrik	yes
[14:40] sustrik	end-to-end reliability
[14:40] sustrik	same as TCP
[20:06] TPL	my first time in 0mq chat
[20:06] TPL	anyone here?
[20:10] travlr	hi tpl.. welcome.. whats up?
[20:34] TPL	earlier i was investigating a problem using 0mq with visual studio 2008 in debug mode
[20:34] TPL	BUT
[20:34] TPL	i've now worked it out
[20:34] TPL	this is my first time to 0mq chatroom
[20:34] TPL	is this a good place to work these kind of problems out?
[20:34] travlr	cool. glad you got it worked out
[20:34] TPL	thanks
[20:35] TPL	actually helping a colleague
[20:35] travlr	yeah.. here and on the mail list to ask for help
[20:35] TPL	i wasn't directly working on it
[20:37] TPL	do you use 0mq under windows with visual studio 2008?
[20:37] travlr	no i don't.. sorry
[20:38] TPL	where should i go for windows knowledge related to 0mq?
[20:39] travlr	on the website, mail list, or here
[20:39] TPL	then i guess i'm in
[20:39] TPL	thanks for taking time to answer my questions
[20:39] travlr	sure
[21:15] ModusPwnens	Hi, I'm trying to get a basic public subscribe server/client setup
[21:15] ModusPwnens	published*
[21:16] ModusPwnens	I read the documentation, but I wasn't sure about the endpoint parameter for the bind function
[22:37] ModusPwnens	hello?