Tuesday August 24, 2010

[Time] NameMessage
[02:00] zedas sustrik: so i'm back on the trail of that bug with the 100% cpu in mongrel2.
[02:00] zedas sustrik: so far, the EINTR in the zmq_poll is actually coming from one of the io threads. it's not a real signal.
[02:01] emacsen I suspect sustrik is asleep :)
[02:01] zedas sustrik: also, when it happens, the receiver gets duplicate messages. i've checked the mongrel2 code i'm only sending the message once. right after they get the double message, the handler dies and mongrel2 goes 100%
[02:01] emacsen it's quite late there
[02:01] zedas emacsen: ah, well he'll see them when he gets up
[04:49] shekispeaks hi
[04:49] shekispeaks an
[04:49] shekispeaks any examples around using zmq_queue in python
[05:19] sustrik zedas: morning
[05:19] sustrik you still there?
[05:20] sustrik shekispeaks: it's an executable, it is used the same in all languages
[05:21] zedas sustrik: yep
[05:21] zedas sustrik: so i've been running tests thrashing with IO threads set to 1 and no problems.
[05:22] zedas definitely a problem with multiple IO threads, duplicate message gets sent, and then EINTR in zmq_poll and the receiving handler stalls.
[05:22] sustrik ok, do you have a test program?
[05:23] zedas i'll work something up. right now the test program is thrashing the hell out of mongrel2
[05:23] zedas with threads > 1 it locks immediately
[05:23] zedas so i should be able to reproduce
[05:23] sustrik ok
[06:12] shekispeaks what is the python binding for zmq-device
[06:13] sustrik zmq_device is an executable
[06:14] sustrik you can launch it from the command line
[06:14] sustrik oops
[06:14] sustrik sorry, i thought you meant zmq_queue
[06:15] sustrik as for python binding for zmq_deivce, i don't know, sorry
[06:17] shekispeaks the documentation is lacking for the bindings
[06:17] shekispeaks :)
[06:18] sustrik maybe it's not there
[06:18] sustrik you can provide a patch then
[06:39] xrfang hi, I get this error: Failed to send message: Operation cannot be accomplished in current state, the program must receive the reply from server before it can send second message, why?
[06:45] zedas xrfang: different types of sockets have different usage pattersn. A REQ has to send, then receive, but a REP socket receives then sends. PUB can only send, SUB can only receive. and so on.
[06:45] zedas xrfang: you probably tried to do something that the socket type didn't support.
[06:50] travlr zedas: just got done reading i'll like what you've done with the concept, source as well as the docs.
[06:51] travlr *
[07:04] zedas travlr: cool glad you like it. shooting for a minimalist 1.0 aug 31 and looks like we might do it
[07:04] travlr i'm looking forward to using it..
[07:39] CIA-20 zeromq2: 03Martin Sustrik 07wip-shutdown * r2a52455 10/ src/session.cpp : sessions created by listerner are correctly shut down -
[07:57] CIA-20 zeromq2: 03Martin Sustrik 07wip-shutdown * r2022dbd 10/ (src/zmq_engine.cpp src/zmq_listener.cpp src/zmq_listener.hpp): listener object unregisters its fd correctly -
[08:08] rbraley zedas, say, where in your code is your coroutine implementation? Do you have an M:N threading model?
[08:11] rbraley hehe, libtask eh? I thought there might be something to do with plan9 in there. Is this derived from libthread?
[08:12] zedas rbraley: not sure, i know russ cox wrote it years ago but probably just inspired by various other coroutine libs
[08:12] zedas rbraley: i also don't do N:M threads explicitly. they're pretty hard to coordinate, so instead i just have a fast as hell I/O coroutine processing in the main thread, then 0mq gives me threading on the IO it works with.
[08:13] zedas next up is to let you load handlers that are inproc and taken from .so files, so you can offload work into threads safely
[08:14] rbraley zedas, I was thinking about sending coroutines over zeromq to different threads with fair queuing. That way M:N is easier.
[08:14] rbraley The trick is getting coroutines to serialize
[08:15] zedas oh good luck with that! :-)
[08:16] zedas you do know that *Lua* can do that though right?
[08:16] zedas you can have a Lua vm start a bit of code, have the code yield in a coroutine, then back in C land pull that coroutine out and make it into a new VM, then send that whole kit over to a separate thread and let it continue there
[08:17] rbraley I suppose Lua is worth a try. I wish it's syntax was more like python but at least it doesn't have the GIL
[08:17] zedas yep, lua is very threadsafe, as long as you don't share vms between threads you can go fairly crazy with it.
[08:18] zedas and since it's so easy to spawn off new vms you just don't have a problem.
[08:18] zedas but, as for getting something like libtask to swap around between threads, it'd be a trick and a half.
[08:18] zedas i'm sure you could make it work, but coordinating the shuffle will be very error prone.
[08:19] rbraley zedas, yeah I was looking at the Go code to see if I could gain anything. I found the G struct and it is pretty nifty
[08:19] rbraley
[08:21] CIA-20 zeromq2: 03Martin Sustrik 07wip-shutdown * rc573c5c 10/ (src/zmq_connecter.cpp src/zmq_connecter.hpp): connecter object unregisters its fd correctly -
[08:21] zedas uh why is Go not written in Go?
[08:21] rbraley apparently they have Goroutines move to a different or new system thread if one of them blocks on a system call so they don't have to wait :D
[08:22] zedas sigh. i need to write my book on C.
[08:22] zedas i mean yeah, the guy who wrote C works on this code.
[08:22] rbraley zedas, do tell
[08:22] zedas but you think he'd stop naming shit M, G, cret, tls
[08:22] rbraley hehe
[08:23] zedas i mean are they missing fingers or something?
[08:23] rbraley I kinda like the laconic naming conventions
[08:24] rbraley as long as there are adequate comments
[08:24] zedas nah, it's why people think C is hard. i mean, if the experts write it this way, obviously you can't write it clean and readably
[08:25] zedas and then weird things like trying to use tabs to line stuff up, but then naming variables and functions so nobody can read them.
[08:25] rbraley zedas, if you think this code is bad try the Erlang source!
[08:25] zedas oh well, i'm sure if google's using it everyone will pick it up.
[08:26] zedas oh i know, i've looked at it.
[08:26] rbraley
[08:26] rbraley yuk
[08:26] zedas it's like the inverse of javascript.
[08:26] zedas when good programmers code javascript their sense of style gets turned off and they crank out unreadable barely working crap.
[08:27] zedas with C, their sense of style turns off and they write unreadable very magic crap.
[08:27] rbraley I wonder how big a Lua vm is? I wonder if I can spawn a bunch of em or if it is just too heavy.
[08:27] rbraley zedas, haha
[08:28] zedas lua's C code is pretty good last time i read it.
[08:29] zedas yep good old erlang. 9000+ lines of ifdef convolution
[08:29] rbraley I'm making a game engine where every entity is an actor (as in actor model of concurrency) which communicate by message passing with 0MQ.
[08:29] rbraley I want an M:N threading model for optimal use of the hardware
[08:30] rbraley it seems like you are after a similar architecture with mongrel2
[08:31] travlr rbraley: btw -- another tidbit for your topical-vid viewing pleasure:
[08:31] keffo rbraley, lua is very performant
[08:32] zedas rbraley: you want lua
[08:32] zedas especially if you're doing a game.
[08:33] rbraley for those that don't know I have done some work on a Brain/Machine interface, that's why that's a topical vid
[08:34] rbraley alright, I'll see if I can make a bunch of lua + zeromq actor coroutines and see how much memory they take up
[08:34] keffo a coroutine is just a stack + ip
[08:35] rbraley well in this case it would also be a Lua vm :P
[08:38] keffo it mostly depends on what the vm does, starting it is very fast, loading libraries usually isnt
[08:40] rbraley travlr, yeah, a dude at my hackerspace has a neural headset like that.
[08:41] rbraley keffo, most actors will be just running functions and sending messages, the libraries will be loaded in the components which the entities will talk to.
[08:42] keffo then I would just do all of them as normal coroutines in one vm
[08:43] rbraley oh right :P that is smarter.
[08:43] keffo starting a vm from scratch and loading all the standard libs will take time (and ram obviously), but starting a coroutine is near instant
[08:44] keffo your cant use any blocking calls though :)
[08:44] rbraley travlr, actually that's the exact same neural headset that he has. Same software and everything.
[08:44] travlr yeah its cool chit
[08:44] rbraley keffo, well, I can spawn another thread if I have to make a blocking call
[08:45] keffo rbraley, why bother? Just dont block and be done with it :)
[08:46] keffo unless you're aiming for linear gain in performance, spawning another thread is just a lazy way to not block..
[08:47] rbraley ;)
[08:47] rbraley I don't want any blocking calls between frames ;)
[08:48] keffo then dont impose such a design :)
[08:49] rbraley I wouldn't dream of it :D
[08:54] jsimmons Lua is pretty light on memory, rbraley. ~300k with the whole shebang.
[08:54] rbraley nice!
[08:54] jsimmons you can cut it down really easily too
[08:55] jsimmons I have it down to 200k here just stripping out libraries and compiling for size.
[08:55] jsimmons but you most likely don't want to do that
[08:58] rbraley it'd be neat if Lua had pattern matching like Scala or Erlang
[08:59] rbraley or Haskell for that matter
[08:59] jsimmons lua has pro pattern matching
[08:59] rbraley really?
[08:59] jsimmons well, if by pro you mean really light, fast and covers most real use-cases.
[08:59] jsimmons and there's lpeg if you need more.
[09:01] jsimmons oh and if you need more performance, you can drop in LuaJIT 2 and get ~ the same speeds as the Java 6 vm
[09:02] rbraley interesting
[09:03] jsimmons it's a pretty cool platform, and for it's minimalism it's often used in the game/other embedded world.
[09:03] keffo jsimmons, 200kb?
[09:03] keffo on what platform?
[09:04] jsimmons yeah keffo linux, that's the .a, though.
[09:04] keffo oh
[09:06] keffo furthest I've come is a 17.8kb executable(win32), and printing hello world in a loop sits the process at around 568kb .. Could probably squeeze out more though, but not without a lot of fuzz
[09:08] rbraley woohoo it's a register-based VM
[09:08] rbraley meaning string manipulation won't take 3-4 instructions per character
[09:09] jsimmons the string manipulation is generally done in C anyway
[09:10] keffo and the lua string hash algo is very fast
[09:13] jsimmons and the c api is pretty fantastic
[09:14] keffo as long as you grasp the stack stuff properly, the language wont matter :)
[09:15] rbraley I like what I see. It may not be as pretty as python, but I already like two lua syntax decisions better than python ^ instead of ** and not needing to put self everywhere.
[09:16] keffo you'll use self soon enough in lua too
[09:19] CIA-20 zeromq2: 03Martin Sustrik 07wip-shutdown * rbdc4adc 10/ (6 files): elementary fixes to the named session -
[11:39] shekispeaks i am using the req/rep paradigm
[11:39] shekispeaks it is a basic server client architecture
[11:40] shekispeaks i know the requests are queued at the Server
[11:40] shekispeaks any idea how long are the messages queued. Can I break the processing into a seperate queue
[13:59] CIA-20 zeromq2: 03Martin Sustrik 07wip-shutdown * r713fbdd 10/ (3 files in 2 dirs): MSVC build fixed -
[14:29] mato sustrik: what does lb_t::active (number of active pipes) actually mean?
[14:29] mato sustrik: i.e. what is an "active pipe"?
[14:29] sustrik number of outbound pipes able to accept a message
[14:30] sustrik "high watermark is not yet reached"
[14:30] mato able, i.e. not full
[14:30] sustrik yes
[14:31] mato sustrik: so, you see, we had the "xreq drops messages" behaviour in the core for a while
[14:31] mato sustrik: you may remember a thread where i took that out
[14:31] sustrik yes
[14:32] mato sustrik: now, i think that what i actually want is for xreq to drop message if and only if there are no pipes at all
[14:32] mato sustrik: if they're all full, then it must block since otherwise HWM would be useless
[14:32] sustrik or if all the pipes are full
[14:32] mato no
[14:32] sustrik HWM limits the memory used
[14:32] sustrik that's it single purpose
[14:33] mato huh? it limits the queue length
[14:33] sustrik yes
[14:33] sustrik so it's not useless
[14:33] sustrik it limits the queue length
[14:34] sustrik think of HWM as SO_SNDBUF
[14:34] mato but SO_SNDBUF being full blocks your socket
[14:35] mato which is what i want, short queues on one end which block if they are full
[14:35] mato my problem is the case where there is "no queue"
[14:36] sustrik what you are doing is a hack, so do as you please
[14:37] mato yes, but i'm trying to understand the problem at the same time
[14:38] sustrik ok, in that case the message should be dropped in pipes are full
[14:38] sustrik this would make it behave as expected in devices
[14:38] sustrik i.e. "never block the execution"
[14:39] mato ok, and in the hypothetical case of a req/rep with resend, who's job would it then be to resend the request?
[14:40] mato the original requester's ?
[14:40] sustrik yes
[14:40] sustrik end-to-end reliability
[14:40] sustrik same as TCP
[20:06] TPL my first time in 0mq chat
[20:06] TPL anyone here?
[20:10] travlr hi tpl.. welcome.. whats up?
[20:34] TPL earlier i was investigating a problem using 0mq with visual studio 2008 in debug mode
[20:34] TPL BUT
[20:34] TPL i've now worked it out
[20:34] TPL this is my first time to 0mq chatroom
[20:34] TPL is this a good place to work these kind of problems out?
[20:34] travlr cool. glad you got it worked out
[20:34] TPL thanks
[20:35] TPL actually helping a colleague
[20:35] travlr yeah.. here and on the mail list to ask for help
[20:35] TPL i wasn't directly working on it
[20:37] TPL do you use 0mq under windows with visual studio 2008?
[20:37] travlr no i don't.. sorry
[20:38] TPL where should i go for windows knowledge related to 0mq?
[20:39] travlr on the website, mail list, or here
[20:39] TPL then i guess i'm in
[20:39] TPL thanks for taking time to answer my questions
[20:39] travlr sure
[21:15] ModusPwnens Hi, I'm trying to get a basic public subscribe server/client setup
[21:15] ModusPwnens published*
[21:16] ModusPwnens I read the documentation, but I wasn't sure about the endpoint parameter for the bind function
[22:37] ModusPwnens hello?