Saturday November 20, 2010

[Time] NameMessage
[00:00] doug the need for identity seeming to nullify the benefit i'd otherwise realize via zmq.
[00:01] doug i think my app is better modeled as a collection of pairs, instead of a pub/sub app.
[00:01] doug with more shared state than pub/sub generally implies.
[00:02] travlr you probably need multiple zmq channels.. each for their own purposes
[00:02] doug yeah, i'm not sure how to abstract the purposes while still getting benefit out of zmq.
[00:03] travlr abstraction of 'purpose' is exactly what is fundamental to prodatalab
[00:04] travlr just consider a buffer that has a description attached to it..
[00:15] travlr doug, i just your discussion with Guther, a couple of things to mention..
[00:16] travlr zmq 'message' is just a blob of binary data on top of tcp or other transport layer
[00:16] travlr each message can be any size you want it to be
[00:16] travlr also..
[00:18] travlr zmq (in itself) is currently only designed for intranet purposes, there is no mechanism as of yet for internet boundries
[00:18] travlr here is where you might use zmq_poll() to interface with a normal BSD stlyle socket
[00:19] travlr on the edge... and use some other application layer for point to point on the internet
[00:19] travlr or use zmq over the internet and provide your own security mechanism on top of it
[00:20] travlr this is what the recent discussions have been about.. as well as _maybe_ incorporating some security features into the zmq infrastructure.
[00:21] travlr security == latency.. so careful consideration is being tossed about
[00:31] travlr huh. i thought i got my layers mixed up.. tcp-network and i meant transport not application over internet. sometimes i think/type to fase :P
[00:32] travlr see.. s/fast/fase/ .. lol
[00:48] nettok I can't connect to the zeromq log, the endpoint is "inproc://log" right?
[00:49] travlr is there a 'log' device? that's news to me.
[00:50] travlr where's that hosted?
[00:51] nettok travlr: I read it in the mailing list, it's supposed to be in 2.1 trunk
[00:52] travlr nice.. i'll check it out. thanks
[05:43] lantins hey guys; is PUSH ment to still block when I've got a HWM set and used ZMQ::NOBLOCK when sending?
[06:16] lantins i think the answers no, and im doing summets wrong
[08:08] lantins yup, something in a thread was causing the global vm lock (in ruby)
[10:06] mikko lestrrat: hi
[10:06] mikko lestrrat: the perl builds are currently failing
[10:07] lestrrat goodie
[10:08] lestrrat I haven't changed a line of code :/
[10:08] lestrrat oh well
[10:10] mikko lestrrat: it's been failing for a long time
[10:10] mikko i got all the modules installed but it says that i don't have the needed modules
[10:10] lestrrat after the last fix?
[10:10] mikko and it says im missing ''
[10:10] lestrrat oh.
[10:10] mikko
[10:10] mikko see the output
[10:10] mikko last success was Nov 8
[10:11] lestrrat The message is bad, but I think you'r missing Module::Install::TestTarget
[10:12] lestrrat you know what, why don't I write a build script
[10:12] lestrrat I'll do all the module installation there.
[10:12] lestrrat so you don't have to worry about it
[10:13] mikko installed the module and rebuilding
[10:14] mikko succeeds
[10:15] lestrrat how often do these tests run? is the trigger a new zmq commit?
[10:15] mikko 5am and 5pm on GMT
[10:15] lestrrat k.
[10:17] mikko should i disable the build against zeromq maint for perl?
[10:17] mikko it seems that several tests are failing
[10:21] lestrrat grr
[10:21] lestrrat sure go ahead
[10:22] lestrrat I won't be able to get everything done for a few days
[10:24] sustrik morning
[10:25] sustrik well, people can use master instead of main if they need perl
[11:04] lestrrat mikko: no hurries, but could you try running autobuild/ in hudson? (use the proper ZMQ_H env et al when you run this script: i.e. ZMQ_H=/path/to/zmq.h ./autobuild/
[11:04] lestrrat this should solve the dependency issues in the future
[11:04] mikko ok, ill give a go
[11:07] mikko lestrrat:
[11:07] mikko building there
[11:10] lestrrat seems like it worked :)
[11:10] lestrrat coolness
[11:33] clkao lestrrat!
[11:36] mikko all the builds that are expected succeed are now succeeding
[11:36] mikko thats a good thing
[11:37] mikko lestrrat: do you want emails for build failures in perl bindings?
[11:39] lestrrat hi clkao :)
[11:39] lestrrat yeah, I think that would be good
[11:39] lestrrat > re: emails
[11:40] mikko lestrrat: can you give me an email in which to send them?
[11:41] lestrrat
[11:41] mikko added
[11:42] lestrrat thanks!
[12:34] Guthur when does the logs get updated?
[12:42] lantins i can't figure out what the hell's going on here...
[12:43] lantins I'm forking some workers, 4 of them, when there doing something like sleep() they seem to get a message, then start sleeping. now when i swap the sleep() out with a http request, it only seems to send the request once I've PUSH'ed 4 messages
[12:44] lantins tried swapping out the http stuff a couple of times, so either they all have the same issue. or I'm not doing somthing right with zmq
[12:44] lantins any ideas?
[13:09] travlr Guthur: hi. I haven't put the log updater app on a cron yet. I'll have that done in a day or two.
[13:11] travlr lantins: do you have a chunk of code you can post on gist or pastebin or something?
[13:13] travlr lantins: are you using zmq_poll() for the http stuff.. are interfacing with a bsd socket for the http stuff or is this all pure zmq?
[13:15] lantins travlr: i can gist the code, its a total hack at the moment though mind =)
[13:16] travlr sure..
[13:16] lantins travlr: not using zmq_poll. got ruby eventmachine dealing with sockets, that PUSH' to a buffer, that does some things, then PUSH' to the 'workers'. there sending http requests out
[13:16] lantins i'll stick it up in a sec
[13:17] travlr well if your using ruby i'm not familiar with it
[13:20] travlr Guthur: I just uploaded the changes. it'll take a few minutes (maybe) for github servers to host them.
[13:22] lantins tried to remove some of the gruff. i started off trying to do this all in threads, but ended up pulling it out into seperate processes to test
[13:22] lantins travlr:
[13:23] lantins buffer_thread.rb, http_workers.rb and socket_server.rb would be the 3 'components'
[13:24] travlr sorry i can't get a grip that easily on the ruby stuff. don't have time right now to dig into it deeper.
[13:25] lantins travlr: no worries
[13:25] lantins i've a feeling its a threading issue or something other then zmq tbh
[13:26] travlr yeah i doubt its zmq itself. it has a small learning curve but i don't know how it works at all with eventmachine.
[13:27] travlr i
[13:27] lantins how it is in the gist, the side im having issues with isn't running inside EM at all
[13:28] lantins im gonna fire up the workers as individual processes rather then threads, if the problem goes away... its threading problems
[13:28] travlr yeah processes will probably easier from the start
[13:28] travlr are there ruby examples on the zmq website... use those to start with.
[13:37] lantins yup works perfect when im not threading
[13:37] travlr cool
[13:37] travlr were you using the inproc protocol for the thread communication?
[13:37] lantins i was to begin with
[13:38] lantins then moved to ipc
[13:38] travlr ipc for process, inproc for threads right?
[13:38] lantins yeah
[13:39] lantins worked fine if i was just calling sleep() in the thread, must be something to do with Net::HTTP and threading
[13:40] lantins to be fair, at the moment I'm forking for every 'message'. so just forking once for however many workers I need would work much better i'm hoping
[13:40] lantins or just fire them up seperately
[13:40] travlr yeah play around with your use case
[13:41] lantins this is very much throw away code =)
[13:41] travlr ya
[13:41] lantins was lotta fun until i hit this threading issue, then atleast 4 hours of my time just seemed to go AWOL!
[13:41] travlr have you worked with examples yet
[13:41] travlr on the website
[13:42] lantins spent the best part of a day going through the epic guide
[13:42] lantins still keep going back to it... like i said... EPIC!
[13:43] travlr yeah i'm still working on intricacies.. but pieter did a really nice job
[13:43] lantins props to the people who wrote that, its been a good insight into this 'messaging' lark =)
[13:43] travlr pieter h
[13:44] travlr hes good at that stuff. i'm a good doc advocate, big time
[13:44] lantins think thats the guy who has epic long blog posts too heh
[13:44] travlr he is the man.. lol
[13:44] lantins yeah, i like how its got humour throughout, given there is so much content
[13:45] travlr i'm more of a just the facts kinda guy myself... give me a good succinct outline any time.
[13:46] travlr with plenty of hyper cross-links
[13:46] lantins not having played with messaging stuff before, its good for getting going
[13:46] travlr ya
[13:46] lantins ruby ffi code comments have been very useful to get the context moved over from C to Ruby
[13:47] travlr zmq is a blessing imo, especially for what i'm using it for.
[13:48] travlr yeah, i haven't played with ruby at all yet
[13:48] lantins i think its gonna be very useful for us, initial benchmarks with this toy code look much nicer then anything were currently getting
[13:49] travlr yup
[13:49] lantins what language you using zmq with mostely?
[13:49] travlr c c++ and python
[13:49] lantins =)
[13:49] travlr c++ through qt mostly... i like qt alot
[13:50] travlr especially now that someone recently mashed up zmq with qt
[13:50] travlr zeromqt
[13:50] lantins im pondering down the line, if it all looks a good fit... about implementing some of this in C, the rest can stay ruby though
[13:50] travlr sure
[13:50] lantins yeah i seen that on the labs page
[13:51] travlr i'm playing with it right now... nice
[13:51] lantins :)
[13:52] lantins obj-c I'd like to get around to playing with at some point
[13:52] travlr sure, i'm not a apple guy so i don't see where else it's used much
[13:54] travlr what are you going to be using zmq for
[13:54] lantins although, right now I'm pondering how to better 'route' these PUSH/PULL messages, rather then fair-routing? but i guess this is where you start building 'device' like stuff to wire that kinda thing up?
[13:54] lantins at the moment, I've got a couple bits of software that are part of a 'system'. they talk to one another via json/http
[13:54] lantins we also push out lots of data to customers via json/http
[13:55] lantins so, im looking to remove some of the http stuff in our backend
[13:55] travlr high speed data or just a "normal" system?
[13:55] lantins the current system does work, but it could be made simpler
[13:56] lantins we got seconds to play with =)
[13:56] travlr cool
[13:56] travlr so your crossing internet boundries?
[13:57] lantins a little bit of the system is, and the part where we talk to customer servers
[13:57] lantins but the rest is internal on the lan
[13:57] lantins were dealing with vehicle tracking/diagnostics data
[13:58] travlr oh cool. that sounds very interesting
[13:58] lantins most vehicles update between 60sec and 5mins
[13:58] travlr very cool.
[13:58] lantins so if it takes 30sec to move through the system it doesn't really matter
[13:58] travlr what kind of 'tracking/diagnostics'
[13:59] lantins oh all kinds =)
[13:59] lantins we design/build the hardware our self
[13:59] travlr are you doing embeddable stuff too
[13:59] travlr well that kills that quest
[13:59] travlr ion
[13:59] travlr i'm learning that stuff too
[13:59] lantins we have gear with 4 CANBus channels on them, so anything you can get on the vehicle ECU's we can get in real time
[14:00] travlr yes yes....
[14:00] lantins heh
[14:00] travlr electronics is cool shit, imo
[14:00] travlr i wish i more time to play with it
[14:00] lantins I'm _not_ smart enough to be one of the guys building the hardware, but it has given me the chance to learn about it all.. and ask questions from people much smarter then me
[14:01] travlr very cool
[14:01] lantins i just shovel data and make web apps with it lol
[14:01] travlr sensor systems are also really cool
[14:01] lantins AVR chips are very good to play around with
[14:01] travlr avr.... awesome!
[14:02] travlr linux32 embedded
[14:02] travlr i want to play with all that stuff real bad.. no time right now
[14:02] lantins was gifted a development AVR board that runs linux form work, with two ethernet's!
[14:02] lantins not got around to toying with that yet :/
[14:03] travlr to start with, i want to build my programmer.. but that's a chicken and egg dilema.. huh.. lol
[14:03] lantins I'll be able to buzz people into the flat without leaving my computer once I get around to soldering my first 'official' electronics project! via bluetooth!
[14:04] travlr what electronics supplier would you use?
[14:04] lantins not quite got the BT cracked yet, and I need to get work to mount it for me... when i seen the £9 price, i didn't pay any attention to the fact it was bloody surface mount!
[14:04] lantins not sure where you are? I'm in the UK
[14:05] travlr surface mount is very tricky for a beginner i would imaging
[14:05] travlr in usa
[14:05] lantins is where I've got all my stuff from
[14:05] travlr that's uk?
[14:05] lantins US too
[14:05] lantins they have _Everything_ on that site
[14:05] travlr i'll check it out..
[14:05] lantins if you got a part number, you can find it
[14:06] travlr is it more for hobbiests... are they fast on delivery?
[14:06] lantins now, the problem is when you don't have a part number and have to go through everything to figure out if the component does what you want it to!
[14:07] lantins they are a big company, not really aimed at hobbiests, you do kinda have to know what to start looking for
[14:07] travlr fast delivery?
[14:07] lantins not that I really do, but its the only place I've found where I got options
[14:07] lantins yeah super fast for me
[14:07] lantins can't comment about the states though
[14:07] travlr thats good... small orders ok?
[14:07] travlr projects/
[14:08] lantins yeah have been for me
[14:08] travlr good
[14:08] lantins what i did was this:
[14:08] lantins
[14:08] lantins that's a real nice document about making an Arduino clone
[14:09] lantins I looked at the components needed to build a handful, then prospected into some other 'usual' sized components
[14:10] travlr there's a lot of great info out on the web
[14:10] travlr that proj looks great
[14:10] travlr i wish i had the time
[14:10] travlr i love teaching myself shit
[14:10] lantins they use a Resonator in that design, I opted for a crystal
[14:10] lantins yeah
[14:10] lantins i know the feeling
[14:11] travlr yeah crystal.. mmmm gooooood.
[14:11] travlr lol
[14:11] lantins i have to say, i sunk a good week or so getting going with all this stuff, and I know I still know naff all about it
[14:11] travlr i've studied a shit load
[14:12] travlr i have four tv's opened up in my 'project' room
[14:12] lantins yeah crystal's smalled cooler then Resonators ;-) (crystal were the better option from what I descovered anyhow)
[14:12] travlr need caps
[14:12] lantins lol
[14:12] travlr radio shack sucks for parts
[14:12] travlr i haven't ordered them yet
[14:12] lantins my problem is when I keep blowing shit up
[14:12] travlr _lol_
[14:12] lantins "ah fuck that cost £4 and I've run out of them!!!!"
[14:13] travlr lol
[14:13] travlr damn, i want to play
[14:13] lantins i think the smell of burning electronics is part of the learning process
[14:13] travlr yup!
[14:13] lantins hell man i managed to keep that stuff at bay for ages and now you've got me all thinking about it!
[14:13] travlr i repeat.. i want to play
[14:13] travlr no time
[14:13] lantins I got real work to be doing lol
[14:14] travlr speaking of which
[14:14] travlr gotta go
[14:14] lantins any free time has been spent with iOS/iPad development
[14:14] lantins take it easy travlr
[14:14] lantins catch you later =)
[14:14] travlr see ya
[15:06] Guthur travlr, Cheers for the heads up on the logs
[15:09] travlr sure
[16:06] bfrog is there a sane way to get queues to work with an existing "reactor" like libevent or whatever?
[16:07] bfrog I mean, whats the point of sockets if you can't do event driven programming :-P
[16:08] travlr bfrog, existing reactor... sure... libevent, i think i remember that that's been done. did you check the mail list and irc archives?
[16:09] bfrog no, I looked on the website
[16:09] travlr there's a nice mashup with qt signal and slots if you do qt
[16:10] travlr the mail list is easily found on the website, here's the irc log:
[16:14] mikko bfrog:
[16:28] bfrog I see it, thanks
[16:29] mikko bfrog: and zmq provides it own polling mechnism as well
[16:29] mikko so you are not really enforced to use external event loop
[16:30] bfrog I get that
[16:31] bfrog but thats again, really limiting
[16:31] bfrog well not again, but yeah, thats faily limiting if I can't use my own
[16:42] bfrog it doesn't look like I can just provide callbacks for when messages are available, call this function sort of thing?
[17:25] Guthur bfrog you could easily build it ontop of polling
[17:25] Guthur clrzmq2 uses event delegates for polling
[17:27] bfrog I don't want to poll, I like my callbacks and selects() :-(
[17:31] mikko bfrog: the zmq_poll is an abstraction over different polling methods such as epoll and select
[17:31] mikko adding callbacks to that is a couple of lines of code
[17:32] mikko but if you want to use an external event loop just use ZMQ_FD sockopt
[17:32] bfrog but I might not have the whole message in the queue then?
[17:32] mikko not necessarily
[17:33] bfrog right, so the recv may still block
[17:33] mikko you can use ZMQ_EVENTS to check that
[17:33] mikko
[17:42] bfrog what if I get a partial message, then I'm constantly looping on ZMQ_EVENTS, how would I break that?
[17:42] bfrog I can't touch the fd itself, can I tell zeromq to just give up somehow?
[17:46] Guthur you can just stop calling poll, and that's it surely
[17:46] Guthur The example code just immediately timeouts, you might want something slightly different
[18:14] cremes bfrog: you are *guaranteed* to have a whole message; there is no such thing in 0mq as a partial message
[18:14] cremes therefore, no worries about it blocking
[18:14] cremes if you are still concerned, pass the NO_BLOCK flag to your send or receive and go on your merry way
[18:17] cremes lantins: you may also be interested in looking at zmqmachine which is essentially eventmachine for 0mq sockets; find it on github
[18:22] mikko cremes: if you poll on ZMQ_FD you could have false positives
[18:22] mikko you need to check for ZMQ_EVENTS for the socket as well if you want to make sure that you dont block
[18:23] cremes mikko: right; so what's the problem? it's a pretty clear 2-step process
[18:23] cremes alternately, try to send/recv with NO_BLOCK and check for EAGAIN
[18:23] cremes either way, there is no concern about blocking
[18:25] mikko no problem there
[18:25] mikko just wanted to indicate that a 'positive' on polling on ZMQ_FD doesn't mean that you can recv()
[18:25] cremes ah, i see
[18:26] cremes i agree with that
[18:26] cremes hopefully bfrog reads through this at some point and this answers his/her question
[18:26] bfrog yeah it does
[18:26] cremes great
[18:27] cremes bfrog: fyi, the FD & EVENTS opts were added *specifically* to allow for 0mq socket integration with other event loops
[18:27] cremes if you find a case where they are inadequate, please speak up
[19:26] rgl hi