ZeroMq IRC Log

Saturday November 20, 2010

[Time] Name	Message
[00:00] doug	the need for identity seeming to nullify the benefit i'd otherwise realize via zmq.
[00:01] doug	i think my app is better modeled as a collection of pairs, instead of a pub/sub app.
[00:01] doug	with more shared state than pub/sub generally implies.
[00:02] travlr	you probably need multiple zmq channels.. each for their own purposes
[00:02] doug	yeah, i'm not sure how to abstract the purposes while still getting benefit out of zmq.
[00:03] travlr	abstraction of 'purpose' is exactly what is fundamental to prodatalab
[00:04] travlr	just consider a buffer that has a description attached to it..
[00:15] travlr	doug, i just your discussion with Guther, a couple of things to mention..
[00:16] travlr	zmq 'message' is just a blob of binary data on top of tcp or other transport layer
[00:16] travlr	each message can be any size you want it to be
[00:16] travlr	also..
[00:18] travlr	zmq (in itself) is currently only designed for intranet purposes, there is no mechanism as of yet for internet boundries
[00:18] travlr	here is where you might use zmq_poll() to interface with a normal BSD stlyle socket
[00:19] travlr	on the edge... and use some other application layer for point to point on the internet
[00:19] travlr	or use zmq over the internet and provide your own security mechanism on top of it
[00:20] travlr	this is what the recent discussions have been about.. as well as _maybe_ incorporating some security features into the zmq infrastructure.
[00:21] travlr	security == latency.. so careful consideration is being tossed about
[00:31] travlr	huh. i thought i got my layers mixed up.. tcp-network and i meant transport not application over internet. sometimes i think/type to fase :P
[00:32] travlr	see.. s/fast/fase/ .. lol
[00:48] nettok	I can't connect to the zeromq log, the endpoint is "inproc://log" right?
[00:49] travlr	is there a 'log' device? that's news to me.
[00:50] travlr	where's that hosted?
[00:51] nettok	travlr: I read it in the mailing list, it's supposed to be in 2.1 trunk http://blog.gmane.org/gmane.network.zeromq.devel/month=20100901
[00:52] travlr	nice.. i'll check it out. thanks
[05:43] lantins	hey guys; is PUSH ment to still block when I've got a HWM set and used ZMQ::NOBLOCK when sending?
[06:16] lantins	i think the answers no, and im doing summets wrong
[08:08] lantins	yup, something in a thread was causing the global vm lock (in ruby)
[10:06] mikko	lestrrat: hi
[10:06] mikko	lestrrat: the perl builds are currently failing
[10:07] lestrrat	goodie
[10:08] lestrrat	I haven't changed a line of code :/
[10:08] lestrrat	oh well
[10:10] mikko	lestrrat: it's been failing for a long time
[10:10] mikko	i got all the modules installed but it says that i don't have the needed modules
[10:10] lestrrat	after the last fix?
[10:10] mikko	and it says im missing ''
[10:10] lestrrat	oh.
[10:10] mikko	http://build.valokuva.org/job/ZeroMQPerl-master_ZeroMQ2-master_GCC/128/console
[10:10] mikko	see the output
[10:10] mikko	last success was Nov 8
[10:11] lestrrat	The message is bad, but I think you'r missing Module::Install::TestTarget
[10:12] lestrrat	you know what, why don't I write a build script
[10:12] lestrrat	I'll do all the module installation there.
[10:12] lestrrat	so you don't have to worry about it
[10:13] mikko	installed the module and rebuilding
[10:14] mikko	succeeds
[10:15] lestrrat	how often do these tests run? is the trigger a new zmq commit?
[10:15] mikko	5am and 5pm on GMT
[10:15] lestrrat	k.
[10:17] mikko	should i disable the build against zeromq maint for perl?
[10:17] mikko	it seems that several tests are failing
[10:21] lestrrat	grr
[10:21] lestrrat	sure go ahead
[10:22] lestrrat	I won't be able to get everything done for a few days
[10:24] sustrik	morning
[10:25] sustrik	well, people can use master instead of main if they need perl
[11:04] lestrrat	mikko: no hurries, but could you try running autobuild/run.sh in hudson? (use the proper ZMQ_H env et al when you run this script: i.e. ZMQ_H=/path/to/zmq.h ./autobuild/run.sh)
[11:04] lestrrat	this should solve the dependency issues in the future
[11:04] mikko	ok, ill give a go
[11:07] mikko	lestrrat: http://build.valokuva.org/job/ZeroMQPerl-master_ZeroMQ2-master_GCC/130/console
[11:07] mikko	building there
[11:10] lestrrat	seems like it worked :)
[11:10] lestrrat	coolness
[11:33] clkao	lestrrat!
[11:36] mikko	all the builds that are expected succeed are now succeeding
[11:36] mikko	thats a good thing
[11:37] mikko	lestrrat: do you want emails for build failures in perl bindings?
[11:39] lestrrat	hi clkao :)
[11:39] lestrrat	yeah, I think that would be good
[11:39] lestrrat	> re: emails
[11:40] mikko	lestrrat: can you give me an email in which to send them?
[11:41] lestrrat	lestrrat@gmail.com
[11:41] mikko	added
[11:42] lestrrat	thanks!
[12:34] Guthur	when does the logs get updated?
[12:42] lantins	i can't figure out what the hell's going on here...
[12:43] lantins	I'm forking some workers, 4 of them, when there doing something like sleep() they seem to get a message, then start sleeping. now when i swap the sleep() out with a http request, it only seems to send the request once I've PUSH'ed 4 messages
[12:44] lantins	tried swapping out the http stuff a couple of times, so either they all have the same issue. or I'm not doing somthing right with zmq
[12:44] lantins	any ideas?
[13:09] travlr	Guthur: hi. I haven't put the log updater app on a cron yet. I'll have that done in a day or two.
[13:11] travlr	lantins: do you have a chunk of code you can post on gist or pastebin or something?
[13:13] travlr	lantins: are you using zmq_poll() for the http stuff.. are interfacing with a bsd socket for the http stuff or is this all pure zmq?
[13:15] lantins	travlr: i can gist the code, its a total hack at the moment though mind =)
[13:16] travlr	sure..
[13:16] lantins	travlr: not using zmq_poll. got ruby eventmachine dealing with sockets, that PUSH' to a buffer, that does some things, then PUSH' to the 'workers'. there sending http requests out
[13:16] lantins	i'll stick it up in a sec
[13:17] travlr	well if your using ruby i'm not familiar with it
[13:20] travlr	Guthur: I just uploaded the changes. it'll take a few minutes (maybe) for github servers to host them.
[13:22] lantins	tried to remove some of the gruff. i started off trying to do this all in threads, but ended up pulling it out into seperate processes to test
[13:22] lantins	travlr: https://gist.github.com/fff5bf54b84e784610a0
[13:23] lantins	buffer_thread.rb, http_workers.rb and socket_server.rb would be the 3 'components'
[13:24] travlr	sorry i can't get a grip that easily on the ruby stuff. don't have time right now to dig into it deeper.
[13:25] lantins	travlr: no worries
[13:25] lantins	i've a feeling its a threading issue or something other then zmq tbh
[13:26] travlr	yeah i doubt its zmq itself. it has a small learning curve but i don't know how it works at all with eventmachine.
[13:27] travlr	i
[13:27] lantins	how it is in the gist, the side im having issues with isn't running inside EM at all
[13:28] lantins	im gonna fire up the workers as individual processes rather then threads, if the problem goes away... its threading problems
[13:28] travlr	yeah processes will probably easier from the start
[13:28] travlr	are there ruby examples on the zmq website... use those to start with.
[13:37] lantins	yup works perfect when im not threading
[13:37] travlr	cool
[13:37] travlr	were you using the inproc protocol for the thread communication?
[13:37] lantins	i was to begin with
[13:38] lantins	then moved to ipc
[13:38] travlr	ipc for process, inproc for threads right?
[13:38] lantins	yeah
[13:39] lantins	worked fine if i was just calling sleep() in the thread, must be something to do with Net::HTTP and threading
[13:40] lantins	to be fair, at the moment I'm forking for every 'message'. so just forking once for however many workers I need would work much better i'm hoping
[13:40] lantins	or just fire them up seperately
[13:40] travlr	yeah play around with your use case
[13:41] lantins	this is very much throw away code =)
[13:41] travlr	ya
[13:41] lantins	was lotta fun until i hit this threading issue, then atleast 4 hours of my time just seemed to go AWOL!
[13:41] travlr	have you worked with examples yet
[13:41] travlr	on the website
[13:42] lantins	spent the best part of a day going through the epic guide
[13:42] lantins	still keep going back to it... like i said... EPIC!
[13:43] travlr	yeah i'm still working on intricacies.. but pieter did a really nice job
[13:43] lantins	props to the people who wrote that, its been a good insight into this 'messaging' lark =)
[13:43] travlr	pieter h
[13:44] travlr	hes good at that stuff. i'm a good doc advocate, big time
[13:44] lantins	think thats the guy who has epic long blog posts too heh
[13:44] travlr	he is the man.. lol
[13:44] lantins	yeah, i like how its got humour throughout, given there is so much content
[13:45] travlr	i'm more of a just the facts kinda guy myself... give me a good succinct outline any time.
[13:46] travlr	with plenty of hyper cross-links
[13:46] lantins	not having played with messaging stuff before, its good for getting going
[13:46] travlr	ya
[13:46] lantins	ruby ffi code comments have been very useful to get the context moved over from C to Ruby
[13:47] travlr	zmq is a blessing imo, especially for what i'm using it for.
[13:48] travlr	yeah, i haven't played with ruby at all yet
[13:48] lantins	i think its gonna be very useful for us, initial benchmarks with this toy code look much nicer then anything were currently getting
[13:49] travlr	yup
[13:49] lantins	what language you using zmq with mostely?
[13:49] travlr	c c++ and python
[13:49] lantins	=)
[13:49] travlr	c++ through qt mostly... i like qt alot
[13:50] travlr	especially now that someone recently mashed up zmq with qt
[13:50] travlr	zeromqt
[13:50] lantins	im pondering down the line, if it all looks a good fit... about implementing some of this in C, the rest can stay ruby though
[13:50] travlr	sure
[13:50] lantins	yeah i seen that on the labs page
[13:51] travlr	i'm playing with it right now... nice
[13:51] lantins	:)
[13:52] lantins	obj-c I'd like to get around to playing with at some point
[13:52] travlr	sure, i'm not a apple guy so i don't see where else it's used much
[13:54] travlr	what are you going to be using zmq for
[13:54] lantins	although, right now I'm pondering how to better 'route' these PUSH/PULL messages, rather then fair-routing? but i guess this is where you start building 'device' like stuff to wire that kinda thing up?
[13:54] lantins	at the moment, I've got a couple bits of software that are part of a 'system'. they talk to one another via json/http
[13:54] lantins	we also push out lots of data to customers via json/http
[13:55] lantins	so, im looking to remove some of the http stuff in our backend
[13:55] travlr	high speed data or just a "normal" system?
[13:55] lantins	the current system does work, but it could be made simpler
[13:56] lantins	we got seconds to play with =)
[13:56] travlr	cool
[13:56] travlr	so your crossing internet boundries?
[13:57] lantins	a little bit of the system is, and the part where we talk to customer servers
[13:57] lantins	but the rest is internal on the lan
[13:57] lantins	were dealing with vehicle tracking/diagnostics data
[13:58] travlr	oh cool. that sounds very interesting
[13:58] lantins	most vehicles update between 60sec and 5mins
[13:58] travlr	very cool.
[13:58] lantins	so if it takes 30sec to move through the system it doesn't really matter
[13:58] travlr	what kind of 'tracking/diagnostics'
[13:59] lantins	oh all kinds =)
[13:59] lantins	we design/build the hardware our self
[13:59] travlr	are you doing embeddable stuff too
[13:59] travlr	well that kills that quest
[13:59] travlr	ion
[13:59] travlr	i'm learning that stuff too
[13:59] lantins	we have gear with 4 CANBus channels on them, so anything you can get on the vehicle ECU's we can get in real time
[14:00] travlr	yes yes....
[14:00] lantins	heh
[14:00] travlr	electronics is cool shit, imo
[14:00] travlr	i wish i more time to play with it
[14:00] lantins	I'm _not_ smart enough to be one of the guys building the hardware, but it has given me the chance to learn about it all.. and ask questions from people much smarter then me
[14:01] travlr	very cool
[14:01] lantins	i just shovel data and make web apps with it lol
[14:01] travlr	sensor systems are also really cool
[14:01] lantins	AVR chips are very good to play around with
[14:01] travlr	avr.... awesome!
[14:02] travlr	linux32 embedded
[14:02] travlr	i want to play with all that stuff real bad.. no time right now
[14:02] lantins	was gifted a development AVR board that runs linux form work, with two ethernet's!
[14:02] lantins	not got around to toying with that yet :/
[14:03] travlr	to start with, i want to build my programmer.. but that's a chicken and egg dilema.. huh.. lol
[14:03] lantins	I'll be able to buzz people into the flat without leaving my computer once I get around to soldering my first 'official' electronics project! via bluetooth!
[14:04] travlr	what electronics supplier would you use?
[14:04] lantins	not quite got the BT cracked yet, and I need to get work to mount it for me... when i seen the Â£9 price, i didn't pay any attention to the fact it was bloody surface mount!
[14:04] lantins	not sure where you are? I'm in the UK
[14:05] travlr	surface mount is very tricky for a beginner i would imaging
[14:05] travlr	in usa
[14:05] lantins	farnell.com is where I've got all my stuff from
[14:05] travlr	that's uk?
[14:05] lantins	US too
[14:05] lantins	they have _Everything_ on that site
[14:05] travlr	i'll check it out..
[14:05] lantins	if you got a part number, you can find it
[14:06] travlr	is it more for hobbiests... are they fast on delivery?
[14:06] lantins	now, the problem is when you don't have a part number and have to go through everything to figure out if the component does what you want it to!
[14:07] lantins	they are a big company, not really aimed at hobbiests, you do kinda have to know what to start looking for
[14:07] travlr	fast delivery?
[14:07] lantins	not that I really do, but its the only place I've found where I got options
[14:07] lantins	yeah super fast for me
[14:07] lantins	can't comment about the states though
[14:07] travlr	thats good... small orders ok?
[14:07] travlr	projects/
[14:08] lantins	yeah have been for me
[14:08] travlr	good
[14:08] lantins	what i did was this:
[14:08] lantins	http://cdn.shopify.com/s/files/1/0038/9582/files/RBBB_Instructions_06.pdf?1260749296
[14:08] lantins	that's a real nice document about making an Arduino clone
[14:09] lantins	I looked at the components needed to build a handful, then prospected into some other 'usual' sized components
[14:10] travlr	there's a lot of great info out on the web
[14:10] travlr	that proj looks great
[14:10] travlr	i wish i had the time
[14:10] travlr	i love teaching myself shit
[14:10] lantins	they use a Resonator in that design, I opted for a crystal
[14:10] lantins	yeah
[14:10] lantins	i know the feeling
[14:11] travlr	yeah crystal.. mmmm gooooood.
[14:11] travlr	lol
[14:11] lantins	i have to say, i sunk a good week or so getting going with all this stuff, and I know I still know naff all about it
[14:11] travlr	i've studied a shit load
[14:12] travlr	i have four tv's opened up in my 'project' room
[14:12] lantins	yeah crystal's smalled cooler then Resonators ;-) (crystal were the better option from what I descovered anyhow)
[14:12] travlr	need caps
[14:12] lantins	lol
[14:12] travlr	radio shack sucks for parts
[14:12] travlr	i haven't ordered them yet
[14:12] lantins	my problem is when I keep blowing shit up
[14:12] travlr	_lol_
[14:12] lantins	"ah fuck that cost Â£4 and I've run out of them!!!!"
[14:13] travlr	lol
[14:13] travlr	damn, i want to play
[14:13] lantins	i think the smell of burning electronics is part of the learning process
[14:13] travlr	yup!
[14:13] lantins	hell man i managed to keep that stuff at bay for ages and now you've got me all thinking about it!
[14:13] travlr	i repeat.. i want to play
[14:13] travlr	no time
[14:13] lantins	I got real work to be doing lol
[14:14] travlr	speaking of which
[14:14] travlr	gotta go
[14:14] lantins	any free time has been spent with iOS/iPad development
[14:14] lantins	take it easy travlr
[14:14] lantins	catch you later =)
[14:14] travlr	see ya
[15:06] Guthur	travlr, Cheers for the heads up on the logs
[15:09] travlr	sure
[16:06] bfrog	is there a sane way to get queues to work with an existing "reactor" like libevent or whatever?
[16:07] bfrog	I mean, whats the point of sockets if you can't do event driven programming :-P
[16:08] travlr	bfrog, existing reactor... sure... libevent, i think i remember that that's been done. did you check the mail list and irc archives?
[16:09] bfrog	no, I looked on the website
[16:09] travlr	there's a nice mashup with qt signal and slots if you do qt
[16:10] travlr	the mail list is easily found on the website, here's the irc log: http://travlr.github.com/zmqirclog/
[16:14] mikko	bfrog: https://github.com/zeromq/zeromq2/blob/master/doc/zmq_getsockopt.txt#L255
[16:28] bfrog	I see it, thanks
[16:29] mikko	bfrog: and zmq provides it own polling mechnism as well
[16:29] mikko	so you are not really enforced to use external event loop
[16:30] bfrog	I get that
[16:31] bfrog	but thats again, really limiting
[16:31] bfrog	well not again, but yeah, thats faily limiting if I can't use my own
[16:42] bfrog	it doesn't look like I can just provide callbacks for when messages are available, call this function sort of thing?
[17:25] Guthur	bfrog you could easily build it ontop of polling
[17:25] Guthur	clrzmq2 uses event delegates for polling
[17:27] bfrog	I don't want to poll, I like my callbacks and selects() :-(
[17:31] mikko	bfrog: the zmq_poll is an abstraction over different polling methods such as epoll and select
[17:31] mikko	adding callbacks to that is a couple of lines of code
[17:32] mikko	but if you want to use an external event loop just use ZMQ_FD sockopt
[17:32] bfrog	but I might not have the whole message in the queue then?
[17:32] mikko	not necessarily
[17:33] bfrog	right, so the recv may still block
[17:33] mikko	you can use ZMQ_EVENTS to check that
[17:33] mikko	https://github.com/zeromq/zeromq2/blob/master/doc/zmq_getsockopt.txt#L272
[17:42] bfrog	what if I get a partial message, then I'm constantly looping on ZMQ_EVENTS, how would I break that?
[17:42] bfrog	I can't touch the fd itself, can I tell zeromq to just give up somehow?
[17:46] Guthur	you can just stop calling poll, and that's it surely
[17:46] Guthur	The example code just immediately timeouts, you might want something slightly different
[18:14] cremes	bfrog: you are guaranteed to have a whole message; there is no such thing in 0mq as a partial message
[18:14] cremes	therefore, no worries about it blocking
[18:14] cremes	if you are still concerned, pass the NO_BLOCK flag to your send or receive and go on your merry way
[18:17] cremes	lantins: you may also be interested in looking at zmqmachine which is essentially eventmachine for 0mq sockets; find it on github
[18:22] mikko	cremes: if you poll on ZMQ_FD you could have false positives
[18:22] mikko	you need to check for ZMQ_EVENTS for the socket as well if you want to make sure that you dont block
[18:23] cremes	mikko: right; so what's the problem? it's a pretty clear 2-step process
[18:23] cremes	alternately, try to send/recv with NO_BLOCK and check for EAGAIN
[18:23] cremes	either way, there is no concern about blocking
[18:25] mikko	no problem there
[18:25] mikko	just wanted to indicate that a 'positive' on polling on ZMQ_FD doesn't mean that you can recv()
[18:25] cremes	ah, i see
[18:26] cremes	i agree with that
[18:26] cremes	hopefully bfrog reads through this at some point and this answers his/her question
[18:26] bfrog	yeah it does
[18:26] cremes	great
[18:27] cremes	bfrog: fyi, the FD & EVENTS opts were added specifically to allow for 0mq socket integration with other event loops
[18:27] cremes	if you find a case where they are inadequate, please speak up
[19:26] rgl	hi