IRC Log


Tuesday December 14, 2010

[Time] NameMessage
[05:07] gandhijee hey, installed zmq on my system, and for my cross compiler, now my c++ programs won't compile unless i link in zmq, any ideas?
[05:21] EricL I am debating using 0mq as a possible 'logserver'. Basically just listen for messages and write them to a file (but these messages are JSON and will be coming at a very high rate of speed). Is there a good use-case?
[09:08] mikko good morning
[09:09] Steve-o good afternoon
[09:09] Steve-o :P
[09:12] Steve-o quiet morning, nothing but net splits
[09:13] mikko indeed
[09:13] mikko europe is just waking up
[09:14] Steve-o I released OpenPGM 5.0.93 and 5.1.99, hopefully the latter is more friendly towards Solaris/x86
[09:15] Steve-o builds in ICC and Sun ONE strict mode, along with OSX and Cygwin too
[09:16] Steve-o I keep on messing up byte order macros on Solaris, the builds definitely work on Sparc
[09:18] mikko nice!
[09:19] Steve-o OSX is tedious as the libc is really old
[09:19] mikko i'll fix zeromq --with-pgm to build properly on solaris x86 at some point
[09:19] mikko haven't got access to sparc yet
[09:20] Steve-o remember there are 3 compilers on sparc to test
[09:20] Steve-o Sun ONE, Sun GCC, and GCC
[09:20] Steve-o all each with their own quirks
[09:20] mikko i can test all those on x86
[09:21] mikko hmm, currently the win7 builds using visual studio 2008
[09:21] mikko might upgrade to 2010 at some point
[09:22] Steve-o PGM should work in 2008, it is just that I found some really annoying bugs in the CRT socket libraries
[09:22] mikko that's not surprising
[09:23] Steve-o haven't tried mingw32 on Windows, only cross compiles
[09:23] Steve-o Scons on Cygwin is pretty terrible, although I think it's Python at fault
[09:23] mikko i could add mingw32 on windows to daily builds
[09:23] mikko and possibly a BSD box
[09:24] Steve-o FreeBSD is nice to catch portability bugs
[09:24] Steve-o but the IP header quirk is annoying
[09:24] mikko should i go for OpenBSD might be even a bit more exotic
[09:25] mikko FreeBSD is a bit more user-friendly though
[09:25] Steve-o good luck, I've only used OpenBSD on Sparc
[09:25] mikko OpenBSD doesn't use GCC iirc
[09:25] Steve-o can be annoying to install, especially since getting the install is inconvenient
[09:25] Steve-o GCC 2.95 I believe
[09:25] mikko Portable C Compiler
[09:26] mikko ermm
[09:27] Steve-o maybe the change was recent
[09:27] mikko added to openbsd source tree 2007
[09:27] mikko was able to compile x86 kernel image 2009
[09:27] mikko according to wikipedia
[09:28] mikko but probably not the 'default' compiler yet
[09:28] Steve-o ok
[09:28] Steve-o tea time, catch you laters
[09:29] Steve-o I thnk Theo wants compiler that isn't GCC 3
[09:29] Steve-o hence the search for others
[09:29] mikko see you later
[09:42] sustrik morning
[09:42] mikko sustrik: did you see the patch for swap?
[09:43] sustrik juse seen it
[09:43] sustrik i have to do some administrative stuff
[09:44] sustrik then i'll have a look
[09:44] mikko col
[09:44] mikko +o
[11:05] mikko sustrik: does HWM option only work if set before bind/connect? If yes should the setsockopt fail?
[11:09] sustrik yes
[11:09] sustrik why should it fail?
[11:10] mikko if it has no effect
[11:11] sustrik it has effect on all subsequent connects and binds
[11:56] mikko hmm
[11:57] mikko how does that work?
[11:57] mikko i got a PUSH socket
[11:57] mikko connect, set hwm, connect
[11:57] mikko so that allows to do weighted load-balancing
[11:59] sustrik then the two connections have different HWMs
[11:59] sustrik the first is unlimited
[11:59] sustrik the second one has the hwm you've set
[12:00] mikko yeah, so in a scenario where second has HWM of 5 and it fills up the messages would go to first one
[12:00] mikko which makes sense, if you for example you have slower consumer on second socket
[12:01] mikko second connection i mean
[13:53] EricL I am debating using 0mq as a possible 'logserver'. Basically just listen for messages and write them to a file (but these messages are JSON and will be coming at a very high rate of speed). Is this a good use-case?
[14:02] spht EricL: Do you have control over what is sending the log entries? If yes, perfect match (including possibilites for distributed logging), if no...uncertain
[14:03] EricL spht: I have 100% control over what's sending the logs. I am writing what is sending the logs.
[14:03] spht EricL: Then it's perfect
[14:03] spht :)
[14:04] EricL spht: That's great news. Problem is that I need to get it working quickly. The client (sending piece) is in Ruby so that should be easy. But I feel like the receiver should be in C for as much speed as possible (unless the receiver language doesn't matter).
[14:04] spht EricL: Prototype it in Ruby (I mean, a prototype will be just a few lines) and benchmark it. won't cost you much time
[14:05] spht EricL: Disclaimer: I'm a 0mq-newbie ;)
[14:05] EricL spht: Fair enough. How long did it take you to get something up and running?
[14:06] spht EricL: I read the guide back to back, from there it's a matter of minutes in python (which is my weapon of choice), should be the same in ruby. Read the guide though
[14:07] EricL I read the beginner part already, not the other 2 parts.
[14:08] spht I would bet that disk io will be the limiting factor not the messaging overhead...
[14:08] spht depending on what network you have of course :)
[14:08] EricL spht: I am thinking the same thing.
[14:08] EricL The network is a local GigE network.
[14:12] guido_g so split the work between multiple writers each with its own disk
[14:13] EricL guido_g: Sounds like that's the advanced stuff I need to work out. Right now I am fine with a single machine. The heavy load will be a month or two down the road.
[14:13] guido_g PUSH/PULL sockets would do the trick out of the box
[14:14] guido_g EricL: you were worried about the speed
[14:17] EricL guido_g: I am. That's why I am asking the questions.
[14:18] brandyn Hey I have a pub/sub setup where the sub is too busy to read all of the pub's messages. What is happening now is that sub seems to be buffering everything. How do I fix that? I am using the python wrapper (I tried release and now dev from git)
[14:19] brandyn I have already set the HWM to 1, that didn't help. And it is the client that seems to be buffering as it's memory usage balloons out of control
[14:19] guido_g if the sub is to slow, make it faster
[14:19] brandyn guido_g, not an option I'm shooting video frames at it and some devices can't handle all ofit
[14:19] guido_g if you set HWM you'll lose messages
[14:19] brandyn guido_g, yes that is what I want
[14:19] guido_g brandyn: too bad
[14:19] brandyn but I am not losing them enough clearly, the client is still buffering
[14:20] brandyn I guess I'm in the wrong place
[14:30] spht Uhm, 2 minute attention span is pretty short for asking questions on IRC :)
[14:34] spht EricL: You could also make the writer a pub socket and have different loggers listening to different messages, one way to partition the writes
[14:35] spht push socket on the other hand will give you round-robin style load balancing between different clients if you don't want to partition by message type
[14:37] guido_g don't forget that for now, the topic is filtered at the sub side
[14:37] guido_g so the message will be received and at least partially processed
[14:38] spht guido_g: true!
[14:38] guido_g but i would go this route anyway, i'm a pub/sub fan :)
[14:40] EricL So the clients would be the pubs and the server would sub to multiiple clients?
[14:40] EricL I know that sounds backwards, but do I have the concept right?
[14:41] spht EricL: no the clients should be sub sockets of course :)
[14:41] guido_g don't think in terms of sockets or connections
[14:41] guido_g start to think in messages instead
[14:41] EricL spht: The clients are the ones publishing the connections.
[14:41] guido_g makes life much easier
[14:42] guido_g publishing connections?
[14:43] spht why?
[14:43] guido_g why what?
[14:44] spht guido_g: why the clients should bind and not the server
[14:44] guido_g spht: huh? where did i say that?
[14:44] spht guido_g: you didn't, EricL did :)
[14:44] guido_g ahhh
[14:45] spht or, I didn't get enough coffee and is confused as usual ;)
[14:45] EricL spht: Maybe I should re-read the beginner stuff. I guess I don't understand it.
[14:46] guido_g EricL: don't overreate the socket/connection/bind stuff
[14:46] EricL I am thinking that clients (the ones pushing the log messages) are the publishers and the server can subscribe to each client.
[14:46] guido_g as i said, to think in terms of messages is much more important
[14:46] guido_g client and server are way to overloaded
[14:47] guido_g i suggest using words like sender and receiver
[14:47] EricL guido_g: Ok. Let me go back and do some reading.
[14:47] EricL I have to jump into a few meetings. I'll be back later today if I can't figure this out.
[14:48] spht same here, bbl
[14:48] guido_g ahhh... the joys of working in a professional environment... ]:->
[16:38] mikko good evening
[17:07] sustrik mikko: i haven't had chance to review the patch :(
[17:08] sustrik i have to leave now
[17:08] sustrik will do tomorrow
[17:08] sustrik sorry
[17:08] mikko no worries
[17:08] gandhijee hey, i installed zmq on my system, now my C++ programs won't compile unless i use -lzmq
[17:09] sustrik you mean unrelated C++ programs?
[17:09] mikko gandhijee: what system is that?
[17:09] gandhijee sustrik: right, even if i don't have the zmq header in, i still need -lzmq
[17:10] gandhijee 2 systems, my gentoo laptop, plus my ARM toolchain thats installed on it
[17:10] mikko what is the error message you get?
[17:10] gandhijee one second
[17:12] gandhijee hmm, actually i guess it might be the way i am invoking the compile
[17:12] gandhijee i was calling gcc instead of g++ to compile to code
[17:13] mikko have you got CLFAGS, LDFLAGS or similar set?
[17:20] gandhijee no
[19:41] brandyn I have a pub/sub setup where the sub is too busy to read all of the pub's messages. What is happening now is that sub seems to be buffering everything. How do I fix that? I am using the python wrapper (I tried release and now dev from git)
[19:44] cremes brandyn: question: is the sub unable to keep up due to processing in python? or it can't read the messages fast enough (and you are not processing them at all)?
[19:45] brandyn cremes, it can't keep up
[19:45] brandyn cremes, I would like it to drop ones that it can't process, like the HWM
[19:45] cremes brandyn: can it keep up if you drop *all* messages?
[19:45] brandyn cremes, but it seems like there isn't an option like that
[19:45] brandyn cremes, yes if I just keep recv'ing
[19:45] cremes ok
[19:46] brandyn it is from a streaming sensor
[19:46] brandyn and some devices can't handle the data
[19:46] brandyn and it is ok to get some of the packets
[19:46] cremes then you're only hope is to increase the speed of your python processor; use a faster language or throw hardware at it
[19:46] brandyn but shouldn't there be a way to leak the buffer
[19:46] brandyn like the HWM on the PUB side
[19:46] cremes no, you can't tell how many messages are queued; that information is not exposed
[19:47] brandyn sure, but it knows
[19:47] brandyn so if I said "don't queue more than X messages"
[19:47] cremes the best you could do is process 1 out of X packets and drop the rest; that logic should be easy
[19:47] brandyn then it should work, just like HWM does now
[19:47] brandyn I see, so I can just close and reopen the socket?
[19:48] brandyn that seems like a hack, if the SUB listened to the HWM, then it could just drop packets and keep the newest ones
[19:48] cremes brandyn: no, you should not close & reopen the socket
[19:49] brandyn how do you drop the packets then?
[19:49] brandyn just recv really fast? or something
[19:49] cremes i meant that your python logic should drop x-1 of x packets; e.g. don't process them, just read and toss
[19:49] cremes yes
[19:49] brandyn ah gotcha, ok yeah I guess that can work
[19:49] brandyn so what I can probably do is set to to nonblocking
[19:49] cremes if (counter MOD 10 == 0) process else drop end
[19:49] brandyn and then keep recv'ing until I get the exception
[19:49] brandyn and just use the last one
[19:50] cremes non-blocking reads/writes aren't going to help you here
[19:50] cremes a regular blocking read should be fine
[19:50] brandyn what I mean is that I just want my app to get access to the newest message
[19:50] brandyn so I can just read all of them
[19:50] brandyn which will flush the buffer
[19:50] brandyn and then use the last one
[19:51] brandyn the nonblocking would tell me when the buffer is done
[19:51] cremes sure, that's fine
[19:51] brandyn cremes, you rock man, thank you
[19:51] cremes pay it forward... :)
[19:52] cremes brandyn: are you sure setting HWM on the publisher didn't help?
[19:52] cremes when a pub socket hits HWM for a subscriber, it should drop messages
[19:52] cremes try HWM = 1 if you haven't
[19:54] brandyn yeah I did
[19:54] brandyn but the client can read them
[19:54] brandyn that's the problem
[19:54] brandyn it appears there is no similar HWM for the sub
[19:54] brandyn but that's ok
[19:55] brandyn this is good enough, basically I offload ZMQ's SUB buffer to myself, and then apply my own restrictions
[19:55] cremes i'm not sure i follow; if a sub socket has more than HWM messages in its queue then the pub socket should detect that and drop packets
[19:55] cremes but whatever...
[19:55] cremes you have a solution
[19:56] brandyn oh I see, yeah it doesn't appear to do that
[19:56] brandyn I think the HWM applies only to the queue on the PUB size
[19:57] brandyn for each SUB
[19:57] brandyn like if I don't call recv, then my buffer blows up
[19:57] brandyn so it tells me that in the background is it buffering lots of goodies for me
[19:57] brandyn and I just didn't ask for them
[19:59] cremes hmmm, that's surprising; you might want to ask on the ML for clarification with your use-case
[20:00] brandyn yeah for sure, this is just a rushed project so I need a fix. It's the first time I've used ZMQ
[20:14] brandyn cremes, ok so strange problem. If I read the socket opt for HWM, it is zero, if I set it, I get a totally different number
[20:15] cremes 0 is the default
[20:15] cremes what value are you trying to set it to? and what do you get back when you read that option again?
[20:15] cremes btw, 0 means no limit
[20:15] brandyn yeah I know that
[20:16] brandyn so I set it to 1, and got thishttp://pastebin.com/xh9dCw6C
[20:16] brandyn crazy right?
[20:17] cremes HWM takes an unsigned 64-bit integer; maybe the python bindings aren't handling that right?
[20:18] brandyn yeah I see that
[20:18] brandyn I'll fix it an put in a patch
[20:19] brandyn yeah simple fix
[20:22] brandyn so I fixed that, it now returns the correct HWM, but same semantics
[20:25] cremes cool
[21:10] mikko evening ladies and gents
[21:12] cremes evening
[21:18] brandyn anyone here a python wrapper contributor?
[21:18] mikko brandyn: MinRK and bgranger pop in occasionally
[21:19] brandyn thanks