[Time] Name | Message |
[05:07] gandhijee
|
hey, installed zmq on my system, and for my cross compiler, now my c++ programs won't compile unless i link in zmq, any ideas?
|
[05:21] EricL
|
I am debating using 0mq as a possible 'logserver'. Basically just listen for messages and write them to a file (but these messages are JSON and will be coming at a very high rate of speed). Is there a good use-case?
|
[09:08] mikko
|
good morning
|
[09:09] Steve-o
|
good afternoon
|
[09:09] Steve-o
|
:P
|
[09:12] Steve-o
|
quiet morning, nothing but net splits
|
[09:13] mikko
|
indeed
|
[09:13] mikko
|
europe is just waking up
|
[09:14] Steve-o
|
I released OpenPGM 5.0.93 and 5.1.99, hopefully the latter is more friendly towards Solaris/x86
|
[09:15] Steve-o
|
builds in ICC and Sun ONE strict mode, along with OSX and Cygwin too
|
[09:16] Steve-o
|
I keep on messing up byte order macros on Solaris, the builds definitely work on Sparc
|
[09:18] mikko
|
nice!
|
[09:19] Steve-o
|
OSX is tedious as the libc is really old
|
[09:19] mikko
|
i'll fix zeromq --with-pgm to build properly on solaris x86 at some point
|
[09:19] mikko
|
haven't got access to sparc yet
|
[09:20] Steve-o
|
remember there are 3 compilers on sparc to test
|
[09:20] Steve-o
|
Sun ONE, Sun GCC, and GCC
|
[09:20] Steve-o
|
all each with their own quirks
|
[09:20] mikko
|
i can test all those on x86
|
[09:21] mikko
|
hmm, currently the win7 builds using visual studio 2008
|
[09:21] mikko
|
might upgrade to 2010 at some point
|
[09:22] Steve-o
|
PGM should work in 2008, it is just that I found some really annoying bugs in the CRT socket libraries
|
[09:22] mikko
|
that's not surprising
|
[09:23] Steve-o
|
haven't tried mingw32 on Windows, only cross compiles
|
[09:23] Steve-o
|
Scons on Cygwin is pretty terrible, although I think it's Python at fault
|
[09:23] mikko
|
i could add mingw32 on windows to daily builds
|
[09:23] mikko
|
and possibly a BSD box
|
[09:24] Steve-o
|
FreeBSD is nice to catch portability bugs
|
[09:24] Steve-o
|
but the IP header quirk is annoying
|
[09:24] mikko
|
should i go for OpenBSD might be even a bit more exotic
|
[09:25] mikko
|
FreeBSD is a bit more user-friendly though
|
[09:25] Steve-o
|
good luck, I've only used OpenBSD on Sparc
|
[09:25] mikko
|
OpenBSD doesn't use GCC iirc
|
[09:25] Steve-o
|
can be annoying to install, especially since getting the install is inconvenient
|
[09:25] Steve-o
|
GCC 2.95 I believe
|
[09:25] mikko
|
Portable C Compiler
|
[09:26] mikko
|
ermm
|
[09:27] Steve-o
|
maybe the change was recent
|
[09:27] mikko
|
added to openbsd source tree 2007
|
[09:27] mikko
|
was able to compile x86 kernel image 2009
|
[09:27] mikko
|
according to wikipedia
|
[09:28] mikko
|
but probably not the 'default' compiler yet
|
[09:28] Steve-o
|
ok
|
[09:28] Steve-o
|
tea time, catch you laters
|
[09:29] Steve-o
|
I thnk Theo wants compiler that isn't GCC 3
|
[09:29] Steve-o
|
hence the search for others
|
[09:29] mikko
|
see you later
|
[09:42] sustrik
|
morning
|
[09:42] mikko
|
sustrik: did you see the patch for swap?
|
[09:43] sustrik
|
juse seen it
|
[09:43] sustrik
|
i have to do some administrative stuff
|
[09:44] sustrik
|
then i'll have a look
|
[09:44] mikko
|
col
|
[09:44] mikko
|
+o
|
[11:05] mikko
|
sustrik: does HWM option only work if set before bind/connect? If yes should the setsockopt fail?
|
[11:09] sustrik
|
yes
|
[11:09] sustrik
|
why should it fail?
|
[11:10] mikko
|
if it has no effect
|
[11:11] sustrik
|
it has effect on all subsequent connects and binds
|
[11:56] mikko
|
hmm
|
[11:57] mikko
|
how does that work?
|
[11:57] mikko
|
i got a PUSH socket
|
[11:57] mikko
|
connect, set hwm, connect
|
[11:57] mikko
|
so that allows to do weighted load-balancing
|
[11:59] sustrik
|
then the two connections have different HWMs
|
[11:59] sustrik
|
the first is unlimited
|
[11:59] sustrik
|
the second one has the hwm you've set
|
[12:00] mikko
|
yeah, so in a scenario where second has HWM of 5 and it fills up the messages would go to first one
|
[12:00] mikko
|
which makes sense, if you for example you have slower consumer on second socket
|
[12:01] mikko
|
second connection i mean
|
[13:53] EricL
|
I am debating using 0mq as a possible 'logserver'. Basically just listen for messages and write them to a file (but these messages are JSON and will be coming at a very high rate of speed). Is this a good use-case?
|
[14:02] spht
|
EricL: Do you have control over what is sending the log entries? If yes, perfect match (including possibilites for distributed logging), if no...uncertain
|
[14:03] EricL
|
spht: I have 100% control over what's sending the logs. I am writing what is sending the logs.
|
[14:03] spht
|
EricL: Then it's perfect
|
[14:03] spht
|
:)
|
[14:04] EricL
|
spht: That's great news. Problem is that I need to get it working quickly. The client (sending piece) is in Ruby so that should be easy. But I feel like the receiver should be in C for as much speed as possible (unless the receiver language doesn't matter).
|
[14:04] spht
|
EricL: Prototype it in Ruby (I mean, a prototype will be just a few lines) and benchmark it. won't cost you much time
|
[14:05] spht
|
EricL: Disclaimer: I'm a 0mq-newbie ;)
|
[14:05] EricL
|
spht: Fair enough. How long did it take you to get something up and running?
|
[14:06] spht
|
EricL: I read the guide back to back, from there it's a matter of minutes in python (which is my weapon of choice), should be the same in ruby. Read the guide though
|
[14:07] EricL
|
I read the beginner part already, not the other 2 parts.
|
[14:08] spht
|
I would bet that disk io will be the limiting factor not the messaging overhead...
|
[14:08] spht
|
depending on what network you have of course :)
|
[14:08] EricL
|
spht: I am thinking the same thing.
|
[14:08] EricL
|
The network is a local GigE network.
|
[14:12] guido_g
|
so split the work between multiple writers each with its own disk
|
[14:13] EricL
|
guido_g: Sounds like that's the advanced stuff I need to work out. Right now I am fine with a single machine. The heavy load will be a month or two down the road.
|
[14:13] guido_g
|
PUSH/PULL sockets would do the trick out of the box
|
[14:14] guido_g
|
EricL: you were worried about the speed
|
[14:17] EricL
|
guido_g: I am. That's why I am asking the questions.
|
[14:18] brandyn
|
Hey I have a pub/sub setup where the sub is too busy to read all of the pub's messages. What is happening now is that sub seems to be buffering everything. How do I fix that? I am using the python wrapper (I tried release and now dev from git)
|
[14:19] brandyn
|
I have already set the HWM to 1, that didn't help. And it is the client that seems to be buffering as it's memory usage balloons out of control
|
[14:19] guido_g
|
if the sub is to slow, make it faster
|
[14:19] brandyn
|
guido_g, not an option I'm shooting video frames at it and some devices can't handle all ofit
|
[14:19] guido_g
|
if you set HWM you'll lose messages
|
[14:19] brandyn
|
guido_g, yes that is what I want
|
[14:19] guido_g
|
brandyn: too bad
|
[14:19] brandyn
|
but I am not losing them enough clearly, the client is still buffering
|
[14:20] brandyn
|
I guess I'm in the wrong place
|
[14:30] spht
|
Uhm, 2 minute attention span is pretty short for asking questions on IRC :)
|
[14:34] spht
|
EricL: You could also make the writer a pub socket and have different loggers listening to different messages, one way to partition the writes
|
[14:35] spht
|
push socket on the other hand will give you round-robin style load balancing between different clients if you don't want to partition by message type
|
[14:37] guido_g
|
don't forget that for now, the topic is filtered at the sub side
|
[14:37] guido_g
|
so the message will be received and at least partially processed
|
[14:38] spht
|
guido_g: true!
|
[14:38] guido_g
|
but i would go this route anyway, i'm a pub/sub fan :)
|
[14:40] EricL
|
So the clients would be the pubs and the server would sub to multiiple clients?
|
[14:40] EricL
|
I know that sounds backwards, but do I have the concept right?
|
[14:41] spht
|
EricL: no the clients should be sub sockets of course :)
|
[14:41] guido_g
|
don't think in terms of sockets or connections
|
[14:41] guido_g
|
start to think in messages instead
|
[14:41] EricL
|
spht: The clients are the ones publishing the connections.
|
[14:41] guido_g
|
makes life much easier
|
[14:42] guido_g
|
publishing connections?
|
[14:43] spht
|
why?
|
[14:43] guido_g
|
why what?
|
[14:44] spht
|
guido_g: why the clients should bind and not the server
|
[14:44] guido_g
|
spht: huh? where did i say that?
|
[14:44] spht
|
guido_g: you didn't, EricL did :)
|
[14:44] guido_g
|
ahhh
|
[14:45] spht
|
or, I didn't get enough coffee and is confused as usual ;)
|
[14:45] EricL
|
spht: Maybe I should re-read the beginner stuff. I guess I don't understand it.
|
[14:46] guido_g
|
EricL: don't overreate the socket/connection/bind stuff
|
[14:46] EricL
|
I am thinking that clients (the ones pushing the log messages) are the publishers and the server can subscribe to each client.
|
[14:46] guido_g
|
as i said, to think in terms of messages is much more important
|
[14:46] guido_g
|
client and server are way to overloaded
|
[14:47] guido_g
|
i suggest using words like sender and receiver
|
[14:47] EricL
|
guido_g: Ok. Let me go back and do some reading.
|
[14:47] EricL
|
I have to jump into a few meetings. I'll be back later today if I can't figure this out.
|
[14:48] spht
|
same here, bbl
|
[14:48] guido_g
|
ahhh... the joys of working in a professional environment... ]:->
|
[16:38] mikko
|
good evening
|
[17:07] sustrik
|
mikko: i haven't had chance to review the patch :(
|
[17:08] sustrik
|
i have to leave now
|
[17:08] sustrik
|
will do tomorrow
|
[17:08] sustrik
|
sorry
|
[17:08] mikko
|
no worries
|
[17:08] gandhijee
|
hey, i installed zmq on my system, now my C++ programs won't compile unless i use -lzmq
|
[17:09] sustrik
|
you mean unrelated C++ programs?
|
[17:09] mikko
|
gandhijee: what system is that?
|
[17:09] gandhijee
|
sustrik: right, even if i don't have the zmq header in, i still need -lzmq
|
[17:10] gandhijee
|
2 systems, my gentoo laptop, plus my ARM toolchain thats installed on it
|
[17:10] mikko
|
what is the error message you get?
|
[17:10] gandhijee
|
one second
|
[17:12] gandhijee
|
hmm, actually i guess it might be the way i am invoking the compile
|
[17:12] gandhijee
|
i was calling gcc instead of g++ to compile to code
|
[17:13] mikko
|
have you got CLFAGS, LDFLAGS or similar set?
|
[17:20] gandhijee
|
no
|
[19:41] brandyn
|
I have a pub/sub setup where the sub is too busy to read all of the pub's messages. What is happening now is that sub seems to be buffering everything. How do I fix that? I am using the python wrapper (I tried release and now dev from git)
|
[19:44] cremes
|
brandyn: question: is the sub unable to keep up due to processing in python? or it can't read the messages fast enough (and you are not processing them at all)?
|
[19:45] brandyn
|
cremes, it can't keep up
|
[19:45] brandyn
|
cremes, I would like it to drop ones that it can't process, like the HWM
|
[19:45] cremes
|
brandyn: can it keep up if you drop *all* messages?
|
[19:45] brandyn
|
cremes, but it seems like there isn't an option like that
|
[19:45] brandyn
|
cremes, yes if I just keep recv'ing
|
[19:45] cremes
|
ok
|
[19:46] brandyn
|
it is from a streaming sensor
|
[19:46] brandyn
|
and some devices can't handle the data
|
[19:46] brandyn
|
and it is ok to get some of the packets
|
[19:46] cremes
|
then you're only hope is to increase the speed of your python processor; use a faster language or throw hardware at it
|
[19:46] brandyn
|
but shouldn't there be a way to leak the buffer
|
[19:46] brandyn
|
like the HWM on the PUB side
|
[19:46] cremes
|
no, you can't tell how many messages are queued; that information is not exposed
|
[19:47] brandyn
|
sure, but it knows
|
[19:47] brandyn
|
so if I said "don't queue more than X messages"
|
[19:47] cremes
|
the best you could do is process 1 out of X packets and drop the rest; that logic should be easy
|
[19:47] brandyn
|
then it should work, just like HWM does now
|
[19:47] brandyn
|
I see, so I can just close and reopen the socket?
|
[19:48] brandyn
|
that seems like a hack, if the SUB listened to the HWM, then it could just drop packets and keep the newest ones
|
[19:48] cremes
|
brandyn: no, you should not close & reopen the socket
|
[19:49] brandyn
|
how do you drop the packets then?
|
[19:49] brandyn
|
just recv really fast? or something
|
[19:49] cremes
|
i meant that your python logic should drop x-1 of x packets; e.g. don't process them, just read and toss
|
[19:49] cremes
|
yes
|
[19:49] brandyn
|
ah gotcha, ok yeah I guess that can work
|
[19:49] brandyn
|
so what I can probably do is set to to nonblocking
|
[19:49] cremes
|
if (counter MOD 10 == 0) process else drop end
|
[19:49] brandyn
|
and then keep recv'ing until I get the exception
|
[19:49] brandyn
|
and just use the last one
|
[19:50] cremes
|
non-blocking reads/writes aren't going to help you here
|
[19:50] cremes
|
a regular blocking read should be fine
|
[19:50] brandyn
|
what I mean is that I just want my app to get access to the newest message
|
[19:50] brandyn
|
so I can just read all of them
|
[19:50] brandyn
|
which will flush the buffer
|
[19:50] brandyn
|
and then use the last one
|
[19:51] brandyn
|
the nonblocking would tell me when the buffer is done
|
[19:51] cremes
|
sure, that's fine
|
[19:51] brandyn
|
cremes, you rock man, thank you
|
[19:51] cremes
|
pay it forward... :)
|
[19:52] cremes
|
brandyn: are you sure setting HWM on the publisher didn't help?
|
[19:52] cremes
|
when a pub socket hits HWM for a subscriber, it should drop messages
|
[19:52] cremes
|
try HWM = 1 if you haven't
|
[19:54] brandyn
|
yeah I did
|
[19:54] brandyn
|
but the client can read them
|
[19:54] brandyn
|
that's the problem
|
[19:54] brandyn
|
it appears there is no similar HWM for the sub
|
[19:54] brandyn
|
but that's ok
|
[19:55] brandyn
|
this is good enough, basically I offload ZMQ's SUB buffer to myself, and then apply my own restrictions
|
[19:55] cremes
|
i'm not sure i follow; if a sub socket has more than HWM messages in its queue then the pub socket should detect that and drop packets
|
[19:55] cremes
|
but whatever...
|
[19:55] cremes
|
you have a solution
|
[19:56] brandyn
|
oh I see, yeah it doesn't appear to do that
|
[19:56] brandyn
|
I think the HWM applies only to the queue on the PUB size
|
[19:57] brandyn
|
for each SUB
|
[19:57] brandyn
|
like if I don't call recv, then my buffer blows up
|
[19:57] brandyn
|
so it tells me that in the background is it buffering lots of goodies for me
|
[19:57] brandyn
|
and I just didn't ask for them
|
[19:59] cremes
|
hmmm, that's surprising; you might want to ask on the ML for clarification with your use-case
|
[20:00] brandyn
|
yeah for sure, this is just a rushed project so I need a fix. It's the first time I've used ZMQ
|
[20:14] brandyn
|
cremes, ok so strange problem. If I read the socket opt for HWM, it is zero, if I set it, I get a totally different number
|
[20:15] cremes
|
0 is the default
|
[20:15] cremes
|
what value are you trying to set it to? and what do you get back when you read that option again?
|
[20:15] cremes
|
btw, 0 means no limit
|
[20:15] brandyn
|
yeah I know that
|
[20:16] brandyn
|
so I set it to 1, and got thishttp://pastebin.com/xh9dCw6C
|
[20:16] brandyn
|
crazy right?
|
[20:17] cremes
|
HWM takes an unsigned 64-bit integer; maybe the python bindings aren't handling that right?
|
[20:18] brandyn
|
yeah I see that
|
[20:18] brandyn
|
I'll fix it an put in a patch
|
[20:19] brandyn
|
yeah simple fix
|
[20:22] brandyn
|
so I fixed that, it now returns the correct HWM, but same semantics
|
[20:25] cremes
|
cool
|
[21:10] mikko
|
evening ladies and gents
|
[21:12] cremes
|
evening
|
[21:18] brandyn
|
anyone here a python wrapper contributor?
|
[21:18] mikko
|
brandyn: MinRK and bgranger pop in occasionally
|
[21:19] brandyn
|
thanks
|