[Time] Name | Message |
[03:40] michelp
|
has anyone yet in the community explored distributed shared memory stores in python over 0mq?
|
[03:41] michelp
|
thinking in terms of doing transactional changes to objects that get propagated to N nodes
|
[03:43] michelp
|
it's something i've always taken an interest in, but the problem of distributing changes to threads, processes, and boxes uniformly has stumped me
|
[08:46] guido_g
|
good morning
|
[09:20] sustrik
|
morning
|
[09:28] Evet
|
morning
|
[09:29] gambi
|
morning :)
|
[09:44] pieter_hintjens
|
g'morning
|
[10:31] private_meta
|
sustrik: mind i talk to you about it outside the mailing lisT?
|
[10:31] private_meta
|
*list
|
[10:31] private_meta
|
sustrik: the zmq::version i mean
|
[10:39] sustrik
|
sure
|
[10:39] sustrik
|
what's up?
|
[10:41] private_meta
|
about the pointer vs reference thing
|
[10:42] private_meta
|
imho the only valid reason to use that would be to be coherent with the other function prototypes
|
[10:44] sustrik
|
right
|
[10:44] sustrik
|
it's a matter of style
|
[10:44] private_meta
|
Bad style, according to a lot of people
|
[10:44] sustrik
|
sure
|
[10:44] sustrik
|
the point is that it's the style that C++ API uses now
|
[10:44] sustrik
|
you can fork and restyle it
|
[10:46] private_meta
|
Wouldn't really help anyway
|
[10:46] sustrik
|
the thing with the style is you have to decide on it once and then keep it
|
[10:47] sustrik
|
some may consider it ugly
|
[10:47] sustrik
|
but that's how it goes
|
[10:48] private_meta
|
hmm... it's just nowhere near defensive programming, maybe that's what bothers me, I was taught to do that over and over
|
[10:49] sustrik
|
shrug
|
[10:49] sustrik
|
you can fork it
|
[10:50] sustrik
|
breaking backward comatibility because of styling issues is not worth it
|
[10:50] private_meta
|
Yeah, but as I said, it wouldn't really help, and I would break my own compatibility
|
[10:50] private_meta
|
It's not worth it
|
[10:50] sustrik
|
ack
|
[10:53] private_meta
|
well ok, even if I'm not quite agree with the reasoning, I can't change it anyway (and no, changing it by forking isn't what I call a helpful solution)
|
[10:54] sustrik
|
would you prefer breaking everyone's applications instead?
|
[10:54] private_meta
|
That's why I said I can't change it
|
[10:54] sustrik
|
exactly
|
[10:54] sustrik
|
nothing to do here
|
[10:54] private_meta
|
"Go on, nothing to see"
|
[10:54] sustrik
|
:)
|
[10:55] private_meta
|
"Change what you can't accept, accept what you can't change" <-- accepting can be hard
|
[11:38] pieter_hintjens
|
consistency is more important than accuracy in some cases
|
[11:41] pieter_hintjens
|
private_meta: actually, it would be possible to change the C++ style for 3.0
|
[11:58] private_meta
|
pieter_hintjens: ad consistency, I understand
|
[11:59] private_meta
|
pieter_hintjens: ad 3.0, do you not plan backwards compatibility for 3.0?
|
[12:11] CIA-103
|
zeromq2: 03Martin Sustrik 07master * r7045a4a 10/ (src/named_session.cpp src/named_session.hpp):
|
[12:11] CIA-103
|
zeromq2: Dead code removed from named_session.cpp
|
[12:11] CIA-103
|
zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/fkOsT0
|
[13:19] Evet
|
sustrik: have you benchmarked zeromq on debian-kfreebsd?
|
[13:24] sustrik
|
Evet: no
|
[13:24] sustrik
|
did you?
|
[13:27] Evet
|
sustrik: downloading it now
|
[13:28] Evet
|
it seems an interesting project
|
[13:34] sustrik
|
yes
|
[13:34] sustrik
|
have a look at http://www.zeromq.org/area:results
|
[13:34] sustrik
|
if you get any numbers we can post the results on the website
|
[14:06] private_meta
|
pieter_hintjens: What might be nice (and not too much work) for the guide, imho, would be a small example output of your code. 1) for those who go through the code to make sure they understood it right without running it 2) for those who want to translate to see a rough outline what the intended result is
|
[14:32] private_meta
|
actually, overloading the C++ binding with by reference methods would be a good way to add the by reference functionality without breaking backwards compatibility
|
[14:35] pieter_hintjens
|
private_meta: back, sorry, in meetings all day...
|
[14:35] pieter_hintjens
|
actually you can make a new C++ binding at any time at all
|
[14:35] private_meta
|
np, IRC is a patient protocol
|
[14:36] pieter_hintjens
|
just as we have improvements to the other language bindings, orthogonally to 0MQ releases
|
[14:37] pieter_hintjens
|
re the guide, yes, this is a good idea, I've done it in a few places but not systematically
|
[14:38] Evet
|
do you think something like mongrel2 for xitami?
|
[14:39] pieter_hintjens
|
Evet: xitami is dead, for several reasons
|
[14:40] pieter_hintjens
|
i'm not sure what your question is... :-)
|
[14:41] Evet
|
oh, xitami was my very first webserver
|
[14:41] Evet
|
i have learnt perl on it :)
|
[14:43] private_meta
|
pieter_hintjens: sorry for being slow and only arriving at the paranoid pirate pattern now, but I have a small question there. According to Figure 59 of the guide, the Lazy Pirate Client, the Paranoid Pirate Queue and the Paranoid Pirate Worker run at the same time, am I correct?
|
[14:43] pieter_hintjens
|
yes, that's right
|
[14:44] pieter_hintjens
|
I made them as three processes, rather than three threads in one process, so you can stop and start them independently
|
[14:44] private_meta
|
So, I seem to be doing something wrong. I run ppqueue, then I run ppworker and I get an "Address already in use" error for the lpserver (which ought to run 2nd)
|
[14:45] private_meta
|
ah wait
|
[14:45] pieter_hintjens
|
you don't want to run lpserver
|
[14:45] private_meta
|
i just realized that
|
[14:45] private_meta
|
stupid me
|
[14:45] private_meta
|
I wanted lpclient and ran lpserver
|
[14:45] pieter_hintjens
|
it's not as simple as it could be...
|
[14:45] private_meta
|
>_<
|
[14:46] private_meta
|
argh
|
[14:46] private_meta
|
I'm looking through it and thinking and thinking and I don't realize I use the wrong one
|
[14:46] private_meta
|
lemme see
|
[14:47] private_meta
|
Sorry for bothering you :/
|
[14:48] sustrik
|
guys, an idea: what about factoring the C++ binding out of 0mq core in 3.0?
|
[14:48] yrashk
|
sustrik: ....and merging erlzmq2 in :D haha, kidding, sorry
|
[14:49] sustrik
|
well, i mean, all the other bindings are separate projects, why should c++ differ
|
[14:50] yrashk
|
yeah, it makes some sense
|
[14:51] yrashk
|
although I am ok with c++ binding living in there
|
[14:51] guido_g
|
good idea
|
[14:51] sustrik
|
the downside is that it ties the C++ binding to the core too much
|
[14:52] sustrik
|
so there's little space for alternative implementations
|
[14:52] sustrik
|
binding-only releases (without dragging the core along)
|
[14:52] sustrik
|
etc.
|
[14:54] yrashk
|
sustrik: I'd vote for keeping c++ binding
|
[14:54] pieter_hintjens
|
sustrik: that was the idea, forking it out, but it needs to be folded back in for a real distribution
|
[14:55] sustrik
|
sure, same as other bindings
|
[14:55] pieter_hintjens
|
exactly the same
|
[14:55] pieter_hintjens
|
makes it much easier to contribute to, improve, etc.
|
[14:55] sustrik
|
ack
|
[14:55] sustrik
|
i am +1 myself
|
[14:56] pieter_hintjens
|
I'm +1 for making it consistent with other languages
|
[14:56] sustrik
|
so far 3 +1's and a single -1
|
[14:56] pieter_hintjens
|
in any case I want to explore how to make packages that include the popular bindings
|
[14:57] sustrik
|
that's something i am wondering about for a long time
|
[14:57] sustrik
|
core + java + ruby + python + .net
|
[14:57] pieter_hintjens
|
the languages are: C++, C#, clisp, Java, Perl, PHP, Python, Ruby
|
[14:57] pieter_hintjens
|
going by the % of translations of the guide examples
|
[14:58] sustrik
|
ack
|
[14:58] mato
|
pieter_hintjens: you'll want to keep in mind that $LANGUAGE generally has it's own way of distributing extensions...
|
[14:58] pieter_hintjens
|
mato: indeed, this is the fun part
|
[14:58] mato
|
pieter_hintjens: e.g. Perl -> CPAN, Ruby -> gem, etc.
|
[14:58] pieter_hintjens
|
exactl
|
[14:58] pieter_hintjens
|
y
|
[14:58] sustrik
|
and on win32 people expect monolithic installer, i would say
|
[14:59] mato
|
then you have the linux distribution people packaging stuff on top of that separately
|
[14:59] pieter_hintjens
|
so Windows is the easiest case, in many ways
|
[15:00] pieterh
|
the process can't replace existing $LANGUAGE distribution processes but it can automate them
|
[15:01] pieterh
|
i'll spend some time studying this and make a few proposals
|
[15:02] pieterh
|
it probably will need cooperation from binding authors / communities to make work well
|
[15:02] sustrik
|
that will be useful
|
[15:02] pieterh
|
ok
|
[15:04] pieterh
|
sustrik: the poll for zmq_term is tied afaics, 4/4
|
[15:04] pieterh
|
there was one vote for infinity here on irc afair
|
[15:06] Guthur
|
+1 the extraction of C++ from core
|
[15:06] Guthur
|
+1 for extraction of C++ from the world
|
[15:06] sustrik
|
:)
|
[15:06] sustrik
|
pieterh: let's wait couple of days more
|
[15:07] pieterh
|
I do want to close the 2.1 release
|
[15:07] pieterh
|
there is not sufficient mandate afaics to revert the behavior
|
[15:07] sustrik
|
looks like
|
[15:07] sustrik
|
let's go for current solution then
|
[15:08] pieterh
|
well, this saves me a lot of work in the Guide, but I have this sinking feeling about it
|
[15:08] pieterh
|
these semantics won't change again in 3.0, right?
|
[15:08] sustrik
|
nope
|
[15:09] pieterh
|
then it's defensible, and the poll shows we did discuss the issue ad nauseam
|
[15:09] pieterh
|
if anyone still complains, they should have spoken up in time
|
[15:09] sustrik
|
ok
|
[15:09] pieterh
|
excellent
|
[15:10] private_meta
|
aw not again
|
[15:10] pieterh
|
private_meta: port in use?
|
[15:10] private_meta
|
no
|
[15:10] private_meta
|
Bad file descriptor
|
[15:10] private_meta
|
nbytes != -1 (mailbox.cpp:241)
|
[15:10] private_meta
|
don't remember right now what the error was when I had that one previously
|
[15:11] mato
|
pieterh: so what is the decision on zmq_term?
|
[15:11] mato
|
pieterh: keep the current behaviour?
|
[15:11] pieterh
|
mato: yes
|
[15:11] sustrik
|
private_meta: is it a reproducible use case?
|
[15:11] mato
|
ok, understood
|
[15:11] pieterh
|
the problem, and the reason for the discussion, is that people migrating from 2.0 _will_ be systematically hit by this
|
[15:11] private_meta
|
sustrik: it's my translation of ppworker (not yet submitted)
|
[15:11] cremes
|
private_meta: i have seen that assert when i ran out of sockets and/or ran out of file descriptors
|
[15:12] cremes
|
should be easy to reproduce
|
[15:12] sustrik
|
private_meta: iirc such a problem can happen if you close the fd you get fromZMQ_FD
|
[15:12] sustrik
|
socket option i mean
|
[15:12] cremes
|
yep, that causes it too
|
[15:12] sustrik
|
cremes: if you can reproduce it, i would fix it
|
[15:14] bpl29
|
Is there a reference page for ZMQ objects such as context and socket?
|
[15:14] pieterh
|
bpl29: yes, at http://rfc.zeromq.org
|
[15:14] private_meta
|
happens when calling zmq::poll
|
[15:14] pieterh
|
bpl29: sorry! http://api.zeromq.org...
|
[15:14] private_meta
|
wait, i think I know
|
[15:15] private_meta
|
k, got it
|
[15:15] bpl29
|
thanks pieterh. Am I wrong or is this just a function reference though?
|
[15:15] pieterh
|
bpl29: these objects are not accessed except via functions
|
[15:15] pieterh
|
opaque structures
|
[15:16] bpl29
|
Ah, I see! Thank you!
|
[15:17] cremes
|
sustrik: i'll submit a small C example that reproduces the EBADF
|
[15:26] Guthur
|
Is 0MQ buildable on solaris 8
|
[15:26] Guthur
|
and is Sun Studio the only option
|
[15:26] private_meta
|
pieterh: again a (for you maybe opaque) question: In the ppqueue, ppworker, lpclient scenario, shouldn't lpclient connect at least ONCE? it only tells me it's trying to connect
|
[15:33] Guthur
|
pieter: which pattern would you suggest for a N client to N servers, where the servers are processing requests to retrieve data from files
|
[15:49] cremes
|
Guthur: i'd use a REQ socket on the client, REP on the server and a QUEUE device in the middle to load balance
|
[15:56] Guthur
|
cremes: Probably a wise choice, I have to resist the temptation to over engineer
|
[15:57] cremes
|
Guthur: i suspect that over the next several months the "lazy pirate" patterns will become dominant
|
[15:57] Guthur
|
I was being drawn to Ch4 patterns actually
|
[15:57] Guthur
|
I've only started reading it though
|
[15:58] Guthur
|
I got hung up debugging peering3 in C#
|
[15:58] Guthur
|
massive 'doh' moment on my part there
|
[16:04] private_meta
|
I'm still curious if the way the zguide C variant of the paranoid pirate is intended behavior. If i run the c example, the lazy pirate client never connects
|
[16:04] private_meta
|
It shouldn't be intended
|
[16:04] private_meta
|
But somehow I don't think it should be an error on my side, I didn't really change anything
|
[16:47] michelp
|
morning
|
[16:48] michelp
|
i asked this q last night but i think everyone was in bed. has anyone in the community experimented with using 0mq to do shared memory objects across threads/procs/boxes in python?
|
[16:48] michelp
|
it's something that's always interested me but i've never found a suitable transport for the underlying data transfer
|
[16:49] michelp
|
seems like 0mq is ideal for things like distributing new versions of objects and doing two or three phase commit between a group of data managers
|
[17:15] Guthur
|
michelp: sounds like an interesting project
|
[17:16] Guthur
|
you could maybe utilize something like tokyo cabinet as well
|
[17:17] michelp
|
for storage? i was thinking of shipping around pickles so those would go well into a key value store where the key is the oid
|
[17:18] michelp
|
it would be nice for there to be a way for a new node to easily bootstrap objects and for objects to survive cluster destruction
|
[17:19] michelp
|
might even be possible for there to be more objects in the store than nodes have memory if there was a way to manage that. thanks for the idea Guthur :)
|
[17:20] Guthur
|
also check out some of the stuff from Ch4 of the guide
|
[17:20] Guthur
|
for reliability
|
[17:20] michelp
|
yeah i'm about halfway through ch 3 :)
|
[17:21] michelp
|
almost there, it's a big guide, which is totally awesome
|
[17:21] michelp
|
the best documentation i've found on rabbit is a blog post. the guide alone has convinced me that 0mq is the future, i haven't even written any code yet :)
|
[17:22] michelp
|
the guide would make an excellent 5 day course. does anyone do 0mq training?
|
[17:23] Guthur
|
not that I am aware
|
[17:23] Guthur
|
I'm sure for corporate customers Imatix might be available for such
|
[17:23] Guthur
|
but don't quote me on that
|
[17:24] michelp
|
their services page does have various support level costs, but we're a US non-profit so it's a little out of our price range for a whole week
|
[17:25] michelp
|
if i worked for a corporation i'd be spending their money on it in a minute :)
|
[17:26] michelp
|
know if anyone in the 0mq community is going to OSCON this year? It's here in my home town and we could get together for a BOF session
|
[17:27] Guthur
|
I'm not sure
|
[17:27] Guthur
|
there is sometimes meetups
|
[17:28] Guthur
|
there was one in San Fran recently
|
[17:35] Guthur
|
anyone built jzmq on windows recently
|
[17:38] pieterh
|
michelp: iMatix does 0MQ training, sure
|
[17:39] michelp
|
pieterh, is it the usual services rate quoted here? http://www.imatix.com/services
|
[17:39] pieterh
|
private_meta: (a) lpclient should connect, yes, I had '-v' to enable verbose tracing of messages in the C code, that helps. (b) let me retest the C code...
|
[17:40] pieterh
|
michelp: yes, those are the usual rates
|
[17:40] pieterh
|
however if you're a non-profit, and you need help, you will find that it's readily available here
|
[17:41] michelp
|
yeah you guys have definitely be a great help so far!
|
[17:41] pieterh
|
mostly, you will benefit most by also contributing somehow to the community
|
[17:41] Guthur
|
I'm getting an error building the jzmq binding, error copying files
|
[17:41] Guthur
|
and Java users around
|
[17:41] pieterh
|
Guthur: I've not tried it in a long time... what's the error?
|
[17:42] Guthur
|
Error 1 error MSB3073: The command "copy ..\config.hpp ..\..\..\src javac ..\..\..\src\org\zeromq\ZMQ.java ..\..\..\src\org\zeromq\ZMQException.java ..\..\..\src\org\zeromq\ZMQForwarder.java ..\..\..\src\org\zeromq\ZMQQueue.java ..\..\..\src\org\zeromq\ZMQStreamer.java :VCEnd" exited with code 9009. C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppCommon.targets 103 6 jzmq
|
[17:42] pieterh
|
michelp: with respect to shared objects across threads/processes, it's a design I'm going to make for Ch5 of the Guide
|
[17:42] private_meta
|
pieterh: thanks for retesting
|
[17:44] pieterh
|
Guthur: I'm searching what exit code 9009 means...
|
[17:44] pieterh
|
try running that command by hand in a shell, does it work?
|
[17:45] pieterh
|
"9009 DNS server not authoritative for zone."???
|
[17:45] Guthur
|
2 secs
|
[17:46] michelp
|
pieterh, wow awesome. i'm tinkering with the idea here in python, what language were you going to focus on?
|
[17:46] pieterh
|
michelp: I always make the examples in C, but they can be translated into other languages quite simply
|
[17:47] pieterh
|
the idea is to make a distributed key-value store using a mix of pubsub and req/rep
|
[17:47] Guthur
|
umm syntax is incorrect
|
[17:48] pieterh
|
Guthur: syntax looks correct, yes, and the command runs successfully?
|
[17:48] pieterh
|
private_meta: indeed, it doesn't work... checking why...
|
[17:53] Guthur
|
oh wait
|
[17:53] michelp
|
pieterh, sound very interesting, i'll keep tinkering on my end for learning purposes until it comes out. I'm also happy to proof read anything from a beginners point of view if you need that
|
[17:53] pieterh
|
michelp: what really helps is running the examples from the guide, each one, and verifying that they work
|
[17:54] michelp
|
yeah i've been doing that, which is why i'm only up to ch 3 at this point
|
[17:54] michelp
|
i've been running them in python but I know C so i've been using both as a guide to understanding
|
[17:54] pieterh
|
:-) you won't need any training
|
[17:55] michelp
|
hopefully not, but we have 7 folks on our team and i was thinking it might be worth having a group training session in the future if we can swing it
|
[17:55] Guthur
|
pieterh: the javac wasn't in the Path
|
[17:55] Guthur
|
hopefully should work now
|
[17:55] michelp
|
which is one reason why i was hoping maybe someone from imatix would be at OSCON :)
|
[17:56] pieterh
|
Guthur: worth noting on the Java bindings page, perhaps (you can edit it)
|
[17:56] Guthur
|
sure
|
[17:56] pieterh
|
michelp: we didn't plan to go
|
[17:56] Guthur
|
How does one generate a lib file again?
|
[17:56] Guthur
|
oh wait I think that is documented
|
[17:56] michelp
|
yeah it's expensive. not to mention travel. are you guys in the netherlands?
|
[17:57] pieterh
|
Brussels
|
[17:57] pieterh
|
Belgium
|
[17:58] pieterh
|
It's more a question of time really, plus we've found that presentations from actual 0MQ users are more effective than our own, often
|
[17:59] michelp
|
ah nice, i've never been. been to just about every country around it except Luxembourg
|
[17:59] michelp
|
one of these days hopefully
|
[18:00] Guthur
|
ok binding page updated
|
[18:02] pieterh
|
private_meta: there was a bug in ppqueue
|
[18:02] pieterh
|
it was inserting an extra null frame at the front of the message
|
[18:03] pieterh
|
line 165, in the C code:
|
[18:03] pieterh
|
- zmsg_wrap (msg, identity, "");
|
[18:03] pieterh
|
+ zmsg_push (msg, identity);
|
[18:03] pieterh
|
thanks for catching this!
|
[18:06] private_meta
|
thanks for fixing it
|
[18:06] private_meta
|
I'll test it a lil while later
|
[18:06] sustrik
|
pieterh: an idea about version numbering
|
[18:07] Guthur
|
is there something on the wiki about building libzmq as a static lib
|
[18:07] sustrik
|
the guarantees for minor version number bump is not clear atm
|
[18:07] Guthur
|
the java binding seems to want one
|
[18:07] sustrik
|
what if it says: "minor version number is incremented when stable branch is forked from the master"
|
[18:09] sustrik
|
Guthur: what OS?
|
[18:09] Guthur
|
sustrik: Win32
|
[18:10] Guthur
|
I think I have it though
|
[18:10] Guthur
|
sorry
|
[18:10] Guthur
|
sustrik: btw why is there an @ before your name
|
[18:10] Guthur
|
are you ops
|
[18:10] sustrik
|
Guthur: iirc building a static lib on win32 is a problem
|
[18:10] sustrik
|
yup
|
[18:11] Guthur
|
umm yeah it does seem to be an issue
|
[18:11] Guthur
|
this is problematic
|
[18:11] Guthur
|
jzmq wants a lib
|
[18:11] sustrik
|
but jzmq used to work with dlls before
|
[18:11] sustrik
|
have that changed?
|
[18:12] Guthur
|
umm not sure, it's complaining at me about not finding a libzmq.lib
|
[18:13] sustrik
|
ah
|
[18:14] sustrik
|
win32 is strange
|
[18:14] Guthur
|
yes, that it is
|
[18:14] Guthur
|
not my first choice to be sure
|
[18:14] sustrik
|
.lib is used even when doing dynamic linking
|
[18:14] sustrik
|
it's basically a table of entry points
|
[18:14] sustrik
|
it's located in lib subdir afterm msvc build
|
[18:14] sustrik
|
iirc
|
[18:17] Guthur
|
so it should be getting generated
|
[18:22] Guthur
|
where is the platform tweaks
|
[18:27] Guthur
|
i can generate a lib but then I get loads of unresolved external symbols
|
[18:42] sustrik
|
like zmq_init() and such?
|
[18:44] Guthur
|
__imp__zmq_init
|
[18:44] Guthur
|
etc
|
[18:48] sustrik
|
can you report that on the mailing list
|
[18:48] sustrik
|
it looks like it is possibly connected to the recent changes regarding symbol visibility
|
[18:49] Guthur
|
ah sure, I'll an old version to check
|
[18:49] Guthur
|
I'd really like to have something for tomorrw
|
[18:49] Guthur
|
I colleague wants to bridge from JMS to C#
|
[18:50] Guthur
|
with possibility for client/server setup
|
[18:50] Guthur
|
I'd like to show him a 0MQ solution
|
[18:51] Guthur
|
sustrik: when was that update you mentioned
|
[19:22] mikko
|
pieterh: there?
|
[20:00] sustrik
|
Guthur: december, i think
|
[20:04] Guthur
|
sustrik, ok, I'll try an earlier version
|
[20:05] Guthur
|
I'm back home now though, on a much saner platform, hehe
|
[21:11] mikko
|
pieterh: there now?
|
[22:05] mikko
|
Guthur: are you michael?
|
[22:07] Guthur
|
mikko, Indeed
|
[22:07] mikko
|
http://www.mansysadmin.com/2011/03/using-zeromq-framework-with-visual-studio-2010-tutorial/
|
[22:08] mikko
|
is this related to your problem?
|
[22:08] mikko
|
i remember something related to VS2010 being funny
|
[22:08] Guthur
|
I haven't been encountering any issues until I tried jzmq
|
[22:09] Guthur
|
I'll double check that though
|
[22:09] Guthur
|
but I am confident it was outputting to the correct directory
|
[22:09] Guthur
|
I'm just freeing up some space on my wins VM here, then I will try again
|
[22:23] sed
|
Is anyone around to answer a couple of questions about ZMQ
|
[22:23] sed
|
?
|
[22:28] Guthur
|
ask and you shall receive...
|
[22:28] Guthur
|
...maybe
|
[22:31] cremes
|
sed: that means yes
|
[22:31] sed
|
we have been testing out ZMQ for about a month and are very impressed with what it can do. However we have noticed memory issues with the bindings in C C++ and perl.
|
[22:32] sed
|
over the course of several weeks we see memory usage rise uniformally across multiple servers
|
[22:32] sed
|
also when running tests on single nodes top shows usage increasing
|
[22:33] mikko
|
sed: are you closing the messages ?
|
[22:33] sed
|
qwe are trying to identify the cause of this and what steps can be taken to avoid having to restart every 2 weeks
|
[22:33] sed
|
yes
|
[22:33] cremes
|
sed: are you closing and recreating sockets often?
|
[22:34] sed
|
no the sockets remain open
|
[22:35] cremes
|
it's hard to say; an example that illustrates the problem in C or C++ would be great
|
[22:35] cremes
|
i have seen some memory leaks but they are due to closing/reopening sockets very rapidly
|
[22:35] cremes
|
i haven't heard about memory leakage with sockets that remain open indefinitely
|
[22:36] cremes
|
are you running 2.1.x or 2.0.x?
|
[22:36] sed
|
2.1.x
|
[22:37] cremes
|
what kind of sockets? (btw, feel free to provide more details without requiring us to interrogate you)
|
[22:38] sed
|
pulling up example code now 1 sec
|
[22:39] sed
|
#include <stdio.h>
|
[22:39] sed
|
#include <iostream>
|
[22:39] sed
|
#include "zmq.hpp"
|
[22:39] sed
|
int main(int argc, char *argv[]) {
|
[22:39] sed
|
int type, tm, uid, pid, city;
|
[22:39] sed
|
char msg[20000], ip[20], host[100], cont[100], act[100], post[2000], get[2000];
|
[22:39] sed
|
// prepare our sockets and context with 1 I/O thread
|
[22:39] sed
|
zmq::context_t context(1);
|
[22:39] sed
|
// Connect to subscriber
|
[22:39] sed
|
zmq::socket_t subscriber(context, ZMQ_SUB);
|
[22:39] sed
|
subscriber.setsockopt(ZMQ_SUBSCRIBE, "", 0);
|
[22:39] sed
|
subscriber.connect("tcp://s2hs-mw-cas1:5566");
|
[22:39] sed
|
while(1) {
|
[22:39] sed
|
zmq::message_t* message = new zmq::message_t;
|
[22:39] sed
|
subscriber.recv(message, ZMQ_NOBLOCK);
|
[22:39] sed
|
strncpy(msg, static_cast<char *>(message->data()), message->size());
|
[22:39] sed
|
delete message;
|
[22:39] sed
|
type = atoi(&msg[0]);
|
[22:39] sed
|
if (type == 3)
|
[22:40] sed
|
{
|
[22:40] sed
|
sscanf(msg,"%d %d %d %s %s %d %d %s %s %s %s", &type, &tm, &uid, &ip, &host, &pid, &city, &cont, &act, &post, &get);
|
[22:40] sed
|
std::cout << msg << "\n";
|
[22:40] sed
|
}
|
[22:40] sed
|
memset ( (void *) &msg, 0, sizeof(msg));
|
[22:40] sed
|
}
|
[22:40] sed
|
free (subscriber);
|
[22:40] sed
|
return 0;
|
[22:40] sed
|
}
|
[22:40] sed
|
that is a simple example script that we see the leak on
|
[22:40] Guthur
|
sed: you really really, should use pastebin
|
[22:40] cremes
|
sed: yeah, i can't read that; use gist.github.com or pastie.org to post code
|
[22:41] sed
|
http://pastebin.com/mp9uJpCs
|
[22:44] cremes
|
sed: how quickly does it leak? a few bytes for each message? is it a time based leak? e.g. it leaks even when messages aren't coming in
|
[22:44] sed
|
we have 40000 messages/sec at all times. Would have to run seperate tests to get that data
|
[22:46] Guthur
|
i know it's not really relevant to the issue, but should that recv not be a blocking call
|
[22:47] cremes
|
sed: does the publisher leak memory or just the subscriber?
|
[22:47] sed
|
est 24k per minute
|
[22:47] cremes
|
cool
|
[22:47] jond
|
sed: I don't think recv should be non blocking, the message doesnt need to be on the heap and the free at end is just wrong
|
[22:47] cremes
|
i see this is over tcp transport; any chance you could do a test where publisher & subscriber communicate via inproc transport?
|
[22:50] sed
|
going to get a script running on the same box
|
[22:50] Guthur
|
sed: Which platform, out of curiosity
|
[22:52] sed
|
centos
|
[22:54] mikko
|
sed: you shouldnt free subscriber
|
[22:54] mikko
|
at the ed
|
[22:55] mikko
|
end*
|
[22:55] mikko
|
sed: also, you dont need to dynamically allocate 'message'
|
[22:56] sed
|
yeah that was just testing if it was related
|
[22:56] mikko
|
that shouldnt leak
|
[22:56] mikko
|
as far as i can see
|
[22:57] jond
|
mikko: what time are you going to the pub tomorrow? hoping not to get my bag stolen this time.....
|
[22:57] mikko
|
jond: dont take a bag with you!
|
[22:57] mikko
|
jond: i probably get off from work around 17:30
|
[22:57] mikko
|
heading to white horse after that so will be there around 18:00
|
[22:58] jond
|
mikko: i have started doing that, learned the hard way
|
[22:58] mikko
|
jond: hows life nowadays?
|
[22:58] mikko
|
i guess we can chat tomorrow
|
[22:59] jond
|
better than before xmas .....
|
[22:59] mikko
|
sed: the only thing i can think of is if messages pile up
|
[22:59] jond
|
i've found a bug in xrep today
|
[22:59] mikko
|
jond: is it related to identities?
|
[23:01] jond
|
when the pipe is full when processing identity message leaves socket in bad state
|
[23:01] jond
|
i also don't think the HWM is honoured correctly but I'd need to show martin
|
[23:02] jond
|
I'm just reading up on how to submit a patch cos I havent done one for months....
|
[23:12] mikko
|
clone the repo, commit the change
|
[23:12] mikko
|
and do git format-patch -s
|
[23:19] cremes
|
this project has the most F'ed up submission policy; it's different from nearly every other github project
|
[23:24] jond
|
mikko: I've sent patch to list. I think I got it right ...
|
[23:26] jond
|
cremes: yep, it's like the version numbering. using master seems to be safest!!!
|
[23:26] cremes
|
jond: don't get me started on that either :)
|
[23:27] cremes
|
this is the only project i know that spawns a *new* repository for a branch
|
[23:30] Guthur
|
a conscientious should be reached on this issue
|
[23:30] Guthur
|
it seems to annoy a few people
|
[23:30] cremes
|
concensus?
|
[23:30] Guthur
|
cremes, yep
|
[23:30] cremes
|
:)
|
[23:30] Guthur
|
hehe trusted the spell checker too much
|
[23:31] cremes
|
i don't feel like fighting this particular battle; i'm not much of a C guy so my patches (if any) will be rare
|
[23:31] mikko
|
jond: didnt look at the contents but the format of the patch looks good
|
[23:31] cremes
|
but it seems to me like it increases the learning curve quite a bit for people new to the project
|
[23:32] mikko
|
im not fan of the dual repos either
|
[23:32] mikko
|
but that seems to work for pieterh and sustrik so i guess its ok
|
[23:32] mikko
|
the version numbering though is very odd for 2.1.x
|
[23:33] cremes
|
yes
|
[23:37] jond
|
mikko: just posted a note to the list about the operator void * on socket which allows that call to free to compile in example code above
|