[Time] Name | Message |
[04:53] CIA-75
|
libzmq: 03Martin Sustrik 07master * rfe2e772 10/ src/pgm_socket.cpp : pgm_socket uses binary version of UUID ...
|
[04:53] CIA-75
|
libzmq: 03Martin Sustrik 07master * reb9bc1b 10/ (src/dist.cpp src/dist.hpp): Message atomicity problem in PUB socket fixed. ...
|
[05:39] pieterh
|
sustrik: issue 191 confirmed fixed on 2.1... :)
|
[05:58] sustrik
|
great
|
[06:22] pieterh
|
sustrik: IMO we need to become more pedantic about test cases for such things
|
[06:22] pieterh
|
without a test case it'd have been impossible to get this working
|
[06:22] sustrik
|
what do you propose
|
[06:22] sustrik
|
?
|
[06:22] pieterh
|
well... we need somewhere to hold test cases
|
[06:23] pieterh
|
issue tracker isn't great for this
|
[06:23] pieterh
|
I'd prefer a git
|
[06:23] sustrik
|
/tests?
|
[06:23] pieterh
|
can't be in libzmq IMO, perhaps a separate git
|
[06:23] sustrik
|
why so?
|
[06:23] sustrik
|
if it's elsewhere it'll bit-rot
|
[06:24] pieterh
|
they'll be historical anyhow, per issue
|
[06:24] pieterh
|
now we have issue 199 with tests
|
[06:24] pieterh
|
a. random contributors (usually person reporting the issue)
|
[06:24] pieterh
|
b. random languages
|
[06:24] pieterh
|
c. random versions of 0MQ (2.1, 3.0 at least)
|
[06:25] pieterh
|
d. perhaps, testing other layers
|
[06:25] pieterh
|
I don't see this as a regression test layer but just an aide for fixing issues as we go along
|
[06:26] pieterh
|
see th's test cases for issue 199... random code on some random URL
|
[06:26] sustrik
|
why do you need those to be persistent?
|
[06:26] pieterh
|
i don't need persistence, I need consistency
|
[06:26] sustrik
|
they'll be completely bit-rotten in 6 months
|
[06:26] pieterh
|
that's fine
|
[06:27] pieterh
|
I don't care about bitrot
|
[06:27] pieterh
|
I care about 'where the heck are the test cases for issue 204 for 2.1?'
|
[06:27] pieterh
|
searching email and irc and issue trackers is a waste of time
|
[06:28] pieterh
|
also, not having a consistent approach means we often don't even have test cases
|
[06:28] pieterh
|
clearly, without a test case for 191 we'd have allowed fairly major breakage
|
[06:28] sustrik
|
hm, you have to look to issue tracker anyway, otherwise you won't have an idea what issue 204 is meant to be
|
[06:28] pieterh
|
also, we need a way to share test code
|
[06:29] pieterh
|
e.g. the gist repo I made for 191 but where anyone can write back to it
|
[06:29] pieterh
|
so the issue tracker is fine for discussion
|
[06:29] pieterh
|
e. multiple authors on repo
|
[06:32] sustrik
|
feel free to try, but imo people will consider it too heavyweight and will use gist/bugtracker instead
|
[06:32] pieterh
|
well, this would only work if we adopt it and enforce it
|
[06:33] pieterh
|
people would like to just email the list saying "hey, 0mq crashes!"
|
[06:34] pieterh
|
ok, I'll make an email to zeromq-dev, everyone will ignore it, then I'll start doing this anyhow, and after a month or two people will bitch at me for fragmenting the repos...
|
[06:34] pieterh
|
sigh
|
[06:37] sustrik
|
hm
|
[06:37] sustrik
|
so once again, what's the rationale?
|
[06:37] pieterh
|
test-driven change, martin
|
[06:38] pieterh
|
every bug has an issue, and test case, which is in C, and ported to 2.1 and 3.0
|
[06:38] pieterh
|
every test case is in a 100% predictable location
|
[06:38] sustrik
|
why not in bug tracker
|
[06:38] pieterh
|
before a fix is applied to a master branch it can be 100% tested
|
[06:38] sustrik
|
or a gist
|
[06:38] sustrik
|
?
|
[06:39] pieterh
|
bug tracker has no attachments
|
[06:39] pieterh
|
a gist has no name
|
[06:39] sustrik
|
gist linked form bug tracker?
|
[06:39] pieterh
|
you provided two C test cases
|
[06:39] pieterh
|
they had bugs
|
[06:39] pieterh
|
did not run on 2.1
|
[06:39] pieterh
|
if I fix this, where do I put the improved versions?
|
[06:39] pieterh
|
where do I put listings, stack dumps, etc.?
|
[06:40] pieterh
|
how does someone download this bundle of stuff?
|
[06:40] pieterh
|
click click click on dozens of links?
|
[06:40] pieterh
|
srsly?
|
[06:40] pieterh
|
gist linked from bug tracker... no predictable location
|
[06:41] pieterh
|
dozens of messy gist repos in working directoruy
|
[06:41] pieterh
|
no obvious naming of cloned repos
|
[06:44] sustrik
|
well, i am not feeling the pain yet
|
[06:44] sustrik
|
clicking on link works ok for me
|
[06:45] th
|
did i do that wrong?
|
[06:45] pieterh
|
th: no, you did exactly what we asked but we're asking for the wrong thing afaics
|
[06:45] th
|
i thought it would be easiest to put all information in one place and link it
|
[06:45] th
|
ahh i see - infrastructure discussion
|
[06:45] pieterh
|
yes
|
[06:45] th
|
i just noted github being slow ;)
|
[06:46] th
|
did not grok all the gist yet
|
[06:46] pieterh
|
sustrik: you don't feel the pain because you avoid a lot of the process
|
[06:46] pieterh
|
that however just means the way you work can't scale
|
[06:46] pieterh
|
e.g. we can't work like that and maintain stable versions
|
[06:46] sustrik
|
ok, give it a try and we'll see how it works
|
[06:47] pieterh
|
I'll make a demo and an email explaining it
|
[06:47] th
|
pieterh: i replied to what i get via email from you via github, but thats not supposed to show up as comment?
|
[06:47] sustrik
|
yes
|
[06:47] pieterh
|
th: it should appear as a comment, yes
|
[06:48] th
|
at least it did not (yet)
|
[06:49] th
|
anyways - added it via the web
|
[06:49] th
|
pieterh: the issue only happens under some load... so while ./stress;do;done
|
[06:50] pieterh
|
th: we definitely need more infrastructure for issue tests
|
[06:50] th
|
and i also tried to port this to 3.0, but i get EAGAIN on send although i dont use NOBLOCK
|
[06:50] pieterh
|
hmm
|
[06:50] th
|
pieterh: the only change i did to "port" was to use zmq_{send,recv}msg instead
|
[06:51] pieterh
|
th: should be sufficient, you don't use a lot of the API
|
[06:51] sustrik
|
th: note the change in the return value
|
[06:51] sustrik
|
in 2.0 send/recv returned 0/-1
|
[06:51] sustrik
|
in 3.0 it returns message size
|
[06:52] th
|
dang
|
[06:52] th
|
if (zmq_sendmsg()) { ... error ... }
|
[06:52] th
|
ok - i'm on it
|
[06:53] th
|
there is no manpage installed of zmq_sendmsg with libzmq (there is zmq_send.3 though)
|
[06:54] pieterh
|
th: also, don't use asserts like that, if you compile in release mode the whole assert is removed
|
[06:54] sustrik
|
th: yes, it's still missing
|
[06:54] th
|
pieterh: yea - but this test case will never be compiled in release mode, right?
|
[06:54] pieterh
|
th: sure
|
[06:54] sustrik
|
should be: if (zmq_sendmsg() < 0) error;
|
[06:54] th
|
sustrik: ok
|
[06:55] th
|
sustrik: same for recv?
|
[06:56] sustrik
|
yes
|
[06:56] th
|
pieterh: i'll add some `assert(test=1);if (test!=1) exit(1);`
|
[07:00] pieterh
|
th: can you clone https://github.com/zeromq/issues
|
[07:00] pieterh
|
and then add your changes, and then make a pull request?
|
[07:00] th
|
Unpacking objects: 100% (23/23), done.
|
[07:00] pieterh
|
I've already create a space for issue 199 and added your current C test cases
|
[07:00] pieterh
|
you can add everything there
|
[07:00] th
|
pieterh: excluding any output.{txt,html} and Makefiles?
|
[07:01] pieterh
|
anything other people need to run the tests, I'd say
|
[07:01] pieterh
|
This is an experiment to see if we can use a git repo for communication instead of pastebins
|
[07:02] th
|
pieterh: i'll do that.
|
[07:02] pieterh
|
tgh: much appreciated
|
[07:02] th
|
but first i'll fix this up for 3.0
|
[07:02] th
|
sustrik: any change on return value of [gs]etsockopt?
|
[07:02] pieterh
|
no hurry
|
[07:02] pieterh
|
th: nope, but most arguments changed
|
[07:03] pieterh
|
e.g. from uint64_t to int
|
[07:03] pieterh
|
so if you use the old types you'll get an EINVAL error
|
[07:03] th
|
well that direction (making it smaller) does not hurt in my case where i use ZMQ_XXXMORE
|
[07:04] th
|
pieterh: i dont use type - i pass a pointer and a pointer to a size, right?
|
[07:04] pieterh
|
th: in C?
|
[07:04] th
|
ZMQ_EXPORT int zmq_getsockopt (void *s, int option, void *optval, size_t *optvallen);
|
[07:04] th
|
wait you're talking about `int option`??
|
[07:05] pieterh
|
see https://github.com/zeromq/czmq/blob/master/src/zsockopt.c
|
[07:05] th
|
and not the value?
|
[07:05] th
|
pieterh: i see that... zmq_getsockopt (socket, ZMQ_RCVMORE, &rcvmore, &type_size);
|
[07:06] th
|
int64_t
|
[07:06] pieterh
|
rcvmore changed from int64_t to int
|
[07:06] pieterh
|
see line 761
|
[07:06] th
|
what about line 403?
|
[07:06] pieterh
|
the zsockopt class (czmq) has both 2.1 and 3.0 support (two large conditional blocks)
|
[07:07] pieterh
|
line 403 is 2.1, line 761 is 3.0...
|
[07:07] th
|
ahhhhh
|
[07:07] th
|
ok
|
[07:08] th
|
pieterh: so i get EINVAL if last argument's value is 8 instead of 4?
|
[07:08] pieterh
|
yes
|
[07:08] th
|
although - only the pointer to type_size is passed.
|
[07:09] th
|
but thats always of size_t?
|
[07:09] pieterh
|
yes
|
[07:09] th
|
ok
|
[07:09] th
|
why a pointer at all? would i get the amount of bytes needed if i pass to small buffer?
|
[07:09] th
|
s/to small/too small/
|
[07:09] pieterh
|
th: this is the standard POSIX API for get/setsockopt
|
[07:12] th
|
`and modified on return to indicate the actual size of the value returned.`
|
[07:12] th
|
okay
|
[07:13] th
|
will a recv in 3.0 ever return 0 to indicate EOF or something like this?
|
[07:14] th
|
pieterh: ok - same effect with 3.0
|
[07:14] pieterh
|
well, hang on, there's no EOF in 0MQ
|
[07:15] pieterh
|
:)
|
[07:15] th
|
pieterh: thats why i said "or something like this?" ;)
|
[07:16] th
|
i wonder if this issue would still be an issue if i would not be on the VSM side...
|
[07:18] pieterh
|
th: I don't get the crash on my box
|
[07:18] pieterh
|
what OS are you using?
|
[07:19] pieterh
|
I'll let it run for a while...
|
[07:21] th
|
pieterh: you also dont get the "STUPID RECV (JUST FOR SHOWING WHATS WAITING)"?
|
[07:22] th
|
pieterh: i replaced some assert(0) with an endless loop of juse receiving and printing, to see the order of messages
|
[07:22] th
|
pieterh: Ubuntu 10.04.2 LTS on 64bit
|
[07:23] th
|
pieterh: i run it |& tee log and cancel the client-while-loop when i see the "STUPID RECV"
|
[07:47] rgl
|
pieterh, I loved the intro on http://zguide.zeromq.org/page:all thx for putting it up!
|
[07:51] pieterh
|
rgl: glad you liked it, some people seem to hate it...
|
[07:51] pieterh
|
th: I eventually got a crash in the server: Assertion failed: fetched (xrep.cpp:248)
|
[07:52] th
|
pieterh: i assume - that is just because you ran out of resources
|
[07:52] rgl
|
pieterh, oh.. oh why?
|
[07:52] th
|
pieterh: it fails for me within the first 3 seconds
|
[07:53] Toba
|
pieterh: too bad sockets in pollitem are void*, makes it easy to mess that up
|
[07:53] th
|
pieterh: although as said above - i dont let it crash when fail happens
|
[07:53] pieterh
|
rgl: no idea...
|
[07:53] th
|
pieterh: to document the order of the rest of the parts
|
[07:54] pieterh
|
th: I think there's a bug in xrep, not sure...
|
[07:55] th
|
pieterh: so did you get the "STUPID RECV..." messages as well
|
[07:55] th
|
?
|
[07:55] pieterh
|
sustrik: is it normal to do msg_->close () and then access msg_->flags ()?
|
[07:55] pieterh
|
th: sure, loads of them
|
[07:55] th
|
pieterh: then it was intermixing
|
[07:56] pieterh
|
th: is that good or bad? I've no idea what I'm supposed to be seeing...
|
[07:56] th
|
pieterh: it means that atomicity of multiparts was violated.
|
[07:56] pieterh
|
it'd be useful to be explicit in any output you make, e.g. "YOU SHOULD NOT SEE THIS"
|
[07:57] th
|
pieterh: ok - see the line above the first "STUPID"
|
[07:57] th
|
pieterh: i'll make it clearer
|
[07:57] pieterh
|
also, it should simply stop at the first error
|
[07:57] th
|
pieterh: but i wanted to print the "in queue" messages as well to show if there was a duplicate
|
[07:57] pieterh
|
so when I run it, I know immediately if it fails
|
[08:44] th
|
pieterh: shall i use markdown in the README?
|
[08:44] pieterh
|
th: as you like
|
[08:44] th
|
then plain txt
|
[08:44] pieterh
|
indeed...
|
[09:04] th
|
pieterh: git pull http://thzn.de/0mq_git/issues/ master
|
[09:05] pieterh
|
th: cool, it works nicely
|
[09:05] pieterh
|
merged into https://github.com/zeromq/issues
|
[09:05] th
|
pieterh: because i just rebased, after your last 2 commits ;)
|
[09:05] pieterh
|
does this seem workable for making / sharing test cases?
|
[09:05] th
|
pieterh: for me - yes.
|
[09:06] th
|
pieterh: not sure about the common php tinker
|
[09:06] pieterh
|
what is that?
|
[09:06] pieterh
|
ah, you mean for average folk
|
[09:06] th
|
sort of
|
[09:07] pieterh
|
well, in any case we need to be able to make C test cases for real work
|
[09:07] pieterh
|
I'm usually able to make that if someone has a test case in another language
|
[09:08] pieterh
|
let me try to test 199 again now that I know what to look for... :)
|
[09:08] th
|
pieterh: in my case c, c++, 2-1, 3-0 - it's all the same, yes
|
[09:08] pieterh
|
yup
|
[09:08] th
|
although it's not as broken down, as i had liked.
|
[09:10] pieterh
|
th: so I get the 'STUPID' stuff in the server but it doesn't assert
|
[09:10] th
|
pieterh: there should be no more "STUPID" output
|
[09:10] th
|
pieterh: recompile!
|
[09:10] th
|
i changed the code
|
[09:10] pieterh
|
hmm
|
[09:10] th
|
199/3-0/
|
[09:11] th
|
grep for STUPID
|
[09:11] th
|
no stupidity in there
|
[09:11] pieterh
|
th: ah, I'm using 2-1
|
[09:11] th
|
pieterh: ignore that - let's care about backporting after it's fixed upstream
|
[09:11] pieterh
|
well, I need the testcase on 2-1 in any case, otherwise I can't test downstreamed patches
|
[09:12] pieterh
|
I'll port the test cases back to 2-1, hang on...
|
[09:12] rgl
|
how does one authenticate the client of a connection like traditional sockets? we have to use some kind of message authentication?
|
[09:12] th
|
pieterh: but could we do that after 3-0?
|
[09:12] pieterh
|
rgl: it depends on the pattern you're using
|
[09:12] pieterh
|
there's no authentication as such in 0MQ
|
[09:13] pieterh
|
th: in fact we could make portable test cases if we used czmq
|
[09:13] th
|
pieterh: it's a s/0 <= zmq_/0 == zmq_/ for the asserts and then the s/msg// for the name-change
|
[09:13] pieterh
|
rgl: for an example of secure pubsub, see the Salt project (at http://www.zeromq.org/docs:labs)
|
[09:13] rgl
|
pieterh, for instance, pub/sub, where not everyone can do pub
|
[09:13] th
|
rgl: how about applying asymmetric cryptography?
|
[09:13] pieterh
|
:)
|
[09:14] rgl
|
oh :)
|
[09:14] rgl
|
will look at it. thx :)
|
[09:19] pieterh
|
th: ok, failure reproduced on 2.1... excellent
|
[09:19] pieterh
|
we can pass this to sustrik now, get a patch and downstream that to 2.1
|
[09:20] th
|
;-)
|
[09:20] th
|
pieterh: and reproduced on 3-0 as well, i suppose?
|
[09:21] pieterh
|
th: hang on... (I don't really care about 3-0 so much in this process)
|
[09:22] pieterh
|
yes, it hits the same way on 3-0
|
[09:22] pieterh
|
th: so I sent an email to zeromq-dev proposing we hold test cases in a single repo, you may want to comment on that thread
|
[09:23] th
|
pieterh: ok - and you need to comment on your last mail in the #199 thread
|
[09:27] th
|
pieterh: commented
|
[09:30] pieterh
|
I've replied to the 199 thread now
|
[09:46] rgl
|
can you point me to the document that describes the wire format of the several socket types?
|
[09:46] pieterh
|
rgl: unfortunately, there's no single document
|
[09:47] pieterh
|
there is some information in the zmq_tcp man page
|
[09:48] pieterh
|
we really do need to make a proper documentation of the wire format, no real excuses
|
[09:49] rgl
|
indeed :)
|
[09:49] rgl
|
even after writting this "But really, Zookeeper should be using a generic messaging layer and an explicitly documented wire level protocol" hehe
|
[09:50] pieterh
|
yes, absolutely true
|
[09:51] pieterh
|
I did start on a wlp at http://rfc.zeromq.org/spec:2
|
[09:51] pieterh
|
but I need to convince the core developers to actually commit to such a contract
|
[09:51] rgl
|
the wire protcol described at http://api.zeromq.org/2-1:zmq-tcp is not the whole story?
|
[09:51] pieterh
|
otherwise it's just documenting code behaviour, which isn't acceptable IMO
|
[09:52] pieterh
|
there is a bunch of stuff missing from the zmq-tcp page
|
[09:52] pieterh
|
sockets exchange identities when they connect
|
[09:52] pieterh
|
it's on some tutorial somewhere but I don't remember which one
|
[09:52] pieterh
|
don't get me started on the failure of this project to write decent specs in advance... :-)
|
[09:53] rgl
|
I think I did heheh
|
[09:53] pieterh
|
it's one of the topics we hope to nail down at the Brussels event later this month
|
[09:54] pieterh
|
but basically until the development process uses contracts properly, it won't happen IMO
|
[09:54] rgl
|
what do you mean by "uses contracts"?
|
[09:54] pieterh
|
otherwise it's one set of people changing code, a second set trying to document that
|
[09:54] pieterh
|
contracts = upfront discussion & documentation of APIs and WLPs
|
[09:55] pieterh
|
with running code to demo, as needed
|
[09:55] rgl
|
WLP?
|
[09:55] pieterh
|
but not after-the-fact documentation of code
|
[09:55] pieterh
|
wire level protocols
|
[09:56] th
|
pieterh: http://rfc.zeromq.org/spec:2 shouldn't the 3rd occurance of "Message size" be actually "Message body"?
|
[09:56] rgl
|
I see. it makes sense :)
|
[09:56] pieterh
|
th: the text was copied as-is from the tcp man page
|
[09:56] pieterh
|
it is message size, afaics
|
[09:56] pieterh
|
it's a 64-bit size,
|
[09:57] th
|
then i totally dont grok that sketch
|
[09:57] pieterh
|
th: the sketch is pretty nasty
|
[09:57] pieterh
|
what it says is, either a 1-byte length, or 0xff followed by an 8-byte length
|
[09:58] th
|
pieterh: the upper part is the < 255octet case, right?
|
[09:58] pieterh
|
yes
|
[09:58] th
|
pieterh: yes- good. so i got the 0xff+message size. but why is it followed by another message size?
|
[09:58] pieterh
|
th: it's not, that last byte is the 8th byte
|
[09:59] pieterh
|
the diagram is lousy
|
[09:59] pieterh
|
god, let me fix it...
|
[09:59] th
|
ahhh
|
[09:59] th
|
the columns are for 32 bit
|
[09:59] th
|
so it is three rows for the 64bit size, because the first octet is for the 0xff
|
[10:00] th
|
gosh - got it now :)
|
[10:00] th
|
perhaps you should add a "(cont.)" after the continuations of the "message size"
|
[10:00] th
|
which would not fit in the 8 bit field ;)
|
[10:02] pieterh
|
th: check it now
|
[10:02] pieterh
|
this is still not how I'd draw it, but whatever
|
[10:04] rgl
|
it would be nice to have some wireshark inspectors on the zeromq sockets, that is, somehow make wireshark interpret the data inside a 0mq socket.
|
[10:04] pieterh
|
rgl: there's a whole bunch of nice things we could make if we had decent WLP specs
|
[10:05] pieterh
|
there's about one request for these specs every two weeks
|
[10:10] th
|
pieterh: at least it got less confusing in my opinion ;)
|
[10:10] pieterh
|
I'm going to start on a new RFC, this annoys the heck out of me
|
[10:10] pieterh
|
at least we can document the 2.1 format properly
|
[10:10] pieterh
|
and then bug sustrik to do the same for 3.0
|
[10:40] mikko
|
pieterh: hi
|
[10:40] pieterh
|
hi mikko
|
[10:40] pieterh
|
how's up?
|
[10:40] pieterh
|
what's life?
|
[10:40] mikko
|
it's something that happens when you don't work?
|
[10:41] pieterh
|
dunno... sounds weird to me
|
[10:41] mikko
|
anyway, you can use gists for simple test-cases as well
|
[10:41] pieterh
|
yes, I tried that, but
|
[10:41] mikko
|
they are stored as git repositories and can be forked, cloned etc
|
[10:41] pieterh
|
a. you get weird names in your directories
|
[10:42] pieterh
|
b. you can never remember what the gist is called so need to keep referring back to some random URI somewhere
|
[10:42] pieterh
|
c. you can't have multiple people working on the same gist
|
[10:43] pieterh
|
d. it's very hard to enforce consistency when every test case is a separate repo
|
[10:43] pieterh
|
e. it's annoying when I have 20 test cases to have 20 random hash-encoded directories to work with
|
[10:44] pieterh
|
f. /me can go on all day
|
[10:51] pieterh
|
mikko: sorry for ranting :)
|
[10:53] mikko
|
i'm getting worried by the amount of repos :)
|
[10:53] mikko
|
ever growing
|
[10:54] pieterh
|
mikko: we had a similar issue at wikidot
|
[10:54] pieterh
|
originally, all the official stuff was on one site, maintained by a couple of people
|
[10:55] pieterh
|
I slowly broke that into 20-30 sites, each with a distinct group of contributors
|
[10:55] pieterh
|
it was quite interesting, the tension that caused
|
[10:55] pieterh
|
OTOH people complained they couldn't navigate the results
|
[10:55] pieterh
|
OTOH, the number of contributors overall increased by 50x or more
|
[10:56] pieterh
|
since every gist is a repo, we already have massive numbers of repos, except they're unknown
|
[10:57] pieterh
|
anyhow, I don't see how https://github.com/zeromq is any different from subdirectories within one repo
|
[11:00] mikko
|
i think i was thinking a workflow like:
|
[11:01] mikko
|
each bigger issue gets branched off to a separate branch, the test cases are added to branch and when everything works it gets merged back
|
[11:01] mikko
|
so that also test cases get back to main repo
|
[11:02] pieterh
|
mikko: that would work nicely except we have to work across multiple topic branches
|
[11:02] pieterh
|
multiple main repos
|
[11:03] pieterh
|
how do we handle an issue that affects both pyzmq and libzmq?
|
[11:03] pieterh
|
or 3.0 and 2.1
|
[11:03] pieterh
|
or erlzmq2 and erlzmq3
|
[11:03] pieterh
|
etc.
|
[11:03] mikko
|
when i've had such issues with php-zmq i've created a reproducible test-case in C or C++
|
[11:04] pieterh
|
at the least, we need a way to maintain C/C++ test cases to work on both 2.1 and 3.0
|
[11:04] pieterh
|
can't use a topic branch in 3.0 for that
|
[11:04] pieterh
|
wouldn't work if 2.1 and 3.0 were branches in same repo, either
|
[11:05] pieterh
|
it'd require one branch for each test case / version combination
|
[11:05] pieterh
|
ugh
|
[11:05] pieterh
|
also, as sustrik pointed out, we don't want test cases to go into the main repo because they rot really fast
|
[11:06] pieterh
|
unless someone's actually maintaining them, which is a role no-one has volunteered for...
|
[11:07] pieterh
|
ok, one of my hopes is that a separate issues repo can become the basis for an independent regression test suite
|
[11:11] mikko
|
what do you mean they rot really fast?
|
[11:12] mikko
|
we've been using the same set of tests through out 2.x
|
[11:16] pieterh
|
mikko: old test cases don't keep working unless they're maintained
|
[11:16] pieterh
|
I know they *should* keep working, for regression testing, but that demands effort
|
[11:16] mikko
|
does that mean that old software doesnt keep working unless maintained all the time?
|
[11:17] pieterh
|
indeed
|
[11:17] pieterh
|
since today, test cases are transient and discarded, we don't see the cost
|
[11:17] pieterh
|
e.g. there are test cases that apply to 2.0.4
|
[11:18] pieterh
|
actually I'd be happy to maintain the regression tests, if the tests were all in one place and properly structured
|
[11:18] pieterh
|
e.g. always started with a consistent command
|
[11:18] pieterh
|
built using a consistent tooling
|
[11:18] pieterh
|
etc.
|
[11:18] pieterh
|
it is doable (and still, a lot of work) if they're in their own repo IMO
|
[12:17] pieterh
|
ok, finished draft of full wire level protocol
|
[12:18] pieterh
|
rgl: if you're still around: http://rfc.zeromq.org/spec:13
|
[13:03] xbmc_fan
|
hi
|
[13:04] xbmc_fan
|
any experts for the .net 0mq-binding available?
|
[13:07] iFire
|
anyone made a pdf of zguide?
|
[13:09] iFire
|
"only" a 105 pages
|
[13:15] mikko
|
xbmc_fan: Guthur is i think
|
[13:17] mikko
|
iFire: seems to print to pdf well
|
[13:19] mikko
|
http://valokuva.org/~mikko/zguide.pdf
|
[13:19] mikko
|
that's what i get out from print to pdf
|
[13:20] xbmc_fan
|
thanks mikko.. Maybe Guthur, you are able to answer this Stackoverflow question: http://stackoverflow.com/questions/5841896/0mq-how-to-use-zeromq-in-a-threadsafe-manner ?
|
[13:22] mikko
|
xbmc_fan: you cannot use sockets concurrently
|
[13:22] mikko
|
xbmc_fan: but with zeromq 2.1 you can migrate socket from one thread to another
|
[13:23] xbmc_fan
|
mikko, I understood that I must not use sockets from different threads
|
[13:23] mikko
|
that was the case with 2.0
|
[13:23] mikko
|
you cannot use a socket concurrently from multiple threads
|
[13:24] mikko
|
you have to ensure a full memory barrier before accessing from another thread
|
[13:24] xbmc_fan
|
i got that. In order to not use it concurrently, I have to use semapohores, locks- which the guide explicitly warns about
|
[13:25] mikko
|
what does it say?
|
[13:25] xbmc_fan
|
please read the question I aksed at stackoverflow
|
[13:25] Guthur
|
xbmc_fan, just don't share them across threads would be the easy option
|
[13:25] mikko
|
xbmc_fan: can you have socket per thread?
|
[13:26] mikko
|
that is usually ideal if multiple threads need to communicate with same endpoint
|
[13:26] mikko
|
mutexes / locks don't necessarily make your application slow and fragile. they are just very hard to get right
|
[13:26] iFire
|
mikko just finished printing
|
[13:26] Guthur
|
sometimes what I do is create the socket in one thread and give it to another to use within it's lifetime
|
[13:26] iFire
|
ha 6 minutes
|
[13:27] xbmc_fan
|
mikko, I could but is this really performant / the way to do things properly? e.g. I might get called from a different temporary-threadpool thread
|
[13:27] xbmc_fan
|
and I *think* there is overhead involved to create a 0mq-socket each time my method gets called
|
[13:27] mikko
|
xbmc_fan: is there going to be lock contention?
|
[13:28] mikko
|
xbmc_fan: why do you have multiple threads handling events?
|
[13:28] mikko
|
not sure if that is common in C#
|
[13:29] xbmc_fan
|
mikko, it is quite common: the class registering for an event does not know which thread invokes the eventhandler
|
[13:30] xbmc_fan
|
I have multiple threads throwing events. and the eventhandler gets called from those threads.
|
[13:31] Guthur
|
I did create a socket pool once, never really used it though
|
[13:31] Guthur
|
I was considering including it in clrzmq2, but I don't think I did in the end
|
[13:34] xbmc_fan
|
http://pastebin.com/EdywPxWa <-- maybe this clearifies my problem
|
[13:35] Guthur
|
xbmc_fan, if you are worried about the setup cost of sockets you could try the pool approach
|
[13:38] Guthur
|
or I suppose just lock the socket
|
[13:39] xbmc_fan
|
so I would have a number of sockets in a pool. But since every socket has to be used from the same thread it was created on, this won't help much, or do I miss something? e.g.: If the eventhandler is invoked from temporary threads, I would create a socket for those calls because there isn't a socket for that thread in the pool yet. I can see that if there is a limited amount of (long running) threads calling, pooling of sockets might ma
|
[13:39] xbmc_fan
|
But I think in my scenario, in the end it would leak ressources - so one had to implement something like "pool timeouts"... too much afford isn't it?!
|
[13:39] Guthur
|
I have the socket pool code if you want me to post it
|
[13:40] xbmc_fan
|
sure, maybe that helps
|
[13:40] Guthur
|
xbmc_fan, in the new version of 0MQ it can be used from a different thread
|
[13:40] Guthur
|
just not from 2 threads at the same time
|
[13:40] Guthur
|
mikko, correct?
|
[13:40] xbmc_fan
|
Guthur or I suppose just lock the socket <-- I could do that, even though the guide says that evil.
|
[13:40] xbmc_fan
|
Guthur just not from 2 threads at the same time <-- ok, that would be nice.
|
[13:41] Guthur
|
xbmc_fan, yeah, pool would be cooler than lock, hehe
|
[13:42] rgl
|
pieterh, oh, you actually did it :)
|
[13:42] Guthur
|
xbmc_fan, https://gist.github.com/949677
|
[13:42] pieterh
|
rgl: yeah, it did take a whole two hours...
|
[13:43] Guthur
|
that is very much as is, no warranty or anything
|
[13:43] Guthur
|
I can't remember if I even completed it, but might serve as some inspiration
|
[13:43] rgl
|
pieterh, humm "The principal issue with ZMTP/1.0 is the lack of content types in the wire format" bummer. so there is no easy way to create a wireshark interpreter.
|
[13:43] pieterh
|
rgl: you can make it but it'll only work at the framing level
|
[13:45] rgl
|
pieterh, I see. I'll have to read it with more attention :)
|
[13:49] xbmc_fan
|
thanks Guthur. Well, the collection and handing out of sockets seems to be threadsafe, but the the pool might run empty eventually. Nevertheless, thanks for posting it!
|
[13:51] xbmc_fan
|
Guthur sometimes what I do is create the socket in one thread and give it to another to use within it's lifetime <-- and that does work? So the author of the docs had .net in mind when he/she wrote: "The only place where it's remotely sane to share sockets between threads are in language bindings that need to do magic like garbage collection on sockets." ?!
|
[13:53] Guthur
|
xbmc_fan, I think it's that some people might abuse the fact that it can be moved from one thread to another
|
[13:53] pieterh
|
iFire: I've made a quick PDF conversion
|
[13:53] Guthur
|
it still doesn't support shared access
|
[13:53] pieterh
|
am uploading it now, 30 seconds...
|
[13:54] Guthur
|
the .net binding does nothing extra to facilitate moving the socket from one thread to another
|
[13:54] Guthur
|
it came with one of the later version of libzmq
|
[13:55] xbmc_fan
|
ok, thanks for the clarifcation guthur. maybe this should be stated also in the docs.
|
[13:57] pieterh
|
xbmc_fan: the reason for this text is that people often tried to share sockets between threads, and it crashes libzmq horribly
|
[13:58] pieterh
|
you can create a socket in thread A and pass it to thread B if, and only if, you do a full memory barrier
|
[13:59] xbmc_fan
|
with "full memory barrier" you mean that I have to make sure that thread A doesn't access the socket at the same time as thread B?
|
[14:01] pieterh
|
"full memory barrier" is a technical term meaning an instruction that causes all CPUs to synchronize and flush caches
|
[14:02] pieterh
|
it's definitely not enough to "not use at the same time"
|
[14:10] xbmc_fan
|
ok, I think I understood pieterh. Therefore it seems I have to use locks to synchronize acces to a single socket if I have no control about the caller's thread.
|
[14:10] pieterh
|
something like that, I've never actually done that
|
[14:11] pieterh
|
xbmc_fan: I'd probably look for an alternative design than that though...
|
[14:13] xbmc_fan
|
pieterh: well, in my case, the architecture is inheritly multithreaded and thread-synchronization takes only place at the outer edges. Currently only for the GUI thread.
|
[14:13] xbmc_fan
|
however, the gui dispatches information which is to be send in a worker thread.
|
[14:13] xbmc_fan
|
I would like to send this data via 0mq.
|
[14:14] pieterh
|
xbmc_fan: have you already tried making multithreaded apps that communicate over inproc sockets?
|
[14:14] pieterh
|
if you are using locks (and IMO if you're sharing sockets between threads) then your design is bogus somewhere
|
[14:16] xbmc_fan
|
pieterh: I haven't used "inproc sockets" for that yet. But I am using a kind of serice bus / event driven architecture. I am sure that - under the hood - this can be implemented using inproc sockets.
|
[14:16] pieterh
|
xbmc_fan: when you discuss multithreading and 0MQ you have to know how inproc sockets work
|
[14:17] pieterh
|
otherwise 100% certain you are looking at the wrong picture
|
[14:17] pieterh
|
please look at the asynchronous server example
|
[14:27] xbmc_fan
|
I think you mean that one: http://zguide.zeromq.org/cs:asyncsrv ?
|
[14:30] xbmc_fan
|
If I understand both the example and your statement correctly, you wanted me to recognize that there is no locking/synchronization of threads involved when using inproc sockets
|
[14:34] xbmc_fan
|
however, at the outer edge of that example, there is in fact synchronization, since the call to Console.Writelline featrues a [MethodImpl(MethodImplOptions.Synchronized)] attribute under the hood. That's the same in my case: the Gui has to synchronize threads to the gui layer. And I have to synchronize non-0mq callers (events, not inproc sockets) if I want to use 0mq at another outer edge of the architecture. From that edge onwards, I c
|
[14:34] xbmc_fan
|
inproc sockets and only have to synchronize again if there is another "edge"
|
[14:36] xbmc_fan
|
I also understood that I could have used 0MQ-Socket throughout the software and would only need to synchronize for the GUI, if I used inproc sockets.
|
[14:40] xbmc_fan
|
am I missing something?
|
[14:46] rgl
|
pieterh, "Each side of the connection consists of an greeting message followed by zero or more content messages. An identity message consists of one frame containing the peer's identity" humm what is an "identity" message? is the "greeting" message?
|
[14:46] pieterh
|
greeting = anonymous / identity
|
[14:46] pieterh
|
let me fix that text
|
[14:47] pieterh
|
I've gotten other review comments as well, new version coming RSN
|
[14:47] xbmc_fan
|
anonynous = %0x01 %x00 <-- isn't it anonyMous
|
[14:47] xbmc_fan
|
"Full ZMTP Grammar"
|
[14:51] xbmc_fan
|
pieterh , did I miss something or did I missinterpreted something?
|
[14:51] pieterh
|
xbmc_fan: fixed that, thanks
|
[14:52] xbmc_fan
|
ah, yeah, np. I mean a few sentences back.. I tried to understand what you wanted me to learn when you were pointing to the asyncserver example
|
[14:53] pieterh
|
xbmc_fan: ah, that's an example showing how to distribute work from one thread to others over inproc
|
[14:54] pieterh
|
to be honest I'
|
[14:54] pieterh
|
I've no idea how the C# code works
|
[14:55] xbmc_fan
|
oh, ok.
|
[14:55] pieterh
|
rgl: OK, I've updated rfc:13
|
[15:37] Seta00
|
Guthur, I'm looking at the asyncserver example, and I must say clrzmq's syntax at https://github.com/imatix/zguide/blob/master/examples/C%23/asyncsrv.cs#L33 is very nice, does clrzmq2 offer similar functionality?
|
[16:05] rgl
|
pieterh, nice. still, the second diagram octets 1-9 should read 1-8
|
[16:05] rgl
|
pieterh, and the other, octet 9 and octets 10+
|
[16:06] pieterh
|
rgl: thanks, fixed
|
[16:07] pieterh
|
ok, bbl
|
[16:07] rgl
|
bye
|
[17:52] Guthur
|
Seta00, sorry was away there, that is clrzmq2, hehe
|
[17:52] Guthur
|
I ported the example
|
[17:53] Guthur
|
I made it more idiomatic to the clrzmq2 binding
|