Saturday April 30, 2011

[Time] NameMessage
[04:53] CIA-75 libzmq: 03Martin Sustrik 07master * rfe2e772 10/ src/pgm_socket.cpp : pgm_socket uses binary version of UUID ...
[04:53] CIA-75 libzmq: 03Martin Sustrik 07master * reb9bc1b 10/ (src/dist.cpp src/dist.hpp): Message atomicity problem in PUB socket fixed. ...
[05:39] pieterh sustrik: issue 191 confirmed fixed on 2.1... :)
[05:58] sustrik great
[06:22] pieterh sustrik: IMO we need to become more pedantic about test cases for such things
[06:22] pieterh without a test case it'd have been impossible to get this working
[06:22] sustrik what do you propose
[06:22] sustrik ?
[06:22] pieterh well... we need somewhere to hold test cases
[06:23] pieterh issue tracker isn't great for this
[06:23] pieterh I'd prefer a git
[06:23] sustrik /tests?
[06:23] pieterh can't be in libzmq IMO, perhaps a separate git
[06:23] sustrik why so?
[06:23] sustrik if it's elsewhere it'll bit-rot
[06:24] pieterh they'll be historical anyhow, per issue
[06:24] pieterh now we have issue 199 with tests
[06:24] pieterh a. random contributors (usually person reporting the issue)
[06:24] pieterh b. random languages
[06:24] pieterh c. random versions of 0MQ (2.1, 3.0 at least)
[06:25] pieterh d. perhaps, testing other layers
[06:25] pieterh I don't see this as a regression test layer but just an aide for fixing issues as we go along
[06:26] pieterh see th's test cases for issue 199... random code on some random URL
[06:26] sustrik why do you need those to be persistent?
[06:26] pieterh i don't need persistence, I need consistency
[06:26] sustrik they'll be completely bit-rotten in 6 months
[06:26] pieterh that's fine
[06:27] pieterh I don't care about bitrot
[06:27] pieterh I care about 'where the heck are the test cases for issue 204 for 2.1?'
[06:27] pieterh searching email and irc and issue trackers is a waste of time
[06:28] pieterh also, not having a consistent approach means we often don't even have test cases
[06:28] pieterh clearly, without a test case for 191 we'd have allowed fairly major breakage
[06:28] sustrik hm, you have to look to issue tracker anyway, otherwise you won't have an idea what issue 204 is meant to be
[06:28] pieterh also, we need a way to share test code
[06:29] pieterh e.g. the gist repo I made for 191 but where anyone can write back to it
[06:29] pieterh so the issue tracker is fine for discussion
[06:29] pieterh e. multiple authors on repo
[06:32] sustrik feel free to try, but imo people will consider it too heavyweight and will use gist/bugtracker instead
[06:32] pieterh well, this would only work if we adopt it and enforce it
[06:33] pieterh people would like to just email the list saying "hey, 0mq crashes!"
[06:34] pieterh ok, I'll make an email to zeromq-dev, everyone will ignore it, then I'll start doing this anyhow, and after a month or two people will bitch at me for fragmenting the repos...
[06:34] pieterh sigh
[06:37] sustrik hm
[06:37] sustrik so once again, what's the rationale?
[06:37] pieterh test-driven change, martin
[06:38] pieterh every bug has an issue, and test case, which is in C, and ported to 2.1 and 3.0
[06:38] pieterh every test case is in a 100% predictable location
[06:38] sustrik why not in bug tracker
[06:38] pieterh before a fix is applied to a master branch it can be 100% tested
[06:38] sustrik or a gist
[06:38] sustrik ?
[06:39] pieterh bug tracker has no attachments
[06:39] pieterh a gist has no name
[06:39] sustrik gist linked form bug tracker?
[06:39] pieterh you provided two C test cases
[06:39] pieterh they had bugs
[06:39] pieterh did not run on 2.1
[06:39] pieterh if I fix this, where do I put the improved versions?
[06:39] pieterh where do I put listings, stack dumps, etc.?
[06:40] pieterh how does someone download this bundle of stuff?
[06:40] pieterh click click click on dozens of links?
[06:40] pieterh srsly?
[06:40] pieterh gist linked from bug tracker... no predictable location
[06:41] pieterh dozens of messy gist repos in working directoruy
[06:41] pieterh no obvious naming of cloned repos
[06:44] sustrik well, i am not feeling the pain yet
[06:44] sustrik clicking on link works ok for me
[06:45] th did i do that wrong?
[06:45] pieterh th: no, you did exactly what we asked but we're asking for the wrong thing afaics
[06:45] th i thought it would be easiest to put all information in one place and link it
[06:45] th ahh i see - infrastructure discussion
[06:45] pieterh yes
[06:45] th i just noted github being slow ;)
[06:46] th did not grok all the gist yet
[06:46] pieterh sustrik: you don't feel the pain because you avoid a lot of the process
[06:46] pieterh that however just means the way you work can't scale
[06:46] pieterh e.g. we can't work like that and maintain stable versions
[06:46] sustrik ok, give it a try and we'll see how it works
[06:47] pieterh I'll make a demo and an email explaining it
[06:47] th pieterh: i replied to what i get via email from you via github, but thats not supposed to show up as comment?
[06:47] sustrik yes
[06:47] pieterh th: it should appear as a comment, yes
[06:48] th at least it did not (yet)
[06:49] th anyways - added it via the web
[06:49] th pieterh: the issue only happens under some load... so while ./stress;do;done
[06:50] pieterh th: we definitely need more infrastructure for issue tests
[06:50] th and i also tried to port this to 3.0, but i get EAGAIN on send although i dont use NOBLOCK
[06:50] pieterh hmm
[06:50] th pieterh: the only change i did to "port" was to use zmq_{send,recv}msg instead
[06:51] pieterh th: should be sufficient, you don't use a lot of the API
[06:51] sustrik th: note the change in the return value
[06:51] sustrik in 2.0 send/recv returned 0/-1
[06:51] sustrik in 3.0 it returns message size
[06:52] th dang
[06:52] th if (zmq_sendmsg()) { ... error ... }
[06:52] th ok - i'm on it
[06:53] th there is no manpage installed of zmq_sendmsg with libzmq (there is zmq_send.3 though)
[06:54] pieterh th: also, don't use asserts like that, if you compile in release mode the whole assert is removed
[06:54] sustrik th: yes, it's still missing
[06:54] th pieterh: yea - but this test case will never be compiled in release mode, right?
[06:54] pieterh th: sure
[06:54] sustrik should be: if (zmq_sendmsg() < 0) error;
[06:54] th sustrik: ok
[06:55] th sustrik: same for recv?
[06:56] sustrik yes
[06:56] th pieterh: i'll add some `assert(test=1);if (test!=1) exit(1);`
[07:00] pieterh th: can you clone
[07:00] pieterh and then add your changes, and then make a pull request?
[07:00] th Unpacking objects: 100% (23/23), done.
[07:00] pieterh I've already create a space for issue 199 and added your current C test cases
[07:00] pieterh you can add everything there
[07:00] th pieterh: excluding any output.{txt,html} and Makefiles?
[07:01] pieterh anything other people need to run the tests, I'd say
[07:01] pieterh This is an experiment to see if we can use a git repo for communication instead of pastebins
[07:02] th pieterh: i'll do that.
[07:02] pieterh tgh: much appreciated
[07:02] th but first i'll fix this up for 3.0
[07:02] th sustrik: any change on return value of [gs]etsockopt?
[07:02] pieterh no hurry
[07:02] pieterh th: nope, but most arguments changed
[07:03] pieterh e.g. from uint64_t to int
[07:03] pieterh so if you use the old types you'll get an EINVAL error
[07:03] th well that direction (making it smaller) does not hurt in my case where i use ZMQ_XXXMORE
[07:04] th pieterh: i dont use type - i pass a pointer and a pointer to a size, right?
[07:04] pieterh th: in C?
[07:04] th ZMQ_EXPORT int zmq_getsockopt (void *s, int option, void *optval, size_t *optvallen);
[07:04] th wait you're talking about `int option`??
[07:05] pieterh see
[07:05] th and not the value?
[07:05] th pieterh: i see that... zmq_getsockopt (socket, ZMQ_RCVMORE, &rcvmore, &type_size);
[07:06] th int64_t
[07:06] pieterh rcvmore changed from int64_t to int
[07:06] pieterh see line 761
[07:06] th what about line 403?
[07:06] pieterh the zsockopt class (czmq) has both 2.1 and 3.0 support (two large conditional blocks)
[07:07] pieterh line 403 is 2.1, line 761 is 3.0...
[07:07] th ahhhhh
[07:07] th ok
[07:08] th pieterh: so i get EINVAL if last argument's value is 8 instead of 4?
[07:08] pieterh yes
[07:08] th although - only the pointer to type_size is passed.
[07:09] th but thats always of size_t?
[07:09] pieterh yes
[07:09] th ok
[07:09] th why a pointer at all? would i get the amount of bytes needed if i pass to small buffer?
[07:09] th s/to small/too small/
[07:09] pieterh th: this is the standard POSIX API for get/setsockopt
[07:12] th `and modified on return to indicate the actual size of the value returned.`
[07:12] th okay
[07:13] th will a recv in 3.0 ever return 0 to indicate EOF or something like this?
[07:14] th pieterh: ok - same effect with 3.0
[07:14] pieterh well, hang on, there's no EOF in 0MQ
[07:15] pieterh :)
[07:15] th pieterh: thats why i said "or something like this?" ;)
[07:16] th i wonder if this issue would still be an issue if i would not be on the VSM side...
[07:18] pieterh th: I don't get the crash on my box
[07:18] pieterh what OS are you using?
[07:19] pieterh I'll let it run for a while...
[07:21] th pieterh: you also dont get the "STUPID RECV (JUST FOR SHOWING WHATS WAITING)"?
[07:22] th pieterh: i replaced some assert(0) with an endless loop of juse receiving and printing, to see the order of messages
[07:22] th pieterh: Ubuntu 10.04.2 LTS on 64bit
[07:23] th pieterh: i run it |& tee log and cancel the client-while-loop when i see the "STUPID RECV"
[07:47] rgl pieterh, I loved the intro on thx for putting it up!
[07:51] pieterh rgl: glad you liked it, some people seem to hate it...
[07:51] pieterh th: I eventually got a crash in the server: Assertion failed: fetched (xrep.cpp:248)
[07:52] th pieterh: i assume - that is just because you ran out of resources
[07:52] rgl pieterh, oh.. oh why?
[07:52] th pieterh: it fails for me within the first 3 seconds
[07:53] Toba pieterh: too bad sockets in pollitem are void*, makes it easy to mess that up
[07:53] th pieterh: although as said above - i dont let it crash when fail happens
[07:53] pieterh rgl: no idea...
[07:53] th pieterh: to document the order of the rest of the parts
[07:54] pieterh th: I think there's a bug in xrep, not sure...
[07:55] th pieterh: so did you get the "STUPID RECV..." messages as well
[07:55] th ?
[07:55] pieterh sustrik: is it normal to do msg_->close () and then access msg_->flags ()?
[07:55] pieterh th: sure, loads of them
[07:55] th pieterh: then it was intermixing
[07:56] pieterh th: is that good or bad? I've no idea what I'm supposed to be seeing...
[07:56] th pieterh: it means that atomicity of multiparts was violated.
[07:56] pieterh it'd be useful to be explicit in any output you make, e.g. "YOU SHOULD NOT SEE THIS"
[07:57] th pieterh: ok - see the line above the first "STUPID"
[07:57] th pieterh: i'll make it clearer
[07:57] pieterh also, it should simply stop at the first error
[07:57] th pieterh: but i wanted to print the "in queue" messages as well to show if there was a duplicate
[07:57] pieterh so when I run it, I know immediately if it fails
[08:44] th pieterh: shall i use markdown in the README?
[08:44] pieterh th: as you like
[08:44] th then plain txt
[08:44] pieterh indeed...
[09:04] th pieterh: git pull master
[09:05] pieterh th: cool, it works nicely
[09:05] pieterh merged into
[09:05] th pieterh: because i just rebased, after your last 2 commits ;)
[09:05] pieterh does this seem workable for making / sharing test cases?
[09:05] th pieterh: for me - yes.
[09:06] th pieterh: not sure about the common php tinker
[09:06] pieterh what is that?
[09:06] pieterh ah, you mean for average folk
[09:06] th sort of
[09:07] pieterh well, in any case we need to be able to make C test cases for real work
[09:07] pieterh I'm usually able to make that if someone has a test case in another language
[09:08] pieterh let me try to test 199 again now that I know what to look for... :)
[09:08] th pieterh: in my case c, c++, 2-1, 3-0 - it's all the same, yes
[09:08] pieterh yup
[09:08] th although it's not as broken down, as i had liked.
[09:10] pieterh th: so I get the 'STUPID' stuff in the server but it doesn't assert
[09:10] th pieterh: there should be no more "STUPID" output
[09:10] th pieterh: recompile!
[09:10] th i changed the code
[09:10] pieterh hmm
[09:10] th 199/3-0/
[09:11] th grep for STUPID
[09:11] th no stupidity in there
[09:11] pieterh th: ah, I'm using 2-1
[09:11] th pieterh: ignore that - let's care about backporting after it's fixed upstream
[09:11] pieterh well, I need the testcase on 2-1 in any case, otherwise I can't test downstreamed patches
[09:12] pieterh I'll port the test cases back to 2-1, hang on...
[09:12] rgl how does one authenticate the client of a connection like traditional sockets? we have to use some kind of message authentication?
[09:12] th pieterh: but could we do that after 3-0?
[09:12] pieterh rgl: it depends on the pattern you're using
[09:12] pieterh there's no authentication as such in 0MQ
[09:13] pieterh th: in fact we could make portable test cases if we used czmq
[09:13] th pieterh: it's a s/0 <= zmq_/0 == zmq_/ for the asserts and then the s/msg// for the name-change
[09:13] pieterh rgl: for an example of secure pubsub, see the Salt project (at
[09:13] rgl pieterh, for instance, pub/sub, where not everyone can do pub
[09:13] th rgl: how about applying asymmetric cryptography?
[09:13] pieterh :)
[09:14] rgl oh :)
[09:14] rgl will look at it. thx :)
[09:19] pieterh th: ok, failure reproduced on 2.1... excellent
[09:19] pieterh we can pass this to sustrik now, get a patch and downstream that to 2.1
[09:20] th ;-)
[09:20] th pieterh: and reproduced on 3-0 as well, i suppose?
[09:21] pieterh th: hang on... (I don't really care about 3-0 so much in this process)
[09:22] pieterh yes, it hits the same way on 3-0
[09:22] pieterh th: so I sent an email to zeromq-dev proposing we hold test cases in a single repo, you may want to comment on that thread
[09:23] th pieterh: ok - and you need to comment on your last mail in the #199 thread
[09:27] th pieterh: commented
[09:30] pieterh I've replied to the 199 thread now
[09:46] rgl can you point me to the document that describes the wire format of the several socket types?
[09:46] pieterh rgl: unfortunately, there's no single document
[09:47] pieterh there is some information in the zmq_tcp man page
[09:48] pieterh we really do need to make a proper documentation of the wire format, no real excuses
[09:49] rgl indeed :)
[09:49] rgl even after writting this "But really, Zookeeper should be using a generic messaging layer and an explicitly documented wire level protocol" hehe
[09:50] pieterh yes, absolutely true
[09:51] pieterh I did start on a wlp at
[09:51] pieterh but I need to convince the core developers to actually commit to such a contract
[09:51] rgl the wire protcol described at is not the whole story?
[09:51] pieterh otherwise it's just documenting code behaviour, which isn't acceptable IMO
[09:52] pieterh there is a bunch of stuff missing from the zmq-tcp page
[09:52] pieterh sockets exchange identities when they connect
[09:52] pieterh it's on some tutorial somewhere but I don't remember which one
[09:52] pieterh don't get me started on the failure of this project to write decent specs in advance... :-)
[09:53] rgl I think I did heheh
[09:53] pieterh it's one of the topics we hope to nail down at the Brussels event later this month
[09:54] pieterh but basically until the development process uses contracts properly, it won't happen IMO
[09:54] rgl what do you mean by "uses contracts"?
[09:54] pieterh otherwise it's one set of people changing code, a second set trying to document that
[09:54] pieterh contracts = upfront discussion & documentation of APIs and WLPs
[09:55] pieterh with running code to demo, as needed
[09:55] rgl WLP?
[09:55] pieterh but not after-the-fact documentation of code
[09:55] pieterh wire level protocols
[09:56] th pieterh: shouldn't the 3rd occurance of "Message size" be actually "Message body"?
[09:56] rgl I see. it makes sense :)
[09:56] pieterh th: the text was copied as-is from the tcp man page
[09:56] pieterh it is message size, afaics
[09:56] pieterh it's a 64-bit size,
[09:57] th then i totally dont grok that sketch
[09:57] pieterh th: the sketch is pretty nasty
[09:57] pieterh what it says is, either a 1-byte length, or 0xff followed by an 8-byte length
[09:58] th pieterh: the upper part is the < 255octet case, right?
[09:58] pieterh yes
[09:58] th pieterh: yes- good. so i got the 0xff+message size. but why is it followed by another message size?
[09:58] pieterh th: it's not, that last byte is the 8th byte
[09:59] pieterh the diagram is lousy
[09:59] pieterh god, let me fix it...
[09:59] th ahhh
[09:59] th the columns are for 32 bit
[09:59] th so it is three rows for the 64bit size, because the first octet is for the 0xff
[10:00] th gosh - got it now :)
[10:00] th perhaps you should add a "(cont.)" after the continuations of the "message size"
[10:00] th which would not fit in the 8 bit field ;)
[10:02] pieterh th: check it now
[10:02] pieterh this is still not how I'd draw it, but whatever
[10:04] rgl it would be nice to have some wireshark inspectors on the zeromq sockets, that is, somehow make wireshark interpret the data inside a 0mq socket.
[10:04] pieterh rgl: there's a whole bunch of nice things we could make if we had decent WLP specs
[10:05] pieterh there's about one request for these specs every two weeks
[10:10] th pieterh: at least it got less confusing in my opinion ;)
[10:10] pieterh I'm going to start on a new RFC, this annoys the heck out of me
[10:10] pieterh at least we can document the 2.1 format properly
[10:10] pieterh and then bug sustrik to do the same for 3.0
[10:40] mikko pieterh: hi
[10:40] pieterh hi mikko
[10:40] pieterh how's up?
[10:40] pieterh what's life?
[10:40] mikko it's something that happens when you don't work?
[10:41] pieterh dunno... sounds weird to me
[10:41] mikko anyway, you can use gists for simple test-cases as well
[10:41] pieterh yes, I tried that, but
[10:41] mikko they are stored as git repositories and can be forked, cloned etc
[10:41] pieterh a. you get weird names in your directories
[10:42] pieterh b. you can never remember what the gist is called so need to keep referring back to some random URI somewhere
[10:42] pieterh c. you can't have multiple people working on the same gist
[10:43] pieterh d. it's very hard to enforce consistency when every test case is a separate repo
[10:43] pieterh e. it's annoying when I have 20 test cases to have 20 random hash-encoded directories to work with
[10:44] pieterh f. /me can go on all day
[10:51] pieterh mikko: sorry for ranting :)
[10:53] mikko i'm getting worried by the amount of repos :)
[10:53] mikko ever growing
[10:54] pieterh mikko: we had a similar issue at wikidot
[10:54] pieterh originally, all the official stuff was on one site, maintained by a couple of people
[10:55] pieterh I slowly broke that into 20-30 sites, each with a distinct group of contributors
[10:55] pieterh it was quite interesting, the tension that caused
[10:55] pieterh OTOH people complained they couldn't navigate the results
[10:55] pieterh OTOH, the number of contributors overall increased by 50x or more
[10:56] pieterh since every gist is a repo, we already have massive numbers of repos, except they're unknown
[10:57] pieterh anyhow, I don't see how is any different from subdirectories within one repo
[11:00] mikko i think i was thinking a workflow like:
[11:01] mikko each bigger issue gets branched off to a separate branch, the test cases are added to branch and when everything works it gets merged back
[11:01] mikko so that also test cases get back to main repo
[11:02] pieterh mikko: that would work nicely except we have to work across multiple topic branches
[11:02] pieterh multiple main repos
[11:03] pieterh how do we handle an issue that affects both pyzmq and libzmq?
[11:03] pieterh or 3.0 and 2.1
[11:03] pieterh or erlzmq2 and erlzmq3
[11:03] pieterh etc.
[11:03] mikko when i've had such issues with php-zmq i've created a reproducible test-case in C or C++
[11:04] pieterh at the least, we need a way to maintain C/C++ test cases to work on both 2.1 and 3.0
[11:04] pieterh can't use a topic branch in 3.0 for that
[11:04] pieterh wouldn't work if 2.1 and 3.0 were branches in same repo, either
[11:05] pieterh it'd require one branch for each test case / version combination
[11:05] pieterh ugh
[11:05] pieterh also, as sustrik pointed out, we don't want test cases to go into the main repo because they rot really fast
[11:06] pieterh unless someone's actually maintaining them, which is a role no-one has volunteered for...
[11:07] pieterh ok, one of my hopes is that a separate issues repo can become the basis for an independent regression test suite
[11:11] mikko what do you mean they rot really fast?
[11:12] mikko we've been using the same set of tests through out 2.x
[11:16] pieterh mikko: old test cases don't keep working unless they're maintained
[11:16] pieterh I know they *should* keep working, for regression testing, but that demands effort
[11:16] mikko does that mean that old software doesnt keep working unless maintained all the time?
[11:17] pieterh indeed
[11:17] pieterh since today, test cases are transient and discarded, we don't see the cost
[11:17] pieterh e.g. there are test cases that apply to 2.0.4
[11:18] pieterh actually I'd be happy to maintain the regression tests, if the tests were all in one place and properly structured
[11:18] pieterh e.g. always started with a consistent command
[11:18] pieterh built using a consistent tooling
[11:18] pieterh etc.
[11:18] pieterh it is doable (and still, a lot of work) if they're in their own repo IMO
[12:17] pieterh ok, finished draft of full wire level protocol
[12:18] pieterh rgl: if you're still around:
[13:03] xbmc_fan hi
[13:04] xbmc_fan any experts for the .net 0mq-binding available?
[13:07] iFire anyone made a pdf of zguide?
[13:09] iFire "only" a 105 pages
[13:15] mikko xbmc_fan: Guthur is i think
[13:17] mikko iFire: seems to print to pdf well
[13:19] mikko
[13:19] mikko that's what i get out from print to pdf
[13:20] xbmc_fan thanks mikko.. Maybe Guthur, you are able to answer this Stackoverflow question: ?
[13:22] mikko xbmc_fan: you cannot use sockets concurrently
[13:22] mikko xbmc_fan: but with zeromq 2.1 you can migrate socket from one thread to another
[13:23] xbmc_fan mikko, I understood that I must not use sockets from different threads
[13:23] mikko that was the case with 2.0
[13:23] mikko you cannot use a socket concurrently from multiple threads
[13:24] mikko you have to ensure a full memory barrier before accessing from another thread
[13:24] xbmc_fan i got that. In order to not use it concurrently, I have to use semapohores, locks- which the guide explicitly warns about
[13:25] mikko what does it say?
[13:25] xbmc_fan please read the question I aksed at stackoverflow
[13:25] Guthur xbmc_fan, just don't share them across threads would be the easy option
[13:25] mikko xbmc_fan: can you have socket per thread?
[13:26] mikko that is usually ideal if multiple threads need to communicate with same endpoint
[13:26] mikko mutexes / locks don't necessarily make your application slow and fragile. they are just very hard to get right
[13:26] iFire mikko just finished printing
[13:26] Guthur sometimes what I do is create the socket in one thread and give it to another to use within it's lifetime
[13:26] iFire ha 6 minutes
[13:27] xbmc_fan mikko, I could but is this really performant / the way to do things properly? e.g. I might get called from a different temporary-threadpool thread
[13:27] xbmc_fan and I *think* there is overhead involved to create a 0mq-socket each time my method gets called
[13:27] mikko xbmc_fan: is there going to be lock contention?
[13:28] mikko xbmc_fan: why do you have multiple threads handling events?
[13:28] mikko not sure if that is common in C#
[13:29] xbmc_fan mikko, it is quite common: the class registering for an event does not know which thread invokes the eventhandler
[13:30] xbmc_fan I have multiple threads throwing events. and the eventhandler gets called from those threads.
[13:31] Guthur I did create a socket pool once, never really used it though
[13:31] Guthur I was considering including it in clrzmq2, but I don't think I did in the end
[13:34] xbmc_fan <-- maybe this clearifies my problem
[13:35] Guthur xbmc_fan, if you are worried about the setup cost of sockets you could try the pool approach
[13:38] Guthur or I suppose just lock the socket
[13:39] xbmc_fan so I would have a number of sockets in a pool. But since every socket has to be used from the same thread it was created on, this won't help much, or do I miss something? e.g.: If the eventhandler is invoked from temporary threads, I would create a socket for those calls because there isn't a socket for that thread in the pool yet. I can see that if there is a limited amount of (long running) threads calling, pooling of sockets might ma
[13:39] xbmc_fan But I think in my scenario, in the end it would leak ressources - so one had to implement something like "pool timeouts"... too much afford isn't it?!
[13:39] Guthur I have the socket pool code if you want me to post it
[13:40] xbmc_fan sure, maybe that helps
[13:40] Guthur xbmc_fan, in the new version of 0MQ it can be used from a different thread
[13:40] Guthur just not from 2 threads at the same time
[13:40] Guthur mikko, correct?
[13:40] xbmc_fan Guthur or I suppose just lock the socket <-- I could do that, even though the guide says that evil.
[13:40] xbmc_fan Guthur just not from 2 threads at the same time <-- ok, that would be nice.
[13:41] Guthur xbmc_fan, yeah, pool would be cooler than lock, hehe
[13:42] rgl pieterh, oh, you actually did it :)
[13:42] Guthur xbmc_fan,
[13:42] pieterh rgl: yeah, it did take a whole two hours...
[13:43] Guthur that is very much as is, no warranty or anything
[13:43] Guthur I can't remember if I even completed it, but might serve as some inspiration
[13:43] rgl pieterh, humm "The principal issue with ZMTP/1.0 is the lack of content types in the wire format" bummer. so there is no easy way to create a wireshark interpreter.
[13:43] pieterh rgl: you can make it but it'll only work at the framing level
[13:45] rgl pieterh, I see. I'll have to read it with more attention :)
[13:49] xbmc_fan thanks Guthur. Well, the collection and handing out of sockets seems to be threadsafe, but the the pool might run empty eventually. Nevertheless, thanks for posting it!
[13:51] xbmc_fan Guthur sometimes what I do is create the socket in one thread and give it to another to use within it's lifetime <-- and that does work? So the author of the docs had .net in mind when he/she wrote: "The only place where it's remotely sane to share sockets between threads are in language bindings that need to do magic like garbage collection on sockets." ?!
[13:53] Guthur xbmc_fan, I think it's that some people might abuse the fact that it can be moved from one thread to another
[13:53] pieterh iFire: I've made a quick PDF conversion
[13:53] Guthur it still doesn't support shared access
[13:53] pieterh am uploading it now, 30 seconds...
[13:54] Guthur the .net binding does nothing extra to facilitate moving the socket from one thread to another
[13:54] Guthur it came with one of the later version of libzmq
[13:55] xbmc_fan ok, thanks for the clarifcation guthur. maybe this should be stated also in the docs.
[13:57] pieterh xbmc_fan: the reason for this text is that people often tried to share sockets between threads, and it crashes libzmq horribly
[13:58] pieterh you can create a socket in thread A and pass it to thread B if, and only if, you do a full memory barrier
[13:59] xbmc_fan with "full memory barrier" you mean that I have to make sure that thread A doesn't access the socket at the same time as thread B?
[14:01] pieterh "full memory barrier" is a technical term meaning an instruction that causes all CPUs to synchronize and flush caches
[14:02] pieterh it's definitely not enough to "not use at the same time"
[14:10] xbmc_fan ok, I think I understood pieterh. Therefore it seems I have to use locks to synchronize acces to a single socket if I have no control about the caller's thread.
[14:10] pieterh something like that, I've never actually done that
[14:11] pieterh xbmc_fan: I'd probably look for an alternative design than that though...
[14:13] xbmc_fan pieterh: well, in my case, the architecture is inheritly multithreaded and thread-synchronization takes only place at the outer edges. Currently only for the GUI thread.
[14:13] xbmc_fan however, the gui dispatches information which is to be send in a worker thread.
[14:13] xbmc_fan I would like to send this data via 0mq.
[14:14] pieterh xbmc_fan: have you already tried making multithreaded apps that communicate over inproc sockets?
[14:14] pieterh if you are using locks (and IMO if you're sharing sockets between threads) then your design is bogus somewhere
[14:16] xbmc_fan pieterh: I haven't used "inproc sockets" for that yet. But I am using a kind of serice bus / event driven architecture. I am sure that - under the hood - this can be implemented using inproc sockets.
[14:16] pieterh xbmc_fan: when you discuss multithreading and 0MQ you have to know how inproc sockets work
[14:17] pieterh otherwise 100% certain you are looking at the wrong picture
[14:17] pieterh please look at the asynchronous server example
[14:27] xbmc_fan I think you mean that one: ?
[14:30] xbmc_fan If I understand both the example and your statement correctly, you wanted me to recognize that there is no locking/synchronization of threads involved when using inproc sockets
[14:34] xbmc_fan however, at the outer edge of that example, there is in fact synchronization, since the call to Console.Writelline featrues a [MethodImpl(MethodImplOptions.Synchronized)] attribute under the hood. That's the same in my case: the Gui has to synchronize threads to the gui layer. And I have to synchronize non-0mq callers (events, not inproc sockets) if I want to use 0mq at another outer edge of the architecture. From that edge onwards, I c
[14:34] xbmc_fan inproc sockets and only have to synchronize again if there is another "edge"
[14:36] xbmc_fan I also understood that I could have used 0MQ-Socket throughout the software and would only need to synchronize for the GUI, if I used inproc sockets.
[14:40] xbmc_fan am I missing something?
[14:46] rgl pieterh, "Each side of the connection consists of an greeting message followed by zero or more content messages. An identity message consists of one frame containing the peer's identity" humm what is an "identity" message? is the "greeting" message?
[14:46] pieterh greeting = anonymous / identity
[14:46] pieterh let me fix that text
[14:47] pieterh I've gotten other review comments as well, new version coming RSN
[14:47] xbmc_fan anonynous = %0x01 %x00 <-- isn't it anonyMous
[14:47] xbmc_fan "Full ZMTP Grammar"
[14:51] xbmc_fan pieterh , did I miss something or did I missinterpreted something?
[14:51] pieterh xbmc_fan: fixed that, thanks
[14:52] xbmc_fan ah, yeah, np. I mean a few sentences back.. I tried to understand what you wanted me to learn when you were pointing to the asyncserver example
[14:53] pieterh xbmc_fan: ah, that's an example showing how to distribute work from one thread to others over inproc
[14:54] pieterh to be honest I'
[14:54] pieterh I've no idea how the C# code works
[14:55] xbmc_fan oh, ok.
[14:55] pieterh rgl: OK, I've updated rfc:13
[15:37] Seta00 Guthur, I'm looking at the asyncserver example, and I must say clrzmq's syntax at is very nice, does clrzmq2 offer similar functionality?
[16:05] rgl pieterh, nice. still, the second diagram octets 1-9 should read 1-8
[16:05] rgl pieterh, and the other, octet 9 and octets 10+
[16:06] pieterh rgl: thanks, fixed
[16:07] pieterh ok, bbl
[16:07] rgl bye
[17:52] Guthur Seta00, sorry was away there, that is clrzmq2, hehe
[17:52] Guthur I ported the example
[17:53] Guthur I made it more idiomatic to the clrzmq2 binding