Thursday September 9, 2010

[Time] NameMessage
[18:30] pieterh PerfDave, sorry, i was kidding around
[18:31] pieterh sd88g93, ok, test case is now clear, lemme think about it a sec
[19:45] pieterh sd88g93, what is 'qskt' for?
[19:45] sd88g93 that's the rep socket, in the worker thread, that recieves the messages
[19:45] sd88g93 from the client
[19:45] pieterh there is already a terrible lack of vowels here in Slovakia
[19:46] sd88g93 tough times, gotta conservee
[19:46] pieterh that's kind of like "who is that guy?" answer: "two legs, body, arms, head"
[19:47] pieterh cause it does not in fact talk to the client at all
[19:47] sd88g93 couldnt come up with a more descriptive variable, sorry, i blame my lack of imagination
[19:47] sd88g93 it doesnt ? it does on mine
[19:47] pieterh it talks to the main thread
[19:47] pieterh via inproc
[19:47] sd88g93 yeah, then there's a queue device to the main socket
[19:47] sd88g93 the client skt
[19:48] pieterh why is the client not talking to the worker directly?
[19:48] sd88g93 if you have more than one thread
[19:48] sd88g93 use the device , at least that's the scenario
[19:48] pieterh well...
[19:49] pieterh it
[19:49] pieterh the device is really when you have an N-to-N case
[19:49] pieterh and don't want all the N's to have to know about each other
[19:49] sd88g93 yeah, there's just one client now, but there could be more
[19:49] pieterh ah... you are in fact developing the MT server model...
[19:49] sd88g93 yeah
[19:49] pieterh i hardly recognized it... :-/
[19:49] sd88g93 there could be more than one client
[19:50] sd88g93 well, a diffferent form of it, i guess
[19:53] sd88g93 zhelpers.h:25 i thought sprintf was "evil", suppose to use snprintf ?
[19:54] keffo pieterh, I've done the lru now!
[19:54] pieterh keffo: wow!
[19:54] pieterh sd88g93, this is what the worker loop becomes:
[19:54] pieterh char *command = s_recv (control);
[19:54] pieterh char *count = s_recv (control);
[19:54] pieterh puts ("Worker got command: %s\n", command);
[19:54] pieterh s_send (publisher, "msg.x|");
[19:54] pieterh s_sendmore (publisher, command);
[19:54] pieterh s_send (publisher, count);
[19:54] pieterh free (command);
[19:54] pieterh free (count);
[19:54] pieterh s_send (control, "OK");
[19:54] pieterh there is a bug, i see immediately...
[19:55] pieterh in my code, not yours
[19:55] ModusPwnens Wait a sec pieter, i was referring to something on the site
[19:55] sd88g93 oh, in s_send and s_sendmore you do init the message and close it each time,
[19:56] sd88g93 yeah, that's what i thought, you were just referrring to coding style, eh ? not actuall instructions needed
[19:56] sd88g93 i get ya now, ok
[19:56] pieterh ModusPwnens, even better, join the site and click 'edit'
[19:56] pieterh it's a wiki :-)
[19:57] ModusPwnens Oh okay, well that's easy to fix then
[19:57] keffo pieterh, it seems to be working very well.. I just need to post back the results now, but I'll probably use a new socketlayer for that, with some persistance as well
[19:57] pieterh hmm, nice
[19:57] pieterh keffo: i still find working with envelopes to be too painful
[19:57] pieterh but we'll make abstractions for those later
[19:58] keffo I have a Route class for that
[19:59] pieterh yeah, right, you told me... nice
[19:59] keffo which can construct itself from incoming, and also send it
[19:59] keffo yeah
[19:59] pieterh were the names in the routing section too weird?
[19:59] pieterh i was trying to get mnemonics that worked
[19:59] keffo where?
[19:59] pieterh ch3
[19:59] keffo mama,papa? =)
[19:59] pieterh yeah
[19:59] jhawk28 the mama and papa were hilarious
[19:59] keffo papa was a bit confusing, didnt really see the application to be honest, but I was mostly interested in the mama anyway :)
[20:00] pieterh well, good, it's hard to make this material even remotely funny
[20:00] jhawk28 we enjoyed it at work
[20:00] psino pieterh: you should move some more reasoning about the naming in ch3 to the place where you introduce them
[20:00] keffo pieterh, Another interesting sideffect of the mama topology you presented there is that there is no longer any immediate necessity for a heartbeat system
[20:00] pieterh psino: did I get the order wrong?
[20:01] pieterh yup, but you need heartbeats for another reason
[20:01] pieterh i was going to write that today but got distracted by our friend sd88g93's curious case here
[20:01] pieterh also, ISP guys getting lost
[20:01] keffo sure, but not in the same way as a topdown design
[20:02] psino no, but when you say that REQ is a MAMA socket, making insistent demands, you might want to add something more that helps the reader remember the terminology
[20:02] pieterh hmm, lemme think about that for a second...
[20:02] pieterh i wrote that text very late on tuesday evening after a glass or two of wine... :-/
[20:03] psino you describe the usage of mama pretty good in the "Router-to-Mama"-section
[20:03] psino (personal opinion of course)
[20:03] pieterh i think the character is based on my friend Fabio' Italian mother
[20:03] pieterh ok...
[20:03] pieterh opinion is better than silence
[20:04] jhawk28 I was going through the swap and hwm parts of the user guide and I got an exception after the pub ran for a while (from yquery:108)
[20:05] pieterh jhawk28, interesting!
[20:05] jhawk28 It died when I was using the connect, but not the bind
[20:06] jhawk28 not sure if it matters, but I was using the Java bindings
[20:06] pieterh do you mean yqueue?
[20:06] jhawk28 yes, it was an assertion that failed
[20:06] jhawk28 going from memory - so it could be wrong
[20:06] pieterh could you please log this on the issue tracker, along with the hardware setup you're using?
[20:07] jhawk28 I think so
[20:07] pieterh this should absolutely never never happen
[20:07] jhawk28 figured as much
[20:07] pieterh there is indeed an assertion on yqueue.hpp:108
[20:07] pieterh woa, hang on
[20:07] pieterh it's an out of memory condition
[20:08] jhawk28 hwm was set to 1 and swap was set to 25000000
[20:09] pieterh you sure you didn't reverse those? :-)
[20:09] pieterh it's worth logging, we'll run that on some big boxes here and see if we can reproduce it
[20:09] jhawk28 :) was following the guide and then started playing around seeing what it would do
[20:09] pieterh thanks
[20:15] daggertip so
[20:23] pieterh sd88g93, you still there? I award you the "obfuscated 0MQ contest" award :-)
[20:24] pieterh you have six sockets and two devices in your minimal test case
[20:46] sd88g93 i'm always up for an award
[20:47] sd88g93 think sustrik can have a looksie ?
[20:54] pieterh sd88g93, lol, are you trolling me?
[20:54] sd88g93 trolling ?
[20:54] pieterh i've stripped it right down and removing the queue device lets it work
[20:54] pieterh :-)
[20:54] pieterh gnash, i will find this... gnash
[20:55] sd88g93 how do you read the messages into the thread ?
[20:55] pieterh well, connect worker directly to client
[20:55] pieterh next time, develop such code step by step so you can test it gradually
[20:56] pieterh and when you add a new piece if it does not work you know exactly what is questionable
[20:56] sd88g93 i'm suspicious that the messages are being altered in some way, like martin is slipping coded messages in there
[20:56] sd88g93 yeah, that's what i did , but ....
[20:56] pieterh i doubt it
[20:56] sd88g93 I realized zmq can do more
[20:57] sd88g93 I had a little different setup before, and then i put the forwarder back in , because i really didnt want to move zmq_msg_t 's around between threads
[20:57] pieterh I've cleaned up (rewritten :-() the server now:
[20:57] pieterh that is a messed up version but it shows you what it looks like
[20:57] pieterh see how the queue thread creates its own sockets
[20:58] pieterh not necessary to do that in main, kind of pointless and... in fact... illegal
[20:58] pieterh you cannot create socket in thread A and use it in thread B
[21:05] pieterh ok, sd88g93 that's the problem
[21:05] sd88g93 oh really
[21:06] pieterh dude, you owe me a case of beer
[21:06] pieterh corrected code is at
[21:06] sd88g93 yes yes, i do hahaa
[21:06] pieterh the publisher socket is not there, you can put it back of course
[21:07] pieterh so problem was that your queue thread wasn't working reliably
[21:07] pieterh when you test code, don't run it once, run it 10-100 times
[21:07] sd88g93 so you just create the skts for the device in that thread then,
[21:07] pieterh threading issues are nasty like that
[21:07] pieterh pls dnt sy skts t mks m brn hrt
[21:08] pieterh socks if you really have to save on letters
[21:08] pieterh but yes, you do
[21:08] sd88g93 what's wrong with skts ?
[21:08] pieterh what's wrong with the letters 'o' and 'e'?
[21:08] pieterh :-)
[21:08] pieterh seriously, if you remove essential letters from words you create extra parsing costs for the reader
[21:08] sd88g93 sorry, that's just how i do variables, i tend to leave out vowels
[21:09] pieterh english may look redundant but it's been elegantly evolved for lazy readers
[21:09] pieterh if you want a hint for style: make your code readable
[21:09] pieterh please compare my versions with yours
[21:09] sd88g93 those zhelper.h are helpful, but unfortunately, i need it for binary data
[21:10] pieterh you seriously cost me several hours just to untangle your code...
[21:10] pieterh zhelpers.h is an example, modify/extend it for binary data
[21:10] pieterh s_recv already works for binary data
[21:10] sd88g93 well, i was hoping for zero copy as well
[21:10] pieterh add s_send_bin (socket, data, size) and it's done
[21:10] sd88g93 ok,
[21:10] pieterh just make sure to copy the data
[21:11] pieterh otherwise you'll get weirdness too
[21:11] pieterh unless you are doing a lot of traffic (100K+ / second) or very large messages, zerocopy won't be an advantage
[21:11] pieterh but it will make your code harder to write and read and fix
[21:11] sd88g93 well, there's no way to do zero copy though, when i can find a way to do zerocopy and those functions , then we've got something
[21:12] pieterh do not optimize code that isn't working
[21:12] pieterh it is really a bad idea
[21:12] pieterh actually I need to totally remove all reference to zerocopy from the user guide
[21:12] sd88g93 well, one message part can be pretty big
[21:12] pieterh people should not even know it exists until chapter 4 or so...
[21:12] pieterh doesn't matter
[21:12] sd88g93 like 100k
[21:13] pieterh doesn't matter
[21:13] sd88g93 100KB , easily
[21:13] pieterh seriously, do the test yourself
[21:13] sd88g93 really ? it wont matter ?
[21:13] pieterh for (namewithvowels = 0; nwv < 1000; nwv++) { do large copy here }
[21:13] sd88g93 hmm ok , i'll have to think on that some
[21:13] pieterh run, test, prove it to yourself
[21:13] pieterh it matters if you are into microsecond latency issues and CPUs are running at 75% full
[21:14] pieterh i'll bet you that case of beer x 2 that's not your case
[21:14] pieterh premature optimization is one of the many roads to evil and it's not even a particularly fun one
[21:15] gebi zero-copy can also easily be more expensive than plain copy if not aligned on cache-lines and between different cpus
[21:18] pieterh ok, folk, /me has to go sleepy byes