Tuesday November 30, 2010

[Time] NameMessage
[02:46] xunker the page recommended I ask questions where if they aren't made clear in the docs, I've got one. I can't find somewhere where it says, yes or no, if you can use topics with something other than PUB/SUB sockets. Can I use 'topics' with say, PUSH/PULL socket pairs?
[15:15] mikko mato: are you applying against master?
[15:15] mikko mato: it complains about whitespace but applies here
[15:15] mato mikko: yup, master
[15:16] mato mikko: hmm, just looking at my git log
[15:16] mikko mato: because i took a fresh git clone
[15:16] mikko and it applies
[15:16] mato mikko: looks like my master branch may have diverged, damn....
[15:16] mato let me try and fix this...
[15:17] mikko
[15:17] mikko there it is with whitespace cleaned
[15:17] mikko it's a bit heavy to send to mailing-list (>40k patch)
[15:18] mato mikko: ok, it was some divergence on my master branch
[15:18] mato mikko: sorry for the confusion...
[15:18] mato am going to test it now
[15:18] mikko no problem
[15:18] mato by the way, we can remove the check for pkg-config in too I guess
[15:19] mato all the pkg-config stuff was there due to the OpenPGM 2.x build depending on it
[15:19] mikko ok
[15:34] mato mikko: you wouldn't happen to have an oldish RHEL/CentOS installation lying around?
[15:35] mikko i got centos 5.5 here
[15:35] mato mikko: say some 5.x or 4.x? I'd like to see if this last set of changes breaks/fixes anything
[15:36] mikko 32 or 64 ?
[15:36] mato can you try a random out-of-the-box build on it just to sanity check it works?
[15:36] mikko or makes no difference
[15:36] mato no difference
[15:36] mikko gimme a min
[15:36] mikko need to make dist on a newer machine
[15:36] mato oh, and try --with-pgm as well if you can
[15:36] mato if all that works then i think we're good to go
[15:42] mato mikko: This isn't right:
[15:42] mato checking for asciidoc... no
[15:42] mato checking for xmlto... no
[15:42] mato checking whether to build documentation... yes
[15:42] mato this is on some FreeBSD box w/o asciidoc on it, building from a "make dist" with your patch
[15:43] mato obviously fails later on because it tries to use asciidoc
[15:44] mato oh, typo in your patch
[15:45] mato fixing....
[15:48] mikko builds on 5.5 with and without pgm
[15:48] mato good
[15:49] mikko yes
[15:50] mikko + if test "x$ac_zmq_cv_have_asciidoc" = "xno" -o "x$ac_zmq_cv_have_xmlto" = "xno"; then
[15:50] mato yup
[15:50] mikko cv is too much
[15:50] mato by the way, which variables are supposed to have _cv_ and which arent?
[15:50] mato
[15:50] mato this should make it work, am going to test
[15:52] mikko cv are ones that are cached
[15:53] mikko i named most of the publicly visible ones "_cv_ because it's easier to add caching to them later
[15:53] mikko cv = cache value
[15:53] mato right
[15:53] mato checking whether C compiler supports -fvisibility=hidden... no
[15:53] mato checking whether C compiler supports "-xldscope=hidden"... no
[15:53] mato checking whether C++ compiler supports -fvisibility=hidden... no
[15:53] mato checking whether C++ compiler supports "-xldscope=hidden"... no
[15:53] mato checking whether C++ compiler supports -mcpu=v9... yes
[15:54] mato are those quotation marks supposed to be there?
[15:54] mato (this is GCC 3.4.3 on Solaris 10 SPARC)
[15:54] mato not that I expect it to support visibility, but the quotes look weird
[15:54] mikko the quotes can be removed
[15:54] mikko they shouldn't make a difference, looks cleaner without them
[15:55] mato up to you, I think I have Sun CC on here as well so will try with that oo
[15:55] mato too
[15:55] mikko as those go into compiler line gcc "-xldscope=hidden" == gcc -xldscope=hidden
[15:55] mato right
[15:55] mikko better put them off
[16:01] mato sustrik__: Assertion failed: nbytes == sizeof (command_t) (mailbox.cpp:193)
[16:01] mato sustrik__: in test_reqrep_tcp
[16:02] mato sustrik__: looks like the auto-resizing code in mailbox.cpp is completely bogus on FreeBSD too :-(
[16:02] mato sustrik__: anyway, not something I want to solve now, but it seems that the auto-resizing approach is no good...
[16:03] mato let's see if the tests pass on Solaris at least...
[16:03] mato hmm, Too many open files :-)
[16:04] mato ok, the tests pass if I raise the # of open files to 1024 from Solaris' default 256
[16:06] mikko mato:
[16:06] mikko mato: autogen check for pkg-config removed
[16:06] mikko fixed the variable name typos
[16:07] mato great
[16:07] mato building with Sun C on Solaris 10 SPARC now....
[16:13] mato and it all works, nice ...
[16:18] mikko there should be some macros that can be used in the future
[16:18] mato yes... this is really good work, thanks!
[16:19] mato just trying -xtarget=native -fast with Sun C for good measure
[16:19] mato and to see if we still honor CFLAGS/CXXFLAGS correctly
[16:31] mato mikko: ok, so the current version on is the latest?
[16:41] mikko y
[16:45] mato mikko: ok, great, it all checks out...
[16:45] mato mikko: Please send the latest version to the ML, I'll reply to it that it's the correct version that Martin should apply.
[16:46] mato mikko: The plan is to make a first release of master tomorrow (2.1.0).
[16:58] mikko mato: ill mail a link to it
[16:58] mikko it's a bit big to spam to everyones mailbox
[16:59] mato mikko: ok, sure
[17:56] mato mikko: any reason you've not yet sent the final patch? something left to do?
[18:00] mikko mato: work stuff
[18:00] mato mikko: ah, ok, no worries... just checking :)
[18:00] mikko sent now
[18:08] mato thx
[18:40] mikko i need to talk to pieterh at some point as well
[18:40] mikko ZFL on win32 + generic build cleanups
[20:52] boothead can anyone give me a hand handling messages in c++?
[20:52] boothead (i'm a c++ newb i'm afraid)
[20:56] drbobbeaty I'll try... what are you looking to do?
[20:58] boothead i'm recieving a message from a SUB socket which is the sub string and then a video frame 480*640
[20:58] boothead what i'm trying to do is get just that video frame into some other data structure i can use (I'm using opencv for the frame eventually)
[20:59] boothead drbobbeaty, here's the line that gets the message:
[20:59] boothead depth_socket.recv(&depth_msg, ZMQ_NOBLOCK);
[21:00] boothead what I want to do is drop the first 6 characters from the message and use the remainder, but I don't know the best (or actually any) way to do that
[21:00] drbobbeaty OK... I've not used the ZMQ_NOBLOCK as typically I have a very tight thread/loop that takes data off the socket and puts it into a single-producer/single-consumer lockless queue.
[21:00] boothead it would be depth_msg[6:] in python :-)
[21:01] drbobbeaty But you should be getting data into the zmq::message_t depth_msg.
[21:01] drbobbeaty Are you?
[21:01] drbobbeaty What's the depth_msg.size() after the recv() call?
[21:01] boothead the NOBLOCK is a bit of a red herring there - recieving the message is fine, i just don't know enough c++ to deal with the message afterward
[21:01] drbobbeaty OH... now I get it.
[21:02] boothead zmq::message_t depth_msg; is the line before that one
[21:02] drbobbeaty OK... what do you want to put it into? What C/C++ datastructure - specifically?
[21:02] drbobbeaty std::string? or something else?
[21:02] boothead well I need a pointer to the data part with I will pass to opencv's cv::Mat class
[21:03] drbobbeaty Then you need to use -- it returns a (char *) to the data in the message
[21:03] boothead it's a 480*640 array of uint_16
[21:03] drbobbeaty depth_msg.size() is the number of bytes in the "payload" of the message.
[21:03] boothead what i'm trying is this:
[21:04] kisielk sounds like you just need to offset the pointer by 6
[21:05] boothead here's what I have so far:
[21:05] boothead
[21:05] boothead kisielk, how would i then use the rest of the data as a uint_16?
[21:05] kisielk okay, I see you have a struct for the message type already
[21:06] kisielk you should be able to cast the message to a DepthMsg struct
[21:06] kisielk and then just read the frame member from it
[21:06] boothead i don't necessarily want to use that struct, i was scratching around trying to get somewhere!
[21:06] kisielk sure, but having a type for your messages is not a bad way to do it
[21:07] boothead kisielk, how do i do the cast?
[21:07] boothead glad to hear that I'm not totally barking up the wrong tree!
[21:08] drbobbeaty DepthMsg *depth_payload = (DepthMsg *);
[21:08] drbobbeaty should work for casting the data in the message to the struct you have. The question is - is it the right struct? Don't know.
[21:08] kisielk yeah, depends what you plan to do with your messages
[21:09] drbobbeaty But if you want to do anything with it, you need to copy that data out and then put it in a queue for processing so you can receive the next message. Assuming speed is important.
[21:10] drbobbeaty Also, I would not do anything if the depth_msg.size() == 0 -- there's nothing there, so why do anything. You have the "if" message... just put the processing in there - or a return to skip the rest of the processing on an empty message.
[21:10] boothead excellent it runs... but the line:
[21:10] boothead LOGM(MO_DEBUG, "depth msg address: " << depth_payload->address << "end");
[21:11] boothead is spitting binary data onto the screen... i thought address would only be 6 chars?
[21:11] kisielk is it null-terminated?
[21:12] kisielk like, is it 6 real chars? or 6 chars including null?
[21:12] drbobbeaty That binary data doesn't surprise me... you're asking it to print whatever it has - and if it's not ASCII, then it's binary junk.
[21:12] boothead what needs to happen is all the stuff in the commented section from the gist - although I think that the cv::Mat class has logic to do most of it.
[21:12] drbobbeaty You might want to format that nicer.
[21:12] drbobbeaty ... like 6 hex values - or something. Up to you, but it sounds like it's working.
[21:13] boothead basically i need to convert the 16 bit image data to 8 bit and then send it along to an out buffer
[21:13] kisielk hard to say what the format is without seeing the sender side
[21:13] drbobbeaty agreed - the format of the sender is crucial.
[21:15] boothead the format of the sender is the string 'depth ' and then a matrix of 480*640 uint_16
[21:16] boothead here's a working recieve side in python:
[21:17] boothead it's the depth stuff i'm interested in at the moment
[21:20] drbobbeaty I'm not a Python expert, but it appears that this 'cv' module is doing a lot of work for you. I think you have the data from the message just fine, but to get it to an image seems to belong to the 'cv' module. In C++, I have no idea what simple libraries there are for this kind of graphics work. Sorry.
[21:20] boothead shouldn't my log message be printing "depth msg address: depthend"?
[21:21] kisielk where is the code for the sender?
[21:21] boothead drbobbeaty, yes the cv module will be doing the heavy lifting cv::Mat had a constructor that you give a width, a height, a type and a pointer to the data and it's the pointer to the data that I'm trying to get here
[21:22] boothead kisielk,
[21:22] kisielk okay, and how do you know what length the "channel" portion of the string is?
[21:22] drbobbeaty Your Python code appears to be explicitly skipping the first 6 bytes in the message's payload - and those are the one that 'address' maps to. So while those first 6 bytes MIGHT be 'depth', it's possible that the sender is not sending ASCII, or maybe the first 6 bytes aren't 'depth '... Don't know.
[21:23] kisielk if it can be either "depth" or "rgb", then you can't just offset by 6 bytes
[21:23] drbobbeaty What happens if you skip the depth_payload->address and just use the depth_payload->frame pointer for the data?
[21:24] boothead drbobbeaty, fair point, kisielk the socket that I'm recieving from is sub'd to 'depth' i will have a similar set up for 'rgb' but a different processing pipeline
[21:24] kisielk yes, but the code to receive the messages should presumably be the same
[21:24] kisielk otherwise it's not very robust
[21:25] kisielk if you want to just stick with creating your own message protocol, you should use to do things on the Python side
[21:26] boothead drbobbeaty, if i do depth_payload->frame i get: depth msg address: 1end
[21:26] kisielk then you can be sure you'll get the same size/layout of struct in both Python and C++
[21:26] kisielk so basically you'd create a struct in Python, pack it with the appropriate data, and then have the same struct layout defined in C++
[21:27] boothead kisielk, the python stuff is just a quick proof of concept to get it up and running - I'll switch it over to c/c++ when I get the recv side working
[21:28] kisielk well, it would certainly be easier if you had a consistent definition of your messages, which you currently do not
[21:28] kisielk the message you are sending is not the same as what you are receiving
[21:29] drbobbeaty I wasn't suggesting that you print it out, but here's an idea - you need to be able to print out the hex data that's in that message. Here's a very simple method that will print out the hex contents of the zmq::message_t. I would suggest that you put this into your code and then see what the entire message is really saying. Then, you can see the first six bytes as well as all the others. (writing it now... give me a minute)
[21:32] boothead ok thanks drbobbeaty
[21:33] boothead kisielk, can you explain what you mean by that? it's the same enough that the python stuff works, is it different underneath in c++?
[21:34] kisielk what you're sending in Python isn't consistent. When you do "%s %s", the first string is variable length and not terminated
[21:34] kisielk which makes it hard to determine from the C++ code where the first field of the message ends
[21:35] kisielk the reason you probably get a bunch of binary data
[21:35] kisielk is you're copying the first 6 chars in to the first field of the message, which in your case are "depth "
[21:35] kisielk there's no \0 to terminate the string, so when you print it out, the stream doesn't know where to stop so just keeps right on going in to the second field of the struct
[21:36] kisielk so one way to fix it would be to make the first field 7 characters, and ensure the 7th is \0, but that's still a hack
[21:38] boothead ah i see - but if i use struct then i get the null termination
[21:38] boothead doesn't char address [6] say give me the first 6 characters?
[21:41] drbobbeaty
[21:42] drbobbeaty This is something I just whipped up... I haven't even tried to compile it - but I think it's going to work. If you run into issues, let me know. It's going to generate a lot of data for that message, but it's going to give you the best view inside the message's contents.
[21:42] drbobbeaty Far better than the std::cout you were using.
[21:42] boothead wow... you just wrote all that?!
[21:43] drbobbeaty Yeah, I had to answer a few emails too... sorry it took so long
[21:43] drbobbeaty I couldn't use tabs for indentation as it was the gist web page, but you get the idea.
[21:45] boothead drbobbeaty, i guess you've been doing this a slightly longer time than me!!
[21:45] boothead here are a few problems:
[21:45] boothead error: passing 'const zmq::message_t' as 'this' argument of 'void* zmq::message_t::data()' discards qualifiers
[21:46] boothead this line (10 in the gist) const char *buff =;
[21:47] drbobbeaty hold on a sec...
[21:51] drbobbeaty updated... there's a (const char *) in front of the call.
[21:51] drbobbeaty SHould compile OK now... sorry for the work you're doing, I just don't have the tools on this machine to do the work.
[21:51] boothead no appologies - can't thank you enough for your help!!
[21:52] boothead same deal: error: passing 'const zmq::message_t' as 'this' argument of 'void* zmq::message_t::data()' discards qualifiers
[21:52] boothead not complaining about the types anymore
[21:53] boothead what does that actually mean?
[21:53] boothead there's the same error message for the next line too for int size = aMessage.size();
[21:56] mikko boothead_: const zmq::message_t & aMessage
[21:57] mikko you are passing a const and you are calling a method that is not declared as const
[21:57] mikko so the compiler complains
[21:58] boothead mikko, ah i see... should i just remove the const?
[21:58] mikko boothead_: either remove the const or make a local copy of it
[21:58] drbobbeaty You can remove the const on the arg if you want - that will simplify things, but allow you to mess with the message. It's a simple method, so go ahead.
[21:59] mikko drbobbeaty: but ->data() is declared as non const method
[21:59] mikko you can't call that as aMessage is const
[21:59] mikko as far as i understand C++
[22:00] drbobbeaty Fair enough. I've updated my gist to cast the message to a non-const. Some don't like that - up to you.
[22:00] drbobbeaty Gotta run - be back in a little bit
[22:00] mikko why is the parameter passed as const?
[22:01] mikko it's used as non const in the method so i would assume passing it as normal argument would make sense
[22:07] mikko
[22:07] mikko i added a comment there but formatting messed up
[22:17] boothead i added a little bit to be able to cut down on the size of the dump - it segfaults
[22:17] boothead can anyone see anything glaring?
[22:21] mikko got a backtrace?
[22:21] mikko compile with -O0 -g
[22:21] mikko gdb --args ./yourprog
[22:21] mikko run
[22:21] mikko and when it crashes "bt"
[22:26] boothead it drops me straight into gdb... is that right? sorry - I'm a total newb at the c stuff
[22:26] mikko yes
[22:26] mikko now type 'run'
[22:26] mikko and press enter
[22:32] boothead ok i got snprintf(hex, 3, "%02x", (uint8_t)(*p));
[22:32] boothead as the place where it crashed
[22:39] mikko boothead_: works here
[22:39] mikko boothead_: in an isolated test case
[22:41] mikko boothead_:
[22:41] mikko if you test that alone
[22:44] boothead oh.
[22:44] boothead yep that works fine for me
[22:45] mikko can you show me the full code?
[22:49] boothead
[22:49] boothead thanks man!
[22:52] mikko you are initializing const char * inside a scope
[22:52] mikko that would leave end in the outer scope uninitialized
[22:52] boothead ah...
[22:52] mikko lines 64, 66, 70
[22:53] mikko your inner variable hides the one in the outer scope
[22:53] boothead so how should i do this part? just do const char end* = _end; after all of the if(num) stuff?
[22:53] mikko no
[22:53] mikko remove declaration and just assign
[22:53] mikko on line 64 end = _end;
[22:53] mikko instead of const char *end = _end;
[22:55] boothead i didn't realize you could assign to a const? though you had to do it when initializing..
[22:57] drbobbeaty I removed the const from the method signature... and the size() call... I'm not sure if there are any lingering problems, but I think you're on your way.
[22:58] boothead it's working now... and look what's in the message: 65 70 74 68 20 bb 03 bc 03 bc 03 bc 03 bd 03 be 03 bf depth .............
[22:58] boothead i was expecting depth to be the first characters
[22:58] drbobbeaty Yup... there you go - hex and asci data side-by-side.
[22:58] boothead wft?!
[22:58] drbobbeaty Good... it sounds like you have found the problem.
[22:58] boothead wtf even!
[22:59] drbobbeaty I was just noticing that the hex data was on the left and the ascii on the right - meaning I didn't mess up the code that bad. (const-ness aside)
[22:59] mikko boothead_: with const char *p you cannot modify where p points to but you can change where it points to
[22:59] mikko boothead_: with const char * const p you cannot modify the pointer or where it points to
[23:01] mikko i need to head to bed
[23:01] mikko good night all
[23:01] drbobbeaty mikko's right -- technically, the 'const' is supposed to mean "can't change". In linux on x86, it's not strictly enforced, but on the likes of Sun hardware, it is.
[23:02] boothead thanks a lot for your help mikko!
[23:02] drbobbeaty I've removed all the const-ness on my little gist as it's just causing too many issues. My point was only to give you a simple tool to inspect the data in the message without any formatting bias. Sorry if it caused more grief than it was worth.
[23:03] boothead so - going back to my struct - if i change the start of my message to 'depth\0<rest of the data>' then my struc will have address == depth and frame == the rest of the data?
[23:03] mikko drbobbeaty: yes, it seems sun pro compiler is a lot stricter about it
[23:03] boothead drbobbeaty, not at all - it's been an education!
[23:04] boothead will frame be a pointer to the start of the data?
[23:04] mikko no, wait
[23:04] mikko
[23:04] mikko compiles without errors or warnings with sun pro
[23:04] drbobbeaty mikko, it's also the hardware - SPARC uses a wonderful MMU that strictly enforces read-only sections of memory where all const variables reside.
[23:05] mikko drbobbeaty: ah
[23:05] mikko drbobbeaty: but technically the pointer itself is not modified
[23:05] mikko drbobbeaty: sounds like SPARC would go against standard?
[23:06] drbobbeaty mikko, I honestly think that linux/x86 is too liberal, and leads to the possibility that you make serious errors. However, it's where all development is headed (at least for my little world), so I go with it.
[23:07] mikko drbobbeaty: but it's const char *x; x = "test"; legal according to standard?
[23:07] drbobbeaty mikko, I think if you try to do a: p[3] = 'x'; in your code, Sun would not like it at runtime. It might compile, but it would not run.
[23:07] mikko const char * const x; x = "test"; shoudl generate error
[23:07] mikko drbobbeaty: it wont compile
[23:07] mikko drbobbeaty: as you are modifying where p points to
[23:07] mikko rather than the pointer istself
[23:08] mikko const char * const p; p = "test"; won't compile either
[23:08] drbobbeaty mikko, I'm not 100% sure - the second form may be considered the 'creation' point as opposed to the 'definition'. I'm not that strong on the standard.
[23:10] mikko the first one marks where the pointer points to as read-only location
[23:10] mikko
[23:11] mikko line 7 is actually the error reported on line 8
[23:11] drbobbeaty mikko: yeah... can't modify it. Nice to know. Have a good evening.