ZeroMq IRC Log

Tuesday November 30, 2010

[Time] Name	Message
[02:46] xunker	the 0mq.org page recommended I ask questions where if they aren't made clear in the docs, I've got one. I can't find somewhere where it says, yes or no, if you can use topics with something other than PUB/SUB sockets. Can I use 'topics' with say, PUSH/PULL socket pairs?
[15:15] mikko	mato: are you applying against master?
[15:15] mikko	mato: it complains about whitespace but applies here
[15:15] mato	mikko: yup, master
[15:16] mato	mikko: hmm, just looking at my git log
[15:16] mikko	mato: because i took a fresh git clone
[15:16] mikko	and it applies
[15:16] mato	mikko: looks like my master branch may have diverged, damn....
[15:16] mato	let me try and fix this...
[15:17] mikko	http://valokuva.org/~mikko/0001-Prefix-all-custom-macros-with-AC_ZMQ_.patch
[15:17] mikko	there it is with whitespace cleaned
[15:17] mikko	it's a bit heavy to send to mailing-list (>40k patch)
[15:18] mato	mikko: ok, it was some divergence on my master branch
[15:18] mato	mikko: sorry for the confusion...
[15:18] mato	am going to test it now
[15:18] mikko	no problem
[15:18] mato	by the way, we can remove the check for pkg-config in autogen.sh too I guess
[15:19] mato	all the pkg-config stuff was there due to the OpenPGM 2.x build depending on it
[15:19] mikko	ok
[15:34] mato	mikko: you wouldn't happen to have an oldish RHEL/CentOS installation lying around?
[15:35] mikko	i got centos 5.5 here
[15:35] mato	mikko: say some 5.x or 4.x? I'd like to see if this last set of changes breaks/fixes anything
[15:36] mikko	32 or 64 ?
[15:36] mato	can you try a random out-of-the-box build on it just to sanity check it works?
[15:36] mikko	or makes no difference
[15:36] mato	no difference
[15:36] mikko	gimme a min
[15:36] mikko	need to make dist on a newer machine
[15:36] mato	oh, and try --with-pgm as well if you can
[15:36] mato	if all that works then i think we're good to go
[15:42] mato	mikko: This isn't right:
[15:42] mato	checking for asciidoc... no
[15:42] mato	checking for xmlto... no
[15:42] mato	checking whether to build documentation... yes
[15:42] mato	this is on some FreeBSD box w/o asciidoc on it, building from a "make dist" with your patch
[15:43] mato	obviously fails later on because it tries to use asciidoc
[15:44] mato	oh, typo in your patch
[15:45] mato	fixing....
[15:48] mikko	builds on 5.5 with and without pgm
[15:48] mato	good
[15:49] mikko	yes
[15:50] mikko	+ if test "x$ac_zmq_cv_have_asciidoc" = "xno" -o "x$ac_zmq_cv_have_xmlto" = "xno"; then
[15:50] mato	yup
[15:50] mikko	cv is too much
[15:50] mato	by the way, which variables are supposed to have _cv_ and which arent?
[15:50] mato	http://pastebin.com/cGqKqavz
[15:50] mato	this should make it work, am going to test
[15:52] mikko	cv are ones that are cached
[15:53] mikko	i named most of the publicly visible ones "_cv_ because it's easier to add caching to them later
[15:53] mikko	cv = cache value
[15:53] mato	right
[15:53] mato	checking whether C compiler supports -fvisibility=hidden... no
[15:53] mato	checking whether C compiler supports "-xldscope=hidden"... no
[15:53] mato	checking whether C++ compiler supports -fvisibility=hidden... no
[15:53] mato	checking whether C++ compiler supports "-xldscope=hidden"... no
[15:53] mato	checking whether C++ compiler supports -mcpu=v9... yes
[15:54] mato	are those quotation marks supposed to be there?
[15:54] mato	(this is GCC 3.4.3 on Solaris 10 SPARC)
[15:54] mato	not that I expect it to support visibility, but the quotes look weird
[15:54] mikko	the quotes can be removed
[15:54] mikko	they shouldn't make a difference, looks cleaner without them
[15:55] mato	up to you, I think I have Sun CC on here as well so will try with that oo
[15:55] mato	too
[15:55] mikko	as those go into compiler line gcc "-xldscope=hidden" == gcc -xldscope=hidden
[15:55] mato	right
[15:55] mikko	better put them off
[16:01] mato	sustrik__: Assertion failed: nbytes == sizeof (command_t) (mailbox.cpp:193)
[16:01] mato	sustrik__: in test_reqrep_tcp
[16:02] mato	sustrik__: looks like the auto-resizing code in mailbox.cpp is completely bogus on FreeBSD too :-(
[16:02] mato	sustrik__: anyway, not something I want to solve now, but it seems that the auto-resizing approach is no good...
[16:03] mato	let's see if the tests pass on Solaris at least...
[16:03] mato	hmm, Too many open files :-)
[16:04] mato	ok, the tests pass if I raise the # of open files to 1024 from Solaris' default 256
[16:06] mikko	mato: valokuva.org/~mikko/0001-Prefix-variables-with-ac_zmq_.patch
[16:06] mikko	mato: autogen check for pkg-config removed
[16:06] mikko	fixed the variable name typos
[16:07] mato	great
[16:07] mato	building with Sun C on Solaris 10 SPARC now....
[16:13] mato	and it all works, nice ...
[16:18] mikko	there should be some macros that can be used in the future
[16:18] mato	yes... this is really good work, thanks!
[16:19] mato	just trying -xtarget=native -fast with Sun C for good measure
[16:19] mato	and to see if we still honor CFLAGS/CXXFLAGS correctly
[16:31] mato	mikko: ok, so the current version on valokuva.org is the latest?
[16:41] mikko	y
[16:45] mato	mikko: ok, great, it all checks out...
[16:45] mato	mikko: Please send the latest version to the ML, I'll reply to it that it's the correct version that Martin should apply.
[16:46] mato	mikko: The plan is to make a first release of master tomorrow (2.1.0).
[16:58] mikko	mato: ill mail a link to it
[16:58] mikko	it's a bit big to spam to everyones mailbox
[16:59] mato	mikko: ok, sure
[17:56] mato	mikko: any reason you've not yet sent the final patch? something left to do?
[18:00] mikko	mato: work stuff
[18:00] mato	mikko: ah, ok, no worries... just checking :)
[18:00] mikko	sent now
[18:08] mato	thx
[18:40] mikko	i need to talk to pieterh at some point as well
[18:40] mikko	ZFL on win32 + generic build cleanups
[20:52] boothead	can anyone give me a hand handling messages in c++?
[20:52] boothead	(i'm a c++ newb i'm afraid)
[20:56] drbobbeaty	I'll try... what are you looking to do?
[20:58] boothead	i'm recieving a message from a SUB socket which is the sub string and then a video frame 480*640
[20:58] boothead	what i'm trying to do is get just that video frame into some other data structure i can use (I'm using opencv for the frame eventually)
[20:59] boothead	drbobbeaty, here's the line that gets the message:
[20:59] boothead	depth_socket.recv(&depth_msg, ZMQ_NOBLOCK);
[21:00] boothead	what I want to do is drop the first 6 characters from the message and use the remainder, but I don't know the best (or actually any) way to do that
[21:00] drbobbeaty	OK... I've not used the ZMQ_NOBLOCK as typically I have a very tight thread/loop that takes data off the socket and puts it into a single-producer/single-consumer lockless queue.
[21:00] boothead	it would be depth_msg[6:] in python :-)
[21:01] drbobbeaty	But you should be getting data into the zmq::message_t depth_msg.
[21:01] drbobbeaty	Are you?
[21:01] drbobbeaty	What's the depth_msg.size() after the recv() call?
[21:01] boothead	the NOBLOCK is a bit of a red herring there - recieving the message is fine, i just don't know enough c++ to deal with the message afterward
[21:01] drbobbeaty	OH... now I get it.
[21:02] boothead	zmq::message_t depth_msg; is the line before that one
[21:02] drbobbeaty	OK... what do you want to put it into? What C/C++ datastructure - specifically?
[21:02] drbobbeaty	std::string? or something else?
[21:02] boothead	well I need a pointer to the data part with I will pass to opencv's cv::Mat class
[21:03] drbobbeaty	Then you need to use depth_msg.data() -- it returns a (char *) to the data in the message
[21:03] boothead	it's a 480*640 array of uint_16
[21:03] drbobbeaty	depth_msg.size() is the number of bytes in the "payload" of the message.
[21:03] boothead	what i'm trying is this:
[21:04] kisielk	sounds like you just need to offset the pointer by 6
[21:05] boothead	here's what I have so far:
[21:05] boothead	https://gist.github.com/722397
[21:05] boothead	kisielk, how would i then use the rest of the data as a uint_16?
[21:05] kisielk	okay, I see you have a struct for the message type already
[21:06] kisielk	you should be able to cast the message to a DepthMsg struct
[21:06] kisielk	and then just read the frame member from it
[21:06] boothead	i don't necessarily want to use that struct, i was scratching around trying to get somewhere!
[21:06] kisielk	sure, but having a type for your messages is not a bad way to do it
[21:07] boothead	kisielk, how do i do the cast?
[21:07] boothead	glad to hear that I'm not totally barking up the wrong tree!
[21:08] drbobbeaty	DepthMsg depth_payload = (DepthMsg )depth_msg.data();
[21:08] drbobbeaty	should work for casting the data in the message to the struct you have. The question is - is it the right struct? Don't know.
[21:08] kisielk	yeah, depends what you plan to do with your messages
[21:09] drbobbeaty	But if you want to do anything with it, you need to copy that data out and then put it in a queue for processing so you can receive the next message. Assuming speed is important.
[21:10] drbobbeaty	Also, I would not do anything if the depth_msg.size() == 0 -- there's nothing there, so why do anything. You have the "if" message... just put the processing in there - or a return to skip the rest of the processing on an empty message.
[21:10] boothead	excellent it runs... but the line:
[21:10] boothead	LOGM(MO_DEBUG, "depth msg address: " << depth_payload->address << "end");
[21:11] boothead	is spitting binary data onto the screen... i thought address would only be 6 chars?
[21:11] kisielk	is it null-terminated?
[21:12] kisielk	like, is it 6 real chars? or 6 chars including null?
[21:12] drbobbeaty	That binary data doesn't surprise me... you're asking it to print whatever it has - and if it's not ASCII, then it's binary junk.
[21:12] boothead	what needs to happen is all the stuff in the commented section from the gist - although I think that the cv::Mat class has logic to do most of it.
[21:12] drbobbeaty	You might want to format that nicer.
[21:12] drbobbeaty	... like 6 hex values - or something. Up to you, but it sounds like it's working.
[21:13] boothead	basically i need to convert the 16 bit image data to 8 bit and then send it along to an out buffer
[21:13] kisielk	hard to say what the format is without seeing the sender side
[21:13] drbobbeaty	agreed - the format of the sender is crucial.
[21:15] boothead	the format of the sender is the string 'depth ' and then a matrix of 480*640 uint_16
[21:16] boothead	here's a working recieve side in python: https://github.com/boothead/zmqnect/blob/master/zmqnect/sub.py
[21:17] boothead	it's the depth stuff i'm interested in at the moment
[21:20] drbobbeaty	I'm not a Python expert, but it appears that this 'cv' module is doing a lot of work for you. I think you have the data from the message just fine, but to get it to an image seems to belong to the 'cv' module. In C++, I have no idea what simple libraries there are for this kind of graphics work. Sorry.
[21:20] boothead	shouldn't my log message be printing "depth msg address: depthend"?
[21:21] kisielk	where is the code for the sender?
[21:21] boothead	drbobbeaty, yes the cv module will be doing the heavy lifting cv::Mat had a constructor that you give a width, a height, a type and a pointer to the data and it's the pointer to the data that I'm trying to get here
[21:22] boothead	kisielk, https://github.com/boothead/zmqnect/blob/master/zmqnect/pub.py
[21:22] kisielk	okay, and how do you know what length the "channel" portion of the string is?
[21:22] drbobbeaty	Your Python code appears to be explicitly skipping the first 6 bytes in the message's payload - and those are the one that 'address' maps to. So while those first 6 bytes MIGHT be 'depth', it's possible that the sender is not sending ASCII, or maybe the first 6 bytes aren't 'depth '... Don't know.
[21:23] kisielk	if it can be either "depth" or "rgb", then you can't just offset by 6 bytes
[21:23] drbobbeaty	What happens if you skip the depth_payload->address and just use the depth_payload->frame pointer for the data?
[21:24] boothead	drbobbeaty, fair point, kisielk the socket that I'm recieving from is sub'd to 'depth' i will have a similar set up for 'rgb' but a different processing pipeline
[21:24] kisielk	yes, but the code to receive the messages should presumably be the same
[21:24] kisielk	otherwise it's not very robust
[21:25] kisielk	if you want to just stick with creating your own message protocol, you should use http://docs.python.org/library/struct.html to do things on the Python side
[21:26] boothead	drbobbeaty, if i do depth_payload->frame i get: depth msg address: 1end
[21:26] kisielk	then you can be sure you'll get the same size/layout of struct in both Python and C++
[21:26] kisielk	so basically you'd create a struct in Python, pack it with the appropriate data, and then have the same struct layout defined in C++
[21:27] boothead	kisielk, the python stuff is just a quick proof of concept to get it up and running - I'll switch it over to c/c++ when I get the recv side working
[21:28] kisielk	well, it would certainly be easier if you had a consistent definition of your messages, which you currently do not
[21:28] kisielk	the message you are sending is not the same as what you are receiving
[21:29] drbobbeaty	I wasn't suggesting that you print it out, but here's an idea - you need to be able to print out the hex data that's in that message. Here's a very simple method that will print out the hex contents of the zmq::message_t. I would suggest that you put this into your code and then see what the entire message is really saying. Then, you can see the first six bytes as well as all the others. (writing it now... give me a minute)
[21:32] boothead	ok thanks drbobbeaty
[21:33] boothead	kisielk, can you explain what you mean by that? it's the same enough that the python stuff works, is it different underneath in c++?
[21:34] kisielk	what you're sending in Python isn't consistent. When you do "%s %s", the first string is variable length and not terminated
[21:34] kisielk	which makes it hard to determine from the C++ code where the first field of the message ends
[21:35] kisielk	the reason you probably get a bunch of binary data
[21:35] kisielk	is you're copying the first 6 chars in to the first field of the message, which in your case are "depth "
[21:35] kisielk	there's no \0 to terminate the string, so when you print it out, the stream doesn't know where to stop so just keeps right on going in to the second field of the struct
[21:36] kisielk	so one way to fix it would be to make the first field 7 characters, and ensure the 7th is \0, but that's still a hack
[21:38] boothead	ah i see - but if i use struct then i get the null termination
[21:38] boothead	doesn't char address [6] say give me the first 6 characters?
[21:41] drbobbeaty	https://gist.github.com/722469
[21:42] drbobbeaty	This is something I just whipped up... I haven't even tried to compile it - but I think it's going to work. If you run into issues, let me know. It's going to generate a lot of data for that message, but it's going to give you the best view inside the message's contents.
[21:42] drbobbeaty	Far better than the std::cout you were using.
[21:42] boothead	wow... you just wrote all that?!
[21:43] drbobbeaty	Yeah, I had to answer a few emails too... sorry it took so long
[21:43] drbobbeaty	I couldn't use tabs for indentation as it was the gist web page, but you get the idea.
[21:45] boothead	drbobbeaty, i guess you've been doing this a slightly longer time than me!!
[21:45] boothead	here are a few problems:
[21:45] boothead	error: passing 'const zmq::message_t' as 'this' argument of 'void* zmq::message_t::data()' discards qualifiers
[21:46] boothead	this line (10 in the gist) const char *buff = aMessage.data();
[21:47] drbobbeaty	hold on a sec...
[21:51] drbobbeaty	updated... there's a (const char *) in front of the call.
[21:51] drbobbeaty	SHould compile OK now... sorry for the work you're doing, I just don't have the tools on this machine to do the work.
[21:51] boothead	no appologies - can't thank you enough for your help!!
[21:52] boothead	same deal: error: passing 'const zmq::message_t' as 'this' argument of 'void* zmq::message_t::data()' discards qualifiers
[21:52] boothead	not complaining about the types anymore
[21:53] boothead	what does that actually mean?
[21:53] boothead	there's the same error message for the next line too for int size = aMessage.size();
[21:56] mikko	boothead_: const zmq::message_t & aMessage
[21:57] mikko	you are passing a const and you are calling a method that is not declared as const
[21:57] mikko	so the compiler complains
[21:58] boothead	mikko, ah i see... should i just remove the const?
[21:58] mikko	boothead_: either remove the const or make a local copy of it
[21:58] drbobbeaty	You can remove the const on the arg if you want - that will simplify things, but allow you to mess with the message. It's a simple method, so go ahead.
[21:59] mikko	drbobbeaty: but ->data() is declared as non const method
[21:59] mikko	you can't call that as aMessage is const
[21:59] mikko	as far as i understand C++
[22:00] drbobbeaty	Fair enough. I've updated my gist to cast the message to a non-const. Some don't like that - up to you.
[22:00] drbobbeaty	Gotta run - be back in a little bit
[22:00] mikko	why is the parameter passed as const?
[22:01] mikko	it's used as non const in the method so i would assume passing it as normal argument would make sense
[22:07] mikko	https://gist.github.com/722469#comments
[22:07] mikko	i added a comment there but formatting messed up
[22:17] boothead	https://gist.github.com/722518 i added a little bit to be able to cut down on the size of the dump - it segfaults
[22:17] boothead	can anyone see anything glaring?
[22:21] mikko	got a backtrace?
[22:21] mikko	compile with -O0 -g
[22:21] mikko	gdb --args ./yourprog
[22:21] mikko	run
[22:21] mikko	and when it crashes "bt"
[22:26] boothead	it drops me straight into gdb... is that right? sorry - I'm a total newb at the c stuff
[22:26] mikko	yes
[22:26] mikko	now type 'run'
[22:26] mikko	and press enter
[22:32] boothead	ok i got snprintf(hex, 3, "%02x", (uint8_t)(*p));
[22:32] boothead	as the place where it crashed
[22:39] mikko	boothead_: works here
[22:39] mikko	boothead_: in an isolated test case
[22:41] mikko	boothead_: https://gist.github.com/68a12c20ecf80f759f03
[22:41] mikko	if you test that alone
[22:44] boothead	oh.
[22:44] boothead	yep that works fine for me
[22:45] mikko	can you show me the full code?
[22:49] boothead	https://github.com/boothead/Movid/blob/zmq-broken/src/modules/moKinectZMQModule.cpp
[22:49] boothead	thanks man!
[22:52] mikko	you are initializing const char * inside a scope
[22:52] mikko	that would leave end in the outer scope uninitialized
[22:52] boothead	ah...
[22:52] mikko	lines 64, 66, 70
[22:53] mikko	your inner variable hides the one in the outer scope
[22:53] boothead	so how should i do this part? just do const char end* = _end; after all of the if(num) stuff?
[22:53] mikko	no
[22:53] mikko	remove declaration and just assign
[22:53] mikko	on line 64 end = _end;
[22:53] mikko	instead of const char *end = _end;
[22:55] boothead	i didn't realize you could assign to a const? though you had to do it when initializing..
[22:57] drbobbeaty	I removed the const from the method signature... and the size() call... I'm not sure if there are any lingering problems, but I think you're on your way.
[22:58] boothead	it's working now... and look what's in the message: 65 70 74 68 20 bb 03 bc 03 bc 03 bc 03 bd 03 be 03 bf depth .............
[22:58] boothead	i was expecting depth to be the first characters
[22:58] drbobbeaty	Yup... there you go - hex and asci data side-by-side.
[22:58] boothead	wft?!
[22:58] drbobbeaty	Good... it sounds like you have found the problem.
[22:58] boothead	wtf even!
[22:59] drbobbeaty	I was just noticing that the hex data was on the left and the ascii on the right - meaning I didn't mess up the code that bad. (const-ness aside)
[22:59] mikko	boothead_: with const char *p you cannot modify where p points to but you can change where it points to
[22:59] mikko	boothead_: with const char * const p you cannot modify the pointer or where it points to
[23:01] mikko	i need to head to bed
[23:01] mikko	good night all
[23:01] drbobbeaty	mikko's right -- technically, the 'const' is supposed to mean "can't change". In linux on x86, it's not strictly enforced, but on the likes of Sun hardware, it is.
[23:02] boothead	thanks a lot for your help mikko!
[23:02] drbobbeaty	I've removed all the const-ness on my little gist as it's just causing too many issues. My point was only to give you a simple tool to inspect the data in the message without any formatting bias. Sorry if it caused more grief than it was worth.
[23:03] boothead	so - going back to my struct - if i change the start of my message to 'depth\0<rest of the data>' then my struc will have address == depth and frame == the rest of the data?
[23:03] mikko	drbobbeaty: yes, it seems sun pro compiler is a lot stricter about it
[23:03] boothead	drbobbeaty, not at all - it's been an education!
[23:04] boothead	will frame be a pointer to the start of the data?
[23:04] mikko	no, wait
[23:04] mikko	https://gist.github.com/53728e300262c4b2bd9f
[23:04] mikko	compiles without errors or warnings with sun pro
[23:04] drbobbeaty	mikko, it's also the hardware - SPARC uses a wonderful MMU that strictly enforces read-only sections of memory where all const variables reside.
[23:05] mikko	drbobbeaty: ah
[23:05] mikko	drbobbeaty: but technically the pointer itself is not modified
[23:05] mikko	drbobbeaty: sounds like SPARC would go against standard?
[23:06] drbobbeaty	mikko, I honestly think that linux/x86 is too liberal, and leads to the possibility that you make serious errors. However, it's where all development is headed (at least for my little world), so I go with it.
[23:07] mikko	drbobbeaty: but it's const char *x; x = "test"; legal according to standard?
[23:07] drbobbeaty	mikko, I think if you try to do a: p[3] = 'x'; in your code, Sun would not like it at runtime. It might compile, but it would not run.
[23:07] mikko	const char * const x; x = "test"; shoudl generate error
[23:07] mikko	drbobbeaty: it wont compile
[23:07] mikko	drbobbeaty: as you are modifying where p points to
[23:07] mikko	rather than the pointer istself
[23:08] mikko	const char * const p; p = "test"; won't compile either
[23:08] drbobbeaty	mikko, I'm not 100% sure - the second form may be considered the 'creation' point as opposed to the 'definition'. I'm not that strong on the standard.
[23:10] mikko	the first one marks where the pointer points to as read-only location
[23:10] mikko	https://gist.github.com/6adb476a4c2f11408f21
[23:11] mikko	line 7 is actually the error reported on line 8
[23:11] drbobbeaty	mikko: yeah... can't modify it. Nice to know. Have a good evening.