IRC Log

Wednesday September 1, 2010

[Time] NameMessage
[00:05] ModusPwnens can anyone tell me about the size method for zeromq?
[06:01] CIA-20 zeromq2: 03Martin Sustrik 07master * r47c064f 10/ (src/fq.cpp src/lb.cpp): hangup when closing socket with no pipes attached -- fixed - http://bit.ly/9iy2zX
[06:01] CIA-20 zeromq2: 03Martin Sustrik 07master * raaa0761 10/ src/pair.cpp : pipe being attached to the PAIR socket during its termination process is immediately asked to terminate itself - http://bit.ly/bwIpo1
[06:01] CIA-20 zeromq2: 03Martin Sustrik 07master * rdb73c76 10/ src/pub.cpp : assert when pipe attaches to PUB socket in process of termination -- fixed - http://bit.ly/bCxg7v
[06:01] CIA-20 zeromq2: 03Martin Sustrik 07master * rce0972d 10/ (src/ctx.cpp src/ctx.hpp src/object.cpp src/object.hpp): context creates an inproc endpoint ('inproc://log') to distribute 0MQ's log messages - http://bit.ly/9YrEbe
[07:02] guido_g howdy
[07:05] guido_g cool, logging...
[07:09] guido_g does the core itself use the logging?
[07:20] guido_g hmmm
[07:20] guido_g doesn't seem so
[07:27] sustrik only the core can use the logging
[07:27] sustrik however, atm it's not logging anything
[07:27] sustrik logs are to be added gradually
[07:31] guido_g well... better then nothing :)
[08:50] Mikael_H_Kjaer Sustrik: I don't have a small example that triggers the bug I've posted about on the mailing list.
[08:51] sustrik hi mikael
[08:51] sustrik that's a problem
[08:51] sustrik from inspecting the backtrace it looks like it's not a 0mq problem
[08:51] Mikael_H_Kjaer I know. I have been trying to boil it down
[08:51] sustrik maybe a memory overwrite?
[08:52] sustrik some random code overwriting the socket?
[08:52] Mikael_H_Kjaer Maybe. Currently I am not hitting the issue anymore
[08:52] Mikael_H_Kjaer But I shouldn't have any wild pointers or memory accesses in my code. I am currently using smart pointers for everything and my cycle checker seems to think I am okay too.
[08:54] Mikael_H_Kjaer I haven't valgrinded my code in a few weeks but I will soon try it with zmq added in.
[08:54] sustrik ok. good
[08:55] sustrik btw, maybe try to compile with debug into so that backtrace is more informative
[08:56] Mikael_H_Kjaer I do :) what additional information do you want
[08:58] sustrik hm
[08:59] sustrik can you possibly add a printf statement to socket's destructor
[08:59] sustrik so that we know whether the socket was destructed or overwritten?
[08:59] sustrik printing out this pointer would be good so that we can correlate it with the backtrace
[09:00] Mikael_H_Kjaer Not a problem
[09:00] Mikael_H_Kjaer I will get back with more. Any other printf's you want?
[09:01] sustrik not yet
[09:02] sustrik let's first see whrther the socket is destructed
[09:12] Mikael_H_Kjaer sustrik: I can't get the error to show up any more. I believe my fix of waiting for all sends to finish before attempting a shutdown of zmq has resolved it.
[09:12] sustrik hm
[09:13] sustrik ok, so let's leave it for now
[09:13] sustrik if you ever see it again, we'll continue
[09:16] Mikael_H_Kjaer Agreed, I was thinking the same.
[09:51] mcxx I'm building a system that will use architecture similar to this one: http://github.com/imatix/zguide/raw/master/images/fig16.png
[09:52] mcxx however, in this setup, the broker is the single point of failure
[09:52] mcxx can the architecture be altered somehowe? e.g. can the clients connect to multiple brokers so that when one goes down, they switch to a working one?
[09:54] sustrik yes, you can load balance via multiple brokers
[09:55] sustrik just start 2 brokers
[09:55] sustrik and connect all components to both of them
[09:55] sustrik you also have to set HWM on clients
[09:55] sustrik so that messages are not queued for the unavailable broker indefinitely
[09:57] mcxx what's HWM? Google doesn't show reasonable results
[09:57] mcxx oh, I see noe
[09:57] mcxx now
[10:39] mcxx when I set HWM on the client and one of the brokers goes down, will the clients' sockets find out the broker is no longer available and automatically use others?
[10:39] guido_g huh? what broker?
[10:40] guido_g you mean the pipes of the underlying connections?
[10:40] mcxx guido_g: it's a follow up to my previous question
[10:40] mcxx in this arch http://github.com/imatix/zguide/raw/master/images/fig16.png
[10:40] guido_g as far as i know, full pipes (hwm reached) are skipped
[10:41] guido_g i see
[10:41] guido_g still, full pipes are skipped
[10:42] guido_g more should be found on http://api.zeromq.org/zmq_socket.html describption of the socket types
[10:43] mcxx re-reading that helps :)
[10:46] guido_g :)
[11:32] CIA-20 zeromq2: 03Martin Sustrik 07master * r6a0c323 10/ (builds/msvc/libzmq/libzmq.vcproj src/ctx.cpp): MSVC build fixed - http://bit.ly/cEMTGs
[11:39] pieterh I've posted a blog that hopefully clears up the naming confusion for once and for all...
[11:39] pieterh http://www.zeromq.org/blog:not-to-be-confused
[11:39] pieterh anyone from Denmark here?
[11:43] guido_g :)
[12:04] pieterh sustrik: random question about 2.1
[12:05] sustrik yes?
[12:07] pieterh how solid is it?
[12:07] pieterh should I start to be using it for the examples in the guide?
[12:08] sustrik it haven't been tested extensively
[12:08] sustrik but the API is the same
[12:08] sustrik so, for the guide, it's irrelevant which one you use
[12:08] pieterh there's the 'insert random sleep here' which i'd like to remove
[12:08] sustrik ah, yes, that can go with 2.1
[12:09] pieterh i'll start to test it then
[12:09] pieterh it'd be fun to release a 2.1.0 unstable IMO
[12:09] sustrik btw, there's a test suite in the package now
[12:09] pieterh especially if you have stuff like sys:// in there...
[12:09] sustrik so if you write a test
[12:09] sustrik jsut put it in there
[12:10] sustrik well, the infrastructure is there, but no notifications are passed atm
[12:10] pieterh :-) i saw some emails on that but missed the original discussion
[12:10] pieterh great idea to have a test suite!
[12:10] sustrik it's in tests subdir
[12:10] sustrik just put your file there
[12:10] sustrik add it to Makefila.am
[12:10] sustrik and make check will run it
[12:10] pieterh ack
[12:11] guido_g ah, while talking about tests
[12:12] guido_g i plan to write the basic tests for the missing socket types over the weekend
[12:12] sustrik missing socket types?
[12:12] sustrik ah, those not yet in test suite
[12:12] guido_g afaik only req/rep and pair are tested atm
[12:12] sustrik nice
[12:13] pieterh ZMQ_∅
[12:13] guido_g if there are any corner cases that should be tested, please leave me a note
[12:13] pieterh the secret socket typ
[12:13] pieterh *type
[12:13] pieterh if you can't spell it, you can't use it...
[12:13] guido_g pieterh seems to be in a funny mood today
[12:13] guido_g :)
[12:13] sustrik you really got the letter or have you made it up?
[12:14] pieterh that's a real letter
[12:14] pieterh i couldn't invent something _that_ ... unique
[12:14] guido_g justifies the number of beers you had
[12:14] pieterh that's why the guide made that strange reference to annoying the Danes
[12:15] guido_g ahhh
[12:15] pieterh next we're going to get email from the mathematicians accusing us of soiling Bourbaki's name
[12:16] pieterh guido_g: what language are you using for the tests?
[12:16] guido_g c++
[12:17] guido_g but they can be in any language the core understands
[12:17] pieterh there's one example that would work for push/pull and pair
[12:17] guido_g for these sockets we have tests already
[12:17] pieterh the mtrelay
[12:17] pieterh you just said only req/rep and pair...?
[12:18] guido_g oh sorry
[12:18] guido_g parsed the pub/sub wrong
[12:18] guido_g push/pull
[12:18] pieterh agh, I realized that example in fact doesn't use push/pull finally, i switched to using pair
[12:19] guido_g and i thought they do have good beer in .sk
[12:19] pieterh it's ok, not real beer like we make in Belgium
[12:19] guido_g ah
[12:20] pieterh kind of halfway between real beer (belgian) and fake beer (anglo)
[12:20] pieterh just kidding
[12:20] pieterh it's the home of beer, of course
[12:20] pieterh ZeroMQ is a big hit in Japan it seems
[12:21] pieterh まだ結構α版なんで、問題とかあったら教えて下さい RT !heavenshell: おお。perl の ZeroMQ のサンプルは動い
[12:21] guido_g ahh... sure
[12:21] sustrik there are some japanese guys working on the perl binding afaiu
[12:21] pieterh I think they're complaining about the spelling... :-/
[12:21] pieterh my Japanese isn't so good anymore
[12:22] sustrik btw "where 0mq goes, google cannot follow" is a nice slogan
[12:22] sustrik wrt distributed vs. centralised services
[12:22] sustrik privacy etc.
[12:22] sustrik のMQ would be read NoMQ :)
[12:25] pieterh Hehe, "can't follow this"...
[12:27] pieterh we need to prepare an ultrasecret joke for next April 1st
[12:27] pieterh where we rename 0MQ to のMQ
[12:27] pieterh get lots of people to blog and twitter about it
[12:27] pieterh confuse the heck out of everyone...
[12:31] pieterh sustrik: could you send me a .zip of the zeromq2 master (clean checkout)?
[12:31] pieterh i'm on a slow 2G connection and have failed 4 times to do a git pull
[12:31] pieterh maybe email will work better
[12:39] pieterh sustrik: recv'd! thanks... git seems to have a problem working across slow links
[12:39] pieterh too much chatter or something... they aren't using 0MQ....
[13:10] pieterh sustrik: what's your view of when 2.1.0 would be releasable?
[13:10] pieterh would it help if I stress tested it with some real applications?
[13:10] pieterh btw, it all build and passed the test suite fine, of course
[13:11] pieterh *built
[13:11] mikko you can run tests in language bindings as well
[13:11] guido_g not a real suprise, the tests are really basic
[13:11] mikko they probably contain fair amount of tests
[13:12] pieterh gudio_g: i'd like to try breakage tests...
[13:13] guido_g nevertheless do we need good test coverage in the core
[13:13] pieterh yes, we can't refer up the stack
[13:13] pieterh core can't test using language bindings, this is up to the binding author/community
[13:13] guido_g pieterh: what does that mean?
[13:14] guido_g breakage tests
[13:14] pieterh well, stuff like send message, close, check that it's delivered or not according to socket policy
[13:14] pieterh exceed HWM, check messages are delayed/discarded
[13:14] pieterh send invalid stuff, check 0MQ does not die
[13:14] pieterh do bad operations, invalid socket mixes, illegal sequences
[13:15] pieterh all the stuff normal apps do all the time... :-)
[13:15] guido_g ahh ok
[13:15] guido_g normal tests :)
[13:15] pieterh normal regression is of course also very useful
[13:15] pieterh "it still works"
[13:15] pieterh but "i can't break it" is also valuable
[13:16] guido_g that you get with every build and make check
[13:16] guido_g the regression test
[13:16] pieterh yes
[13:16] guido_g ok, would be nice to have a list of tests to do
[13:16] pieterh it's sometimes possible to collect test cases for issues
[13:16] guido_g i'll go over it as time permits and write the test programs
[13:17] pieterh oh, sorry, i'm not trying to propose work... :-/
[13:17] guido_g it has to be done
[13:17] pieterh i'm more than happy to help though it would be in C not C++
[13:17] pieterh well, let's start by collecting specs?
[13:17] pieterh sorry s/start/continue/
[13:18] guido_g yes
[13:18] pieterh perhaps this can help language binding authors make coherent test sets too
[13:18] guido_g to make it leight and easy to start with, a simple collection of ideas what to test would be ok i think
[13:18] pieterh i.e. standardized test set spec
[13:18] pieterh yes
[13:19] pieterh it would give users a tool for validating bindings
[13:19] pieterh "has test suite"
[13:20] pieterh actually it should then be written both in C++ and C
[13:20] pieterh i volunteer to make the C test suite then
[13:22] guido_g :)
[13:22] guido_g i'd like to stick with c++ for now
[13:22] guido_g it's hard enough to relearn it
[13:23] pieterh sure, so we will double up the test suite, that also gives people examples to compare?
[13:23] pieterh sustrik: do you have any problem with a C test suite in parallel to the C++ one?
[13:23] pieterh if it's a stupid idea, i'm happy to retract it... :-)
[13:26] guido_g hehe
[13:26] guido_g see you later
[13:29] CIA-20 zeromq2: 03Martin Sustrik 07master * r651c1ad 10/ (src/ctx.cpp src/socket_base.cpp): sys transport introdced; inproc://log moved to sys://log - http://bit.ly/bmO6tM
[13:29] CIA-20 zeromq2: 03Martin Sustrik 07master * r0fe7d3c 10/ (builds/msvc/libzmq/libzmq.vcproj src/ctx.cpp): conflicts resolved - http://bit.ly/aV1tLE
[14:02] guido_g re
[14:03] sustrik pieterh: C test suite is OK
[14:03] sustrik but afaik the files have to have .cpp extension
[14:03] sustrik so that autotools know it should link c++ runtime with it
[14:04] sustrik (needed by 0mq library, in case of static linking)
[14:04] pieterh yes, indeed
[14:04] pieterh great
[14:04] guido_g why is the logging now a special transport?
[14:04] pieterh will have to use a naming convention to avoid conflicts, NP
[14:05] sustrik guido_g: two reasons
[14:05] sustrik 1. it's a different beast, communicating with 0mq internals rather than communicating with a socket in a different thread
[14:05] sustrik 2. full backward compatibility
[14:06] guido_g ok
[14:06] pieterh sustrik: i see one problem with making it a separate transport
[14:06] guido_g how will users attach to this type of transport?
[14:07] sustrik guido_g: connect
[14:07] guido_g just a SUB w/ sys://log?
[14:07] sustrik pieterh: what problem?
[14:07] sustrik guido_g: yes
[14:07] pieterh take the syslog service as an analogy
[14:07] guido_g great! thanks
[14:07] pieterh it's a bus, any component can participate
[14:08] pieterh ok, that's not quite accurate but I mean you might want to allow arbitrary components to inject traffic into this
[14:08] sustrik i don't :)
[14:08] pieterh and have it picked up by log clients without them needing to be changed
[14:08] guido_g :)
[14:08] sustrik what i mean is how would you scale that kind of thing?
[14:08] sustrik right now it's easy
[14:09] pieterh it's inproc or ipc, what's the scaling issue?
[14:09] sustrik it's constrained to the process
[14:09] sustrik the business logic
[14:09] pieterh attach device to sub, export as pub... np
[14:09] sustrik what you want with a business logic is a distributed logging service
[14:09] guido_g ack
[14:09] guido_g the way with a forwarder is ok
[14:10] pieterh i can play devil's advocate here
[14:10] sustrik pieterh: i have no idea how you would implement it
[14:10] pieterh it looks like you are claiming a semantic difference in order to solve a naming clash
[14:10] pieterh it's unclear what the value is of a new transport
[14:10] pieterh no clear problem you're solving wrt to using inproc:
[14:10] sustrik you mean between sys & inproc or between 0mq logging and application logging?
[14:10] pieterh or ipc:
[14:12] pieterh sustrik: ... sorry, I don't follow this thread
[14:12] sustrik pieterh> it looks like you are claiming a semantic difference in order to solve a naming clash
[14:12] pieterh ah, yes
[14:12] pieterh between sys: and inproc or ipc
[14:13] pieterh is sys: like inproc or like ipc?
[14:13] sustrik it's different from both
[14:13] sustrik it has no peer actually
[14:13] pieterh yes, but can you connect to it from another process?
[14:13] pieterh is it an in or ex-process endpoint?
[14:14] sustrik it's process-scoped
[14:14] pieterh so it's like inproc
[14:14] sustrik sure, same scope
[14:14] sustrik different semantics
[14:15] pieterh now if i am being evil and minimalist i'll ask what the value is in defining a totally new protocol instead of a protected namespace within inproc
[14:15] pieterh given that every new construct is something expensive to document, understand...
[14:15] sustrik same question: why not make ipc a separate namespace within tcp?
[14:15] pieterh but the semantics fit perfectly to an inproc SUB socket from the users' POV
[14:16] sustrik no: there's no peer socket
[14:16] pieterh that's an implementation detail
[14:16] pieterh that's behind the wall, and not relevant to users
[14:16] pieterh it looks like there is a peer socket
[14:16] sustrik ok, let me explain the internal mechanism of sys
[14:16] pieterh ah, hang on
[14:16] pieterh i'm at the API level
[14:17] pieterh internal mechanisms should not be relevant
[14:17] guido_g this is what sys: says imho, don't expect the usual, this comming from the kernel
[14:17] pieterh but why?
[14:17] pieterh why the surprise?
[14:17] guido_g i can't see any, sorry
[14:17] pieterh why is the kernel a 'special' component with its own transport that looks exactly like inproc: but has a different name?
[14:17] pieterh it's surprising
[14:18] sustrik it's administration/business logic distinction
[14:18] sustrik inproc carries business logic
[14:18] sustrik sys carries admin data
[14:18] pieterh not necessarily
[14:18] sustrik necessarily
[14:18] pieterh as soon as i make a framework over 0MQ
[14:18] pieterh then inproc carries admin data
[14:18] guido_g for an application using the core
[14:18] guido_g not the ømq core itself
[14:18] pieterh ok... perhaps it would help if there was some significant semantic difference between admin data and business data
[14:19] pieterh such as priority
[14:19] guido_g i see sys: as the core namespace
[14:19] pieterh or blocking/ error handling
[14:19] pieterh guido_g: as a namespace this is a lousy solution IMO
[14:19] pieterh namespaces should be orthogonal to transports
[14:19] guido_g pieterh: why?
[14:19] pieterh well, this makes it impossible to run a separate logger to inspect a running process
[14:20] pieterh every single application needs a special thread that exports the sys: data over ipc: or tcp:
[14:20] sustrik the main problem i have with the whole log thing is that people are now going to use it for their business logic and thus break the patterns
[14:20] sustrik if so, i'll rather remove it completely
[14:20] pieterh sustrik: no, you can prevent that easily
[14:20] sustrik ?
[14:20] guido_g sustrik: hold it, we need it!
[14:20] pieterh you can prevent that trivially
[14:20] pieterh use a protected name space
[14:20] guido_g like sys: ? :)
[14:21] pieterh recall in amqp how exchanges beginning with amq. were protected
[14:21] pieterh transports are not namespaces... :-(
[14:21] guido_g sure
[14:21] sustrik what i mean is people are now going to ask me to report disconnections via sys
[14:21] pieterh inproc://zmq/log and ipc://zmq/log would be perfect
[14:21] pieterh allows me to start separate logger
[14:21] sustrik so that they can handle each connection separately
[14:21] pieterh perhaps
[14:22] sustrik to prevent that, the admin part should be as much separated from the business part as possible
[14:22] pieterh that's a use case but it's possibly not the right solution
[14:22] pieterh hmm
[14:22] sustrik different API would be ideal
[14:22] guido_g but the application developer doe *not* know what makes sys: work technically, so the obvious similarity to inproc might be wrong
[14:22] sustrik but that's too much work
[14:22] pieterh sustrik: do you see the value in allowing this namespace to work over both ipc and inproc?
[14:22] sustrik no
[14:23] sustrik it's just adminstration stuff
[14:23] pieterh i don't think logging should be mixed with out of band control of sockets
[14:23] pieterh that's a different issue
[14:23] sustrik right
[14:23] sustrik but it's an obvious trap
[14:24] pieterh avoidable trap
[14:24] sustrik i think someone have already suggested something like that
[14:24] pieterh yes
[14:24] sustrik i can avoid it by saying "NO"
[14:24] sustrik as i am core maintainer atm
[14:24] pieterh indeed
[14:24] pieterh but that's not going away, your veto
[14:24] sustrik no mixing of business logic and administration
[14:25] pieterh anyone can fork 0MQ and add whatever they like
[14:25] pieterh but it won't be the official version
[14:25] sustrik sure, that's ok
[14:25] pieterh unless and until you vote it in
[14:25] pieterh no exceptions to this, period.
[14:25] guido_g are we still at the logging thing or do i have to move myself to a safe place?
[14:26] sustrik good, so unless i hear a good argument for mixing business logic and maitenence/debugging stuff, i would keep it apart
[14:26] sustrik it's logging still :)
[14:26] pieterh sustrik: i'd like, really love, to be able to do logging from a separate process
[14:26] pieterh otherwise every application becomes N x more complex
[14:26] sustrik write an app that does that
[14:26] sustrik np
[14:26] sustrik 5 lines of code
[14:27] pieterh you said it was in process
[14:27] pieterh did I misunderstand you?
[14:27] sustrik you can create a device that would read from sys and republish on tcp
[14:27] sustrik you can even use forwarder
[14:28] sustrik when you do that you are effectively converting 0mq admin data into business logic data
[14:28] sustrik which is ok
[14:28] sustrik but at 0mq level i need them separated
[14:28] sustrik to avoid contamination
[14:28] guido_g if anyone can access the *internal* data flow w/o authorization it would be a nightmare
[14:29] sustrik you mean the logs?
[14:29] guido_g yes
[14:29] sustrik :(
[14:29] sustrik no authorisation as for now
[14:29] sustrik otoh
[14:29] sustrik it's in-process only
[14:29] sustrik so nobody from outside can read from sys
[14:29] guido_g given the "one forwarder per program" model, the program decides on what to publish and where etc.
[14:30] guido_g sustrik: sure, but as far as i understood, pieterh wanted access from an external process w/o a forwarder inbetween
[14:30] sustrik yes, that was the original discussion
[14:31] sustrik actually it would be almost impossible from implementation point of view
[14:31] sustrik that's black magic going on in the background :)
[14:31] guido_g details! :)
[14:31] sustrik well, sys is technically different from all the other sockets
[14:31] sustrik maybe pgm excluded
[14:32] sustrik becasue it's a real bus
[14:32] sustrik ie. multiple writers
[14:32] sustrik multiple readers
[14:32] sustrik no device in the middle
[14:32] guido_g as i said "<guido_g> but the application developer doe *not* know what makes sys: work technically, so the obvious similarity to inproc might be wrong"
[14:32] guido_g therefore the sys: transport makes sense in my eyes
[14:33] sustrik yes, that was my argument as well
[14:33] guido_g good
[14:33] guido_g because i really like the logging
[14:33] sustrik it's good but can be dangerous if misused
[14:34] guido_g ømq is notoriously weak on giving information on what's going on
[14:34] sustrik :|
[14:34] guido_g that's a thing nearly every library has
[14:34] sustrik right
[14:34] sustrik that's because it's much more async that probably any other library around
[14:35] sustrik zillions of moving parts
[14:35] sustrik even without proper synchronisation
[14:35] guido_g and for real distributed applications there is rule 0: log everything, all the time
[14:35] sustrik right
[14:35] sustrik so, this should solve it
[14:35] sustrik hopefully
[14:35] guido_g it's at least a clean interface
[14:36] pieterh sustrik: makes sense but it's a shame that we'd need a dedicated thread in every app to export log data
[14:36] pieterh still, this can be extended over time
[14:36] pieterh minimalism demands I shut up now :-)
[14:36] pieterh though a new question comes to mind...
[14:36] pieterh if a bus transport is useful for 0MQ internally
[14:37] pieterh surely it is also useful for applications as well
[14:37] pieterh :-)
[14:37] sustrik it doesn't scale
[14:37] sustrik it can work on level on 1 process
[14:37] sustrik even on LAN
[14:37] sustrik once you try to stretch it further it breaks
[14:38] sustrik the number of transfers grows quadratically with increasding number of components
[14:38] sustrik pieterh: ad dedicated thread
[14:38] pieterh sustrik: as an inproc bus, I meant
[14:38] sustrik yes, that's why it's a bus
[14:38] pieterh bus://
[14:38] sustrik to avoid speacial thread
[14:39] sustrik yes, can be
[14:39] sustrik but i like that sys is explicit abour "don't pass your business logic through this"
[14:39] pieterh for some reason 'magic system stuff you use here' seems not ideal
[14:40] pieterh it's like the core gets to use magic stuff the apps don't get
[14:40] sustrik yes
[14:40] guido_g that's why it's the core
[14:40] sustrik because core doesn't have to scale
[14:40] sustrik applications do
[14:41] pieterh imo you will get people demanding access to this transport for their essential internal signalling
[14:41] pieterh but i have no proof, just baseless speculation
[14:42] sustrik i expect it will be so
[14:42] sustrik thus i want it be named something really scary
[14:42] guido_g but the line is drawn "sorry, no access for you here"
[14:42] pieterh e.g. "how do I do a clean shutdown"... answer: signal every thread
[14:42] guido_g as with system calls into the kernel
[14:42] pieterh bah. that was the idea with PAIR as well
[14:42] sustrik that's baked into 0mq as well
[14:42] sustrik ETERM
[14:42] pieterh but it turns out we really need PAIR for some use cases
[14:43] sustrik ok, the use case for pair:
[14:43] pieterh yeah, i know, it's a bad example
[14:43] sustrik you have an application that does accounting and HR
[14:43] sustrik you want to split it into two applications
[14:43] sustrik one does accounting
[14:43] sustrik other does HR
[14:43] sustrik they have a PAIR socket between them
[14:44] sustrik because each of the peer is singleton
[14:44] sustrik and doesn't scale
[14:44] pieterh my use case for PAIR is just interthread signalling
[14:44] sustrik hm
[14:44] sustrik what use case?
[14:44] pieterh i explained it in the guide, end of chapter 2
[14:45] pieterh thread a wants to signal to thread b that it's done
[14:45] pieterh no scaling required
[14:45] sustrik let me see
[14:45] pieterh no reconnection required
[14:45] pieterh PAIR gives the best semantics afaics
[14:48] sustrik this one: "Thread Coordination" ?
[14:51] sustrik well, there's no business logic in the example
[14:52] sustrik so it's hard to say what's the right socket types to use
[14:52] sustrik it the three components are by definition singletons, then PAIR is appropriate
[14:52] sustrik it say step2 can be parallelised the pipeline would be better
[14:53] sustrik the problem i have with pair is that it solves a problem (the accounting vs. HR problem)
[14:53] sustrik but it doesn't scale
[14:54] sustrik by throwing more boxes on your accounting/HR application you won't get more power
[14:54] sustrik even thought you paired the two by a PAIR socket
[15:29] guido_g sustrik: latest master shows poll problem again
[15:30] guido_g test code is at http://github.com/guidog/cpp/tree/master/zmqcpp/
[15:31] guido_g output of zmqcpp when sender was executed: http://gist.github.com/560863
[16:11] ModusPwnens hello
[16:18] ModusPwnens i have a question about the zmq_msg_size function
[16:26] cremes ModusPwnens: go ahead and ask
[16:26] cremes if someone knows the answer, they'll respond
[16:26] ModusPwnens ok
[16:26] ModusPwnens so as i have said before, I am using google protocl buffers to encode messages and send them
[16:27] ModusPwnens and on the clientsize, I do message.size(), where message is the size of the object encoded by google protocol buffers
[16:27] ModusPwnens however, on the server side
[16:27] ModusPwnens I use the zmq_msg_size function
[16:27] ModusPwnens but no matter how large the message actually is, zmq_msg_size always gives 36 bytes
[16:28] ModusPwnens I am wondering not only why that is, but how to get the actual size of the payload of the message
[16:28] cremes ok
[16:28] cremes it sounds to me like your client is not allocating the message correctly
[16:29] cremes do you have a code snippet showing how you are allocating the zmq_msg_t and assigning your protocol buffer to it?
[16:29] ModusPwnens rc = zmq_msg_init_data(&msg, cstr, message.size(), NULL, NULL);
[16:29] ModusPwnens something like that?>
[16:30] cremes hold on a sec...
[16:30] ModusPwnens k
[16:32] cremes the variable 'cstr' points to the data; are you sure its size is the same as message.size() ?
[16:32] Steve-o shouldn't it be message.ByteSize() in Google protobufs?
[16:34] ModusPwnens hmm, that's a good point. I wasn't aware that function existed..
[16:34] ModusPwnens but in any case, that doesn't seem like it would solve my problem.
[16:35] cremes does the server *always* receive packets of length 36 exactly? if so, the client must be producing messages of that length
[16:35] cremes focus on the client side for now... something is likely wrong there
[16:35] cremes on the server, are the bytes received garbage? try printing the data the client sends and comparing it to the data the server receives
[16:36] ModusPwnens It seems so, but when i print the value of message.size() in the client, it prints outs omething much larger
[16:36] ModusPwnens and the bytes received are not garbage
[16:36] ModusPwnens i have verified the message coming through is valid
[16:37] cremes why not try: rc = zmq_msg_init_data(&msg, cstr, strlen(cstr), NULL, NULL);
[16:37] cremes i don't understand the relationship between cstr and message.size()
[16:38] ModusPwnens Hmm, i changed that for a reason...
[16:38] ModusPwnens I think it's because the zmq syntax doesn't want a string there or something
[16:38] ModusPwnens so I had to convert a string into a c-style string
[16:38] ModusPwnens and thats what cstr is
[16:39] cremes ah ha... that is your problem
[16:39] ModusPwnens it is?
[16:39] cremes c style strings are terminated by a NUL \0
[16:39] ModusPwnens Yeah. Does that interfere with zeromq somehow?
[16:39] cremes hmmm, wait, i got ahead of myself there...
[16:40] cremes i'm going to think out loud
[16:40] ModusPwnens ok
[16:40] cremes so a c style string is really just a char* or a pointer to a byte buffer
[16:40] ModusPwnens correct
[16:40] ModusPwnens oh right
[16:40] cremes you need to pass a byte buffer pointing to the beginning of your encoded message
[16:40] ModusPwnens sorry you're thinking out loud >_<
[16:41] cremes is the cstr pointing to that?
[16:41] ModusPwnens yeah
[16:41] ModusPwnens and then i strcpy the message into cstr
[16:41] cremes e.g. cstr = (char*) message.encode().get_pointer()
[16:41] ModusPwnens beforehand
[16:41] ModusPwnens yeah something like that
[16:41] cremes any embedded NULs will terminate the string copy
[16:42] cremes so you may only be copying 36 bytes even though your byte buffer is larger
[16:42] cremes use memcpy instead
[16:42] cremes does that sound reasonable?
[16:42] ModusPwnens but..wouldn't that reflect on the serverside as far as displaying the message?
[16:42] ModusPwnens i.e. it wouldn't display the whole thing?
[16:43] ModusPwnens furthermore
[16:43] ModusPwnens embedded nuls will terminate the string copy
[16:43] ModusPwnens but doesnt' that only apply to the encoded message that is being copied?
[16:43] cremes ModusPwnens: you are going to have to show more code, i think
[16:44] ModusPwnens ok i will just copy the entire function into a pastebin
[16:44] cremes there are lots of folks successfully slinging around large strings and large byte buffers so there is likely an issue in your code
[16:44] cremes cool
[16:44] ModusPwnens That is definitely a possibility
[16:44] cremes you are using 2.0.8 release, yes? or the 2.1 master branch?
[16:44] ModusPwnens 2.0.7
[16:45] CIA-20 zeromq2: 03Martin Sustrik 07master * r99ddfa7 10/ (builds/msvc/platform.hpp configure.in): maint: will become 2.0.9 - http://bit.ly/dgicJb
[16:45] CIA-20 zeromq2: 03Martin Sustrik 07master * r47aaf10 10/ :
[16:45] CIA-20 zeromq2: Merge branch 'maint'
[16:45] CIA-20 zeromq2: * maint:
[16:45] CIA-20 zeromq2: maint: will become 2.0.9
[16:45] CIA-20 zeromq2: Conflicts:
[16:45] CIA-20 zeromq2: builds/msvc/platform.hpp
[16:45] CIA-20 zeromq2: configure.in - http://bit.ly/9JfwEL
[16:45] CIA-20 zeromq2: 03Mikael Helbo Kjær 07master * r59315eb 10/ src/select.cpp :
[16:45] CIA-20 zeromq2: Erasure of retired fd's in select.cpp causes an assertion in MSVC 2008 STL
[16:45] CIA-20 zeromq2: I was hitting an issue with an SCL enabled STL library in connection with the
[16:45] CIA-20 zeromq2: way select_t::loop was erasing retired fd's. The problem as identified by the
[16:45] CIA-20 zeromq2: SCL assertion was that by the time the iterator given to the erase method was
[16:45] CIA-20 zeromq2: called it was considered invalid by the library. I am not sure this isn't just
[16:45] CIA-20 zeromq2: a "quirk" of the MSVC STL library as the other code looks valid to me as well. - http://bit.ly/dzUsD3
[16:45] CIA-20 zeromq2: 03Martin Sustrik 07master * ra81a373 10/ src/select.cpp :
[16:45] CIA-20 zeromq2: Merge branch 'maint'
[16:45] cremes 2.0.7 should be okay too; i don't recall any data truncation bugs for any sockets
[16:46] ModusPwnens http://pastebin.com/NMyvuBmn
[16:46] ModusPwnens if i am doing anything stupid in general
[16:46] ModusPwnens please let me know
[16:46] ModusPwnens I am actually only a stupid so im not as experienced as most people here probably are..
[16:47] ModusPwnens wow
[16:47] ModusPwnens i meant student
[16:47] cremes line 36 confuses me...
[16:47] ModusPwnens oh yeah ignore that
[16:47] cremes is that supposed to be commented out?
[16:48] cremes because it is overwriting the size of the msg on the prior line
[16:48] ModusPwnens I think so. We put that in for debugging purposes.
[16:48] cremes with 36 bytes which matches up with how many you are receiving on the server side
[16:49] cremes well, that's a bug
[16:49] cremes remove that and test again
[16:49] ModusPwnens *sigh* i hate asking stupid questions
[16:51] cremes don't worry about it; you'll be repaying the favor here in a few short weeks
[16:51] ModusPwnens ahhhhh gosh darn it
[16:51] ModusPwnens that was it
[16:51] ModusPwnens I'm sorry for wasting your time.
[16:51] ModusPwnens thanks for your help tho
[16:51] cremes btw, line 34 looks to me like a bug waiting to happen
[16:51] cremes i would use memcpy
[16:52] ModusPwnens because of the nul string thing?
[16:52] cremes right
[16:53] cremes memcpy(cstr, message.c_str(), message.size())
[16:53] ModusPwnens Ok, i will change it.
[16:53] cremes or use strncpy
[16:53] cremes but memcpy is likely quicker since it isn't looking for NULs
[16:53] cremes ModusPwnens: and all this from a Ruby guy who hasn't used C in anger for many years... :)
[16:54] ModusPwnens heh, well it looks like you still got it then.
[16:54] ModusPwnens oh, and i have another question this time
[16:54] ModusPwnens for the send and receive topology
[16:55] ModusPwnens is there a flag you can set or something that prevents the send and receive messages from blocking?
[16:55] cremes yes
[16:56] cremes check out ZMQ_NOBLOCK; http://api.zeromq.org/zmq_send.html
[16:56] cremes btw, when using PUB sockets, it's a good idea to put your topic in a separate message part
[16:57] ModusPwnens what do you mean?
[16:57] cremes the example code you pastied showed that you are sending the AddressBook as the first message part, so subscribers will match on it
[16:57] cremes i'll pastie some pseudo-code showing what i mean
[16:57] ModusPwnens ok.
[16:58] ModusPwnens and as far as the no blocking, we want to use it so we can constantly send messages and then receive a message every once in a while, or vice-versa. But would it be better to just use multiple sockets and a publish-subscribe topology instead?
[16:58] jond cremes: if it's std::string why not use the copy method std::string::copy?
[16:59] mikko do you actually need a copy there?
[16:59] ModusPwnens well i need to store the message I am sending in that string, so I would need to copy it in, right?
[17:01] cremes ModusPwnens: http://pastebin.com/EEe7zujL unlikely legal C, but you should get the idea
[17:01] cremes jond: i don't know; i'm not a C++ guy but memcpy is pretty easy to figure out for me
[17:02] cremes mikko: you probably need the copy unless you can guarantee the buffer won't go away until the message is *actually* sent
[17:02] ModusPwnens Hmm, that makes sense.
[17:02] Steve-o Does it really need to be a string? I used buf.SerializeToArray() and buf.ParseFromArray().
[17:03] cremes ModusPwnens: by sending a topic, your SUB sockets can use it to filter out unwanted messages
[17:04] ModusPwnens Yes that makes sense. I can see the usefulness of that.
[17:04] ModusPwnens And steve-o, you make a good point
[17:04] jond cremes: ok, there is a method on string, so you could do message.copy(cstr,message.size()), but the code id as you say mainly c
[17:04] mikko ModusPwnens: is that C++ or C ?
[17:05] cremes jond: i'm a ruby guy so take my C suggestions with a grain of salt ;)
[17:05] ModusPwnens well, I am using visual studio C++, but I am actually using the C bindings
[17:05] mikko cremes: yeah, true
[17:05] mikko on the earlier
[17:05] ModusPwnens mostly because I was having trouble with the C++ ones
[17:06] ModusPwnens and steve-o, that would probably be a better/more efficient way of doing this, right?
[17:06] ModusPwnens then I wouldn't have to do all the c-string manipulation
[17:07] Steve-o it should be the best as of v2.1, but I think Google has changed their api a few times since
[17:07] Steve-o my example of using protobufs here, http://code.google.com/p/openpgm/source/browse/trunk/openpgm/pgm/examples/pgmping.cc
[17:09] Steve-o noting ByteSize() is only valid after you have actually filled in values for protobufs, it has caught me a few times
[17:09] ModusPwnens you mean after you have encoded a message?
[17:10] Steve-o I needed the message size before serializing
[17:11] ModusPwnens hmm
[17:11] ModusPwnens for the serializetoarray method
[17:12] ModusPwnens you would need to get the size of the data you are serializing before you can serialize it..?
[17:13] Steve-o For my code I need to reserve space in a socket buffer for the serialized form, so I need the size. Obviously usage case varies for everyone.
[17:14] ModusPwnens You mean the size is always known beforehand? What if you don't know the size before, can you still use that?
[17:15] Steve-o probably if you know the protobuf object is always going to be smaller than your buffer
[17:15] ModusPwnens Hmm. okay.
[17:15] Steve-o it's a bit safer to check the size before hand, much like vasprintf()
[17:19] ModusPwnens Yeah that makes sense.
[17:19] ModusPwnens Ok, thanks for your help guys!
[17:20] ModusPwnens I will probably be back again in a few days with something else..hopefully something less stupid.
[17:55] pieterh folks: if anyone wants to review http://www.zeromq.com, feedback would be welcome
[17:55] pieterh this is meant to be the welcome site for new 0MQ users
[17:56] pieterh next I'll remove a lot of the fluff from the zeromq.org front page and menus
[18:00] pieterh my idea is to make the zeromq.org site an index of all projects
[18:01] pieterh kind of ranked by activity somehow
[18:05] pieterh no-one minds? :-)
[18:12] Steve-o inproc is still a new word
[18:13] guido_g the onhover bold is a little overdone
[18:16] guido_g on the page linked from 04READ should show the canonical doku first (the guide & the manual)
[18:16] jond steve-o: agree about inproc, not widely used AFAIK
[19:44] mcxx I'm trying to build pyzmq, but I get this error: http://pastebin.ca/1930723
[19:45] mcxx I got freshly cloned zeromq2 and pyzmq repos from github

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy