Thursday March 3, 2011

[Time] NameMessage
[01:16] cremes ha, figured out why my zmq_connect() calls were returning an error
[01:17] cremes i was trying to connect two sockets in *different* contexts via inproc
[01:17] cremes that doesn't work :)
[02:58] cremes i'm digging this inproc transport
[02:58] cremes it reduces latency by at least 30%... me likey
[03:07] Steve-o just need a Windows version
[06:51] Steve-o mikko: pingu
[06:58] Steve-o mikko: ready for bumping zeromq with autoconf of openpgm
[07:44] guido_g mornig all
[07:44] pieterh guido_g: g'morning
[07:44] guido_g hi pieterh
[07:44] guido_g just saw that you've been busy
[07:46] guido_g will have a look at the mdp broker later
[07:48] pieterh it does everything but leaks a little memory, and is slow
[07:50] guido_g "The biggest improvement in performance is the non-working-to-working
[07:50] guido_g transition."
[07:50] guido_g -- John Ousterhout
[07:50] pieterh what's nice is how it all handles when you stop/restart pieces
[07:51] guido_g will use it as counterpart for my python mdp implementation
[07:52] pieterh I'd start by implementing the client and worker APIs
[07:52] pieterh you can then test against the existing broker
[07:53] guido_g that was the idea
[07:53] guido_g i'm on the client atm
[07:53] pieterh I'm keen to see what you come up with
[07:53] pieterh mdp will also solve the problem of reliable pipelines
[07:54] pieterh today I profile to see where the CPU time is going
[07:54] pieterh I suspect too much envelope stuffing
[07:54] guido_g my idea is not to write another mdp example, but a lib that can be used for apps
[07:54] pieterh this is what the two APIs are meant to be
[07:55] guido_g so i figured that right at least :)
[07:55] pieterh so an actual client looks like:
[07:55] pieterh and the worker:
[07:56] guido_g sweet
[07:56] pieterh yeah, I think so too... :-)
[07:56] guido_g no really
[07:56] pieterh especially since it's like 3-4 days' total work so far from concept to running code
[07:57] guido_g as one who is not used to c anymore, this is very easy to read
[07:57] pieterh well, it's not about the language, just the semantics you implement
[07:57] pieterh the class-based C style we developed at iMatix is pleasant to work with
[07:58] pieterh i'm curious to see how that maps into Python
[07:58] guido_g sure, but you need to know a language quite well to come up w/ a good api for something
[07:58] pieterh indeed
[08:13] yrashk I forgot this again -- in the latest 0mq can you or can you not use sockets in diff threads?
[08:13] yrashk this was getting confusing at some point
[08:13] guido_g you can, but you should do it *very* carefully
[08:13] guido_g as always when using threads
[08:13] yrashk what consitutes carefulness in this case?
[08:14] sustrik no parallel access
[08:14] yrashk so if mutexed around, we're fine?
[08:15] eyecue moin
[08:15] guido_g which means the same usage-pattern as before, but you can migrate a socket from thread to another
[08:15] guido_g yrashk: no
[08:15] guido_g yrashk: doesn't make sense
[08:15] sustrik guido_g: it would work
[08:16] sustrik basically it means you migrate the socket each time the mutex is locked
[08:16] eyecue pieterh; nice coincidence, a vendor we're engaging today uses zeromq to arbitrate their mail and messaging infrastructure :)
[08:16] sustrik yrashk: it's going to be slow
[08:16] yrashk sustrik: yeah
[08:16] yrashk mutexes add a penalty
[08:16] sustrik yes
[08:16] guido_g sustrik: technically yes, but you should be the one pointing out that it is broken by design and start to protect the innocent developers...
[08:16] pieterh eyecue: nice
[08:16] eyecue pieterh; taguchimail :]
[08:16] sustrik exactly
[08:16] eyecue in case youre interested
[08:17] sustrik that's why we added scary stuff about memory barriers into docs :)
[08:17] guido_g obviously didn't work
[08:17] guido_g ]:->
[08:17] eyecue i might need to pick some irc brains soon for how i can apply 0mq to our stuff
[08:17] pieterh sustrik: problem is that 'memory barriers' isn't very explanatory
[08:17] guido_g i'm available for hire ,)
[08:17] sustrik that's the point
[08:18] sustrik for most people it translates to "don't do this"
[08:18] pieterh sustrik: doesn't help to be mysterious
[08:18] sustrik which is exactly the message
[08:18] pieterh passing sockets from thread to thread is a valid pattern
[08:18] pieterh bleh
[08:18] eyecue pieterh; would this sentence make sense: i would get a developer to build a daemon'able queueing application using the 0mq library and add support for some form of persistant storage?
[08:19] pieterh eyecue: depends whom you're talking to
[08:19] sustrik pieterh: we can remove the memory barrier text
[08:19] pieterh sustrik: the explanation needs to be precise
[08:19] sustrik but actually, you *do* have to do the memory barrier
[08:19] pieterh you can create a socket in one thread and pass that to another thread
[08:19] eyecue pieterh; in the strictest sense that 0mq is a programming 'library' which one can use to then build daemons (instead of say a client app)
[08:19] pieterh you MUST NOT read/write/close the same socket in multiple threads
[08:19] sustrik the thing is that in 99% of cases it's done for you
[08:20] sustrik so you don't have to care about it
[08:20] eyecue pieterh; compared to say your beanstalkd's, which implement and provide a daemon out of the box
[08:20] pieterh sustrik: not one person has hit a problem related to memory barriers afaics
[08:20] yrashk the proble is that we DON'T WANT to do this
[08:20] pieterh eyecue: yes, 0MQ is a toolkit with which you build frameworks
[08:20] yrashk but we have to deal with the way erlang uses schedulers
[08:20] sustrik right, so should we remove the text?
[08:20] yrashk each scheduler is a differnet OS thread
[08:20] eyecue pieterh; ta :]
[08:20] yrashk and we can't tell Erlang to use one specific scheduler only
[08:20] pieterh eyecue: look at for an example, that's 3-4 days from concept to running code
[08:21] pieterh with full APIs etc, and in C
[08:21] eyecue roger that
[08:21] pieterh that's less time than it takes to *learn* a conventional messaging API
[08:21] eyecue pieterh; i read the guide last night, and the most intriguing part was the self-healing slash self adapting node network concept
[08:21] pieterh sustrik: any text in the docs has to reference people's real questions and needs
[08:22] eyecue pieterh; yeh, i got one of our django guys to take a look at 0mq today
[08:22] pieterh sustrik: if you like I'll review that man page and propose changes
[08:22] eyecue pieterh; i have a particular problem to solve where i dont know what the load of the input system will be ahead of time, nor do i want to care necessarily about the message/payload size (email bodies * mailing list members)
[08:23] sustrik sure, give it a try
[08:23] eyecue pieterh; i want to avoid serially queueing campaigns, and i dont really care about fair queueing, so im toying with the concept of an abstract fifo queue per campaign. the issue that i have is the idea that any campaign can 'start' or be scheduled at any time
[08:23] pieterh eyecue: my advice would be to ask here for someone to spend a short time training & helping your team use 0MQ
[08:23] sustrik reference should be complete though, so the memory barrier issue should be at least mentioned
[08:23] eyecue pieterh; we're in .au, know anyone here? :)
[08:23] sustrik maybe in a note?
[08:24] eyecue pieterh; id be more than happy to entertain that idea
[08:24] eyecue *reads mdp*
[08:25] pieterh sustrik: do we know anyone in .au?
[08:25] sustrik let me see
[08:25] eyecue do you have have an active/developed advocacy network/framework/contact list going?
[08:26] sustrik i have kind of vague feeling that we do, but not sure
[08:26] eyecue pieterh; well, apart from taguchimail ;) i may go and pick his brain a bit (since we're already paying him)
[08:26] eyecue haha very cute -> The Majordomo pattern has no relation to the open source mailing list software with the same name.
[08:27] pieterh eyecue: someone pointed out the risk of confusion
[08:27] eyecue that was me last night :]
[08:27] eyecue too much beer? :]
[08:27] pieterh ah!
[08:27] pieterh hey, it's all one big blur...
[08:27] eyecue amen, and so it should be
[08:28] eyecue actually mdp may well solve for our problem domain
[08:28] eyecue each worker can say how many active campaigns are available to dequeue
[08:28] eyecue it could assign itself one of the 'service names'
[08:28] pieterh eyecue: yes, mdp seems to cover a lot of problems
[08:28] eyecue ensuring a 1:1
[08:29] sejo if I would want to persist all the messages (untill they are processed) what backend would you suggest (need a fast backing)
[08:29] eyecue pieterh; i could then interleave/loadbalance the MTA injection form that point
[08:29] pieterh sejo: it's not that simple but you might look at Tokyo Cabinet or similar
[08:29] sejo pieterh: tokio cabinet, mongodb, couchdb etc?
[08:29] eyecue hmm, could even spawn new workers based on workload that mdp knows about
[08:30] eyecue id say threadpool, but i only mean in concept.
[08:30] eyecue this may be a kicker though -> Workers are idempotent, i.e. it is safe to execute the same request more than once.
[08:30] pieterh eyecue: that's a matter of shared database
[08:30] eyecue we need to ensure non-duplicate delivery of mail/messages
[08:31] pieterh sure
[08:31] eyecue im not sure how it applies, but yeh
[08:31] guido_g sejo: redis for speed only, if you have the memory
[08:31] sejo guido_g: needs to be persistable and distributed :p
[08:31] sejo (not that I'm asking a lot :p)
[08:32] guido_g sure, go read
[08:32] pieterh sejo: what're the performance requirements?
[08:32] guido_g sejo: mongodb is quite fast too
[08:32] pieterh eyecue: if you're up for sponsoring work on MDP and/or broker implementations, drop me a line
[08:32] sejo pieterh: 100-500 reqs a second
[08:32] sejo might even go up
[08:32] pieterh sejo: can you wait a week or two? I'm working on a rust-based reliability pattern
[08:33] pieterh fire and forget
[08:33] eyecue pieterh; i was just about to ask you about 'pausing' queues, or something semantically similar. i note that some of your socket types block or drop, depending on which is chosen, so that may be a way
[08:33] pieterh client sends to rust-based broker
[08:33] pieterh broker then sends to workers
[08:33] pieterh whole thing is brute-force ack'd
[08:33] sejo cool, looks what I need, but sorry no time for wait (startup here and my money isn't unlimited)
[08:34] pieterh sejo: where do you want your persistence, in client, or in center?
[08:34] sejo on the broker
[08:34] pieterh ugh
[08:34] pieterh that's the worst choice
[08:34] sejo worker should just execute and send results
[08:35] pieterh so the problem here is that when you want the broker to hold the state
[08:35] sejo well not much states just a group of key-value's
[08:35] eyecue im out, thanks for the help pieter
[08:35] pieterh you need extra work to speak to that state reliably
[08:35] pieterh eyecue: ciao
[08:36] pieterh sejo: if you place the persistence in the client API, for example
[08:36] pieterh you can use dumb brokers
[08:37] sejo ok, basicly i need a group of brokers (that share data) which is pulled by workers, the same workers will push new tasks generated from the result to the broker. State is kept in worker, the minute it reaches the broker, only the messagedata is state
[08:37] pieterh sure
[08:38] sejo btw redis is master-slave
[08:39] pieterh sustrik: random question about style
[08:39] sejo couchdb is master:master
[08:39] pieterh the man pages are written in the form of specifications
[08:39] pieterh (which is excellent IMO)
[08:40] pieterh to make this more contractual, we could use
[08:41] pieterh e.g. "Applications MAY create a socket in one thread with _zmq_socket()_ and then pass it to a _newly created_ thread as part of thread initialization, for example via a structure passed as an argument to _pthread_create()_. Applications MUST NOT do stupid stuff."
[08:44] sejo nice did you guys use couchdb already?
[08:45] sejo looks pretty good
[08:54] pieterh sustrik: ok, patch to zmq_socket sent
[08:54] pieterh that should help IMO
[08:56] djc sejo: couchdb is awesome, we use it a lot at work
[08:57] sejo djc: Yeah going to use it also
[08:57] sejo already started playing with it :p
[08:57] sejo =80
[08:58] sustrik pieterh: does that help in any way?
[08:58] sustrik it doesn't answer yrashk's question
[08:58] pieterh does what help?
[08:58] sustrik the patch
[08:58] pieterh what's yrashk's question? sorry, I missed something
[08:58] sustrik the one that started the disussion
[08:58] sustrik whether sockets can be used from 2 threads
[08:59] pieterh " can you or can you not use sockets in diff threads?"
[08:59] pieterh I think the patch makes this very clear
[08:59] sustrik then the argument was that speaking about memory barriers is not clear
[08:59] pieterh sure
[08:59] pieterh the man page started by talking about contexts
[08:59] pieterh now it says "sockets are not thread safe. period".
[08:59] sustrik and that haven't changed
[09:00] sustrik ?
[09:00] sustrik +0MQ 'sockets' are _not_ thread safe. Applications MAY create a socket in one
[09:00] sustrik +thread with _zmq_socket()_ and then pass it to a _newly created_ thread as
[09:00] sustrik +part of thread initialization, for example via a structure passed as an
[09:00] sustrik +argument to _pthread_create()_. Applications MUST NOT otherwise use a socket
[09:00] sustrik +from multiple threads except after migrating a socket from one thread to
[09:00] sustrik +another with a "full fence" memory barrier.
[09:00] pieterh the first sentence is the most important
[09:00] sustrik yep, that's ok
[09:00] pieterh the second sentence provides the ONE valid use case
[09:00] pieterh the third sentence explains it for those who care
[09:00] pieterh and fourth, some bla blah about contexts
[09:00] sustrik why singling out one valid use case?
[09:01] pieterh it's the only one I know of
[09:01] pieterh for normal apps
[09:01] sustrik that's more of a guide stuff
[09:01] pieterh that was already in the man page... I just trimmed it a little
[09:01] pieterh plus it really does need to be there
[09:01] guido_g ha! client just passed first send unittest
[09:01] sustrik you mean the memory barrier stuff?
[09:02] sustrik yes
[09:02] sustrik the one use case should be moved to guide imo
[09:02] pieterh sustrik: please, no
[09:02] pieterh don't make life harder for users than it has to be
[09:02] pieterh i agree that this is explanatory and not a specification
[09:02] pieterh but this is so important for MT apps
[09:03] pieterh you can't afford to hide it somewhere in a 1000-page book
[09:03] sustrik i mean there are many use cases, so singling one of them out gives wrong impression
[09:03] sustrik what about making it an example?
[09:03] pieterh there are not many use cases!
[09:03] pieterh sorry,
[09:03] pieterh that's just not accurate
[09:03] pieterh i've hit precisely one, in 40-50 examples that cover every angle
[09:04] pieterh you may imagine use cases, that's not the same
[09:04] sustrik garbage collecting the socket is pretty common
[09:04] pieterh if you fix the inproc connect/bind issue, this use case disapepars
[09:04] pieterh *disappears
[09:04] pieterh that would be ideal IMO
[09:05] pieterh only 5 people, globally, will ever write a socket garbage collector
[09:05] pieterh maybe 10, ever
[09:05] sustrik wait a sec, you are saying that this use case is *necessary*?
[09:05] sustrik how so?
[09:05] pieterh yes
[09:05] sustrik you can pass context to the other thread
[09:05] pieterh because there's no other way to create a working inproc socket pair
[09:05] sustrik and create the socket there
[09:05] sustrik no?
[09:05] pieterh due to the bind/connect issue
[09:05] pieterh sustrik: have you read the Guide?
[09:05] sustrik yes
[09:06] pieterh read it again until you understand 0MQ
[09:06] pieterh :-)
[09:06] pieterh heh
[09:06] sustrik what's wrong with this:
[09:06] sustrik c = context()
[09:06] sustrik s = socket (c);
[09:06] sustrik s.bind (...);
[09:06] sustrik pthread_create (c);
[09:07] sustrik and in the worker thread:
[09:07] sustrik s = socket(c);
[09:07] sustrik s.connect (...)
[09:07] pieterh the example in the Guide has 3 stages
[09:07] sustrik same thing, no?
[09:08] pieterh let me double check
[09:08] pieterh if there is a valid pattern, this use case disappears
[09:09] pieterh ok, you're right afaics
[09:09] pieterh bind before creating child threads
[09:09] pieterh I need to change some stuff in the guide
[09:10] pieterh we can indeed remove that example from the man page... hang on a sec then...
[09:12] pieterh sustrik: patch sent, this is much cleaner
[09:12] sustrik ok, thanks
[09:13] sustrik pieterh: you've sent the old patch i think
[09:14] sustrik looks the same
[09:14] pieterh git did something... hang on
[09:15] pieterh weird, git produces the wrong patch
[09:16] sustrik never mind, i can do it myself
[09:16] sustrik it's just reversing two paragraphs
[09:16] pieterh sigh, just use this paragraph:
[09:16] sustrik and adding a period
[09:16] sustrik right?
[09:16] pieterh 0MQ 'sockets' are _not_ thread safe. Applications MUST NOT use a socket
[09:16] pieterh from multiple threads except after migrating a socket from one thread to
[09:16] pieterh another with a "full fence" memory barrier.
[09:16] pieterh and move the context stuff below
[09:17] pieterh i forgot a 'git add'
[09:17] pieterh too many little bitty steps
[09:18] pieterh BTW "full fence" is the proper jargon afaics
[09:18] sustrik possibly
[09:18] sustrik btw, we've changed the wording a bit
[09:19] sustrik but haven't addressed the original issue
[09:19] sustrik that speaking about 'fences' is mysterious
[09:19] sustrik rather than helpful
[09:19] pieterh the question was 'are sockets thread safe'?
[09:19] pieterh and the answer is 'no'... how is that not answering it?
[09:19] sustrik the docs said so even before
[09:19] pieterh they hid that in a lot of other text
[09:20] pieterh obviously people didn't see it clearly
[09:20] sustrik ok
[09:20] sustrik the period helps
[09:21] pieterh shrug, I'm not sure what you're asking, yrashk's question was clear, and it seems clear the man page had way too much wrapping around the essential statement, "don't do it, but if you must, use full fence memory barriers"
[09:23] sustrik well, the only real change seems to be the period; what i'm asking is: should it say something like "you can use socket from multiple threads given you synchronise the access"
[09:24] pieterh people will start using mutexes all over the place
[09:24] sustrik it's antipattern in most cases, but technically, the statement is sound
[09:24] sustrik exactly
[09:24] sustrik that was the original discussion, whether it's ok to scare people using terms like "memory barrier"
[09:25] sustrik or rather be technically precise
[09:25] pieterh i don't think fear is a valid tool
[09:25] pieterh this is a contract
[09:25] pieterh it should simply state what is allowed, and what is not
[09:25] sustrik ok, then it should be "you can use socket from multiple threads given you synchronise the access"
[09:26] sustrik that's more comprehensible than memory barrier stuff
[09:26] pieterh every contract aims to force the signer to behave in some way
[09:26] pieterh look, we don't want people to share sockets between threads, period
[09:26] pieterh it's the cause of repeated failures
[09:27] sustrik ok, good
[09:27] pieterh we see one or two bizarre 0MQ crash reports a week due to this
[09:34] pieterh I'm ripping out all explanation of socket migration from the guide, we'll use the pattern you explained, bind before starting child thread
[09:49] sejo if you have a zmq.PULL and recv() something, is it possible to know what worker sent that?
[09:49] pieterh sejo: you have to add the information yourself to the message
[09:50] pieterh otherwise, use a ROUTER (XREP) socket instead
[09:50] sejo pieterh: ok good! thanks!
[09:50] sejo I think it'll be better to put it in the message
[09:52] pieterh sustrik: so, I've removed all examples of socket migration from the guide
[09:53] pieterh your pattern is actually much clearer and simpler than passing sockets around
[10:03] sustrik ok, good
[10:03] sustrik i'll update the reference accordingly
[10:58] guido_g re with new internet :)
[11:03] pieterh sustrik: is there any recommended way to trap Ctrl-C in a 0MQ program (C++ or C)?
[11:04] sustrik standard C way
[11:04] sustrik no specifics
[11:04] pieterh ok, I'll give it a shot...
[11:16] CIA-21 zeromq2: 03Martin Sustrik 07master * r97add1e 10/ (doc/zmq_init.txt doc/zmq_socket.txt):
[11:16] CIA-21 zeromq2: Documentation wrt thread-safety cleaned up.
[11:16] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[11:25] pieterh sustrik: cool, signal handling works perfectly to shut down 0MQ...
[11:25] pieterh I'll document it, it's simple and clean
[11:38] sejo hmm I created a small zmq.PULL and try to send with telnet a message to it... when debugging it it never gets to recv()
[11:38] AlexB Hello. :) I've found an issue with having inproc sockets that are bound but without any connected end-point. With recent versions of ZeroMQ and PyZMQ, closing a context with socket like that hangs. Here's a short test-case:
[11:39] CIA-21 zeromq2: 03Martin Sustrik 07master * r184bdb8 10/ src/xrep.cpp :
[11:39] CIA-21 zeromq2: Bug caused by interaction of REQ/REP routing and HWM fixed.
[11:39] CIA-21 zeromq2: Signed-off-by: Martin Sustrik <> -
[11:39] guido_g sejo: ømq does have its own wire-format
[11:40] sejo guido_g: ach ok, so i'd better write my client then :p
[11:40] sejo thanks!
[11:40] pieterh AlexB: this is a known issue
[11:40] pieterh you need to close the socket before closing the context
[11:46] AlexB I see. I found it in the docs now.
[11:46] AlexB Thanks. :)
[11:50] pieterh sustrik: can 184bdb8 go to 2.0.x?
[11:52] sustrik yes, if applicable
[11:53] pieterh :-) do you really trust me to figure that out and make the change?
[11:53] pieterh the code in xrep.cpp looks similar enough
[11:54] pieterh but I'm not comfortable backporting patches that I did not write myself
[11:54] private_meta pieterh: hmm... renaming the .c files to .class somewhat broke online viewing on github :/
[11:54] sustrik this is why i said maintaining a stable branch needs a dedicated person :)
[11:55] pieterh sustrik: the process can work fine if we are consistent about patch flow
[11:55] pieterh i.e. anyone wants a patch to version X they send it
[11:55] pieterh separate the work of making the release, and making the code
[11:55] pieterh it should be two separate hats / people
[11:56] pieterh imagine it's not you...
[11:56] pieterh who would be best to backport OpenPGM 5 support to 2.0?
[11:57] pieterh clearly, the person who made that work in 2.1
[11:57] pieterh private_meta: well, you were the one trying to compile these classes
[11:57] guido_g private_meta: i bet that's because .class files are compiled .java files
[11:57] pieterh lol java
[11:58] pieterh private_meta: indeed, nothing shows at all... :-/
[11:58] guido_g so another rename...
[11:58] pieterh ok, alternate suggestions?
[11:58] private_meta pieterh: Well, I wouldn't mind just ".h"
[11:58] private_meta if it were c++, I'd go with ".hpp" with those classes
[11:59] pieterh i suppose it's consistent with zhelpers.h
[11:59] private_meta It would be, yes
[11:59] pieterh sigh, 'clever' software...
[12:00] private_meta heh
[12:00] guido_g hehe
[12:00] pieterh sustrik: if you want that patch to go into 2.0.11 (which it should IMO), please spend 2 minutes at
[12:00] pieterh it's not going to work if you ask the release maintainer to do backports
[12:01] pieterh (a) I refuse, it would be insane, and (b) it's impossible if you have a real community of developers
[12:01] pieterh just like you ask contributors to provide you with patches, I'm asking you
[12:01] pieterh pretty please
[12:07] sustrik well, i don't have resources to maintain 3 branches, sorry
[12:07] sustrik i've warned you
[12:12] pieterh this is not about maintaining branches, martin
[12:12] pieterh it's about how we backport changes when we have N contributors and N release versions
[12:12] pieterh did you understand my example of OpenPGM?
[12:13] pieterh if you want this to scale, it's the only way I can see
[12:13] pieterh it's literally two minutes for you, you know the code perfectly, it's a 10-line patch
[12:14] pieterh and I take care of delivering that to the user community
[12:14] pieterh but if you comingle these two tasks, there will never be properly maintained stable releases
[12:14] pieterh period
[12:14] pieterh do you have another suggestion?
[12:15] sustrik it's not a 2 minute work
[12:16] sustrik the code has changed in the meanwhile and works somewhat differentlu
[12:16] sustrik so it needs, careful patching, testing etc.
[12:16] sustrik backporting is hard work
[12:16] pieterh this is perfectly acceptable as an answer
[12:16] pieterh "no"
[12:18] pieterh you have two sets of people, those able to make release packages (follow procedures perfectly) and those able to backport patches
[12:18] pieterh the intersection of those two sets is close to zero, if not literally zero
[12:18] pieterh that's the major issue here
[12:18] Steve-o :O
[12:18] pieterh in the 0MQ world, specifically
[12:19] Steve-o backporting networking code is hard and tedious work
[12:19] jsimmons backporting sucks everywhere
[12:19] pieterh it gets much worse if you consider a 'release' to include multiple projects
[12:19] Steve-o I did it once for you guys with pgm 5.0, not again :/
[12:20] jsimmons forwardport dont go backwards :D
[12:20] Steve-o I only maintain critical fixes to old branches now
[12:20] pieterh well, the way I see it, someone *wants* fix X on version Y
[12:20] pieterh customer needs it, perhaps
[12:21] pieterh so the backport has an economic basis
[12:21] Steve-o which is what support contracts are about
[12:21] pieterh yes
[12:22] pieterh my goal with 2.0 and 2.1 is to accept and manage backports provided by people who *want* them to be in a specific released version
[12:22] pieterh so, sustrik, my question was really "do you *want* 184bdb8 to go into 2.0.x?"
[12:24] Guthur surely it a customer wants a new feature you say 'upgrade'
[12:27] Steve-o Looks like mikko not about today?
[12:30] private_meta pieterh: A question about zmsg_recv in the zmsg.[h|class] file. You exit the entire application if a message isn't received, is that intentional?
[12:30] pieterh private_meta: nope, I've changed that, just now
[12:30] private_meta oh
[12:30] pieterh it should return NULL
[12:30] private_meta ok
[12:30] private_meta I'll think of something equivalent in C++
[12:37] sustrik pieterh: it would be nice if someone backported it
[12:38] pieterh sustrik: that's what I felt but someone has to want to, enough to do it, and then convince the maintainer (me) to accept the patch
[12:38] pieterh it's the same workflow for all projects, right?
[12:38] pieterh I think there are compromises possible here...
[12:38] sustrik yes, i think so
[12:38] pieterh for example, I'm happy to test a release heavily before it goes out
[12:39] pieterh but that demands proper test cases for every (new) patch
[12:39] sustrik right, lot of work
[12:39] pieterh I'm happy to coordinate with mikko to ensure the rc builds
[12:39] pieterh divide the work, then it's not so much
[12:39] pieterh the only thing I totally refuse to do is patch code I did not write
[12:40] pieterh not in stable production releases
[12:40] sustrik so no backports?
[12:40] pieterh again: divide the work
[12:40] sustrik then i would suggest dropping the stable repos and using tags on master instead
[12:40] pieterh author of code can confidently do a backport if there is incentive
[12:40] pieterh sigh
[12:40] sustrik ah, yes, sure
[12:41] pieterh if you insist on mixing the two roles, we're going to get badly stuck
[12:41] pieterh that's MHO
[12:41] pieterh s/you/we/
[12:42] pieterh look, you provide me with a patch to 2.1, it applies cleanly, ok
[12:42] pieterh I'm still going to test the thing before it's released
[12:42] pieterh but you can't assume I know the code well enough to do that manualluy
[12:42] pieterh *manually
[12:42] pieterh so there must also be proper test cases for new patches
[12:43] pieterh all of this makes it possible to scale
[12:43] pieterh if you depend on unique people able to do *everything*, you cannot scale
[12:43] pieterh s/you/we/
[12:43] sustrik ok, let me explain my pov
[12:44] pieterh shoot
[12:44] sustrik it's a problem of cost
[12:44] sustrik the process we've had before was the minimal-cost process
[12:44] pieterh in what sense was is cheap before?
[12:44] pieterh *it
[12:44] sustrik no backporting
[12:44] sustrik there was maint
[12:45] pieterh well, there should have been backporting of critical fixes
[12:45] sustrik the patches to maint were upstreamed to master
[12:45] sustrik but not other way round
[12:45] pieterh ... ok, and how often did that happen?
[12:45] pieterh last fix to maint was 1 december afaict
[12:45] sustrik rarely
[12:45] sustrik that's why the cost was low
[12:45] pieterh the process was cheap because the process was never used
[12:45] sustrik so the user was faced with following choice:
[12:45] pieterh hey, I can do that now as well
[12:46] pieterh and we can upstream today just as easily
[12:46] pieterh it's no different
[12:46] sustrik 1. stay with maint, patch the problem there, submit the patch
[12:46] sustrik 2. move to master
[12:46] pieterh 1. stay with 2.0, patch the problem there, submit the patch
[12:46] pieterh 2. move to 2.1, patch the problem there, submit the patch
[12:46] pieterh 3. move to 2.2 master, etc...
[12:46] sustrik yeah
[12:46] pieterh it is exactly the same process
[12:47] sustrik nope, it's unidirectional
[12:47] pieterh ok, theoretical question
[12:47] pieterh if someone provides a patch to 2.0, how does that get upstreamed?
[12:47] pieterh who does that work?
[12:47] sustrik myself
[12:48] sustrik i'm maintainer of master
[12:48] sustrik so i want it to be bug-free
[12:48] pieterh but you're not following the old releases
[12:48] pieterh how do you even know there was a change?
[12:48] sustrik so if someone finds a bug in maint and submits a patch
[12:48] sustrik i want it to get to master
[12:48] sustrik all the patches flew via myself
[12:48] pieterh ah
[12:48] pieterh so you also maintain maint?
[12:49] pieterh and if there are three branches, you maintain all 3?
[12:49] pieterh afaics we agree on the problem and the solution
[12:50] sustrik 'maintain' in terms of applying patches that i get from other people
[12:50] pieterh yes, but you don't want to get patches to old versions, do you
[12:50] pieterh because that makes you maintainer of those
[12:50] sustrik git apply
[12:50] sustrik git commit
[12:50] sustrik git push
[12:50] sustrik easy
[12:51] pieterh that assumes the patch is tested and perfect
[12:51] pieterh ok, let's assume that
[12:51] sustrik yes, that's up to submitter
[12:51] pieterh so we do the same today, except you don't have to do that
[12:51] pieterh the patch must be tested and perfect
[12:52] pieterh then *anyone* who follows the process can apply/commit/push
[12:52] pieterh does not require your time
[12:52] pieterh that then becomes scalable
[12:52] sustrik it's up to you what process you use for stable branches
[12:53] pieterh and if the patch is not perfect, you don't want to be fixing up old branches, do you
[12:53] pieterh retesting 2.0?
[12:53] pieterh so what we've done with the separate gits, and why we both like it
[12:54] pieterh is that you have delegated the grunt work of applying *perfect* patches to old versions
[12:54] pieterh followed by documentation, release packaging, uploading, emails, etc. etc.
[12:54] pieterh ack?
[12:54] pieterh second aspect is upstream vs. downstream
[12:55] pieterh totally orthogonal, this applies equally if we had 1 repo or 3 repos
[12:55] sustrik yeah, i've delegated hat work to you
[12:55] sustrik now i don't care about it amymore
[12:55] pieterh yes, and that is a big improvement
[12:55] sustrik life is beautiful now :)
[12:55] pieterh well, that is what I call a saving in time and cost
[12:55] pieterh so now we look at the orthogonal issue of up/down streaming
[12:56] pieterh as maintainer, I'll accept *perfect* patches from anyone I trust
[12:57] pieterh I don't necessarily see 2.0 as downstream from 2.1, they act as two independent projects, almost
[12:57] sustrik sure, it's up to you
[12:58] pieterh do you understand why I want to separate the role of coding from release management?
[12:58] ianbarber in that process there must be an agent who wants a patch from <new version> in <maintenance version> and is willing to do the work to test and submit the perfect patch. But, the people most likely to be trusted by the maintainer are also mostly likely to be working on the later/latest versions, I would guess.
[12:59] ianbarber and the people that are on the older versions don't necessarily have the awareness that there are improvements that would benefit them
[12:59] pieterh ianbarber: to a point, but if we are smarter, it scales better
[12:59] pieterh for example, better test cases
[12:59] ianbarber yeah
[12:59] sustrik it's hard to get someone to do that kind of work
[12:59] sustrik linux has GregKH
[12:59] pieterh sustrik: there has to be an economic incentive in every case
[12:59] ianbarber better test cases would let you apply patches more speculatively
[13:00] ianbarber if you could know things will shout if you break stuff
[13:00] sustrik but i would guess people like that are pretty rare
[13:00] pieterh ianbarber: indeed, I'd rather trust a test case I can understand than someone's patch
[13:00] ianbarber definitely.
[13:00] pieterh sustrik: not only rare, but counter-productive in the end
[13:00] pieterh if you know too much, you cut corners
[13:01] pieterh being naive forces better process in many cases
[13:01] pieterh e.g. I don't know the internals so I insist on better test cases
[13:01] sustrik having a dedicated maintainer of stable is counter-productive?
[13:01] pieterh if I know the code really well, I won't use test cases
[13:02] pieterh and the product and process will suffer (this has already happened a lot)
[13:02] pieterh sustrik: you say "maintainer" to mean two different things in my view
[13:02] pieterh I am the dedicated maintainer but not in the sense of "I own the code"
[13:02] pieterh I own the release and the process
[13:03] pieterh the code is owned by its authors
[13:03] sustrik maintainer in terms of "i am willing to spend all my free time to maintain stable"
[13:03] pieterh define "maintain" please
[13:03] sustrik that's pretty rare i would say :)
[13:04] pieterh look, I maintain the Guide, yes?
[13:04] pieterh that is, I'm responsible for the repository, publishing it, and I write the text
[13:04] pieterh but every set of translations has their authors
[13:05] pieterh they own that code
[13:05] pieterh I don't maintain it
[13:05] sustrik this discussion is pointless
[13:05] sustrik the process in stable is entirely up to you
[13:05] pieterh if you want that patch to go into 2.0, send it my way, thanks
[13:05] Guthur pieterh: Do you think it would be nice to have some readily available statistic of the guide example coverage for each language binding?
[13:06] pieterh Gurthur: hang on...
[13:06] Guthur it's ok i'm not going anywhere soon
[13:06] Guthur hehe, and I know you guys are having a heated debate
[13:06] pieterh Translations (48 in total):
[13:06] pieterh Ada 0, 0%
[13:06] pieterh Basic 0, 0%
[13:06] pieterh C 48, 100%
[13:06] pieterh C++ 33, 68%
[13:06] pieterh C# 26, 54%
[13:06] pieterh Common Lisp 32, 66%
[13:06] pieterh Erlang 0, 0%
[13:06] pieterh Go 2, 4%
[13:06] pieterh Haskell 0, 0%
[13:06] pieterh Java 17, 35%
[13:06] pieterh Lua 0, 0%
[13:06] pieterh Objective-C 0, 0%
[13:06] pieterh ooc 0, 0%
[13:06] Guthur oh whao
[13:07] Guthur I wasn't expecting there to be any
[13:07] Guthur Would that be worth making visible, do you think?
[13:08] pieterh where?
[13:08] Guthur top of the guide maybe
[13:08] pieterh hang on...
[13:08] pieterh I'll post it next time I do an update, if I remember
[13:08] Guthur cool
[13:09] Guthur I need to bump that C# number up a bit, hehe
[13:21] private_meta pieterh: about zmsg, is it made to cater other types but standard character strings?
[13:21] pieterh nope
[13:21] pieterh this is not meant to be production code, it's for the examples
[13:21] private_meta pieterh: in the dump function you differ between non-character blob and string, that's why I ask
[13:22] pieterh yes, because a socket can get anything
[13:22] pieterh the class *could* be expanded to allow binary message parts
[13:22] pieterh but that gets complex quickly, and was not worth it for here
[13:22] pieterh I'd do it in zfl_msg, which is a more serious implementation
[13:23] private_meta hmm ok. If it were string only, i would use std::string and the implementation would be a little more clean
[13:23] private_meta but the example should follow the same functionality as the C version i guess
[13:24] pieterh you can implement it any way you like, I'd suggest keeping the API similar so people can follow
[13:25] pieterh the methods that muck with envelopes... wrap/unwrap/push/ etc.
[14:38] guido_g pieterh: ping?
[14:38] pieterh hi guido_g
[14:38] guido_g question about mdp spec
[14:39] pieterh shoot
[14:39] guido_g worker send ready message to broker
[14:39] guido_g there will be no response to that, right?
[14:39] pieterh there is a response but it's not immediate
[14:39] pieterh it's the lru pattern
[14:40] guido_g i'm talking about the ready message
[14:40] pieterh worker waits, indefinitely, for a client request to arrive, that's the response
[14:40] guido_g from the worker
[14:40] guido_g to the broker
[14:40] guido_g aka handshake
[14:40] guido_g ok, half of a handshake
[14:40] pieterh yes, I do understand what you are asking
[14:40] pieterh it is the lru pattern
[14:41] guido_g i can't find any answer to the ready message is the spec
[14:41] guido_g *in the spec
[14:41] pieterh there is no answer to the ready spec, as such
[14:41] pieterh the first REQUEST is the answer
[14:41] guido_g ahh we're getting closer
[14:41] pieterh a DISCONNECT would be a negative acknowledgement
[14:41] pieterh sorry, when I say "lru pattern" does that make any sense?
[14:42] guido_g ok, so till a request arrives the woker does not know if the broker has his service registered correctly, right?
[14:42] guido_g no it doesn't, because we're on mdp protcol level
[14:42] pieterh it can safely assume all is OK unless it (a) gets a disconnect or (b) no heartbeat within whatever time
[14:43] pieterh ok, good point, I'll add this explicitly
[14:43] guido_g so a received hb from the broker will count as "registration is fine"
[14:44] guido_g so we need a basic timeout value for the handshake
[14:44] guido_g or a dedicated "got your ready" reply
[14:45] pieterh what is the value of a dedicated "OK" command?
[14:46] pieterh I'm not against it, but it seems unnecessary
[14:46] guido_g the value would be that the worker from this point on knows that his service is registered and he does receive heartbeats etc.
[14:47] pieterh what is the value of the worker knowing?
[14:47] guido_g would allow for small timeouts in the handshake phase
[14:47] pieterh I'm being serious, remember we're on disconnected TCP
[14:48] pieterh so the broker can arrive 30 minutes late
[14:48] pieterh do we want to break that? I'd rather not, it's valuable
[14:48] guido_g wouldn't break that, just needs a longer timeout
[14:48] pieterh which is the heartbeat
[14:49] guido_g right
[14:49] pieterh so you don't need a positive ack
[14:49] guido_g but not the "i'm ok" hb, but the one for the connection handshake
[14:49] pieterh a negative ack gives you just the same semantics
[14:50] guido_g example: no broker , worker starts, broker comes 30 seconds later
[14:50] guido_g general hb interval is 1 sec
[14:50] guido_g client died due to lots of missed hbs
[14:50] pieterh worker will reconnect over and over and eventually get a HB back
[14:50] pieterh clients have different timescales
[14:51] pieterh the client could easily retry like the worker does
[14:51] pieterh that's not MDP's problem
[14:51] guido_g shit, s/client/worker/
[14:51] pieterh worker does not die, if you look at my API
[14:51] guido_g again
[14:52] pieterh it keeps reconnecting forever (or until Ctrl-C in my latest code)
[14:52] guido_g i do look at the spec
[14:52] pieterh right
[14:52] pieterh "If the worker detects that the broker has disconnected, it MUST restart a new conversation."
[14:52] pieterh I'm not sure that's accurate
[14:52] guido_g ok
[14:53] pieterh but it allows the broker to kick workers that come in without doing 'ready'
[14:53] guido_g would mean: after x missed hbs close socket and start from the beginning
[14:53] pieterh yes
[14:53] pieterh "Both broker and worker MUST send heartbeats at regular and agreed-upon intervals. A peer can consider the other peer "disconnected" if no heartbeat arrives within some multiple of that interval (usually 3-5)."
[14:53] guido_g ok, i'll implement it that way
[14:53] pieterh s/can/SHOULD/
[14:54] guido_g MUST even
[14:54] guido_g otherwise the whole thing would be useless
[14:54] guido_g i'll start the hb timer after sending the ready message then
[14:55] pieterh ack, MUST it is
[14:55] guido_g fine
[14:55] pieterh I need to reboot this box, it's starting to act funny
[14:55] pieterh cy in a minute or two
[14:55] guido_g np
[14:59] pieter_hintjens back
[14:59] pieter_hintjens what did I miss? :-)
[15:00] cremes is your new laptop with the ssd giving you fits?
[15:01] pieter_hintjens cremes: it all works... :-)
[15:01] pieter_hintjens it's silent and uses a little less power, and it's faster
[15:01] guido_g pieter_hintjens: why are you talking about missing x hbs? why not simply set the hb interval and barf if one is missed?
[15:02] pieter_hintjens guido_g: bitter experience
[15:02] guido_g uh
[15:02] guido_g could you explain that a bit?
[15:02] pieter_hintjens it's relatively easy to miss single HBs due to the two peers each sending out at intervals
[15:03] pieter_hintjens it's also relatively easy to get them delayed when there's congestion
[15:03] pieter_hintjens we did extensive testing of this for AMQP
[15:04] pieter_hintjens you basically really do not want false positives
[15:04] guido_g ah its ok
[15:04] pieter_hintjens try it yourself, set the liveness to 1, run it in real use
[15:07] guido_g so basically the hb interval is the max. time w/o a message divided by the liveness as you call it
[15:07] pieter_hintjens yes
[15:07] pieter_hintjens this is a simpleminded design, I'm sure we can do better over time
[15:08] guido_g me too :)
[15:08] pieter_hintjens here's one key aspect...
[15:08] pieter_hintjens say the clock is 1 second
[15:08] pieter_hintjens and a HB comes in at 1.001 seconds
[15:08] pieter_hintjens and your liveness is 1
[15:08] pieter_hintjens you now disconnect
[15:09] pieter_hintjens so liveness must be at least 2 to allow for fractional delays in HBs
[15:09] guido_g you have to draw the border somwwhere
[15:09] guido_g *somewhere
[15:09] pieter_hintjens what border?
[15:09] guido_g connection alive or not
[15:09] pieter_hintjens yes, how delayed can a HB be...
[15:09] pieter_hintjens if it's 0.1% delayed, that's not a dead peer
[15:09] guido_g infinite?
[15:10] pieter_hintjens if it's infinite, you don't have heartbeating any more
[15:10] guido_g by a femto second?
[15:10] pieter_hintjens by a couple of heartbeats, is the best answer we found
[15:10] pieter_hintjens it's like your own heart can miss one beat, two if you're really shocked, but three means you're dead
[15:11] guido_g this is more a problem of how to define the border between alive and not
[15:12] pieter_hintjens yes, and it's a heuristic you can't really make until you have real apps
[15:12] guido_g you get the same problem when counting hbs
[15:12] pieter_hintjens that's why it's configurable in my latest api
[15:12] guido_g ok
[15:13] guido_g i'll implement that as a simple counter decremented by a periodically called method
[15:13] guido_g so one can adjust the interval and the count as he/she/it sees fit
[15:19] pieter_hintjens kiss...
[15:19] guido_g thanks :)
[15:19] pieter_hintjens it just has to detect real failure
[15:19] pieter_hintjens :-)
[15:20] guido_g i need to read
[15:24] pieter_hintjens guido_g: there may also be some threads on the websocket heartbeating discussions
[15:27] guido_g first i'll implement the simple version
[15:32] sustrik guido_g: tcp's keepalives in just a afterthought
[15:32] sustrik even explictly discouraged by the spec
[15:32] guido_g ah ok
[15:32] sustrik if you want to read about sane heartbeating mechanism, have a look at SCTP heartbeats
[15:32] guido_g one tab less open in the browser then :)
[15:33] guido_g ok, will do, but later
[15:33] sustrik just a tip :)
[16:00] ok2 hi!
[16:02] ok2 question, how can i see with zmq, on send() that the other end is not here anymore?
[16:02] ok2 zmq simply hangs for me, if the other side crashes
[16:05] cremes ok2: you need to use zmq_poll(), zmq_send() with ZM_NOBLOCK and setup your own timeout logic
[16:06] cremes there is some work being done to wrap up this pattern (it gets asked about a lot)
[16:06] cremes but i have no idea what the status is on that work
[16:06] cremes i've built this myself but using ruby instead of C
[16:08] ok2 it means that i make zmq_poll(x, NOBLOCK) and wait on zmq_poll for POLLOUT?
[16:08] ok2 zmq_send(x, NONBLOCK)
[16:09] ok2 no, on POLLERR ...
[16:20] cremes ok2: no, that isn't correct
[16:21] cremes let's say you are using a req/rep socket pair and you want to timeout on the request
[16:21] cremes you would do something like:
[16:22] cremes zmq_send(req_socket, message, ZM_NOBLOCK)
[16:22] cremes register req_socket with zmq_poll()
[16:23] cremes and POLLIN (you are looking for a reply, so that will be a recv operation)
[16:24] cremes when req_socket returns POLLIN, call zmq_recv(req_socket, replymsg, 0)
[16:24] cremes OR
[16:24] cremes generate an application error if your timeout has expired before req_socket returns POLLIN
[16:24] cremes make sense?
[16:39] pieter_hintjens ok2: it's explained here:
[16:42] guido_g pieter_hintjens: any chance that the frame order in spec:7 is wrong?
[16:42] pieter_hintjens where exactly?
[16:42] guido_g given that the broker is a XREQ type, it needs to prepend the target identity
[16:42] pieter_hintjens guido_g: you know, I'm so pleased to see you implementing this
[16:43] guido_g sarcasm?
[16:43] pieter_hintjens no
[16:43] guido_g puh
[16:43] guido_g :)
[16:43] pieter_hintjens seriously, it's magic to see someone implement a spec the day after it's published
[16:43] pieter_hintjens what command are you looking at?
[16:44] guido_g worker <-> broker
[16:44] guido_g broker is XREQ
[16:44] pieter_hintjens this is why all the framing says "on the wire"
[16:45] guido_g pardon?
[16:45] pieter_hintjens when you send a message to a router socket you always prepend the address
[16:45] pieter_hintjens but that does not go on the wire
[16:45] pieter_hintjens it's not sent, thus is not relevant to the protocol
[16:45] pieter_hintjens however I'll explain this, it's confusing
[16:46] guido_g it is
[16:53] pieter_hintjens guido_g: ok, I've added a note, can you see if it's clear?
[16:53] pieter_hintjens
[16:55] guido_g GREAT!
[17:01] guido_g getting close now
[17:01] guido_g *closer
[17:03] ok2 cremes and pieter_hintjens: thank you very much, i think i got what i need :-)
[17:03] pieter_hintjens ok2, np
[17:05] guido_g first success for the worker
[17:09] pieter_hintjens broker is tricky to get right
[17:10] pieter_hintjens am still finding undispatched messages in some cases
[17:11] guido_g so, short break and then testing my worker against your broker
[17:12] pieter_hintjens I'll need to commit, almost ready
[17:12] guido_g ok, no hurry
[17:24] Altxp Hello there.
[17:36] pieter_hintjens guido_g: ok, done
[17:36] pieter_hintjens the broker is, afaict, complete and robust
[17:40] pieter_hintjens I've pushed this to the website, and committed all the source code
[17:41] djc okay, I have some questions with my distro hat on
[17:41] djc are there options to rely on external openpgm instead of the bundled one?
[17:41] djc what versions of openpgm are compatible with zeromq-2.1.1?
[17:42] djc does zeromq actually depend on the xmlParser, or can that be disableD?
[17:45] guido_g pieter_hintjens: thanks
[17:45] guido_g after some phineas and ferb i'll check it out
[17:49] private_meta pieter_hintjens: so it's basically finished?
[17:52] guido_g pieter_hintjens: simple hb works
[17:54] guido_g ahh my worker crashes when the broker is restarted
[18:08] guido_g ha! works now
[18:52] guido_g pieter_hintjens: python client and worker are running against your broker
[19:21] yrashk according to our latest fixes to erlzmq tests, ezmq is still way faster
[20:07] pieter_hintjens guido_g: I'm back... that's great!
[20:07] pieter_hintjens private_meta: yes, it's finished for now
[20:11] guido_g pieter_hintjens: wb
[20:11] pieter_hintjens guido_g: I added you as contributor to the spec, if that's ok
[20:12] guido_g small thing, the mdbroker segfaults sometimes :)
[20:12] guido_g thanks
[20:12] pieter_hintjens can you get me a backtrace?
[20:12] pieter_hintjens or reproducible case
[20:12] guido_g i'll try my best
[20:12] travlr pieter_hintjens: hey pieter, how would suggest I implement a mail list service for an open source community project i'm ready to start?
[20:12] pieter_hintjens ulimit -u
[20:12] guido_g it's related to a worker beeing killed
[20:12] pieter_hintjens and then you'll get core files
[20:13] pieter_hintjens yeah, cleaning up dead workers is tricky
[20:13] pieter_hintjens travlr: hmm... I'd avoid email lists altogether, personally
[20:13] guido_g are your build scripts compiling w/ debug on?
[20:13] pieter_hintjens guido_g: should be, now
[20:13] pieter_hintjens try 'c -C' for sure
[20:13] travlr pieter_hintjens: what might be your suggestion then.
[20:13] pieter_hintjens travlr: you might take one of the wikis at
[20:14] pieter_hintjens that works well for community projects
[20:14] pieter_hintjens forum + email alerts + wiki combinations
[20:14] travlr is that just using wikidot's forum widgets?
[20:14] pieter_hintjens no, pages + comments in those ones
[20:15] travlr ah, ok so its irongiant specific... let me look at the link, thank you.
[20:15] pieter_hintjens email lists are ok but a bit useless for dynamic work
[20:15] pieter_hintjens thus we all hang out here on irc...
[20:16] travlr yes, i was hoping for forum, email notication integration
[20:16] pieter_hintjens you got it all there
[20:16] pieter_hintjens there is an open source project template afair
[20:16] travlr k
[20:16] travlr thanks
[20:16] pieter_hintjens clickety-clone and it's done, very simple
[20:16] pieter_hintjens np
[20:25] guido_g pieter_hintjens: is not enough info i guess
[20:26] pieter_hintjens guido_g: when did you pull the branch from git?
[20:26] pieter_hintjens this is a bug I fixed afaik
[20:26] guido_g ok, i'll check
[20:26] pieter_hintjens also it's not debug, so I think the previous version...
[20:26] pieter_hintjens I'll have to put a version number in the broker :-)
[20:30] guido_g it's up-to-date when i do a pull
[20:30] pieter_hintjens hmm
[20:31] pieter_hintjens master branch...
[20:31] pieter_hintjens ok, let me add a version number print...
[20:31] guido_g <- output of run
[20:31] guido_g trying to get more info out of gdb
[20:33] pieter_hintjens can you grab the latest master, rebuild the broker?
[20:35] guido_g grml the c script ignores the -g
[20:35] pieter_hintjens how are you building?
[20:35] pieter_hintjens './build mdbroker' should work
[20:36] guido_g the c script does not pass the -g to the compiler
[20:36] pieter_hintjens nope, it doesn't
[20:36] pieter_hintjens that's not how it works
[20:36] guido_g ??
[20:36] pieter_hintjens sigh... it's a long story
[20:36] guido_g build calls this script
[20:36] pieter_hintjens can't you just './build all'?
[20:36] pieter_hintjens ah hang on
[20:36] guido_g *sigh*
[20:36] guido_g yes, i can but w/o debug info
[20:36] pieter_hintjens yes, but it doesn't pass a -g option
[20:37] guido_g then the header of c should be changed
[20:37] guido_g there is a -g mentioned
[20:37] pieter_hintjens yes, that's wrong
[20:37] guido_g double *sigh*
[20:38] pieter_hintjens what git commit are you on?
[20:38] pieter_hintjens git log -1
[20:39] guido_g new clone and fresh broker (w/o debug info)
[20:39] guido_g testing...
[20:39] pieter_hintjens what git commit are you on?
[20:40] pieter_hintjens the c script is ok, it doesn't pass -g to the compiler as such, it processes it
[20:40] pieter_hintjens but all this is beside the point
[20:40] guido_g git log -1
[20:40] guido_g commit a8121994b0c25f4d242bbf06074dc6cc389ff336
[20:40] pieter_hintjens ok
[20:40] pieter_hintjens take hammer, break glass
[20:40] pieter_hintjens export BOOM_MODEL=debug; build all
[20:41] djc pieter_hintjens: if you have a moment later, could you take a peek at my questions from 3h ago?
[20:41] pieter_hintjens djc: let me look...
[20:42] pieter_hintjens ok... for openpgm, I don't know
[20:42] pieter_hintjens for the XML dependency, it's there in some random main programs still being built with 0MQ
[20:42] pieter_hintjens they resist being killed because they theoretically form part of the 'API' and thus are holy
[20:42] djc ah yes, that is how it goes
[20:42] pieter_hintjens guido_g: the build script _should_ be making debug builds
[20:43] guido_g it does not
[20:43] djc so for openpgm I should probably stick to .92 and wait for you guys to test something newer?
[20:43] pieter_hintjens djc: you need to ask steve-o
[20:43] djc pieter_hintjens: okay, thanks
[20:43] djc does he come around here, and if so, in what tz?
[20:43] pieter_hintjens guido_g: ok, can you add 'c -C' to build just before the first if statement?
[20:44] pieter_hintjens that'll produce a report of the actual C compiler syntax it uses
[20:44] guido_g w/ debug info
[20:44] guido_g hehe
[20:44] guido_g i did the export hingie and it seemed to work
[20:44] guido_g *thingie
[20:45] pieter_hintjens which the dratted build script is meant to do itself if necessary
[20:45] pieter_hintjens one line of shell, how complex can it be... :-(
[20:46] pieter_hintjens ok, looking at that backtrace, thanks
[20:46] guido_g this happens when the (only) worker exits w/o disconnect (just disapears)
[20:47] guido_g oops
[20:47] guido_g tested this w/ my worker
[20:47] guido_g will try the c worker now
[20:48] pieter_hintjens if your worker makes it crash, send that to me
[20:48] pieter_hintjens I can't get the crash using the C worker
[20:48] guido_g i can :)
[20:49] guido_g you stopped the worker w/ ctrl-c?
[20:49] pieter_hintjens yes
[20:49] guido_g tz tz tz
[20:49] pieter_hintjens kill -9?
[20:49] pieter_hintjens should not make any difference
[20:49] guido_g it looks like the woker sends a disconnect on sigint
[20:50] pieter_hintjens nope, don't think so...
[20:50] guido_g ok
[20:50] pieter_hintjens well, valgrind is also real good for this
[20:51] pieter_hintjens valgrind --tool=memcheck mdbroker
[20:51] guido_g crash when killing the worker w/ ctrl-c too
[20:51] pieter_hintjens hmm, can't be a timing issue, it's all one thread
[20:51] pieter_hintjens are you also running a client?
[20:51] guido_g bt looks like its the same place
[20:52] guido_g no, client has been disconnected before
[20:52] pieter_hintjens but do you start it in the test case?
[20:52] guido_g i've a one shot client that sends one request
[20:52] pieter_hintjens this makes a difference
[20:52] pieter_hintjens yes, I got the crash now
[20:53] pieter_hintjens need to have a client request
[20:53] guido_g even a processed one?
[20:53] pieter_hintjens it all affects the broker state
[20:53] guido_g obviously
[20:54] pieter_hintjens ok, I got the error, but will look at it tomorrow
[20:54] pieter_hintjens i appreciate the feedback enormously, guido_g
[20:54] guido_g ømq is like a time machine, didn't use gdb for 10 years
[20:54] pieter_hintjens heh
[20:57] guido_g any idea where to put the python version of mdp?
[20:58] guido_g i'll make a repo for it
[21:05] pieter_hintjens guido_g: I'd hope for Python translations that can be used for the Guide
[21:06] pieter_hintjens these are minimal implementations, we can make a real product & repository after
[21:06] guido_g sure, but there not translations of the c code, as i said
[21:06] guido_g this a mdp implementation using the async features of pyzmq
[21:07] pieter_hintjens that sounds fine
[21:07] pieter_hintjens the point isn't to write C in Python
[21:07] pieter_hintjens the point is to show Python programmers how to make the example in question
[21:07] pieter_hintjens so, client API, worker API, broker
[21:07] pieter_hintjens using whatever the language offers
[21:07] guido_g <- the test worker
[21:09] pieter_hintjens did you read mdwrkapi.c?
[21:09] pieter_hintjens and mdworker.c
[21:09] guido_g no
[21:09] pieter_hintjens probably worth doing
[21:09] guido_g why?
[21:09] pieter_hintjens simply to see if we can offer a consistent API
[21:10] pieter_hintjens worker API in C or Python can look similar
[21:10] guido_g pieter_hintjens> the point isn't to write C in Python <- ...
[21:10] pieter_hintjens the API I made is not C
[21:10] pieter_hintjens it's implemented in C
[21:10] pieter_hintjens I'd thought of specifying the API as a rfc
[21:10] pieter_hintjens both APIs
[21:11] pieter_hintjens for example, in mdworker.c there is no 0MQ context, it's invisible
[21:12] pieter_hintjens see
[21:12] pieter_hintjens and yes it includes a .c, please don't complain... :-)
[21:15] guido_g <- first go
[21:16] guido_g see, the api is completely different because of the mechanisms used
[21:16] guido_g as i said, i want to make use of the async features built into pyzmq
[21:17] pieter_hintjens for sure
[21:18] pieter_hintjens though I've no idea what the value is in offering such a complex API to app developers...
[21:18] guido_g complex?
[21:18] pieter_hintjens look, my API has three methods
[21:18] pieter_hintjens create, send/recv, destroy
[21:19] pieter_hintjens yours exposes the internals of MDP
[21:19] pieter_hintjens every timeout, every message
[21:19] pieter_hintjens it's not a wrapper at all, just a deconfobulatorix
[21:19] guido_g the create is line 31 in the example worker
[21:19] pieter_hintjens i made that up, btw
[21:19] pieter_hintjens sorry :-) it's late
[21:19] guido_g the worker method is the on_request
[21:20] guido_g and the reply is sent by self.reply
[21:20] pieter_hintjens i'm not complaining, this is fantastic work
[21:20] guido_g very complicated, i've to admit
[21:20] pieter_hintjens please do look at that gist I posted just now
[21:20] pieter_hintjens that's what the app developer sees
[21:20] pieter_hintjens when my C code is shorter than your Python code, to do the same work... that's... well...
[21:21] pieter_hintjens ok
[21:21] pieter_hintjens projects...
[21:21] pieter_hintjens MDP is the protocol, not software
[21:21] guido_g shorter?
[21:21] pieter_hintjens do you see value in a project with mixed pieces, e.g. brokers / apis in different languages
[21:21] pieter_hintjens
[21:21] guido_g yes
[21:22] pieter_hintjens that's the echo worker
[21:22] pieter_hintjens ok, let'
[21:22] pieter_hintjens let's invent a name
[21:22] guido_g i might be shocked, but can still read
[21:22] pieter_hintjens majordomo already been taken by some random FOSS project apparently
[21:23] guido_g random... *cough*
[21:23] pieter_hintjens do you prefer memorable or meaningful?
[21:23] guido_g the interoperability is there
[21:23] pieter_hintjens yes
[21:23] guido_g i'm using my worker w/ your broker and and client
[21:23] pieter_hintjens interop is a funny thing though, it operates at multiple levels
[21:23] pieter_hintjens e.g. 0mq api is interop
[21:24] Guthur pieter_hintjens, what's the link to this protocol?
[21:24] pieter_hintjens
[21:24] guido_g
[21:24] Guthur cheers
[21:24] pieter_hintjens :-)
[21:24] Guthur did you find that majordomo is trademarked?
[21:25] pieter_hintjens nope, it's not afaics
[21:25] guido_g but still, the name is known for a quite long time
[21:25] pieter_hintjens but it's a bad idea to mix the pattern - protocol - implementation
[21:26] pieter_hintjens oh god, a bank email hit the list
[21:26] pieter_hintjens you can tell them by the 1000-word disclaimers
[21:27] pieter_hintjens And someone's acidic reply, "It took me a moment to realize your ridiculous disclaimer is not the message itself..."
[21:27] pieter_hintjens hehe
[21:27] guido_g hehe
[21:29] Guthur it then has another link to even more disclaimer talk
[21:30] pieter_hintjens I think I just sold my soul to BoA by replying to that email
[21:31] pieter_hintjens ok, guido, let's make a project called Popcorn
[21:31] pieter_hintjens cause I'm finishing a large bowl I made a while ago
[21:31] pieter_hintjens there are dozens of FOSS projects called popcorn so there's no chance of confusion
[21:31] guido_g there was a song called like that... back when music came these black large discs...
[21:32] pieter_hintjens 8" floppies?
[21:32] guido_g no, harder
[21:32] pieter_hintjens 12" removable winchesters?
[21:32] pieter_hintjens did they have MP3s in those days? don't think so...
[21:32] guido_g but drive was open mostly
[21:32] pieter_hintjens you're making this up as you're going along!
[21:33] pieter_hintjens so, guido_g, shall I make this as an official zeromq community project?
[21:33] guido_g let it cook a bit more first
[21:33] pieter_hintjens embrace the chaos!
[21:33] pieter_hintjens it burns if you leave it cooking too long
[21:34] guido_g if we have full interop w/ the c version, then it might be worth to be announced
[21:34] guido_g for this it needs a broker
[21:34] pieter_hintjens ah, but the trick with projects is to announce a stunning goal way before you can make it happen
[21:34] Guthur what is this new project
[21:34] pieter_hintjens guido_g: see, we have interest already...
[21:34] pieter_hintjens Guthur: sorry, can't tell you, it's confidential
[21:34] guido_g *sigh*
[21:34] pieter_hintjens :-)
[21:35] Guthur I could try the logs hehe
[21:35] Guthur they are quite long though
[21:35] guido_g
[21:35] Guthur you guys were chatting for ages, hehe
[21:35] pieter_hintjens guido_g: cook it a while, anyhow
[21:36] pieter_hintjens if it was my project, I'd insist on common APIs in all languages
[21:36] guido_g at the least the broker should be there and working with your worker and client
[21:36] pieter_hintjens yes
[21:36] pieter_hintjens defining good, language neutral APIs is quite fun
[21:36] pieter_hintjens and extraordinarily useful to app developers
[21:36] guido_g pieter_hintjens: then i'd remove the repo immediately
[21:37] pieterh i can make event driven APIs in C, that's not the issue
[21:37] pieterh the issue is just the semantics you show to users, and whether these can be consistent between languages
[21:38] pieterh I'm pretty adamant they can be consistent
[21:38] guido_g there is more than language
[21:38] pieterh guido_g: are you in London by any chance?
[21:38] guido_g nope
[21:38] pieterh sigh...
[21:39] guido_g too expensive just for a beer
[21:39] pieterh I'd actually take a train somewhere to thrash this out
[21:39] guido_g wouldn't make it cheaper
[21:40] pieterh ok, let's cook this a while.
[21:41] pieterh I need to get onto the rest of Ch4 anyhow
[21:41] pieterh there is a lot still to do
[21:41] pieterh I'll fix that broker crash tomorrow when I find an hour, it's a busy day
[21:42] guido_g fine with me
[21:42] pieterh ok, gnite !
[21:43] guido_g good night!
[21:55] guido_g has been a long day, need some sleep
[21:55] guido_g cya
[22:32] Guthur pieterh, umm did the implementation of zhelpers.h not use to be in the guide text
[22:33] pieterh Guthur: (a) it got very long and (b) the translation model only works for .c files so it was showing C code in the PHP/whatever versions
[22:33] Guthur ok
[22:34] pieterh Did you like having it in the guide?
[22:34] Guthur doesn't matter too much, I just wanted to check the dump implementation
[22:35] pieterh I think the Guide is moving towards being less focused on C
[22:35] Guthur true
[22:35] pieterh Though I think C is the future... but still...
[22:35] pieterh :-)
[22:35] Guthur I thought C was the past, hehe
[22:35] Guthur Lisp is the future, hehe
[22:36] pieterh Lisp is older than C afair
[22:37] Guthur that was LISP
[22:37] Guthur but true, Lisps ancestor is the second oldest high level language
[22:38] Guthur after Fortran
[22:38] Guthur First with a Garbage Collector as well
[22:38] Guthur afaik
[22:39] pieterh gawk rules
[22:39] Guthur Lisp is a very enlightening experience as they say
[22:40] Guthur Javascript is pretty awesome as well, it's just a pity it always lives in a browser, hehe
[22:46] Guthur pieterh, have you ever tried Forth? last language mention, I swear
[22:47] sp4ke hi guys
[22:47] pieterh Guthur: I used to build Forth machines in 6502 assembler, quite freaky
[22:47] pieterh hi sp4ke
[22:47] Guthur pieterh, cool, 6502 was one of my first programming experiences
[22:47] pieterh The sign of a Real Programmer
[22:47] Guthur BBC micros
[22:48] sp4ke want just to post this link for people who want to use zmq with MSVC 2010 i made a tutorial after struggling to make it work
[22:48] sp4ke
[22:48] sp4ke it could be useful on irc archive
[22:49] pieterh sp4ke: ... it's ...
[22:49] pieterh it comes down to "VS2010 made some changes to output properties, go to projects properties -> General -> , then change the Output Directory field to"..\..\..\lib\" (the same path as in (Linker -> General -> Output File) property"
[22:49] pieterh why not add that to
[22:49] sp4ke yes indeed but for a new comer to MSVC it is not obvious
[22:50] sp4ke ok i will do
[22:50] sp4ke did not know that link
[22:50] pieterh lol
[22:52] Guthur it always works without any bother for me
[22:52] Guthur I convert and compile
[22:53] Guthur maybe I'm just lucky, hehe
[22:53] pieterh yeah, but he made a *screenshot*
[23:08] Guthur pieterh, the identity.c is setting the identity after connecting
[23:09] Guthur no sorry
[23:09] Guthur my bad
[23:09] Guthur tired eyes
[23:09] pieterh yeah, it does it twice
[23:10] pieterh get some sleep, man!
[23:10] Guthur soon, just want to get this dump working right
[23:11] pieterh :-) well, I fixed the Last Bug in mdbroker, am now crashing...