Wednesday June 22, 2011

[Time] NameMessage
[02:08] cody-somerville So whats the catch with ZeroMQ? It seems too good to be true. I mean, how has it taken this long for us to get ZeroMQ? :S
[02:17] cremes cody-somerville: yes, definitely too good to be true... keep walking! :)
[02:22] CIA-32 pyzmq: 03MinRK 07master * rf34775a 10/ (16 files in 8 dirs): Merge remote branch 'dln/master' -
[02:22] CIA-32 pyzmq: 03MinRK 07master * r1dd8a3c 10/ zmq/core/constants.pyx : remove tabs in constants.pyx ...
[02:23] cody-somerville Are the things proposed on being proposed for inclusion with zeromq or just proposing a standard that people can implement themselves?
[04:38] pieterh cody-somerville: the RFCs cover both aspects of 0MQ, and layers built on top
[07:37] whack There's no way to start a socket on an ephemeral port, is there? Like normally you'd bind on port 0 and get a random port, zmq binding on "tcp://*:0" aborts
[07:41] whack well, it doesn't abort, it just gets on a high horse with EINVAL and such
[07:42] sustrik no, you can't bind to ephemeral port
[07:43] whack sustrik: nod, I also saw there's no way to ask what port a thing was bound on. Worth filing a feature request on?
[07:44] whack I could also probably produce a patch that adds this feature as it should be pretty easy
[07:45] sustrik it's not
[07:45] whack what's not?
[07:45] sustrik you don't have an API to find out what the port number is
[07:45] pieterh whack: I've implemented that in CZMQ
[07:45] pieterh it's pretty simple
[07:45] whack sustrik: right, but like ZMQ_FD, it could be added to the zmq_getsockopt (which seems overloaded already)
[07:45] sustrik nope, because you can have several binds on a single socket
[07:46] pieterh sustrik: the bind command could return the port it actually used
[07:46] pieterh since bind is synchronous
[07:46] whack pieterh: that would break backwards compat, though
[07:46] pieterh whack: yup
[07:47] pieterh getsockopt is the alternative, it'd work, for the last bind each time
[07:47] pieterh how does TCP return ephemeral ports?
[07:47] whack you bind on it, then call get getsockname(2) to get the poor
[07:47] whack *port
[07:48] pieterh right
[07:48] whack pieterh: honestly though, I don't have preference on implementation (whether the kernel does the ephemeral port selection or userspace does it like your czmq)
[07:48] pieterh so analogous with getsockopt
[07:48] whack nod
[07:48] pieterh it's so easy to do in the language binding...
[07:48] pieterh for (port = ZSOCKET_DYNFROM; port < ZSOCKET_DYNTO; port++) {
[07:49] pieterh you're not the first person to ask for this in libzmq...
[07:49] pieterh obviously that would make it available to all bindings for free
[07:49] whack I'd be happy with a 2nd zmq_bind method
[07:49] sustrik i don't like the idea
[07:49] whack if you have a preferred implementation, I'm happy to write the patch
[07:49] sustrik it would turn into next "manual identity" issue in the future
[07:50] pieterh sustrik: ?
[07:50] whack sustrik: ephemeral ports? or specifics of the implementation being discussed?
[07:50] pieterh that is a weird argument against a useful feature
[07:50] sustrik i mean a feature added because of popular pressure that turns to be a pain in the ass
[07:50] pieterh actually, not just useful, but mandatory for some use cases
[07:50] sustrik but cannot be removed because of backward compatibility
[07:50] pieterh why on earth would this be a pita?
[07:51] pieterh all you're going to do is force people to layer it on top
[07:51] sustrik binds designate stable pieces in topology
[07:51] pieterh no they do no
[07:51] pieterh not
[07:51] pieterh that is one scenario
[07:51] pieterh it's absolutely not the only case
[07:51] whack sustrik: in my case, I the main inteface to my program is req/rep
[07:51] pieterh binds designate addressability
[07:51] whack where one api call is "I want a stream of the following things"
[07:52] pieterh in some cases, unstable pieces have to bind because they are the only pieces that are addressable
[07:52] whack and pre-selecting a port number for that (and other) streams is a huge pain
[07:52] pieterh indeed
[07:52] whack when all I care about is "an unused port, then tell me what port that is"
[07:52] pieterh obviously
[07:52] pieterh whack: what language are you working in?
[07:52] whack like at loggly, we have some rediculous port ranges "reserved" for zmq "just in case" we need them
[07:52] whack pieterh: C right now
[07:52] pieterh whack: I'd suggest using CZMQ
[07:52] pieterh and then, send me a patch for zeromq/2.2
[07:52] sustrik it'll make a mess when combined with naming service in the future
[07:53] sustrik anyway, it's available in czmq
[07:53] pieterh sustrik: I disagree most strongly
[07:53] pieterh any name service has to be dynamic
[07:53] whack sustrik: isn't it perfect for the naming service?
[07:53] sustrik it's terrible
[07:53] pieterh whack: it's pointless to discuss this with sustrik since he doesn't actually *use* zeromq
[07:53] sustrik because it makes the topology dynamic
[07:53] pieterh sigh
[07:53] whack haha ;)
[07:53] pieterh whack: seriously, you'll get stuck in a twisty maze of theoretical "can'ts"
[07:54] pieterh just send me a patch, if it's clean I'll add it to 2.2
[07:54] whack sustrik: on the other hand, it guarantees zero port conflicts.
[07:54] whack pieterh: sold, I'll try with czmq for a bit
[07:54] pieterh yeah
[07:54] sustrik sure, but my point is that it's easy to add a feature but almost impossible to remove it
[07:54] pieterh btw the syntax there is tcp://*:*
[07:54] pieterh feel free to send me a patch to change that to tcp://*:0
[07:54] whack pieterh: I like that better than *:0 anyway
[07:54] pieterh it is more consistent with "any", yes
[07:55] pieterh also, please take CZMQ from the git master, the tarballs are out of date now
[07:55] pieterh I need to make a new release
[07:55] whack pieterh: righto :)
[07:55] whack pieterh: what's the license on czmq? i don't see one
[07:55] pieterh lgpl, it's there somewhere... hang on
[07:55] sustrik pieterh: btw, speaking about removing features
[07:56] pieterh sustrik: you need the ability to experiment
[07:56] sustrik what about manual identities?
[07:56] pieterh add, try, remove
[07:56] pieterh yes, manual identities should go
[07:56] sustrik how?
[07:56] pieterh well... this is the process...
[07:56] sustrik i would say first ste would be to remove any mention of them from the guide
[07:56] pieterh first, you gather the use cases to understand what people are really trying to do
[07:56] sustrik step
[07:57] pieterh second, you provide better alternative solutions
[07:57] sustrik removing it from docs would guarantee that no new users will start using it
[07:57] pieterh third, you deprecate the feature and disdocument it
[07:57] sustrik then we can take any further steps
[07:57] pieterh fourth, once you have successfully migrated users, you remove it
[07:58] pieterh this is the process, version numbering can help but is not a replacement for it
[07:58] sustrik while you do steps 1-4 there will be new users of the feature coming it
[07:58] pieterh that's fine
[07:58] sustrik so you'll never finish the process
[07:58] pieterh well, sure
[07:58] pieterh the process will finish
[07:59] pieterh this is not magic, it starts by collecting use cases
[07:59] pieterh for durable pubsub, for example, we have the Clone pattern
[07:59] sustrik so you want to keep manual identities in the guide for now?
[07:59] pieterh yes, until I have a clearly better alternative
[07:59] sustrik ok
[08:00] pieterh I'm happy to remove it any time, once we have a proper plan
[08:00] pieterh regarding adding new features, I think it's really important to listen to users
[08:01] sustrik that's how we've got the manual identities :)
[08:01] pieterh well, perhaps they were necessary
[08:01] pieterh it's all a process
[08:02] pieterh I mean, perhaps they were a necessary step on the way to learning how to properly make this stuff
[08:03] sustrik sure
[08:03] sustrik but the consequence is that i'm dealing with terrible codebase now
[08:03] sustrik with no real propspect of simplification
[08:04] pieterh well, I have a proposal...
[08:05] pieterh you do realize there is quite a lot of back pressure for specific new features in libzmq
[08:05] pieterh - queue size visibility
[08:05] pieterh - ephemeral sockets
[08:05] sustrik sure
[08:05] pieterh - protocol documentation
[08:05] sustrik ack
[08:05] pieterh give a little, take a little
[08:06] pieterh if the cost of making 3.0 really better is we lose explicit identities, that's a fair deal
[08:06] pieterh especially if we (users) get functionality we desperately need
[08:06] sustrik there's subscription forwarding there people were asking about for 2 years
[08:06] pieterh yes, for example
[08:06] sustrik can i drop manual identities now?
[08:06] pieterh yes
[08:07] sustrik c'mon people are using it
[08:07] pieterh not in 3.0, they're not
[08:07] pieterh and afaics all the use cases for that are bogus
[08:07] pieterh with a very few exceptions
[08:07] sustrik hm, so we can keep people like brian granger on 2.0
[08:07] sustrik with no prospect of upgrading
[08:08] pieterh well, first of all, anyone using 2.x in production will probably *never* upgrade
[08:08] sustrik unless they rewrite their platform
[08:08] sustrik how long do you plan to maintain 2.x?
[08:08] pieterh second, have you actually asked Brian what he's doing with identities?
[08:08] sustrik nope
[08:08] pieterh 2.x will be maintained until there's a stable replacement, I'd guess 1-2 years
[08:08] pieterh ask
[08:08] sustrik the point it they have it deployed
[08:08] sustrik and cannot change it
[08:09] pieterh that's not the point
[08:09] pieterh if the deployment is really fixed, there'll be no upgrades anyhow
[08:09] pieterh if there will be upgrades, it's not fixed
[08:09] pieterh look, I'm speaking as a user here, with several dozen applications to maintain
[08:10] pieterh you need more accurate understanding of how much change people are willing to accept, on what basis
[08:10] pieterh random change "because sustrik wants it"... not so good
[08:10] pieterh improvements ... yes, definitely
[08:11] sustrik what we are speaking about now?
[08:11] pieterh mix of the two... well, ok, sure, I'll try it
[08:11] sustrik manual identities still?
[08:11] pieterh yes
[08:11] sustrik ah: "sustrik wants to remove manual ids"
[08:11] sustrik i see
[08:11] pieterh if you really want to drop these, just make sure you give something back
[08:12] sustrik i think i already did
[08:12] pieterh ok, here's my wishlist...
[08:12] sustrik nope. there's subscription forwarding there
[08:12] pieterh ... please document your protocols before you code them
[08:13] pieterh ... please do listen when people ask for stuff like ephemeral ports
[08:13] sustrik it brings 0mq to whole stock trading industry
[08:13] pieterh if both Mato and myself think feature X is valuable, you should listen
[08:13] pieterh ... please do find a way to make sys://log work, and log significant events
[08:13] pieterh ... please keep ROUTER working, we really need it
[08:14] sustrik sys://log works
[08:14] pieterh ... please continue on reliability, as you've been doing for REQ/REP
[08:14] sustrik ROUTER works
[08:14] pieterh yes, but the code base doesn't use sys://log, nor do language bindings know how to use it
[08:14] sustrik it's a standard API, so anyone can use it in any language
[08:15] sustrik the codebase has to log more events
[08:15] pieterh it's not documented, is it
[08:15] sustrik ah
[08:15] sustrik should be
[08:15] sustrik hm
[08:15] pieterh so, stability, monitoring, tweaking of existing patterns
[08:15] pieterh this is all valuable, excellent stuff
[08:16] pieterh and so we lose manual identities... sniff...
[08:16] sustrik as for the reliability thing: i am working on it right now
[08:16] pieterh yes, I'm following that
[08:16] pieterh it would be cool if you documented the WLP with labels and such since I'm working on the very same stuff in VTX
[08:16] sustrik it's in state of flux
[08:16] pieterh reverse engineering C++ is not a sane way to get interoperability
[08:17] pieterh c'mon
[08:17] pieterh mark it as "draft"
[08:17] sustrik this morning i've implemented 32-bit elements in the backtrace stack
[08:17] pieterh so?
[08:17] pieterh then you edit the draft and click Save
[08:17] pieterh it takes you 5 minutes
[08:17] sustrik i would have to rewrite the spec every day
[08:17] pieterh 'edit' not rewrite
[08:17] pieterh and yes, that's why it's a wiki
[08:18] sustrik wait while it's ready
[08:18] sustrik you are not using 3.0 anyway
[08:18] pieterh then I'm not going to support dropping manual identities, sorry
[08:18] sustrik ok
[08:18] pieterh for which, btw, there is at least one real use case
[08:19] pieterh like I said, four steps
[08:19] sustrik what's the real use case?
[08:20] pieterh router to router connections
[08:20] pieterh which we need for certain p2p topologies
[08:20] pieterh since routers can't proactively address a peer, one of the routers has to use an explicit identity
[08:20] pieterh we use the endpoint string
[08:21] pieterh in fact if ROUTER switched to using a schemed identity, that might solve it
[08:21] pieterh then you could route proactively
[08:21] sustrik that's exactly what 0mq should do abstract you from
[08:21] pieterh yes, in all patterns except router
[08:21] sustrik if you want to send to specific peer use TCP, UDP etc.
[08:21] sustrik there's plethora of means to do that
[08:21] pieterh oh, sure
[08:22] pieterh I'll just write 1000 lines of TCP for fun
[08:22] pieterh excellent idea
[08:22] pieterh let me add that to the FAQ
[08:22] sustrik there are many products that make it easy for you
[08:22] sustrik ACE
[08:22] pieterh please stop
[08:22] sustrik Boost.ASIO
[08:22] sustrik etc.
[08:23] pieterh cool
[08:23] pieterh I'll make my network out of a mix of 0MQ and Boost.ASIO
[08:23] pieterh excellent idea
[08:24] pieterh hey, why not remove the whole req/rep pattern, in fact?
[08:24] pieterh just turn 0MQ into a pure pub/sub framework
[08:24] pieterh after all, Boost.ASIO can do req/rep just as well
[08:24] sustrik the thing is that with strictly defined patterns i can provide certain guarantees
[08:24] sustrik such as scalability
[08:24] pieterh can you?
[08:24] sustrik reliability etc.
[08:25] pieterh can you?
[08:25] sustrik yes, because user is constrained in what he can do
[08:25] pieterh perhaps with pubsub, yes
[08:25] sustrik so there's no way to insane things
[08:25] pieterh with req-rep not a single person who answered that survey used multiply hops
[08:25] sustrik huh?
[08:25] pieterh except over hand-made router devices
[08:25] pieterh no-one is doing what you think they want to do, with req-rep
[08:25] sustrik multi-threaded server?
[08:26] pieterh a toy example
[08:26] sustrik anyway
[08:26] whack pieterh: load balancers are good examples of multiple-hop req/rep
[08:26] sustrik yes
[08:26] pieterh whack: sure, but as soon as you want to make them even semi-reliable you have to write your own using ROUTER sockets
[08:26] pieterh that's the whole issue here
[08:27] sustrik that's why i am trying to implement the reliability part that was missing so far
[08:27] pieterh good luck, especially if you ignore the mass of work already done
[08:27] pieterh sorry if I sound annoyed
[08:27] pieterh I am
[08:27] sustrik i think it's the same thing as you paranoid pirate pattern
[08:27] pieterh you think
[08:28] pieterh rtfm
[08:28] pieterh I don't have energy to repeat this, if you pretend to know what people want and need without (a) using the tool yourself and (b) asking them or (c) reading their work
[08:28] pieterh then what you make is based largely on your own prejudicies
[08:29] pieterh *prejudices
[08:29] pieterh and these will be accurate only to 70% or so
[08:29] sustrik definitely
[08:29] sustrik but the problem here is about layering not functionality
[08:29] sustrik so you've implemented N patterns on top of 0MQ
[08:29] sustrik i've implemented 3 pattern inside 0MQ
[08:30] pieterh the reason for implementing these patterns was to capture the knowledge of real use cases
[08:30] sustrik which way is the right way?
[08:30] pieterh the goal at the time was to experiment on top, and provide knowledge that could be internalized
[08:30] pieterh have you looked at the heartbeating in the various HL patterns?
[08:30] pieterh have you discussed this anywhere?
[08:30] pieterh if not, how on earth can you implement it properly?
[08:31] pieterh it took the websockets list a year to make heartbeating
[08:31] sustrik the laternative is to work on topic branches
[08:31] pieterh the problem is not layering
[08:31] pieterh that's a solution
[08:31] sustrik to a political problem
[08:31] sustrik you vs. me
[08:31] sustrik i handle the lib stuff
[08:31] sustrik you handel the stuff on top
[08:31] pieterh well, you vs. a lot of people, to be honest
[08:32] sustrik technically, patterns should be at a single layer
[08:32] sustrik not two of them
[08:32] sustrik so there are 2 options
[08:32] pieterh well, it's technically impossible to experiment in libzmq
[08:32] sustrik 1. make 0mq a generic framework such as ACE and build patterns on top
[08:32] pieterh and without experimentation you cannot design the right solutions
[08:33] pieterh libzmq's complexity makes it impossible to open up and play with
[08:33] sustrik you can experiment with libzmq
[08:33] sustrik the problem is that's it's not easy
[08:33] pieterh look at subscription forwarding, in the end only you could make it work
[08:33] sustrik yes, the codebase is to complex
[08:33] pieterh well, then don't ask people to participate
[08:33] pieterh they'll just feel stupid
[08:34] sustrik i would like to make it simpler
[08:34] sustrik say by removing manual identities
[08:34] pieterh here is the problem, martin
[08:34] pieterh you do stuff without consultation
[08:34] pieterh good design would be much easier if you started by writing it down
[08:34] pieterh instead of communicating in code, as Mato said
[08:34] pieterh meeting, cyl
[08:35] sustrik see you
[09:09] kventil aloha
[09:20] sustrik hi
[12:36] mile does anyone know wether JZMQ is thread-safe?
[12:52] eintr mile: it should be. but sockets are not.
[12:52] eintr mile: as a rule, don't share sockets between threads
[12:53] mile ok, so either a socket per thread, or synchronize per socket
[12:53] mile thanks
[12:53] eintr socket per thread definitely.. then aggregate using an inproc device
[12:53] john I want to be able to send a message out from one point to many nodes at the same time and I want to be able to ensure the delivery of the message to all the nodes. Is there a way you can esure the delivery of a message using pub/sub?
[12:54] eintr john__: only by sending an ack back (through another socket, presumably)
[12:55] john yeah that's how I have it set up now. I was just wondering if pub/sub has some feature I didn't know about.
[12:55] eintr mile: it's very convenient to use zmq as an "actor" library, relying on it for concurrency and sharing immutable structures... it lends itself very well to this.
[12:56] mile that would speak for one socket per thread?
[12:56] eintr mile: definitely.. a socket is also a queue / mailbox, so it's a 1-1 match w/actor design
[16:57] michelp another huge backscroll morning :)
[16:57] michelp always fun to see what happens overnight
[17:01] sustrik good morning :)
[17:02] michelp for some reason xchat paints you and pieterh as the same color, and you have the same number of chars in your names, reading your conversations is a little eye-crossing
[17:02] whack any recommendations on doing authentication? On both req/req and pub/sub? I'd guess encryption and key rotation, but any alternatives?
[17:02] michelp hmac?
[17:02] michelp that's what i've been using
[17:03] pieterh whack: have a read here:
[17:03] whack michelp: that would just produce signatures, so I hold it under the crypto umbrella with encryption
[17:03] pieterh michelp: I hope it's 'neocortex red'
[17:03] whack pieterh: ahh cool
[17:03] michelp heh, unfortunately i don't think xchat is that configureable, and if it were i wouldn't waste my time figuring it out
[17:03] pieterh whack: we could use some constructive criticism of that design
[17:04] whack pieterh: I remember you talking about something similar at the sf zmq meetup
[17:04] pieterh whack: yes, this is the Salt design, somewhat generalized
[17:08] ianbarber i had a look at that, other than the comments already made nothing much occurred - it even fairly naturally extends to a multi-master tree afaics
[17:11] pieterh so we think it'll work?
[17:13] sustrik +1
[17:17] pieterh whack: do you have a use case that needs pubsub security?
[17:19] whack pieterh: yep; I'm working on a process runner similar to daemontools/supervisord/runit/etc, and I'd like clients to be able to subscribe to events (log messages, state changes, etc)
[17:20] whack but only allowing that information to authenticated (and authorized) clients
[17:20] pieterh whack: the ideal would be to test that security design with real code, and when we are happy with it, turn it into an rfc
[17:21] whack pieterh: nod, I'll try to implement the pubsub-security thing you pointed me at
[17:21] pieterh ok, that'd be great... as you do so, you'll find gaps in the text, which I'll fill in
[17:21] pieterh at the end we should have a reference implementation and a proven draft
[17:21] pieterh you'll be making your implementation open source?
[17:21] whack perhaps with extensions, since I'd prefer to have only one pubsub socket open for sanity with N clients, which means each event type may need it's own key
[17:21] whack pieterh: yep
[17:21] pieterh lovely
[17:22] pieterh I'd like to cover any extensions etc. so we have real interoperability
[17:22] pieterh in the few cases where we've properly documented protocols, we do get people reusing stacks
[17:22] pieterh or reimplementing and interoperating
[17:23] whack I think the use case I'm targeting is multiple channels on the same pubsub (with one rotating key per channel)
[17:24] pieterh whack: you might find the whitepaper at helpful
[17:25] pieterh this is the security model proposed for that
[17:25] pieterh (there is no software for whaleshark but we're collecting the pieces slowly)
[17:27] michelp whack, just out of curiosity, why are you writing your own instead of using one of the ones you mentioned?
[17:27] whack michelp: using one of the what?
[17:27] michelp process runner
[17:28] whack michelp: because they're all piles of meh
[17:28] whack I've used supervisord and daemontools for years; supervisord has terrible design flaws and daemontools is missing features
[17:28] whack and because I really like coding and want to see what I can come up with that's better :)
[17:28] michelp yeah daemontools was never my friend. i've had luck with supervisord, but haven't strained it too hard
[17:29] michelp adding 0mq event publishing to it was pretty trivial
[17:29] michelp (to supervisor)
[17:29] whack michelp: supervisord, for example, implements 'restart' as "stop, start" - so if the stop stalls, and the 'restart' times out, your process never starts again.
[17:30] whack among other bugfeatures ;)
[17:30] michelp we have seen some very slow restart performance when we do rolls
[17:31] whack I also like some of the neater features of upstart/systemd, but both of those are terrible, so I'll cherrypick features
[17:32] whack afk for a bit
[17:51] michelp whack, well when you come up with something usable let me know, if you want other people to kick the tires on it, i use supervisor to run a cluster of 0mq peers and wouldn't mind something more.... native.
[17:51] michelp currently using supervisor event listener processes to publish 0mq events to brokers when workers die, and it's definitely not elegant
[22:55] al_nunn can someone help setup zeromq for use with c#?
[22:55] al_nunn I am stuck with "N.B. The ZeroMQ library file will need to be made available for the binding to work."
[22:55] al_nunn when trying to build the clrzmq project
[22:58] michelp al_nunn, i can't help you specifically, sorry. is that an error message your getting?
[22:59] michelp in general any binding that uses 0mq will need to have access to the libzmq file, which on unix, for example, would probably be someplace like /usr/lib (or wherever you told it to install)
[22:59] al_nunn Ordinarily i would have thought you would just "reference" the library
[23:00] michelp what platform are you running on?
[23:00] michelp C#, i presume, windows?
[23:00] al_nunn but i am getting errors when i try to do that from MS Visual C# 2010 Express that it is not a valid .net or COM library (which i imagine is correct so it fails)
[23:01] al_nunn but i dont see how to make use of the library from any C# code
[23:01] michelp hmm.. sorry that whole platform is pretty much all alien to me.
[23:02] michelp but someone might chime in with some helpful advice
[23:02] michelp as a totally wild ass guess, i would presume it would be looking for some .DLL file somewhere in a well known path
[23:03] michelp the 'reference' you were talking about might just be telling the development environment file you are using where that file is
[23:03] al_nunn i have tried putting it in C:\Windows\System32
[23:04] al_nunn thats what i meant by reference
[23:04] al_nunn but it wont allow me to add it as i would a normal .net assembly
[23:05] michelp there's probably a few .net users about, but you might get more platform specific help in a .net specific channel
[23:05] al_nunn i would think it is something very simple. but i cannot find any simple documentation for getting it all built and started
[23:05] al_nunn how can i change channels
[23:05] michelp that depends on your client, but you could try someting like '/j #somechannel'
[23:06] michelp whatever the .net channel might be
[23:06] al_nunn I am just using the web client on the zeromq site...
[23:06] michelp oh i see
[23:06] michelp sorry can't help you there then
[23:06] al_nunn thanks for trying anyway
[23:07] michelp i'm sure there are .net channel on this network, try downloading an irc client and reconnecting
[23:07] al_nunn i will plod on and see if i can find some infor