Friday April 2, 2010

[Time] NameMessage
[00:24] mikko Skaag:
[00:24] mikko initial api docs
[00:24] mikko getting a bit late
[01:03] dermoth mikko, I am myself working on a php binding, but i'm focusing on the procedural interface with access to persistent sockets and thread safety... I wasn't planning an OO interface.
[01:05] dermoth mikko, persistence is very important for what I want as I will send data in short-lived request and expect best-effort delivery while the SAPI is running...
[01:07] dermoth mikko, I haven't written much code yet but I've already read nearly half a book on php extentions and was planning to start real coding very soon... i'm wondering if you'd like any help from me...
[01:11] dermoth hummm I see you have persistence too. I was thinking of doing it explicitly though, with an optional key to allow re-using unique contexts, sockets, etc.
[01:17] mikko dermoth: in my case context is always tied to ZeroMQSocket object
[01:17] mikko and the ZeroMQSocket is stored with persistent_id
[01:18] mikko i didn't see much point for multiple threads as there is no userland threading in php
[01:19] mikko dermoth: hmmm.. what was the idea behind using contexts explicitly?
[01:20] mikko allow setting thread counts and choose different context for a socket?
[01:20] dermoth mikko, this would be my first module, but I was trying to do everything the right way... do you think it would be worth completing it then we can take the best of both? I've also been told OO interface are often built on top or procedural extentions...
[01:20] dermoth mikko, at first I just wanted proper thread separation
[01:20] dermoth i.e. each thread acting as a standalone process
[01:21] dermoth ZMQ sockets cannot be shared anyway, even with locking (according to the api dioc) so only the xcontext could be shared, it there's any advantage with that.
[01:21] mikko which threads?
[01:21] dermoth mpm_worker for example...
[01:22] mikko i'm not that fuzzed about php + threading as large amounts of libraries php interacts with are not thread-safe
[01:22] dermoth and before you ask the fact that many modules still crash in mpm_worker doesn't mean a new module don't need to be thread-safe ;)
[01:22] mikko and with large amounts of globals state the threaded version of php is often slower than single proces
[01:22] dermoth awww you asked! :)
[01:22] mikko globals=global
[01:22] dermoth yeah I know
[01:23] dermoth but i beliebe it's still the best wai under IIS though
[01:23] dermoth I believe it's still the best way...
[01:23] mikko i think microsoft is now recommending fcgi since IIS7
[01:23] dermoth really?
[01:23] mikko yep
[01:23] dermoth I though fcgi was nearly dead...
[01:23] mikko most of their php related development efforts seem to go to fcgi
[01:24] dermoth any chances they fear php competing their .NET market?
[01:24] dermoth lol
[01:24] mikko ISAPI is faster but less stable
[01:25] dermoth yes, but that's only because of php
[01:25] dermoth fix php and isapi will be much better, imho
[01:25] mikko well, by definition native SAPI is always going to be faster
[01:25] mikko fix php and libraries that it uses
[01:26] dermoth yes...
[01:27] dermoth back to php-zeromq, itn't it would be wasteful of resources to have one context per socket, when an application use many sockets?
[01:29] mikko but the sockets are persisted so the context is created only once
[01:29] mikko i had problems allocating the context globally
[01:29] mikko i think the persistence should use EG(persistent_list) instead of random hashtable though
[01:30] dermoth I was wondering, in a threaded environment, if there would be any advantage to have a single context shared amongst threads. but anyway my plan was to have contexts handled by the php code like in the c and python APIs that I use already.
[01:32] dermoth and make persistence explicit trough separate function or optional arguments - persistent contexts/sockets would also have an associated key so that could could have separate persistent sockets going to the same endpoint - and same soe ctx.
[01:33] dermoth I'm not sure how useful this is but I fear trouble when multiple application start doing advanced things using the same sockets...
[01:34] dermoth ex what if cone app gets a rep intended for another...
[01:35] mikko but that's always going to be problematic if you run in a shared environment
[01:36] dermoth yes, but at least I planned an api that allow explicit separation...
[01:36] mikko explicit separation of sockets?
[01:38] dermoth ex: zmq_socket_p($context, $type [, $key]);
[01:38] mikko the only difference is that it allows setting context excplicitly
[01:38] mikko i guess that would make sense in some scenarios
[01:40] dermoth I think we should mimic the zmq api as much as possible. There's a reason behind its design, alter all. copying it gives as much versatility to the language it's being bound to
[01:42] dermoth and adding to it where it makes sense, like adding optional persistence to php to accommodate get general workflow in web servers...
[01:44] mikko not sure how much the threading makes sense without userland threading
[01:44] mikko which C and python have
[01:49] dermoth python threading isn't so much better... although at least it doesn't crash ;)
[01:50] mikko and there is a difference between userland threading and engine level threading
[01:51] mikko to the users point of view php is always 'single threaded'
[01:51] dermoth yea... by thread-safe I mean separating the threads so that they don't clobber each-other
[01:51] dermoth yes
[01:52] dermoth i.e. instead of declaring statics, you have to use macros so that the threaded-version will allocate per-thread structures in a hash.
[01:52] dermoth without zts, though, the macros evaluate to a simple static
[01:54] dermoth ok maybe I will get back to coding then, and than we can see what comes out.
[01:55] mikko but still, i don't get what the point of having different contexts would be
[01:55] mikko as you would store those into resource lists
[01:55] mikko which would be shared between threads
[01:56] mikko and you end up into same problem
[01:56] dermoth no
[01:56] dermoth now shared between threads
[01:56] dermoth not
[01:56] dermoth each thread will have its own contextes...
[01:57] mikko yes, i tried allocating context in GINIT phase but it seemed to mess up everything
[01:57] mikko GINIT should be called per globals invocation, on single process when the process starts and with threading when a thread starts
[01:58] dermoth contexts will be initialized on demand
[01:58] dermoth i will expost zmw_init and zmq_init_p to userland...
[01:58] mikko where do you store them internally?
[01:59] dermoth in EG(persistent_list)
[01:59] mikko i thought persistent_list is shared between threads
[01:59] mikko i might be wrong on that though
[01:59] dermoth is it? you can add the thread_id in that case...
[02:00] dermoth but i hope not as it would all lots of useless locking...
[02:00] dermoth would add
[02:01] dermoth and worst case i'm just enclose a die between #ifdef ZTS :p
[02:01] mikko no, it has to be thread-safe
[02:01] dermoth problem solved...
[02:01] dermoth lol
[02:01] mikko executor globals contains stuff like user erorr handlers
[02:01] mikko and so on
[02:01] mikko they couldn't be shared between threads
[02:02] mikko so effectively my implementation is fairly thread-safe then
[02:02] mikko you live you learn :)
[02:03] dermoth my primary goal is learning anyway... if it might be useful it's a plug :)
[02:03] dermoth plus
[02:03] mikko yes, i think combined procedural + oo is very good for learning
[02:03] mikko you can probably learn alot from pecl/memcache
[02:03] mikko or ext/mysqli
[02:04] dermoth awwww I prefer not to think about oo just yet...
[02:04] dermoth i don't even code PHP! Actually I manage to write simple scripts but that'S all...
[02:05] mikko if the context is in a single thread does it make sense to allocate more than 1 app thread and 1 io thread?
[02:05] dermoth i'm not the one that the use it... I'm the sysadmin
[02:05] dermoth the contex threads are on the library side
[02:06] dermoth the library use threading internally
[02:07] dermoth so yes, someone might want to set more threads if that's the only way to squeeze the last bits of performance out of his cpu cores...
[02:07] dermoth but I have doubts this would apply to php...
[02:09] dermoth it's like putting a huge truck engine on a f1... the vehicle it fast but the engine won't allow it!
[02:11] dermoth i might rip your header... is that the "standard" php extention header?
[02:12] dermoth i mean the copyright block...
[02:16] mikko nope
[02:16] mikko thats the BSD license
[02:17] mikko normally the PHP stuff is under PHP license
[02:17] mikko the stuff in at least
[02:18] dermoth and it that more like LGPL?
[02:19] mikko
[02:19] dermoth no, the php license
[02:19] mikko the only difference between PHP license and BSD license is that PHP license has a naming clause
[02:20] mikko
[02:20] mikko see 4th clause
[02:20] dermoth ok, like the 3-clause BSD
[02:22] mikko added an ini-setting for setting thread counts
[02:23] dermoth so technically php-zeromq wouldn't even be allowed without permission!
[02:24] mikko that's just the name of the repository
[02:24] mikko the product is not called 'php-zeromq'
[02:26] mikko committed a fix to the header to indicate it more clearly :)
[02:27] dermoth i see... i'm not saying that they would though, but packaging it in something like php-zeromq-x.xx.tar.gz would be very borderline...
[02:28] mikko it's just to clarify on github as there is "zeromq2" project already and the language isn't really clear from the context
[02:29] mikko i'm done, 18 hours straight coding is enough for a day
[02:30] mikko good night and good luck with your php expedition
[06:21] desrt hello.
[06:22] desrt what's the amqp story?
[06:25] desrt hm. i'll be back tomorrow. :)
[06:31] sustrik desrt: no big deal, imatix has no stakes in amqp anymore so it's leaving the amqp working group
[06:52] sustrik dermoth, mikko: what about bringing the issue of explicit/implicit context in PHP on the mailing list?
[06:52] sustrik i belive other languages with limited thread support (python? ruby?) are facing the same dilemma
[06:53] dermoth sustrik, sure... I'd like to have something at least partially working first though :)
[06:54] dermoth talking about amqp, why was support for it dropped, I though that was a really nice feature
[06:54] dermoth no patent should cover the basic functionality anyway...
[06:57] sustrik hm, it's a long story...
[06:57] sustrik basically the participation in WG was pain
[06:58] sustrik it's a design-by-committee thing so it's political rather than technical
[06:58] sustrik furthermore, the resulting design is extremely complex
[06:58] dermoth I couldn't find much about it exceps some obscure "patent" issue, but I really liked the feature for one thing I had in mind...
[06:58] sustrik so making it work is really costly
[06:59] sustrik so you have to invest in political fight in the working group as well as in implementation
[06:59] dermoth there is no standard yet?
[06:59] sustrik several versions
[06:59] sustrik mutually incompatible :)
[06:59] dermoth backward-compatible?
[06:59] sustrik nope
[06:59] dermoth ouch
[07:00] sustrik design-be-committee
[07:00] sustrik anyway, so you are investing a lot in the thing
[07:00] sustrik and then you find out the partners are patenting around of the standard
[07:00] sustrik so you get mad and quit
[07:00] sustrik that's basically it
[07:01] dermoth so could it possibly be re-implemented if the WG settles on a final protocol?
[07:01] sustrik you've seen the specification?
[07:01] dermoth no
[07:01] dermoth lol
[07:01] sustrik latest version is 270 pages long
[07:01] sustrik who's going to pay for that?
[07:01] dermoth yes but I guess you don't need all of it...
[07:02] sustrik hm, i think that's the basic spec, let me check...
[07:02] dermoth nm
[07:03] dermoth well anyway it wouldn't be hard to add a glue to it... write a piece of code that use both APIs.
[07:04] dermoth like the zmq devices...
[07:04] sustrik they've got it down to 106 pages, but it's core spec, you have to implement it all
[07:04] sustrik well, you can try
[07:05] sustrik but it's not as easy as it seems
[07:05] sustrik let me explain
[07:05] dermoth i'l talking of using an existing library...
[07:05] sustrik something like a bridge?
[07:05] dermoth yes
[07:05] sustrik yes, that's doable
[07:06] dermoth like a forwarded that use both zmq and an AMQP lib, like RabbitMQ
[07:06] sustrik yes that makes sense
[07:06] dermoth a forwarder
[07:06] sustrik combine rabbitma C client with 0MQ devices
[07:06] sustrik rabbitmq*
[07:07] sustrik if there's an Erlang binding to 0MQ
[07:07] sustrik you would be even able to create a "custome exchange" in rabbitmq server
[07:07] sustrik that would publish messages in 0MQ format
[07:07] dermoth I think rabbitmq had sifferent binding, including python
[07:08] sustrik yes, it does
[07:08] dermoth or could be another broker, but I remember I tested some stuff with python
[07:08] sustrik i think it's real python AMQP client
[07:10] dermoth But in about the same time I found an usable AMQP API I already has finished a working client/server test with ZeroMQ... that's what I like about it :)
[07:11] dermoth had
[07:11] sustrik yup, it's simpler
[07:12] sustrik the basic difference is that AMQP was developed by adding features
[07:12] sustrik while 0MQ was developed by ditching features
[07:12] sustrik :)
[07:12] dermoth btw I was thinking this could be a nice fit for adding authentication/encryption to zeromq:
[07:13] sustrik definitely
[07:13] dermoth another project built on simplicity and speed... unfortunately it doesn't seem very active lately.
[07:14] sustrik i am not very familiar with security
[07:14] sustrik but you may speak to Martin Lucina, he's looking at security aspects of 0mq
[07:14] sustrik As for me, I was thinking of it in terms of messaging on Internet
[07:14] sustrik imagine a network of devices
[07:15] sustrik two terminal application points trust each other
[07:15] sustrik but they don't trust the devices in the middle
[07:15] sustrik what should they do?
[07:15] sustrik it's an interesting area
[07:15] dermoth well I don't think a single open-source crypto package out there takes security as seriously and professionally as this project, which is based on work and participation by D. J. Bernstein
[07:16] dermoth pki is likely what you want...
[07:16] sustrik quite possible, i am not an expert though, feel free to discuss it on the mailing list
[07:16] dermoth share secret keys securely the first time, then tou can trust each other<
[07:17] dermoth err
[07:17] sustrik yes, that's the basic idea
[07:17] dermoth share public keys
[07:17] sustrik not think of 1 to many data distribution
[07:17] sustrik each subscriber had paid for the content
[07:17] sustrik got the key
[07:17] sustrik now, couple of months later
[07:18] sustrik one of the subscribers haven't paid
[07:18] sustrik what now?
[07:18] sustrik change the keys each month?
[07:18] dermoth if you don'T multicast you can use each subscriber public key to encrypt their feed
[07:19] sustrik you can't because you don't trust devices in the middle
[07:19] dermoth that won't wotk with a PUB socket unless you tie in in tight with NaCL though
[07:19] sustrik they cannot do the encryption
[07:19] desrt sustrik: ya... but you almost get the impression that [something] happened
[07:19] dermoth you just have to trust the public key
[07:19] desrt sustrik: just wondering what the something is :)
[07:20] dermoth then ouy know the private key associated with it is the only one that can ecrypt it
[07:20] sustrik desrt: have a look above, we've just discussed it with dermoth
[07:20] desrt oh. ok. i'll read :)
[07:22] sustrik dermoth: i mean, if the middle node has to encrypt messages, original provider has to deliver unecrypted content to the middle node - now recall we don't trust the middle node
[07:22] desrt sustrik: hmm. that seems pretty much like how i imagined
[07:23] dermoth the original provider has to encrypt, or it won't work... or you can di a trust chain between each forwarder
[07:23] sustrik desrt: shrug
[07:23] desrt sustrik: it's not so hard to believe large standards descending into design-by-comittee patent messes
[07:24] sustrik dermoth: if you are interested in the area, take it to the mailing list; you'll get much more educated audience there
[07:24] sustrik desrt: it happens all the time
[07:24] sustrik think of telco standards
[07:24] dermoth ok, but it's not on my roadmap just yet
[07:24] sustrik sure
[07:26] desrt my only concern is taht amqp will become a widely-deployed standard nonetheless
[07:26] desrt and 0mq will be this thing that nobody uses
[07:26] desrt although the community around 0mq looks pretty active at the moment, which is nice
[07:27] sustrik desrt: think of it this way:
[07:27] sustrik "standard" means formalisation of best practices in the area
[07:27] sustrik it's strictly non-experimental
[07:27] sustrik so what AMQP does is to formalise best practices for messaging in corporate environment
[07:28] dermoth well i'm out, it's pretty late here. see you...
[07:28] sustrik cyl
[07:28] desrt ta
[07:28] sustrik which are kind of broken anyway
[07:28] sustrik or maybe not broken
[07:28] sustrik but require large IT depts
[07:28] desrt the other thing is that 0mq seems not to support as rich of a routing model...
[07:29] sustrik you mean complex routing criteria, say a SQL-like subscriptions
[07:29] sustrik ?
[07:29] desrt no. not criteria
[07:29] desrt but more like network traversal/propagation
[07:30] desrt the stuff that brokers did
[07:30] sustrik there are devices that do the same thing
[07:30] sustrik say zmq_forwarder is an app that acts a simple broker
[07:31] desrt so there is still support for asynchronous message queuing, etc
[07:31] sustrik definitely
[07:31] desrt well
[07:31] desrt definitely seems like it's worth looking into for me
[07:31] sustrik sure, have a look
[07:31] desrt you guys made reddit, btw
[07:31] sustrik ah, let me see...
[07:32] desrt
[07:32] desrt that's what brought me here
[07:32] desrt well, it made the frontpage of the programming reddit, anyway
[07:34] sustrik 1000 hits on the website yesterday
[07:34] sustrik not that bad for esoteric networking product
[07:34] sustrik :)
[07:41] desrt you guys need to get ubuntu packages
[07:47] sustrik desrt: there are debian packages atm
[07:47] sustrik should work with ubuntu
[07:47] desrt hm nice
[07:50] desrt i meant more like having stuff in the repository :)
[07:51] sustrik you mean the package build sources?
[07:51] sustrik it's in /debian subdirectory
[07:51] desrt no
[07:51] sustrik a download?
[07:51] desrt i mean like, i should be able to type apt-get install zeromq
[07:52] sustrik ah
[07:52] sustrik it's moving upstream
[07:52] sustrik it takes time till it gets there
[07:52] desrt probably too late for this cycle anyway
[07:53] sustrik it's in "unstable" state
[07:53] sustrik
[07:53] desrt bin, eh?
[07:53] desrt what utils are those?
[07:54] sustrik ah, those are just the devices
[07:55] sustrik source package:
[07:55] sustrik
[07:55] sustrik there are links to lib and dev packages there
[07:56] desrt thanks
[08:05] desrt sustrik: is this debian package up-to-date and in good standing?
[08:07] desrt (ie: no terrible bugs in it that have been fixed in master)
[08:07] sustrik it's the last release (2.0.6)
[08:07] sustrik the master is a bit ustable now
[08:07] sustrik so rather use the last release
[08:08] desrt ok
[08:15] desrt ok. i'm going to push my luck one more time :)
[08:21] desrt sustrik: it seems that the freeze exception is likely to be granted
[08:27] sustrik desrt: i'm not familir with the process
[08:27] sustrik what does that mean?
[08:27] desrt sustrik: normally ubuntu is frozen for new packages at this point (the release is coming out later this month)
[08:27] desrt but i've requested a freeze break
[08:27] sustrik desrt: nice
[08:27] sustrik thanks
[08:28] desrt i want to use this stuff and having packages in the OS obviously makes that a lot easier for me :)
[08:28] sustrik helping with packaging is appreciated
[08:29] desrt heh. i'm not helping. just asking for favours. :)
[08:29] sustrik i'm leaving now; see you all after the holiday
[08:29] sustrik happy Easter!
[08:29] desrt have a good weekend
[08:29] desrt thanks for your help this evening
[10:57] mikko sustrik: good morning
[11:59] jldupont hi - where is the wire-format for 0mq? I'd like to build an Erlang binding for it.
[11:59] jldupont been browsing the site like crazy!!!
[12:01] jldupont anybody??
[12:04] mikko jldupont: can erlang use C libraries?
[12:07] jldupont mikko: yes... but the comm between the C library and erlang will be through.... sockets!
[12:08] jldupont mikko: that's one way of doing it... there is a direct binding but the polling issue surfaces then....
[12:08] jldupont I'd rather bind directly over TCP/UDP
[12:09] mikko jldupont: ok, i think if you wait for a while you might get some help
[12:09] mikko it's good friday and people might be with their families
[12:11] jldupont mikko: understood... thanks!
[12:15] mikko jldupont: it might even be that you need to wait until holiday is over. although you can probably extract the wire protocol from the source
[12:37] FlaPer87 hey guys, are there some guidlines to install 0mq on windows?
[12:38] FlaPer87 I built it with visual studio but I can't find the zeromq_server command, how do I start the server?
[13:37] FlaPer87 is there a sample configuration file somewhere?
[21:51] jomofo Any pointers for troubleshooting UnsatisfiedLinkError (no jzmq on java.library.path) on 64-bit Ubuntu 9.10 ? I installed both libzmq and libjzmq to /usr/local/lib, ran ldconfig, set java.library.path and go, but no luck