ZeroMq IRC Log

Friday April 2, 2010

[Time] Name	Message
[00:24] mikko	Skaag: http://wiki.github.com/mkoppanen/php-zeromq/api-documentation
[00:24] mikko	initial api docs
[00:24] mikko	getting a bit late
[01:03] dermoth	mikko, I am myself working on a php binding, but i'm focusing on the procedural interface with access to persistent sockets and thread safety... I wasn't planning an OO interface.
[01:05] dermoth	mikko, persistence is very important for what I want as I will send data in short-lived request and expect best-effort delivery while the SAPI is running...
[01:07] dermoth	mikko, I haven't written much code yet but I've already read nearly half a book on php extentions and was planning to start real coding very soon... i'm wondering if you'd like any help from me...
[01:11] dermoth	hummm I see you have persistence too. I was thinking of doing it explicitly though, with an optional key to allow re-using unique contexts, sockets, etc.
[01:17] mikko	dermoth: in my case context is always tied to ZeroMQSocket object
[01:17] mikko	and the ZeroMQSocket is stored with persistent_id
[01:18] mikko	i didn't see much point for multiple threads as there is no userland threading in php
[01:19] mikko	dermoth: hmmm.. what was the idea behind using contexts explicitly?
[01:20] mikko	allow setting thread counts and choose different context for a socket?
[01:20] dermoth	mikko, this would be my first module, but I was trying to do everything the right way... do you think it would be worth completing it then we can take the best of both? I've also been told OO interface are often built on top or procedural extentions...
[01:20] dermoth	mikko, at first I just wanted proper thread separation
[01:20] dermoth	i.e. each thread acting as a standalone process
[01:21] dermoth	ZMQ sockets cannot be shared anyway, even with locking (according to the api dioc) so only the xcontext could be shared, it there's any advantage with that.
[01:21] mikko	which threads?
[01:21] dermoth	mpm_worker for example...
[01:22] mikko	i'm not that fuzzed about php + threading as large amounts of libraries php interacts with are not thread-safe
[01:22] dermoth	and before you ask the fact that many modules still crash in mpm_worker doesn't mean a new module don't need to be thread-safe ;)
[01:22] mikko	and with large amounts of globals state the threaded version of php is often slower than single proces
[01:22] dermoth	awww you asked! :)
[01:22] mikko	globals=global
[01:22] dermoth	yeah I know
[01:23] dermoth	but i beliebe it's still the best wai under IIS though
[01:23] dermoth	I believe it's still the best way...
[01:23] mikko	i think microsoft is now recommending fcgi since IIS7
[01:23] dermoth	really?
[01:23] mikko	yep
[01:23] dermoth	I though fcgi was nearly dead...
[01:23] mikko	most of their php related development efforts seem to go to fcgi
[01:24] dermoth	any chances they fear php competing their .NET market?
[01:24] dermoth	lol
[01:24] mikko	ISAPI is faster but less stable
[01:25] dermoth	yes, but that's only because of php
[01:25] dermoth	fix php and isapi will be much better, imho
[01:25] mikko	well, by definition native SAPI is always going to be faster
[01:25] mikko	fix php and libraries that it uses
[01:26] dermoth	yes...
[01:27] dermoth	back to php-zeromq, itn't it would be wasteful of resources to have one context per socket, when an application use many sockets?
[01:29] mikko	but the sockets are persisted so the context is created only once
[01:29] mikko	i had problems allocating the context globally
[01:29] mikko	i think the persistence should use EG(persistent_list) instead of random hashtable though
[01:30] dermoth	I was wondering, in a threaded environment, if there would be any advantage to have a single context shared amongst threads. but anyway my plan was to have contexts handled by the php code like in the c and python APIs that I use already.
[01:32] dermoth	and make persistence explicit trough separate function or optional arguments - persistent contexts/sockets would also have an associated key so that could could have separate persistent sockets going to the same endpoint - and same soe ctx.
[01:33] dermoth	I'm not sure how useful this is but I fear trouble when multiple application start doing advanced things using the same sockets...
[01:34] dermoth	ex what if cone app gets a rep intended for another...
[01:35] mikko	but that's always going to be problematic if you run in a shared environment
[01:36] dermoth	yes, but at least I planned an api that allow explicit separation...
[01:36] mikko	explicit separation of sockets?
[01:38] dermoth	ex: zmq_socket_p($context, $type [, $key]);
[01:38] mikko	the only difference is that it allows setting context excplicitly
[01:38] mikko	i guess that would make sense in some scenarios
[01:40] dermoth	I think we should mimic the zmq api as much as possible. There's a reason behind its design, alter all. copying it gives as much versatility to the language it's being bound to
[01:42] dermoth	and adding to it where it makes sense, like adding optional persistence to php to accommodate get general workflow in web servers...
[01:44] mikko	not sure how much the threading makes sense without userland threading
[01:44] mikko	which C and python have
[01:49] dermoth	python threading isn't so much better... although at least it doesn't crash ;)
[01:50] mikko	and there is a difference between userland threading and engine level threading
[01:51] mikko	to the users point of view php is always 'single threaded'
[01:51] dermoth	yea... by thread-safe I mean separating the threads so that they don't clobber each-other
[01:51] dermoth	yes
[01:52] dermoth	i.e. instead of declaring statics, you have to use macros so that the threaded-version will allocate per-thread structures in a hash.
[01:52] dermoth	without zts, though, the macros evaluate to a simple static
[01:54] dermoth	ok maybe I will get back to coding then, and than we can see what comes out.
[01:55] mikko	but still, i don't get what the point of having different contexts would be
[01:55] mikko	as you would store those into resource lists
[01:55] mikko	which would be shared between threads
[01:56] mikko	and you end up into same problem
[01:56] dermoth	no
[01:56] dermoth	now shared between threads
[01:56] dermoth	not
[01:56] dermoth	each thread will have its own contextes...
[01:57] mikko	yes, i tried allocating context in GINIT phase but it seemed to mess up everything
[01:57] mikko	GINIT should be called per globals invocation, on single process when the process starts and with threading when a thread starts
[01:58] dermoth	contexts will be initialized on demand
[01:58] dermoth	i will expost zmw_init and zmq_init_p to userland...
[01:58] mikko	where do you store them internally?
[01:59] dermoth	in EG(persistent_list)
[01:59] mikko	i thought persistent_list is shared between threads
[01:59] mikko	i might be wrong on that though
[01:59] dermoth	is it? you can add the thread_id in that case...
[02:00] dermoth	but i hope not as it would all lots of useless locking...
[02:00] dermoth	would add
[02:01] dermoth	and worst case i'm just enclose a die between #ifdef ZTS :p
[02:01] mikko	no, it has to be thread-safe
[02:01] dermoth	problem solved...
[02:01] dermoth	lol
[02:01] mikko	executor globals contains stuff like user erorr handlers
[02:01] mikko	and so on
[02:01] mikko	they couldn't be shared between threads
[02:02] mikko	so effectively my implementation is fairly thread-safe then
[02:02] mikko	you live you learn :)
[02:03] dermoth	my primary goal is learning anyway... if it might be useful it's a plug :)
[02:03] dermoth	plus
[02:03] mikko	yes, i think combined procedural + oo is very good for learning
[02:03] mikko	you can probably learn alot from pecl/memcache
[02:03] mikko	or ext/mysqli
[02:04] dermoth	awwww I prefer not to think about oo just yet...
[02:04] dermoth	i don't even code PHP! Actually I manage to write simple scripts but that'S all...
[02:05] mikko	if the context is in a single thread does it make sense to allocate more than 1 app thread and 1 io thread?
[02:05] dermoth	i'm not the one that the use it... I'm the sysadmin
[02:05] dermoth	the contex threads are on the library side
[02:06] dermoth	the library use threading internally
[02:07] dermoth	so yes, someone might want to set more threads if that's the only way to squeeze the last bits of performance out of his cpu cores...
[02:07] dermoth	but I have doubts this would apply to php...
[02:09] dermoth	it's like putting a huge truck engine on a f1... the vehicle it fast but the engine won't allow it!
[02:11] dermoth	i might rip your header... is that the "standard" php extention header?
[02:12] dermoth	i mean the copyright block...
[02:16] mikko	nope
[02:16] mikko	thats the BSD license
[02:17] mikko	normally the PHP stuff is under PHP license
[02:17] mikko	the stuff in php.net at least
[02:18] dermoth	and it that more like LGPL?
[02:19] mikko	http://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_.28.22New_BSD_License.22.29
[02:19] dermoth	no, the php license
[02:19] mikko	the only difference between PHP license and BSD license is that PHP license has a naming clause
[02:20] mikko	http://www.php.net/license/3_01.txt
[02:20] mikko	see 4th clause
[02:20] dermoth	ok, like the 3-clause BSD
[02:22] mikko	added an ini-setting for setting thread counts
[02:23] dermoth	so technically php-zeromq wouldn't even be allowed without permission!
[02:24] mikko	that's just the name of the repository
[02:24] mikko	the product is not called 'php-zeromq'
[02:26] mikko	committed a fix to the header to indicate it more clearly :)
[02:27] dermoth	i see... i'm not saying that they would though, but packaging it in something like php-zeromq-x.xx.tar.gz would be very borderline...
[02:28] mikko	it's just to clarify on github as there is "zeromq2" project already and the language isn't really clear from the context
[02:29] mikko	i'm done, 18 hours straight coding is enough for a day
[02:30] mikko	good night and good luck with your php expedition
[06:21] desrt	hello.
[06:22] desrt	what's the amqp story?
[06:25] desrt	hm. i'll be back tomorrow. :)
[06:31] sustrik	desrt: no big deal, imatix has no stakes in amqp anymore so it's leaving the amqp working group
[06:52] sustrik	dermoth, mikko: what about bringing the issue of explicit/implicit context in PHP on the mailing list?
[06:52] sustrik	i belive other languages with limited thread support (python? ruby?) are facing the same dilemma
[06:53] dermoth	sustrik, sure... I'd like to have something at least partially working first though :)
[06:54] dermoth	talking about amqp, why was support for it dropped, I though that was a really nice feature
[06:54] dermoth	no patent should cover the basic functionality anyway...
[06:57] sustrik	hm, it's a long story...
[06:57] sustrik	basically the participation in WG was pain
[06:58] sustrik	it's a design-by-committee thing so it's political rather than technical
[06:58] sustrik	furthermore, the resulting design is extremely complex
[06:58] dermoth	I couldn't find much about it exceps some obscure "patent" issue, but I really liked the feature for one thing I had in mind...
[06:58] sustrik	so making it work is really costly
[06:59] sustrik	so you have to invest in political fight in the working group as well as in implementation
[06:59] dermoth	there is no standard yet?
[06:59] sustrik	several versions
[06:59] sustrik	mutually incompatible :)
[06:59] dermoth	backward-compatible?
[06:59] sustrik	nope
[06:59] dermoth	ouch
[07:00] sustrik	design-be-committee
[07:00] sustrik	anyway, so you are investing a lot in the thing
[07:00] sustrik	and then you find out the partners are patenting around of the standard
[07:00] sustrik	so you get mad and quit
[07:00] sustrik	that's basically it
[07:01] dermoth	so could it possibly be re-implemented if the WG settles on a final protocol?
[07:01] sustrik	you've seen the specification?
[07:01] dermoth	no
[07:01] dermoth	lol
[07:01] sustrik	latest version is 270 pages long
[07:01] sustrik	who's going to pay for that?
[07:01] dermoth	yes but I guess you don't need all of it...
[07:02] sustrik	hm, i think that's the basic spec, let me check...
[07:02] dermoth	nm
[07:03] dermoth	well anyway it wouldn't be hard to add a glue to it... write a piece of code that use both APIs.
[07:04] dermoth	like the zmq devices...
[07:04] sustrik	they've got it down to 106 pages, but it's core spec, you have to implement it all
[07:04] sustrik	well, you can try
[07:05] sustrik	but it's not as easy as it seems
[07:05] sustrik	let me explain
[07:05] dermoth	i'l talking of using an existing library...
[07:05] sustrik	something like a bridge?
[07:05] dermoth	yes
[07:05] sustrik	yes, that's doable
[07:06] dermoth	like a forwarded that use both zmq and an AMQP lib, like RabbitMQ
[07:06] sustrik	yes that makes sense
[07:06] dermoth	a forwarder
[07:06] sustrik	combine rabbitma C client with 0MQ devices
[07:06] sustrik	rabbitmq*
[07:07] sustrik	if there's an Erlang binding to 0MQ
[07:07] sustrik	you would be even able to create a "custome exchange" in rabbitmq server
[07:07] sustrik	that would publish messages in 0MQ format
[07:07] dermoth	I think rabbitmq had sifferent binding, including python
[07:08] sustrik	yes, it does
[07:08] dermoth	or could be another broker, but I remember I tested some stuff with python
[07:08] sustrik	i think it's real python AMQP client
[07:10] dermoth	But in about the same time I found an usable AMQP API I already has finished a working client/server test with ZeroMQ... that's what I like about it :)
[07:11] dermoth	had
[07:11] sustrik	yup, it's simpler
[07:12] sustrik	the basic difference is that AMQP was developed by adding features
[07:12] sustrik	while 0MQ was developed by ditching features
[07:12] sustrik	:)
[07:12] dermoth	btw I was thinking this could be a nice fit for adding authentication/encryption to zeromq: http://nacl.cace-project.eu/
[07:13] sustrik	definitely
[07:13] dermoth	another project built on simplicity and speed... unfortunately it doesn't seem very active lately.
[07:14] sustrik	i am not very familiar with security
[07:14] sustrik	but you may speak to Martin Lucina, he's looking at security aspects of 0mq
[07:14] sustrik	As for me, I was thinking of it in terms of messaging on Internet
[07:14] sustrik	imagine a network of devices
[07:15] sustrik	two terminal application points trust each other
[07:15] sustrik	but they don't trust the devices in the middle
[07:15] sustrik	what should they do?
[07:15] sustrik	it's an interesting area
[07:15] dermoth	well I don't think a single open-source crypto package out there takes security as seriously and professionally as this project, which is based on work and participation by D. J. Bernstein
[07:16] dermoth	pki is likely what you want...
[07:16] sustrik	quite possible, i am not an expert though, feel free to discuss it on the mailing list
[07:16] dermoth	share secret keys securely the first time, then tou can trust each other<
[07:17] dermoth	err
[07:17] sustrik	yes, that's the basic idea
[07:17] dermoth	share public keys
[07:17] sustrik	not think of 1 to many data distribution
[07:17] sustrik	each subscriber had paid for the content
[07:17] sustrik	got the key
[07:17] sustrik	now, couple of months later
[07:18] sustrik	one of the subscribers haven't paid
[07:18] sustrik	what now?
[07:18] sustrik	change the keys each month?
[07:18] dermoth	if you don'T multicast you can use each subscriber public key to encrypt their feed
[07:19] sustrik	you can't because you don't trust devices in the middle
[07:19] dermoth	that won't wotk with a PUB socket unless you tie in in tight with NaCL though
[07:19] sustrik	they cannot do the encryption
[07:19] desrt	sustrik: ya... but you almost get the impression that [something] happened
[07:19] dermoth	you just have to trust the public key
[07:19] desrt	sustrik: just wondering what the something is :)
[07:20] dermoth	then ouy know the private key associated with it is the only one that can ecrypt it
[07:20] sustrik	desrt: have a look above, we've just discussed it with dermoth
[07:20] desrt	oh. ok. i'll read :)
[07:22] sustrik	dermoth: i mean, if the middle node has to encrypt messages, original provider has to deliver unecrypted content to the middle node - now recall we don't trust the middle node
[07:22] desrt	sustrik: hmm. that seems pretty much like how i imagined
[07:23] dermoth	the original provider has to encrypt, or it won't work... or you can di a trust chain between each forwarder
[07:23] sustrik	desrt: shrug
[07:23] desrt	sustrik: it's not so hard to believe large standards descending into design-by-comittee patent messes
[07:24] sustrik	dermoth: if you are interested in the area, take it to the mailing list; you'll get much more educated audience there
[07:24] sustrik	desrt: it happens all the time
[07:24] sustrik	think of telco standards
[07:24] dermoth	ok, but it's not on my roadmap just yet
[07:24] sustrik	sure
[07:26] desrt	my only concern is taht amqp will become a widely-deployed standard nonetheless
[07:26] desrt	and 0mq will be this thing that nobody uses
[07:26] desrt	although the community around 0mq looks pretty active at the moment, which is nice
[07:27] sustrik	desrt: think of it this way:
[07:27] sustrik	"standard" means formalisation of best practices in the area
[07:27] sustrik	it's strictly non-experimental
[07:27] sustrik	so what AMQP does is to formalise best practices for messaging in corporate environment
[07:28] dermoth	well i'm out, it's pretty late here. see you...
[07:28] sustrik	cyl
[07:28] desrt	ta
[07:28] sustrik	which are kind of broken anyway
[07:28] sustrik	or maybe not broken
[07:28] sustrik	but require large IT depts
[07:28] desrt	the other thing is that 0mq seems not to support as rich of a routing model...
[07:29] sustrik	you mean complex routing criteria, say a SQL-like subscriptions
[07:29] sustrik	?
[07:29] desrt	no. not criteria
[07:29] desrt	but more like network traversal/propagation
[07:30] desrt	the stuff that brokers did
[07:30] sustrik	there are devices that do the same thing
[07:30] sustrik	say zmq_forwarder is an app that acts a simple broker
[07:31] desrt	so there is still support for asynchronous message queuing, etc
[07:31] sustrik	definitely
[07:31] desrt	well
[07:31] desrt	definitely seems like it's worth looking into for me
[07:31] sustrik	sure, have a look
[07:31] desrt	you guys made reddit, btw
[07:31] sustrik	ah, let me see...
[07:32] desrt	http://www.reddit.com/r/programming/comments/bl9rb/while_imatix_was_the_original_designer_of_amqp_we/
[07:32] desrt	that's what brought me here
[07:32] desrt	well, it made the frontpage of the programming reddit, anyway
[07:34] sustrik	1000 hits on the website yesterday
[07:34] sustrik	not that bad for esoteric networking product
[07:34] sustrik	:)
[07:41] desrt	you guys need to get ubuntu packages
[07:47] sustrik	desrt: there are debian packages atm
[07:47] sustrik	should work with ubuntu
[07:47] desrt	hm nice
[07:50] desrt	i meant more like having stuff in the repository :)
[07:51] sustrik	you mean the package build sources?
[07:51] sustrik	it's in /debian subdirectory
[07:51] desrt	no
[07:51] sustrik	a download?
[07:51] desrt	i mean like, i should be able to type apt-get install zeromq
[07:52] sustrik	ah
[07:52] sustrik	it's moving upstream
[07:52] sustrik	it takes time till it gets there
[07:52] desrt	probably too late for this cycle anyway
[07:53] sustrik	it's in "unstable" state
[07:53] sustrik	http://packages.debian.org/unstable/zeromq-bin
[07:53] desrt	bin, eh?
[07:53] desrt	what utils are those?
[07:54] sustrik	ah, those are just the devices
[07:55] sustrik	source package:
[07:55] sustrik	http://packages.debian.org/source/sid/zeromq
[07:55] sustrik	there are links to lib and dev packages there
[07:56] desrt	thanks
[08:05] desrt	sustrik: is this debian package up-to-date and in good standing?
[08:07] desrt	(ie: no terrible bugs in it that have been fixed in master)
[08:07] sustrik	it's the last release (2.0.6)
[08:07] sustrik	the master is a bit ustable now
[08:07] sustrik	so rather use the last release
[08:08] desrt	ok
[08:15] desrt	ok. i'm going to push my luck one more time :)
[08:21] desrt	sustrik: it seems that the freeze exception is likely to be granted
[08:27] sustrik	desrt: i'm not familir with the process
[08:27] sustrik	what does that mean?
[08:27] desrt	sustrik: normally ubuntu is frozen for new packages at this point (the release is coming out later this month)
[08:27] desrt	but i've requested a freeze break
[08:27] sustrik	desrt: nice
[08:27] sustrik	thanks
[08:28] desrt	i want to use this stuff and having packages in the OS obviously makes that a lot easier for me :)
[08:28] sustrik	helping with packaging is appreciated
[08:29] desrt	heh. i'm not helping. just asking for favours. :)
[08:29] sustrik	i'm leaving now; see you all after the holiday
[08:29] sustrik	happy Easter!
[08:29] desrt	have a good weekend
[08:29] desrt	thanks for your help this evening
[10:57] mikko	sustrik: good morning
[11:59] jldupont	hi - where is the wire-format for 0mq? I'd like to build an Erlang binding for it.
[11:59] jldupont	been browsing the site like crazy!!!
[12:01] jldupont	anybody??
[12:04] mikko	jldupont: can erlang use C libraries?
[12:07] jldupont	mikko: yes... but the comm between the C library and erlang will be through.... sockets!
[12:08] jldupont	mikko: that's one way of doing it... there is a direct binding but the polling issue surfaces then....
[12:08] jldupont	I'd rather bind directly over TCP/UDP
[12:09] mikko	jldupont: ok, i think if you wait for a while you might get some help
[12:09] mikko	it's good friday and people might be with their families
[12:11] jldupont	mikko: understood... thanks!
[12:15] mikko	jldupont: it might even be that you need to wait until holiday is over. although you can probably extract the wire protocol from the source
[12:37] FlaPer87	hey guys, are there some guidlines to install 0mq on windows?
[12:38] FlaPer87	I built it with visual studio but I can't find the zeromq_server command, how do I start the server?
[13:37] FlaPer87	is there a sample configuration file somewhere?
[21:51] jomofo	Any pointers for troubleshooting UnsatisfiedLinkError (no jzmq on java.library.path) on 64-bit Ubuntu 9.10 ? I installed both libzmq and libjzmq to /usr/local/lib, ran ldconfig, set java.library.path and go, but no luck