IRC Log

Tuesday April 13, 2010

[Time] NameMessage
[01:08] iFire mato, sustrik if I decide to make a custom udp layer for game communication i.e. custom (reliability, ordered) where should I start?
[01:27] iFire i.e. making 0mq do that
[03:06] enki yo, i'm sure you guys are biased - but any opinions on beanstalkd?
[05:06] sustrik iFire: reliable ordered UDP or unreliable unordered UDP?
[05:06] iFire both
[05:07] iFire http://www.jenkinssoftware.com/raknet/manual/reliabilitytypes.html
[05:07] iFire example
[05:07] sustrik why not use TCP for the former?
[05:08] sustrik as for the latter, clone the PGM part of the codebase
[05:08] iFire well sometimes if you miss you get a visual error
[05:08] iFire there's unreliable, unreliable sequenced, reliable, reliable sequenced
[05:08] iFire in that raknet page
[05:08] sustrik tcp is reliable sequenced
[05:09] sustrik that's what you want, no?
[05:09] iFire I want unreliable sequenced
[05:09] iFire and unrealiable
[05:10] sustrik ok, then clone the pgm part
[05:10] sustrik (4 files)
[05:10] sustrik ditch the entrails
[05:10] sustrik and replace them by UDP code
[05:10] sustrik to get sequenced you'll have to number the packets and drop out-of-order packets
[05:11] sustrik enki: sorry, never used it
[05:12] iFire well it's be symmetrical if I have all four combinations in udp
[05:13] iFire sustrik it's in zeromq2/src?
[05:13] sustrik reliabilty is not easy to implement
[05:13] sustrik and it's provided by tcp anyway
[05:13] sustrik iFire: yes
[05:13] sustrik pgm_sender/pgm_receiver
[05:16] iFire there's not problem with mixing messages right?
[05:17] iFire no problem
[05:17] sustrik mixing?
[05:17] iFire udp/tcp
[05:17] sustrik no, it's different protocol
[05:17] sustrik btw, for a wire format you have to implement have a look here
[05:17] sustrik http://api.zeromq.org/zmq_pgm.html
[05:18] sustrik it's already in the pgm part so you can leave that in
[05:20] iFire well I want the benefits of being packet based rather than stream based
[05:20] iFire but just implementing unreliable and unreliable sequenced I can see how much work to do
[05:22] sustrik iFire: designing a sane raliable protocol is hard; unless you are a networking researcher you don't even want to try
[05:23] sustrik you would have to implement tx/rx windowing
[05:23] sustrik congestion control
[05:23] sustrik etc.
[05:23] sustrik all that stuff have been done over the years by tcp folks
[05:23] iFire I see
[05:24] sustrik as for the unreliable thing, i would estimate it as 1-2 days of work
[05:29] iFire I think you can migate the problem of older packets blocking newer packets by having different channels of tcp
[05:30] iFire hmm
[05:30] iFire sustrik are you familiar with sctp
[05:35] sustrik iFire: yes
[05:36] iFire the last message I received is iFire sustrik are you familiar with sctp
[05:36] iFire last message I saw
[05:37] sustrik yes, i am familiar with SCTP
[05:37] iFire sustrik last message was iFire sustrik are you familiar with sctp
[05:37] iFire well tcp is notorious for packet blocking
[05:38] iFire so unreliable udp and sctp
[05:38] sustrik that's the price for reliability
[05:38] sustrik same thing with sctp btw
[05:38] iFire well sctp is reliable and has this concept of channel
[05:39] sustrik what do you need exactly?
[05:39] iFire well it counts in terms of messages
[05:39] iFire I want something that works well in a 3d online game aspect
[05:39] sustrik i am not familiar with game development
[05:39] sustrik what are the requirements?
[05:40] iFire sctp has this
[05:40] iFire Multi-streaming is an important feature of SCTP, especially when you consider some of the control and data issues in protocol design. In TCP, control and data typically share the same connection, which can be problematic because control packets can be delayed behind data packets. If control and data were split into independent streams, control data could be dealt with in a more timely manner, resulting in better utilization of available resources.
[05:40] sustrik yes, it's same as having several tcp connections
[05:43] sustrik also, you should be aware that sctp is not available on windows as a part of OS
[05:44] iFire I did not know that
[05:44] iFire what does using several tcp connections require?
[05:44] iFire different ports?
[05:44] sustrik yes
[05:45] sustrik also, there's a possibility of opening several connections to the same source port
[05:46] sustrik then you would have to tag the messages so that you know which connection it went through
[05:46] sustrik - if you need to know it at all
[05:49] iFire found http://www.bluestop.org/SctpDrv/
[05:50] iFire no 64bit windows signing
[06:02] iFire Good Morning sustrik
[06:06] sustrik morning
[06:15] iFire sustrik here's a good list of requirements http://stackoverflow.com/questions/1200901/why-is-udp-a-software-reliable-ordering-system-faster-than-tcp/2445227#2445227
[06:15] iFire but I not sure if zeromq is the right solution
[06:18] sustrik which part of the discussion makes you believe you want to implement reliability on top of UDP>
[06:18] sustrik ?
[06:18] iFire so I would have to add a unreliable sequenced/unsequenced to the 0mq message
[06:18] iFire sustrik I found a library which did that
[06:19] iFire routing
[06:20] iFire http://enet.bespin.org/ but how would a routing message in zeromq work
[06:21] sustrik udp packets are routed in ip layer
[06:21] iFire I mean if I put enet beneath 0mq, I'll need some way of signaling which 0mq messages have to be routed some way
[06:22] iFire i.e. reliability/sequenced
[06:23] sustrik if you have a library that does what you need why bother about 0mq at all?
[06:24] iFire I don't
[06:24] iFire it's just a udp library
[06:24] iFire doesn't do messenging
[06:24] iFire let's go back to that discussion about adding unreliable to zeromq
[06:25] iFire I'll take a look a the code and talk about it later
[06:25] sustrik sure
[06:28] enki ha, you guys could try using uTP for this :p
[06:48] iFire sustrik where are the zmq_msg_t flag definitions?
[06:49] sustrik zmq.h
[06:49] sustrik ZMQ_MSG_MORE
[06:49] sustrik ZMQ_MSG_SHARED
[07:29] mikko sndmore/recvmore added
[10:01] sustrik mato: ESHUTDOWN is not POSIX, it's Linux-specific, leaving ETERM as is
[11:12] sjampoo Martin, just read your message regarding the Polling interface on the mailinglist, seems like you read my mind.
[11:13] sjampoo Do you suggest to have functions that register a socket within the ZMQ api?
[11:15] sjampoo Also, wouldn't a linkedlist be a much nicer datastructure to store the pollable items?
[11:31] mikko sjampoo: the php bindings currently have "add"
[11:31] mikko i dont know how orthodox it is
[11:32] mikko http://github.com/mkoppanen/php-zmq/blob/master/examples/poll-server.php
[11:32] mikko might be subject to change in future
[11:38] sjampoo yes, well I really wished that ZMQ had native support for managing a pollset, thus a level on top of the raw zmq_poll so that each binding does not have to implement its own pollset
[11:40] sjampoo pyzmq has register / unregister
[11:42] sjampoo but on each call to poll() the pollitems array gets rebuild from scratch, which will be expensive with lots of sockets
[11:45] mikko at the moment i just realloc up
[11:45] mikko never down
[11:46] sjampoo what do you do when you remove a socket?
[11:46] mikko why do you build the array on each poll?
[11:46] sjampoo (i don't, pyzmq does)
[11:46] mikko you only need full rebuild on remove
[11:46] mikko add should only cause append
[11:47] mikko currently there is no remove in php
[11:48] mikko i got a local branch for making some changes to pollset which contains remove as well
[11:49] sustrik re
[11:49] sjampoo I use the same approach currently as you are suggesting.
[11:49] sjampoo I just think it would be nicer if it was implemented in ZMQ directly, ie as a linked list or whatever to get great performance.
[11:50] mikko yes, it would make sense
[11:50] sjampoo I don't think the API directly matters, as the binding can then provide a thin layer on top of it.
[11:51] sustrik i'll definitely move it to 0mq core once the api is agreed on
[11:51] sustrik sjampoo: i wouldn't worry about creating the pollset each time
[11:51] sustrik there should never be more than couple of sockets in the pollset
[11:52] sjampoo Well, there are going to be lots if you are going to use it in some sort of IOLoop that also polls regular TCP sockets.
[11:53] sustrik true
[11:54] sustrik mikko, sjampoo: so what about the api, how do you believe the results should be retrieved from the pollset?
[12:00] mikko int revents; int id; zmq_pollset_t set; zmq_pollset_init(&set); id = zmq_pollset_add_socket(set, s, ZMQ_POLLOUT); zmq_poll(set, -1); revents = zmq_get_events(set, id); if (revents & ZMQ_POLLOUT) {}
[12:01] mikko top of my head
[12:01] mikko probably not any more flexible than the current one
[12:06] sustrik i don't like the 'id' part
[12:06] sustrik it somehow presuposes a hash inside the poller to map id to actual pollitem
[12:07] sustrik => performance impact
[12:32] sjampoo sustrik, it would be nice if i could simply iterate over some sort of result set, but i am not sure what the best approach would be.
[12:34] sustrik what about getting events one by one?
[12:35] sustrik event = pollset.next_event ();
[12:37] sjampoo i would like that i suppose, fe in python one could simply turn that into a generator and yield ready sockets
[12:43] mikko how do you know which socket the event ties to?
[12:47] sustrik that's the question: what is the event?
[12:48] sustrik maybe a triple: socket, event, hint?
[12:48] sustrik any better ideas?
[12:48] sjampoo Event is actually an object consisting of 'socket', 'event' and a 'hint', as you suggested in your mailing list.
[12:48] sjampoo And then the hint can be set while 'adding' the pollitem.
[12:48] sustrik that's how epoll does it
[12:48] sjampoo What are the restrictions on the hint?
[12:49] sjampoo What type is it?
[12:49] sustrik for epoll it's void* or something like that
[12:49] sjampoo aha
[12:49] sjampoo well that would be perfect imho
[12:50] sustrik presumably not every language has 'any' type
[12:53] sjampoo well, it shouldn't be that difficult to translate it to something the language does understand by the binding maintainer
[13:06] mato re
[13:06] mato sustrik: ack
[13:55] mato sustrik: you're following the pgm discussion on the mailing list?
[13:55] mato sustrik: i think we have finally hit on the problem and everyone agrees it's there, which makes me happy
[13:55] mato sustrik: the question is now, what to do about it ...
[13:57] sustrik mato: yup, i am following it
[13:58] sustrik afaics what openpgm does is ok
[13:58] sustrik you advertise limit of x
[13:58] sustrik then publish >x
[13:58] sustrik you experience message loss
[13:58] sustrik fair enough
[13:58] sustrik the problem is that 0mq adds queueing to the picture
[13:59] sustrik so you have queue full of data
[13:59] sustrik and want it pushed out through multicast
[13:59] sustrik you set the rate limit
[13:59] vtl sustrik: commit 716f4ac8714d33d21f9853f58482e35c1e3ad934 why do you use int64_t when it should be size_t?
[13:59] sustrik and publish much more than that
[14:00] sustrik my feeling is that steven may be right about us having to implement some rathe control on top of openpgm
[14:00] sustrik vtl: wait a sec
[14:01] sustrik vtl: where exactly?
[14:02] mato sustrik: us implementing rate control seems silly to me
[14:02] mato sustrik: openpgm already has all the information it needs
[14:02] sustrik note that what we do now when we hit the limit is busy looping
[14:02] sustrik that feels kind of strange anyway
[14:03] vtl sustrik: almost everywhere. socket options has type of size_t
[14:03] sustrik ?
[14:03] mato sustrik: my point is that the current situation is broken. AFAICS pgm with 0mq does not work. at all.
[14:03] vtl or I miss something?
[14:03] sustrik you mean POSIX socket options?
[14:04] vtl yeah, I messed it up. sorry
[14:04] sustrik np
[14:04] sustrik mato: it works unless the limit is hit, no?
[14:05] mato sustrik: yes, but the application has no way of controlling that
[14:05] mato sustrik: therefore it does not work
[14:05] sustrik then it's not completely broken
[14:05] sustrik and btw pub/sub doesn't guarantee delivery anyway
[14:06] mato sustrik: it's not about guaranteeing delivery
[14:06] sustrik so it may be less reliable than expected but not broken
[14:06] mato sustrik: yes, but with the latest changes steven made, you saw what happened when i tried the performance tests
[14:06] sustrik hang up?
[14:06] mato sustrik: you run the _thr tests and the moment you hit the limit everything just hangs
[14:06] mato yes
[14:06] mato hangs == broken in my book
[14:07] sustrik thr tests should be modified to use P2P sockets
[14:07] mato huh?
[14:07] sustrik P2P sovkets don't allow for PGM as underlying protocol
[14:07] sustrik that solves the problem
[14:07] mato you're side-stepping the issue
[14:08] sustrik nope
[14:08] sustrik PUB/SUB is unreliable
[14:08] sustrik thus you cannot base the test on it
[14:08] sustrik there's always a chance of message being dropped
[14:08] sustrik that's the nature of pub/sub
[14:09] sustrik "rather drop messages then let the slow consumer block the whole system"
[14:09] mato the problem here is *not* about slow consumers
[14:09] sustrik i know
[14:09] sustrik but the semantics of pub/sub based on the slow consumer scenario
[14:10] sustrik fit even the case we are seeing niw
[14:10] sustrik now*
[14:10] sustrik what's happening is annoying
[14:10] sustrik bet technically sound
[14:10] sustrik but*
[14:10] mato it might be technically sound according to the (vague) PGM spec, and according to the implementation
[14:11] mato but it seems to me that the current behaviour is totally useless from an application developer's point of view
[14:11] sustrik ack
[14:12] mato the question remains, what to do about it...
[14:12] sustrik my feeling is that another rate limit has to be implemeted on 0mq level
[14:13] mato what's wrong with reserving x% of the bandwidth for RDATA at the openpgm level?
[14:14] sustrik nothing, but you still end up with busy looping
[14:15] mato why are we busy looping?
[14:15] sustrik because when you try to send openPGM says: EAGAIN (because od the rate limit)
[14:16] sustrik you cannot poll for POLLOT
[14:16] sustrik POLLOUT
[14:19] mato well, i don't know what to do now
[14:19] mato presumably 2.0.7 should include steven's changes post openpgm 2.0.24 that fix the crashes, but introduce the hangs
[14:20] mato but as for what to do with the current behaviour...
[14:20] sustrik it's not easy
[14:20] sustrik what i am saying it's good enough to let it be for now
[14:21] sustrik if the actual rate is lower than specified max rate everyting works as expected
[14:21] sustrik it you publish faster, messages get lost
[14:21] mato no, if you publish faster your publisher hangs
[14:21] sustrik publisher?
[14:21] mato sorry, consumer
[14:22] sustrik it doesn't hang, it only looses some messages, no?
[14:22] mato no, it hangs
[14:22] mato forever
[14:22] sustrik a-ha
[14:22] sustrik how does that happen?
[14:23] mato that was discussed in the long thread between me and steven
[14:23] sustrik i;ve missed the consumer hanging part
[14:23] mato due to the packet loss, and the absence of RDATA, the receiver is unable to recover, ever
[14:24] sustrik if says "unrecoverable message loss", no?
[14:24] mato no
[14:24] sustrik yuck
[14:24] mato well, i'm not 100% sure
[14:24] mato i'd have to retest at your place again
[14:25] sustrik i though local_the hangs because it does not get all the messages
[14:25] sustrik hm
[14:26] mato no, the behaviour i saw with local_thr was it gets ~2k messages, then (presumably) some packet loss occurs and it just sits there forever
[14:26] sustrik right, because the rest of the messages are lost
[14:27] sustrik it happens because you publish faster than the rate is set
[14:28] sustrik i am not saying it is great
[14:29] sustrik but it's not a real bug
[14:29] sustrik it's reliability deficiency problem :)
[14:36] sustrik mato: btw, there's a pull request from mikko for the RPM packaging
[14:38] mato sustrik: yes, requires license-foo
[14:39] sustrik are you sure these are needed for packaging scripts/
[14:39] sustrik ?
[14:39] mato yes
[14:39] mato anything that goes into the 0mq git
[14:39] sustrik mikko: are you there?
[14:40] sustrik he seems to be out
[14:46] sustrik sent him a message
[14:52] mikko almost here
[14:52] mikko @work
[14:52] mikko is there a CLA i can sign?
[14:52] mikko so that all my contributions would be automatically under the license
[14:53] mato mikko: we haven't really formulated one yet
[14:54] mato adrian sent me this, for example:
[14:54] mato +++
[14:54] mato I, Adrian von Bidder, Switzerland, license my work on the Debian packaging of
[14:54] mato the zeromq library to iMatrix under the MIT/X11 licsense.
[14:54] mato +++
[14:54] mato by email, with the relevant patch attached
[14:54] sustrik shouldn't it go to the mailing list?
[14:54] mato yes, he sent it to the mailing list
[14:55] mato sustrik: we need to decide how to handle this, and ph needs to agree
[14:55] sustrik yes, it's a mess
[14:55] sustrik i am trying to direct this kind of thing to the mailing list
[14:56] sustrik as it's presumably more persistent than personal mail boxes
[14:56] mato yeah, but even there the relevant statements will get lost
[14:56] sustrik shrug
[14:56] mikko CLA would make most sense
[14:57] mikko that in theory would eliminate the need for patches etc
[14:57] mato mikko: I'm thinking of two things
[14:58] mato mikko: 1) Simple template "CLA" or whatever you want to call it by email
[14:58] mato mikko: 2) Signed-off-by: on any git commits which is explicitly defined as "falls under the CLA"
[14:58] mikko 1) probably needs a signature (fax / scan / etc)
[14:58] mikko 2) makes sense
[14:59] mato no, we want to deliberately avoid the whole fax/scan/etc
[14:59] mato email is enough, this is 2010 after all
[14:59] mikko is it legally binding? i've had to scan+email to many places
[14:59] mato mikko: the consensus is that it's "Good enough"
[15:00] sustrik what's signed-off-by?
[15:00] mato mikko: see the commentary by Pieter Hintjens at http://www.zeromq.org/blog:why-the-mit-license-for-contributions
[15:00] mato sustrik: it's just an extra header you can add to git commits/patches
[15:00] mikko http://www.apache.org/licenses/icla.txt
[15:01] mikko thats apache one
[15:01] sustrik isn't the signed-off-by better then?
[15:01] mato sustrik: the Linux kernel people use it to mean "I sign off this patch and state that it came from me" or something similar
[15:01] sustrik everyone going to commit anyway
[15:01] sustrik so adding one field is not big a hassle
[15:01] mato sustrik: yeah, but you need to define somewhere *else* what the Signed-Off-By in the context of 0mq git means
[15:01] sustrik and we can pull the patches directly
[15:02] sustrik hm
[15:03] mikko http://www.oss-watch.ac.uk/resources/cla.xml#body.1_div.4
[15:03] mikko 4. Associating commits to CLAs
[15:03] mato sustrik: hence the combination of a simple email statement "I, XXX, of YYY, hereby state that any commits of mine containing Signed-Off-By: ..., are licensed under the MIT license.
[15:03] mato sustrik: with actual Signed-Off-By headers
[15:03] sustrik mikko: right
[15:04] sustrik mato: yes, seems to make sense
[15:04] mato sustrik: it should also be possible to put a hook in github to reject commits w/o a Signed-Off-By
[15:04] sustrik mikko's link suggest that's it's necessary to be able to blame each line of code
[15:05] sustrik that can be done only by direct pulling => singed-off-by
[15:05] mato no, it can work for all models
[15:05] mato at least i think so, here's an example
[15:05] sustrik ah, the --author option
[15:05] sustrik right
[15:06] mato the point is you can set up your git so that *all* your commits to zeromq2 contain the signed-off-by tag
[15:06] sustrik that would be good, no?
[15:06] mato then, when those commits are pushed/pulled/whatever, the signed-off-by tags propagate
[15:06] mato sustrik: yes, that is what i have in mind
[15:07] sustrik mato: post a proposal to the mailing list and let's see what will people say
[15:08] mato i'll email a proposal to pieter first, and get him to make that post
[15:08] mato i don't speak for imatix
[15:08] sustrik ok
[15:09] sustrik however, this is discussion of technical aspect of contributing so community should have a say
[15:09] mato right, well then i can make a proposal
[15:52] mikko hmm
[15:53] mikko if i add fd to pollset and close the remote peer the socket is marked readable in a loop
[15:55] mikko ermm confusing terms
[15:55] mikko adding normal fd that is connected to a remote peer. remote peer closes and poll marks it readable in in a loop
[15:55] mikko and chews 100% cpu
[15:57] mikko i guess i need to check for that
[15:59] sustrik mikko: "if i add fd to pollset and close the remote peer the socket is marked readable"
[15:59] sustrik that's standard POSIX behaviour
[16:00] sustrik in case of orderly shutdown POLLIN is signaled
[16:00] sustrik and subsequent read returns 0 bytes
[16:00] sustrik iirc
[16:03] mikko indeed
[16:04] mikko i need to work on this a bit on the evening
[16:05] mikko to make the removing of objects more fluent
[16:09] mato argh
[16:09] mato the whole signed-off-by thing with git is complicated :-(
[16:09] mato i don't want to deal with this right now, i'd rather get some real work done
[16:10] sustrik ok, mikko, won't you mind simply sending the patch to the mailing list and stating it's MIT licensed?
[16:10] mato mikko: in the interim, can you just send a copy of your patch made with git-format-patch to the mailing list, and include one line that says "This patch is submitted under the MIT license"?
[16:10] sustrik :)
[16:10] mato sustrik: my words exactly :-)
[16:11] mato sustrik: ok, ad openpgm: i guess i should just email steven to make a 2.0.25 release with his latest changes, since that stops the crashes
[16:11] mato sustrik: that can go into our 2.0.7
[16:11] sustrik ack
[16:11] mato sustrik: therefore the only other thing is looking at the options/flags stuff, which i will write about
[16:12] mato and then i can get onto writing that nginx plugin which is actually interesting :-)
[16:12] mato sustrik: ack?
[16:12] sustrik sure, more useful even
[16:12] sustrik mato, mikko: the rpm spac file in the patch in in builds subdirectory
[16:12] mato good, we're agreed then
[16:13] sustrik shouldn't it go to 'packages' ?
[16:13] sustrik mato: yes
[16:13] mato sustrik: why create packages when we already have builds, it doesn't really matter
[16:13] mato sustrik: the point is at make dist time that spec file will get copied into the tarball root anyway
[16:14] sustrik dunno, msvc build system and rpm packaging script seem to be distinct kind of thing
[16:14] mato sustrik: this is similar to why debian is in /
[16:14] sustrik up to you guys, i'm giving up :)
[16:14] mato who cares
[16:14] mato it's just trivia
[16:14] mato less directories is better :-)
[16:14] mikko "redhat builds", "debian builds", "windows builds"
[16:14] mikko sounds logical
[16:15] mato exactly
[16:15] sustrik ah, it also builds the thing?
[16:15] mikko it creates binaries
[16:15] mikko or source rpms
[16:15] sustrik then it makes perfect sense
[16:15] mikko spec-file defines how it's built
[16:15] sustrik sorry for trolling
[16:15] mikko and creates installable rpm package
[16:16] mikko i guess you are entitled for a bit of trolling :)
[16:54] iFire wow vmware is on a roll. They acquired rabbitmq
[16:54] iFire previously it was redis
[16:57] sustrik yup, looks like they are making a portfolio of cloud infrastructure
[16:58] iFire I hope vmware deals with rabbitmq the same way they dealt with redis. i.e. keep the same license, except pay the developers for work
[16:59] iFire (background info: redis is in bsd license)
[17:01] sustrik I've just spoke to rabbitmq guys, seems that that's the plan
[17:02] sustrik from the press release: "RabbitMQ will continue to be open source and distributed in the same way
[17:02] sustrik as before. The RabbitMQ community can expect to see increased investment
[17:02] sustrik in this outstanding technology which should result in significant
[17:02] sustrik improvements to the open source release."
[18:04] dirtmcgirt hi all.
[18:05] dirtmcgirt can zeromq be used along side libevent?
[18:05] mikko dirtmcgirt: we actually discussed the other day about 0MQ and libev
[18:06] dirtmcgirt mikko: ah! i missed it.
[18:06] mikko http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#code_ev_prepare_code_and_code_ev_che
[18:07] mikko libev provides something like that looks like it could be used
[18:07] mikko you could check whether libevent provides something similar
[18:09] dirtmcgirt interesting, thanks
[18:09] mikko i've written most of my stuff using libev lately (instead of libevent)
[18:09] mikko libevent has a nice http library though
[18:09] mikko evhttp i think it's called
[18:10] dirtmcgirt yes, we are using that
[18:10] dirtmcgirt it's great

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy