Wednesday July 20, 2011

[Time] NameMessage
[05:56] sustrik mikko: hi
[07:29] mikko sustrik: the patch seems ok
[07:29] sustrik :)
[07:29] sustrik ok, let me me commit it
[07:30] mikko
[07:30] mikko you can see the results there
[07:32] sustrik sure, i just wanted your approval as you seem to be maintaining the package
[07:38] mikko looks like pieter fixed the windows builds
[07:39] mikko some of them
[07:40] mikko
[07:40] mikko looks like signaler missing from solution file?
[07:55] mikko pieterh: add signaler.cpp to msvc.sln and change signaler.cpp line 196 from #if ZMQ_HAVE_WINDOWS to #ifdef ZMQ_HAVE_WINDOWS
[08:10] sustrik mikko: oops, my fault
[08:10] sustrik the actual patch was done in 2 steps
[08:10] sustrik 1. patch
[08:11] sustrik 2. adjust the win build
[08:11] sustrik and i incidentally passed only thr 1st patch to pieter for backporting
[08:11] sustrik pieterh: here?
[08:14] mpales i need a help with pub sub over inproc
[08:14] mpales it works fine over tcp or ipc, but not over inproc transport
[08:14] sustrik intersting
[08:15] sustrik what does it do?
[08:15] sustrik is reconnection involved?
[08:15] mpales another pub sub between different threads works fine
[08:15] sustrik or starting connect side befor bing side?
[08:15] sustrik these two are known problems with inproc
[08:15] mpales i believe i start bind before connect
[08:15] sustrik check that
[08:15] mpales ok
[08:16] mpales connect/bind is asynchronous?
[08:16] mpales i mean over inproc
[08:18] sustrik no, it's sync actually
[08:23] mpales ok, bind happens 1 sec before connect
[08:23] mpales still i receive no message
[08:25] mpales send looks good
[08:25] mpales for receiving side, i do a select, but i get no event
[08:33] sustrik you mean zmq_poll?
[08:36] sustrik quite possible there's a bug there
[08:36] sustrik can you create a minimal test case?
[08:48] mpales not zmq_poll, select syscall
[08:48] mpales i have my own reactor
[08:49] mpales it looks like there is a bug in zeromq
[08:50] mpales i need to do reads from socket to start getting events on the file descriptor\
[08:51] mpales but this happens only with inproc
[09:05] sustrik sure, can you create a test program to reproduce?
[09:05] sustrik i'll have a look at it then
[09:41] mpales sure
[09:52] zmqBeginner Hi everyone. I'm a ZeroMQ beginner having trouble getting the JAVA binding to work on Ubuntu. I have followed all the steps in [The C/C++ bindings work fine]. However, when I try to find where the libjzmq is located, I am unable to find it in the system! Hence, I can't compile anything, since for javac, "package org.zeromq does not exist". Please help me out.
[09:55] zmqBeginner Is anyone online to help?
[10:03] G2P zmqBeginner: make install should tell you where it got installed
[11:07] pieterh sustrik: re
[11:12] pieterh mikko: ok, made those fixes to 2.1/2.2
[11:13] mikko thanks
[11:13] mikko should be ok in 5pm build
[11:13] pieterh there is a weird issue with czmq on win32 still, I don't know what the cause is
[11:13] pieterh presumably due to last changes
[11:14] mikko zclock.c:137: error: ‘int64_t’ undeclared (first use in this function) ?
[11:28] pieterh mikko: yes
[11:33] pieterh mikko: ok, it's definitely the last changes by Brandon Ehle, they stop int64_t from being defined
[11:40] pieterh mikko: ironically, those changes were "to make CZMQ compile with MinGW", anyhow I've reverted them
[12:07] sustrik pieterh: hi
[12:07] pieterh sustrik: hi!
[12:08] sustrik i messed up with the patch i've sent you recently
[12:08] pieterh sustrik: which one? for windows?
[12:08] sustrik there is another associated patch that fixes the win build
[12:08] pieterh I've fixed it with Mikko's help, no problem
[12:08] sustrik ok then
[12:43] cremes sustrik: will 3.x have the fix so that inproc can connect first (then bind) just like other sockets?
[12:43] cremes or will it be restricted to binding first as it is now?
[12:44] sustrik well, i should fix ti
[12:44] sustrik it
[12:44] sustrik but it's not a trivial fix
[12:44] sustrik do you need it?
[12:44] cremes i do... right now i have to special-case some code when i use inproc whereas every other
[12:44] cremes t
[12:44] cremes ransport "just works" without any special sequencing
[12:45] cremes it kind of blows the whole "start with inproc and scale to other transports without changing *any* code" promise
[12:45] cremes :)
[12:45] pieterh cremes: this is true, but there is a plausible explanation
[12:46] pieterh first of all, inproc and tcp are about the same speed (~3M operations / second)
[12:46] pieterh maybe 6M, it depends
[12:47] pieterh second, if you really want to do the "scale from threads to process" thing, you need detached threads
[12:47] pieterh i.e. threads with their own context
[12:47] pieterh otherwise you have to make changes when you move them around
[12:48] pieterh so I end up using inproc only for attached threads, i.e. which are architecturally part of the same process
[12:48] cremes i don't agree with your second point
[12:49] cremes i have no issues at all with sharing a context within a process and then moving work/threads to
[12:49] cremes another context and have it continue working
[12:49] cremes or am i misunderstanding your point?
[12:49] sustrik in any case, the functionality is missing for inproc transport
[12:49] sustrik and should be added
[12:49] cremes sustrik: agreed
[12:50] sustrik the only problem is that it requires a lot of work
[12:50] cremes well, since you are making so many architecture changes, at least plan to support this
[12:50] cremes even if the work cannot be done right away
[12:50] sustrik i am planning for it for 2 years already :)
[12:50] cremes excellent
[12:51] sustrik the problem with inproc is that there's no I/O thread in the background
[12:51] sustrik so the whole reconnection mechanism is very different from what is done with TCP et al.
[12:51] cremes right, i recall that from earlier discussions
[12:52] cremes so how do you plan to resolve this? spin up a temporary i/o thread until at least 2 inproc endpoints are linked?
[12:52] sustrik i think you can do without that
[12:53] sustrik just doing it in sync manner
[12:53] sustrik i.e. connect request is stored in a queue
[12:53] sustrik when bind happens, the binding thread looks into the queue
[12:53] sustrik and creates the "connections"
[12:53] sustrik the biggest problem is that it interacts in various ways with shutdown code
[12:54] sustrik which is the most complex part of the codebase
[12:54] cremes aha
[12:54] sustrik the problem with shutdown code in turn is that it's holistic
[12:54] sustrik it's not well delimited part of the system
[12:55] cremes it's scattered throughout all of the objects, yes?
[12:55] sustrik exactly
[12:55] sustrik so what happens
[12:55] cremes and a bit on non-determinism makes things extra difficult?
[12:55] sustrik is that any change has repercussions in random parts of the system
[12:55] cremes s/on/of/
[12:55] sustrik yes
[12:55] cremes sounds like fun :)
[12:55] sustrik ;)
[12:56] cremes so is your approach to enforce a shutdown order or a particular sequence on shutdown?
[12:56] cremes or do you just try to cover every case?
[12:57] sustrik 0mq is fully async
[12:57] sustrik so the shutdown has to be fully async as well
[12:57] cremes sure, but you can still use a statemachine to enforce an order
[12:57] sustrik statemachine of what?
[12:57] sustrik individual components are fully async
[12:58] mpales +1 for fixing inproc setup :-)
[12:58] sustrik so you can have a state machine for socket
[12:58] sustrik for session
[12:58] sustrik for engine
[12:58] sustrik etc.
[12:58] cremes a statemachine to enforce a shutdown order and coordinate termination amongst all components
[12:58] sustrik global one
[12:58] sustrik ?
[12:58] cremes yes
[12:59] mpales e.g. I have several inproc connections between threads in both directions and it is quite difficult to manage thread creation process
[12:59] sustrik then you would have to synchronise the whole system using sync primitives such as mutexes
[12:59] sustrik which would in turn cut the performance to a fraction of what it is today
[12:59] mpales i must be carefukl that there are no cyclic dependencies, e.g. first thread must bind everything and second one connect
[12:59] cremes sustrik: who signals the beginning of termination? can it originate inside of any of those components?
[12:59] sustrik it's messy
[12:59] cremes mpales: i have the same difficulties
[13:00] sustrik there can be several shutdowns going on at once
[13:00] sustrik say, a peer disconnected
[13:00] sustrik and at same time user closed a socket
[13:00] sustrik the algorithm has to account for such corner cases
[13:00] mpales i fear the day when i add another thread that *will* cause cyclic dependencies :)
[13:00] cremes yes, in my ignorance it looks simpler to solve than the reality :)
[13:01] cremes i trust you when you say it's a hard problem
[13:01] sustrik it is
[13:01] sustrik of course, if anyone needs it so badly to actually sponsor the development, just ping me :)
[13:02] cremes of course :)
[13:02] mpales sustrik: maybe it would be simpler to change inproc transport handling so it mimics tcp or ipc
[13:02] sustrik mpales: yes, i am aware of the problem
[13:02] cremes mpales: look at the irc channel history for the last 20 minutes... it covers the issue quite nicely
[13:02] mpales it would use i/o thread normally
[13:02] sustrik mpales: you can use ipc to get that kind of behaviour
[13:03] mpales sustrik: true
[13:03] sustrik the only reason for inproc to exist is that it circumvents all that cost
[13:04] cremes mpales: a technique i have experimented with to solve the issue is as follows:
[13:05] cremes each thread that tries to connect to inproc has a second socket (REQ) that connects to a "well known" service
[13:05] cremes that acts as a synchronizer; the service has a XREP socket
[13:05] cremes the thread that binds the inproc connects to the same service and informs it that the bind has completed
[13:06] cremes the service then replies to everyone else that they can now connect to inproc
[13:06] cremes it's a bit of work to set this up, but it obviously scales to dozens of threads
[13:06] mpales from another perspective, inproc does not cross process boundaries and i can use "well-known" endpoins like "control" and "data"
[13:06] mpales so 2 applications can run simultaneously
[13:07] cremes yes
[13:07] mpales with ipc, enpoints must be unique and communicated within applications
[13:07] mpales cremes: nice solution
[13:07] mpales cremes: i could use it as well, thanks!
[13:08] cremes sure... it's essentially turning the well-known service into a mutex (or condition var) for these other threads
[13:08] cremes so the concept is pretty simple
[13:08] cremes for all i know, this pattern is in the guide and i just forgot the name (the angry pimp? the sinking dolphin? i can't keep track...)
[13:10] pieterh cremes: I was thinking of explicitly documenting the concepts of 'attached' and 'detached' threads
[13:10] pieterh but couldn't think of a cute name, so abandoned the idea
[13:10] pieterh tbh I don't see the requirement to make inproc work differently than it does
[13:11] cremes pieterh: you have 2 suggestions for naming just above here :)
[13:11] pieterh well, in fact I chose the names with great care and attention... :)
[13:11] cremes i know... i'm just teasing you
[13:11] pieterh i have no sense of humour
[13:12] cremes that's because you are misspelling it (humor)
[13:12] pieterh anyhow, wrt inproc, it really does look to me that if you want detached threads, use tcp, and if you want attached threads, use inproc
[13:12] pieterh there's not a significant performance difference unless you are doing stuff like sending pointers
[13:12] pieterh and in that case you can't move threads around anyhow
[13:13] sustrik see LIBZMQ-6 ticket
[13:13] sustrik "Auto-reconnect for inproc transport"
[13:13] sustrik Created:
[13:13] sustrik 31/Mar/10 8:37 AM
[13:13] pieterh yup
[13:14] pieterh and if something is _that_ hard to do, it's probably not worth doing
[13:14] mpales pieterh: anyhow, making connect/bind work like for other transport would simplify multithread applications a lot
[13:14] sustrik i am postponing it all the time :(
[13:14] pieterh srsly, I don't see a use case except "orthogonality"
[13:14] pieterh and I'm using inproc really a lot
[13:15] pieterh one could even argue that for attached threads, forcing the bind/connect order is a benefit, not a cost
[13:15] pieterh one does not expect threads to come and go randomly
[13:55] taotetek hi pieterh
[14:25] pieterh taotetek: hi!
[14:35] taotetek pieterh: hoping to find time to start working on a jruby majordomo implementation today :)
[14:36] pieterh excellent... there are already a few implementations if you need inspiration
[14:37] taotetek pieterh: I was looking over the C implementations in the guide yesterday
[14:38] pieterh there's also an independent PHP implementation
[14:38] pieterh on github, you can probably find it by searching
[14:45] taotetek pieterh: the lazy pirate client I whipped up yesterday to play with
[14:45] taotetek pieterh: going to work my way up through simple pirate, paranoid pirate, and then start in on majordomo
[14:55] cremes taotetek: please take a look at this project:
[14:55] cremes i'm sure that if you pinged andrewvc he would add you as a project committer/owner
[14:55] cremes it might be a better way to consolidate all 0mq knowledge for ruby... just a suggestion
[14:58] taotetek cremes: thanks for the heads up! bookmarked it
[20:09] skvidal is there any/much provision for authenticating clients beyond just ip-based restrictions?
[20:10] mikko skvidal: depends on the requirements
[20:10] mikko skvidal: you could use PKI for the messages if you need anything beyond simple restrictions
[20:10] skvidal mikko: so handle everything internally to the msg
[20:10] skvidal okay
[20:10] mikko skvidal: yes, encrypt the messages
[20:10] skvidal nod
[20:11] mikko in that case it doesnt matter who connects
[20:11] skvidal but replay is possible at that point, I guess.
[20:11] mikko and i've noticed that ip based restrictions are doomed to fail in larger environments
[20:11] skvidal nod
[20:11] skvidal definitely.
[20:11] skvidal I was wondering if establishing the connection could be ssl client cert restricted
[20:12] mikko you always end up having a network hop on the "safe range" and it all fails
[20:12] mikko skvidal: that is a very complicated topic
[20:12] skvidal mikko: I agree
[20:13] skvidal mikko: I've worked on a system which use xmlrpc over ssl authed both ways using ssl certs and there is a fair amount of pain and suffering involved
[20:13] skvidal mikko: it's why I was asking if y'all had already solved it - for purposes of porting some existing code
[20:14] mikko it gets very very complicated when it comes to transport layer security and zeromq
[20:14] mikko hence i tend to lean towards encrypting messages
[20:14] mikko that way you don't need to care about middle-hops
[20:14] mikko or the topology
[20:15] mikko if you add intermediates it raises an interesting question of where the transport layer security should be terminated
[20:15] mikko on the intermediate?
[20:15] mikko that would make the middle node a trusted one
[20:15] mikko and that is not always the case
[20:15] mikko etc
[20:15] skvidal that's a good point
[20:16] skvidal thanks for the response. I'll have to figure out nicer-ish modules for encrypting the messages
[20:16] mikko i think pieterh wrote a draft protocol for secure pub-sub
[20:16] skvidal I read that
[20:18] mikko i think you might want to look at salt as well
[20:18] mikko i think that uses some sort of encryption for the messages
[20:19] mikko
[20:22] p0lt so for my implementation of security, I use pre-shared keys and an offset bit that I contain in an rpm, the stream (publisher stream) is encrypted using the key, and of course if you have the key, and my offset bit you can decrypt my messages... however pushing new keys / restarting my encrypted workers is pretty easy to do
[20:24] skvidal I suspect I'd need preshared public keys/CA and some kind of CRL
[20:24] p0lt yeah, injecting encryption should happen before you send your message, encryption in zeromq sorta kills it a bit because you'd be encrypting routing information possibly and you need that to move the msg along ;)
[20:27] p0lt my stream is aes256 bit encrypted with sha1 signatures... I only send the encrypted contents of my msg, no offset bits / keys are ever distributed without my knowledge / doing... Other things you can do is salt the encrypted stream, so it uses the key, plus contents of the stream to actually decrypt the msg... that makes things even funner for a hacker because they need to a) get your keys / offset bits, and then b) figure out your salt and what
[20:27] p0lt is salting the msg ;)
[20:28] skvidal do you have a deployed/available library or call to handle this?
[20:28] p0lt unfortunately no, because I'm building an internal project for my company...
[20:29] skvidal understood. sorry.
[20:29] p0lt no worries
[20:30] p0lt good luck, definitely look at salt, and if you have questions fire away...
[21:26] bitcycle Hey all. I want to use zeromq with cassandra to make a fast and highly available message queue without the need for broker. However, the problem that I'm running into is the "highly availalbe" part. What would you all recommend to address that need, architecturally?
[21:34] cremes bitcycle: there's a whole chapter on that in the guide;
[21:35] cremes it lays out a whole architecture for you
[22:00] bitcycle cremes: Which chapter, that link is just to the guide... and the guide is pretty long.