Monday May 9, 2011

[Time] NameMessage
[03:40] aspidites how would i modify the asycsrv example to send reply back to client?
[03:41] aspidites if i'm reading it correctly, it would appear that it passes messags between worker threads and the server indefinitely
[04:38] sustrik asyncsrv sounds like it's meant to be async, ie. no replies
[04:38] sustrik REQ/REP is what you want foe replies
[05:00] pieterh aspidites: the asyncsrv example already sends replies back to the client
[05:01] pieterh sustrik: async means 'processing messages without wait states', it's orthogonal to the pattern
[05:01] pieterh so happens async request-reply is exactly what that example does
[05:01] sustrik ok, sorry
[05:01] pieterh np, you're up early!
[05:01] sustrik i tend to wake at 6am
[05:02] sustrik so are you, btw :)
[05:02] sustrik what about the evening?
[05:02] pieterh like my 8-month old son, who wakes at 6am
[05:02] pieterh you're arriving today?
[05:02] sustrik yep
[05:02] sustrik afterrnoon
[05:02] pieterh excellent!
[05:02] sustrik the same flight as mato
[05:02] lt_schmidt_jr are you guys getting together in Moscow?
[05:02] sustrik brussels
[05:02] pieterh so you can come to iMatix if you like, to work here
[05:03] sustrik moscow meetup was last week
[05:03] pieterh this evening, meetup at 7pm in town, same location as the conference
[05:03] sustrik ok
[05:03] lt_schmidt_jr ah good - cause it will be a zoo tomorrow, nevermind
[05:03] sustrik i think it'll take some time to get from charleroi
[05:03] pieterh sustrik: if you can buy but tickets online that saves a lot of time
[05:03] pieterh *bus tickets
[05:03] sustrik ok
[05:04] pieterh it's one bus every 45 mins for often too many people
[05:04] sustrik the end of war celebrations?
[05:04] lt_schmidt_jr Yup
[05:04] pieterh ah, I thought it was the celebration of 150 years since they switched to the modern rail track size...
[05:05] pieterh all those railway fanatics going crazy with vodka
[05:05] lt_schmidt_jr he he he
[05:05] guido_g good morning!
[05:05] sustrik morning
[05:05] pieterh hi guido_g!
[05:05] guido_g pieterh: what did you do to the weather? forecast says thunderstorms...
[05:06] sustrik yeah, need for warm clothing?
[05:06] pieterh guido_g: all the sun got used up this weekend for barbequeues :-(
[05:06] pieterh sustrik: it'll be 23c or so
[05:06] guido_g sustrik: 24°C they say, so i think no
[05:06] sustrik nice
[06:24] guido_g coffee machine shut down, going to leave soon
[06:29] guido_g off i go, see you in brusssels!
[06:34] th thinking about DoS, is there any way to decide whether to accept a connection (POSIX socket accept())? would it be really out of 0mq's scope to have the possibility for a callback method there?
[06:39] sustrik what's the criteria you would like to use for that?
[06:40] sustrik overl number of connections?
[06:45] th no
[06:45] th i want some information as parameter to the callback
[06:46] th remote peer address for example
[06:46] th then i could do checks myself
[06:46] th (did we have 10 failed authentications attempts before?)
[06:48] th technically thats not possible for deciding "whether to accept a connection", because thats only known after accept()
[06:48] th but that would be good enough. even if accept()ed, i could immediately drop the message, and if necessary send a message to the system to firewall the subject
[06:52] th well - overall number of connections is something a user could evaluate for him/her-self if he/she had such a callback
[06:55] th i'm not sure in which thread the callback should be running, though.
[07:54] th sustrik: whats your opinion on this?
[07:56] mikko good morning
[07:57] mato morning
[07:59] mato th: you are asking for a admin/monitoring infrastructure, right?
[07:59] th mato: you could call it like that. but i'd call it a way to refuse connections.
[08:00] th mato: i dont want to monitor with it. i really want to fight DoS
[08:00] mato th: you're using zmq on the public internet? brave man :-)
[08:00] th mato: i'm not. but it would be one step towards it.
[08:01] mato th: well, the question is, should that be zmq's job or not... you can of course rate-limit with iptables and suchlike.
[08:01] mato th: if you do it at the zmq level then all of a sudden it becomes part of your application policy
[08:01] th mato: but i want to do limits based on information that is known inside 0mq
[08:01] mato th: which is not necessarily a good idea.
[08:02] th mato: what i do includes authenticated+encrypted messages already. so i can already drop everything which is invalid/illegal
[08:02] mato th: well, why not just rate-limit your # of connections to the port in question with iptables?
[08:03] mato that way, it's also *external* to your app. so if conditions change you can tweak your firewall rules on the fly accordingly.
[08:05] mato otherwise, getting information out of zmq about # of peers/connection rate/... and otherwise manipulating individual peers, well, there's no support for that at the moment.
[08:05] mato and you'll have to do a lot of convicing of el sustrik to get him to add anything like that since he's afraid of people abusing it for business logic :-)
[08:07] drakkan1000 Hi, I'm evaluating zeromq for my project, in my app I need to share file descriptors between different process. The main process must accept network connection and open an fd, this fd must be passed to a worker process that write data on the fd. Can this be done with zeromq? thanks
[08:08] th mato: because i dont want to rate limit # of connections for valid connections
[08:08] mato th: how do you determine that a connection is valid?
[08:08] mato th: you know the IPs in advance?
[08:09] th mato: it is valid if the first message is asymmetrically signed+encrypted with the correct key
[08:10] mato th: okay, right... i see your problem.
[08:10] mato th: so for your purposes you need to know which peer that message came from...
[08:12] th mato: i think i would want to have a std::map or something which stores authenticated peers; then i give a penalty to new (not yet authenticated) peers (like greylisting); and i block (probably by notifying the systems firewall) repeated unauthenticated connection attempts
[08:12] th with block i mean blocking new connections from that IP address
[08:13] mato th: right, can't you just stick the peer IP inside the message? then start out by rate-limiting everyone (since that won't hurt the initial connection from an IP anyway), and then add the IP to a whitelist once you know it's good?
[08:13] mato th: I believe that sort of thing is possible with the iptables rate-limit stuff
[08:16] th mato: i dont really like to depend so much on the hosts firewall system
[08:17] th mato: actually the blocking part i suggested is only a last resort which i hope to no need. (because the greylisting for initial connections)
[08:17] mato th: well, abstract out the bit that talks to the firewall into a separate process, talk to it over zmq, and if/when you decide to change things it'll be easy :-)
[08:17] th mato: what about the possibility to run the code unprivileged?
[08:18] mato th: the bit that talks to the firewall?
[08:18] th mato: no the bit that modifies the firewall
[08:18] th mato: that would be a sad requirement for this 0mq-based solution (yea, we need access to your firewall)
[08:19] mato th: right... i don't know what the state of linux capabilities (assuming you're on linux) is like these days. so running it unprivileged, not sure.
[08:19] mato th: but privileged or not, if it's going to be modifying firewall rules, you're going to have to audit and secure that code anyway.
[08:20] th mato: i dont want that to be limited to linux at all... solaris, windows, bsd
[08:20] th mato: thats why i dont want to have the requirement to modify the firewall
[08:20] th mato: if i had such a callback, i would not need the firewall to protect myself
[08:21] mato th: well, i'm just offering alternative solutions that would work now.
[08:21] th mato: yes, i see that :)
[08:22] th mato: i'm still designing...
[08:22] mato th: if you abstract out the firewall bit and talk to it via zmq, there's nothing stopping you from having the other end run on windows/solaris/whatever.
[08:22] mato th: but yes, you end up with some rather system-dependent work you need to do.
[08:22] th yea
[08:57] pouete Hello :)
[08:58] pouete is it the right channel to ask a question about pyzmq ?
[09:24] djc pouete: I think so, yes
[09:30] pouete great :) I have a question about the "poller"
[09:34] pouete i would like my python program to unblock for exemple every 2 seconds if the zmq socket do not receiver anything.i found the "poller.poll" but i dont understand how to use it (where in my code and how using it ) .
[09:34] mikko pouete: are you using tornado or twisted or something like that?
[09:34] mikko if not you would create a loop like this:
[09:36] mikko while (true) { int num = zmq_poll(two_sec_timeout); if (num > 0) { if (socket [0].revents & ZMQ_POLLIN) { // something to read } } else { // do something during idle time } }
[09:36] mikko in pseudocode
[09:39] pouete just "zmq_poll(timeout)" ? nothing else ?
[09:48] djc pouete: in the python case, poller.poll(timeout)
[09:48] djc but yeah, that should unblock it every two seconds
[09:48] djc and then check if the returned list is empty
[09:52] pouete yep :) great
[13:03] aspidites thanks pieterh (about the async answer)
[13:03] pieterh aspidites: np, it's not always easy to understand the examples
[13:03] aspidites ah the joins of not doing network programming for 8 years then diving in head first
[13:03] pieterh in the asyncsrv example the worker sends a handful of replies back
[13:08] aspidites so the client sends request to fontend, which then distributes that to all workers in backend which returns the result to the front end which sends reply back to client?
[13:08] pieterh yes
[13:09] pieterh the simplest way to follow this is to add tracing to the code and run it
[13:11] aspidites thanks. i mostly get it. i just need to wake up now
[17:31] ptrb if I have a coredump, and I suspect a ZMQ PUB/SUB socket with no HWM has consumed a bunch of memory, is there some way to check eg. how many messages are in the socket buffer, from within gdb?
[18:08] p0lt hey everyone, wondering if someone might be able to expand upon why czmq needs autoconf v2.61 or higher? I'd love to compile this on my CentOS 5.4 machines, but sadly CentOS5.4 comes with autoconf v2.59...
[18:25] jer so tell yum to update autoconf? =]
[18:27] p0lt umm yeah, updating autoconf isn't a simple thing
[18:43] jer just out of curiosity, why not? it's a build time dependency, won't affect anything already installed.
[19:21] p0lt jer, mainly cause of all the other packages I'd need to upgrade to build that out, and then touching the thousands of machines with those rpms adds a bit more complexity to the mix...
[19:22] p0lt figured it wouldn't hurt to ask ahead of time...
[19:59] taotetek p0lt: you shouldn't need to update your autoconf on any box other than the one you're building your rpms on
[20:00] p0lt so its only required at build time... there aren't any gotcha dependencies that will be necessary in my prod tier
[20:18] jer p0lt, i didn't figure oyu'd be building on more than one system; to me it's a "wtf's the problem" type of problem =]
[20:19] p0lt I guess in the commotion I hadn't thought everything through... I'll see what I can put together, thanks jer and taotetek ... its been a busy day