ZeroMq IRC Log

Monday May 9, 2011

[Time] Name	Message
[03:40] aspidites	how would i modify the asycsrv example to send reply back to client?
[03:41] aspidites	if i'm reading it correctly, it would appear that it passes messags between worker threads and the server indefinitely
[04:38] sustrik	asyncsrv sounds like it's meant to be async, ie. no replies
[04:38] sustrik	REQ/REP is what you want foe replies
[05:00] pieterh	aspidites: the asyncsrv example already sends replies back to the client
[05:01] pieterh	sustrik: async means 'processing messages without wait states', it's orthogonal to the pattern
[05:01] pieterh	so happens async request-reply is exactly what that example does
[05:01] sustrik	ok, sorry
[05:01] pieterh	np, you're up early!
[05:01] sustrik	i tend to wake at 6am
[05:02] sustrik	so are you, btw :)
[05:02] sustrik	what about the evening?
[05:02] pieterh	like my 8-month old son, who wakes at 6am
[05:02] pieterh	you're arriving today?
[05:02] sustrik	yep
[05:02] sustrik	afterrnoon
[05:02] pieterh	excellent!
[05:02] sustrik	the same flight as mato
[05:02] lt_schmidt_jr	are you guys getting together in Moscow?
[05:02] sustrik	brussels
[05:02] pieterh	so you can come to iMatix if you like, to work here
[05:03] sustrik	moscow meetup was last week
[05:03] pieterh	this evening, meetup at 7pm in town, same location as the conference
[05:03] sustrik	ok
[05:03] lt_schmidt_jr	ah good - cause it will be a zoo tomorrow, nevermind
[05:03] sustrik	i think it'll take some time to get from charleroi
[05:03] pieterh	sustrik: if you can buy but tickets online that saves a lot of time
[05:03] pieterh	*bus tickets
[05:03] sustrik	ok
[05:04] pieterh	it's one bus every 45 mins for often too many people
[05:04] sustrik	the end of war celebrations?
[05:04] lt_schmidt_jr	Yup
[05:04] pieterh	ah, I thought it was the celebration of 150 years since they switched to the modern rail track size...
[05:05] pieterh	all those railway fanatics going crazy with vodka
[05:05] lt_schmidt_jr	he he he
[05:05] guido_g	good morning!
[05:05] sustrik	morning
[05:05] pieterh	hi guido_g!
[05:05] guido_g	pieterh: what did you do to the weather? forecast says thunderstorms...
[05:06] sustrik	yeah, need for warm clothing?
[05:06] pieterh	guido_g: all the sun got used up this weekend for barbequeues :-(
[05:06] pieterh	sustrik: it'll be 23c or so
[05:06] guido_g	sustrik: 24Â°C they say, so i think no
[05:06] sustrik	nice
[06:24] guido_g	coffee machine shut down, going to leave soon
[06:29] guido_g	off i go, see you in brusssels!
[06:34] th	thinking about DoS, is there any way to decide whether to accept a connection (POSIX socket accept())? would it be really out of 0mq's scope to have the possibility for a callback method there?
[06:39] sustrik	what's the criteria you would like to use for that?
[06:40] sustrik	overl number of connections?
[06:45] th	no
[06:45] th	i want some information as parameter to the callback
[06:46] th	remote peer address for example
[06:46] th	then i could do checks myself
[06:46] th	(did we have 10 failed authentications attempts before?)
[06:48] th	technically thats not possible for deciding "whether to accept a connection", because thats only known after accept()
[06:48] th	but that would be good enough. even if accept()ed, i could immediately drop the message, and if necessary send a message to the system to firewall the subject
[06:52] th	well - overall number of connections is something a user could evaluate for him/her-self if he/she had such a callback
[06:55] th	i'm not sure in which thread the callback should be running, though.
[07:54] th	sustrik: whats your opinion on this?
[07:56] mikko	good morning
[07:57] mato	morning
[07:59] mato	th: you are asking for a admin/monitoring infrastructure, right?
[07:59] th	mato: you could call it like that. but i'd call it a way to refuse connections.
[08:00] th	mato: i dont want to monitor with it. i really want to fight DoS
[08:00] mato	th: you're using zmq on the public internet? brave man :-)
[08:00] th	mato: i'm not. but it would be one step towards it.
[08:01] mato	th: well, the question is, should that be zmq's job or not... you can of course rate-limit with iptables and suchlike.
[08:01] mato	th: if you do it at the zmq level then all of a sudden it becomes part of your application policy
[08:01] th	mato: but i want to do limits based on information that is known inside 0mq
[08:01] mato	th: which is not necessarily a good idea.
[08:02] th	mato: what i do includes authenticated+encrypted messages already. so i can already drop everything which is invalid/illegal
[08:02] mato	th: well, why not just rate-limit your # of connections to the port in question with iptables?
[08:03] mato	that way, it's also external to your app. so if conditions change you can tweak your firewall rules on the fly accordingly.
[08:05] mato	otherwise, getting information out of zmq about # of peers/connection rate/... and otherwise manipulating individual peers, well, there's no support for that at the moment.
[08:05] mato	and you'll have to do a lot of convicing of el sustrik to get him to add anything like that since he's afraid of people abusing it for business logic :-)
[08:07] drakkan1000	Hi, I'm evaluating zeromq for my project, in my app I need to share file descriptors between different process. The main process must accept network connection and open an fd, this fd must be passed to a worker process that write data on the fd. Can this be done with zeromq? thanks
[08:08] th	mato: because i dont want to rate limit # of connections for valid connections
[08:08] mato	th: how do you determine that a connection is valid?
[08:08] mato	th: you know the IPs in advance?
[08:09] th	mato: it is valid if the first message is asymmetrically signed+encrypted with the correct key
[08:10] mato	th: okay, right... i see your problem.
[08:10] mato	th: so for your purposes you need to know which peer that message came from...
[08:12] th	mato: i think i would want to have a std::map or something which stores authenticated peers; then i give a penalty to new (not yet authenticated) peers (like greylisting); and i block (probably by notifying the systems firewall) repeated unauthenticated connection attempts
[08:12] th	with block i mean blocking new connections from that IP address
[08:13] mato	th: right, can't you just stick the peer IP inside the message? then start out by rate-limiting everyone (since that won't hurt the initial connection from an IP anyway), and then add the IP to a whitelist once you know it's good?
[08:13] mato	th: I believe that sort of thing is possible with the iptables rate-limit stuff
[08:16] th	mato: i dont really like to depend so much on the hosts firewall system
[08:17] th	mato: actually the blocking part i suggested is only a last resort which i hope to no need. (because the greylisting for initial connections)
[08:17] mato	th: well, abstract out the bit that talks to the firewall into a separate process, talk to it over zmq, and if/when you decide to change things it'll be easy :-)
[08:17] th	mato: what about the possibility to run the code unprivileged?
[08:18] mato	th: the bit that talks to the firewall?
[08:18] th	mato: no the bit that modifies the firewall
[08:18] th	mato: that would be a sad requirement for this 0mq-based solution (yea, we need access to your firewall)
[08:19] mato	th: right... i don't know what the state of linux capabilities (assuming you're on linux) is like these days. so running it unprivileged, not sure.
[08:19] mato	th: but privileged or not, if it's going to be modifying firewall rules, you're going to have to audit and secure that code anyway.
[08:20] th	mato: i dont want that to be limited to linux at all... solaris, windows, bsd
[08:20] th	mato: thats why i dont want to have the requirement to modify the firewall
[08:20] th	mato: if i had such a callback, i would not need the firewall to protect myself
[08:21] mato	th: well, i'm just offering alternative solutions that would work now.
[08:21] th	mato: yes, i see that :)
[08:22] th	mato: i'm still designing...
[08:22] mato	th: if you abstract out the firewall bit and talk to it via zmq, there's nothing stopping you from having the other end run on windows/solaris/whatever.
[08:22] mato	th: but yes, you end up with some rather system-dependent work you need to do.
[08:22] th	yea
[08:57] pouete	Hello :)
[08:58] pouete	is it the right channel to ask a question about pyzmq ?
[09:24] djc	pouete: I think so, yes
[09:30] pouete	great :) I have a question about the "poller"
[09:34] pouete	i would like my python program to unblock for exemple every 2 seconds if the zmq socket do not receiver anything.i found the "poller.poll" but i dont understand how to use it (where in my code and how using it ) .
[09:34] mikko	pouete: are you using tornado or twisted or something like that?
[09:34] mikko	if not you would create a loop like this:
[09:36] mikko	while (true) { int num = zmq_poll(two_sec_timeout); if (num > 0) { if (socket [0].revents & ZMQ_POLLIN) { // something to read } } else { // do something during idle time } }
[09:36] mikko	in pseudocode
[09:39] pouete	just "zmq_poll(timeout)" ? nothing else ?
[09:48] djc	pouete: in the python case, poller.poll(timeout)
[09:48] djc	but yeah, that should unblock it every two seconds
[09:48] djc	and then check if the returned list is empty
[09:52] pouete	yep :) great
[13:03] aspidites	thanks pieterh (about the async answer)
[13:03] pieterh	aspidites: np, it's not always easy to understand the examples
[13:03] aspidites	ah the joins of not doing network programming for 8 years then diving in head first
[13:03] pieterh	in the asyncsrv example the worker sends a handful of replies back
[13:08] aspidites	so the client sends request to fontend, which then distributes that to all workers in backend which returns the result to the front end which sends reply back to client?
[13:08] pieterh	yes
[13:09] pieterh	the simplest way to follow this is to add tracing to the code and run it
[13:11] aspidites	thanks. i mostly get it. i just need to wake up now
[17:31] ptrb	if I have a coredump, and I suspect a ZMQ PUB/SUB socket with no HWM has consumed a bunch of memory, is there some way to check eg. how many messages are in the socket buffer, from within gdb?
[18:08] p0lt	hey everyone, wondering if someone might be able to expand upon why czmq needs autoconf v2.61 or higher? I'd love to compile this on my CentOS 5.4 machines, but sadly CentOS5.4 comes with autoconf v2.59...
[18:25] jer	so tell yum to update autoconf? =]
[18:27] p0lt	umm yeah, updating autoconf isn't a simple thing
[18:43] jer	just out of curiosity, why not? it's a build time dependency, won't affect anything already installed.
[19:21] p0lt	jer, mainly cause of all the other packages I'd need to upgrade to build that out, and then touching the thousands of machines with those rpms adds a bit more complexity to the mix...
[19:22] p0lt	figured it wouldn't hurt to ask ahead of time...
[19:59] taotetek	p0lt: you shouldn't need to update your autoconf on any box other than the one you're building your rpms on
[20:00] p0lt	so its only required at build time... there aren't any gotcha dependencies that will be necessary in my prod tier
[20:18] jer	p0lt, i didn't figure oyu'd be building on more than one system; to me it's a "wtf's the problem" type of problem =]
[20:19] p0lt	I guess in the commotion I hadn't thought everything through... I'll see what I can put together, thanks jer and taotetek ... its been a busy day