Thursday September 1, 2011

[Time] NameMessage
[00:00] Seta00 scripts don't segfault
[00:00] Seta00 attach to the interpreter
[00:02] sigmonsay turns out you can't share a context between procs. Makes sense, code should have raise an exception instead of segfaulting
[00:02] sigmonsay immature libraries.
[00:03] sigmonsay Using a inproc with multiple callers to bind on PUSH/PULL setup doesn't load balance requests =/
[00:09] sigmonsay anyone use multiple inproc PULL "consumers" in python and have them load balance?
[00:09] sigmonsay woops, s/inproc/ipc/ *
[00:46] mikko sigmonsay: how does the code look like?
[00:47] AlbertoEAF hello everyone :)
[00:48] jhawk28 hello
[00:49] AlbertoEAF I am doing a simple client/server app to learn 0mq but apparently I can't even do the first steps right. I'm using C++
[00:49] AlbertoEAF I am using REQ/REP
[00:49] mikko hi
[00:49] mikko jhawk28: did you open libzmq-105?
[00:50] mikko AlbertoEAF: have you read the guide?
[00:50] AlbertoEAF yes
[00:50] jhawk28 mikko: yes
[00:50] mikko is there a specific problem you are facing?
[00:50] mikko jhawk28: there is a problem with throwing an error from connect
[00:50] mikko due to the actual connection happening in another thread
[00:50] AlbertoEAF I have the server printing the messages it receives from the client
[00:50] mikko throwing exception from there would go unhandled
[00:51] AlbertoEAF and the client just prompts me in a infinite loop for commands
[00:51] mikko as the zmq_connect has already returned
[00:51] jhawk28 no, I didnt open 105
[00:51] mikko AlbertoEAF: this far sounds reasonable
[00:51] AlbertoEAF however if the commands include spaces; the server executes 1 recv per word
[00:51] mikko jhawk28: ok
[00:51] jhawk28 I did open the one related to invalid zmq connect
[00:51] jhawk28 that does an assertion instead
[00:52] mikko jhawk28: i might remember wrong ticket number
[00:52] mikko AlbertoEAF: can i see the code?
[00:52] AlbertoEAF it seems that zeromq is separating the messages into words and sending those down the line
[00:52] mikko http://gist/
[00:52] mikko ermm
[00:52] AlbertoEAF sure
[00:52] jhawk28 mikko: 155
[00:53] mikko yeah
[00:53] jhawk28 they fixed it in 3.0.1 and it should be fixed on the trunk for 2.1
[00:54] mikko it got pushed to 2.1 eventually?
[00:54] mikko i've lost track of issues lately
[00:54] jhawk28
[00:55] AlbertoEAF Public Clone URL:git://
[00:57] AlbertoEAF damn it, I believe it is the cin that's doing this
[00:57] mikko AlbertoEAF: its std::cin
[00:57] AlbertoEAF :D
[00:58] mikko you might want std::getline
[00:58] AlbertoEAF ok, thanks for the tip :)
[00:58] mikko std::string in; std::getline(std::cin, in); iirc
[00:59] AlbertoEAF ah, by the way
[01:00] AlbertoEAF I want to do another dropbox-like clone for linux+windows+mac (yes I know that there are already many out there :p )
[01:01] mikko sounds like a nice project
[01:01] AlbertoEAF and I was wondering if zeromq was appropriate for this or if I should choose libev
[01:02] mikko are you handling messages or streams?
[01:02] mikko (logically)
[01:03] mikko i dont see why you couldn't use zeromq and chunk the files into smaller messages
[01:03] AlbertoEAF i don't know how to answer to that one
[01:03] mikko with libev and bsd sockets you need to do quite a lot of boilerplate
[01:04] AlbertoEAF basically I have already created a protocol to communicate with the server
[01:04] mikko reconnections, read/write retrying, etc
[01:04] AlbertoEAF it supports certain types of requests/actions like MOVE, ADD_FILE, ADD_FOLDER, UPDATE_FILE, etc
[01:04] mikko how do you handle security?
[01:04] mikko pki?
[01:05] AlbertoEAF and then I would have to do another connection just for sending files keeping the request socket open
[01:05] AlbertoEAF I haven't dealt with that yet :)
[01:06] AlbertoEAF I was thinking of encrypting every message with openssl
[01:07] AlbertoEAF (but zeromq doesn't yet support it or at least that's what I've read)
[01:08] mikko encryption is more of an application layer thing
[01:08] AlbertoEAF ok
[01:08] AlbertoEAF so those are my questions: scalability; security and throughtput
[01:09] AlbertoEAF supposedly zeromq is great at scalability and security can be put at the application level
[01:09] AlbertoEAF but to do the transfer of the files
[01:09] AlbertoEAF is it fast or should I use other libraries, etc.
[01:09] AlbertoEAF ?
[01:16] mikko AlbertoEAF: you can easily saturate normal users bw
[01:20] AlbertoEAF mikko: "bw"? ; then I shouldn't be using zeromq?
[01:29] AlbertoEAF well, I've got to go but thanks for the help :)
[02:36] sigmonsay mikko, want to see a paste.. if you're still around. I'm quite curious of why this doesn't balance
[05:27] CIA-32 libzmq: 03Martin Sustrik 07master * r65bb758 10/ (src/dist.cpp src/socket_base.cpp): Fixed warnings on Win64 ...
[05:34] sustrik sigmonsay: context is a local object, how can you possibly pass it to a different process?
[05:34] sustrik using some kind of python serialisation?
[05:35] sustrik if so, the python context should be marked as non-serialisable
[08:47] mikko pieterh: there?
[08:48] mikko you are a hard man to catch
[08:48] pieterh mikko: hi :-)
[10:05] ntelford mikko, seems I'm going to by in NYC during the pubsub huddle after all
[10:06] ntelford shame, is it going to be an annual thing/
[10:06] ntelford *?
[10:27] mikko ntelford: well see how it goes
[10:28] mikko if it goes well
[11:04] CIA-32 libzmq: 03Martin Sustrik 07master * r82ab08d 10/ src/xsub.cpp : Premature deallocation bug in XSUB fixed (issue 244) ...
[16:31] sigmonsay Will zmq use epoll if possible?
[16:49] felipejcruz sigmonsay: yes.. take a look
[16:56] sigmonsay thanks felipejcruz. Have you used the python zmq lib? It seems rather unstable
[16:56] sigmonsay Ie, i've segfaulted python twice doing the "wrong thing"
[16:58] felipejcruz yes..
[16:59] Plouj- wait, zmq is written in C++??
[16:59] Plouj- (as opposed to C)
[16:59] felipejcruz sigmonsay: if you pastebin your code maybe someone can help.. :)
[17:00] sigmonsay I'm good at debugging, so I Figured it out. It's just not cool to segfault =P
[17:01] felipejcruz sigmonsay: I'm workin on a zmq python lib using ctypes on top of czmq
[17:01] felipejcruz but still in early development
[17:01] michelp felipejcruz, nice
[17:01] michelp seems like that would be a good fit
[17:02] felipejcruz michelp: great learning experience
[17:02] felipejcruz plus i'll work with pypy
[17:02] felipejcruz it will*
[17:02] michelp i've always wanted to do a ctypes project but haven't had the opportunity (or time)
[17:03] felipejcruz do you have time nowdays?
[17:03] michelp i suppose the api would look more like czmq than the current pyzmq api?
[17:03] sigmonsay Nice
[17:03] felipejcruz I'm trying to follow pyzmq.. so far, i got some tests from pyzmq running
[17:04] felipejcruz but i'm facing a "problem" now.. czmq has a zloop approach insted of a poller approach
[17:04] michelp felipejcruz, i'd certainly be willing to try it with some of my code if the api isn't too incredibly different
[17:04] felipejcruz if your code only send/recv strings, maybe it will work
[17:05] michelp hmm, is the problem that it's hard for zloop to callback to python functions?
[17:05] michelp my code uses poller extensively
[17:05] felipejcruz i can emulate a poll creating a thread and creating this loop on this thread
[17:06] felipejcruz because zloop take the control of running program..
[17:06] felipejcruz but it can be changed to expose zmq_poll func
[17:06] felipejcruz i'll probably do that
[17:06] michelp hmm... yeah, i'm not against that paradigm, i've used Twisted a lot in the past and have enjoyed it
[17:06] felipejcruz i'm not agains, but i don't really like.. matter os taste to me :)
[17:06] michelp seems like exposing a zmq_poll wouldn't be that hard... but obviously I'm not speaking from experience
[17:07] michelp you have a public repo?
[17:09] felipejcruz not yet.. but if you really want I can share with you.. I want implement zmq.Poll before publish (i don't know if this word is the right one:) )
[17:10] felipejcruz good thing about using czmq is that it hides differences between different zmq versions
[17:10] felipejcruz so it should work with 2.x to 3.x (maybe with libzmq 4.0 unstable too) but I'm not sure
[17:10] michelp i would encourage you to push to a repo, no one is going to judge your code :) it would make it easier for sharing
[17:11] felipejcruz michelp: i'm afraid of judging :D
[17:12] felipejcruz but since at least one person got interested, i'll put on my github
[17:12] felipejcruz :)
[17:13] michelp awesome, i don't think you'll regret it, the more people can see that it is out there, the more help you'll get
[17:14] michelp also there's no saying that maybe the pyzmq authors will get ideas from your code like you've gotten ideas from theirs, that's the beauty of shared code. mixing ideas.
[17:17] dcolish ctypes are not a good choice for a performance oriented python module
[17:19] felipejcruz dcolish: pypy jit optimize ctypes calls now
[17:20] dcolish not everyone is using pypy
[17:20] dcolish i'd say most applications are still using cpython
[17:20] felipejcruz fair.. but some people are
[17:20] felipejcruz :)
[17:20] dcolish sure
[17:21] dcolish however, i've yet to see an actual comparison of ctypes in pypy vs cython
[17:21] dcolish using the C-API has benefits
[17:21] felipejcruz if I got sometime maybe I'll do that
[17:22] felipejcruz but if you you have some heavy python processing, you can benefit of using pypy.. even if cython zmq is faster than ctypes on pypy..
[17:23] dcolish sure, depends on where your bottleneck is
[17:23] Plouj- felipejcruz: you should use cython
[17:23] dcolish you could easily have issues with the marshaling of datatypes if you have a high message volume
[17:23] dcolish thats why i'd avoid ctypes
[17:24] Plouj- oh
[17:24] Plouj- felipejcruz: what is the benefit of using ctypes over cython for you?
[17:25] felipejcruz as I said.. to me, this is a learning experience
[17:25] Plouj- ok, nvm
[17:28] dcolish yeah all that said, ctypes are a great way to experiment
[17:28] dcolish and if that new pypy stackless branch lands you might really have something there
[17:29] felipejcruz Plouj-: to me it that cython isn't compatible with pypy
[17:29] felipejcruz it = is
[17:31] Plouj- I see
[17:31] felipejcruz plus it's fun :)
[17:48] kilpatds I've perhaps a stupid question (yes, I've read the guide) if someone could help me figure it out...
[17:48] kilpatds I've got a router socket to handle serveral clients. A given client request could generate a ... large amount of response (split over many messages).
[17:48] kilpatds I'd like to throttle my replies. The source data ain't going anywhere.
[17:48] kilpatds Any obvious way to do that? Or is this just the wrong connection pattern?
[17:55] kilpatds (er, to clarify. I'd like to get blocking behavior, but I have multiple clients and need to be able to route messages correctly)
[18:13] felipejcruz kilpatds: you want to block all yours server send() until this specific client gets this large reply?
[18:14] kilpatds I want to know to stop sending to the client right now (cause not all the data's gotten there yet), and I want to know when a good time to start sending again would be
[18:15] kilpatds while still handling other clients
[18:16] kilpatds basically, I've got, say 4G of stuff here that I need to send to client, in nice 64k chunks. I'd rather not duplicate it in zmq buffers, or cause 4G of ram to be allocated, or drop the last 3G onto the floor
[18:19] felipejcruz i really don't know how to help you with this.. sorry
[18:33] pieterh kilpatds: this may help:
[18:48] sigmonsay Is there UDP support in 0MQ?
[19:00] pieterh sigmonsay: there's provisional support via the VTX project in
[19:07] ParadoxG Would performance be better if you had many ZMQ threads doing blocking IO, or have a single thread with non-blocking IO?
[19:14] pieterh ParadoxG: you won't get extraordinary performance if you increase the number of threads
[19:15] pieterh but it's not smart to (try to) optimize a design for performance until you actually have testable code
[19:23] ParadoxG I would think it would be easier to write an application that uses blocking io with ZMQ threads, than it would be to write a fancy non-blocking one
[19:44] sigmonsay pieterh, interesting. Thanks for that. imma watching that project
[19:56] kilpatds pieterh: yes, that looks like it could be helpful. Thanks
[22:09] ahbritto How can I use ssl with pyzmq?
[22:11] michelp ahbritto, you could use openvpn or stunnel
[22:11] ahbritto Ow. :(
[22:11] michelp or even ssh port forwarding
[22:12] michelp what were you expecting?
[22:12] ahbritto ssl is such an easy thing in python. You just specify a cert when opening a socket.
[22:16] michelp ah i see what you're saying. you can still use pyopenssl to encrypt your messages before sending them over the wire
[22:19] xristos pyopenssl is a pain in the ass tho ;p
[22:21] michelp well, maybe the core devs would entertain the idea of a patch adding optional support that's as simple as what python's socket module provides
[22:21] ahbritto michelp: It would be better all communication was encrypted, I could use the other parties ssl cert for authenticaton too.
[22:21] ahbritto michelp: But all bindings would need to support it. :/
[22:22] michelp yeah it would be a pain. and honestly i don't want all my traffic encrypted for my use cases, it would be much better if it were optional
[22:30] ahbritto What I'd like to see is something like websocket that let me ride https so that I would not need to worry about firewalls.
[22:33] michelp i'm no expert, but as i understand it websockets are binary clean, the data frames contain utf-8 data. maybe the technology has advanced beyond my understanding of it
[22:33] michelp 'are not binary clean'
[22:34] ahbritto michelp: I think that has been resolved.
[22:34] michelp i should probably shut up since this gets into levels of stack i have less knowledge of ;)
[22:35] ahbritto Trying to find the right combination of features in a single solution is turning out to be a night mare for me. :/
[22:35] ahbritto zeromq looked so good.
[22:35] ahbritto I'll probably use it internally.
[22:35] ahbritto A real shame I can't use it public facing.
[22:40] michelp ah "while binary frames use a length prefix."
[22:40] michelp wikipedia is not as up to date as
[22:41] michelp maybe websocket isn't such a crazy idea, depending on if your client can support it
[22:41] michelp or use something like Mongrel2 to translate the http part of your protocol into 0mq messages
[22:41] ahbritto I'd love to use, but it seems little imature with no client side python support.
[22:41] michelp Mongrel2 grew ssl support recently IIRC
[22:42] ahbritto I'd like clients to easily be javascript or python.
[22:42] ahbritto And, I'd like to do this with stable technology. ;)
[22:42] ahbritto I'm thinking about what would have to happen to zeromq to do ssl...
[22:43] ahbritto You need to know the identity of a message sender.
[22:43] ahbritto So, you'd want to put the ssl finger print in messages.
[22:43] ahbritto This is after the stunnel connection.
[22:58] ahbritto Is zeromq easily runnable with Apache (poll integration)?
[22:59] ahbritto Nevermind, mongrel2 might be fine.