[Time] Name | Message |
[00:00] Seta00
|
scripts don't segfault
|
[00:00] Seta00
|
attach to the interpreter
|
[00:02] sigmonsay
|
turns out you can't share a context between procs. Makes sense, code should have raise an exception instead of segfaulting
|
[00:02] sigmonsay
|
immature libraries.
|
[00:03] sigmonsay
|
Using a inproc with multiple callers to bind on PUSH/PULL setup doesn't load balance requests =/
|
[00:09] sigmonsay
|
anyone use multiple inproc PULL "consumers" in python and have them load balance?
|
[00:09] sigmonsay
|
woops, s/inproc/ipc/ *
|
[00:46] mikko
|
sigmonsay: how does the code look like?
|
[00:47] AlbertoEAF
|
hello everyone :)
|
[00:48] jhawk28
|
hello
|
[00:49] AlbertoEAF
|
I am doing a simple client/server app to learn 0mq but apparently I can't even do the first steps right. I'm using C++
|
[00:49] AlbertoEAF
|
I am using REQ/REP
|
[00:49] mikko
|
hi
|
[00:49] mikko
|
jhawk28: did you open libzmq-105?
|
[00:50] mikko
|
AlbertoEAF: have you read the guide?
|
[00:50] AlbertoEAF
|
yes
|
[00:50] jhawk28
|
mikko: yes
|
[00:50] mikko
|
is there a specific problem you are facing?
|
[00:50] mikko
|
jhawk28: there is a problem with throwing an error from connect
|
[00:50] mikko
|
due to the actual connection happening in another thread
|
[00:50] AlbertoEAF
|
I have the server printing the messages it receives from the client
|
[00:50] mikko
|
throwing exception from there would go unhandled
|
[00:51] AlbertoEAF
|
and the client just prompts me in a infinite loop for commands
|
[00:51] mikko
|
as the zmq_connect has already returned
|
[00:51] jhawk28
|
no, I didnt open 105
|
[00:51] mikko
|
AlbertoEAF: this far sounds reasonable
|
[00:51] AlbertoEAF
|
however if the commands include spaces; the server executes 1 recv per word
|
[00:51] mikko
|
jhawk28: ok
|
[00:51] jhawk28
|
I did open the one related to invalid zmq connect
|
[00:51] jhawk28
|
that does an assertion instead
|
[00:52] mikko
|
jhawk28: i might remember wrong ticket number
|
[00:52] mikko
|
AlbertoEAF: can i see the code?
|
[00:52] AlbertoEAF
|
it seems that zeromq is separating the messages into words and sending those down the line
|
[00:52] mikko
|
http://gist/github.com
|
[00:52] mikko
|
ermm gist.github.com
|
[00:52] AlbertoEAF
|
sure
|
[00:52] jhawk28
|
mikko: 155
|
[00:53] mikko
|
yeah
|
[00:53] jhawk28
|
they fixed it in 3.0.1 and it should be fixed on the trunk for 2.1
|
[00:54] mikko
|
it got pushed to 2.1 eventually?
|
[00:54] mikko
|
i've lost track of issues lately
|
[00:54] jhawk28
|
https://github.com/zeromq/zeromq2-1/commit/a41271a0c1840729f4a439b0bd240ca78af04f02
|
[00:55] AlbertoEAF
|
Public Clone URL:git://gist.github.com/1185151.git
|
[00:57] AlbertoEAF
|
damn it, I believe it is the cin that's doing this
|
[00:57] mikko
|
AlbertoEAF: its std::cin
|
[00:57] AlbertoEAF
|
:D
|
[00:58] mikko
|
you might want std::getline
|
[00:58] AlbertoEAF
|
ok, thanks for the tip :)
|
[00:58] mikko
|
std::string in; std::getline(std::cin, in); iirc
|
[00:59] AlbertoEAF
|
ah, by the way
|
[01:00] AlbertoEAF
|
I want to do another dropbox-like clone for linux+windows+mac (yes I know that there are already many out there :p )
|
[01:01] mikko
|
sounds like a nice project
|
[01:01] AlbertoEAF
|
and I was wondering if zeromq was appropriate for this or if I should choose libev
|
[01:02] mikko
|
are you handling messages or streams?
|
[01:02] mikko
|
(logically)
|
[01:03] mikko
|
i dont see why you couldn't use zeromq and chunk the files into smaller messages
|
[01:03] AlbertoEAF
|
i don't know how to answer to that one
|
[01:03] mikko
|
with libev and bsd sockets you need to do quite a lot of boilerplate
|
[01:04] AlbertoEAF
|
basically I have already created a protocol to communicate with the server
|
[01:04] mikko
|
reconnections, read/write retrying, etc
|
[01:04] AlbertoEAF
|
it supports certain types of requests/actions like MOVE, ADD_FILE, ADD_FOLDER, UPDATE_FILE, etc
|
[01:04] mikko
|
how do you handle security?
|
[01:04] mikko
|
pki?
|
[01:05] AlbertoEAF
|
and then I would have to do another connection just for sending files keeping the request socket open
|
[01:05] AlbertoEAF
|
I haven't dealt with that yet :)
|
[01:06] AlbertoEAF
|
I was thinking of encrypting every message with openssl
|
[01:07] AlbertoEAF
|
(but zeromq doesn't yet support it or at least that's what I've read)
|
[01:08] mikko
|
encryption is more of an application layer thing
|
[01:08] AlbertoEAF
|
ok
|
[01:08] AlbertoEAF
|
so those are my questions: scalability; security and throughtput
|
[01:09] AlbertoEAF
|
supposedly zeromq is great at scalability and security can be put at the application level
|
[01:09] AlbertoEAF
|
but to do the transfer of the files
|
[01:09] AlbertoEAF
|
is it fast or should I use other libraries, etc.
|
[01:09] AlbertoEAF
|
?
|
[01:16] mikko
|
AlbertoEAF: you can easily saturate normal users bw
|
[01:20] AlbertoEAF
|
mikko: "bw"? ; then I shouldn't be using zeromq?
|
[01:29] AlbertoEAF
|
well, I've got to go but thanks for the help :)
|
[02:36] sigmonsay
|
mikko, want to see a paste.. if you're still around. I'm quite curious of why this doesn't balance
|
[05:27] CIA-32
|
libzmq: 03Martin Sustrik 07master * r65bb758 10/ (src/dist.cpp src/socket_base.cpp): Fixed warnings on Win64 ...
|
[05:34] sustrik
|
sigmonsay: context is a local object, how can you possibly pass it to a different process?
|
[05:34] sustrik
|
using some kind of python serialisation?
|
[05:35] sustrik
|
if so, the python context should be marked as non-serialisable
|
[08:47] mikko
|
pieterh: there?
|
[08:48] mikko
|
you are a hard man to catch
|
[08:48] pieterh
|
mikko: hi :-)
|
[10:05] ntelford
|
mikko, seems I'm going to by in NYC during the pubsub huddle after all
|
[10:06] ntelford
|
shame, is it going to be an annual thing/
|
[10:06] ntelford
|
*?
|
[10:27] mikko
|
ntelford: well see how it goes
|
[10:28] mikko
|
if it goes well
|
[11:04] CIA-32
|
libzmq: 03Martin Sustrik 07master * r82ab08d 10/ src/xsub.cpp : Premature deallocation bug in XSUB fixed (issue 244) ...
|
[16:31] sigmonsay
|
Will zmq use epoll if possible?
|
[16:49] felipejcruz
|
sigmonsay: yes.. take a look https://github.com/zeromq/zeromq3-0/blob/master/src/epoll.cpp
|
[16:56] sigmonsay
|
thanks felipejcruz. Have you used the python zmq lib? It seems rather unstable
|
[16:56] sigmonsay
|
Ie, i've segfaulted python twice doing the "wrong thing"
|
[16:58] felipejcruz
|
yes..
|
[16:59] Plouj-
|
wait, zmq is written in C++??
|
[16:59] Plouj-
|
(as opposed to C)
|
[16:59] felipejcruz
|
sigmonsay: if you pastebin your code maybe someone can help.. :)
|
[17:00] sigmonsay
|
I'm good at debugging, so I Figured it out. It's just not cool to segfault =P
|
[17:01] felipejcruz
|
sigmonsay: I'm workin on a zmq python lib using ctypes on top of czmq
|
[17:01] felipejcruz
|
but still in early development
|
[17:01] michelp
|
felipejcruz, nice
|
[17:01] michelp
|
seems like that would be a good fit
|
[17:02] felipejcruz
|
michelp: great learning experience
|
[17:02] felipejcruz
|
plus i'll work with pypy
|
[17:02] felipejcruz
|
it will*
|
[17:02] michelp
|
i've always wanted to do a ctypes project but haven't had the opportunity (or time)
|
[17:03] felipejcruz
|
do you have time nowdays?
|
[17:03] michelp
|
i suppose the api would look more like czmq than the current pyzmq api?
|
[17:03] sigmonsay
|
Nice
|
[17:03] felipejcruz
|
I'm trying to follow pyzmq.. so far, i got some tests from pyzmq running
|
[17:04] felipejcruz
|
but i'm facing a "problem" now.. czmq has a zloop approach insted of a poller approach
|
[17:04] michelp
|
felipejcruz, i'd certainly be willing to try it with some of my code if the api isn't too incredibly different
|
[17:04] felipejcruz
|
if your code only send/recv strings, maybe it will work
|
[17:05] michelp
|
hmm, is the problem that it's hard for zloop to callback to python functions?
|
[17:05] michelp
|
my code uses poller extensively
|
[17:05] felipejcruz
|
i can emulate a poll creating a thread and creating this loop on this thread
|
[17:06] felipejcruz
|
because zloop take the control of running program..
|
[17:06] felipejcruz
|
but it can be changed to expose zmq_poll func
|
[17:06] felipejcruz
|
i'll probably do that
|
[17:06] michelp
|
hmm... yeah, i'm not against that paradigm, i've used Twisted a lot in the past and have enjoyed it
|
[17:06] felipejcruz
|
i'm not agains, but i don't really like.. matter os taste to me :)
|
[17:06] michelp
|
seems like exposing a zmq_poll wouldn't be that hard... but obviously I'm not speaking from experience
|
[17:07] michelp
|
you have a public repo?
|
[17:09] felipejcruz
|
not yet.. but if you really want I can share with you.. I want implement zmq.Poll before publish (i don't know if this word is the right one:) )
|
[17:10] felipejcruz
|
good thing about using czmq is that it hides differences between different zmq versions
|
[17:10] felipejcruz
|
so it should work with 2.x to 3.x (maybe with libzmq 4.0 unstable too) but I'm not sure
|
[17:10] michelp
|
i would encourage you to push to a repo, no one is going to judge your code :) it would make it easier for sharing
|
[17:11] felipejcruz
|
michelp: i'm afraid of judging :D
|
[17:12] felipejcruz
|
but since at least one person got interested, i'll put on my github
|
[17:12] felipejcruz
|
:)
|
[17:13] michelp
|
awesome, i don't think you'll regret it, the more people can see that it is out there, the more help you'll get
|
[17:14] michelp
|
also there's no saying that maybe the pyzmq authors will get ideas from your code like you've gotten ideas from theirs, that's the beauty of shared code. mixing ideas.
|
[17:17] dcolish
|
ctypes are not a good choice for a performance oriented python module
|
[17:19] felipejcruz
|
dcolish: pypy jit optimize ctypes calls now
|
[17:20] dcolish
|
not everyone is using pypy
|
[17:20] dcolish
|
i'd say most applications are still using cpython
|
[17:20] felipejcruz
|
fair.. but some people are
|
[17:20] felipejcruz
|
:)
|
[17:20] dcolish
|
sure
|
[17:21] dcolish
|
however, i've yet to see an actual comparison of ctypes in pypy vs cython
|
[17:21] dcolish
|
using the C-API has benefits
|
[17:21] felipejcruz
|
if I got sometime maybe I'll do that
|
[17:22] felipejcruz
|
but if you you have some heavy python processing, you can benefit of using pypy.. even if cython zmq is faster than ctypes on pypy..
|
[17:23] dcolish
|
sure, depends on where your bottleneck is
|
[17:23] Plouj-
|
felipejcruz: you should use cython
|
[17:23] dcolish
|
you could easily have issues with the marshaling of datatypes if you have a high message volume
|
[17:23] dcolish
|
thats why i'd avoid ctypes
|
[17:24] Plouj-
|
oh
|
[17:24] Plouj-
|
felipejcruz: what is the benefit of using ctypes over cython for you?
|
[17:25] felipejcruz
|
as I said.. to me, this is a learning experience
|
[17:25] Plouj-
|
ok, nvm
|
[17:28] dcolish
|
yeah all that said, ctypes are a great way to experiment
|
[17:28] dcolish
|
and if that new pypy stackless branch lands you might really have something there
|
[17:29] felipejcruz
|
Plouj-: to me it that cython isn't compatible with pypy
|
[17:29] felipejcruz
|
it = is
|
[17:31] Plouj-
|
I see
|
[17:31] felipejcruz
|
plus it's fun :)
|
[17:48] kilpatds
|
I've perhaps a stupid question (yes, I've read the guide) if someone could help me figure it out...
|
[17:48] kilpatds
|
I've got a router socket to handle serveral clients. A given client request could generate a ... large amount of response (split over many messages).
|
[17:48] kilpatds
|
I'd like to throttle my replies. The source data ain't going anywhere.
|
[17:48] kilpatds
|
Any obvious way to do that? Or is this just the wrong connection pattern?
|
[17:55] kilpatds
|
(er, to clarify. I'd like to get blocking behavior, but I have multiple clients and need to be able to route messages correctly)
|
[18:13] felipejcruz
|
kilpatds: you want to block all yours server send() until this specific client gets this large reply?
|
[18:14] kilpatds
|
I want to know to stop sending to the client right now (cause not all the data's gotten there yet), and I want to know when a good time to start sending again would be
|
[18:15] kilpatds
|
while still handling other clients
|
[18:16] kilpatds
|
basically, I've got, say 4G of stuff here that I need to send to client, in nice 64k chunks. I'd rather not duplicate it in zmq buffers, or cause 4G of ram to be allocated, or drop the last 3G onto the floor
|
[18:19] felipejcruz
|
i really don't know how to help you with this.. sorry
|
[18:33] pieterh
|
kilpatds: this may help: http://unprotocols.org/blog:15
|
[18:48] sigmonsay
|
Is there UDP support in 0MQ?
|
[19:00] pieterh
|
sigmonsay: there's provisional support via the VTX project in https://github.com/imatix/vtx/v3.
|
[19:07] ParadoxG
|
Would performance be better if you had many ZMQ threads doing blocking IO, or have a single thread with non-blocking IO?
|
[19:14] pieterh
|
ParadoxG: you won't get extraordinary performance if you increase the number of threads
|
[19:15] pieterh
|
but it's not smart to (try to) optimize a design for performance until you actually have testable code
|
[19:23] ParadoxG
|
I would think it would be easier to write an application that uses blocking io with ZMQ threads, than it would be to write a fancy non-blocking one
|
[19:44] sigmonsay
|
pieterh, interesting. Thanks for that. imma watching that project
|
[19:56] kilpatds
|
pieterh: yes, that looks like it could be helpful. Thanks
|
[22:09] ahbritto
|
How can I use ssl with pyzmq?
|
[22:11] michelp
|
ahbritto, you could use openvpn or stunnel
|
[22:11] ahbritto
|
Ow. :(
|
[22:11] michelp
|
or even ssh port forwarding
|
[22:12] michelp
|
what were you expecting?
|
[22:12] ahbritto
|
ssl is such an easy thing in python. You just specify a cert when opening a socket.
|
[22:16] michelp
|
ah i see what you're saying. you can still use pyopenssl to encrypt your messages before sending them over the wire
|
[22:19] xristos
|
pyopenssl is a pain in the ass tho ;p
|
[22:21] michelp
|
well, maybe the core devs would entertain the idea of a patch adding optional support that's as simple as what python's socket module provides
|
[22:21] ahbritto
|
michelp: It would be better all communication was encrypted, I could use the other parties ssl cert for authenticaton too.
|
[22:21] ahbritto
|
michelp: But all bindings would need to support it. :/
|
[22:22] michelp
|
yeah it would be a pain. and honestly i don't want all my traffic encrypted for my use cases, it would be much better if it were optional
|
[22:30] ahbritto
|
What I'd like to see is something like websocket that let me ride https so that I would not need to worry about firewalls.
|
[22:33] michelp
|
i'm no expert, but as i understand it websockets are binary clean, the data frames contain utf-8 data. maybe the technology has advanced beyond my understanding of it
|
[22:33] michelp
|
'are not binary clean'
|
[22:34] ahbritto
|
michelp: I think that has been resolved.
|
[22:34] michelp
|
i should probably shut up since this gets into levels of stack i have less knowledge of ;)
|
[22:35] ahbritto
|
Trying to find the right combination of features in a single solution is turning out to be a night mare for me. :/
|
[22:35] ahbritto
|
zeromq looked so good.
|
[22:35] ahbritto
|
I'll probably use it internally.
|
[22:35] ahbritto
|
A real shame I can't use it public facing.
|
[22:40] michelp
|
ah "while binary frames use a length prefix."
|
[22:40] michelp
|
wikipedia is not as up to date as websocket.org
|
[22:41] michelp
|
maybe websocket isn't such a crazy idea, depending on if your client can support it
|
[22:41] michelp
|
or use something like Mongrel2 to translate the http part of your protocol into 0mq messages
|
[22:41] ahbritto
|
I'd love to use socket.io, but it seems little imature with no client side python support.
|
[22:41] michelp
|
Mongrel2 grew ssl support recently IIRC
|
[22:42] ahbritto
|
I'd like clients to easily be javascript or python.
|
[22:42] ahbritto
|
And, I'd like to do this with stable technology. ;)
|
[22:42] ahbritto
|
I'm thinking about what would have to happen to zeromq to do ssl...
|
[22:43] ahbritto
|
You need to know the identity of a message sender.
|
[22:43] ahbritto
|
So, you'd want to put the ssl finger print in messages.
|
[22:43] ahbritto
|
This is after the stunnel connection.
|
[22:58] ahbritto
|
Is zeromq easily runnable with Apache (poll integration)?
|
[22:59] ahbritto
|
Nevermind, mongrel2 might be fine.
|