[Time] Name | Message |
[06:33] sustrik
|
mikko: any idea how to reproduce the problem?
|
[06:34] sustrik
|
keffo: can you report the problem with more diagnostics (short test program to reproduce, the stack trace)?
|
[06:52] keffo
|
sustrik, I doubt it was related to zmq to be honest, it looked very strange actually..
|
[06:54] keffo
|
class with a thread, kept the context obj in the threadfunc.. if I moved it out as a class member instead, everything worked.. NO clue why
|
[06:54] keffo
|
(threadfuc did a while(true), so the obj should stick around)
|
[07:16] mikko
|
sustrik: it happens within one of the PHP tests, i can try to isolate
|
[07:23] sustrik
|
mikko: that would be great
|
[07:23] sustrik
|
i've had a look at the code and i have no idea how it can happen
|
[07:23] sustrik
|
i need a test program to find out
|
[07:35] iFire
|
sustrik when is pieterh usually around?
|
[07:38] sustrik
|
iFire: randomly :)
|
[07:38] sustrik
|
use mailing list presumably...
|
[08:09] mikko
|
sustrik: http://github.com/mkoppanen/php-zmq/blob/master/tests/007-addremovepoll.phpt
|
[08:10] sustrik
|
mikko: thx
|
[08:12] mikko
|
sustrik: ill try to get c code out of that
|
[08:12] mikko
|
or maybe a backtrace
|
[08:12] mikko
|
let's see
|
[08:13] mikko
|
i posted a backtrace earlier?
|
[08:13] sustrik
|
yes, you did
|
[08:14] mikko
|
been a bit distracted since yesterday
|
[08:14] mikko
|
i was supposed to move to a new flat on sunday and yesterday my agent called that the flat wont be available
|
[08:17] mikko
|
sustrik: no wait
|
[08:17] mikko
|
it might be if i connect to a bound libzmq socket with normal posix socket call
|
[08:24] sustrik
|
mikko: it shouldn't fail this way anyway
|
[08:24] sustrik
|
if you find out that to trigger the problem you need to connect by telnet of somesuch, let me know
|
[08:34] mikko
|
doesnt come out with telnet
|
[08:34] mikko
|
simple tc
|
[08:48] pieterh
|
iFire: hi
|
[08:49] iFire
|
waves
|
[08:49] pieterh
|
what's up?
|
[08:49] iFire
|
pieterh I guess I wanted to know about that authenticated pubsub thing
|
[08:50] pieterh
|
well, i made a small prototype
|
[08:50] pieterh
|
there are two ways, currently, with 0MQ
|
[08:50] pieterh
|
1. use pubsub sockets but encrypt messages, and do key exchange on a secure channel after authentication
|
[08:51] pieterh
|
2. use xrep sockets and explicitly route messages to authenticated subscribers
|
[08:51] pieterh
|
i tested design 2, and it's fairly straight-forward
|
[08:52] iFire
|
don't you have to do authentication and encryption with #1 ?
|
[08:52] iFire
|
of each message
|
[08:52] pieterh
|
yes, that's what I meant to say
|
[08:52] pieterh
|
with 2 you don't need encryption unless you actually want to
|
[08:53] pieterh
|
in short, with pub/sub sockets the security models are (a) open access and (b) encryption of each message
|
[08:54] iFire
|
well isn't there a bandwith problem with #1 ?
|
[08:54] pieterh
|
not really
|
[08:54] pieterh
|
a CPU cost, for sure
|
[08:54] pieterh
|
and key exchange is far from simple
|
[08:54] iFire
|
I mean don't you get all the messages from #1
|
[08:54] iFire
|
ideally you want to be able to do both
|
[08:55] pieterh
|
yes, pubsub would send all messages to all subscribers
|
[08:56] iFire
|
so I guess #2 should be the first thing done, then you can layer #1 onto it
|
[08:56] pieterh
|
they are different approaches
|
[08:56] pieterh
|
but you could layer key exchange and encryption on #2, yes
|
[08:57] pieterh
|
#2 is not real pubsub any more, so no prefix matching etc.
|
[08:57] pieterh
|
you could however do that at the publisher side
|
[08:57] pieterh
|
in your own code
|
[08:58] pieterh
|
so the design is you open two sockets to the server, a REQ and an XREQ
|
[08:58] pieterh
|
you set the same identity on both sockets, before connecting
|
[08:58] pieterh
|
you authenticate on the REQ
|
[08:58] pieterh
|
the server uses two XREP sockets
|
[08:58] pieterh
|
it knows your identity when you authenticate and it adds this to a list of authenticated subscribers
|
[08:59] pieterh
|
then on the second socket it sends updates to all authenticated subscribers
|
[08:59] pieterh
|
finally the subscribers need to heartbeat the server every so often, on that second XREP
|
[08:59] pieterh
|
otherwise they are considered dead and taken off the list
|
[09:00] pieterh
|
you can use this design to do custom pubsub routing too
|
[09:00] pieterh
|
i guess i'll cover this in Ch4
|
[09:01] iFire
|
pieterh we can't change pubsub to send messages to everyone? and add a authentication envelope that is user defined
|
[09:02] iFire
|
to don't send*
|
[09:02] pieterh
|
sure, send a patch :-)
|
[09:02] pieterh
|
it's not that simple unfortunately
|
[09:03] pieterh
|
doing an efficient pubsub is a complex problem
|
[09:03] pieterh
|
there is work in progress to upstream subscriptions, i.e. do the routing at the pub side
|
[09:04] pieterh
|
adding authentication to the socket layer would make it significantly more complex
|
[09:04] pieterh
|
there is an option, perhaps to use TLS as a transport and then do authentication via certificates
|
[09:05] pieterh
|
but these are non-trivial changes
|
[09:05] pieterh
|
so i assume your question was "how do we do authenticated pubsub with today's 0MQ"
|
[09:05] iFire
|
I see
|
[09:06] pieterh
|
does this help?
|
[09:07] iFire
|
I want to read the details of #2
|
[09:08] iFire
|
I mean a guide
|
[09:08] iFire
|
I get the concept
|
[09:09] iFire
|
>.<
|
[09:10] pieterh
|
ok, it'll come in the guide, hopefully
|
[12:27] nisbus
|
hi, is there some logical explanation why my consumers would only receive some messages using pub/sub with hwm =1?
|
[12:32] nisbus
|
I'm sending 10000 messages but my consumer is only receiving 2-4000?
|
[12:43] guido_g
|
because they're dropped?
|
[12:44] nisbus
|
but the listener is active at the time I start sending
|
[12:44] nisbus
|
why would they get dropped?
|
[12:44] guido_g
|
did you read what hwm=1 means?
|
[12:44] nisbus
|
I took it to mean that messages that are sent when no one is listening would get dropped.
|
[12:45] nisbus
|
not that it would drop messages for active listeners
|
[12:45] guido_g
|
go read http://api.zeromq.org/zmq_socket.html
|
[12:47] nisbus
|
what is the default hwm?
|
[12:47] guido_g
|
0 aka none
|
[12:48] nisbus
|
ok, so none will be dropped?
|
[12:49] guido_g
|
but filling your memory, as can be seen on the ml
|
[12:50] nisbus
|
ok, I'm now sending without setting hwm and out of a 1000 messages sent I'm only receiving >500
|
[12:51] nisbus
|
it seems to be dropping about half of my messages
|
[12:51] guido_g
|
given the fact that you didn't tell what you're doing and hwo, it's hard to tell where the problem is
|
[12:52] guido_g
|
pub/sub is not reliable in the way you might think
|
[12:54] nisbus
|
I have a simple Erlang publisher that just sends out an incremented number in a loop.
|
[12:55] nisbus
|
I then have a python/.net listener that just prints out the messages as they arrive.
|
[12:56] guido_g
|
so?
|
[12:56] guido_g
|
<guido_g> pub/sub is not reliable in the way you might think
|
[12:58] nisbus
|
should I use rep/req instead?
|
[12:58] guido_g
|
i don't know?
|
[12:58] guido_g
|
what is the intention of the application you write?
|
[12:59] nisbus
|
I just want every message I send to be received
|
[12:59] nisbus
|
when there is someone connected, otherwise drop it
|
[12:59] guido_g
|
oh, while you're thinking... read the guide, it explains a lot of things http://zguide.zeromq.org/chapter:all
|
[13:45] nisbus
|
ok, I'm checking the reply code of my sends and I'm not getting ok's for all of them.
|
[13:47] guido_g
|
see
|
[13:47] nisbus
|
the other ones don't really tell me anything in the way of what's wrong.
|
[13:48] guido_g
|
one of the darker sides of ømq
|
[13:50] mikko
|
nisbus: in your scenario the subscriber might not consume the messages as fast as the publisher is sending
|
[13:50] mikko
|
that causes some messages to be dropped with hwm 1
|
[13:50] nisbus
|
for sure, it gets all messages when I send just 10 of them but starts failing when msgcount > 100
|
[13:50] nisbus
|
I removed the hwm setting
|
[13:52] nisbus
|
it's a tight loop where both sender and receiver are on the same box.
|
[14:03] nisbus
|
I'm printing out the replycode for messages and when I don't get ok I get "Sent 99 with repcode 98", where 98 is the previous message.
|
[14:28] nisbus
|
I tried sleeping for 1 sec between sending messages and still I'm getting errors on send?
|
[14:29] nisbus
|
just trying to send more than 5 messages starts generating errors
|