[Time] Name | Message |
[00:48] TokyoDan
|
In the Lua binding i don't see an equivalent for zmq_msg_close(). Why?
|
[00:53] TokyoDan
|
In the Lua bindings readme i don't see an equivalent to zmq_msg_close(). Why?
|
[01:00] TokyoDan
|
hello?
|
[01:12] Guthur
|
TokyoDan, It possibly cleans up itself
|
[01:12] Guthur
|
not familiar with Lua or the binding, but in clrzmq2 the GC actually takes care of that
|
[01:12] Guthur
|
there maybe some dispose equivalent
|
[01:13] tokyodan
|
I was thinking the same thing as Lua has garbage collection. But I'm not sure.
|
[01:13] Guthur
|
you could take a look at the source code
|
[01:13] Guthur
|
it would probably be evident pretty quickly
|
[01:13] tokyodan
|
Good idea.
|
[01:14] Guthur
|
sorry i'd have a look as well but I need to get to bed
|
[01:14] tokyodan
|
Good Night.
|
[01:14] tokyodan
|
Thanks again
|
[01:14] Guthur
|
g'night
|
[01:14] Guthur
|
your welcome, happy coding
|
[01:14] Guthur
|
you're*
|
[08:09] mikko
|
good morning
|
[08:10] sustrik
|
morning
|
[08:11] sustrik
|
hi mikko, what about the pgm patch?
|
[08:11] Steve-o
|
afternoon
|
[08:12] Steve-o
|
afternoon
|
[08:12] mikko
|
sustrik: i sent pull request to 201
|
[08:12] mikko
|
ermm
|
[08:12] mikko
|
2-1
|
[08:12] sustrik
|
ok, so it's up to pieter to upstream it to master, right?
|
[08:13] mikko
|
sustrik: i don't know
|
[08:13] sustrik
|
:)
|
[08:13] sustrik
|
never mind
|
[08:13] mikko
|
i can prepare it for master as well
|
[08:13] mikko
|
i think it should've gone to master first
|
[08:13] mikko
|
and then backported to 2-1
|
[08:13] sustrik
|
both directions are possible
|
[08:13] sustrik
|
depends what you are doing
|
[08:14] sustrik
|
with new development i would assume you are right
|
[08:14] sustrik
|
with bugfixes the direction is opposite sometimes
|
[08:14] sustrik
|
ie. stable version is fixed, then the fix is upstreamed to master (dev)
|
[08:17] jond
|
morning all
|
[08:17] jond
|
sutrik
|
[08:18] jond
|
sustrik: let me know what you think on that xrep hwm, pollout stuff and i'll fix if necessary
|
[08:21] sustrik
|
jond: it's a complex issue, i'll produce a lengthy email and send it to the ml later on
|
[08:22] jond
|
sustrik: yes i have been thinking about it and it's not easy!
|
[08:24] jond
|
i've gotta go now, can't see irc in the day so catch up later on list
|
[08:39] pieterh
|
hi folks
|
[08:40] pieterh
|
hi mikko
|
[10:39] yrashk
|
I just realized why saleyn never replied to my email about erlzmq/erlzmq2 :-(
|
[10:44] sustrik
|
you can try to ping him once more...
|
[10:46] sustrik
|
but iirc the problem with erlang bindings is that each of the three has useful pieces of functionality that others don't have, but nobody is willing to work on merging the three featuresets into a single codebase
|
[10:48] yrashk
|
sustrik: I don't think he'll be able to
|
[10:49] yrashk
|
for the next 10 years or so
|
[10:49] sustrik
|
oh my, is that definitive?
|
[10:49] yrashk
|
sustrik: it's impossible to merge erlzmq & erlzmq2, they use different binding strategies
|
[10:49] yrashk
|
sustrik: pretty much
|
[10:49] sustrik
|
:(
|
[10:50] yrashk
|
he has been jailed since mar 2 or so
|
[10:50] sustrik
|
:(((
|
[10:50] yrashk
|
"On March 2, 2011, it was reported that Mr. Aleynikov was incarcerated on request of the government, as he was judged to be more of a flight risk after separating from his wife and growing distant from his kids. [5]
|
[10:50] yrashk
|
"
|
[10:50] yrashk
|
also his wife filed a divorce
|
[10:50] yrashk
|
not a good time for Sergey
|
[10:50] yrashk
|
He was convicted in U.S. Federal court in December 2010 of the two counts of theft of stolen property and transportation of stolen property, and faces up to 10 years in prison. Sentencing is set for March 18, 2011. [4]
|
[10:50] yrashk
|
which is like, today
|
[10:51] sustrik
|
oh well
|
[10:51] sustrik
|
so it's up to the rest of erlzmq community to drive the project forward
|
[10:51] yrashk
|
yup
|
[10:52] yrashk
|
but again
|
[10:52] yrashk
|
erlzmq and erlzq2 can't be merged by any means
|
[10:52] yrashk
|
they are way too different
|
[10:52] sustrik
|
yes, the codebase itself cannot be merged
|
[10:52] yrashk
|
erlzmq2 won't work with older Erlang VMs
|
[10:52] yrashk
|
as it uses a new binding technique
|
[10:52] sustrik
|
what could be merged is functionality
|
[10:52] yrashk
|
I thiknk they are very close functionally
|
[10:53] sustrik
|
close = equivalent?
|
[10:53] yrashk
|
we don't have an active mode (like saleyn had for the main erlzmq)
|
[10:53] yrashk
|
APIs is not absolutely equivalent, no
|
[10:53] yrashk
|
module names are different, app name is different
|
[10:53] sustrik
|
the problem is that as long as there's functionality missing, those using one binding
|
[10:53] yrashk
|
some calls might look a little bit different
|
[10:53] sustrik
|
cannot switch to anotherone
|
[10:53] yrashk
|
I don't think we have any functionality missing, really
|
[10:53] yrashk
|
active mode was controversial anyway
|
[10:54] yrashk
|
I think jugg opposed it a lot
|
[10:54] sustrik
|
doesn't it have to do with pushback?
|
[10:54] sustrik
|
won't passive mode just exhaust all the memory
|
[10:54] sustrik
|
if sender is faster than receiver?
|
[10:54] yrashk
|
well
|
[10:55] yrashk
|
passive mode is pretty much reliant on the way 0mq works
|
[10:55] yrashk
|
recv/poll
|
[10:55] yrashk
|
there are certain issues why we haven't implemented active mode yet
|
[10:56] yrashk
|
we could have done a workaround.. but still not sure if it is worth it
|
[10:56] yrashk
|
it probably is worth a try
|
[10:56] sustrik
|
is there any other way to implement pushback?
|
[10:56] yrashk
|
what od you mean by a pushback?
|
[10:56] sustrik
|
meaning that when the receiving end is not able to keep up with the sender
|
[10:57] sustrik
|
the sender gets blocked
|
[10:57] sustrik
|
until the receiver catches up
|
[10:57] yrashk
|
not sure how to do anything about this
|
[10:57] sustrik
|
i think that was what active mode was for
|
[10:57] yrashk
|
basically what we do is we just pass messages back to erlang
|
[10:57] yrashk
|
either directly or as actual essages
|
[10:58] yrashk
|
active mode is just something that does recv for you, basically
|
[10:58] yrashk
|
so I am not sure
|
[10:58] sustrik
|
the problem is that erlang stores the messages in erlang mailbox
|
[10:58] yrashk
|
yep
|
[10:58] sustrik
|
which grows indefinitely
|
[10:58] yrashk
|
but we don't send them unless you request
|
[10:59] yrashk
|
active is what sends them regardless of whether you can keep up
|
[10:59] yrashk
|
our recv() algo is simple
|
[10:59] yrashk
|
1. recv noblock
|
[10:59] yrashk
|
2. if there was something, return
|
[10:59] yrashk
|
3. if not, let the reader thread poll it
|
[10:59] sustrik
|
yes, that's ok
|
[10:59] yrashk
|
4. when something appears there, send a message to the original requesting process
|
[10:59] sustrik
|
what about the sender side?
|
[11:00] yrashk
|
sender just does send() and that's it
|
[11:00] sustrik
|
it blocks the thread?
|
[11:00] yrashk
|
you mean the scheduler?
|
[11:00] sustrik
|
then you can end up with deadlocked erlang VM
|
[11:00] sustrik
|
i mean physicall thread
|
[11:00] sustrik
|
from erlang's thread pool
|
[11:01] yrashk
|
if you don't pass noblock, yes it will block the scheduler
|
[11:01] yrashk
|
we don't use erlang's thead pool
|
[11:01] yrashk
|
NIF calls block the scheduler
|
[11:01] sustrik
|
i mean, erlang VM has to execute in an OS thread(s)
|
[11:01] yrashk
|
so if zmq_send blocks, the VM is blocked (well, one cpu core is blocked, to be precise)
|
[11:01] sustrik
|
exactly
|
[11:02] yrashk
|
yes, erlang vm usually has # of schedulers equal to number of cores
|
[11:02] sustrik
|
so, if there's 1 CPU
|
[11:02] yrashk
|
and 1 core?
|
[11:02] sustrik
|
and you call invoke 0mq in a blocking way
|
[11:02] sustrik
|
the whole OS process gets bloecked
|
[11:02] yrashk
|
it will fuck up, yes
|
[11:02] yrashk
|
we were thinking about non-blocking send()
|
[11:02] yrashk
|
just haven't done it yet
|
[11:03] yrashk
|
there's nothing that prevents us from doing it
|
[11:03] sustrik
|
that would mean busy-looping and 100% CPU usage, no?
|
[11:03] yrashk
|
nah
|
[11:04] yrashk
|
we poll on internal inproc sockets
|
[11:04] yrashk
|
for commands
|
[11:04] yrashk
|
in the receiver thread
|
[11:04] yrashk
|
which does async recv
|
[11:04] yrashk
|
so we can do the same thing for send, pretty much
|
[11:04] yrashk
|
may be even in the same thread
|
[11:04] sustrik
|
aha
|
[11:04] yrashk
|
no biggie
|
[11:05] sustrik
|
that way the polling doesn't block the erlang scheduler, right?
|
[11:05] yrashk
|
will take probabyl an hour or two to implement
|
[11:05] yrashk
|
it does not
|
[11:05] yrashk
|
it never does
|
[11:05] sustrik
|
nice
|
[11:05] yrashk
|
it's always in that second thread
|
[11:05] yrashk
|
we have one thread per context
|
[11:05] yrashk
|
which I guess is good enough
|
[11:05] yrashk
|
originally I had a thread per socket
|
[11:05] yrashk
|
and that was a tad nasty
|
[11:05] yrashk
|
but evax refactored this nicely
|
[11:06] sustrik
|
so erlang allows you to say "please invoke this code when this file descriptor becomes active"?
|
[11:06] sustrik
|
and erlang scheduler will do that for you
|
[11:06] yrashk
|
no, it's all manual
|
[11:06] yrashk
|
there's some FD stuff in there, but not for NIFs afaik
|
[11:06] yrashk
|
basically, when we poll in that thread
|
[11:07] sustrik
|
the question is whether waiting for FD blocks the OS thread or not
|
[11:07] yrashk
|
you mean zmq_poll?
|
[11:07] sustrik
|
are you using zmq_poll?
|
[11:07] yrashk
|
yes
|
[11:07] sustrik
|
zmq_poll is blocking
|
[11:07] yrashk
|
yes, and that's fine
|
[11:07] yrashk
|
because we're in a separate thread
|
[11:07] yrashk
|
which erlang has no idea about
|
[11:07] sustrik
|
outside of erlang VM?
|
[11:07] sustrik
|
i see
|
[11:08] yrashk
|
well, it's in the same process
|
[11:08] yrashk
|
but it's separate from the scheduler
|
[11:08] yrashk
|
so all NIF calls are generally very short
|
[11:08] sustrik
|
yep, that could work
|
[11:08] yrashk
|
except for send() which might theoretically block
|
[11:08] yrashk
|
but that's something we can definitely fix
|
[11:08] yrashk
|
I just never saw zmq_send() blocking
|
[11:08] sustrik
|
but can be done in same way as recv()
|
[11:08] yrashk
|
yes
|
[11:08] yrashk
|
absolutely
|
[11:09] sustrik
|
you get that for high loads
|
[11:09] yrashk
|
like I said, it's an easy change
|
[11:09] yrashk
|
since we have worked out the approach
|
[11:09] sustrik
|
would be good to get that in
|
[11:09] sustrik
|
it's essential for some applications
|
[11:09] yrashk
|
I see
|
[11:09] sustrik
|
such as rabbitmq/0mq bridge
|
[11:09] yrashk
|
I am adding a github issue right now
|
[11:10] yrashk
|
https://github.com/zeromq/erlzmq2/issues/1
|
[11:10] sustrik
|
:)
|
[11:10] sustrik
|
what about the third erlzmq binding?
|
[11:10] sustrik
|
any additional functionality there?
|
[11:11] yrashk
|
cstar/jugg's?
|
[11:11] sustrik
|
i guess
|
[11:11] yrashk
|
I don't remember any special functionality there
|
[11:11] yrashk
|
it looked more polished than the official one
|
[11:11] yrashk
|
to my eye
|
[11:11] yrashk
|
so I guess it's good for older erlangs
|
[11:12] sustrik
|
aha, that's the catch, right?
|
[11:12] yrashk
|
or if you have any reasons to use port driver instead of NIF
|
[11:12] sustrik
|
old arlangs don't have NIF
|
[11:12] yrashk
|
yeah they don't
|
[11:12] sustrik
|
well, if you believe there's all the functionality in erlzmq2
|
[11:12] yrashk
|
and some of those have (like last R13Bs and early R14As) but the API was changing a lot
|
[11:12] yrashk
|
so we're pretty much >=R14B01 or so
|
[11:13] sustrik
|
let's discuss making it a primary erlang binding on the mailing list
|
[11:13] yrashk
|
also, jugg was never able to clock performance comparable to erlzmq2
|
[11:13] sustrik
|
yes, i recall that
|
[11:13] yrashk
|
lets do that
|
[11:13] yrashk
|
we have like 1.3mln/sec for 1bytes
|
[11:13] yrashk
|
which is not bad
|
[11:13] yrashk
|
not great, but decent
|
[11:13] sustrik
|
it's very nice
|
[11:13] yrashk
|
(on my mac pro)
|
[11:13] sustrik
|
comparable to other non-C languages
|
[11:14] yrashk
|
c-to-c on my mac pro is like twice as fast
|
[11:14] yrashk
|
2.5+mln/sec
|
[11:14] yrashk
|
but we weren't able to squeeze out more yet
|
[11:14] yrashk
|
and I am not sure we can
|
[11:14] sustrik
|
you have to keep in mind that erlang runtime does a lot of stuff
|
[11:15] sustrik
|
scheduling etc.
|
[11:15] sustrik
|
which takes some time itself
|
[11:15] yrashk
|
of course
|
[11:15] yrashk
|
I know erlang fairly well
|
[11:15] yrashk
|
or so I think
|
[11:15] sustrik
|
with throughput over 1M/sec
|
[11:15] sustrik
|
you are getting to nanosecond ranges
|
[11:15] yrashk
|
well that's for 1bytes
|
[11:15] yrashk
|
1Kb is like 300K/sec
|
[11:16] yrashk
|
which is still okay I guess
|
[11:16] sustrik
|
100% bandwidth utilisation i would guess
|
[11:16] sustrik
|
1GbE?
|
[11:16] yrashk
|
it's all localhost tests :)
|
[11:16] yrashk
|
both for c-to-c and erlang-to-erlang
|
[11:17] sustrik
|
ack
|
[11:17] yrashk
|
for comparison
|
[11:17] yrashk
|
tcp transport
|
[11:17] sustrik
|
the larger the message, the less the erlang overhead
|
[11:17] yrashk
|
ipc was about the same
|
[11:17] sustrik
|
anyway, 1.3M is great
|
[11:19] yrashk
|
anyway
|
[11:19] yrashk
|
lets discuss binding issue on the mailing list?
|
[11:19] yrashk
|
I am subscribed (digest mode)
|
[11:19] yrashk
|
will be back in about 30 mins
|
[11:21] sustrik
|
yrashk: yes, send the proposal to the mailing listr
|
[11:27] Guthur
|
talking of bindings, sustrik do you think it would be wise to discuss marking clrzmq as deprecated and that users should consider either the interop binding or clrzmq2
|
[11:27] Guthur
|
the rational being that the original binding isn't really actively developed
|
[11:28] sustrik
|
it's a community project, discuss it on the mailing list
|
[11:28] Guthur
|
sure
|
[11:28] sustrik
|
and if there's no opposition, let's do it
|
[11:28] Guthur
|
just interested in what you think
|
[11:28] sustrik
|
i would rather have actively maintained bindings as "official" :)
|
[11:28] Guthur
|
ok, I'm raise it on the list later today
|
[11:48] pieterh
|
sustrik: ping
|
[11:48] sustrik
|
pong
|
[11:48] pieterh
|
random question about pub/sub...
|
[11:48] pieterh
|
if I send a two-frame message, first key then value, I expect to receive two frames, right?
|
[11:49] sustrik
|
yes
|
[11:49] pieterh
|
ok, I have a test case where it appears to sometimes fail at connection startup
|
[11:49] pieterh
|
loses first frame, sometimes
|
[11:49] pieterh
|
i
|
[11:49] sustrik
|
ok
|
[11:49] pieterh
|
I'll post the server and client, hang on...
|
[11:50] Guthur
|
pieterh_: with your lazy pirate patterns did you notice that reconnecting after CPU overload rarely happened?
|
[11:51] Guthur
|
I've noticed this behaviour, even with the C examples
|
[11:51] pieterh
|
Guthur: what do you mean?
|
[11:51] pieterh
|
the client doesn't reconnect?
|
[11:51] Guthur
|
yeah, it tries and thinks the server is not there
|
[11:51] pieterh
|
normal, because its timeouts are quite aggressive, and simulation of CPU overload is quite a long pause afair
|
[11:51] Guthur
|
if I remove the 1sec sleep it connects
|
[11:52] pieterh
|
try making the client more patient...
|
[11:52] Guthur
|
1sec (heavy work)
|
[11:52] pieterh
|
i'll retest, hang on a sec...
|
[11:53] pieterh
|
sustrik: so, https://gist.github.com/875932
|
[11:53] pieterh
|
start the server, then the client
|
[11:53] sustrik
|
ok, will give it a look
|
[11:53] pieterh
|
thanks, this is hitting one of my examples and I can't see my fault
|
[11:53] pieterh
|
so made a simplistic case, it still hits...
|
[11:58] pieterh
|
Guthur: it's just because the client is too impatient
|
[11:59] Guthur
|
pieterh_: ok, cheers for testing
|
[11:59] pieterh
|
Guthur: I'll change the 'CPU overload' delay to 2 seconds instead of 5
|
[11:59] pieterh
|
this lets the client recover
|
[12:00] Guthur
|
sounds good
|
[12:01] yrashk
|
sustrik: what exactly should be in that proposal?
|
[12:01] Guthur
|
it had me scratching my head for a while yesterday, for a moment I thought the Mono runtime was just insanely slow
|
[12:01] Guthur
|
but then seen the same in C
|
[12:02] pieterh
|
yeah, it's good to be able to compare the different languages
|
[12:02] sustrik
|
yrashk: "i propose making erlzmq2 an official erlang binding while keeping erlzmq as a binding for old versions of erlang"
|
[12:02] sustrik
|
then list your arguments
|
[12:02] yrashk
|
ok
|
[12:02] sustrik
|
and ask people to comment
|
[12:02] pieterh
|
sustrik: I think that was already agreed...
|
[12:02] yrashk
|
can I refer to you as it was your idea?
|
[12:02] pieterh
|
so erlzmq kind of became v1
|
[12:03] sustrik
|
did it?
|
[12:03] yrashk
|
wel
|
[12:03] yrashk
|
well
|
[12:03] pieterh
|
yrashk: I missed something, why are you asking for people to comment on this?
|
[12:03] yrashk
|
ont the wiki, eys
|
[12:03] yrashk
|
pieterh_: I am not, sustrik does
|
[12:03] pieterh
|
sustrik: we already afaik got agreement from the erlzmq authors to make v2
|
[12:03] yrashk
|
pieterh_: just one of them
|
[12:04] pieterh
|
yrashk: ah... did you ask the others?
|
[12:04] yrashk
|
pieterh_: saleyn is unable to answer for some sad reasons
|
[12:04] pieterh
|
ok
|
[12:04] sustrik
|
if it was discussed already, then there's no point in discussing it anew
|
[12:04] pieterh
|
indeed
|
[12:04] yrashk
|
ok
|
[12:04] sustrik
|
i just missed the original discussion
|
[12:04] yrashk
|
ya it was discussed here
|
[12:04] pieterh
|
it's already an official repository, and the name was made coherent, the wiki correct, etc.
|
[12:05] sustrik
|
then there's no problem
|
[12:05] yrashk
|
sweet
|
[12:05] yrashk
|
less work for me then
|
[12:05] yrashk
|
we'll then improve send() some time later
|
[12:05] pieterh
|
given that it's much faster and sticks to the same API, there was no real argument...
|
[12:05] sustrik
|
i see
|
[12:05] yrashk
|
to make it behave better on blocking sends
|
[12:05] evax
|
yrashk, sustrik: doing it now
|
[12:05] yrashk
|
evax: oh the issue #1?
|
[12:05] evax
|
yes
|
[12:06] yrashk
|
evax: I bet it should be easy to replicate the same method as we do with recv
|
[12:06] yrashk
|
may be even the same thread
|
[12:06] evax
|
same thread, yes
|
[12:06] yrashk
|
sweet
|
[12:06] yrashk
|
should be fairly trivial then
|
[12:06] yrashk
|
just adding an enum flag to recv structure
|
[12:07] yrashk
|
and renaming recv structure to something like command or someting
|
[12:07] yrashk
|
also, evax
|
[12:07] evax
|
yes, along those lines
|
[12:07] yrashk
|
do you plan to do exactly the same scheme?
|
[12:07] yrashk
|
i.e.
|
[12:07] yrashk
|
recv noblock first
|
[12:07] evax
|
sure
|
[12:07] yrashk
|
err
|
[12:07] yrashk
|
send*
|
[12:07] yrashk
|
if it says EAGAIN
|
[12:07] yrashk
|
then pool it
|
[12:07] evax
|
shouldn't I ?
|
[12:08] yrashk
|
so that we don't pool when it is unnecessary
|
[12:08] yrashk
|
I think it's a good approach
|
[12:08] yrashk
|
zmq_send(.., ZMQ_NOBLOCK) -> if EGAGAIN, pool
|
[12:08] yrashk
|
otherwise, be done with it
|
[12:08] yrashk
|
EAGAIN*
|
[12:08] yrashk
|
that looks really very much the same way recv does
|
[12:11] evax
|
except we have to store the msg somewhere
|
[12:12] yrashk
|
in the same struct
|
[12:12] yrashk
|
recv, but renamed
|
[12:12] yrashk
|
and extended accordingly
|
[12:12] evax
|
this goes over the ipc socket, so for big messages its ugly
|
[12:13] yrashk
|
just copy the zms_msg
|
[12:13] yrashk
|
it is
|
[12:13] yrashk
|
but its not ipc
|
[12:13] yrashk
|
inproc
|
[12:13] evax
|
yeah
|
[12:13] yrashk
|
we just pass a pointer
|
[12:13] yrashk
|
how about that?
|
[12:13] yrashk
|
and then zmq_msg_close it when done sending it in a loop
|
[12:14] yrashk
|
theres more important problem
|
[12:14] yrashk
|
in that thread we poll
|
[12:14] yrashk
|
but if we do the blocking send there
|
[12:15] yrashk
|
then receivers will get blocked untill the send is over
|
[12:15] yrashk
|
mhm
|
[12:15] evax
|
reading zmq.h shouldn't be a problem to pass the msg in, ideed
|
[12:15] yrashk
|
yeah, so pointer in that struct for msg is cool
|
[12:15] yrashk
|
but this problem above bothers me
|
[12:15] yrashk
|
not nice
|
[12:16] yrashk
|
also separate sending thread isnt solving this completely
|
[12:17] yrashk
|
it will help us from blocking poller
|
[12:17] yrashk
|
but it will not b able to send more than one thing at a time
|
[12:17] yrashk
|
since sending is blocked
|
[12:17] yrashk
|
if only we had a reverse poll
|
[12:17] yrashk
|
wait
|
[12:18] yrashk
|
dont we have that kind of event in poll?
|
[12:18] evax
|
it has to be in the same thread
|
[12:18] yrashk
|
not sure how it will help tho
|
[12:18] evax
|
and I'm not sure sending and recving are allowed at the same time
|
[12:19] evax
|
if one must not use a socket in multiple threads
|
[12:19] yrashk
|
hey
|
[12:19] yrashk
|
how about using zmq_pollout event?
|
[12:20] evax
|
sure
|
[12:20] yrashk
|
it guarantees you can send without blocking
|
[12:20] yrashk
|
so we can embed this into our poller
|
[12:21] yrashk
|
this way we wont be blocking the poller ever
|
[12:21] yrashk
|
i think it is a sound idea
|
[12:21] yrashk
|
sustrik, wdyt? will it work fine? any precautions?
|
[12:23] yrashk
|
i really like it
|
[12:42] sustrik
|
yes, that's what POLLOUT is for
|
[12:43] yrashk
|
sweet
|
[12:43] yrashk
|
so hopefully we'll have this non-blocking very soon then
|
[12:43] yrashk
|
as it will not change our poller much
|
[12:45] Steve-o
|
mikko: looks like all the platforms are good now
|
[13:06] pieterh
|
sustrik: I think your email is inaccurate in some key ways
|
[13:06] pieterh
|
most critically, you are claiming a purpose for XREP that was never documented before
|
[13:06] sustrik
|
feel free to discuss
|
[13:07] sustrik
|
that's why i sent the email out
|
[13:07] pieterh
|
as a user, and the developer of many devices, the notion that you would break this, is... shocking
|
[13:07] sustrik
|
that's why it has to be solved
|
[13:07] pieterh
|
it suggests (a) you don't know wtf you were doing, (b) you don't care about your users, (c) you don't understand what they are doing
|
[13:07] pieterh
|
none of this is true, but it's what your email suggests
|
[13:07] sustrik
|
i would start with ZMQ)ROUTER socket type
|
[13:08] sustrik
|
shrug, it has to be solved somehow
|
[13:08] pieterh
|
it's unavoidable that people use this as a ROUTER
|
[13:08] pieterh
|
so, embrace that and make it work nicely
|
[13:08] sustrik
|
feel free to propose a solution
|
[13:08] pieterh
|
I've done that many times
|
[13:08] sustrik
|
ZMQ_ROUTER, right?
|
[13:08] pieterh
|
call it by the right name, make it work a little better
|
[13:08] pieterh
|
yes
|
[13:08] pieterh
|
of course
|
[13:08] sustrik
|
that's option 2 i think
|
[13:08] pieterh
|
you will see the overwhelming consensus on the list
|
[13:09] sustrik
|
sure
|
[13:09] pieterh
|
you did read that thread, right?
|
[13:09] sustrik
|
np
|
[13:09] sustrik
|
yes
|
[13:09] pieterh
|
I like your solution of a verbose compile option
|
[13:09] sustrik
|
i am just explaining the internals
|
[13:09] sustrik
|
opt.2 is probably the best
|
[13:09] pieterh
|
well, you're saying you might make a different implementation of "return to sender"
|
[13:10] sustrik
|
definitely
|
[13:10] pieterh
|
which (and I give you the raw emotion) is ... shokcing
|
[13:10] pieterh
|
shocking
|
[13:10] sustrik
|
that's the XREP should be separated from REOUTER
|
[13:10] sustrik
|
ROUTER*
|
[13:10] sustrik
|
they are two distinct things
|
[13:10] pieterh
|
you have a better idea for the REP-REQ chain?
|
[13:10] sustrik
|
not at the moment
|
[13:10] sustrik
|
but it's conceptually different from router
|
[13:10] pieterh
|
precisely, and afaics it's impossible without creating stateful fabric
|
[13:11] sustrik
|
say the dropping of unroutable messages ->
|
[13:11] pieterh
|
conceptually it's the same
|
[13:11] pieterh
|
it's a use case
|
[13:11] sustrik
|
ROUTER should not drop
|
[13:11] sustrik
|
XREP should
|
[13:11] pieterh
|
yes it should
|
[13:11] pieterh
|
ROUTER should drop BUT it should be able to warn you
|
[13:11] pieterh
|
there is nothing useful you can do with an unroutable message but since it often suggests a programming error, you need visibility
|
[13:11] pieterh
|
that's all
|
[13:11] sustrik
|
sure
|
[13:12] pieterh
|
I literally go into src/xrep.cpp and add a printf
|
[13:12] pieterh
|
so I can debug a complex router
|
[13:12] pieterh
|
doing a req-rep chain is just one use case for routing
|
[13:12] pieterh
|
it's not a core pattern
|
[13:12] sustrik
|
it is
|
[13:13] pieterh
|
not a *separate* core pattern
|
[13:13] pieterh
|
afaics
|
[13:14] pieterh
|
tbh the envelope concept is ugly as heck but I can't see anything simpler or better
|
[13:14] sustrik
|
the one described in the email is nice
|
[13:14] sustrik
|
but works on only for req/rep
|
[13:14] sustrik
|
not for router
|
[13:15] sustrik
|
and it has some scaling issues
|
[13:15] sustrik
|
so it's not a real alternative, i think
|
[13:16] pieterh
|
XPUB/XSUB are hop-by-hop?
|
[13:16] sustrik
|
yes
|
[13:16] pieterh
|
the fact you implement PUB on XPUB, REQ on XREQ, is irrelevant to the user
|
[13:17] pieterh
|
it's vital to you, irrelevant to me
|
[13:17] pieterh
|
in fact, worse than irrelevant, it's negative data
|
[13:17] pieterh
|
you basically ask me as user to think about your implementation design, which breaks my brain
|
[13:18] pieterh
|
"The end-to-end type is always layered on top of the corresponding X socket"
|
[13:18] sustrik
|
it works like IP and TCP
|
[13:18] pieterh
|
that's rubbish, sorry
|
[13:18] pieterh
|
I don't know IP
|
[13:18] sustrik
|
a TCP user doesn't have to be aware of IP
|
[13:18] sustrik
|
but it's there
|
[13:18] pieterh
|
sorry, it does not work for me, not at all
|
[13:18] pieterh
|
what I want are a set of socket types with clear names and clear semantics
|
[13:18] Steve-o
|
+1
|
[13:18] pieterh
|
I don't want some layering and abstraction I need to learn first
|
[13:18] sustrik
|
that's the option 2
|
[13:18] sustrik
|
ZMQ_ROUTER
|
[13:19] pieterh
|
do not try to sell me your abstractions, it's what AMQP does and it's bogus
|
[13:19] sustrik
|
what do you want exactly then?
|
[13:19] sustrik
|
i am not following
|
[13:19] pieterh
|
I want clear socket names
|
[13:19] sustrik
|
ZMQ_ROUTER
|
[13:19] pieterh
|
clear semantics that are documented up-front
|
[13:19] pieterh
|
yes
|
[13:19] pieterh
|
this is, sadly, reverse engineering
|
[13:19] sustrik
|
what's the problem then?
|
[13:19] pieterh
|
it's me trying to make names that fit your semantics
|
[13:20] pieterh
|
there's no problem so long as you don't come back with abstractions that hurt my brain
|
[13:20] pieterh
|
:-)
|
[13:20] pieterh
|
it is already hard enough
|
[13:20] sustrik
|
that's an abstraction on which 0mq is working from very beginning
|
[13:20] sustrik
|
just ignore it
|
[13:20] sustrik
|
as a user you don't have to care
|
[13:20] pieterh
|
do you understand that your semantics are counter-productive for mine?
|
[13:21] pieterh
|
we have a conflict, as user vs. designer?
|
[13:21] sustrik
|
i don't think so
|
[13:21] pieterh
|
it's been that way so far
|
[13:21] pieterh
|
e.g. The end-to-end type is always layered on top of the corresponding X socket
|
[13:21] sustrik
|
yes
|
[13:22] pieterh
|
that is toxic for me as user
|
[13:22] sustrik
|
you don't have to care
|
[13:22] sustrik
|
use just the non-X types
|
[13:22] sustrik
|
that's what proposition 2 is about
|
[13:22] sustrik
|
people want to use routing
|
[13:22] sustrik
|
so let's make routing a first-class non-X socket
|
[13:22] sustrik
|
ZMQ_ROUTER
|
[13:22] pieterh
|
well, you can make internal socket types that are undocumented
|
[13:23] pieterh
|
but ZMQ_ROUTER and ZMQ_DEALER cannot be internal and undocumented
|
[13:23] pieterh
|
since they cover 40%, as I said, of real application use cases
|
[13:23] sustrik
|
X-sockets must be documented because they are used to make devices
|
[13:23] pieterh
|
devices are apps
|
[13:23] sustrik
|
i.e. intermdiate nodes with no end-to-end functionalitty
|
[13:23] pieterh
|
X-sockets are your terminology
|
[13:23] pieterh
|
I'm against that terminology
|
[13:24] sustrik
|
what would you want instead?
|
[13:24] pieterh
|
especially when you explain it
|
[13:24] sustrik
|
i admitt the X- thing is stupid
|
[13:24] pieterh
|
just a clear set of sockets that are properly documented and match real requirements
|
[13:24] pieterh
|
it's not hard
|
[13:24] pieterh
|
design from the use case backwards
|
[13:24] sustrik
|
haven't though of anything better though
|
[13:24] pieterh
|
not from the implementation forwards
|
[13:24] pieterh
|
I don't *care* how you implement stuff
|
[13:24] pieterh
|
what I do care about is the clarity of documentation as a contract
|
[13:25] pieterh
|
clear names
|
[13:25] sustrik
|
the problem is that X- sockets have to be exposed because of device implementers
|
[13:25] pieterh
|
clear semantics that directly speak to user requirements
|
[13:25] pieterh
|
?
|
[13:25] sustrik
|
so they can't be hidden
|
[13:25] pieterh
|
ZMQ_ROUTER is not a problem
|
[13:25] pieterh
|
sigh
|
[13:25] pieterh
|
devices are apps
|
[13:25] sustrik
|
no, the problem is layering
|
[13:25] pieterh
|
that's your problem, not mine
|
[13:25] pieterh
|
that's an implementation detail
|
[13:25] pieterh
|
look... just make small steps here
|
[13:26] sustrik
|
ok, so just ignore the X types
|
[13:26] pieterh
|
rename XREP and XREQ to ROUTER and DEALER
|
[13:26] pieterh
|
and stop using weird names that make no sense
|
[13:26] sustrik
|
it's mine problem, not yours
|
[13:26] pieterh
|
if you want to use undocumented internal socket types, fine
|
[13:26] pieterh
|
but they don't exist for me
|
[13:26] sustrik
|
they must be documented because of devices
|
[13:26] sustrik
|
there's no way to create a device with REQ/REP
|
[13:27] sustrik
|
it would blockl
|
[13:27] pieterh
|
then they are not internal
|
[13:27] pieterh
|
well, I do know that
|
[13:27] pieterh
|
when I say "undocumented internal socket types" I do not mean ROUTER/DEALER
|
[13:27] pieterh
|
obviously
|
[13:27] pieterh
|
I mean e.g. XPUB/XSUB
|
[13:27] sustrik
|
so my proposition is let distiquish XREP/XREQ
|
[13:27] sustrik
|
from ROUTER/DEALER
|
[13:28] sustrik
|
you can then use ROUTER/DEALER
|
[13:28] pieterh
|
why?
|
[13:28] pieterh
|
why keep the old names at all?
|
[13:28] sustrik
|
because it's a different messaging pattern
|
[13:28] pieterh
|
sigh
|
[13:28] pieterh
|
this is just bogus
|
[13:28] sustrik
|
it SMTP-like messaging
|
[13:28] pieterh
|
it's not a different pattern
|
[13:28] pieterh
|
it's just one use case
|
[13:28] sustrik
|
specify an address, the message will get there
|
[13:28] pieterh
|
you have said it, there's no better way to implement it
|
[13:28] sustrik
|
while req/rep is:
|
[13:28] pieterh
|
"message will get there" is not a socket semantic
|
[13:29] sustrik
|
specify the name of the service, you task will be executed
|
[13:29] pieterh
|
that is Majordomo
|
[13:29] pieterh
|
have you actually read the Guide?
|
[13:29] sustrik
|
most of it
|
[13:29] pieterh
|
you should perhaps see what people do with your work
|
[13:29] sustrik
|
anyway, let's move this discussion on the mailing list
|
[13:29] pieterh
|
it's all based on real questions on the list
|
[13:29] pieterh
|
SOA is not a socket semantic
|
[13:30] pieterh
|
my advice, honestly, is to make small, careful evolutionary steps
|
[13:30] pieterh
|
do not make radical changes to things that took years to learn
|
[13:30] sustrik
|
#define ZMQ_ROUTER ZMQ_XREP
|
[13:30] pieterh
|
people will not like you for it
|
[13:30] pieterh
|
sure
|
[13:30] sustrik
|
that's as small as it gets
|
[13:30] pieterh
|
I've done that in 2.1
|
[13:30] pieterh
|
precisely that
|
[13:30] sustrik
|
i'll do that in trunk
|
[13:30] pieterh
|
it's sufficient for me
|
[13:30] sustrik
|
but i would like it to be discussed first
|
[13:31] pieterh
|
the naming has been discussed several times
|
[13:31] sustrik
|
i assume that would be the preferred option
|
[13:31] pieterh
|
it's not the same issue as semantic changes
|
[13:31] sustrik
|
but i don't want to do it without discussion on the list
|
[13:31] pieterh
|
you are mixing two issues and making the discussion harder
|
[13:31] pieterh
|
the naming *has* been discussed several times, with clear consensus
|
[13:31] pieterh
|
shall I point you to the thread?
|
[13:31] sustrik
|
explain your point of view on the list then
|
[13:31] sustrik
|
we are a community
|
[13:31] pieterh
|
I've done that, of course
|
[13:31] pieterh
|
we're also on IRC
|
[13:32] pieterh
|
this is logged
|
[13:32] sustrik
|
yep, but lot of people are not here at the moment
|
[13:32] pieterh
|
the list is also fairly passive
|
[13:32] pieterh
|
nonetheless, I have asked about renaming the sockets and there was *clear* consensus
|
[13:32] pieterh
|
please do respect that
|
[13:32] pieterh
|
we are a community, as you say
|
[13:32] sustrik
|
it's polite to allow even those in other timezones to discuss
|
[13:33] pieterh
|
also, reopening a discussion because you weren't there, or don't like the outcome is unpolite
|
[13:33] sustrik
|
yep, i am for ZMQ_ROUTER options as well
|
[13:33] pieterh
|
so, we make that
|
[13:33] pieterh
|
and we consider the semantics as a separate issue
|
[13:33] sustrik
|
it's supplying the missing pieces of information
|
[13:33] pieterh
|
i'm always in favor of better semantics but they must be carefully worked out of use cases
|
[13:33] sustrik
|
yup
|
[13:33] pieterh
|
abstract "it's like XYZ" discussions are pointless IMO
|
[13:33] pieterh
|
we are not like XYZ
|
[13:34] sustrik
|
sure, say so
|
[13:34] pieterh
|
XYZ is like us, done badly
|
[13:34] pieterh
|
period
|
[13:34] pieterh
|
what, again?
|
[13:34] pieterh
|
:-)
|
[13:34] sustrik
|
:)
|
[13:34] pieterh
|
btw, for IPC, what do you think about making a shared memory transport?
|
[13:35] pieterh
|
one can make a portable shmem layer, then use that
|
[13:35] sustrik
|
there's an API problem with that
|
[13:35] pieterh
|
all we need is a portable shmem api people can implement
|
[13:35] sustrik
|
when you allocate a message
|
[13:35] sustrik
|
(zmq_msg_init_size)
|
[13:35] sustrik
|
it's allocated in process heap
|
[13:36] pieterh
|
right, that's fine, we copy data
|
[13:36] sustrik
|
if you copy data then there's not much point in using shmem
|
[13:36] pieterh
|
much, much faster than IPC
|
[13:36] pieterh
|
also, am I wrong, but with ZMQ you *always* have to copy data on recv?
|
[13:36] sustrik
|
if that's the case, then sure
|
[13:36] sustrik
|
it have been discussed serveral times
|
[13:37] pieterh
|
shmem?
|
[13:37] sustrik
|
yep
|
[13:37] sustrik
|
but nobody really tried to implement it
|
[13:37] pieterh
|
I've not seen any attempt to slice it into two
|
[13:37] sustrik
|
at least afaik
|
[13:37] sustrik
|
two what?
|
[13:37] pieterh
|
two layers
|
[13:37] pieterh
|
so, abstract shmem api, and implementations thereof, and use thereof
|
[13:38] pieterh
|
the problem is today, you'd have to know both sides of the cake to make it
|
[13:38] pieterh
|
both how to make a transport in 0MQ, and how to do shmem on system X
|
[13:38] pieterh
|
I'd like to slice this into two problems
|
[13:38] sustrik
|
how would you do that?
|
[13:38] Steve-o
|
re: shmem I got as far as this and haven't looked at it again: https://gist.github.com/876073
|
[13:38] pieterh
|
well, classic portable API solution
|
[13:39] pieterh
|
like how I made portable threading in ZFL
|
[13:39] sustrik
|
currently 0MQ uses ZMQ_HAVE_... macros to provide OS-specific functionality
|
[13:39] pieterh
|
first step is to agree on what constitutes the necessary shmem api for a portable shmem transport in 0MQ
|
[13:39] pieterh
|
sure, but that's not helpful here, forget it
|
[13:40] pieterh
|
you do that in some module, shmem.cpp or whatever
|
[13:40] sustrik
|
ah, you are speaking of internal apis, right?
|
[13:40] pieterh
|
yes
|
[13:40] sustrik
|
abstraction for transports
|
[13:40] pieterh
|
a portability API for shmem
|
[13:40] pieterh
|
yes
|
[13:40] sustrik
|
that would be great
|
[13:40] pieterh
|
it means one person writes the transport, another group makes it work on random boxes
|
[13:40] sustrik
|
i was thinking about it for a long time
|
[13:40] sustrik
|
but it's rather complex
|
[13:40] pieterh
|
all we need is a simple abstract API
|
[13:40] pieterh
|
shmem_get_block ()
|
[13:40] pieterh
|
shmem_write ()
|
[13:41] pieterh
|
etc. I have no idea...
|
[13:41] sustrik
|
ah, at that level
|
[13:41] sustrik
|
so not generic API for transports
|
[13:41] mato
|
guys, re ROUTER, if we are going to formalize this then rather than just renaming XREP shouldn't we think about what the real semantics are for people who wish to do routing by design?
|
[13:41] pieterh
|
no, only for shmem
|
[13:41] sustrik
|
rather generic API for shmem
|
[13:41] mato
|
since the current use of router has simply evolved out of what XREP provides.
|
[13:41] pieterh
|
sustrik: yes, just to solve this case
|
[13:41] sustrik
|
spieterh: sure, feel free to implement
|
[13:41] pieterh
|
mato: yes, but it seems there's no better way
|
[13:42] pieterh
|
sustrik: my problem is I have zero idea what a transport would need
|
[13:42] mato
|
pieterh_: why so? there's no hurry, 3.0 is a way away yet
|
[13:42] sustrik
|
mato: ack
|
[13:42] pieterh
|
I'm happy to make shmem implementations of some defined API
|
[13:42] mato
|
pieterh_: aren't we supposed to be master programmers doing this well, rather than making rash decisions?
|
[13:42] sustrik
|
pieterh: great
|
[13:42] pieterh
|
mato: there's no hurry except the current names are painful
|
[13:42] sustrik
|
mato: my feeling is people want SMTP like-semantivc
|
[13:42] pieterh
|
mato: lol
|
[13:42] pieterh
|
double lol
|
[13:42] pieterh
|
the man pages document code rather than code implementing man pages
|
[13:43] mato
|
huh?
|
[13:43] pieterh
|
shrug
|
[13:43] pieterh
|
if you don't know what I mean, forget it
|
[13:43] pieterh
|
anyhow, where is the rash decision?
|
[13:43] pieterh
|
there has been extensive discussion about the names of sockets
|
[13:43] sustrik
|
mato: there's one issue in the bugtracker to do with man pages
|
[13:44] mato
|
but the names don't fit with the big picture
|
[13:44] sustrik
|
could you have a look at it?
|
[13:44] pieterh
|
and renaming XREP and XREQ has been agreed by the list, with total consensus
|
[13:44] mato
|
i love your dogmatism
|
[13:44] pieterh
|
mato: what big picture? show me the design docs...
|
[13:44] mato
|
4 people replied
|
[13:44] pieterh
|
if it's just in someone's head, it's not on paper
|
[13:44] mato
|
total consensus
|
[13:44] pieterh
|
4 people replied, no-one said no, that's total consensus, yes
|
[13:44] pieterh
|
it's not exactly a hypertalkative list
|
[13:44] mato
|
right.
|
[13:45] pieterh
|
but if you want to veto it, go for it
|
[13:45] pieterh
|
I'm going to switch to the new names ASAP in any case
|
[13:45] mato
|
right, so you're going to fork the api
|
[13:45] pieterh
|
no, I'm implementing the consensus
|
[13:45] mato
|
thanks for that.
|
[13:45] pieterh
|
if you don't want to follow, fork the API
|
[13:45] pieterh
|
please don't be silly, mato
|
[13:46] pieterh
|
this was discussed ad nauseam
|
[13:46] pieterh
|
Just like a 3.0 version will 'fork the api'?
|
[13:46] pieterh
|
improvements that solve real problems people face
|
[13:46] pieterh
|
if you find this offensive, well... sigh
|
[13:47] mato
|
forget it
|
[13:47] pieterh
|
whatever
|
[13:47] pieterh
|
discuss on that thread, it's public
|
[13:47] pieterh
|
give your reasons
|
[13:47] mato
|
sure.
|
[13:52] drbobbeaty
|
Guys, I don't mean to poke my ideas in in the middle of a discussion, but would there be a significant issue if the 2.x codebase used BOTH names, and then the 3.x codebase used just the 'new' one? That way, people that use the names for a significant part of their understanding of their function can have both while not breaking backward compatibility.
|
[13:53] drbobbeaty
|
I can see that names are important to understand, I deal with that from the users of my libraries every day. I sweat the names for a long time to make sure they are concise, descriptive, and clear.
|
[13:53] ianbarber
|
upstream downstream were (are?) still valid
|
[13:54] drbobbeaty
|
But I can also see that the original design was based on these names, and they, too, have value to a group of users for just that reason. They understand the library and function in terms of these names.
|
[13:54] drbobbeaty
|
Just my USD $0.02
|
[13:55] sustrik
|
drbobbeaty: i think almost everybody agrees that ZMQ_ROUTER being introduced
|
[13:56] sustrik
|
the discussion was about whether original XREP should remain as a different socket type
|
[13:56] drbobbeaty
|
Oh... there... I should have stayed quiet. I thought it was about a name change. Please forgive my ignorance.
|
[13:57] mato
|
sustrik: that is precisely the point.
|
[14:00] jhawk28
|
I suspect that it will become more clear as more "devices" are written
|
[14:04] mato
|
sustrik: SMTP, yes, that's an interesting insight
|
[14:04] mato
|
sustrik: I think you're right
|
[14:05] mato
|
sustrik: what is the manpage issue? i can't find it...
|
[14:05] sustrik
|
jhaw28: i would say so
|
[14:05] sustrik
|
mato: let me see
|
[14:08] sustrik
|
mato: issue 130
|
[14:10] mile
|
hi
|
[14:10] jhawk28
|
b.....tlllrrrrr
|
[14:10] jhawk28
|
]
|
[14:10] mile
|
does anyone have hands-on experience with erlang bindings?
|
[14:11] mato
|
sustrik: hmm, i can see how that might a bug in the manpage, but isn't it just exposing an inconsistency in the internal implementation?
|
[14:11] sustrik
|
mile: ask yrashk
|
[14:12] sustrik
|
he should be here i think
|
[14:12] mato
|
sustrik: shouldn't the blocking behaviour be identical irrespective of the use of bind/connect?
|
[14:12] sustrik
|
the problem is that on bind side
|
[14:12] sustrik
|
when there's no connection
|
[14:12] sustrik
|
there's no queue
|
[14:12] mile
|
sustrik, tx, will do
|
[14:12] mato
|
sustrik: yes, you're describing the implementation
|
[14:12] sustrik
|
and thus nowhere to store the message in
|
[14:13] sustrik
|
the alternative would be to have one pipe pre-prepared even on bind side
|
[14:13] sustrik
|
and assign it to first connected peer
|
[14:13] mato
|
that would seem to result in more consistent behaviour on the user side
|
[14:14] sustrik
|
ok, then create an issue for that
|
[14:14] yrashk
|
sustrik: mile I am here
|
[14:14] sustrik
|
so that we don't forget
|
[14:14] mato
|
alright.
|
[14:14] mile
|
yrashk, hi, great :)
|
[14:14] mile
|
i'm looking into connecting yaws to 0mq
|
[14:14] mile
|
as a http front-end
|
[14:15] mile
|
but I'm not sure about the state of the erlang bindings
|
[14:15] yrashk
|
mile: what do you mean?
|
[14:15] mile
|
since the alternative is hacking nginx, which I would very much like to avoid :)
|
[14:16] yrashk
|
mile: what exactly are you worried about re: state?
|
[14:16] mile
|
the bindings page states the both versions are not API compatible
|
[14:16] mile
|
and I couldn't find much of the benchmarks regarding the performance
|
[14:17] yrashk
|
mile: short version: erlzmq is slow, erlzmq2 is fast
|
[14:17] yrashk
|
on 1byte messages erlzmq2 does 1.3mln/sec (localhost) [on my mac pro]
|
[14:18] yrashk
|
on 1Kbyte messages -- about 300K/sec
|
[14:18] mile
|
that should saturate a gbit ethernet, right?
|
[14:18] yrashk
|
their API are close to each other, though -- but not exactly the same, but fairly close
|
[14:18] yrashk
|
hopefully, I guess
|
[14:18] yrashk
|
never tried yet
|
[14:19] mile
|
ok, thanks!
|
[14:19] mile
|
i will be looking into this
|
[14:19] mile
|
and report if I find something interesing ;)
|
[14:19] sustrik
|
300K/sec = 2.45Gb/sec
|
[14:20] mile
|
yeah :)
|
[14:21] yrashk
|
mile: for erlzmq2, you have to have >=R14B01
|
[14:21] yrashk
|
as it is a NIF-based binding
|
[14:21] mato
|
sustrik: i guess i should close the original issue then?
|
[14:22] yrashk
|
mile: sure, I'd be pleased to see reports, erlzmq2 is fairly young
|
[14:22] yrashk
|
but seems to work well
|
[14:22] sustrik
|
yes, just mention the number of the new one when closing it
|
[14:22] yrashk
|
if anything evax and I always ready to fix it :)
|
[14:22] mile
|
yrashk, greta :)
|
[14:22] mile
|
I'll try to make a good benchmark for aws+0mq
|
[14:23] mile
|
yaws*
|
[14:23] mile
|
kind of like mongrail2
|
[14:23] mile
|
thats what I'm aiming for
|
[14:23] mile
|
only simpler and hopefully much faster
|
[14:23] sustrik
|
mile: as for intreraction with nginx, meto mentioned he's going to have a look at that
|
[14:24] mato
|
i will, if i get the funding :-)
|
[14:24] yrashk
|
mile: just please understand that those beforementioned benchmarks are from localhost
|
[14:24] mile
|
I looked into it a couple of days ago, but it kind of freaked me out
|
[14:24] yrashk
|
I haven't tested anything else yet
|
[14:25] mile
|
yrashk, sure, I will be setting up 2 boxes on a gbit switch
|
[14:25] mile
|
and see how they perform
|
[14:25] yrashk
|
sweet
|
[14:25] yrashk
|
mile: please send them over if possible (yrashk@gmail.com)
|
[14:25] mile
|
and try to replicate the ck1mil
|
[14:26] mile
|
I hope I'll manage to do it next week
|
[14:26] mile
|
then I'm gone for like 10 days :)
|
[14:27] mile
|
but both nginx and yaws plugins would be very very cool stuff :)
|
[14:27] mile
|
hopefully I'll get something useful to opensource :)
|
[14:28] mato
|
mile: i've been poring over the nginx internals, it looks doable but non-trivial
|
[14:28] mile
|
mato, I was looking into proxy-modules for that, but my C is not good enough...
|
[14:29] mile
|
since it in deed appears to be nontrivial :)
|
[14:29] mato
|
mile: my C is more than good enough, but it's still a non-trivial amount of work.
|
[14:29] mile
|
but yaws has a very elegant way of attaching stuff
|
[14:29] mato
|
yaws? /me goes off to google that
|
[14:29] mile
|
erlang yaws
|
[14:30] mile
|
so I'll be looking into the performance of that combo
|
[14:30] mato
|
aha, ok
|
[14:31] mile
|
my erlang is not that good
|
[14:31] mile
|
but it's straitforward as hell
|
[14:31] mile
|
as long as one had something to do with functional programming
|
[14:31] mile
|
:)
|
[14:32] yrashk
|
it is indeed very straightforward
|
[14:33] mile
|
the only problem is that you don't get to find many erlang developers
|
[14:33] mile
|
so its hard to sell it
|
[14:33] mile
|
in small companies - hence the 0mq
|
[14:34] mile
|
make the frontend in less then 100lines of erlang
|
[14:34] yrashk
|
mile: my company is about to change it (i.e. make erlang more popular)
|
[14:34] mile
|
and let others use whatever they want on the other side of 0mq
|
[14:34] yrashk
|
:)
|
[14:35] mile
|
thats a good thing! :)
|
[14:35] mile
|
I had contact with haskell some time ago, and I'm sincerely charmed by erlang :)
|
[14:35] yrashk
|
nice!
|
[14:36] yrashk
|
I've been doing erlang on and off since 2001
|
[14:37] mile
|
that's a while :)
|
[14:37] mile
|
I pinged you, hopefully next week I'll have some data
|
[14:38] yrashk
|
sweet
|
[14:38] yrashk
|
thanks
|
[14:39] mile
|
I'm off for a weekend, thanks for the advice!
|
[15:03] Guthur
|
pieterh & sustrik: was that long discussion just about the naming of XREP and XREQ
|
[15:04] sustrik
|
it was not about naming :)
|
[15:04] sustrik
|
ultimately, it was about what should be done by 0mq itself and what should be done by the layers on top of it
|
[15:05] Guthur
|
sustrik: ok, sorry I just came back and it was very long and I found it hard to figure out what was going on
|
[15:06] sustrik
|
there's a thread on the ML if you are interested
|
[15:09] Guthur
|
ok, I'll check it out
|
[15:24] sustrik
|
pieterh: still here?
|
[15:25] sustrik
|
the use case for the error you've reported have scrolled out of my irc window
|
[15:25] sustrik
|
can you repost it?
|
[15:29] sustrik
|
<sustrik> the use case for the error you've reported have scrolled out of my irc window
|
[15:29] sustrik
|
<sustrik> can you repost it?
|
[15:29] sustrik
|
pieterh_: ^^
|
[15:29] pieterh
|
sustrik: sorry, was offline for a while
|
[15:30] sustrik
|
np
|
[15:30] pieterh
|
you lost the URIs?
|
[15:30] sustrik
|
yep
|
[15:30] pieterh
|
https://gist.github.com/875932
|
[15:30] sustrik
|
thanks
|
[15:30] pieterh
|
it looks like the first frame after a handshake gets lost
|
[15:30] pieterh
|
but randomly...
|
[15:30] pieterh
|
thanks, I appreciate it, it's blocking me
|
[15:40] evax
|
working on erlzmq2 some weird behaviour happens at times: after calling zmq_term and our own poller thread has ended, one thread enters a busy loop because SIGFE is trapped but keeps being raised
|
[15:40] evax
|
*SIGFPE
|
[15:43] evax
|
when zmq_term returns, all zmq threads have ended ?
|
[15:46] pieterh
|
evax: it should say so in the man page
|
[15:48] mikko-
|
pieter: all 2-1 builds succeed now
|
[15:49] pieterh
|
mikko: yup
|
[15:49] evax
|
pieterh_: thanks, I was reading an outdated man page...
|
[15:49] pieterh
|
there are a couple of issues still to close, then 2-1 is ready to release
|
[15:50] pieterh
|
evax: does it say something useful in the man page?
|
[15:51] evax
|
pieterh_: that it can fail with EINTR now
|
[15:52] pieterh
|
ok
|
[16:26] evax
|
pieterh_: in the case of inproc sockets, is it possible that a zmq_recv succeeds before the associated send returns ?
|
[16:28] evax
|
this is with a PUSH/PULL pair and the zmq_recv if right after a zmq_poll
|
[16:30] sustrik
|
yes, it's possible
|
[16:30] evax
|
sustrik: thanks, then that's what's biting us
|
[16:33] Guthur
|
evax: out of curiosity what are you working on for erlzmq?
|
[16:33] Guthur
|
it is elrzmq, correct?
|
[16:34] evax
|
we send a message to shut down the system, and sometimes before the zmq_send returns our polling thread recvs the request, closes the sockets and terminates the context, and this fires the SIGFPE busyloop in the original thread
|
[16:34] evax
|
Guthur: erlzmq2, the NIF base version
|
[16:35] Guthur
|
ah ok
|
[16:36] Guthur
|
so you are using zmq to manage the state of your zmq binding
|
[16:36] Guthur
|
many kudos there, hehe
|
[16:39] evax
|
Guthur: I wanted to implementing non blocking send, but I stumbled on this busy loop issue
|
[16:39] evax
|
Guthur: yeah, it felt really natural
|
[16:40] michelp
|
wow you guys overflowed my backscroll :)
|
[16:40] Guthur
|
it has been very busy today
|
[16:41] Guthur
|
I'm still digesting the XREP vs. ROUTER stuff, the details are on the mailing list though
|
[16:42] Guthur
|
sustrik: if XREP really could suddenly change, it probably should never have been exposed
|
[16:43] Guthur
|
but I suppose that's been argued already
|
[16:43] sustrik
|
yup
|
[16:54] Guthur
|
and those cool devices that guide demonstrates are pretty cool
|
[16:54] Guthur
|
would be a shame if they broke
|
[16:58] sustrik
|
i'm not proposing to break them
|
[16:58] sustrik
|
it was just illustration
|
[18:09] evax
|
sustrik: erlzmq2 can no more block the VM on send
|
[18:09] sustrik
|
nice!
|
[18:10] sustrik
|
thanks for quick fix
|
[18:11] evax
|
np, let me know if there are other features missing
|
[18:15] sustrik
|
sure
|
[18:49] pieterh
|
sustrik: sorry, was away... did you manage to reproduce the pub-sub issue?
|
[19:16] pieterh
|
ok, reproduced on second box
|
[20:17] Guthur
|
pieterh, do you mean to not increment the sequence in that gist?
|
[22:08] rfw
|
hi there, does the pyzmq.eventloop ioloop have epoll support like tornado's?
|
[22:53] yrashk
|
so 8 years for saleyn :-(((
|
[23:43] Guthur
|
yrashk, have you seen the first paragraph of the wiki page
|
[23:44] Guthur
|
Goldman Sachs code, which is described as "there is a danger that somebody who knew how to use this program could use it to manipulate markets in unfair ways."
|
[23:44] Guthur
|
but it's fine for Goldman to have it
|