| [Time] Name | Message |
|
[00:48] pieterh
|
prodatalab is concept i've been working on for quite some time. needless to say 0mq makes it all come together very well. i think it can be revolutionary in more than a few ways. can i possibly interest you in discussing it with me. i'm considering a partnership atm for the potential to market it.
|
|
[01:23] rbraley
|
i was looking for that link where the guy described his google interview process and a google search provides hundreds of them.
|
|
[01:36] jhawk28
|
any of the zeromq devs here?
|
|
[01:39] jhawk28
|
has any work been done to migrate openamq to 0mq?
|
|
[04:07] CIA-20
|
rbzmq: 03Brian Buchanan 07master * r2aa02db 10/ (rbzmq.c zmq.gemspec):
|
|
[04:07] CIA-20
|
rbzmq: Fix potential memory leak in ZMQ.select()
|
|
[04:07] CIA-20
|
rbzmq: Reported by: Eric Wong <normalperson@yhbt.net> - http://bit.ly/coN2GW
|
|
[06:01] sustrik
|
mikko: that's bad, it looks like freebsd bug
|
|
[06:01] sustrik
|
what version of freebsd are you using?
|
|
[07:05] mikko
|
sustrik: 8.1 i think
|
|
[07:06] mikko
|
FreeBSD fbsd.localdomain 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
|
|
[07:10] mikko
|
sustrik: i can't reproduce in a simple program
|
|
[07:22] sustrik
|
mikko: what about showing the traceback on a freebsd forum?
|
|
[07:23] sustrik
|
someone may have had an experience with it there
|
|
[07:25] sustrik
|
the two links i've posted seem to indicate that there used to be such bug in freebsd...
|
|
[08:14] mikko
|
sustrik: some progress
|
|
[08:14] mikko
|
yep
|
|
[08:14] mikko
|
got it
|
|
[08:14] sustrik
|
mikko: a solution?
|
|
[08:14] mikko
|
yes
|
|
[08:15] mikko
|
php needs to link against libthr
|
|
[08:15] sustrik
|
several thread libraries out there?
|
|
[08:15] mikko
|
it is not enough that libzmq links against libthr
|
|
[08:15] mikko
|
this is my theory this far:
|
|
[08:16] mikko
|
single threaded php linked against non-reentrant libraries
|
|
[08:16] mikko
|
loads up and loads libzmq
|
|
[08:16] sustrik
|
got it
|
|
[08:16] mikko
|
which ends up using non-reentrant things
|
|
[08:16] mikko
|
due to loading order
|
|
[08:16] mikko
|
but this is just a theory not knowing much about freebsd internals
|
|
[08:17] sustrik
|
seems plausible to mwe
|
|
[08:17] sustrik
|
me
|
|
[08:17] mikko
|
libthr.so.3 => /lib/libthr.so.3 (0x800883000)
|
|
[08:17] mikko
|
is what libzmq links against
|
|
[08:17] mikko
|
and php compilation gives option "Do you want to link against libthr?"
|
|
[08:17] sustrik
|
aha
|
|
[08:18] sustrik
|
it should be noted somewhere
|
|
[08:18] sustrik
|
so that the knowlefge is not lost
|
|
[08:18] mikko
|
it's in github issue tracker
|
|
[08:18] mikko
|
at least
|
|
[08:19] mikko
|
it is odd that i can't reproduce this with a simple program
|
|
[08:19] sustrik
|
timing issue maybe
|
|
[08:19] sustrik
|
btw, is the php binding built using autotools?
|
|
[08:19] mikko
|
possibly and php uses dlopen/load to load the php-zmq
|
|
[08:19] mikko
|
yes
|
|
[08:20] sustrik
|
maybe it can be made automatic
|
|
[08:20] mikko
|
but thats not the thing
|
|
[08:20] sustrik
|
FreeBSD -> link with libthr
|
|
[08:20] mikko
|
there are three moving parts
|
|
[08:20] mikko
|
php, php extension for zmq and libzmq
|
|
[08:20] mikko
|
so roughly:
|
|
[08:21] mikko
|
php <- dynamic loading -> php-zmq <- runtime linker loading > libzmq
|
|
[08:21] mikko
|
dynamic loading being dlopen or what the platform has
|
|
[08:21] mikko
|
i think the boundary of dlopen actually causes the confusion
|
|
[08:21] sustrik
|
looks like
|
|
[08:22] sustrik
|
but does in make sense to compile php-zmq without libthr?
|
|
[08:22] mikko
|
i could try to create a test case
|
|
[08:22] mikko
|
for php-zmq it doesnt matter
|
|
[08:22] mikko
|
because libzmq links against it
|
|
[08:22] sustrik
|
ah
|
|
[08:22] sustrik
|
it's php build that has to be modified
|
|
[08:22] mikko
|
for php it matters because it doesnt 'hard-link' against libthr at any point
|
|
[08:22] mikko
|
yes
|
|
[08:23] sustrik
|
my god
|
|
[08:23] sustrik
|
ok, i see
|
|
[08:23] mikko
|
i'll try to create a test case for this
|
|
[08:23] sustrik
|
ack
|
|
[08:28] anarchocap
|
hi
|
|
[08:36] mikko
|
sustrik: oh joy
|
|
[08:36] mikko
|
can't reproduce
|
|
[08:39] sustrik
|
:|
|
|
[08:42] mikko
|
in a very simple test case it work
|
|
[08:42] mikko
|
s
|
|
[08:43] pieterh
|
mikko: this is kind of typical, you chopped out whatever it is that provokes the problem
|
|
[08:44] mikko
|
pieterh: there are so many moving parts in the final solution, could be a race condition, could be just about anything
|
|
[08:44] mikko
|
at least there is a work-around
|
|
[08:44] mikko
|
http://github.com/mkoppanen/php-zmq/issues#issue/7
|
|
[08:44] mikko
|
i'll add it to the wiki as well
|
|
[08:48] mikko
|
http://www.zeromq.org/bindings:php#toc3
|
|
[08:48] mikko
|
done
|
|
[08:58] pieterh
|
sustrik: do you have a moment?
|
|
[08:59] pieterh
|
could you explain why you think subscription idempotency would let one subscriber unsubscribe another one?
|
|
[09:06] sustrik
|
pieterh: think of a pub node connected to 2 sub nodes
|
|
[09:06] pieterh
|
ack
|
|
[09:06] sustrik
|
sub node 1: SUB(ABC)
|
|
[09:06] sustrik
|
sub node 2: SUB(ABC)
|
|
[09:07] sustrik
|
sub node 1: UNSUB(ABC)
|
|
[09:07] sustrik
|
sub node2: wtf?!?
|
|
[09:07] pieterh
|
that is, if you literally take the algorithm from the SUB socket and move it to the PUB socket
|
|
[09:08] sustrik
|
yes, any such algorithm
|
|
[09:08] pieterh
|
if idempotency was part of the spec, would it be possible?
|
|
[09:08] sustrik
|
sure
|
|
[09:08] pieterh
|
(trick question: answer is "of course")
|
|
[09:08] sustrik
|
the question is whether it is desirable
|
|
[09:08] sustrik
|
there are 2 use cases afaics:
|
|
[09:09] pieterh
|
seen from the app pov it's very desirable
|
|
[09:09] pieterh
|
it removes the need to duplicate state
|
|
[09:09] sustrik
|
1. a GUI app, user nervously clicking on sub & unsub buttons
|
|
[09:09] sustrik
|
2. a automated app with a consistent algorithm
|
|
[09:09] pieterh
|
consistency may not be always possible
|
|
[09:10] sustrik
|
if you have idempotency, 2 has to duplicate state
|
|
[09:10] sustrik
|
if you have refcounting 1 has to duplicate state
|
|
[09:10] pieterh
|
if you don't have idempotency, 2 has to duplicate state
|
|
[09:10] sustrik
|
exactly
|
|
[09:10] pieterh
|
:-)
|
|
[09:10] pieterh
|
how can you say 'exactly' when i say the exact opposite of you?
|
|
[09:11] pieterh
|
i'm confused now
|
|
[09:11] pieterh
|
ok, here's a point of view on this...
|
|
[09:11] sustrik
|
ah, sorry misread 2 as 1
|
|
[09:11] sustrik
|
you are NOT right
|
|
[09:11] sustrik
|
:)
|
|
[09:11] pieterh
|
that's better
|
|
[09:12] pieterh
|
ok, do you have 5 mins to walk through this?
|
|
[09:12] sustrik
|
yes
|
|
[09:12] pieterh
|
so... idempotent apis are robust against stateloss in the network
|
|
[09:12] pieterh
|
e.g. REST
|
|
[09:13] pieterh
|
that's just a desirable property... any code that handles sub/unsub control commands will be more robust if the api it talks to is idempotent
|
|
[09:13] pieterh
|
now
|
|
[09:13] pieterh
|
this is for TCP, not multicast
|
|
[09:13] pieterh
|
the PUB knows each SUB connection
|
|
[09:13] pieterh
|
has the subscriptions for each one
|
|
[09:13] sustrik
|
don't go into technical details
|
|
[09:13] sustrik
|
that's clear
|
|
[09:14] pieterh
|
it can trivially ensure that anything coming from SUB A does not interfere with SUB B
|
|
[09:14] pieterh
|
and it needs to, anyhow
|
|
[09:14] pieterh
|
since anyone can craft sub/unsub messages
|
|
[09:14] sustrik
|
sure, but not so user implemetned nodes
|
|
[09:14] sustrik
|
such as a node aggregating FIX data feeds
|
|
[09:14] sustrik
|
or similar
|
|
[09:14] pieterh
|
devices is next, we were starting with direct sub-to-pub
|
|
[09:15] sustrik
|
the only guestion is:
|
|
[09:15] sustrik
|
do we want to provide the GUI use case, or the "device" use case as default
|
|
[09:15] pieterh
|
i don't see this as the choice
|
|
[09:15] pieterh
|
that looks like a false dichotomy
|
|
[09:15] sustrik
|
shrug
|
|
[09:16] pieterh
|
based on assumptions that i'm not sharing / seeing
|
|
[09:16] sustrik
|
how would you integrate say FIX feeds?
|
|
[09:16] pieterh
|
can i refer to previous known use cases?
|
|
[09:16] sustrik
|
yes
|
|
[09:16] pieterh
|
take AMQP bindings
|
|
[09:16] pieterh
|
idempotent
|
|
[09:16] pieterh
|
federable
|
|
[09:17] sustrik
|
that's because they are encapsulated inside of the broker
|
|
[09:17] pieterh
|
work for both the GUI and the device use case
|
|
[09:17] sustrik
|
not for the latter
|
|
[09:17] pieterh
|
ah, ok
|
|
[09:18] pieterh
|
you are saying you don't want devices to keep any state
|
|
[09:18] sustrik
|
yes
|
|
[09:18] pieterh
|
thus state must be in the edges
|
|
[09:18] sustrik
|
so that they can come and go
|
|
[09:18] sustrik
|
right
|
|
[09:18] sustrik
|
end-to-end principle
|
|
[09:18] pieterh
|
ok, so sub sockets can do the refcounting and give idempotency
|
|
[09:18] pieterh
|
with amqp there is no state in the API
|
|
[09:19] sustrik
|
yes, amqp is designed other way round
|
|
[09:19] pieterh
|
devices in any case must store subscriptions
|
|
[09:19] pieterh
|
in their sub sockets
|
|
[09:19] sustrik
|
it's just transient
|
|
[09:19] sustrik
|
when the connection goes, associated subscriptions are deleted
|
|
[09:19] pieterh
|
right
|
|
[09:19] sustrik
|
when device fails, it looses them of course
|
|
[09:20] sustrik
|
when the peer reconnects it resends the subscriptions
|
|
[09:20] pieterh
|
there is an easy, trivial, stupid, scalable design
|
|
[09:20] sustrik
|
yes?
|
|
[09:20] pieterh
|
do not forward ubsub messages
|
|
[09:20] pieterh
|
*unsub
|
|
[09:20] pieterh
|
and ignore duplicate subs
|
|
[09:21] sustrik
|
that would aggregate subs in the network until congestion happens
|
|
[09:21] pieterh
|
it fails in some use cases but those would normally be on multicast anyhow
|
|
[09:21] pieterh
|
where it does not apply
|
|
[09:21] pieterh
|
scenarios that do unsubscription are arguably very rare
|
|
[09:22] sustrik
|
true
|
|
[09:22] sustrik
|
do you want to drop UNSUBSCRIBE then?
|
|
[09:22] pieterh
|
it's just a thought
|
|
[09:22] sustrik
|
"if you want to unsubscribe, close the socket"
|
|
[09:22] pieterh
|
for example, yes
|
|
[09:22] sustrik
|
it's viable
|
|
[09:23] sustrik
|
ask on list, IMO
|
|
[09:23] pieterh
|
or else don't forward unsubscribe over TCP and handle it locally
|
|
[09:23] sustrik
|
no
|
|
[09:23] sustrik
|
that's the congestion scenario
|
|
[09:23] pieterh
|
well, it's what we do today :-)
|
|
[09:23] sustrik
|
invisible traffic flowing on
|
|
[09:24] sustrik
|
yes, buth it's definitely not desirable
|
|
[09:24] pieterh
|
true
|
|
[09:24] pieterh
|
is it out of the question that forwarder devices keep track of their downstream subscribers?
|
|
[09:26] sustrik
|
?
|
|
[09:26] sustrik
|
ah, i see
|
|
[09:26] pieterh
|
it would require that they replicate the prefix tree
|
|
[09:26] pieterh
|
and intelligently sub/unsub as needed
|
|
[09:26] pieterh
|
it is scalable and if the device dies, it works as expected
|
|
[09:27] pieterh
|
a proper proxy for subscriptions
|
|
[09:27] sustrik
|
yes, that's an option
|
|
[09:27] pieterh
|
i'm happy to help code that up in 21 years or so :-)
|
|
[09:28] sustrik
|
:)
|
|
[09:28] sustrik
|
so how about the birth?
|
|
[09:28] pieterh
|
still waiting
|
|
[09:28] sustrik
|
have it happened already?
|
|
[09:28] sustrik
|
ah
|
|
[09:28] sustrik
|
ok, cyl
|
|
[09:29] pieterh
|
sylvie is in the hospital and they're giving her stuff to provoke it but the baby is like "no way! that's a hostile world out there!"
|
|
[09:29] pieterh
|
thanks for this explanation
|
|
[09:51] travlr
|
pieterh: yup, prodatalab is me. do you find the concept interesting?
|
|
[09:51] pieterh
|
travlr: will it be free software?
|
|
[09:52] travlr
|
i'm trying to decide what route to take atm.. would care to discuss it with me?
|
|
[09:52] pieterh
|
certainly...
|
|
[10:01] CIA-20
|
zeromq2: 03Martin Sustrik 07master * r7aba2d1 10/ doc/zmq.txt : documentation leftover from v2.0.6 cleaned in zmq(7) - http://bit.ly/9Tz5PI
|
|
[11:57] pietia
|
is it possible to use zeromq as cron?
|
|
[12:10] sustrik
|
pietia: there are no timers
|
|
[12:10] sustrik
|
it's just an interconnect
|
|
[12:11] mikko
|
http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#code_ev_timer_code_relative_and_opti
|
|
[16:05] pieterh
|
mikko: nice link
|
