| [Time] Name | Message |
|
[00:31] Guthur
|
out of curiosity do many specifically build 0MQ for 64bit platform?
|
|
[00:33] Guthur
|
oh nvm, it's not really the issue I had mind
|
|
[00:38] klestes
|
greetings
|
|
[00:38] klestes
|
anyone here ?
|
|
[00:39] Guthur
|
klestes, a few
|
|
[00:39] Guthur
|
i wont be here long though, just about to hit the sack
|
|
[00:39] Guthur
|
what's up?
|
|
[00:39] klestes
|
good. I am preparing to do my first patch submission
|
|
[00:39] klestes
|
where are you, Guth ?
|
|
[00:39] Guthur
|
UK
|
|
[00:40] klestes
|
ah. Hows the weather ?
|
|
[00:40] Guthur
|
I'm not much of a git gurur
|
|
[00:40] Guthur
|
it's actually been quite good recently
|
|
[00:40] klestes
|
me either !
|
|
[00:40] klestes
|
great !
|
|
[00:40] Guthur
|
pieterh, ping
|
|
[00:40] Guthur
|
pieterh might be able to help, if he's about
|
|
[00:41] klestes
|
yep, if you hear the 0Mgods gripe about that damn newbie patch, I was susccessful :)
|
|
[00:41] klestes
|
I think the SF meetup was today, so he may be out of it. or having beer.
|
|
[00:41] klestes
|
anyway, if I botch it, there's always tomorrow.
|
|
[00:41] Guthur
|
oh ok, well there is a couple of resources on the web on how to make git patches
|
|
[00:41] klestes
|
yes, I am reading one now.
|
|
[00:42] Guthur
|
it's actually not too hard, I just can't remember otoh
|
|
[00:42] Guthur
|
I really need sleep though, sorry I can't be more help
|
|
[00:42] klestes
|
thats ok.
|
|
[00:42] klestes
|
I wonder if I can submit just a plain diff the first time ?
|
|
[00:43] Guthur
|
they do like signed off patches to be honest
|
|
[00:43] klestes
|
ah, ok.
|
|
[00:43] Guthur
|
anyhoo must go, later klestes
|
|
[00:43] klestes
|
now,what exactly qualifies a patch as being 'signed off' - use of the git format-patch command ?
|
|
[00:43] klestes
|
nice talking to you, guthur. Have a good night !
|
|
[00:44] Guthur
|
we should probably document this process somewhere
|
|
[00:44] Guthur
|
g'night
|
|
[01:06] klestes
|
well, my first patch has been submitted. Bombs away !!
|
|
[01:06] klestes
|
and to all, a good nyte.
|
|
[06:03] Freejack
|
Has anyone tried plugging ZeroMQ into anything besides sockets? (i.e. TIPC, DIPC, LINX, Streams, etc...)?
|
|
[06:07] sustrik
|
Freejack: nope
|
|
[06:08] sustrik
|
you are free to try
|
|
[06:08] sustrik
|
however, if the underlying transport doesn't use file descriptors, it will be pretty hard to integrate into 0mq
|
|
[06:08] Freejack
|
Sounds like a plan. Yeah, it uses file descriptors.
|
|
[06:09] Freejack
|
I've been playing with message queues for a while now...and find posix queues and Linx Queues to be extremely efficient.
|
|
[06:11] Freejack
|
Linx Queues especially. Their like traveling at light speed compared to sockets over the LAN.
|
|
[06:13] Freejack
|
sustrik: Besides...since ZeroMQ is in fact a high level MQ API, I figure the two would be a good match.
|
|
[06:14] sustrik
|
sure
|
|
[06:14] sustrik
|
go on with it
|
|
[06:17] Freejack
|
Guess I'd better start grokking the code. bbl.
|
|
[08:30] pieterh
|
sustrik: how easy is it to add a transport layer to 0MQ?
|
|
[08:44] sustrik
|
pieterh: depends
|
|
[08:44] sustrik
|
what transport do you have in mind?
|
|
[08:45] pieterh
|
well, of course TLS/SSL for a start
|
|
[08:45] pieterh
|
but more generally, any virtual stream
|
|
[08:46] pieterh
|
we get these questions about implementing UDP, SCTP, etc. regularly
|
|
[08:46] pieterh
|
and there is no good answer except "go study the code and send us a patch"
|
|
[08:47] pieterh
|
have you ever looked at e.g. the Linux user-space file system (FUSE)?
|
|
[08:50] sustrik
|
it would be nice to have standardised interface for transports
|
|
[08:50] sustrik
|
however, it's hard to define it
|
|
[08:50] sustrik
|
given the pretty large differences between trasnsports (inproc, pgm, tcp)
|
|
[08:51] sustrik
|
as FUSE: no
|
|
[08:59] pieterh
|
FUSE is just an analogy
|
|
[08:59] pieterh
|
you can keep a bunch of native transports tightly bound to whatever...
|
|
[08:59] pieterh
|
but it would be neat to have a virtual interface for stream transports
|
|
[09:00] pieterh
|
e.g. I'd like to explore making a shmem transport, but where do I start?
|
|
[09:00] sustrik
|
it's a connection-based transport, so have a look how tcp transport works
|
|
[09:01] sustrik
|
zmq_listener_t, zmq_connecter_t, zmq_init_t, zmq_engine_t
|
|
[09:01] pieterh
|
no, I'm not asking for documentation now, here, on IRC... :-)
|
|
[09:02] pieterh
|
if tcp is the right model for all connected stream transports
|
|
[09:02] sustrik
|
probably not
|
|
[09:03] sustrik
|
no idea actually
|
|
[09:03] pieterh
|
right
|
|
[09:03] pieterh
|
IMO this is a major barrier to contributors able to extend 0MQ's transports
|
|
[09:03] sustrik
|
agreed
|
|
[09:04] Guthur
|
+1 that, hehe
|
|
[09:05] sustrik
|
the question is how to define the interface
|
|
[09:05] sustrik
|
right now 0mq doesn't do much for you
|
|
[09:05] pieterh
|
well, it can IMO map the 0MQ API semantics
|
|
[09:05] sustrik
|
there's a framework for launching async objects
|
|
[09:05] pieterh
|
bind, connect, send, recv
|
|
[09:05] sustrik
|
nope, it's event-driven
|
|
[09:06] sustrik
|
another part of functionality, is pollers
|
|
[09:06] sustrik
|
which allow you to register your fds
|
|
[09:06] sustrik
|
and get events such as pollin and pollout
|
|
[09:06] pieterh
|
so any transport layer has to export FDs
|
|
[09:06] pieterh
|
is that how the pgm transport works?
|
|
[09:06] sustrik
|
well, it can run a thread of it's own
|
|
[09:06] sustrik
|
and ignore the I/O threads
|
|
[09:06] pieterh
|
that's a good start IMO
|
|
[09:06] sustrik
|
= standard pollers
|
|
[09:07] pieterh
|
how about... a transport as a bridge?
|
|
[09:07] sustrik
|
?
|
|
[09:07] pieterh
|
hehe
|
|
[09:07] pieterh
|
ok, this is crazy, but...
|
|
[09:08] pieterh
|
transport X runs as its own thread, OK?
|
|
[09:08] pieterh
|
it communicates purely over inproc
|
|
[09:08] pieterh
|
define an inproc API back to 0MQ core
|
|
[09:08] pieterh
|
*any* thread can register, use it, become a transport
|
|
[09:09] sustrik
|
can you be more specific?
|
|
[09:09] pieterh
|
I've not had coffee yet, I know close to nothing about the 0MQ internals, and you want more details?
|
|
[09:09] pieterh
|
:-)
|
|
[09:09] sustrik
|
i don't understand what you meant, that's all
|
|
[09:09] ianbarber
|
is that better than just writing a 0MQ bridge in 'userland'?
|
|
[09:09] ianbarber
|
(not sure why i added scare quotes there :))
|
|
[09:10] pieterh
|
ianbarber: it's exactly the same except it can be hidden as a transport
|
|
[09:10] pieterh
|
i.e. you can plug in and plug out without changing code
|
|
[09:10] pieterh
|
whereas if it's in userland, its architecturally significant
|
|
[09:10] pieterh
|
it's FUSE by analogy
|
|
[09:10] ianbarber
|
maybe have that functionality as a device or similar
|
|
[09:11] pieterh
|
sustrik: specifics...
|
|
[09:11] pieterh
|
a transport thread has to register, using some name
|
|
[09:11] pieterh
|
that name becomes available to applications for binding and connecting
|
|
[09:11] sustrik
|
that's inproc
|
|
[09:11] pieterh
|
hang on
|
|
[09:11] pieterh
|
the transport thread then accepts a set of control messages over inproc
|
|
[09:11] pieterh
|
- bind to endpoint
|
|
[09:12] pieterh
|
- connect to endpoint
|
|
[09:12] pieterh
|
- send message
|
|
[09:12] pieterh
|
- ?
|
|
[09:12] pieterh
|
it returns events asynchronously
|
|
[09:12] pieterh
|
obviously there has to be a peer inside 0MQ able to manage these threads and events
|
|
[09:12] pieterh
|
so it's a virtualized transport
|
|
[09:13] pieterh
|
now, I can write an SSL transport really simply
|
|
[09:13] pieterh
|
or a transport that logs everything
|
|
[09:13] sustrik
|
looks like an obvious case for layering on top of 0mq
|
|
[09:13] pieterh
|
of course you can layer it on top but that's missing the point
|
|
[09:13] sustrik
|
basically, 0mq is used as a transport between the "transport" and the user
|
|
[09:14] pieterh
|
if you layer it at the bottom you get different effects
|
|
[09:14] pieterh
|
again, please understand FUSE
|
|
[09:14] pieterh
|
I can implement a file system in user space
|
|
[09:14] pieterh
|
all my tools work on that file system, since it's just a virtualized transport layer
|
|
[09:15] pieterh
|
in 0MQ jargon, my devices, brokers, etc. would automatically work over any virtualized transport
|
|
[09:15] sustrik
|
ack, what's needed is an interface from kernel back to user space
|
|
[09:15] pieterh
|
it's much, much, much cleaner than implementing the transport on top of 0MQ
|
|
[09:15] pieterh
|
yes
|
|
[09:15] sustrik
|
but we already have that :)
|
|
[09:15] sustrik
|
it's inproc
|
|
[09:15] pieterh
|
that's the transport for the interface, it's not the interface
|
|
[09:16] pieterh
|
what I want is to inject user space code inside the 0MQ API, not on top of it
|
|
[09:16] pieterh
|
but instead of going and modifying core and sending you a patch
|
|
[09:16] sustrik
|
ok
|
|
[09:16] pieterh
|
I want to write a thread that talks to your code, over inproc
|
|
[09:16] sustrik
|
have a look at 0mq internals and find a way to do that
|
|
[09:17] pieterh
|
i've just made it really easy for you to offer dozens of contributors a virtualized transport layer
|
|
[09:17] sustrik
|
ack
|
|
[09:17] pieterh
|
you know full well I'm not competent to look at the 0MQ internals and find a way to actually implement this
|
|
[09:18] pieterh
|
if you want people to invest in new transport layers, do *not* tell them to go read the code
|
|
[09:18] pieterh
|
it will not work
|
|
[09:18] pieterh
|
it's basically saying "piss off" with a smile
|
|
[09:18] sustrik
|
the point is that's it not obvious how to do it
|
|
[09:18] sustrik
|
given that internally, 0mq uses very different model
|
|
[09:18] sustrik
|
it's completely event driven
|
|
[09:18] sustrik
|
so there's no such thing as say "connect"
|
|
[09:18] pieterh
|
you can clearly make event driven APIs over inproc
|
|
[09:19] sustrik
|
it's different kind of events, those that are passed via mailbox_t
|
|
[09:19] pieterh
|
how many user contributed transports since 0MQ went fully open source?
|
|
[09:19] sustrik
|
see command.hpp
|
|
[09:19] sustrik
|
none
|
|
[09:19] sustrik
|
you don't have to convince me
|
|
[09:19] pieterh
|
and this is the reason IMO, no abstract interface
|
|
[09:20] sustrik
|
i am all for standardised transport API
|
|
[09:20] sustrik
|
i just don't know how to implement it
|
|
[09:20] sustrik
|
that's it
|
|
[09:20] pieterh
|
well, forget efficiency as such, this is about extensibility, not performance
|
|
[09:20] pieterh
|
what are the events passed via mailboxes?
|
|
[09:21] sustrik
|
internal events, such as "there's a new pipe you should start dealing with"
|
|
[09:21] pieterh
|
so how about pipes extended across inproc?
|
|
[09:21] pieterh
|
logically, I mean
|
|
[09:21] sustrik
|
the concept is that the pipe itself is agnostic about who's on the either end of it
|
|
[09:22] pieterh
|
we already have this virtualized interface internally, with mailboxes and pipes
|
|
[09:22] sustrik
|
with inproc, there's a socket on both ends
|
|
[09:22] sustrik
|
with tcp, there's a socket on one end and a session on the other
|
|
[09:22] pieterh
|
yes, but you can hide that, obviously
|
|
[09:22] sustrik
|
propose a way
|
|
[09:22] sustrik
|
it's not easy
|
|
[09:22] sustrik
|
i've spent a lot of time thinking about it
|
|
[09:22] sustrik
|
but haven't get to anything sensible so far
|
|
[09:23] pieterh
|
well, thinking by yourself is not often a very productive process...
|
|
[09:23] sustrik
|
sure, feel free to help
|
|
[09:23] pieterh
|
this is what I'm doing now
|
|
[09:23] pieterh
|
or trying to, anyhow
|
|
[09:23] sustrik
|
well, have a look at the internals first
|
|
[09:23] sustrik
|
so that you have an idea of how it works
|
|
[09:24] pieterh
|
is this documented somewhere?
|
|
[09:24] sustrik
|
source code
|
|
[09:24] pieterh
|
asking me to read C++ code simply to discuss architecture is a waste of my time, sorry
|
|
[09:24] sustrik
|
ok, let it be then
|
|
[09:25] ianbarber
|
one thing that maybe is worth considering for connectors is just sticking the files into a different directory
|
|
[09:25] pieterh
|
ianbarber: what connectors, and what files? I'm confused
|
|
[09:26] ianbarber
|
for the transports, sorry
|
|
[09:26] pieterh
|
actually I'd like to not even use inproc but IPC
|
|
[09:26] pieterh
|
so a transport can be a separate process
|
|
[09:26] ianbarber
|
so like, once i realised there would be a connector and a socket and listener, it was easier to browse around
|
|
[09:26] pieterh
|
ah, you mean organization of the 0MQ source code?
|
|
[09:26] ianbarber
|
and in terms of people building new ones, the best reference is usually other examples
|
|
[09:26] ianbarber
|
yeah
|
|
[09:27] pieterh
|
yeah, nice
|
|
[09:27] pieterh
|
so all the code for one transport is actually identifiable in one place
|
|
[09:27] ianbarber
|
yeah
|
|
[09:27] pieterh
|
excellent suggestion
|
|
[09:27] pieterh
|
send a patch to the list
|
|
[09:27] pieterh
|
:-) sorry, just being facetious
|
|
[09:28] pieterh
|
sustrik: if this all ends up with me learning C++ I'm seriously never going to forgive you
|
|
[09:28] sustrik
|
:)
|
|
[09:29] sustrik
|
here's a beginning of a design document:
|
|
[09:29] sustrik
|
http://www.zeromq.org/whitepapers:architecture
|
|
[09:29] sustrik
|
i never had time to finish it though
|
|
[09:30] ianbarber
|
oh, this looks like a good start!
|
|
[09:30] ianbarber
|
already helpful
|
|
[09:30] sustrik
|
if only it was at least 5x longer :|
|
|
[09:31] pieterh
|
it's a good start, I just clicked the 'rating +1' button
|
|
[09:31] pieterh
|
I think ianbarber's idea of moving submodules to their own directory is neat, though just using naming conventions is often simpler
|
|
[09:32] pieterh
|
i.e. zmq_tcp_whatever
|
|
[09:32] sustrik
|
the problem is that's it's a mess atm
|
|
[09:32] sustrik
|
there are pieces of tcp even in other files
|
|
[09:32] pieterh
|
yes, this is often how it happens
|
|
[09:32] sustrik
|
the consequence of not having a clean interface
|
|
[09:33] pieterh
|
part of the benefit of making a pluggable architecture is you're forced to make it clean
|
|
[09:33] sustrik
|
full-heartedly agreed
|
|
[09:33] pieterh
|
sigh, I regret not being able to help, really
|
|
[09:36] sustrik
|
np, it'll get sorted out eventually
|
|
[09:37] sustrik
|
pieterh: an idea, maybe you can move the rationale for zmq_msg_t design into the arch document?
|
|
[09:37] sustrik
|
it kind of belongs there
|
|
[09:38] pieterh
|
yes, that's where it belongs, indeed... let me do that
|
|
[09:38] pieterh
|
I will also link to the arch document from the main wiki page
|
|
[09:38] pieterh
|
we need more focus on the core codebase IMO
|
|
[09:39] sustrik
|
ok
|
|
[09:46] pieterh
|
ok, all done
|
|
[09:46] pieterh
|
I also renamed the GSoC page to be reusable
|
|
[09:50] Guthur
|
is the labs page the only source for examples of people using 0MQ in anger?
|
|
[10:04] pieterh
|
Guthur: yes, that and Google
|
|
[10:08] pieterh
|
sustrik: random idea, sorry to interrupt your accounting...
|
|
[10:09] valmac
|
no body
|
|
[10:09] valmac
|
wow! how many people
|
|
[10:09] valmac
|
good day!
|
|
[10:11] mikko--
|
hi
|
|
[10:12] pieterh
|
mikko--: hello sir
|
|
[10:20] valmac
|
poka poka
|
|
[10:36] arvind
|
hi guys need a small info regarding ZMQ_PULL queue.. anyone around
|
|
[10:38] pieterh
|
arvind: shoot...
|
|
[10:48] arvind
|
hi
|
|
[10:48] arvind
|
thanks
|
|
[10:48] arvind
|
well i have a PUSH PULL queue working great using IPCmode..
|
|
[10:49] pieterh
|
ok
|
|
[10:49] arvind
|
but when i have set up my push pull queue, the PUSH side has no ZMQ_HWM or SWAP setup
|
|
[10:49] arvind
|
and so is the PULL side..
|
|
[10:50] pieterh
|
ok
|
|
[10:50] pieterh
|
what is your question?
|
|
[10:50] arvind
|
noticed that when the PULL side gets a bit slower then PUSH side, i don't lose any messages, but the MEMORY size on the PULL side process starts increasing
|
|
[10:50] pieterh
|
yes, this is normal
|
|
[10:51] arvind
|
and the issue is the memory does not seem to come down even after the PULL side process picks up everything..
|
|
[10:51] pieterh
|
that is a side-effect of the operating system, probably
|
|
[10:51] arvind
|
as in ?? sorry did not get that..
|
|
[10:52] arvind
|
i am using freebsd by the way..
|
|
[10:52] pieterh
|
either it does not reduce VM size of process, or it does not return smaller allocations to heap
|
|
[10:52] pieterh
|
alternatively
|
|
[10:53] pieterh
|
you are not properly closing messages in your PULL code
|
|
[10:53] pieterh
|
you may have a simple leak
|
|
[10:53] arvind
|
oh ok... actually its the top command output that shows this increase in MEM
|
|
[10:53] arvind
|
well i thought so, and verified it ..
|
|
[10:53] pieterh
|
it is simple to test, run under valgrind
|
|
[10:54] pieterh
|
if that reports all memory properly freed, it is an artifact of FreeBSD and/or runtime memory management
|
|
[10:54] arvind
|
ok.. so just wanted to confirm that i had missed any flag for this kind of behavior..
|
|
[10:54] arvind
|
sorry "NOT MISSED"
|
|
[10:55] pieterh
|
nothing special, this seems to happen... there is some work going on to try to make this work better
|
|
[10:55] arvind
|
great and last question...
|
|
[10:56] arvind
|
should i set the HWM and SWAP config on the PULL side socket too ??
|
|
[10:56] pieterh
|
if you use top, note that VIRT and RES are very different
|
|
[10:57] pieterh
|
you can set HWM and SWAP on the PULL socket but I've not actually tried that to see what happens
|
|
[10:57] arvind
|
yes both VIRT and RES both seem to increase and stay there even when there are no more writes into the queue and everything in the queue is read
|
|
[10:58] pieterh
|
it doesn't look like a happy process, but I've no more info on the causes than "this happens, it's not a leak"
|
|
[10:58] arvind
|
ok
|
|
[10:58] arvind
|
hey thanks for sharing the info..
|
|
[10:59] arvind
|
will debug more and maybe update the behavior on the mailing list too..
|
|
[10:59] pieterh
|
anything you can dig up would be useful
|
|
[10:59] arvind
|
sure thanks
|
|
[11:00] onceuponatime
|
Hi everybody
|
|
[11:01] onceuponatime
|
I just downloaded the latest version of zeromq
|
|
[11:01] onceuponatime
|
and installed the php binding
|
|
[11:01] onceuponatime
|
and when loading the zmq extension the following error is logged:
|
|
[11:01] onceuponatime
|
PHP Warning: PHP Startup: Unable to load dynamic library '/Applications/MAMP/bin/php5.3/lib/php/extensions/no-debug-non-zts-20090626/zmq.so' - dlopen(/Applications/MAMP/bin/php5.3/lib/php/extensions/no-debug-non-zts-20090626/zmq.so, 9): no suitable image found. Did find:
|
|
[11:01] onceuponatime
|
/Applications/MAMP/bin/php5.3/lib/php/extensions/no-debug-non-zts-20090626/zmq.so: mach-o, but wrong architecture in Unknown on line 0
|
|
[11:01] onceuponatime
|
does anyone have any idea what is going on?
|
|
[11:02] onceuponatime
|
Im doing it in Mac OS X
|
|
[11:03] pieterh
|
not an expert in PHP on OS X but the message seems self-explanatory
|
|
[11:03] onceuponatime
|
so it is not supported for mac os x 64 bit
|
|
[11:03] pieterh
|
you're trying, it seems, to use a zmq.so built for a different architecture
|
|
[11:03] onceuponatime
|
?
|
|
[11:04] pieterh
|
I'd suggest rebuilding the PHP binding on your box
|
|
[11:04] onceuponatime
|
I built it on my box
|
|
[11:05] pieterh
|
hmm
|
|
[11:05] pieterh
|
ianbarber: ever used 0MQ/PHP on OS X?
|
|
[11:05] so_solid_moo
|
it probably just guessed the wrong arch when it build
|
|
[11:06] so_solid_moo
|
I would use 'file' to check other php extensions and see what that is compiled as
|
|
[11:09] onceuponatime
|
ok thanks, I just had a wrong reference to the pecl binnary and it took the wrong config file when installing the extension
|
|
[11:10] pieterh
|
onceuponatime: would you like to add this to the tips page?
|
|
[11:10] pieterh
|
http://zero.mq/tips
|
|
[11:11] onceuponatime
|
well, it is just use the pear && pecl files within MAMP dir
|
|
[11:12] pieterh
|
perhaps add this to the PHP binding page then
|
|
[11:13] mikko--
|
CWW scheduled for 1 April 2011: yes
|
|
[11:13] mikko--
|
ermm
|
|
[11:13] mikko--
|
onceuponatime
|
|
[11:13] mikko--
|
i got a fix for that
|
|
[11:13] mikko--
|
steve-o !
|
|
[11:14] Steve-o
|
yo
|
|
[11:14] onceuponatime
|
what fix?
|
|
[11:14] mikko--
|
onceuponatime: https://github.com/mkoppanen/php-zmq/issues/20
|
|
[11:14] mikko--
|
this reported had the same issue
|
|
[11:15] onceuponatime
|
cool, thx mikko--
|
|
[11:15] mikko--
|
i boils down that both libzmq and php extension need to built with 32bit or 64 bit
|
|
[11:15] mikko--
|
what ever matches your apache
|
|
[11:22] Steve-o
|
mikko: had me concerned I broke Solaris again there, just checked it's fine :P
|
|
[11:23] sustrik
|
re
|
|
[11:23] pieterh
|
sustrik: I'm making the zapi context wrapper threadsafe
|
|
[11:23] sustrik
|
ok
|
|
[11:24] pieterh
|
am inspired by your explanation "ØMQ's concurency model may a bit confusing at first. The reason is that we eat our own dogfood and use message passing to achieve concurrency and internal scalability."
|
|
[11:24] pieterh
|
:-)
|
|
[11:24] sustrik
|
wait a sec, context wrapper?
|
|
[11:24] pieterh
|
haha
|
|
[11:24] sustrik
|
context is thread safe
|
|
[11:24] sustrik
|
no need to add any synchronisation there
|
|
[11:24] tjgillies
|
http://draw.nostat.us
|
|
[11:24] pieterh
|
zapi's context wrapper does things like automatically close all open sockets, emulate 2.0 zmq_term if you want it, etc.
|
|
[11:25] sustrik
|
ok
|
|
[11:26] mikko--
|
Steve-o_: thanks for the tip on automake thing
|
|
[11:26] mikko--
|
works like a dream
|
|
[11:26] pieterh
|
so the approach is to start an agent thread that does the real work, and talk to it over inproc
|
|
[11:26] Steve-o
|
mikko: I saw, Google can be awesome at times
|
|
[11:26] mikko--
|
Steve-o_: i knew that something like that must exist but used the wrong search terms
|
|
[11:27] mikko--
|
i searched for something like "prevent automake install in subdirs"
|
|
[11:27] mikko--
|
came up with a lot of irrelevant results
|
|
[11:30] Steve-o
|
Your Google-fu is weak master Mikko
|
|
[11:30] Steve-o
|
I used "stop recursive make install"
|
|
[11:31] pieterh
|
heh, I love IRC
|
|
[11:31] pieterh
|
meanwhile people are still struggling with stuff like this:
|
|
[11:31] pieterh
|
"Reminder: Due to the DST clock change in the US, a couple of weeks ago, phone conference will again commence 1 hour earlier for those dialling in from Europe/Asia."
|
|
[12:02] Guthur
|
pieterh: do you have a phone conference re 0MQ?
|
|
[12:03] pieterh
|
Guthur: no, I get invited to these things, which I try to ignore
|
|
[12:03] pieterh
|
it's most definitely not 0MQ... :-)
|
|
[12:12] mikko--
|
Steve-o_: yeah, that is true
|
|
[12:12] mikko--
|
Steve-o_: anyway, thanks for the work on autoconf stuff. it makes things million times easier on zeromq side
|
|
[12:15] djc
|
I
|
|
[12:15] djc
|
'm hitting an assert in a Python client
|
|
[12:15] djc
|
I can see where the assert comes from, but not sure why it hits
|
|
[12:16] djc
|
it's in 2.0 zeromq, req.cpp:225
|
|
[12:20] sustrik
|
looks like someone is sending junk to the socket
|
|
[12:20] sustrik
|
maybe you are connecting it by an incompatible socket type?
|
|
[12:24] djc
|
mmm, pretty sure I'm not
|
|
[12:25] djc
|
ah, wait
|
|
[12:25] pieterh
|
djc: can you please post a minimal test case?
|
|
[12:25] pieterh
|
this is explained on http://www.zeromq.org/intro:ask-for-help
|
|
[12:25] djc
|
already fixed it, sorry about that
|
|
[12:25] djc
|
was referencing an endpoint of the wrong type
|
|
[12:26] djc
|
at least now my server segfaults :P
|
|
[12:26] pieterh
|
sustrik: can we not assert when getting junk on a socket?
|
|
[12:26] pieterh
|
or this has already been removed in 2.1 and later?
|
|
[12:26] djc
|
2.1 has very different code, already looked at that
|
|
[12:26] sustrik
|
there were several related fixes in 2.1
|
|
[12:27] pieterh
|
djc: any specific reason you're still using 2.0?
|
|
[12:29] djc
|
because it's the stable version
|
|
[12:30] djc
|
or, I guess, was, until two days ago :P
|
|
[12:30] djc
|
I'll see about upgrading :)
|
|
[12:30] pieterh
|
please do so, 2.1 fixes a *lot* of issues in 2.0
|
|
[12:31] djc
|
I definitely wanted to, but decided to wait until y'all would tag it stable
|
|
[12:31] pieterh
|
plus we have no intention of fixing any issues reported in 2.0 unless they are also in 2.1
|
|
[12:31] pieterh
|
ok, thanks for the explanation, was just curious
|
|
[12:47] Guthur
|
You know after the 'discussion' last night on the mailing list I started to think more about the XREP/XREQ, and I was wondering what exactly they were intended for if not for ROUTER/DEALER type patterns
|
|
[12:48] pieterh
|
Guthur: me too, and afaics routing is exactly a core mechanism for returning stuff to sender, i.e. what xrep does
|
|
[12:50] Guthur
|
I think it might be a prime example that if you expose something like this, people will use and abuse it
|
|
[12:50] pieterh
|
that was an original argument for not documenting xrep/xreq
|
|
[12:50] pieterh
|
but I don't see how to make real code without this
|
|
[12:51] pieterh
|
e.g. I'm making a request-reply chain right now, between two threads
|
|
[12:51] pieterh
|
if I use req/rep, it has to be lock-step, every request MUST send a reply
|
|
[12:51] Guthur
|
well I could possibly see some saying that you need to use a different product for routing
|
|
[12:51] pieterh
|
but using dealer/router I can send requests that don't need replies
|
|
[12:51] Guthur
|
not that I would necessarily agree just that I can see that arguement
|
|
[12:52] pieterh
|
also I can't see how user space divides into cases that want router semantics and those that want xrep semantics
|
|
[12:52] pieterh
|
that is a senseless division, it just isn't there
|
|
[12:52] pieterh
|
unless you count 100% and 0% as a division
|
|
[12:53] Guthur
|
hehe, I stillhave to keep checking back to the guide to see which is the router and the dealer
|
|
[12:53] Guthur
|
I need to write it on my hand or something
|
|
[12:53] pieterh
|
well, the one exception I can see is a simplistic req/rep device, but there are no real use cases for those
|
|
[12:53] pieterh
|
dealer is push+pull, remember that
|
|
[12:53] pieterh
|
it does nothing with envelopes
|
|
[12:54] sustrik
|
the problem is that req/rep is going to evolve
|
|
[12:54] Guthur
|
oh that bit is fine I just keep forgetting which XREQ/XREP is the DEALER/ROUTER
|
|
[12:54] sustrik
|
and can undeliberately break the other use case
|
|
[12:54] pieterh
|
sustrik: you need to clarify that, it's much too vague as an argument
|
|
[12:54] sustrik
|
cases
|
|
[12:54] pieterh
|
i agree in theory, just can't see it in practice
|
|
[12:54] sustrik
|
well, imagine the alternative routing algorithm
|
|
[12:54] pieterh
|
anything could evolve
|
|
[12:54] sustrik
|
i've described it in one email
|
|
[12:55] pieterh
|
you described something that had no design afaics...
|
|
[12:55] sustrik
|
?
|
|
[12:55] Guthur
|
the other case where REQ/REP falls down is the strict request/reply pattern
|
|
[12:55] pieterh
|
well, sorry, I didn't understand the design then
|
|
[12:55] pieterh
|
to get a reply back to a sender, across a chain, you need an address stack
|
|
[12:55] sustrik
|
the idea was to keep routing table in nodes
|
|
[12:56] sustrik
|
instead of in the message itself
|
|
[12:56] sustrik
|
similar to how IP behaves
|
|
[12:56] pieterh
|
and if a node crashes?
|
|
[12:56] sustrik
|
same as now
|
|
[12:56] sustrik
|
the message is dropped
|
|
[12:56] pieterh
|
no, now I can resend via a different route
|
|
[12:56] sustrik
|
anyway, it's probably not viable
|
|
[12:56] pieterh
|
if it's not viable, please don't raise it as a realistic case... :-(
|
|
[12:56] sustrik
|
the point, however, is that it would break req/rep used for something else than req/rep
|
|
[12:57] pieterh
|
it's all req/rep
|
|
[12:57] pieterh
|
routing is req/rep
|
|
[12:57] pieterh
|
there are no illegal semantics here
|
|
[12:57] sustrik
|
no point in discussing this
|
|
[12:57] sustrik
|
once it break, we'll have to find the solution
|
|
[12:57] pieterh
|
well, you say that when you don't want to discuss stuff
|
|
[12:57] sustrik
|
we don't have the problem yet
|
|
[12:57] pieterh
|
yet this is very sensitive to people and when you say you might break it all... you'll get discussion
|
|
[12:58] pieterh
|
I wrote two entire chapters on req-rep
|
|
[12:58] sustrik
|
well, you are asking for the problem
|
|
[12:58] sustrik
|
there's none yet
|
|
[12:58] sustrik
|
will be in the future
|
|
[12:58] Guthur
|
sustrik: we will face this problem for 3.0 though, correct?
|
|
[12:58] Guthur
|
or is it going to be deferred for later
|
|
[12:58] sustrik
|
not immediately
|
|
[12:58] pieterh
|
'there will be change in the future' is kind of ... anyhow...
|
|
[12:59] sustrik
|
well, as i said, no immediate problem
|
|
[12:59] pieterh
|
i'm all open to new routing semantics
|
|
[12:59] sustrik
|
however, separating the routing pattern from req/rep pattern would help to prevent the clash in the future
|
|
[12:59] pieterh
|
but the ones we have, are precious
|
|
[13:00] pieterh
|
sustrik: please, it's not separable
|
|
[13:00] pieterh
|
it is part of the same package... sigh
|
|
[13:00] sustrik
|
shrug
|
|
[13:00] sustrik
|
never mind
|
|
[13:00] Guthur
|
hehe, alot of circling going on
|
|
[13:00] pieterh
|
you are suggesting making divisions that won't help anyone
|
|
[13:00] pieterh
|
leave it, it works, just let the syntax get cleaner
|
|
[13:00] Guthur
|
sorry for stirring it up again
|
|
[13:00] sustrik
|
Guthur: here's some background to req/rep design
|
|
[13:00] sustrik
|
see Request/reply chapter
|
|
[13:01] pieterh
|
sustrik: I'll make the analogy with AMQP and the desire of architects to "get it right"
|
|
[13:01] pieterh
|
it takes years to collectively learn these patterns properly
|
|
[13:02] pieterh
|
if you arbitrarily put that investment into question, you will get into trouble
|
|
[13:02] pieterh
|
change has to be worth it for users
|
|
[13:02] Guthur
|
sustrik: chapter in the manual
|
|
[13:02] pieterh
|
this is the cost of success
|
|
[13:02] Guthur
|
?
|
|
[13:03] sustrik
|
Guthur: http://www.250bpm.com/hits
|
|
[13:05] pieterh
|
sustrik: one difficulty with this discussion is you are thinking 10-50 years into the future
|
|
[13:05] sustrik
|
someone has to :)
|
|
[13:05] pieterh
|
yes
|
|
[13:06] pieterh
|
but you see Guthur's question, "will this change affect 3.0?"
|
|
[13:06] pieterh
|
the timescales here, in this community, are usually weeks or months
|
|
[13:06] sustrik
|
yup
|
|
[13:07] pieterh
|
goodo, we'll embrace this tension, it's healthy
|
|
[13:07] sustrik
|
i think so
|
|
[13:07] sustrik
|
more amusing as well
|
|
[13:07] pieterh
|
just don't go do a JOH on us...
|
|
[13:07] sustrik
|
not my intention
|
|
[13:07] pieterh
|
yeah, forgive my paranoia
|
|
[13:07] sustrik
|
however, i have to have clear pattern separation to get anything sensible in the long term
|
|
[13:08] sustrik
|
btw, i've recalled you "content classes" in AMQP
|
|
[13:08] pieterh
|
I think stretching 0MQ across these scales is perhaps the most interesting problem I've seen in decades
|
|
[13:08] sustrik
|
it's basically the same idea
|
|
[13:08] pieterh
|
oh, content classes... lol
|
|
[13:08] sustrik
|
vertical separation of the stack
|
|
[13:08] pieterh
|
shot in the back of the head, thrown into the ditch of ignorance... :-)
|
|
[13:09] sustrik
|
so the idea is that there a vertical segment of the stack called req/rep
|
|
[13:09] sustrik
|
which is a dumb 1 request followed by 1 reply thing
|
|
[13:09] Guthur
|
what's a JOH?
|
|
[13:09] pieterh
|
this stops working real fast, though
|
|
[13:09] sustrik
|
not imo, but that's not the point
|
|
[13:10] pieterh
|
Guthur: a JOH is a mythical beast that creates havoc by randomly changing fundamental architecture for no good reason
|
|
[13:10] sustrik
|
the point is that's it vertically separated from different patterns
|
|
[13:10] sustrik
|
such as pub/sub
|
|
[13:10] pieterh
|
yes, agreed
|
|
[13:10] sustrik
|
now, given that people are using req/rep to do something that is not strictly speaking req/rep
|
|
[13:10] pieterh
|
ah
|
|
[13:10] pieterh
|
but router/dealer are strictly part of req/rep
|
|
[13:10] sustrik
|
i am proposing to create a new vertical slot
|
|
[13:10] pieterh
|
this is where you are purely wrong IMO
|
|
[13:11] sustrik
|
can router route two replies to a single request?
|
|
[13:11] pieterh
|
speaking as a user of 0MQ, router/dealer are essentially just async req/rep
|
|
[13:11] sustrik
|
if so, it's not "strict" req/rep
|
|
[13:11] pieterh
|
and async req/rep is the majority pattern
|
|
[13:11] sustrik
|
async is ok
|
|
[13:11] mikko--
|
pieterh: heads up: there are some things i need to clean up on the build. trying to get patch to you today
|
|
[13:11] pieterh
|
simple lock-step req/rep stops working real fast
|
|
[13:11] pieterh
|
mikko--: no hurry
|
|
[13:11] sustrik
|
what's not ok is breaking 1:1 request/reply relation
|
|
[13:12] pieterh
|
no plans to release 2.1.4 for a few days yet
|
|
[13:12] sustrik
|
anyway, back to accounting, otherwide i'll never have it ready
|
|
[13:12] pieterh
|
sustrik: in my email I listed 4-5 cases where sync req/rep just stops working
|
|
[13:12] pieterh
|
very rapidly
|
|
[13:12] pieterh
|
you cannot make real req/rep without router/dealer
|
|
[13:12] pieterh
|
thus these are intrinsically part of that vertical stack
|
|
[13:13] pieterh
|
it's like saying the car and the engine are two different products
|
|
[13:14] sustrik
|
does router guarantee that there's 1:1 relationship between requests and replies?
|
|
[13:14] sustrik
|
i suppose it does not
|
|
[13:14] pieterh
|
that relationship is not intrinsic to req-rep
|
|
[13:14] pieterh
|
does every email I send get 1 reply?
|
|
[13:14] sustrik
|
it is
|
|
[13:14] pieterh
|
not in the real world
|
|
[13:14] Guthur
|
I'm not sure if router/dealer should be so intrinsically linked to req-rep
|
|
[13:14] pieterh
|
only in some fictional world
|
|
[13:14] sustrik
|
it's the only way to allow end-to-end reliability
|
|
[13:14] sustrik
|
the reply works as an ack
|
|
[13:14] pieterh
|
end-to-end reliability is a subcase too
|
|
[13:15] pieterh
|
are you suggesting that's the only use case for req-rep?
|
|
[13:15] pieterh
|
what if the ack is lost?
|
|
[13:15] sustrik
|
timeout and resend
|
|
[13:15] pieterh
|
so heartbeating
|
|
[13:15] sustrik
|
kind of
|
|
[13:15] pieterh
|
look, I'm an expert in this, really
|
|
[13:15] pieterh
|
and you cannot do it with synchronous endpoints
|
|
[13:15] sustrik
|
where reply works as a heartbeat
|
|
[13:16] pieterh
|
you may find other patterns and socket types
|
|
[13:16] sustrik
|
you can, but it has to be done inside 0mq rather than on top of it
|
|
[13:16] pieterh
|
but today, realistic req-rep needs those four socket types
|
|
[13:16] pieterh
|
actually it barely needs req and rep at all
|
|
[13:16] pieterh
|
router/dealer is the real core of request-reply
|
|
[13:16] sustrik
|
yes, because it's unfinished
|
|
[13:17] pieterh
|
you speak like an artist sometimes
|
|
[13:17] sustrik
|
well, the resend thing is missing in REQ socket
|
|
[13:17] pieterh
|
the issue here is brains learning patterns, and the cost of that
|
|
[13:18] pieterh
|
it's about people, not technicalities
|
|
[13:18] sustrik
|
exactly
|
|
[13:18] pieterh
|
you publish ideas, people invest decades in learning them
|
|
[13:18] pieterh
|
and then you say, "aha, but it's unfinished! "
|
|
[13:18] pieterh
|
I'm not a fan
|
|
[13:18] pieterh
|
there is a lot missing in REQ
|
|
[13:19] Guthur
|
I'm confused why REQ/REP keeps getting drawn into it, that socket type does a job and works well, the issue is people also want a socket that provides less restrictive send/recv and control over message routing
|
|
[13:19] sustrik
|
i am just a human being, i cannot finish everything at once
|
|
[13:19] Guthur
|
or am I missing something
|
|
[13:19] pieterh
|
sustrik: then don't threaten change no-one is asking for...
|
|
[13:19] pieterh
|
if you didn't want people to use router/dealer you had to make req/rep actually work in real code
|
|
[13:20] pieterh
|
since they don't, people have had no choice except to learn and use router/dealer
|
|
[13:20] sustrik
|
ack
|
|
[13:20] pieterh
|
that is now the state of the art
|
|
[13:20] pieterh
|
I'd much prefer not having four socket types to learn
|
|
[13:20] pieterh
|
but this is where we are today
|
|
[13:20] pieterh
|
it works, it's usable, it's actually elegant enough to defend
|
|
[13:20] sustrik
|
the problem is that aside of legitimate uses of xreq/xrep such as building relianility
|
|
[13:21] sustrik
|
people use it in a following way
|
|
[13:21] sustrik
|
i have a XREP socket
|
|
[13:21] sustrik
|
i connect three apps to it
|
|
[13:21] pieterh
|
I don't think you actually know how people use it
|
|
[13:21] sustrik
|
i set an identity to each app, say A, B and C
|
|
[13:21] pieterh
|
the terminology "legitimate" is like an artist
|
|
[13:21] sustrik
|
when i want to send message to A, i pushlish A+payload
|
|
[13:21] pieterh
|
there is no illegitimate use, only bad design
|
|
[13:21] sustrik
|
well, it doesn't scale
|
|
[13:22] pieterh
|
repeat after me, when users misuse a product, it's the designer's fault
|
|
[13:22] pieterh
|
always
|
|
[13:22] sustrik
|
right
|
|
[13:22] pieterh
|
this is a key tenet of good design
|
|
[13:22] sustrik
|
mea culpa
|
|
[13:22] sustrik
|
i should have left xreq/xrep undocumented
|
|
[13:22] sustrik
|
too late now
|
|
[13:22] pieterh
|
that was not an option, you know it... sigh
|
|
[13:22] pieterh
|
let me explain a simple router use case I am making right now
|
|
[13:23] Guthur
|
sustrik: is that routing to specific clients bad in your eyes?
|
|
[13:23] sustrik
|
pieterh: sure
|
|
[13:23] sustrik
|
Gurthur: yes, because it's not scalable
|
|
[13:23] pieterh
|
so in fact setting identities is a Bad Thing with some very few exceptions
|
|
[13:23] sustrik
|
yes
|
|
[13:24] sustrik
|
i should not have introduced them at all
|
|
[13:24] pieterh
|
there are some necessary use cases
|
|
[13:24] pieterh
|
I've found two, actually
|
|
[13:24] sustrik
|
well, they give an impression of "reliability" in AMQP sense
|
|
[13:24] Guthur
|
sustrik: but surely it's essential in some cases unless you use some different layer to expose to clients
|
|
[13:24] pieterh
|
persistent subscribers, and ... hehe... router-to-router routing
|
|
[13:24] sustrik
|
whereas, in reality, 0mq is more like TCP
|
|
[13:24] sustrik
|
transient in nature
|
|
[13:25] pieterh
|
indeed, reliable network buffers is just weird
|
|
[13:25] sustrik
|
the mix of the two is pretty awkward
|
|
[13:25] pieterh
|
I agree
|
|
[13:25] pieterh
|
I'd vote to kill explicit identities
|
|
[13:25] pieterh
|
except that one use case absolutely needs them, and it's safe
|
|
[13:25] pieterh
|
distributed name service, see Ch4
|
|
[13:25] pieterh
|
but this is divergence
|
|
[13:25] sustrik
|
that would be nice
|
|
[13:26] pieterh
|
my current use case is using a thread to isolate an object, the context wrapper in zapi
|
|
[13:26] sustrik
|
the code for identities is a nightmare
|
|
[13:26] sustrik
|
a pile of special cases basically :(
|
|
[13:26] pieterh
|
yes, we can IMO aim to remove that functionality
|
|
[13:26] pieterh
|
especially if we provide layers on top that do the same only better
|
|
[13:26] pieterh
|
e.g. Clone pattern for reliable pub/sub
|
|
[13:26] pieterh
|
which I've made, and which is really easy finally
|
|
[13:27] pieterh
|
so I have threads that send commands to this object thread
|
|
[13:27] pieterh
|
it's what you do in 0MQ core, right
|
|
[13:27] pieterh
|
now commands are async
|
|
[13:27] pieterh
|
only one or two have replies
|
|
[13:27] pieterh
|
if I use req/rep, I have to artificially send a reply to each request
|
|
[13:27] pieterh
|
I did that
|
|
[13:27] pieterh
|
it's horrid
|
|
[13:27] pieterh
|
using dealer/router I can do it right, reply only when needed
|
|
[13:28] sustrik
|
yes, sure there are use cases for it
|
|
[13:28] pieterh
|
the reply address has to be delivered to my code because the reply can happen later
|
|
[13:28] pieterh
|
it can't be 'reply to previous request'
|
|
[13:29] sustrik
|
the main question imo is what 0mq should provide
|
|
[13:29] pieterh
|
just this
|
|
[13:29] sustrik
|
there are two mindsets:
|
|
[13:29] pieterh
|
what it provides *works*
|
|
[13:29] sustrik
|
1. generic networking
|
|
[13:29] sustrik
|
such as ACE, Asio etc.
|
|
[13:29] sustrik
|
2. scalability
|
|
[13:29] pieterh
|
both, obviously
|
|
[13:29] pieterh
|
that's a false dichotomy
|
|
[13:29] sustrik
|
well, the second means restricting unscalable behaviour
|
|
[13:29] pieterh
|
I want scalable generic networking
|
|
[13:29] sustrik
|
no way
|
|
[13:30] pieterh
|
yes, way :-)
|
|
[13:30] Guthur
|
hehe, I think that's the issue, we all like making connections with 0MQ we want to use it everwhere
|
|
[13:30] pieterh
|
the trick is generic networking at multiple scales
|
|
[13:30] sustrik
|
it does not
|
|
[13:30] pieterh
|
i.e. same semantic for inproc as for clusters
|
|
[13:30] pieterh
|
yes, it does
|
|
[13:30] sustrik
|
see the A, B, C example
|
|
[13:30] sustrik
|
it does not scale
|
|
[13:30] sustrik
|
if there's a lot of clients
|
|
[13:30] pieterh
|
ah, I see what you mean
|
|
[13:30] pieterh
|
look, you have this kind of backwards IMO
|
|
[13:30] sustrik
|
you cannot simply interject a device to scale it up
|
|
[13:31] pieterh
|
allow me to explain
|
|
[13:31] sustrik
|
it's fixed 1-hop architecture
|
|
[13:31] pieterh
|
hang on :-)
|
|
[13:31] pieterh
|
speaking as a 0MQ user, who knows about scale,...
|
|
[13:31] pieterh
|
I don't really expect, or need, my architectures to scale without some redesign
|
|
[13:31] pieterh
|
however I want that redesign to be cheap
|
|
[13:32] pieterh
|
and I really, really want one set of semantics at all scales
|
|
[13:32] pieterh
|
e.g. I know that multicast is not just click-click
|
|
[13:32] pieterh
|
it's a *lot* of extra work
|
|
[13:32] pieterh
|
but my code will basically not change (too much)
|
|
[13:32] pieterh
|
this is the key to 0MQ
|
|
[13:33] pieterh
|
it's not that apps magically scale just by adding devices and IMO it's not helpful to aim for that
|
|
[13:33] pieterh
|
it's that you can re-architect code really easily to make it fit arbitrary scales
|
|
[13:33] sustrik
|
well, i am aiming for that
|
|
[13:33] pieterh
|
I can give a real example from Ch4
|
|
[13:33] pieterh
|
where this works beautifully
|
|
[13:33] pieterh
|
I make a whole reliability design, based on a broker, right?
|
|
[13:34] sustrik
|
broker is inherently inscalable
|
|
[13:34] pieterh
|
then at the end, like a magic trick, I move the broker into the client and it disappears
|
|
[13:34] sustrik
|
when you want to add another broker
|
|
[13:34] sustrik
|
you run into AMQP-style problems
|
|
[13:34] pieterh
|
and I get full scalability across an N-to-N network of clients and workers
|
|
[13:34] pieterh
|
this is what works
|
|
[13:34] sustrik
|
ah, so it's the device interhjection pattern
|
|
[13:34] pieterh
|
federating brokers is just how banks waste money :-)
|
|
[13:34] pieterh
|
it's the "rescale my code" pattern
|
|
[13:34] sustrik
|
yes, that the only viable way to scale imo
|
|
[13:35] sustrik
|
adding hops
|
|
[13:35] sustrik
|
without changing semantics
|
|
[13:35] pieterh
|
this is why these socket types cannot be hidden
|
|
[13:35] sustrik
|
yes, there's good reason to keep them visible
|
|
[13:35] sustrik
|
however, they can be used only in certain ways
|
|
[13:35] pieterh
|
this is true
|
|
[13:35] sustrik
|
to keep the overall design scalable
|
|
[13:35] pieterh
|
it's about education
|
|
[13:35] sustrik
|
well, that's my point
|
|
[13:35] pieterh
|
i.e. please drive on the right
|
|
[13:36] pieterh
|
however it is justifiable to build unscalable designs
|
|
[13:36] pieterh
|
because it's easy to refactor them as needed
|
|
[13:37] pieterh
|
e.g. starting with a broker is just much simpler, conceptually
|
|
[13:37] pieterh
|
to be honest, I think 0MQ is already magically good at this
|
|
[13:38] sustrik
|
yeah, i haven't expected it to be this universal
|
|
[13:38] sustrik
|
back in 2007
|
|
[13:38] sustrik
|
back to serious stuff!
|
|
[13:38] pieterh
|
yeah, it was not really the initial vision :-)
|
|
[13:38] pieterh
|
accounting! code!
|
|
[13:41] pieterh
|
mikko--: ping
|
|
[13:42] pieterh
|
if you clone https://github.com/zeromq/zeromq2-1 and then try to build with openpgm, you get an error
|
|
[13:42] pieterh
|
config.status: error: cannot find input file: `Makefile.in'
|
|
[13:51] pieterh
|
mikko--: np, solved it I think
|
|
[13:53] Steve-o
|
:P
|
|
[13:53] Steve-o
|
That error occurs when using a openpgm svn snapshot without bootstrap
|
|
[13:54] Steve-o
|
for 114 and 115 I now bootstrap Autoconf
|
|
[13:55] Steve-o
|
only annoying bit is autogen.sh not creating the pgm staging directory
|
|
[13:55] pieterh
|
Steve-o_: weird thing is the missing file is zeromq's root file, not in openpgm
|
|
[13:55] pieterh
|
though the message may be inaccurate
|
|
[13:56] Steve-o
|
more importantly you didn't read Bob's response, it's already working for him
|
|
[13:56] pieterh
|
orly?
|
|
[13:56] Steve-o
|
your last two mails are out of date
|
|
[13:56] Steve-o
|
:P
|
|
[13:57] pieterh
|
nope, his last email (that I have) says, "Can you verify that it configures properly and it's me and not the tarball?"
|
|
[13:57] Steve-o
|
lol, ok two to me :D
|
|
[13:57] pieterh
|
maybe he replied to you offlist
|
|
[13:57] Guthur
|
pieterh: we drive on the left
|
|
[13:59] Steve-o
|
I reproduced the error here: http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/010041.html
|
|
[14:01] pieterh
|
Guthur: thanks?
|
|
[14:08] cremes
|
i'd like to ask a question about 3.0; any chance the project will get a major refactoring
|
|
[14:08] cremes
|
and reorg so that the internal classes can be fully tested?
|
|
[14:09] cremes
|
right now the only testing we can do is 'black box' style where we give it some input and hope
|
|
[14:09] cremes
|
that all the plumbing works well enough to get an answer on the other side
|
|
[14:09] Guthur
|
cremes: there was a discussion earlier about something related
|
|
[14:09] Guthur
|
more to ease the understanding of 0MQ core
|
|
[14:09] cremes
|
Guthur: what was the outcome of the discussion? or is it still ongoing?
|
|
[14:10] Guthur
|
'yeah that would be nice' hehe
|
|
[14:10] sustrik
|
cremes: can be done, but it requires a significant effor writing unit tests
|
|
[14:10] sustrik
|
exactly, 'yeah that would be nice'
|
|
[14:11] sustrik
|
if you want to help with that we can give it a try
|
|
[14:12] pieterh
|
cremes: what about writing unit tests on top, not in core?
|
|
[14:12] pieterh
|
any refactoring of core is going to be expensive
|
|
[14:13] pieterh
|
I know it's not the same...
|
|
[14:14] pieterh
|
I'm using systematic unit tests in zapi and it's a really nice way of working
|
|
[14:20] mikko--
|
Steve-o_: thats my fault
|
|
[14:20] mikko--
|
i got a patch soon(ish) for it
|
|
[14:22] mikko--
|
mkoppanen/zeromq2-1 has a branch fix-make-dist
|
|
[14:22] mikko--
|
can you test that?
|
|
[14:23] Steve-o
|
k
|
|
[14:25] Guthur
|
pieterh: imo retrospectively adding adequate unit test coverage is usually difficult or next to impossible
|
|
[14:25] Guthur
|
the code really needs to be design with it in mind from the beginning
|
|
[14:25] pieterh
|
Guthur: I believe you're right, and it's not going to be possible to refactor core to make this happen
|
|
[14:26] pieterh
|
not without major, major effort, i.e. manyears of work
|
|
[14:26] pieterh
|
so what we need is a separate tests project IMO
|
|
[14:27] pieterh
|
where anyone can bung a test case
|
|
[14:27] pieterh
|
and which we can use when doing daily builds
|
|
[14:28] Steve-o
|
mikko--: looks fine, just finishing the build
|
|
[14:29] Steve-o
|
well zeromq should go for two test beds, an external one sitting above the apis and one living next to the code testing each class
|
|
[14:30] pieterh
|
Steve-o_: indeed, but the second case is probably impossible to make
|
|
[14:30] pieterh
|
and right now the external one is part of core, which is a problem IMO
|
|
[14:30] Steve-o
|
depends, have you looked at Chromium unit tests? They're pretty fruity
|
|
[14:31] Steve-o
|
mine too for OpenPGM can only go so far
|
|
[14:31] Steve-o
|
mikko--: all done, no problems encountered.
|
|
[14:32] mikko--
|
make dist works as well?
|
|
[14:32] Steve-o
|
yup
|
|
[14:33] mikko--
|
if anyone has boxes that can execute network tests they can be plugged into the central jenkins cluster
|
|
[14:33] Steve-o
|
I'm thinking of investigating upgrading my network testing to use Python and zeromq
|
|
[14:34] Steve-o
|
the bit issue with network testing is stopping and cleaning up when things go wrong
|
|
[14:34] Steve-o
|
s/bit/big/
|
|
[14:34] mikko--
|
you mean going with too high rates?
|
|
[14:35] Steve-o
|
usually dead locks
|
|
[14:35] Steve-o
|
I have a few years ago suffered quite funny network problems
|
|
[14:35] Steve-o
|
i.e. too high data rate, literally millions of packets per second
|
|
[14:35] Steve-o
|
turned the switch into a hub and floored the entire network
|
|
[14:35] pieterh
|
mikko--: did you see drbobbeaty's offer of OpenPGM testing space?
|
|
[14:35] Steve-o
|
fcuk.
|
|
[14:36] drbobbeaty
|
pieterh: it's actually an offer of financial support - for ZMQ to have it's own OpenPGM testing space so we don't have these same kinds of issues in the future.
|
|
[14:37] pieterh
|
drbobbeaty: please meet Mikko, who is starting his own ZMQ support company
|
|
[14:37] drbobbeaty
|
pieterh: it could be machines, if that's all that's needed, but my original intention was to help fund an expansion to the testing facilities.
|
|
[14:38] pieterh
|
It would be excellent if your organization would fund an expansion of the testing facilities via mikko--
|
|
[14:38] drbobbeaty
|
mikko-: OK, if you're the guy, great. I think we want the same thing - a tested tarball for users. I'm just ready to see what it'll take and try to swing that through the company I work for.
|
|
[14:39] Steve-o
|
I might be able to move some of my hardware in the future
|
|
[14:39] Guthur
|
I don't mind offering some capacity on my VPS if that is what is being talked about
|
|
[14:39] pieterh
|
I think what we're talking about is two things,
|
|
[14:40] pieterh
|
a. buying or finding some PGM-capable switches
|
|
[14:40] Steve-o
|
I have a ibm blade chassis and a ultra sparc which need new homes when I move to NY
|
|
[14:40] pieterh
|
b. setting up and managing a test process
|
|
[14:41] pieterh
|
drbobbeaty: I think funding would have to cover time as well as materials
|
|
[14:42] pieterh
|
it would not be huge amounts, but mikko-- is a professional
|
|
[14:42] pieterh
|
his firm is at http://kuut.io/
|
|
[14:42] drbobbeaty
|
pieterh: I'm not against that. I benefit from ZMQ, and me not having to spend a day debugging a new release is worth real $$ to my employer.
|
|
[14:42] pieterh
|
indeed
|
|
[14:42] pieterh
|
it's win-win-win
|
|
[14:43] pieterh
|
I suggest you can discuss with Mikko privately, I'm available if you need me, but this does not need to involve iMatix
|
|
[14:44] drbobbeaty
|
pieterh: you and mikko chat - I really don't care who does it, who owns it, etc. When you were here on your talk, you talked about support contracts. We're not against that - towards getting the testing as part of this.
|
|
[14:44] drbobbeaty
|
I'd just like someone that's going to stay in the project for a long time be the one we fund because I'd hate to be in this situation in 6 months.
|
|
[14:45] pieterh
|
ok, so there are two ways to approach this
|
|
[14:45] drbobbeaty
|
like I said - I'm very easy on this. You know what I want... you let me know if there's a way I can help get us all there.
|
|
[14:45] pieterh
|
either, you fund work done by the community, which means working with Mikko, who is the owner of all that is testing
|
|
[14:46] pieterh
|
or, you ask iMatix for a proper liong
|
|
[14:46] pieterh
|
*long-term support contract, which we're more than happy to do, and we take care of everything
|
|
[14:48] sustrik
|
mikko: nice website
|
|
[14:48] sustrik
|
the design, i mean
|
|
[14:54] sustrik
|
pieterh: accepting a new functionality into stable without testing it on master of elsewhere is not a good idea imo
|
|
[14:55] pieterh
|
sustrik: you mean openpgm?
|
|
[14:55] sustrik
|
yup
|
|
[14:55] sustrik
|
btw, it's still not in master
|
|
[14:55] sustrik
|
:(
|
|
[14:55] pieterh
|
true, but basically it was broken before
|
|
[14:56] sustrik
|
well, it worked for drbobbeaty
|
|
[14:56] pieterh
|
i'm not sure why it's not yet in master... just waiting for a patch from mikko?
|
|
[14:56] sustrik
|
yes
|
|
[14:56] pieterh
|
ah, rc2 worked, yes
|
|
[14:56] pieterh
|
it's not easy to know 'new functionality' vs 'necessary patch' in foreign imports
|
|
[14:57] pieterh
|
the solution is proper testing of daily builds off github
|
|
[14:57] Steve-o
|
you get both :D
|
|
[14:57] cremes
|
sorry about not responding to the unit test stuff an hour ago; we had a little outage here that i had to attend to
|
|
[14:57] pieterh
|
mostly, I expect the release process to self-stabilize
|
|
[14:57] pieterh
|
this is my experience
|
|
[14:57] cremes
|
so i agree that refactoring core would be many man-years of effort
|
|
[14:58] cremes
|
but i think the idea of having some unit tests that live "underneath" the public api and test internal classes
|
|
[14:58] cremes
|
is a good idea
|
|
[14:58] cremes
|
i am willing to write some code (i.e. put my money/time where my mouth is)
|
|
[14:59] cremes
|
i think that over a period of time as unit tests are written, we'll find the pain points that require redesign/refactoring
|
|
[14:59] sustrik
|
pieterh: pushing _anything_ untested into stable is no good
|
|
[14:59] cremes
|
of the internal classes and it can be done piece-meal
|
|
[14:59] pieterh
|
sustrik: it's only testable in master now because it's the same codebase
|
|
[14:59] sustrik
|
cremes: well, do you have an idea of a specific unit test you want to write
|
|
[14:59] pieterh
|
as soon as there's functional changes, that stops being possible
|
|
[14:59] sustrik
|
then test it elsewhere
|
|
[15:00] pieterh
|
this is not a problem, really
|
|
[15:00] cremes
|
sustrik: i do; i thought a good initial set of tests would be to exercise the yqueue class since it is
|
|
[15:00] cremes
|
so important to internal communications
|
|
[15:00] sustrik
|
pieterh: it's up to you, but unstable stable is a problem afaics
|
|
[15:00] Steve-o
|
cremes: I think you're jumping into a different conversation
|
|
[15:00] sustrik
|
cremes: sounds good
|
|
[15:00] pieterh
|
sustrik: you are stressed because you want a perfect product, but that is not how it happens
|
|
[15:01] sustrik
|
if you write one unit test we can have a look at how to incorporate it into the codebase
|
|
[15:01] cremes
|
Steve-o_: just answering a direct query from sustrik; but yeah, the channel has a few topics going simultaneously :)
|
|
[15:01] pieterh
|
cremes: it sounds like a good plan
|
|
[15:01] cremes
|
i will endeavor to do a proof-of-concept this weekend so we can discuss on monday/tuesday
|
|
[15:02] pieterh
|
sustrik: the other option, which I wanted to discuss, was to start a 2.2 release cycle
|
|
[15:02] sustrik
|
what would that be?
|
|
[15:02] pieterh
|
well, in strict principle, no new functionality in 2.1 at all...
|
|
[15:02] pieterh
|
so 2.2 allows us to add new functionality without changing the api
|
|
[15:03] sustrik
|
well, if you feel like maintaining 3 separate repos :)
|
|
[15:03] Steve-o
|
pieterh: re: "it's not easy to know 'new functionality' vs 'necessary patch' in foreign imports", new functionality would be a openpgm 6.x, I only push small changes down
|
|
[15:03] pieterh
|
i've no problem with that, I maintain 20-30 repositories
|
|
[15:03] sustrik
|
wait till you start doing real backports
|
|
[15:03] sustrik
|
:)
|
|
[15:04] pieterh
|
sustrik: somewhere along the line you missed the trick that as maintainer I don't do ports at all
|
|
[15:04] pieterh
|
look, let me give you an example...
|
|
[15:04] pieterh
|
2.1 will be used in production for a very long time
|
|
[15:04] pieterh
|
now say you want to deliver some new socket options to users
|
|
[15:05] pieterh
|
nothing dangerous
|
|
[15:05] pieterh
|
but if you only provide them via 3.0, it'll be ages before you get proper feedback
|
|
[15:05] pieterh
|
so you convince me, as maintainer, to release them in 2.2
|
|
[15:05] pieterh
|
everyone wins
|
|
[15:05] sustrik
|
if it's fully backward compatible it can go into 2.1
|
|
[15:05] pieterh
|
nope
|
|
[15:05] pieterh
|
it cannot
|
|
[15:06] sustrik
|
up to you
|
|
[15:06] Steve-o
|
I thought the definition was new features = minor++, API breakage = major++?
|
|
[15:06] pieterh
|
the point is that 2.1 is "stable" and thus does not get new functionality
|
|
[15:06] pieterh
|
Steve-o_: exactluy
|
|
[15:06] pieterh
|
new functionality = new minor release
|
|
[15:06] Guthur
|
pieterh: only a day or two ago you mentioned there would be no 2.2, iirc
|
|
[15:06] pieterh
|
Guthur: I said, we had no plans
|
|
[15:06] pieterh
|
plans change and develop
|
|
[15:07] Guthur
|
so if there was a 2.2 would that be 4 main repos of 0MQ
|
|
[15:07] Guthur
|
your 3 and sustrik's one
|
|
[15:07] pieterh
|
Guthur: which is fine, seriously
|
|
[15:08] pieterh
|
it is scalable
|
|
[15:08] pieterh
|
it's perhaps unusual but it's very simple, predictable, and scalable
|
|
[15:08] pieterh
|
plus there are dozens of 0MQ repos, in fact
|
|
[15:09] pieterh
|
we have a distributed development process and the repos map that
|
|
[15:09] pieterh
|
there is no "main"
|
|
[15:09] pieterh
|
there is only the rather more valuable repos maintained by El Sustrik
|
|
[15:09] Guthur
|
Sure. I'll have to trust you on it to be honest
|
|
[15:09] pieterh
|
and the cheap & nasty imitations made by other folk
|
|
[15:10] pieterh
|
sustrik: do you plan to develop 3.0 inside the zeromq2 repository?
|
|
[15:11] sustrik
|
depends on how painful it is to rename it
|
|
[15:11] sustrik
|
i assume people have links to the repo
|
|
[15:11] pieterh
|
millions of internet links break
|
|
[15:11] pieterh
|
there is no forwarding
|
|
[15:11] sustrik
|
yeah
|
|
[15:11] sustrik
|
:|
|
|
[15:11] pieterh
|
however, everyone watching the old site still watches the new one
|
|
[15:11] pieterh
|
so my idea was to rename it, and then make a new zeromq2 with a cover page
|
|
[15:12] mikko--
|
pieterh: saw it
|
|
[15:12] pieterh
|
i mentioned this already, you may recall
|
|
[15:12] mikko--
|
havent had time to respond yet
|
|
[15:12] pieterh
|
mikko--: ok
|
|
[15:12] sustrik
|
dunno whather it's worth of the trouble
|
|
[15:12] sustrik
|
it's just a name anyway
|
|
[15:13] pieterh
|
sigh... you want to break an entire naming scheme just because "it's just a name?"
|
|
[15:13] pieterh
|
how on earth can we manage a family of repositories if the names don't make sense?
|
|
[15:14] sustrik
|
let's find out what exactly would break
|
|
[15:14] sustrik
|
obviosuly, existing web links
|
|
[15:14] pieterh
|
like I said, that can be fixed
|
|
[15:14] sustrik
|
mikko's auto builds
|
|
[15:14] pieterh
|
did you understand my proposal?
|
|
[15:14] sustrik
|
rename
|
|
[15:14] pieterh
|
and recreate
|
|
[15:14] sustrik
|
no way
|
|
[15:15] pieterh
|
rename and recreate
|
|
[15:15] pieterh
|
yes, obviously
|
|
[15:15] sustrik
|
that's kicking out 350+ people watching the repo
|
|
[15:15] pieterh
|
sigh...
|
|
[15:15] pieterh
|
please assume I'm not an idiot
|
|
[15:15] pieterh
|
you rename, the watchers follow along to the new zeromq3 repo, ok?
|
|
[15:15] pieterh
|
everything is cool
|
|
[15:16] pieterh
|
now you create a *new* zeromq2 repo to take the place of the old one
|
|
[15:16] sustrik
|
yuck
|
|
[15:16] sustrik
|
sorry, no
|
|
[15:16] pieterh
|
you make a cover page, which provides a map for visitors
|
|
[15:16] pieterh
|
all internet links work
|
|
[15:16] pieterh
|
this is far better than doing 3.0 work in a repo called zeromq2, seriously
|
|
[15:17] pieterh
|
"we were afraid of renaming it, that's why it's still weird in 2015, sorry"
|
|
[15:17] pieterh
|
better, rename it now to 'zeromq'
|
|
[15:18] pieterh
|
when we get to making stable releases of 3.0, there will be repos like zeromq3-0
|
|
[15:18] pieterh
|
based off zeromq2?????
|
|
[15:19] sustrik
|
i am not against renaming
|
|
[15:19] pieterh
|
it's so bad I'm tempted to take my ball and stop playing
|
|
[15:19] sustrik
|
just re-creating is a nonsense
|
|
[15:19] pieterh
|
you're against making a replacement cover page?
|
|
[15:19] pieterh
|
you prefer broken links?
|
|
[15:19] pieterh
|
shrug, I have no more to offer
|
|
[15:19] sustrik
|
rename zeromq2->zeromq
|
|
[15:19] sustrik
|
then do whatever with zeromq2 namespace
|
|
[15:20] sustrik
|
that way the watchers remain subscribed
|
|
[15:20] sustrik
|
no point in annying 350 people
|
|
[15:20] pieterh
|
how is it possible to say, "yuck no way:" while agreeing with my proposal?
|
|
[15:20] sustrik
|
you said, "recreate"
|
|
[15:20] sustrik
|
recreate what?
|
|
[15:21] pieterh
|
my fault for not communicating this more clearly
|
|
[15:21] pieterh
|
please excuse me, I'll try again
|
|
[15:21] pieterh
|
a. you rename the repo, watchers are not annoyed, ack?
|
|
[15:22] sustrik
|
yes
|
|
[15:22] pieterh
|
b. there is now no repo called 'zeromq2', ack?
|
|
[15:22] sustrik
|
ah, you want to re-create that one?
|
|
[15:22] sustrik
|
as a strawman
|
|
[15:22] pieterh
|
the word 'recreate' seems pretty clear
|
|
[15:22] pieterh
|
yes, create an empty repo with the old name
|
|
[15:23] sustrik
|
why i thouth you were suggesting was "re-create the master"
|
|
[15:23] pieterh
|
and fill it with one page, a cover sheet that helps people navigate the new structure
|
|
[15:23] sustrik
|
ok
|
|
[15:23] sustrik
|
that makes sense
|
|
[15:23] pieterh
|
well, you assume I'm stupid, maybe
|
|
[15:23] pieterh
|
don't know...
|
|
[15:23] pieterh
|
it's ok
|
|
[15:23] pieterh
|
:-)
|
|
[15:23] sustrik
|
will do, once i move to 3.0
|
|
[15:24] pieterh
|
if we do rename the repo, I'd consider the new name carefully
|
|
[15:24] pieterh
|
it's very important to get this right
|
|
[15:24] sustrik
|
zeromq?
|
|
[15:24] sustrik
|
zeromq-master
|
|
[15:24] pieterh
|
hang on
|
|
[15:24] sustrik
|
zeromq-dev
|
|
[15:24] sustrik
|
?
|
|
[15:24] sustrik
|
dunno
|
|
[15:24] pieterh
|
core
|
|
[15:24] pieterh
|
or libzmq
|
|
[15:24] sustrik
|
core sounds more like a component rather than branch
|
|
[15:25] pieterh
|
it's not a branch, at all
|
|
[15:25] pieterh
|
it is *the* living project within the 0MQ community
|
|
[15:25] pieterh
|
there are two issues I see here
|
|
[15:25] pieterh
|
a. confusion between zeromq as a community, and zeromq as a library
|
|
[15:25] sustrik
|
good point
|
|
[15:25] pieterh
|
b. zeromq as a package will end up including multiple projects
|
|
[15:26] pieterh
|
my vision of the packaging is we package the community
|
|
[15:26] sustrik
|
yup
|
|
[15:26] pieterh
|
and that matches the organization name, zeromq
|
|
[15:26] sustrik
|
libzmq+pyzmq+clrzmq etc.
|
|
[15:26] pieterh
|
yes
|
|
[15:26] pieterh
|
exactly
|
|
[15:26] pieterh
|
so this is also an explanation why we have multiple repositories
|
|
[15:27] pieterh
|
it is only possible to increase the scope of packaging if we have separate projects for that
|
|
[15:27] sustrik
|
libzmq would make sense then, however, i am not sure whether it's worth of renaming everything now
|
|
[15:27] pieterh
|
I'd really enjoy seeing libzmq, to be honest
|
|
[15:27] sustrik
|
it would be nice
|
|
[15:28] pieterh
|
I'd rename zapi to libzapi in the same sense then
|
|
[15:28] sustrik
|
the question is how annoying would the renaming be
|
|
[15:28] pieterh
|
it's going to have to happen anyhow
|
|
[15:28] pieterh
|
and this is very helpful to people IMO
|
|
[15:28] pieterh
|
change that makes things better is always acceptable
|
|
[15:29] sustrik
|
the benefit should be weighted against all the annoyance the rename will reate
|
|
[15:29] sustrik
|
create
|
|
[15:30] sustrik
|
actually, current naming doesn't necessarily clash with package naming
|
|
[15:30] pieterh
|
we won't know the annoyance until we've done the rename
|
|
[15:30] sustrik
|
the name of the project on github is kind of invisible to end users
|
|
[15:30] pieterh
|
yes
|
|
[15:31] sustrik
|
it's of interest only to devs
|
|
[15:31] pieterh
|
we need to check that forks of the repo don't break when it's renamed
|
|
[15:31] pieterh
|
forks and watches, that's the real value
|
|
[15:31] sustrik
|
+1
|
|
[15:32] pieterh
|
and IMO if we recreate a useful map for visitors who follow the old link, it's solved
|
|
[15:32] pieterh
|
I'd like to make a map of the community anyhow
|
|
[15:32] pieterh
|
every project
|
|
[15:32] pieterh
|
with a person and a URI
|
|
[15:33] pieterh
|
ok, we can come back to this when you start on 3.0
|
|
[15:36] Guthur
|
pieterh: are you building whaleshark at the moment?
|
|
[15:36] pieterh
|
Guthur: in a manner of speaking, I'm working on the API that I'll need to build Whaleshark
|
|
[15:36] pieterh
|
parts are almost done, like the name service
|
|
[15:36] Guthur
|
is that related to MDP
|
|
[15:37] pieterh
|
well, the name service is the Freelance pattern described at the end of Ch4
|
|
[15:37] pieterh
|
the client does broker-style work but the workers are simpler than MDP workers
|
|
[15:37] pieterh
|
it's distributed client-server
|
|
[15:38] pieterh
|
you don't really want to access a name service via a broker, too heavy
|
|
[15:39] Guthur
|
just reading the protocol now
|
|
[15:47] Guthur
|
sorry to bring this up again, I'm having trouble seeing how explicitly routing to client A or B or C... goes against scalability, you could surely use a DEALER at the client side which load-balances across a number of ROUTER based devices
|
|
[15:47] Guthur
|
am I missing something
|
|
[15:47] pieterh
|
Guthur: the point is using explicit identities
|
|
[15:48] pieterh
|
if you're using automatic identities, and a stack, you don't route to a client A, B, or C
|
|
[15:48] pieterh
|
but rather you route backwards along a path
|
|
[15:48] pieterh
|
which is scalable
|
|
[15:49] pieterh
|
does that help?
|
|
[15:49] pieterh
|
e.g. in MDP, we never use our own identities, only those provided by 0MQ
|
|
[15:50] pieterh
|
it's scalable
|
|
[15:50] Guthur
|
ok, but if you take same routing pattern with 0MQ generated UUIDs it is fine?
|
|
[15:51] pieterh
|
well, I see two issues
|
|
[15:51] pieterh
|
one, using durable sockets is not scalable, as we
|
|
[15:51] pieterh
|
*we've learned
|
|
[15:52] pieterh
|
two, treating an envelope as 1 frame rather than a path is probably not scalable
|
|
[15:52] pieterh
|
probably, but not necessarily
|
|
[15:53] pieterh
|
I've not actually seen real use cases for req-rep with more than 2 hops
|
|
[15:54] Guthur
|
yeah I think outside the req-rep pattern tbh
|
|
[15:54] pieterh
|
sustrik's whitepaper does present some but they seem contrived to me
|
|
[15:54] Guthur
|
it's in relation to my endeavours with FIX, this is very much a disjointed async communication
|
|
[15:55] Guthur
|
you may send off 1 request and async recv N replies
|
|
[15:55] pieterh
|
right
|
|
[15:55] Guthur
|
So the X sockets are the only way to go really
|
|
[15:56] Guthur
|
without maybe using multiple sockets
|
|
[16:14] pieterh
|
sp|ke: the sub client will reconnect automatically
|
|
[16:14] pieterh
|
but it will lose data obviously
|
|
[16:54] cremes
|
sp|ke: no, you can't do that with req/rep when using blocking send/recv; you have to create your own timeout
|
|
[16:54] cremes
|
and use non-blocking send/recv along with zmq_poll() to detect a timeout and to handle that situation
|
|
[19:09] mikko
|
drbobbeaty: did you manage to test the new version of openpgm?
|
|
[19:10] drbobbeaty
|
mikko: yup, sure did. Works fine.
|
|
[19:10] mikko
|
cool
|
|
[19:10] drbobbeaty
|
mikko: I used 0MQ 2.1.3 and the 5.1.115 OpenPGM from Steve-o
|
|
[19:11] mikko
|
the --with-pgm=pgknamehere ?
|
|
[21:14] drbobbeaty
|
mikko: yes, sorry I didn't see it earlier, I used the name of the tarball in the config line.
|
|
[21:16] mikko
|
drbobbeaty: cool
|
|
[21:16] mikko
|
that feature proved to be useful quite quickly
|
|
[21:23] pieterh
|
mikko: the new openpgm packaging is really nicely done, congrats
|
|
[21:25] mikko
|
pieterh: i will send pull request for the make dist fixes
|
|
[21:25] mikko
|
and clear hudson job workspaces so that it will build from empty
|
|
[21:27] pieterh
|
mikko: we still need to get this into the libzmq core repository
|
|
[21:27] pieterh
|
sustrik's repo
|
|
[21:27] mikko
|
yes
|
|
[21:27] mikko
|
as soon as it works properly i'll get it ready for el sustrik
|
|
[21:28] pieterh
|
I figured out (I'm so thick) how to pull individual commits across repos
|
|
[21:28] mikko
|
cherry pick?
|
|
[21:28] pieterh
|
don't need to fetch and cherry-pick
|
|
[21:28] pieterh
|
gah
|
|
[21:29] pieterh
|
just define the remote and then use 'git pull' with the commit number
|
|
[21:29] pieterh
|
the precise syntax is explained by github when you have a pull request
|
|
[21:29] pieterh
|
in your face the whole time, so to speak
|
|
[21:29] mikko
|
yeah
|
|
[21:30] mikko
|
been thinking about this situation for some time
|
|
[21:30] pieterh
|
github pull requests are just issues that wrap manual git commands
|
|
[21:30] mikko
|
pull request effectively isn't much more than github thing around pulling from other request
|
|
[21:30] mikko
|
yes
|
|
[21:30] pieterh
|
exactly
|
|
[21:30] mikko
|
we could use the underlying mechanism but just make the requests via mailing-list
|
|
[21:31] pieterh
|
exactly
|
|
[21:31] pieterh
|
that's what I'd like to aim for
|
|
[21:31] pieterh
|
precisely this:
|
|
[21:31] pieterh
|
git checkout -b mkoppanen-fix-make-dist master
|
|
[21:31] pieterh
|
git pull https://github.com/mkoppanen/zeromq2-1.git fix-make-dist
|
|
[21:31] pieterh
|
git checkout master
|
|
[21:31] pieterh
|
git merge mkoppanen-fix-make-dist
|
|
[21:32] pieterh
|
to take your last pull request as example
|
|
[21:33] mikko
|
very simple process
|
|
[21:33] pieterh
|
it seems so
|
|
[21:33] pieterh
|
now we just have to convince el sustrik to try it ... :-)
|
|
[21:35] pieterh
|
let me document this quickly on the contributing page...
|
|
[21:46] Guthur
|
pieterh, could you quickly take a look at the issue this guy describes https://github.com/zeromq/clrzmq/issues/11#comment_909819
|
|
[21:46] pieterh
|
Guthur: ack, on it...
|
|
[21:46] Guthur
|
Would it be possible it is a slow subscriber issue
|
|
[21:46] Guthur
|
I understand there is very little in the way of concrete details
|
|
[21:47] Guthur
|
just your gut feeling would be suffice
|
|
[21:50] pieterh
|
Guthur: do you expose the zmq_msg_init_data call in your API?
|
|
[21:50] pieterh
|
gut feeling is he's sending stuff that he meanwhile frees
|
|
[21:53] pieterh
|
Guthur: ok, comment added
|
|
[22:06] Guthur
|
pieterh, sorry was afk there
|
|
[22:06] Guthur
|
my flatmates made some food
|
|
[22:06] pieterh
|
curry?
|
|
[22:06] Guthur
|
pizza and pasta
|
|
[22:07] pieterh
|
sounds excellent
|
|
[22:07] pieterh
|
and you seriously still want to discuss 0MQ? :-)
|
|
[22:07] Guthur
|
it was
|
|
[22:07] Guthur
|
well I knew you might be answer so felt bad leaving so quickly
|
|
[22:07] pieterh
|
hehe
|
|
[22:07] Guthur
|
I'm just about to go back for tea an chocolates
|
|
[22:08] pieterh
|
go for it, I'm writing some docs
|
|
[22:08] Guthur
|
ok, back in a bit
|
|
[23:31] Guthur
|
Hi jeckstein, did you raise the issue for clrzmq2 on github
|
|
[23:31] jeckstein
|
yes i did, hi michael
|
|
[23:31] Guthur
|
are you using the latest version of clrzmq2 from github?
|
|
[23:32] jeckstein
|
i'm going to try to come up with a simple scenario to reproduce the error
|
|
[23:32] Guthur
|
that would be best
|
|
[23:32] Guthur
|
best isolating the issue much easier
|
|
[23:33] jeckstein
|
i'm not actually, i'm using the clrzmq2 v2.0.0
|
|
[23:33] jeckstein
|
since you said all it did was add the newer socket options, i didn't want to introduce too many new components into my app
|
|
[23:33] Guthur
|
jeckstein, there was actually a couple of bug fixes after that
|
|
[23:34] Guthur
|
are you using polling?
|
|
[23:35] jeckstein
|
i'm not setting polling explicitly
|
|
[23:36] Guthur
|
ok, best to get some code together so I can run an identical test to try and reproduce the bug
|
|
[23:37] jeckstein
|
ok, I'll update my clrzmq2
|
|
[23:39] jeckstein
|
guthur, are you familiar with reading windows memory dump files?
|
|
[23:40] Guthur
|
to be honest no
|
|
[23:41] jeckstein
|
no problem, i was able to get the dump but couldn't get too much out of it..one thing I did see interesting was that part of the stack trace for the error included 2 back to back calls to zmq_stopwatch_stop
|
