[Time] Name | Message |
[04:52] motk
|
howdy people
|
[04:52] snerd
|
would there be any interest in adding a zeromq mirror?
|
[04:52] snerd
|
I'm running a large academic mirror and am looking for decent content
|
[04:53] guido_g
|
i don't think that the hit rate on the web-site makes this necessary :)
|
[04:54] guido_g
|
but you should ask the imatix guys
|
[04:54] snerd
|
heh, worth a shot
|
[04:55] snerd
|
wish I could've been using 0mq back at my last investment banking job
|
[04:55] guido_g
|
same here for the tradings system project i'vew been on last year :)
|
[04:56] guido_g
|
ibm wllm is a beast
|
[04:56] snerd
|
cough mutter 'alladin' cough mutter
|
[04:56] snerd
|
before that it was mqseries
|
[04:56] snerd
|
guh
|
[04:58] guido_g
|
oh btw, there discussions going to redesign the web site, so if you're interested stay tuned
|
[04:59] snerd
|
we're already mirror apachemq, rabbitmq, amqp stuff, so might as well
|
[04:59] guido_g
|
as i said, imatix is taking care of that
|
[04:59] snerd
|
nod
|
[05:00] guido_g
|
the whole thing is a wikidot page
|
[05:00] snerd
|
eep
|
[05:25] sustrik
|
snerd: we've actually talked about where to host the downloads
|
[05:25] sustrik
|
uploading them to the wiki is a bit weird
|
[05:26] sustrik
|
as for the mirroring, why not
|
[05:26] sustrik
|
it won't hurt anybody
|
[05:27] sustrik
|
Blafasel_: everything is FIFO in 0MQ
|
[05:27] snerd
|
if anyone is up for it, hit me at rob.kearey@aarnet.edu.au
|
[05:29] sustrik
|
ok
|
[05:43] CIA-20
|
rbzmq: 03Martin Sustrik 07master * re742398 10/ rbzmq.c : PUSH and PULL constants added - http://bit.ly/ctHC2I
|
[05:49] CIA-20
|
clrzmq: 03Martin Sustrik 07master * rcf394ea 10/ clrzmq/zmq.cs : PUSH and PULL constants added - http://bit.ly/9G3Gqh
|
[07:49] mrm2m
|
guido_g: Rückmeldung zur Seite: Der "all languages"-button unter den Beispielen ist genial!
|
[07:50] guido_g
|
great
|
[07:51] mrm2m
|
Sotty
|
[07:51] mrm2m
|
Sorry, did't recorgnize it's english here
|
[07:56] guido_g
|
np
|
[07:57] guido_g
|
for info about crosscompiling you might want have a lookt at the maling list
|
[09:05] mrm2m
|
just an Idea: You already use JS for this All languages. Why not replace the content of the corresponding example in place? A set Language as default would be cool, too.
|
[09:38] mrm2m
|
Test of pyzmq failes: http://paste.pocoo.org/show/256240/
|
[09:38] mrm2m
|
Any ideas?
|
[09:39] mrm2m
|
/usr/local/lib/python2.6/dist-packages/zmq/_zmq.so looks good - as far as I can tell
|
[09:41] mrm2m
|
oh - never mind. I didn't recognize that python was looking in "/home/moritz/GEU/pyzmq/zmq/" instead of "/usr/local/lib/python2.6/dist-packages/zmq/"
|
[09:47] guido_g
|
re
|
[09:53] mrm2m
|
Hi, I just ran into this problem: http://blog.boxedice.com/2010/05/23/building-zeromq-and-pyzmq-on-red-hat/
|
[09:54] mrm2m
|
It's the same for Ubuntu. Solution worked, but isn't pretty.
|
[09:55] guido_g
|
no probs on original debian
|
[09:56] mrm2m
|
http://mail.scipy.org/pipermail/ipython-dev/2010-March/005900.html
|
[09:58] guido_g
|
so?
|
[09:58] guido_g
|
what's the problem?
|
[10:01] mrm2m
|
Nothing. Just an info, that this topic still exists and also affects Ubuntu
|
[10:10] CIA-20
|
zeromq2: 03Martin Sustrik 07master * rfba90af 10/ src/socket_base.cpp : Issue 54 - socket_base.cpp:162 comparison error - http://bit.ly/9M5Xs2
|
[10:23] guido_g
|
ahh, ryanair again? :)
|
[10:24] pieterh
|
guido_g: yeah :-)
|
[10:24] guido_g
|
so good luck, take care
|
[10:24] pieterh
|
http://new.zeromq.org is something I hacked up this morning
|
[10:25] pieterh
|
this would become www.zeromq.org and the current www. would become dev.zeromq.org
|
[10:25] guido_g
|
nice
|
[10:25] pieterh
|
sustrik and i will strip it down until it's minimalistic
|
[10:26] guido_g
|
but for more input you should ask mrm2m, he's the new kid on the block :)
|
[10:27] jond
|
pieter: looks better. I am beginning to see yr point about the colours
|
[10:27] pieterh
|
the strong red/black works for simple sites but seems painful for link-rich text
|
[10:28] sustrik
|
the guide?
|
[10:28] pieterh
|
?
|
[10:28] pieterh
|
that's a question?
|
[10:28] pieterh
|
sorry., have to leave, cyl
|
[10:28] sustrik
|
cyl
|
[10:29] sustrik
|
ugh, the voting box hurts eyes
|
[10:30] guido_g
|
the box won't be in the final version, i assume
|
[10:32] sustrik
|
dunno, but it should be of that poisonous pink hue
|
[10:35] guido_g
|
at least, it *does* catch attention :)
|
[10:37] sustrik
|
guido_g: have a look at www.zeromq.org now
|
[10:37] sustrik
|
i've removed the red
|
[10:37] sustrik
|
pretty dull
|
[10:38] guido_g
|
can't see the links anymore
|
[10:38] sustrik
|
they should be grey
|
[10:38] guido_g
|
links should stand out a little
|
[10:39] sustrik
|
ah, you see them as black?
|
[10:39] guido_g
|
no grey
|
[10:39] guido_g
|
but it's hard to read
|
[10:40] sustrik
|
ok, reverting it back to red
|
[10:41] jond
|
what would the whole background look like in that grey?
|
[10:41] sustrik
|
it would make the text less readable imo
|
[10:42] sustrik
|
the contrast is good
|
[10:42] sustrik
|
people just complained about the red
|
[10:43] jond
|
0MQ logo looks very close to helvetica, does it look any good in black
|
[10:47] sustrik
|
not really, and it's not even helvetica
|
[10:47] sustrik
|
it's custom-designed font :)
|
[10:52] mrm2m
|
OMG!
|
[10:53] mrm2m
|
No, I like the design.
|
[10:53] mrm2m
|
The TOC could be a little bit more structured.
|
[10:53] mrm2m
|
But so far I found everything I was looking for.
|
[10:59] sustrik
|
mrm2m: thanks for feeback
|
[10:59] sustrik
|
yes, it's not about content missing, it's about messy structure
|
[11:00] mrm2m
|
Well - The Navigation looks a litttle bit like a tag cloud.
|
[11:01] mrm2m
|
A lots of people have trouble communicating with a cloud, you know ;.)
|
[11:01] keffo
|
Personally I find dancing to be a great way!
|
[11:02] sustrik
|
:)
|
[11:07] keffo
|
sustrik, I just replied to the lb thread :)
|
[11:38] jond
|
I still think the user comments and blog stuff should go and the text from the what's 0MQ spread two columns
|
[11:38] jond
|
and there's too much red
|
[11:53] sustrik
|
jond: yes, that kind of stuff, but it's more about ergonomy
|
[11:53] sustrik
|
90% of visitors want to solve a problem rather then browse around
|
[11:54] sustrik
|
and the page is not particularly well designed to help you solve your proble quickly and efficiently
|
[11:54] sustrik
|
joel spolsky had few nice articles on this topic
|
[11:55] mrm2m
|
by the way: Got it running at my embedded ARM platform now.
|
[11:55] guido_g
|
woot! :9
|
[11:55] mrm2m
|
I will give some summery, of what I've done If you like
|
[11:56] guido_g
|
isn't there a wiki page for this kind of stories?
|
[11:56] sustrik
|
mrm2m: please, do post it on the mailing list
|
[11:56] sustrik
|
to make it available
|
[11:57] sustrik
|
the wiki would be a good choice if it's a howto rather than story
|
[11:58] guido_g
|
ok
|
[12:14] mrm2m
|
The user-guide is NSFW. My boss is looking strange, while I lay down in laughter.
|
[12:16] sustrik
|
:)
|
[13:16] theICEBear
|
Hi I wonder if anyone knows about an issue with sessions in zmq. I am hitting an assert in session::process_attach line 287
|
[13:21] sustrik
|
theICEBear: what version is that?
|
[13:21] theICEBear
|
2.0.8
|
[13:21] sustrik
|
zmq_assert (!engine);
|
[13:21] theICEBear
|
Yup
|
[13:21] sustrik
|
are you using identities?
|
[13:22] sustrik
|
ZMQ_IDENTITY?
|
[13:22] theICEBear
|
I am trying to set my own yeah
|
[13:23] theICEBear
|
I am probably setting them in the wrong place then
|
[13:23] sustrik
|
i think what's happening is that there are two peers trying to connect to the same socket with the same identity
|
[13:24] sustrik
|
this case is not yet handled decently
|
[13:24] theICEBear
|
peers as in?
|
[13:25] sustrik
|
the applications over the network
|
[13:25] theICEBear
|
Ah, thanks for the tip I will check
|
[13:28] CIA-20
|
zeromq2: 03Martin Sustrik 07master * rf0a36f9 10/ doc/zmq_cpp.txt : Minor patch to zmq_cpp(7) - http://bit.ly/bfgJOm
|
[14:29] sustrik
|
pieterh: are you in bts already?
|
[14:46] pieterh
|
sustrik: ano, ja jestem z bratislava
|
[14:50] pieterh
|
sorry, i was using polish notation, it should be "som w bratislave"
|
[14:50] pieterh
|
0MQ/5.0 will have Slavic language bindings
|
[15:05] CIA-20
|
zeromq2: 03Martin Sustrik 07master * re45583c 10/ src/semaphore.hpp : OSX build fixed -- semaphore replaced by mutex - http://bit.ly/9AYi6Y
|
[15:05] sustrik
|
pieteh: cool
|
[15:05] sustrik
|
just w is spelled v in slovak
|
[15:06] sustrik
|
pieterh: in the office?
|
[15:08] pieterh
|
sustrik: i'm with mato, we're debuggine weird pubsub issues
|
[15:09] pieterh
|
*debugging
|
[15:09] sustrik
|
ack
|
[15:09] sustrik
|
i leave you in peace then
|
[18:21] Steve-o
|
anyone see the new LBM 4,1 features from 29 West?
|
[18:21] Steve-o
|
zero copy for c# api
|
[18:26] Steve-o
|
surprised at the choice of terminology, I can't tell whether it is official MS API or out of someones orifice,
|
[18:26] Steve-o
|
"Zero Object Delivery (ZOD) has been implemented for .NET, which allows .NET messaging receivers to deliver messages to an application with no per-message object creation."
|
[18:26] Steve-o
|
"Zero Incoming Copy (ZIC) has been implemented for .NET, which provides access to message data directly through a byte pointer returned by the LBMMessage.dataPointer() method."
|
[18:32] sustrik
|
nope, haven't seen it yet
|
[18:32] sustrik
|
what's the link?
|
[18:44] Steve-o
|
marketing email, looks like most detail is here, http://blogs.informatica.com/perspectives/index.php/2010/08/27/lbm-4-1-ume-3-1-umq-1-1-released/
|
[18:46] guido_g
|
zero object delivery sounds good :)
|
[18:48] Steve-o
|
and presumably if 29West are using it it must show some improvement somewhere
|
[18:49] guido_g
|
sorry, forgot the sarcasm tags
|
[18:49] Steve-o
|
:P
|
[18:49] pieterh
|
Lol
|
[18:49] guido_g
|
zero objects usccessfully delivered
|
[18:49] Steve-o
|
you could use a zero object for confirmation reply of delivery
|
[18:50] Steve-o
|
or just heart beating
|
[18:50] pieterh
|
ZOD & ZIC vs ZMQ
|
[18:50] pieterh
|
OMQ!
|
[18:50] guido_g
|
Steve-o: depends on the protocol information in this "zero object"
|
[18:51] Steve-o
|
guido_g: in PGM you can delivery 0 byte messages too
|
[18:52] guido_g
|
as i said, depends on the information in the protocol if this is useful or not
|
[18:52] guido_g
|
one can't send 0 bytes, so there *is* something on the wire
|
[18:53] guido_g
|
heartbeat allone isn't that useful
|
[18:53] Steve-o
|
would be a good idea for virtual circuits
|
[18:53] guido_g
|
but if you attach the seq. no. of the last data packet, we start to have something
|
[18:57] guido_g
|
afair, wllm docs didn't say a word about zero copy for the bindings
|
[18:58] sustrik
|
i like this: http://www.29west.com/products/gateway.html
|
[18:58] sustrik
|
it looks like 29west have independently arrived at the concept of "device"
|
[18:59] sustrik
|
that kind of proves the design
|
[18:59] guido_g
|
hehe
|
[19:00] Steve-o
|
reads like P2PS, i.e. a TIC without caching
|
[19:10] rossij
|
I have a requirement for a single port to be opened between the agent and the server. The agent needs to be able to connect the server and send messages on demand, but the server needs to be able to supply updates, config changes, other data to all the agents over the established connection. I don't see a simple way of doing this without using a REQ/REP and a second SUB/PUB with zeromq. Does anyone have a suggestions on how to do this w
|
[19:10] rossij
|
ZeroMQ?
|
[19:12] pieterh
|
rossij: it works better when you separate the different types of message flow
|
[19:12] pieterh
|
request-response is not publish-subscribe
|
[19:12] pieterh
|
in old-fashioned tcp you would mix these all on one port
|
[19:12] pieterh
|
but that becomes more complex and less scalable
|
[19:12] rossij
|
pieterh: I wish i could it would be sooo much simpler, but two ports is goign to be a no go
|
[19:12] pieterh
|
imagine you want to supply updates to some components across a multicast bust
|
[19:12] pieterh
|
*bus
|
[19:13] pieterh
|
why is two ports a no go?
|
[19:13] rossij
|
pieterh: it's what current ly is in place in a huge number of installations.
|
[19:14] pieterh
|
this is across a LAN?
|
[19:14] rossij
|
pieterh: zeromq plus two ports would make this simple
|
[19:14] pieterh
|
in any case you will need to migrate components to a new stack
|
[19:15] pieterh
|
presumably hidden inside whatever framework you use
|
[19:15] pieterh
|
one, two, 10 ports... it does not change that
|
[19:15] pieterh
|
existing components cannot talk to 0MQ without having 0MQ linked with them
|
[19:16] rossij
|
understand that I am working to rip on the udp network code and put in zeromq for tcp/multicast and its just so nice to work with
|
[19:16] pieterh
|
sure
|
[19:17] pieterh
|
do you understand that the number of ports you end up using is not a problem
|
[19:17] pieterh
|
unless you have firewall issues
|
[19:17] pieterh
|
or someone who really, really hates you opening new ports
|
[19:18] rossij
|
yes - i understand that - inside the firewall it's all ok - but - for devices and installs in the DMZ and other areas
|
[19:18] pieterh
|
right...
|
[19:18] pieterh
|
what I'd look at is bridging this into the DMZ using a device
|
[19:19] pieterh
|
there is no literature yet on how to tunnel 0MQ connections over 0MQ but IMO it's possible
|
[19:20] rossij
|
that is something I will have to try. makes the setup a little more complex but could work
|
[19:20] pieterh
|
i'd say, a pair of devices that create a tunnel
|
[19:20] pieterh
|
one inside the DMZ, one on the LAN
|
[19:21] pieterh
|
XREQ/XREP between the two
|
[19:21] Steve-o
|
ideally would be nicer to tunnel it all over http, easier to firewall
|
[19:21] rossij
|
humm anyone know if TCP virtual services / virtual endpoints is being worked on ?
|
[19:21] pieterh
|
yes, devices could be bridges, speaking HTTP at one edge, 0MQ at the other
|
[19:21] pieterh
|
it's an idea for 0MQ/3.0
|
[19:21] pieterh
|
i assume you mean subports kind of thing?
|
[19:21] Steve-o
|
is there any http/zeromq gateway?
|
[19:22] rossij
|
pieterh: yes
|
[19:22] pieterh
|
Steve-o: not as far as I know
|
[19:22] pieterh
|
rossij: yes, 0MQ/3.0 concept
|
[19:22] pieterh
|
see http://www.zeromq.org/docs:3_0
|
[19:22] pieterh
|
"TCP virtual services / virtual endpoints"
|
[19:22] pieterh
|
oh... :-)
|
[19:23] pieterh
|
you quoted that page
|
[19:23] pieterh
|
lol
|
[19:23] rossij
|
yeah that is where I got - i just noticed and hoped - some might move to 2.1 :)
|
[19:23] pieterh
|
it might be moved to 2.1
|
[19:23] sustrik
|
Steve-o: 0MQ/HTTP gateway => mongrel2
|
[19:24] pieterh
|
sustrik: not really... kind of but no...
|
[19:24] pieterh
|
it uses 0MQ to implement a higher-level API
|
[19:24] sustrik
|
ok
|
[19:24] sustrik
|
i've did some thinking about the virtual endpoints
|
[19:25] sustrik
|
subports
|
[19:25] rossij
|
pieterh: thanks will keep a watch out - and see what i can do myself
|
[19:25] pieterh
|
rossij: IMO a pair of tunneling devices will be quite easy to make, if you don't try to be generic
|
[19:26] sustrik
|
the basic concept is simple, the only implementation problem i see is how to share a single TCP listener socket between all the sockets and i/o threads
|
[19:26] pieterh
|
sustrik: use 0MQ messaging, internally... :-)
|
[19:26] sustrik
|
it would require non-trivial work to solve all the synchronisation issues
|
[19:27] Steve-o
|
the recent google work on shared sockets?
|
[19:27] sustrik
|
pieterh: actually, that's what 0mq is doing :)
|
[19:27] pieterh
|
so a socket with virtual endpoints becomes a device, no?
|
[19:27] sustrik
|
inter-thread synchronisation messages are passed using same code that is used to pass real 0mq messages
|
[19:27] pieterh
|
sustrik: neat
|
[19:28] sustrik
|
what you are speaking about is tunnel
|
[19:28] sustrik
|
which is doable,sure
|
[19:28] sustrik
|
but it's nasty
|
[19:28] sustrik
|
you get head of line blocking problem etc.
|
[19:29] sustrik
|
real solution is to have real separate connections
|
[19:29] sustrik
|
sharing same source port
|
[19:29] sustrik
|
sorry for the rant btw
|
[19:30] sustrik
|
i have another one about zero-copy
|
[19:30] Steve-o
|
:D
|
[19:30] sustrik
|
:)
|
[19:30] sustrik
|
well, there are interesting things i've learned hard way with zero copy
|
[19:30] sustrik
|
i can summarise it in an email
|
[19:31] sustrik
|
anyone interested in that?
|
[19:31] pieterh
|
sustrik: if you have a good rant, make it a blog posting
|
[19:31] sustrik
|
it stems from you \x00 email, so it would be logical to reply
|
[19:32] pieterh
|
lol, ok, go for it...
|
[19:32] sustrik
|
goodo
|
[19:32] pieterh
|
note that i only mentioned this because some poor user hit the door with his nose
|
[19:33] sustrik
|
i know, the problem is that it's user feature that has profound implementation impact on the whole stack
|
[19:33] sustrik
|
basically the feature would be "stretched" all the way down to kernel/userspace boundary
|
[19:34] sustrik
|
and -- if kernel support is added one day -- even further down into kernel networking stack
|
[19:34] sustrik
|
which is pretty nasty
|
[19:35] pieterh
|
you're saying...
|
[19:35] pieterh
|
that allocating one extra byte
|
[19:35] pieterh
|
means the end of the universe...
|
[19:36] pieterh
|
:-)
|
[19:36] sustrik
|
kind of
|
[19:36] sustrik
|
it's not obvious
|
[19:36] sustrik
|
but it's a trap i;ve already fell in
|
[19:36] pieterh
|
i understand that when you do zerocopy on read, it's not possible to mess with what you provide
|
[19:36] sustrik
|
wasted weeks of coding that way
|
[19:36] pieterh
|
claro
|
[19:37] pieterh
|
and we always do zero copy on read?
|
[19:38] sustrik
|
yes
|
[19:38] sustrik
|
no point in having 2 stacks
|
[19:38] sustrik
|
zero-copy and non-zero-copy
|
[19:38] pieterh
|
fair enough, it obviously does not work with zerocopy reads
|
[19:39] pieterh
|
for some reason i was imagining we allocated and copied on recv unless the app provided a buffer
|
[19:39] pieterh
|
i'm stupid
|
[19:39] sustrik
|
i've did the same mistake
|
[19:39] sustrik
|
but there was noone to stop me back then :)
|
[19:40] pieterh
|
lol
|
[19:40] sustrik
|
i've wasted a lot of time, but otoh the multi-part messages emerged from the mess
|
[19:41] Steve-o
|
it's very conducive for memory managment
|
[19:41] sustrik
|
which are actually a pretty good way to retain zero copy while still being able to modify messages
|
[19:41] pieterh
|
indeed, multipart messages are a clever solution
|
[19:41] sustrik
|
don't work for the terminal zero though
|
[19:41] sustrik
|
shrug
|
[19:44] sustrik
|
pieterh: btw i though of writing little pieces of implementation experience down somehow
|
[19:44] sustrik
|
so that it's not lost
|
[19:44] sustrik
|
maybe a mino-blog?
|
[19:44] sustrik
|
mini
|
[19:47] jond
|
sustrik: think that would be an excellent idea. as i have discovered with investigating pub/sub filtering the code is quite terse and clever in places but can take quite a while to get into the headspace of 0mq internals and why it is written like it is.
|
[19:48] sustrik
|
jond: yes, that's exactly my point
|
[19:48] sustrik
|
there are comments in the code
|
[19:48] sustrik
|
but that missed the big picture
|
[19:48] sustrik
|
well, the larger picture
|
[19:49] sustrik
|
actually, it would be great if you could help with it
|
[19:49] sustrik
|
you had to dissect the pub/sub code
|
[19:49] sustrik
|
so you know what the stumbling blocks there are
|
[19:50] jond
|
hmm, my patmatch branch is so full of cout calls for me to figure out what was going on, i can barely find the code i changed now ;-)
|
[19:50] sustrik
|
i mean, what were the things that were not obvious immediately
|
[19:50] sustrik
|
what have taken you a long time to figure out
|
[19:50] sustrik
|
etc.
|
[19:51] sustrik
|
i can only guess what's hard and what's not
|
[19:51] sustrik
|
anyway, it's just an idea
|
[19:52] jond
|
i'll try and get some notes down, the comments you have just made about zero copy lead me to think i better study that stuff a bit more
|
[19:52] sustrik
|
ok. good
|
[19:53] jond
|
that prefix tree find, was because i have my own prefix tree now and i needed to be sure that i hadnt introduced the bug.
|
[19:54] sustrik
|
that's what peer review is for :)
|
[19:55] jond
|
I would still prefer on the pub side to be able to use bits rather than refcount bytes. each sub socket connect results in a single write pipe on the pub side
|
[19:55] jond
|
that is the case?
|
[19:56] sustrik
|
yes
|
[19:56] sustrik
|
yes, probably it can be compressed to a bit
|
[19:57] sustrik
|
depends on how the subscription forwarding algorithm looks like
|
[19:57] sustrik
|
if you have a single bit
|
[19:57] sustrik
|
you have to assure that same subscription won't be sent twice over the same "logical connection"
|
[19:57] jond
|
that is what i think, if write-pipe == sub endpoint effectively
|
[19:58] sustrik
|
sure, try it that way
|
[19:58] sustrik
|
we'll see what emerges
|
[19:59] jond
|
think i might package up that pub matcher class and see it for review. it seems to get the right answers
|
[19:59] sustrik
|
sure
|
[19:59] jond
|
s/see/send/g
|
[19:59] sustrik
|
btw
|
[19:59] sustrik
|
the bitmap thing
|
[20:00] sustrik
|
i've seen a performance study on exactly this problem
|
[20:00] sustrik
|
and the interesting point there was that compressing the bitmaps pays of
|
[20:00] jond
|
url?
|
[20:00] sustrik
|
not public yet i think
|
[20:00] sustrik
|
in any case
|
[20:01] sustrik
|
the point was that the bitmaps tend to be very sparse
|
[20:01] sustrik
|
and often pretty large
|
[20:01] jond
|
i was wondering if dreppers memory paper might have any useful info.
|
[20:01] sustrik
|
that way you hit cpu cache boundary very soone
|
[20:01] sustrik
|
and your performance goes down by order of magnitude
|
[20:02] sustrik
|
maybe even two orders of magnitude
|
[20:02] sustrik
|
jond: it's definitely worth of reading
|
[20:02] sustrik
|
but pretty long
|
[20:02] jond
|
wrt the bitmap, i think that it will be very sparse and large and that is the problem. that's also why i worry about the vectors of refcount, n pipes wide at every node in the trie....
|
[20:02] jond
|
in the current implementation
|
[20:03] sustrik
|
right
|
[20:03] sustrik
|
so what about compressing the vector
|
[20:03] sustrik
|
making it say a vector of indices
|
[20:03] sustrik
|
hm
|
[20:03] sustrik
|
ordered vector of indices
|
[20:03] sustrik
|
that way merge is O(n)
|
[20:05] jond
|
it might be easier if i were to send the code. let me go look. need to be sure we are talking about the same thing...
|
[20:06] sustrik
|
sure send it, review is good
|
[20:06] pieterh
|
jond: how large are the bitmaps?
|
[20:06] sustrik
|
what i said was that instead of bit vector 0000100001001001
|
[20:06] sustrik
|
you can have a vector of indices like this: 5|10|13|16
|
[20:07] pieterh
|
sustrik: we tested this quite extensively when making openamq
|
[20:07] pieterh
|
we used compressed bit maps in SFL
|
[20:07] sustrik
|
and?
|
[20:07] sustrik
|
it must have been much faster for large amount of subscribers, no?
|
[20:07] pieterh
|
if you use fast bit operations, uncompressed is faster
|
[20:08] sustrik
|
how large bitmaps have you tested?
|
[20:08] pieterh
|
8k bytes
|
[20:08] pieterh
|
iirc
|
[20:08] sustrik
|
too small
|
[20:08] pieterh
|
let me check, it's in IPR
|
[20:08] sustrik
|
the performance hit comes what cache size is hit
|
[20:08] pieterh
|
so how large are the bitmaps?
|
[20:08] sustrik
|
several MB these days
|
[20:09] sustrik
|
number of topics x number of subscribers
|
[20:09] pieterh
|
?
|
[20:09] pieterh
|
surely you have one bitstring per topic
|
[20:09] sustrik
|
es
|
[20:09] sustrik
|
yes
|
[20:09] pieterh
|
that's the 8k
|
[20:09] sustrik
|
what 8k?
|
[20:10] pieterh
|
you asked how large our bitmaps were, I said 8K
|
[20:10] pieterh
|
that is one bitstring
|
[20:10] sustrik
|
ah
|
[20:10] pieterh
|
equals 64k subscribers
|
[20:10] pieterh
|
one bitstring per topic
|
[20:10] pieterh
|
or per expression in OpenAMQ
|
[20:10] sustrik
|
how sparse was the bitmap?
|
[20:10] pieterh
|
you might want to look at ipr_bits
|
[20:11] sustrik
|
i recall it
|
[20:11] pieterh
|
the bitmap can be very sparse but subscribers are not randomly scattered
|
[20:11] pieterh
|
they cluster
|
[20:11] sustrik
|
how does that affect performance?
|
[20:11] pieterh
|
it means you need to treat 111111111 as efficiently as 000000000
|
[20:12] pieterh
|
well, not in all cases but afair doing that helps
|
[20:12] sustrik
|
it's up to choosing the compression algorithm
|
[20:12] pieterh
|
we compared sflbits, which did compression
|
[20:12] pieterh
|
against ipr_bits, which rakan el-khalil wrote, which did not do compression
|
[20:12] pieterh
|
ipr_bits was much faster
|
[20:13] sustrik
|
i am not sure what happened, but the issue here is that once you start getting cache misses
|
[20:13] sustrik
|
the performance goes down something like 70 times
|
[20:14] sustrik
|
accessing memory is sloooooooooow
|
[20:14] pieterh
|
well, I don't have data on that
|
[20:14] sustrik
|
anyway, it should be retested
|
[20:14] pieterh
|
just to point out that we already solved these problems (probably poorly) more than once
|
[20:15] sustrik
|
yes, i recall the algos in openamq
|
[20:15] pieterh
|
inverted bitmaps go back a long way
|
[20:15] sustrik
|
afair they should work very fast when the whole bitmap is in cache
|
[20:15] pieterh
|
i've seen implementations in algol-68 :-)
|
[20:15] sustrik
|
there haven't been caches back then :)
|
[20:15] pieterh
|
on paper, in a museum...
|
[20:16] pieterh
|
the whole RAM was a cache, real memory was rust
|
[20:16] sustrik
|
an alternative to compression would be a cache oblivious algorithm...
|
[20:16] sustrik
|
but i'm not sure that can be done with tries & bitmaps
|
[23:09] ffred
|
hello. Where can I find the coded examples from: http://www.zeromq.org/code:_start ?
|
[23:10] ffred
|
I looked in the 2.0.8 download, but (at least to me) it wasn't obvious if these larger examples were included somewhere.
|