IRC Log


Monday August 30, 2010

[Time] NameMessage
[04:52] motk howdy people
[04:52] snerd would there be any interest in adding a zeromq mirror?
[04:52] snerd I'm running a large academic mirror and am looking for decent content
[04:53] guido_g i don't think that the hit rate on the web-site makes this necessary :)
[04:54] guido_g but you should ask the imatix guys
[04:54] snerd heh, worth a shot
[04:55] snerd wish I could've been using 0mq back at my last investment banking job
[04:55] guido_g same here for the tradings system project i'vew been on last year :)
[04:56] guido_g ibm wllm is a beast
[04:56] snerd cough mutter 'alladin' cough mutter
[04:56] snerd before that it was mqseries
[04:56] snerd guh
[04:58] guido_g oh btw, there discussions going to redesign the web site, so if you're interested stay tuned
[04:59] snerd we're already mirror apachemq, rabbitmq, amqp stuff, so might as well
[04:59] guido_g as i said, imatix is taking care of that
[04:59] snerd nod
[05:00] guido_g the whole thing is a wikidot page
[05:00] snerd eep
[05:25] sustrik snerd: we've actually talked about where to host the downloads
[05:25] sustrik uploading them to the wiki is a bit weird
[05:26] sustrik as for the mirroring, why not
[05:26] sustrik it won't hurt anybody
[05:27] sustrik Blafasel_: everything is FIFO in 0MQ
[05:27] snerd if anyone is up for it, hit me at rob.kearey@aarnet.edu.au
[05:29] sustrik ok
[05:43] CIA-20 rbzmq: 03Martin Sustrik 07master * re742398 10/ rbzmq.c : PUSH and PULL constants added - http://bit.ly/ctHC2I
[05:49] CIA-20 clrzmq: 03Martin Sustrik 07master * rcf394ea 10/ clrzmq/zmq.cs : PUSH and PULL constants added - http://bit.ly/9G3Gqh
[07:49] mrm2m guido_g: Rückmeldung zur Seite: Der "all languages"-button unter den Beispielen ist genial!
[07:50] guido_g great
[07:51] mrm2m Sotty
[07:51] mrm2m Sorry, did't recorgnize it's english here
[07:56] guido_g np
[07:57] guido_g for info about crosscompiling you might want have a lookt at the maling list
[09:05] mrm2m just an Idea: You already use JS for this All languages. Why not replace the content of the corresponding example in place? A set Language as default would be cool, too.
[09:38] mrm2m Test of pyzmq failes: http://paste.pocoo.org/show/256240/
[09:38] mrm2m Any ideas?
[09:39] mrm2m /usr/local/lib/python2.6/dist-packages/zmq/_zmq.so looks good - as far as I can tell
[09:41] mrm2m oh - never mind. I didn't recognize that python was looking in "/home/moritz/GEU/pyzmq/zmq/" instead of "/usr/local/lib/python2.6/dist-packages/zmq/"
[09:47] guido_g re
[09:53] mrm2m Hi, I just ran into this problem: http://blog.boxedice.com/2010/05/23/building-zeromq-and-pyzmq-on-red-hat/
[09:54] mrm2m It's the same for Ubuntu. Solution worked, but isn't pretty.
[09:55] guido_g no probs on original debian
[09:56] mrm2m http://mail.scipy.org/pipermail/ipython-dev/2010-March/005900.html
[09:58] guido_g so?
[09:58] guido_g what's the problem?
[10:01] mrm2m Nothing. Just an info, that this topic still exists and also affects Ubuntu
[10:10] CIA-20 zeromq2: 03Martin Sustrik 07master * rfba90af 10/ src/socket_base.cpp : Issue 54 - socket_base.cpp:162 comparison error - http://bit.ly/9M5Xs2
[10:23] guido_g ahh, ryanair again? :)
[10:24] pieterh guido_g: yeah :-)
[10:24] guido_g so good luck, take care
[10:24] pieterh http://new.zeromq.org is something I hacked up this morning
[10:25] pieterh this would become www.zeromq.org and the current www. would become dev.zeromq.org
[10:25] guido_g nice
[10:25] pieterh sustrik and i will strip it down until it's minimalistic
[10:26] guido_g but for more input you should ask mrm2m, he's the new kid on the block :)
[10:27] jond pieter: looks better. I am beginning to see yr point about the colours
[10:27] pieterh the strong red/black works for simple sites but seems painful for link-rich text
[10:28] sustrik the guide?
[10:28] pieterh ?
[10:28] pieterh that's a question?
[10:28] pieterh sorry., have to leave, cyl
[10:28] sustrik cyl
[10:29] sustrik ugh, the voting box hurts eyes
[10:30] guido_g the box won't be in the final version, i assume
[10:32] sustrik dunno, but it should be of that poisonous pink hue
[10:35] guido_g at least, it *does* catch attention :)
[10:37] sustrik guido_g: have a look at www.zeromq.org now
[10:37] sustrik i've removed the red
[10:37] sustrik pretty dull
[10:38] guido_g can't see the links anymore
[10:38] sustrik they should be grey
[10:38] guido_g links should stand out a little
[10:39] sustrik ah, you see them as black?
[10:39] guido_g no grey
[10:39] guido_g but it's hard to read
[10:40] sustrik ok, reverting it back to red
[10:41] jond what would the whole background look like in that grey?
[10:41] sustrik it would make the text less readable imo
[10:42] sustrik the contrast is good
[10:42] sustrik people just complained about the red
[10:43] jond 0MQ logo looks very close to helvetica, does it look any good in black
[10:47] sustrik not really, and it's not even helvetica
[10:47] sustrik it's custom-designed font :)
[10:52] mrm2m OMG!
[10:53] mrm2m No, I like the design.
[10:53] mrm2m The TOC could be a little bit more structured.
[10:53] mrm2m But so far I found everything I was looking for.
[10:59] sustrik mrm2m: thanks for feeback
[10:59] sustrik yes, it's not about content missing, it's about messy structure
[11:00] mrm2m Well - The Navigation looks a litttle bit like a tag cloud.
[11:01] mrm2m A lots of people have trouble communicating with a cloud, you know ;.)
[11:01] keffo Personally I find dancing to be a great way!
[11:02] sustrik :)
[11:07] keffo sustrik, I just replied to the lb thread :)
[11:38] jond I still think the user comments and blog stuff should go and the text from the what's 0MQ spread two columns
[11:38] jond and there's too much red
[11:53] sustrik jond: yes, that kind of stuff, but it's more about ergonomy
[11:53] sustrik 90% of visitors want to solve a problem rather then browse around
[11:54] sustrik and the page is not particularly well designed to help you solve your proble quickly and efficiently
[11:54] sustrik joel spolsky had few nice articles on this topic
[11:55] mrm2m by the way: Got it running at my embedded ARM platform now.
[11:55] guido_g woot! :9
[11:55] mrm2m I will give some summery, of what I've done If you like
[11:56] guido_g isn't there a wiki page for this kind of stories?
[11:56] sustrik mrm2m: please, do post it on the mailing list
[11:56] sustrik to make it available
[11:57] sustrik the wiki would be a good choice if it's a howto rather than story
[11:58] guido_g ok
[12:14] mrm2m The user-guide is NSFW. My boss is looking strange, while I lay down in laughter.
[12:16] sustrik :)
[13:16] theICEBear Hi I wonder if anyone knows about an issue with sessions in zmq. I am hitting an assert in session::process_attach line 287
[13:21] sustrik theICEBear: what version is that?
[13:21] theICEBear 2.0.8
[13:21] sustrik zmq_assert (!engine);
[13:21] theICEBear Yup
[13:21] sustrik are you using identities?
[13:22] sustrik ZMQ_IDENTITY?
[13:22] theICEBear I am trying to set my own yeah
[13:23] theICEBear I am probably setting them in the wrong place then
[13:23] sustrik i think what's happening is that there are two peers trying to connect to the same socket with the same identity
[13:24] sustrik this case is not yet handled decently
[13:24] theICEBear peers as in?
[13:25] sustrik the applications over the network
[13:25] theICEBear Ah, thanks for the tip I will check
[13:28] CIA-20 zeromq2: 03Martin Sustrik 07master * rf0a36f9 10/ doc/zmq_cpp.txt : Minor patch to zmq_cpp(7) - http://bit.ly/bfgJOm
[14:29] sustrik pieterh: are you in bts already?
[14:46] pieterh sustrik: ano, ja jestem z bratislava
[14:50] pieterh sorry, i was using polish notation, it should be "som w bratislave"
[14:50] pieterh 0MQ/5.0 will have Slavic language bindings
[15:05] CIA-20 zeromq2: 03Martin Sustrik 07master * re45583c 10/ src/semaphore.hpp : OSX build fixed -- semaphore replaced by mutex - http://bit.ly/9AYi6Y
[15:05] sustrik pieteh: cool
[15:05] sustrik just w is spelled v in slovak
[15:06] sustrik pieterh: in the office?
[15:08] pieterh sustrik: i'm with mato, we're debuggine weird pubsub issues
[15:09] pieterh *debugging
[15:09] sustrik ack
[15:09] sustrik i leave you in peace then
[18:21] Steve-o anyone see the new LBM 4,1 features from 29 West?
[18:21] Steve-o zero copy for c# api
[18:26] Steve-o surprised at the choice of terminology, I can't tell whether it is official MS API or out of someones orifice,
[18:26] Steve-o "Zero Object Delivery (ZOD) has been implemented for .NET, which allows .NET messaging receivers to deliver messages to an application with no per-message object creation."
[18:26] Steve-o "Zero Incoming Copy (ZIC) has been implemented for .NET, which provides access to message data directly through a byte pointer returned by the LBMMessage.dataPointer() method."
[18:32] sustrik nope, haven't seen it yet
[18:32] sustrik what's the link?
[18:44] Steve-o marketing email, looks like most detail is here, http://blogs.informatica.com/perspectives/index.php/2010/08/27/lbm-4-1-ume-3-1-umq-1-1-released/
[18:46] guido_g zero object delivery sounds good :)
[18:48] Steve-o and presumably if 29West are using it it must show some improvement somewhere
[18:49] guido_g sorry, forgot the sarcasm tags
[18:49] Steve-o :P
[18:49] pieterh Lol
[18:49] guido_g zero objects usccessfully delivered
[18:49] Steve-o you could use a zero object for confirmation reply of delivery
[18:50] Steve-o or just heart beating
[18:50] pieterh ZOD & ZIC vs ZMQ
[18:50] pieterh OMQ!
[18:50] guido_g Steve-o: depends on the protocol information in this "zero object"
[18:51] Steve-o guido_g: in PGM you can delivery 0 byte messages too
[18:52] guido_g as i said, depends on the information in the protocol if this is useful or not
[18:52] guido_g one can't send 0 bytes, so there *is* something on the wire
[18:53] guido_g heartbeat allone isn't that useful
[18:53] Steve-o would be a good idea for virtual circuits
[18:53] guido_g but if you attach the seq. no. of the last data packet, we start to have something
[18:57] guido_g afair, wllm docs didn't say a word about zero copy for the bindings
[18:58] sustrik i like this: http://www.29west.com/products/gateway.html
[18:58] sustrik it looks like 29west have independently arrived at the concept of "device"
[18:59] sustrik that kind of proves the design
[18:59] guido_g hehe
[19:00] Steve-o reads like P2PS, i.e. a TIC without caching
[19:10] rossij I have a requirement for a single port to be opened between the agent and the server. The agent needs to be able to connect the server and send messages on demand, but the server needs to be able to supply updates, config changes, other data to all the agents over the established connection. I don't see a simple way of doing this without using a REQ/REP and a second SUB/PUB with zeromq. Does anyone have a suggestions on how to do this w
[19:10] rossij ZeroMQ?
[19:12] pieterh rossij: it works better when you separate the different types of message flow
[19:12] pieterh request-response is not publish-subscribe
[19:12] pieterh in old-fashioned tcp you would mix these all on one port
[19:12] pieterh but that becomes more complex and less scalable
[19:12] rossij pieterh: I wish i could it would be sooo much simpler, but two ports is goign to be a no go
[19:12] pieterh imagine you want to supply updates to some components across a multicast bust
[19:12] pieterh *bus
[19:13] pieterh why is two ports a no go?
[19:13] rossij pieterh: it's what current ly is in place in a huge number of installations.
[19:14] pieterh this is across a LAN?
[19:14] rossij pieterh: zeromq plus two ports would make this simple
[19:14] pieterh in any case you will need to migrate components to a new stack
[19:15] pieterh presumably hidden inside whatever framework you use
[19:15] pieterh one, two, 10 ports... it does not change that
[19:15] pieterh existing components cannot talk to 0MQ without having 0MQ linked with them
[19:16] rossij understand that I am working to rip on the udp network code and put in zeromq for tcp/multicast and its just so nice to work with
[19:16] pieterh sure
[19:17] pieterh do you understand that the number of ports you end up using is not a problem
[19:17] pieterh unless you have firewall issues
[19:17] pieterh or someone who really, really hates you opening new ports
[19:18] rossij yes - i understand that - inside the firewall it's all ok - but - for devices and installs in the DMZ and other areas
[19:18] pieterh right...
[19:18] pieterh what I'd look at is bridging this into the DMZ using a device
[19:19] pieterh there is no literature yet on how to tunnel 0MQ connections over 0MQ but IMO it's possible
[19:20] rossij that is something I will have to try. makes the setup a little more complex but could work
[19:20] pieterh i'd say, a pair of devices that create a tunnel
[19:20] pieterh one inside the DMZ, one on the LAN
[19:21] pieterh XREQ/XREP between the two
[19:21] Steve-o ideally would be nicer to tunnel it all over http, easier to firewall
[19:21] rossij humm anyone know if TCP virtual services / virtual endpoints is being worked on ?
[19:21] pieterh yes, devices could be bridges, speaking HTTP at one edge, 0MQ at the other
[19:21] pieterh it's an idea for 0MQ/3.0
[19:21] pieterh i assume you mean subports kind of thing?
[19:21] Steve-o is there any http/zeromq gateway?
[19:22] rossij pieterh: yes
[19:22] pieterh Steve-o: not as far as I know
[19:22] pieterh rossij: yes, 0MQ/3.0 concept
[19:22] pieterh see http://www.zeromq.org/docs:3_0
[19:22] pieterh "TCP virtual services / virtual endpoints"
[19:22] pieterh oh... :-)
[19:23] pieterh you quoted that page
[19:23] pieterh lol
[19:23] rossij yeah that is where I got - i just noticed and hoped - some might move to 2.1 :)
[19:23] pieterh it might be moved to 2.1
[19:23] sustrik Steve-o: 0MQ/HTTP gateway => mongrel2
[19:24] pieterh sustrik: not really... kind of but no...
[19:24] pieterh it uses 0MQ to implement a higher-level API
[19:24] sustrik ok
[19:24] sustrik i've did some thinking about the virtual endpoints
[19:25] sustrik subports
[19:25] rossij pieterh: thanks will keep a watch out - and see what i can do myself
[19:25] pieterh rossij: IMO a pair of tunneling devices will be quite easy to make, if you don't try to be generic
[19:26] sustrik the basic concept is simple, the only implementation problem i see is how to share a single TCP listener socket between all the sockets and i/o threads
[19:26] pieterh sustrik: use 0MQ messaging, internally... :-)
[19:26] sustrik it would require non-trivial work to solve all the synchronisation issues
[19:27] Steve-o the recent google work on shared sockets?
[19:27] sustrik pieterh: actually, that's what 0mq is doing :)
[19:27] pieterh so a socket with virtual endpoints becomes a device, no?
[19:27] sustrik inter-thread synchronisation messages are passed using same code that is used to pass real 0mq messages
[19:27] pieterh sustrik: neat
[19:28] sustrik what you are speaking about is tunnel
[19:28] sustrik which is doable,sure
[19:28] sustrik but it's nasty
[19:28] sustrik you get head of line blocking problem etc.
[19:29] sustrik real solution is to have real separate connections
[19:29] sustrik sharing same source port
[19:29] sustrik sorry for the rant btw
[19:30] sustrik i have another one about zero-copy
[19:30] Steve-o :D
[19:30] sustrik :)
[19:30] sustrik well, there are interesting things i've learned hard way with zero copy
[19:30] sustrik i can summarise it in an email
[19:31] sustrik anyone interested in that?
[19:31] pieterh sustrik: if you have a good rant, make it a blog posting
[19:31] sustrik it stems from you \x00 email, so it would be logical to reply
[19:32] pieterh lol, ok, go for it...
[19:32] sustrik goodo
[19:32] pieterh note that i only mentioned this because some poor user hit the door with his nose
[19:33] sustrik i know, the problem is that it's user feature that has profound implementation impact on the whole stack
[19:33] sustrik basically the feature would be "stretched" all the way down to kernel/userspace boundary
[19:34] sustrik and -- if kernel support is added one day -- even further down into kernel networking stack
[19:34] sustrik which is pretty nasty
[19:35] pieterh you're saying...
[19:35] pieterh that allocating one extra byte
[19:35] pieterh means the end of the universe...
[19:36] pieterh :-)
[19:36] sustrik kind of
[19:36] sustrik it's not obvious
[19:36] sustrik but it's a trap i;ve already fell in
[19:36] pieterh i understand that when you do zerocopy on read, it's not possible to mess with what you provide
[19:36] sustrik wasted weeks of coding that way
[19:36] pieterh claro
[19:37] pieterh and we always do zero copy on read?
[19:38] sustrik yes
[19:38] sustrik no point in having 2 stacks
[19:38] sustrik zero-copy and non-zero-copy
[19:38] pieterh fair enough, it obviously does not work with zerocopy reads
[19:39] pieterh for some reason i was imagining we allocated and copied on recv unless the app provided a buffer
[19:39] pieterh i'm stupid
[19:39] sustrik i've did the same mistake
[19:39] sustrik but there was noone to stop me back then :)
[19:40] pieterh lol
[19:40] sustrik i've wasted a lot of time, but otoh the multi-part messages emerged from the mess
[19:41] Steve-o it's very conducive for memory managment
[19:41] sustrik which are actually a pretty good way to retain zero copy while still being able to modify messages
[19:41] pieterh indeed, multipart messages are a clever solution
[19:41] sustrik don't work for the terminal zero though
[19:41] sustrik shrug
[19:44] sustrik pieterh: btw i though of writing little pieces of implementation experience down somehow
[19:44] sustrik so that it's not lost
[19:44] sustrik maybe a mino-blog?
[19:44] sustrik mini
[19:47] jond sustrik: think that would be an excellent idea. as i have discovered with investigating pub/sub filtering the code is quite terse and clever in places but can take quite a while to get into the headspace of 0mq internals and why it is written like it is.
[19:48] sustrik jond: yes, that's exactly my point
[19:48] sustrik there are comments in the code
[19:48] sustrik but that missed the big picture
[19:48] sustrik well, the larger picture
[19:49] sustrik actually, it would be great if you could help with it
[19:49] sustrik you had to dissect the pub/sub code
[19:49] sustrik so you know what the stumbling blocks there are
[19:50] jond hmm, my patmatch branch is so full of cout calls for me to figure out what was going on, i can barely find the code i changed now ;-)
[19:50] sustrik i mean, what were the things that were not obvious immediately
[19:50] sustrik what have taken you a long time to figure out
[19:50] sustrik etc.
[19:51] sustrik i can only guess what's hard and what's not
[19:51] sustrik anyway, it's just an idea
[19:52] jond i'll try and get some notes down, the comments you have just made about zero copy lead me to think i better study that stuff a bit more
[19:52] sustrik ok. good
[19:53] jond that prefix tree find, was because i have my own prefix tree now and i needed to be sure that i hadnt introduced the bug.
[19:54] sustrik that's what peer review is for :)
[19:55] jond I would still prefer on the pub side to be able to use bits rather than refcount bytes. each sub socket connect results in a single write pipe on the pub side
[19:55] jond that is the case?
[19:56] sustrik yes
[19:56] sustrik yes, probably it can be compressed to a bit
[19:57] sustrik depends on how the subscription forwarding algorithm looks like
[19:57] sustrik if you have a single bit
[19:57] sustrik you have to assure that same subscription won't be sent twice over the same "logical connection"
[19:57] jond that is what i think, if write-pipe == sub endpoint effectively
[19:58] sustrik sure, try it that way
[19:58] sustrik we'll see what emerges
[19:59] jond think i might package up that pub matcher class and see it for review. it seems to get the right answers
[19:59] sustrik sure
[19:59] jond s/see/send/g
[19:59] sustrik btw
[19:59] sustrik the bitmap thing
[20:00] sustrik i've seen a performance study on exactly this problem
[20:00] sustrik and the interesting point there was that compressing the bitmaps pays of
[20:00] jond url?
[20:00] sustrik not public yet i think
[20:00] sustrik in any case
[20:01] sustrik the point was that the bitmaps tend to be very sparse
[20:01] sustrik and often pretty large
[20:01] jond i was wondering if dreppers memory paper might have any useful info.
[20:01] sustrik that way you hit cpu cache boundary very soone
[20:01] sustrik and your performance goes down by order of magnitude
[20:02] sustrik maybe even two orders of magnitude
[20:02] sustrik jond: it's definitely worth of reading
[20:02] sustrik but pretty long
[20:02] jond wrt the bitmap, i think that it will be very sparse and large and that is the problem. that's also why i worry about the vectors of refcount, n pipes wide at every node in the trie....
[20:02] jond in the current implementation
[20:03] sustrik right
[20:03] sustrik so what about compressing the vector
[20:03] sustrik making it say a vector of indices
[20:03] sustrik hm
[20:03] sustrik ordered vector of indices
[20:03] sustrik that way merge is O(n)
[20:05] jond it might be easier if i were to send the code. let me go look. need to be sure we are talking about the same thing...
[20:06] sustrik sure send it, review is good
[20:06] pieterh jond: how large are the bitmaps?
[20:06] sustrik what i said was that instead of bit vector 0000100001001001
[20:06] sustrik you can have a vector of indices like this: 5|10|13|16
[20:07] pieterh sustrik: we tested this quite extensively when making openamq
[20:07] pieterh we used compressed bit maps in SFL
[20:07] sustrik and?
[20:07] sustrik it must have been much faster for large amount of subscribers, no?
[20:07] pieterh if you use fast bit operations, uncompressed is faster
[20:08] sustrik how large bitmaps have you tested?
[20:08] pieterh 8k bytes
[20:08] pieterh iirc
[20:08] sustrik too small
[20:08] pieterh let me check, it's in IPR
[20:08] sustrik the performance hit comes what cache size is hit
[20:08] pieterh so how large are the bitmaps?
[20:08] sustrik several MB these days
[20:09] sustrik number of topics x number of subscribers
[20:09] pieterh ?
[20:09] pieterh surely you have one bitstring per topic
[20:09] sustrik es
[20:09] sustrik yes
[20:09] pieterh that's the 8k
[20:09] sustrik what 8k?
[20:10] pieterh you asked how large our bitmaps were, I said 8K
[20:10] pieterh that is one bitstring
[20:10] sustrik ah
[20:10] pieterh equals 64k subscribers
[20:10] pieterh one bitstring per topic
[20:10] pieterh or per expression in OpenAMQ
[20:10] sustrik how sparse was the bitmap?
[20:10] pieterh you might want to look at ipr_bits
[20:11] sustrik i recall it
[20:11] pieterh the bitmap can be very sparse but subscribers are not randomly scattered
[20:11] pieterh they cluster
[20:11] sustrik how does that affect performance?
[20:11] pieterh it means you need to treat 111111111 as efficiently as 000000000
[20:12] pieterh well, not in all cases but afair doing that helps
[20:12] sustrik it's up to choosing the compression algorithm
[20:12] pieterh we compared sflbits, which did compression
[20:12] pieterh against ipr_bits, which rakan el-khalil wrote, which did not do compression
[20:12] pieterh ipr_bits was much faster
[20:13] sustrik i am not sure what happened, but the issue here is that once you start getting cache misses
[20:13] sustrik the performance goes down something like 70 times
[20:14] sustrik accessing memory is sloooooooooow
[20:14] pieterh well, I don't have data on that
[20:14] sustrik anyway, it should be retested
[20:14] pieterh just to point out that we already solved these problems (probably poorly) more than once
[20:15] sustrik yes, i recall the algos in openamq
[20:15] pieterh inverted bitmaps go back a long way
[20:15] sustrik afair they should work very fast when the whole bitmap is in cache
[20:15] pieterh i've seen implementations in algol-68 :-)
[20:15] sustrik there haven't been caches back then :)
[20:15] pieterh on paper, in a museum...
[20:16] pieterh the whole RAM was a cache, real memory was rust
[20:16] sustrik an alternative to compression would be a cache oblivious algorithm...
[20:16] sustrik but i'm not sure that can be done with tries & bitmaps
[23:09] ffred hello. Where can I find the coded examples from: http://www.zeromq.org/code:_start ?
[23:10] ffred I looked in the 2.0.8 download, but (at least to me) it wasn't obvious if these larger examples were included somewhere.