Tip of the Iceberg

1 - Great series Tim. Thanks for taking the time to put this in a blog series. I am not quite sure if/when I would need to use this but I learnt a lot from the general techniques you were applying to solve the overall need. I am guessing the way we are going with mobile devices, SaaS etc. this will soon become an important technique to have in our developer kit. At some time I would like to encapsulate what you have done as a new set of methods for my own ·dominoFramework.

2 - Karsten, Squawk { Link } originally used no views at all. Eventually we decided that the "Community Squawks" transcript might as well be a view, because it's just all squawks listed in descending order, so it seemed silly not to just use a view. But every other query - "My Flock's Squawks", "My Replies", etc. - uses a variation on the digest search technique (the article you linked to was our original inspiration for some of our design techniques in Squawk). That's the reason that, without even bothering to use scope caching, it's so much faster than it would be if we were using views to do our indexing for us.

In that application, we don't have to worry (as much) about replication conflicts because not only does the nature of the application dictate that, once a document is created, it's never modified again, but the digest is also designed to be user-based: when you post a squawk, you're updating a digest that is specific to you, so no user ever modifies the same document as another user. It's possible, of course, that the same user could get routed to multiple servers for separate transactions that each result in a digest modification, but that's most likely to occur in a cluster, which means that replication should be occurring almost immediately instead of on a schedule. Obviously, though, offline usage would present other challenges.

Although we didn't go this route with Squawk, another approach would be to maintain some or all of the digest data in memory: for example, when a user submits data, after writing it back to the NSF, also make an update to some SortedMap in an applicationScope'd managed bean that also does lazy loading so that, if the data being queried isn't already stored in scope (or has been purged due to a reboot or design modification), it does a fresh pull from the actual documents. You'd have to be careful with this approach for several reasons: you'd want to limit how much data you store about each record in memory, or the server would eventually (if not rapidly) explode; you can only store metadata about each document, not actual document handles, since Domino will recycle those handles after each request; finally, if the data model also includes Reader fields or other security restrictions, storing too much in scope would allow users to see data they shouldn't, even if they're unable to access the full document that data represents.

But it's definitely possible, though I'm not sure I'd recommend designing every application this way - just applications that would benefit from extremely rapid response times and are already conceptually structured in a way that fits this model well. But it's an extremely powerful way to approach maximizing performance... we just have to be willing to be "programmers", not just "developers".

Speaking of which, Nathan's already publicly thanked you for this, but I also wanted to thank you for the articles you posted last year about managed beans. That information has been absolutely essential to how we've designed our applications ever since.

3 - Thanks for the article. The digest search (also described here: { Link } ) is a very interesting approach to speed up lookup operations.
What do you think - is it possible to create a whole application based on digest search without using ANY views and in addition support replicated environments (be able to handle replication conflicts)?
For that scenario, you would have to create all document lists on your own, e.g. with special documents that represent the list position of a document and pointers to the previous and next entry to use it like views.
I'm curious if that would increase or decrease performance, because you would loose the benefit of a stored view index. Writing data takes longer because you need to update all document lists on your own (which could be done in a background thread). The question is whether reading data via digest is so much faster compared to using Notes views and how to detect changes coming in via replication in order to update the index. Looks like it would also mean to create an index for each server and not to replicate it between replicas.
Interesting thought.

4 - "What do you think - is it possible to create a whole application based on digest search without using ANY views and in addition support replicated environments (be able to handle replication conflicts)?"

Yes it is. And Tim and I have done it before. It's probably easier than you think, since the whole "replication conflict" story is based on the assumption of updating existing documents, when you don't really have to. Keeping transactional audit trails makes that a lot easier.