DATA & INDEXING
What is an index server?
A copy of your data organized for finding and counting, not for storing and updating - and the reason the fast screens in large applications are fast.
What an index is
A phone book is an index. It holds a copy of the same facts the telephone company keeps - names, streets, numbers - sorted by surname, so that finding one name among a million takes a few page turns instead of a read through every entry. Print a second copy sorted by street and the same data answers a different question quickly: who lives on Lake Road. Nothing about the underlying facts changed. What changed is the organization, and organization is what decides which questions are cheap.
That is the whole idea. An index is a copy of data arranged for a particular kind of lookup. A database table is arranged for storing and updating rows safely. The indexes a database keeps on the side are arranged for finding a row by one value. A search index is arranged for the questions an application screen asks: which records mention this word, how many of them fall in each category, what the average price is of the ones left after these filters.
- Anand, R.Ashram Rd
- Bhatt, K.Lake Rd
- Desai, P.Mill Rd
- Joshi, M.Lake Rd
- Mehta, S.Ashram Rd
- Patel, V.Lake Rd
- Find Mehta - a few page turns
- Who lives on Lake Road? - read every entry
- Ashram RdAnand, R.
- Ashram RdMehta, S.
- Lake RdBhatt, K.
- Lake RdJoshi, M.
- Lake RdPatel, V.
- Mill RdDesai, P.
- Find Mehta - read every entry
- Who lives on Lake Road? - one place to look
The cost of a copy is that it has to be kept in step with the original. The benefit is that each copy can be organized for its own job without compromising the other. Every system that grows past a certain size ends up with more than one organization of the same data, whether it calls them indexes, caches, replicas or warehouses.
What an index server adds
An index server takes that second copy out of the database and gives it a home of its own: its own process, its own storage, its own API. Three things follow.
It has its own copy. Records are sent to the index server, or pulled by it, and stored in the shape its indexes need. The database keeps its rows exactly as before and is never asked to hold a second structure for someone else's queries.
It is refreshed on a schedule, not on every write. A collection of orders might be brought up to date every hour, a product master once a day. The schedule is a decision made per collection, based on how fresh that screen has to be, rather than a side effect of every transaction.
It is built to be read from, not written to. Nothing an application does through the index server's API changes the source systems. Writes go where they always went. The index server is the read side.
The word server matters less than the word index. The same idea goes by search engine, search index, indexed data layer or indexed foundation. Titles here say index server because that is what people search for; the rest of this site mostly says indexed data layer. They are the same thing.
What it is good at
Four kinds of question, and ideally all four in one request.
Full-text search with typo tolerance. A query for jaket finds the jackets, ranks the best matches first, and can suggest Denim jacket and Rain jacket as the user types. The index knows what a word is, which a pattern match on characters does not.
Filtering on any field. Category, brand, size, status, date, store - any field that was indexed can narrow the set, in any combination, without a composite index planned in advance for that combination.
Facets. For the records left after filtering: how many fall in each brand, how many in each price band, what the average price is. These are counts and totals computed on the filtered set and returned beside the results. In the products example used across this site, a search for jaket in Clothing finds 531 records and returns, in the same response, Polo 302 · Levis 144 · Nike 85 by brand, Small 412 · Medium 88 · Large 31 by size, three price bands of 120, 245 and 166 records, and an average price of 42.60.
Sorting and paging at depth. Sort by relevance, by price, by date, in either direction, and fetch page 40 about as cheaply as page 1.
The reason to list them together is that a single screen usually needs all of them at once. A catalog page shows results, the counts in its filter sidebar and a total. A back-office listing shows rows, a filter bar and a sum. An index server answers that whole screen with one request, which is the difference people notice first.
What it is not
It is not a system of record. The index holds a copy; the database holds the truth. If the two disagree, the database is right and the index gets refreshed.
It has no transactions. There is no way to debit one record and credit another atomically, and there should not be. Anything that must be exactly right at the moment of writing stays in the database.
It is not instantly consistent with the source. A record changed at 09:04 appears in the index at the next import, which might be 09:05 or 10:00 depending on the schedule. For a listing screen, a report or a catalog, a short lag is invisible or acceptable, and the screen can say as of 09:00. For a stock check at the till it is not acceptable, and that screen should keep reading the database.
Knowing which screens tolerate a lag is most of the design work. The ones that do are usually the slow ones, which is why moving them is worth it.
How data gets in and stays fresh
Two directions. Your systems can push records to the index through an API whenever something changes, or the index server can pull from the database on a schedule by running a query against it. Push suits systems that already emit events. Pull suits the common case where the database is the only reliable record of what changed.
Two scopes. A full import rebuilds a collection from nothing: every record, read once, indexed once. It is the right move for the first load, after a schema change, or after a large backfill. An incremental import brings only new and changed records, found by an updated-at timestamp, a change flag or a rising id, and is what runs on the schedule the rest of the time.
One schedule per collection. Orders hourly, customers every four hours, products daily, invoices weekly. Each collection has its own index and its own clock, so the busy ones stay fresh without rebuilding the quiet ones.
Nothing comes back. The flow from your systems into the index is one way. The mechanics of change detection, deletes and schedules are in Incremental indexing: keeping an index fresh.
Where it fits
The diagram at the top of this article is the whole architecture. Business systems on the left - an ERP, a CRM, an accounting package, custom software - send data one way into the index server. Inside it, one collection per entity: products, orders, customers, invoices, each with its own index and schedule. On the right, the applications that read from it - web, mobile, internal consoles, other services - through an API that returns records, counts, ranges and totals together.
Nothing on the left is replaced. The ERP still owns orders; the index server holds a copy organized for the screens that search and count them. Nothing on the right talks to the database for those screens any more, which is the second benefit: the production database stops carrying read traffic it was never built for. Why not query the production database directly? covers that side of the argument.
Integration is screen by screen. The slow listing page moves first; the checkout stays where it is. If you are weighing the index against the database you already run, RDBMS vs index server puts them side by side.
Inside the index
The structure that makes searching cheap is the inverted index. A database table maps a record to its fields. An inverted index maps each term to the list of records that contain it. Index the title of every product and the entry for jacket becomes a list of record ids - 7, 41, 302 and so on - sorted and compressed. A search for denim jacket fetches two lists and intersects them: records 7 and 302 are in both. Nothing is scanned. The index is opened at two terms and the work is proportional to the matching records, not to the size of the collection.
Every field gets its own structure. Text fields are tokenized - split into words, lowercased, reduced to a stem - before they go into the inverted index, which is why jaket can be matched to jacket by edit distance and why Jackets matches jacket. Numeric, date and keyword fields are indexed as exact values and ranges, so a filter on price or on a status code is a lookup rather than a comparison across rows.
Sorting and faceting use a third layout: doc values, a column-oriented store where each field's values sit together in record order. Counting how many of the 531 matching records are in each brand is one pass over the brand column, with the matching ids already known. The SQL equivalent, COUNT(*) … GROUP BY brand over the filtered set, has to visit the rows, which is why facets are cheap here and grow expensive there as the table grows.
Relevance comes from the same structures. How rare a term is across the collection, how often it appears in a record, and which field it appeared in - a hit in the title outranks a hit in the description - combine into a score per result, and results come back ordered by it. A database has no equivalent, because a row either matches a predicate or it does not.
This family of techniques comes from Apache Lucene, the library underneath most search engines, and Apache Solr, the server built on it. Solr adds the HTTP API, the schema, the collections and the scheduling that turn a library into an index server. For the B-tree side of the comparison, read Database index vs search index.