MARKLOGIC SERVER

CLUSTERING AND CACHING
As your data size grows and your request load increases, you might hear one of two
things from a software vendor. The vendor might tell you that they have a monolithic
server design requiring you to buy ever larger boxes (each one exponentially more
expensive than the last), or they might tell you they've architected their system to
cluster—to take a group of commodity servers and have them operate together as a
single system (thus keeping your costs down). MarkLogic? It's designed to cluster.
Clustering provides four key advantages:
1. The ability to use commodity servers, bought for reasonable prices
2. The ability to incrementally add (or remove) new servers as needed (as data or
users grow)
3. The ability to maximize cache locality, by having different servers optimized for
different roles and managing different parts of the data
4. The ability to include failover capabilities, to handle server failures
MarkLogic servers placed in a cluster specialize in one of two roles. They can be
Evaluators (E-nodes) or Data Managers (D-nodes). E-nodes listen on a socket, parse
requests, and generate responses. D-nodes hold data along with its associated indexes
and support E-nodes by providing them with the data they need to satisfy requests, as
well as to process updates.
As your user load grows, you can add more E-nodes. As your data size grows, you can
add more D-nodes. A mid-sized cluster might have two E-nodes and 10 D-nodes, with
each D-node responsible for about 10% of the total data.
A load balancer usually spreads incoming requests across the E-nodes. An E-node
processes the request and delegates to the D-nodes any subexpression of the request
involving data retrieval or storage. If the request needs, for example, the 10 most recent
items matching a query, the E-node sends the query constraint to every D-node with
a forest in the database, and each D-node responds with the most recent results for
its portion of the data. The E-node coalesces the partial answers into a single unified
answer: the most recent items across the full cluster. It's an intra-cluster back-and-forth
that happens frequently as part of every request. It happens any time the request
includes a subexpression requiring index work, document locking, fragment retrieval, or
fragment storage.
D-nodes, for the sake of efficiency, don't send full fragments across the wire unless
they're truly needed by the E-node. When doing a relevance-based search, for example,
each D-node forest returns an iterator, within which there's an ordered series of
fragment IDs and scores, extracted from indexes. The E-node pulls entries from each
47