MARKLOGIC SERVER

DATABASES, FORESTS, AND STANDS
MarkLogic organizes data hierarchically, with the database as the logical unit at the
top. A system can have multiple databases, and generally will, but each query or update
request typically executes against a particular database. (The exception is with a superdatabase,
which lets you query multiple databases as a single unit. That's covered later.)
Installing MarkLogic creates several auxiliary databases by default: App-Services,
Documents, Extensions, Fab, Last-Login, Meters, Modules, Schemas,
Security, and Triggers.
A database consists of one or more forests. 5 A forest is a collection of documents and
is implemented as a physical directory on disk. In addition to the documents, a forest
stores the indexes for those documents and a journal that tracks transaction operations
as they occur. A single machine may manage several forests, or when in a cluster
(acting as an E-node, an evaluator), it might manage none. Forests can be queried in
parallel, so placing more forests on a multi-core server will help with concurrency. The
optimal number of forests depends on a variety of factors, not all of which are related
to performance. On modern server hardware, six primary forests per server (with six
replica forests for failover, as discussed later) is a good starting point. The size of each
forest will vary depending on if it's a high-performance or high-capacity environment.
For high performance (e.g., a user-facing search site with facets), each forest might
contain 8 million documents and 100 gigabytes on disk. For high capacity (e.g., an
internal analytical application), each forest might contain 150 million documents and
500 gigabytes on disk. In a clustered environment, you can have a set of servers, each
managing its own set of forests, all unified into a single database. 6 For more about
setting up forests, see Performance: Understanding System Resources.
Each forest is made up of stands. A stand (like a stand of trees) holds a subset of the
forest data. Each forest has an in-memory stand and some number of on-disk stands
that exist as physical subdirectories under the forest directory. Having multiple stands
helps MarkLogic efficiently ingest data. A stand contains a set of compressed binary
files with names like TreeData, IndexData, Frequencies, and Qualities. This is
where the actual compressed XML data (in TreeData) and indexes (in IndexData)
can be found.
5 Why are they called "forests"? Because the documents they contain are stored as tree structures.
6 Is a forest like a relational database partition? Yes and no. Yes because both hold a subset of data. No
because you don't typically allocate data to a forest based on a particular aspect of the data. Forests aren't
about pre-optimizing a certain query. They instead allow a request to be run in parallel across many forests, on
many cores, and possibly many disks.
34