An Integrated Data Analysis Suite and Programming ... - TOBIAS-lib

22 CHAPTER 2. A HIGH-THROUGHPUT DNA SEQUENCING DATA ANALYSIS SUITE
language of choice, readout of binary formats is therefore often too laborious or beyond basic programming
skills, eectively limiting users of the format to choose from a narrow set of supported
programming languages.
In contrast, basic parsers are straightforward to implement for simple tab-delimited table
formats. Requirements such as string tokenization and conversion between character strings and
numeric types are well supported by high level scripting languages popular in the bioinformatics
eld such as Perl or Python, and considered a basic skill in next to any programming language.
The line-based formats are furthermore accessible to shell scripting and readily ltered through
widely used Unix text processing utilities such as grep, awk, sed or sort. While conversion
between le and in-memory representation is less ecient compared to optimized binary formats,
the introduced overhead is marginal in relation to the overall CPU consumption of data analysis
algorithms.
Furthermore, delimited text tables can serve as a basic framework for le format denitions
such as SAM, GFF or VCF. This common foundation enables implementation of generic algorithms
applicable to a large variety of dierent le formats, as demonstrated for example by the
tool Tabix [125] for range indexing of genomic feature le formats. Adapting similar algorithms to
dierent specialized binary formats is laborious, requiring reimplementation or implementation
of sophisticated adapter mechanisms.
Format issues are easily identiable with text-based le formats without requirement of expert
knowledge or implementation of specialized format verication routines. Software dealing with
data produced by a rapidly evolving technology as is high-throughput sequencing instrumentation
is likely required to undergo constant adaptation. Besides, at high rates of data production
storage systems may operate close to the limits of their capacity. In such potentially unstable
settings, simplicity of le formats constitutes a signicant advantage.
On grounds of these considerations, the SHORE analysis suite relies on tab-delimited table
formats for storing reads, read mapping data and analysis results. Users may further choose to
store les either directly as plain text, or between one of two widely used compression formats,
GZIP [116] and XZ [127]. While such generic compression le formats themselves represent
binary formats, decoding commands and routines are widely available across platforms and
programming languages and are employed merely as a ltering layer, thus retaining most of the
advantages of a text-based format. With compression, SHORE implements block-wise encoding
for both the GZIP and the XZ format to enable near-random read access at any decompressed
le oset. Our implementation enables free adjustment of random access and space eciency
trade-os (section 2.2.2).
SHORE's text-based read alignment le format was extended to support reference compression
as introduced by the binary CRAM format (section 1.6), as well as further simple preprocessing
lters serving to boost compressibility in combination with generic compression algorithms.
Depending on the choice of preprocessing lters, compression algorithm and random access resolution,
our compressed text le formats enable similar or better read alignment compression
ratios compared to CRAM (section 2.2.5).
Another common requirement in sequencing data analysis is the retrieval of data set elements
with a certain attribute value, e. g. extraction of the set of single nucleotide polymorphisms
included in a certain range of the reference genome or retrieving a read with a specic identier
for a short read data set. However, standard linear search is slow when processing large data sets.
Given the data set is in sorted order with respect to the key attribute, accelerated queries are
possible. SHORE sort oers text le sorting functionality similar to the standard sort command
line utility, optimized for use with typical high-throughput sequencing le formats. Additionally,
the tool is capable of performing binary accelerated queries on arbitrary sorted tab-delimited
text les.