New Statistical Algorithms for the Analysis of Mass - FU Berlin, FB MI ...

5.3. QAD GRID PLATFORM SERVER 123
Data elements (e.g. a single 1D spectrum, or a peak picking result). This
is the most basic element reflecting the actual information physically
stored in the Grid. There exist two classes of data:
1. RAW Data: This kind of data usually exists as text or binary files
on the platform server. These files can get quite large and easily
exceed several gigabytes per file. Depending on the network path
transferring these files can take several minutes to several hours.
Especially if large quantities of data is requested from a single node
network traffic speed can decrease drastically.
2. Analysis (intermediate) Result Data: This type of data is usually
stored in database tables and fetched by database queries. The size
of data fetched per query is usually comparatively low. However,
if many workers query a database the overall performance drops as
in the file based case.
To model this we have chosen to use a rooted directed acyclic graph (DAG).
This naturally introduces hierarchies and links between elements and allows for
fast searches. Studies are directly connected to the root, groups and datasets
are part of a study and data elements are modeled as the leafs. Each leaf in
a tree corresponds to an actual physical file or database entry of a result.
A nice feature of this structure is that is allows for linking any kind of
information to each element (node) without losing the DAG’s properties. For
example, metadata (e.g. patient metadata, timestamp when a datum was
added to the system, or information about physical location of a file) can be
directly linked to a leaf. Especially the latter will be of importance for data
distribution and (subsequent) data location in the Grid (see section 5.3.2).
Task Management
To split up a large problem some problem specific method must break this up
into smaller tasks. These tasks contain parameters describing
� system data, such as: task priority, task owner, draft flag and linked
task (see section 5.3.4)
� target specification: sometimes a particular worker has to handle a task
� the type of this task (e.g. peak picking or file copy)
� (optional) dependencies on another tasks
� (optional) target worker that has to handle this particular task
� what input data is needed
� where results should be written to (e.g. database or file)
� further task dependent parameters - such as fitting variables or window
sizes