An Evaluation of Knowledge Base Systems for Large OWL Datasets

P q = max (1 – N
T q
,
T q is the response time (ms) for query q and N is the total number of triples in the
dataset concerned. To allow for comparison of the metric values across datasets of different
sizes, we use the response time per triple (i.e., N
T q
) in the calculation. Also we
use a timeout value to eliminate undue affect of those query response time that is extremely
far away from others in the test results: if to a certain query q, a system’ s response
time per triple is greater than 1- where is a very small positive value, we
ZLOOXVH IRU3 q instead. has the same role as in F q .
Generally speaking, the CM metric will reward those systems that can answer queries
faster, more completely and more soundly.
2.5 Benchmarking Architecture
Fig. 2 depicts the architecture of the benchmarking. LUBM prescribes an interface to
be instantiated by each target system. Through the interface, the benchmark test module
launches the loading process, requests operations on the repository (e.g. open and
close), issues queries and obtains the results. Users inform the test module of the target
systems and test queries by defining them in the KBS configuration file and query definition
file respectively. It needs to be noted that queries are translated from the above mentioned
KIF-like language into the query language supported by the system prior to being issued
to the system. In this way, we want to eliminate affect of query translation to the query
response time. The translated queries are fed to the tester through the query definition file.
The tester just reads the lines of each query from the definition file and passes them to the
system.
LUBM
Test Module
Data Generator
KBS
Configuration
Query
Definition
Benchmark Interface
Benchmark Interface
Benchmark Data
KBS 1
KBS n