%--------------------------------------------------------------------------
%--------------------------------------------------------------------------
%                 RAWSEEDS METRICS COMPUTATION TOOLKIT
%                       http://www.rawseeds.org/
%--------------------------------------------------------------------------
%--------------------------------------------------------------------------
%   Authors: C. Cadena, J.D.Tardos, J. Civera, O. Garcia
%            University of Zaragoza, Spain
%--------------------------------------------------------------------------
%   Version: 1.1    15-dec-2009
%      Tested with: Matlab 7.0.1 with the Optimization Toolbox. 
%--------------------------------------------------------------------------
%   Version:    1.0    31-nov-2009
%               1.1    17-dec-2009
%--------------------------------------------------------------------------
%   History:    1.1    added choice of interval used for GT alignment in ATE
%--------------------------------------------------------------------------


Introduction: ------------------------------------------------------------
This is a toolkit for the automatic computation of some of the evaluation 
metrics for SLAM algorithms defined by RAWSEEDS. The algorithms represented in
this toolkit have been used to build some of the Benchmark Solutions associated
to RAWSEEDS' Benchmark Problems. A Benchmark Solution is basically the package
obtained by combining a SLAM algorithm with the results of its application to
one of RAWSEEDS' datasets, and includes the values of some of the metrics.
For more information, please refer to http://www.rawseeds.org/

The metrics computation toolkit includes:
* a set of MATLAB scripts to load the output of a SLAM algorithm and compare it
  with the ground truth provided with RAWSEEDS' datasets: the comparison is
  done according to the specifications of the metrics;
* a set of ground truth data files, partly copied from RAWSEEDS' datasets, and
  partly obtained by suitably post-processing those files in order to build an
  "extended ground truth";
* a set of trajectory files, representing the output of specific SLAM 
  algorithms when applied to some of RAWSEEDS' datasets.
  
By adding the trajectory files generated by your own SLAM algorithm to those
provided by this metrics computation toolkit, you will be able to use the 
scripts to automatically compute the values of the metrics for that algorithm.


About extended ground truth: ---------------------------------------------
The ground truth included with each of RAWSEEDS' datasets has been generated
with systems which are completely independent from the sensors mounted on board
the robot. However, available ground truth data do not cover the whole path of
the robot, due to the limitations of the equipment used to collect them.
This metrics computation toolkit includes an "extended ground truth", available
for some of the datasets. Such "extended GT" covers the whole trajectory of the
robot, but is *not* independent from sensor data streams.
The files have been generated by University of Freiburg, applying manual scan
matching to data from the robot's own Sick laser range finders.

For the datasets where "extended GT" is available, two files are provided:
- a file called SESSION_NAME-GT-extended.csv, containing extended GT in terms of
  poses (x, y, theta);
- a file called SESSION_NAME.relations, containing extended GT in terms of 
  relations between poses (timestamp1 timestamp2 x y z roll pitch yaw). Each
  line of this file describes the relation from the pose associated to
  timestamp1 to the pose associated to timestamp2.


Scripts: -----------------------------------------------------------------
The scripts are in the ./programs directory.
The main script (benchmark.m) loads the output of a SLAM algorithm and the
ground truth files from the selected RAWSEEDS dataset and computes two metrics:
  - Absolute Trajectory Error (ATE)
  - Relative Pose Error (RPE) 
By changing the values in the global variable PARAMETERS, the behaviour of the
script can be changed. In particular, the dataset to use, the algorithm to be 
assessed and the ground truth to be used as a reference can be specified.
For more information on the metrics, refer to http://www.rawseeds.org/


Data and trajectory files: --------------------------------------------
The files are in the ./datasets directory. It includes a separate subdirectory
for each of the data collection sessions; the names are the same used in
http://www.rawseeds.org/
Please note that the content of each subdirectory is only a small part of the
RAWSEEDS session with the same name. Note also that trajectory files (i.e., the
output of SLAM algorithms as applied to the data of a session) are present only
for a subset of the sessions, basically as examples of use of this toolkit.

Subdirectory SESSION_NAME can include the following files:

* SESSION_NAME-GROUNDTRUTH.csv
    - indoor ground truth obtained with external lasers and cameras, covering
      only part of the path of the robot [copied from the dataset]
* SESSION_NAME-GPS.csv 
    - outdoor ground truth obtained with an RTK-GPS system, covering only part
      of the path of the robot [copied from the dataset]
* SESSION_NAME-GT-extended.csv, SESSION_NAME.relations
    - extended ground truth for the complete trajectory of the robot [obtained
      with manual scan matching, performed by U. Freiburg on onboard LRF data]
* CI-GRAPH Stereo.csv        
    - SLAM Solution provided by U. Zaragoza.
* GraphSLAM Laser.csv	   
    - SLAM Solution provided by U. Freiburg.
* H-SLAM Trinocular.csv	   
    - SLAM Solution provided by Politecn. Milano and U. Milano Bicocca
* EKF Monocular.csv          
    - SLAM Solution provided by U. Zaragoza
--------------------------------------------------------------------------
