Rawseeds will generate and publish two categories of structured benchmarks:

• Benchmark Problems (BPs), defined as the union of: (i) the detailed and unambiguous description of a task; (ii) a collection of raw multisensor data, gathered through experimental activity, to be used as the input for the execution of the task; (iii) a set of rating methodologies for the evaluation of the results of the task execution. The application of the given methodologies to the output of an algorithm or piece of software designed to solve a Benchmark Problem produces a set of scores that can be used to assess the performance of the algorithm or compare it with other algorithms.
• Benchmark Solutions (BSs), defined as the union of: (i) a BP; (ii) the detailed description of an algorithm for the solution of the BP (possibly including the source code of its implementation and/or executable code); (iii) the complete output of the algorithm when applied to the BP; (iv) the scores associated to this output, calculated with the methodology specified in the BP.

The set of sensor data (or dataset) associated to each BP is subject to a validation procedure prior to publication, to ensure absence of defects (such as data drop-outs) and correct time-syncing between separate sensor data streams. In addition to that, each dataset includes the corresponding ground truth, time-synchronized with the sensor data, and – where applicable – the results of sensor calibration. Finally, the data required to calibrate the sensors are included into the datasets as well, for users who prefer to perform calibration personally.

The collection of datasets, BPs and BSs published by Rawseeds is called the Rawseeds Benchmarking Toolkit and is freely downloadable through this website.

The main use of a BP is to test existing algorithms and compare their performance with that of alternative algorithms. The fact that a common ground for comparison exists is assured by the use of the rating methodology defined by the BP itself. A BS can be used in many ways, as it is possible to:

• compare the scores obtained by the algorithm included in the BS with the scores obtained by another algorithm applied to the same BP (remember that the rating methodology is defined by the BP, and so can be applied to different BSs associated to the same BP);

• use the output of the algorithm included in the BS to get pre-processed input data for higher level algorithms to be tested (e.g. planners);

• use the algorithm and/or the software of the BS as a “building block” to design a multi-layer system for the processing of sensor data;

• use the algorithm included in the BS as a starting point for the design of new algorithms.

The Rawseeds Project will generate the BPs and a set of BSs based on state-of-the-art robotics algorithms, but users of the Rawseeds website are encouraged to contribute to Rawseeds by submitting their own Benchmark Solutions for publication. We hope that a vital community of Rawseeds users and contributors will build up, using our Forum to communicate. Rawseeds includes mechanisms to safeguard the intellectual property of the contributors.

It is also possible for Rawseeds users to submit new Benchmark Problems for publication. However, they will be accepted only if the associated datasets have been subjected to the same exacting validation procedures that we applied on our own datasets, and if an associated ground truth with sufficient accuracy is present.

Click here to go to Rawseeds’ Benchmarking Toolkit.