Rawseeds will generate and publish two categories of structured benchmarks:

• Benchmark Problems (BPs), defined as the union of: (i) the detailed and unambiguous description of a task; (ii) a collection of raw multisensor data, gathered through experimental activity, to be used as the input for the execution of the task; (iii) a set of rating methodologies for the evaluation of the results of the task execution. The application of the given methodologies to the output of an algorithm or piece of software designed to solve a Benchmark Problem produces a set of scores that can be used to assess the performance of the algorithm or compare it with other algorithms.
• Benchmark Solutions (BSs), defined as the union of: (i) a BP; (ii) the detailed description of an algorithm for the solution of the BP (possibly including the source code of its implementation and/or executable code); (iii) the complete output of the algorithm when applied to the BP; (iv) the scores associated to this output, calculated with the methodology specified in the BP.

The set of sensor data (or dataset) associated to each BP is subject to a validation procedure prior to publication, to ensure absence of defects (such as data drop-outs) and correct time-syncing between separate sensor data streams. In addition to that, each dataset includes the corresponding ground truth, time-synchronized with the sensor data, and – where applicable – the results of sensor calibration. Finally, the data required to calibrate the sensors are included into the datasets as well, for users who prefer to perform calibration personally.

The collection of datasets, BPs and BSs published by Rawseeds is called the Rawseeds Benchmarking Toolkit and is freely downloadable through this website.

The main use of a BP is to test existing algorithms and compare their performance with that of alternative algorithms. The fact that a common ground for comparison exists is assured by the use of the rating methodology defined by the BP itself. A BS can be used in many ways, as it is possible to:

• compare the scores obtained by the algorithm included in the BS with the scores obtained by another algorithm applied to the same BP (remember that the rating methodology is defined by the BP, and so can be applied to different BSs associated to the same BP);

• use the output of the algorithm included in the BS to get pre-processed input data for higher level algorithms to be tested (e.g. planners);

• use the algorithm and/or the software of the BS as a “building block” to design a multi-layer system for the processing of sensor data;

• use the algorithm included in the BS as a starting point for the design of new algorithms.

The Rawseeds Project will generate the BPs and a set of BSs based on state-of-the-art robotics algorithms, but users of the Rawseeds website are encouraged to contribute to Rawseeds by submitting their own Benchmark Solutions for publication. We hope that a vital community of Rawseeds users and contributors will build up, using our Forum to communicate. Rawseeds includes mechanisms to safeguard the intellectual property of the contributors.

It is also possible for Rawseeds users to submit new Benchmark Problems for publication. However, they will be accepted only if the associated datasets have been subjected to the same exacting validation procedures that we applied on our own datasets, and if an associated ground truth with sufficient accuracy is present.

Click here to go to Rawseeds’ Benchmarking Toolkit.

A Benchmark Solution is, in a nutshell, an algorithm used to solve one of Rawseeds’ Benchmark Problems, together with its output and some additional information (such as the values of the rating methodologies associated to the BP when applied to the output of the BS). To know what is a BS in practice, and what information you should include in your own, please see this document, also included into the description of each rating methodology.

The BSs that Rawseeds will initially publish are developed by the project and based on state-of-the-art algorithms; but (here is the important bit) you are invited to add your own BSs. To do that, it is sufficient that you put your algorithms in the form of solutions to specific BPs, and submit them to Rawseeds.
To submit a new Benchmark Solution for publication, use the “Got a solution?” form on the page of the Benchmark Problem that your BS is targeted to. You need to be a registered user to use the form: if you aren’t, you can register by going to Rawseeds’ forum, clicking on “Sign In” and then on “Apply for membership” (or go directly to this page).

There are many reasons why publishing your results in the form of BSs is good for you, either if you are a researcher or a company. If you are a company, you are probably worrying about how this can coexist with the protection of your intellectual property: well, don’t. We devised specific solutions for this issue, exactly to avoid excluding companies from the benefits of publishing their results in the form of Benchmark Solutions. Rawseeds lets you publish your achievements without jeopardizing your investments… or reducing your contribution to a scant press release.