BitTorrent is a protocol for efficient peer-to-peer file sharing. The term “peer-to-peer” means that each user requesting a specific file also acts as a distributor for that file towards other users. This leads to massively faster download speed for every user, something that is very needed for the case of Rawseeds, where extremely large files have to be distributed within reasonable time. An explanation of why and how the use of BitTorrent to distribute the datasets is a key element of Rawseeds can be read here.

The key concepts behind BitTorrent are the following:

• every file to be distributed is subdivided into chunks
• the chunks are distributed to different users requesting that file
• users exchange already downloaded portions as well as downloading new ones
• “smart” distribution policies (e.g., rarest chunk first) are implemented to further speed up things

The elements of a BitTorrent system for the distribution of a file are:

1. the peers, i.e., the PCs requesting the file;
2. the seeds, i.e., the peers possessing a complete copy of the file;
3. the tracker, i.e., the PC coordinating the sharing of the file.

The Rawseeds project set up the tracker and a few seeds. However, many other seeds are needed, so your help is crucial! In fact, we are a small project: we do not have the resources to set up many seeds by ourselves, or to buy ultra-wideband hosting services. So we ask you to share a little bit of your upload bandwidth to help other users get the files (just as you were helped by previous users). How? In a nutshell, keeping your BitTorrent client (i.e., the program you used to get the data) running after you have finished the download, and ensuring that it is restarted at every boot. Read here for the details. Thank you!

What you need to get Rawseeds’ datasets

To be able to download Rawseeds’ datasets, you need two things:

1. a BitTorrent client program;
2. an open TCP port towards that program (this is only required if you want fast download).

First, let’s see what a BitTorrent client is.
It is a little program that is able to manage the exchange of data through the internet using the BitTorrent protocol. BitTorrent clients are available for free for any software platform (and particularly for Windows, Mac, Linux); actually, they have become so commonplace that they are frequently installed by default together with the OS (this is the case of many Linux distributions, for instance). Here you can find a list of available BitTorrent clients. Some of the most commonly used are μTorrent (Windows only), Transmission (Mac and Linux only), KTorrent ( Linux, Mac, Windows): however, everyone has her preferences here, so let’s just say that any of them is good for Rawseeds (if we can say a word about this… prefer open source software!). Just ask your friends and colleagues: we’re sure that some (or possibly all…) of them already use BitTorrent.

Now, let’s describe what an open TCP port is and how to get it.
Your BitTorrent client will point out to you if the port it is trying to use to receive connections is closed. If it isn’t, you don’t even need to read this paragraph. If it is closed, this means that your PC is behind a firewall (be it hardware or software, such as Windows’ own), and that an external PC cannot try and connect to your running applications unless you instruct your firewall to accept incoming connections. Incoming connections are directed to one of the 65536 ports defined by the TCP protocol, each of which is associated to a specific service or application (or to none). One of this ports is associated to your BitTorrent client: you must open it in order to receive connections from other Rawseeds users: in this way, your download speeds will be much greater.
The specific port to open is set in the Preferences (or Options or whatever) of your BitTorrent client. If you don’t have a reason to change it, use the default value set by the program. To open the port, you have to access your firewall and specify that incoming connections to that port must be allowed. Unfortunately, we can’t give you detailed instructions about this process, because it is heavily dependent on how your firewall is implemented. Usually, hardware firewalls are fitted with a web interface accessible with a web browser, while software firewalls have their own configuration menus.
Anyway, if you work in a company, an university, or any other large-scale structure the firewalls are almost surely not operated by you, but by your network administrators. So you simply have to ask them to open for your machine the port specified by the BitTorrent client you chose. A word of warning about this… network administrators do not usually like people using BitTorrent within their network. In fact, this kind of protocol is mostly used to share media such as music and movies. Therefore, be prepared to be asked by a frowning tecnician: “Exactly, why do you need this port to be opened?”.

How to get Rawseeds’ datasets

Once you have installed the BitTorrent client of your choice, you’re done.
While browsing Rawseeds’ website, if you find the link to a file you want to dowload, just click on it. As usual, you will be asked by your web browser what to do with it (e.g., save or open it). If the file has a “.torrent” extension, choose “Open with [your BitTorrent client program]. The program will open, and download of the file will begin (possibly after you have been asked for a further confirmation by the BitTorrent client itself).
The amount of resources (processor, RAM, …) used by a BitTorrent client is very small: so much so that they can be considered as negligible with a modern PC. Therefore, you can leave the program running all the time, while you do your work as usual. You can stop the BitTorrent program whenever you like, without losing the already downloaded data; you can also, of course, turn off the PC. However be sure to re-run the BitTorrent client as you turn on the PC, or the download will not resume (the best way to do this is to put the BitTorrent client in the list of applications that are automatically started every time the PC is turned on).
The BitTorrent client will put the downloaded files into a specific directory that is set by its own “Preferences” (or “Options” or whatever). It will also inform you about how much of each file you have already downloaded. For extremely large files, such as some of Rawseeds’ ones, download can take many hours or even days. Download speed for a given file depends (on the download bandwidth of your internet connection and) on how many other users are helping you by seeding the same file. This is why your help as a seed is crucial: please, take away some minutes from your work to read how easily you can become a seed for Rawseeds’ datasets and why you should do it.

• Benchmark Problems (BPs), defined as the union of: (i) the detailed and unambiguous description of a task; (ii) a collection of raw multisensor data, gathered through experimental activity, to be used as the input for the execution of the task; (iii) a set of rating methodologies for the evaluation of the results of the task execution. The application of the given methodologies to the output of an algorithm or piece of software designed to solve a Benchmark Problem produces a set of scores that can be used to assess the performance of the algorithm or compare it with other algorithms.
• Benchmark Solutions (BSs), defined as the union of: (i) a BP; (ii) the detailed description of an algorithm for the solution of the BP (possibly including the source code of its implementation and/or executable code); (iii) the complete output of the algorithm when applied to the BP; (iv) the scores associated to this output, calculated with the methodology specified in the BP.

The set of sensor data (or dataset) associated to each BP is subject to a validation procedure prior to publication, to ensure absence of defects (such as data drop-outs) and correct time-syncing between separate sensor data streams. In addition to that, each dataset includes the corresponding ground truth, time-synchronized with the sensor data, and – where applicable – the results of sensor calibration. Finally, the data required to calibrate the sensors are included into the datasets as well, for users who prefer to perform calibration personally.

The collection of datasets, BPs and BSs published by Rawseeds is called the Rawseeds Benchmarking Toolkit and is freely downloadable through this website.

The main use of a BP is to test existing algorithms and compare their performance with that of alternative algorithms. The fact that a common ground for comparison exists is assured by the use of the rating methodology defined by the BP itself. A BS can be used in many ways, as it is possible to:

• compare the scores obtained by the algorithm included in the BS with the scores obtained by another algorithm applied to the same BP (remember that the rating methodology is defined by the BP, and so can be applied to different BSs associated to the same BP);

• use the output of the algorithm included in the BS to get pre-processed input data for higher level algorithms to be tested (e.g. planners);

• use the algorithm and/or the software of the BS as a “building block” to design a multi-layer system for the processing of sensor data;

• use the algorithm included in the BS as a starting point for the design of new algorithms.

The Rawseeds Project will generate the BPs and a set of BSs based on state-of-the-art robotics algorithms, but users of the Rawseeds website are encouraged to contribute to Rawseeds by submitting their own Benchmark Solutions for publication. We hope that a vital community of Rawseeds users and contributors will build up, using our Forum to communicate. Rawseeds includes mechanisms to safeguard the intellectual property of the contributors.

It is also possible for Rawseeds users to submit new Benchmark Problems for publication. However, they will be accepted only if the associated datasets have been subjected to the same exacting validation procedures that we applied on our own datasets, and if an associated ground truth with sufficient accuracy is present.

Click here to go to Rawseeds’ Benchmarking Toolkit.

A Benchmark Solution is, in a nutshell, an algorithm used to solve one of Rawseeds’ Benchmark Problems, together with its output and some additional information (such as the values of the rating methodologies associated to the BP when applied to the output of the BS). To know what is a BS in practice, and what information you should include in your own, please see this document, also included into the description of each rating methodology.

The BSs that Rawseeds will initially publish are developed by the project and based on state-of-the-art algorithms; but (here is the important bit) you are invited to add your own BSs. To do that, it is sufficient that you put your algorithms in the form of solutions to specific BPs, and submit them to Rawseeds.
To submit a new Benchmark Solution for publication, use the “Got a solution?” form on the page of the Benchmark Problem that your BS is targeted to. You need to be a registered user to use the form: if you aren’t, you can register by going to Rawseeds’ forum, clicking on “Sign In” and then on “Apply for membership” (or go directly to this page).

There are many reasons why publishing your results in the form of BSs is good for you, either if you are a researcher or a company. If you are a company, you are probably worrying about how this can coexist with the protection of your intellectual property: well, don’t. We devised specific solutions for this issue, exactly to avoid excluding companies from the benefits of publishing their results in the form of Benchmark Solutions. Rawseeds lets you publish your achievements without jeopardizing your investments… or reducing your contribution to a scant press release.