Benchmarking

euRathlon aims to be an important milestone for robotics research. Not only will it provide opportunities to put the latest robotics technologies to test under realistic disaster scenario conditions, we also recognise that there is currently an urgent need to develop useful benchmarks that will advance the field of robotics.

Benchmarking
Benchmarking can be used effectively to measure progress with respect to the state of the art. Benchmarking is well established in other disciplines such as medicine where carefully designed experiments, tailored to remove any statistical and external environment bias, are used to assess the benefits of a new technique or a new drug. In the robotics field, it is less established and there are good reasons for this. First, algorithms are embodied into a specific physical robot with its particular physical, sensing and computing constraints. Second, in any real outdoor scenario, environmental constraints cannot be controlled.

The objective of euRathlon consortium is to devise and propose assessment mechanisms to establish the benchmark in robotic systems as well as to provide well-defined scoring scales to mark competitors (teams) for the euRathlon Competition. The approach is based on metrics and criteria that are applied to robot performance in specific scenarios where all the robotic platforms face identical challenges under the same contextual conditions.

For detailed information on the benchmarking approaches and evaluation of euRathlon 2013 land and 2014 sea competitions, please download the document: https://drive.google.com/file/d/0B2fnwAhpxyZ7Tks5THI1N3ZtRFE/view?usp=sharing

For detailed information on the benchmarking approach and evaluation of euRathlon 2015 air+land+sea competition, please download the document: https://drive.google.com/file/d/0B2fnwAhpxyZ7VGlBMzFqSVpRckU/view?usp=sharing