Evaluation of grid maps produced by a SLAM algorithm

When evaluating the performance of a SLAM algorithm, quantifying the produced map quality is one of the most important criteria. Often, the produced map is compared with (1) a ground-truth map (which can be easily obtained in simulation) or (2) with another existing map that is considered accurate (in case of real world experiment where the ground-truth is not always available ).

Basically, grid maps are images, so image similarity measurement metrics can be used in this case. In this post, we consider three different metrics: Mean Square Error (MSE), K-nearest based normalized error (NE) and Structure Similarity Index (SSIM)

Setting up a software stack for autonomous mono robot exploration and mapping using ROS

Prerequisites

This setup is performed and documented on an Ubuntu system with the following software stack:

Ubuntu 16.04 LTS (may work on other distribution though)
ROS kinetic: if you don't have ROS pre-installed, please refer to this tutorial http://wiki.ros.org/kinetic/Installation/Ubuntu. I suggest to use the full desktop installation configuration, this will take a while (> 2 GB)
Gazebo for simulation (it is installed by default if you choose the full desktop installation when installing ROS)
GIT (sudo apt-get install git)
Catkin for package building (installed by default when installing ROS)

To follow this post, some basic knowledge on ROS is needed:

http://wiki.ros.org/ROS/StartGuide
ROS launch: http://wiki.ros.org/roslaunch/XML
SLAM: https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping
Books about ROS: http://wiki.ros.org/Books