Tom Van Haute

Platform for benchmarking of RF-based indoor localization solutions

Over the last few years, the number of indoor localization solutions has grown exponentially, and a wide variety of different technologies and approaches are being explored. Unfortunately, there is currently no established standardized evaluation method for comparing their performance. As a result, each solution is evaluated in a different environment using proprietary evaluation metrics. Consequently, it is currently extremely hard to objectively compare the performance of multiple localization solutions with each other. To address the problem, we present the EVARILOS Benchmarking Platform, which enables automated evaluation and comparison of multiple solutions in different environments using multiple evaluation metrics. We propose a testbed-independent benchmarking platform, combined with multiple testbed-dependent plugins for executing experiments and storing performance results. The platform implements the standardized evaluation method described in the EVARILOS Benchmarking Handbook, which is aligned with the upcoming ISO/IEC 18305 standard “Test and Evaluation of Localization and Tracking Systems.” The platform and plug-ins can be used in real time on existing wireless testbed facilities, while also supporting a remote offline evaluation method using precollected data traces. Using these facilities, and analyzing and comparing the performance of three different localization solutions, we demonstrate the need for objective evaluation methods that consider multiple evaluation criteria in different environments.

Web-based platform for evaluation of RF-based indoor localization algorithms

The experimental efforts for optimizing the performance of RF-based indoor localization algorithms for specific environments and scenarios is time consuming and costly. In this work, we address this problem by providing a publicly accessible platform for streamlined experimental evaluation of RF-based indoor localization algorithms, without the need of a physical testbed infrastructure. We also offer an extensive set of raw measurements that can be used as input data for indoor localization algorithms. The datasets are collected in multiple testbed environments, with various densities of measurement points, using different measuring devices and in various scenarios with controlled RF interference. The platform encompasses two core services: one focused on storage and management of raw data, and one focused on automated calculation of metrics for performance characterization of localization algorithms. Tools for visualization of the raw data, as well as software libraries for convenient access to the platform from MATLAB and Python, are also offered. By contrasting its fidelity and usability with respect to remote experiments on dedicated physical testbed infrastructure, we show that the virtual platform produces comparative performance results while offering significant reduction in the complexity, time and labor overheads.

Comparability of RF-based indoor localisation solutions in heterogeneous environments: an experimental study

The growing popularity of indoor localisation research has resulted in a significant amount of research papers describing and evaluating innovative localisation solutions. Unfortunately, the results from most of these research papers cannot easily be compared since they are evaluated in different environments, use different evaluation criteria and typically tailor their solutions towards a single testbed environment. To evaluate how these different conditions influence the localisation performance, in this paper an exhaustive set of experiments has been performed, in which three different localisation solutions have been evaluated using multiple metrics in three different test environments: two types of office environments and an industry-like factory environment. None of the used localisation solutions was previously optimised for any of these test environments and they were all evaluated under similar conditions. The results reveal several weaknesses in the evaluation methods used in the majority of existing scientific literature of indoor localisation solutions.

Toward Extrapolation of WiFi Fingerprinting Performance Across Environments

Out of the plethora of approaches for indoor localization, WiFi-based fingerprinting offers attractive trade-off between deployment overheads and accuracy. This has motivated intense research interest resulting in many proposed algorithms which are typically evaluated only in a single or small number of discrete environments. When the end-users environment is not part of the evaluated set, it remains unclear if and to what extent the reported performance results can be extrapolated to this new environment. In this paper, we aim at establishing a relationship between the similarities among a set of different deployment environments and parameterizations of fingerprinting algorithms on one side, and the performance of these algorithms on the other. We hypothesize about the factors that can be used to capture the degree of similarity among environments and parameterizations of the algorithms, and proceed to systematically analyze the performance of two fingerprinting algorithms across four environments with different levels of similarity. The results show that the localization error distributions have small statistical difference across environments and parameterizations that are considered similar according to our hypothesis. As the level of similarity is decreased, we demonstrate that the relative performance of the algorithms can still be preserved across environments. For dissimilar environments, the localization errors demonstrate larger statistical differences.

A hybrid indoor localization solution using a generic architectural framework for sparse distributed wireless sensor networks

Indoor localization and navigation using wireless sensor networks is still a big challenge if expensive sensor nodes are not involved. Previous research has shown that in a sparse distributed sensor network the error distance is way too high. Even room accuracy can not be guaranteed. In this paper, an easy-to-use generic positioning framework is proposed, which allows users to plug in a single or multiple positioning algorithms. We illustrate the usability of the framework by discussing a new hybrid positioning solution. The combination of a weighted (range-based) and proximity (range-free) algorithm is made. Both solutions separately have an average error distance of 13.5m and 2.5m respectively. The latter result is quite accurate due to the fact that our testbeds are not sparse distributed. Our hybrid algorithm has an average error distance of 2.66m only using a selected set of nodes, simulating a sparse distributed sensor network. All our experiments have been executed in the iMinds testbed: namely at “de Zuiderpoort”. These algorithms are also deployed in two real-life environments: “De Vooruit” and “De Vijvers”.

Benchmarking of Localization Solutions

Indoor localization solutions are key enablers for next-generation indoor navigation and track and tracing solutions. As a result, an increasing number of different localization algorithms have been proposed and evaluated in scientific literature. However, many of these publications do not accurately substantiate the used evaluation methods. In particular, many authors utilize a different number of evaluation points, but they do not (i) analyze if the number of used evaluation points is sufficient to accurately evaluate the performance of their solutions and (ii) report on the uncertainty of the published results. To remedy this, this paper evaluates the influence of the selection of evaluation points. Based on statistical parameters such as the standard error of the mean value, an estimator is defined that can be used to quantitatively analyze the impact of the number of used evaluation points on the confidence interval of the mean value of the obtained results. This estimator is used to estimate the uncertainty of the presented accuracy results, and can be used to identify if more evaluations are required. To validate the proposed estimator, two different localization algorithms are evaluated in different testbeds and using different types of technology, showing that the number of required evaluation points does indeed vary significantly depending on the evaluated solution.

Demonstration Abstract: Platform for Benchmarking RF-Based Indoor Localization Solutions

This demonstration presents a user-friendly Web-based platform that supports intuitive remote benchmarking of different indoor localization solutions. It reduces the barriers for experimental evaluation and fair comparison of their performance across a set of testing environments. The platform is aimed at addressing the limitations in the current praxis of publishing indoor localization research evaluated only in local, potentially biased environments. To this end, it provides a holistic support for the benchmarking process offering: (i) multiple pre-collected raw data-traces from different RF technologies, (ii) high-level interface to control remote wireless testbed facilities, and (iii) a set of tools for creating, storing, comparing and visualizing the performance results of multiple indoor localization solutions.

Optimizing Time-of-Arrival Localization Solutions for Challenging Industrial Environments

Since Global Positioning System technologies cannot be used indoors, a significant amount of research focuses on developing radio-frequency-based alternatives for indoor localization. Unfortunately, most of the suggested solutions for indoor localization consist of theoretical work or have been evaluated in non-industrial environments, typically office spaces. To evaluate the influence of industrial environments on localization accuracy, in this paper, a time-of-arrival (ToA) approach was used to determine the stationary locations of a robot inside the w-iLab.t II testbed, an open industrial-like environment containing several metal obstacles. The ToA method utilizes the measured propagation time of a radio wave between a sender and receiver to estimate their corresponding distance. This paper evaluates several industrial-related deployment aspects that influence location accuracy and describes how their negative impact can be reduced, resulting in an almost 50% accuracy improvement in industrial environments.