Filip Lemic

Platform for benchmarking of RF-based indoor localization solutions

Over the last few years, the number of indoor localization solutions has grown exponentially, and a wide variety of different technologies and approaches are being explored. Unfortunately, there is currently no established standardized evaluation method for comparing their performance. As a result, each solution is evaluated in a different environment using proprietary evaluation metrics. Consequently, it is currently extremely hard to objectively compare the performance of multiple localization solutions with each other. To address the problem, we present the EVARILOS Benchmarking Platform, which enables automated evaluation and comparison of multiple solutions in different environments using multiple evaluation metrics. We propose a testbed-independent benchmarking platform, combined with multiple testbed-dependent plugins for executing experiments and storing performance results. The platform implements the standardized evaluation method described in the EVARILOS Benchmarking Handbook, which is aligned with the upcoming ISO/IEC 18305 standard “Test and Evaluation of Localization and Tracking Systems.” The platform and plug-ins can be used in real time on existing wireless testbed facilities, while also supporting a remote offline evaluation method using precollected data traces. Using these facilities, and analyzing and comparing the performance of three different localization solutions, we demonstrate the need for objective evaluation methods that consider multiple evaluation criteria in different environments.

Experimental decomposition of the performance of fingerprinting-based localization algorithms

Despite their popularity, the current praxis of comparative experimental evaluation of fingerprinting-based localization algorithms is lacking rigor, with studies typically following an ad-hoc evaluation process and focusing on black-box comparison of complete algorithms. In this paper we present a systematic benchmarking methodology that is focused on gaining finegrained insight about the relative contributions of the individual phases of the fingerprinting-based localization algorithms to their overall performance. To this end, we decompose the localization algorithms in common phases (collection of raw measurements, creation of fingerprints, pattern matching and post-processing) and systematically asses the performance of different procedures that can be applied in each of these phases. We illustrate the application of the proposed methodology using a comprehensive experimental case-study of 3 WiFi fingerprinting algorithms with 4 raw RSSI collection procedures, 3 fingerprint creation and pattern matching procedures, 4 different post-processing procedures in 3 testbeds and 4 evaluation scenarios, resulting in 36 individual experiments. The results demonstrate that in the evaluated scenarios, a lower number of WiFi APs and rather simple fingerprint creation and pattern matching can achieve better performance in terms of location accuracy than more sophisticated alternatives. The results also show that postprocessing steps like k-Nearest Neighbours (kNN) procedure are indeed effective in reducing the localization error variability and extremes, thus increasing the stability of the location estimation.

Infrastructure for benchmarking RF-based indoor localization under controlled interference

The proliferation of RF-based indoor localization solutions raises the need for testing systems that enable objective evaluation of their functional and non functional properties. We introduce a testbed and cloud infrastructure for supporting automatized benchmarking of RF-based indoor localization solutions under controlled interference. For evaluating the impact of RF interference on the performance of benchmarked solution, the infrastructure leverages various interference generation and monitoring devices. The infrastructure obtains location estimates from the System Under Test (SUT) using a well defined interface, and the estimates are subsequently processed in a dedicated metrics computation engine and stored in the dedicated engine for storing the results of benchmarking experiments. The infrastructure further includes a robotic mobility platform which serves as a reference localization system and can transport the localized device of the evaluated indoor localization solution in an autonomous and repeatable manner. We present the accuracy of our autonomous mobility platform in two different setups, showing that, due to the high accuracy, the location estimation provided by the platform can be considered as the reference localization system for benchmarking of RF-based indoor localization solutions. The results, as well as the raw data from the benchmarking experiments, can be stored into the dedicated publicly available services which gives the opportunity of reusing the same data for benchmarking different solutions. Finally, we present the capabilities of the testbed and cloud infrastructure on the use-case of benchmarking of an example WiFi fingerprinting-based indoor localization solution in four different interference scenarios.

Web-based platform for evaluation of RF-based indoor localization algorithms

The experimental efforts for optimizing the performance of RF-based indoor localization algorithms for specific environments and scenarios is time consuming and costly. In this work, we address this problem by providing a publicly accessible platform for streamlined experimental evaluation of RF-based indoor localization algorithms, without the need of a physical testbed infrastructure. We also offer an extensive set of raw measurements that can be used as input data for indoor localization algorithms. The datasets are collected in multiple testbed environments, with various densities of measurement points, using different measuring devices and in various scenarios with controlled RF interference. The platform encompasses two core services: one focused on storage and management of raw data, and one focused on automated calculation of metrics for performance characterization of localization algorithms. Tools for visualization of the raw data, as well as software libraries for convenient access to the platform from MATLAB and Python, are also offered. By contrasting its fidelity and usability with respect to remote experiments on dedicated physical testbed infrastructure, we show that the virtual platform produces comparative performance results while offering significant reduction in the complexity, time and labor overheads.

Comparability of RF-based indoor localisation solutions in heterogeneous environments: an experimental study

The growing popularity of indoor localisation research has resulted in a significant amount of research papers describing and evaluating innovative localisation solutions. Unfortunately, the results from most of these research papers cannot easily be compared since they are evaluated in different environments, use different evaluation criteria and typically tailor their solutions towards a single testbed environment. To evaluate how these different conditions influence the localisation performance, in this paper an exhaustive set of experiments has been performed, in which three different localisation solutions have been evaluated using multiple metrics in three different test environments: two types of office environments and an industry-like factory environment. None of the used localisation solutions was previously optimised for any of these test environments and they were all evaluated under similar conditions. The results reveal several weaknesses in the evaluation methods used in the majority of existing scientific literature of indoor localisation solutions.

Interference Effect on Localization Solutions: Signal Feature Perspective

We study the effect of interference on localization algorithms through the study of the interference effect on signal features that are used for localization. Particularly, the effect of interference on packet-based Received Signal Strength Indicator (RSSI), reported by IEEE 802.11 and IEEE 802.15.4 technologies, and on Time of Flight (ToF), reported by IEEE 802.15.4 technology, is studied using both theoretical discussions and experimental verifications. As for the RSSI values, using an information theoretic formulation, we distinguish three operational regimes and we show that the RSSI values, in dBm, remain unchanged in the noise-limited regime, increase almost linearly with interference power in dBm in the interference- limited regime and cannot be obtained due to packet loss in the collision regime. The maximum observable RSSI variation is dependent on the transmission rate and Signal to Noise Ratio (SNR). We also show that ToF is, interestingly, decreased under interference which is caused in the symbol synchronization procedure at the receiver. After providing the experimental results, we discuss how the localization algorithms are affected by interference.

Toward Extrapolation of WiFi Fingerprinting Performance Across Environments

Out of the plethora of approaches for indoor localization, WiFi-based fingerprinting offers attractive trade-off between deployment overheads and accuracy. This has motivated intense research interest resulting in many proposed algorithms which are typically evaluated only in a single or small number of discrete environments. When the end-users environment is not part of the evaluated set, it remains unclear if and to what extent the reported performance results can be extrapolated to this new environment. In this paper, we aim at establishing a relationship between the similarities among a set of different deployment environments and parameterizations of fingerprinting algorithms on one side, and the performance of these algorithms on the other. We hypothesize about the factors that can be used to capture the degree of similarity among environments and parameterizations of the algorithms, and proceed to systematically analyze the performance of two fingerprinting algorithms across four environments with different levels of similarity. The results show that the localization error distributions have small statistical difference across environments and parameterizations that are considered similar according to our hypothesis. As the level of similarity is decreased, we demonstrate that the relative performance of the algorithms can still be preserved across environments. For dissimilar environments, the localization errors demonstrate larger statistical differences.

Increasing Interference Robustness of WiFi Fingerprinting by Leveraging Spectrum Information

RF-based indoor localization is constantly gaining popularity, and WiFi-based fingerprinting algorithms belong to the most promising candidates due to their well-known advantages. While it is known that RF interference can adversely influence the accuracy of those algorithms, it is still unclear if this effect could be efficiently mitigated. To this end, we demonstrate the impact and propose a procedure for reducing the influence of RF interference on WiFi beacon packets RSSI-based fingerprinting algorithms. The proposed procedure adjusts the RSSI measurements based on estimates of their variability caused by RF interference. For estimating the variability in RSSI measurements, the procedure leverages information about the spectrum power levels in the frequency band on which the fingerprinting algorithm performs. The proposed procedure can be inserted in the usual workflow of fingerprinting algorithms. We experimentally compared the performance of two well-known WiFi-based fingerprinting algorithms without and with the proposed interference mitigation procedure. Our experimental evaluation in different interference scenarios shows that the proposed procedure for mitigating the influence of RF interference significantly improves the localization accuracy, while it does not notably increase the latency of evaluated fingerprinting algorithms.

Localization as a feature of mmWave communication

mmWave (millimeter-Wave) is a very promising technology for the future wireless communication. To mitigate its high attenuation characteristics, mmWave communication frequently employs directional beamforming for both transmission and reception. Localization commonly takes advantage of directionality in RF frequencies in urban and indoor environments. In this paper, we use lessons learned from classical RF-based localization for discussing a set of feasible localization approaches in the context of mmWave bands. We further map the requirements of each discussed localization approach to design requirements for future mmWave devices and assess the expected accuracy of such approaches for a set of realistic scenarios. Our results show that mmWave-based localization is promising in both its availability and accuracy, even in the presence of a limited number of localization anchor nodes.

Toward standardized localization service

Localization services available on todays mobile devices are proprietary and leverage a limited set of sources of location information. Integration of new location estimation methods is therefore cumbersome, requiring adaptation to the specific interfaces of the proprietary location service. In addition, location-based applications are tightly interwoven with the location service that is typically provided by the operating system, hence these applications require significant restructuring to be able to run with another location service. To address these problems, we propose a modular localization service architecture that consists of location-based applications, an integrated location service enabling a fusion of so-called elementary location services, and resources for generating location information. A unified style of interaction among these components is enabled by a set of well-defined Application Programming Interfaces (APIs). The practicability and advantages of the proposed design is demonstrated by outlining how the APIs can be realized using modern types of component interactions.