Diego Martín

On the measurement of test collection reliability

The reliability of a test collection is proportional to the number of queries it contains. But building a collection with many queries is expensive, so researchers have to find a balance between reliability and cost. Previous work on the measurement of test collection reliability relied on data-based approaches that contemplated random what if scenarios, and provided indicators such as swap rates and Kendall tau correlations. Generalizability Theory was proposed as an alternative founded on analysis of variance that provides reliability indicators based on statistical theory. However, these reliability indicators are hard to interpret in practice, because they do not correspond to well known indicators like Kendall tau correlation. We empirically established these relationships based on data from over 40 TREC collections, thus filling the gap in the practical interpretation of Generalizability Theory. We also review the computation of these indicators, and show that they are extremely dependent on the sample of systems and queries used, so much that the required number of queries to achieve a certain level of reliability can vary in orders of magnitude. We discuss the computation of confidence intervals for these statistics, providing a much more reliable tool to measure test collection reliability. Reflecting upon all these results, we review a wealth of TREC test collections, arguing that they are possibly not as reliable as generally accepted and that the common choice of 50 queries is insufficient even for stable rankings.

Cyber–physical systems: Extending pervasive sensing from control theory to the Internet of Things

Abstract Essentially, the emerging term “Cyber–Physical Systems (CPS)” is an architectural paradigm in which the pervasive sensing technologies represent a fundamental part. Originally defined in the computer sciences domain, the term Cyber–Physical Systems has been adapted to very different domains such as the control theory or electronic engineering. Even, some authors understand CPS as a particular scenario of the Internet of Things (IoT) based on pervasive sensing. Furthermore, recently, some works propose a definition for CPS including all the features described in the different domains. In this paper we provide a comprehensive analysis of the nature and characteristics of the different proposals, discuss the recent attempts to standardize CPS, and review the state-of-the-art on CPS for each technological domain. We compare those different proposals on CPS, discuss about some related terms and technologies and conclude by describing the main research challenges in the area.Essentially, the emerging term CyberPhysical Systems (CPS) is an architectural paradigm in which the pervasive sensing technologies represent a fundamental part. Originally defined in the computer sciences domain, the term CyberPhysical Systems has been adapted to very different domains such as the control theory or electronic engineering. Even, some authors understand CPS as a particular scenario of the Internet of Things (IoT) based on pervasive sensing. Furthermore, recently, some works propose a definition for CPS including all the features described in the different domains. In this paper we provide a comprehensive analysis of the nature and characteristics of the different proposals, discuss the recent attempts to standardize CPS, and review the state-of-the-art on CPS for each technological domain. We compare those different proposals on CPS, discuss about some related terms and technologies and conclude by describing the main research challenges in the area.

Enhancing Evacuation Plans with a Situation Awareness System Based on End-User Knowledge Provision.

Recent disasters have shown that having clearly defined preventive procedures and decisions is a critical component that minimizes evacuation hazards and ensures a rapid and successful evolution of evacuation plans. In this context, we present our Situation-Aware System for enhancing Evacuation Plans (SASEP) system, which allows creating end-user business rules that technically support the specific events, conditions and actions related to evacuation plans. An experimental validation was carried out where 32 people faced a simulated emergency situation, 16 of them using SASEP and the other 16 using a legacy system based on static signs. From the results obtained, we compare both techniques and discuss in which situations SASEP offers a better evacuation route option, confirming that it is highly valuable when there is a threat in the evacuation route. In addition, a study about user satisfaction using both systems is presented showing in which cases the systems are assessed as satisfactory, relevant and not frustrating.

A comparison of the optimality of statistical significance tests for information retrieval evaluation

Previous research has suggested the permutation test as the theoretically optimal statistical significance test for IR evaluation, and advocated for the discontinuation of the Wilcoxon and sign tests. We present a large-scale study comprising nearly 60 million system comparisons showing that in practice the bootstrap, t-test and Wilcoxon test outperform the permutation test under different optimality criteria. We also show that actual error rates seem to be lower than the theoretically expected 5%, further confirming that we may actually be underestimating significance.

Self-configuration in humanized Cyber-Physical Systems

Most works on Cyber-Physical Systems (CPS) are based on classic hardware infrastructures made of sensors, actuators and processing devices. Usual self-configuration technologies, then, do not allow humans to be integrated in CPS as service providers. Therefore, in this work we propose a new self-configuration technology for humanized CPS. The proposed technology uses simple binary and mathematical operations in order to reduce the convergence time, improve the scalability and address the dynamism introduced by humans into CPS. Besides, a human-oriented quality-of-service algorithm based on the Maslow pyramid is also introduced. Moreover, an experimental validation is conducted in order to validate the proposed solution as a useful and scalable self-configuration technology for humanized Cyber-Physical Systems.

Supporting variability dependencies for rule-based service compositions in prosumer environments

Prosumer service provision is characterised by the creation, composition and sharing of services by non-expert users. Among the various creation strategies, rule-based service composition offers many advantages for prosumers. Variability in service composition and its implications regarding creation flexibility and domain adaptation can be associated to service templates in the form of rules, and shared by prosumers in a cooperative way. In this work we define a variability model, integrable in composable and customisable service templates, which supports adaptability in the form of variations and their dependencies. We consider this model in the development of a rule-based service composition environment integrating a creation wizard, guiding the prosumer through the customisation process, and some solutions for dependency problems, detecting and correcting prosumer undesired behaviour in a fault-tolerant approach. Finally, we implement the variability model by using web service technologies and evaluate how these technologies improve the dynamic provision of these variations, enriching the service composition process in prosumer environments.

An Ambient Intelligence Framework for End-User Service Provisioning in a Hospital Pharmacy: a Case Study

End-user development is a new trend to provide tailored services to dynamic environments such as hospitals. These services not only facilitate daily work for pharmacy personnel but also improve self-care in elder people that are still related to hospital, such as discharged patients. This paper presents an ambient intelligence (AmI) environment for End-user service provisioning in the pharmacy department of Gregorio Marañón Hospital in Madrid, composed of a drug traceability infrastructure (DP-TraIN) and a ubiquitous application for enabling the pharmacy staff to create and execute their own services for facilitating drug management and dispensing. The authors carried out a case study with various experiments where different roles from the pharmacy department of Gregorio Marañón Hospital were involved in activities such as drug identification, dispensing and medication administering. The authors analyzed the effort required to create services by pharmacy staff, the discharged patients’ perception of the AmI environment and the quantifiable benefits in reducing patient waiting time for drug dispensing.

Theoretical Comparison between Diffused and Epitaxial GaSb TPV Cells

This work deals with the analysis and optimization of GaSb TPV cells. First, the best configuration for the classical Zn‐diffused structures is discussed, regarding both the semiconductor structure and grid designs for different illumination conditions. Then, the theoretical performance of both p/n and n/p GaSb epitaxial cell structures is calculated and compared to the diffused‐cell approach, weighing up the potential benefits of each option.

Building Smart Adaptable Cyber-Physical Systems: Definitions, Classification and Elements

The provision of systems that join the information technologies with the physical world has been one of the most popular issues in research in the last fifteen years. Nevertheless, the complexity associated with these systems prevented many authors from providing a theoretical formalization. Even it is difficult to find a consensus name or a definition for this new type of systems. Therefore, in this work we propose a theoretical and technical formalization for these solutions, which includes a name at the forefront of research: smart adaptable cyber-physical systems (SACPS). We also present a complete definition for the SACPS, and explain the elements and subsystem interaction.

Improving Learning Tasks for Mentally Handicapped People Using AmI Environments Based on Cyber-Physical Systems

In this research work it is presented a preliminary prototype for an ambient intelligence scenario based on cyber-physical systems for improving learning tasks. The system proposed is composed of a cyber-glove, a worktable (both with RFID and NFC detection zones) and a AmI software application for modeling and workflow guidance. The authors carried out a case study where 12 mentally handicapped people and 3 trainers were involved executing workflows creation and performing and controlling tasks. The results obtained indicate that this kind of solutions are feasible, but due to the problem complexity and to the fact that the proposed solution is a preliminary version, we have found many issues to be solved in next versions. This research helped us to uncover these issues and design a better system.