Susana Brandão

CoBots: Collaborative robots servicing multi-floor buildings

In this video we briefly illustrate the progress and contributions made with our mobile, indoor, service robots CoBots (Collaborative Robots), since their creation in 2009. Many researchers, present authors included, aim for autonomous mobile robots that robustly perform service tasks for humans in our indoor environments. The efforts towards this goal have been numerous and successful, and we build upon them. However, there are clearly many research challenges remaining until we can experience intelligent mobile robots that are fully functional and capable in our human environments.

Development and validation of a visual body condition scoring system for dairy goats with picture-based training

Body condition scoring (BCS) is the most widely used method to assess changes in body fat reserves, which reflects its high potential to be included in on-farm welfare assessment protocols. Currently used scoring systems in dairy goats require animal restraint for body palpation. In this study, the Animal Welfare Indicators project (AWIN) proposes to overcome this constraint by developing a scoring system based only on visual assessment. The AWIN visual body condition scoring system highlights representative animals from 3 categories: very thin, normal, and very fat, and was built from data sets with photographs of animals scored by a commonly used 6-point scoring system that requires palpation in 2 anatomical regions. Development of the AWIN scoring system required 3 steps: (1) identification and validation of a body region of interest; (2) sketching the region from photographs; and (3) creation of training material. The scoring systems reliability was statistically confirmed. An initial study identified features in the rump region from which we could compute a set of body measurements (i.e., measures based on anatomical references of the rump region) that showed a strong correlation with the assigned BCS. To validate the result, we collected a final data set from 171 goats. To account for variability in animal size and camera position, we mapped a subset of features to a standard template and aligned all the rump images before computing the body measurements. Scientific illustrations were created from the aligned images of animals identified as representative of each category to increase clarity and reproducibility. For training material, we created sketches representing the threshold between consecutive categories. Finally, we conducted 2 field reliability studies. In the first test, no training was given to 4 observers, whereas in the second, training using the threshold images was delivered to the same observers. In the first experiment, interobserver results was substantial, showing that the visual scoring system is clear and unambiguous. Moreover, results improved after training, reaching almost perfect agreement for the very fat category. The visual body condition scoring system is not only a practical tool for BCS in dairy goats but also shows potential to be fully automated, which would enhance its use in welfare assessment schemes and farm management.

The Partial View Heat Kernel descriptor for 3D object representation

We introduce the Partial View Heat Kernel (PVHK) descriptor, for the purpose of 3D object representation and recognition from partial views, assumed to be partial object surfaces under self occlusion. PVHK describes partial views in a geometrically meaningful way, i.e., by establishing a unique relation between the shape of the view and the descriptor. PVHK is also stable with respect to sensor noise and therefore adequate for sensors, such as the current active 3D cameras. Furthermore, PVHK takes full advantage of the dual 3D/RGB nature of current sensors and seamlessly incorporates appearance information onto the 3D information. We formally define the PVHK descriptor, discuss related work, provide evidence of the PVHK properties and validate them in three purposefully diverse datasets, and demonstrate its potential for recognition tasks.

SMARTcycling: Assessing cyclists' driving experience

Due to economic and environmental issues, bicycles have been regaining their significance as a transportation vehicle in urban scenarios. To further drive this desirable trend, policy makers must have the tools to access current bicycle infrastructures and road safety concerns. Fundamental for this assessment is a deeper understanding of how cyclists use current infrastructures, if the cycling experience results in stressful events, and the conditions of the current infrastructure. We here introduce a new platform, SMARTcycling, that, by taking advantage of the mobile power available to a smartphone, captures and stores data from several sensors, namely an action camera, a cardio signal acquisition belt, and smartphones Global Positioning System (GPS) coordinates. The data is further processed and, through visual cues, we access the cyclist driving events and road condition cues. SMARTcycling also detects the cyclist stress using the electrocardiograms (ECG) from the belt. We further contribute by making available a dataset containing the sensors data from 10 paths over two cities in Portugal. On this dataset, we show our initial promising results on event detection, road condition identification and stress assessment.

Global localization by soft object recognition from 3D Partial Views

Global localization is a widely studied problem, and in essence corresponds to the online robot pose estimation based on a given map with landmarks, an odometry model, and real robot sensory observations and motion. In most approaches, the map provides the position of visible objects, which are then recognized to provide the robot pose estimation. Such object recognition with noisy sensory data is challenging. In this paper, we present an effective global localization technique using soft 3D object recognition to estimate the pose with respect to the landmarks in the given map. A depth sensor acquires a partial view for each observed object, from which our algorithm extracts the robot pose relative to the objects, based on a library of 3D Partial View Heat Kernel descriptors. Our approach departs from methods that require classification and registration against complete 3D models, which are prone to errors due to noisy sensory data and object misclassifications in the recognition stage. We experimentally validate our method in different robot paths with different common 3D environment objects. We also show the improvement of our method compared to when the partial view information is not used.

Multiple Hypothesis for Object Class Disambiguation from Multiple Observations

The current paper addresses the problem of object identification from multiple3D partial views, collected from different view angles with the objective of disambiguating between similar objects. We assume a mobile robot equipped with a depth sensor that autonomously collects observations from an object from different positions, with no previous known pattern. The challenge is to efficiently combine the set of observations into a single classification. We approach the problem with a multiple hypothesis filter that allows to combine information from a sequence of observations given the robot movement. We further innovate by off-line learning neighborhoods between possible hypothesis based on the similarity of observations. Such neighborhoods translate directly the ambiguity between objects, and allow to transfer the knowledge of one object to the other. In this paper we introduce our algorithm, Multiple Hypothesis for Object Class Disambiguation from Multiple Observations, and evaluate its accuracy and efficiency.

Combining 3D Shape and Color for 3D Object Recognition

We present new results in object recognition based on color and 3D shape obtained from 3D cameras. Namely, we further exploit diffusion processes to represent shape and the use of color/texture as a perturbation to the diffusion process. Diffusion processes are an effective tool to replace shortest path distances in the characterization of 3D shapes. They also provide effective means for the seamlessly representation of color and shape, mainly because they provide information both the color and on their distribution over surfaces. While there have been different approaches for incorporating color information in the diffusion process, this is the first work that explores different parameterizations of color and their impact on recognition tasks. We present results using very challenging datasets, where we propose to recognize different instances of the same object class assuming a very limited a-priori knowledge on each individual object.

Hot Tiles: A Heat Diffusion Based Descriptor for Automatic Tile Panel Assembly

We revisit the problem of forming a coherent image by assembling independent pieces, also known as the jigsaw puzzle. Namely, we are interested in assembling tile panels, a relevant task for art historians, currently facing many disassembled panels. Existing jigsaw solving algorithms rely strongly on texture alignment to locally decide if two pieces belong together and build the complete jigsaw from local decisions. However, pieces in tile panels are handmade, independently painted, with poorly aligned patterns. In this scenario, existing algorithms suffer from severe degradation. We here introduce a new heat diffusion based affinity measure to mitigate the misalignment between two abutting pieces. We also introduce a global optimization approach to minimize the impact of wrong local decisions. We present experiments on Portuguese tile panels, where our affinity measure performs considerably better that state of the art and we can assemble large parts of a panel.

Fast object detection by regression in robot soccer

Visual object detection in robot soccer is fundamental so the robots can act to accomplish their tasks. Current techniques rely on manually highly polished definitions of object models, that lead to accurate detection, but are quite often computationally inefficient. In this work, we contribute an efficient object detection through regression (ODR) method based on offline training. We build upon the observation that objects in robot soccer are of a well defined color and investigate an offline learning approach to model such objects. ODR consists of two main phases: (i) offline training, where the objects are automatically labeled offline by existing techniques, and (ii) online detection, where a given image is efficiently processed in real-time with the learned models. For each image, ODR determines whether the object is present and provides its position if so. We show comparing results with current techniques for precision and computational load.