Filipe de O. Costa

Open set source camera attribution and device linking

Camera attribution approaches in digital image forensics have most often been evaluated in a closed set context, whereby all devices are known during training and testing time. However, in a real investigation, we must assume that innocuous images from unknown devices will be recovered, which we would like to remove from the pool of evidence. In pattern recognition, this corresponds to what is known as the open set recognition problem. This article introduces new algorithms for open set modes of image source attribution (identifying whether or not an image was captured by a specific digital camera) and device linking (identifying whether or not a pair of images was acquired from the same digital camera without the need for physical access to the device). Both algorithms rely on a new multi-region feature generation strategy, which serves as a projection space for the class of interest and emphasizes its properties, and on decision boundary carving, a novel method that models the decision space of a trained SVM classifier by taking advantage of a few known cameras to adjust the decision boundaries to decrease false matches from unknown classes. Experiments including thousands of unconstrained images collected from the web show a significant advantage for our approaches over the most competitive prior work.

Image Phylogeny Forests Reconstruction

Today, a simple search for an image on the Web can return thousands of related images. Some results are exact copies, some are variants (or near-duplicates) of the same digital image, and others are unrelated. Although we can recognize some of these images as being semantically similar, it is not as straightforward to find which image is the original. It is not easy either to find the chain of transformations used to create each modified version. There are several approaches in the literature to identify near-duplicate images, as well as to reconstruct their relational structure. For the latter, a common representation uses the parent-child relationship, allowing us to visualize the evolution of modifications as a phylogeny tree. However, most of the approaches are restricted to the case of finding the tree of evolution of the near-duplicates, with few works dealing with sets of trees. Since one set of near-duplicates can contain n independent subsets, it is necessary to reconstruct not only one phylogeny tree, but several trees that will compose a phylogeny forest. In this paper, through the analysis of the state-of-the-art image phylogeny algorithms, we introduce a novel approach to deal with phylogeny forests, based on different combinations of these algorithms, aiming at improving their reconstruction accuracy. We analyze the effectiveness of each combination and evaluate our method with more than 40 000 testing cases, using quantitative metrics.

Phylogeny reconstruction for misaligned and compressed video sequences

In the last few years, the amount of videos distributed online has dramatically increased due to the popularity of media sharing platforms (e.g., YouTube, Vimeo, etc.). However, distributed videos are often edited copies of original content, typically referred to as near duplicates. In this paper, we face the problem of reconstructing a video phylogeny tree, i.e., given a set of near-duplicate videos, we want to reconstruct the relationships between every pair of videos to detect which one generated the others and trace back their evolution history. Solving this problem is of paramount importance when the first published video within a set is sought, e.g., to solve copyright infringement cases or to pinpoint criminal impersonation online. The technique we propose exploits the same rationale of previous works in the field of image and video phylogeny. However, we embed in the commonly used pipeline of operations the possibility of dealing with temporally misaligned and encoded video sequences, thus making the method applicable to user-generated videos shared on online platforms. Results computed on a wide dataset of video sequences highlight the importance of taking care of both coding and misalignment in the reconstruction pipeline.

New dissimilarity measures for image phylogeny reconstruction

Image phylogeny is the problem of reconstructing the structure that represents the history of generation of semantically similar images (e.g., near-duplicate images). Typical image phylogeny approaches break the problem into two steps: (1) estimating the dissimilarity between each pair of images and (2) reconstructing the phylogeny structure. Given that the dissimilarity calculation directly impacts the phylogeny reconstruction, in this paper, we propose new approaches to the standard formulation of the dissimilarity measure employed in image phylogeny, aiming at improving the reconstruction of the tree structure that represents the generational relationships between semantically similar images. These new formulations exploit a different method of color adjustment, local gradients to estimate pixel differences and mutual information as a similarity measure. The results obtained with the proposed formulation remarkably outperform the existing counterparts in the literature, allowing a much better analysis of the kinship relationships in a set of images, allowing for more accurate deployment of phylogeny solutions to tackle traitor tracing, copyright enforcement and digital forensics problems.

Hash-based frame selection for video phylogeny

Multimedia phylogeny is a research field that aims at tracing back past history of multimedia documents to discover their ancestral relationships. As an example, it might leverage, with the aid of other side information, forensic analysts to detect who was the first user that published online an illegal content (e.g., child pornography). Although relatively well developed for images, this field is still not fully fledged when it comes to analyzing ancestral and evolutionary relationships among digital videos. Dealing with videos is much more challenging, especially as temporal dimension comes into play. In this vein, one of the pivotal tasks for video phylogeny reconstruction is video synchronization in order to compare temporally coherent near-duplicate frames among pairs of sequences. In this work, we propose an algorithm to efficiently select synchronized frame pairs among videos before calculating their phylogenetic relationships. This approach underpins the video phylogeny reconstruction and leverages the analysis on a reduced subset of frames rather than on the full set, thus decreasing the overall computational time. Experimental results show the effectiveness of the proposed method when temporal transformations are considered (i.e., change of frame rate and temporal clipping at any point in the stream).

Face Verification Based on Relational Disparity Features and Partial Least Squares Models

Face verification approaches aim at determining whether two given faces are from the same person. This scenario has several applications, such as information security, forensics, surveillance and smart cards. Several works extract features independently from each face image, i.e., any sort of relation between the two faces is not modeled a priori to either training or classification stages. In this work, we propose an approach that compares a pair of faces by extracting relational features, assuming the hypothesis that modeling the relation between two faces can be useful for increasing the robustness and performance of the face verification task. Then, we employ multiple classification models based on Partial Least Squares to verify whether a given pair of images belongs the same subject (genuine) or belongs to different subjects (impostor). We validate our approach on the Labeled Faces in the Wild (LFW) and on the Public Figures (Pubfig) datasets, using only few images for training. According to the experiments, our approach achieves results up to 0.966 of area under the curve (AUC) for the LFW dataset using its unrestricted, labeled outside data protocol and an average equal error (EER) of 13.65% on PubFig dataset.