Nevenka Dimitrova

Applications of video-content analysis and retrieval

Managing multimedia data requires more than collecting the data into storage archives and delivering it via networks to homes or offices. We survey technologies and applications for video-content analysis and retrieval. We also give specific examples.

Multimedia content processing through cross-modal association

Multimodal information processing has received considerable attention in recent years. The focus of existing research in this area has been predominantly on the use of fusion technology. In this paper, we suggest that cross-modal association can provide a new set of powerful solutions in this area. We investigate different cross-modal association methods using the linear correlation model. We also introduce a novel method for cross-modal association called Cross-modal Factor Analysis (CFA). Our earlier work on Latent Semantic Indexing (LSI) is extended for applications that use off-line supervised training. As a promising research direction and practical application of cross-modal association, cross-modal information retrieval where queries from one modality are used to search for content in another modality using low-level features is then discussed in detail. Different association methods are tested and compared using the proposed cross-modal retrieval system. All these methods achieve significant dimensionality reduction. Among them CFA gives the best retrieval performance. Finally, this paper addresses the use of cross-modal association to detect talking heads. The CFA method achieves 91.1% detection accuracy, while LSI and Canonical Correlation Analysis (CCA) achieve 66.1% and 73.9% accuracy, respectively. As shown by experiments, cross-modal association provides many useful benefits, such as robust noise resistance and effective feature selection. Compared to CCA and LSI, the proposed CFA shows several advantages in analysis performance and feature usage. Its capability in feature selection and noise resistance also makes CFA a promising tool for many multimedia analysis applications.

DNA methylation patterns in luminal breast cancers differ from non-luminal subtypes and can identify relapse risk independent of other clinical variables

The diversity of breast cancers reflects variations in underlying biology and affects the clinical implications for patients. Gene expression studies have identified five major subtypes– Luminal A, Luminal B, basal‐like, ErbB2+ and Normal‐Like. We set out to determine the role of DNA methylation in subtypes by performing genome‐wide scans of CpG methylation in breast cancer samples with known expression‐based subtypes. Unsupervised hierarchical clustering using a set of most varying loci clustered the tumors into a Luminal A majority (82%) cluster, Basal‐like/ErbB2+ majority (86%) cluster and a non‐specific cluster with samples that were also inconclusive in their expression‐based subtype correlations. Contributing methylation loci were both gene associated loci (30%) and non‐gene associated (70%), suggesting subtype dependant genome‐wide alterations in the methylation landscape. The methylation patterns of significant differentially methylated genes in luminal A tumors are similar to those identified in CD24 + luminal epithelial cells and the patterns in basal‐like tumors similar to CD44 + breast progenitor cells. CpG islands in the HOXA cluster and other homeobox (IRX2, DLX2, NKX2‐2) genes were significantly more methylated in Luminal A tumors. A significant number of genes (2853, p < 0.05) exhibited expression–methylation correlation, implying possible functional effects of methylation on gene expression. Furthermore, analysis of these tumors by using follow‐up survival data identified differential methylation of islands proximal to genes involved in Cell Cycle and Proliferation (Ki‐67, UBE2C, KIF2C, HDAC4), angiogenesis (VEGF, BTG1, KLF5), cell fate commitment (SPRY1, OLIG2, LHX2 and LHX5) as having prognostic value independent of subtypes and other clinical factors.

Video keyframe extraction and filtering: a keyframe is not a keyframe to everyone

In this paper, we describe the keyframe extraction and filtering process within the video content indexing system called Vitamin. The video content filtering system analyzes the source video and presents to the user a visual table of contents using thumbnail images. The tiltering process eliminates keyframes which do not contribute to the overall comprehension of the video contents. The user should he able to access particular points on a VHS tape, or MPEG file using this visual table of contents. We have analyzed over ten hours of video content from dierent movies, home videos, serials. and sitcoms. Our experhnents show that the number of keyframes is reduced to a manageable size, thus enabling only important visual information to be presented to the user.

Expression-based network biology identifies alteration in key regulatory pathways of type 2 diabetes and associated risk/complications.

Type 2 diabetes mellitus (T2D) is a multifactorial and genetically heterogeneous disease which leads to impaired glucose homeostasis and insulin resistance. The advanced form of disease causes acute cardiovascular, renal, neurological and microvascular complications. Thus there is a constant need to discover new and efficient treatment against the disease by seeking to uncover various novel alternate signalling mechanisms that can lead to diabetes and its associated complications. The present study allows detection of molecular targets by unravelling their role in altered biological pathways during diabetes and its associated risk factors and complications. We have used an integrated functional networks concept by merging co-expression network and interaction network to detect the transcriptionally altered pathways and regulations involved in the disease. Our analysis reports four novel significant networks which could lead to the development of diabetes and other associated dysfunctions. (a) The first network illustrates the up regulation of TGFBRII facilitating oxidative stress and causing the expression of early transcription genes via MAPK pathway leading to cardiovascular and kidney related complications. (b) The second network demonstrates novel interactions between GAPDH and inflammatory and proliferation candidate genes i.e., SUMO4 and EGFR indicating a new link between obesity and diabetes. (c) The third network portrays unique interactions PTPN1 with EGFR and CAV1 which could lead to an impaired vascular function in diabetic nephropathy condition. (d) Lastly, from our fourth network we have inferred that the interaction of β-catenin with CDH5 and TGFBR1 through Smad molecules could contribute to endothelial dysfunction. A probability of emergence of kidney complication might be suggested in T2D condition. An experimental investigation on this aspect may further provide more decisive observation in drug target identification and better understanding of the pathophysiology of T2D and its complications.

Identification of Tumor Suppressors and Oncogenes from Genomic and Epigenetic Features in Ovarian Cancer

The identification of genetic and epigenetic alterations from primary tumor cells has become a common method to identify genes critical to the development and progression of cancer. We seek to identify those genetic and epigenetic aberrations that have the most impact on gene function within the tumor. First, we perform a bioinformatic analysis of copy number variation (CNV) and DNA methylation covering the genetic landscape of ovarian cancer tumor cells. We separately examined CNV and DNA methylation for 42 primary serous ovarian cancer samples using MOMA-ROMA assays and 379 tumor samples analyzed by The Cancer Genome Atlas. We have identified 346 genes with significant deletions or amplifications among the tumor samples. Utilizing associated gene expression data we predict 156 genes with altered copy number and correlated changes in expression. Among these genes CCNE1, POP4, UQCRB, PHF20L1 and C19orf2 were identified within both data sets. We were specifically interested in copy number variation as our base genomic property in the prediction of tumor suppressors and oncogenes in the altered ovarian tumor. We therefore identify changes in DNA methylation and expression for all amplified and deleted genes. We statistically define tumor suppressor and oncogenic features for these modalities and perform a correlation analysis with expression. We predicted 611 potential oncogenes and tumor suppressors candidates by integrating these data types. Genes with a strong correlation for methylation dependent expression changes exhibited at varying copy number aberrations include CDCA8, ATAD2, CDKN2A, RAB25, AURKA, BOP1 and EIF2C3. We provide copy number variation and DNA methylation analysis for over 11,500 individual genes covering the genetic landscape of ovarian cancer tumors. We show the extent of genomic and epigenetic alterations for known tumor suppressors and oncogenes and also use these defined features to identify potential ovarian cancer gene candidates.

Optimizing sparse sequencing of single cells for highly multiplex copy number profiling

Genome-wide analysis at the level of single cells has recently emerged as a powerful tool to dissect genome heterogeneity in cancer, neurobiology, and development. To be truly transformative, single-cell approaches must affordably accommodate large numbers of single cells. This is feasible in the case of copy number variation (CNV), because CNV determination requires only sparse sequence coverage. We have used a combination of bioinformatic and molecular approaches to optimize single-cell DNA amplification and library preparation for highly multiplexed sequencing, yielding a method that can produce genome-wide CNV profiles of up to a hundred individual cells on a single lane of an Illumina HiSeq instrument. We apply the method to human cancer cell lines and biopsied cancer tissue, thereby illustrating its efficiency, reproducibility, and power to reveal underlying genetic heterogeneity and clonal phylogeny. The capacity of the method to facilitate the rapid profiling of hundreds to thousands of single-cell genomes represents a key step in making single-cell profiling an easily accessible tool for studying cell lineage.

CONIVAS: content-based image and video access system

We have developed a CO Ntent-based Image and Video Access System (CO NIVAS). The system consists of a collection of tools to help users retrieve still images and videos from databases by content. This system includes new algorithms for image and video retrieval in addition to algorithms that are adopted from the literature. The system can be used in many applications that require searching large collections of images and video clips such as digital video library, professional video editing, TV news retrieval and copyright protection applications,

Integrated multimedia processing for topic segmentation and classification

We describe integrated multimedia processing for Video Scout, a system that segments and indexes TV programs according to their audio, visual, and transcript information. Video Scout represents a future direction for personal video recorders. In addition to using electronic program guide metadata and a user profile, Scout allows the users to request specific topics within a program. For example, users can request the video clip of the USA president speaking from a half-hour news program. Video Scout has three modules: (i) video pre-processing, (ii) segmentation and indexing, and (iii) storage and user interface. Segmentation and indexing, the core of the system, incorporates a Bayesian framework that integrates information from the audio, visual, and transcript (closed captions) domains. This framework uses three layers to process low, mid, and high-level multimedia information. The high-level layer generates semantic information about TV program topics. This paper describes the elements of the system and presents results from running Video Scout on real TV programs.

TV program classification based on face and text processing

In this paper we describe a system to classify TV programs into predefined categories based on the analysis of their video contents. This is very useful in intelligent display and storage systems that can select channels and record or skip contents according to the consumers preference. Distinguishable patterns exist in different categories of TV programs in terms of human faces and superimposed text. By applying face and text tracking to a number of training video segments, including commercials, news, sitcoms, and soaps, we have identified patterns within each category of TV programs in a predefined feature space that reflects the face and text characteristics of the video. A given video segment is projected to the feature space and compared against the distribution of known categories of TV programs. Domain-knowledge is used to help the classification. Encouraging results have been achieved so far in our initial experiments.