Mohamed H. Ali

Microsoft CEP server and online behavioral targeting

In this demo, we present the Microsoft Complex Event Processing (CEP) Server, Microsoft CEP for short. Microsoft CEP is an event stream processing system featured by its declarative query language and its multiple consistency levels of stream query processing. Query composability, query fusing, and operator sharing are key features in the Microsoft CEP query processor. Moreover, the debugging and supportability tools of Microsoft CEP provide visibility of system internals to users. Web click analysis has been crucial to behavior-based online marketing. Streams of web click events provide a typical workload for a CEP server. Meanwhile, a CEP server with its processing capabilities plays a key role in web click analysis. This demo highlights the features of Microsoft CEP under a workload of web click events.

Geospatial stream query processing using Microsoft SQL Server StreamInsight

Microsoft SQL Server spatial libraries contain several components that handle geometrical and geographical data types. With advances in geo-sensing technologies, there has been an increasing demand for geospatial streaming applications. Microsoft SQL Server StreamInsight (StreamInsight, for brevity) is a platform for developing and deploying streaming applications that run continuous queries over high-rate streaming events. With its extensibility infrastructure, StreamInsight enables developers to integrate their domain expertise within the query pipeline in the form of user defined modules. This demo utilizes the extensibility infrastructure in Microsoft StreamInsight to leverage its continuous query processing capabilities in two directions. The first direction integrates SQL spatial libraries into the continuous query pipeline of StreamInsight. StreamInsight provides a well-defined temporal model over incoming events while SQL spatial libraries cover the spatial properties of events to deliver a solution for spatiotemporal stream query processing. The second direction extends the system with an analytical refinement and prediction layer. This layer analyzes historical data that has been accumulated and summarized over the years to refine, smooth and adjust the current query output as well as predict the output in the near future. The demo scenario is based on transportation data in Los Angeles County.

The similarity join database operator

Similarity joins have been studied as key operations in multiple application domains, e.g., record linkage, data cleaning, multimedia and video applications, and phenomena detection on sensor networks. Multiple similarity join algorithms and implementation techniques have been proposed. They range from out-of-database approaches for only in-memory and external memory data to techniques that make use of standard database operators to answer similarity joins. Unfortunately, there has not been much study on the role and implementation of similarity joins as database physical operators. In this paper, we focus on the study of similarity joins as first-class database operators. We present the definition of several similarity join operators and study the way they interact among themselves, with other standard database operators, and with other previously proposed similarity-aware operators. In particular, we present multiple transformation rules that enable similarity query optimization through the generation of equivalent similarity query execution plans. We then describe an efficient implementation of two similarity join operators, ε-Join and Join-Around, as core DBMS operators. The performance evaluation of the implemented operators in PostgreSQL shows that they have good execution time and scalability properties. The execution time of Join-Around is less than 5% of the one of the equivalent query that uses only regular operators while ε-Joins execution time is 20 to 90% of the one of its equivalent regular operators based query for the useful case of small ε (0.01% to 10% of the domain range). We also show experimentally that the proposed transformation rules can generate plans with execution times that are only 10% to 70% of the ones of the initial query plans.

The extensibility framework in Microsoft StreamInsight

Microsoft StreamInsight (StreamInsight, for brevity) is a platform for developing and deploying streaming applications, which need to run continuous queries over high-data-rate streams of input events. StreamInsight leverages a well-defined temporal stream model and operator algebra, as the underlying basis for processing long-running continuous queries over event streams. This allows StreamInsight to handle imperfections in event delivery and to provide correctness guarantees on the generated output. StreamInsight natively supports a diverse range of off-the-shelf streaming operators. In order to cater to a much broader range of customer scenarios and applications, StreamInsight has recently introduced a new extensibility infrastructure. With this infrastructure, StreamInsight enables developers to integrate their domain expertise within the query pipeline in the form of user defined modules (functions, operators, and aggregates). This paper describes the extensibility framework in StreamInsight; an ongoing effort at Microsoft SQL Server to support the integration of user-defined modules in a stream processing system. More specifically, the paper addresses the extensibility problem from three perspectives: the query writers perspective, the user defined module writers perspective, and the systems internal perspective. The paper introduces and addresses a range of new and subtle challenges that arise when we try to add extensibility to a streaming system, in a manner that is easy to use, powerful, and practical. We summarize our experience and provide future directions for supporting stream-oriented workloads in different business domains.

Similarity queries: their conceptual evaluation, transformations, and processing

Many application scenarios can significantly benefit from the identification and processing of similarities in the data. Even though some work has been done to extend the semantics of some operators, for example join and selection, to be aware of data similarities, there has not been much study on the role and implementation of similarity-aware operations as first-class database operators. Furthermore, very little work has addressed the problem of evaluating and optimizing queries that combine several similarity operations. The focus of this paper is the study of similarity queries that contain one or multiple first-class similarity database operators such as Similarity Selection, Similarity Join, and Similarity Group-by. Particularly, we analyze the implementation techniques of several similarity operators, introduce a consistent and comprehensive conceptual evaluation model for similarity queries, and present a rich set of transformation rules to extend cost-based query optimization to the case of similarity queries.

Similarity Group-By

Group-by is a core database operation that is used extensively in OLTP, OLAP, and decision support systems. In many application scenarios, it is required to group similar but not necessarily equal values. In this paper we propose a new SQL construct that supports similarity-based Group-by (SGB). SGB is not a new clustering algorithm, but rather is a practical and fast similarity grouping query operator that is compatible with other SQL operators and can be combined with them to answer similarity-based queries efficiently. In contrast to expensive clustering algorithms, the proposed similarity group-by operator maintains low execution times while still generating meaningful groupings that address many application needs. The paper presents a general definition of the similarity group-by operation and gives three instances of this definition. The paper also discusses how optimization techniques for the regular group-by can be extended to the case of SGB. The proposed operators are implemented inside PostgreSQL. The performance study shows that the proposed similarity-based group-by operators have good scalability properties with at most only 25% increase in execution time over the regular group-by.

Differential Regulation of Peripheral CD4+ T Cell Tolerance Induced by Deletion and TCR Revision

In Vβ5 transgenic mice, mature Vβ5+CD4+ T cells are tolerized upon recognition of a self Ag, encoded by a defective endogenous retrovirus, whose expression is confined to the lymphoid periphery. Cells are driven by the tolerogen to enter one of two tolerance pathways, deletion or TCR revision. CD4+ T cells entering the former pathway are rendered anergic and then eliminated. In contrast, TCR revision drives gene rearrangement at the endogenous TCR β locus and results in the appearance of Vβ5−, endogenous Vβ+, CD4+ T cells that are both self-tolerant and functional. An analysis of the molecules that influence each of these pathways was conducted to understand better the nature of the interactions that control tolerance induction in the lymphoid periphery. These studies reveal that deletion is efficient in reconstituted radiation chimeras and is B cell, CD28, inducible costimulatory molecule, Fas, CD4, and CD8 independent. In contrast, TCR revision is radiosensitive, B cell, CD28, and inducible costimulatory molecule dependent, Fas and CD4 influenced, and CD8 independent. Our data demonstrate the differential regulation of these two divergent tolerance pathways, despite the fact that they are both driven by the same tolerogen and restricted to mature CD4+ T cells.

ACM SIGSPATIAL GIS Cup 2012

The 20th ACM SIGSPATIAL Conference on Advances in Geographic Information Systems (GIS) was held in November of 2012. In conjunction with this conference, we organized the conferences first competition, called the SIGSPATIAL GIS Cup 2012. The subject of the competition was map matching, which is the problem of correctly matching a sequence of noisy GPS points to roads. We describe the details of the contest, the results of the competition, and the lessons we learned in running a contest like this.

An introduction to Microsoft SQL server StreamInsight

Microsoft StreamInsight is a powerful platform that you can use to develop and deploy complex event processing (CEP) applications. Its high-throughput stream processing architecture and the Microsoft .NET Framework-based development platform enable you to quickly implement robust and highly efficient event processing applications. Event stream sources typically include data from manufacturing applications, financial trading applications, Web analytics, and operational analytics. By using StreamInsight, you can develop CEP applications that derive immediate business value from this raw data by reducing the cost of extracting, analyzing, and correlating the data; and by allowing you to monitor, manage, and mine the data for conditions, opportunities, and defects almost instantly. By using StreamInsight to develop CEP applications, you can achieve the following tactical and strategic goals for your business: • Monitor your data from multiple sources for meaningful patterns, trends, exceptions, and opportunities. • Analyze and correlate data incrementally while the data is in-flight -- that is, without first storing it--yielding very low latency. Aggregate seemingly unrelated events from multiple sources and perform highly complex analyses over time. • Manage your business by performing low-latency analytics on the events and triggering response actions that are defined on your business key performance indicators (KPIs). • Respond quickly to areas of opportunity or threat by incorporating your KPI definitions into the logic of the CEP application, thereby improving operational efficiency and your ability to respond quickly to business opportunities. • Mine events for new business KPIs. • Move toward a predictive business model by mining historical data to continuously refine and improve your KPI definitions. This course covers the key concepts in Microsoft StreamInsight and provides developers with a step-by-step guidance to build their first data streaming applications. The course is expected to run for two hours and is expected to cover the following topics: 20 minutes - Welcome and Introductions 20 minutes - StreamInsight use cases, architecture and demo 20 minutes - Getting the data in and out: writing adapters for StreamInsight 20 minutes - StreamInsight queries: writing continuous queries and analytics for StreamInsight in LINQ 20 minutes - Building rich StreamInsight applications 20 minutes - Deploying and managing StreamInsight applications

Data Stream Management Systems for Computational Finance

Because financial applications rely on a continual stream of time-sensitive data, any data management system must be able to process complex queries on the fly. Although many organizations turn to custom solutions, data stream management systems can offer the same low-latency processing with the flexibility to handle a range of applications.