Orlando Belo

Catching web crawlers in the act

This paper recommends a new approach to the detection and containment of Web crawler traverses based on clickstream data mining. Timely detection prevents crawler abusive consumption of Web server resources and eventual site contents privacy or copyrights violation. Clickstream data differentiation ensures focused usage analysis, valuable both for regular users and crawler profiling. Our platform, named ClickTips, sustains a site-specific, updatable detection model that tags Web crawler traverses based on incremental Web session inspection and a decision model that assesses eventual containment. The goal is to deliver a model flexible enough to keep up with crawling continuous evolving and that is capable of detecting crawler presence as soon as possible. We use a real-world Web site case study as a support for process description, as well as, to evaluate the accuracy of the obtained classification models and their ability for discovering previously unknown Web crawlers.

BPMN patterns for ETL conceptual modelling and validation

ETL systems continue to suffer from a lack of a simple and rigorous approach for modelling and validation of populating processes for data warehouses. In spite of the efforts that researchers have been done, there is not yet a convinced and simply approach for modelling (conceptual and logical views), validating and testing ETL processes before conduct them to implementation and roll out. In this paper we explored the use of BPMN for ETL conceptual modelling and validation. Basically, we intended to provide a set of BPMN meta-models (patterns) especially designed to map standard data warehousing ETL processes and testing them before constructing the final system. We think this is a practical approach to reduce significantly the inadequacy of an ETL system in its real world scenario. A surrogate key pipelining process was selected as a case study to demonstrate the use and utility of the ETL modelling approach presented here.

Path planning for complex 3D multilevel environments

The continuous development of graphics hardware is contributing to the creation of 3D virtual worlds with high level of detail, from models of large urban areas, to complete infrastructures, such as residential buildings, stadiums, industrial settings or archaeological sites, to name just a few. Adding virtual humans or avatars adds an extra touch to the visualization providing an enhanced perception of the spaces, namely adding a sense of scale, and enabling simulations of crowds. Path planning for crowds in a meaningful way is still an open research field, particularly when it involves an unknown polygonal 3D world. Extracting the potential paths for navigation in a non automated fashion is no longer a feasible option due to the dimension and complexity of the virtual environments available nowadays. This implies that we must be able to automatically extract information from the geometry of the unknown virtual world to define potential paths, determine accessibilities, and prepare a navigation structure for real time path planning and path finding. A new image based method is proposed that deals with arbitrarily a priori unknown complex virtual worlds, namely those consisting of multilevel passages (e.g. over and below a bridge). The algorithm is capable of extracting all the information required for the actual navigation of avatars, creating a hierarchical data structure to help both high level path planning and low level path finding decisions. The algorithm is image based, hence it is tessellation independent, i.e. the algorithm does not rely on the underlying polygonal structure of the 3D world. Therefore, the number of polygons does not have a significant impact on the performance, and the topology has no weight on the results.

An Evolutionary Approach to the Selection and Allocation of Distributed Cubes

The materialization of multidimensional structures is a common way to speed up OLAP queries. Since there might be a huge number of those structures, a variety of proposals tried to select the most beneficial set, based on the profile of the queries and observing some constraints as materializing space and maintenance time, addressing a centralized storage facility. Only recently, the distributed scenario came to stage on this area, introducing the space dimension (and corresponding communication costs) into the equation to minimize costs. This paper focuses on the selection and allocation of distributed OLAP cubes, using evolutionary algorithms, having an extended aggregation lattice as framework to capture the distributed semantics. Moreover, the evaluation of the fitness of evolutionary solutions is based on cost estimation algorithms that simulate the execution of parallel tasks, using time units as cost metric

Multi-Agent Forex Trading System

Automated trading is a novel field of study in which computer programs are put in charge of deciding when and how to trade financial instruments. Intelligent agents, with their ability to act autonomously and to adapt and interact with the environment, seem like an obvious choice for the development of automated trading systems. The aim of this article is to analyze how well intelligent agents suit this task. We implemented a set of autonomous currency trading agents, using an architecture that consists of an ensemble of classification and regression models, a case-based reasoning system and an expert system. A total of six trading agents were implemented, each being responsible for trading one of the following currency pair in the Forex market: EUR/USD, EUR/JPY, EUR/CHF, USD/JPY, USD/CHF and CHF/JPY. The agents simulated trades over a period of 23 months, having all achieved a reasonable profit trading independently. However, their strategies resulted in relatively high drawdows. In order to decrease the risk inherent to these high drawdowns, the same simulation was performed while making the agents share the monetary resources. As expected, this strategy of investment diversification originated better results. Still, when the trading costs were taken into consideration, the overall trading performance was less than impressive. That was due to the fact that each agent performed too many trades, and the cost associated with the trading commissions became prohibitively high. We were able to lessen the impact of the trading costs in the total profit by integrating the agents in a multi-agent system, in which the agents communicated with each other before opening new trades. This allowed them to calculate the intended exposure to the market, which in turn enabled them to avoid redundant trades. Under simulation and using low leverage, this multi-agent system obtained a 55.7% profit in 23 months of trading, with a 9.0% maximum drawdown.

QuerySheet: A bidirectional query environment for model-driven spreadsheets

This paper presents a tool, named QuerySheet, to query spreadsheets. We defined a language to write the queries, which resembles SQL, the language to query databases. This allows to write queries which are more related to the spreadsheet content than with current approaches.

Using inheritance in a metadata based approach to data quality assessment

Currently available data quality tools provide development environments that significantly decrease the effort in dealing with common data problems, such as those related with attribute domain validation, syntax checking, or value matching against a reference master data repository. On the contrary, more complex and specific data quality functionalities, whose requirements usually derive from application domain business rules, have to be developed from scratch, usually leading to high costs of development and maintenance. This paper introduces the concept of inheritance in a metadata-driven approach to simplified data quality rule management. The approach is based on the belief that even complex data quality rules very often adhere to recurring patterns that can be encoded and encapsulated as reusable, abstract templates. The approach is supported by a metamodel developed on top of OMGs Common Warehouse Metamodel, herein extended with the ability to derive new rule patterns from existing ones, through inheritance. The inheritance metamodel is presented in UML and its application is illustrated with a running example.

ETL Standard Processes Modelling - A Novel BPMN Approach

ETL systems modelling have been a topic quite explored by researchers in Data Warehousing. However, we believe that there isn’t yet a convinced and simply approach that provides the necessary bridges to validate conceptual and logical models and testing them before its real implementation. In this work we explore the use of BPMN for ETL conceptual modelling, presenting an extension to the BPMN 2.0 meta-model and notation to support modelling and visualization of ETL activities. We intend to provide a set of BPMN meta-models especially designed to map standard ETL processes, providing the necessary bridges to translate conceptual models into their correspondent implementation testing correctness and effectiveness of its execution. For this particular work, we specially designed a standard ETL process – Change Data Capture based on log files – to demonstrate the viability and effectiveness of the approach presented.

Applying user signatures on fraud detection in telecommunications networks

Fraud in telecommunications is increasing dramatically with the expansion of modern technology, resulting in the loss of billions of dollars worldwide each year. Although prevention technologies are the best way to reduce fraud,. Fraudsters are adaptive, searching systematically for new ways to commit fraud and, in most of the cases, will usually find some way to circumvent companies prevention measures. In this paper we expose some of the ways in which fraud is being used against organizations, evaluating the limitations of existing strategies and methods to detect and prevent it in todays telecommunications companies. Additionally, we expose a data mining profiling technique based on signatures that was developed for a real mobile telecommunications network operator and integrated into one of its Fraud Management Systems (FMS), currently under operation.

Predicting the Performance of a GRID Environment: An Initial Effort to Increase Scheduling Efficiency

GRID environments are privileged targets for computation-intensive problem solving in areas from weather forecasting to seismic analysis. Mainly composed by commodity hardware, these environments can deliver vast computational capacity, at relatively low cost. In order to take full advantage of their power we need to have efficient task schedulers with the ability to maximize resource effectiveness, shortening execution times. GRID schedulers must not only decide taking a snapshot of the GRIDs status into account, but should also consider the output involved in past decisions. In this work, we intend to show how resource usage can be analyzed, through the use of data mining techniques, to predict performance availability of a GRID environment, as a preliminary work to increase scheduling efficiency as well as adequate resource provisioning.