Fernando Bellas

Extracting lists of data records from semi-structured web pages

Many web sources provide access to an underlying database containing structured data. These data can be usually accessed in HTML form only, which makes it difficult for software programs to obtain them in structured form. Nevertheless, web sources usually encode data records using a consistent template or layout, and the implicit regularities in the template can be used to automatically infer the structure and extract the data. In this paper, we propose a set of novel techniques to address this problem. While several previous works have addressed the same problem, most of them require multiple input pages while our method requires only one. In addition, previous methods make some assumptions about how data records are encoded into web pages, which do not always hold in real websites. Finally, we have also tested our techniques with a high number of real web sources and we have found them to be very effective.

Standards for second-generation portals

By integrating applications and resources, portals let users access information in a simple, straightforward manner. Currently, most portals let users create one or more personal pages composed of portlets - interactive Web mini-applications. Until recently, no standards for portlets existed, and thus consuming remote portlets in a generic way or deploying portlets in one portal server that were developed in a different one has been impossible. Two standards released in Fall 2003 - the Web Services for Remote Portlets (WSRP) and the Java portlet specification - address these problems. This articles outlines these standards and presents a typical architecture for a standards-based portal.

A Component-Based Approach for Engineering Enterprise Mashups

Mashup applications combine pieces of functionality from several existing, heterogeneous sources to provide new integrated functionality. This paper presents the design of an enterprise-oriented mashup tool that fully supports the construction of mashup applications. Our tool provides generic, reusable components to engineer mashup applications. It includes components for accessing heterogeneous sources, a component to combine data from different sources and components for building the graphical interface. The user builds graphically the mashup application by selecting, customizing, and interconnecting components. Unlike other proposals we: (1) use the Data Federation/Mediation pattern (instead of Pipes and Filters pattern) to express the data combination logic, (2) follow the RESTful architectural style to improve component reusability, and (3) reuse Java standard portal technology to implement the graphical interface of the mashup application.

Automating Navigation Sequences in AJAX Websites

Web automation applications are widely used for different purposes such as B2B integration, automated testing of web applications or technology and business watch. One crucial part in web automation applications is to allow easily generating and reproducing navigation sequences. Previous proposals in the literature assumed a navigation model today turned obsolete by the new breed of AJAX-based websites. Although some open-source and commercial tools have also addressed the problem, they show significant limitations either in usability or their ability to deal with complex websites. In this paper, we propose a set of new techniques to deal with this problem. Our main contributions are a new method for recording navigation sequences supporting a wider range of events, and a novel method to detect when the effects caused by a user action have finished. We have evaluated our approach with more than 100 web applications, obtaining very good results.

Automated browsing in AJAX websites

Web automation applications are widely used for different purposes such as B2B integration, automated testing of web applications or technology and business watch. One crucial part in web automation applications is for them to easily generate and reproduce navigation sequences. This problem is specially complicated in the case of the new breed of AJAX-based websites. Although recently some tools have also addressed the problem, they show some limitations either in usability or their ability to deal with complex websites. In this paper, we propose a set of new techniques to build an automatic web navigation system able to deal with these complexities. Our main contributions are: a new method for recording navigation sequences able to scale to a wider range of events, an algorithm to identify in a change-resilient manner the target element of a user action, and a novel method to detect when the effects caused by a user action (including the effects of scripting code and AJAX requests) have finished. In addition, we have also tested our approach with a high number of real web sources and have compared it with other relevant web automation tools obtaining very good results.

Finding and Extracting Data Records from Web Pages

Many HTML pages are generated by software programs by querying some underlying databases and then filling in a template with the data. In these situations the metainformation about the data structure is lost, so automated software programs cannot process these data in such powerful manners as information from databases. We propose a set of novel techniques for detecting structured records in a web page and extracting the data values that constitute them. Our method needs only an input page. It starts by identifying the data region of interest in the page. Then it is partitioned into records by using a clustering method that groups similar subtrees in the DOM tree of the page. Finally, the attributes of the data records are extracted by using a method based on multiple string alignment. We have tested our techniques with a high number of real web sources, obtaining high precision and recall values.

Using clustering and edit distance techniques for automatic web data extraction

Many web sources provide access to an underlying database containing structured data. These data can be usually accessed in HTML form only, which makes it difficult for software programs to obtain them in structured form. Nevertheless, web sources usually encode data records using a consistent template or layout, and the implicit regularities in the template can be used to automatically infer the structure and extract the data. In this paper, we propose a set of novel techniques to address this problem. While several previous works have addressed the same problem, most of them require multiple input pages while our method requires only one. In addition, previous methods make some assumptions about how data records are encoded into web pages, which do not always hold in real websites. Finally, we have tested our techniques with a high number of real web sources and we have found them to be very effective.

Web Navigation Sequences Automation in Modern Websites

Most todays web sources are designed to be used by humans, but they do not provide suitable interfaces for software programs. That is why a growing interest has arisen in so-called web automation applications that are widely used for different purposes such as B2B integration, automated testing of web applications or technology and business watch. Previous proposals assume models for generating and reproducing navigation sequences that are not able to correctly deal with new websites using technologies such as AJAX: on one hand existing systems only allow recording simple navigation actions and, on the other hand, they are unable to detect the end of the effects caused by an user action. In this paper, we propose a set of new techniques to record and execute web navigation sequences able to deal with all the complexity existing in AJAX-based web sites. We also present an exhaustive evaluation of the proposed techniques that shows very promising results.

Finding and extracting data records from web pages

Many HTML pages are generated by software programs by querying some underlying databases and then filling in a template with the data. In these situations the metainformation about the data structure is lost, so automated software programs cannot process these data in such powerful manners as information from databases. We propose a set of novel techniques for detecting structured records in a web page and extracting the data values that constitute them. Our method needs only an input page. It starts by identifying the data region of interest in the page. Then it is partitioned into records by using a clustering method that groups similar subtrees in the DOM tree of the page. Finally, the attributes of the data records are extracted by using a method based on multiple string alignment. We have tested our techniques with a high number of real web sources, obtaining high precision and recall values.

Towards Semi-automatic Construction of Data Mashups

Mashup applications combine pieces of functionality from several existing web data sources to provide new integrated functionality. In this work-in-progress paper, we outline a new method for creating data mashups that suggests to users new relevant data sources and the most suitable ways of combining them. This makes the data mashup process much easier for non-technical users.