Del T. Scott

Extracting Data behind Web Forms

A significant and ever-increasing amount of data is accessible only by filling out HTML forms to query an underlying Web data source. While this is most welcome from a user perspective (queries are relatively easy and precise) and from a data management perspective (static pages need not be maintained and databases can be accessed directly), automated agents must face the challenge of obtaining the data behind forms. In principle an agent can obtain all the data behind a form by multiple submissions of the form filled out in all possible ways, but efficiency concerns lead us to consider alternatives. We investigate these alternatives and show that we can estimate the amount of remaining data (if any) after a small number of submissions and that we can heuristically select a reasonably minimal number of submissions to maximize the coverage of the data. Experimental results show that these statistical predictions are appropriate and useful.

Covariance analyses with heterogeneity of slopes in fixed models.

Techniques that describe the use of covariance when heterogeneity of slopes exists are severely limited. Although a few procedures for model selection have been recommended, none, except the hierarchical approach, is straightforward and usable with present computer programs. The hierarchical subset selection procedure presented in this paper is based on the proposition that heterogeneity may be present only for certain terms in the model. After hierarchical selection, those terms which do not involve heterogeneity are interpreted as in the usual analysis for covariance. The interpretations of those terms which do involve heterogeneity are modified with respect to significance tests performed at various values of the covariate. The hierarchical subset selection method allows one to investigate heterogeneity of slopes in covariance models as functions of the classification variables present in the design.

Can Programmers Reuse Software

An experiment asked programmers untrained in reuse to evaluate component reusability. They did poorly. Are reusabilitys promises hollow? Or are there some answers?

Estimation and hypothesis in linear models-A reparameterization approach to the cell mlans modll

A general approach to analysis of fixed effects models through a reparameterization of the cell means model is outlined. After applying a Q-op3rator, distribution theory for the model is developed. Methods of estimation and hypothesis testing are given and the exact form of each hypothesis is shown. The implementation of these results in a currently available computer package is discussed.

HyKSS: Hybrid Keyword and Semantic Search

Keyword search suffers from a number of issues: ambiguity, synonymy, and an inability to handle semantic constraints. Semantic search helps resolve these issues but is limited by the quality of annotations which are likely to be incomplete or imprecise. Hybrid search, a search technique that combines the merits of both keyword and semantic search, appears to be a promising solution. In this paper we describe and evaluate HyKSS, a hybrid search system driven by extraction ontologies for both annotation creation and query interpretation. For displaying results, HyKSS uses a dynamic ranking algorithm. We show that over data sets of short topical documents, the HyKSS ranking algorithm outperforms both keyword and semantic search in isolation, as well as a number of other non-HyKSS hybrid approaches to ranking.

Simple Plots Improve Software Reliability Prediction Models

Software reliability prediction is accomplished by fitting a nonhomogeneous Poisson process (NHPP) model to data from software testing. The data consist of the cumulative time and the cumulative number of failures found in software testing. The NHPP model can be used to predict the reliability of the software product at the time of release or to determine how much further testing must be done to reach a specified failure rate. Models are normally fitted to software testing data using Poisson regression by the method of maximum likelihood. We encountered difficulties fitting models when numerical algorithms failed to converge or when we were unable to discriminate among several models with the same number of parameters. These difficulties were the result of having no likelihood ratio test to compare models with the same number of parameters and anomalies in the data that caused numerical algorithms to fail. We found that a simple cumulative plot of the data (cumulative failures on the vertical axis vs. cumulative test time on the horizontal axis) helped in spotting anomalies in the data and selecting an appropriate model to fit. A second plot of running products of ratios of the probability densities for the predictions made from competing models, called the prequential likelihood ratio, helped in discriminating between models. Use of these plots helped resolve the difficulties we experienced in fitting models to the software testing data.

Orthogonalization-Triangularization Methods in Statistical Computations

Abstract Procedures are presented for reducing a data matrix to triangular form by using orthogonal transformations. It is shown how an analysis of variance can be constructed from the triangular reduction of the data matrix. Procedures for calculating sums of squares, degrees of freedom, and expected mean squares are presented. It is demonstrated that all statistics needed for inference on linear combinations of parameters of a linear model may be calculated from the triangular reduction of the data matrix.

Likelihood estimation of missing cell means in the fixed model analysis of variance

This paper examines the formation of maximum likelihood estimates of cell means in analysis of variance problems for cells with missing observations. Methods of estimating the means for missing cells has a long history which includes iterative maximum likelihood techniques, approximation techniques and ad hoc techniques. The use of the EM algorithm to form maximum likelihood estimates has resolved most of the issues associated with this problem. Implementation of the EM algorithm entails specification of a reduced model. As demonstrated in this paper, when there are several missing cells, it is possible to specify a reduced model that results in an unidentifiable likelihood. The EM algorithm in this case does not converge, although the slow divergence may often be mistaken by the unwary as convergence. This paper presents a simple matrix method of determining whether or not the reduced model results in an identifiable likelihood, and consequently in an EM algorithm that converges. We also show the EM algo...