Alexey Radul

Nuggeteer: Automatic Nugget-Based Evaluation using Descriptions and Judgements

The TREC Definition and Relationship questions are evaluated on the basis of information nuggets that may be contained in system responses. Human evaluators provide informal descriptions of each nugget, and judgements (assignments of nuggets to responses) for each response submitted by participants. While human evaluation is the most accurate way to compare systems, approximate automatic evaluation becomes critical during system development.We present Nuggeteer, a new automatic evaluation tool for nugget-based tasks. Like the first such tool, Pourpre, Nuggeteer uses words in common between candidate answer and answer key to approximate human judgements. Unlike Pourpre, but like human assessors, Nuggeteer creates a judgement for each candidate-nugget pair, and can use existing judgements instead of guessing. This creates a more readily interpretable aggregate score, and allows developers to track individual nuggets through the variants of their system. Nuggeteer is quantitatively comparable in performance to Pourpre, and provides qualitatively better feedback to developers.

Report on the probabilistic language scheme

Reasoning with probabilistic models is a widespread and successful technique in areas ranging from computer vision, to natural language processing, to bioinformatics. Currently, these reasoning systems are either coded from scratch in general-purpose languages or use formalisms such as Bayesian networks that have limited expressive power. In both cases, the resulting systems are difficult to modify, maintain, compose, and interoperate with. This work presents Probabilistic Scheme, an embedding of probabilistic computation into Scheme. This gives programmers an expressive language for implementing modular probabilistic models that integrate naturally with the rest of Scheme.

A RESTful messaging system for asynchronous distributed processing

Traditionally, distributed computing problems have been solved by partitioning data into chunks small enough to be handled by commodity hardware. However, such partitioning is not possible in cases where there are a high number of dependencies or high dimensionality, such as in reasoning and expert systems, rendering such problems less tractable for distributed systems. By instead partitioning the problem, rather than the data, we can achieve a more general application of distributed computing. Partitioning the problem rather than the data may require tighter communication between members of the network, even though many networks can only be assumed to be weakly-connected. We believe that a decentralized implementation of propagator networks may resolve the problem. By placing several constraints on the merging of data transmitted over the network, we can easily synchronize information and achieve eventual convergence without implementing mechanisms needed for serialization. To this end, we present the design of a RESTful messaging mechanism, currently in the process of being implemented, that allows distributed propagator networks to be created, using mechanisms that result in eventual convergence of knowledge across a weakly-connected network. By utilizing a RESTful design of the mechanism, we can also achieve a reduction of bandwidth usage during synchronization through the use of caching.

Probabilistic programming with programmable inference

We introduce inference metaprogramming for probabilistic programming languages, including new language constructs, a formalism, and the rst demonstration of e ectiveness in practice. Instead of relying on rigid black-box inference algorithms hard-coded into the language implementation as in previous probabilistic programming languages, infer- ence metaprogramming enables developers to 1) dynamically decompose inference problems into subproblems, 2) apply in- ference tactics to subproblems, 3) alternate between incorpo- rating new data and performing inference over existing data, and 4) explore multiple execution traces of the probabilis- tic program at once. Implemented tactics include gradient- based optimization, Markov chain Monte Carlo, variational inference, and sequental Monte Carlo techniques. Inference metaprogramming enables the concise expression of proba- bilistic models and inference algorithms across diverse elds, such as computer vision, data science, and robotics, within a single probabilistic programming language.