Hugo Liu

Commonsense Reasoning in and Over Natural Language

ConceptNet is a very large semantic network of commonsense knowledge suitable for making various kinds of practical inferences over text. ConceptNet captures a wide range of commonsense concepts and relations like those in Cyc, while its simple semantic network structure lends it an ease-of-use comparable to WordNet. To meet the dual challenge of having to encode complex higher-order concepts, and maintaining ease-of-use, we introduce a novel use of semi-structured natural language fragments as the knowledge representation of commonsense concepts. In this paper, we present a methodology for reasoning flexibly about these semi-structured natural language fragments. We also examine the tradeoffs associated with representing commonsense knowledge in formal logic versus in natural language. We conclude that the flexibility of natural language makes it a highly suitable representation for achieving practical inferences over text, such as context finding, inference chaining, and conceptual analogy.

Beating Common Sense into Interactive Applications

A long-standing dream of artificial intelligence has been to put commonsense knowledge into computers -- enabling machines to reason about everyday life. Some projects, such as Cyc, have begun to amass large collections of such knowledge. However, it is widely assumed that the use of common sense in interactive applications will remain impractical for years, until these collections can be considered sufficiently complete and commonsense reasoning sufficiently robust. Recently, at the Massachusetts Institute of Technologys Media Laboratory, we have had some success in applying commonsense knowledge in a number of intelligent interface agents, despite the admittedly spotty coverage and unreliable inference of todays commonsense knowledge systems. This article surveys several of these applications and reflects on interface design principles that enable successful use of commonsense knowledge.

Unraveling the Taste Fabric of Social Networks

Popular online social networks such as Friendster and MySpace do more than simply reveal the superficial structure of social connectedness — the rich meanings bottled within social network profiles themselves imply deeper patterns of culture and taste. If these latent semantic fabrics of taste could be harvested formally, the resultant resource would afford completely novel ways for representing and reasoning about web users and people in general. This paper narrates the theory and technique of such a feat — the natural language text of 100,000 social network profiles were captured, mapped into a diverse ontology of music, books, films, foods, etc., and machine learning was applied to infer a semantic fabric of taste. Taste fabrics bring us closer to improvisational manipulations of meaning, and afford us at least three semantic functions — the creation of semantically flexible user representations, cross-domain taste-based recommendation, and the computation of taste-similarity between people — whose use cases are demonstrated within the context of three applications — the InterestMap, Ambient Semantics, and IdentityMirror. Finally, we evaluate the quality of the taste fabrics, and distill from this research reusable methodologies and techniques of consequence to the semantic mining and Semantic Web communities.

Metafor: visualizing stories as code

Every program tells a story. Programming, then, is the art of constructing a story about the objects in the program and what they do in various situations. So-called programming languages, while easy for the computer to accurately convert into code, are, unfortunately, difficult for people to write and understand.We explore the idea of using descriptions in a natural language as a representation for programs. While we cannot yet convert arbitrary English to fully specified code, we can use a reasonably expressive subset of English as a visualization tool. Simple descriptions of program objects and their behavior generate scaffolding (underspecified) code fragments, that can be used as feedback for the designer. Roughly speaking, noun phrases can be interpreted as program objects; verbs can be functions, adjectives can be properties. A surprising amount of what we call programmatic semantics can be inferred from linguistic structure. We present a program editor, Metafor, that dynamically converts a users stories into program code, and in a user study, participants found it useful as a brainstorming tool.

Visualizing the affective structure of a text document

This paper introduces an approach for graphically visualizing the affective structure of a text document. A document is first affectively analyzed using a unique textual affect sensing engine, which leverages commonsense knowledge to classify text more reliably and comprehensively than can be achieved with keyword spotting methods alone. Using this engine, sentences are annotated using six basic Ekman emotions. Colors used to represent each of these emotions are sequenced into a color bar, which represents the progression of affect through a text document. Smoothing techniques allow the user to vary the granularity of the affective structure being displayed on the color bar. The bar is hyperlinked in a way such that it can be used to easily navigate the document. A user evaluation demonstrates that the proposed method for visualizing and navigating a documents affective structure facilitates a users within-document information foraging activity.

Teaching Machines about Everyday Life

In order to build software that can deeply understand people and our problems, we require computational tools that give machines the capacity to learn and reason about everyday life. We describe three commonsense knowledge bases that take unconventional approaches to representing, acquiring, and reasoning with large quantities of commonsense knowledge. Each adopts a different approach — ConceptNet is a large-scale semantic network, LifeNet is a probabilistic graphical model, and StoryNet is a database of story-scripts. We describe the evolution, architecture and operation of these three systems, and conclude with a discussion of how we might combine them into an integrated commonsense reasoning system.

NLP (natural language processing) for NLP (natural language programming)

Natural Language Processing holds great promise for making computer interfaces that are easier to use for people, since people will (hopefully) be able to talk to the computer in their own language, rather than learn a specialized language of computer commands. For programming, however, the necessity of a formal programming language for communicating with a computer has always been taken for granted. We would like to challenge this assumption. We believe that modem Natural Language Processing techniques can make possible the use of natural language to (at least partially) express programming ideas, thus drastically increasing the accessibility of programming to non-expert users. To demonstrate the feasibility of Natural Language Programming, this paper tackles what are perceived to be some of the hardest cases: steps and loops. We look at a corpus of English descriptions used as programming assignments, and develop some techniques for mapping linguistic constructs onto program structures, which we refer to as programmatic semantics.

Toward a Programmatic Semantics of Natural Language

Natural language is imbued with a rich semantics but unfortunately its complex elegance is often mistaken for mere imprecision. Because complete parsers of English are not yet achievable, people assume that it is not feasible to use English directly as a means of instructing computers. However, in this paper, we show that English descriptions of procedures often contain programmatic semantics-linguistic features that can be easily mapped into programming language constructs. Some linguistic features can even inspire new ways of thinking about specifying programs. Far from being hopelessly ambiguous, natural languages exhibit important principles of communication that could be used to make human-computer communication more natural

FEASIBILITY STUDIES FOR PROGRAMMING IN NATURAL LANGUAGE

We think it is time to take another look at an old dream -- that one could program a computer by speaking to it in natural language. Programming in natural language might seem impossible, because it would appear to require complete natural language understanding and dealing with the vagueness of human descriptions of programs. But we think that several developments might now make programming in natural language feasible. First, improved broad coverage natural language parsers and semantic extraction techniques permit partial understanding. Second, mixed-initiative dialogues can be used for meaning disambiguation. And finally, where direct understanding techniques fail, we hope to fall back on Programming by Example, and other techniques for specifying the program in a more fail-soft manner. To assess the feasibility of this project, as a first step, we are studying how non-programming users describe programs in unconstrained natural language. We are exploring how to design dialogs that help the user make precise their intentions for the program, while constraining them as little as possible.

Unpacking meaning from words: a context-centered approach to computational lexicon design

The knowledge representation tradition in computational lexicon design represents words as static encapsulations of purely lexical knowledge. We suggest that this view poses certain limitations on the ability of the lexicon to generate nuance-laden and context-sensitive meanings, because word boundaries are obstructive, and the impact of non-lexical knowledge on meaning is unaccounted for. Hoping to address these problematics, we explore a contextcentered approach to lexicon design called a Bubble Lexicon. Inspired by Ross Quillians Semantic Memory System, we represent word-concepts as nodes on a symbolic-connectionist network. In a Bubble Lexicon, a words meaning is defined by a dynamically grown context-sensitive bubble; thus giving a more natural account of systematic polysemy. Linguistic assembly tasks such as attribute attachment are made context-sensitive, and the incorporation of general world knowledge improves generative capability. Indicative trials over an implementation of the Bubble Lexicon lends support to our hypothesis that unpacking meaning from predefined word structures is a step toward a more natural handling of context in language.