Igor Labutov

Evaluation methods for unsupervised word embeddings

We present a comprehensive study of evaluation methods for unsupervised embedding techniques that obtain meaningful representations of words from text. Different evaluations result in different orderings of embedding methods, calling into question the common assumption that there is one single optimal vector representation. We present new evaluation techniques that directly compare embeddings with respect to specific queries. These methods reduce bias, provide greater insight, and allow us to solicit data-driven relevance judgments rapidly and accurately through crowdsourcing.

Deep Questions without Deep Understanding

We develop an approach for generating deep (i.e, high-level) comprehension questions from novel text that bypasses the myriad challenges of creating a full semantic representation. We do this by decomposing the task into an ontologycrowd-relevance workflow, consisting of first representing the original text in a low-dimensional ontology, then crowdsourcing candidate question templates aligned with that space, and finally ranking potentially relevant templates for a novel region of text. If ontological labels are not available, we infer them from the text. We demonstrate the effectiveness of this method on a corpus of articles from Wikipedia alongside human judgments, and find that we can generate relevant deep questions with a precision of over 85% while maintaining a recall of 70%.

Learning Student and Content Embeddings for Personalized Lesson Sequence Recommendation

Students in online courses generate large amounts of data that can be used to personalize the learning process and improve quality of education. In this paper, we present the Latent Skill Embedding (LSE), a probabilistic model of students and educational content that can be used to recommend personalized sequences of lessons with the goal of helping students prepare for specific assessments. Akin to collaborative filtering for recommender systems, the algorithm does not require students or content to be described by features, but it learns a representation using access traces. We formulate this problem as a regularized maximum-likelihood embedding of students, lessons, and assessments from historical student-content interactions. Empirical findings on large-scale data from Knewton, an adaptive learning technology company, show that this approach predicts assessment results competitively with benchmark models and is able to discriminate between lesson sequences that lead to mastery and failure.

Unbounded Human Learning: Optimal Scheduling for Spaced Repetition

In the study of human learning, there is broad evidence that our ability to retain information improves with repeated exposure and decays with delay since last exposure. This plays a crucial role in the design of educational software, leading to a trade-off between teaching new material and reviewing what has already been taught. A common way to balance this trade-off is spaced repetition, which uses periodic review of content to improve long-term retention. Though spaced repetition is widely used in practice, e.g., in electronic flashcard software, there is little formal understanding of the design of these systems. Our paper addresses this gap in three ways. First, we mine log data from spaced repetition software to establish the functional dependence of retention on reinforcement and delay. Second, we use this memory model to develop a stochastic model for spaced repetition systems. We propose a queueing network model of the Leitner system for reviewing flashcards, along with a heuristic approximation that admits a tractable optimization problem for review scheduling. Finally, we empirically evaluate our queueing model through a Mechanical Turk experiment, verifying a key qualitative prediction of our model: the existence of a sharp phase transition in learning outcomes upon increasing the rate of new item introductions.

Design and calibration of single-camera catadioptric omnistereo system for miniature aerial vehicles (MAVs)

Stereo system plays an important role in the navigation of MAVs. In this paper, we design a single-camera catadioptric omnistereo system for MAV, which consists of one hyperboloidal mirror, one hyperboloidal-planar combined mirror, and one conventional camera. System parameters are optimized based on the analysis of constraints and each parameters influence on performance. Projective model of this system is derived, which provides a foundation for sphere-based calibration algorithm. It calibrates not only the conventional camera parameters, but also the mirror parameters. We also prove that a minimum of two spheres are needed to calibrate the seven parameters.

Generating Code-switched Text for Lexical Learning

A vast majority of L1 vocabulary acquisition occurs through incidental learning during reading (Nation, 2001; Schmitt et al., 2001). We propose a probabilistic approach to generating code-mixed text as an L2 technique for increasing retention in adult lexical learning through reading. Our model that takes as input a bilingual dictionary and an English text, and generates a code-switched text that optimizes a defined “learnability” metric by constructing a factor graph over lexical mentions. Using an artificial language vocabulary, we evaluate a set of algorithms for generating code-switched text automatically by presenting it to Mechanical Turk subjects and measuring recall in a sentence completion task.

Teaching Agents When They Fail: End User Development in Goal-Oriented Conversational Agents

This chapter introduces an end user development (EUD) approach for handling common types of failures encountered by goal-oriented conversational agents. We start with identifying three common sources of failures in human-agent conversations: unknown concepts, out-of-domain tasks and wrong fulfillment means or level of generalization in task execution. To handle these failures, it is useful to enable the end user to program the agent and to “teach” the agent what to do as a fallback strategy. Showing examples for this approach, we walk through our two integrated systems: Sugilite and Lia. Sugilite uses the programming by demonstration (PBD) technique, allowing the user to program the agent by demonstrating new tasks or new means for completing a task using the GUIs of third-party smartphone apps, while Lia learns new tasks from verbal instructions, enabling the user to teach the agent through breaking down the procedure verbally. Lia also enables the user to verbally define unknown concepts used in the commands and adds those concepts into the agent’s ontology. Both Sugilite and Lia can generalize what they have learned from the user across related entities and perform a task with new parameters in a different context.

A Queueing Network Model for Spaced Repetition

Flashcards are a popular study tool for exploiting the spacing effect -- the phenomenon in which periodic, spaced review of educational content improves long-term retention. The Leitner system is a simple heuristic algorithm for scheduling reviews such that forgotten items are reviewed more frequently than recalled items. We propose a formalization of the Leitner system as a queueing network model, and formulate optimal review scheduling as a throughput-maximization problem. Through simulations and theoretical analysis, we find that the Leitner Queue Network (LQN) model has desirable properties and gives insight into general principles for spaced repetition.

Fusing optical flow and stereo in a spherical depth panorama using a single-camera folded catadioptric rig

We present a novel catadioptric-stereo rig consisting of a coaxially-aligned perspective camera and two spherical mirrors with distinct radii in a “folded” configuration. We recover a nearly-spherical dense depth panorama (360°×153°) by fusing depth from optical flow and stereo. We observe that for motion in a horizontal plane, optical flow and stereo generate nearly complementary distributions of depth resolution. While optical flow provides strong depth cues in the periphery and near the poles of the view-sphere, stereo generates reliable depth in a narrow band about the equator. We exploit this principle by modeling the depth resolution of optical flow and stereo in order to fuse them probabilistically in a spherical panorama. To aid the designer in achieving a desired field-of-view and resolution, we derive a linearized model of the rig in terms of three parameters (radii of the two mirrors plus axial separation from their centers). We analyze the error due to the violation of the Single Viewpoint (SVP) constraint and formulate additional constraints on the design to minimize the error. Performance is evaluated through simulation and with a real prototype by computing dense spherical panoramas in cluttered indoor settings.