A Manifold Alignment Approach to Grounded Language Learning

Abstract

As robots become advanced and affordable enough to have in our daily lives, the next question is: How do we make using these machines as intuitive as possible? Language offers an approachable and relatively accessible interface without requiring prior training on the part of the user. We have seen the integration of voice-assistant speakers in homes drastically increase in the recent years. Voice, and more specifically language, is proving to be a preferred method for interacting with AI-enabled assistants. However, understanding how natural language applies to the physical world is still very much an open problem. Combining language and robotics creates unique challenges that much of the current work on grounded language learning has not addressed. Our proposed approach is to jointly learn language and world representations by learning a projection of both the language and sensor data into a joint space, using a process known as manifold alignment. This will enable learning of more complex grounded language in a domain-independent way. Once completed, this work will provide a bridge between the noisy, multimodal perceived world of the robotic agent and unconstrained natural language.