Maj Stenmark

Natural language programming of industrial robots

In this paper, we introduce a method to use written natural language instructions to program assembly tasks for industrial robots. In our application, we used a state-of-the-art semantic and syntactic parser together with semantically rich world and skill descriptions to create high-level symbolic task sequences. From these sequences, we generated executable code for both virtual and physical robot systems. Our focus lays on the applicability of these methods in an industrial setting with real-time constraints.

On Distributed Knowledge Bases for Robotized Small-Batch Assembly

The flexibility demands in manufacturing are severe, e.g., for rapid-change-over to new product variants, while robots are flexible machines that potentially can be adapted to a large variety of production tasks. Task definitions such as explicit robot programs are hardly reusable from an application point-of-view. To improve the situation, a knowledge-based approach exploiting distributed declarative information and cloud computing offers many possibilities for knowledge exchange and reuse, and it has the potential to facilitate new business models for industrial solutions. However, there are many unresolved questions yet, e.g., those related to reliability, consistency, or legal responsibility. To investigate some of these issues, different knowledge-based architectures have been prototyped and evaluated by confronting the solution candidates with key application demands. The conclusion is that distributed cloud-based approaches offer many possibilities, but there is still a need for further research and better infrastructure before this approach can become industrially attractive.

Describing constraint-based assembly tasks in unstructured natural language

Task-level industrial robot programming is a mundane, error-prone activity requiring expertise and skill. Since humans easily communicate with natural language (NL), it may be attractive to use speech or text as instruction means for robots. However, there has to be a substantial amount of knowledge in the system to translate the high-level language instructions to executable robot programs. In this paper, the method of Stenmark and Nugues (2013) for natural language programming of robotized assembly tasks is extended. The core idea of the method is to use a generic semantic parser to produce a set of predicate-argument structures from the input sentences. The algorithm presented here facilitates extraction of more complicated, advanced task instructions involving cardinalities, conditionals, parallelism and constraint-bounded programs, besides plain sequences of commands. The bottleneck of this approach is the availability of easily parametrizable robotic skills and functionalities in the system, rather than the natural language understanding by itself. (Less)

Knowledge-Based Industrial Robotics

When robots are working in dynamic environments, close to humans lacking extensive knowledge of robotics, there is a strong need to simplify the user interaction and make the system execute as autonomously as possible. For industrial robots working side-by-side with humans in manufacturing industry, AI systems are necessary to lower the demand on programming time and expertise. We are convinced that only by building a system with appropriate knowledge and reasoning services, we can simplify the robot programming sufficiently to meet those demands and still get a robust and efficient task execution. In this paper, we present a system we have realized that aims at fulfilling the above demands. The paper focuses on the ontologies we have created for robotic devices and manufacturing tasks, and presents examples of AI-related services using the semantic descriptions of the skills to help the user instruct the robot adequately. (Less)

From High-Level Task Descriptions to Executable Robot Code

For robots to be productive co-workers in the manufacturing industry, it is necessary that their human colleagues can interact with them and instruct them in a simple manner. The goal of our research is to lower the threshold for humans to instruct manipulation tasks, especially sensorcontrolled assembly. In our previous work we have presented tools for high-level task instruction, while in this paper we present how these symbolic descriptions of object manipulation are translated into executable code for our hybrid industrial robot controllers.

Industrial Robot Skills

When robots are working in dynamic environments, close to humans lacking extensive knowledge of robotics, there is a strong need to simplify the user interaction and make the system execute as autonomously as possible. For industrial robots working side-by-side with humans in manufacturing industry, AI systems are necessary to lower the demand on programming time and expertise. One central concept in knowledge modeling for robots is action representation. In this paper, we describe our representation of robot skills. The skills have resource requirements, logical and procedural information from which executable code can be generated.

Simplified Programming of Re-usable Skills on a Safe Industrial Robot: Prototype and Evaluation

This paper presents a study on iconic programming support for mainly position-based lead-through programming of an ABB YuMi collaborative robot. A prototype tool supporting a hybrid programming and execution mode was developed and evaluated with 21 non-expert users with varying programming and robotics experience. We also present a comparison of the programming times for an expert robot programmer using traditional tools versus the new tool. The expert programmed the same tasks in 1/5 of the time compared to traditional tools and the non-experts were able to program and debug a LEGO building task using the robot within 30 minutes.

Bilingual robots: Extracting robot program statements from Swedish natural language instructions

In the English-speaking world, the idea of human-robot interaction in natural language has been well established. The tools for other languages are lacking, more specifically, Scandinavian languages are not supported by robot programming environments. The RobotLab at Lund University has a programming environment with English natural language programming. In this paper a module for Swedish natural language programming is presented. Program statements for force-based assembly tasks for an industrial robot are extracted from unstructured Swedish text. The goal is to create action sequences with motion and force constraints for the robot. The method produces tuples with actions and objects and uses the dependency relations to find nested temporal conditions. (Less)

Supporting Semantic Capture during Kinesthetic Teaching of Collaborative Industrial Robots

Industrial robot systems being deployed today do not contain domain knowledge to aid robot operators in setup and operational use. To gather such knowledge in a robot context requires mechanisms for entering and capturing semantic data, that will gradually build a working vocabulary while interacting with environment and operators, for bootstrapping system knowledge and ensuring data collection over time. This paper presents a prototype user interface, assisting kinesthetic teaching of a collaborative industrial robot, that allows for capturing semantic information while working with the robot in day-to-day use. A graphical user interface with natural language processing builds a working vocabulary of the environment while modifying and/or creating robot programs. A simple demonstration illustrates the approach.