Yizhe Chang

REAL-TIME 3D MODEL RECONSTRUCTION AND INTERACTION USING KINECT FOR A GAME-BASED VIRTUAL LABORATORY

Over the last few years, academic institutions have started to explore the potential of using computer game engines for developing virtual laboratory environments. Recent studies have shown that developing a realistic visualization of a physical laboratory space poses a number of challenges. A significant number of modifications are required for adding customized interactions that are not built into the game engine itself. For example, a major challenge in creating a realistic virtual environment using a computer game engine is the process of preparing and converting custom models for integration into the environment, which is too complicated to be performed by untrained users. This paper describes the usage of the Microsoft Kinect for rapidly creating a 3D model of an object for implementation in a virtual environment by retrieving the object’s depth and RGB information. A laboratory experiment was selected to demonstrate how real experimental components are reconstructed and embedded into a game-based virtual laboratory by using the Kinect. The users are then able to interact with the experimental components. This paper presents both the technical details of the implementation and some initial results of the system validation.

Integration of a Real-Time Remote Experiment Into a Multi-Player Game Laboratory Environment

While real-time remote experiments have been used in engineering and science education for over a decade, more recently virtual learning environments based on game systems have been explored for their potential usage in educational laboratories. However, combining the advantages of both these approaches and integrating them into an effective learning environment has not been reported yet. One of the challenges in creating such a combination is to overcome the barriers between real and virtual systems, i.e. to select compatible platforms, to achieve an efficient mapping between the real world and the virtual environment and to arrange for efficient communications between the different system components.This paper will present a pilot implementation of a multi-player game-based virtual laboratory environment that is linked to the remote experimental setup of an air flow rig. This system is designed for a junior-level mechanical engineering laboratory on fluid mechanics. In order to integrate this remote laboratory setup into the virtual laboratory environment, an existing remote laboratory architecture had to be redesigned. The integrated virtual laboratory platform consists of two main parts, namely an actual physical experimental device controlled by a remote controller and a virtual laboratory environment that was implemented using the ‘Source’ game engine, which forms the basis of the commercially available computer game ‘Half-Life 2’ in conjunction with ‘Garry’s Mod’ (GM). The system implemented involves a local device controller that exchanges data in the form of shared variables and Dynamical Link Library (DLL) files with the virtual laboratory environment, thus establishing the control of real physical experiments from inside the virtual laboratory environment. The application of a combination of C++ code, Lua scripts [1] and LabVIEW Virtual Instruments makes the platform very flexible and expandable. This paper will present the architecture of this platform and discuss the general benefits of virtual environments that are linked with real physical devices.Copyright

Capturing Assembly Constraints of Experimental Setups in a Virtual Laboratory Environment

Recently, multi-player game engines have been explored regarding their potential for implementing virtual laboratory environments for engineering and science education. In these developments, the virtual assembly process of the laboratory equipment is a critical step, and a detailed formalized description of how different components of the experimental equipment are to be joined in the assembly process is necessary. This description includes the joint types (lower and upper kinematic pairs) and the associated degrees of freedom, the resulting mobility of the assembly as well as the joint fit requirements.In this paper, a formalized representation of the assembly process that captures the information on the joint kinematics and the components’ degrees of freedom generated when assembling laboratory equipment in a virtual laboratory environment will be discussed. A planetary gear train system will be used as an example to illustrate the proposed method. In particular, the structure of the assembly of a planetary gear train system involves assembly constraints between a group of components (sun, planet and ring gears, shafts, planet carrier assembly, etc.) that generate the desired relationship between the input and output motions. This paper will identify important requirements for modeling different configurations of planetary gear train assemblies within a game-based virtual laboratory environment. These requirements include the positioning and the orienting of the components, the verification of the kinematic joints, the propagation of the mating constraints and the capturing of the joint attributes.Copyright

Virtual Mechanical Assembly Training Based on a 3D Game Engine

ABSTRACTDesigned for personal computers, 3D video games are powerful tools with respect to graphics rendering, real world physics simulation, human-computer interaction and multi-user communication. Because of these favorable features of 3D games, their adaption for serious applications has been widely researched recently. These applications generally focus on topics such as real world scenario reconstruction, which require no or only minor development efforts on the game engines themselves. Contrary to this, the development of virtual education and training environments requires the integration of complex engineering systems into games, which poses greater challenges and thus causes this topic to be covered less frequently. This article presents a framework for authoring virtual environments (VEs) for mechanical assembly training using a commercially available 3D game engine. The VE presented here allows multiple users to conduct simulations of assembly procedures in a collaborative manner and provides a...

A Smart Method for Developing Game-Based Virtual Laboratories

Virtual laboratories are one popular form of implementation of virtual reality. They are now widely used at various levels of education. Game-based virtual laboratories created using game engines take advantage of the resources of these game engines. While providing convenience to developers of virtual laboratories, game engines also exhibit the following shortcomings: (1) They are not realistic enough. (2) They require long design and modification periods. (3) They lack customizability and flexibility. (4) They are endowed with limited artificial intelligence. These shortcomings render game-based virtual laboratories (and other virtual laboratories) inferior to traditional laboratories.This paper proposes a smart method for developing game-based virtual laboratories that overcomes these shortcomings. In this method, 3D reconstruction and pattern recognition techniques are adopted. 3D reconstruction techniques are used to create a virtual environment, which includes virtual models of real objects and a virtual space. These techniques can render this virtual environment fairly realistic, can reduce the time and effort of creating the virtual environment and can increase the flexibility in the creation of the virtual environment. Furthermore, pattern recognition techniques are used to endow game-based virtual laboratories with general artificial intelligence. The scanned objects can be recognized, and certain attributes of real objects can be added automatically to their virtual models. In addition, the emphasis of the experiments can be adjusted according to the users’ abilities in order to get better training results. As a prototype, an undergraduate student laboratory was developed and implemented. Finally, additional improvements in the approach for creating game-based virtual laboratories are discussed.Copyright

A Platform for Mechanical Assembly Education Using the Microsoft Kinect

Mechanical assembly activities involve multiple factors including humans, mechanical parts, tools and assembly environments. In order to simulate assembly processes by computers for educational purposes, all these factors should be considered. Virtual reality (VR) technology, which aims to integrate natural human motion into real-world scenarios, provides an ideal simulation medium. Novel VR devices such as 3D glasses, motion-tracking gloves, haptic sensors, etc. are able to fulfill fundamental assembly simulation needs. However, most of these implementations focus on assembly simulations for computer-aided design, which are geared toward professionals rather than students, thus leading to complicated assembly procedures not suitable for students. Furthermore, the costs of these novel VR devices and specifically designed VR platforms represent an untenable financial burden for most educational institutions.In this paper, a virtual platform for mechanical assembly education based on the Microsoft Kinect sensor and Garry’s Mod (GMod) is presented. With the help of the Kinect’s body tracking function and voice recognition technology in conjunction with the graphics and physics simulation capabilities of GMod, a low-cost VR platform that enables educators to author their own assembly simulations was implemented. This platform utilizes the Kinect as the sole input device. Students can use voice commands to navigate their avatars inside of a GMod powered virtual laboratory as well as use their body’s motions to integrate pre-defined mechanical parts into assemblies. Under this platform, assembly procedures involving the picking, placing and attaching of parts can be performed collaboratively by multiple users. In addition, the platform allows collaborative learning without the need for the learners to be co-located. A pilot study for this platform showed that, with the instructor’s assistance, mechanical engineering undergraduate students are able to complete basic assembly operations.Copyright

Collaborative Virtual Laboratory Environments with Hardware in the Loop

Over the last decade, the research community has expanded substantial efforts aiming at designing, agreeing on, and rolling out technical standards and powerful universal development tools that allow the rapid and cost-effective integration of specific experimental devices into standardized remote laboratory platforms. In this chapter, a virtual laboratory system with experimental hardware in the loop is described.

A Virtual Laboratory Combined With Biometric Authentication and 3D Reconstruction

Virtual laboratories are used in professional skill development, online education, and corporate training. There are several aspects that determine the effectiveness and popularity of virtual laboratories: (i) the benefits brought to the users compared with those provided by traditional physical hands-on laboratories, (ii) the cost of adopting a virtual laboratory which includes the costs of creating the virtual environment and developing virtual experiments, and (iii) the operation which includes the communication between trainers and trainees, the authentication and remote proctoring of the trainees, etc. At present, the procedures of building and operating a virtual laboratory are still tedious, time-consuming and resource-intense, thus considerably limiting the potential applications and popularization of virtual laboratories.In this paper, a virtual laboratory built and operated with 3D reconstruction and biometric authentication is introduced and an evaluation of the feasibility of the proposed approaches is presented.3D reconstruction techniques are used to create the virtual environment of this virtual laboratory. The traditional tools used to survey the real world are replaced by a hand-held camera. Then, all of the information acquired by this hand-held camera is processed. Finally, the virtual environment of the virtual laboratory is generated automatically in real-time.The biometric authentication techniques (here facial recognition techniques) are used to create a remote proctor. The general logic and basic algorithms used to enable biometric authentication and remote proctoring are described. When using this virtual laboratory, the students log in by capturing their face with a camera. While performing a laboratory exercise, they sit in front of the camera and the virtual laboratory system monitors their facial expressions and the motion of their head in order to identify suspicious behaviors. Upon detection of such suspicious behaviors, the system records a video for further analysis by the laboratory administrator.Copyright

Overcoming the Limitations of Current Online Laboratory Systems Using Game-Based Virtual Environments

Current online laboratory systems based on either remotely operated hardware or pure software simulations are often criticized for lacking experimental setup experience and for being limited to the data collection and result analysis aspects of traditional hands-on laboratories. One possible solution for overcoming these shortcomings is to use a multi-player computer game engine for implementing interactive laboratory environments. Taking advantage of game technology, the students are enabled to simultaneously interact with laboratory equipment in the virtual laboratory and to communicate with each other through voice or chat. In particular, assembling the experimental equipment before using it in some experimental procedure can become part of the laboratory exercise. In this article, a virtual gear design laboratory environment for a junior-level undergraduate mechanical engineering course on mechanisms and machine dynamics is presented. This virtual laboratory environment implemented using a multiplayer computer game engine provides the students with the flexibility to perform various experiments related to the concepts of the fundamental law of gearing and to planetary motion of gears. In this environment, the students can work in teams to interactively select and cooperatively assemble gear train components piece by piece into a more complex experimental apparatus. This article describes the assembly logic that is used to combine individual components into experimental setups as part of the experimental procedure in student laboratories. Furthermore, the game environment can be equipped with functionality for monitoring the students’ progress and learning outcomes, thus enabling skill-based assessment.Copyright

Adapting Computer Game Technology to Support Engineering Laboratories

Today’s youth has become accustomed to and skilled in computer gaming. The potential use of multi-player computer games as an educational tool has drawn significant attention. Computer games exhibit several features that could provide the students with an effective learning experience. This paper introduces different methodologies for adopting computer game technologies such as the “Source” game engine (used in “Half-Life 2”) and “Second Life” to develop game-based interactive student laboratory environments. A number of predefined scenarios can be scripted which exercise the students’ problem solving skills by mimicking typical problems that might occur when carrying out actual hands-on experiments in a traditional student laboratory. The suitability of several gaming platforms for implementing such laboratory environments is discussed. The article concludes with some observations on the most relevant aspects of game platforms that can help to create engaging and effective digital learning environments to support engineering laboratories.Copyright