Naresh Marturi

Visual servoing-based approach for efficient autofocusing in scanning electron microscope

Fast and reliable autofocusing methods are essential for performing automatic nano-objects positioning tasks using a scanning electron microscope (SEM). So far in the literature, various autofocusing algorithms have been proposed utilizing a sharpness measure to compute the best focus. Most of them are based on iterative search approaches; applying the sharpness function over the total range of focus to find an image in-focus. In this paper, a new, fast and direct method of autofocusing has been presented based on the idea of traditional visual servoing to control the focus step using an adaptive gain. The visual control law is validated using a normalized variance sharpness function. The obtained experimental results demonstrate the performance of the proposed autofocusing method in terms of accuracy, speed and robustness.

Visual servoing schemes for automatic nanopositioning under scanning electron microscope

This paper presents two visual servoing approaches for nanopositioning in a scanning electron microscope (SEM). The first approach uses the total pixel intensities of an image as visual measurements for designing the control law. The positioning error and the platform control are directly linked with the intensity variations. The second approach is a frequency domain method that uses Fourier transform to compute the relative motion between images. In this case, the control law is designed to minimize the error i.e. the 2D motion between current and desired images by controlling the positioning platform movement. Both methods are validated at different experimental conditions for a task of positioning silicon microparts using a piezo-positioning platform. The obtained results demonstrate the efficiency and robustness of the developed methods.

Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videos

Abstract This paper addresses the problems of both general and also fine-grained human action recognition in video sequences. Compared with general human actions, fine-grained action information is more difficult to detect and occupies relatively small-scale image regions. Our work seeks to improve fine-grained action discrimination, while also retaining the ability to perform general action recognition. Our method first estimates human pose and human parts positions in video sequences by extending our recent work on human pose tracking, and crops different scaled patches to obtain richer action information in a variety of different scales of appearance and motion cues. We then utilize a Convolutional Neural Network (CNN) to process each such image patch. Instead of using the output one dimension feature from the full-connection layer, we utilize the outputs of the pooling layer of CNN structure, which contains more spatial information. Then the high dimension of the pooling features is reduced by encoding, to generate the final human action descriptors for classification. Our method reduces feature dimension while also effectively combining appearance and motion information in a unified framework. We have carried out empirical experiments using two publicly available human action datasets, comparing the human action recognition result of our algorithm against six recent state-of-the-art methods from the literature. The results suggest comparatively strong performance of our method.

Scanning electron microscope image signal-to-noise ratio monitoring for micro-nanomanipulation.

As an imaging system, scanning electron microscope (SEM) performs an important role in autonomous micro-nanomanipulation applications. When it comes to the sub micrometer range and at high scanning speeds, the images produced by the SEM are noisy and need to be evaluated or corrected beforehand. In this article, the quality of images produced by a tungsten gun SEM has been evaluated by quantifying the level of image signal-to-noise ratio (SNR). In order to determine the SNR, an efficient and online monitoring method is developed based on the nonlinear filtering using a single image. Using this method, the quality of images produced by a tungsten gun SEM is monitored at different experimental conditions. The derived results demonstrate the developed methods efficiency in SNR quantification and illustrate the imaging quality evolution in SEM.

Visual Servoing-Based Depth-Estimation Technique for Manipulation Inside SEM

Depth estimation for micronanomanipulation inside a scanning electron microscope (SEM) is always a major concern. So far, in the literature, various methods have been proposed based on stereoscopic imaging. Most of them require external hardware unit or manual interaction during the process. In this paper, solely relying on image sharpness information, we present a new technique to estimate the depth in real time. To improve the accuracy as well as the rapidity of the method, we consider both autofocus and depth estimation as visual servoing paradigms. The major flexibility of the method lies in its ability to compute the focus position and the depth using only the acquired image information, i.e., sharpness. The feasibility of the method is shown by performing various ground truth experiments: autofocus achievements, depth estimation, focus-based nanomanipulator depth control, and sample topographic estimation at different scenarios inside the vacuum chamber of a tungsten gun SEM. The obtained results demonstrate the accuracy, rapidity, and efficiency of the developed method.

Fast image drift compensation in scanning electron microscope using image registration

Scanning Electron Microscope (SEM) image acquisition is mostly affected by the time varying motion of pixel positions in the consecutive images, a phenomenon called drift. In order to perform accurate measurements using SEM, it is necessary to compensate this drift in advance. Most of the existing drift compensation methods were developed using the image correlation technique. In this paper, we present an image registration-based drift compensation method, where the correction on the distorted image is performed by computing the homography, using the keypoint correspondences between the images. Four keypoint detection algorithms have been used for this work. The obtained experimental results demonstrate the methods performance and efficiency in comparison with the correlation technique.

Towards advanced robotic manipulation for nuclear decommissioning: A pilot study on tele-operation and autonomy

We present early pilot-studies of a new international project, developing advanced robotics to handle nuclear waste. Despite enormous remote handling requirements, there has been remarkably little use of robots by the nuclear industry. The few robots deployed have been directly teleoperated in rudimentary ways, with no advanced control methods or autonomy. Most remote handling is still done by an aging workforce of highly skilled experts, using 1960s style mechanical Master-Slave devices. In contrast, this paper explores how novice human operators can rapidly learn to control modern robots to perform basic manipulation tasks; also how autonomous robotics techniques can be used for operator assistance, to increase throughput rates, decrease errors, and enhance safety. We compare humans directly teleoperating a robot arm, against human-supervised semi-autonomous control exploiting computer vision, visual servoing and autonomous grasping algorithms. We show how novice operators rapidly improve their performance with training; suggest how training needs might scale with task complexity; and demonstrate how advanced autonomous robotics techniques can help human operators improve their overall task performance. An additional contribution of this paper is to show how rigorous experimental and analytical methods from human factors research, can be applied to perform principled scientific evaluations of human test-subjects controlling robots to perform practical manipulative tasks.

Vision-guided state estimation and control of robotic manipulators which lack proprioceptive sensors

This paper presents a vision-based approach for estimating the configuration of, and providing control signals for, an under-sensored robot manipulator using a single monocular camera. Some remote manipulators, used for decommissioning tasks in the nuclear industry, lack proprioceptive sensors because electronics are vulnerable to radiation. Additionally, even if proprioceptive joint sensors could be retrofitted, such heavy-duty manipulators are often deployed on mobile vehicle platforms, which are significantly and erratically perturbed when powerful hydraulic drilling or cutting tools are deployed at the end-effector. In these scenarios, it would be beneficial to use external sensory information, e.g. vision, for estimating the robot configuration with respect to the scene or task. Conventional visual servoing methods typically rely on joint encoder values for controlling the robot. In contrast, our framework assumes that no joint encoders are available, and estimates the robot configuration by visually tracking several parts of the robot, and then enforcing equality between a set of transformation matrices which relate the frames of the camera, world and tracked robot parts. To accomplish this, we propose two alternative methods based on optimisation. We evaluate the performance of our developed framework by visually tracking the pose of a conventional robot arm, where the joint encoders are used to provide ground-truth for evaluating the precision of the vision system. Additionally, we evaluate the precision with which visual feedback can be used to control the robots end-effector to follow a desired trajectory.

Toward Fast Calibration of Global Drift in Scanning Electron Microscopes with Respect to Time and Magnification

It is a well-known fact that scanning electron microscopic (SEM) image acquisition is mainly affected by nonlinearities and instabilities of the column and probe-specimen interaction; in turn, producing a shift in the image points with respect to many parameters and time, in particular. Even though this drift is comparatively less in modern SEMs, it is still an important factor to consider in most of the SEM-based applications. In this airticle, a simple and real-time method is proposed to estimate the global drift from a set of target images using image phase correlation, and to model its evolution by using the recursive equations of time and magnification. Based on the developed model, it is opted to use a Kalman filter in real time for accurate estimation and removal of the drift from the images. The developed method is tested using the images from a tungsten filament gun SEM (Jeol JSM 820) and a field effect gun SEM (FEI Quanta 200). The derived results show the effectiveness of the developed algorithm and also demonstrates its ability to be used in robotics as well as in material characterization under SEM.

A Q-learning-based memetic algorithm for multi-objective dynamic software project scheduling

Abstract Software project scheduling is the problem of allocating employees to tasks in a software project. Due to the large scale of current software projects, many studies have investigated the use of optimization algorithms to find good software project schedules. However, despite the importance of human factors to the success of software projects, existing work has considered only a limited number of human properties when formulating software project scheduling as an optimization problem. Moreover, the changing environments of software companies mean that software project scheduling is a dynamic optimization problem. However, there is a lack of effective dynamic scheduling approaches to solve this problem. This work proposes a more realistic mathematical model for the dynamic software project scheduling problem. This model considers that skill proficiency can improve over time and, different from previous work, it considers that such improvement is affected by the employees’ properties of motivation and learning ability, and by the skill difficulty. It also defines the objective of employees’ satisfaction with the allocation. It is considered together with the objectives of project duration, cost, robustness and stability under a variety of practical constraints. To adapt schedules to the dynamically changing software project environments, a multi-objective two-archive memetic algorithm based on Q-learning (MOTAMAQ) is proposed to solve the problem in a proactive-rescheduling way. Different from previous work, MOTAMAQ learns the most appropriate global and local search methods to be used for different software project environment states by using Q-learning. Extensive experiments on 18 dynamic benchmark instances and 3 instances derived from real-world software projects were performed. A comparison with seven other meta-heuristic algorithms shows that the strategies used by our novel approach are very effective in improving its convergence performance in dynamic environments, while maintaining a good distribution and spread of solutions. The Q-learning-based learning mechanism can choose appropriate search operators for the different scheduling environments. We also show how different trade-offs among the five objectives can provide software managers with a deeper insight into various compromises among many objectives, and enabling them to make informed decisions.