Jongyoo Kim

Fully Deep Blind Image Quality Predictor

In general, owing to the benefits obtained from original information, full-reference image quality assessment (FR-IQA) achieves relatively higher prediction accuracy than no-reference image quality assessment (NR-IQA). By fully utilizing reference images, conventional FR-IQA methods have been investigated to produce objective scores that are close to subjective scores. In contrast, NR-IQA does not consider reference images; thus, its performance is inferior to that of FR-IQA. To alleviate this accuracy discrepancy between FR-IQA and NR-IQA methods, we propose a blind image evaluator based on a convolutional neural network (BIECON). To imitate FR-IQA behavior, we adopt the strong representation power of a deep convolutional neural network to generate a local quality map, similar to FR-IQA. To obtain the best results from the deep neural network, replacing hand-crafted features with automatically learned features is necessary. To apply the deep model to the NR-IQA framework, three critical problems must be resolved: 1) lack of training data; 2) absence of local ground truth targets; and 3) different purposes of feature learning. BIECON follows the FR-IQA behavior using the local quality maps as intermediate targets for conventional neural networks, which leads to NR-IQA prediction accuracy that is comparable with that of state-of-the-art FR-IQA methods.

Epstein–Barr virus reactivation in extranodal natural killer/T-cell lymphoma patients: a previously unrecognized serious adverse event in a pilot study with romidepsin

BACKGROUND Romidepsin, a histone deacetylase (HDAC) inhibitor, has been approved for the treatment of relapsed and refractory peripheral T-cell lymphoma. However, the efficacy and safety of romidepsin has never been studied in patients with relapsed or refractory extranodal natural killer (NK)/T-cell lymphoma (ENKTL). PATIENTS AND METHODS We conducted an open-label, prospective pilot study to evaluate the efficacy and feasibility of romidepsin in the treatment of patients with ENKTL. The treatment was intravenous infusion of romidepsin (14 mg/m(2)) for 4 h on days 1, 8, and 15 of a 28-day cycle, and was repeated until disease progression or the occurrence of unacceptable toxicity. RESULTS A total of five patients enrolled on to this pilot study. However, three patients developed fever and elevated liver enzyme and bilirubin levels immediately after their first administration of romidepsin. We suspected that these events were associated with Epstein-Barr virus (EBV) reactivation because of the rapidly elevated EBV DNA titers in blood from these patients. An in vitro study with the ENKTL cell line SNK-6 cells also showed that HDAC inhibitors including romidepsin increased the copy number of EBV DNA in a dose-dependent manner. These findings suggested that romidepsin-induced histone acetylation reversed the repressed state of the genes required for EBV reactivation and that romidepsin treatment may have caused EBV reactivation in EBV-infected tumor cells in ENKTL patients. Therefore, we discontinued the enrollment of patients into this pilot study. CONCLUSIONS Our study suggests that the use of romidepsin may cause severe EBV reactivation in patients with ENKTL.

Minimal residual disease-based effect and long-term outcome of first-line dasatinib combined with chemotherapy for adult Philadelphia chromosome-positive acute lymphoblastic leukemia

BACKGROUND The use of imatinib combined with chemotherapy has demonstrated improved outcome in adults with Philadelphia chromosome-positive acute lymphoblastic leukemia (Ph-positive ALL). However, a substantial proportion of patients continue to die as a result of disease progression. PATIENTS AND METHODS We assessed the minimal residual disease (MRD)-based effect and long-term outcome of first-line incorporation of dasatinib (100 mg once daily) into chemotherapy alternatively for adults with Ph-positive ALL. The primary end point was the major molecular response (MMR) rate by the end of the second dasatinib cycle. Patients with a donor proceeded to allogeneic stem cell transplantation (SCT) as early as possible. MRD monitoring was centrally evaluated by real-time quantitative polymerase chain reaction (4.5-log sensitivity) using bone marrow samples. RESULTS Fifty-one patients (median age, 46 years) were enrolled and treated with this strategy. After the first dasatinib cycle, 50 patients (98.0%) achieved complete remission (CR). By the end of the second dasatinib cycle, 46 (93.9%) of 49 assessable patients had persistent CR, and 38 (77.6%) had MMR (32.7%) or undetectable MRD (44.9%). On the basis of the MRD kinetics by this time point, the numbers of early-stable, late, and poor molecular responders were 23 (46.9%), 15 (30.7%), and 11 (22.4%), respectively. Thirty-nine patients (76.5%) underwent allogeneic SCT in CR1. After a median follow-up of 54 months, the 4-year cumulative incidence of relapse and disease-free survival (DFS) rate for all patients were 30.0% and 52.0%, respectively, and the corresponding outcomes among those receiving allogeneic SCT in CR1 were 20.5% and 64.1%, respectively. Poor molecular responders had a higher risk of relapse and DFS than those of early-stable molecular responders. CONCLUSION This dasatinib-based protocol was effective for achieving a good quality molecular response and durable DFS in adults with Ph-positive ALL. TRIAL REGISTRATION clinicaltrials.gov, NCT01004497.

Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework

Since human observers are the ultimate receivers of digital images, image quality metrics should be designed from a human-oriented perspective. Conventionally, a number of full-reference image quality assessment (FR-IQA) methods adopted various computational models of the human visual system (HVS) from psychological vision science research. In this paper, we propose a novel convolutional neural networks (CNN) based FR-IQA model, named Deep Image Quality Assessment (DeepQA), where the behavior of the HVS is learned from the underlying data distribution of IQA databases. Different from previous studies, our model seeks the optimal visual weight based on understanding of database information itself without any prior knowledge of the HVS. Through the experiments, we show that the predicted visual sensitivity maps agree with the human subjective opinions. In addition, DeepQA achieves the state-of-the-art prediction accuracy among FR-IQA models.

Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having ~ 91% correlation in terms of MOS.Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having ~ 91% correlation in terms of MOS.

Enhancement of Visual Comfort and Sense of Presence on Stereoscopic 3D Images

Conventional stereoscopic 3D (S3D) displays do not provide accommodation depth cues of the 3D image or video contents being viewed. The sense of content depths is thus limited to cues supplied by motion parallax (for 3D video), stereoscopic vergence cues created by presenting left and right views to the respective eyes, and other contextual and perspective depth cues. The absence of accommodation cues can induce two kinds of accommodation vergence mismatches (AVM) at the fixation and peripheral points, which can result in severe visual discomfort. With the aim of alleviating discomfort arising from AVM, we propose a new visual comfort enhancement approach for processing S3D visual signals to deliver a more comfortable 3D viewing experience at the display. This is accomplished via an optimization process whereby a predictive indicator of visual discomfort is minimized, while still aiming to maintain the viewer’s sense of 3D presence by performing a suitable parallax shift, and by directed blurring of the signal. Our processing framework is defined on 3D visual coordinates that reflect the nonuniform resolution of retinal sensors and that uses a measure of 3D saliency strength. An appropriate level of blur that corresponds to the degree of parallax shift is found, making it possible to produce synthetic accommodation cues implemented using a perceptively relevant filter. By this method, AVM, the primary contributor to the discomfort felt when viewing S3D images, is reduced. We show via a series of subjective experiments that the proposed approach improves visual comfort while preserving the sense of 3D presence.

Implementation of Human Action Recognition System Using Multiple Kinect Sensors

Human action recognition is an important research topic that has many potential applications such as video surveillance, human-computer interaction and virtual reality combat training. However, many researches of human action recognition have been performed in single camera system, and has low performance due to vulnerability to partial occlusion. In this paper, we propose a human action recognition system using multiple Kinect sensors to overcome the limitation of conventional single camera based human action recognition system. To test feasibility of the proposed system, we use the snapshot and temporal features which are extracted from three-dimensional (3D) skeleton data sequences, and apply the support vector machine (SVM) for classification of human action. The experiment results demonstrate the feasibility of the proposed system.

An identification framework for print-scan books in a large database

In this paper, we propose an identification framework to determine copyright infringement in the form of illegally distributed print-scan books in a large database. The framework contains following main stages: image pre-processing, feature vector extraction, clustering, and indexing, and hierarchical search. The image pre-processing stage provides methods for alleviating the distortions induced by a scanner or digital camera. From the pre-processed image, we propose to generate feature vectors that are robust against distortion. To enhance the clustering performance in a large database, we use a clustering method based on the parallel-distributed computing of Hadoop MapReduce. In addition, to store the clustered feature vectors efficiently and minimize the searching time, we investigate an inverted index for feature vectors. Finally, we implement a two-step hierarchical search to achieve fast and accurate on-line identification. In a simulation, the proposed identification framework shows accurate and robust in the presence of print-scan distortions. The processing time analysis in a parallel computing environment gives extensibility of the proposed framework to massive data. In the matching performance analysis, we empirically and theoretically find that in terms of query time, the optimal number of clusters scales with O(N) for N print-scan books.

Blind Sharpness Prediction for Ultrahigh-Definition Video Based on Human Visual Resolution

We explore a no-reference sharpness assessment model for predicting the perceptual sharpness of ultrahigh-definition (UHD) videos through analysis of visual resolution variation in terms of viewing geometry and scene characteristics. The quality and sharpness of UHD videos are influenced by viewer perception of the spatial resolution afforded by the UHD display, which depends on viewing geometry parameters including display resolution, display size, and viewing distance. In addition, viewers may perceive different degrees of quality and sharpness according to the statistical behavior of the visual signals, such as the motion, texture, and edge, which vary over both spatial and temporal domains. The model also accounts for the resolution variation associated with fixation and foveal regions, which is another important factor affecting the sharpness prediction of UHD video over the spatial domain and which is caused by the nonuniform distribution of the photoreceptors. We calculate the transition of the visually salient statistical characteristics resulting from changing the display’s screen size and resolution. Moreover, we calculated the temporal variation in sharpness over consecutive frames in order to evaluate the temporal sharpness perception of UHD video. We verify that the proposed model outperforms other sharpness models in both spatial and temporal sharpness assessments.

3D visual discomfort predictor based on neural activity statistics

Visual discomfort assessment (VDA) on stereoscopic images is of fundamental importance for making decisions regarding visual fatigue caused by unnatural binocular alignment. Nevertheless, no solid framework exists to quantify this discomfort using models of the responses of visual neurons. Binocular vision is realized by means of neural mechanisms that subserve the sensorimotor control of eye movements. We propose a neuronal model-based framework called Neural 3D Visual Discomfort Predictor (N3D-VDP) that automatically predicts the level of visual discomfort experienced when viewing stereoscopic 3D (S3D) images. The N3D-VDP model extracts features derived by estimating the neural activity associated with the processing of binocular disparities. In this regard we deploy a model of disparity processing in the extra-striate middle temporal (MT) region of occipital lobe. We compare the performance of N3D-VDP with other recent VDA algorithms using correlations against reported subjective visual discomfort, and show that N3D-VDP is statistically superior to the other methods.