Depth Completion with RGB Prior
DDepth Completion with RGB Prior
Yuri Feldman , Yoel Shapiro , Dotan Di Castro Technion - Israel Institute of Technology [email protected] Bosch Center for Artificial Intelligence {Shapiro.Yoel,Dotan.DiCastro}@bosch.com
Abstract.
Depth cameras are a prominent perception system for robotics,especially when operating in natural unstructured environments. Indus-trial applications, however, typically involve reflective objects under harshlighting conditions, a challenging scenario for depth cameras, as it in-duces numerous reflections and deflections, leading to loss of robustnessand deteriorated accuracy. Here, we developed a deep model to correctthe depth channel in RGBD images, aiming to restore the depth in-formation to the required accuracy. To train the model, we created anovel industrial dataset that we now present to the public. The data wascollected with low-end depth cameras and the ground truth depth wasgenerated by multi-view fusion.
Keywords: depth, depth camera, RGBD, 3D localization, image en-hancement, industrial robotics, manufacturing robotics, robotics
Robotic manipulation has been an active research field for a few decades and isgaining renewed traction with novel learning-based methods. Industrial roboticshas especially benefited from Deep Reinforcement Learning (DRL; [43]) and deepcomputer vision. A typical setup for industrial systems is a robotic arm servedby a perception system that drives a visual signal into the robot’s high-levelcontroller. Industrial applications, and manufacturing in particular, require pre-cise localization and pose estimation for successful grasping of the manipulatedobject. This enhanced perception requirement is often catered by depth cameras[37] which yield RGBD images, i.e. RGB and an additional depth map.Here, we investigate the usage of a 3D camera in an industrial use-case. Byusing a relatively inexpensive 3D camera worth around $150, we collected over3,500 samples of 45 different industrial objects. Industrial scenarios are predom-inantly reflective since they are composed of smooth metallic objects, includingparts, tools, and jigs. Under these conditions, many low-end 3D cameras providepoor depth data with high noise and many pixels with invalid values. These We use Intel RealSense 415 3D camera. a r X i v : . [ c s . C V ] A ug Yuri Feldman , Yoel Shapiro , Dotan Di Castro issues arise from limitations of the Infra-Red (IR) beam, which is used in Time-of-Flight (ToF) depth cameras to scan the field of view or to produce patterns forpoint-correlations in structured-light depth cameras. Reflections between scenecomponents can confuse the sensors and create artifacts. For example, a reflec-tive table top may appear to have dents, mirroring the objects laid upon it.Attentive work station design can minimize such reflections, but they cannotbe entirely eliminated and are considered difficult to handle. A more pervasiveproblem is encountered on corners and edges of reflective objects, where the IRbeams often deflect away from the sensor’s field of view, leaving a ”hole” inthe depth channel. To alleviate the missing values issue, depth camera manu-facturers usually provide post-processing algorithms, including smoothing filtersand interpolations. However, the accuracy of these traditional methods tends tofall short of robotic manufacturing requirements. The system we used calculatesdepth using stereo cameras, enhanced by a static-pattern IR projector. High-endsystem can achieve better accuracy and robustness, for example by projecting aseries of carefully crafted patterns at the cost of a longer acquisition time. More-over, the cost of high-end systems can reach two orders of magnitude greater.Taken together, high end depth cameras are not an acceptable solution for mostindustrial applications.An alternative to depth-cameras is multi-view perception, using ten or moredifferent view-points in space (and possibly time) [42,27,46]. The most notablevariants of multi-view methods are Simultaneous Localization and Mapping (SLAM;[35,15]) and
Structure from Motion (SfM; [10,19]), which are quite popular formobile robots. However, SLAM and SfM are inclined to intensive computationsand are inherently time-consuming, raising doubts on their applicability to in-dustrial robotics and manufacturing. In the opposite direction, previous studieson monocular depth estimation applied a deep model on a single view image toestimate depth. While monocular depth estimation has been successful in sev-eral domains [14,21,25], it has not yet reached sufficient accuracy for industrialapplications.A promising avenue is to employ depth cameras for acquisition, followed bypost-processing by depth corrections algorithms that exploit the RGB informa-tion as a prior for the correct depth values. In the next section, we provide detailson two notable works [53,41] that follow this direction. These studies break downthe solution into two steps. The first step applies a deep model on the RGB infor-mation to estimate surface normals and additional tasks. The second step usesoptimization techniques to find a depth surface that fulfills a mixed objective,combined from the depth information and the outputs of the first step.In this work, we use an end-to-end deep learning (DL; [28]) approach wherewe train a network to complete the depth in the missing places. We appliedan encoding-decoding architecture, with a ResNet [17] or VGG [44] backbone,adding a U-net style skip connections [38] between the corresponding encoderand decoder layers. We collected industrial real-world training data with a staticrig of multiple (4) depth cameras. The ground truth depth maps are generatedusing TSDF [7], which is an imperfect approximation of the true depth values. epth Completion with RGB Prior 3
Nonetheless, our model is trained to improve the input depth to the level of theattainable ground truth estimation. In the results section (Sec. 4), we quantifythe average error between the input and ground-truth and show that our modelreduces it by 1-2 orders of magnitude. Overall, our contributions are two-fold:1. We provide a depth completion dataset of several thousand samples for afew dozen industrial objects, primarily challenging and specular objects. Thedata consists of RGBD measurements from a cost-effective device, embody-ing the real-world issues which hinder industrial adoption of depth cameras.We also introduce the fused 3D meshes, a by-product of our ground-truthgeneration algorithm.2. We present a depth-correction model for the industrial manufacturing do-main. By incorporating ideas from previous studies and adjacent domains,we show that our model achieves better accuracy than the closest competitor(see Sec. 4).In the next section (Sec. 2), we review related work and describe the researchlandscape. In Sec. 3, we expand on the methods we applied and how we craftedthem to attain the best result. In Sec. 4, we describe our results and providea discussion. We conclude the work (Sec. 5) by summarizing our contributionsand proposing future work directions.
In this section, we review previous work related to depth completion, all of whichmake use of the RGB information. We begin from monocular depth estimationfrom RGB, which clearly demonstrates how pertinent RGB is to depth. Wecontinue to sparse-to-dense depth completion, which has a lot in common withour task and demonstrates how much is achievable with only a sparse sample ofdepth information. We conclude with dense depth completion, where the depthdata is dense but has missing or erroneous regions.
An extensive body of work provided a multitude of solutions for monoculardepth estimation from RGB images. The premise is that RGB contains all theinformation needed to estimate depth. This notion is supported by the observa-tion that people can estimate 3D from RGB using prior knowledge on objectsshapes and sizes, shading and obstruction cues, etc. Furthermore, RGB camerasare considered remarkably cost effective, ubiquitous, and resilient to phenomenathat adversely affect active acquisition modalities such as depth-cameras.In [31], the authors employ Continuous Random Fields (CRF) and Convo-lutional Neural Network (CNN) in order to estimate depth. They over-segmentthe image into super pixels and predict a depth value for the super-pixel. TheirCRF objective is closely related to image matting [48]. In [9], the authors useda pair of coarse-to-fine deep networks. The first network is a CNN with a Fully
Yuri Feldman , Yoel Shapiro , Dotan Di Castro Convolutional (FC) head that predicts a coarse depth map. The input imageand the intermediate coarse outputs are fed into a Fully Convolutional Network(FCN) that yields a refined depth map. In [26], the authors use an encoder-decoder architecture [18] with a Residual Network (ResNet 50; [17]) backbone.They introduce a novel up-convolution block, a hybrid of upsampling and con-volution, which allows addressing several (4) scales concurrently at a given layer.Other multi-scale blocks can be seen in [45,11].Several works [12,54,10,19,14] developed unsupervised training pipelines, ex-ploiting either prior knowledge of camera motion in combination with multiple-view geometry or SfM techniques. Here, we applied multiple-geometry to gen-erate an explicit ground-truth and train in a supervised manner. Some of theseunsupervised-training works solved a subsidiary task, for example warping a leftstereo image to match the right image. Multi Task Learning (MTL) has beenused in top scoring works (e.g. [22]), and it is considered to be beneficial by itsown merit.
A related task is predicting a dense depth image, given its corresponding RGBimage and sparse samples of the target depth image, around 5% of the pixels.This problem setting originates in driving applications, for sensor fusion of RGBcameras and LiDAR scanners. This setting is akin to a regression problem, wherea piece wise continuous solution needs to satisfy the sampled depth points. Mostof the sparse-to-dense works test on the outdoor driving KITTI dataset [47] anda few test additionally on the indoor NYUv2 dataset [36], after simulating theLiDAR sampling behavior. A dominant architecture in this domain is Encoder-Decoder [34,33,50] and sometimes even a double Encoder-Decoder [13,25]. In[33], the Encoder-Decoder was embellished with U-connections [39], where theauthors introduced a smoothness loss term and self-supervised training tech-niques.Other groups employed Multi Task Learning (MTL), including posteriorprobability of the prediction [50] (which is analogues to confidence), semanticsegmentation [13], and image alignment [25]. The later [25] applied supervisedtraining for depth prediction and added the image-alignment loss in an unsuper-vised manner. In [13], five different tasks were provided: semantic segmentationfrom RGB, semantic segmentation from depth (D), semantic segmentation fromRGBD, depth prediction from RGB (monocular), and sparse-to-dense depthcompletion.
Depth cameras typically fail on reflective objects such as metals and mirrors,and deflective objects such as glass objects and clear plastics. Specular objectsresult in connected regions (blobs) of missing or erroneous depth values, whilethe remaining depth map retains its integrity. This comprehension is reflectedin auxiliary tasks, shared across the projects described in this subsection. The epth Completion with RGB Prior 5 first commonality is using RGB to partition the images; for example, in [8] themodel predicts semantic segmentation and in [53,41] occlusion boundaries. Themodel in [41] predicts additionally a binary mask for clear objects. The secondcommonality is using the RGB to produce an initial estimation for the depth,only with a surprising twist. Unlike monocular models, [8,53,41] predict surfacenormals, i.e. the gradient of the depth which in some sense can be thought ofas a ”spatial residual”. Both in [53] and [41], the input is RGBD, unlike [8]which only used RGB. They applied a two-stage solution: the first stage consistsof two or three parallel encoder-decoders applied only to the RGB data. Theoutputs are passed on to the second stage along with the depth data. The secondstage resorts to global optimization for predicting the depth map. The optimizerminimizes an objective that corresponds to the surface normals and the validregions of the depth input. In [41], the first stage provides an explicit mask forinvalid regions, while in [53] it seems to be implicitly embedded in the depthinput since the corrupted regions have outlying values. Notably, [41] provides anovel dataset for transparent objects with high quality depth maps. The datasetconsists of over 50k synthetic images (5 objects) and 500 real world images (4objects) collected in a painstaking process of replacing transparent objects withopaque replicas. In light of this, it is compelling to incorporate surface normalprediction.
We collected depth measurements of 45 industrial and 7 domestic objects, against3 backgrounds, with 4 or 5 camera viewpoints and 4 poses for each item; yieldingabout 3,500 data samples. A few samples from the dataset are displayed in Fig. 1.The camera rig was calibrated at the beginning of each data collection session,using an AprilTags matrix [24]. Our registration algorithm applies tag detection,extracts the depth at the tag centers, and then uses the a priori knowledge of thetag matrix to fit the calibration target with a 3D coordinate system and ensurethe quality of the depth data.The measured depth data was particularly poor for the metallic background,due to ambient lighting reflections. In this case, we employed a trick to improveour data, and collected the data at two lighting levels without moving the objects;RGB is always taken from the bright conditions while depth is taken accordingto usage context either from the dark conditions (in which case it suffers lessfrom ambient reflections or from the bright conditions).The ground-truth depth is generated by fusing the information from fourcamera view-points, using an implementation of TSDF [7] by Zeng et al. [52],followed by re-projection to each of the original camera view points. Under thisdata-collection scheme, our depth inputs preserve real-world issues, while theground-truth is not perfect but significantly better. We show a few data samplesin Fig. 1. This approach yields lower quality results than the data collection
Yuri Feldman , Yoel Shapiro , Dotan Di Castro Fig. 1:
Samples from our dataset. From left to right: Input RGB, input (measured)depth, fused depth (ground truth), reconstructed mesh (not directly used in training).White ”holes” in the measured depth are filled thanks to the depth fusion from multipleviews. Zero values in the depth maps correspond to missing pixels which the fusion didnot allow to fill. Clearly, the metal background examples (bottom rows) are the morechallenging ones, still - some of the missing values are filled.epth Completion with RGB Prior 7 solutions used in [41] but is much easier to apply. Despite our efforts, our ground-truth still has holes, so for practical reasons, the training loss is computed onlywhere the ground-truth is valid.
Fig. 2: Architectures examined in our workOur deep model is an assemblage of “best practices” for deep computer vision.We used an encoder-decoder architecture [18] (Fig. 2.1; our implementation isbased on [34]), which is a common choice for dense outputs [16] and a prevailingarchitecture for depth estimation (see Sec. 2).
Backbones
We experimented with ResNet [17] and VGG [44] backbones; As the latter gavelower accuracy, we exclude it from further discussion.
Improving the Sharpness of Predicted Depth
Output blurriness is a known issue of encoder-decoder architectures (it was alsoshown to be inherent to sparse depth completion by [32], however for a differentsetting of much sparser inputs). To mitigate that, we applied two modifications tothe baseline model: U-Net [38] -style connections between corresponding encoderand decoder layers of the same resolution (see Fig. 2.2; also commonly used indepth prediction, e.g. [12,3,9,33,29,55]) and replacing de-convolution layers withmax-unpooling for upsampling in the decoder [30].
Early Feedback and Auxiliary Outputs
To expedite training, we used early feedback [45], i.e. added loss terms for pre-dictions of down-sampled depth at internal decoder layers (Fig. 2.3). To enhancethe accuracy of the model, we explored Multi-Task-Learning (MTL) [40,22,8,4]
Yuri Feldman , Yoel Shapiro , Dotan Di Castro and added an auxiliary task, RGB prediction, or reconstruction (Fig. 2.4). It hasbeen reported in MTL literature that learning correlated tasks can give rise tosynergy and higher performance than learning each single task separately. Residual Prediction
The RealSense depth camera and its alternatives come with correction algo-rithms to improve their raw depth acquisition. This post-processing step usuallygives gross but reasonable estimations in our (industrial) scenarios of interest.Acknowledging these capabilities, we also experimented with restating the prob-lem as residual prediction. Formally, denoting input RGB frame as I and post-processed depth as ˜ d , we fit our model f to predict a correction (”delta”) to theinput depth, so that final depth prediction ˆ d is expressed as:ˆ d = ˜ d + f ( I, ˜ d ) . (1)In some sense residual-prediction is an easier task conceptually and numerically,since the output lies in a smaller dynamic range. Residual learning is emergingas a promising approach in control and reinforcement learning [51], where it isused to learn corrections for a parametric control policy. Validity Mask
Residual prediction is supported by appending an additional input channel,quantifying the pixel-wise confidence for each depth value. In our case, we foundit practical to use binary values, valid-invalid, but the confidence values couldbe continuous in the general case.The Encoder-Decoder with U-Net connections (Fig. 2.2) was the basis for amodel search and experimentation with different combinations of the other fea-tures mentioned in the above subsections. We describe these experiments inSec. 4.2.
We trained all models on a mixture of our industrial data and on the NYUv2indoor data [36] at a 1:1 ratio, while the validation set was slanted towardsindustrial data (85% to 15%). These data domains are rather different, so we(a) scaled the NYUv2 depth values by a constant factor to match our workingrange ( ∼ epth Completion with RGB Prior 9 Objective Function
The main loss in our work is prediction accuracy, i.e. the difference betweenthe predicted depth and ground truth. Another useful loss was on the magnitudeof the prediction’s Laplacian, encouraging smoothness (as developed in [32,33]).We also experimented with losses on the difference between the gradients of theprediction and the ground-truth but this did not prove out to be beneficial.The general form of the loss we use is: L = w L depth + w L early + w L rgb , (2)where L depth comprises penalize difference in depth prediction w.r.t. to groundtruth, L early penalizes difference of intermediate depth outputs w.r.t. groundtruth depth, and L rgb difference in reconstructed RGB (auxiliary task, Sec. 3.2)w.r.t. input frame. { w i } are constant weights chosen via hyperparameter search.Specifically, the depth term is defined as: L depth ( d, ˆ d ) = w p l p ( d, ˆ d ) + w g l g ( ∇ d, ∇ ˆ d ) + w s l s ( ∆ ˆ d ) , (3)where d , ˆ d are the ground truth and predicted depth respectively, and we denoteby ∇ d the image of gradients and by ∆d the Laplacian, i.e. ∆d = d xx + d yy , perpixel. The weights w p , w g , w s corresponding to the prediction, gradient (normals)and smoothness terms, respectively, are again chosen via hyperparameter search.The early feedback and auxiliary task terms are defined as: L early = l e ( ˆ d early , d ) , and L rgb = l rgb ( ˆ I, I ) , (4)for ˆ d early - intermediate depth outputs, and ˆ I the predicted RGB image (asbefore, d is the ground truth depth and I the input RGB frame).Different choices are possible and discussed in the literature for the distancemetrics ( l { p,g,s,d,rgb } ). After experimentation (see next clause), we used L totrain models for comparison. Hyperparameter Search
We conducted a hyperparameter search to determine Learning-Rate (LR),LR decay parameters, weight decay, etc. Our hyper parameter search includeddifferent types of distance metrics and their relative weights: L , L , Huber,Adaptive Huber, and RHuber [20]. We found negligible differences between themetrics, so from here on we will only mention results for L . In this section, we compare our model to that of Ma and Karaman [34] (Sec. 4.1)using standard metrics (see Table 1). Subsequently we present the results of anablation study (Sec. 4.2) outlined in Sec. 3.2. All compared models are trained , Yoel Shapiro , Dotan Di Castro Metric Definition Full NameRMSE (cid:113) N (cid:80) i ( d i − ˆ d i ) Root Mean Square ErrorMAE N (cid:80) i | d i − ˆ d i | Mean Absolute ErrorRel . N (cid:80) i | d i − ˆ d i | /d i Relative Absolute Error
Table 1:
Evaluation metrics. The sums exclude invalid ground truth pixels. We use d and ˆ d to denote the target (ground truth) and predicted depth, respectively. The index i runs over N , the total number of valid (non-zero) ground truth pixels. from scratch on the same training set, which is a mixture of random samples ata 1:1 ratio from the NYU Depth v2 (NYUDv2) [36] and our data. The modelsare compared on a validation set which is a random mixture preferring our data(85%) over NYUDv2 (15%). We comapre our model against Ma and Karaman [34] in Table 2, on the valida-tion set. The 1st row lists the total acquisition error, i.e raw input vs. ground-truth. The 2nd row shows prediction errors of Ma and Karaman’s model aftertraining from scratch on our data, using their best hyperparameters. For faircomparison, we performed a hyperparameter search on our data for their model,and show the results on the 3rd row; this was sufficient to reduce their errors by ∼ To assess the contribution of each modification listed in Sec. 3.2, we reporttraining results from two sets of experiments. In the first set of experiments(“incremetal”; Fig. 4, left panel) we add one modifications to a baseline modelconsisting of Ma and Karaman’s encoder-decoder with added (“U-Net”) skipconnections. In the second set of experiments (“decremental”; Fig. 4, right panel)we remove one modification at a time from our best-performing model, withthe latter (best-performing model) taken as the baseline for comparison. Dueto the high co-dependence of a subset of features we introduces a added twotrials with more than one modification. Each of the models in the comparison istrained from scratch with the same training hyperparameters. Each experimentcorresponds to a feature addressed in Sec. 3.2 or in Sec. 3.3. Further details oneach experiment are given in Table 1.The incremental experiments (Fig. 4, left panel) show that a noted improve-ment was obtained w.r.t. the baseline model by adding a validity mask input epth Completion with RGB Prior 11
Model RMSE [m] MAE [m] Rel.Input 0.226 0.226 0.119Ma & Karaman 0.052 0.039 0.092Ma & Karaman* 0.016 0.011 0.025Ours 0.012 0.004 0.009
Table 2:
Validation errors. 1st row: errors in raw inputs w.r.t. ground truth. 2nd row:Ma and Karaman [34] trained with best hyperparameters from the original paper, and(3rd row) trained with results of our hyperparameter search. 4th row: our best model(see Sec. 3.2).
Fig. 3: Progress of training (left) and validation (right) MAE scores for our bestmodel compared to [34], denoted as Ma & Karaman 18, after re-training onour data with their original hyper parameters. For fairness, we performed ahyper parameter search, denoted with an asterisk, which improved their model’sperformance.Fig. 4: Training scores (MAE) for our ablation study.
Left: “incremental” ex-periments. +
UNet is the baseline model, i.e. [34] with the added U-connections.In each trial, a single feature is added, i.e. feature significance corresponds to a reduction in error.
Right: “decremental” experiments. the baseline is our bestmodel with all features included. For each trial, a single feature is removed,hence feature significance corresponds to an increase in error. See further clar-ifications in Table 3, in regard to the modifications from Sec. 3.2 and Sec. 3.3.For discussion and analysis see Sec. 4.2. , Yoel Shapiro , Dotan Di Castro Experiment Name Incremental Decremental+Unet [34] with added U-connections –criterion Criterion from Sec. 3.3 L between predicted depth and GTdol Use early (depth) feedback No early feedbackmask Use binary validity mask No validity mask inputunpool Use max-unpooling Use deconvolution layersdelta Residual prediction, input interpolation Depth predictioninterp – Predict depth, with input interpolationdelta-interp – Predict depth, no input interpolationdelta-interp-mask Residual, interpolation, validity mask – Table 3:
Ablation experiments, correspondence to Sec. 3.2. Experiment name is asappears in Fig. 4. Incremental column lists corresponding modification(s) to baselinefor incremental experiments (Fig. 4 left). Decremental column lists corresponding mod-ifications to baseline for decremental experiments (Fig. 4 right). channel. Not surprisingly, the most significant improvement was obtained by in-terpolating depth measurements input to the model and predicting the residual,while the rest of the modifications do not seem to have a significant effect on theirown. Residual prediction over interpolated input and validity mask accounts formost of the improvement w.r.t. the baseline. The same behavior can be seen inthe decremental experiments (Fig. 4, right panel), where performance was af-fected by removing residual prediction, i.e. training to directly predict depth in-stead. Interestingly, removing the residual prediction reduced accuracy (“delta”,“delta-interp” experiments) even when model was still input with interpolateddepth (“delta”). However, predicting residual depth from non-interpolated in-puts caused an even more severe degradation (“interp” experiment).
In this paper, we provide a novel industrial objects dataset of RGBD imagesand a method to complete missing depth points due to reflections of specularobjects. The main limitations of the dataset are: – Variability: 45 objects, 3 backgrounds. – Size: ∼ ∼ – Fidelity: the ground truth depth suffers from holes.A compelling avenue for future work is automating the data collection usinga robot mounted camera. This will make it easier to acquire a larger variety ofobjects, increase the number of view-points, and open interesting secondary re-search questions. With a camera-in-hand, one can generate the ground-truth, ei-ther by applying multiple-camera geometry on the robot’s poses or by Structure-from-Motion techniques. The efficiency of this processes could benefit from ap- epth Completion with RGB Prior 13 plying
Active Perception principles [2,5,1]. Moreover, we expect active percep-tion to improve data fidelity, as [49] has demonstrated in the domestic domain,since different view angles modify the perceived reflections and may completelyeliminate them. This will also allow us to do surface normals prediction, whichcurrently is problematic with the data we have at hand. Additionally, the robotwill manipulate the objects and randomize the scenes to expand variability andboost generalization capability of trained models.One alternative to this automation is to generate photo-realistic syntheticdata. This has been done for transparent consumer products [41] and couldbe attempted for industrial applications, largely thanks to the availability oflarge industrial CAD datasets [23]. The flexibility of synthetic frameworks isappealing but it must be noted that they require large engineering and artistic-design efforts to render a large variety of scenes and bridge the synthetic-realdomain gap. In our case. we have two potential domain-gaps which are slightlydifferent from each other: – Photo-realism: the RGB images need to appear real. – Depth issues: it is straightforward to produce complete synthetic depth forground-truth, but it is more challenging to imitate the device-specific issuesand artifacts that arise in real-world acquisition systems.A practical approach might be to mix synthetic and real data, as well as applyingdomain-adaptation techniques [6].Notably, the model that we developed performed better than related counter-parts, owing to the synergy between the best-practices assembled from previousanalyses. The paramount modifications are the additional U-connections andrephrasing the problem in residual terms, that is, predicting a small additivecorrection on top of a reasonable initial estimation. It might be possible to drivethe model performance further by tracking the latest improvements in deepmodels, e.g. higher capacity ResNet blocks [11]. Prior research [8,41] achievedcompelling progress by taking surface normals into consideration. Consequently,an interesting continuation of the current research would be to add an auxil-iary task of predicting surface-normals from RGB. We note that in this workwe refrained from doing so due to lack of data, namely the ground-truth beingrelatively noisy and scarce.Depth completion is a rapidly evolving research topic, but it still lacks clearindustrial requirements or benchmarks to test against. As a by-pass, we suggestthat it would be interesting to gauge the impact of this post-processing stepon physical manufacturing tasks, such as object grasping and manipulations. Itcould be exciting to demonstrate the practical implications of this seeminglytepid depth-completion task. We believe that it is a pivotal enabler for Industry4.0, with vast consequences to global economy. , Yoel Shapiro , Dotan Di Castro References
1. Bajcsy, R., Aloimonos, Y., Tsotsos, J.K.: Revisiting active perception. AutonomousRobots (2), 177–196 (2018)2. Bajcsy, R., Campos, M.: Active and exploratory perception. CVGIP: Image Un-derstanding (1), 31–40 (1992)3. Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., Davison, A.J.: Codeslam-learning a compact, optimisable representation for dense visual slam. In: Proceed-ings of the IEEE conference on computer vision and pattern recognition. pp. 2560–2568 (2018)4. Cadena, C., Dick, A.R., Reid, I.D.: Multi-modal auto-encoders as joint estimatorsfor robotics scene understanding. In: Robotics: Science and Systems. vol. 7 (2016)5. Chen, S., Li, Y., Kwok, N.M.: Active vision in robotic systems: A survey of recentdevelopments. The International Journal of Robotics Research (11), 1343–1377(2011)6. Csurka, G.: Domain Adaptation in Computer Vision Applications. Springer Pub-lishing Company, Incorporated, 1st edn. (2017)7. Curless, B., Levoy, M.: A volumetric method for building complex models fromrange images. In: Proceedings of the 23rd annual conference on Computer graphicsand interactive techniques. pp. 303–312 (1996)8. Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels witha common multi-scale convolutional architecture. In: Proceedings of the IEEE in-ternational conference on computer vision. pp. 2650–2658 (2015)9. Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image usinga multi-scale deep network. In: Advances in neural information processing systems.pp. 2366–2374 (2014)10. Faugeras, O.D., Lustman, F.: Motion and structure from motion in a piecewiseplanar environment. International Journal of Pattern Recognition and ArtificialIntelligence (03), 485–508 (1988)11. Gao, S., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.H.: Res2net: Anew multi-scale backbone architecture. IEEE Transactions on Pattern Analysis andMachine Intelligence p. 11 (2019). https://doi.org/10.1109/tpami.2019.2938758, http://dx.doi.org/10.1109/TPAMI.2019.2938758
12. Garg, R., BG, V.K., Carneiro, G., Reid, I.: Unsupervised cnn for single view depthestimation: Geometry to the rescue. In: European Conference on Computer Vision.pp. 740–756. Springer (2016)13. Giannone, G., Chidlovskii, B.: Learning common representation from rgb anddepth images. In: Proceedings of the IEEE Conference on Computer Vision andPattern Recognition Workshops. pp. 0–0 (2019)14. Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth es-timation with left-right consistency. In: Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition. pp. 270–279 (2017)15. Handa, A., Whelan, T., McDonald, J., Davison, A.J.: A benchmark for rgb-d visualodometry, 3d reconstruction and slam. In: 2014 IEEE international conference onRobotics and automation (ICRA). pp. 1524–1531. IEEE (2014)16. He, K., Gkioxari, G., Dollr, P., Girshick, R.: Mask r-cnn (2017)17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In:Proceedings of the IEEE conference on computer vision and pattern recognition.pp. 770–778 (2016)epth Completion with RGB Prior 1518. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionalityof data with neural networks. Science (5786), 504–507 (2006).https://doi.org/10.1126/science.1127647, https://science.sciencemag.org/content/313/5786/504
19. Huang, T.S., Netravali, A.N.: Motion and structure from feature correspondences:A review. In: Advances In Image Processing And Understanding: A Festschrift forThomas S Huang, pp. 331–347. World Scientific (2002)20. Irie, G., Kawanishi, T., Kashino, K.: Robust learning for deep monocular depthestimation. In: 2019 IEEE International Conference on Image Processing (ICIP).pp. 964–968. IEEE (2019)21. Jrgensen, E., Zach, C., Kahl, F.: Monocular 3d object detection and box fittingtrained end-to-end using intersection-over-union loss (2019)22. Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weighlosses for scene geometry and semantics. In: Proceedings of the IEEE conferenceon computer vision and pattern recognition. pp. 7482–7491 (2018)23. Koch, S., Matveev, A., Jiang, Z., Williams, F., Artemov, A., Burnaev, E., Alexa,M., Zorin, D., Panozzo, D.: ABC: A big CAD model dataset for geometric deeplearning. CoRR abs/1812.06216 (2018), http://arxiv.org/abs/1812.06216
24. Krogius, M., Haggenmiller, A., Olson, E.: Flexible layouts for fiducial tags. In:Proceedings of the IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS) (2019)25. Kuznietsov, Y., Stuckler, J., Leibe, B.: Semi-supervised deep learning for monoc-ular depth map prediction. In: Proceedings of the IEEE conference on computervision and pattern recognition. pp. 6647–6655 (2017)26. Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depthprediction with fully convolutional residual networks. In: 2016 Fourth internationalconference on 3D vision (3DV). pp. 239–248. IEEE (2016)27. Lazaros, N., Sirakoulis, G.C., Gasteratos, A.: Review of stereo vision algorithms:from software to hardware. International Journal of Optomechatronics (4), 435–462 (2008)28. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature (7553), 436–444(2015)29. Li, J., Klein, R., Yao, A.: A two-streamed network for estimating fine-scaled depthmaps from single rgb images. In: Proceedings of the IEEE International Conferenceon Computer Vision. pp. 3372–3380 (2017)30. Li, Y., Liu, M.Y., Li, X., Yang, M.H., Kautz, J.: A closed-form solution to photo-realistic image stylization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y.(eds.) Computer Vision – ECCV 2018. pp. 468–483. Springer International Pub-lishing, Cham (2018)31. Liu, F., Shen, C., Lin, G.: Deep convolutional neural fields for depth estimationfrom a single image. In: Proceedings of the IEEE conference on computer visionand pattern recognition. pp. 5162–5170 (2015)32. Ma, F., Carlone, L., Ayaz, U., Karaman, S.: Sparse sensing for resource-constraineddepth reconstruction. In: 2016 IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS). pp. 96–103. IEEE (2016)33. Ma, F., Cavalheiro, G.V., Karaman, S.: Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera. In: 2019 Inter-national Conference on Robotics and Automation (ICRA). pp. 3288–3295. IEEE(2019)6 Yuri Feldman , Yoel Shapiro , Dotan Di Castro
34. Ma, F., Karaman, S.: Sparse-to-dense: Depth prediction from sparse depth sam-ples and a single image. In: 2018 IEEE International Conference on Robotics andAutomation (ICRA). pp. 1–8. IEEE (2018)35. Mur-Artal, R., Tard´os, J.D.: Orb-slam2: An open-source slam system for monoc-ular, stereo, and rgb-d cameras. IEEE Transactions on Robotics (5), 1255–1262(2017)36. Nathan Silberman, Derek Hoiem, P.K., Fergus, R.: Indoor segmentation and sup-port inference from rgbd images. In: ECCV (2012)37. ten Pas, A., Platt, R.: Using geometry to detect grasp poses in 3d point clouds.In: Robotics Research, pp. 307–324. Springer (2018)38. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi-cal image segmentation. In: International Conference on Medical image computingand computer-assisted intervention. pp. 234–241. Springer (2015)39. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed-ical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F.(eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI2015. pp. 234–241. Springer International Publishing, Cham (2015)40. Ruder, S.: An overview of multi-task learning in deep neural networks (2017)41. Sajjan, S.S., Moore, M., Pan, M., Nagaraja, G., Lee, J., Zeng, A., Song, S.: Clear-grasp: 3d shape estimation of transparent objects for manipulation (2019)42. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereocorrespondence algorithms. International journal of computer vision (1-3), 7–42(2002)43. Schoettler, G., Nair, A., Luo, J., Bahl, S., Ojea, J.A., Solowjow, E., Levine, S.:Deep reinforcement learning for industrial insertion tasks with visual inputs andnatural rewards. arXiv preprint arXiv:1906.05841 (2019)44. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scaleimage recognition. arXiv preprint arXiv:1409.1556 (2014)45. Szegedy, C., Wei Liu, Yangqing Jia, Sermanet, P., Reed, S., Anguelov, D., Erhan,D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: 2015 IEEEConference on Computer Vision and Pattern Recognition (CVPR). pp. 1–9 (June2015). https://doi.org/10.1109/CVPR.2015.729859446. Tippetts, B., Lee, D.J., Lillywhite, K., Archibald, J.: Review of stereo vision al-gorithms and their suitability for resource-limited systems. Journal of Real-TimeImage Processing (1), 5–25 (2016)47. Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., Geiger, A.: Sparsityinvariant cnns. In: International Conference on 3D Vision (3DV) (2017)48. Wang, J., Cohen, M.: Image and video matting: A survey. Founda-tions and Trends in Computer Graphics and Vision3