CLOI: An Automated Benchmark Framework For Generating Geometric Digital Twins Of Industrial Facilities
CCLOI: AN AUTOMATED BENCHMARK FRAMEWORK FORGENERATING GEOMETRIC DIGITAL TWINS OF INDUSTRIALFACILITIES
Eva Agapaki and Ioannis Brilakis Senior Software Developer, PTC Inc.,U.S.A. Email: [email protected] Laing O’Rourke Reader, Department of Engineering, University of Cambridge, CB2 1PZ, U.K.
ABSTRACT
This paper devises, implements and benchmarks a novel framework, named
CLOI , that canaccurately generate individual labelled point clusters of the most important shapes of existingindustrial facilities with minimal manual effort in a generic point-level format.
CLOI employsa combination of deep learning and geometric methods to segment the points into classes andindividual instances. The current geometric digital twin generation from point cloud data incommercial software is a tedious, manual process. Experiments with our
CLOI framework revealthat the method can reliably segment complex and incomplete point clouds of industrial facilities,yielding 82% class segmentation accuracy. Compared to the current state-of-practice, the proposedframework can realize estimated time-savings of 30% on average.
CLOI is the first framework ofits kind to have achieved geometric digital twinning for the most important objects of industrialfactories. It provides the foundation for further research on the generation of semantically enricheddigital twins of the built environment.
INTRODUCTION
The industrial sector and especially the oil and gas is an industry with the highest potentialgrowth in terms of worker productivity and economic value of the sector within the next couple1 Agapaki, January 6, 2021 a r X i v : . [ c s . C Y ] J a n f years. The Global Infrastructure Initiative forecasts that heavy industrial buildings and the oiland gas sector are among the construction sectors with the highest potential for investments withan average Compound Annual Growth Rate (CAGR) of 3.4% (McKinsey Global Institute 2015).Therefore, it is crucial that the industrial sector is properly maintained given the high value of theindustrial assets for our economies.Maintenance, safety management and retrofitting are vital operations in the life-cycle of existingindustrial facilities. Corrective or poor maintenance incurs unplanned downtime costs, whichare estimated to be $50 billion per year (National Institute of Standards and Technology 2018).The primary reasons for these incidents are ineffective and inefficient facility management andpoor mapping of the existing industrial equipment. Faster digital industrial documentation isurgently required to reduce unscheduled equipment downtimes and boost the Overall EquipmentEffectiveness (OEE) of a factory, which is currently estimated to be between 5 to 20% (PECI 1999).There are limits on the acceptable shut down duration that will not impede production. Theselimits cannot be violated without incurring extra costs. This is why adoption of Digital Twins (DTs)is crucial for the industrial sector. The greatest value of using DTs is that they are projected to savesubstantial costs for facility managers by automating the preventive maintenance process whichwill enable accurate positioning of each industrial object and timely maintenance decisions. Forexample, DTs can help to keep records of the inventory, processes, historical data and additionalequipment. This allows owners to identify inefficiencies and ways to address them. Studies showthat the wider adoption of DTs will unlock 15-25% savings to the global infrastructure market by2025 (Barbosa et al. 2017; Gerbert et al. 2016).The concept of DTs is not new. NASA first generated the term “twin” when building two identicalspace vehicles for its Apollo program (Glaessgen and Stargel 2012). The modern terminology of a“digital twin” has been attributed to Dr Michael Grieves as part of his research in Product LifecycleManagement (PLM) (Grieves 2014). Reports based on the digitization index have shown that theoil and gas industry has been highly digitised as compared to the construction industry, which is inthe bottom of the list (Agarwal et al. 2016). Despite the high value DTs have in the industrial sector,2 Agapaki, January 6, 2021et, industrial facilities do not have DTs for existing industrial factories due to the high perceivedcost which outweighs their benefits (West and Blackburn 2017).The generation of a geometric Digital Twin (gDT) is the core and first step in the DT generation(Borrmann and Berkhahn 2018). The inputs for the generation of gDTs are usually point cloudsscanned with Terrestrial Laser Scanners (TLS) (Marshall 2016). 90% of the gDT generation costis spent on converting point cloud data to 3D models due to the sheer number of objects of eachindustrial facility (Fumarola and Poelman 2011; Hullo et al. 2015). Hence, cost reduction is onlypossible by automating the generation of gDTs. However, automatically classifying millions ofobjects is a very hard classification problem due to the very large number of classes and the strongsimilarities between them. We provided in our previous work (Agapaki et al. 2018) a comprehensivetechnical assessment and viable evaluation of existing state-of-the-art software tools available. Inthe following paragraphs, we summarize the state-of-practice based on this evaluation. State-of-practice
In our previous work (Agapaki et al. 2018), we identified the most frequent and laborious tomodel object types, which are cylindrical objects (straight pipes, electrical conduit and circularhollow sections), valves, elbows, I-beams, angles, channels and flanges. Cylinders require 80%of the total modelling time of the ten most important object types in EdgeWise (ClearEdge 2019)and represent 45.5% of the total number of objects in an industrial plant on average. EdgeWisewas selected compared to other state-of-the-art software, because it is the only commerciallyavailable tool that attempts to automatically extract cylinders from the point cloud of an industrialplant without significant user assistance. EdgeWise has significantly accelerated 3D modelling ofindustrial plants according to the findings discussed above. However, it has some limitations, whichcan be summarized as follows:1. Structural elements (I-beams, angles, channels) should be manually modelled and theirlocation in the point cloud is roughly defined based on the modeler’s discretion.2. Segmentation of cylinders has been partially achieved with detection rates being 75% recall3 Agapaki, January 6, 2021nd 62% precision on average (Agapaki et al. 2018). The same metrics for cylindricalobjects labelled as pipes are 58% and 47% respectively. It is also important to note thatEdgeWise erroneously includes points that do not belong to a geometric shape. This isdue to fitting errors, which occur since primitive shapes are perfect shapes, whereas thescanned, physical objects are imperfect (e.g. a cylindrical pipe may be bent).3. EdgeWise is not designed to output geometric shapes in an open and generic format.As such, modelers cannot easily exchange data between different operational-phase gDTplatforms due to data inconsistency between them.Therefore, the evaluation of EdgeWise uncovered (a) the substantial performance of this softwarein detecting cylinders with its pitfalls, (b) the inability of the software to (i) further classify cylindersinto conduit or pipes or CHSs and (ii) detect and further classify I-beams, channels, flanges, valvesand angles in spite of their high frequency in an industrial facility.This performance of EdgeWise has substantial room for improvement and this paper intendsto address the above-mentioned limitations in order to automatically generate gDTs of industrialfacilities and assist the tedious current practice. We propose a geometric twinning frameworkfor existing industrial facilities and bench-mark it with the current state of practice. In the fol-lowing section, the state-of-the-art research methods related to the above-mentioned limitationsare presented. We then outline the framework in the proposed solution, which is followed by theexperiments and results. The conclusions are then derived in the last section.
BACKGROUND
There are two distinct gDT generation strategies investigated in the literature as presented inFigure 1. The first one ( S1 ) involves two steps: (a) primitive industrial shape detection and (b)fitting. The second one ( S2 ) has three steps: (a) class segmentation, (b) instance segmentation and(c) fitting. Class segmentation describes the procedure of partitioning the TLS point cloud datasetto clusters of points with class labels assigned per point (such as cylinder, elbow, I-beam, valve,flange, angle and channel) (Li et al. 2019). Instance segmentation assigns a label per point based4 Agapaki, January 6, 2021n the individual object that the point belongs to. For reasons explained in (Agapaki and Brilakis2020a), the S2 DT strategy was selected in this paper. Therefore, the literature review is elaboratingon: (a) S2 class segmentation methods and (b) S2 instance segmentation methods. Fitting methodsare not discussed, since they are out of scope of this paper. Class segmentation
Class segmentation methods applied on industrial shapes have been widely investigated. Wecategorize them into three groups: (a) attribute based methods, (b) machine learning and (c) deeplearning methods. A comprehensive review of class segmentation methods based on hand-craftedfeatures is provided by (Agapaki and Nahangi 2020) and some of the most important methods areexplained in the paragraphs that follow.
Attribute-based
Attribute-based methods are bottom-up approaches that cluster base elementsto generate complex systems in successive higher levels until a top-level system is formed (e.g.bridge, facility) (Borenstein and Ullman 2008). These methods cluster points with similar attributesinto subsets. An 𝑛 -dimensional attribute space is created to extract the attributes in the parameterdomain, where 𝑛 represents the estimated number of attributes. These methods process a pointcloud starting from point-wise features and generate higher-level features, such as surface normals(Rusu et al. 2009; Sampath and Shan 2010), mesh (Marton et al. 2009) or patches (Vosselman 2009;Zhang et al. 2015). The estimated attributes are clustered and extracted in the parameter domain.Attribute based methods can be divided in two broad categories based on the shape descriptorsthey use: global or local. Local descriptors allow for partial matching of features, therefore arepreferred for occluded scenes compared to global descriptors. Global descriptors describe the sceneas a whole. For instance, local descriptors of a cylinder are curvature and normal vectors, whereasglobal descriptors are its length and diameter, which correspond to properties for the whole cylinder.Curvature has been extensively used as a local feature for industrial piping segmentation (Dimitrovand Golparvar-Fard 2015; Perez-Perez et al. 2016). However, substantial manual segmentation isneeded to pre-process the input TLS data. 5 Agapaki, January 6, 2021 achine learning We review one of the most widely used parametric supervised machine learn-ing methods in the class segmentation literature, which is
Support Vector Machines (SVMs) .(Li et al. 2016) used SVMs on TLS urban point clouds and then a multi-classification graph-cutalgorithm to optimize the initial segmentation result. Similarly, (Zhang et al. 2013) used a region-growing algorithm before applying an SVM for urban point cloud segmentation. (Huang and You2013) and (Armeni et al. 2016) use SVM classifiers with local features to segment cylindrical andindoor space objects. The use of SVMs in these approaches though has inherently two limitations:(1) SVM is not designed for imbalanced classes. Weights inversely proportional to the class fre-quency are applied to the imbalanced classes. Industrial facility datasets are highly imbalancedwith respect to the most important object types they have, since their distribution follows the Zipf’slaw as proved in (Agapaki et al. 2018). For this reason, the application of SVMs on TLS industrialfacility data is not preferred, unless one oversamples the object types that appear less frequently.(2) the success of SVMs depends on the selection of hand-crafted features, the type of kernelfunction and the parameters to the kernel function. Improper selection of features can result inmisclassifications, whereas application of different kernel functions for a dataset gives differentresults.
3D Class Segmentation Deep Learning methods
CNNs have been widely used for a variety oftasks in image segmentation (Krizhevsky et al. 2012; LeCun et al. 2008; Taha and Hanbury 2015;Pang et al. 2012; Wang et al. 2018a; Teichmann et al. 2018). We group these methods in threemain categories as suggested by (Wang et al. 2019): (
DLa ) view-based (Su et al. 2015; Kalogerakiset al. 2017; Wei et al. 2016), (
DLb ) volumetric (Maturana and Scherer 2015; Wu et al. 2015; Zhouand Tuzel 2017; Klokov and Lempitsky 2017; Tatarchenko et al. 2017) and (
DLc ) geometric deeplearning methods (Qi et al. 2017b; Qi et al. 2017a; Wang et al. 2019).Geometric deep learning methods are chosen as the most suitable for class segmentation asexplained by (Agapaki and Brilakis 2020a), since they address the following challenges that TLSindustrial point cloud processing has: (1) irregularity in the TLS data structure, (2) TLS data spar-6 Agapaki, January 6, 2021ity, noise, presence of outliers and occlusions as well as density variations especially in industrialsettings and (3) differences in industrial object scales, rotation and translation variant objects aswell as geometric similarities between objects of the same class. PointNETs (Qi et al. 2017b; Qiet al. 2017a) and their derivatives (Wang et al. 2019; Wang et al. 2018b; Landrieu and Simonovsky2018; Thomas et al. 2019) have solved these challenges by applying permutation invariant functionsas well as local 3D filters in their network architectures. PointNET networks concatenate globaland local features into point feature vectors based on which class labels are predicted. PointNET++improves the PointNET architecture by adding local neighbourhood geometric features.
Instance Segmentation
3D instance segmentation is based on 3D geometric class segmentation networks. Thesemethods can be grouped into shape-based (top-down) or shape-free (bottom-up). Our readerscan refer to (Agapaki and Brilakis 2020b) for a comprehensive literature review of each of thesemethods. We elaborate on the state-of-the-art literature on shape-free methods, since these aremore suitable for the generation of gDTs from TLS industrial data (Agapaki and Brilakis 2020b).Shape-free methods are based on deep learning networks, which aggregate features per pointand output instance labels per point given a similarity matrix between pairs of points (Wang et al.2018b; Wang et al. 2019) or embedding another network measuring point-wise distances (Phamet al. 2019). PointNET (Qi et al. 2017b) and PointNET++ (Qi et al. 2017a) is the backbonenetwork for these methods, meaning that they achieve class segmentation as well. Although thesenetworks take into consideration the local neighbourhoods of points, they cannot explicitly definethe boundaries of complex industrial shapes. Object boundaries can be taken into account byconsidering the class and instance segmentation labels. The readers can refer to (Xie et al. 2019)for a detailed review of all the instance segmentation methods.
PROPOSED SOLUTION
We target to solve the problem of the generation of gDTs of existing industrial facilities withrespect to cost and modelling time reduction. The main objective of this paper is to develop abenchmark framework as the foundation for future research.7 Agapaki, January 6, 2021 verview
The proposed framework consists of two major parts. Specifically, these parts are (1) classsegmentation and (2) instance level segmentation that intend to answer the research questions asoutlined in the Background section and aim to outperform the existing state of practice and researchin the industrial modelling space.We propose a novel hybrid framework which develops deep learning networks and leveragestheir detected outputs with industrial engineering knowledge, in order to automatically extractlabelled point clusters corresponding to industrial shape components without generating surfaceprimitives ( class point clusters ) and then to efficiently detect individual industrial shapes from thelabelled point clusters ( instance point clusters ).Real-world industrial environments are more challenging than buildings that have been exten-sively studied and scanned in previous research efforts as mentioned in the Background section.Industrial components do not comply with a universal colour scheme, rather colours depend oneach manufacturer’s specifications (Agapaki and Brilakis 2020a). Industrial spaces are typicallylarge and unstructured with shapes that may span across their whole length/width and they areheterogeneous spaces where there are usually no direct contextual rules in separate systems (pip-ing, structural, electrical) and only the components that belong to the same system are internallyconnected with strong context. For example, the relative location of a cylinder with respect to anI-beam in a factory does not imply that the locations of these objects should comply to specificspatial rules. We propose a 3D-slicing facility window method, CLOI-NET-class based networksand CLOI-Instance graph-connectivity algorithms to tackle these challenges. The 3D windows areused to segment the TLS dataset in non-overlapping parts, so that a portion of these windows willbe used for training. These windows should be non-overlapping, so that the training and test set aredisjoint. These algorithms are the core foundation of the methods built upon them to enhance thesegmentation and detection results. The proposed algorithms can deal with the challenges outlinedabove and can accurately detect the majority of
CLOI industrial shapes.Most of the
CLOI shapes match 1 to 1 to a component class, (i.e. the shape is unique to this8 Agapaki, January 6, 2021omponent), but for cylinders the shape is not unique. So the method focuses on segmenting the
CLOI shapes , and by default, equivalently segments their component classes except for cylinders.Segmentation of the subcategories of cylindrical shapes (i.e. pipes, circular hollow sections,handrails, electrical conduit) is beyond the scope of this research. The proposed framework isnot applicable for connections of steel members (welding and bolting). The proposed algorithmsaddress scale variance (The algorithms are scale invariant, since we feed them with objects atdifferent scales (from a few centimeters to some meters.) of industrial objects and intra-classvariations. For instance, there are many types of valves as expressed above, which are grouped inone class and the proposed algorithms should be able to segment valves of all the above mentionedcategories.We illustrate the developed hybrid framework in Figure 2. It consists of two major processes:
Process 1 , class segmentation of
CLOI industrial point clusters, and
Process 2 , instance segmen-tation of
CLOI industrial shapes from point clusters.The proposed framework starts with a raw, laser-scanned, PCD of an existing industrial facility(data format: points in .pcd, .txt, .las, .xyz). External noise such as vegetation, adjacent buildings isremoved using commercial software as explained in (Agapaki and Brilakis 2020a). The industrialPCD contains
CLOI -shapes and any other industrial shapes inside a factory (data format: pointsin .pcd, .txt, .las, .xyz). The first step of the framework is to automatically split the PCD facility in3D windows and the 3D windows in “3D blocks”. Then, the 3D blocks are aligned in the globalcoordinate system. As such, the outputs of this step are 3D block PCDs (data format: points in .pcd,.txt, .las or .xyz). Then, we manually annotate industrial facilities to generate a benchmark datasetand the outputs of this step are class and instance segmentation labels and points. It is important tonote that this is an essential offline step needed for training purposes and serves as the ground truthfor the validation of the framework.Next, we propose a three-step class segmentation method (Process 1) to segment the
CLOI point clusters from the 3D blocks. The final outputs of this process are seven industrial shapes,namely cylinders, elbows, channels, I-beams, angles, flanges and valves, in the form of labelled9 Agapaki, January 6, 2021oint clusters (data format: points in .pcd, .txt, .las, .xyz). Then, we suggest an optimal manualannotation (if the users select it) to remove the erroneous point clusters maintained from Process1 followed by proposing an efficient instance segmentation method (Process 2) through which theseven
CLOI classes (in point cluster format) can be directly segmented to individual shapes. Thefinal outputs of this process are point data corresponding to the points, class and instance labels perpoint. We elaborate on each process in the following sections.We validate Process 1 on the
CLOI benchmark dataset (Agapaki et al. 2019), which is composedof four laser scanned industrial facilities. The original number of laser scanned points, the numberof instances, the area and the manual labor hours to manually annotate (with class and instancelabels) each facility are documented in Figure 3.
Process 1: CLOI-NET-Class segmentation
The methods of Process 1 bypass the stage of surface generation altogether and directly outputsegmented and labelled point clusters. The 3D window parsing method breaks down the wholeindustrial facility into subset windows for more efficient processing. The key insight behind Process1 is to formulate a high dimensional feature space to automatically assign labels per point so thatthe target point clusters can be quickly located in the point cloud.The inputs of the method are the spatial coordinates of TLS points and the outputs are labelled,segmented point clusters with confidence levels of the predictions. Here we define segmented pointclusters as all the points that belong to one class i.e. all cylinder points is one class point cluster.The method consists of three major steps:
Step 1 partitions each facility into smaller spaces using a3D sliding window/block approach and prepares the data for training,
Step 2 predicts a class labelper point using a modified version (SFR) of a geometric deep learning network for point cloudsegmentation (PointNET++) with the goal to accurately segment the
CLOI shapes. In
Step 2 , theuser has two options on how to train the network, either training with no data from the test facility ormanually annotating data of the test facility and including those for training. The latter is based onthe assumption that, inevitably, any class segmentation algorithm will have errors, which will haveto be manually corrected eventually. Therefore the goal is to minimize the total manual annotation10 Agapaki, January 6, 2021ime.
Step 3 refines the predicted class labels by improving class level predictions with strongercontextual relationships.The success of the proposed pipeline is measured not by maximising the point-wise accuracyof the method, rather by minimising the cost that it incurs to the modelers when using it. Thisnovel method leverages the advances in point cloud deep learning segmentation, contextual shapespecific attributes and active learning in order to accurately predict point-wise class labels withno significant difference in performance for diverse industrial environments. A critical part ofthis method’s novel design is the stage-wise annotation, which permits both human-annotatedand automatically annotated points to influence the system’s view of what needs the most humanattention next. Details of our methodology, named
CLOI-NET-Class , can be found in (Agapaki andBrilakis 2020a).
Process 2: CLOI-Ins instance segmentation
The inputs of Process 2 are the predicted point clusters from the CLOI-NET-Class method forthe evaluation of the proposed framework. The same 3D block generation method from Process 1is used for segmenting the input data. The outputs of this process are point-wise instance labels(individual point clusters of
CLOI shapes).Process 2 consists of two major steps: Step 1 predicts an instance label per point by using agraph-based method, namely Breadth First Search (BFS) that was originally introduced by (Bauerand Wössner 1972). Step 2 is a boundary segmentation method that is used to enhance the instancesegmentation results of Step 1. An assumption of the method is that the initial TLS industrial datais partitioned in 3D non-overlapping sliding windows with overlapping 3D blocks. The outputsof Step 1 are connected components based on connectivity relationships in order to segment theinstances as output. The boundary segmentation method in Step 2 outputs binary labels on whethera point is a boundary point or not. These instance point clusters present industrial shapes at Levelof Detail (LOD) 300.The novelty of Process 2 is two-fold: 11 Agapaki, January 6, 2021. the efficiency of the BFS algorithm by applying it on the entire PCD and connectivitybetween points2. the intelligence of the boundary segmentation method to account for boundary points androbustly process points in small regions.Readers can refer to (Agapaki and Brilakis 2020b) for details of the CLOI-Ins instance segmen-tation process.
EXPERIMENTS AND RESULTS
Implementation
The author generated the first dataset of class labelled point clusters of industrial facilities,
CLOI , (Agapaki et al. 2019) to validate Processes 1 and 2.
CLOI consists of 10 classes that cover awide range of industrial scenes (both indoor and outdoor). The TLS datasets of four laser scannedindustrial facilities are used for the generation of
CLOI as shown in Figure 3. One facility is awarehouse, one is a petrochemical plant, one an oil refinery and the fourth a processing unit. Thesefacilities are anonymized since rights are reserved by AVEVA Group Plc. and British Petroleum.All datasets were obtained using static terrestrial laser scanners. This research provides the (to thebest of our knowledge) hitherto largest collection of terrestrial laser scans of industrial facilitieswith point-level (a) class and (b) instance ground truth annotations. (a) refers to one of the ten
CLOI classes and (b) is an index number that refers to a specific individual shape and is not furtherused in this work. In total, it consists of 12,497 shapes and 7.1 billion points with their class andinstance labels for each point. To this end, this research provides
CLOI , the largest annotated datasetbased on already existing datasets (Agapaki and Brilakis 2020a) and the only dataset of industrialenvironments that is captured with more than one sensors. This means that processing
CLOI pointcloud data is independent of the data capturing system that was used to generate the data.
CLOI isalso the only dataset available for processing PCDs of industrial environments. Detailed statisticsand scanner specifications of the data can be found in (Agapaki et al. 2019).Two research platforms were developed for the framework validation; one capable of high12 Agapaki, January 6, 2021omputing for training deep neural networks and one for visualisations of large scale TLS industrialdatasets. Training of the CLOI-NET-Class method was performed on Google Cloud instances. Weimplemented the deep learning class segmentation experiments on Tensorflow 2.0 as a proof ofconcept prototype and ran experiments on Google Cloud (Deep Learning VM image) with NVIDIATesla P100 GPUs. Visualizations of point clouds and segmentation results were implemented onthe
CLOI platform which is based on the Potree Viewer (http://potree.org/) in JavaScript. Potreeis built upon ThreeJS and allows for rendering of large point clouds in a WebGL web browser(Schuetz 2016; Devaux et al. 2012). We created the user interface to select the TLS dataset ofa
CLOI facility, then segment the
CLOI classes and validate with the ground truth class labels.The user can also select a point and only view the points associated with that
CLOI class. Furtherdetails about the implementation of Process 1 and Process 2 can be found in (Agapaki and Brilakis2020a) and (Agapaki and Brilakis 2020b) respectively.
Manual annotation
The
CLOI dataset was generated by manually annotating the four industrial facilities. TheGround Truth (GT) datasets are the desired outputs to compare against those generated by theproposed methodology and also used for training. The following GT datasets were created for the
CLOI dataset validation.
GT class:
A given industrial facility, TLS scanned, point cloud input is segmented into theeight
CLOI classes. Each individual point was assigned a class point-wise label. Figure 3 showseach
CLOI facility coloured with one of the eight class labels and the manual annotation timeinvolved to generate the GT per facility. The number of shapes (instances), original number of 3Dpoints and the area per facility are also provided. One can distinguish that even if a small facilityarea is scanned, the density of the scans may be so high that the number of points is much highercompared to a sparsely scanned facility. For instance, the oil refinery is only 300 𝑚 , making it thesmallest facility of the dataset, but it has the largest number of surveyed 3D points. GT instance:
A given point cloud input is assigned to an individual instance point cluster.13 Agapaki, January 6, 2021
T boundary:
A given point is classified as a “boundary” point if there is more than oneinstance in a neighbourhood of radius 4 𝑐𝑚 around it. The data structure used to define theneighbourhoods around each point is a kDTree. Experiments
The performance of the framework was evaluated based on:1. the performance of the
CLOI-NET-Class segmentation network2. the performance of the
CLOI-Instance segmentation network.The inputs of the proposed framework are the class segmented clusters of Process 1. The classsegmentation experiments showed average accuracy and mIoU of 79.8% and 44.65% when allthe
CLOI facilities are included for training except the one of interest to segment that is tested.CLOI-NET-Class has been proven to be consistent, reliable and without significant bias when testedon all the
CLOI facilities. The author validated the theoretical active learning model as outlinedin (Agapaki and Brilakis 2020a). Results showed that the total cost annotation function and thevalidation accuracy follow the theoretical model and the optimal data pre-annotation percentagethat minimized the total annotation cost is between 20 ± CLOI facilities using the ground truth class labels as inputs (Agapaki and Brilakis 2020b). Forthe evaluation of the framework, we compared the state-of-the-art instance segmentation networks(SGPN (Wang et al. 2018b; Wang et al. 2019)), the BFS algorithm and the proposed
CLOI framework in Table 1. The results illustrated in Table 1 show that SGPN has very low performanceon the oil refinery data with the ASIS network performing better in all efficiency metrics. The oilrefinery is used to compare the state-of-the-art deep learning instance segmentation networks, theBFS algorithm and the
CLOI framework methodology. For the application of the BFS algorithm,the minimum instance size was selected for the predicted
CLOI class point clusters based on14 Agapaki, January 6, 2021erformance. Therefore, the author conducted experiments to determine the minimum instancesize based on the performance in terms of precision and recall on the
CLOI datasets. The results inFigure 4 illustrate that the optimal trade-off between precision and recall is for minimum instancesize 200 points instead of the minimum instance size of 20 points that was computed based onthe ground truth class segmentation labels (Agapaki and Brilakis 2020b). This is attributed tonoisy predicted class labels compared to the ground truth class labels used to evaluate Process2 independently. There is an exception for the minimum instance size ( 𝜇 ) and the minimumneighbourhood size ( 𝜖 ) for the case of cylinders. The results indicate to set the instance size at50 points and the minimum neighbourhood size ( 𝜖 ) at 3cm (instead of 4cm) only for cylinderinstance point clusters due to the observation that cylinders have higher class segmentation labelpredictions and the CLOI-Instance methodology benefits from that. We also observe in Table 1 a10% increase in precision due to the class boundary constraint on the BFS algorithm for a minimumneighbourhood of 4cm.The author then tested the performance of the same methods per CLOI shape in Table 2. Wepresent these results for the oil refinery dataset as an example for comparison of the best performingexisting instance segmentation methods and the proposed
CLOI framework. The illustrated resultsin Table 1 and 2 demonstrate that the CLOI-Instance methodology clearly outperforms the currentstate-of-the-art research.Another important note is that the CLOI framework results are calculated assuming that theusers pre-annotate X% of the test facility with X% being the value from Table 3 depending on thefacility. These percentages are based on the active learning curves of (Agapaki and Brilakis 2020a).Then, we present the precision and recall per
CLOI class and the average precision and recallcurves in Figure 5 as a reference. The results for the other three facilities are included in theAppendix. It is evident that for all datasets the recall metric of all the
CLOI classes outperformsthe precision metric for all the IoU threshold values. The greater difference between the meanprecision and mean recall is for the oil refinery (Figure 5(c)), which is attributed to the highcomplexity of this dataset. This leads to reduced performance for all classes. Although the CLOI-15 Agapaki, January 6, 2021nstance proposed methodology has promising results compared to the state-of-the-art methods forthe instance segmentation task, the results demonstrate that the predicted class labels significantlyreduce the precision and recall metrics compared to the same results presented given the groundtruth class labels (Agapaki and Brilakis 2020b).The CLOI framework performance of cylinders is relatively high across the
CLOI facilitiesgiven their high class segmentation performance (Agapaki and Brilakis 2020a) for all the IoUthreshold values. We remind the reader that the cylinder class segmentation performance was81.25% precision, 81.75% recall and 68.25% IoU on average. There are though some cases wherethe cylinder instance point clusters are over- or under-segmented. These cases are the
Cyl casespresented in (Agapaki and Brilakis 2020b). The results of the
CLOI framework show an additionalpain point. This is the uncertainty of the CLOI-NET-Class segmentation on predicting the classlabels of the points. This leads to erroneous instance label predictions and mostly impacts the
CLOI classes that have low class segmentation performance (the reader can refer to (Agapaki andBrilakis 2020a) for a detailed discussion).Another achievement of the
CLOI framework is that it correctly segments sub-instances of aninstance point cluster that has the “other” class label and even outperforms the manual instancesegmentation in cases where a ground truth instance is under-segmented (Figure 6(a) and Figure6(b)). This particularly applies for instances close to the floor or roof of a facility. The superiorperformance of the
CLOI framework is attributed to the connectivity information that the BFSalgorithm uses to segment instances. Another case where the
CLOI framework outperforms themanual instance segmentation is for sequences of pipe components that have different radii. Anexample of that is Figure 6(c) where the
CLOI framework correctly segmented the cylinder from apump and a flange with steel rods.We then recommend to use the 25% IoU threshold that gives slightly improved results (50%mPrec and 35.3% mRec for all the
CLOI facilities). The
CLOI shapes that have significantly highermetrics are those with higher class segmentation results as explained above. These are cylinders(53.6% mPrec and 44% mRec), elbows (66.8% mPrec) and I-beams (63% mPrec and 64.3% mRec).16 Agapaki, January 6, 2021 ime savings in Geometric Digital Twinning
One of the main goals of Process 2 was to prove that the CLOI-Instance method requirescompetitively less manual segmentation time compared to the current practice. We validated thishypothesis for the overall framework given that the class segmentation labels are predicted fromthe CLOI-NET-Class method (Process 1). We use the percentage of
CLOI shapes that the CLOI-Instance method correctly predicts as a proxy to approximate the number of manual labour hoursthat are still needed in order to achieve an accurate gDT generation. The results are summarized foreach
CLOI dataset in Table 5. A comparison of the manual instance segmentation time for the
CLOI benchmark dataset generation and the
CLOI overall framework segmentation time is presented inFigure 7. The total number of man hours needed when deploying the overall framework is calculatedas follows. The number of manually segmented
CLOI shapes is computed as the product of thenumber of shapes that are missed by the framework (1 − 𝑟 𝑒𝑐𝑎𝑙𝑙 ) and the average time it takes amodeller to manually segment a given shape. An assumption for the simplification of the calculationhere that each CLOI shape takes the same time regardless of its complexity. The results illustratethat 35% of the manual labour hours are saved on average. The oil refinery dataset is one of themost complex
CLOI datasets and this is reflected in reduced savings in labour hours for instancesegmentation. It is noteworthy that for all
CLOI facilities, the cylinder
CLOI shapes have relativelylow recall ( ≈ CLOI dataset. However, the number of cylinders correctly detected by EdgeWisecan be compared with the number of cylinders segmented by the proposed framework. The resultsin Table 6 demonstrate that the proposed framework correctly segments more cylinders than thosedetected by EdgeWise. The proposed framework is designed to better segment conduits and evenwith the discussed limitations, Table 6 illustrates its superiority to EdgeWise which is mostly in the17 Agapaki, January 6, 2021orrectly predicted conduits that EdgeWise does not identify.The performance of the proposed framework is then compared directly with EdgeWise assumingthat the modeling of
CLOI shapes will be manually performed in EdgeWise. Therefore, the averagemodeling labour time per object is taken from (Agapaki et al. 2018) and multiplied with the numberof objects that are not automatically segmented. The output in labour hours in shown in Figure 8and compared with the manual labour hours for the objects that EdgeWise cannot automaticallydetect (a fraction of cylinders and the rest of
CLOI shapes). Figure 8 shows that 21% and 39%more time savings are achieved when the proposed framework is utilized for the warehouse andpetrochemical plant respectively.The warehouse and the petrochemical plant datasets are then used as a proxy to estimate theaverage percentage of labour hour reduction of the CLOI framework compared to EdgeWise per
CLOI class. The average percentage per class is shown in Table 7. An assumption was made thatthe modeling time of all cylindrical shapes is the same, since our framework detects cylinders andnot their sub-classes, i.e. pipes. Then, the
CLOI framework is directly compared with EdgeWisefor the petrochemical plant with 240,687 objects that was used for manual modeling in (Agapakiet al. 2018). The same assumptions are used here for consistency of the results. The results inFigure 9 reveal that 12 person-months are needed when using the
CLOI framework instead of the17 person months that are needed when using EdgeWise. In particular,
CLOI saves 10% moreman-hours for cylinder modeling, which is translated in 773 labour hours saved. Although there isstill time required for manual cylinder extraction, the proposed framework clearly outperforms thecommercial software EdgeWise.
CONCLUSIONS
This paper presents
CLOI , an automated benchmarking framework for generating gDTs ofexisting industrial facilities from point cloud data. This work focuses on the generation of instancepoint clusters in a cost-effective approach compared to the current practice. The framework consistsof two main processes: the
CLOI-NET-Class segmentation (Process 1), which generates the tenmost important industrial objects in the format of class point clusters and
CLOI-Ins segmentation18 Agapaki, January 6, 2021Process 2), which segments the class point clusters into individual point clusters. The
CLOI framework was experimentally validated on the largest published industrial point cloud dataset,which consists of four TLS industrial point clouds. The consistent results on the
CLOI datasetdemonstrate that the proposed framework can reduce the onerous, repetitive manual work ofsegmenting industrial shapes and therefore reduce the modelling time of the resulting models. Inthe following paragraphs, we present the contributions (
Con ) and limitations (
Lim ) of the
CLOI framework in detail.
Con 1
This is the first framework of its kind to achieve significantly high and reliable perfor-mance (50% mPrec and 35.3% mRec) compared to current state-of-the-art research and commer-cially available software. It is the first framework to provide significant improvements on cylindersegmentation (53.6% mPrec and 44% mRec) and the first to segment the rest of the
CLOI classes.It, therefore, provides a solid foundation for future work in generating DTs of industrial facilities.
Con 2
This research moves forward the state of automated class and instance segmentation fromTLS point cloud datasets as well as promotes the value of adding “intelligence” to the PCD data.The interpretation of the results strongly suggest that the performance of both the CLOI-NET-Classand the CLOI-Instance methods are significantly improved by using the optimal amount of dataduring training ( ≈ CLOI classes.
Con 3
It is the first framework of its kind to significantly reduce the manual labour hours (by at least33%) compared to the state of practice, EdgeWise. It also has 21% and 39% more time savingswhen segmenting the warehouse and the petrochemical facility dataset compared to EdgeWise.
Con 5
The connectivity of pipe components or members of steel frames assist the modeller inidentifying all the connected components of a pipe spool or steel frame when using the outputs ofthis framework. Figure 10 shows characteristic examples from the warehouse and the oil refinerydatasets. The confidence level of the predicted class labels from the CLOI-NET-Class methodis also an indicator of whether the performance of the instance segmentation under-segments in-stances. Figure 10(aiii) shows that the elbows of the pipe spool were predicted with uncertainty(confidence level score ≤ Lim 1
The
CLOI dataset, although the largest available dataset of TLS industrial point clouds,is not enough to fully validate the proposed framework. More industrial facility point cloudswith various configurations are needed to enhance the statistical validity of the framework with anincreased confidence level and decrease the bias between facilities especially for the
CLOI classesthat are underrepresented in the dataset. As demonstrated in (Agapaki and Brilakis 2020a) moredata is not always beneficial, so careful experimental set-up should be conducted to alleviate fromnegatively impacting the performance.
Lim 2
Manual annotation of TLS industrial point cloudsaccording to the data preparation explained in the experiments section is an onerous task. In theseefforts, an automated segmentation interface should be adopted to enable for easy generation oflabelled class and instance point clusters.
Lim 3
Finally, the framework is not designed to segmentobjects of the same geometric group, for instance pipes, conduits and circular hollow sections orfurther object types within the same class i.e. globe valves, gate valves. This could be an interestingdirection for future research.
DATA AVAILABILITY
Some or all data, models, or code used during the study were provided by a third party. Directrequests for these materials may be made to the provider as indicated in the Acknowledgements.
ACKNOWLEDGEMENTS
We thank our colleague Graham Miatt, who has provided insight, expertise and data that greatlyassisted this research. We also express our gratitude to Bob Flint from BP International Centre forBusiness and Technology (ICBT), who provided data for evaluation. The research leading to theseresults has received funding from the Engineering and Physical Sciences Research Council (EPSRC)and the US National Academy of Engineering (NAE). AVEVA Group Plc. and BP InternationalCentre for Business and Technology (ICBT) partially sponsor this research under grant agreementsRG83104 and RG90532 respectively. We gratefully acknowledge the collaboration of all academicand industrial project partners. Any opinions, findings and conclusions or recommendations20 Agapaki, January 6, 2021xpressed in this material are those of the authors and do not necessarily reflect the views of theinstitutes mentioned above.
APPENDIX
Appendix figures.
REFERENCES
Agapaki, E. and Brilakis, I. (2020a). “Cloi-net: Class segmentation of industrial facilities’ pointcloud datasets.”
Advanced Engineering Informatics , 45, 101121.Agapaki, E. and Brilakis, I. (2020b). “Instance segmentation of industrial point cloud data.Agapaki, E., Glyn-Davies, A., Mandoki, S., and Brilakis, I. (2019). “CLOI: A Shape ClassificationBenchmark Dataset for Industrial Facilities.” .Agapaki, E., Miatt, G., and Brilakis, I. (2018). “Prioritizing object types for modelling existingindustrial facilities.”
Automation in Construction .Agapaki, E. and Nahangi, M. (2020). “Chapter 3 - Scene understanding and model generation.”
Infrastructure Computer Vision , I. Brilakis and C. Haas, eds., Elsevier, 1 edition, Chapter 3.Agarwal, R., Chandrasekaran, S., and Sridhar, M. (2016). “The digital future of construc-tion, < > .Armeni, I., Sener, O., Jiang, H., Fischer, M., and Savarese, S. (2016). “3D Semantic Parsingof Large-Scale Indoor Spaces.” Proceedings of the IEEE Conference on Computer Vision andPattern Recognition , 1534–1543.Barbosa, F., Woetzel, J., Mischke, J., Joao Ribeirinho, M., Sridhar, M., Parsons, M.,and Brown, S. (2017). “Reinventing Construction: A Route to Higher Productiv-ity, < > .21 Agapaki, January 6, 2021auer, F. L. and Wössner, H. (1972). “The “Plankalkül” of Konrad Zuse: A Forerunner of Today’sProgramming Languages.” Communications of the ACM .Borenstein, E. and Ullman, S. (2008). “Combined top-down/bottom-up segmentation.”
IEEE Trans-actions on Pattern Analysis and Machine Intelligence , 30(12), 2109–2125.Borrmann, A. and Berkhahn, V. (2018). “Principles of geometric modeling.”
Building InformationModeling , Springer, 27–41.ClearEdge (2019). “Plant Modeling Capabilities, < > .Devaux, A., Br??dif, M., and Paparoditis, N. (2012). “A web-based 3D mapping application usingWebGL allowing interaction with images, point clouds and models.” GIS: Proceedings of theACM International Symposium on Advances in Geographic Information Systems .Dimitrov, A. and Golparvar-Fard, M. (2015). “Segmentation of building point cloud models in-cluding detailed architectural/structural features and MEP systems.”
Automation in Construction ,51(C), 32–45.Fumarola, M. and Poelman, R. (2011). “Generating virtual environments of real world facilities:Discussing four different approaches.”
Automation in Construction , Vol. 20, 263–269.Gerbert, P., Castagnino, S., Rothballer, C., Renz, A., and Filitz, R. (2016). “Digital in Engineer-ing and Construction, < http://futureofconstruction.org/content/uploads/2016/09/BCG-Digital-in-Engineering-and-Construction-Mar-2016.pdf > .Glaessgen, E. H. and Stargel, D. S. (2012). “The digital twin paradigm for future NASA and U.S.Air force vehicles.” Collection of Technical Papers - AIAA/ASME/ASCE/AHS/ASC Structures,Structural Dynamics and Materials Conference .Grieves, M. (2014). “Digital Twin: Manufacturing Excellence Through Virtual Factory Replica-tion.”
Nc-Race 18 .Huang, J. and You, S. (2013). “Detecting objects in scene point cloud: A combinational approach.”
Proceedings - 2013 International Conference on 3D Vision, 3DV 2013 , 175–182.Hullo, J.-F., Thibault, G., Boucheny, C., Dory, F., and Mas, A. (2015). “Multi-Sensor As-Built22 Agapaki, January 6, 2021odels of Complex Industrial Architectures.”
Remote Sensing , 7(12), 16339–16362.Kalogerakis, E., Averkiou, M., Maji, S., and Chaudhuri, S. (2017). “3D Shape segmentation withprojective convolutional networks.”
Proceedings - 30th IEEE Conference on Computer Visionand Pattern Recognition, CVPR 2017 .Klokov, R. and Lempitsky, V. (2017). “Escape from Cells: Deep Kd-Networks for the Recognitionof 3D Point Cloud Models.”
Proceedings of the IEEE International Conference on ComputerVision .Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). “ImageNet Classification with DeepConvolutional Neural Networks.”
Advances In Neural Information Processing Systems .Landrieu, L. and Simonovsky, M. (2018). “Large-scale point cloud semantic segmentation withsuperpoint graphs.LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.(2008). “Backpropagation Applied to Handwritten Zip Code Recognition.”
Neural Computation .Li, B., Shi, Y., Qi, Z., and Chen, Z. (2019). “A survey on semantic segmentation.”
IEEE Interna-tional Conference on Data Mining Workshops, ICDMW .Li, Z., Zhang, L., Tong, X., Du, B., Wang, Y., Zhang, L., Zhang, Z., Liu, H., Mei, J., Xing, X.,and Mathiopoulos, P. T. (2016). “A three-step approach for tls point cloud classification.”
IEEETransactions on Geoscience and Remote Sensing , 54(9), 5412–5424.Marshall, G. F. (2016).
Handbook of Optical and Laser Scanning .Marton, Z. C., Rusu, R. B., and Beetz, M. (2009). “On fast surface reconstruction methods forlarge and noisy point clouds.”
Robotics and Automation, 2009. ICRA ’09. IEEE InternationalConference on , 3218–3223.Maturana, D. and Scherer, S. (2015). “Voxnet: A 3d convolutional neural network for real-timeobject recognition.”
IROS .McKinsey Global Institute (2015). “Digital America: A tale of the Haves and Have-Mores.”
Report no. , < > .National Institute of Standards and Technology (2018). “The Costs and Benefits of Ad-vanced Maintenance in Manufacturing.” Report no. , U.S. Department of Commerce, < https://nvlpubs.nist.gov/nistpubs/ams/NIST.AMS.100-18.pdf > .Pang, Y., Li, L., Hu, W., Peng, Y., Liu, L., and Shao, Y. (2012). “Computerized segmentation andcharacterization of breast lesions in dynamic contrast-enhanced MR images using fuzzy c-meansclustering and snake algorithm.” Computational and Mathematical Methods in Medicine .PECI (1999). “Portable Data Loggers Diagnostic Tools for Energy-Efficient Building Op-erations.”
Report no. , Prepared for the U.S. Environmental Protection Agency and U.S.Department of Energy by Portland Energy Conservation, Incorporated, Portland, Oregon, < > .Perez-Perez, Y., Golparvar-Fard, M., and El-Rayes, K. (2016). “Semantic and Geometric Labelingfor Enhanced 3D Point Cloud Segmentation.” Construction Research Congress 2016 , 2542–2552.Pham, Q., Nguyen, D. T., Hua, B., Roig, G., and Yeung, S. (2019). “JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-ValueConditional Random Fields.”
CVPR .Qi, C. R., Yi, L., Su, H., and Guibas, L. J. (2017a). “PointNet++: Deep Hierarchical FeatureLearning on Point Sets in a Metric Space.”
Computer Vision and Pattern Recognition (CVPR) .Qi, R., Su, H., K., M., and J., G. L. (2017b). “PointNET: Deep Learning on Point Sets for 3DClassification and Segmentation.”
Computer Vision and Pattern Recognition (CVPR) .Rusu, R. B., Blodow, N., Marton, Z. C., and Beetz, M. (2009). “Close-range scene segmentationand reconstruction of 3D point cloud maps for mobile manipulation in domestic environments.” , 1–6.Sampath, A. and Shan, J. (2010). “Segmentation and reconstruction of polyhedral building roofsfrom aerial lidar point clouds.”
IEEE Transactions on Geoscience and Remote Sensing , 48(3PART2), 1554–1567. 24 Agapaki, January 6, 2021chuetz, M. (2016). “Potree: Rendering Large Point Cloudin Web Browsers.” Ph.D. thesis, University of TU Wien, , < https://pdfs.semanticscholar.org/0d9d/db7335331d28a4a23e086e960396fd4e1b65.pdf > .Su, H., Maji, S., Kalogerakis, E., and Learned-Miller, E. (2015). “Multi-view convolutional neuralnetworks for 3D shape recognition.” Proceedings of the IEEE International Conference onComputer Vision .Taha, A. A. and Hanbury, A. (2015). “Metrics for evaluating 3D medical image segmentation:Analysis, selection, and tool.”
BMC Medical Imaging .Tatarchenko, M., Dosovitskiy, A., and Brox, T. (2017). “Octree Generating Networks: EfficientConvolutional Architectures for High-resolution 3D Outputs.”
Proceedings of the IEEE Interna-tional Conference on Computer Vision .Teichmann, M., Weber, M., Zöllner, M., Cipolla, R., and Urtasun, R. (2018). “MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving.”
IEEE Intelligent Vehicles Symposium,Proceedings .Thomas, H., Qi, C. R., Deschaud, J.-E., Marcotegui, B., Goulette, F., and Guibas, L. J. (2019).“Kpconv: Flexible and deformable convolution for point clouds.Vosselman (2009). “Advanced Point Cloud Processing.”
In Photogrammetric Week ’09 , 137–146.Wang, S., Suo, S., Ma, W. C., Pokrovsky, A., and Urtasun, R. (2018a). “Deep Parametric ContinuousConvolutional Neural Networks.”
Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition .Wang, W., Yu, R., Huang, Q., and Neumann, U. (2018b). “SGPN: Similarity Group ProposalNetwork for 3D Point Cloud Instance Segmentation.”
Computer Vision and Pattern Recognition .Wang, X., Shen, X., Shen, C., and Jia, J. (2019). “Associatively Segmenting Instances and Semanticsin Point Clouds.”
CVPR .Wei, L., Huang, Q., Ceylan, D., Vouga, E., and Li, H. (2016). “Dense human body correspon-dences using convolutional networks.”
Dense human body correspondences using convolutionalnetworks , CVPR. 25 Agapaki, January 6, 2021est, T. and Blackburn, M. (2017). “Is Digital Thread/Digital Twin Affordable? A SystemicAssessmet of the Cost of DoD’s Latest Manhattan Project.”
Procedia Computer Science , 114,47–56.Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2015). “3D ShapeNets: Adeep representation for volumetric shapes.”
Proceedings of the IEEE Computer Society Confer-ence on Computer Vision and Pattern Recognition , Vol. 07-12-June, 1912–1920.Xie, Y., Tian, J., and Zhu, X. X. (2019). “A review of point cloud semantic segmentation.” arXivpreprint arXiv:1908.08854 .Zhang, J., Huang, Q., and Peng, X. (2015). “3D Reconstruction of Indoor Environment Usingthe Kinect Sensor.” , 538–541 (9).Zhang, J., Lin, X., and Ning, X. (2013). “Svm-based classification of segmented airborne lidarpoint clouds in urban areas.”
Remote Sensing , 5(8), 3749–3775.Zhou, Y. and Tuzel, O. (2017). “VoxelNet: End-to-End Learning for Point Cloud Based 3D ObjectDetection.” arXiv . 26 Agapaki, January 6, 2021 ist of Tables
CLOI shape in the oil refinerydataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Optimal class segmentation pre-annotation percentage of test facility data for activelearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Performance of the CLOI-Instance method per
CLOI shape for all the
CLOI datasets(IoU=25%) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Manual labour hours and total segmentation savings of the overall framework per
CLOI facility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 Correctly predicted cylinders of the petrochemical plant and warehouse point cloudsusing EdgeWise and our framework. . . . . . . . . . . . . . . . . . . . . . . . . . 337 Percentage (%) of the reduction of the labour hours of the
CLOI framework com-pared to EdgeWise per class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3427 Agapaki, January 6, 2021
ABLE 1.
CLOI framework performance for the oil refinery dataset
Method mPrec(%) mRec(%)SGPN (Wang et al. 2018b) 5.3 6.5
ASIS (Wang et al. 2019) 16.7 4.5
CLOI-Framework (without boundary)
CLOI-Framework
28 Agapaki, January 6, 2021
ABLE 2.
Performance of instance segmentation networks per
CLOI shape in the oil refinerydataset
Prec (%) Angles Channels Cylinders Elbows I-beams Valves FlangesASIS
SGPN
BFS
CLOI-Instance
Rec (%) Angles Channels Cylinders Elbows I-beams Valves FlangesASIS
SGPN
BFS
CLOI-Instance
29 Agapaki, January 6, 2021
ABLE 3.
Optimal class segmentation pre-annotation percentage of test facility data for activelearning
Test facility Optimalpre-annotated data(%)Warehouse Processing unit Oil refinery Petrochemical
30 Agapaki, January 6, 2021
ABLE 4.
Performance of the CLOI-Instance method per
CLOI shape for all the
CLOI datasets(IoU=25%)
Oil refinery Angles Channels Cylinders Elbows I-beams Valves FlangesPrec (%)
Rec (%)
Warehouse Angles Channels Cylinders Elbows I-beams Valves FlangesPrec (%)
56 67.1 64.7 76.9 44.4 29.4 30.8
Rec (%)
Petrochemical Angles Channels Cylinders Elbows I-beams Valves FlangesPrec (%)
50 52.6 51.1 70 77.8 29.7 40
Rec (%)
35 46.2 48.2 20 61.8 91.7 8.3
Processing unit Angles Channels Cylinders Elbows I-beams Valves FlangesPrec (%)
Rec (%)
31 Agapaki, January 6, 2021
ABLE 5.
Manual labour hours and total segmentation savings of the overall framework per
CLOI facility.
Oil refinery
Angles Channels Cylinders Elbows I-beams Valves Flanges OtherRecall (%) 26 19 43 20 49 36 11 25Total
Angles Channels Cylinders Elbows I-beams Valves Flanges OtherRecall (%) 16.5 34.6 56 18.6 100 41.7 28.6 27.9Total
Angles Channels Cylinders Elbows I-beams Valves Flanges OtherRecall (%) 35 46.2 41.8 20 61.8 91.7 8.3 29Total
Angles Channels Cylinders Elbows I-beams Valves Flanges OtherRecall (%) 8.7 23.7 35.5 9.1 46.4 43.5 0.4 25.1Total number of shapes 188 34 1100 382 274 341 229 370Manually segmented
32 Agapaki, January 6, 2021
ABLE 6.
Correctly predicted cylinders of the petrochemical plant and warehouse point cloudsusing EdgeWise and our framework.
EdgeWise 468 164Proposed framework 510 62333 Agapaki, January 6, 2021
ABLE 7.
Percentage (%) of the reduction of the labour hours of the
CLOI framework comparedto EdgeWise per class.
CLOI class % of labour hour reduction
Cylinders 22.3Channels 40.4I-beams 81Valves 67Elbows 19.3Flanges 18.5Angles 25.7 34 Agapaki, January 6, 2021 ist of Figures
CLOI framework . . . . . . . . . . . . . . . . . . . . . 373
CLOI benchmark dataset specifications . . . . . . . . . . . . . . . . . . . . . . . 384 Performance of the BFS algorithm with respect to the minimum instance size ( 𝜇 )for IoU=50% and 𝜖 = CLOI class and (c) mean precisionand recall for different IoU thresholds for the oil refinery facility. . . . . . . . . . . 406 Examples where the CLOI framework outperforms the manual instance segmenta-tion. (i) refers to ground truth instances and (ii) refers to predicted instances withthe CLOI framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 Manual and our framework’s total labour hours per
CLOI dataset. . . . . . . . . . 428 Comparison of EdgeWise and our framework with respect to manual labour hoursper
CLOI shape for the (a) petrochemical and (b) the warehouse dataset. . . . . . . 439 Average modelling labour hours per object type for the most important objects of asample facility with shown numbers of objects. . . . . . . . . . . . . . . . . . . . 4410 Examples of ground truth and predicted instances of piping elements (a,b) and (c)I-beams of a steel frame. (aiii) Predicted class label predictions (predictions with ≤
80% confidence score coloured in red). . . . . . . . . . . . . . . . . . . . . . . 4511 (a) BFS instance precision and (b) recall per
CLOI class and (c) mean precisionand recall for different IoU thresholds for the processing unit facility. . . . . . . . . 4612 (a) BFS instance precision and (b) recall per
CLOI class and (c) mean precisionand recall for different IoU thresholds for the petrochemical plant facility. . . . . . 4713 (a) BFS instance precision and (b) recall per
CLOI class and (c) mean precisionand recall for different IoU thresholds for the warehouse facility. . . . . . . . . . . 4835 Agapaki, January 6, 2021 ig. 1.
Automated geometric Digital Twinning strategies36 Agapaki, January 6, 2021 ig. 2.
Workflow of the proposed
CLOI framework37 Agapaki, January 6, 2021 ig. 3.
CLOI benchmark dataset specifications38 Agapaki, January 6, 2021 ig. 4.
Performance of the BFS algorithm with respect to the minimum instance size ( 𝜇 ) forIoU=50% and 𝜖 = ig. 5. (a) BFS instance precision and (b) recall per CLOI class and (c) mean precision and recallfor different IoU thresholds for the oil refinery facility.40 Agapaki, January 6, 2021 ig. 6.
Examples where the CLOI framework outperforms the manual instance segmentation. (i)refers to ground truth instances and (ii) refers to predicted instances with the CLOI framework.41 Agapaki, January 6, 2021 ig. 7.
Manual and our framework’s total labour hours per
CLOI dataset.42 Agapaki, January 6, 2021 ig. 8.
Comparison of EdgeWise and our framework with respect to manual labour hours per
CLOI shape for the (a) petrochemical and (b) the warehouse dataset.43 Agapaki, January 6, 2021 ig. 9.
Average modelling labour hours per object type for the most important objects of a samplefacility with shown numbers of objects. 44 Agapaki, January 6, 2021 ig. 10.
Examples of ground truth and predicted instances of piping elements (a,b) and (c) I-beamsof a steel frame. (aiii) Predicted class label predictions (predictions with ≤
80% confidence scorecoloured in red). 45 Agapaki, January 6, 2021 ig. 11. (a) BFS instance precision and (b) recall per
CLOI class and (c) mean precision and recallfor different IoU thresholds for the processing unit facility.46 Agapaki, January 6, 2021 ig. 12. (a) BFS instance precision and (b) recall per
CLOI class and (c) mean precision and recallfor different IoU thresholds for the petrochemical plant facility.47 Agapaki, January 6, 2021 ig. 13. (a) BFS instance precision and (b) recall per