Tackling problems of marker-based augmented reality under water
TTackling problems of marker-based augmented reality un-der water
Authors: Jan Č ejka, Fotis Liarokapis Abstract
Underwater sites are a harsh environment for augmented reality applications. Obstacles that must be battled include poor visibility conditions, difficult naviga-tion, and hard manipulation with devices under water. This chapter focuses on the problem of localizing a device under water using markers. It discusses various fil-ters that enhance and improve images recorded under water, and their impact on marker-based tracking. It presents various combinations of 10 image improving algorithms and 4 marker detecting algorithms, and tests their performance in real situations. All solutions are designed to run real-time on mobile devices to provide a solid basis for augmented reality. Usability of this solution is evaluated on loca-tions in Mediterranean Sea. It is shown that image improving algorithms with carefully chosen parameters can reduce the problems with visibility under water and improve the detection of markers. The best results are obtained with marker detecting algorithms that are specifically designed for underwater environments.
Introduction
Cultural heritage sites and artefacts are spread all around the word, and people search them to learn more about their history and lives. Today, they are not limited only to observe these objects in their current state and read about their story, but thanks to modern technologies like augmented reality (AR), they can see these ob-jects as virtual models superimposed into the real world to see how they fit into the scene and how they interact with other objects. These technologies are able to show even missing parts of settlements or whole buildings [1, 2]. Historical artefacts are not only on land, but many of them are hidden under water. This includes wrecks of ancient ships transporting goods between cities, or seaside settlements that submerged over the last thousands of years. Unfortunate- ly, underwater sites are not only harder to access for people that wish see the arte-facts, but they also impose many problems for technology to work. Localization techniques based on GPS, Wi-Fi, or Bluetooth technology do not work as their signal is absorbed very quickly. Augmented reality and other computer vision so-lutions that require visual input struggle with problems like low contrast of imag-es, sensor noise caused by recording images in low light, occlusions caused by small particles and fish floating in water, unnatural colors due to uneven absorp-tion of light, and short visibility limited due to turbidity. The idea of using AR under water is not new [3], however, solutions are lim-ited mostly on the clear water of swimming pools [4, 5, 6]. In marine areas, AR solutions use acoustic beacons to replace the visual input [7, 8], but they are lim-ited only to show a map and a textual information about the area, since they are not able to track precise position of the diver required to accurately superimpose virtual objects. Impact of bad visibility conditions on algorithms of computer vi-sion was tested in laboratory conditions [9] or as a part of an evaluation of a single image improving algorithm [10, 11], but such evaluation was not focused on sea environments. This chapter describes solutions used for improving marker-based tracking for project iMareCULTURE [12] and is based on results of Ž u ž i et al. [13] and Č ejka et al. [14, 15]. These works focus on postprocessing images taken under water to increase their quality to improve detection of markers for AR. It is divided into three parts. First, it evaluates nine algorithms for improving images to assess their performance for enhancing a quality of images before detecting markers for AR. Second, it chooses the most promising solutions and performs a deeper analysis in various sea environments. The final part inspects which components of marker de-tecting algorithms are affected in in bad visibility conditions and presents a results of a cultural heritage use case scenario. Performance of image-enhancing algorithms
Visibility under water is affected by many factors that are hard to separate to explore their impact individually, so for this reason, all algorithms in this chapter are evaluated using a data recorded in sea environments. First, to get an initial in-sight about the performance of image-enhancing algorithms, this section presents a brief analysis of nine solutions by recording a video with markers placed under water, processing it offline, and comparing the number of detected markers. The most promising solutions are chosen and studied in more details in the following sections. The test was conducted with a video with two divers holding a sheet with markers (see Figure 1). A camera operated by another diver is moving closer to them to decrease the impact of bad under water visibility conditions when the dis-tance to the markers gets smaller. The video is recorded with a GoPro camera with resolution of 1920 × Figure 1: Two divers holding sheets with markers for initial assessment of image-improving algorithms.
The video was processed off-line on a standard PC with processor Intel Core i5 760, 8 GB of operating memory, and operating system Windows 10. Each frame was processed separately, first increasing its quality by applying image-improving algorithms, and then detecting markers.
Tested image enhancing algorithms
In total, nine image-improving algorithms were evaluated: two algorithms de-signed to improve the colors of underwater images (fusion based on [17] , and bright channel prior based on [11]), three algorithms for denoising (Gaussian fil-ter, median filter, and bilateral filter), three algorithms for increasing contrast (his-togram equalization, contrast limited adaptive histogram equalization, and white balancing), and one algorithm for image sharpening (debluring), see Figure 2. The algorithms were implemented in MATLAB, OpenCV 3.2.0, and C++. To assess them, different parameters were explored to find out how they affect tracking ac-curacy. Algorithms based on neural networks [18, 19] were not considered in this experiment, as they require a large amount of data for training. Figure 2: Image enhancing algorithms. Top row from left to right: original image, fusion, bright channel prior, Gaussian filter, median filter; bottom row from left to right: bilateral filter, histogram equalization, contrast limited adaptive histogram equalization, white bal-ancing, debluring.
Fusion
The fusion algorithm is based on [17]. It restores the colors and enhances the contrast of images taken under water by deriving two improved versions of the in-put image and combines them together using a multi-scale fusion process with a weight function computed from each image. The restoration process is simplified by using general image-improving tech-niques, instead of choosing a more sophisticated technique based on the physical properties of the scene. The first enhanced version of the input image is obtained by using a white balancing algorithm that corrects the shift of colors caused by various illuminations of the scene. This algorithm is based on the Gray-World ap-proach [20], which was found as the most appropriate method for underwater im-ages. The second enhanced version of the input image is aiming at reducing noise and improving contrast of the underwater image, by applying the bilateral filter on the color-corrected image created in previous step. The contrast is further im-proved by applying a local adaptive histogram equalization method. Fusion of the derived images is controlled by local contrast weights derived from the luminance of the images, and saliency weight to highlight the objects with higher importance. The final image is created by blending the two derived input images by their normalized weight values. The implementation was done in Matlab, where most of the filters and operations are available directly, using its Image Processing Toolbox.
Bright channel prior
Bright channel prior (BCP) is a technique developed by [11], which is based on a method of [21] to dehaze images taken on land that are affected by fog. [21] ob-serves that with images taken on land with no fog, intensity of one color channel is very low. However, this assumption fails to work for images taken under water, since the colors are shifted due to an uneven absorption of light under water. Visi-ble light with longer wavelength is absorbed more quickly, which results in un-derwater scenes being green or blue due to very low intensities of red color chan-nel. Because of this, [11] define the bright channel image as follows: 𝐽 " = max *∈,(.) 𝐼 (𝑦) ; where Ω(𝑥) denotes a window neighborhood centered at pixel 𝑥 and 𝐼 is the input image with original red color channel and inverted green and blue channels. Intensities of all color channels of this bright channel image are close to one for pixels without haze, and lower for pixels affected by haze and turbidity. To estimate atmospheric light, which is defines the color distortion of the im-age, [11] takes one percent of darkest pixels, and a pixel with the least variance is selected as the color of atmospheric light. Transmittance is then derived from the bright channel image and estimated atmospheric light by following equation 𝑡 (𝑥) = (𝐽 " (𝑥) − 𝐴 )1 − 𝐴 where 𝑐 denotes red, green, and blue color channels, 𝑡 denotes the transmit-tance of this channel, and 𝐴 denotes the value of atmospheric light. The final transmittance is obtained by averaging values across the all color channels, which is then processed with guide filter [22] to remote halos. The final haze-free image is restored by formula: 𝐽(𝑥) = 𝐼(𝑥) − 𝐴𝑡(𝑥) + 𝐴 where 𝐽 is the haze-free image, 𝐼(𝑥) is the degraded underwater image, 𝐴 is the previously estimated atmospheric light and 𝑡(𝑥) is the transmittance of the scene. BCP algorithm was implemented in Matlab using Image Processing Toolkit. Gaussian filter
Gaussian filter is a standard filter for reducing noise in images. It is defined as: 𝐼 DEF (𝑥) = 1𝑊 H 𝐼 I7 (𝑥 I )𝐺 K$L (|𝑥 I − 𝑥|) . N ∈, with normalizing factor: 𝑊 = H 𝐺
K$L (|𝑥 I − 𝑥|) . N ∈, where 𝐺 K$L is a Gaussian function with parameter 𝜎 K$L . Gaussian filter is a lin-ear filter that sums neighbor pixels using weights based on their mutual distance in image space. It decreases the number of edges found by edge detecting algorithms, and thus improves the image for marker detection. Gaussian filtering was applied on the Y channel of our images after their conversion to YUV color space.
Median filter
Median filter is another standard filter for reducing noise in images and can be defined as: 𝐼 DEF (𝑥) = 𝑚𝑒𝑑𝑖𝑎𝑛 . N ∈, (𝐼 I7 (𝑥 I )) Median filter is used for removing a noise in image like Gaussian filter, but un-like it, it provides good results when removing impulse noise. Median filtering was applied on the Y channel after the images are converted to YUV color space.
Bilateral filter
Bilateral filter is a variation of Gaussian filter, which preserves edges while simultaneously removing noise in smooth areas. It is defined as: 𝐼 DEF (𝑥) = 1𝑊 H 𝐼 I7 (𝑥 I )𝐺 K$L (|𝑥 I − 𝑥|)𝐺 (|𝐼(𝑥 I ) − 𝐼(𝑥)|) . N ∈, with normalizing factor: 𝑊 = H 𝐺
K$L (|𝑥 I − 𝑥|) . N ∈, 𝐺 (|𝐼(𝑥 I ) − 𝐼(𝑥)|) where 𝐺 K$L is a Gaussian function with parameter 𝜎 K$L , and 𝐺 is a Gaussi-an function with parameter 𝜎 . As Gaussian filter, it sums the weighted intensi-ties of neighbor pixels, but unlike it, the weights do not depend only on the spacial distance of the weighted pixels in image space, but also on the different between intensities of these pixels in color space, allowing the filter to decrease the amount of blurring over edges. The implementation of bilateral filter is based on [23] and is applied to Y channel after the input image is converted to YUV color space. Histogram equalization
Histogram equalization technique is a method that maps intensities of pixels of the original image to different values to balance the histogram of the filtered im-age. This test uses an ordinary histogram equalization implementation from OpenCV. It was applied on the Y channel after the images were converted to YUV color space.
Contrast limited adaptive histogram equalization
Contrast limited adaptive histogram equalization (CLAHE) [24] is an adapta-tion of histogram equalization, which works with a histogram of a small window around each pixel and reduces the contrast of output image by clipping the highest values of the histogram. CLAHE is applied to Y channel after the images are con-verted to the YUV color space. CLAHE has a single parameter (clip limit), which influences the amount of values clipped in the histogram.
White balancing
White balancing algorithm changes the colors of input image to render white objects correctly under different illuminations like sun or clouded sky. In this test, we used an algorithm presented in [25]: foreach color channel do compute histogram of this channel; channel min ← black -th percentile of values in histogram; channel max ← white -th percentile of values in histogram; linearly transform all intensities so that channel min = 0 and channel max = 255; end The main advantages of this method include speed and simplicity. Though the re-stored image may not represent the colors of objects properly, due to the simplici-ty of the algorithm, this is not a problem, since the image is not presented to a viewer, but it is only processed by a marker detection algorithm. The algorithm was applied to all channels of RGB image before it is converted to YUV space for marker detection.
Debluring
The last filter used is a debluring filter (or unsharp mask filter [26]), which em-phasizes high frequencies in input image by subtracting its low frequencies from itself as shown below: 𝐼 DEF = (1 + 𝑤) ⋅ 𝐼 I7 − 𝑤 ⋅ 𝐺𝑎𝑢𝑠𝑠𝑖𝑎𝑛[𝐼 I7 , 𝜎 K$L \ where 𝑤 represents the weight of subtracted low frequencies, and 𝐺𝑎𝑢𝑠𝑠𝑖𝑎𝑛 [𝐼 I7 , 𝜎 K$L \ is a Gaussian filter with standard deviation 𝜎 K$L applied to the input image 𝐼 I7 . This method was implemented and applied on the Y channel after the image was transformed into the YUV color space. Results of enhancing underwater images
We experimented with different values of parameters for each technique and counted the number of newly found markers that were detected in the enhanced video and not detected in the original video, and a number of lost markers that were detected in the original video and not in the enhanced video. All image en-hancing algorithms were also compared between each other, counting the number of detected markers in images processed with one algorithm and not detected in images processed with the other algorithm. To reduce the number results to pre-sent, the following tables show the results of only the best parameters for each al-gorithm that obtain the highest number of newly detected markers while keeping the number of lost markers as low as possible. The following parameters were se-lected: • Gaussian filter, 𝜎 K$L = 0.4; • Median filter, window size = 3; • Bilateral filter, 𝜎 = 2.0 and 𝜎 K$L = 4.0; • CLAHE, clip limit 2; • White balancing, black / white percentile 2/98; • Debluring, 𝜎 K$L = 2.8 and weight = 1.9. The evaluation of the results is shown in Table 1 and demonstrates that the white balancing algorithm provides the best outcome, followed by debluring, then CLAHE, and BCP. Other algorithms provided similar or worse results then with the original image. Also, it is worth noting that Gaussian filter parameter 𝜎 K$L is very low, which indicates that high amount of blur worsens the results. Performance of the algorithms depend on the chosen parameters. Improper pa-rameters for CLAHE and white balancing algorithms make the results of detection worse than when detecting markers in the original images. It should be also noted that the best results are obtained when using very low values of parameters, e.g., in case of debluring, sufficiently good results are available even with smaller val-ues. This test was not focused on measuring the time to process the images, howev-er, we see that real-time algorithms CLAHE, white balancing, and debluring out- performed more sophisticated BCP. This shows that even a simple real-time algo-rithm can improve the detection of markers under water.
Summary of algorithms enhancing underwater images
We see that there is no algorithm that would be strictly better than other algo-rithms. Many algorithms were found to improve the detection of markers when, but in all cases, some markers from the original image were lost. Also, sophisti-cated offline solutions are not necessarily better than real-time general solutions. This test shows that the most promising algorithms are algorithms that improve sharpness of the image (debluring), colors of the image (white balancing), and its contrast (CLAHE). In the following section, these algorithms are tested into more depth with more videos and more marker detecting algorithms.
Algorithm Found Markers Number of found markers that were not found by following algorithms O r i g i n a l i m a g e F u s i on BC P G a u ss i a n f ilt e r M e d i a n f ilt e r B il a t e r a l f ilt e r H i s t . e qu a li za ti on C L AH E W h it e b a l a n c i ng D e b l u r i ng Original image 4811 0 231 35 57 312 30 625 43 28 20 Fusion 4611 31 0 16 26 152 27 457 17 9 8 BCP 5097 321 502 0 336 592 309 876 91 83 71 Gaussian filter 4795 41 210 34 0 298 48 612 40 24 19 Median filter 4515 16 56 10 18 0 13 395 13 7 7 Bilateral filter 4818 37 234 30 71 316 0 628 35 29 20 Histogram equalization 4238 52 84 17 55 118 48 0 20 14 12 CLAHE 5120 352 526 114 365 618 337 902 0 75 58 White balancing 5264 481 662 250 493 756 475 1040 219 0 116 Debluring 5249 458 646 223 473 741 451 1023 187 101 0
Table 1: Results of various image improving algorithms Methodology of the second test
The second test focuses on the image-enhancing algorithms that gave the best results in the first test and analyses them in additional underwater environments. Also, it adds the results of an improved version of which balancing algorithm, marker-based underwater white balancing (MBUWWB) [14] that is better adapted to underwater conditions. Additionally, it compares the results with AprilTag [27], a marker detecting algorithm that provides better results than ARUco at a higher detection time, as shown in [9].
Testing sites
Three additional sets of videos were tested, to the total number of four, includ-ing the video from the previous experiment, see Table 2. The first set of videos is the video tested in the previous section. As already noted, it was taken in the depth of approximately 7 to 9 meters in a moderately turbid environment with a GoPro camera with resolution of 1920 × × × Environment 1 Environment 2 Environment 3 Environment 4 Moderate turbidity High turbidity Moderate turbidity Low turbidity Depth 5 – 7 meters Depth 5 – 6 meters Depth 20–22 meters Depth 7 – 9 meters GoPro camera iPad Pro 9.7 GARMIN VIRB XE NVIDIA SHIELD 1920 × × × × Table 2: Four different testing sites were recorded using MPEG-4 compression, and decoded into RGB frames. This set of videos contains not only videos with the camera moving towards the mark-ers, but also videos with markers recorded from multiple directions and distances. The fourth set of videos consists of eight videos taken in the depth of 7 to 9 me-ters in a moderately turbid environment with a NVIDIA SHIELD tablet. We refer to this set as Environment 4. Two of these videos were recorded with resolution of 1920 × ×
720 and stored without using any com-pression as YUV frames. As with the third set of videos, this set of videos also contains not only videos with the camera moving towards the markers, but also videos with markers recorded from multiple directions and distances.
Tested algorithms
In addition to the best algorithms tested in the previous test, CLAHE, deblur-ring, and white balancing, this section contains also results of a fourth algorithm, marker-based underwater white balancing (MBUWWB), that is adapted for mark-er-based tracking under water [14]. This algorithm solves an intrinsic part of white balancing algorithms, which is finding the colors that are subsequently mapped to the white and black in the filtered image. White balancing algorithm define in the previous section chooses these colors as a percentile of values in input image his-togram, but MBUWWB assumes that the marker is black and white, and instead of computing the histogram of the whole image, it computes the histogram only of the part of the image which contains markers. In this experiment, MBUWWB is applied to all channels of RGB image, and afterwards, the improved is converted into YUV space, similarly as with the sim-ple white balancing. The images stored in YUV format are also converted to RGB before they were processed by MBUWWB.
Results in various environments
Image enhancing algorithms and marker detection algorithms were compared in two aspects: the total number of detected markers, and performance in different visibility conditions. Summed number of all markers found by ARUco in every set of enhanced vid-eos are illustrated in Table 3. They clearly demonstrate that all tested algorithms improve detection of markers in strongly turbid environments. In moderately tur-bid environments, the visibility is better, so the improvement in detection is not very apparent. The behavior of CLAHE algorithm is hard to predict. Using CLAHE for en-hancing images can lead to an improvement in the detection of markers (Environ-ment 1, Environment 2), but also it is able to decrease the number of detected markers (Environment 3, Environment 4). The result is also highly dependent on the value of clip limit. On the other hand, debluring, white balancing, and MBUWWB provide much stable results. The results of debluring clearly shows that using value 4 for subtraction weight w leads to higher number of detected markers. The results also illustrate that MBUWWB provided better overall results than original white balancing, although the difference is much lower in moderately tur-bid environments. Careful choice of percentile affects the performance of both these algorithms, especially in highly turbid environments. This difference is more visible in case of WB algorithm, while in case of MBUWWB, percentile 3/97 provided better or nearly the same results when compared to percentile 0/100. The results for AprilTag are in Table 4. Unlike ARUco, AprilTag algorithm is much more robust to environments with strong turbidity (a similar result was ob-served by [9]). The detection provides already very good results in original video, and the improvement obtained by enhancing the image is not very large, if any. Environment 1 Environment 2 Environment 3 Environment 4 Original video 5338 512 5847 9925 CLAHE, clip limit 2 5670 3904 5405 8227 CLAHE, clip limit 4 5541 4916 4801 7276 CLAHE, clip limit 6 5399 5010 4190 7057 Debluring, weight 1 5696 1852 6033 10105 Debluring, weight 4 5804 4622 6050 10372 White bal., perc. 0/100 5557 3603 6053 9988 White bal., perc. 3/97 5668 5159 5950 9558 MBUWWB, perc. 0/100 5781 5842 6094 10026 MBUWWB, perc. 3/97 5787 6351 6069 10085
Table 3: Total number of markers detected with ARUco in tested sets of videos en-hanced by tested algorithms.
Environment 1 Environment 2 Environment 3 Environment 4 Original video 5925 6884 5775 9141 CLAHE, clip limit 2 5868 7032 5552 8119 CLAHE, clip limit 4 5667 6679 4974 6655 CLAHE, clip limit 6 5381 6186 4362 6326 Debluring, weight 1 5850 6652 5781 9351 Debluring, weight 4 5474 5542 5847 9526 White bal., perc. 0/100 5922 6873 5648 9091 White bal., perc. 3/97 5479 6476 5597 8769 MBUWWB, perc. 0/100 5925 6914 5705 9067 MBUWWB, perc. 3/97 5887 6695 5697 8925
Table 4: Total number of markers detected with AprilTag in tested sets of videos en-hanced by tested algorithms. The best results in individual sets were obtained by CLAHE with clip parameter 2 (Environment 1) and debluring (Environment 3 and Environment 4), although these two algorithms provided much worse results on other videos. Unlike the results of ARUco, these results show that proper choice of subtrac-tion weight w in debluring algorithm is not easy to get the highest number of de-tected markers. It can be shown that in environments Environment 1 and Envi-ronment 2, using weight 1 lead to a higher number of detected markers than using weight 4, but in environments Environment 3 and Environment 4 the result is the opposite. The results also show that WB and MBUWWB were not improving the detec-tion, but they are also not making it worse. This is true only when we used percen-tile 0; the results when using percentile 3 were always worse. Camera moves towards and away from the markers in some videos of all sets. With decreasing distance, the marker gradually emerges from the turbid, which al-lows us to evaluate the behavior of image enhancing algorithms in different levels of marker visibility. We focused on ARUco detector and the video from Environment 1, and plotted the progress of the number of detected markers in time, see Figure 3. In the video, markers start partially in turbid, and as the camera moves towards them, the dis-tance where all the markers are visible is reached very soon. This experiment shows that in worse visibility conditions (first frames of the video), all image enhancing algorithms improves the detection of the markers, with debluring giving the best results, only slightly better than MBUWWB. Since
Figure 3: Number of markers detected by ARUco in a video from Environment 2. approximately 150 th frame, the visibility is good for all algorithms to find all mark-ers. Summary of algorithms performing in various environments
Results indicate the detection of markers can be improved by adding an image improving step, however, proper choice of the algorithm and its parameters is heavily dependent on the environment and the marker-detecting algorithm. If the marker-detecting algorithm is robust, there is no need to improve the input image, its “enhancement” may make the results even worse, but robust algorithms run very slowly. In the final section of this chapter, we will focus on ARUco marker detector to find, which parts are affected by bad visibility conditions.
Underwater marker-detecting algorithms
The last part of this chapter is focused on the structure of ARUco marker-detecting algorithm to identify parts that are affected by underwater conditions. It also presents results of a cultural heritage use case scenario from Baiae, Italy.
Structure of marker-detecting algorithm ARUco
The ARUco detector runs fast and reliably recognize markers. First, it thresh-olds the input image using an adaptive thresholding algorithm, then it finds all contours to detect marker-like shapes, and filters out non-square polygons. After that, it reprojects them to remove perspective distortion, obtains the inner binary marker code, compares it with a dictionary to remove errors, and if correct, it computes the relative position of the marker using its corners. Č ejka et al. in [15] investigates, which parts of ARUco are influenced by image improving algorithms and finds that the most vulnerable part is the initial thresh-olding. Figure 4 shows that when an underwater image is thresholded by ARUco, the border often breaks, and the marker is not recognized as a square-like object. This can be avoided by changing the parameters of the thresholding, but this step increases the number of small contours, which increases the processing time (a Figure 4: Left: input image taken under water; Middle: thresholded by ARUco, notice a broken contour; Right: the contour is solid with better parameters of ARUco, but the image contains a lot of noise similar method is used in the robust AprilTag algorithm). All four image-improving algorithms tested in previous sections increased the contrast of the im-age, with approximately the same output with a lesser number of false contours, and so they do not increase the detection time very much. Č ejka et al. in [15] presents an improved version of ARUco called UWARUco that is adapted to underwater environments. This algorithm creates a mask to re-move image parts that contain no contour, and applies it to an image thresholded with proper parameters. This approach provides results that are comparable with robust marker detecting algorithms, but their algorithm runs faster. Use case: Presenting submerged ancient buildings with AR
The last part of this chapter is focused on using marker-based augmented reali-ty to present virtual structures to divers that dive in Baiae in Italy at locations of ancient villas. The focus of the testing was limited to one building, Villa a Protiro, with a characteristic mosaic in one of the rooms. Using the improved approach, divers were able to perceive a 3D reconstruction of Villa a Protiro in AR. The test used nine markers from the ARUco DICT_6X6_50 dictionary, forming a grid of 3 × Figure 5: Left: nine markers placed at the location of a room with mosaic of Villa a Protiro in Baiae, Italy. Right: a virtual model superimposed in augmented reality at the place of these markers.
ARUco ARUco with MBUWWB UWARUco ARUco3 AprilTag
Table 5: Number of detected markers and computation time for various augmented reality solutions. and is less robust to underwater environment, which is clearly visible in the re-sults. Robust AprilTag detects high number of markers without the necessity of improving images, but its computation time is very high. Conclusion
This chapter described problems with detecting markers in marine underwater environments when targeting applications that use augmented reality to present additional information about cultural heritage sites and superimpose ancient virtu-al objects. First, it presented a comparison of general solutions that improve imag-es affected by bad visibility conditions under water. It was shown that there is no solution that would perform better than every other solution, and that more elabo-rated algorithms designed for underwater images do not provide strictly better re-sults than general solutions. Then, the chapter focused deeply on the most promis-ing solutions in multiple underwater environments. Here, the results showed that image improving step can improve the detection of markers if the parameters of the image improving algorithm are correctly set. Finally, it investigated the impact of underwater visibility conditions on a single marker detecting algorithm, and showed that a properly adapted algorithm can outperform general algorithms combined with image enhancing algorithms.
Acknowledgements
This research is a part of the i-MareCulture project (Advanced VR, iMmersive Serious Games and Augmented REality as Tools to Raise Awareness and Access to European Underwater CULTURal heritagE, Digital Heritage) that has received funding from the European Union’s Horizon 2020 research and innovation pro-gram under Grant Agreement No. 727153.
References [1]
V. Vlahakis, N. Ioannidis, J. Karigiannis, M. Tsotros, M. Gounaris, D. Stricker, T. Gleue, P. Daehne and L. Almeida, "Archeoguide: An Augmented Reality Guide for Archaeological Sites,"
IEEE Computer Graphics and Applications, vol. 22, pp. 52-60, 2002. [2] C. Panou, L. Ragia, D. Dimelli and K. Mania, "An Architecture for Mobile Outdoors Augmented Reality for Cultural Heritage,"
ISPRS International Journal of Geo-Information, vol. 7, no. 12, 2018. [3] D. Gallagher, "Development of Miniature, Head-Mounted, Virtual Image Displays for Navy Divers,"
OCEANS ’99, vol. 3, pp. 1098-1104, 1999. [4] A. Bellarbi, C. Domingues, S. Otmane, S. Benbelkacem and A. Dinis, "Augmented Reality for Underwater Activities with the Use of the DOLPHYN," pp. 409-412, 2013. [5] L. Oppermann, L. Blum, J.-Y. Lee and J.-H. Seo, "AREEF Multi-player Underwater Augmented Reality experience," pp. 199-202, 2013. [6] L. Oppermann, L. Blum and M. Shekow, "Playing on AREEF: Evaluation of an Underwater Augmented Reality Game for Kids,"
Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 330-340, 2016. [7] F. Bruno, A. Lagudi, L. Barbieri, M. Muzzupappa, G. Ritacco, A. Cozza, M. Cozza, R. Peluso, M. Lupia and G. Cario, "Virtual and Augmented Reality Tools to Improve the Exploitation of Underwater Archaeological Sites by Diver and Non-Diver Tourists,"
Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection: 6th International Conference, EuroMed, pp. 269-280, 2016. [8] F. Bruno, A. Lagudi, M. Muzzupappa, M. Lupia, G. Cario, L. Barbieri, S. Passaro and R. Saggiomo, "Project VISAS: Virtual and Augmented Exploitation of Submerged Archaeological Site - Overview and First Results,"
Marine Technology Society Journal, vol. 50, pp. 119-129, 2016. [9] D. B. d. S. Cesar, C. Gaudig, M. Fritsche, M. A. d. Reis and F. Kirchner, "An Evaluation of Artificial Fiducial Markers in Underwater Environments,"
OCEANS, pp. 1-6, 2015. [10] C. Ancuti and A. Codruta, "Effective Ccontrast-Based Dehazing for Robust Image Matching,"
IEEE Geoscience and Remote Sensing Letters, vol. 11, pp. 1871-1875, 2014. [11] Y. Gao, H. Li and S. Wen, "Restoration and Enhancement of Underwater Images Based on Bright Channel Prior,"
Mathematical Problems in Engineering, pp. 1-15, 2016. [12] D. Skarlatos, P. Agrafiotis, T. Balogh, F. Bruno, F. Castro, B. Petriaggi, S. Demesticha, A. Doulamis, P. Drap, A. Georgopoulos, F. Kikillos, P. Kyriakidis, F. Liarokapis, C. Poullis and S. Rizvic, "Project iMARECULTURE: Advanced VR, iMmersive Serious Games and Augmented REality as Tools to Raise Awareness and Access to European Underwater CULTURal heritagE.,"
Digital Heritage. Progress in Cultural Heritage, pp. 805-813, 2016. [13] M. Ž u ž i, J. Č ejka, F. Bruno, D. Skarlatos and F. Liarokapis, "Impact of Dehazing on Underwater Marker Detection for Augmented Reality," Frontiers in Robotics and AI, vol. 5, pp. 1-13, 2018. [14] J. Č ejka, M. Ž u ž i, P. Agrafiotis, D. Skarlatos, F. Bruno and F. Liarokapis, "Improving Marker-Based Tracking for Augmented Reality in Underwater Environments," Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage, pp. 21-30, 2018. [15] J. Č ejka, F. Bruno, D. Skarlatos and F. Liarokapis, "Detecting Square Markers in Underwater Environments," Remote Sensing, vol. 11, no. 4, 2019. [16] S. Garrido-Jurado, R. n. Salinas, F. Madrid-Cuevas and M. Marín-Jiménez, "Automatic Generation and Detection of Highly Reliable Fiducial Markers under Occlusion,"
Pattern Recognition, vol. 47, pp. 2280-2292, 2014. [17] C. Ancuti, C. O. Ancuti, T. Haber and P. Bekaert, "Enhancing Underwater Images and Videos by Fusion,"
Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 81-88, 2012. [18] Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu and M. S. Lew, "Deep learning for visual understanding: A review,"
Neurocomputing, pp. 27-48, 2016. [19] A. Voulodimos, N. Doulamis, A. Doulamis and E. Protopapadakis, "Deep Learning for Computer Vision: A Brief Review,"
Computational Intelligence and Neuroscience, pp. 1-13, 2018. [20] G. Buchsbaum, "A Spatial Processor Model for Object Colour Perception,"
Journal of the Franklin Institute, pp. 1-26, 1980. [21] K. He, J. Sun and X. Tang, "Single Image Haze Removal Using Dark Channel Prior,"
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp. 2341-2353, 2011. [22] K. He, J. Sun and X. Tang, "Guided Image Filtering,"
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, pp. 1397-1409, 2013. [23] C. Tomasi and R. Manduchi, "Bilateral Filtering for Gray and Color Images,"
International Conference on Computer Vision,
Computer Vision, Graphics, and Image Processing, vol. 39, pp. 355-368, 1987. [25] N. Limare, J. Lisani, J. Morel, A. Petro and C. Sbert, "Simplest Color Balance,"
Image Processing On Line,
IEEE Transactions on Image Processing, vol. 26, pp. 1496-1508, 2017. [27] J. Wang and E. Olson, "AprilTag 2: Efficient and Robust Fiducial Detection,"
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),