Automatic consecutive context perceived transformer GAN for serial sectioning image blind inpainting

Abstract

BACKGROUND AND OBJECTIVE\nSerial sectioning is the routine method in histology study. In order to restore the defective images in section stack, and overcome the limitations of manual annotation of broken areas, we developed a fully automatic approach for locating and restoring the defective image stack.\n\n\nMETHODS\nWe proposed a novel end-to-end framework named automatic consecutive context perceived transformer GAN (ACCP-GAN) for fully automatic serial sectioning image blind inpainting. The first stage network (auto-detection module) was designed to detect the broken areas and repair them roughly, then guided the second stage network (refined inpainting module) to generate these expected patches precisely; therefore, the segmentation part was integrated into restoring part. The transformer module (SPTransformer), based on self-attention mechanism, was introduced to make the refined inpainting module focus on the features from neighboring images to help in correcting inpainting results. Moreover, gated convolution was largely used to extract features from normal parts in the defective image. The framework was trained and validated on the N7 dataset (803 images), and the generalization ability of the model was tested on the E17 (701 images) and N5 (413 images) datasets, all of these images were collected for previous kidney study.\n\n\nRESULTS\nN7 dataset was divided into training, validation, and test sets with a ratio of 6:2:2. Our model performed well in broken areas segmentation with the accuracy\xa0=\xa00.9995. The final restoration got the best performance with FSIM\xa0=\xa00.9478, MS-SSIM\xa0=\xa00.9592, PSNR\xa0=\xa029.7903, VIF\xa0=\xa00.8543, and FID\xa0=\xa047.2252 compared to the popular inpainting methods. The model was further tested on E15 and N5 datasets, and the generalization ability was satisfying.\n\n\nCONCLUSIONS\nOur method could detect and restore the defective serial sectioning image stack automatically, even the broken patches were large on an individual image. The newly designed SPTransformer performed well in feature extraction. This method reduced the workload of manual annotation and improved the analysis or application of large scale sectioning image stack in histology research.