DefocusTracker: A modular toolbox for defocusing-based, single-camera, 3D particle tracking
UUP JORS software Latex paper template version 0.1
Software paper for submission to the Journal of Open Research Software
To complete this template, please replace the blue text with your own. The paper hasthree main sections: (1) Overview; (2) Availability; (3) Reuse potential.Please submit the completed paper to: [email protected] (1) OverviewTitle
DefocusTracker : A modular toolbox for defocusing-based, single-camera, 3D particletracking
Paper Authors
1. Barnkob, Rune;2. Rossi, Massimiliano;
Paper Author Roles and Affiliations
1. Heinz-Nixdorf-Chair of Biomedical Electronics, Department of Electrical and Com-puter Engineering, Technical University of Munich, Center for Translational CancerResearch (TranslaTUM), Munich, Germany2. Department of Physics, Technical University of Denmark, DTU Physics Building309, DK-2800 Kongens Lyngby, Denmark
Abstract
The need for single-camera 3D particle tracking methods is growing, among others,due to the increasing focus in biomedical research often relying on single-plane mi-croscopy imaging. Defocusing-based methods are ideal for a wide-spread use as theyrely on basic microscopy imaging rather than requiring additional non-standard optics.However, a wide-spread use has been limited by the lack of accessible and easy-to-usesoftware.
DefocusTracker is an open-source toolbox based on the universal principlesof General Defocusing Particle Tracking (GDPT) relying solely on a reference look-uptable and image recognition to connect a particle’s image and its respective out-of-plane depth coordinate. The toolbox is built in a modular fashion, allowing for easyaddition of new image recognition methods, while maintaining the same workflow andexternal user interface.
DefocusTracker is implemented in MATLAB, while a parallelimplementation in Python is in the preparation.
Keywords particle tracking; velocimetry; PTV; general defocusing particle tracking; fluid dynam-ics; MATLAB; Python a r X i v : . [ phy s i c s . f l u - dyn ] F e b P JORS software Latex paper template version 0.1
Introduction
The use of single-camera 3D particle tracking analysis is receiving increasing interest,among others, due to the rapid development of bio-engineering and biomedical sci-ences where single-access imaging, such as with microscopes, is a standard researchtool [15, 16]. For this, methods based on the principle of particle image defocusing areparticularly attractive as no special optics or cameras are required, and have potentialfor wide-spread use. However, until now most of the software for defocused-based par-ticle tracking has been developed in-house for private use of research groups, and thereare only few examples of user-friendly software that can be accessible to a larger au-dience, including researchers outside the engineering or computer-science community.One example is GDPTlab, a MATLAB GUI implementation written by the authorsand released in 2015 [5]. GDPTlab was used by few research groups and cited in sev-eral peer-reviewed journals (see also Reuse potential section). GDPTlab, however, wasnot distributed under an open-source license, thus its potential for collaboration andexpansion was limited by that.To accommodate this need, we developed
DefocusTracker , which is a modular andopen-source toolbox for defocusing-based 3D particle tracking.
DefocusTracker usesdifferent architecture and functions and is not compatible with GDPTlab, howeverthey are based on the same method, namely the General Defocusing Particle Tracking(GDPT). GDPT relies on a reference look-up table, with known defocused particleimages and depth positions, and on a image recognition method, that matches targetand reference particle images [1, 3, 15]. Following machine learning terminology, wewill refer here to the look-up table as the training set, whereas the image recognitionmethod will be part of the chosen calibration model. The GDPT principle is shownin Fig. 1. Panel (a) shows the creation and training of a calibration model throughcalibration/training images of defocused particle images with known 3D particle posi-tions. In Panel (b) the trained model is used to reconstruct the 3D particle positionsfrom the particles’ defocused images in 2D measurement images. For more details onthe GDPT method, experiments, and uncertainty assessment, we refer to Refs. 1, 3, 5.
DefocusTracker is built in a modular fashion to allow for a continuous addition ofstate-of-the-art image recognition methods, e.g. methods based on convolutional neuralnetworks and deep learning, while maintaining general method-independent features.The current implemented method, referred to as Method 1, is based on the normalizedcross-correlation function and described in Ref. 10.
DefocusTracker is implemented inMATLAB, while a parallel implementation in Python is in the preparation.
Defocus-Tracker is accompanied by a website, https://defocustracking.com/ , to facilitatethe research community with a platform for sharing of user guides, experiences, andapplications, as well as for data for training and validation.
Implementation and architectureGeneral architecture
P JORS software Latex paper template version 0.1
Defocused particle images(b) (a) Calibration images k n o w n z Defocus-Tracker Reconstruction of3D particle positions z ' ( µ m ) x' (µm) y' (µm) train() Defocus-Tracker(c) imageset model datasetimagesetdataset
Create postprocess()create() process() show()
Training set
Figure 1:
DefocusTracker working principle and general architecture. (a) A set ofcalibration images with known 3D particle positions are used to (b) determine theunknown 3D positions of particles through the comparison of their defocused particleimages. (c) General architecture and workflow of the
DefocusTracker toolbox. Thetoolbox uses three types of data structures (rectangles) and five main functions (roundshapes, green). The toolbox is modular, allowing for addition and use of differentmodels for the image processing and particle tracking.The general architecture and workflow is shown in Fig. 1(c). The toolbox is based onthree types of data structures and five primary functions for their creation, processing,and manipulation:Data structures: imageset
Link and description to a set of images, i.e. image paths, number ofimages, and image type. dataset
Particle data, i.e. 3D spatial positions and displacements, trajectories,connection to image frames, and detection accuracy estimation. model
Data and settings required to process the images, i.e. method-specific pa-rameters as well as training and processing settings.Functions: create()
Creates a data structure. show()
Opens GUIs to inspect data structures. train()
Trains a model on a specific training set ( imageset + dataset ).P JORS software Latex paper template version 0.1 process() Processes an imageset using a given model . postprocess() Manipulates a dataset , e.g. merge datasets, apply scaling, re-move outliers, perform particle tracking, filter trajectories, and estimate uncer-tainties.Workflow:As seen in Fig. 1(c), a typical workflow starts with the creation of a model using the create() function. Each model refers to a specific method (e.g. Method 1) and itis at first created with some default values. The model is trained using the train() function by feeding a training set as input. A training set consists of a dataset andan imageset (made with create() ), corresponding to a set of images containing oneor more particles of known 3D position. As illustrated in Fig. 1(a), such a trainingset can be obtained experimentally by taking subsequent images of particles displacedat known positions, e.g. by observing particles sedimented on a microchannel bottom,while taking images at known objective distances using a focusing stage. If only asubset of the calibration particles are used for a training, the remaining particles canbe used as validation to make a pre-measurement uncertainty estimation.With a trained model as input, the process() function can take one or more mea-surement images in an imageset and output a dataset containing the measured 3Dparticle positions. The dataset can be further manipulated via the postprocess() function for purposes such as outlier removal or trajectory smoothing. Throughoutthe entire workflow, the show() function can be used to visualize and inspect the datastructures.Modularity and Method 1:The
DefocusTracker toolbox is modular in the sense that a model can be createdbased on different methods for the image recognition. If a new method is added to thetoolbox, the data structures, primary functions, and workflow will remain the same.The current implementation provides only one method for the creation of a model ,namely Method 1. For a full description of Method 1, we refer to [10]. Briefly, Method1 is based on training images of a single particle (the so-called calibration stack) anduses the normalized cross-correlation function for image recognition. The normalizedcross-correlation is used to rate the similarity between a target particle image and thecalibration stack images, using its maximum peak value as the similarity coefficient,referred to as C m . The values of C m can range from 0 to 1, with 1 corresponding to aperfect match between the target image and a calibration image. MATLAB implementation
DefocusTracker is implemented in MATLAB and additionally requires the image pro-cessing, curve fitting, and statistics toolboxes. The implementation is script-based withcertain features using GUI-based pop-up windows for visualization. The data struc-tures are so-called MATLAB structs, namely structure arrays where data is grouped us-ing containers called fields. The data in a field is accessed using dot notation of the formP JORS software Latex paper template version 0.1Figure 2: Overview of data structures and functions in the
DefocusTracker
MATLABimplementation, following the general toolbox architecture shown in Fig. 1. structName.fieldName , e.g. the path of an imageset is called with imageset.path .The primary toolbox functions follow the form of standard MATLAB functions and arenamed as dtracker functionName() , e.g. dtracker create() . A full overview of thedata structures and functions are shown in Fig. 2. An exemplary practical applicationof
DefocusTracker , including a script and few results, is presented in Fig. 3. Moredetails about this application are given in the Quality control section.
Python implementation
A Python implementation of
DefocusTracker is planned and under development andit will the lines of the MATLAB implementation. The data structures will be im-plemented using Python dictionaries, whereas the functions will be part of the mod-ule dtracker . Following the above example, the path of an imageset will be calledin Python with imageset[’path’] , whereas the create() function will be called as dtracker.create() .The Python implementation is available on the Gitlab repository: https://gitlab.com/defocustracking/defocustracker-python . Updates and release informationcan be followed on https://defocustracking.com .P JORS software Latex paper template version 0.1 (b) (a) >> dtracker_show(model) >> dtracker_show(images_meas,data_meas)>> dtracker_show(data_train_test)>> data_meas
Figure 3: Example workflow of the toolbox MATLAB implementation. The exampleworkflow is based on part of the provided Work-Through Example 2 that takes theuser through the processing and analysis of particle trajectories inside an evaporatingdroplet [12]. (a) Illustration of the code used to create and train a model (greenframe), to process image test data (blue frame), and to process measurement images(red frame). (b) Illustration of the implemented pop-up GUIs used to visualize andinspect the data structures.
Quality control
The MATLAB toolbox has been tested functionally on Windows 10 with MATLABreleases R2018b and 2020a. The toolbox performance has been tested and investigatedextensively in two recent publications [3, 10]. In Barnkob and Rossi [3], guidelines forassessing the uncertainty of GDPT analyses were given. Synthetic images were used totest the toolbox in terms of measurement uncertainty and relative number of measuredparticles as a function of image signal-to-noise ratio, particle image concentration, andvariations in image intensity. In Rossi and Barnkob [10], different toolbox settingswere tested on synthetic and experimental images to outline the measurement uncer-tainties, detection rates, and processing times. The results were benchmarked againstthe GDPTlab software [5], which has been extensively-tested and used in high-impactresearch publications, see more in the Section Reuse potential.The MATLAB toolbox contains two work-through examples (WTE1 and WTE2)that serve as tutorials to get new users quickly started, but also as test scripts in casenew functionalities or methods are added. The scripts of each WTE are included inP JORS software Latex paper template version 0.1the
DefocusTracker package, while the relative datasets can be acquired via https://defocustracking.com/datasets/ . WTE1 is based on synthetic images and givesa first introduction to the basic building blocks of the toolbox. The use of syntheticimages allow for an exact estimation of the uncertainty using the postprocessing method ’compare true values’ . WTE2 is based on a state-of-the art microfluidic experiment,namely the 3D flow inside an evaporating droplet [12], and guides the user toward amore advanced use of
DefocusTracker , including postprocessing and bias correction.We report in Fig. 3 a shortened version of the WTE2 script, including few screenshotsof GUI panels obtained with the dtracker show() function. For the full commentedversion we refer to the script
Work through ex2.m . As the community grows, we expectthat more WTEs will be added by the users and developers. (2) AvailabilityOperating system
Windows, UNIX/Linux, Macintosh (and any operating system supporting MATLAB).
Programming language
MATLAB 9.4.0 (R2018a), upward compatible.
Additional system requirements
N/A
Dependencies
The MATLAB implementation requires the additional MATLAB toolboxes:‘curve fitting toolbox‘, ‘image toolbox‘, ‘statistics toolbox‘
List of contributors
N/A
Software location:ArchiveName:
Gitlab (master)
Persistent identifier: https://gitlab.com/defocustracking/defocustracker-matlab/-/releases/v1.0.2
Licence:
MIT
Publisher:
Massimiliano Rossi and Rune Barnkob
Version published:
Date published:
Code repositoryName:
Gitlab (develop)
Persistent identifier: https://gitlab.com/defocustracking/defocustracker-matlab
Licence:
MIT
Date published:
Language
English (3) Reuse potential
The analysis of particle positions, velocities, and trajectories is an integral part of manyresearch disciplines. This includes analyses in 2D as well as in 3D, and with the rapidgrowth in fields relying on microscopy, such single-camera methods can provide uniqueand important information. One example is within the field of microfluidics where highcontrol of fluid flow and externally-applied forces is becoming an important tool inbiomedical research and applications. Here, 3D detection and tracking of particles andcells can provide the necessary information needed for optimization, standardization,and real-time inspection and control [13, 14].GDPT has shown to be an excellent candidate for a wide-spread technique as isa simple and universal defocusing-based method and requires no special optics andcan be used in standard microscope setups. Here, the development of free, accessible,user-friendly, and accurate tools can greatly enhance the practicability and availabilityof the method. One example is the MATLAB implementation GDPTlab (also by theauthors), which has been distributed to researchers since 2015 and has proven its valuein a number of research projects including work in journals such as Proceedings of theNational Academy of Sciences, Physical Review Letters, and Scientific Reports [2, 4, 6–9, 11, 16–18]. Note that most of this research were done in laboratories with no previousexperience of 3D single-camera particle tracking prior to the use of GDPTlab. GDPT-lab has thus shown the huge potential such methods hold, if free and user-friendlyimplementations are available. Though GDPTlab has enabled many researchers to getstarted using GDPT, it has unfortunately not been fully accessible as an open-sourceproject, limiting its further development and adaptation by the community.
Defocus-Tracker fills this need as it is fully open-source. The toolbox contains a readme-fileand work-through examples for users to get quickly started.
Modification and support
DefocusTracker is set up in a versatile and modular fashion allowing for easy expan-sions and improvements, such as extensions of custom functionalities and features, e.g.using MATLAB’s GUI editor and pre-built functions. In the Python implementation,such expansions could involve the use of popular libraries for data analysis and ma-chine learning, such as SciKit, Keras, and TensorFlow.
DefocusTracker is supportedby https://defocustracking.com/ which is an online platform created to assist thedevelopment and support of
DefocusTracker as well as to facilitate the research com-munity with a place for sharing of data and experiences related to single-camera 3Dparticle tracking. The platform contains several forums, e.g. for new developers torequest access and for users to ask the community for support.
Funding statement
The research leading to these results has received funding from the European Union’sHorizon 2020 research and innovation programme under the Marie Sklodowska-Curie
EFERENCES
UP JORS software Latex paper template version 0.1grant agreement no. 713683 (COFUNDfellows-DTU).
Competing interests
The authors declare that they have no competing interests.
References [1] Barnkob, R., K¨ahler, C. J. and Rossi, M. [2015], ‘General defocusing particletracking’,
Lab on a Chip (17), 3556–3560. URL: http: // dx. doi. org/ 10. 1039/ C5LC00562K [2] Barnkob, R., Nama, N., Ren, L., Huang, T. J., Costanzo, F. and K¨ahler, C. J.[2018], ‘Acoustically driven fluid and particle motion in confined and leaky sys-tems’,
Physical Review Applied (1), 014027. URL: https: // doi. org/ 10. 1103/ PhysRevApplied. 9. 014027 [3] Barnkob, R. and Rossi, M. [2020], ‘General defocusing particle tracking: funda-mentals and uncertainty assessment’,
Experiments in Fluids , 1–14. URL: https: // doi. org/ 10. 1007/ s00348-020-2937-5 [4] Bod´e, W. N., Jiang, L., Laurell, T. and Bruus, H. [2020], ‘Microparticleacoustophoresis in aluminum-based acoustofluidic devices with pdms covers’,
Mi-cromachines (3), 292. URL: https: // doi. org/ 10. 3390/ mi11030292 [5]
GDTPlab — how to get it, Institut f¨ur Str¨omungs-mechanik und Aerody-namik, Univerist¨at der Bundeswher, Werner Heisenberg Weg 39, 85577 Neu-biberg, Germany [n.d.], . Accessed: 3 June 2020.[6] Karlsen, J. T., Qiu, W., Augustsson, P. and Bruus, H. [2018], ‘Acousticstreaming and its suppression in inhomogeneous fluids’,
Physical review letters (5), 054501.
URL: https: // doi. org/ 10. 1103/ PhysRevLett. 120. 054501 [7] Liu, J., Reisbeck, M. and Hayden, O. [2019], ‘Investigation of mechanical andmagnetophoretic focusing for magnetic flow cytometry’,
Current Directions inBiomedical Engineering (1), 353–355. URL: https: // doi. org/ 10. 1515/ cdbme-2019-0089 [8] Qiu, W., Bruus, H. and Augustsson, P. [2020], ‘Particle-size-dependentacoustophoretic motion and depletion of micro-and nano-particles at longtimescales’,
Physical Review E (1), 013108.
URL: https: // doi. org/ 10. 1103/ PhysRevE. 102. 013108
EFERENCES
UP JORS software Latex paper template version 0.1[9] Qiu, W., Karlsen, J. T., Bruus, H. and Augustsson, P. [2019], ‘Experimentalcharacterization of acoustic streaming in gradients of density and compressibility’,
Physical Review Applied (2), 024018. URL: https: // doi. org/ 10. 1103/ PhysRevApplied. 11. 024018 [10] Rossi, M. and Barnkob, R. [2020], ‘A fast and robust algorithm for general defo-cusing particle tracking’,
Meas. Sci. Technol, in print .[11] Rossi, M., Cicconofri, G., Beran, A., Noselli, G. and DeSimone, A. [2017], ‘Kine-matics of flagellar swimming in euglena gracilis: Helical trajectories and flagellarshapes’,
Proceedings of the National Academy of Sciences (50), 13085–13090.
URL: https: // doi. org/ 10. 1073/ pnas. 1708064114 [12] Rossi, M., Marin, A. and K¨ahler, C. J. [2019], ‘Interfacial flows in sessile evapo-rating droplets of mineral water’,
Physical Review E (3), 033103.
URL: https: // doi. org/ 10. 1103/ PhysRevE. 100. 033103 [13] Sapudom, J., Waschke, J., Franke, K., Hlawitschka, M. and Pompe, T. [2017],‘Quantitative label-free single cell tracking in 3d biomimetic matrices’,
Scientificreports (1), 1–9. URL: https: // doi. org/ 10. 1038/ s41598-017-14458-x [14] Sitters, G., Kamsma, D., Thalhammer, G., Ritsch-Marte, M., Peterman, E. J. andWuite, G. J. [2015], ‘Acoustic force spectroscopy’,
Nature methods (1), 47–50. URL: https: // doi. org/ 10. 1038/ nmeth. 3183 [15] Taute, K., Gude, S., Tans, S. and Shimizu, T. [2015], ‘High-throughput 3D track-ing of bacteria on a standard phase contrast microscope’,
Nature communications , 8776. URL: https: // doi. org/ 10. 1038/ ncomms9776 [16] Van Assche, D., Reithuber, E., Qiu, W., Laurell, T., Henriques-Normark, B.,Mellroth, P., Ohlsson, P. and Augustsson, P. [2020], ‘Gradient acoustic focusingof sub-micron particles for separation of bacteria from blood lysate’,
Scientificreports (1), 1–13. URL: https: // doi. org/ 10. 1038/ s41598-020-60338-2 [17] Volk, A. and K¨ahler, C. J. [2018], ‘Size control of sessile microbubbles for repro-ducibly driven acoustic streaming’,
Physical Review Applied (5), 054015. URL: https: // doi. org/ 10. 1103/ PhysRevApplied. 9. 054015 [18] Westerbeek, E. Y., Bomer, J., Olthuis, W., Eijkel, J. C. and De Malsche, W.[2020], ‘Reduction of taylor-aris dispersion by lateral mixing for chromatographicapplications’,
Lab on a Chip . URL: https: // doi. org/ 10. 1039/ D0LC00773K
EFERENCES
UP JORS software Latex paper template version 0.1