Just-Enough Interaction Approach to Knee MRI Segmentation: Data from the Osteoarthritis Initiative
JJust-Enough Interaction Approach to Knee MRISegmentation: Data from the OsteoarthritisInitiative
Satyananda Kashyap, Honghai Zhang, and Milan Sonka
Iowa Institute for Biomedical Imaging, The University of Iowa, Iowa City IA, USA [email protected]
Abstract.
State-of-the-art automated segmentation algorithms are not100% accurate especially when segmenting difficult to interpret datasetslike those with severe osteoarthritis (OA). We present a novel interac-tive method called just-enough interaction (JEI), which adds a fast cor-rection step to the automated layered optimal graph segmentation ofmultiple objects and surfaces (LOGISMOS). After LOGISMOS segmen-tation in knee MRI, the JEI user interaction does not modify boundarysurfaces of the bones and cartilages directly. Local costs of underlyinggraph nodes are modified instead and the graph is re-optimized, provid-ing globally optimal corrected results. Significant performance improve-ment ( p (cid:28) . Keywords: just-enough interaction, LOGISMOS, graph search, KneeMRI,Osteoarthritis
Osteoarthritis (OA) is one of the most widely afflicted diseases among the agingpopulation with no interventional drugs available [1]. The current standard carefor the disease is prescription of analgesics and using joint braces to ease thepain with worsening of OA eventually leading to total knee replacement. MRIas a non-invasive bio-marker has potential to detect the structural changes earlyto help predict the onset of OA and monitor the effects of treatments in drugtrials. For any analysis the crucial first step is the segmentation of the bonesand cartilages of the femur and tibia. Manual segmentation of a MR volumetakes several hours and is subjected to inter and intra observer biases. Severalautomated algorithms exist, which are robust for most of the cases but cannotbe relied upon for subjects with severe pathology. With disease progression,automated algorithms face a challenging problem of delineating the bones andcartilages in presence of bone marrow lesions, cartilage surface thinning, mesicalextrusion and synovial fluid leakage. Many of these artifacts are symptoms of thedisease and appear similar in texture and intensity to cartilage on MR volumes. a r X i v : . [ c s . C V ] M a r Kashyap et. al
Interactive correction methods are designed to help ease the post-processingneeded. Several such techniques have been proposed in the literature such as thin-plate splines [2], active shape models for interactions [3] and live-wires [4]. Live-wires have embedded user interactions that require a seed based initializationwhich is user inputted or based on some initialization algorithm. However theyare inherently 2D mechanisms which were later extended to 3D with a drawbackof being unable to maintain global optimality for multiple surfaces and objects.Several of the above mentioned methods correct for segmentation inaccuracies bydirectly matching the object boundaries with the interaction which after severalrepetitions results in the final surfaces having local topological errors.The proposed interaction algorithm uses a graph based LOGISMOS frame-work [5,6] with the user-clicked points hereafter called nudge points interactingdirectly with the underlying graph framework. This method has guarantees ofglobal optimality for every interaction and differs from the traditional voxel-by-voxel editing by requiring just enough (i.e., limited) interaction (JEI) to correctthe original automated segmentation if needed. The proposed method may ap-pear similar to the Boykov’s graph cut [7], however their interaction algorithmis not able to guarantee global optimality for multiple surfaces and objects.LOGISMOS-JEI always guarantees global optimality when handling multipleobjects and surfaces. The JEI architecture and GUI are designed to be platformand application agnostic and their details are covered in [8]. The algorithm isextended to handle longitudinal JEI (multi-3D, or 4D) which enables correctionof multiple time-points of the single patient sequence. It is based on an existingautomated 4D LOGISMOS framework [9]. The GUI was also extended to en-able simultaneous examination and interaction using all the time-points givinga progressive view of cartilage losses.
The JEI method starts with an initial automated LOGISMOS segmentation.Post segmentation, the resulting optimized graph state (called residual graph)is saved for the purposes of JEI. The JEI algorithm was extended from [10]to handle multi-surface multiple object interactions. Further, a new interactionmechanism was developed along with a faster graph optimization library toprovide almost immediate feedback on the interaction.
The automated LOGISMOS algorithm was detailed in [6]. The algorithm identi-fied the volumes of interest (VOI’s) which pertain to a smaller regions around thefemur and tibia. This VOI identification was done using an Adaboost classifierwhich was trained on example VOI’s using 3D Haar-like features. The VOI de-tection helped reduce the computation time by operating on the smaller region.Further the VOI bounds were used for affine fitting the mean shape mesh S forthe femur and tibia bone respectively. Since a patient specific bone shape affects EI Approach to Knee MRI Segmentation 3 the final segmentation, a single surface single object LOGISMOS segmentationwas computed using S .Each surface of the object to be segmented had a geometric graph con-structed. A set of graph nodes were built up as a column from the patient specificmesh surface S which represented the set of locations for the final surfaces. Ev-ery node was associated with either a bone or a cartilage surface cost. The bonecosts along the column direction were 1D derivative operator ∇ ( x, y, z ). Sim-ilarly the cartilage costs was an empirically weighted ( w ) combination of firstand second order derivatives given by w ∗ ∇ ( x, y, z ) + (1 − w ) ∗ ∇ ( x, y, z ). Toensure topologically correct segmentation non-intersecting geometric graph con-struction is crucial. This was enforced by replicating the column construction asnon-intersecting electric lines of force with the mesh surface points denoting thepositively charged particles. The surface segmentation task was solved by rep-resenting the problem as a max-flow optimization which was accomplished byconnecting intra and inter-columns arcs to enforce the smoothness constraints.Further inter-object and inter-surface constraints were constructed for the finalmulti-surface multi-object segmentation.For the 4D automated multi-object multi-surface multi-3D segmentation [9],inter-time-point graph arcs were introduced which helped establish the maximumand minimum allowable changes between the time-points to be within physio-logically permissible ranges. The inter-time point arcs longitudinally linked thecorresponding column positions of the bones and the cartilages of the femur andtibia temporally. The electric lines of force (ELF) based geometric graph, image volume and resid-ual graph were loaded into the GUI (see Fig. 1) to inspect segmentation qualityand perform JEI. The work-flow is presented as a video provided in the supple-mentary material with an example subject with severe OA. The details of thework-flow were as follows:1.
User provided nudge points:
The user identified correction is providedas a set of nudge points which guide the segmentation to the correct posi-tion. Fig. 3a shows the GUI magnified with the volume and the automatedLOGISMOS segmentation results overlayed. The particular slice indicatedis a case with severe OA having bright fluid regions improperly segmentedas cartilage. The blue line with points are the nudge points indicated by theuser approximately identifying the correct cartilage region.2.
3D local graph cost modification:
To identify the underlying graphcolumns influenced by the nudge points (defined as a contour), a k -dimensionaltree algorithm is used which stores all the geometric graphs positions. In pre-vious JEI applications [10] the graph was constructed on a regular 3D gridwhere the nearest graph columns could be identified quite easily. Howevergiven the complex shape of the knee objects and the ELF graph constructedbased on it, a more sophisticated query of the closest columns is needed which Kashyap et. al does not compromise on speed. The k D tree allows for a O ( log n ) query onthe N nearest graph nodes (empirically determined) for every nudge point.Once identified the costs (i.e. unlikeliness) at corresponding columns associ-ated with these nodes are modified as c ( i, j ) = (cid:40) , if D (( i, j ) , n ( i, j )) < ∆ , otherwise , with c ( i, j ) defined as the cost of node j on column i , D (( i, j ) , n ( i, j )) thedistance between node closest to the nudge point ( i, j ) and its nearby inter-secting nodes n ( i, j ) within the ∆ tolerance.3. Max-flow re-computation:
Following the local graph cost modificationthe max-flow is recomputed in 3D within a few milliseconds and the updatedsurfaces rendered onto the GUI. As seen in Fig. 3b the correction made bythe nudge points are reflected in the updated cartilage surface overlayed onthe image volume.The above work-flow is repeated to correct the tibial cartilage errors as well.In the intermediate steps following the correction of the femur, the tibia boneand cartilage surfaces appear to worsen. This can be attributed to a combinationof the existing graph costs and the graph constraints. Since the tibia cartilagesurface has no clear defined edge cost in that region, the surface result movedalong with the femur corrected cartilage surface. Subsequently due to the inter-surface distance constraints between the tibial surfaces the tibial bone surfacealso changed. However once the nudge points provided the appropriate locationsfor cost modification the erroneous surfaces were corrected (Figs. 3c,d). Note thatthe corrections made on a single 2D slice resulted in the entire locally affected3D neighborhood being corrected. This can be appreciated in the correspondingcircled regions of the surface model.
Undo-Redo Interaction Capabilities
For a more consistent segmentationinteraction we designed into the GUI the ability to save the user interactionsteps. A stack was used to save the inputted nudge points and the surface ID foreach editing step. The editing was reverted by popping the stack which restoredthe previous costs on the local graph columns. Re-optimizing the graph resultedin the previous surfaces. However the popped stack was not deleted unless adifferent interaction was continued after being reverted. If the interaction neededto be redone, the pointer simply moved to the previous position on the stackand repeated the same steps as above to redo the correction.When a new automated LOGISMOS surface is loaded that was previouslyedited the user can load the editing stack to bring the interaction to most up-todate edited state. The edits can be continued from that point. We anticipatethat this feature would be very useful in reducing the inter-observer and intra-observer variability. A video demonstration of this feature is provided in thesupplementary material. http://bit.ly/2blYXFzEI Approach to Knee MRI Segmentation 5 Fig. 1.
The graphical user interface for 3D JEI with the image volumes and the surfacemeshes overlayed.
The longitudinal JEI GUI was extended to enable visualization of all patienttime-points simultaneously. The viewer also enabled synchronized scrolling acrossdatasets. Fig. 2a shows eight time-points of the same patient (baseline, 12, 18,24, 36, 48, 72 and 96 month follow-ups) being simultaneously visualized. Eachindividual thumbnail view can be expanded (see Fig. 2b) to a detailed largerGUI (identical to 3D GUI) for interaction.The interaction mechanism is similar to the 3D JEI where a set of nudgepoints on a single 2D slice modifies the graph node costs in the local 3D neigh-borhood columns of the given time-point. Further since the longitudinal JEI hashas a single large underlying residual graph with temporal inter-time-point con-straints the corresponding local 3D neighborhood column locations at the othertime-points are also corrected.
MRI volumes used in this study were acquired from the osteoarthritis initiativewhich also had a limited number of datasets with independent standard available.All subjects were scanned using the DESS protocol with a voxel resolution of0 . × . × . . 19 baseline subjects with varying degrees of OA severitywere used in this study. They were segmented using the automated LOGISMOSfollowed by 3D JEI correction. The geometric graph for the tibia and femur Kashyap et. al (a) (b)
Fig. 2.
4D LOGISMOS-JEI. (a) Longitudinal JEI viewer screen-shot showing a thumb-nail of eight time-points of a single patient simultaneously. (b) Smaller editing windowfor each 3D time-point. objects had 8006 and 8002 graph columns, respectively. The graph parametersused in this experiment are listed in Table 1.
Table 1.
Parameters used for graph construction. Minimum inter-surface inter-objectand inter-time-point separations are zero. Note that the inter-time point constraintswere only used in the longitudinal JEI.
Inter-surface Inter-object Inter-time point Smoothness Column size Node spacingmax (nodes) max (nodes) max (nodes) (nodes) (nodes) (mm)20 60 5 2 61 0.20
The surface positioning errors of the automated LOGISMOS and the JEI cor-rected surfaces were computed against the independent standard which weremanually traced cartilage and bone borders of the knee joint provided by theOAI. The bone surface segmentation results were very robust. The presented re-sults focused on the cartilage surface. Table 2 shows the 3D JEI versus automated
EI Approach to Knee MRI Segmentation 7
LOGISMOS for 19 baseline datasets. The signed errors for the JEI-corrected sur-faces were close to zero and were significantly smaller ( p (cid:28) . p (cid:28) . Table 2.
Surface positioning errors (signed, unsigned, in mm) of 3D JEI-correctedversus automated LOGISMOS segmentation. Bold entries mark statistical significantlybetter performance of the pairwise comparisons.
JEI-Corrected Automated p-value JEI-Corrected Automated p-valueFemur Signed -0.03 ± -0.37 ± (cid:28) .
001 Femur Unsigned ± ± (cid:28) . ± ± ± ± (cid:28) .
3D JEI and longitudinal JEI methods for knee MRI were presented. The interac-tive corrections for 3D JEI took on an average 15 min per dataset in comparisonto several hours of effort that are needed for traditional voxel-by-voxel edit-ing. The surface positioning errors of JEI-corrected results showed significantimprovements over the automated LOGISMOS.Another example application, not reported above, used the JEI-correctedsurfaces to train machine learning classifiers to identify cartilage regions in thepresence of pathology. For the fully automated LOGISMOS, using JEI-trainedclassifiers yielded significantly better performance than employing simple costfunctions. The learning based method will be presented in [11]. Further quanti-tative analysis of longitudinal JEI, comparing, and quantifying the inter/intraobserver bias and validating JEI corrected results in a larger clinical study areplanned for future work.
Acknowledgments
This research was supported by NIH grant R01-EB-004640. The OAI is a public-private partnership comprised of five contracts (N01-AR-2-2258; N01-AR-2-2259;N01-AR-2-2260; N01-AR-2-2261; N01-AR-2-2262) funded by the National Insti-tutes of Health, a branch of the Department of Health and Human Services, andconducted by the OAI Study Investigators. Private funding partners includeMerck Research Laboratories; Novartis Pharmaceuticals Corporation, Glaxo-SmithKline; and Pfizer, Inc. Private sector funding for the OAI is managedby the Foundation for the National Institutes of Health. This manuscript wasprepared using an OAI public use data set and does not necessarily reflect the
Kashyap et. al opinions or views of the OAI investigators, the NIH, or the private funding part-ners
References