BrainPainter: A software for the visualisation of brain structures, biomarkers and associated pathological processes
Razvan V. Marinescu, Arman Eshaghi, Daniel C. Alexander, Polina Golland
BBrainPainter: A software for the visualisation of brain structures, biomarkersand associated pathological processes
R˘azvan V. Marinescu a,b , Arman Eshaghi b,c , Daniel C. Alexander b, ∗ , Polina Golland a, ∗ a Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, USA, MA 02139 b Centre for Medical Image Computing, University College London, Gower Street, London, United Kingdom, WC1E 6BT c Queen Square MS Centre, UCL Institute of Neurology, UK
User-defined color gradient0 1 2 3INPUTS
Biomarkers Hippocampus Inferior Superior ...(.csv file) temporal parietal ...Brain 1 0.6 2.3 1.3 ..Brain 2 1.2 0.0 3.0 ..... ...
OUTPUTSBrain 1Brain 2BrainPainter
Figure 1:
Given a .csv file with region-of-interest (ROI) biomarkers and a user-defined color gradient, BrainPainter canautomatically generate brain images with the cortical surface (left and middle) as well as with subcortical structures (right).The input .csv file can have multiple rows, one for each set of output images. The color gradient is a list of RGB colours givenby the user. Final colours are interpolated using the numbers from the input .csv file based on the color gradient – e.g. if thehippocampus has a value of 1.2, its final color will be an interpolation of colors 1 (yellow) and 2 (orange) from the gradient.
Abstract
We present BrainPainter, a software that automatically generates images of highlighted brain structures given a list of numberscorresponding to the output colours of each region. Compared to existing visualisation software (i.e. Freesurfer, SPM, 3DSlicer), BrainPainter has three key advantages: (1) it does not require the input data to be in a specialised format, allowingBrainPainter to be used in combination with any neuroimaging analysis tools, (2) it can visualise both cortical and subcorticalstructures and (3) it can be used to generate movies showing dynamic processes, e.g. propagation of pathology on the brain.We highlight three use cases where BrainPainter was used in existing neuroimaging studies: (1) visualisation of the degreeof atrophy through interpolation along a user-defined gradient of colours, (2) visualisation of the progression of pathologyin Alzheimer’s disease as well as (3) visualisation of pathology in subcortical regions in Huntington’s disease. Moreover,through the design of BrainPainter we demonstrate the possibility of using a powerful 3D computer graphics engine such asBlender to generate brain visualisations for the neuroscience community. Blender’s capabilities, e.g. particle simulations,motion graphics, UV unwrapping, raster graphics editing, raytracing and illumination e ff ects, open a wealth of possibilitiesfor brain visualisation not available in current neuroimaging software. BrainPainter is customisable, easy to use, and canrun straight from the web browser: https://brainpainter.csail.mit.edu , as well as from source-code packaged in adocker container: https://github.com/mrazvan22/brain-coloring . It can be used to visualise biomarker data fromany brain imaging modality, or simply to highlight a particular brain structure for e.g. anatomy courses. ∗ Joint senior authors with equal contribution
Email address: [email protected]
1. Introduction E ffi cient visualisation of brain structure, function andpathology is crucial for understanding the mechanisms under- a r X i v : . [ c s . G R ] A ug ying neurodegenerative diseases and eases the interpretationof results in neuroimaging studies. This is especially impor-tant in populations studies, where two or more populations arecompared for group di ff erences in biomarkers derived frome.g. Magnetic Resonance Imaging, Positron Emission To-mography (PET) or Computer Tomography (CT). The resultsare best visualised as brain images, where regions-of-interest(ROIs) are highlighted based on the magnitude of the di ff er-ence between the two groups. These visualisation are gener-ally done by the same software that performs the registration,segmentation and statistical analysis. However, for traumaticbrain injury or less common neurodegenerative diseases suchas Parkinson’s disease and Multiple Sclerosis, visualisationsof statistical results is sometimes not performed due to theinability to register images to a common template or lack ofrobust registration software. Therefore, many studies such as(Coughlin et al. (2015); Schoonheim et al. (2012)) only reportdi ff erences between patients and controls in tables or as boxplots. There is therefore a lack of visualisation tools that canhighlight neuroimaging findings for these complex diseases.When registration to a common population template is pos-sible, e.g. in Alzheimer’s disease (AD), excellent 3D vi-sualisation software exists which allows interactive visual-isation of population di ff erences – e.g. 3D Slicer (Pieperet al. (2004)), Freesurfer (Fischl (2012)) or SPM (Penny et al.(2011)). However, they have several inherent limitations.First, such software – e.g. Freesurfer – generally require in-puts in their proprietary data format, which is usually di ffi cultand time-consuming to create without using their pipeline.While creating these proprietary data formats is necessarywhen users need to display voxelwise visualisations, oftenusers only need to highlight entire ROIs – in this simpler casethe user could only provide a list of RGB colors for each ROIin a csv file, removing the need to create input data in a spe-cialised format. Another limitation of existing visualisationsoftware is their di ffi culty in highlighting complex patternsof pathology in a single slice of a 3D volumetric image. Toovercome this, some authors show multiple slices (sometimesup to 8 slices, e.g. Migliaccio et al. (2015)), although thistakes too much space on the academic paper being published.While Freesurfer solves this problem using a cortical surface-based representation that captures most of the complexity ofpathology patterns in a single image, this surface represen-tation is not supported for subcortical structures such as thehippocampus. Third, current visualisation software cannot beeasily used to generate e.g. a movie showing a dynamic pro-cess, e.g. propagation of pathology within the human brain,as most of them have been intended for interactive visuali-sation and have no application programming interface (API)that allows automatic generation of hundreds of images usingpre-defined settings.We present BrainPainter, a software for easy visualisationof structures, pathology and biomarkers in the brain. As op-posed to previous visualisation software, the input data isvery simple: a generic .csv file containing numbers for eachROI, each number mapping to a di ff erent colour to be as- signed to that ROI – such a simple input allows BrainPainterto be used in conjunction with any other neuroimaging anal-ysis software. Secondly, BrainPainter can visualise both cor-tical and subcortical structures using a surface representation,removing the need to show multiple slices of the same 3Dscan. Third, the images are generated automatically from pre-defined view-points, and can be easily used to create a movieshowing e.g. the propagation of pathology, without the needto write any extra software code or interface with an API.Written in Python, BrainPainter is open-source and can beextended in many ways. In this paper we describe the capa-bilities of the software, and showcase three use cases where itwas used in neuroimaging studies.
2. Design
BrainPainter has a very simple yet e ff ective workflow.Given an input csv file with biomarkers for each region, itproduces high-quality visualisations of cortical and subcorti-cal structures. For this, it uses Blender as a rendering engine,and loads 3D meshes from a template brain (one 3D meshfor each ROI), which are then coloured according to the inputnumbers. Instead of providing a list of RGB colours for eachROI, we choose a simpler interface of providing one num-ber for each ROI which maps to an RGB color using a user-defined color gradient. For example, the gradient can rangefrom white → yellow → orange → red, as in the example fromFig. 1. In this case, the input numbers for each ROI need tobe in the range [0,3], where a value of 1.3 would interpolatebetween colour 1 (yellow) and color 2 (orange).BrianPainter uses open-source software Blender as the ren-dering engine. We chose Blender for three reasons. First, itis open-source, allowing us to distribute it already integratedwith BrainPainter, thus requiring no further installation. Sec-ondly, Blender is a powerful 3D graphics software, which al-lowed us to create realistic lightning conditions and handletransparency required for the glass-brain. Third, it also sup-ports creating movies of complex temporal processes such aspathology spread along the brain. The software also supportsa variety of object formats for the brain template, includingthe popular .obj mesh format. As BrainPainter is written inPython, it allows interfacing with any Blender function.The software is able to colour and visualise regions belong-ing to a pre-defined atlas. Currently, we support three widely-used atlases: (i) the Desikan-Killiany (DK) atlas (Desikanet al. (2006)), (ii) the Destrieux atlas (Destrieux et al. (2010))and (iii) the Tourville atlas (Klein and Tourville (2012)).However, a custom atlas can also be used by mapping thoseregions to any of the three atlases currently supported, throughthe modification a simple mapping in the main configurationfile. https://github.com/mrazvan22/brain-coloring/blob/master/config.py igure 2: BrainPainter website interface at http://brainpainter.csail.mit.edu , showing how it can be easily cus-tomised. Here, the user selects the brain type, colours and resolution, and finally uploads the input .csv file with ROI biomark-ers. The server then generates the output images, which can be downloaded by the user. More customisation features will beadded in future versions.
3. Customisation
BrainPainter can be easily customised in several ways, asshown in Fig 2. First of all, the colours assigned to each regioncan be changed by modifying both the control points of thecolor gradient and the input numbers selecting colors alongthe gradient. The background colour and image resolutioncan also be changed.The 3D structures being visualised can also be customised.We currently support three atlases (Desikan-Killiany, De-strieux and Tourville) as well as two types of brain surfaces:inflated, which is a brain surface that is smoothed out andwhere no gyri appear, and also pial, the standard brain sur-face with gyri. The software allows one to remove some 3Dstructures – for example, Fig. 5 shows the subcortical struc-tures with the cerebellum removed from the visualisation –contrast this with Fig. 4.BrainPainter also support two types of surfaces, corticaland subcortical structures. For the cortical surface, we onlyshow the left hemisphere (although the right hemisphere canalso be added), and provide two default viewing angles (frontand back). For the subcortical structures, we show them forboth hemispheres and also show the right hemisphere as aglass brain, for reference.More complex settings such as the viewing angle and lu-minosity can also be customised, but currently require minormodifications to the source code. In the future, we plan toenable these customisations from the main configuration file.
Figure 3:
Demonstration of BrainPainter for showing extentof pathology on a vertical bar, where colours towards blueshow increased severity. Source: Young et al. (2018).
4. First use case: Visualising the degree of pathology
In the first use case, we want to visualise the degree ofpathology in Alzheimer’s disease. During the progressionof Alzheimer’s disease, some regions of the brain such asthe hippocampus and temporal lobes will be more a ff ectedcompared to other regions of the brain such as the occipitallobe. Visualisation of pathology in AD is important in orderto understand its underlying mechanisms and generate newhypotheses.The notion of pathology here is abstract, and can refer to at-rophy as measured by volume loss or cortical thinning, whitematter degradation as measured by di ff usion tensor imaging(DTI) changes in fractional anisotropy (FA), or the level ofabnormal conformations of proteins such as amyloid-beta ortau as measured by Positron Emission Tomography. However,BrainPainter is agnostic to the meaning of these biomarkers3 igure 4: Demonstration of our software for showing the temporal progression of pathology, as a sequence of snapshot atdi ff erent stages. Images used by Young et al. (2018).and can be used with any imaging modality, including mark-ers derived from several modalities together.Fig. 3 shows an application of BrainPainter by (Young et al.(2018)) to highlight the degree of atrophy in Alzheimer’s dis-ease. Regions with no atrophy are coloured in white, whileregions with severe atrophy are coloured in blue. The gradi-ent on the right shows, for every color, the number of standarddeviations away from controls.
5. Second use case: Visualising the temporal progressionof neurodegenerative diseases
In the second use case, we would like to visualise the tem-poral progression of Alzheimer’s disease (AD). Alzheimer’sdisease is characterised by a slow, continuous progression –while it’s mechanisms are still not fully understood, it is cur-rently believed that initial abnormalities in the amyloid andtau proteins cause a cascade of events that eventually lead toaxonal degradation, neural death and cognitive decline (Mud-her and Lovestone (2002)). Therefore, being able to visu-alise the progression of these events, including their timingand speed, is crucial for understanding the mechanisms ofAlzheimer’s disease.Fig. 4, reproduced and adapted from Young et al. (2018),demonstrates the ability of BrainPainter to visualise the evo-lution of atrophy in two subtypes of Alzheimer’s disease – cortical and subcortical – characterised by prominent atrophyin the cortical and subcortical regions respectively. This studydone by Young et al. (2018) used data from the Alzheimer’sdisease Neuroimaging Initiative to disentangle the hetero-geneity of AD into subtypes with di ff erent progression. Here,visualisations provided by BrainPainter were able to charac-terise not only the degree of atrophy in each region (white / redto blue colors), but also the timing of atrophy events. For example, even in the cortical subtype, the hippocampus be-comes a ff ected by stage 13, while similarly, in the subcortical subtype the temporal lobe becomes a ff ected by stage 13.
6. Third use case: Visualising pathology in subcorticalstructures
The ability to visualise subcortical structures is crucial forneurodegenerative diseases that cause damage to these re-gions. Apart from Alzheimer’s disease, Huntington’s disease(HD) is also known for targetting subcortical regions (Douaudet al. (2009); Wijeratne et al. (2018)). The neurodegenera-tion in HD is believed to begin in the striatum and pallidum,and later followed by other subcortical and cortical regions(Douaud et al. (2009)).Fig. 5, reproduced and adapted with permission from Wi-jeratne et al. (2018), shows visualisations generated by Brain-Painter of atrophy progression in subcortical areas, for Hunt-ington’s disease. The images show early involvement of theputamen, caudate and pallidum in the progression of Hunting-ton’s disease, and demonstrate the potential of BrainPainter invisualising pathology dynamics in subcortical regions usingparsimonious glass-brain images.
7. Conclusion
We presented BrainPainter, an open-source software thatcan be used to visualise structures, biomarkers and patholo-gies in the human brain. The visualisations generated byBrainPainter can be used to significantly enhance the interpre-tation of neuroimaging research and can be easily embeddedby researchers into scientific articles. While not demonstratedhere, BrainPainter can also easily generate movies showingdynamic processes, e.g. propagation of brain pathology.4 a) Stage 0 (b)
Stage 3 (c)
Stage 7 (d)
Stage 10
Figure 5:
Progression of pathology in subcortical regionswithin a glass brain, using images generated with our method.Images used by Wijeratne et al. (2018).Our software has several limitations that can be addressedin future versions. First, it can currently only highlight entireregions-of-interest from an atlas. However, this was a designchoice, as it removes the need for users to create specialisedinput files with voxelwise measurements, thus increasing us-ability. Nevertheless, in future versions we might add the abil-ity to highlight fine-grained patterns of pathology. Yet anotherlimitation of BrainPainter is that it cannot visualise more com-plex structures such as white-matter tracts, although we planto add such functionality in future releases.The use of the powerful Blender engine opens numerousavenues not possible with current neuroimaging software:motion graphics can be used to generate realistic moviesshowing e.g. the evolution of biomarkers, particle simulationscan be used to visualise toxic proteins accumulating in certainregions, soft-body simulations can be used to model brain de-formations due to head trauma, while camera-based renderingallows the creation of educational videos.
8. Acknowledgements
RVM was supported by the NIH grants NIBIB NACP41EB015902 and NINDS R01NS086905, as well as the EP-SRC Centre For Doctoral Training in Medical Imaging withgrant EP / L016478 /
9. ReferencesReferences
Jennifer M Coughlin, Yuchuan Wang, Cynthia A Munro,Shuangchao Ma, Chen Yue, Shaojie Chen, Raag Airan,Pearl K Kim, Ashley V Adams, Cinthya Garcia, et al. Neu-roinflammation and brain atrophy in former NFL players:an in vivo multimodal imaging pilot study.
Neurobiologyof disease , 74:58–65, 2015.Rahul S Desikan, Florent S´egonne, Bruce Fischl, Brian TQuinn, Bradford C Dickerson, Deborah Blacker, Randy LBuckner, Anders M Dale, R Paul Maguire, Bradley T Hy-man, et al. An automated labeling system for subdividingthe human cerebral cortex on MRI scans into gyral basedregions of interest.
Neuroimage , 31(3):968–980, 2006.Christophe Destrieux, Bruce Fischl, Anders Dale, and EricHalgren. Automatic parcellation of human cortical gyriand sulci using standard anatomical nomenclature.
Neu-roimage , 53(1):1–15, 2010.Gwena¨elle Douaud, Timothy E Behrens, Cyril Poupon, YannCointepas, Saˆad Jbabdi, V´eronique Gaura, Narly Golestani,Pierre Krystkowiak, Christophe Verny, Philippe Damier,et al. In vivo evidence for the selective subcortical degener-ation in Huntington’s disease.
Neuroimage , 46(4):958–966,2009.Bruce Fischl. Freesurfer.
Neuroimage , 62(2):774–781, 2012.Arno Klein and Jason Tourville. 101 labeled brain images anda consistent human cortical labeling protocol.
Frontiers inneuroscience , 6:171, 2012.Ra ff aella Migliaccio, Federica Agosta, Katherine L Possin,Elisa Canu, Massimo Filippi, Gil D Rabinovici, Howard JRosen, Bruce L Miller, and Maria Luisa Gorno-Tempini.Mapping the progression of atrophy in early-and late-onsetalzheimers disease. Journal of Alzheimer’s Disease , 46(2):351–364, 2015.Amritpal Mudher and Simon Lovestone. Alzheimer’sdisease–do tauists and baptists finally shake hands?
Trendsin neurosciences , 25(1):22–26, 2002.William D Penny, Karl J Friston, John T Ashburner, Stefan JKiebel, and Thomas E Nichols.
Statistical parametric map-ping: the analysis of functional brain images . Elsevier,2011. https: // brainder.org / research / brain-for-blender / , pages632–635. IEEE, 2004.Menno M Schoonheim, Veronica Popescu, Fernanda C RuedaLopes, Oliver T Wiebenga, Hugo Vrenken, Linda Douw,Chris H Polman, Jeroen JG Geurts, and Frederik Barkhof.Subcortical atrophy and cognition: sex e ff ects in multiplesclerosis. Neurology , 79(17):1754–1761, 2012.Peter A Wijeratne, Alexandra L Young, Neil P Oxtoby, Raz-van V Marinescu, Nicholas C Firth, Eileanoir B John-son, Amrita Mohan, Cristina Sampaio, Rachael I Scahill,Sarah J Tabrizi, et al. An image-based model of brain vol-ume biomarker changes in Huntington’s disease.
Annals ofclinical and translational neurology , 5(5):570–582, 2018.Alexandra L Young, Razvan V Marinescu, Neil P Oxtoby,Martina Bocchetta, Keir Yong, Nicholas C Firth, David MCash, David L Thomas, Katrina M Dick, Jorge Cardoso,et al. Uncovering the heterogeneity and temporal complex-ity of neurodegenerative diseases with Subtype and StageInference.