Branches Tracking
“Determining the branchings of 3D structures from respective 2D projections"
Jorge J. G. Leandro, Roberto M. Cesar. Jr.
Luciano da Fontoura Costa
Institute of Mathematics and Statistics – USP
Institute of Phisics of São Carlos – USP
Department of Computer Science
Department of Physics and Informatics
Rua do Matão, 1010
Av. Trabalhador Sãocarlense, 400
São Paulo, SP - 05508-090 – Brazil
São Carlos, SP - 13560-970 – Brazil
{jleandro,cesar}@vision.ime.usp.br
[email protected]
Summary • • • • • •
Introduction Concepts and Overview General Framework • Preprocessing • Branches Tracking Results Concluding Remarks References
Motivation
1 - Introduction
•Neural shape analysis •Fenotype versus genotype •Phenotypic features understanding •Environment influences on cells and tissues development •New research areas:post-genomics, neuro-informatics, system biology •Nervous System: good for studies in relationship among phenotypic characteristics, genetics, environment, form and function!
Neuron Image
Motivation
Neuron Image
1 - Introduction
Motivation
1 - Introduction
•First works [Ramon y Cajal, 1909] •Sholl diagrams [Sholl, 1953] •Cat ganglion cells [Boycott and Wässle, 1974] •Fractal and multifractal analysis [Caserta et al., 1990, Jelinek and Fernandez, 1998, Smith et al., 1996] •Curvature, wavelets and multiscale energies [Cesar and Costa, 1998, Costa and Velte, 1999]
Motivation
1 - Introduction
•Shape Analysis •There are a large number of descriptors and models •Contour (1D parametric curve) based measurements •Curvature, •Wavelet, •Orientation Entropy, •Bending energy, •Tangent Fields, etc •Advantages: locally assessment of curvature degree and concavity [Costa & Cesar, 2001, Loncaric, 1998, Pavlidis 1980]
Motivation
1 - Introduction
Parametric contour extraction
[Costa & Cesar, 2001, Bennett and Mac Donald, 1975, Chassery and Montanvert, 1991; Shahraray and Anderson, 1985, 1989]
Motivation
Parametric contour extraction
1 - Introduction
Motivation
1 - Introduction
Problem: • Available data: 2D projection of a non-planar (often complex) 3D shape •Critical regions (crossings and bifurcations) constrain the access to innermost neuron regions for contour extraction •Briefly, chain-code based contour following algorithms cannot deal with Non-Jordan curves
Motivation
1 - Introduction
Neuron skeleton (black), contour (red) and unreachable areas (green) for the contour following algorithm based on the chain-code
Motivation
1 - Introduction
Motivation
1 - Introduction
Motivation
1 - Introduction
Motivation
1 - Introduction
Retina image analysis [Hoover et al., 2000, Zana & Klein, 2001, Soares et al., 2006]
Motivation
Retina image analysis
1 - Introduction
2 – Concepts and Overview Concepts •Image: •2D projection from a 3D shape •Lacking of depth information •Branching structure images (neurons, vessel trees,…) •Projection gives rise to intersecting branches •Intersecting branches give rise to critical regions •How to set apart intersecting branches in critical regions?
2 – Concepts and Overview Concepts •Definitions: •Branching Structure •Binary image of the 8-connected one-wide pixel skeleton, where critical regions are present •Branch •Binary image comprised of a set of pixels starting from either the soma up to its termination or a critical region up to its termination
2 – Concepts and Overview Concepts •Definitions: •Critical Region •Bifurcation Region is a set of pixels where an inward branch splits into two outward branches, one of them quite distinct in orientation •Crossing Region is a set of pixels where an inward branch splits into N>2 outward branches, with N-1 outward branches in quite distinct orientations
2 – Concepts and Overview Overview
The method is comprised of three great steps, that is: •Pre processing: •Branches tracking •Contour following
2 – Concepts and Overview Overview Preprocessing: •Mathematical morphology: dilations, erosions, skeletonization, pruning, hit-or-miss and etc, yielding : •an 8-connected skeleton image •an image with critical regions and •a queue with branches sources (seeds).
2 – Concepts and Overview Overview •Branches tracking •A BTA (Branching Tracking Algorithm) call for each source point •Labeling iterativetly all the current seed valid neighbor pixels until reaching a critical region •Valid pixels: non-labeled object pixels, whose vicinity equals two pixels •Taking a decision to continue with the labeling procedure in the most probable outward branch portion
3– General Framework Preprocessing Binarization: thresholding the input graylevel image Soma segmentation: erosion of dentrites by a disk area filtering to separate the soma structure soma dilation to retrieve its original size
3– General Framework Preprocessing
3– General Framework Preprocessing Skeletonization: morphological thining pruning to filter noise out hit-or-miss to filter redundancies out in order to achieve na 8connected one-pixel wide skeleton End points: difference between soma and the 8-connected one-pixel wide skeleton
3– General Framework Preprocessing
3– General Framework
Preprocessing
3– General Framework
Preprocessing
3– General Framework
Preprocessing
Structuring element (left) for hit-or-miss filtering the prunned skeleton to yield an 8-connected one-pixel wide skeleton. Light gray pixels (right) at the left hand should be removed yielding the non-redundant structure at the right hand.
3– General Framework
Preprocessing
skeleton soma
end points
3– General Framework
Preprocessing
Branch point detection based on the number of neighbors
3– General Framework Branches Tracking
3– General Framework Branches Tracking
3– General Framework Branches Tracking
3– General Framework Branches Tracking
3– General Framework Branches Tracking
Large inner product
3– General Framework Branches Tracking
Smaller inner product
3– General Framework Branches Tracking
Small inner product
3– General Framework Branches Tracking
Oposite directions: vectors should not be considered for the current dendritic tree
3– General Framework Branches Tracking Branch Tracking Algorithm (BTA) •
Seed points (adjacent to soma) are stacked, each one defining a dendritic tree to be processed
•
Each seed point starts a labeling region-growing procedure
•
Each time a branching region is reached, the algorithm calculates the inneward and outward vectors
•
Branch directions are pushed in the BTA stack
•
Crossing directions are ignored: they are associated to another dendritic tree
3– General Framework Branches Tracking
3– General Framework Branches Tracking
4 – Results Implementation details •Code implementation: Matlab scripts •SDC Morph: Mathematical Morphology toolbox
Database •Rat hippocampal cells from Southampton Archive: http://www.compneuro.org/CDROM/nmorph/index/topindex_tn.html
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results Contour following algorithm (preliminary)
4 – Results Contour following algorithm (preliminary)
4 – Results Contour following algorithm (preliminary)
5 – Concluding Remarks ¾New method for contour extraction of non-planar branching structures ¾Better contours may be extracted (w.r.t. previous algorithms), even if the correct 3D branching structure is not recovered ¾It is not always possible to solve the ambiguity due to 3D to 2D projection ¾Preprocessing •Any skeletonization may be applied (morphological thinning, exact dilations, medial axis transform) •Parameters empirically chosen
5 – Concluding Remarks ¾Branch Tracking Algorithm •high critical regions densities 9problem: BTA fails with some branches 9possible solution: search by using a tree data structure to keep memory of the shape topology •superposition of branches 9problem: BTA mistakes superposition of branches for a critical region, since it is neither a crossing region nor a branching region 9possible solution: multiscale analysis
5 – Concluding Remarks ¾Branch Tracking Algorithm •branches parallelism 9problem: BTA identifies incorrectly short cycles 9current workaround: dilation preceding skeletonization to shrink short cycles into a critical region 9possible solution: multiscale analysis
References •
M. F. Carmo Differential geometry of curves and surface Prentice-Hall, Englewood Cliffs, N.J., 1976.
•
F. Caserta, H. Stanley, W. Eldred, G. Daccord, R. Haussman, and J. Nittmann. Physical mechanisms underlying neurite outgrowth: A quantitative analysis of neuronal shape. Physical Review Letters}, 64(1):95--98, 1990.
•
R. M. Cesar-Jr. and L.F. Costa. Neural cell classification using wavelets and multiscale curvature. Biological Cybernetics , 79(4):347--360, 1998
•
L. da F.Costa. Bioinformatics: Perspectives for the future. 3(4):564--574, 2004.
•
L. da F. Costa. Morphological complex networks: Can individual morphology determine the general connectivity and dynamics of networks? 2005..
•
E. R. Dougherty and R. A. Lotufo. Hands-On Morphological Image Processing . SPIEInternational Society for Optical Engine, 2003.
•
G. T. Herman.
•
H. Jelinek and E. Fernandez. Neurones and fractals: how reliable and useful are calculations of fractal dimensions? Journal of Neuroscience Methods , 81(1-2):9--18, 1998.
Genetic and Molecular Biology ,
Geometry of Digital Spaces . Birkhauser Boston, 1998.
References •
J.R. Bennett and J.S. Mac Donald, On the Measurement of Curvature in a Quantized Environment, IEEE Transactions on Computers, C 24(8):803-820, Aug 1975.
•
J.M. Chassery and A. Montanvert, Géométrie Discrète, Hermes, Paris, 1991 (in French).
•
L. da F. Costa and T.J. Velte, Automatic Characterization and Classification of Ganglion Cells from the Salamander Retina, Journal of Comparative Neurology, 404(1): 35-51, 1999.
•
Morigiwa, M. Tauci, and Y. Fukuda. Fractal analysis of ganglion cell dendritic branching patterns of the rat and cat retinae. Neuroscience Research Suppl. , 10:S131--S140, 1989.
•
T. Pavlidis, Algorithms for Shape Analysis of Contours and Waveforms, IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-2(4): 301-312, July 1980.
•
B. Shahraray and D. J. Anderson, Uniform Resampling of Digitized Contours, IEEE Transactions on Pattern Analysis and Machine Intelligence, 7(6):674-682, 1985.
•
B. Shahraray and D.J. Anderson, Optimal Estimation of Contour Properties by Cross-Validated Regularization, IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6): 600610, 1989.
Jorge J. G. Leandro, Roberto M. Cesar. Jr.
Luciano da Fontoura Costa
Institute of Mathematics and Statistics – USP
Institute of Phisics of São Carlos – USP
Department of Computer Science
Department of Physics and Informatics
Rua do Matão, 1010
Av. Trabalhador Sãocarlense, 400
São Paulo, SP - 05508-090 – Brazil
São Carlos, SP - 13560-970 – Brazil
{jleandro,cesar}@vision.ime.usp.br
[email protected]
Summary • • • • • •
Introduction Concepts and Overview General Framework • Preprocessing • Branches Tracking Results Concluding Remarks References
Motivation
1 - Introduction
•Neural shape analysis •Fenotype versus genotype •Phenotypic features understanding •Environment influences on cells and tissues development •New research areas:post-genomics, neuro-informatics, system biology •Nervous System: good for studies in relationship among phenotypic characteristics, genetics, environment, form and function!
Neuron Image
Motivation
Neuron Image
1 - Introduction
Motivation
1 - Introduction
•First works [Ramon y Cajal, 1909] •Sholl diagrams [Sholl, 1953] •Cat ganglion cells [Boycott and Wässle, 1974] •Fractal and multifractal analysis [Caserta et al., 1990, Jelinek and Fernandez, 1998, Smith et al., 1996] •Curvature, wavelets and multiscale energies [Cesar and Costa, 1998, Costa and Velte, 1999]
Motivation
1 - Introduction
•Shape Analysis •There are a large number of descriptors and models •Contour (1D parametric curve) based measurements •Curvature, •Wavelet, •Orientation Entropy, •Bending energy, •Tangent Fields, etc •Advantages: locally assessment of curvature degree and concavity [Costa & Cesar, 2001, Loncaric, 1998, Pavlidis 1980]
Motivation
1 - Introduction
Parametric contour extraction
[Costa & Cesar, 2001, Bennett and Mac Donald, 1975, Chassery and Montanvert, 1991; Shahraray and Anderson, 1985, 1989]
Motivation
Parametric contour extraction
1 - Introduction
Motivation
1 - Introduction
Problem: • Available data: 2D projection of a non-planar (often complex) 3D shape •Critical regions (crossings and bifurcations) constrain the access to innermost neuron regions for contour extraction •Briefly, chain-code based contour following algorithms cannot deal with Non-Jordan curves
Motivation
1 - Introduction
Neuron skeleton (black), contour (red) and unreachable areas (green) for the contour following algorithm based on the chain-code
Motivation
1 - Introduction
Motivation
1 - Introduction
Motivation
1 - Introduction
Motivation
1 - Introduction
Retina image analysis [Hoover et al., 2000, Zana & Klein, 2001, Soares et al., 2006]
Motivation
Retina image analysis
1 - Introduction
2 – Concepts and Overview Concepts •Image: •2D projection from a 3D shape •Lacking of depth information •Branching structure images (neurons, vessel trees,…) •Projection gives rise to intersecting branches •Intersecting branches give rise to critical regions •How to set apart intersecting branches in critical regions?
2 – Concepts and Overview Concepts •Definitions: •Branching Structure •Binary image of the 8-connected one-wide pixel skeleton, where critical regions are present •Branch •Binary image comprised of a set of pixels starting from either the soma up to its termination or a critical region up to its termination
2 – Concepts and Overview Concepts •Definitions: •Critical Region •Bifurcation Region is a set of pixels where an inward branch splits into two outward branches, one of them quite distinct in orientation •Crossing Region is a set of pixels where an inward branch splits into N>2 outward branches, with N-1 outward branches in quite distinct orientations
2 – Concepts and Overview Overview
The method is comprised of three great steps, that is: •Pre processing: •Branches tracking •Contour following
2 – Concepts and Overview Overview Preprocessing: •Mathematical morphology: dilations, erosions, skeletonization, pruning, hit-or-miss and etc, yielding : •an 8-connected skeleton image •an image with critical regions and •a queue with branches sources (seeds).
2 – Concepts and Overview Overview •Branches tracking •A BTA (Branching Tracking Algorithm) call for each source point •Labeling iterativetly all the current seed valid neighbor pixels until reaching a critical region •Valid pixels: non-labeled object pixels, whose vicinity equals two pixels •Taking a decision to continue with the labeling procedure in the most probable outward branch portion
3– General Framework Preprocessing Binarization: thresholding the input graylevel image Soma segmentation: erosion of dentrites by a disk area filtering to separate the soma structure soma dilation to retrieve its original size
3– General Framework Preprocessing
3– General Framework Preprocessing Skeletonization: morphological thining pruning to filter noise out hit-or-miss to filter redundancies out in order to achieve na 8connected one-pixel wide skeleton End points: difference between soma and the 8-connected one-pixel wide skeleton
3– General Framework Preprocessing
3– General Framework
Preprocessing
3– General Framework
Preprocessing
3– General Framework
Preprocessing
Structuring element (left) for hit-or-miss filtering the prunned skeleton to yield an 8-connected one-pixel wide skeleton. Light gray pixels (right) at the left hand should be removed yielding the non-redundant structure at the right hand.
3– General Framework
Preprocessing
skeleton soma
end points
3– General Framework
Preprocessing
Branch point detection based on the number of neighbors
3– General Framework Branches Tracking
3– General Framework Branches Tracking
3– General Framework Branches Tracking
3– General Framework Branches Tracking
3– General Framework Branches Tracking
Large inner product
3– General Framework Branches Tracking
Smaller inner product
3– General Framework Branches Tracking
Small inner product
3– General Framework Branches Tracking
Oposite directions: vectors should not be considered for the current dendritic tree
3– General Framework Branches Tracking Branch Tracking Algorithm (BTA) •
Seed points (adjacent to soma) are stacked, each one defining a dendritic tree to be processed
•
Each seed point starts a labeling region-growing procedure
•
Each time a branching region is reached, the algorithm calculates the inneward and outward vectors
•
Branch directions are pushed in the BTA stack
•
Crossing directions are ignored: they are associated to another dendritic tree
3– General Framework Branches Tracking
3– General Framework Branches Tracking
4 – Results Implementation details •Code implementation: Matlab scripts •SDC Morph: Mathematical Morphology toolbox
Database •Rat hippocampal cells from Southampton Archive: http://www.compneuro.org/CDROM/nmorph/index/topindex_tn.html
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results
4 – Results Contour following algorithm (preliminary)
4 – Results Contour following algorithm (preliminary)
4 – Results Contour following algorithm (preliminary)
5 – Concluding Remarks ¾New method for contour extraction of non-planar branching structures ¾Better contours may be extracted (w.r.t. previous algorithms), even if the correct 3D branching structure is not recovered ¾It is not always possible to solve the ambiguity due to 3D to 2D projection ¾Preprocessing •Any skeletonization may be applied (morphological thinning, exact dilations, medial axis transform) •Parameters empirically chosen
5 – Concluding Remarks ¾Branch Tracking Algorithm •high critical regions densities 9problem: BTA fails with some branches 9possible solution: search by using a tree data structure to keep memory of the shape topology •superposition of branches 9problem: BTA mistakes superposition of branches for a critical region, since it is neither a crossing region nor a branching region 9possible solution: multiscale analysis
5 – Concluding Remarks ¾Branch Tracking Algorithm •branches parallelism 9problem: BTA identifies incorrectly short cycles 9current workaround: dilation preceding skeletonization to shrink short cycles into a critical region 9possible solution: multiscale analysis
References •
M. F. Carmo Differential geometry of curves and surface Prentice-Hall, Englewood Cliffs, N.J., 1976.
•
F. Caserta, H. Stanley, W. Eldred, G. Daccord, R. Haussman, and J. Nittmann. Physical mechanisms underlying neurite outgrowth: A quantitative analysis of neuronal shape. Physical Review Letters}, 64(1):95--98, 1990.
•
R. M. Cesar-Jr. and L.F. Costa. Neural cell classification using wavelets and multiscale curvature. Biological Cybernetics , 79(4):347--360, 1998
•
L. da F.Costa. Bioinformatics: Perspectives for the future. 3(4):564--574, 2004.
•
L. da F. Costa. Morphological complex networks: Can individual morphology determine the general connectivity and dynamics of networks? 2005..
•
E. R. Dougherty and R. A. Lotufo. Hands-On Morphological Image Processing . SPIEInternational Society for Optical Engine, 2003.
•
G. T. Herman.
•
H. Jelinek and E. Fernandez. Neurones and fractals: how reliable and useful are calculations of fractal dimensions? Journal of Neuroscience Methods , 81(1-2):9--18, 1998.
Genetic and Molecular Biology ,
Geometry of Digital Spaces . Birkhauser Boston, 1998.
References •
J.R. Bennett and J.S. Mac Donald, On the Measurement of Curvature in a Quantized Environment, IEEE Transactions on Computers, C 24(8):803-820, Aug 1975.
•
J.M. Chassery and A. Montanvert, Géométrie Discrète, Hermes, Paris, 1991 (in French).
•
L. da F. Costa and T.J. Velte, Automatic Characterization and Classification of Ganglion Cells from the Salamander Retina, Journal of Comparative Neurology, 404(1): 35-51, 1999.
•
Morigiwa, M. Tauci, and Y. Fukuda. Fractal analysis of ganglion cell dendritic branching patterns of the rat and cat retinae. Neuroscience Research Suppl. , 10:S131--S140, 1989.
•
T. Pavlidis, Algorithms for Shape Analysis of Contours and Waveforms, IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-2(4): 301-312, July 1980.
•
B. Shahraray and D. J. Anderson, Uniform Resampling of Digitized Contours, IEEE Transactions on Pattern Analysis and Machine Intelligence, 7(6):674-682, 1985.
•
B. Shahraray and D.J. Anderson, Optimal Estimation of Contour Properties by Cross-Validated Regularization, IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6): 600610, 1989.
