Self-dual morphology on tree semilattices and applications

Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 Self-dual morphology on tree semilattices and applications Alla Vichik∗, 1 , Renato Keshet†, 2 and David Malah1 1 Electrical Engineering Department, Technion–Israel Institute of Techlonogy, Haifa, Israel malah@ee.technion.ac.il 2 Hewlett Packard Laboratories–Israel, Technion City, Haifa, Israel renato.keshet@hp.com Abstract We present a new tree-based framework for producing self-dual morphological operators. For any given tree representation of images, one can associate a complete inf-semilattice (CISL) in the corresponding tree-representation domain, where the operators can then be derived. We also present a particular case of this general framework, involving a new tree representation, the ExtremaWatershed Tree (EWT). The operators obtained by using the EWT in the above framework behave like classical morphological operators, but in addition are self-dual. Some application examples are provided: pre-processing for OCR and dust & scratch removal algorithms, and image denoising. Keywords: complete inf-semilattices, self-dual operators, tree representation of images. 1. Introduction One of the main approaches for producing self-dual1 morphological operators is by means of a tree representation. For instance, Salembier and Garrido proposed a Binary Partition Tree for hierarchical segmentation in [12, 15]. A tree of shapes was proposed by Monasse and Guichard [10, 11] (see also [1, 2]). These tree representations are usually used for performing connected filtering operations on an image; however, they do not yield nonconnected operators, such as erosions, dilations or openings by a structuring element. In [7] (see also [8]) a new complete inf-semilattice (CISL), called the shape-tree semilattice, was introduced. This semilattice provides non-connected morphological operations, based on the above-mentioned “tree of ∗ alla.vichik@gmail.com † Renato Keshet is also an adjunct lecturer at the EE dep., Technion–Israel Institute of Techlonogy. 1 An operator ψ is self-dual when ψ(−f ) = −ψ(f ) for all input f . 49 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 shapes”. As a consequence, self-dual erosions and openings were obtained. Similar operators had been developed earlier on the so-called Reference Semilattices (introduced in [6], and further studied by Heijmans and Keshet in [4]); however, they require a reference image, which somewhat limits the usage of these operators. The self-dual operators in the shape-tree semilattice provide erosions and openings without the need for a reference image. In this paper, we present a general framework for tree-based morphological image processing, which generalizes the shape-tree operators. This framework yields a set of new morphological operators (erosion, dilation, opening, etc.), for each given tree representation of images. The heart of the proposed approach is a novel complete inf-semilattice of tree representations of images. Because many of the properties of the tree are inherited by the corresponding operators, the choice of the tree representation is of high importance. We focus mostly on self-dual trees, which represent dark and bright elements equally. A particular case of the proposed framework is also presented, based on a novel tree representation, the Extrema-Watershed Tree (EWT). Following the general framework, we derive self-dual morphological operators from the EWT. Examples of applications discussed here are pre-processing for OCR (Optical Character Recognition) algorithms, de-noising of images, and preprocessing for dust and scratch removal. 2. Theoretical background 2.1 Complete inf-semilattices A complete inf-semilattice (CISL) is a partially-ordered set S, where the infimum operation (∧) is always well-defined (but the supremum ∨ is not necessarily so). The theory of mathematical morphology on complete semilattices was introduced in [5], and is an almost-straightforward extension of the traditional morphology on complete lattices. It mathematically supports intuitive observations, such as the fact that erosions are naturally extended from complete lattices to CISLs, whereas dilations are not universally well-defined on CISLs. On the other hand, some results may not be necessarily intuitive. The main ones are as follows: (a) it is always possible to associate an opening V γ to a given erosion ε by means of γ(x) = {y | ε(y) = ε(x)}, (b) even though the adjoint dilation δ is not universally well-defined, it is always well defined for elements on the image of S by ε, and (c) γ = δε. 2.2 Rooted trees and their corresponding CISL This section reviews basic graph theory notions (given in [3, chapter 1]), including the natural partial ordering on rooted trees, which provide them 50 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 with a CISL structure. A graph is a pair of sets G = (V, E) satisfying E ⊆ [V ]2 . A path is a non-empty graph P = (V, E) of the form: V = {x0 , x1 , ..., xk }, E = {x0 x1 , x1 x2 , ..., xk−1 xk }, where the xi are all distinct. A cycle is a path where k ≥ 2 and (x0 , xk ) ∈ E. A graph not containing any cycles, is called a forest. A connected forest is called a tree (thus, a forest is a graph whose components are trees). Sometimes it is convenient to consider one vertex of a tree as special; such a vertex is then called the root of this tree. A tree with a fixed root is a rooted tree. Choosing a root r in a tree t imposes the following partial ordering on V(t): x t y ⇐⇒ x ∈ rty, where rty is the unique path in t that connects y to the root. Note that (V, t ) is a CISL, where r is the least element, and the maximal elements are the leafs of t. The infimum between vertices is the nearest common ancestor vertex. We say that a tree t1 is smaller than another tree t2 if t1 ⊆ t2 . 3. Tree semilattices This section presents the proposed general framework for tree-based morphological image processing (introduced in [16]). This framework enables the definition of new morphological operators that are based on tree representations. The proposed image processing scheme is shown is Figure 1. Figure 1. Tree-based morphology. 3.1 CISL of tree representations The heart of the proposed approach is a novel complete inf-semilattice of tree representations of images. Let L be an arbitrary set of “labels”, and let t = (V, E) be a rooted tree, with root r, such that V ⊆ L. Therefore t is a tree of labels. Moreover, let M : E 7→ V be an image of vertices, mapping each point in E to a vertex of t. As usual, E may be an Euclidean space or a discrete rectangular grid within the image area. Definition 1. (Tree representation) The structure T = (t, M ) shall be called a tree representation. The set of all tree representations associated with the label set L and with the root r shall be denoted by TrL . 51 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 Consider the following relation between tree representations: For all T1 = (t1 , M1 ) and T2 = (t2 , M2 ) in TrL ,  t1 ⊆ t2 and (1) T1 ≤ T2 ⇐⇒ M1 (x) t2 M2 (x), ∀x ∈ E, where ⊆ is the usual graph inclusion, and t2 is the partial ordering of vertices within the tree t2 (see Subsection 2.2). Proposition 1. The above tree relation ≤ is a partial ordering on TrL , and
△ TrL , ≤ is a CISL. The least element is T0 = (({r}, {}), M0 (x) ≡ r)). The proof is given in [16]. The general format of the corresponding infimum and supremum operators are also derived in [16]. Here, however, we focus on the particular case where all tree presentations involved in an infimum or supremum operation have a common tree associated with them: Proposition 2. Let {Ti = (t, Mi )} be a collection of tree representations with a common tree t. In this case, ^ Ti = (t, ft {Mi }) , (2) i and _ Ti = (t, gt {Mi }) , (3) i where ft and gt are the point-wise infimum and supremum associated to vertex order t , respectively. Notice that gt {Mi } may not always exist. The situation where the set of tree representations share the same tree is what one encounters when defining flat erosions and dilations on the complete inf-semilattice of tree representations. The flat erosion can be defined as the operator ε given by: ^ △ ^ εB (T ) = T−b = (t, M−b ), (4) b∈B b∈B where B is a structuring element. It is easy to verify that the above operator is indeed an erosion on TrL . Using Proposition 2, one obtains that εB (T ) = (t, ft {M−b |b ∈ B}) . (5) As reminded in Section 2.1, on a complete inf-semilattice, one can associate to any given erosion ε an opening γ (and, in fact, any morphological operator that is derived from compositions of erosions and openings, such as the internal gradient, dark top-hat transform, and skeletons). Furthermore, the adjoint dilation δ exists, and, even though it is not well defined for all 52 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 complete inf-semilattice elements, it is always well-defined for elements that are mapped by the erosion, and γ = δε. In the case of the above tree-representation flat erosion, the adjoint dilation is given by: δB (T ) = (t, gt {Mb |b ∈ B}) . (6) We also define the tree-representation reconstruction of T from a marker T = (t, M ) ≤ T as the infinite iteration of the conditional dilation 
△ (7) δB (T |T ) = t, gt Mb ft M |b ∈ B . Notice that δB (T |T ) is always well defined, since it consists of a supremum of bounded elements. 3.2 Image processing on tree semilattices Now that morphology on the tree representation domain has been established, we can turn to our ultimate goal, which is to process a given grayscale image f . Let us assume that f is an integer-valued function on E, i.e., f ∈ Fun(E, Z). Moreover, let τ by an operator that transforms f into a pair (T, ℓ) , where T = (t, M ) ∈ TrL is a tree representation, and ℓ : L 7→ Z is a function that maps labels into graylevels. The tree transformation τ should be invertible, and the inversion be given by: τ −1 (ℓ, M (x)) = ℓ (M (x)). We propose the following approach for processing f , using the CISL of tree representations: 1. compute τ (f ) = (T, ℓ); 2. perform one or more morphological operations on T to obtain a processed tree representation T̂ = (t, M̂ ); 3. transform (T̂ , ℓ) back into a new image fˆ ∈ Fun(E, Z), using:   fˆ(x) = τ −1 (ℓ, M̂ (x)) = ℓ M̂ (x) . (8) If the morphological operation in Step 2 above is the erosion εB , then all three steps can be collapsed into the following equation: fˆ(x) = ℓ (ft {M−b (x)|b ∈ B}) . (9) △ Proposition 3. For any vertex v in V , let R(v) = {x ∈ E|M (x) t v} and △ R̂(v) = {x ∈ E|M̂ (x) t v}, where M̂ is again the mapping function after the erosion εB . Then, for all v: R̂(v) = R(v) ⊖ B, where (.) ⊖ B is the traditional binary erosion by the s.e. B. 53 (10) Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 Proposition 3 (which is proven in [16]) suggests an alternative algorithm for computing the erosion. For any v, (a) compute R(v), (b) S compute R̂(v) = R(v)⊖B, and (c) assign ℓ(v) to all points within R̂(v)\ v≺v′ R̂(v ′ ). 3.3 Particular cases and examples In order for the tree transform to be invertible, τ should be such that it assigns a common label to each flat zone of f . This is because τ −1 maps each label to a single graylevel. This suggests that special attention should be paid to the flat zones of f . One way of addressing the flat zones of a given image is by considering its Regional Adjacency Graph (RAG). The RAG is a graph, where V is the set of all flat zones of the image, and E contains all pairs of flat zones that are adjacent to each other. A spanning tree is a subgraph of a RAG that should, obviously, be a tree, and have the same vertex set V as the RAG. A spanning tree creates a hierarchy in the RAG, defining father/son relationships between adjacent flat zones. The proposed morphological scheme is of particular interest when t is a spanning tree of the RAG. In this case, the associated morphological operators do not create new grey/color values. One particular group of trees are the Max- and Min-Trees [13]. When a tree vertex is always brighter (resp., darker) then its sons, as in the Max-Tree (resp., Min-Tree), the infimum operation always changes the gray level to the local minimum (resp., maximum), which is precisely what the traditional grayscale erosion (resp., dilation) does. In other words, for these trees, the proposed tree approach becomes the traditional grayscale mathematical morphology (resp., its dual version). More interesting particular cases are the Boundary Topographic Variation (BTV) Tree (see [16]), which is built from the RAG using a minimal topographic distance criterion. Another one is the shape-tree defined in [9] and the resulting semilattice defined in [7, 8]. Both provide self-dual morphological operators, based on some inclusion criterion. 3.4 Image semilattice What we would really like is the CISL of tree representation (using a tree τ ) to induce a CISL in the image domain. That is, we would like, for instance, the composite operation of τ −1 ετ to be an erosion in the image domain. However, that is not guaranteed. In fact, the partial ordering in the treerepresentation domain does induce a partial ordering for images, for any τ ; however, the infimum operation is not guaranteed to be well defined. This issue is still under study. 54 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 4. Extrema-watershed tree Based on the general framework of Section 3, all that is needed in order to obtain a new set of morphological operators is a given tree representation. In this section, we explore a particular case of the proposed framework, using a novel self-dual tree representation, which we call the Extrema-Watershed Tree (EWT). The EWT is a particular case of “Binary Partitioning Tree” [12]; in particular, the proposed representation is built using a particular case of the iterative merging process presented by Salembier, Garrido and Garcia in [14], as follows: Input all the extrema of a given image (i.e., all regions associated to a local minimum or maximum) into a list, sorted by increasing area2 . Also, initiate the EWT by setting each flat zone as a leaf vertex. The main loop for the computation of the EWT is as follows: Take the first extremum from the ordered list (the one with smallest area), and merge it with the adjacent neighbor that is the closest one in terms of graylevels. Then, set the merged region as the parent vertex of the above two regions (the extremum and its neighbor) in the EWT. Select the graylevel of the non-extremum region to be the graylevel of the new merged region. Finally, check whether the newly merged region and all its neighbors are extrema, and insert those that are into the sorted list (in their corresponding place, according to the listing order). This loop runs until the list has just one element, which then becomes the EWT root. Figure 2 illustrates the computation of the EWT. Consider the image in Figure 2(a), which contains two extrema with the same area: v1 and v3 . The first step of the procedure, shown in Figure 2(b), consists of merging v1 with v2 , since the difference in graylevel between v1 and v2 is smaller than the one between v3 and v4 . This merger produces a new flat zone – v5 , with the same graylevel as v2 – which is a new extremum in the image. In the next step, shown in Figure 2(c), the extremum v3 is merged with v4 to create v6 . The procedure continues until all extrema (old and new) are merged. Figure 2(d) shows the final EWT. As described in Section 3, once a tree transform is defined, morphological operations (such as erosion and opening) in the tree-domain can be derived. The new operators typically inherit some of the properties of the tree, such as self-duality, for instance. Figure 3 shows the result of the EWT erosion and EWT opening. Notice that very small features are removed, whereas the larger ones shrunk, in a self-dual manner. The average gray level of the picture does not change; in particular, the picture does not become darker, which is what usually happens after a standard erosion or opening. 2 If two extrema have the same area, input first the one who has the smallest grayscale distance to its closest neighbor. 55 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 (a) (b) (c) (d) Figure 2. Example of the EWT computation. (a) Input image, (b) first merging step, (c) second merging step, (d) the final EWT. 5. Application examples The EWT has many potential applications; in this section we list just a few. One application that requires image simplification is pre-processing for OCR (Optical Character Recognition). We have chosen a specific OCR algorithm, used for recognition of license plate numbers, that was developed in [17]. This algorithm uses a mask for each digit and looks for the best correlation among these masks with an image. The algorithm also outputs a confidence grade, which can be used for comparing algorithms. Any noise that exists in the image degrades the correlation value and interferes with the recognition. Consider the example license plate shown in Figure 4(a), which has been artificially corrupted with blobs of different sizes. Without pre-processing the algorithm fails to read the correct number. Several different algorithms (including linear filtering and traditional grayscale morphology) has been applied to this image. In order to compensate for the lack of duality in classical morphology, we have also compared the EWT with the “quasi-self-dual” Opening-Closing by reconstruction operator. The 56 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 (a) (b) (c) Figure 3. (a) Original image, (b) EWT erosion by square SE 5 × 5, (c) EWT opening by square SE 5 × 5. (a) (b) (c) (d) Figure 4. (a) Input image, artificially corrupted, (b) filtered with a median filter, (c) filtered with regular self dual opening by reconstruction, and (d) filtered by the EWT-based opening by reconstruction, using circle SE of radius 4. only algorithms that cause the algorithm to correctly read the number were the median filter, quasi-self-dual filter and the EWT opening by reconstruction (see Figure 4(b), 4(c) and 4(d), respectively). The confidence grades associated with the EWT pre-processed image were higher than those for the median filter and the quasi-self-dual filter. Further details on this experiment can be found in [16]. Another example uses opening by reconstruction as an initial step for an application that removes dust and scratches from images. The elements filtered by the opening by reconstruction are completely extracted, including their edges. This enables one to extract candidates for dust and scratch removal, without corrupting their shapes. The proposed operation is a EWT top-hat filter. Figure 5 shows an example. Later steps (not considered here) can then make further analysis of the image in order to decide which candidates should be removed. We have compared the proposed approach to linear and median filters. Subjective and objective criteria were used. 57 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 (a) (b) Figure 5. Top hat, using cross SE 3 × 3, as a pre-processing stage for dust and scratch removal. (a) Original image (b) Top hat by reconstruction based on EWT. The subjective criterion is the overall corruption of the candidate shapes. The objective criterion is the measured energy of the filtered images. The EWT performed better in both criteria. On one hand, for the relevant structuring elements, the energy of the EWT filtered image was lower than for the linear and median filters. On the other hand, the linear and median filters do not completely extract the artifacts, as can be seen in Figure 6 for the “cross” structuring element. 6. Conclusion We have presented a general framework for producing new morphological operators that are compatible to given tree representations. Furthermore, a useful particular case is provided, based on a new tree representation, the Extrema Watershed Tree. The resulting morphological erosion and opening operators were applied to a number of application examples, giving better results in comparison to other filtering techniques, including classical morphological filtering. In general, EWT-based filtering performs well in tasks suitable for classical morphological filtering, especially when self-duality is required. 7. Acknowledgments We would like to thank the reviewers for their thorough remarks, and, in particular, the 2nd reviewer, for the insights regarding the reconstruction operator. 58 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 (a) (b) (c) (d) Figure 6. Zoom in. Top hat, using cross SE 3 × 3, as a pre-processing stage for dust and scratch removal. (a) Original image (b) Top hat by reconstruction based on EWT (c) Top hat using median (d) Top hat using an averaging filter. References [1] C. Ballester, V. Caselles, and P. Monasse, The Tree of Shapes of an Image, ESAIM: Control, Optimization and Calculus of Variations 9 (2003), 1-18. [2] V. Caselles and P. Monasse, Grain filters, Journal of Mathematical Imaging and Vision 17 (November 2002), no. 3, 249-270. [3] R. Diestel, Graph Theory, Electronic Edition, Springer-Verlag New York, 2000. [4] H. J. A. M. Heijmans and R. Keshet, Inf-semilattice approach to self-dual morphology, Journal of Mathematical Imaging and Vision 17 (July 2002), 55-80. [5] R. Keshet, Extension of Morphological Operations to Complete Semilattices and Its Applications to Image and Video Processing, Mathematical Morphology and its Applications to Image and Signal Processing (Proc. of ISMM’98) (June 1998), 35-42. [6] , Mathematical Morphology on Complete Semilattices and its Applications to Image Processing, Fundamenta Informaticæ 41 (January 2000), 33-56. [7] , Shape-Tree Semilattice, Journal of Mathematical Imaging and Vision 22 (May 2005), 309 - 331. [8] , Adjacency lattices and shape-tree semilattices, Image & Vision Computing, Special Issue on ISMM05 25 (April 2007), no. 4. [9] P. Monasse, Morphological representation of digital images and application to registration, 2000. Ph.D. Thesis, Université Paris IX-Dauphine. [10] P. Monasse and F. Guichard, Fast computation of a contrast-invariant image representation, IEEE Transactions on Image Processing 9 (2000), 860-872. [11] , Scale-Space from a Level Lines Tree, Journal of Visual Communication and Image Representation 11 (2000), 224-236. 59 Proceedings of the 8th International Symposium on Mathematical Morphology, Rio de Janeiro, Brazil, Oct. 10 –13, 2007, MCT/INPE, v. 1, p. 49–60. http://urlib.net/dpi.inpe.br/ismm@80/2007/03.02.08.07 [12] P. Salembier, Binary Partition Tree as an Efficient Representation for Image Processing, Segmentation, and Information Retrieval, IEEE Transactions on Image Processing 9 (April 2000), 561-576. [13] P. Salembier and L. Garrido, Connected operators based on region-tree prunning, 15th International Conference on Pattern Recognition (ICPR’00) 3 (September 2000), 367 - 370. [14] P. Salembier, L. Garrido, and D. Garcia, Auto-Dual connected operators based on iterative merging algorithms, IEEE Transactions on Image Processing 7 (April 1998), no. 4. [15] P. Salembier, J. Llach, and L. Garrido, Visual Segment Tree Creation for MPEG-7 Description Schemes, Pattern Recognition 35 (March 2002), 563-579. [16] A. Vichik, Self-dual Morphological Operators Based on Tree Representations of Images, 2006. Master’s Thesis, EE Dept., Technion-IIT, Israel. Available from: <http://sipl.technion.ac.il/new/Research/Publications/Graduates/Alla_ Vichik/Alla_main_revised_f.pdf>. [17] S. Vichik, R. Sandler, and A. Rosen, Moving Car License Plate Recognition, 1999. Semestrial project, Technion-IIT, Available from: <www.cs.technion.ac.il/Labs/ Isl/Project/Projects_done/cars_plates/finalreport.htm>. 60