Encoding visual information using anisotropic transformations

Encoding Visual Information Using Anisotropic Transformations∗ Giuseppe Boccignone (corresponding author) Dipartimento di Ingegneria dell’Informazione e Ingegneria Elettrica and INFM - Universitá di Salerno via Ponte Don Melillo 1, 84084 Fisciano (SA), Italy. E-mail:boccig@diiie.unisa.it Mario Ferraro Dipartimento di Fisica Sperimentale and INFM Universitá di Torino via Giuria 1, 10125 Torino, Italy. Terry Caelli Department of Computing Science The University of Alberta, General Services Building Edmonton AB T6G 2E9 Canada 1 A bst r act The evolution of information in images undergoing fine-to-coarse anisotropic transformations is analyzed by using an approach based on the theory of irreversible transformations. In particular, we show that, when an anisotropic diffusion model is used, local variation of entropy production over space and scale provides the basis for a general method to extract relevant image features. Index terms. Scale space, anisotropic diffusion, entropy production, feature encoding. 1 I nt r oduct ion and Pr elim inar ies In a previous paper [3] we have proposed a method, based on the theory of thermodynamics of irreversible transformations, to compute variations of local entropy in images undergoing fine-to-coarse transformations. We have applied this method using an isotropic diffusion model to produce image segmentation where region boundaries were depicted by different rates of local entropy variation [3]. Here, we combine the method with an anisotropic diffusion model. It will be shown that the measure of local entropy variation, which gauges image structure dissipation, contains a term accounting for the bias or a priori information introduced by anisotropy. As a result, more precise and more stable feature encoding is obtained than by using isotropic diffusion [3]. ∗ Published in: IEEE Transactions on Pattern Analysis and Machine I ntelligence , vol.23, no.2, pp.207-211, (2001) 2 Let Ω be a subset of R 2 , (x, y) denote a point in Ω, and the scalar field I : (x, y) × t → I(x, y, t) represent a gray-level image. The non-negative parameter t defines the scale of resolution at which the image is observed; small values of t correspond to fine scales, while large values correspond to coarse scales. A scale transformation is assumed here to be a non-invertible or irreversible transformation of I, which preserves the total intensity, i.e., RR Ω I(x, y, t)dxdy = const. Let I ∗ denote the fixed point of the transformation, that is, I ∗ (·, 0) = I ∗ (·, t), for all t, and let I ∗ be stable, that is limt→∞ I(·, t) = I ∗ (·). Let f be the normalized version of RR I, that is f (x, y, t) = I(x, y, t) [ Ω I(x, y, t)dxdy]−1 , then f (x, y) can be interpreted as an estimate of the probability that a photon, from the 3D scene, impinge on the image domain Ω at the point (x, y). In [3], the conditional entropy H(f | f ∗ ) was introduced as the negative of the Kullback-Leibler distance. That is, H(f | f ∗ ) = − RR Ω f (x, y, t) ln ff(x,y,t) ∗ (x,y) dxdy, and ∂H(f ) ∂ Z Z ∂H(f |f ∗ ) = + f (x, y, t) ln f ∗ (x, y)dxdy, ∂t ∂t ∂t Ω where H(f ) = − RR Ω f (x, y, t) ln f(x, y, t)dxdy (1) is the Boltzmann-Gibbs entropy mea- sure. In this case the fine-to-coarse transformation can be modeled by a diffusion equation [4], namely ∂I(x, y, t)/∂t = ∇2 I(x, y, t), and the evolution of H(f |f ∗ ) is determined solely by P = ∂H(f )/∂t = σ(x, y, t) = RR Ω f(x, y, t)σ(x, y, t)dxdy, where ∇f(x, y, t) · ∇f (x, y, t) f (x, y, t)2 3 (2) is the density of entropy production in the thermodynamical sense. Since P ≥ 0 is non- negative, H(f) is an increasing function of t, and lim σ = lim P = 0. t→∞ 2 t→∞ I nfor mat ion fr om A nisot r opic Tr ansfor m at ions Anisotropic fine-to-coarse transformations can be modeled by generalizing the isotropic diffusion equation as ∂ f (x, y, t) = div(χ(x, y)∇f (x, y, t)), ∂t (3) where the function χ is used to force convergence of the diffusion process toward some desired image representation. For instance, Perona and Malik [7] assume χ to be a nonnegative, monotonically decreasing function of the magnitude of local image gradient. In this way the diffusion mainly takes place in areas where intensity is constant or varies slowly, whereas it does not affect areas with large intensity transitions. As a result, small variations in f such as noise can be smoothed while edges are retained. Unfortunately, this diffusion process does not incorporate a convergence criterion. Nordstrom [6] conjectured that, at the limit t → ∞, the image would converge to a piecewise constant image, and developed an algorithm called ”biased anisotropic diffusion”, closely related to Perona and Malik’s, which converges to a steady state solution without prespecifying the number of iterations. In discrete simulations, the diffusion process may diverge depending on the difference schemes and grid sizes [5], [8], [10]; a method was presented by Black et al. [1], 4 which is well-posed and robust, in the sense that, given a piecewise constant image, diffusion will leave the image unchanged. In this note we assume Nordstrom’s conjecture in order to avoid the use of specific well behaved diffusion, such as the one detailed in [1], which may not apply to the many different types of images of interest here. That is, we assume that, under a suitable choice of χ, and for t → ∞ the stationary point f ∗ of the transformation corresponds to a piecewise constant function with sharp boundaries between regions of constant intensity. In practice, such ideal fixed point can be obtained either by using a well-behaved diffusion such as the ones proposed in [1], [11], or by defining f ∗ (·) = f (·, t∗ ), with t∗ À 0 (as it is actually done in the simulations). Consider now the general form of ∂H(f |f ∗ )/∂t (see (1)); by taking into account that ∂ ∂t RR Ω f (x, y,t)dxdy = 0, after some algebra one obtains, by making use of (3): Z Z ∂ ∗ H(f |f ) = [ln f ∗ (x, y) − ln f (x, y, t)]div(χ(x, y)∇f(x, y,t))dxdy. ∂t Ω (4) Applying the Gauss divergence theorem and making use of Neumann’s boundary conditions, we eventually obtain the formula for the evolution of the conditional entropy H(f|f ∗ ) in the anisotropic case as: Z Z ∂ ∇f (x, y, t) · ∇f (x, y, t) ∗ H(f |f ) = dxdy f (x, y, t)χ(x, y) ∂t f(x, y, t)2 Ω Z Z − Ω f (x, y, t)χ(x, y) (5) ∇f (x, y, t) · ∇f ∗ (x, y) dxdy, f (x, y, t)f ∗ (x, y, t) thus, during anisotropic diffusion, the evolution of the conditional entropy H(f |f ∗ ) 5 is not determined solely by the entropy production P = ∂H(f )/∂t. Let P = RR 0 Ω f (x, y, t)σan (x, y, t)dxdy and S = RR 00 Ω f (x, y, t)σan (x, y, t)dxdy. A local measure σan of the variation rate of conditional entropy H(f |f ∗ ) can be defined by setting Z Z ∂ ∗ H(f|f ) = P − S = f (x, y, t)σan (x, y, t)dxdy, ∂t Ω (6) 00 0 and σan can be written as the sum of two terms, denoted by σan and σan respectively, 00 0 σan (x, y, t) = σan (x, y, t) − σan (x, y, t) = χ(x, y) ∇f(x, y, t) · ∇f ∗ (x, y) ∇f (x, y, t) · ∇f(x, y, t) − χ(x, y) .(7) f (x, y, t)2 f (x, y, t)f ∗ (x, y, t) It is worth noting that σan as defined in (7) is not necessarily always positive; however, numerical calculations of ∂H(f |f ∗ )/∂t and P, S, performed on a data set of 120 natural images provide evidence that, if χ is a non-negative decreasing function of |∇f |,then 0 < S < P, so that P − S > 0. We have used different kinds of images (animals, flowers, landscapes, objects, submarine, aerial) of various sizes and resolutions to evaluate the model. An example is provided by the image “Aerial1” presented in Fig. 1 (left image). ∗ (x,y,t)·∇f (x,y) is calculated by using an approximated The term σan = χ(x, y) ∇ff (x,y,t)f ∗ (x,y,t) 00 fixed point image fe∗ (x, y) ' f ∗ (x, y) obtained by letting the anisotropic diffusion run for large t. As previously discussed, for simulation purposes, large values of t provide a suitable fixed point approximation. 6 Figure 1: Left image: the original ”Aerial1” image. Right image: the fixed point of ”Aerial1” obtained through 2000 iterations of anisotropic diffusion. The graph below the images plots ∂H(f|f ∗ )/∂t, P, and S as a function of scale, represented by iterations of the anisotropic diffusion process. Units are nats/t. 7 The right image in Fig. 1 shows the fixed point for the ”Aerial1” image obtained for t = 2000 iterations and using a “balanced” backward sharpening diffusion as detailed in [11].In the same figure, the graphs show the plots of ∂H(f |f ∗ )/∂t and P, S, as functions of t. By definition it is clear that σan is a function of (x, y) and depends on the local characteristics of the image. This property holds also for the entropy production σ (see (2)), and it has been the basis for a method of region identification [3], which exploits the measurement of the activity of aσ across scales, defined as aσ (x, y) = R∞ 0 (x,y,t) dt. Consider a generalization of the activity concept to f (x, y, t) ∇f (x,y,t)·∇f f (x,y,t)2 the case of anisotropic diffusion. Since the anisotropic process performs information selection (see (7)), the region identification algorithm is, to some extent, simplified while increasing precision in identifying different information content structures. Define the activity of σan as aσa n (x, y) = define aσa0 n = R∞ 0 R∞ 0 f (x, y, t)σan (x, y, t)dt, and likewise 0 f (x, y, t)σan(x, y, t)dt, and aσ00 = an R∞ 0 00 f (x, y, t)σan (x, y, t)dt; then, aσa n = aσa0 n − aσa00n . 0 the activity aσa0 n is relatively large First note that, ideally, by definition of σan at points where different kinds of structures are present: textures, edges, etc. On the other hand, aσa00n is approximately different from zero only where | ∇f ∗ |6= 0 and this happens only along strong edges. Thus the total activity aσa n is different from zero only along weak edges. In other terms, aσa n and aσa00n naturally encode medium information content (m-type) and high information content (h-type) regions, respectively. Clearly, this is an ideal model. When dealing with real images, various sources of noise and degradation must be taken into account; however, the 8 information provided by aσa n and aσa0 n can be recovered by means of a statistical technique, as follows. Consider the probability distributions Pb (aσa n ), Pb (aσa00n ) estimated by means of the histograms of aσa n and aσa00n , respectively. Note that, in contrast with the isotropic case, anisotropic diffusion allows for the definition of two probability distributions which contain contributions from high and medium activity respectively - making the separation between the activity types clearer. Here, an iterative thresholding [3] is not needed (in the sense that preliminary h-type / m-type identification is incorporated in the process itself) but, rather, a onepass thresholding, applied to Pb (aσa n ) and to Pb (aσa00n ). Results using the anisotropic process are shown, for the ”Aerial1” image, in the right column of Fig. 2. They demonstrate that the use of anisotropic activity makes localization of different features in the image more precise; in particular, localization of edges is improved, since anisotropic diffusion avoids edge blurring and displacement. Moreover, the identification of h-type and m- type regions is much more biased to detect proper edge and texture regions, respectively. Clearly, the procedure discussed above encompasses more general aspects than just edge detection. However, since by using anisotropic activity, differently from isotropic activity proposed in [3], h-type regions should represent reliably edges, it may be of some interest to compare such results with those achieved by other methods. Fig. 3 provides an example where edge detection has been performed on the fixed point image of Fig.1, by adopting the same procedure used 9 Figure 2: Results of region identification by activity for image ”Aerial1” shown in Figure 1. The left column displays results obtained experimenting with the isotropic process [3]: the top image presents h-type regions, whereas the bottom image presents m-type regions. The two images in the right column are the corresponding results obtained exploiting the anisotropic process. In both experiments, the activity has been computed by integrating over 100 iterations; the anisotropic process used the fixed point achieved after 2000 iterations. 10 Figure 3: Edge detection results obtained on the fixed point of image ”Aerial1” shown in Figure 1, using the procedure reported in [7] 11 Figure 4: Results of region identification by means of anisotropic activity on the natural image ”Old Lady” characterized for varying details, low resolution and limited dynamic range. 100 diffusion iterations have been used, and a fixed point of 500 iterations. The top image is the original ”Old Lady”, images in the middle and on the bottom show, respectively, h-type regions and m-type regions. 12 in [7]. By comparing Fig. 3 with the top right image of Fig. 2, it can be noted a slight improvement, as regards the saliency of edge regions encoded. A further example of results obtained with the procedure proposed here is presented in Fig. 4. 3 D iscussion and Conclusion In this note we have considered the case of anisotropic diffusion for the extraction of features over space and scale. It has been shown that, in this case, the rate of change of information across scales does not depend solely on entropy production, but that, due to the characteristics of the process, the loss of information is, at least, partially prevented by a term that depends on the degree of parallelism between the gradient of the image at scale t and that of the image representing the fixed point of the anisotropic diffusion process. It has also been shown that different types of regions, features, can be derived from local measures of variation of conditional entropy and that this new model is an improvement over the isotropic case. The reason for this is that the anisotropic diffusion imposes constraints that limit the destruction of structures in the image, which represent strongly oriented features. Then the term S, which tends to prevent entropy growth, can be interpreted as accounting for a priori knowledge or information introduced by the anisotropic process as opposed to the isotropic process. The idea of applying information theory to scale-space representations has been discussed in a recent and interesting paper by Sporring and Weickert [9], who also 13 present a generalization to previous methods by studying the behavior of Renyi’s entropy throughout linear and non linear scale-spaces. However, a basic limitation of such approach is that only global measures can be defined for an image. Although this can be useful when general characteristics (e.g., finding the fingerprint of a texture [9]) are of interest, it cannot encode structures localized in space and spatial scale. An issue that remains to be investigated is the extension to vector-valued images of the framework developed here and in [3]. In this case, the transformation from fine-to-coarse representations can be modelled by a generalization of the diffusion equation, namely ∂Ii ∂t = −div J~i , where J~i is a flow and the index i labels the bands of the image I~ = {I1 , ..., In }, and by considering flows as functions of the thermodynamical forces expressed by the phenomenological laws of irreversible processes. Preliminary results for color images have been reported in [2]. R efer ences [1] M. J. Black, G. Sapiro, D.H. Marimont, and D. Heeger, ”Robust Anisotropic Diffusion”, IEEE Trans. on Image Processing, vol. 7, n.3, pp. 421-432, 1998 [2] G. Boccignone, M. Ferraro and T. Caelli, ”Entropy Production in Colour Images”, Proc. 15th Int’l Conf. Pattern Recog., IEEE Computer Soc. Press, vol.1, pp.202-205, Barcelona, Spain, September 2000. [3] M. Ferraro, G. Boccignone and T. Caelli, “On the Representation of Image Structures via Scale Space Entropy Conditions”, IEEE Trans. Pattern Analysis 14 and Machine I ntelligence,vol.21, no. 11 , pp. 1199-1203, 1999. [4] T. Lindeberg, Scale-space Theory in Computer Vision. Dordrecht, The Netherlands: Kluwer Academic Press, 1994. [5] M. Nitzberg and T. Shiota, ”Nonlinear image filtering with edge and corner enhancement”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol.14, pp.826-833, 1992. [6] N. Nordstrom, ”Biased anisotropic diffusion: A unified regularization and diffusion approach to edge detection”, I mage Vis. Comput., vol. 8, pp.318-327, 1990. [7] P. Perona and J. Malik, ”Scale-space and Edge Detection Using Anisotropic Diffusion”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol.12, pp.629-639, 1990. [8] L.I. Rudin, S. Osher, and E. Fatemi, ”Nonlinear total variation based noise removal algorithms”, Physica D, vol. 60, pp. 259-268, 1992. [9] J. Sporring and J. Weickert, ”Information Measures in Scale-Spaces”, IEEE Trans. Information Theory, vol. 45, n.3, pp.1051-58 , 1999. [10] R.T. Whitaker and S.M. Pizer, ”A multi-scale approach to non-uniform diffusion”, CVIGP: Image Understanding, vol. 57, no 1, pp. 99-110, 1993. 15 [11] Y. L.You, X. Wenyuan, A. Tannenbaum and M. Kaveh, ”Behavioral Analysis of Anisotropic Diffusion in Image Processing”, IEEE Trans. Image Processing, vol. 5, no.11, pp. 1539-1552, 1996. 16