ITERATIVE METHODS FOR TOTAL VARIATION DENOISING. C. R. VOGEL AND M. E. OMAN y

Transkript

1 ITERATIVE METHODS FOR TOTAL VARIATION DENOISING C. R. VOGEL AND M. E. OMAN y Abstract. Total Variation (TV) methods are very eective for recovering \blocky", possibly discontinuous, images from noisy data. A xed point algorithm for minimizing a TV-penalized least squares functional is presented and compared with existing minimization schemes. A variant of the cell-centered nite dierence multigrid method of Ewing and Shen is implemented for solving the (large, sparse) linear subproblems. Numerical results are presented for one- and two-dimensional examples; in particular, the algorithm is applied to actual data obtained from confocal microscopy. Key words. Total Variation, denoising, image reconstruction, multigrid methods, confocal microscopy, xed point iteration.. Introduction. The problem of denoising, or estimating an underlying function from error-contaminated observations, occurs in a number of important applications, particularly in probability density estimation and image reconstruction. Consider the model equation (.) z = u + ; where u represents the desired true solution, represents error, and z represents the observed data. A number of approaches can be taken to estimate u. These include spline smoothing (see [8]), ltering using Fourier and wavelet transforms, and Total Variation (TV) based denoising. Figure below illustrates the qualitative dierences between these various approaches on a simple one-dimensional test case. It is not the goal of this paper to carry out an exhaustive comparison of TV with standard denoising methods. For that, see [5] and the analysis in [5]. Suce it to say that TV denoising is extremely eective for recovering \blocky", possibly discontinuous, functions from noisy data. It is the goal of this paper to present a new algorithm for TV denoising and to compare it to some existing TV-based methods. In their seminal paper on TV denoising, Rudin, Osher, and Fatemi [5] considered the constrained minimization problem, (.) min u jruj dx subject to ku? zk = ; where the parameter describes the magnitude of the error in the data in the model equation (.). Here is a bounded, convex region in d-dimensional space, j j denotes the Euclidean norm in R d, and k k denotes the norm on L (). Here we will consider a closely related problem the minimization of the TVpenalized least squares functional (.3) f(u) = ku? zk + J (u); Department of Mathematical Sciences, Montana State University, Bozeman MT 5977, vogel@math.montana.edu. Research was supported in part by the NSF under Grant DMS-933. y Scalable Computing Laboratory, Ames Laboratory (USDOE), Iowa State University, Ames, IA 5, oman@scl.ameslab.gov. Research was supported in part by a DOE-EPSCoR graduate fellowship at Montana State University.

2 A) Exact and Noisy Data B) Sobolev H Reconstruction.5 C) TV Reconstruction.5 D) Fourier Reconstruction.5 E) Haar Reconstruction.5 F) Daubechie Reconstruction.5.5 Fig.. Denoised reconstructions obtained using a variety of ltering techniques. Dotted lines represent noisy data. Solid line in subplot A is exact solution. Solid lines in subplots B-F are reconstructions. Reconstructions E and F were obtained using Haar and Daubechie wavelets, respectively. See section 4 for details.

3 where (.4) J (u) = q jruj + dx and and are (typically small) positive parameters. The parameter controls the tradeo between goodness of t to the data, as measured by ku? zk, and the variability of the solution, measured by J (u). When =, J (u) represents the surface area of the graph of u, while = gives the total variation of u. When =, TV-penalized least squares can be viewed as a penalty method (see []) to solve the constrained problem (.). The penalty parameter in (.3) is inversely proportional to the Lagrange multiplier for (.). This penalty approach is standard in the inverse problems community, and is commonly referred to as Tikhonov regularization. Provided the parameters are selected appropriately, the solutions obtained by these two methods are identical. However, from a computational standpoint, unconstrained problems are much easier to implement than constrained problems. To solve their constrained minimization problem, Rudin, et al, applied articial time evolution. In the context of the unconstrained problem (.3), this amounts to assuming u is a function of time t (as well as space) and then time-integrating the dierential equation t > ; (.6) u = u () ; t = ; to steady state. Here g(u) denotes the gradient (derivative with respect to u) of the TV-penalized least squares functional (.3), and u () is an initial guess for the solution. Formally, g is a nonlinear elliptic partial dierential operator with homogeneous Neumann boundary conditions (.7) (.8) g(u) = u? r q jruj + A? z; = ; After spatial discretization, Rudin, et al, applied explicit (forward Euler) time marching to obtain a gradient descent scheme. In the context of (.5), this approach yields (.9) u (k+) = u (k)? k g(u (k) ); k = ; ; : : : : A line search (see [4]) can be added to select the step size k in a manner which gives sucient decrease in the objective functional in (.3) to guarantee convergence to a minimizer. This gives the method of steepest descent (see []). While numerical implementation is straightforward, steepest descent has rather undesirable asymptotic convergence properties which can make it very inecient. Obviously, one can apply 3

4 other standard unconstrained optimization methods with better convergence properties, like the nonlinear conjugate gradient method or Newton's method. These methods converge rapidly near a minimizer provided the objective functional depends smoothly on u. When =, the term J in (.3) is not dierentiable. For small values of, the near nondierentiability of the objective functional results in a loss of robustness and eciency for higher order methods like Newton's method. In this paper we introduce an alternative approach to minimizing (.3) which we call \lagged diusivity xed point iteration", denoted by FP. At a minimizer, g(u) =, or equivalently, (.) u + L(u)u = z; where L(u) is the diusion operator whose action on a function v is given by (.) L(u)v q rva : jruj + FP iteration can be expressed as + L(u (k) ) u (k+) = z; k = ; ; :::: (.) Note that at each iteration, one must solve a linear diusion equation, whose diusivity depends on the previous iterate u (k), to obtain the new iterate u (k+). In our numerical experiments, we have observed global convergence of FP iteration. We suspect this is true in general because the mapping u 7! L(u)u is monotone (see [9]). Hence, there appears to be no need for a \globalization" procedure like a line search to guarantee convergence, as is the case with standard optimization methods. In addition, this method exhibits rapid linear convergence for a broad range of the parameters and. In the following section, we discuss the mathematical structure of the TV-penalized least squares functional (.3) and the equations which arise in its minimization. Section 3 deals with numerical implementation issues like discretization, stopping criteria, and iterative methods to solve (large, sparse) linear systems. In the nal section we present a numerical comparison of FP iteration with Newton's method and steepest descent. Results of a numerical study of the eects of various parameters (e.g., and in (.3)) are also presented in this section. Finally, we apply FP iteration to denoise actual data obtained from a confocal scanning microscope [9].. Mathematical Structure. In this section we discuss the mathematical structure of the TV-penalized least squares functional (.3) and the implications of this structure for numerical methods. An analysis of a similar functional, (.) kku? zk + J (u); has been carried out in [], where K is a compact linear operator mapping L p () into L (), with p < d=(d? ). d is the spatial dimension. This analysis relies on the relative compactness of sets of the form (.) fu L p () : kuk p + J (u) Bg; 4

5 where B is a xed constant, in L p (). Perhaps the most important dierence in the analysis of (.3) is the fact that sets of the form (.) are not relatively compact in L () for dimensions d >. Following section of [], dene the functional (.3) Also dene Q(u; ~v) = (u? z)? u r ~v + q? j~vj dx: (.4) f(u) = sup Q(u; ~v) ~vv where (.5) V = f~v C (; R d ) : j~v(x)j g: For u suciently smooth (say, in C ()), this coincides with the functional in (.3)- (.4). We wish to nd u L () for which (.6) f(u ) = inf f(u): ul () Theorem.. Problem (.6) has a unique solution. Proof. Note that f(u) is L -coercive, i.e., f(u)! whenever kuk!. This combined with the weak compactness of closed balls in L () and the weak lowersemicontinuity of f yields existence. Uniqueness follows from the strict convexity of the L norm. While we have not yet been able to rigorously establish this fact, it appears that the minimizer is stable with respect to perturbations in the data z and the parameters and. This has the following implications: (i) Taking small but positive in (.3) gives minimizers which are close (in an L sense) to minimizers obtained with =. This is illustrated in Figure A. (ii) As!, the minimizer u = u in (.3) tends to the (noisy) data z. On the other hand, as becomes large, u tends to the mean value of z. This is illustrated in Figure B. We next examine the operators arising in the minimization of (.3). Taking the rst Gateaux derivative in the direction v and assuming u and v are smooth, one obtains the gradient (.7) hg(u); vi (u? z)v q ru rv jruj + A A dx: Similarly, one obtains the Hessian (.8) hh(u)v; wi =! rv rw vw + dx: (jruj + ) 3= 5

6 .6 A) for various beta.5 B) for various alpha Fig.. Subplot A shows TV reconstructions for various 's with xed = :. Solid line corresponds to = :, dashed line to = :, and dotted line to =?6. Subplot B shows reconstructions for various 's with xed =?4. Solid line corresponds to = :, dashed line to = :, and dotted line to = :. See Section 4 for details. The quadratic approximation (.9) f(u + s) = f(u) + hg(u); si + hh(u)s; si + o(ksk ) is the basis for the analysis of standard optimization methods. In particular, the spectrum of the Hessian H(u) determines the asymptotic convergence rate of the methods of steepest descent and the (nonlinear) conjugate gradient method (see []). One obtains local quadratic convergence for Newton's method assuming that H(u) is Lipschitz continuous and has a bounded inverse (see [4]). The size of the convergence region is proportional to the inverse of the Lipschitz constant and the bound on the inverse. Note that if = and ru vanishes anywhere in, then f(u) is not dierentiable. This is a consequence of the nondierentiability of the Euclidean norm. This diculty is overcome when >, but the Lipschitz constant behaves like?3 when is small. This explains the deterioration in the performance of Newton's method observed in section 4 below for small values of. Again assuming sucient smoothness and applying Green's identity (integration by parts) in (.7), one obtains (.) hg(u); vi = (u? z)? ru n q jruj + ds; ru jruj + A A v dx where n is the outward unit normal to the Consequently, a minimizer 6

7 for (.3) is a weak solution to the nonlinear second order elliptic PDE (.) g(u) def = u? r q jruj + A? z = ; x ; (.) This can be expressed in = ; (.3) A(u)u = ( + L (u))u = z where (.4) and (.5) L (u)v =?r ( (u)rv) (u) = q jruj + Note that the operator L is symmetric and positive semidenite. Its null space consists of the constant functions. The diusivity (u) is bounded above by and below by zero. Finally, observe that the Hessian is related to the operators A(u) and L (u) by (.6) H(u) = + L (u) + L (u)u = A(u) + L (u)u; where denotes dierentiation with respect to u. 3. Numerical Implementation. Any discretization of (.3) should allow for very sharp gradients without introducing spurious oscillations in the solution u. We have implemented several Finite Element Methods (which are based on (.7)) with piecewise linear basis functions and the rst order accurate nite dierence scheme described in [5] (based on (.)-(.)). The resulting discrete systems are quite similar. These schemes yield discrete Hessians H, c.f., (.8), and matrix operators A, c.f., (.3), which are symmetric and positive denite (SPD) and sparse tridiagonal in one space dimension and block tridiagonal in two dimensions. Let u and z denote the vectors (mesh functions) obtained from the discretization of u and z, respectively. Also, denote by f(u), g(u), H(u), and A(u), respectively, the discretization of the objective functional (.3), its gradient, Hessian, and the operator A in (.3). What follows is a generic algorithm for the minimization of f(u). The superscript (k) denotes iteration count. Let u () be an initial guess. For k = ; ; : : : ;. Compute a descent direction d (k) for f at u (k).. u (k+) = u (k) + d (k), where = argmin > f(u (k) + d (k) ): 7

8 3. Check stopping criteria. For the method of steepest descent, d (k) SD =?g method, (k) def =?g(u (k) ), while for Newton's (3.) d (k) N =?H(u(k) )? g (k) : Note that the line search in step may be replaced by a trust region method [4]. Either \globalization" technique will guarantee convergence to the minimizer of f. For our FP iteration, steps and are combined. One obtains u (k+) directly by solving the linear system (3.) A(u (k) )u (k+) = z: Setting d (k) F P = u (k+)? u (k) yields (3.3) (3.4) d (k) F P =?A(u (k) )? A(u (k) ) u (k)? z =?A(u (k) )? g(u (k) ): The second equality follows from (.) and (.4). Hence FP iteration is of quasi- Newton form, and existing convergence theory in [4] can be applied. Since the matrix A(u) is SPD with its minimum eigenvalue bounded away from zero, each of the d (k) F P 's is a descent direction, and global convergence can be guaranteed by \globalization", i.e., appropriate step size control. In our computational experiments, we have not found globalization to be necessary. Comparing (3.) and (3.3) and observing that the term L (u)u in (.6) does not vanish, the asymptotic convergence rate is linear. The following stopping criteria are standard (see [4]). ;, and 3 are user-dened tolerances, k max is an iteration limit, and k k denotes the ` norm. (3.5) ku (k+)? u (k) k ; (3.6) kg (k+) )k ; (3.7) f(u (k) )? f(u (k+) ) 3 ku (k+)? u (k) k; (3.8) k k max : Solving the Linear Systems. With both Newton's method (c.f., (3.)) and FP iteration (c.f., (3.)), one must solve a sparse SPD linear system at each iteration. In one space dimension, these systems are tridiagonal and can be solved directly in O(n) operations, where n is the order of the system. In two space dimensions, these systems are block tridiagonal. Direct banded system solvers, which have complexity O(n 3= ), may be applied. One may also apply a variety of iterative methods, like preconditioned 8

9 conjugate gradient (PCG) methods (see [, ]) and multigrid methods (see [] and the references therein). For certain linear SPD systems arising in the discretization of elliptic PDE's, one can achieve O(n) complexity with multigrid methods []. Our early multigrid implementations, which were based on standard nite dierence or nite element discretizations and standard intergrid transfer operators, yielded very disappointing results when u was not smooth. This seems to be due to properties of the diusion coecient (u), c.f., equation (.5). For nonsmooth u of bounded variation, (u) is not smooth. Moreover, on the set (having Lebesgue measure zero) where u is discontinuous, (u) vanishes. To overcome these diculties, we employed a cell-centered nite dierence (CCFD) discretization (see [7] and the references therein). To solve the resulting linear systems, we applied a variant of the multigrid algorithm developed by Ewing and Shen [8]. While pure multigrid iterations were considered in [8], we applied a PCG iteration with a CCFD-based multigrid method as a preconditioner. See [7] for implementation details. Numerical results appear in the following section. 4. Numerical Results. In this section we rst present a numerical comparison of FP iteration, Newton's method, and Steepest Descent applied to minimize the TVpenalized least squares functional (.3). Results are also presented which illustrate the eects of varying the parameters and on these methods. First consider the one-dimensional test problem of denoising the data presented in Figure. The exact (noise-free) solution is (4.) u exact (x) = [=6;=4] + 3 [=3;5=8]; where [a;b] denotes the indicator function for the interval a x b. The data was generated by evaluating u exact at N = 57 equispaced points in the interval x and adding pseudo-random, uncorrelated error (so-called \discrete white noise") f i g N i= having a Gaussian distribution with mean and variance selected so the noise to signal ratio (4.) vu u t E(P N P N i= u(x i) = :5: i= i ) Here E() denotes mathematical expectation. Subplot C shows the minimizer of the TV-penalized least squares functional (.3). Subplot B shows the minimizer of a related H -penalized least squares functional, which is commonly used in data smoothing (see [8]), (4.3) ku? zk + jruj dx: Here the penalty term is the square of the Sobolev H seminorm. It does not allow discontinuous minimizers. On the other hand, it is easy to compute minimizers and is appropriate for denoising smooth functions. Subplot D was obtained by Fourier transforming the data, applying a low pass lter, and then applying the inverse transform. 9

10 Subplots E and F were created with the aid of the software package wavethresh, as documented by Nason and Silverman [4]. Each of these reconstructions was obtained by applying a wavelet transform to the data, applying the universal lter of Donoho and Johnstone [6], and then applying an inverse wavelet transform. In Subplot E, Haar wavelets are used in the transformations (see [6] or [3] for a discussion of Haar wavelets). These wavelets are generated by a discontinuous mother wavelet and are of regularity level (see [4]). This reconstruction clearly maintains the discontinuities of the true image; however, there appear to be extraneous eects similar to ringing which are not a part of the original image. Subplot F uses Daubechies' \extremal phase" wavelets (see [3]) which have regularity level (see [4]). Here, both smoothing and ringing eects are apparent. In all cases, the lter parameters were selected so that (4.4) NX i= (u(x i )? z i ) =N : Figure shows the qualitative eects of varying the parameters and on the minimizer of the TV-penalized least squares functional (.3). In subplot A, the xed = : is selected so (4.4) is satised, and is varied. Larger values of have the eect of \rounding o sharp edges" in the reconstructions. In subplot B, =? is xed and is varied. Solutions tend to be piecewise constant. Larger values of have the eect of reducing the number of piecewise constant regions. Figure 3 illustrates the convergence behavior of the various methods for minimizing (.3), as measured in the ` norm of the gradient. In each case, the initial guess was taken to be the zero solution, i.e., u () (x) =, and = :. Subplots A and B also show the eect of varying the parameter on the performance of Newton's method with a line search and FP iteration. Note that the Newton iteration converges rapidly for relatively large values of. However, as decreases, the performance decreases markedly. The line search restricts the size of steps in order to maintain a steady decrease in f(u (k) ), but quadratic convergence is not attained until a very large number of iterations have been performed. See the discussion in Section for an explanation. FP performance also drops o as decreases, but unlike Newton's Method, the drop o is gradual and there is no dramatic change as becomes very small. Subplot C illustrates performance of the method of steepest descent. There is a substantial decrease in the norm of the gradient in the rst few iterations, but after that the decrease is extremely slow. A thousand steepest descent iterations were required to obtain reconstructions comparable to those obtained with 4 or 5 Newton or FP iterations. Figure 4 shows FP performance as measured by the objective functional (.3). Note that f(u (k) ) decreases monotonically. Finally, we present a two-dimensional example from confocal microscopy (see [9]). The images in subplots A and B of Figure 5 show rod-shaped bacteria on a stainless steel surface. The vertical axis represents recorded light intensity, while the horizontal axes represent scaled pixel locations on a 6464 grid. Figure 5A is an actual image recorded with a scanning confocal microscope. Figure 5B shows a TV reconstruction obtained from the FP algorithm with our CCFD multigrid PCG method used to solve the linear

11 A) FP Iteration B) Newton Iteration - gk / g - -3 gk / g iteration k C) Steepest Descent iteration k gk / g - 5 iteration k Fig. 3. Performance of methods measured by the scaled ` norm of the gradient, kg (k) k =kg () k. = : is xed throughout. Subplot C shows Steepest Descent performance for = only. Subplot A shows FP performance for = (solid line), = : (dashed line), =? (dash-dotted line), and =?3 (dotted line). Subplot B shows Newton performance for = (solid line), = : (dashed line), =? (dash-dotted line), and =?3 (dotted line). f(uk) - f(u_exact) for Fixed Point Iteration - - delta f(uk) iteration k Fig. 4. Performance of Fixed Point Iteration measured by f(u (k) )? f(u ), where u is the minimizer of f. Solid line is for = and dashed line is for = :. is xed at :.

12 A) Noisy data B) TV denoised data C) PCG convergence factor D) PCG residual norms Fixed point iteraton PCG iteration Fig. 5. Subplot A shows a scanning confocal microscope image of rod-shaped bacteria on a stainless steel surface. Subplot B shows a TV reconstruction obtained using FP iteration. Subplots C and D illustrate performance of the linear system solver. systems at each xed point iteration. The actual computations were performed on a pixel grid, with n 65; unknowns. So that ne details are not obscured, only the upper left hand subgrid is actually displayed. Parameter values are = = :. Subplots C and D describe the performance of the CCFD multigrid PCG linear system solver. Subplot D shows the norms of residuals r j = z? Au j as a function of PCG iteration count at FP iteration. A = A(u () ) is xed throughout for this subplot. Dene the PCG convergence factor to be the ratio j = kr j k=kr j? k. Subplot C shows the geometric mean of the convergence factors, = exp( P m j ln j =m), as a function of FP iteration count. Finally, we note that far fewer that xed point iterations and PCG iterations per FP iteration were required to obtain comparable denoised images. The purpose of the large number of iterations was to demonstrate the asymptotic convergence properties of the linear iterative method.

13 REFERENCES [] R. Acar and C. R. Vogel, Analysis of Total Variation penalty methods for ill-posed problems, Inverse Problems, vol. (994), pp [] O. Axelsson and V.A. Barker, Finite Element Solution of Boundary Value Problems, Academic Press, 984. [3] I. Daubechies, Orthonormal Bases of Compactly Supported Wavelets, Communications on Pure and Applied Mathematics, vol. 4 (988), pp [4] J. E. Dennis, Jr., and R. B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, 983. [5] D. Dobson and F. Santosa, Recovery of blocky images from noisy and blurred data, Tech. Report No. 94-7, Center for the Mathematics of Waves, University of Delaware (994). [6] D. L. Donoho and I. M. Johnstone, Ideal spatial adaptation by wavelet shrinkage, Biometrika, vol. 8, (994), pp [7] R. E. Ewing and J. Shen, A discretization scheme and error estimate for second-order elliptic problems with discontinuous coecients, preprint, Institute for Scientic Computation, Texas A & M University. [8] R. E. Ewing and J. Shen, A multigrid algorithm for the cell-centered nite dierence scheme, in the Proceeding of the 6 th Copper Mountain Conference on Multigrid Methods, April 993. [9] S. Fucik and A. Kufner, Nonlinear Dierential Equations, Elsevier, 98. [] G. H. Golub and C. F. VanLoan, Matrix Computations, nd ed., The Johns Hopkins University Press, 989. [] D. Luenberger, Introduction to Linear and Nonlinear Programming, Addison-Wesley, 965. [] S. F. McCormick, ed., Multigrid Methods, SIAM, 987. [3] Y. Meyer, Wavelets: Algorithms and Applications, SIAM, 993. [4] G. P. Nason and B. W. Silverman, The Discrete Wavelet Transform in S, Journal of Computational and Graphical Statistics, vol. 3 (994), pp [5] L.I. Rudin, S. Osher, and E. Fatemi, Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, vol 6 (99), pp [6] G. Strang, Wavelet Transforms Versus Fourier Transforms, American Mathematical Society Bulletin, vol. 8 (993), pp [7] C. R. Vogel, A multigrid method for Total Variation-based image denoising, in Computation and Control IV, conference proceedings to be published by Birkhauser. [8] G. Wahba, Spline Models for Observational Data, SIAM, 99. [9] T. Wilson, ed., Confocal Microscopy, Academic Press, 99. 3