Introduction to Image Processing§

author:	David Coeurjolly

Value based Approaches§

Principles§

We only focus on values of the functions $\, f: S\subset\mathbb{Z}^n\rightarrow Q$ using stochastic representation

Important object: histograms

$H: Q \rightarrow [0,1]$

$H(i) = \frac{Card \{ p\in S \,|\, f(p) =i \}}{Card S}$

The image can be seen as a random variable with probability density function $H$

Exemple: $S=[0,185]\times[0,85],\, Q=[0,255]$

Histogram based image processing§

Histograms sum up colorimetric information of the image without considering spatial relationships.

Image description/indexing
- Image characterized by its histogram (ou by descriptors consturcted from the histogram)
- distance between two images == distance between image histograms == distance between two probability distributions
Structure recognition
- Foreground/background analysis from histogram shape analysis
- Noise modeling
Histogram optimization, equalization, …
Perform transformations on the histogram and propagate the changes to image values

Histogram transformations§

Few examples:

Translation

global luminosity shift
$H'(i) = H(i+t)$

Histogram remapping

e.g. using linear form $\,l: [a,b]\subset[0,1] \rightarrow [0,1]$
$H'(i) = H( a + i(b-a) )$

Inversion

$H'(i) = H(1 - i )$

…

More generally

$\phi: [0,1] \rightarrow [0,1]$

$H'(i) = H(\phi(i))$

Propagation to image $f'(p) = \phi(f(p))$ <demo>

Histogram Equalization and Histogram Specifications§

In previous examples, $\phi$ was prescribed. Now, we are looking for $\phi$ to map histogram $H$ (empiric or observed) to $H'$ (model).

Example: Equalization

Goals : equalize the intensity values dynamic using a spreading of the histogram
Target distribution $H'$ : uniform distribution
Effect: contrast enhancing

Equalization (bis)§

Be careful not always a good idea

Histogram based Segmentation§

Image Segmentation classify pixels into classes such that intra-class pixels share the same visual properties (colorimetric information, geometrical properties …)

Let’s go back to initial image:

Histogram Thresholding

Naive approach to get binary segmentation (2 classes)
We can analyze statistical properties of each class to find the best threshold (variance minimization, entropic thresholding, …)
Use cases are quite limited

Color transfer using Optimal Transport§

Principles§

Take into account spatial relationships between pixel values

Key tool: convolution

Image as a function $f:\,\mathbb{R}^2\rightarrow \mathbb{R}$
We consider the result of the convolution product of $f$ by $g:\,\mathbb{R}^2\rightarrow \mathbb{R}$

$(f * g)(x,y) = \int_{-\infty}^\infty f(x-s,y-t)g(s,t)dsdt$
or a discrete version for compact support kernel $g$

$(f * g)(x,y) = \sum_{i=-M}^M\sum_{j=-M}^M f(x-i,y-j)g(i,j)$

Algorithmically

Direct discrete computation of the support is small
Or by regular product on Fourier space:

$f * g = \mathcal{F}^{-1}( \mathcal{F}(f)\cdot\mathcal{F}(g))$

Fourier Space§

$\mathcal{F}(u)=\int_{-\infty}^\infty f(x)\cdot e^{-i u x}dx \qquad {f}(x)=\int_{-\infty}^\infty \mathcal{F}(u)\cdot e^{i u x}du$

$F_k=\sum _{{n=0}}^{{{\mathrm {N}}-1}}x_n\cdot e^{{-2i\pi kn / N}}\qquad x_n={\frac {1}{{\mathrm {N}}}}\sum _{{k=0}}^{{{\mathrm {N}}-1}}{ {F}}_k\cdot e^{{2i\pi n k /{N}}}$

(for $0\leqslant k<{\mathrm {N}}$ )

Few Fourier Space Properties§

Linearity $\quad h(x)= af(x)+bg(x) \Leftrightarrow \mathcal{H}(u)=a\mathcal{F}(u)+b\mathcal{G}(u)$
Scaling

$h(x) = f(a\cdot x) \Leftrightarrow \mathcal{H}(u)=\frac{1}{|a|}\mathcal{F}\left(\frac{u}{a}\right)$
Transaltion

$h(x) = f(x_0 + x) \Leftrightarrow \mathcal{H}(u)=e^{i2\pi x_0 u} \mathcal{F}(u)$
Modulation for real $u_0$

$\mathcal{H}(u)=\mathcal{F}(u+u_0) \Leftrightarrow h(x) = e^{i2\pi xu_0}f(x)$
Convolution

$h(x) = (f*g)(x) = \int_{-\infty}^\infty f(t)g(x-t)dx\Leftrightarrow \mathcal{H}(u)=\mathcal{F}(u)\cdot\mathcal{G}(u)$
DCT component

$\mathcal{F}(0) = \int_{-\infty}^\infty f(x)dx$

(magnitude of Fourier transform leads to translation and rotational invariant descriptors)

Illustrations§

(http://www.cs.unm.edu/~brayer/vision/fourier.html)


Pure cosine signals	Magnitude vs. Phase	Real image example

Illustrations§

(http://www.cs.unm.edu/~brayer/vision/fourier.html)


Continuous vs. discrete FT	Edge effect reduction through convolution

Periodic structure:

Illustrations§

(http://www.cs.unm.edu/~brayer/vision/fourier.html)


Lowpass filter	Highpass filter

Elementary Image Filtering§

Be careful: $\int_{-\infty}^\infty g(s,t)dsdt = 1$

Smoothing / Low-pass filters

$M_{3\times 3} = \frac{1}{9} \begin{bmatrix} 1 & 1 & 1\\1 &1 & 1\\1 & 1 & 1\\ \end{bmatrix}$

$G_{3\times 3} = \frac{1}{16} \begin{bmatrix} 1 & 2 & 1\\2 &4 & 2\\1 & 2 & 1\\ \end{bmatrix}$

$G_{5\times 5} = \frac{1}{256} \begin{bmatrix} 1 & 4 & 6 &4 & 1\\ 4 & 16 & 24 & 16 &4\\6 & 24 & 36 & 24 & 6\\ 4 & 16 & 24 & 16 &4\\1 & 4 & 6 &4 & 1\\ \end{bmatrix}$

Mask Filtering Design§

G : Gaussian kernel approximation function

$g(x,y) = \frac{1}{\sqrt{2\pi}\sigma}\exp^{- \frac{x^2+y^2}{2\sigma} }$

$\Rightarrow$ binomial coefficients, ….

Examples§

$M_{3\times 3}$
$G_{3\times 3}$
$G_{5\times 5}$

Noise Models§

Match between filtering and noise model

Either as a theoretical hypothesis on the signal
Or from knowledge on the device or acquisition system

Gaussian noise
Specular noise salt/pepper
Contour spread

High pass filter§

Complementary filters to low-pass filters

$M_{3\times 3} = \frac{1}{9} \begin{bmatrix} -1 & -1 & -1\\-1 &8 & -1\\-1 & -1 & -1\\ \end{bmatrix}$

$G_{3\times 3} = \frac{1}{16} \begin{bmatrix} -1 & -2 & -1\\-2 &12 & -2\\-1 & -2 & -1\\ \end{bmatrix}$

Non-linear filters§

Median filter

In the support, we compute the median intensity

example: $\begin{bmatrix} 12 & 13 & 24\\1 &30 & 43\\3 & 15 & 20\\ \end{bmatrix}$

$\Rightarrow$ 15

perfect for specular noise
good preservation of contours

…but…

non-linear
non analytical model in Fourier space

complexity for a size $m$ domain?

Contour Detection and Differential Estimators§

Objectives§

Context

Region : set of pixels sharing the same colorimetric information
By analogy, contours are the loci high colorimetric variations

$\Rightarrow$ loci if high gradient vector norm of the function $f(x,y)$

$\Rightarrow$ Zero-crossing of the Laplacian $\Delta f$ …

We will focus later to segmentation process.

Image Gradient§

Definition

$\nabla f(x,y) = \left (\frac{\partial f}{\partial x}(x,y), \frac{\partial f}{\partial y}(x,y)\right)$

Finite difference approximation

$Prewitt_x = \frac{1}{3} \begin{bmatrix} -1 & 0 & 1\\-1 &0 & 1\\-1 & 0 & 1\\ \end{bmatrix} Sobel_x = \frac{1}{4} \begin{bmatrix} -1 & 0 & 1\\-2 &0 & 2\\-1 & 0 & 1\\ \end{bmatrix} Kirsch_x = \frac{1}{15} \begin{bmatrix} -3 & -3 & 5\\-3 &0 & 5\\-3 & -3 & 5\\ \end{bmatrix}$

(other masks exist along specific directions $\theta$ )

Image Gradient (bis)§

Amplitude = gradient vector norm

$\|\nabla f\|_2 = \sqrt{\left(\frac{\partial f}{\partial x}\right)^2 + \left(\frac{\partial f}{\partial y}\right)^2} \|\nabla f\|_1 = \left |\frac{\partial f}{\partial x}\right| + \left |\frac{\partial f}{\partial y} \right| \|\nabla f\|_\infty = \max\left(\left |\frac{\partial f}{\partial x}\right|, \left |\frac{\partial f}{\partial y}\right|\right)$

Orientation

$\theta = atan\left( \frac{\frac{\partial f}{\partial y}}{\frac{\partial f}{\partial x}}\right)$

$\Rightarrow$ a contour can be characterized has pixels with high gradient vector norm

Example§

(Prewitt)

Mask Design§

Gradient of ta filterer image and Gaussian smoothing filters

$\nabla (f*g) = \nabla f * g = f*\nabla g$

$\Rightarrow$ let us consider Gaussian kernel filter $g(x,y) = \frac{1}{\sqrt{2\pi}\sigma}\exp^{- \frac{x^2+y^2}{2\sigma} }$ , we have

$\frac{\partial g(x,y)}{\partial x} = -\frac{x}{\sigma^2\sqrt{2\pi}\sigma}\exp^{- \frac{x^2+y^2}{2\sigma} }$


g	g’	g’‘

Separable Filters§

Let $g(x,y)$ be a convolution filter ( $\equiv$ Impulse response of a filter). The filter is separable if

$g(x,y) = g_x(x,y)g_y(x,y)$

Thus:

$f*g = g_x * (g_y*f)$

and for partial derivatives:

$\frac{\partial (f*g)(x,y)}{\partial x} = f(x,y)* \left( g_x(x)\frac{dg_y}{dy}(y)\right)$

$\Rightarrow$ direct consequences when implementing filters (only 1D convolutions)

If $g=[1\, 1 \,1]$ and $d = \nabla h = [-1\, 0 \,1]$ then $Prewitt_x = g(x).d(y)$

i.e. Prewitt’s filter corresponds to finite difference approximation of the result of a constant smoothing filtering of $f$

Gaussisan filters are separable (and isotropic)

Theoretical Design of a Contour Detector§

Canny’s criteria (1983)

Detection: minimize false answers
Localization: the contour must be located with high precision (minimize the distance to real contour)
Unique answer: a single answer by contour (no ghosts)

Noise model + signal model + Criteria modeling + initial conditions $\Rightarrow$ Optimal filter as solution of a PDE

$2g(x) - 2\lambda_1g'(x) + 2\lambda_2g''(x) + \lambda_3 = 0$


Shen-Castan : $c\exp^{-\alpha\|x\|}$	Deriche : $c(\alpha\|x\|+1)\exp^{-\alpha\|x\|}$

The Gaussian filter is a good approximation of such optimal filters

Contour Extraction§

Contour extraction requires gradient vector norm (or zero-crossing laplacian) thresholding

Naive approach: $(x,y)$ belongs to a contour iff

$\|\nabla f(x,y)\| > \sigma$
From hysteresis : $(x,y)$ belongs to a contour iff

$\|\nabla f(x,y)\| > \sigma_1$

$\|\nabla f(x,y)\| > \sigma_2$ and $\exists (x',y')$ neighbor of $\exists (x,y)$ such that $\|\nabla f(x',y')\| > \sigma_1$

Using gradient orientation (we follow the contour in a direction perpendicular to the gradient)

More Advanced Image Processing§

Bilateral Filter§

[Tomasi & Manduchi, 98]

Non-linear filter with a pair of spatial intensity kernels

$I^*(x) = \frac{1}{W} \sum_{y \in \Omega_x} I(y)f(|I(y)-I(x)|)g(|y-x|)$

$W= \sum_{y\in \Omega_x}{f(\|I(y)-I(x)\|)g(\|y-x\|)}$
$\Omega_x$ is a window around $x$
$f$ and $g$ are decreasing functions $[0,+\infty]\rightarrow\mathbb{R}^+$ (e.g. Gaussian kernels)

Bilateral Filter (bis)§

(From [Bilateral Filtering: Theory and Applications Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand Foundations and Trends in Computer Graphics and Vision, 2009])

Question: How to efficiently implement bilateral filtering ?

Variational Approaches: Anisotropic diffusion§

[Perona, Malik]

Images as functions $\Omega\rightarrow\mathbb{R}^2$
Diffusion process defined on time-varying images: $I(\cdot ,t):\Omega \rightarrow {\mathbb {R}}$
Diffusion operator defined as:

${\frac {\partial I}{\partial t}}={\mathrm {div}}\left(c(x,y,t)\nabla I\right)=\nabla c\cdot \nabla I+c(x,y,t)\Delta I$

with for example:

$c\left(\|\nabla I\|\right)=e^{{-\left(\|\nabla I\|/K\right)^{2}}}$

Anisotropic diffusion process that removes noise preserving edges

Rationale: for some $g\mathbb{R}\rightarrow \mathbb{R}$ we want to minimize

$E[I]={\frac {1}{2}}\int _{{\Omega }}g\left(\|\nabla I(x)\|^{2}\right)\,dx$

and thus

${\frac {\partial I}{\partial t}}=-\nabla E_{I}={\mathrm {div}}(g'\left(\|\nabla I(x)\|^{2}\right)\nabla I)$

Examples§

N.B Bilateral filtering is asymptotically close to Perona-Malik’s diffusion

Mumford-Shah functional§

Idea Variational formulation edge aware smoothing

$\mathcal{MS} (K,u)= \underbrace{ \alpha \int_{{M\backslash K}} | u - g |^2 dx}_{\text{attachment term}} + \underbrace{\int_{{M\backslash K}} | \nabla u |^2 dx}_{\text{smoothness term}} + {\underbrace{\lambda \mathcal{H}^{1}(K \cap M)}_{\text{discontinuities length}}}$

$\mathcal{M}$ is the function domain
${g}$ is the input raw image
${u}$ is the regularized image
${K}$ is the discontinuities support set
$\mathcal{H}^{1}$ is the Hausdorff measure

But:

Needs an explicit evaluation of $\mathcal{H}^{1}(K\cap M)$
We integrate over ${M\backslash K}$
$K$ has to be optimized

Ambrosio-Tortorelli functional§

$\Gamma-$ convergence mathemtatical framework

$\mathcal{AT}_\epsilon(u,v) = {\alpha} \underbrace{\int_{{M}} |u - g|^{{2}} dx}_{\text{attachment term}} + \underbrace{\int_{{M}} |{v} \nabla u|^{{2}} dx}_{\text{smoothness term}} + \underbrace{{\lambda} \int_{{M}} {\epsilon} |\nabla {v}|^{{2}} + \frac{1}{4 {\epsilon}} |1 - {v}|^{{2}} dx}_{\text{discontinuities length}}$

Two functionsTwo functions to optimize: u (regularized image) and v (feature scalar map to [0,1])
v≈1 on smooth parts, v≈0 near features
Integration domain does not change
No Hausdorff measure
Quadratic terms
the AT functional Γ−converges to Mumford-Shah’s functional as ϵ→0

:-)

Mathematical Morphology§

Principles§

[Matheron, Serra, …]

Idea

Objects defined as sets
Elementary operators based on boolean operations (union, difference)
Notion of structuring elements
Study of operator properies, e.g.:
- Idempotence $f\circ f=f$
- Non-linear
- Not reversible
Generalizations exist to gray-level images and to any ordered lattice

Operators§

Translation

$X_p = \{ x + p\,|\, x\in X\}$

Dilation by a structuring element B

$\delta_B(X) = X \oplus B = \bigcup_{x\in X} B_x = \bigcup_{b\in B} X_b$

Erosion by a structuring element B

$\epsilon_B(X) = X \ominus B = \bigcap_{b\in B} X_{-b}$

Properties

with $\check{B} = \{ -p \,|\, p \in B\}$ and $X^c = E \setminus X$

$(X \oplus B)^c = X^c \ominus \check{B}$

$(X \ominus B)^c = X^c \oplus \check{B}$

Illustrations§

Structuring element: Euclidean disc

(blue: set $X$ , gray circle: structuring element and cyan: result of the operators)

More operators§

Opening B

$A \circ B = (A \ominus B)\oplus B \quad (\gamma)$

Closing

$A \bullet B = (A \oplus B)\ominus B \quad (\phi)$

Examples§

3x3 structuring element

Properties§

Properties

Opening is anti-extensive: $A\circ B\subseteq A$
Closing is extensive: $A \subseteq A\bullet B$
By duality

$A \bullet B = (A^c \circ B^c)^c$

Useful tool for granulometric analysis

Sequence of increasing structuring elements $B_k= B\oplus\ldots\oplus B$ k times

$\gamma_k(X) = X \circ B_k$

$G_k = |\gamma_k(X)|$

$PS_k = G_k(X) - G_{k+1}(X)$

$G_k$ is called the granulometry function of $X$ and $PS_k$ the spectrum

Intuitive explanation

$X$ is defined as the union of grains and $G_k$ is the size of the set $\gamma_k(X)$ defined by grains larger than k

Generalizations§

Operators on gray-level images

$F,G: E\rightarrow E$

$(F\oplus G)(x) = \sup_{y\in E} \{ F(y) + G(x-y)\} (F\ominus G)(x) = \inf_{y\in E} \{ F(y) - G(x-y)\}$

Example

Let suppose a grayscale image and a constant structuring element (whose origin is it mid-point, so-called flat structuring element)

$B = \begin{bmatrix} 0 & 0^* & 0 \end{bmatrix}$
Then

$\begin{bmatrix}13 & 16 & 17\\ 15 & 10 & 13\\ 16 & 9 & 15\\ \end{bmatrix} \oplus B = ?$

Generalization (bis)§

$\begin{bmatrix}13 & 16 & 17\\ 15 & 10 & 13\\ 16 & 9 & 15\\ \end{bmatrix} \oplus B = \begin{bmatrix}16 & 17 & 17\\ 15 & 15 & 13\\ 16 & 16 & 15\\ \end{bmatrix}$

Morphogical Gradient/Laplacian§

Basic “à-la” finite difference definition

Gradient

$Grad_B(X) = X\oplus B - X\ominus B$
Laplacian

$Lap_B(X) = X\oplus B + X\ominus B - 2Id$

Mathemtatical model

Operators acting on complete lattices $(L, \leq)$

Good morpholigical filters§

Principle

Given an specific image

Select the best structuring element(s)
Specify the combination of fundamental operators (e.g. series of opening/closing)
$\psi$ is a filter iff it is increasing and idempotent

$f\leq g \Rightarrow \psi(f) \leq \psi(g)$
$\phi,\gamma,\delta,\epsilon$ are filters
$\phi\gamma, \gamma\phi, \phi\gamma\phi, \gamma\phi\gamma$ are filters and

$\gamma\leq \gamma\phi\gamma \leq \{ \gamma\phi, \phi\gamma \} \leq \phi\gamma\phi \leq \phi$

[Soille]

Introduction to Image Processing§

Value based Approaches§

Principles§

Histogram based image processing§

Histogram transformations§

Histogram Equalization and Histogram Specifications§

Equalization (bis)§

Histogram based Segmentation§

Color transfer using Optimal Transport§

Local Approaches§

Principles§

Fourier Space§

Few Fourier Space Properties§

Illustrations§

Illustrations§

Illustrations§

Elementary Image Filtering§

Mask Filtering Design§

Examples§

Noise Models§

High pass filter§

Non-linear filters§

Example§

Contour Detection and Differential Estimators§

Objectives§

Image Gradient§

Image Gradient (bis)§

Example§

Mask Design§

Separable Filters§

Theoretical Design of a Contour Detector§

Contour Extraction§

More Advanced Image Processing§

Bilateral Filter§

Bilateral Filter (bis)§

Variational Approaches: Anisotropic diffusion§

Examples§

Mumford-Shah functional§

Ambrosio-Tortorelli functional§

Examples§

Examples§

Examples§

3D Examples§

3D Examples§

3D Examples§

Mathematical Morphology§

Principles§

Operators§

Illustrations§

More operators§

Examples§

Properties§

Example§

Example§

Generalizations§

Generalization (bis)§

Example (input)§

Example (output)§

Morphogical Gradient/Laplacian§

Good morpholigical filters§

Example§