Image Sampling and Reconstruction - Cs.princeton.edu
Image Sampling and Reconstruction Thomas Funkhouser Princeton University C0S 426, Fall 2000
Image Sampling • An image is a 2D rectilinear array of samples
Quantization due to limited intensity resolution Sampling due to limited spatial and temporal resolution
Pixels are infinitely small point samples
1
Image Reconstruction • Re-create continuous image from samples
Example: cathode ray tube
Image is reconstructed by displaying pixels with finite area (Gaussian)
Sampling and Reconstruction
Figure 19.9 FvDFH
2
Sampling and Reconstruction
Sampling
Reconstruction
Sources of Error • Intensity quantization Not enough intensity resolution
• Spatial aliasing Not enough spatial resolution
• Temporal aliasing
Not enough temporal resolution
E2 =
∑ (I ( x, y) − P( x, y ))
2
( x, y )
3
Aliasing (in general) • In general: Artifacts due to under-sampling or poor reconstruction
• Specifically, in graphics:
Spatial aliasing Temporal aliasing
Under-sampling
Figure 14.17 FvDFH
Spatial Aliasing • Artifacts due to limited spatial resolution
4
Spatial Aliasing • Artifacts due to limited spatial resolution
“Jaggies”
Temporal Aliasing • Artifacts due to limited temporal resolution
Strobing Flickering
5
Temporal Aliasing • Artifacts due to limited temporal resolution Strobing Flickering
Temporal Aliasing • Artifacts due to limited temporal resolution
Strobing Flickering
6
Temporal Aliasing • Artifacts due to limited temporal resolution Strobing Flickering
Antialiasing • Sample at higher rate Not always possible Doesn’t always solve problem
• Pre-filter to form bandlimited signal
Form bandlimited function (low-pass filter) Trades aliasing for blurring
Must consider sampling theory!
7
Sampling Theory • How many samples are required to represent a given signal without loss of information? • What signals can be reconstructed without loss for a given sampling rate?
Spectral Analysis • Spatial domain:
Function: f(x) Filtering: convolution
• Frequency domain:
Function: F(u) Filtering: multiplication
Any signal can be written as a sum of periodic functions.
8
Fourier Transform
Figure 2.6 Wolberg
Fourier Transform
Figure 2.5 Wolberg
9
Fourier Transform • Fourier transform:
F (u ) =
∞
∫
f ( x)e −i 2πx dx
−∞
• Inverse Fourier transform:
f ( x) =
∞
+ i 2πu F ( u e du ) ∫
−∞
Sampling Theorem • A signal can be reconstructed from its samples, if the original signal has no frequencies above 1/2 the sampling frequency - Shannon • The minimum sampling rate for bandlimited function is called “Nyquist rate”
A signal is bandlimited if its highest frequency is bounded. The frequency is called the bandwidth.
10
Convolution • Convolution of two functions (= filtering):
g ( x ) = f ( x ) ⊗ h( x ) =
∞
∫ f ( λ ) h ( x − λ ) dλ
−∞
• Convolution theorem Convolution in frequency domain is same as multiplication in spatial domain, and vice-versa
Image Processing • Quantization Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither
• Filtering Blur Detect edges "
#
• Warping Scale Rotate Warps $
• Pixel operations
!
Add random noise Add luminance Add contrast Add saturation
%
&
• Combining (
'
Morphs Composite
11
Image Processing • Consider reducing the image resolution
Original image
1/4 resolution
Image Processing • Image processing is a resampling problem
Resampling
Thou shalt avoid aliasing!
12
Antialiasing in Image Processing • General Strategy Pre-filter transformed image via convolution with low-pass filter to form bandlimited signal )
• Rationale *
Prefer blurring over aliasing
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
13
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Continuous Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Discrete Samples
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
14
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Reconstructed Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Transformed Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
15
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Bandlimited Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Discrete samples
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
16
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Display
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Ideal Low-Pass Filter • Frequency domain
• Spatial domain Sinc ( x) =
sin πx πx
Figure 4.5 Wolberg
17
Practical Image Processing • Finite low-pass filters +
,
-
Point sampling (bad) Triangle filter Gaussian filter
Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Triangle Filter • Convolution with triangle filter
Input
Output
Figure 2.4 Wolberg
18
Gaussian Filter • Convolution with Gaussian filter
Input
Output
Figure 2.4 Wolberg
Image Processing • Quantization Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither .
/
0
1
• Filtering Blur Detect edges 6
7
• Warping Scale Rotate Warps 8
• Pixel operations
5
4
3
2
Add random noise Add luminance Add contrast Add saturation
9
:
• Combining
?
@
Blur Detect edges E
F
• Warping Scale Rotate Warps G
• Pixel operations A
B
C
D
Add random noise Add luminance Add contrast Add saturation
H
I
• Combining J
K
Morphs Composite
Adjusting Brightness • Simply scale pixel components L
Must clamp to range (e.g., 0 to 255)
Original
Brighter
20
Adjusting Contrast • Compute mean luminance
for all pixels
luminance = 0.30*r + 0.59*g + 0.11*b M
• Scale deviation from
for each pixel component
Must clamp to range (e.g., 0 to 255) N
Original
More Contrast
Image Processing • Quantization Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither O
P
Q
R
• Filtering Blur Detect edges W
X
• Warping Scale Rotate Warps Y
• Pixel operations
V
U
T
S
Add random noise Add luminance Add contrast Add saturation
Z
[
• Combining ]
\
Morphs Composite
21
Adjust Blurriness • Convolve with a filter whose entries sum to one ^
Each pixel becomes a weighted average of its neighbors
Original
Blur 2 1 1 16 16 16 Filter = 216 416 216 1 2 1 16 16 16
Edge Detection • Convolve with a filter that finds differences between neighbor pixels
Original
Detect edges − 1 − 1 − 1
Filter = − 1 + 8 − 1 − 1 − 1 − 1
22
Image Processing • Quantization
• Filtering
Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither _
`
a
b
Blur Detect edges g
h
• Warping Scale Rotate Warps i
• Pixel operations c
d
e
f
Add random noise Add luminance Add contrast Add saturation
j
k
• Combining m
l
Morphs Composite
Scaling • Resample with triangle or Gaussian filter More on this next lecture!
Original
1/4X resolution
4X resolution
23
Summary • Image processing is a resampling problem o
n
Avoid aliasing Use filtering
24
Image Sampling • An image is a 2D rectilinear array of samples
Quantization due to limited intensity resolution Sampling due to limited spatial and temporal resolution
Pixels are infinitely small point samples
1
Image Reconstruction • Re-create continuous image from samples
Example: cathode ray tube
Image is reconstructed by displaying pixels with finite area (Gaussian)
Sampling and Reconstruction
Figure 19.9 FvDFH
2
Sampling and Reconstruction
Sampling
Reconstruction
Sources of Error • Intensity quantization Not enough intensity resolution
• Spatial aliasing Not enough spatial resolution
• Temporal aliasing
Not enough temporal resolution
E2 =
∑ (I ( x, y) − P( x, y ))
2
( x, y )
3
Aliasing (in general) • In general: Artifacts due to under-sampling or poor reconstruction
• Specifically, in graphics:
Spatial aliasing Temporal aliasing
Under-sampling
Figure 14.17 FvDFH
Spatial Aliasing • Artifacts due to limited spatial resolution
4
Spatial Aliasing • Artifacts due to limited spatial resolution
“Jaggies”
Temporal Aliasing • Artifacts due to limited temporal resolution
Strobing Flickering
5
Temporal Aliasing • Artifacts due to limited temporal resolution Strobing Flickering
Temporal Aliasing • Artifacts due to limited temporal resolution
Strobing Flickering
6
Temporal Aliasing • Artifacts due to limited temporal resolution Strobing Flickering
Antialiasing • Sample at higher rate Not always possible Doesn’t always solve problem
• Pre-filter to form bandlimited signal
Form bandlimited function (low-pass filter) Trades aliasing for blurring
Must consider sampling theory!
7
Sampling Theory • How many samples are required to represent a given signal without loss of information? • What signals can be reconstructed without loss for a given sampling rate?
Spectral Analysis • Spatial domain:
Function: f(x) Filtering: convolution
• Frequency domain:
Function: F(u) Filtering: multiplication
Any signal can be written as a sum of periodic functions.
8
Fourier Transform
Figure 2.6 Wolberg
Fourier Transform
Figure 2.5 Wolberg
9
Fourier Transform • Fourier transform:
F (u ) =
∞
∫
f ( x)e −i 2πx dx
−∞
• Inverse Fourier transform:
f ( x) =
∞
+ i 2πu F ( u e du ) ∫
−∞
Sampling Theorem • A signal can be reconstructed from its samples, if the original signal has no frequencies above 1/2 the sampling frequency - Shannon • The minimum sampling rate for bandlimited function is called “Nyquist rate”
A signal is bandlimited if its highest frequency is bounded. The frequency is called the bandwidth.
10
Convolution • Convolution of two functions (= filtering):
g ( x ) = f ( x ) ⊗ h( x ) =
∞
∫ f ( λ ) h ( x − λ ) dλ
−∞
• Convolution theorem Convolution in frequency domain is same as multiplication in spatial domain, and vice-versa
Image Processing • Quantization Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither
• Filtering Blur Detect edges "
#
• Warping Scale Rotate Warps $
• Pixel operations
!
Add random noise Add luminance Add contrast Add saturation
%
&
• Combining (
'
Morphs Composite
11
Image Processing • Consider reducing the image resolution
Original image
1/4 resolution
Image Processing • Image processing is a resampling problem
Resampling
Thou shalt avoid aliasing!
12
Antialiasing in Image Processing • General Strategy Pre-filter transformed image via convolution with low-pass filter to form bandlimited signal )
• Rationale *
Prefer blurring over aliasing
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
13
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Continuous Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Discrete Samples
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
14
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Reconstructed Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Transformed Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
15
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Bandlimited Function
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Discrete samples
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
16
Image Processing Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function
Display
Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Ideal Low-Pass Filter • Frequency domain
• Spatial domain Sinc ( x) =
sin πx πx
Figure 4.5 Wolberg
17
Practical Image Processing • Finite low-pass filters +
,
-
Point sampling (bad) Triangle filter Gaussian filter
Real world Sample
Discrete samples (pixels) Reconstruct
Reconstructed function Transform
Transformed function Filter
Bandlimited function Sample
Discrete samples (pixels) Reconstruct
Display
Triangle Filter • Convolution with triangle filter
Input
Output
Figure 2.4 Wolberg
18
Gaussian Filter • Convolution with Gaussian filter
Input
Output
Figure 2.4 Wolberg
Image Processing • Quantization Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither .
/
0
1
• Filtering Blur Detect edges 6
7
• Warping Scale Rotate Warps 8
• Pixel operations
5
4
3
2
Add random noise Add luminance Add contrast Add saturation
9
:
• Combining
?
@
Blur Detect edges E
F
• Warping Scale Rotate Warps G
• Pixel operations A
B
C
D
Add random noise Add luminance Add contrast Add saturation
H
I
• Combining J
K
Morphs Composite
Adjusting Brightness • Simply scale pixel components L
Must clamp to range (e.g., 0 to 255)
Original
Brighter
20
Adjusting Contrast • Compute mean luminance
for all pixels
luminance = 0.30*r + 0.59*g + 0.11*b M
• Scale deviation from
for each pixel component
Must clamp to range (e.g., 0 to 255) N
Original
More Contrast
Image Processing • Quantization Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither O
P
Q
R
• Filtering Blur Detect edges W
X
• Warping Scale Rotate Warps Y
• Pixel operations
V
U
T
S
Add random noise Add luminance Add contrast Add saturation
Z
[
• Combining ]
\
Morphs Composite
21
Adjust Blurriness • Convolve with a filter whose entries sum to one ^
Each pixel becomes a weighted average of its neighbors
Original
Blur 2 1 1 16 16 16 Filter = 216 416 216 1 2 1 16 16 16
Edge Detection • Convolve with a filter that finds differences between neighbor pixels
Original
Detect edges − 1 − 1 − 1
Filter = − 1 + 8 − 1 − 1 − 1 − 1
22
Image Processing • Quantization
• Filtering
Uniform Quantization Random dither Ordered dither Floyd-Steinberg dither _
`
a
b
Blur Detect edges g
h
• Warping Scale Rotate Warps i
• Pixel operations c
d
e
f
Add random noise Add luminance Add contrast Add saturation
j
k
• Combining m
l
Morphs Composite
Scaling • Resample with triangle or Gaussian filter More on this next lecture!
Original
1/4X resolution
4X resolution
23
Summary • Image processing is a resampling problem o
n
Avoid aliasing Use filtering
24
