Binocular tone mapping

Binocular tone mapping ACM Transactions on Graphics vol. 31, No. 4, 2012 Xuan Yang, Linling Zhang, Tien-Tsin Wong, Pheng-Ann Heng Presented by Ran Shu

School of Electrical Engineering and Computer Science Kyungpook National Univ.

Abstract  Binocular single vision phenomenon – Fusing two displaced images • With difference in detail and contrast and luminance

– Proposed binocular tone mapping framework • Generating binocular LDR image pair − Preserving more human-perceivable visual content − Left for tone-mapped LDR image » Without loss of generality

− Right for proposed optimally synthesize image

• LDR image pair presenting more human-perceivable visual richness − Without triggering visual discomfort » Using novel binocular viewing comfort predictor

» Based on finding in vision science 2/35

– Binocularly tone-mapped image pair

Fig. 1. LDR image pairs. Left shows global contrast, right shows more local details

3/35

Introduction  Two binocular vision phenomenon – Stereopsis for existing binocular display • 3D movies

– Binocular single vision • Fusing images from two eyes as single percept − Even two images with difference » Complex non-linear neurophysiological process

4/35

 Proposed method – Utilizing existing LDR binocular display • Simultaneously presenting contrast and details in HDR images − By proposed binocular tone mapping framework

– Given an HDR image and tone-mapped LDR image • Generated by existing tone mapping techniques • Optimally synthesizing counterpart LDR image − Using proposed framework BVCP − Presenting more human-perceivable visual content

5/35

 Proposed binocular viewing comfort predictor – Guiding binocular tone mapping • Based on findings in vision science and experimental results − Avoiding visual discomfort − Making more difference in two views

6/35

Related work  Binocular single vision – Combing two different images from two eyes into single vision • Occurring only in small volume of retinal area − Around fixating area

− Non-linear combination of luminance and contrast and color » Regarded as combination of binocular fusion and suppression

– Binocular fusion • Process of superimposing

• Combining similar content from two views into one percept − Under two similar or same image

– Binocular suppression • Under submissive view

– Binocular rivalry 7/35

– Binocular fusion and suppression and rivalry

Fig. 2. Fusion, suppression and rivalry.

8/35

– Assessment for binocular viewing comfort • Existing metrics considering visible difference − Between two images looking with both eyes

» Without considering binocular vision

• Designing brand new metric − Binocular viewing comfort predictor » Based on theories and experimental results

9/35

 Tone mapping – Using previous tone mapping method • Global operators − Adaptive logarithmic mapping

• Local operators − Bilateral filtering approach − Gradient domain HDR compression

− Perceptual based contrast processing

10/35

Overview  Overview of binocular tone mapping – Input HDR image and left tone-mapping LDR image – Generating optimal right LDR image • Using same tone-mapping operator as left • Objective function composed of two metrics − Visible difference predictor − Binocular viewing comfort predictor

Fig. 3. System overview.

11/35

BVCP  Fusional area – Based on visual agreement of neighborhoods • Panum’s fusional area • Occupying constant solid angle subtended at eye • Considering whole area for fusion stability − In terms of contour and contrast and luminance and color difference

Fig. 4. Fusional area.

12/35

– Radius of fusional area rf   tan   d

(1)

where  is the maximal retinal disparity, which is around 60 to 70 arcmin

– Approximating fusional area by rectangle • Go over all pixel pairs from LDR image pair − Considering corresponding fusion areas » For measuring fusion stability

13/35

 Contour fusion – Contour difference more important than contrast or color difference – Corresponding contour no need to exactly same • Motor fusion − Superimposing corresponding points or similar contour » By movement of our eyes

• Sensory fusion − Combing two views into one unified percept

Fig. 5. Contour dominance.

Fig. 6. Motor fusion and sensory fusion. 14/35

– Contour has different definitions in different domains • Applying scale representation to fusional area − Constructing pyramid

» Fourier transform fusional area to frequency domain » Applying pyramid of LPF in frequency domain » Mesa filter as LPF » Inversely Fourier transforming each low-passed frequency images » Obtaining pyramid of low-passed fusional areas

Fig. 7. Mesa pyramid.

15/35

– Measuring color difference between adjacent pixels • Defined as 2-norm distance of colors in LAB color space Lk  p1 , p2   EC  Lk  p1  , Lk  p2  

(2)

k where L is color difference of fusional area, EC is color difference of two pixel.

• Color difference between a pair of pixels

Fig. 9. Contour matching.

16/35

• Looking up decision table − With JND and obvious color difference − Considering count of fusible pixels in fusional area 0,   k Bcf  1,  1,

 else if   if

p1 , p2 

  S k  p1 , p2   0

p1 , p2 

  S k  p1 , p2   0

(3)

otherwise

− Contour fusion state at higher levels override lower Bcfk 1  Bcfk

if Bcfk  0

(4)

− Final fusion state of two fusional areas Bcf  BcfS

(5)

Table 1. Decision table for contour fusion where J=JND, O=OCD

17/35

 Contour and regional contrasts – Two types of contrast influencing binocular single vision • Contour contrast − Coexisting with contour if detected by human eye − Matched contour pair generally helping fusion » Except obviously inversed contrast S k  p1 , p2   1

if C  Lk  p1  , Lk  p2  

OCD and C  R k  p1  , R k  p2  

and C  Lk  p1  , Lk  p2    C  R k  p1  , R k  p2  

OCD

(6)

0

where C  c1 , c2  computes the lightness difference between the pixel pair c1 , c2

• Regional contrast − Contrast between two regions by adopting restrictive constraint 1  1, if  p E  L  p  , R  p    4r 2 Brc   j 1, otherwise 





DCD

where L  p  , R  p  are two corresponding pixels located at position p in L and R, respectively.

(7)

18/35

 Failure of rivalry – Contrast below certain threshold • Stable single percept forming regardless of contour fusion − Happen in low-contrast regions

Fig. 8. Failure of Rivalry. 19/35

– Measuring contrast in log space • As log percentage contrast between pixel pair P  c1 , c2   log10

 Y1    Y2   Y1    Y2 

(8)

where Y1 , Y2 are the normalized luma of c1 , c2 ,  Y  maps the normalized luma Y in [0,1] to the physical measurement in the unit of cd / m 2

• Correlation between luminance and frequency and contrast

(a)

(b)

Fig. 10. Luminance and freq. vs. contrast.

20/35

• Obtaining two continuous curves 2 T  l   min  2,1.999  0.362 log10  l   0.026 log10 l 



T  f   min 2, log10  3.557  1.334 f  1.881 f 2  0.108 f 3 

(9)



(10)

where l is average luminance in cd / m2, f is spatial frequency measured in cycles per degree, T represents log percentage contrast threshold.

• Function of both luminance and spatial frequency



2 T  l , f   min 2, log10  3.557  1.334 f  1.881 f 2  0.108 f 3   0.514  0.362 log10  l   0.026 log10 l   



(11)

where is user parameter and is set to a value in [-0.15,0.15]

Fig. 11. The constructed log percentage contrast threshold.

21/35

– Contrast threshold for left fusional area • Obtained by feeding parameters TLk  T  lL , f k 

(12)

• Take second revision to fusion state variable S k  p1 , p2   0

if S k  p1 , p2 

0 and P  Lk  p1  , Lk  p2    T k and P  R k  p1  , R k  p2    T k

(13)

22/35

 The overall fusion predictor – Predict ultimate viewing comfort • For arbitrary pair of fusional areas 1, if Brc  1 B  Bcf , otherwise

(14)

Fig. 12. Visualization of BVCP assessment. 23/35

Optimization  Adopt visible difference predictor – Detecting visible difference E



1  H V  i, j      i, j



(15)

where  is a user-defined probability threshold and generally set as 75%, H is the Heaviside step function,  is the total number of pixels in the images.

• With VDP and proposed BVCP − Maximizing E without Violating BVCP

24/35

– Visualization of binocular visual of four tone mapping operators

(a) Drago

(b) Mantiuk

(c) Durand

(d) Fattal

Fig. 13. Visualization of binocular visual differences of four tone mapping operators for the HDR example in Fig. 20. 25/35

Results and Discussions  Framework adopted tone mapping operators – Supporting four tone mapping operators • Experimented with four operators

(a) Drago

(b) Mantiuk

(c) Durand

(d) Fattal

Fig. 14. Optimal LDR image pairs generated by our framework using four tone mapping operators.

26/35

 User study – Visual richness evaluating effectiveness of framework • Comparing bioptic image pairs to dichoptic image pairs − Dichoptic image pairs showing more preference

Table 2. User study of visual richness.

27/35

– Binocular symmetry • Binocular tone mapping symmetric to left and right eyes − Comparing statistics from two sets of evaluations

Table 3. User study of binocular symmetry. L/R means the left/right eye sees the images generated by our framework.

28/35

– Predictability of BVCP • Discomfort limit of binocular vision − Based on classical psychophysical methodology

» Method of adjustment

Fig. 15. One test sequence for evaluating BVCP predictability. 29/35

• Statistics of 20 test sequences

(a) Drago

(b) Mantiuk

(c) Durand

(d) Fattal

Fig. 16. Statistics of the predictability of BVCP. 30/35

 Incorporating stereopsis – Feasible to extend framework to incorporate stereopsis

Fig. 17. A stereo LDR image pair with left and right images tone-mapped differently. 31/35

 Limitation – Human-tolerable image pairs may be rejected – Human vision not equally sensitive to all pixel • Due to visual attention − Introducing importance map based on image content

– Contrast threshold function • In modeling failure of rivalry

Fig. 18. A case with small improvement of overall visual richness.

32/35

– Other resulting images

Fig. 19. LDR pair of “BrightonPierOld” using Drago’s operator.

Fig. 20. LDR pair of “GGmusicians” using Durand’s operator.

33/35

– Other resulting images

Fig. 21. LDR pair of “WreathBuilding” using Fattal’s operator.

Fig. 22. LDR pair of “Alhambra” using Mantiuk’s operator.

34/35

Conclusion  Binocular tone mapping framework – Generating binocular LDR image pair • Presenting more visual richness

– Developing novel BVCP metric • Predicting discomfort threshold

35/35