CUSTOM COLOR ENHANCEMENTS BY STATISTICAL LEARNING ...
CUSTOM COLOR ENHANCEMENTS BY STATISTICAL LEARNING Maya R. Gupta University of Washington Dept. of Electrical Engineering Seattle, WA 98195 ABSTRACT We consider the problem of automatically learning color enhancements from a small set of sample color pairs, and describing the enhancement by a three-dimensional look-uptable that can be stored and implemented as an ICC profile. We propose a new method for adapting the neighborhood size for local statistical learning methods such as local linear regression, and show that this leads to relatively accurate descriptions of the desired color transformation and results in images that appear smooth and have natural depth of detail. In a previous work we showed that learning arbitrary color enhancements can result in colored specular highlights, causing images to look unnatural. We show that this can be solved by adding a null sample that maps white to white. 1. INTRODUCTION Surveying professional graphic artists, we have found that they are interested in customizable tools that are easy to use for color image processing, and that they can re-use as personal tools. We proposed [1] implementing custom color enhancements as ICC profiles, where the ICC profile is based on sample color transformations provided by a designer. In [1] we presented the statistical learning issues that arise in estimating the three-dimensional look-uptable (LUT), that forms the core of an ICC profile, based on relatively few input-output color pairs. In this paper, we propose solutions to the issues raised in [1], and show how those solutions make it possible to automatically create ICC profiles based on a small set of example color transformations. One of our solutions is a way to adapt neighborhood size that may be useful for other statistical learning problems. ICC profiles [2] are the standard method for characterizing and correcting for the way devices, such as printers, transform colors. The core of an ICC profile is a threedimensional LUT that defines how input colors are transformed into output colors for a set number of grid points This work was supported in part by the National Science Foundation Grant SBE-0123552.
(e.g. 17 × 17 × 17). The transformation for colors not on the grid is interpolated from the gridpoint transformations. This is a very flexible architecture for defining any color transformation from any device-independent space or device-dependent space to another, and ICC profile usage is implemented in many software and hardware packages. Our goal is to automatically create an ICC profile that accurately captures a user’s custom color enhancement, where the user has specified that color enhancement with a (possibly small) set of sample input and outputs color pairs. Other researchers have considered the problem of “learning” transformations implied by the color palettes of an input and output image [3], or by moving the target image’s palette towards a reference image palette [4]. In a previous paper [1], we showed that statistical learning estimation methods such as linear interpolation with maximum entropy (LIME) and local linear regression could be used to effectively learn color transformations, where we used a 24 sample Gretag MacBeth chart under different illuminations to define the input and output samples. We compared how different learning algorithms estimated the gridpoints of a 25x25x25 LUT, and the resulting color transforms on images, neutrals, and the output gamut. The problems that arose were due to the small number of training samples, and included objectionable colored highlights, inaccurate capture of the desired color enhancement, clipped (flat) regions in enhanced images, and false contours. The key issue behind most of these problems is that the user’s small set of color samples may not cover the full colorspace, causing many gridpoints fell outside the convex hull spanned by the input color samples. In that case, the output colors estimated for those gridpoints have to be extrapolated. Interpolative methods like LIME clip colors outside the convex hull of the input color samples to the gamut defined by that convex hull. Extrapolative methods like local linear regression (which fits a hyperplane to a set of training samples local to each gridpoint), extrapolate in a way that can lose the sense of the desired color transformation, and the linear extrapolation can cause clipping at the boundaries of the display colorspace, creating flat regions
of saturated colors. The most noticeable example is when bright white is mapped to a non-bright-white color, causing object highlights to become colored. The eye is very sensitive to deviations from neutral and in an informal survey subjects rated images with colored highlights as non-natural without even realizing the only difference was the color of the highlights.
the smallest k such that the gridpoint X is within a just noticeable difference (JND) of the convex hull spanned by its local neighborhood, or if no such neighborhood exists, then k is set to some preset maximum neighborhood size. Then N = 1, . . . , k(X) where the training sample set is reindexed for every gridpoint X so that Xi is the ith nearestneighbor to X out of the set of training samples, and k(X) is defined,
2. SOLUTIONS We propose two solutions to the problems that arise when estimating the color transformation from a small sample set: adding null points and automatically adapting the learning neighborhood size. Null points are extra training samples added to the user’s set of training samples, but the null points map to themselves, for example [100 0 0] → [100 0 0] in CIELab coordinates ensures that very bright white colors will map to very bright white, solving the colored specular highlights problem. If null points are added at the corners of the colorspace, then the sample set spans the entire colorspace and interpolative methods will no longer clip. However, adding extra null points thwarts the color enhancement that the sample set is defining, and causes a loss of accuracy in describing the desired color enhancement. For the results in this paper, two null points were added to every sample set: [0 0 0] and [100 0 0] (CIELab). 3. ADAPTIVE NEIGHBORHOOD SIZE Given user-defined samples {Xi , Yi } for i = 1, . . . , n, where Xi and Yi are three-component CIELab values, the local linear regression output value Yˆ corresponding to a grid point X is Yˆ = aT X + b, where a and b are three-component vectors that minimize the least-squares error fit over a local neighborhood centered at gridpoint X: X arg min (kaT Xi + b − Yi k2 )2 , (1) i∈N
where the neighborhood of training samples N is commonly defined by the k nearest-neighbors to the gridpoint X being estimated, or by all sample points falling within a fixed volume centered at X [5]. The larger the neighborhood size, the less local the fit, and the fewer degrees of freedom are used to describe the color enhancement. Thus large neighborhoods will oversmooth the enhancement. Too small a neighborhood will lead to wild extrapolations and create noisy color transformations. For this application, there is no simple objective to optimize to determine the best-sized neighborhood. To satisfy the goal of a tool that “automatically learns” color enhancements defined by a arbitrary sized set of sample pairs, we propose adapting the neighborhood size k to be
k(X) = min (min argk¯ D (C(Xk¯ ), X) < 2.5, kmax ) where C(Xk¯ ) is the convex hull of {X1 , . . . , Xk¯ }, and D(r, s) is the minimum Euclidean distance between points in r and s, and 2.5 is used as an approximate JND in CIELab space. 4. RESULTS We show results for two color enhancements defined by two different sets of 26 sample pairs (24 user-defined and 2 null pairs), called cloudy-to-soft-white and daylight-tosunset. The sample color pairs were taken from photographs of the 24 color Gretag Macbeth chart under different lighting conditions. The results compare local linear regression estimation of the 3D LUT with different neighborhood sizes. The sample color pairs are transformed to CIELab space and a 25 × 25 × 25 3D LUT is estimated to transform CIELab input values to CIELab output values. The images shown here have been converted from CIELab values to sRGB for display, and were produced by interpolating the 3D LUT with trilinear interpolation (a standard fast choice for color management applications) [6]. All the adaptive neighborhood results are shown for kmax = 21. Neighborhood size k=4 k=21 adaptive k
L* mse
a* mse
b* mse
0.67 5.82 2.57
3.90 23.94 8.54
272.05 97.42 30.50
Table 1. For the cloudy-to-soft-light transform: mean squared-errors of the estimated 3D LUT applied to the original color sample pairs. Tables 1 and 2 compare the accuracy of the color enhancements, giving the mean-squared error of using the estimated 3D LUT to estimate the original sample color pairs. Accurately transforming the small set of original values does not necessarily mean the color transformation will be useful, also of concern is smoothness of the transform (judged by lack of color noise or false contours) and the clipping of extreme color points. Fig. 1 shows how the neutral axis is transformed by 3D LUT’s estimated with different neighborhood sizes; k =
Neighborhood size k=4 k=21 adaptive k
L* mse
a* mse
b* mse
7.83 8.30 4.83
206.44 28.88 15.24
176.38 94.99 46.95
D. H. Salesin, “Image analogies,” SIGGRAPH, pp. 327–340, 2001.
Table 2. For the daylight-to-sunset transform: mean squared-errors of the estimated 3D LUT applied to the original color sample pairs.
[4] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, “Color transfer between images,” IEEE Computer Graphics and Applications, vol. 21, pp. 34–41, September 2001. [5] Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, Springer-Verlag, New York, 2001.
100
[6] Henry Kang, Color Technology for Electronic Imaging Devices, SPIE Press, United States of America, 1997.
90 80 70 60 50 40 30 20 10 0 100 0 −100
−10
0
10
20
30
40
50
Fig. 1. Neutral axis transformed by the color enhancements estimated by local linear regression: blue line is k = 4, green line is k = 21, red line is adaptive k. 4 yields a sporadic neutrals transform, k = 21 is oversmoothed, while the adaptive neighborhood is a compromise. In Fig. 3 and Fig. 4, it is clearly seen that too small a neighborhood leads to color noise, and and the tables show that too large a neighborhood does not accurately capture the desired color enhancement. The adaptive neighborhood strategy tries to use a small neighborhood in order to accurately describe the color enhancement. However, the neighborhoods are chosen to be large enough so that the gridpoint X is bounded within a convex hull of training samples, resulting in stable and smooth estimations. In all images and sample pairs considered, the colored highlights issue appears to be solved by the inclusion of the extra null points. 5. REFERENCES [1] M. R. Gupta, S. Upton, and J. Bowen, “Simulating the effect of illumination using color transformation,” Proc. of the SPIE Conf. on Computational Imaging III, January 2005. [2] “ICC webpage,” 2004, www.color.org/profile.html. [3] A. Hertzmann, C. E. Jacobs, B. Curless N. Oliver, and
Fig. 2. Original images used (peppers from Matlab, building from Foveon image gallery).
Fig. 3. Cloudy-to-soft-light custom color enhancement. Top: k = 4 neighbors used to estimate 3D LUT. Middle: k = 21 neighbors used to estimate 3D LUT. Bottom: Adaptive k.
Fig. 4. Daylight-to-sunset custom color enhancement. Top: k = 4 neighbors used to estimate 3D LUT. Middle: k = 21 neighbors used to estimate 3D LUT. Bottom: Adaptive k.
(e.g. 17 × 17 × 17). The transformation for colors not on the grid is interpolated from the gridpoint transformations. This is a very flexible architecture for defining any color transformation from any device-independent space or device-dependent space to another, and ICC profile usage is implemented in many software and hardware packages. Our goal is to automatically create an ICC profile that accurately captures a user’s custom color enhancement, where the user has specified that color enhancement with a (possibly small) set of sample input and outputs color pairs. Other researchers have considered the problem of “learning” transformations implied by the color palettes of an input and output image [3], or by moving the target image’s palette towards a reference image palette [4]. In a previous paper [1], we showed that statistical learning estimation methods such as linear interpolation with maximum entropy (LIME) and local linear regression could be used to effectively learn color transformations, where we used a 24 sample Gretag MacBeth chart under different illuminations to define the input and output samples. We compared how different learning algorithms estimated the gridpoints of a 25x25x25 LUT, and the resulting color transforms on images, neutrals, and the output gamut. The problems that arose were due to the small number of training samples, and included objectionable colored highlights, inaccurate capture of the desired color enhancement, clipped (flat) regions in enhanced images, and false contours. The key issue behind most of these problems is that the user’s small set of color samples may not cover the full colorspace, causing many gridpoints fell outside the convex hull spanned by the input color samples. In that case, the output colors estimated for those gridpoints have to be extrapolated. Interpolative methods like LIME clip colors outside the convex hull of the input color samples to the gamut defined by that convex hull. Extrapolative methods like local linear regression (which fits a hyperplane to a set of training samples local to each gridpoint), extrapolate in a way that can lose the sense of the desired color transformation, and the linear extrapolation can cause clipping at the boundaries of the display colorspace, creating flat regions
of saturated colors. The most noticeable example is when bright white is mapped to a non-bright-white color, causing object highlights to become colored. The eye is very sensitive to deviations from neutral and in an informal survey subjects rated images with colored highlights as non-natural without even realizing the only difference was the color of the highlights.
the smallest k such that the gridpoint X is within a just noticeable difference (JND) of the convex hull spanned by its local neighborhood, or if no such neighborhood exists, then k is set to some preset maximum neighborhood size. Then N = 1, . . . , k(X) where the training sample set is reindexed for every gridpoint X so that Xi is the ith nearestneighbor to X out of the set of training samples, and k(X) is defined,
2. SOLUTIONS We propose two solutions to the problems that arise when estimating the color transformation from a small sample set: adding null points and automatically adapting the learning neighborhood size. Null points are extra training samples added to the user’s set of training samples, but the null points map to themselves, for example [100 0 0] → [100 0 0] in CIELab coordinates ensures that very bright white colors will map to very bright white, solving the colored specular highlights problem. If null points are added at the corners of the colorspace, then the sample set spans the entire colorspace and interpolative methods will no longer clip. However, adding extra null points thwarts the color enhancement that the sample set is defining, and causes a loss of accuracy in describing the desired color enhancement. For the results in this paper, two null points were added to every sample set: [0 0 0] and [100 0 0] (CIELab). 3. ADAPTIVE NEIGHBORHOOD SIZE Given user-defined samples {Xi , Yi } for i = 1, . . . , n, where Xi and Yi are three-component CIELab values, the local linear regression output value Yˆ corresponding to a grid point X is Yˆ = aT X + b, where a and b are three-component vectors that minimize the least-squares error fit over a local neighborhood centered at gridpoint X: X arg min (kaT Xi + b − Yi k2 )2 , (1) i∈N
where the neighborhood of training samples N is commonly defined by the k nearest-neighbors to the gridpoint X being estimated, or by all sample points falling within a fixed volume centered at X [5]. The larger the neighborhood size, the less local the fit, and the fewer degrees of freedom are used to describe the color enhancement. Thus large neighborhoods will oversmooth the enhancement. Too small a neighborhood will lead to wild extrapolations and create noisy color transformations. For this application, there is no simple objective to optimize to determine the best-sized neighborhood. To satisfy the goal of a tool that “automatically learns” color enhancements defined by a arbitrary sized set of sample pairs, we propose adapting the neighborhood size k to be
k(X) = min (min argk¯ D (C(Xk¯ ), X) < 2.5, kmax ) where C(Xk¯ ) is the convex hull of {X1 , . . . , Xk¯ }, and D(r, s) is the minimum Euclidean distance between points in r and s, and 2.5 is used as an approximate JND in CIELab space. 4. RESULTS We show results for two color enhancements defined by two different sets of 26 sample pairs (24 user-defined and 2 null pairs), called cloudy-to-soft-white and daylight-tosunset. The sample color pairs were taken from photographs of the 24 color Gretag Macbeth chart under different lighting conditions. The results compare local linear regression estimation of the 3D LUT with different neighborhood sizes. The sample color pairs are transformed to CIELab space and a 25 × 25 × 25 3D LUT is estimated to transform CIELab input values to CIELab output values. The images shown here have been converted from CIELab values to sRGB for display, and were produced by interpolating the 3D LUT with trilinear interpolation (a standard fast choice for color management applications) [6]. All the adaptive neighborhood results are shown for kmax = 21. Neighborhood size k=4 k=21 adaptive k
L* mse
a* mse
b* mse
0.67 5.82 2.57
3.90 23.94 8.54
272.05 97.42 30.50
Table 1. For the cloudy-to-soft-light transform: mean squared-errors of the estimated 3D LUT applied to the original color sample pairs. Tables 1 and 2 compare the accuracy of the color enhancements, giving the mean-squared error of using the estimated 3D LUT to estimate the original sample color pairs. Accurately transforming the small set of original values does not necessarily mean the color transformation will be useful, also of concern is smoothness of the transform (judged by lack of color noise or false contours) and the clipping of extreme color points. Fig. 1 shows how the neutral axis is transformed by 3D LUT’s estimated with different neighborhood sizes; k =
Neighborhood size k=4 k=21 adaptive k
L* mse
a* mse
b* mse
7.83 8.30 4.83
206.44 28.88 15.24
176.38 94.99 46.95
D. H. Salesin, “Image analogies,” SIGGRAPH, pp. 327–340, 2001.
Table 2. For the daylight-to-sunset transform: mean squared-errors of the estimated 3D LUT applied to the original color sample pairs.
[4] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, “Color transfer between images,” IEEE Computer Graphics and Applications, vol. 21, pp. 34–41, September 2001. [5] Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, Springer-Verlag, New York, 2001.
100
[6] Henry Kang, Color Technology for Electronic Imaging Devices, SPIE Press, United States of America, 1997.
90 80 70 60 50 40 30 20 10 0 100 0 −100
−10
0
10
20
30
40
50
Fig. 1. Neutral axis transformed by the color enhancements estimated by local linear regression: blue line is k = 4, green line is k = 21, red line is adaptive k. 4 yields a sporadic neutrals transform, k = 21 is oversmoothed, while the adaptive neighborhood is a compromise. In Fig. 3 and Fig. 4, it is clearly seen that too small a neighborhood leads to color noise, and and the tables show that too large a neighborhood does not accurately capture the desired color enhancement. The adaptive neighborhood strategy tries to use a small neighborhood in order to accurately describe the color enhancement. However, the neighborhoods are chosen to be large enough so that the gridpoint X is bounded within a convex hull of training samples, resulting in stable and smooth estimations. In all images and sample pairs considered, the colored highlights issue appears to be solved by the inclusion of the extra null points. 5. REFERENCES [1] M. R. Gupta, S. Upton, and J. Bowen, “Simulating the effect of illumination using color transformation,” Proc. of the SPIE Conf. on Computational Imaging III, January 2005. [2] “ICC webpage,” 2004, www.color.org/profile.html. [3] A. Hertzmann, C. E. Jacobs, B. Curless N. Oliver, and
Fig. 2. Original images used (peppers from Matlab, building from Foveon image gallery).
Fig. 3. Cloudy-to-soft-light custom color enhancement. Top: k = 4 neighbors used to estimate 3D LUT. Middle: k = 21 neighbors used to estimate 3D LUT. Bottom: Adaptive k.
Fig. 4. Daylight-to-sunset custom color enhancement. Top: k = 4 neighbors used to estimate 3D LUT. Middle: k = 21 neighbors used to estimate 3D LUT. Bottom: Adaptive k.
