Optimized flash light-emitting diode spectra for mobile phone cameras Felix Kimme,1,* Peter Brick,1 Sangam Chatterjee,2 and Tran Quoc Khanh3 1
OSRAM Opto Semiconductors GmbH, 93055 Regensburg, Germany
2
Department of Physics, Philipps-University Marburg, 35032 Marburg, Germany
3
Institute of Electromechanical Design, Technical University of Darmstadt, 64289 Darmstadt, Germany *Corresponding author: felix.kimme@osram‐os.com Received 20 September 2013; revised 13 November 2013; accepted 20 November 2013; posted 20 November 2013 (Doc. ID 196963); published 16 December 2013
Flash light-emitting diodes (LEDs) of modern mobile phones lack red and cyan spectral parts, however, the color gamut of their respective displays has increased in recent years. The influence of this discrepancy on the color reproduction of smart phones is investigated in this paper. Based on the CIECAM02 color appearance model, a metric is introduced to judge color reproduction of mobile phones under flash LED illumination. An evaluation method is established to compare the visual appearance of a scene under various surrounding illuminations with the reproduction of that scene. To facilitate a comparison with measurements, the evaluation method is based on the raw data of two test cameras and a Digital ColorChecker SG. To reduce the color shift between perception and reproduction, optimized flash LED spectra are presented. A single-LED and a double-LED concept with adjustable color temperature are derived from these results. Additionally, the common characteristics of flash LED spectra giving good results is investigated, identifying the spectral parts with the most influence on camera color reproduction and showing the spectral parts not contributing or even resulting in poor color reproduction. Finally the efficiency of optimized flash LED spectra is compared to standard flash LEDs. © 2013 Optical Society of America OCIS codes: (100.2000) Digital image processing; (120.5240) Photometry; (150.2950) Illumination; (330.1710) Color, measurement; (330.6180) Spectral discrimination. http://dx.doi.org/10.1364/AO.52.008779
1. Introduction
Most modern mobile phones include an integrated camera. To give the phone a slim shape, small cameras with small optics are used. As the resolution of the camera sensors is increasing, the size of each pixel is decreasing and the light sensitivity is dropping. As a result, mobile phones rely on their flash light-emitting diode (LED) to increase illuminance levels, whereas normal compact cameras can take pictures without flash illumination in many cases. The typical mobile phone flash is realized with a white LED containing a yellow phosphor. Compared 1559-128X/13/368779-10$15.00/0 © 2013 Optical Society of America
to xenon bulbs used in compact cameras, the spectrum of those LEDs lacks red (above 640 nm) and cyan (around 490 nm) spectral parts, see Fig. 1. Additionally, mobile phone cameras differ in their spectral measurement compared to the human perception (Fig. 2). In the context of missing spectral parts while taking the picture, the question arises as to which spectral parts have an influence on good color reproduction. The color reproduction of an acquired picture is influenced by the ambient and flash light, the camera color filters, and the white balance algorithms. Ideally, two distinguishable colors when directly perceived should be equally distinguishable in the picture taken. To determine if the color reproduction is affected by those spectral gaps, an appropriate evaluation 20 December 2013 / Vol. 52, No. 36 / APPLIED OPTICS
8779
method is generated. It compares the reproduced image depending on the spectrum of the flash LED to the direct perception of the photographed scene. The goal is to find the optimal flash LED spectrum. The following sections explain this method, its influences, the selection of parameters, the resulting spectra with good color reproduction, and their common characteristics. 2. Method to Evaluate Optimal Flash Spectra
Fig. 1. Illumination used in mobile phone flash photography. Typical LED flash and xenon flash spectrum [1].
The method used to evaluate the color reproduction of flash LEDs is separated into three parts: the direct perception of a scene, the camera color measurement, and the reproduction of it on a display. The ambient illumination creates directly perceived color stimuli for each test color of the scene, which are transformed into a uniform color space. The same test colors under the flash LED illumination are measured by the camera. The radiation of the test colors is filtered by the camera color filters and creates a picture in raw data. This picture is processed by the white balance algorithm and displayed on an ideal monitor. The monitor is modeled with an unlimited color resolution and color space and thus does not restrict the color reproduction. Taking the ambient illumination while viewing the picture into account, the color difference between the direct perception of the scene and the perception of the reproduction in a uniform color space (UCS) can be determined for all test colors. A schematic of the method used to evaluate the color reproduction of flash LEDs is shown in Fig. 3. In the following, the parameters of this method will be described in detail: the test colors used, the uniform color space, the camera, and the assumed ambient illumination. A. Gamut of Test Colors
Fig. 2. Measured camera color filters (RGB) compared to the color matching functions (xyz) [2].
The surface color gamut represents most of the colors of a scene and is a subset of the visible colors. Pointer
Fig. 3. Method to evaluate optimal flash spectra. The direct perception of a scene (top left frame) under ambient light, is compared to the reproduction on an ideal monitor (right frame), created by a camera measurement (bottom left frame) under flash illumination of that scene. The CIE color appearance model (CIECAM02) is used to take account of the different chromatic adaptation. 8780
APPLIED OPTICS / Vol. 52, No. 36 / 20 December 2013
described by Luo et al. [8]. Δa0ucs and Δb0ucs describe the difference in the red/green and blue/yellow plane. ΔEucs is the Euclidean distance of the two stimuli. C.
Surrounding Illuminants
Typically, the illumination is a mixture of the flash LED and an ambient illuminant, such as the sun, halogen, discharge, or LED lamps. A camera can only guess the color temperature of the surrounding illuminant by the ratios of the red and blue color channel. Depending on those values, predefined white balance parameters are then chosen. Furthermore, the in-picture ratio of flash to ambient illuminant is not constant as the light sources have different directional characteristics and origins. To cover all these possibilities, the evaluation of the flash spectra is done for different scenarios. A scenario is characterized by the type of the ambient illuminant, its color temperature T amb , and the ratio ramb of radiometric power of the flash LED and the ambient light (ramb Φe;flash ∕Φe;amb ). Fig. 4. Surface colors measured by Pointer [3] under various illuminations and the color coordinates of the Digital ColorChecker SG und D65, shown in the CIE Diagram from 1931.
[3] created a dataset of 4000 measured stimuli under various illuminations, shown in Fig. 4. For practical purposes, the Digital ColorChecker SG [4] is used, which contains 56 black, white, and gray colors, and 84 test colors, that are well distributed over the surface color space. The spectral reflectivities of this color checker card were measured and included into the evaluation method, so that it can be compared to measurement results. The color coordinates of the color checker under D65 light are shown in Fig. 4. B.
Color Space
To quantify the perceived color difference between a directly viewed and a reproduced color, regardless of the color, a UCS is used. Color spaces that include models for chromatic adaptation and additional characteristics of human vision are called color appearance models (CAMs). There are several color spaces that consider chromatic adoption and are uniform to human perception. The most recent standard from the CIE is called CIECAM02 [5], that stands for CIE CAM from 2002. Xue [6] showed that CIECAM02 performs as well as CIELAB, for some cases it is more uniform than CIELAB. In addition, Moroney and Zeng [7] demonstrated that CIECAM02 has a better hue constancy than CIELAB. The standard CIECAM02 CAM with an extension for color uniformity called CAM02-UCS [8] is used here, including the color difference formula q 02 02 (1) ΔEucs ΔJ 02 ucs Δaucs Δbucs : ΔJ 0ucs is the lightness difference between reproduced and directly perceived stimulus for a test color, as
D. Camera Color Measurement and Artificial White Balance
Camera color measurement and human perception differ in multiple aspects. First, cameras measure intensity linearly to radiometric quantities though humans have a nonlinear perception. Second, as Fig. 2 illustrates, the measured response of a mobile phone camera does not match the color matching functions (CMFs) [2]. The red and blue filters are much wider than the corresponding CMF. This cannot be corrected by a linear transformation and is a major influence on camera color reproduction. Modern mobile phone displays have a wider gamut than the sRGB standard [9] of desktop monitors, so for this research, the monitor is modeled as ideal and does not limit the color space. Therefore, a gamut mapping to XYZ color space is included in this evaluation method. Using mobile phone test cameras, it is possible to obtain raw sensor data from a mobile phone camera. This is useful, as the influences of the color filters and the post processing is often indistinguishable for standard mobile phone pictures. As a result, the method used here is based on the data of the demonstrators. In Figs. 5 and 6 the raw data of two different demonstrators are shown. The pictures illustrate the noise in the initial raw data. The red and yellow color patch in Fig. 7 are the processed data of the raw data in Figs. 5 and 6 by a mobile phone. The two color patches are less noisy than the raw data, pointing to a smoothing algorithm and showing the reproduced color after gamma correction and white balance. The boundary of the gray patch in Fig. 8 is lighter than the middle of the color patch and the adjacent black is darkened, a behavior known from contrast enhancement algorithms. With the color matching functions [2] (¯xλ, y¯ λ, and z¯ λ) and the spectral radiance coefficient qe;n λ of the Digital Colorchecker SG test colors the directly 20 December 2013 / Vol. 52, No. 36 / APPLIED OPTICS
8781
Fig. 5. Noise illustration of the raw data of two test cameras. Color patches red (G4) and yellow (H4) of the Digital ColorChecker SG in their respective camera color space for test camera 1.
Fig. 7. Examples of post processing. Result of the processed raw data of G4 and H4 of mobile phone 1.
perceived stimulus (X, Y, and Z) can be calculated, as shown in Eq. (2) for each test color n. Adding flash illumination Ee;flash λ to the ambient light Ee;amb λ and using the measured spectral sensitivity of the camera (rλ, gλ, and bλ) the camera raw data (R, G, and B) are calculated for each test color with Eq. (3) 8 9 8 9 Z < x¯ λ =
OSRAM Opto Semiconductors GmbH, 93055 Regensburg, Germany
2
Department of Physics, Philipps-University Marburg, 35032 Marburg, Germany
3
Institute of Electromechanical Design, Technical University of Darmstadt, 64289 Darmstadt, Germany *Corresponding author: felix.kimme@osram‐os.com Received 20 September 2013; revised 13 November 2013; accepted 20 November 2013; posted 20 November 2013 (Doc. ID 196963); published 16 December 2013
Flash light-emitting diodes (LEDs) of modern mobile phones lack red and cyan spectral parts, however, the color gamut of their respective displays has increased in recent years. The influence of this discrepancy on the color reproduction of smart phones is investigated in this paper. Based on the CIECAM02 color appearance model, a metric is introduced to judge color reproduction of mobile phones under flash LED illumination. An evaluation method is established to compare the visual appearance of a scene under various surrounding illuminations with the reproduction of that scene. To facilitate a comparison with measurements, the evaluation method is based on the raw data of two test cameras and a Digital ColorChecker SG. To reduce the color shift between perception and reproduction, optimized flash LED spectra are presented. A single-LED and a double-LED concept with adjustable color temperature are derived from these results. Additionally, the common characteristics of flash LED spectra giving good results is investigated, identifying the spectral parts with the most influence on camera color reproduction and showing the spectral parts not contributing or even resulting in poor color reproduction. Finally the efficiency of optimized flash LED spectra is compared to standard flash LEDs. © 2013 Optical Society of America OCIS codes: (100.2000) Digital image processing; (120.5240) Photometry; (150.2950) Illumination; (330.1710) Color, measurement; (330.6180) Spectral discrimination. http://dx.doi.org/10.1364/AO.52.008779
1. Introduction
Most modern mobile phones include an integrated camera. To give the phone a slim shape, small cameras with small optics are used. As the resolution of the camera sensors is increasing, the size of each pixel is decreasing and the light sensitivity is dropping. As a result, mobile phones rely on their flash light-emitting diode (LED) to increase illuminance levels, whereas normal compact cameras can take pictures without flash illumination in many cases. The typical mobile phone flash is realized with a white LED containing a yellow phosphor. Compared 1559-128X/13/368779-10$15.00/0 © 2013 Optical Society of America
to xenon bulbs used in compact cameras, the spectrum of those LEDs lacks red (above 640 nm) and cyan (around 490 nm) spectral parts, see Fig. 1. Additionally, mobile phone cameras differ in their spectral measurement compared to the human perception (Fig. 2). In the context of missing spectral parts while taking the picture, the question arises as to which spectral parts have an influence on good color reproduction. The color reproduction of an acquired picture is influenced by the ambient and flash light, the camera color filters, and the white balance algorithms. Ideally, two distinguishable colors when directly perceived should be equally distinguishable in the picture taken. To determine if the color reproduction is affected by those spectral gaps, an appropriate evaluation 20 December 2013 / Vol. 52, No. 36 / APPLIED OPTICS
8779
method is generated. It compares the reproduced image depending on the spectrum of the flash LED to the direct perception of the photographed scene. The goal is to find the optimal flash LED spectrum. The following sections explain this method, its influences, the selection of parameters, the resulting spectra with good color reproduction, and their common characteristics. 2. Method to Evaluate Optimal Flash Spectra
Fig. 1. Illumination used in mobile phone flash photography. Typical LED flash and xenon flash spectrum [1].
The method used to evaluate the color reproduction of flash LEDs is separated into three parts: the direct perception of a scene, the camera color measurement, and the reproduction of it on a display. The ambient illumination creates directly perceived color stimuli for each test color of the scene, which are transformed into a uniform color space. The same test colors under the flash LED illumination are measured by the camera. The radiation of the test colors is filtered by the camera color filters and creates a picture in raw data. This picture is processed by the white balance algorithm and displayed on an ideal monitor. The monitor is modeled with an unlimited color resolution and color space and thus does not restrict the color reproduction. Taking the ambient illumination while viewing the picture into account, the color difference between the direct perception of the scene and the perception of the reproduction in a uniform color space (UCS) can be determined for all test colors. A schematic of the method used to evaluate the color reproduction of flash LEDs is shown in Fig. 3. In the following, the parameters of this method will be described in detail: the test colors used, the uniform color space, the camera, and the assumed ambient illumination. A. Gamut of Test Colors
Fig. 2. Measured camera color filters (RGB) compared to the color matching functions (xyz) [2].
The surface color gamut represents most of the colors of a scene and is a subset of the visible colors. Pointer
Fig. 3. Method to evaluate optimal flash spectra. The direct perception of a scene (top left frame) under ambient light, is compared to the reproduction on an ideal monitor (right frame), created by a camera measurement (bottom left frame) under flash illumination of that scene. The CIE color appearance model (CIECAM02) is used to take account of the different chromatic adaptation. 8780
APPLIED OPTICS / Vol. 52, No. 36 / 20 December 2013
described by Luo et al. [8]. Δa0ucs and Δb0ucs describe the difference in the red/green and blue/yellow plane. ΔEucs is the Euclidean distance of the two stimuli. C.
Surrounding Illuminants
Typically, the illumination is a mixture of the flash LED and an ambient illuminant, such as the sun, halogen, discharge, or LED lamps. A camera can only guess the color temperature of the surrounding illuminant by the ratios of the red and blue color channel. Depending on those values, predefined white balance parameters are then chosen. Furthermore, the in-picture ratio of flash to ambient illuminant is not constant as the light sources have different directional characteristics and origins. To cover all these possibilities, the evaluation of the flash spectra is done for different scenarios. A scenario is characterized by the type of the ambient illuminant, its color temperature T amb , and the ratio ramb of radiometric power of the flash LED and the ambient light (ramb Φe;flash ∕Φe;amb ). Fig. 4. Surface colors measured by Pointer [3] under various illuminations and the color coordinates of the Digital ColorChecker SG und D65, shown in the CIE Diagram from 1931.
[3] created a dataset of 4000 measured stimuli under various illuminations, shown in Fig. 4. For practical purposes, the Digital ColorChecker SG [4] is used, which contains 56 black, white, and gray colors, and 84 test colors, that are well distributed over the surface color space. The spectral reflectivities of this color checker card were measured and included into the evaluation method, so that it can be compared to measurement results. The color coordinates of the color checker under D65 light are shown in Fig. 4. B.
Color Space
To quantify the perceived color difference between a directly viewed and a reproduced color, regardless of the color, a UCS is used. Color spaces that include models for chromatic adaptation and additional characteristics of human vision are called color appearance models (CAMs). There are several color spaces that consider chromatic adoption and are uniform to human perception. The most recent standard from the CIE is called CIECAM02 [5], that stands for CIE CAM from 2002. Xue [6] showed that CIECAM02 performs as well as CIELAB, for some cases it is more uniform than CIELAB. In addition, Moroney and Zeng [7] demonstrated that CIECAM02 has a better hue constancy than CIELAB. The standard CIECAM02 CAM with an extension for color uniformity called CAM02-UCS [8] is used here, including the color difference formula q 02 02 (1) ΔEucs ΔJ 02 ucs Δaucs Δbucs : ΔJ 0ucs is the lightness difference between reproduced and directly perceived stimulus for a test color, as
D. Camera Color Measurement and Artificial White Balance
Camera color measurement and human perception differ in multiple aspects. First, cameras measure intensity linearly to radiometric quantities though humans have a nonlinear perception. Second, as Fig. 2 illustrates, the measured response of a mobile phone camera does not match the color matching functions (CMFs) [2]. The red and blue filters are much wider than the corresponding CMF. This cannot be corrected by a linear transformation and is a major influence on camera color reproduction. Modern mobile phone displays have a wider gamut than the sRGB standard [9] of desktop monitors, so for this research, the monitor is modeled as ideal and does not limit the color space. Therefore, a gamut mapping to XYZ color space is included in this evaluation method. Using mobile phone test cameras, it is possible to obtain raw sensor data from a mobile phone camera. This is useful, as the influences of the color filters and the post processing is often indistinguishable for standard mobile phone pictures. As a result, the method used here is based on the data of the demonstrators. In Figs. 5 and 6 the raw data of two different demonstrators are shown. The pictures illustrate the noise in the initial raw data. The red and yellow color patch in Fig. 7 are the processed data of the raw data in Figs. 5 and 6 by a mobile phone. The two color patches are less noisy than the raw data, pointing to a smoothing algorithm and showing the reproduced color after gamma correction and white balance. The boundary of the gray patch in Fig. 8 is lighter than the middle of the color patch and the adjacent black is darkened, a behavior known from contrast enhancement algorithms. With the color matching functions [2] (¯xλ, y¯ λ, and z¯ λ) and the spectral radiance coefficient qe;n λ of the Digital Colorchecker SG test colors the directly 20 December 2013 / Vol. 52, No. 36 / APPLIED OPTICS
8781
Fig. 5. Noise illustration of the raw data of two test cameras. Color patches red (G4) and yellow (H4) of the Digital ColorChecker SG in their respective camera color space for test camera 1.
Fig. 7. Examples of post processing. Result of the processed raw data of G4 and H4 of mobile phone 1.
perceived stimulus (X, Y, and Z) can be calculated, as shown in Eq. (2) for each test color n. Adding flash illumination Ee;flash λ to the ambient light Ee;amb λ and using the measured spectral sensitivity of the camera (rλ, gλ, and bλ) the camera raw data (R, G, and B) are calculated for each test color with Eq. (3) 8 9 8 9 Z < x¯ λ =
