A Graduate Level Course on Image Sensors and Digital Cameras

A GRADUATE LEVEL COURSE ON IMAGE SENSORS AND DIGITAL CAMERASAbbas El Gamal and Ting ChenInformation Systems LaboratoryDepartment of Electrical EngineeringStanford UniversityStanford, CA 94305{abbas, tingchen}@stanford.eduABSTRACTA graduate level course on the design and analysis of imagesensors and digital cameras is described. The courseuses a MATLAB based camera simulator. Information onthe course and brief descriptions of several class projectsare presented.1. INTRODUCTIONDigital cameras are rapidly replacing film cameras as thepreferred image capture devices. As a result major effortsare being spent on improving digital camera design. Inparticular CMOS image sensors are emerging as an alternativeto CCDs. Using CMOS technology, image captureand processing can be integrated on a single chip, enablingmany new imaging applications that require low cost, lowpower, and small size. The use of CMOS technology hasalso been attracting many new engineers and researchers tothe field of image sensor and digital camera design. Thisfield, however, has traditionally been highly specialized andproprietary with very few textbooks and research publicationsavailable and almost no courses developed. Anotheraspect of digital camera design that makes it difficult to researchand teach is the breadth of the background needed.The study of digital camera system design requires someworking knowledge of optics, photometry, device physics,circuits, noise analysis, color science, and image processingalgorithms and architectures.To satisfy the needs of interested Stanford graduate studentswe developed a one-quarter course on the design andanalysis of image sensors and digital cameras. This paperwill summarize the objectives, prerequisites, and syllabusof the course. We will describe a camera system simulationtool (vCam) that we developed as part of our research andused in the course. We will also provide several examplecourse projects that used vCam. More information on thecourse can be found on the class webpage [1].2. COURSE OBJECTIVESThe course objectives are(i) to provide an introduction to the design and analysis ofimage sensors especially CMOS imagers,(ii) to develop basic understanding of the signal path throughdigital camera,(iii) to develop understanding of the performance measuresand tradeoffs involved in the design of sensors and digitalcamera systems, and(iv) to have fun using many basic EE knowledge to understanda “cool” system.3. COURSE CONTENTThe course is intended for graduate students with interestin image sensors and image system engineering, e.g., imageprocessing, digital photography, computer vision, andgraphics. Undergraduate level device, circuit, signal andsystem background is required for successful completion ofthe course. Due to the diverse and up-to-date nature of thetopics covered, the course has no required textbook. Instead,a complete set of lecture notes has been developed.<strong>Course</strong> requirements include: weekly homework assignments,a take home midterm, and a class project. The homeworkand midterm exam problems require a mixture of hand analysisand simulation using MATLAB and vCam. The projectinvolves either pre-assigned or self-pursued topics and usuallyrequires three to four weeks to complete for studentsgrouped in pairs.Lecture topics include:1. Photodetectors: photodiode and photogate; photocurrent,dark current, quantum efficiency, and spectral response.2. Introduction to CCDs and CMOS passive and active pixelsensors3. Noise analysis, sensor signal-to-noise ratio (SNR), anddynamic range.4. Fixed pattern noise.5. Spatial resolution and modulation transfer function.6. Introduction to imaging optics.

7. Color Processing: color demosaicing, color correctionand white balance.8. Analysis of digital camera signal path, from the scenethrough the imaging optics, the sensor, the A/D converter,to the different color processing steps using vCam, a MAT-LAB based digital camera simulator.9. Recent research topics, e.g., digital pixel sensor and highdynamic range schemes.4. DIGITAL CAMERA SIMULATORA digital camera, as depicted in Figure 1, is a complex systemcomprising components that use different technologies.Given the complexity of the system, simulation tools areneeded to study any design tradeoffs and their ultimate effectson image quality. To the best of our knowledge nosuch simulation tools are commercially available. To adressthis need, we developed our own camera simulator (vCam).Figure 2 shows vCam’s imaging pipeline. The simulatorprovides models for the scene, the imaging optics, and thesensor. The simulator is written in MATLAB and is modularto facilitate future modifications and extensions.LensCFAImagesensorAGCADCAutoFocusAutoExposureColorProcessingImageEnhancement&CompressionControl &InterfaceFig. 1. Block diagram for a typical digital camera systemstudents resulting in a number of research publications. Thefollowing are brief descriptions of the most recent projects.5.1. Pixel size studyAn important parameter in the design of an image sensoris the choice of pixel size. A small pixel size is desirablebecause it results in a smaller die size and/or higher spatialresolution, while a large pixel size is desirable because it resultsin higher dynamic range and signal-to-noise ratio. Weasked the students to investigate this tradeoff analyticallyand using vCam. This project was pursued further resultingin a conference paper [2].5.2. High dynamic range schemesAn important figure of merit for image sensors is dynamicrange, i.e., the range of illumination that can be captured ina single image. Several schemes have been proposed to enhanceCMOS image sensor dynamic range. The purposeof this project was to compare the performance of theseschemes based on SNR and image quality using vCam. Theprojects confirmed the results of the hand analysis in [3].5.3. “Honey Comb” pixel arrayFuji Film recently introduced a new CCD image sensor [4],which they called “Honey Comb” CCD. The main motivationfor this structure is to enhance spatial resolution withoutincreasing the number of pixels or die size (cost). Thegoal of this project was to use vCam to validate (or dispute)the claimed advantage of the “Honey Comb” array structureover the conventional rectangular pixel array. The projectshave shown that for scenes with mainly vertical and horizontalspatial structures, the “Honey Comb” pattern is indeedbetter. No advantage was observed for other types ofscenes.6. REFERENCES[1] “Introduction to Image Sensors and Digital Cameras,”in http://www.stanford.edu/class/ee392b.[2] T. Chen, P.B. Catrysse, A. El Gamal, and B.A. Wandell,“How small should pixel size be?,” in Proceedings ofSPIE, April 2000, vol. 3965, pp. 451–459.Fig. 2. vCam imaging pipeline5. SAMPLE COURSE PROJECTSAn important requirement for the course is the final classproject. Each year we proposed one or two projects that thestudents did in pairs. Most projects required the studentsto use vCam as a tool for optimizing or comparing designoptions. Several projects have been pursued further by the[3] D.X.D Yang and A. El Gamal, “Comparative analysisof SNR for image sensors with widened dynamicrange,” in Proceedings of SPIE, February 1999, vol.3650, pp. 22–28.[4] T. Yamada, Y.G. Kim, H. Wakoh, T. Toma,T. Sakamoto, K. Ogawa, and E. Okamoto, “A ProgressiveScan CCD Imager for DSC Applications,” 2000IEEE ISSCC Digest of Technical Papers, vol. 43, pp.110–111, February 2000.