Texas Instruments Incorporated Previewer-Resizer Driver Software Design Specification

QRSA # 00011909

Revision ALSP 2.10

20450 Century Boulevard

Germantown, MD20874

Fax: (301) 515-7954

LSP 2.10

Previewer-Resizer Driver

Software Design Specification (SDS)

Revision C

QRSA # 00011952

June15, 2009

Prepared by: Murali Karicheri

Copyright © 2007 Texas Instruments Incorporated

All Rights Reserved

NOTICE OF CONFIDENTIAL AND PROPRIETARY INFORMATION

Information contained herein is subject to the terms of the Non-Disclosure Agreement between Texas Instruments Incorporated and your company, and is of a highly sensitive nature. It is confidential and proprietary to Texas Instruments Incorporated. It shall not be distributed, reproduced, or disclosed orally or in written form, in whole or in part, to any party other than the direct recipients without the express written consent of Texas Instruments Incorporated.

TI Confidential and ProprietaryPage 1

Texas Instruments Incorporated Previewer-Resizer Driver Software Design Specification

QRSA # 00011952

Revision CLSP 2.10

Revision Record
Document Control Number: QRSA # 00011952 / LSP 2.10
Document Title: Software Design Specification
Revision / Description of Change
0.0 / Initial Release
0.1 / Updated review comments on 4/22/2008
0.2 / Updated Niclas comments to add serialization support. 5/8/2008
A / Updated the TOC, added capability to reset module configuration to POR values for PREV_SET_PARAM (section 9.1.2.3)
B / Update after implementation
C / Updates based on final tested functionality at GA

Note: Revision letter increments only upon approval via the Quality Record System.

TI Confidential and ProprietaryPage 1

Texas Instruments Incorporated Previewer-Resizer Driver Software Design Specification

QRSA # 00011952

Revision CLSP 2.10

TABLE OF CONTENTS

1Introduction......

2Scope......

3References......

4Definitions......

5Overview......

6Requirements......

6.1Assumptions......

6.2Constraints......

7Issues......

8Usecases......

8.1.1Previewer device use case......

8.1.2Resizer device usecase......

9Design......

9.1Component Interaction......

9.1.1Initialization......

9.1.1.1 Previewer-Resizer Driver registration/ un-registration......

9.1.2Command Processing......

9.1.2.1 Open/close davinci_previewer device......

9.1.2.2 Open/close davinci_resizer device......

9.1.2.3 Configuration and Control......

9.1.2.3.1PREV_ENUM_CAP......

9.1.3Interrupt Handling......

9.1.3.1 On the fly mode......

9.1.3.2 Single shot mode......

9.1.4Buffer Allocation and Management module......

9.1.5Multiple channel support in Resizer......

9.1.6Multiple Instance Handling for Previewer......

9.1.7Previewer-Resizer Hardware Layer......

9.1.8Sequence diagrams......

9.1.8.1 Preview Engine Configuration......

9.1.8.2 On the Fly mode preview-resize and control......

9.1.8.3 Single Shot mode – Preview & Resize......

9.1.8.4 Single Shot mode – Multiple concurrent Resize operations......

9.2Software Resource Requirements......

9.3Internal Data Structures......

9.4APIs......

10Header files......

10.1Previewer......

10.2Resizer......

11Integration......

12Future Extensions......

TI Confidential and ProprietaryPage 1

Texas Instruments Incorporated Previewer-Resizer Driver Software Design Specification

QRSA # 00011952

Revision CLSP 2.10

1Introduction

The document describes the detailed design specification for thePreviewer-Resizer Driver. This driver is used on the DM355 and DM365 platforms to do Preview and Resize of the image data. Preview is the process of fine tuning the capture image so that it is made displayable. Resize is the process of down scaling or up scaling a image to a desired dimension. The term Imp Previewer – Resizer driver is also used in this document to refers to the same driver.

2Scope

The purpose of the project is to develop a Linux based Previewer-Resizer driver for DM355 and DM365 SoCs.. The overall scope of this document is to cover the detailed design specifications for the driver.

3References

The following references are related to the feature described in this document and shall be consulted as necessary.

No / Referenced Document / Control Number / Description
1 / PSP-LSP 2.10 PRD / QRSA # 00011700 / PSP-LSP 2.10 Product Requirement Document.
2 / DaVinci CCDC Capture driver SDS / QRSA # 00011909 / CCDC Driver Software Design Specification
3 / DM365 IPIPEIF Design Specification / DM365 IPIPEIF Design Specification Rev 1.00.w.04
4 / Image IPIPE 5.1 Specification / Rev 0.4.10 of DM365 Image IPIPE 5.1 Specification
5 / Peripheral Reference Guide for DM355 Subsystem / Version 1.3 of Peripheral Reference Guide for DM355 Subsystem

Table 1. Referenced Materials

4Definitions

Acronym / Description
VPFE / Video Processing Front End
IPIPE / Image Pipe
CCDC / CCD Controller
SD / Standard Definition
HD / High Definition
API / Application Programming Interface
ISIF / Image Sensor Interface
KConfig / A configuration framework available in Linux to change build time configurations

Table 2. Definitions

5Overview

Video Processing Front End (VPFE) provides an input interface for external imaging peripheral such as image sensors, video decoders etc. This subsystem consists of the CCD controller (CCDC) aka Image Sensor Interface (ISIF), Image Pipe (IPIPE) and other image processing IPs. IPIPE is a parameterized hardwired image processing block whose image processing functions can be customized for each sensor type to realize good still image quality.IPIPE hardware has many modules that collectively is called the Previewer which is responsible for doing preview of the image data. In this document, preview means, processing the image to make it viewable. This includes format conversion from Raw Bayer RGB data to YUV data, applying tuning techniques such as Gamma correction, RGB to RGB conversion, Noise filtering etc. Please refer reference #4 & #5 for the details of different modules that are available for preview/fine tuning of the image on DM365 and DM355. IPIPE also has two Resizers that allows for the resize of input image to two different output resolutions.

Figure 1shows the basic block diagram of IPIPE subsystem in DM355/360 VPFE. There is some difference in the IPIPEIF(IPIPE Interface) available in DM355 and DM365. In DM355, CCDC is interfaced with the parallel port. IPIPEIF gets input from CCDC or SDRAM and feeds data only to IPIPEHowever in DM365, both CCDC and IPIPE receives input from the IPIPEIFand IPIPEIF is connected to parallel port as well as Buffer logic (to fetch data from SDRAM). IPIPEIF has multiplexers to allow input come from either the SDRAM or Parallel port. The IPIPEIF also has Dark Frame subtraction module that allows subtracting a dark frame from the input data.

Preview and Resizers can operate in Single Shot mode or On the fly (continuous) mode. In the Single Shot mode, the module reads image data from the SDRAM and writes the processed data back to SDRAM. In the continuous mode, it gets data directly from the ISIF or CCDC and then do processing on the fly. This is mode of operation is called On the fly mode in this document. Note that driver can work in one of these modes and the same is configured through boot argument.

Figure 1 Block diagram of IPIPE in DM365

IPIPE process two types of data.

  1. Raw Bayer image data
  2. YUV image data.

Raw Bayer image data is usually received from a sensor module like MT9T001. The YUV data comes from a decoder peripheral like TVP5146. Previewer takes only Raw Bayer data as input. The Resizer will take both Raw Bayer data as well as YUV data as input.

Following are the data formats supported at the input of IPIPEmodules.

  1. Raw Bayer (SBGR8/SBGR16)

This is the Bayer RGB data (2x2 RGGB block). This uses up to 16 bit data. There is an option to capture 8 bit packed data from the CCDC Driver using A-Law or DPCM compression. In the 16 bit image data, valid data may be 10 or 12 bit as supported by the image sensor.

  1. YUV 422 (YUYV or UYVY)

This is the interleaved YCbCr data. 16 bits per pixel. The chroma order may be different for YUYV and UYVY.

Following are the supported output formats

  1. Raw Bayer (SBGR16)
  2. YUV 422 (YUYV or UYVY)
  3. YUV 420 planar (NV12 or NV21)

6Requirements

Following are the major requirements that will be addressed in this design document:

  • The Previewer-Resizerdriver will support onesoftware channel (logical channel) of preview and one software channel (logical channels) for resize.
  • The Previewer logical channel will support one IO instance and multiple control instances.
  • Each Resizer channel will support one IO instance.
  • One Resize logical channel may be chained with the Previewer channel to do both preview and resize simultaneously on the input image.
  • The drivers will be built as loadable modules. They would be built so that they can be statically built or inserted/removed after boot.
  • The driver will support Single shot and On the fly (continuous) modes of operation.
  • Support for up to 12 bit input data
  • Allow bypass of individual processing blocks if permitted by the hardware.
  • Allow preview and resize processing of VGA, XGA, SVGA, NTSC, PAL, 720p & 1080p (DM365) resolution images
  • The Input shall come from CCDC/ISIF or SDRAM.
  • IOCTLs to support configuration of preview engine parameters before starting streaming as well as selective control operations while streaming.
  • Support resizing of BayerYCbCr. For YCbCr data, only top or bottom field resizing supported. Support image flipping
  • Each Resizerlogical channelcan support 2 resizers (limited to VGA resolution for the second resizer output).
  • Support configuration for following preview engine modules
  • Defect Pixel Correction
  • 2D Noise Filter 1
  • 2D Noise Filter 2 (Only in DM365)
  • Green Imbalance Correction
  • White balance
  • CFA interpolation
  • RGB2RGB blending -1
  • RGB2RGB blending – 2 (Only in DM365)
  • Gamma Correction
  • 3D LUT for RGB data converter
  • RGB2YCbCr Converter
  • Global brightness and Contrast
  • Edge Enhancer (EE)
  • Chroma Artifact Reduction (CAR)
  • The driver will support buffer access mechanism through both memory mapping and user pointers.
  • When memory mapped IO scheme is used, driver allocates contiguous memory for holding the input and output frames.

6.1Assumptions

  • Loading of all the modules/drivers will be done before running the application.
  • Error codes, returned from driver, are standard Linux error codes.
  • When User pointer IO is used, it is assumed that application provides contiguous memory to the driver for holding input or output frames

6.2Constraints

  • CMOS sensor MT9T001/31 will support 10 bit data and MT9P031 will support 12 bit data. So testing will be done only for these data sizes.
  • IPIPE input buffer addresses must be multiple of 32.
  • Operating mode on the preview and resizer device to be the same at any instance if the driver is running on IPIPE as per hardware requirement.
  • Previewer may be used independently (Resizer becomes unavailable) to convert Bayer RGB data to YUV data or Resizer may used independently to convert YUV data to YUV 420 semi planar data (Previewer becomes unavailable) or to do resize of YUV data or Resizer can be chained to previewer to do Bayer RGB to YUV/YUV 420 semi planar data conversion.

7Issues

  1. Attached spreadsheet has the issues / questions related to DM365 ipipe hardware. Would need input from hardware team to resolve these.
  2. Action: These have been discussed over email and resolved with the hardware team.
  3. The current API available for the driver has exposed all of the hardware configuration parameters to user API. This makes the API very complex for configuration. Also it allows only configuring the hardware once before the ipipe process starts. However there is requirement to change configuration or do control operation while ipipe process is ongoing or per ipipe request. In this design, only selective configuration parameters are made available to user application, others are either build time configured or hard coded in the source. Also control ioctls are added. This would require migration of existing applications to use the new set of ioctls provided by Previewer-Resizer Driver in this release and needs to be agreed upon by all parties concerned.
  4. Action: During the review comments discussion, the final decision taken was to expose all parameters to application. Driver sets default values for these parameters and application will GET/SET ioctls to read current parameters and modify and re-apply them at the driver.

8Usecases

The previewer-resizer driver will be used by following classes of applications:-

  1. Streaming applications. These applications essentially use the driver for
  • preview only
  • preview & resize
  • resize only

These applications usually configure the driver to operate in the “continuous” mode. The frame rate is dictated by the standard supported at the image sensor (example:-MT9T001) or decoder (TVP5146)

  1. Resize applications. These applications are expected to configure the resizer to resize a given input image to get a desired resolution output image.
  2. Control applications. These applications are for fine tuning the image properties such as contrast, brightness, gain, white balance etc while streaming is going on. For example when the previewer is operated in the continuous mode, a control application can open the preview device and set/get control ioctl.

Figure shows these different applications using the driver.

8.1.1Previewer device usecase

Following flowchart illustrates the usecase for the Previewer device.

8.1.2Resizer device usecase

Following flowchart illustrates the usecase for the Resizer device.

9Design

The section describes the detail design ofPreviewer-Resizerdriver.The driver provides a character device interface to user application to to do preview and resize. Figure 2 shows the overall design of the driver. Application accesses service from the preview device using the /dev/davinci_previewer device and resizer device using the /dev/davinci_resizer device. The device name has been renamed(dm355_ipipe used in previous release) to hide the details of the underlying hardware from the user application.

Previewer-Resizer driver is implemented in two layers:-

  1. Hardware Independent Layer
  2. Hardware Layer

The hardware independent layer is the upper layer of the driver and provides APIs to the application to open a device, configure the device and to do preview and/or resize of the input image data through IO as well interrupt handling. This layer will remain the sane on different SoCs.

The hardware layer is the lower layer responsible for configuring the hardware by writing into registers.This layer will be different for different SoCs. The term Image Processing (IMP) is used to refer this layer in general. The hardware layer is IPIPE on DM355 and DM365.

Figure 2Overall Design of Previewer-Resizer Driver

9.1Component Interaction

The section describes the detailed design showing the interaction between different layers and functionalities of the Previewer-Resizer Driver. Figure below represents detailed design for the Previewer-Resizer Driver.

Figure 3Detail Diagram for Previwer-Resizerdriver

The Previewer-Resizerdriver is divided into following layers as shown in Figure 3

  • Hardware Independent Layer :

This layer of Previwer-Resizerdriver implements the Linux Kernel device driver interface to support the character device.

  • Hardware Layer:

This layer will be responsible for all hardware related configurations. This layer will provide routines to set/get configuration in hardware.

The Previwer-Resizerdriver can be divided into following modules as per their functionality:

Initialization: This module is responsible to handle all the initialization activities including driver registration, driver un-registration, and initialize device with default values.

Command processing:This module is responsible for receiving commands from user application and perform operations as needed. It is responsible for open/close of the device, and process IOCtl commands from user applications.

Configuration & Control: This module is responsible to configure and control IPIPEhardware.For this the driver will support various IOCtl commands. Application will use these commands to interact with driver.

Interrupt Handling: This is the interrupt handler for Previwer-Resizerdriver. It will handle interrupt generated by IPIPE hardware.

Buffer Management: This module is responsible for exchanging buffers between driver and application and maintaining the buffers queues.

9.1.1Initialization

9.1.1.1Previewer-Resizer Driver registration/ un-registration

Driver registration

The sequence below shows the driver registration process. Drivermodule init function will be called after the driver will be loaded into the kernel. It create a simple class device and add it to sysfs by calling cdev_add()Driver can be loaded (inserted) at the time of booting or dynamically once kernel is up.Init function registers the driver to the Linux kernel. Then the platform registration is carried out...

Driver un-registration

Driver’s cleanup function will be called at the time of removal of the driver or when Linux kernel is shutting down.It removes the class device that was registered at init. , It then un-register with the platform, and finally un-register the driver with the kernel.

9.1.2Command Processing

The driver gets user commands from the device driver interface. These includes open/close and Configuration & Control commands.

9.1.2.1Open/close davinci_previewer device

Open

Application will be able to open the device for doing preview of an image. Only one primary user is allowed. However multiple control instances are allowed. The file handle that is used for configuring the operating mode is the primary user. All other instances will be treated as control instances and will be able to invoke only IOCTls that can be executed during operation, like white balance, gain adjustment etc.

Following sequence happens when user call open() API call.

  1. Pre-condition: User opens davinci_previewer device.
  2. Allocate memory for file handle
  3. save device chan reference in the private_data field of the file handle.
  4. Increment the usage counter

Close

Following sequence happens when user call close() API call.

  1. Pre-condition: User close davinci_previewer device
  2. if this is the primaryuser and usage count is not 1, return error. primary user file handle must be the last one to close.
  3. if this is the primaryuser and last one being closed, Reset primary_user field in the device chan. Also clear configured status.
  4. decrement channel usage counter
  5. free file handle private_data area.
9.1.2.2Open/close davinci_resizer device

Open

Application will be able to open the device for doing Resize of an image. The device support one logical channel. When application wants to do preview and resize using resize chained in the data path, it will be able to open resizer device and set the “chain” parameter in the resizer configuration. Once this is done, IO is disallowed on this channel and application uses previewer device to do the IO.