Adding Sony IMX219 sensor support on AmebaPro2 (RTL8735B), SDK release 9.6_r

FreeRTOS Multimedia
1

Hi,

We are moving our AmebaPro2 (RTL8735BDM) video doorbell from the IMX327 to the Sony IMX219 (IMX219PQH5-C, MIPI CSI-2 2-lane, RAW10). We plan to write the driver ourselves using the ameba-camera-sensor-driver repo, with the IMX327 driver as our template. Could you help with the following?

  1. Existing support
  • Do you already have an IMX219 sensor driver (sensor_imx219.bin) or IQ file (iq_imx219.bin), even as beta or internal-only? If yes, can you share it?
  • If not, can Realtek provide the driver or IQ tuning for this sensor, and what would the timeline and cost be?
  1. Sensor and MIPI support
  • Does the RTL8735B ISP and MIPI receiver fully support RAW10 input (SNR_10BIT)? Are there limits compared with 12-bit sensors, for example in black level, WDR or noise reduction?
  • What is the maximum supported MIPI data rate per lane in 2-lane mode? The IMX219 needs up to 912 Mbps per lane for 1080p30.
  • Which IMX219 mode do you recommend: 1920x1080 crop or 1640x1232 binned (which keeps the full field of view)? Are non-standard sizes like 1640x1232 supported by the ISP and encoder?
  • Is hs_term (MIPI termination) sensor-specific? What value should we use for the IMX219?
  1. Driver development
  • Is there documentation for the sensor driver interface (rts_isp_sensor_ops, rts_isp_sensor_init_info, rts_isp_sync_regs)? In particular, what exactly do pclk, hts/vts, size and start.x/start.y mean, and how do they relate to the sensor’s line length and frame length?
  • How does the ISP’s auto-exposure use get_exposure_range, get_tuned_again, get_tuned_dgain and exp_step? Are there rules for sensors like the IMX219 whose analog gain is not linear (gain = 256 / (256 − code))?
  • Can the sensor driver binary be debugged with logging? How do we view printf output from a sensor driver?
  • Which toolchain version should we use to build sensor drivers for SDK 9.6_r (asdk-10.3.0 newlib, or picolibc)?
  1. Image tuning (IQ)
  • What is the process to create iq_imx219.bin? Can we get the ISP IQ tuning tool and its documentation, and is an NDA required?
  • What calibration do we need to provide (raw captures, lens shading, colour charts, light box)?
  • For a doorbell with IR night vision (850 nm IR LEDs, IR-cut filter, gray mode), how do we create and tune the night IQ table that isp_set_day_night(1) uses?
  • Until a tuned IQ is ready, is the generic iq.bin safe to use for bring-up?
  1. Hardware
  • Does the RTL8735B need to provide the sensor clock (INCK, 24 MHz) through SNR_HCLK, or is an external crystal on the module recommended?
  • Which pins should we use for the sensor reset (XCLR) and the I2C bus? Are there power-sequencing requirements on the chip side?
  • Do you have a reference schematic for an IMX219 (or any 2-lane MIPI sensor) on the RTL8735BDM?
  1. Low light
  • We are concerned about the IMX219’s low-light and IR (850 nm) sensitivity compared with the IMX327. Do you have customers using the IMX219 for night-vision products, or would you recommend a different 1080p sensor that is already supported with good low-light performance?

Thank you.

2

Hi @deepakkumarbinwal,

Thank you for your patience on this matter.

Existing Support

  1. Currently there are no drivers for the IMX219.

  2. Please give us some time to discuss with the team to see if we are able to support the sensor bring up for the IMX219.

Sensor and MIPI support

  1. Yes the ISP and MIPI receiver is capable of supporting up to RAW12 SNR_8BIT/SNR_10BIT/SNR_12BIT inputs.

  2. The AmebaPro2 is capable of handling up to a maximum of 250 MHz for its pixel clock rate. You may wish to adjust the bit depth along with the horizontal/vertical blanking to tweak the clock rate within the limits of the pixel clock rate.

  3. Both options are perfectly fine and can be supported. The 2x2 binning mode is likely produce a better image quality than simply cropping the output width/height.

  4. Yes, sort of. Some sensors have registers settings to fix the hs_term for its D-PHY Tx/Rx. You will only need to adjust the hs_term in the drivers to match the timings.

Driver development

  1. There are some getting started guides here for these functions and here for the bring up procedure. In most cases your values should be: hts = LineLength and vts = FrameLength. You would typically only adjust the start.x and start.y if there is a bayer pattern mismatch.
    Line length relates to the sensor’s pixel width and is almost always much greater than the active pixel width. Similarly, the frame length to the sensor’s pixel height. This gives you the horizontal and vertical blanking that you will see in the sensor’s register. The additional blanking space is to allow the sensor to update its ISP before pushing out the next frame.

  2. You will need to refer to the sensor’s own parameters and derive an appropriate calculation to set its exposure and gain values. Trial and error is often required for this tuning process.

  3. Yes! You can definitely set printf(); statements within the functions and you should see them being output in the logs with the [VOE] tag. It is recommended that you enable #define APP_VOE_LOG_EN 1 in component\video\driver\RTL8735B\video_api.h and #define BOOTLOADER_VOE_LOG_EN 1 in component\video\driver\RTL8735B\video_boot.h.

  4. It is fine if you are using asdk-10.3.0 for the development of the drivers.

Image Tuning (IQ)

  1. Tuning Guide contains the rough steps to tuning the IQ of a particular sensor. It requires our RealCam software to do and you may request for access by following the setup instructions.

  2. The tuning components really depends on the desired quality and you will have to tune accordingly. There are many parameters for you to adjust so there is no straightforward step I can advise on this.

  3. Check if the guide is sufficient to help you get start on this and reach out again in this thread if not.

  4. In fact, we would advise that you begin with the default iq.bin to for tuning purposes.

Hardware

  1. It will be good to allow the RTL8735B to provide the input clock rate as that should give you more control over the PLL clock timings.

  2. The pinout for the MIPI connector is fixed and must not defer from the AmebaPro2’s design.
    Each camera has their own power-up/down sequences to follow, you will have to get the datasheet for reference. However, these sequences are often similar between modules of the same brand, so you could probably reuse the one in IMX327. Do take note that the IMX219’s input clock rate may need to be changed to 24 MHz instead.

  3. No, there are no references for this as we do not have this sensor. Custom sensor manufacturers will only need the matching 24-pin configuration and you may leave the rest of the design to them. Feel free to send your schematics here for a second opinion before making the samples.

Low Light

  1. We do not have any immediate recommendations for this and highly suggest that you look into a few sensors before making a decision. It will be best to try them out before deciding on the final one as different scenes with their lighting conditions will affect your image quality.