ArduCam EK035 Mira220 RGB-IR green image with visible IR layer in Python OpenCV

cloudff · September 29, 2024, 10:52am

Hi all. I’m working with an ArduCam EK035 board and a Mira220 RGB-IR camera, using Python3 + OpenCV. I need a script that captures and saves the RGB layer without the IR layer into one AVI file (captured without the IR LED), and saves the IR layer into a separate AVI file (captured with the IR LED on). It would also be great if both images are available as OpenCV frames in Python for further processing. Has anyone done RGB/IR separation with this setup?

Dion · September 29, 2024, 11:08am

Welcome. Which config file are you using with the Mira220, and could you share the part of your code where you convert the raw buffer into the ‘bayer’ and ‘color’ images? That will help us see whether the issue is in the capture config or in the conversion step.

cloudff · September 29, 2024, 11:39am

Thanks. After the OpenCV operation, both the ‘bayer’ variable and the ‘color’ variable show a green image, and I can clearly see the IR layer when I light the scene with an IR LED. I started wondering if my color settings are wrong, or whether the provided configuration files are correct. Could there be a problem with the config files?

Dion · September 29, 2024, 2:18pm

The configuration files are correct. What you are seeing — a green image with a visible IR layer — is actually the expected result for this sensor with the current processing pipeline.

cloudff · September 30, 2024, 12:00am

Okay, but why is the IR showing up in the color output at all? I would have expected the color channels to stay clean.

Dion · September 30, 2024, 6:00am

The root cause is that the current processing pipeline does not remove the IR signal from the color channels. So when you illuminate with an IR LED, that IR signal remains visible in the color channels, which is why the output looks green and you can see the IR contribution.

The recommended approach is to use the ArduCam EVK Demo image conversion utilities to separate/handle the RGB and IR data. The relevant code is here, around line 84:

github.com

ArduCAM/ArduCam_EVK_Demo/blob/d4db9d03b22a31f680fd1b2bd07b44fa53d3d27a/python/project/img_cvt_utils.py#L84


      
                  image = cv2.cvtColor(image, COLOR_BayerGR2BGR)
              if color_mode == 2:
                  image = cv2.cvtColor(image, COLOR_BayerGB2BGR)
              if color_mode == 3:
                  image = cv2.cvtColor(image, COLOR_BayerBG2BGR)
              if 0 > color_mode > 3:
                  image = cv2.cvtColor(image, COLOR_BayerGB2BGR)
              return image
          
          def RGBIRToMat(data, color_mode):
              bayer, ir = arducam_rgbir_remosaic.rgbir_remosaic(data, rgbir_format[color_mode])
              color = cv2.cvtColor(bayer, cv2.COLOR_BayerRG2BGRA)
              ir_color = cv2.cvtColor(ir, cv2.COLOR_GRAY2BGRA)
              ir_resize = cv2.resize(ir_color, (bayer.shape[1], bayer.shape[0]))
              image = cv2.hconcat([color, ir_resize])
              return image
          
          def convert_image(data, cfg):
              Width = cfg.width
              Height = cfg.height
              bitWidth = cfg.bit_depth

If that code doesn’t fully cover your requirement, a customization service would be needed.

cloudff · September 30, 2024, 8:29am

Thanks. Could you give me the steps to run that demo so I can look at the conversion utilities?

Dion · September 30, 2024, 9:08am

Sure. Pull the project:
git clone GitHub - ArduCAM/ArduCam_EVK_Demo · GitHub

Enter the python directory:
cd ArduCam_EVK_Demo/python/

Install the dependencies:
python -m pip install -r requirements.txt

Enter the project directory:
cd project

Run the demo:
python async.py -c EK035_Mira220_124MHz_RGBIR/Mira220_MIPI_2Lane_640x480_12b_80fps.cfg

The config file for the EK035 Mira220 RGB-IR camera is EK035_Mira220_124MHz_RGBIR/Mira220_MIPI_2Lane_640x480_12b_80fps.cfg. The RGB/IR separation logic is in python/project/img_cvt_utils.py, specifically around line 84. Also keep in mind that IR signal removal from the color channels is not performed automatically, so you need that explicit conversion/processing step.

cloudff · September 30, 2024, 11:54am

Thanks for the details. I’d like to try to separate the RGB and IR layers myself. Could you share any technical documentation for the pixel format? If documentation can’t be shared, a descriptive formula and/or a graphical representation of the pixel bytes would help. The issue is still the same: ‘bayer’ and ‘color’ after the OpenCV operation show a green image with a visible IR layer.

Dion · September 30, 2024, 2:08pm

To summarize the current conclusion: the config files are correct, and the green image with a visible IR layer is expected because the current processing pipeline does not remove the IR signal from the color channels. The recommended path is to use the conversion utilities in python/project/img_cvt_utils.py (around line 84) to separate/handle the RGB and IR data. I don’t have a separate pixel-format document to share publicly, but the source in img_cvt_utils.py shows how the raw bytes are arranged and converted for this sensor, so that should give you the formula/graphical representation you need.