Camarray Quad Camera Depth Mapping OpenCV

1.Which seller did you purchase the product(s) from?
UCTRONICS
2.The Model number of the product(s) you have purchased?
https://www.arducam.com/product/arducam-1mp4-quadrascopic-camera-bundle-kit-for-raspberry-pi-nvidia-jetson-nano-xavier-nx-four-ov9782-global-shutter-color-camera-modules-and-camarray-camera-hat-b0331/
3.Which Platform are you using the product(s) on?
Raspberry Pi 3
4.Which instruction are you following?
Online Documentation at https://www.arducam.com/docs/cameras-for-raspberry-pi/synchronized-stereo-camera-hat/camarray-arducam-1mp-ov9782-color-quad-camera-bundle-kit/
5.Has your product ever worked properly?
Works OK
6.What problems are you experiencing?
I want to use OpenCV and Python to perform depth mapping between two of the cameras whilst having the remaining cameras positioned in the center. Whenever I try to run the example program provided at https://github.com/ArduCAM/3d-camera.git I get the error ‘VIDEOIO ERROR: V4L2: Pixel format of an incoming image is unsupported by OpenCV’.

 

Running v4l2-ctl --list-formats gives me:

'Type: Video Capture

 

[2] ‘RG10’ (10-bit Bayer RGRG/GBGB)"

 

I have ran the arducamstill function and get the 4 video feeds displayed, however they are useless to me unless I can perform depth mapping between two of the cameras of my choice.

 

How do I fix this? How do I get these cameras to work with OpenCV.
7.What attempts at troubleshooting have you already made?
Tried googling the error message given. Little to no information available online
8.How would you like us to help you?

Hi @jamesheaton

try this:

https://drive.google.com/drive/folders/16uYUxuGFStYRs-KpHuTtEaz_J31XC4Kc?usp=sharing

Hi, it’s still me. I had to create another account as I forgot my password and your password reset page does not work.

 

Anyway, I have followed the link and attempted to install the firmware update. However it won’t install - I get the following errors:

 

[ERROR] Failed to write register

[ERROR] Write update flag 2 error, ret: -1

 

I have no idea what these errors are, or how to fix them.

@wong please see above comment

Ok @wong I fixed the install error as I realised there was a typo. However now when I try to run the python script I get ‘open VIDEOIO(V4L2:/dev/video0): can’t open camera by index’

One more thing. I think this might be due to the errors I get with ./install_driver as described in the instructions, however I have found that even with this error it is the only way to get the arducam command able to display the camera feed without errors.

I am on the latest release of the Pi kernal but I followed the instructions to your downgrade patch and installed this before running ./install-driver.

The error comes from v4l.cpp when I run the Python command so I am presuming that this is still a problem with the drivers as this software does not seem to be able to recognise the device. The arducam function can still work fine however.

One more thing, I noticed that when downgrading and installing the drivers, I now have another video port /dev/video1 as well as /dev/vide0, is this expected functionality?

Hi @jamesheaton

sorry for the late reply,
Sorry I didn’t explain clearly, the reason why opencv cannot be opened is that it does not support the current format:
[0]: ‘RGGB’ (8-bit Bayer RGRG/GBGB)

[2] ‘RG10’ (10-bit Bayer RGRG/GBGB)"

The firmware update tool I sent you is used to modify the current format so that opencv can support it.

Regarding the driver problem, it seems that you have been able to list the current camera-related information, then there is no problem with the driver (the video1 node is expected, and it has no effect on this camera)

Hi @wong

I have just done the firmware update again, but after updating the formats remain the same. Is there something I am missing?

 

 

Hi again @wong

 

It appears that the firmware update was not actually installed. Earlier when I did the ./firmware_update command it exited with no display on the command line.

I have moved to a new pi and when I try to do the update I get this ouput:

[INFO] Write update flag 2

[INFO] Write checksum

[INFO] Write update flag

[INFO] Sending… | | ETA: 0h05m58s

[ERROR] Write firmware data failed, ret: -1, block:1

 

Any thoughts as to what is causing this error? I am following the instructions as described on the pdf

Hi @jamesheaton

It seems that the firmware update failed,
You can try to modify /boot/config.txt to look like this:
dtparam=i2c_vc=on,i2c_arm_baudrate=30000
dtparam=i2c_vc_baudrate=30000

When the update fails, you need to power off and restart.

Note: This update program was designed to work on Jetson Nano, so it does not work very well on Raspberry Pi.

Hi @wong that still didn’t work. So is there no way to get this working on the Pi? I will need to purchase a Jetson Nano?

Hello again @wong yesterday I managed to install the firmware update.

It succeeded only when I loaded up a clean version of Raspberry Pi OS and istalled the downgrade patch but not the driver. Then installing the driver allows me to use the camera with OpenCV.

However, the performance is extremely slow, pretty much unusable. I am getting around 0.2fps when I run 1_test.py, and when trying to capture chessboard pictures for calibration using 2_chess_cycle.py it cannot take pictures because it is running so slowly, the countdown timer goes past 0 and down into negative numbers infinitely.

I also now get different formats from running v4l2-ctl --formats-ext-ctl which are unsupported by the arducamstill tool. The three formats I get are BA81, pBAA and BG10 which all give unsupported format errors when running arducamstill.

You say the product is designed for Jetson Nano, can I expect a better performance on that platform?

Also, is there a way to reverse this firmware install? The stereovision python package is no longer being maintained and so I would prefer not to use that, and also I manged to get better performace for capturing images from the camera by extending the arducamstill.c program so I would prefer to go back to that since the performace with opencv and python is terrible.Video Formats & Python Performance

2_chess_cycle.py

Hi @jamesheaton

Yes, if you use opencv to access this camera, then opencv will use cpu for color conversion, which is very slow, and arducamstill uses hardware acceleration, so its speed will be much faster.
But you don’t seem to need to restore the firmware, arducamstill should support these formats, please refer to this:
https://github.com/ArduCAM/Arducam_OBISP_MIPI_Camera_Module/blob/12cd9e9f69a40dbb00c4a01fd7732ed3e6972739/arducamstill/arducamstill.c#L383

Hi @wong

I have question if there is a way how to speed up process on RPi4. I am also using opencv. I am streaming video trough opencv so the same as @jamesheaton. I also can work with black/white / grayscale images. Do you have any idea how to avoid this color conversion and stream more FPS ? I need at least around 25fps.

Thanks

Can you tell me what module you use?
If you are using a bayer format module, it needs to use the CPU to do color conversion, and the speed is difficult to improve(depending on the CPU performance, or you can use hardware acceleration and need to write your own code).

Hi Wong,

I am using B0331 with bayer format.

The output of this camera is in color format, so it needs to use CPU to do color conversion, and the speed of using CPU to convert is not very fast.

You can refer to this, it uses hardware acceleration for format conversion, but it is not very easy.

1 Like