IMX298 low FPS with OpenCV

Raspberry Pi Cameras mipi-camera-modules
1

Hi!

I need to capture video from a camera in a resolution of 1920x1080 with the highest possible frame rate, and then work with the resulting image in opencv.

Using arducamstill I get the maximum possible 60 FPS, however when using capture2opencv the frame rate is about 10 FPS.

To understand why this is happening, I wrote a small project based on capture2opencv, where I simply get an image from the camera and write it to cv::Mat using the get_image() function without displaying it through cv::imshow(). I measured the frame rate and got about 20 FPS.

I managed to figure out that this is happening because of the cv::cvtColor(*image, *image, cv::COLOR_YUV2BGR_I420) in the get_image() function. Removing this conversion, I get 16 ms to capture one frame, with it - 49 ms. Thus, on RPI4, this function takes 33 ms, which, in my opinion, is a very long time. Hence, the drop in the frame rate is exactly 3 times, since two frames are simply lost (I tried to depict it in the diagram).

problem
problem3590×1410 227 KB

Question: is it possible to somehow solve this problem? Speed ​​up cvtColor() execution or avoid using it altogether?

P.S. OpenCV is compiled with TBB, while the program is running with the cvtColor() function, all 4 cores are used.

2

Hi,
It is due to the cvtColor function will do color convert which will need much time.
You can use the following code to get frame
cap = cv2.VideoCapture(0,cv2.CAP_V4L2)
ret, frame = cap.read()

3

I am using C ++ and this does not work as the camera does not support v4l2:

pi@raspberrypi:~ $ v4l2-ctl --list-formats
Cannot open device /dev/video0, exiting.

This is why I used the capture2opencv example from MIPI_Camera repo.

4

Oh I get it.
The capture2opencv demo eixst some memory copy and the speed is not very fast.

5

Ok, how do I then capture the image from the camera as fast as possible? Taking into account the fact that in the future I need to work with him in OpenCV.

6

Hi,
If you want to use opencv, the MIPI_Camera driver is not a good choice. For the imx298, you can cosider to work with our pivariety board, if using pivariety board, it use unicam driver which is basd on v4l2 framework. I think you can use opencv to grep the image through the video0 node. For more about this, please refer to Pivariety IMX298 16MP Motorized Focus Camera Module - Arducam