Maximum Framerate IMX298 + RPI4

Raspberry Pi Cameras mipi-camera-modules
1

What is the maximum framerate for the IMX298 on RPI4 at full resolution?

I wrote some short code in C to test. I set auto exposure off, exposure time to 1, resolution to 4656x3496 and I used IMAGE_ENCODING_RAW_BAYER. I measured 4.4fps (not including the first frame which takes longer).

Is it possible to increase the speed? What is the primary bottleneck?

I also tried IMAGE_ENCODING_I420, but the speed is similar (even though the bandwidth is larger). I also tried IMAGE_ENCODING_JPEG, but it causes a crash:

usrvcsm: [vcsm_malloc_cache]: [5925] [mmal_vc_port buffer]: ioctl mem-alloc FAILED [-1] (hdl: 0) usrvcsm: [vcsm_vc_hdl_from_hdl]: [5925]: invalid device or handle! usrvcsm: [vcsm_lock]: [5925]: invalid device or invalid handle! mmal: mmal_vc_shm_alloc: could not allocate 48944896 bytes of shared memory (handle 0) - mem (nil), vc_hdl 00000000 mmal: mmal_vc_port_payload_alloc: vc.ril.image_encode:out:0(JPEG): failed to allocate 48944896 bytes of shared memory mmal: mmal_pool_initialise_buffer_headers: failed to allocate payload 0/1 mmal: Failed to create isp output pool. <code></code>

2

Hi,

The main reason is the 2 channel bandwidth limitation of PI.

Please change the GPU memory to aviod the malloc error .

3

Thanks. I was able to get the JPEG setting working at full resolution. I can see that you answer this very basic question all the time, so thank you for your patience.

I thought that JPEG compression was performed by the IMX298. But when I set IMAGE_FORMAT fmt = {IMAGE_ENCODING_JPEG, 1}; and arducam_set_control(camera_instance, V4L2_CID_EXPOSURE, 1); so that the images are only 0.2MB, I can see that the framerate is still 4.4fps. So this must mean that the JPEG compression is performed by the arducam library.

When I modify the raw_callback or yuv_callback examples to use the full resolution I also get 4fps.

I read that CSI2 supports 1Gbit/s per lane, for 2Gbit/s total = 2,000,000,000 bits/s. A full frame of IMAGE_ENCODING_RAW is 20,407,296 bytes = 163,258,368 bits. 2,000,000,000 bits/s / 163,258,368 bits = 12.2 fps.

And on this page https://www.arducam.com/docs/cameras-for-raspberry-pi/mipi-camera-modules/ it says that the full resolution framerate is 10fps. This page says the full resolution framerate is 30fps https://www.arducam.com/docs/cameras-for-raspberry-pi/mipi-camera-modules/imx298-16mp-color-rolling-shutter-camera-for-raspberry-pi/

How can I achieve 10-12 fps?

4

Hello,

Yes, you are right. Our configuration currently runs to 8 frames. You can use arudcamstill to test. Due to different exposure times, there will be some delay. The mainly factor is the sensor’s configure. Recently, I will try to optimize the sensor configuration

IF you use the encoder to get the jpeg image, there is a memory copy of the CPU fetch graph, and a lot of time will be lost.

 

5

Hello,

I have optimized the imx298 configuration just now and the max resolution can up to 12 fps.

 

6

Hi bin, I pulled the new code including the new library:

MD5(lib/libarducam_mipicamera.so)= ded62a83575b0a5f61ba13dadb597a0e

Then I did make clean; make arducamstill, but I still see 4fps:

I also checked with my own code and it still shows 4.4fps.

https://gist.github.com/kylemcdonald/eb5c72c19129039ee4f83472d4450298

7

Hi, You should make install to install the new lib at the MIPI_Camera path.

However, some people recently reported that the high frame rate is too fast, and our line will be a little disturbed. I still reduced the frame rate to 8 frames.

 

8

Of course, I’m surprised I forgot make install. Thanks.

I tried the latest code and now I get between 7-8 fps.

I also tried the previous version (after git reset --hard f337a4ddb7430e46e8db70f9cb01ba459c50fd72; make clean; make install) but I was unable to get 12 fps. Instead I get a Segmentation fault.

Would it be possible to expose the internal parameter that is allowing you to set the maximum framerate? I would like to be able to set this myself.

9

Hello,

You can try to reduce the value of the 0x0307 register to increase the clock.

 

10

Thanks! I will look into this. I haven’t set the registers on Raspberry Pi before so I will look for some examples first.

For anyone else coming across this thread, here are some more references that might be relevant:

https://www.raspberrypi.org/forums/viewtopic.php?t=254733

https://www.raspberrypi.org/forums/viewtopic.php?p=630909#p630909

11

Great, thanks for your share.

Just use this API I attached for you to change the value of the register.

int arducam_write_sensor_reg(CAMERA_INSTANCE camera_instance, uint16_t address, uint16_t value);

12

Thanks! I used arducam_read_sensor_reg to get the current value, and I see it is 112.

Assuming this is directly connected to the clock speed, I tried to change it to 74 to get 12fps with arducam_write_sensor_reg(camera_instance, 0x0307, 74); but I don’t see any change.

I also tried changing it to a larger value like 224 but I did not see any change.

I also tried setting the register immediately after arducam_init_camera and immediately before arducam_capture but this showed no difference in results.

https://gist.github.com/kylemcdonald/eb5c72c19129039ee4f83472d4450298#file-capture_fast-cpp-L30

13

Hello,

Oh I know, the sensor is very strange, you should change the clock before it’s streaming on register write 1.

I have help you add the mode. I have tested it on PI3 platform. Please try to download our new lib and make install. Then retry uisng mode 4 .

14

I pulled the new code (library) and ran make install then I replaced the call to arducam_set_resolution with arducam_set_mode(camera_instance, 4); immediately after calling arducam_init_camera.

From my tests, this new mode actually runs slower. When I modified raw_callback.c to test the speed: mode 3 runs around 7.8fps, and mode 4 runs at 4.8fps.

I checked the 0x0307 register and I noticed that it is 160 in mode 4 and 112 in mode 3.

I tried changing the register 0x0307 in mode 4, and it did not have any effect.

15

Hello,

Strange, Have you tested it on PI3 platform?

 

16

Hello,

Strange, Have you tested it on PI3 platform?

 

17

I don’t have a PI3 for testing, and need a PI4 for my application. If there are any updates to bring the PI4 up to the 12fps frame rate please let me know!

18

Hi,

I am sure I have configure the sensor to 12 fps. I just download our new lib and test it again

19

I did a fresh clone and install:

$ git clone https://github.com/ArduCAM/MIPI_Camera.git
$ cd MIPI_Camera/RPI
$ make install
$ make arducamstill
$ ./arducamstill -t 0 -m 4

And I get the 4 fps result. When I use mode 3 I get 8 fps.

$ cat /etc/os-release
PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”
NAME=“Raspbian GNU/Linux”
VERSION_ID=“10”
VERSION=“10 (buster)”
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL=“http://www.raspbian.org/”
SUPPORT_URL=“http://www.raspbian.org/RaspbianForums”
BUG_REPORT_URL=“http://www.raspbian.org/RaspbianBugs”

$ hostnamectl
Static hostname: raspberrypi
Icon name: computer
Machine ID: 3001bc63adc342ff8a14dae4a06e58d6
Boot ID: 131ad9825a4d48dd913fdd4ed5dcf35e
Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 4.19.97-v7l+
Architecture: arm

$ gcc -v
…
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

$ openssl md5 lib/libarducam_mipicamera.so
MD5(lib/libarducam_mipicamera.so)= 6046474cbf30d0fb45be1630a952b819

$ openssl md5 /usr/lib/libarducam_mipicamera.so
MD5(/usr/lib/libarducam_mipicamera.so)= 6046474cbf30d0fb45be1630a952b819

This is on a fresh install of the operating system. I don’t have any other peripherals installed. I haven’t overclocked or modified anything.

One difference I can see is that my Raspberry Pi is not plugged into a monitor. But I think this should make no difference.

20

Hello,

I know the reason. The signal quality of the PI4’s mipi interface is relatively poor, the data error is very serious, and the frame loss phenomenon is serious.

So, on PI4, we can’t use the 12 fps configure mode .