IMX477 driver not installing correctly on R32.6.1

  1. Where did you get the camera module(s)?
    Amazon

  2. Model number of the product(s)?
    B0279

  3. What hardware/platform were you working on?
    Jetson Nano 2GB

  4. Instructions you have followed. (link/manual/etc.)
    IMX477 - How to install the Driver - Arducam

  5. Problems you were having?
    Driver isn’t loading

  6. The dmesg log from your hardware?

[email protected]:~$ dmesg | grep imx477
[    1.245244] imx477 6-001a: tegracam sensor driver:imx477_v2.0.6
[    1.705494] vi 54080000.vi: subdev imx477 6-001a bound
  1. Troubleshooting attempts you’ve made?
    See issue report here - https://github.com/ArduCAM/MIPI_Camera/issues/132

  2. What help do you need?
    Help figuring out what is wrong and how to fix.

Please see my original post linked in #7 above.

[email protected]:~$ cat /etc/nv_tegra_release
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021

Any ideas on how to fix? Happy to provide more info/debug.

It seems that the driver has been loaded correctly.
What is your specific problem? Does the display command not work?

How can you determine the Arducam driver is loaded from that message? With the Arducam driver I would expect to see 4032x3040 as an available sensor mode for use in gstreamer, but I do not. See here for some more notes - https://github.com/ArduCAM/MIPI_Camera/issues/132

This output indicates that the driver has successfully probed.

In fact, for the latest L4T-R32.6.1 we are using the official driver provided by Nvidia:

From what I’ve read on this forum and elsewhere, it appears the Arducam driver used to provide access to other resolution modes. Our use case depends on having access to the full 4:3 sensor readout of 4032x3040 at 30fps. Has that support been deprecated? From the wording on that Arducam page, it sounded like the Nvidia driver works but is limited to the two modes, whereas the Arducam driver was in continued development with the aim of providing additional modes. Is that not correct? Are you saying that the current Arducam driver is 100% functionally the same as the Nvidia one?

On previous versions of the driver (and older L4T) the sensor modes available looked like:

v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 4032x3040
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 2592x1944
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 2560x1440
			Interval: Discrete 0.025s (40.000 fps)

Yes, we will continue to develop, but the two drivers are the same at present, and we have not received any requests from users before, so we think maybe the current mode is enough.
At the moment, what you mainly want is the maximum resolution, right? Maybe we can finish it this week.

Thanks for the info. Yes, having the full 4:3 sensor resolution mode @ 30fps is very important for us, and I imagine other users as well (who are maybe just currently using older L4T / drivers). Please let me know if there’s anything I can do. Having that additional mode this week / next would be extremely helpful. Thanks!

Hi @dustobub

I haven’t officially released it yet, but you can try it first.
arducam-nvidia-l4t-kernel-t210-4.9.253-tegra-32.6.1-20210915100958_arm64_imx477.deb

Nice! The full resolution capture looks great. This is exactly what I need. Thanks!

One side note is that when running gst-launch-1.0 you need to specify sensor-mode=0|1|2 corresponding to the output of v4l2-ctl --list-formats-ext or else the output is just scaled/stretched from the default sensor mode, rather than correctly using a different window of the sensor (ie. 16x9 vs. 4x3). As long as the correct sensor-mode is used (0 in my case with this latest driver), all looks perfect. Thanks again!

On a somewhat related note, when capturing at the full sensor resolution, I am dropping/duping 2-3 frames when I run this command:

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 num-buffers=240 ! 'video/x-raw(memory:NVMM),width=4032,height=3040,framerate=30/1' ! nvvidconv flip-method=2 ! nvv4l2h264enc bitrate=100000000 ! h264parse ! mp4mux ! filesink location=~/out.mp4

Do you happen to know of any ways to minimize the chance of these drops? I’ve run sudo jetson_clocks but that doesn’t seem to have any impact. Are there any gstreamer buffers / settings that can be modified from the command-line that could help? Thanks again.

Sorry, I don’t know why the frame is dropped here, maybe it is caused by the insufficient processing speed of a certain component.
Perhaps it is the H264 encoding component?

It looks like the frames are dropping/duping on the earliest stages of the pipeline. I tested this by removing all encoding / parsing / muxing, filming a stopwatch, and then extracting individual frames and looking for dupes / dropped frames.

Here’s an example that outputs to raw YUV and then extracts the frames:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 num-buffers=30 ! 'video/x-raw(memory:NVMM),width=4032, height=3040, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw,format=I420' ! filesink location=/dev/shm/dump.yuv
ffmpeg -video_size 4032x3040 -r 30 -pixel_format yuv420p -i dump.yuv output-%d.jpg

My question is, if I set framerate=24/1, will that actually force the sensor to produce frames at 24fps, or will it still be producing frames at the defined fps per the sensor mode (ie. 30fps) and then skipping frames to achieve 24fps?

If it is skipping frames to achieve 24fps (or any arbitrary fps), would it be possible to create a new sensor mode at a lower fps to avoid dropped / duped frames?

Thanks again!

PS - Here’s a Dropbox Link that contains the output files from the above YUV output test. You can see that frame 15 and 16 show the same time on the stopwatch, and are dupes.

If you are using yuv, you may encounter SD Card write performance issues. .

It will set the sensor to output 24fps

For the YUV test, I was outputting to /dev/shm/ which is the ramdisk, so there shouldn’t be any throughput concerns. I’ll do a little more testing and get back to you. Thanks again.

Will this happen if you use other resolutions (such as 4k)?

In fact, I don’t know much about the specific performance parameters of Nvidia ISP.

Yah, it could be a limitation in the ISP. Lowering the resolution (and still using sensor-mode=0 to use the full 4:3 aspect ratio) does seem to reduce these frame drops/dupes. Using jetson_clocks also helps, and essentially makes the drops/dupes a non issue (the only drops are at the very beginning of the capture). Will this driver become an official release for Arducam? I hope so. Thanks again!

Thank you for the information,
I have released this driver, you can use the installation script to install it.

Thanks!

Another thing I just noticed with the new Arducam driver is that I can’t get v4l2-ctl working. When I run this command:

v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100

It hangs indefinitely. When uninstalling the Arducam driver and reverting to the Nvidia driver, it correctly outputs frames. It seems like it is behaving similarly to what is described here - Nano 2GB + Raspi HQ Cam nvarguscamerasrc working but not v4l2-ctl - #6 by DicksonChow - Jetson Nano - NVIDIA Developer Forums

Is the latest Arducam driver based on the latest Nvidia driver? Or is it possible that it’s based on an earlier version that had this same v4l2-ctl issue? Thanks again.