IMX477 driver not installing correctly on R32.6.1

Can you send an email with a link to this post to [email protected]?
It is best to attach the required L4T32.6.1 version and the required driver version (e.g. IMX477 or Arducam)

An update after some more testing. Also, the reason I am looking to use v4l2 is that I was told by Nvidia developers that I would be able to achieve arbitrary long exposure captures if I bypassed Argus and the Nvidia Core Camera stack by using v4l2 (possibly with some additional driver tweaks). However, it looks like I am seeing a few issues which prevent me from further testing raw bayer long exposures. Note that I’m exclusively testing on a Nano 2GB Dev Kit.

  1. With Arducam drivers installed via the latest install_full.sh script I am unable to get v4l2-ctl streaming working when sensor_mode == 0 (4032x3040). The other two lower resolution modes work correctly. Dmesg output looks like: vi 54080000.vi: tegra_channel_error_status:error 20022 frame 68 when running the commands below. Note that the error happens even when frame_rate==10, likely due to issue #2. I guess it’s possible that 4032x3040 raw bayer via v4l2 is too much bandwidth, though I’m not sure how Gstreamer is able to handle that same sensor mode with no issues. The fact that Gstreamer can handle 4032x3040 at 30fps (and v4l2 can’t) doesn’t seem to make much sense to me as the MIPI CSI transfer occurs before the ISP procesing.
 v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0,**frame_rate=30** --stream-mmap --stream-count=100
 v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0,**frame_rate=10** --stream-mmap --stream-count=100
  1. Regardless of using the Nvidia / Arducam driver, v4l2-ctl does not seem to respect set frame_rate control. The commands below result in 60fps and 30fps output, respectfully.
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1,**frame_rate=10** --stream-mmap --stream-count=100
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=2,**frame_rate=10** --stream-mmap --stream-count=100
  1. Interestingly, Gstreamer does respect the user-defined frame rate running the following command (and this applies across all sensor modes, from their device tree specified min and max framerates):
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=1 ispdigitalgainrange='1 1' num-buffers=300 wbmode=3 aelock=true gainrange='1 1' exposuretimerange='300000000 300000000' ! 'video/x-raw(memory:NVMM),width=3840,height=2160,**framerate=10/1**' ! nvvidconv ! nvv4l2h265enc bitrate=100000000 ! h265parse ! mp4mux ! filesink location=out.mp4
  1. v4l2 commands sometimes break Gstreamer until the next reboot, and vice versa. Though this doesn’t always happen… Gstreamer seems to break v4l2 more than the other way around. I haven’t isolated whether this behavior is driver dependent. When v4l2 is broken, it will throw this error when running the commands from above and throw this error: video4linux video0: frame start syncpt timeout!0 Note there is a potentially related issue where v4l2 doesn’t ever work, even on the stock Nvidia drivers. It seems to vary from board to board (or camera to camera) and has yet to be isolated. See here for more details.

As per the possibility of enabling trigger mode capture (with long exposures up to sensor max spec), do you think it’s possible Arducam would be interested in helping enable that at the driver level? I’d be happy to put up some money as a bounty…

Thanks again,

Dustin

My guess is that his speed is too fast. Jetson Nano’s CSI interface is D-PHY 1.1, 1.5Gbps per lane, 4032x3040@30 has exceeded this speed.

Yes, I was weird at first, but I think a different pipeline is used when using ISP. maybe you can confirm this with Nvidia?

This may be caused by incorrect order or timing of setting control. It is effective if you enable two terminals to execute frame rate control separately.

By the way, when using 4032x3040, it does not work even if the frame rate is set separately.
Maybe you can try to increase the line_length of 4032x3040 mode,
line length resigter:

{0x0342, 0x23},
{0x0343, 0x40},

I am not very clear about the reason for this. I think maybe Nvidia’s pipeline has not handled it properly.

In fact, we have tried the master-slave mode, but we found that it is impossible to use the MCU to trigger the slave camera. Only the output signal of the master mode camera can trigger the slave camera. We did not go deep into it. If you want to customize it, you You can contact [email protected] and they will handle the customization request.

Hi @dustobub

Good news, I took the time to test it,
By adjusting the timing, the 4032x3040 can work at 15fps.

https://github.com/ArduCAM/MIPI_Camera/releases/download/v0.0.1/arducam-nvidia-l4t-kernel-t210-4.9.253-tegra-32.6.1-20211009074442_arm64_imx477-v4l2.deb

That’s interesting. I’ll report that behavior on their forums. Hopefully it’s something they can fix.

I’ll include this behavior in the same bug report to Nvidia as the post above.

Ok, trigger mode would be ideal, but do you think it’s possible to get long exposures working with streaming mode? The Raspberry Pi implementation of the IMX477 allows the full use of the IMX477’s specified max exposure. Though I do believe it’s in streaming mode because people talk about the long delays - ie. if you just miss the frame you might need to wait up 2*exposure time to get a capture, which for long exposures can be a long wait (and also means you’re not starting the capture at the right time to catch what you’re looking to capture. Maybe there’s a way to smartly use streaming mode without having to restart the entire app/pipeline:

  1. Enable streaming with fast exposure (discard all frames)
  2. When ready to start long exposure, set timing, and capture the immediate next frame
  3. Immediately after long exposure frame (or maybe can be queued…) switch back to fast exposure (and discarding frames)
  4. Repeat…

That would avoid the fixed cost time penalty of having to start the pipeline for each long exposure, and the risk of missing a frame and having to wait for the next (long) frame. If that’s doable, it’s almost as good as a true trigger capture mode. Does that sound feasible with v4l2?

Awesome! I’ll have some time to test later today. Does this include an extra mode in addition to the 4032x3040 at 30fps, or does it replace that mode? It would be nice to have both in my mind, since 30fps does work perfectly fine in Gstreamer (which I also need for my use-case when capturing video), and then when working with longer exposures, switching down to a lower FPS would allow bayer capture for stills. Also, did you happen to have a chance to test 24fps? Thanks again!

Yes, I replaced the original mode, 15fps is the best speed I got, in fact I am not good at configuring camera registers… and we don’t know the specific limits.

This seems feasible in theory, but specific tests are needed to find out whether there are limitations. But this does not seem to be a simple matter.

1 Like

Ok cool. 15fps should be ok for my usecase when I need raw Bayer. Would you be able to compile the driver / patch to have two 4032x3040 modes? That way gstreamer can continue to use 30fps (however that works). Thanks!

I’m not sure, I will take the time to test it.

Quick note, with your latest 15fps driver, I am able to modify the 4032x3040 frame-rate using this separate shell technique down to 5fps. In fact, with all of the modes, I am unable to get below 5fps with v4l2-ctl even though the driver and DTB should allow down to 2fps. I noted this behavior in the Nvidia post I just created - V4l2-ctl not respecting frame_rate control, unlike Gstreamer on IMX477 + Nano 2GB - Jetson Nano - NVIDIA Developer Forums

Also, I think that if v4l2-ctl were to correctly respect the frame_rate setting on initialization, then it wouldn’t be necessary to have two 4032x3040 modes (one at 30fps and one at 15fps). I think since it immediately tries 4032x3040 at 30fps (which it can’t handle for some reason), it doesn’t recover when setting the frame_rate from another shell. But if it were to have initialized correctly at 15fps, then it would work fine. Let’s see what the Nvidia folks say about the frame_rate control in v4l2 and go from there. Thanks for your help.

It’s not correct. I didn’t simply adjust the frame rate in 15fps mode. At the same time, I also adjusted some MIPI timing related registers. This effect cannot be achieved only through frame_rate control.

Got it. So maybe we will need two modes (4032x3040 at 30fps for Argus/Gstreamer and 4032x3040 at 15fps for v4l2 enabling raw bayer capture). Please see this thread on the Nvidia forum - V4l2-ctl not respecting frame_rate control, unlike Gstreamer on IMX477 + Nano 2GB - #5 by ShaneCCC - Jetson Nano - NVIDIA Developer Forums - They are interested in merging in the 4032x3040 support. Would you be able to share the patch with them?

In fact, they have a 4032x3040 configuration, we also use the patch provided by RidgeRun, please check:

Ok I will make sure to ask them about why they removed that mode. What are your thoughts on having both a 30fps and 15fps mode in the official Arducam drivers for 4032x3040?

Currently we have no plans to add it in the release version, which may confuse users.
If you want them to coexist, you can try to modify it yourself by referring to the patch I provided.