Yes, I just added a mode to the Nvidia driver.
We are aware of this problem. As far as we know, this problem is caused by the high MIPI rate of the maximum resolution. You can try other resolutions.
I seem to get the same behavior when trying the other 2 modes:
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=2 --stream-mmap --stream-count=100
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
Width/Height : 1920/1080
Pixel Format : 'RG10'
Field : None
Bytes per Line : 3840
Size Image : 4147200
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
**It hangs here**
Another known issue is that you cannot use v4l2 to access the camera after running nvarguscamerasrc.
This should be the same behavior as the official driver. . Because I did not modify the official driver code, just simply add a mode.
Ok, good to know. Thanks!
Hey Wong, it looks like Iām having trouble with v4l2 again on the stock Nvidia drivers after a fresh reboot. Running either:
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1 --stream-mmap --stream-count=100
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0 --stream-mmap --stream-count=100
Results in an infinite hang with dmesg outputting:
...
[ 31.857862] vdd-fan: disabling
[ 31.857867] vdd-usb-vbus: disabling
[ 31.857869] vdd-usb-vbus2: disabling
[ 31.857880] vddio-sdmmc3-ap: disabling
[ 31.857950] avdd-io-edp-1v05: disabling
[ 31.857953] vdd-usb-hub-en: disabling
[ 191.395885] video4linux video0: frame start syncpt timeout!0
[ 191.603901] video4linux video0: frame start syncpt timeout!0
[ 191.811912] video4linux video0: frame start syncpt timeout!0
[ 192.019926] video4linux video0: frame start syncpt timeout!0
[ 192.227941] video4linux video0: frame start syncpt timeout!0
Iāve ensured that nvarguscamerasrc hasnāt been accessed before running those commands, but it still hangs (as you can see in the dmesg output which is from a fresh reboot). Any ideas on whatās going on? Thanks!
FYI, I have posted about this particular issue (likely unrelated to anything Arducam) here - Nano Dev Kit B01 + Raspi HQ cam doesn't work with v4l2-ctl (JetPack 4.6) - #7 by dustinkerstein - Jetson Nano - NVIDIA Developer Forums
Hey! Would you be able to share the driver patch source? Iād like to try adding in a test sensor mode with a lower frame rate. Thanks!
Can you send an email with a link to this post to [email protected]?
It is best to attach the required L4T32.6.1 version and the required driver version (e.g. IMX477 or Arducam)
An update after some more testing. Also, the reason I am looking to use v4l2 is that I was told by Nvidia developers that I would be able to achieve arbitrary long exposure captures if I bypassed Argus and the Nvidia Core Camera stack by using v4l2 (possibly with some additional driver tweaks). However, it looks like I am seeing a few issues which prevent me from further testing raw bayer long exposures. Note that Iām exclusively testing on a Nano 2GB Dev Kit.
- With Arducam drivers installed via the latest install_full.sh script I am unable to get v4l2-ctl streaming working when sensor_mode == 0 (4032x3040). The other two lower resolution modes work correctly. Dmesg output looks like: vi 54080000.vi: tegra_channel_error_status:error 20022 frame 68 when running the commands below. Note that the error happens even when frame_rate==10, likely due to issue #2. I guess itās possible that 4032x3040 raw bayer via v4l2 is too much bandwidth, though Iām not sure how Gstreamer is able to handle that same sensor mode with no issues. The fact that Gstreamer can handle 4032x3040 at 30fps (and v4l2 canāt) doesnāt seem to make much sense to me as the MIPI CSI transfer occurs before the ISP procesing.
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0,**frame_rate=30** --stream-mmap --stream-count=100
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0,**frame_rate=10** --stream-mmap --stream-count=100
- Regardless of using the Nvidia / Arducam driver, v4l2-ctl does not seem to respect set frame_rate control. The commands below result in 60fps and 30fps output, respectfully.
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1,**frame_rate=10** --stream-mmap --stream-count=100
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=2,**frame_rate=10** --stream-mmap --stream-count=100
- Interestingly, Gstreamer does respect the user-defined frame rate running the following command (and this applies across all sensor modes, from their device tree specified min and max framerates):
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=1 ispdigitalgainrange='1 1' num-buffers=300 wbmode=3 aelock=true gainrange='1 1' exposuretimerange='300000000 300000000' ! 'video/x-raw(memory:NVMM),width=3840,height=2160,**framerate=10/1**' ! nvvidconv ! nvv4l2h265enc bitrate=100000000 ! h265parse ! mp4mux ! filesink location=out.mp4
- v4l2 commands sometimes break Gstreamer until the next reboot, and vice versa. Though this doesnāt always happenā¦ Gstreamer seems to break v4l2 more than the other way around. I havenāt isolated whether this behavior is driver dependent. When v4l2 is broken, it will throw this error when running the commands from above and throw this error: video4linux video0: frame start syncpt timeout!0 Note there is a potentially related issue where v4l2 doesnāt ever work, even on the stock Nvidia drivers. It seems to vary from board to board (or camera to camera) and has yet to be isolated. See here for more details.
As per the possibility of enabling trigger mode capture (with long exposures up to sensor max spec), do you think itās possible Arducam would be interested in helping enable that at the driver level? Iād be happy to put up some money as a bountyā¦
Thanks again,
Dustin
My guess is that his speed is too fast. Jetson Nanoās CSI interface is D-PHY 1.1, 1.5Gbps per lane, 4032x3040@30 has exceeded this speed.
Yes, I was weird at first, but I think a different pipeline is used when using ISP. maybe you can confirm this with Nvidia?
This may be caused by incorrect order or timing of setting control. It is effective if you enable two terminals to execute frame rate control separately.
By the way, when using 4032x3040, it does not work even if the frame rate is set separately.
Maybe you can try to increase the line_length of 4032x3040 mode,
line length resigter:
{0x0342, 0x23},
{0x0343, 0x40},
I am not very clear about the reason for this. I think maybe Nvidiaās pipeline has not handled it properly.
In fact, we have tried the master-slave mode, but we found that it is impossible to use the MCU to trigger the slave camera. Only the output signal of the master mode camera can trigger the slave camera. We did not go deep into it. If you want to customize it, you You can contact [email protected] and they will handle the customization request.
Hi @dustobub
Good news, I took the time to test it,
By adjusting the timing, the 4032x3040 can work at 15fps.
Thatās interesting. Iāll report that behavior on their forums. Hopefully itās something they can fix.
Iāll include this behavior in the same bug report to Nvidia as the post above.
Ok, trigger mode would be ideal, but do you think itās possible to get long exposures working with streaming mode? The Raspberry Pi implementation of the IMX477 allows the full use of the IMX477ās specified max exposure. Though I do believe itās in streaming mode because people talk about the long delays - ie. if you just miss the frame you might need to wait up 2*exposure time to get a capture, which for long exposures can be a long wait (and also means youāre not starting the capture at the right time to catch what youāre looking to capture. Maybe thereās a way to smartly use streaming mode without having to restart the entire app/pipeline:
- Enable streaming with fast exposure (discard all frames)
- When ready to start long exposure, set timing, and capture the immediate next frame
- Immediately after long exposure frame (or maybe can be queuedā¦) switch back to fast exposure (and discarding frames)
- Repeatā¦
That would avoid the fixed cost time penalty of having to start the pipeline for each long exposure, and the risk of missing a frame and having to wait for the next (long) frame. If thatās doable, itās almost as good as a true trigger capture mode. Does that sound feasible with v4l2?
Awesome! Iāll have some time to test later today. Does this include an extra mode in addition to the 4032x3040 at 30fps, or does it replace that mode? It would be nice to have both in my mind, since 30fps does work perfectly fine in Gstreamer (which I also need for my use-case when capturing video), and then when working with longer exposures, switching down to a lower FPS would allow bayer capture for stills. Also, did you happen to have a chance to test 24fps? Thanks again!
Yes, I replaced the original mode, 15fps is the best speed I got, in fact I am not good at configuring camera registersā¦ and we donāt know the specific limits.
This seems feasible in theory, but specific tests are needed to find out whether there are limitations. But this does not seem to be a simple matter.
1 Like
Ok cool. 15fps should be ok for my usecase when I need raw Bayer. Would you be able to compile the driver / patch to have two 4032x3040 modes? That way gstreamer can continue to use 30fps (however that works). Thanks!