Audio support libcamera-vid libav for streaming with hawkeye

Hi everyone,

I am new here and have a big question:
I want to stream video + audio via RTMP from my raspi 4b and so far I did this via “libcamera-vid --codec libav --libav-audio…” with my HQ camera.
Now I would like to get there with the arducam 64mp.
Unfortunately libav is not included?

Is there any way to get libav-audio working with the “Hawkeye” to add synchronized audio?
If this is even impossible (I am not fully aware of how the magic happens, v4l2, libcamera, libav, ffmpeg etc…) is there a better/alternative way streaming video+audio via RTMP?

Many thanks in advance and best regards
Daniel

I have IMX462 low light camera from arducam and Adafruit I2S MEMS Microphone Breakout.
I want to capture video and audio using libcamera-vid and split the feed into files for drive recorder.
Audio is one of very important evidence when accident happens.
So I desperately need audio recording function to libcamera-vid.
I already tried gstreamer but they require large CPU consumption so I need to ise libcamera-vid for my purpose.
Please include this audio recording function (libav support?)

@p.denton

On the Raspberry Pi you need to use the software within their framework,libcamera-vid is a perfectly supported solution

Raspberry Pi now uses media-contoller instead of traditional video0 node

You can refer to the link below:

https://forums.raspberrypi.com/viewtopic.php?t=322076

Thank you for the heads up.
I didn’t know the media controller thing, so I will look into that.

Does the libcamera-vid app provided in pre-compiled binary support libav option?
I use RasbberyPi Zero 2 W, IMX462 from arducam and followed instructions:

I already tried libcamera-vid working OK but I can’t see the option to designate libav documented in :

here is the info:

[email protected]:~ $ uname -a
Linux raspberrypi 5.15.32-v7+ #1538 SMP Thu Mar 31 19:38:48 BST 2022 armv7l GNU/Linux
[email protected]:~ $ libcamera-vid --version
libcamera-apps build: f047a5b01578-intree 12-07-2022 (03:13:06)
libcamera build: v0.0.0+3730-67300b62
[email protected]:~ $ libcamera-vid -h
Valid options are:
  -h [ --help ] [=arg(=1)] (=0)         Print this help message
  --version [=arg(=1)] (=0)             Displays the build version number
  --list-cameras [=arg(=1)] (=0)        Lists the available cameras attached to the system.
  --camera arg (=0)                     Chooses the camera to use. To list the available indexes, use the
                                        --list-cameras option.
  -v [ --verbose ] [=arg(=1)] (=0)      Output extra debug and diagnostics
  -c [ --config ] [=arg(=config.txt)]   Read the options from a file. If no filename is specified, default to
                                        config.txt. In case of duplicate options, the ones provided on the command line
                                        will be used. Note that the config file must only contain the long form
                                        options.
  --info-text arg (=#%frame (%fps fps) exp %exp ag %ag dg %dg)
                                        Sets the information string on the titlebar. Available values:
                                        %frame (frame number)
                                        %fps (framerate)
                                        %exp (shutter speed)
                                        %ag (analogue gain)
                                        %dg (digital gain)
                                        %rg (red colour gain)
                                        %bg (blue colour gain)
                                        %focus (focus FoM value)
                                        %aelock (AE locked status)
  --width arg (=0)                      Set the output image width (0 = use default value)
  --height arg (=0)                     Set the output image height (0 = use default value)
  -t [ --timeout ] arg (=5000)          Time (in ms) for which program runs
  -o [ --output ] arg                   Set the output file name
  --post-process-file arg               Set the file name for configuring the post-processing
  --rawfull [=arg(=1)] (=0)             Force use of full resolution raw frames
  -n [ --nopreview ] [=arg(=1)] (=0)    Do not show a preview window
  -p [ --preview ] arg (=0,0,0,0)       Set the preview window dimensions, given as x,y,width,height e.g. 0,0,640,480
  -f [ --fullscreen ] [=arg(=1)] (=0)   Use a fullscreen preview window
  --qt-preview [=arg(=1)] (=0)          Use Qt-based preview window (WARNING: causes heavy CPU load, fullscreen not
                                        supported)
  --hflip [=arg(=1)] (=0)               Request a horizontal flip transform
  --vflip [=arg(=1)] (=0)               Request a vertical flip transform
  --rotation arg (=0)                   Request an image rotation, 0 or 180
  --roi arg (=0,0,0,0)                  Set region of interest (digital zoom) e.g. 0.25,0.25,0.5,0.5
  --shutter arg (=0)                    Set a fixed shutter speed in microseconds
  --analoggain arg (=0)                 Set a fixed gain value (synonym for 'gain' option)
  --gain arg                            Set a fixed gain value
  --metering arg (=centre)              Set the metering mode (centre, spot, average, custom)
  --exposure arg (=normal)              Set the exposure mode (normal, sport)
  --ev arg (=0)                         Set the EV exposure compensation, where 0 = no change
  --awb arg (=auto)                     Set the AWB mode (auto, incandescent, tungsten, fluorescent, indoor, daylight,
                                        cloudy, custom)
  --awbgains arg (=0,0)                 Set explict red and blue gains (disable the automatic AWB algorithm)
  --flush [=arg(=1)] (=0)               Flush output data as soon as possible
  --wrap arg (=0)                       When writing multiple output files, reset the counter when it reaches this
                                        number
  --brightness arg (=0)                 Adjust the brightness of the output images, in the range -1.0 to 1.0
  --contrast arg (=1)                   Adjust the contrast of the output image, where 1.0 = normal contrast
  --saturation arg (=1)                 Adjust the colour saturation of the output, where 1.0 = normal and 0.0 =
                                        greyscale
  --sharpness arg (=1)                  Adjust the sharpness of the output image, where 1.0 = normal sharpening
  --framerate arg (=30)                 Set the fixed framerate for preview and video modes
  --denoise arg (=auto)                 Sets the Denoise operating mode: auto, off, cdn_off, cdn_fast, cdn_hq
  --viewfinder-width arg (=0)           Width of viewfinder frames from the camera (distinct from the preview window
                                        size
  --viewfinder-height arg (=0)          Height of viewfinder frames from the camera (distinct from the preview window
                                        size)
  --tuning-file arg (=-)                Name of camera tuning file to use, omit this option for libcamera default
                                        behaviour
  --lores-width arg (=0)                Width of low resolution frames (use 0 to omit low resolution stream
  --lores-height arg (=0)               Height of low resolution frames (use 0 to omit low resolution stream
  --mode arg                            Camera mode as W:H:bit-depth:packing, where packing is P (packed) or U
                                        (unpacked)
  --viewfinder-mode arg                 Camera mode for preview as W:H:bit-depth:packing, where packing is P (packed)
                                        or U (unpacked)
  --autofocus [=arg(=1)] (=0)           Trigger Autofocus once.
  --continue-autofocus [=arg(=1)] (=0)  Enable continue autofocus.
  -b [ --bitrate ] arg (=0)             Set the video bitrate for encoding, in bits/second (h264 only)
  --profile arg                         Set the encoding profile (h264 only)
  --level arg                           Set the encoding level (h264 only)
  -g [ --intra ] arg (=0)               Set the intra frame period (h264 only)
  --inline [=arg(=1)] (=0)              Force PPS/SPS header with every I frame (h264 only)
  --codec arg (=h264)                   Set the codec to use, either h264, mjpeg or yuv420
  --save-pts arg                        Save a timestamp file with this name
  -q [ --quality ] arg (=50)            Set the MJPEG quality parameter (mjpeg only)
  -l [ --listen ] [=arg(=1)] (=0)       Listen for an incoming client network connection before sending data to the
                                        client
  -k [ --keypress ] [=arg(=1)] (=0)     Pause or resume video recording when ENTER pressed
  -s [ --signal ] [=arg(=1)] (=0)       Pause or resume video recording when signal received
  -i [ --initial ] arg (=record)        Use 'pause' to pause the recording at startup, otherwise 'record' (the default)
  --split [=arg(=1)] (=0)               Create a new output file every time recording is paused and then resumed
  --segment arg (=0)                    Break the recording into files of approximately this many milliseconds
  --circular [=arg(=4)] (=0)            Write output to a circular buffer of the given size (in MB) which is saved on
                                        exit
  --frames arg (=0)                     Run for the exact number of frames specified. This will override any timeout
                                        set.

If my understanding is right, the pre-compiled libcamera-vid doesn’t support audio.
libcamera-vid is easy to use and is low CPU consumption so I prefer libcamera-vid solution.
Is there any other option for capture video and audio at same time with low CPU consumption?
( or libcamera-vid pre-compiled binary to support audio?)
I think the original author of this thread says basically the same.

@p.denton

I did find this problem after testing.
It shouldn’t be, I’ll fix this and update libcamera.

1 Like

Thank you for reply.
I am waiting for the release.

@p.denton

We have updated and published,please update.

./install_pivariety_pkgs.sh -p libcamera_apps
2 Likes

Thank you for quick release!

$ ./install_pivariety_pkgs.sh -p libcamera_dev
$ ./install_pivariety_pkgs.sh -p libcamera_apps

There is error (missing ffmpeg)

$ libcamera-vid --help
libcamera-vid: error while loading shared libraries: libavdevice.so.58: cannot open shared object file: No such file or directory

install dependency:

$ sudo apt install ffmpeg
$ sudo apt install pulseaudio-utils
$ sudo apt-get install pulseaudio
$ pulseaudio --start
$ pactl list | grep -A2 'Source #' | grep 'Name: '
Name: alsa_output.platform-asoc-simple-card.0.stereo-fallback.monitor
Name: alsa_input.platform-asoc-simple-card.0.stereo-fallback
$ libcamera-vid -t 10000 --inline -n --codec libav --libav-audio  -o sample_with_audio.mp4

Ended up no sound mp4 video.

But I realized ffmpeg backend consumes more CPU(40%) than gstreamer(20%).
So I stick to video only.(CPU load under 5%).
Anyway thank you for the release.

@p.denton

Regarding the problem of no sound, I will test it again.

@p.denton

I tested the audio and found it to function perfectly, both the sound and the video appeared.
Note that I plugged in the mic via usb.

In the video, I tested it with blowing up. Forgive me, I didn’t add any scenes, because in the company

Thank you for testing.
My environment might cause the problem.
I installed pulseaudio and ALSA recording so they might conflict each other.
I already gave up using libav background due to the high CPU load.
I separated recoding video and audio capture, libcamera-vid and arecord respectively.

Again, thank you for quick release.