USB2.0_UC-391_Rev.E + DVP/MT9J003 - Snapshot with 4 cameras stalls/stucks

Hi,

Running the USB2.0_UC-391_Rev.E and the Arducam MT9J003 I have issues to “snapshot” 4 Cameras simultaneously via soft triggers.

I create a thread for each camera and run this in each thread:
call ArducamSDK.Py_ArduCam_softTrigger(self.handle)
check ArducamSDK.Py_ArduCam_isFrameReady(self.handle)
fetch ArducamSDK.Py_ArduCam_getSingleFrame(self.handle, 1000)

When trying that with 4 cameras I see “getSingleFrame” stucking for the 5th of them for a minute (isFrameReady() is OK for all 4 cameras). After around 60 seconds the function returns a black image instead of a actual camera snapshot. The other 3 cameras return a “good” image (dark, blurry… but this is a question of adjusting sensor config).

When using a Mutex (threading) to only call a single “getSingleFrame” simultaneously, it works as it should. But then of course snapshotting for all 4 cameras takes up to 2.5 seconds here… which is not tolerateable (and usb3 can transfer more than 4 HD RAW-datas per second).

Might it be that the API is there not threadsafe (and hence needs this mutex) ?

Sample code triggered from

    def capture_single_image(self):
        global use_arducam_api_mutex, arducam_api_mutex

        #trigger the external trigger via software
        ArducamSDK.Py_ArduCam_softTrigger(self.handle)

        missed_time = 0
        while ArducamSDK.Py_ArduCam_isFrameReady(self.handle) != 1:
            #print("Camera #%i: Py_ArduCam_isFrameReady ... waiting." % (self.idx))
            time.sleep(0.005)
            missed_time += 0.005
        
        #if missed_time > 0:
        #    print("Camera #%i: Waited %f seconds for isFrameReady()." % (self.idx, missed_time))

        if use_arducam_api_mutex:
            arducam_api_mutex.acquire()

        rtn_val,data,rtn_cfg = ArducamSDK.Py_ArduCam_getSingleFrame(self.handle, 1000)
        if rtn_val != 0:
            print("Camera #%i: Error capturing single image, rtn_val = %i" % (self.idx, rtn_val))
            self.running = False

        if use_arducam_api_mutex:
            arducam_api_mutex.release()

        datasize = rtn_cfg['u32Size']
        if datasize == 0:
            print("Camera #%i: Data length zero" % (self.idx))
            return

        self.img = convert_image(data,rtn_cfg, self.color_mode)
        self.captures_done += 1


    def capture_single_image_thread_func(self):
        while self.running == True:
...
                print("Camera #%i: Received command to capture image." % (self.idx))
                self.capture_single_image()      

Chosen config is “ArduCAM_USB_Camera_Shield-master/Config/USB2.0_UC-391_Rev.E/DVP/MT9J003/MT9J003_RAW_8b_1920x1080_20fps.cfg”

Edit:
I tried to attach them to multiple USB-Slots instead of using a single USB3.0-Hub … while it worked in the first loop and try for 4 cameras, further loops (new snapshots) failed with the above problem (3 cameras snapshotted, the 4th timeout/black image after estimated 60 seconds). Further executions of the python script lead to the same issue right on start (3 working, 1 failed snapshot).

Can you please check if the corresponding API call is thoroughfully threadsafe?

Ideas and Thoughts?

Thanks in Advance,
Ron

Hi, @GWRon

USB2.0_UC-391_Rev.E It is USB2.0, even if it is plugged into USB3.0, it still recognizes USB2.0. This may be the cause of the failed image capture.

Hi @yang

Thanks for your swift reply.

it (the board) is USB2.0 - but 4 of these boards are connected to a USB3 hub. Means all of the 4 boards can use their “full speed” to communicate with the host.

Yet I have issues to stream 4 of them (data length errors) - or to “snapshot” 4 of them via soft trigger (should result in way less “data” on the lane “per second”).

I have another thought:

If my usb hub was only 2.0 - or if I connected 4 boards to 4 different usb slots (maybe even only 2.0 too) on the jetson …
shouldn’t it then flawlessly be able to “snapshot” a single image from each of the boards simultaneously? The image data is buffered in the board until I fetch it - is this correct?
So if I request the image/frame from the board, it should not matter how “occupied” the data lane is - it would just transfer it “slower”.

Yet it does not transfer “slower”, but “broken” (black image - after 60 seconds waiting time, while the others only took some hundred milliseconds in total).

You can use one camera for individual testing. Check to see if the camera is normal.

All four cameras work - when using them “alone”.

The “not responding camera” is not the same each time. It is as if the API does not get the response within a certain time and somehow “stucks” then.

As stated I use ArducamSDK.Py_ArduCam_getSingleFrame(self.handle, 1000) means I pass a timeout of 1000 ms. Yet it takes dozens of seconds to “timeout”.

Regarding USB3.0: I use the “Icybox 4-port hub with USB 3.0 Type-A interface” (IB-HUB1409-U3) … which somehow is only detected as USB2.0 via my linux boxes (and so also the Nvidia Jetson). Yet I just wrote to it/read from it via an attached usb3 stick and write speed is the same as directly attached speed (10MB write and 130MB/s read for that particular stick - which indicates USB3.0). So this means it might only communicate with 480 mbit/s. Yet this means all these 4 boards attached to it - should still be able to send a HD (raw) image within a second (each one has 120mbit if equally shared)

Since our device is USB2.0, the device descriptors reported to the host on the final device are USB2.0, and the USB2.0 protocol is used.

The bandwidth of USB2.0 is not enough to support four cameras.

It has nothing to do with your other USB3.0 devices.

To repeat (simply correct me if one of the points is invalid):

  • 1 usb hub (2.0 or 3.0 does not matter)
  • 4 arducam devices, each attached to one of the usb slots of the usb hub
  • the arducam devices are USB 2.0 (so are capable of up to 480 mbit transfer rate)
  • the 4 arducam devices will run on a kind of “usb 2.0” lane of the usb hub
  • so all 4 have to share the 480 mbit/s maximum

So far … no problem. It means I cannot “stream” with 25 FPS HD simultaneously from all 4 boards.

But I did not want to stream - I want to “single image” snapshot from 4 cameras (as much as possible) simultaneously. Requesting a frame (Py_ArduCam_softTrigger()) should inform all 4 boards+cams to “snapshot” and keep the image in their memory. In the next step I wait for Py_ArduCam_isFrameReady() (snapshot done?).
Finally I (try to) fetch the data via Py_ArduCam_getSingleFrame().
This last step should “simply said” just transfer data from “camera/board” to the host. There is no need for it to have a constant data flow, it can happen slower … or faster.
Means I should be able to request (in parallel) 4 times Py_ArduCam_getSingleFrame(handle[n]) with “handle[n]” iterating over the 4 camera handles. Yet this only works for 3 of 4 (or sometimes 4 of 4 … but rarely).

So - again - my question is: How can I simultaneously request “take an image” from the cameras, and then retrieve it - in parallel (so as fast as possible). As stated it works for 3 of 4 cameras (random order).
Why do I have to use a Mutex (for the last step Py_ArduCam_getSingleFrame()) to make it work (aka making it not parallel but processed one after another).

Thanks for your help @yang.

Hi GWRon,
I think you are right, our engineer has confirmed it should work normally. Would you like to attach us the whole code project? We can help you test it in detail.

Hi @lvbin

Thanks for your repyly. To avoid broken links I directly post the two files into this posting here.

Place both files inside “ArduCAM_USB_Camera_Shield-master/Nvidia_Jetson/Python/Streaming_demo” so download from the arducam shield repo at github, extract … and as said place the two files there.

MultiCam.py:

"""
Multi-Camera-Snapshot class
"""

import sys
import os
import time
from ImageConvert import convert_image
import arducam_config_parser
import ArducamSDK
import threading
import signal

arducam_api_mutex = threading.Lock()
use_arducam_api_mutex = False

class singleCam(threading.Thread):   #Class for the Multiple camera thread

    def __init__(self,idx):
        self.idx = idx
        self.handle = {}
        self.cfg = {}
        self.running = False
        self.save_flag = False
        self.save_raw = True
        self.color_mode = None
        self.cam_id = None
        self.img = None
        self.Width = None
        self.Height = None
        self.capture_mutex = threading.Lock()
        self.camera_name = "camera"
        self.captures_done = 0
        self.captures_todo = 0
        #threads
        self.capture_single_image_thread = None
        
    #Method for the USB shield configuration
    def configBoard(self,config):

        ArducamSDK.Py_ArduCam_setboardConfig(self.handle, config.params[0], \
            config.params[1], config.params[2], config.params[3], \
                config.params[4:config.params_length])
        pass
    

    #Method for the Initial setup of the camera
    def camera_initFromFile(self,fileName):
        
        config = arducam_config_parser.LoadConfigFile(fileName)
        camera_parameter = config.camera_param.getdict()
        self.Width = camera_parameter["WIDTH"]
        self.Height = camera_parameter["HEIGHT"]
        BitWidth = camera_parameter["BIT_WIDTH"]
        ByteLength = 1
        if BitWidth > 8 and BitWidth <= 16:
            ByteLength = 2
            self.save_raw = True
        FmtMode = camera_parameter["FORMAT"][0]
        self.color_mode = camera_parameter["FORMAT"][1]
        #print("color mode",self.color_mode)


        # Camera is operated in I2C mode
        I2CMode = camera_parameter["I2C_MODE"]
        I2cAddr = camera_parameter["I2C_ADDR"]
        TransLvl = camera_parameter["TRANS_LVL"]

        # Dictionary to store the configuration data
        self.cfg = {"u32CameraType":0x00,
                "u32Width":self.Width,
                "u32Height":self.Height,
                "usbType":0,
                "u8PixelBytes":ByteLength,
                "u16Vid":0,
                "u32Size":0,
                "u8PixelBits":BitWidth,
                "u32I2cAddr":I2cAddr,
                "emI2cMode":I2CMode,
                "emImageFmtMode":FmtMode,
                "u32TransLvl":TransLvl }


        # Open the available camera in the USB bus interface with the index
        ret,self.handle,rtn_cfg = ArducamSDK.Py_ArduCam_open(self.cfg, self.idx)
        #ret,handle,rtn_cfg = ArducamSDK.Py_ArduCam_autoopen(cfg)
        #print(rtn_cfg)

        #If opened successfully, send the configuration information to the USB board
        if ret == 0:
            #ArducamSDK.Py_ArduCam_writeReg_8_8(handle,0x46,3,0x00)
            usb_version = rtn_cfg['usbType']
            configs = config.configs
            #print(configs)
            configs_length = config.configs_length
            #print(configs_length)
            for i in range(configs_length):
                type = configs[i].type
                if ((type >> 16) & 0xFF) != 0 and ((type >> 16) & 0xFF) != usb_version:
                    continue
                if type & 0xFFFF == arducam_config_parser.CONFIG_TYPE_REG:
                    ArducamSDK.Py_ArduCam_writeSensorReg(self.handle, configs[i].params[0], configs[i].params[1])
                elif type & 0xFFFF == arducam_config_parser.CONFIG_TYPE_DELAY:
                    time.sleep(float(configs[i].params[0])/1000)
                elif type & 0xFFFF == arducam_config_parser.CONFIG_TYPE_VRCMD:
                    self.configBoard(configs[i])

            #brightness
            #0x3012 = "Coarse Integration Time"
            ArducamSDK.Py_ArduCam_writeSensorReg(self.handle, 0x3012, 1000)

            ArducamSDK.Py_ArduCam_registerCtrls(self.handle, config.controls, config.controls_length)
            #ArducamSDK.Py_ArduCam_setCtrl(self.handle, "setFramerate", 5)


            rtn_val,self.cam_id = ArducamSDK.Py_ArduCam_readUserData(self.handle,0x400-16, 16)
            self.cam_id = convert_serial(self.cam_id) #str(self.cam_id)
            return True
        else:
            print("Failed to open camera #%i. rtn_val = %i" % (self.idx, ret))
            return False
    pass
    

    def capture_single_image(self):
        global use_arducam_api_mutex, arducam_api_mutex

        #trigger the external trigger via software
        ArducamSDK.Py_ArduCam_softTrigger(self.handle)

        missed_time = 0
        while ArducamSDK.Py_ArduCam_isFrameReady(self.handle) != 1:
            #print("Camera #%i: Py_ArduCam_isFrameReady ... waiting." % (self.idx))
            time.sleep(0.005)
            missed_time += 0.005
 
        #if missed_time > 0:
        #    print("Camera #%i: Waited %f seconds for isFrameReady()." % (self.idx, missed_time))

        if use_arducam_api_mutex:
            arducam_api_mutex.acquire()

        rtn_val,data,rtn_cfg = ArducamSDK.Py_ArduCam_getSingleFrame(self.handle, 1000)
        if rtn_val != 0:
            print("Camera #%i: Error capturing single image, rtn_val = %i" % (self.idx, rtn_val))
            self.running = False

        if use_arducam_api_mutex:
            arducam_api_mutex.release()

        datasize = rtn_cfg['u32Size']
        if datasize == 0:
            print("Camera #%i: Data length zero" % (self.idx))
            return

        self.img = convert_image(data,rtn_cfg, self.color_mode)
        self.captures_done += 1


    def capture_single_image_thread_func(self):
        while self.running == True:
            if self.captures_todo > self.captures_done:
                print("Camera #%i: Received command to capture image." % (self.idx))
                time0 = time.time()
                self.capture_single_image()
         
                time1 = time.time()
                print("Camera #%i: Image capture required %f seconds.." % (self.idx, (time1 - time0)))


    def capture_start(self):
        if self.running == True:
            return
        print("capture start")
        self.running = True
        # start thread        
        #if self.capture_single_image_thread is not None:
        self.capture_single_image_thread.start()


    def capture_end(self):
        self.running = False
        # kill thread
        if self.capture_single_image_thread is not None and self.capture_single_image_thread.is_alive():
            self.capture_single_image_thread.join()
            print("Camera #%i. Single image thread killed successfully." % (self.idx))
       
        
    def close_cam(self):
        rtn_val = ArducamSDK.Py_ArduCam_close(self.handle)
        if rtn_val == 0:
            print(self.cam_id[:12],"-device close success!")
        else:
            print(self.cam_id[:12],"-device close fail!")


class MultiCam():
    cams = []
    cam_ok = []
    cam_count = 0
    config_file_name = ""


    def __init__(self) -> None:
        pass

    def set_config(self, config):
        self.config_file_name = config


    def scan_cameras(self):
        print("Scanning for cameras...")
        # Scan for the number of camera are available on the USB bus interface
        self.cam_count, handles, serial_numbers = ArducamSDK.Py_ArduCam_scan()

        # TODO: umwandeln in "add_camera(...)" und dann hier fuer jede gefundene
        # camera "add_camera()" nutzen

        # Creating the objects for the camera's available
        self.cam_ok = [False for i in range(self.cam_count)]
        self.cams = [None for i in range(self.cam_count)]
        for i in range(self.cam_count):
            self.cams[i] = singleCam(handles[i])
        print("Scanned for cameras... %i found." % (self.cam_count))

        # Sleep time of atleast one second is required between scan and open of the camera
        time.sleep(1)


    def initialize_camera(self, cam, cam_name):
        if cam.camera_initFromFile(self.config_file_name):
            cam.camera_name = cam_name
            print("camera '%s' initialized. Serial='%s'" % (cam.camera_name, cam.cam_id))
        else:
            print("camera '%s' failed to initialize." % (cam.camera_name))
            return False

        # Set the mode of operation - in our case it is continuous mode for the streaming purposes
        rtn_val = ArducamSDK.Py_ArduCam_setMode(cam.handle, ArducamSDK.EXTERNAL_TRIGGER_MODE)
        #ArducamSDK.Py_ArduCam_setMode(cam.handle,ArducamSDK.CONTINUOUS_MODE)

        # Define the thread
        cam.capture_single_image_thread = threading.Thread(target=cam.capture_single_image_thread_func)        

        return True


    def initialize_cameras(self):
        # Check if initialization setup is successful before starting the capture thread
        for i in range(self.cam_count):
            cam = self.cams[i]

            if not self.initialize_camera(cam, "Kamera " + str(i+1)):
                #break as soon as one camera fails
                break
        # wait a bit before giving back control to others
        time.sleep(1)

    def start_cameras(self):
        if self.cam_count == 0:
            return False

        # Start Threads
        for cam in self.cams:
            if cam.running:
                cam.capture_end()
            cam.capture_start()


    #capture an image from all cams - one after another
    def capture_single_shot(self):
        for cam in self.cams:
            restart = cam.running
            if restart:
                cam.capture_end()

            #actually fetch the image of this cam
            cam.capture_single_image()

            if restart:
                cam.capture_start()


    def close_cameras(self):
        for i in range(self.cam_count):
            self.cams[i].capture_end()
            self.cams[i].close_cam()



def convert_serial(byteArray):
    return (chr(byteArray[0]) + chr(byteArray[1]) + chr(byteArray[2]) + chr(byteArray[3]) 
            + "-" + chr(byteArray[4]) + chr(byteArray[5]) + chr(byteArray[6]) + chr(byteArray[7])
            + "-" + chr(byteArray[8]) + chr(byteArray[9]) + chr(byteArray[10]) + chr(byteArray[11])
           )

def sigint_handler(signum, frame):
    #global running,handle
    #running = False
    exit()

signal.signal(signal.SIGINT, sigint_handler)
#signal.signal(signal.SIGHUP, sigint_handler)
signal.signal(signal.SIGTERM, sigint_handler)


if __name__ == "__main__":

    config_file_name = "/home/nano/Tests/Multicam/ArduCAM_USB_Camera_Shield-master/Config/USB2.0_UC-391_Rev.E/DVP/MT9J003/MT9J003_RAW_8b_1920x1080_20fps.cfg"
    if len(sys.argv) > 1:
        if os.path.exists(sys.argv[1]):
            config_file_name = sys.argv[1]
        else:
            print("Config file does not exist.")

    mc = MultiCam()
    mc.set_config(config_file_name)
    mc.scan_cameras()
    mc.initialize_cameras()
    mc.start_cameras()

    print ("Ready to fetch data ...")

    mc.close_cameras()

    print("Done.")

MultiCam_TestWindow.py

import sys
import MultiCam
import cv2
import time

if __name__ == "__main__":

    config_file_name = "/home/nano/Tests/Multicam/ArduCAM_USB_Camera_Shield-master/Config/USB2.0_UC-391_Rev.E/DVP/MT9J003/MT9J003_RAW_8b_1920x1080_20fps.cfg"
    if len(sys.argv) > 1:
        if os.path.exists(sys.argv[1]):
            config_file_name = sys.argv[1]
        else:
            print("Config file does not exist.")

    mc = MultiCam.MultiCam()
    mc.set_config(config_file_name)
    mc.scan_cameras()
    mc.initialize_cameras()
    mc.start_cameras()

    print ("Starting loop")
    # Exception handling for the termination of the program
    try:
        exit_loop = False
        captures_done = 0
        captures_todo = 0
        next_snapshot_interval = 1000 #ms
        next_snapshot_time = time.time() #now
        snapshot_request_time = None
        while(not exit_loop and mc.cam_count > 0):
            time.sleep(0.10)

            #check if one camera is still running/snapshotting
            all_done_snapshotting = True
            for cam in mc.cams:
                if cam.captures_todo > cam.captures_done:
                    all_done_snapshotting = False
                    break
            

            #current capture done?
            if all_done_snapshotting and captures_todo > captures_done:
                for cam in mc.cams:
                    image = cv2.resize(cam.img, (640,480), interpolation = cv2.INTER_LINEAR)
                    cv2.imshow(cam.camera_name, image)
                cv2.waitKey(10)

                captures_done += 1
                print("Capture done by all cameras in %f seconds." % ((time.time() - snapshot_request_time)))


            #time for a new snapshot?
            if all_done_snapshotting and next_snapshot_time <= time.time():
                print("Time for a new snapshot request...")
                captures_todo += 1
                for cam in mc.cams:
                    cam.captures_todo += 1
                snapshot_request_time = time.time()

            #print("Tick ... time=%f  all_done_snapshotting=%s" % (time.time(), done_string))


            #cv2.waitKey(1)
            #cv2.imwrite('images1/camera1.jpg',cam1.img)
    # Kill the threads and exit the program when Keyboard interrupt happens
    except KeyboardInterrupt:
        print('Exception called')

    mc.close_cameras()
    cv2.destroyAllWindows()

    print("Done.")
  • attach 4 cameras on the same hub (so they share a “USB2 lane”).
  • open “MultiCam_TestWindow.py” and adjust the path to the config file (and save…)
  • execute “MultiCam_TestWindow.py”
  • wait for it to “stuck” in one of the loops, might happen right on the first - or maybe on the second or third (seems like “collisions” - like in non-thread-safe environments)

Open “MultiCam.py” and set use_arducam_api_mutex = False to True so it places a mutex around ArducamSDK.Py_ArduCam_getSingleFrame(self.handle, 1000).

If you need further information, do not hesitate to ask.