I am trying to implement this algo on an embedded system and speed of the deepsort component is much slower compared to yolo. Is it possible to to run deepsort at regular intervals instead of every frame?

Thank you very much for your reply. I would like to know how to run the folder of multi-objective evaluation indicators to get MOT16 evaluation results (such as MOTA).

Evaluating ch_yolov5m_deep_sort

MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-02)     0.3986 sec
MotChallenge2DBox.get_preprocessed_seq_data(pedestrian)                0.4142 sec
CLEAR.eval_sequence()                                                  0.1181 sec
Identity.eval_sequence()                                               0.0189 sec
Count.eval_sequence()                                                  0.0000 sec

1 eval_sequence(MOT16-02, ch_yolov5m_deep_sort) 0.9544 sec MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-04) 1.4155 sec MotChallenge2DBox.get_preprocessed_seq_data(pedestrian) 0.8502 sec CLEAR.eval_sequence() 0.2638 sec Identity.eval_sequence() 0.0354 sec Count.eval_sequence() 0.0000 sec 2 eval_sequence(MOT16-04, ch_yolov5m_deep_sort) 2.5745 sec MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-05) 0.1946 sec MotChallenge2DBox.get_preprocessed_seq_data(pedestrian) 0.4567 sec CLEAR.eval_sequence() 0.1318 sec Identity.eval_sequence() 0.0214 sec Count.eval_sequence() 0.0000 sec 3 eval_sequence(MOT16-05, ch_yolov5m_deep_sort) 0.8120 sec MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-09) 0.1831 sec MotChallenge2DBox.get_preprocessed_seq_data(pedestrian) 0.3084 sec CLEAR.eval_sequence() 0.0786 sec Identity.eval_sequence() 0.0093 sec Count.eval_sequence() 0.0000 sec 4 eval_sequence(MOT16-09, ch_yolov5m_deep_sort) 0.5832 sec MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-10) 0.2463 sec MotChallenge2DBox.get_preprocessed_seq_data(pedestrian) 0.3749 sec CLEAR.eval_sequence() 0.1235 sec Identity.eval_sequence() 0.0108 sec Count.eval_sequence() 0.0000 sec 5 eval_sequence(MOT16-10, ch_yolov5m_deep_sort) 0.7594 sec MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-11) 0.2213 sec MotChallenge2DBox.get_preprocessed_seq_data(pedestrian) 0.4973 sec CLEAR.eval_sequence() 0.1329 sec Identity.eval_sequence() 0.0162 sec Count.eval_sequence() 0.0000 sec 6 eval_sequence(MOT16-11, ch_yolov5m_deep_sort) 0.8738 sec MotChallenge2DBox.get_raw_seq_data(ch_yolov5m_deep_sort, MOT16-13) 0.2492 sec MotChallenge2DBox.get_preprocessed_seq_data(pedestrian) 0.4239 sec CLEAR.eval_sequence() 0.1154 sec Identity.eval_sequence() 0.0150 sec Count.eval_sequence() 0.0000 sec 7 eval_sequence(MOT16-13, ch_yolov5m_deep_sort) 0.8082 sec

All sequences for ch_yolov5m_deep_sort finished in 7.37 seconds

CLEAR: ch_yolov5m_deep_sort-pedestrianMOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
MOT16-02 40.677 91.743 40.778 41.317 98.714 20.37 40.741 38.889 37.266 7368 10465 96 18 11 22 21 19
MOT16-04 65.656 90.874 65.685 65.797 99.831 34.94 36.145 28.916 59.651 31291 16266 53 14 29 30 24 18
MOT16-05 55.471 85.627 55.749 62.027 90.81 27.2 49.6 23.2 46.556 4229 2589 428 19 34 62 29 30
MOT16-09 74.035 89.755 74.13 76.622 96.85 56 40 4 66.185 4028 1229 131 5 14 10 1 6
MOT16-10 62.088 85.56 62.34 66.935 93.576 40.741 46.296 12.963 52.422 8245 4073 566 31 22 25 7 78
MOT16-11 64.214 92.027 64.312 66.961 96.195 27.536 49.275 23.188 58.876 6143 3031 243 9 19 34 16 14
MOT16-13 51.729 87.662 51.956 53.878 96.557 28.972 40.187 30.841 45.082 6169 5281 220 26 31 43 33 34
COMBINED 59.429 89.735 59.54 61.113 97.49 30.948 43.714 25.338 53.156 67473 42934 1737 122 160 226 131 199

Identity: ch_yolov5m_deep_sort-pedestrianIDF1 IDR IDP IDTP IDFN IDFP
MOT16-02 50.045 35.496 84.807 6330 11503 1134
MOT16-04 75.348 62.504 94.835 29725 17832 1619
MOT16-05 63.808 53.696 78.613 3661 3157 996
MOT16-09 77.889 69.755 88.17 3667 1590 492
MOT16-10 69.052 59.222 82.794 7295 5023 1516
MOT16-11 70.334 59.647 85.687 5472 3702 914
MOT16-13 58.972 45.939 82.329 5260 6190 1129
COMBINED 68.379 55.621 88.73 61410 48997 7800

Count: ch_yolov5m_deep_sort-pedestrianDets GT_Dets IDs GT_IDs
MOT16-02 7464 17833 47 54
MOT16-04 31344 47557 72 83
MOT16-05 4657 6818 76 125
MOT16-09 4159 5257 22 25
MOT16-10 8811 12318 58 54
MOT16-11 6386 9174 56 69
MOT16-13 6389 11450 73 107
COMBINED 69210 110407 404 517

Timing analysis: MotChallenge2DBox.get_raw_seq_data 2.9086 sec MotChallenge2DBox.get_preprocessed_seq_data 3.3256 sec CLEAR.eval_sequence 0.9641 sec Identity.eval_sequence 0.1270 sec Count.eval_sequence 0.0000 sec eval_sequence 7.3654 sec Evaluator.evaluate 7.3683 sec

Reopening since it seemed to get some attention. Rebased to latest master, I do not know of any other changes to the repo. Please let me know.

Fixed noise covariance matrices so they are not varied based on bounding box location. Fixed delta time of predictions from constant 1 second to varied based on the frequency of predictions, this should increase performance.

Implemented adaptive kalman filter for Q and R estimation, based on this article.

In my experiments, I found that the additions I made gave better results in a practical scenario. When tracking something, you usually want to take into account the delta time between kalman updates. Also, I made it not necessary any more to have to tune the filter to find the optimal Q and R noise matrix parameters, should hopefully give better results in the end.

Hey a clarification! while running detection over a video, I see that my entire CPU memory is being used. I'm not able to run it on multiple threads as it leads to slowness. Did anyone face this issue ? Any help would be appreciated

in deepsort tracker

# Now Compute the Appearance-based Cost Matrix app_cost = self.metric.distance( np.array([dets[i].feature for i in detection_indices]), np.array([tracks[i].track_id for i in track_indices]), )

Why is line 121 fetch the track_id instead of the feature of the track?

It seems to be wrong since the appearence cost always return way higher than the threshold Edit: The problem is not in the track_id, but in the distance function. See below comments

Thank you for your great work. I have a question--why the id are assigned without any order? I think some object is detected and given an ID, but it is not tracked, so some IDs are not displayed. I am not sure why this is happening. Any help is much appreciated.Thanks.

Search before asking

• [X] I have searched the Yolov5_StrongSORT_OSNet issues and found no similar bug report.

Question

Greetings. I encountered a small error (rather, my knowledge is not enough:( ). When starting the program, the following error occurs: "No URL associated to the chosen DeepSort weights. Choose between:" How to choose? Where? Why and why? Thanks a lot in advance!

Search before asking

• [X] I have searched the Yolov5_StrongSORT_OSNet issues and found no similar bug report.

Question

Hi,

I have a YOLOv5 model, a video and its ground truth. I would like to evaluate YOLOv5 StrongSORT on this video but I do not know how to do it? Is there any tutorial, or someone can explain it to me?

Thank you in advance.

Search before asking

• [X] I have searched the Yolov5_StrongSORT_OSNet issues and found no similar bug report.

Question

I was unable to download the strong_sort weights file online, so I chose to download the weights manually from the model zoo. However, after the downloaded weight is placed in the corresponding folder, the code cannot be read, and the online download is still performed. Is it due to the weight format being .pth? How to solve it?

     IDF1 IDP  IDR Rcll Prcn GT MT PT ML FP   FN IDs  FM MOTA MOTP IDt IDa IDm

MOT16-09 0.0% NaN 0.0% 0.0% NaN 25 0 0 25 0 5257 0 0 0.0% NaN 0 0 0 OVERALL 0.0% NaN 0.0% 0.0% NaN 25 0 0 25 0 5257 0 0 0.0% NaN 0 0 0 I write the code by yolov3_deepsort,that code can run the results.but this get this results.Do you run this code on MOT16 ?

Search before asking

• [x] I have searched the Yolov5_StrongSORT_OSNet issues and found no similar bug report.

Question

Hello,

I want to detect floating plastics on the water surface. The detection for only yolov5 runs goes quite well. The problem is that I want to count the objects passing and remove the duplicate counts. When running the track.py script, there a many detections missing that were detected in the detect.py script of the regular yolov5:

Any idea what might cause this difference & how to fix it?

Search before asking

• [X] I have searched the Yolov5_StrongSORT_OSNet issues and found no similar bug report.

Question

Hello! I need help understanding why StrongSORT losing an object. Any suggestion would be appreciated. Thank you!

1. Reassign tracked object to a newly detected object with the current object_d

2. Losing tracks after collision with another object

3. Although, I have tried to hide behind the wall for a couple of seconds. And after every appearance on the camera tracker assign a new object_id. So person re-id not working well. (I was facing the camera before hiding the wall and appear facing the camera).

As discussed in the previous PR, BoT-SORT tracker is implemented using the ReID architectures used in the repo. Also have made some changes in the README.md file for the installation of cython_bbox.

Demo video:

https://user-images.githubusercontent.com/82194525/209459332-e1c74fca-25b6-4b1f-8cdb-be8338b40777.mp4

Search before asking

• [X] I have searched the Yolov5_StrongSORT_OSNet issues and found no similar bug report.

Question

Hello, I found this repo super helpful and very straight forward to use! Thank you for this amazing work on this repo!

My Question:

I want to get the segmentation masks of the tracked object in the output text file. but, after examining your code repo for a while I came to the conclusion that the track.py file draws and populates save_text file using the boxes from the tracker output. and draws the masks from the yolo model output: lines to draw mask: here.

# Mask plotting
annotator.masks(
    masks,
    colors=[colors(x, True) for x in det[:, 5]],
    im_gpu=torch.as_tensor(im0, dtype=torch.float16).to(device).permute(2, 0, 1).flip(0).contiguous() /
    255 if retina_masks else im[i]
)

lines to draw boxes: here.

if save_txt:
    # to MOT format
    bbox_left = output[0]
    bbox_top = output[1]
    bbox_w = output[2] - output[0]
    bbox_h = output[3] - output[1]
    # Write MOT compliant results to file
    with open(txt_path + '.txt', 'a') as f:
        f.write(('%g ' * 10 + '\n') % (frame_idx + 1, id, bbox_left,  # MOT format
                                       bbox_top, bbox_w, bbox_h, -1, -1, -1, i))

if save_vid or save_crop or show_vid:  # Add bbox to image
    c = int(cls)  # integer class
    id = int(id)  # integer id
    label = None if hide_labels else (f'{id} {names[c]}' if hide_conf else \
        (f'{id} {conf:.2f}' if hide_class else f'{id} {names[c]} {conf:.2f}'))
    color = colors(c, True)
    annotator.box_label(bboxes, label, color=color)

    if save_trajectories and tracking_method == 'strongsort':
        q = output[7]
        tracker_list[i].trajectory(im0, q, color=color)
    if save_crop:
        txt_file_name = txt_file_name if (isinstance(path, list) and len(path) > 1) else ''
        save_one_box(bboxes, imc, file=save_dir / 'crops' / txt_file_name / names[c] / f'{id}' / f'{p.stem}.jpg', BGR=True)

If my understanding is correct the output of the tracker is used to:

1. draw bboxes. and
2. save text file when --save_text is added.

While the yolo model output is used to draw masks...

since my goal is to get an output text file that contains the segments of the tracked objects I thought of maybe trying to match the yolo segments to the trackers output, and then write the matched results to an output text file.

but I've also came to the conclusion that not all object from the detector has to be tracked and for a single video frame, all detected objects will not necessarily have an associated tracking output. (kindly correct me if I'm wrong about this)

do you have any suggestion for me to output a text file that contains segments of tracked object?

thanks in advance!

I have implemented Strong-OCSort, which is a combination between StrongSort and OCSort.

StrongOCSort performs association in 3 steps:

For all detections with confidence above a detection threshold: 1. Associate using feature matching (StrongSort) 2. Associate using trajectory matching (OCSort)

(optional) For all detections with confidence below a detection threshold: 3. Associate using byte association

I have also implemented a resurrection system. This is my attempt to cache features of tracks that have "died". In case a detection with a similar feature shows up again in a sequence (lady with a red shirt in MOT17-05 for example), it should create a track of the same ID as the one that previously "died". This system is not solving this issue at the moment, but I left the code in together with a parameter to toggle the system on and off (default off).

Eval results (can be tuned for better results), based on detector weights crowdhuman_yolov5m.pt

HOTA: exp15-pedestrian             HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
MOT17-04-FRCNN                     60.825    59.138    63.116    64.129    76.426    67.275    80.735    81.156    63.564    80.714    75.474    60.918
MOT17-05-FRCNN                     39.572    38.377    40.885    40.461    77.312    50.886    65.91     82.316    40.667    50.308    78.129    39.305
MOT17-09-FRCNN                     57.6      61.734    53.789    66.385    82.209    57.95     82.977    86.121    59.744    70.633    82.976    58.609
MOT17-10-FRCNN                     50.489    50.549    50.58     53.689    76.855    54.478    79.142    81.023    52.106    66.309    76.676    50.843
MOT17-11-FRCNN                     63.41     60.258    66.942    70.522    75.302    73.393    83.735    86.99     68.697    75.341    83.895    63.207
MOT17-13-FRCNN                     46.94     42.853    51.795    46.001    74.34     56.438    77.363    80.765    48.761    60.994    75.77     46.216
COMBINED                           56.889    54.669    59.764    59.472    76.522    64.677    80.779    82.155    59.525    73.537    77.147    56.731

CLEAR: exp15-pedestrian            MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag
MOT17-04-FRCNN                     70.095    78.644    70.263    77.086    91.868    50.602    36.145    13.253    53.632    36660     10897     3245      80        42        30        11        441
MOT17-05-FRCNN                     45.829    79.61     46.697    49.516    94.613    16.541    57.895    25.564    35.733    3425      3492      195       60        22        77        34        168
MOT17-09-FRCNN                     69.164    84.578    69.746    75.249    93.186    46.154    53.846    0         57.559    4007      1318      293       31        12        14        0         67
MOT17-10-FRCNN                     63.066    77.822    63.595    66.726    95.518    33.333    56.14     10.526    48.267    8567      4272      402       68        19        32        6         558
MOT17-11-FRCNN                     65.112    85.561    65.441    79.546    84.938    45.333    40        14.667    53.627    7506      1930      1331      31        34        30        11        104
MOT17-13-FRCNN                     52.843    77.367    53.548    57.713    93.268    30.909    40.909    28.182    39.781    6719      4923      485       82        34        45        31        278
COMBINED                           64.643    79.592    65.019    71.369    91.829    33.678    47.107    19.215    50.078    66884     26832     5951      352       163       228       93        1616

Identity: exp15-pedestrian         IDF1      IDR       IDP       IDTP      IDFN      IDFP
MOT17-04-FRCNN                     76.335    70.194    83.654    33382     14175     6523
MOT17-05-FRCNN                     53.753    40.943    78.232    2832      4085      788
MOT17-09-FRCNN                     70.919    64.094    79.372    3413      1912      887
MOT17-10-FRCNN                     68.204    57.925    82.919    7437      5402      1532
MOT17-11-FRCNN                     74.449    72.086    76.972    6802      2634      2035
MOT17-13-FRCNN                     62.602    50.67     81.885    5899      5743      1305
COMBINED                           71.768    63.772    82.055    59765     33951     13070

Count: exp15-pedestrian            Dets      GT_Dets   IDs       GT_IDs
MOT17-04-FRCNN                     39905     47557     140       83
MOT17-05-FRCNN                     3620      6917      117       133
MOT17-09-FRCNN                     4300      5325      54        26
MOT17-10-FRCNN                     8969      12839     105       57
MOT17-11-FRCNN                     8837      9436      153       75
MOT17-13-FRCNN                     7204      11642     141       110
COMBINED                           72835     93716     710       484

Previews (MOT17-05):

Without trajectories:

With trajectories :

Blue circles are feature-matched tracks Green circles are trajectory-matched tracks White circles are unmatched tracks No byte association was seen in this sequence (should probably seldom happen for a large enough detector model, will happen more often for smaller detection models).

If the resurrection system is turned on, you will see resurrected tracks as purple circles in the trajectory plot.

Implemented predictions if no detections come from yolo

HOTA: exp-pedestrian               HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
MOT17-04-FRCNN                     61.223    59.34     63.701    64.651    75.783    68.253    80.509    80.925    64.126    81.961    75.043    61.506
MOT17-05-FRCNN                     42.367    40.82     44.104    44.135    73.81     54.626    65.858    81.502    44.109    54.935    76.472    42.01
MOT17-09-FRCNN                     58.03     61.61     54.702    67.461    80.114    60.43     79.769    85.656    60.737    72.13     81.815    59.013
MOT17-10-FRCNN                     51.628    52.627    50.836    56.976    74.417    55.733    76.026    80.314    53.812    69.267    75.347    52.19
MOT17-11-FRCNN                     62.068    58.834    65.678    71.629    71.774    72.308    83.066    86.635    68.58     74.617    83.102    62.008
MOT17-13-FRCNN                     47.269    44.033    51.147    48.209    71.661    57.14     74.165    80.062    49.603    62.66     74.508    46.687
COMBINED                           57.254    55.179    59.953    60.905    74.836    65.477    79.587    81.75     60.347    74.957    76.376    57.249

CLEAR: exp-pedestrian              MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag
MOT17-04-FRCNN                     69.87     78.439    70.002    77.656    91.028    50.602    33.735    15.663    53.126    36931     10626     3640      63        42        28        13        376
MOT17-05-FRCNN                     47.636    78.725    48.605    54.2      90.643    24.06     60.15     15.789    36.105    3749      3168      387       67        32        80        21        153
MOT17-09-FRCNN                     69.446    84.359    70.197    77.202    91.682    53.846    46.154    0         57.371    4111      1214      373       40        14        12        0         70
MOT17-10-FRCNN                     65.558    76.879    66.189    71.376    93.225    31.579    61.404    7.0175    49.055    9164      3675      666       81        18        35        4         331
MOT17-11-FRCNN                     61.615    85.232    61.944    80.871    81.034    49.333    37.333    13.333    49.672    7631      1805      1786      31        37        28        10        96
MOT17-13-FRCNN                     53.659    76.647    54.338    60.806    90.386    31.818    40.909    27.273    39.459    7079      4563      753       79        35        45        30        221
COMBINED                           64.769    79.171    65.154    73.269    90.029    36.777    47.107    16.116    49.508    68665     25051     7605      361       178       228       78        1247

Identity: exp-pedestrian           IDF1      IDR       IDP       IDTP      IDFN      IDFP
MOT17-04-FRCNN                     77.367    71.685    84.028    34091     13466     6480
MOT17-05-FRCNN                     57.251    45.742    76.499    3164      3753      972
MOT17-09-FRCNN                     71.506    65.859    78.211    3507      1818      977
MOT17-10-FRCNN                     69.249    61.134    79.847    7849      4990      1981
MOT17-11-FRCNN                     73.24     73.167    73.314    6904      2532      2513
MOT17-13-FRCNN                     63.695    53.273    79.188    6202      5440      1630
COMBINED                           72.614    65.855    80.919    61717     31999     14553

Count: exp-pedestrian              Dets      GT_Dets   IDs       GT_IDs
MOT17-04-FRCNN                     40571     47557     144       83
MOT17-05-FRCNN                     4136      6917      117       133
MOT17-09-FRCNN                     4484      5325      53        26
MOT17-10-FRCNN                     9830      12839     105       57
MOT17-11-FRCNN                     9417      9436      161       75
MOT17-13-FRCNN                     7832      11642     134       110
COMBINED                           76270     93716     714       484