TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation, CVPR2022

TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation

Paper Links: TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation (CVPR 2022)

by Wenqiang Zhang*, Zilong Huang*, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, Chunhua Shen.

(*) equal contribution, (†) corresponding author.

Introduction

Although vision transformers (ViTs) have achieved great success in computer vision, the heavy computational cost makes it not suitable to deal with dense prediction tasks such as semantic segmentation on mobile devices. In this paper, we present a mobile-friendly architecture named Token Pyramid Vision TransFormer(TopFormer). The proposed TopFormer takes Tokens from various scales as input to produce scale-aware semantic features, which are then injected into the corresponding tokens to augment the representation. Experimental results demonstrate that our method significantly outperforms CNN- and ViT-based networks across several semantic segmentation datasets and achieves a good trade-off between accuracy and latency.

The latency is measured on a single Qualcomm Snapdragon 865 with input size 512×512×3, only an ARM CPU core is used for speed testing. *indicates the input size is 448×448×3.

Updates

  • 04/23/2022: TopFormer backbone has been integrated into PaddleViT, checkout here for the 3rd party implementation on Paddle framework!

Requirements

  • pytorch 1.5+
  • mmcv-full==1.3.14

Main results

The classification models pretrained on ImageNet can be downloaded from Baidu Drive/Google Drive.

ADE20K

Model Params(M) FLOPs(G) mIoU(ss) Link
TopFormer-T_448x448_2x8_160k 1.4 0.5 32.5 Baidu Drive, Google Drive
TopFormer-T_448x448_4x8_160k 1.4 0.5 33.4 Baidu Drive, Google Drive
TopFormer-T_512x512_2x8_160k 1.4 0.6 33.6 Baidu Drive, Google Drive
TopFormer-T_512x512_4x8_160k 1.4 0.6 34.6 Baidu Drive, Google Drive
TopFormer-S_512x512_2x8_160k 3.1 1.2 36.5 Baidu Drive, Google Drive
TopFormer-S_512x512_4x8_160k 3.1 1.2 37.0 Baidu Drive, Google Drive
TopFormer-B_512x512_2x8_160k 5.1 1.8 38.3 Baidu Drive, Google Drive
TopFormer-B_512x512_4x8_160k 5.1 1.8 39.2 Baidu Drive, Google Drive
  • ss indicates single-scale.
  • The password of Baidu Drive is topf

Usage

Please see MMSegmentation for dataset prepare.

For training, run:

sh tools/dist_train.sh local_configs/topformer/<config-file> <num-of-gpus-to-use> --work-dir /path/to/save/checkpoint

To evaluate, run:

sh tools/dist_test.sh local_configs/topformer/<config-file> <checkpoint-path> <num-of-gpus-to-use>

To test the inference speed in mobile device, please refer to tnn_runtime.

Acknowledgement

The implementation is based on MMSegmentation.

Citation

if you find our work helpful to your experiments, please cite with:

@article{zhang2022topformer,
  title     = {TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation},
  author    = {Zhang, Wenqiang and Huang, Zilong and Luo, Guozhong and Chen, Tao and Wang,  Xinggang and Liu, Wenyu and Yu, Gang and Shen, Chunhua.},
  booktitle = {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  year      = {2022}
}
Owner
Hust Visual Learning Team
Hust Visual Learning Team belongs to the Artificial Intelligence Research Institute in the School of EIC in HUST, Lead by @xinggangw
Hust Visual Learning Team
Comments
  • TNN推理无法进行

    TNN推理无法进行

    TNN推理没有办法进行,我用的README里面提供的TopFormer-T_512x512_2x8_160k-33.6.pth,按照tnn_runtime里的流程转换。

    首先我写了一个onnx.sh,用于转换ONNX

    CONFIG=$1
    CHECK=$2
    
    PYTHONPATH="$(dirname $0)/..":$PYTHONPATH \
    python $(dirname "$0")/convert2onnx.py $CONFIG --shape 512 512 --checkpoint $CHECK
    

    convert2onnx.py文件里面我只在torch.onnx.export里面添加了一个opset_version=11,其他未改。

    之后执行命令:bash tools/onnx.sh local_configs/topformer/topformer_tiny_512x512_160k_2x8_ade20k.py <path>/TopFormer-T_512x512_2x8_160k-33.6.pth

    输出以下结果: image

    按照TNN的convert.md自行编译convert2tnn,之后执行命令:python3 converter.py onnx2tnn <path>/topformer_t.onnx -optimize -v=v3.0

    再按照TNN的profiling.md进行Android平台耗时测试。执行的代码是./benchmark_models.sh -c。我用TNN代码库提供一个样例squeezenet_v1.1.tnnproto试了一下是可以正常在我的手机上测试的,但是我转的tnnproto就不能运行,会一直卡住,见下图: image

    如果我把TNN转换时的-optimize参数去掉,就不会卡住,但是会报错: image

    相关工具的版本:

    ndk:android-ndk-r23c
    Cmake:3.21.7
    TNN用的最新版代码库:[20b2b7a](https://github.com/Tencent/TNN/commit/20b2b7a068c1bb6f1efb701009bece82e7a80bb4)
    

    Python环境:

    _libgcc_mutex             0.1                        main    defaults
    _openmp_mutex             5.1                       1_gnu    defaults
    addict                    2.4.0                    pypi_0    pypi
    anyio                     3.6.1            py38h578d9bd_0    conda-forge
    argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
    argon2-cffi-bindings      21.2.0           py38h0a891b7_2    conda-forge
    asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
    attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
    babel                     2.10.3             pyhd8ed1ab_0    conda-forge
    backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
    backports                 1.0                        py_2    conda-forge
    backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
    beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
    blas                      1.0                         mkl    defaults
    bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
    bottleneck                1.3.5            py38h7deecbd_0    defaults
    brotli                    1.0.9                he6710b0_2    defaults
    brotlipy                  0.7.0           py38h0a891b7_1004    conda-forge
    bzip2                     1.0.8                h7f98852_4    conda-forge
    ca-certificates           2022.07.19           h06a4308_0    defaults
    certifi                   2022.6.15        py38h06a4308_0    defaults
    cffi                      1.15.0           py38hd667e15_1    defaults
    charset-normalizer        2.1.0              pyhd8ed1ab_0    conda-forge
    click                     8.1.3                    pypi_0    pypi
    codecov                   2.1.12                   pypi_0    pypi
    colorama                  0.4.5                    pypi_0    pypi
    coloredlogs               15.0.1                   pypi_0    pypi
    commonmark                0.9.1                    pypi_0    pypi
    coverage                  6.4.2                    pypi_0    pypi
    cryptography              35.0.0           py38h3e25421_2    conda-forge
    cudatoolkit               11.1.1              ha002fc5_10    conda-forge
    cycler                    0.11.0             pyhd3eb1b0_0    defaults
    dbus                      1.13.18              hb2f20db_0    defaults
    debugpy                   1.6.0            py38hfa26641_0    conda-forge
    decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
    defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
    entrypoints               0.4                pyhd8ed1ab_0    conda-forge
    executing                 0.9.1              pyhd8ed1ab_0    conda-forge
    expat                     2.4.4                h295c915_0    defaults
    ffmpeg                    4.3                  hf484d3e_0    pytorch
    flake8                    4.0.1                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
    fontconfig                2.13.1               h6c09931_0    defaults
    fonttools                 4.25.0             pyhd3eb1b0_0    defaults
    freetype                  2.11.0               h70c0345_0    defaults
    glib                      2.69.1               h4ff587b_1    defaults
    gmp                       6.2.1                h58526e2_0    conda-forge
    gnutls                    3.6.13               h85f3911_1    conda-forge
    gst-plugins-base          1.14.0               h8213a91_2    defaults
    gstreamer                 1.14.0               h28cd5cc_2    defaults
    humanfriendly             10.0                     pypi_0    pypi
    icu                       58.2                 he6710b0_3    defaults
    idna                      3.3                pyhd8ed1ab_0    conda-forge
    importlib-metadata        4.11.4           py38h578d9bd_0    conda-forge
    importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
    importlib_resources       5.9.0              pyhd8ed1ab_0    conda-forge
    iniconfig                 1.1.1                    pypi_0    pypi
    intel-openmp              2021.4.0          h06a4308_3561    defaults
    interrogate               1.5.0                    pypi_0    pypi
    ipykernel                 6.15.1             pyh210e3f2_0    conda-forge
    ipython                   8.4.0            py38h578d9bd_0    conda-forge
    ipython_genutils          0.2.0                      py_1    conda-forge
    isort                     4.3.21                   pypi_0    pypi
    jedi                      0.18.1           py38h578d9bd_1    conda-forge
    jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
    jpeg                      9b                   h024ee3a_2    defaults
    json5                     0.9.5              pyh9f0ad1d_0    conda-forge
    jsonschema                4.7.2              pyhd8ed1ab_0    conda-forge
    jupyter_client            7.0.6              pyhd8ed1ab_0    conda-forge
    jupyter_core              4.11.1           py38h578d9bd_0    conda-forge
    jupyter_server            1.18.1             pyhd8ed1ab_0    conda-forge
    jupyterlab                3.4.4              pyhd8ed1ab_0    conda-forge
    jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
    jupyterlab_server         2.15.0             pyhd8ed1ab_0    conda-forge
    kiwisolver                1.4.2            py38h295c915_0    defaults
    lame                      3.100             h7f98852_1001    conda-forge
    ld_impl_linux-64          2.38                 h1181459_1    defaults
    libffi                    3.3                  he6710b0_2    defaults
    libgcc-ng                 11.2.0               h1234567_1    defaults
    libgfortran-ng            7.5.0               ha8ba4b0_17    defaults
    libgfortran4              7.5.0               ha8ba4b0_17    defaults
    libgomp                   11.2.0               h1234567_1    defaults
    libiconv                  1.17                 h166bdaf_0    conda-forge
    libpng                    1.6.37               h21135ba_2    conda-forge
    libsodium                 1.0.18               h36c2ea0_1    conda-forge
    libstdcxx-ng              11.2.0               h1234567_1    defaults
    libtiff                   4.1.0                h2733197_1    defaults
    libuuid                   1.0.3                h7f8727e_2    defaults
    libuv                     1.43.0               h7f98852_0    conda-forge
    libxcb                    1.15                 h7f8727e_0    defaults
    libxml2                   2.9.14               h74e7548_0    defaults
    lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
    markupsafe                2.1.1            py38h0a891b7_1    conda-forge
    matplotlib                3.5.1            py38h06a4308_1    defaults
    matplotlib-base           3.5.1            py38ha18d171_1    defaults
    matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
    mccabe                    0.6.1                    pypi_0    pypi
    mistune                   0.8.4           py38h497a2fe_1005    conda-forge
    mkl                       2021.4.0           h06a4308_640    defaults
    mkl-service               2.4.0            py38h95df7f1_0    conda-forge
    mkl_fft                   1.3.1            py38h8666266_1    conda-forge
    mkl_random                1.2.2            py38h1abd341_0    conda-forge
    mmcv-full                 1.3.14                   pypi_0    pypi
    mpmath                    1.2.1                    pypi_0    pypi
    munkres                   1.1.4                      py_0    defaults
    nbclassic                 0.4.3              pyhd8ed1ab_0    conda-forge
    nbclient                  0.6.6              pyhd8ed1ab_0    conda-forge
    nbconvert                 6.5.0              pyhd8ed1ab_0    conda-forge
    nbconvert-core            6.5.0              pyhd8ed1ab_0    conda-forge
    nbconvert-pandoc          6.5.0              pyhd8ed1ab_0    conda-forge
    nbformat                  5.4.0              pyhd8ed1ab_0    conda-forge
    ncurses                   6.3                  h5eee18b_3    defaults
    nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
    nettle                    3.6                  he412f7d_0    conda-forge
    ninja                     1.11.0               h924138e_0    conda-forge
    notebook                  6.4.12             pyha770c72_0    conda-forge
    notebook-shim             0.1.0              pyhd8ed1ab_0    conda-forge
    numexpr                   2.8.3            py38h807cd23_0    defaults
    numpy                     1.22.3           py38he7a7128_0    defaults
    numpy-base                1.22.3           py38hf524024_0    defaults
    olefile                   0.46               pyh9f0ad1d_1    conda-forge
    onnx                      1.12.0                   pypi_0    pypi
    onnx-simplifier           0.4.1                    pypi_0    pypi
    onnxruntime               1.12.0                   pypi_0    pypi
    opencv-python             4.6.0.66                 pypi_0    pypi
    openh264                  2.1.1                h780b84a_0    conda-forge
    openssl                   1.1.1q               h7f8727e_0    defaults
    packaging                 21.3               pyhd3eb1b0_0    defaults
    pandas                    1.4.3            py38h6a678d5_0    defaults
    pandoc                    2.18                 ha770c72_0    conda-forge
    pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
    parso                     0.8.3              pyhd8ed1ab_0    conda-forge
    pcre                      8.45                 h295c915_0    defaults
    pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
    pickleshare               0.7.5                   py_1003    conda-forge
    pillow                    7.1.2            py38hb39fc2d_0    defaults
    pip                       22.1.2           py38h06a4308_0    defaults
    pluggy                    1.0.0                    pypi_0    pypi
    prettytable               3.3.0              pyhd8ed1ab_0    conda-forge
    prometheus_client         0.14.1             pyhd8ed1ab_0    conda-forge
    prompt-toolkit            3.0.30             pyha770c72_0    conda-forge
    protobuf                  3.20.1                   pypi_0    pypi
    psutil                    5.9.1            py38h0a891b7_0    conda-forge
    ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
    pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
    py                        1.11.0                   pypi_0    pypi
    pybind11                  2.10.0                   pypi_0    pypi
    pycodestyle               2.8.0                    pypi_0    pypi
    pycparser                 2.21               pyhd8ed1ab_0    conda-forge
    pyflakes                  2.4.0                    pypi_0    pypi
    pygments                  2.12.0             pyhd8ed1ab_0    conda-forge
    pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
    pyparsing                 3.0.9                    pypi_0    pypi
    pyqt                      5.9.2            py38h05f1152_4    defaults
    pyrsistent                0.18.1           py38h0a891b7_1    conda-forge
    pysocks                   1.7.1            py38h578d9bd_5    conda-forge
    pytest                    7.1.2                    pypi_0    pypi
    python                    3.8.13               h12debd9_0    defaults
    python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
    python-fastjsonschema     2.16.1             pyhd8ed1ab_0    conda-forge
    python_abi                3.8                      2_cp38    conda-forge
    pytorch                   1.9.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
    pytz                      2022.1             pyhd8ed1ab_0    conda-forge
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     19.0.2           py38ha71036d_2    conda-forge
    qt                        5.9.7                h5867ecd_1    defaults
    readline                  8.1.2                h7f8727e_1    defaults
    requests                  2.28.1             pyhd8ed1ab_0    conda-forge
    rich                      12.5.1                   pypi_0    pypi
    scipy                     1.7.3            py38hc147768_0    defaults
    seaborn                   0.11.2             pyhd3eb1b0_0    defaults
    send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
    setuptools                61.2.0           py38h06a4308_0    defaults
    sip                       4.19.13          py38h295c915_0    defaults
    six                       1.16.0             pyh6c4a22f_0    conda-forge
    sniffio                   1.2.0            py38h578d9bd_3    conda-forge
    soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
    sqlite                    3.38.5               hc218d9a_0    defaults
    stack_data                0.3.0              pyhd8ed1ab_0    conda-forge
    sympy                     1.10.1                   pypi_0    pypi
    tabulate                  0.8.10                   pypi_0    pypi
    terminado                 0.15.0           py38h578d9bd_0    conda-forge
    tinycss2                  1.1.1              pyhd8ed1ab_0    conda-forge
    tk                        8.6.12               h1ccaba5_0    defaults
    toml                      0.10.2                   pypi_0    pypi
    tomli                     2.0.1                    pypi_0    pypi
    torchaudio                0.9.0                      py38    pytorch
    torchvision               0.10.0               py38_cu111    pytorch
    tornado                   6.1              py38h27cfd23_0    defaults
    traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
    typing_extensions         4.3.0              pyha770c72_0    conda-forge
    urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
    wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
    webencodings              0.5.1                      py_1    conda-forge
    websocket-client          1.3.3              pyhd8ed1ab_0    conda-forge
    wheel                     0.37.1             pyhd3eb1b0_0    defaults
    xdoctest                  1.0.1                    pypi_0    pypi
    xz                        5.2.5                h7f8727e_1    defaults
    yapf                      0.32.0                   pypi_0    pypi
    zeromq                    4.3.4                h9c3ff4c_1    conda-forge
    zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
    zlib                      1.2.12               h7f8727e_2    defaults
    zstd                      1.4.9                ha95c52a_0    conda-forge
    

    编译日志,删除了一部分正常的:

    -- >>>>>>>>>>>>>
    -- TNN BUILD INFO:
    --      System: Android
    --      Processor: aarch64
    --      Cpu:    ON
    --      X86:    OFF
    --      Arm:    ON
    --      Arm82:  ON
    --      Metal:  OFF
    --      OpenCL: ON
    --      CUDA:   OFF
    --      DSP:    OFF
    --      Atlas:  OFF
    --      TensorRT:       OFF
    --      AppleNPU:       OFF
    --      HuaweiNPU:
    --      RKNPU:  OFF
    --      OpenVINO:       OFF
    --      OpenMP: ON
    --      TEST:   ON
    --      --Unit Test:    OFF
    --      Quantization:   OFF
    --      ModelCheck:     OFF
    --      DEBUG:
    --      PROFILE:        OFF
    --      BENCHMARK:      ON
    --      BENCHMARK Layer:        OFF
    --      Model Converter:        OFF
    --      ONNX2TNN Converter:     OFF
    --      TNN2MEM:        OFF
    --      BENCHMARK Test Lib:     ON
    --      Dynamic Range Quantization:     OFF
    -- Found PythonInterp: /home/qyz/miniconda3/envs/p1/bin/python (found version "3.8.13")
    Generate OpenCL Source done !!!
    
    -- Build TNN OpenCL
    -- Looking for C++ include unistd.h
    -- Looking for C++ include unistd.h - found
    -- Looking for C++ include stdint.h
    -- Looking for C++ include stdint.h - found
    -- Looking for C++ include inttypes.h
    -- Looking for C++ include inttypes.h - found
    -- Looking for C++ include sys/types.h
    -- Looking for C++ include sys/types.h - found
    -- Looking for C++ include sys/stat.h
    -- Looking for C++ include sys/stat.h - found
    -- Looking for C++ include fnmatch.h
    -- Looking for C++ include fnmatch.h - found
    -- Looking for C++ include stddef.h
    -- Looking for C++ include stddef.h - found
    -- Check size of uint32_t
    -- Check size of uint32_t - done
    -- Looking for strtoll
    -- Looking for strtoll - found
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /mnt/c/Users/ASUS/Desktop/TNN/benchmark/benchmark_android/build
    Scanning dependencies of target TNNArm82
    Scanning dependencies of target TNNArm
    
    [ 15%] Built target TNNArm82
    [ 15%] Building CXX object source/tnn/device/arm/CMakeFiles/TNNArm.dir/acc/arm_log_acc_layer_acc.cc.o
    [ 15%] Building CXX object source/tnn/device/opencl/CMakeFiles/TNNOpenCL.dir/acc/opencl_clip_layer_acc.cc.o
    [ 15%] Building CXX object source/tnn/device/cpu/CMakeFiles/TNNCpu.dir/acc/cpu_elu_layer_acc.cc.o
    [ 16%] Building CXX object source/tnn/device/arm/CMakeFiles/TNNArm.dir/acc/arm_log_sigmoid_layer_acc.cc.o
    /mnt/c/Users/ASUS/Desktop/TNN/source/tnn/device/opencl/acc/opencl_clip_layer_acc.cc:42:1: warning: non-void function does not return a value in all control paths [-Wreturn-type]
    }
    ^
    [ 16%] Building CXX object source/tnn/device/arm/CMakeFiles/TNNArm.dir/acc/arm_lstm_layer_acc.cc.o
    1 warning generated.
    
    [ 24%] Linking CXX static library libgflags_nothreads.a
    [ 24%] Built target gflags_nothreads_static
    
    [ 50%] Built target TNNArm
    
    [ 52%] Built target TNNCpu
    
    [ 53%] Built target TNNOpenCL
    
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/blob_dump_utils.cc.o
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/blob_memory_size_utils.cc.o
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/blob_transfer_utils.cc.o
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/cpu_info.cc.o
    /mnt/c/Users/ASUS/Desktop/TNN/source/tnn/utils/cpu_info.cc:618:9: warning: array designators are a C99 extension [-Wc99-designator]
            [cpuinfo_android_chipset_property_proc_cpuinfo_hardware] =
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1 warning generated.
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/cpu_utils.cc.o
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/data_flag_utils.cc.o
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/data_format_converter.cc.o
    [ 95%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/data_type_utils.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/detection_post_process_utils.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/dims_function_utils.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/dims_offset_utils.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/dims_vector_utils.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/exclusive_file.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/half_utils.cc.o
    [ 96%] Building CXX object CMakeFiles/TNN.dir/source/tnn/utils/mat_converter_acc.cc.o
    /mnt/c/Users/ASUS/Desktop/TNN/source/tnn/utils/exclusive_file.cc:181:47: warning: 'open' has superfluous mode bits; missing O_CREAT? [-Wuser-defined-warnings]
        int fd = open(fname, O_RDWR | O_EXCL, 0666);
                                                  ^
    /mnt/c/Users/ASUS/Desktop/Paper/Code/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/bits/fortify/fcntl.h:76:9: note: from 'diagnose_if' attribute on 'open':
            __clang_warning_if(!__open_modes_useful(flags) && modes,
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /mnt/c/Users/ASUS/Desktop/Paper/Code/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/sys/cdefs.h:139:54: note: expanded from macro '__clang_warning_if'
    #define __clang_warning_if(cond, msg) __attribute__((diagnose_if(cond, msg, "warning")))
                                                         ^           ~~~~
    1 warning generated.
    
    [ 98%] Linking CXX shared library libTNN.so
    [ 98%] Built target TNN
    [ 98%] Building CXX object test/CMakeFiles/TNNBenchmarkTest.dir/flags.cc.o
    [ 98%] Building CXX object test/CMakeFiles/TNNBenchmarkTest.dir/test.cc.o
    [ 99%] Building CXX object test/CMakeFiles/TNNBenchmarkTest.dir/test_utils.cc.o
    [ 99%] Building CXX object test/CMakeFiles/TNNTest.dir/flags.cc.o
    [100%] Building CXX object test/CMakeFiles/TNNBenchmarkTest.dir/timer.cc.o
    [100%] Building CXX object test/CMakeFiles/TNNTest.dir/test.cc.o
    [100%] Building CXX object test/CMakeFiles/TNNTest.dir/test_utils.cc.o
    [100%] Building CXX object test/CMakeFiles/TNNTest.dir/timer.cc.o
    [100%] Linking CXX shared library libTNNBenchmarkTest.so
    [100%] Linking CXX executable TNNTest
    [100%] Built target TNNBenchmarkTest
    [100%] Built target TNNTest
    CMake Warning (dev) in CMakeLists.txt:
      No project() command is present.  The top-level CMakeLists.txt file must
      contain a literal, direct call to the project() command.  Add a line of
      code such as
    
        project(ProjectName)
    
      near the top of the file, but after cmake_minimum_required().
    
      CMake is pretending there is a "project(Project)" command on the first
      line.
    This warning is for project developers.  Use -Wno-dev to suppress it.
    
    -- The C compiler identification is Clang 12.0.9
    -- The CXX compiler identification is Clang 12.0.9
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /mnt/c/Users/ASUS/Desktop/Paper/Code/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /mnt/c/Users/ASUS/Desktop/Paper/Code/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /mnt/c/Users/ASUS/Desktop/TNN/benchmark/benchmark_android/build/build_app
    [ 50%] Building CXX object CMakeFiles/tnn_wrapper.dir/src/main/jni/cc/benchmark_model_jni.cc.o
    [100%] Linking CXX shared library libtnn_wrapper.so
    [100%] Built target tnn_wrapper
    Performing Streamed Install
    Success
    /mnt/c/Users/ASUS/Desktop/TNN/benchmark/benchmark_android/...files pushed, 0 skipped. 1.6 MB/s (525286 bytes in 0.309s)
    ./build_app/libtnn_wrapper.so: 1 file pushed, 0 skipped. 14.4 MB/s (410776 bytes in 0.027s)
    ./libTNN.so: 1 file pushed, 0 skipped. 80.9 MB/s (10881800 bytes in 0.128s)
    ./test/libTNNBenchmarkTest.so: 1 file pushed, 0 skipped. 20.8 MB/s (413080 bytes in 0.019s)
    
    benchmark device: ARM
    
    Stopping: com.tencent.tnn.benchmark
    Starting: Intent { cmp=com.tencent.tnn.benchmark/.MainActivity (has extras) }
    Status: timeout
    LaunchState: UNKNOWN (-1)
    Activity: com.tencent.tnn.benchmark/.MainActivity
    WaitTime: 10092
    Complete
    09-01 12:46:42.874 26746 26746 I tnn : void tnn::test::Timer::Print() [File test/timer.cc][Line 60] densenet.tnnproto - ARM TNN Benchmark time cost: min = 525.659 ms | max = 555.620 ms | avg = 531.057 ms
    Stopping: com.tencent.tnn.benchmark
    Starting: Intent { cmp=com.tencent.tnn.benchmark/.MainActivity (has extras) }
    Status: ok
    LaunchState: COLD
    Activity: com.tencent.tnn.benchmark/.MainActivity
    TotalTime: 1373
    WaitTime: 1376
    Complete
    09-01 12:46:50.199 26927 26927 I tnn : void tnn::test::Timer::Print() [File test/timer.cc][Line 60] squeezenet_v1.1.tnnproto - ARM TNN Benchmark time cost: min = 43.554 ms | max = 44.123 ms | avg = 43.744 ms
    Stopping: com.tencent.tnn.benchmark
    Starting: Intent { cmp=com.tencent.tnn.benchmark/.MainActivity (has extras) }
    Status: ok
    LaunchState: COLD
    Activity: com.tencent.tnn.benchmark/.MainActivity
    TotalTime: 357
    WaitTime: 360
    Complete
    09-01 12:46:57.539 27024 27024 I tnn : topformer_t_official2.tnnproto TNN Benchmark time cost failed error code: 4096
    Stopping: com.tencent.tnn.benchmark
    Starting: Intent { cmp=com.tencent.tnn.benchmark/.MainActivity (has extras) }
    Status: ok
    LaunchState: UNKNOWN (0)
    Activity: com.tencent.tnn.benchmark/.MainActivity
    WaitTime: 588
    Complete
    
  • FileNotFoundError

    FileNotFoundError

    Hello, developer, when I try to run in mmsegmentation environment, I get the following error FileNotFoundError: modelzoos/classification/topformer-T-224-66.2.pth can not be found. where should I get this file? topformer-T-224-66.2.pth

  • is there any ImageNet pre-training config of Topformer?

    is there any ImageNet pre-training config of Topformer?

    I want to change some modules when using Topformer to do classification on ImageNet. Could you please provide the ImageNet Pre-training config? Thanks!

  • how to modify the batch size during inference?

    how to modify the batch size during inference?

    I tried to modify the value samples_per_gpu from 2 to 1 in config file, but the elapsed time in inference log seems to change not much. Do I have something wrong?

  • AttributeError: 'ConfigDict' object has no attribute 'dist_params'

    AttributeError: 'ConfigDict' object has no attribute 'dist_params'

    Hello, Thanks for providing such a great repo. But I encountered an error when run bash tools/dist_train.sh local_configs/topformer/topformer_base.py 8 --work-dir /path/to/save/checkpoint Details are as follows: image

  • 对比实验

    对比实验

    请问论文Table 1里面的对比方法全都是用MMSegmentation做的吗,我想跑一下对比方法但是我发现好多模型没有找到,比如DeepLabV3+ EfficientNet、DeepLabV3+ ShuffleNetV2-1.5x、Semantic FPN ConvMLP-S等,configs里面没有相关的代码。

  • The validation set result is 0

    The validation set result is 0

    Hi, thanks for your excellent work. When I used toformer to train the ADE20K dataset, the result of the validation set appeared 0. Have you ever encountered this situation? 337ba331169f76b1985b94671ea2f86 This is the command I use bash tools/dist_train.sh local_configs/topformer/topformer_tiny_448x448_160k_2x8_ade20k.py 1 look forward to you reply.

  • size of output

    size of output

    Hello, As shown in figure 2, 1/4 scale of input is present in the output. so for 448448 input, there is a 5656 image in the output!!! what is the point that I can't see?

  • Deploy test error with onnx backend

    Deploy test error with onnx backend

    Hello,

    Thanks for sharing this great repo. I trained and test with my own dataset, the prediction results are quite good.

    But when I exported the onnx file with command:

    python3 tools/convert2onnx.py  local_configs/topformer/topformer_tiny_512x512_80k_2x8_drive_inviol1k.py --input-img results/0109.png --shape 512 512 --checkpoint results/tiny_20k/latest.pth --output-file results/tiny_80k/tiny_512_512.onnx --show
    

    then try the deploy script with command: python tools/deploy_test.py local_configs/topformer/topformer_tiny_512x512_80k_2x8_drive_inviol1k.py results/tiny_80k/tiny_512_512.onnx --backend=onnxruntime --show

    Then it show error:

    File "tools/deploy_test.py", line 297, in main() File "tools/deploy_test.py", line 268, in main results = single_gpu_test( File "/home/yidong/anaconda3/envs/mask2former/lib/python3.8/site-packages/mmsegmentation-0.19.0-py3.8.egg/mmseg/apis/test.py", line 91, in single_gpu_test result = model(return_loss=False, **data) File "/home/yidong/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/yidong/anaconda3/envs/mask2former/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 42, in forward return super().forward(*inputs, **kwargs) File "/home/yidong/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/yidong/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/yidong/anaconda3/envs/mask2former/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func return old_func(*args, **kwargs) File "/home/yidong/anaconda3/envs/mask2former/lib/python3.8/site-packages/mmsegmentation-0.19.0-py3.8.egg/mmseg/models/segmentors/base.py", line 110, in forward return self.forward_test(img, img_metas, **kwargs) File "/home/yidong/anaconda3/envs/mask2former/lib/python3.8/site-packages/mmsegmentation-0.19.0-py3.8.egg/mmseg/models/segmentors/base.py", line 92, in forward_test return self.simple_test(imgs[0], img_metas[0], **kwargs) File "tools/deploy_test.py", line 84, in simple_test self.sess.run_with_iobinding(self.io_binding) File "/home/yidong/anaconda3/envs/mask2former/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 276, in run_with_iobinding self._sess.run_with_iobinding(iobinding._iobinding, run_options) RuntimeError: Error in execution: Got invalid dimensions for input: input for the following indices index: 2 Got: 1080 Expected: 512 index: 3 Got: 1878 Expected: 512 Please fix either the inputs or the model.

    Does it have any requirements for input images or shapes when export onnx file? the config setting as: img_scale = (1920, 1080) , crop_size = (512, 512).

    I also tried some other shape size, still confused with this error and any helps would be much appreciated.

  • What's SASE

    What's SASE

    Excellent work.In your paper you compare SASE with FFN and ASPP , I read the code also, but I can't find out what is sase, could you please tell me what's SASE or the related paper

  • onnx to tensorrt

    onnx to tensorrt

    Hi, I used this command to get tmp.onnx. python3 tools/pytorch2onnx.py \ ./local_configs/topformer/topformer_tiny_448x448_160k_2x8_ade20k.py \ --input-img ./imgs/999438_0_B_0007_7680-3840-11776-7936.jpg \ --shape 2048 2048 \ --checkpoint ./work_dirs/topformer_tiny_448x448_160k_2x8_ade20k/latest.pth

    then I got an error below when I try to convert tmp.onnx to tensorrt assert is_tensorrt_plugin_loaded(), 'TensorRT plugin should be compiled.' AssertionError: TensorRT plugin should be compiled.

    my environment configuration is cuda : 10.1 tensorrt:8.4.0.6 I also try cuda 11.3, but get the same error,have you tried switching to tensorrt? what is the difference between this and the operation of mmsegmentation docs? https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/useful_tools.md

    look forward to your reply.

  • Some training problems on 1 gpu

    Some training problems on 1 gpu

    I want to train on 1 gpu without using distributed training, so i ran

    python tools/train.py local_configs/topformer/topformer_small_512x512_160k_2x8_ade20k.py 1 --work-dir runs/

    and added

    import sys
    sys.path.append("/home/xx/TopFormer-main")
    

    to train.py Finally i got this:

    assert hasattr(ext, fun), f'{fun} miss in module {name}' AssertionError: ball_query_forward miss in module _ext

    How can i fix the problem? 我重装了1.3.14版本的mmcv解决了上述问题,但是我在使用自己的ade20k数据的时候出现了一个bug

    2022-10-31 18:18:14,190 - mmseg - INFO - Loaded 22945 images
    fatal: not a git repository (or any of the parent directories): .git
    Traceback (most recent call last):
      File "tools/train.py", line 183, in <module>
        main()
      File "tools/train.py", line 179, in main
        meta=meta)
      File "/home/xx/TopFormer-main/mmseg/apis/train.py", line 88, in train_segmentor
        drop_last=True) for ds in dataset
      File "/home/xx/TopFormer-main/mmseg/apis/train.py", line 88, in <listcomp>
        drop_last=True) for ds in dataset
    TypeError: object of type 'int' has no len()
    
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

  • Add the onnxsim support

    Add the onnxsim support

    Hi, onnxsim is a wonderfull onnx graph optimizer. I think it may be useful for the deploy of topformer! It could remove some unuseful ops in onnx graph and sometimes accelerate the computing. After the simplifying, the model could support more deploy ways, like ncnn, etc. image

  • Some questions

    Some questions

    Hi!Excuse me ,I have some question to ask you ?

    1. Can I run the program in WIN10?
    2. I try it but it have a KeyError: ‘Topformer is not in the models registry’ Looking forward to your reply
[CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation
[CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation

RCIL [CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation Chang-Bin Zhang1, Jia-Wen Xiao1, Xialei Liu1, Ying-Cong Chen2

Nov 22, 2022
CVPR2022 (Oral) - Rethinking Semantic Segmentation: A Prototype View
CVPR2022 (Oral) - Rethinking Semantic Segmentation: A Prototype View

Rethinking Semantic Segmentation: A Prototype View Rethinking Semantic Segmentation: A Prototype View, Tianfei Zhou, Wenguan Wang, Ender Konukoglu and

Dec 1, 2022
Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019)
Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019)

Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019) Introduction Official implementation of Adaptive Pyramid Context Network

Nov 9, 2022
Code for the CVPR2022 paper "Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity"
Code for the CVPR2022 paper

Introduction This is an official release of the paper "Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity" (arxiv link). Abstrac

Nov 23, 2022
Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding (CVPR2022)
Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding (CVPR2022)

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding by Qiaole Dong*, Chenjie Cao*, Yanwei Fu Paper and Supple

Nov 25, 2022
Video Frame Interpolation with Transformer (CVPR2022)

VFIformer Official PyTorch implementation of our CVPR2022 paper Video Frame Interpolation with Transformer Dependencies python >= 3.8 pytorch >= 1.8.0

Nov 30, 2022
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)
 Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

Sep 21, 2022
nnFormer: Interleaved Transformer for Volumetric Segmentation Code for paper "nnFormer: Interleaved Transformer for Volumetric Segmentation "

nnFormer: Interleaved Transformer for Volumetric Segmentation Code for paper "nnFormer: Interleaved Transformer for Volumetric Segmentation ". Please

Nov 30, 2022
Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks
 Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks

This is the code associated with the paper Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks, published at CVPR 2020.

Dec 4, 2022
The official implementation of paper Siamese Transformer Pyramid Networks for Real-Time UAV Tracking, accepted by WACV22
The official implementation of paper Siamese Transformer Pyramid Networks for Real-Time UAV Tracking, accepted by WACV22

SiamTPN Introduction This is the official implementation of the SiamTPN (WACV2022). The tracker intergrates pyramid feature network and transformer in

Nov 25, 2022
(IEEE TIP 2021) Regularized Densely-connected Pyramid Network for Salient Instance Segmentation

RDPNet IEEE TIP 2021: Regularized Densely-connected Pyramid Network for Salient Instance Segmentation PyTorch training and testing code are available.

Oct 21, 2022
Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers (arXiv2021)
Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers (arXiv2021)

Polyp-PVT by Bo Dong, Wenhai Wang, Deng-Ping Fan, Jinpeng Li, Huazhu Fu, & Ling Shao. This repo is the official implementation of "Polyp-PVT: Polyp Se

Dec 3, 2022
Unofficial PyTorch implementation of MobileViT based on paper "MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer".
Unofficial PyTorch implementation of MobileViT based on paper

MobileViT RegNet Unofficial PyTorch implementation of MobileViT based on paper MOBILEVIT: LIGHT-WEIGHT, GENERAL-PURPOSE, AND MOBILE-FRIENDLY VISION TR

Dec 2, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Nov 30, 2022
Code for Boundary-Aware Segmentation Network for Mobile and Web Applications

BASNet Boundary-Aware Segmentation Network for Mobile and Web Applications This repository contain implementation of BASNet in tensorflow/keras. comme

Nov 24, 2022
This is an official implementation of the CVPR2022 paper "Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots".

Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots Blind2Unblind Citing Blind2Unblind @inproceedings{wang2022blind2unblind, tit

Dec 4, 2022
PSTR: End-to-End One-Step Person Search With Transformers (CVPR2022)
PSTR: End-to-End One-Step Person Search With Transformers (CVPR2022)

PSTR (CVPR2022) This code is an official implementation of "PSTR: End-to-End One-Step Person Search With Transformers (CVPR2022)". End-to-end one-step

Nov 22, 2022
CVPR2022 paper "Dense Learning based Semi-Supervised Object Detection"
CVPR2022 paper

[CVPR2022] DSL: Dense Learning based Semi-Supervised Object Detection DSL is the first work on Anchor-Free detector for Semi-Supervised Object Detecti

Nov 30, 2022
[CVPR2022] Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos
[CVPR2022] Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos

Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos Created by Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie

Nov 22, 2022