MBW: Multiview Bootstrapping in the Wild (NeurIPS 2022)

Paper | Project page | MBW-Data | Pretrained models

Multiview Bootstrapping in the Wild (MBW) provides a powerful way of generating labeled data in the wild at scale. Thereby, it democratizes the domain of data collection to a plethora of machine learning-driven computer vision applications via its novel self-supervision technique.

 

Requirements (if using GPU)

• Tested in Pytorch 1.11, with CUDA 11.4

 

Setup

1. Create a conda environment and activate it.

   conda env create -f environment_<cpu/gpu>.yml (change the flag within <> based on the available system)
   conda activate mbw
   pip install opencv-python
   

2. Please do a clean install of the submodule robust_loss_pytorch:

   cd modules/helpers/robust_loss_pytorch
   pip install git+https://github.com/jonbarron/robust_loss_pytorch
   

3. Please do a clean install of the submodule torch_batch_svd: (if using GPU)

   cd modules/helpers/torch-batch-svd
   export CUDA_HOME=/your/cuda/home/directory/    
   python setup.py install
   

 

Data & Pre-trained models

1. Fetch the pre-trained flow and detector models from Zenodo using:

   zenodo_get 10.5281/zenodo.7054596
   unzip models.zip
   rm -rf models.zip && rm -rf md5sums.txt
   

2. Download the data from and unzip it in the data directory.

   zenodo_get 10.5281/zenodo.7058567
   unzip data.zip
   rm -rf data.zip && rm -rf md5sums.txt
   

   The final directory after retrieving pre-trained models and sample data should look like this:

   ${mbw}
    `-- data
        `-- Chimpanzee
            |-- annot/
            |-- images/
        
    `-- models
        |-- detector/
        |-- flow/
        |-- mvnrsfm/
   

 

Run unit tests

./scripts/unit_tests.sh

Training (Generate labels from MBW)

./scripts/train.sh

Evaluation and visualization

./scripts/eval.sh
./scripts/visualize.sh    

Jupyter notebook example

Citation

If you use our code, dataset, or models in your research, please cite with:

@inproceedings{dabhi2022mbw,
	title={MBW: Multi-view Bootstrapping in the Wild},
	author={Dabhi, Mosam and Wang, Chaoyang and Clifford, Tim and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
	booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
	year={2022},
	ee = {https://openreview.net/forum?id=i1bFPSw42W0},
	organization={NeurIPS}
}


@inproceedings{dabhi2021mvnrsfm,
	title={High Fidelity 3D Reconstructions with Limited Physical Views},
	author={Dabhi, Mosam and Wang, Chaoyang and Saluja, Kunal and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
	booktitle={2021 International Conference on 3D Vision (3DV)},
	year={2021},
	ee = {https://ieeexplore.ieee.org/abstract/document/9665845},
	organization={IEEE}
}