Code for 3D-RFT reasoning tasks
- Clone Github repo
git clone git@github.com:3D-RFT/3D-RFT-Reasoning.git
cd 3D-RFT-Reasoning- Create
condaenvironment and install dependencies
conda create -n 3d-rft-reasoning python=3.9
conda activate 3d-rft-reasoning
# install PyTorch, take our version for example
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=11.8 -c pytorch -c nvidia
# install other dependencies with pip
pip install -r requirements.txt
# install flash attention, recommend cuda>=11.8
pip install flash-attn==2.7.4.post1 --no-build-isolationData includes scene videos (sceneverse_base) and annotations (annotation_base). The data structure is as follows:
├── ${sceneverse_base}
├── ScanNet
│ └── ...
├── ScanNetPP
│ └── ...
├── ARKitScenes
│ └── ...
└── ...
├── ${annotation_base}
└── qa
└── ...
The pretrained weights include VLMs (Qwen2.5-VL-3B and Qwen2.5-VL-7B), and VGGT-1B for geometry prior.
We also release our model checkpoints, including both before RL and after RL.
Update configs:
configs/data/default.yaml:sceneverse_baseandannotation_baseconfigs/llm/*.yaml:cfg_pathconfigs/vision3d/vggt.yaml:ckpt_pathconfigs/default.yaml:base_dirandlogger.entity
Launch training:
python launch.py --mode accelerate \
--num_nodes 1 \
--gpu_per_node 8 \
--strategy ds \ # ds | ddp
note=grpo_3b_vggt \ # exp name
llm=qwen2.5-3b \ # 3b | 7b
task=grpo \ # sft | cold_start | grpo
vision3d=vggt \ # vggt | dummy
pretrained_ckpt_path=${pretrained_ckpt_path} # required except sftFor evaluation: add an argument mode=test and specify pretrained_ckpt_path
We manually modify some methods of accelerate.Accelerator in common/misc.py, including gather_for_metrics (fix gathering non-tensor objects), get_state_dict (only save learnable parameters when calling save_state).