This repository contains the official implementation of the 3D-RFT Perception track, including reinforcement fine-tuning for 3D visual grounding on ScanRefer and 3D object detection on ScanNet.

Choose the codebase according to the task family.

The reference environment uses Python 3.11, CUDA 12.4, and GCC 11.3 or newer. Create the environment from the repository root:

git clone https://github.com/3D-RFT/3D-RFT-Perception.git
cd 3D-RFT-Perception

conda create -n 3d-rft python=3.11 -y
conda activate 3d-rft
bash setup.sh

The setup script installs the local Qwen-VL utilities with Decord video support.

The training code was developed with the following key package versions:

• vllm==0.7.2
• trl==0.16.0
• flash-attn==2.5.5

Qwen2.5-VL is sensitive to the Transformers version. Download the compatible source archive from Google Drive, then install it into the environment:

unzip transformers-main.zip
pip install ./transformers-main

The VGGT-based training scripts additionally require PyTorch3D. Install a PyTorch3D build compatible with the PyTorch and CUDA versions in your environment.

All processed data, pretrained weights, and released checkpoints will be hosted in the 3D-RFT Hugging Face repository. After the assets are released, download them with:

hf download EricLHK/3D-RFT --local-dir 3D-RFT-assets

Prepare the training annotations and the image/video data referenced by them. The released scripts expect the following annotation files:

3D-RFT-data/
├── ScanRefer_train_64.78K.json
├── ScanNetDet_train_144.16K.json
└── <images and frames referenced by the JSON files>

Place the pretrained weights in the following default locations:

pretrained_weights/
├── Qwen2.5-VL-3B-Instruct/
├── Qwen2.5-VL-3B-Instruct-ScanRefer-Scan2Cap-ScanNet-Det-Template-VGGT/
└── VGGT-1B/

If the Hugging Face assets are kept under 3D-RFT-assets/, point the scripts to them without moving any files:

export DATA_ROOT="$PWD/3D-RFT-assets/3D-RFT-data"
export MODEL_ROOT="$PWD/3D-RFT-assets/pretrained_weights"

The scripts use repository-relative paths by default. You can override them without editing a script by exporting any of the following variables:

• PROJECT_ROOT
• DATA_ROOT
• MODEL_ROOT, MODEL_PATH, and VGGT_PATH
• OUTPUT_ROOT
• DEEPSPEED_CONFIG
• RESUME_FROM_CHECKPOINT (optional; unset by default)

For example, resume a job from a repository-relative checkpoint with:

RESUME_FROM_CHECKPOINT=logs/ScanNetDet-3D-RFT/checkpoint-6000 \
sbatch src/scripts/Sulrm/RL/VGGT/ScanNetDet/run_grpo_slurm_ScanNetDet_standalone.slurm

Submit Slurm jobs from the repository root. The single-GPU debug scripts resolve the repository root from their own locations.

All runnable training scripts are located in src/scripts. The .slurm files are multi-GPU Slurm jobs, while the *_debug.sh files run directly on one GPU for debugging.

Run the standard four-GPU GRPO training job:

sbatch src/scripts/Sulrm/RL/ScanRefer/run_grpo_slurm_ScanRefer.sh

Run 3D-RFT with VGGT and the 3D bounding-box reward on eight GPUs:

sbatch src/scripts/Sulrm/RL/VGGT/ScanRefer/run_qwen2.5-vl-instruct_grpo_pretrained_bbox3d_iou_reward_8GPU_optimize_vgllm_beta_0.04.slurm

For a single-GPU debug run:

bash src/scripts/Sulrm/RL/VGGT/ScanRefer/run_qwen2.5-vl-instruct_grpo_pretrained_bbox3d_iou_reward_8GPU_optimize_vgllm_beta_0.04_debug.sh

Run the eight-GPU 3D-RFT training job:

sbatch src/scripts/Sulrm/RL/VGGT/ScanNetDet/run_grpo_slurm_ScanNetDet_standalone.slurm

For a single-GPU debug run:

bash src/scripts/Sulrm/RL/VGGT/ScanNetDet/run_grpo_slurm_ScanNetDet_standalone_debug.sh

The scripts use a per-device batch size of 1. Adjust --gradient_accumulation_steps, --num_generations, and the Slurm resource requests according to the available GPUs and memory.

If you find this project useful, please cite our paper:

@inproceedings{linghu20263d,
  title={3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding},
  author={Linghu, Xiongkun and Huang, Jiangyong and Jia, Baoxiong and Huang, Siyuan},
  booktitle={International Conference on Machine Learning},
  year={2026}
}

This codebase is developed based on Video-R1. We thank the authors for releasing their code.