DistributedManager initialization fails on CPU-only hosts #1936

Version

2.1.1 (also reproduces on current main)

On which installation method(s) does this occur?

Pip

Describe the issue

DistributedManager cannot initialize on CPU-only hosts (e.g. laptop-scale gloo runs, CPU CI):

1. DistributedManager.setup() passes manager.device as the device_id argument of torch.distributed.init_process_group. On CPU-only hosts that is torch.device("cpu"), which recent PyTorch versions reject with ValueError: init_process_group device_id parameter must be an accelerator with an index.
2. When local_rank is None, setup() computes rank % torch.cuda.device_count(), a division by zero when no accelerator is present (hit via the SLURM/OpenMPI initialization paths, which pass local_rank=None).
3. As a follow-on, the failed initialization leaves _shared_state["_distributed"] = True, so the atexit cleanup() calls dist.destroy_process_group() on a process group that was never created and dies with AssertionError: Process group cannot be None, obscuring the original error.

Note the distributed manager tests are skipped on CPU by an autouse fixture, so CI does not catch this path.

Minimum reproducible example

# repro.py — run on a CPU-only machine:
#   torchrun --standalone --nproc_per_node=2 repro.py
import torch
from physicsnemo.distributed import DistributedManager

DistributedManager.initialize()
dm = DistributedManager()
t = torch.tensor([dm.rank + 1.0])
torch.distributed.broadcast(t, src=0)
DistributedManager.cleanup()

Relevant log output

  File ".../physicsnemo/distributed/manager.py", line 590, in setup
    dist.init_process_group(
  File ".../torch/distributed/distributed_c10d.py", line 1763, in init_process_group
    raise ValueError(
ValueError: init_process_group device_id parameter must be an accelerator with an index

Exception ignored in atexit callback: <staticmethod(<function DistributedManager.cleanup ...)>
  File ".../physicsnemo/distributed/manager.py", line 823, in cleanup
    dist.destroy_process_group()
AssertionError: Process group cannot be None

Environment details

nvidia-physicsnemo 2.1.1, torch 2.13.0, Python 3.12, macOS (arm64) and Linux CPU nodes; gloo backend.