Reading the repo
Opening the box…
Reading the repo
Opening the box…
How to install on Windows
RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs
Saved setup from 2 h ago
Will it work on Windows?
The maintainers included install.sh. Run it from the cloned folder after reading it.
bash install.sh
docs/installation.mdgit clone https://github.com/Maelic/RelateAnything
cd RelateAnything
./install.sh
install.sh creates .venv, installs the package editable, and warms the
Hugging Face cache into ./.hf_cache so later runs can go fully offline. It is
idempotent — re-running it is safe.
Knobs:
PYTHON_BIN=python3.13./install.sh # pick the interpreter
VENV_DIR=/scratch/env./install.sh # put the venv elsewhere
SKIP_HF_WARMUP=1./install.sh # no network at install time
The base install is enough to train and evaluate. Everything else is opt-in:
pip install -e ".[deploy]" # onnxruntime + onnx + full opencv — the demo
pip install -e ".[tensorrt]" # native TensorRT 10 on NVIDIA GPUs; see deployment guide
pip install -e ".[detector]" # ultralytics (AGPL-3.0) — rebuild demo detectors
pip install -e ".[datagen]" # vLLM annotation pipeline — heavy, GPU only
pip install -e ".[monitoring]" # wandb + matplotlib
pip install -e ".[hub]" # huggingface_hub — download weights and packs
pip install -e ".[train]" # scikit-learn + pyarrow — training and the RA-4M image export
pip install -e ".[gradio]" # the server-side Gradio demo (needs [detector] too)
pip install -e ".[release]" # huggingface_hub — publishing
pip install -e ".[dev]" # pytest, black, isort
datagen is deliberately not a core dependency: it pulls autoawq, which
compiles, and nothing in training or inference needs it.
Python ≥ 3.12 (numpy 2.5 requires it); the released checkpoints were trained and verified on 3.13.
Dependencies are pinned with ~= in pyproject.toml
against that verified environment — loosening a pin is fine for
experimentation, but the pins document what is known to reproduce.
A note on pycocotools: under plain pip this resolves to the upstream
package. The project uses a fork, which only uv picks up automatically. With
plain pip:
pip install git+https://github.com/Maelic/pycocotools
Three models, one repository each on the Hugging Face Hub, all under maelic/:
relsgg-vits16, relsgg-vits16plus (the recommended default) and
relsgg-vitb16. Each carries model.pth (torch, EMA weights, backbone config
embedded), text_student.pt (the distilled predicate text encoder),
predicate_embeddings.npz (the training vocabulary already encoded with that
student) and its calibrated predicate_bank.npz. relsgg-vits16plus also
carries the ONNX graph the torch-free demo runs on.
pip install -e ".[hub]"
python - <<'PY'
from huggingface_hub import snapshot_download
print(snapshot_download("maelic/relsgg-vits16plus")) # -> local directory with model.pth
PY
Running a released checkpoint needs no gated download: model.pth embeds
the backbone configuration and all weights. The gated DINOv3 repositories are
only needed to train.
pip install -e ".[train,hub]"
python training/setup_data.py # to train: packs, text space, ~52 GB of images
python benchmark/setup_data.py # to evaluate: OV-SGG-Bench and its ~15 GB of test images
Both download from the Hugging Face Hub, keep only the images the packs name, write
them to $RA_DATASETS (default datasets/), link the packs into runs/packed and
runs/datamix, point every pack at $RA_DATASETS and check that each image is
there. A rerun resumes.
| what | where |
|---|---|
| training packs: megasg, megasg_clean, vg_raw, megasg_proxy50k | maelic/RA-4M packs/ |
| evaluation packs, hicodet, negatives, text student, text space, shared detections | maelic/OV-SGG-Bench |
| MegaSG images | maelic/RA-4M (also release/ra4m/export_images.py) |
| Visual Genome images (vg_raw, VG150), PSG, IndoorVG | maelic/VG150-coco-format, maelic/PSG-coco-format, maelic/IndoorVG-coco-format |
| HICO-DET images | maelic/HICO-DET-coco-format |
| Haystack, SpatialSense images (evaluation only) | maelic/Haystack-coco-format, maelic/SpatialSense-coco-format: mirrors of the authors' releases (SA-1B subset, SA-1B Dataset Research License; Zenodo 8104370, CC BY 2.0) |
training/setup_data.py --proxy takes the 50,000-image development pack
(megasg_proxy50k, 6 GB of MegaSG images instead of 43 GB) for runs on one GPU.
A pack records the directory of its images (meta.json: img_dir); to move the
images later, rerun the setup or training/relocate_packs.py. If the recorded
directory does not exist and RA_DATASETS is set, the loaders look for the images
at the same place under RA_DATASETS and warn, naming both paths, if they are not there.
DINOv3 weights live in gated Hugging Face repositories
(facebook/dinov3-vits16-pretrain-lvd1689m, -vits16plus-, -vitb16-). Two ways
through:
Accept the terms and log in (needed once):
huggingface-cli login # account that accepted Meta's DINOv3 terms
Or convert a local copy of Meta's raw checkpoints and point
--backbone_model at the resulting directory:
python training/convert_dinov3_local.py --help # ViT towers
python training/convert_dinov3_convnext.py --help # ConvNeXt towers
If neither is available, install.sh still succeeds; it prints a warning and
you get the failure later, at model construction, with a clearer message.
Once the cache is warm, nothing needs the network:
export HF_HOME="$PWD/.hf_cache"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
Compute nodes are usually offline. Warm the cache on a login node, then export the offline flags in the job:
source.venv/bin/activate
export HF_HOME="$PWD/.hf_cache"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export WANDB_MODE=offline
export RA_DATASETS=/path/to/datasets
Two conventions worth copying:
/tmp. On most clusters /tmp is node-local,
so a job that logs there silently discards its output.pytest tests/ -q
These are CPU-only and need no weights and no network — the same suite CI runs. They cover the install path, imports, evaluator math, score parity between the training and deploy paths, and edge decoding.
Then run powershell -ExecutionPolicy Bypass -File .\setup.ps1. It stops at the first error and is safe to run again. Servers and tests aren't started; it lists them at the end. Read it first.
Needs Git: winget install Git.Git
git clone https://github.com/Maelic/RelateAnything.git cd RelateAnything
Python >=3.12
winget install Python.Python.3.13
python -m venv .venv .venv\Scripts\Activate.ps1 pip install -e .
pytest
Use Python 3.12+. Install a CUDA-enabled PyTorch build for NVIDIA GPU
inference, or use device="cpu" in the example below.
git clone https://github.com/Maelic/RelateAnything.git
cd RelateAnything
pip install -e ".[hub,detector]" # detector: ultralytics YOLO, used below
Run this from the repository root. A detector finds the objects in the included sample photo; RelateAnything predicts the relations between them.
from PIL import Image
from ultralytics import YOLO
from relsgg import RelateAnything
image = Image.open("assets/reel/images/horse.jpg").convert("RGB")
# 1. Find the objects. Any detector or segmenter works; this one is YOLO26.
detections = YOLO("yolo26m.pt")(image, conf=0.4, verbose=False)[0]
boxes = detections.boxes.xyxy.cpu().numpy() # [N, 4]: x1, y1, x2, y2 in pixels
labels = [detections.names[int(c)] for c in detections.boxes.cls] # for display only
# 2. Predict the relations between them.
model = RelateAnything.from_pretrained("maelic/relsgg-vits16plus", device="cuda")
for triplet in model.predict(image, boxes, box_labels=labels, topk=3):
print(triplet)
# (person) --riding [0.76]--> (horse)
# (horse) --carrying [0.51]--> (person)
# (horse) --walking past [0.43]--> (car)
# 3. Change the relations to look for, without retraining.
model.set_vocabulary(["riding", "carrying a rider", "beside", "in front of"])
graphs = model.predict(image, boxes, box_labels=labels, decompose=True, topk=3)
print(graphs["spatial"]) # where things are, e.g. (horse) --in front of--> (car)
print(graphs["semantic"]) # what they do, e.g. (person) --riding--> (horse)
The relation model reads the image and the boxes; the detector's labels are
never an input. Boxes can come from any detector, a segmenter (pass masks= as
well) or a person: an [N, 4] array in original-image pixels is all it needs.
Released checkpoints include the backbone configuration and text encoder;
no gated DINOv3 login is needed for inference. The detector extra
installs ultralytics, which is
AGPL-3.0 and downloads the YOLO26 weights on first use.
API guide: inputs, vocabularies, masks, and scores · Installation and offline use
No reports for Windows yet.
GitManual AI Agent reads the 4 key files and writes a step-by-step guide for Windows. Once written, it's saved for everyone.