Reading the repo
Opening the box…
Reading the repo
Opening the box…
How to install on Linux
RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs
Saved setup from 2 h ago
Will it work on Linux?
The maintainers included install.sh. Run it from the cloned folder after reading it.
bash install.sh
docs/installation.mdgit clone https://github.com/Maelic/RelateAnything
cd RelateAnything
./install.sh
install.sh creates .venv, installs the package editable, and warms the
Hugging Face cache into ./.hf_cache so later runs can go fully offline. It is
idempotent — re-running it is safe.
Knobs:
PYTHON_BIN=python3.13./install.sh # pick the interpreter
VENV_DIR=/scratch/env./install.sh # put the venv elsewhere
SKIP_HF_WARMUP=1./install.sh # no network at install time
The base install is enough to train and evaluate. Everything else is opt-in:
pip install -e ".[deploy]" # onnxruntime + onnx + full opencv — the demo
pip install -e ".[tensorrt]" # native TensorRT 10 on NVIDIA GPUs; see deployment guide
pip install -e ".[detector]" # ultralytics (AGPL-3.0) — rebuild demo detectors
pip install -e ".[datagen]" # vLLM annotation pipeline — heavy, GPU only
pip install -e ".[monitoring]" # wandb + matplotlib
pip install -e ".[hub]" # huggingface_hub — download weights and packs
pip install -e ".[train]" # scikit-learn + pyarrow — training and the RA-4M image export
pip install -e ".[gradio]" # the server-side Gradio demo (needs [detector] too)
pip install -e ".[release]" # huggingface_hub — publishing
pip install -e ".[dev]" # pytest, black, isort
datagen is deliberately not a core dependency: it pulls autoawq, which
compiles, and nothing in training or inference needs it.
Python ≥ 3.12 (numpy 2.5 requires it); the released checkpoints were trained and verified on 3.13.
Dependencies are pinned with ~= in pyproject.toml
against that verified environment — loosening a pin is fine for
experimentation, but the pins document what is known to reproduce.
A note on pycocotools: under plain pip this resolves to the upstream
package. The project uses a fork, which only uv picks up automatically. With
plain pip:
pip install git+https://github.com/Maelic/pycocotools
Three models, one repository each on the Hugging Face Hub, all under maelic/:
relsgg-vits16, relsgg-vits16plus (the recommended default) and
relsgg-vitb16. Each carries model.pth (torch, EMA weights, backbone config
embedded), text_student.pt (the distilled predicate text encoder),
predicate_embeddings.npz (the training vocabulary already encoded with that
student) and its calibrated predicate_bank.npz. relsgg-vits16plus also
carries the ONNX graph the torch-free demo runs on.
pip install -e ".[hub]"
python - <<'PY'
from huggingface_hub import snapshot_download
print(snapshot_download("maelic/relsgg-vits16plus")) # -> local directory with model.pth
PY
Running a released checkpoint needs no gated download: model.pth embeds
the backbone configuration and all weights. The gated DINOv3 repositories are
only needed to train.
pip install -e ".[train,hub]"
python training/setup_data.py # to train: packs, text space, ~52 GB of images
python benchmark/setup_data.py # to evaluate: OV-SGG-Bench and its ~15 GB of test images
Both download from the Hugging Face Hub, keep only the images the packs name, write
them to $RA_DATASETS (default datasets/), link the packs into runs/packed and
runs/datamix, point every pack at $RA_DATASETS and check that each image is
there. A rerun resumes.
| what | where |
|---|---|
| training packs: megasg, megasg_clean, vg_raw, megasg_proxy50k | maelic/RA-4M packs/ |
| evaluation packs, hicodet, negatives, text student, text space, shared detections | maelic/OV-SGG-Bench |
| MegaSG images | maelic/RA-4M (also release/ra4m/export_images.py) |
| Visual Genome images (vg_raw, VG150), PSG, IndoorVG | maelic/VG150-coco-format, maelic/PSG-coco-format, maelic/IndoorVG-coco-format |
| HICO-DET images | maelic/HICO-DET-coco-format |
| Haystack, SpatialSense images (evaluation only) | maelic/Haystack-coco-format, maelic/SpatialSense-coco-format: mirrors of the authors' releases (SA-1B subset, SA-1B Dataset Research License; Zenodo 8104370, CC BY 2.0) |
training/setup_data.py --proxy takes the 50,000-image development pack
(megasg_proxy50k, 6 GB of MegaSG images instead of 43 GB) for runs on one GPU.
A pack records the directory of its images (meta.json: img_dir); to move the
images later, rerun the setup or training/relocate_packs.py. If the recorded
directory does not exist and RA_DATASETS is set, the loaders look for the images
at the same place under RA_DATASETS and warn, naming both paths, if they are not there.
DINOv3 weights live in gated Hugging Face repositories
(facebook/dinov3-vits16-pretrain-lvd1689m, -vits16plus-, -vitb16-). Two ways
through:
Accept the terms and log in (needed once):
huggingface-cli login # account that accepted Meta's DINOv3 terms
Or convert a local copy of Meta's raw checkpoints and point
--backbone_model at the resulting directory:
python training/convert_dinov3_local.py --help # ViT towers
python training/convert_dinov3_convnext.py --help # ConvNeXt towers
If neither is available, install.sh still succeeds; it prints a warning and
you get the failure later, at model construction, with a clearer message.
Once the cache is warm, nothing needs the network:
export HF_HOME="$PWD/.hf_cache"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
Compute nodes are usually offline. Warm the cache on a login node, then export the offline flags in the job:
source.venv/bin/activate
export HF_HOME="$PWD/.hf_cache"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export WANDB_MODE=offline
export RA_DATASETS=/path/to/datasets
Two conventions worth copying:
/tmp. On most clusters /tmp is node-local,
so a job that logs there silently discards its output.pytest tests/ -q
These are CPU-only and need no weights and no network — the same suite CI runs. They cover the install path, imports, evaluator math, score parity between the training and deploy paths, and edge decoding.
Then run bash setup.sh. It stops at the first error and is safe to run again. Servers and tests aren't started; it lists them at the end. Read it first.
git clone https://github.com/Maelic/RelateAnything.git cd RelateAnything
Python >=3.12
sudo apt install python3 python3-venv # Debian/Ubuntu
python3 -m venv .venv source .venv/bin/activate pip install -e .
pytest
Use Python 3.12+. Install a CUDA-enabled PyTorch build for NVIDIA GPU
inference, or use device="cpu" in the example below.
git clone https://github.com/Maelic/RelateAnything.git
cd RelateAnything
pip install -e ".[hub,detector]" # detector: ultralytics YOLO, used below
Run this from the repository root. A detector finds the objects in the included sample photo; RelateAnything predicts the relations between them.
from PIL import Image
from ultralytics import YOLO
from relsgg import RelateAnything
image = Image.open("assets/reel/images/horse.jpg").convert("RGB")
# 1. Find the objects. Any detector or segmenter works; this one is YOLO26.
detections = YOLO("yolo26m.pt")(image, conf=0.4, verbose=False)[0]
boxes = detections.boxes.xyxy.cpu().numpy() # [N, 4]: x1, y1, x2, y2 in pixels
labels = [detections.names[int(c)] for c in detections.boxes.cls] # for display only
# 2. Predict the relations between them.
model = RelateAnything.from_pretrained("maelic/relsgg-vits16plus", device="cuda")
for triplet in model.predict(image, boxes, box_labels=labels, topk=3):
print(triplet)
# (person) --riding [0.76]--> (horse)
# (horse) --carrying [0.51]--> (person)
# (horse) --walking past [0.43]--> (car)
# 3. Change the relations to look for, without retraining.
model.set_vocabulary(["riding", "carrying a rider", "beside", "in front of"])
graphs = model.predict(image, boxes, box_labels=labels, decompose=True, topk=3)
print(graphs["spatial"]) # where things are, e.g. (horse) --in front of--> (car)
print(graphs["semantic"]) # what they do, e.g. (person) --riding--> (horse)
The relation model reads the image and the boxes; the detector's labels are
never an input. Boxes can come from any detector, a segmenter (pass masks= as
well) or a person: an [N, 4] array in original-image pixels is all it needs.
Released checkpoints include the backbone configuration and text encoder;
no gated DINOv3 login is needed for inference. The detector extra
installs ultralytics, which is
AGPL-3.0 and downloads the YOLO26 weights on first use.
API guide: inputs, vocabularies, masks, and scores · Installation and offline use
No reports for Linux yet.
GitManual AI Agent reads the 4 key files and writes a step-by-step guide for Linux. Once written, it's saved for everyone.