# Clip → 3D Studio Turn any camera view — an uploaded clip, a still frame, or a direct media URL — into an orbitable 3D safety scene: hazard zone reconstructed in 3D, a min-distance cage around it, and per-worker (and per-joint) proximity verdicts. Two ways to use it: | | What you get | Needs | |---|---|---| | **Live site** (`/demos/clip-to-3d/`) | Browse + orbit the archived hero renders | nothing | | **Local engine** | Generate *new* scenes from your own clips | the setup below | Reconstruction runs on a local GPU (Apple Silicon `mps` or CUDA), so the static site can only serve pre-baked scenes. To make your own, run the engine locally. --- ## 1. Setup (one time) **Prerequisites** - Python 3.11 (3.11.15 verified) - `ffmpeg` + `ffprobe` on PATH — `brew install ffmpeg` - ~4 GB free for model weights (downloaded on first run to `~/.cache/huggingface`) **Create the engine venv.** The scripts default to `scripts/generate-belt-mask/.venv/bin/python`; point `CLIP3D_PY` at any other interpreter if you prefer. ```bash cd /path/to/realm-demos python3.11 -m venv scripts/generate-belt-mask/.venv scripts/generate-belt-mask/.venv/bin/pip install -U pip scripts/generate-belt-mask/.venv/bin/pip install \ torch torchvision transformers opencv-python numpy plyfile pillow safetensors \ "git+https://github.com/facebookresearch/sam2.git" ``` Verified versions: torch 2.11, transformers 5.14, opencv 4.11, numpy 2.4, plyfile 1.1, `SAM-2` (imported as `sam2`). **Models** download automatically on first use — nothing to place by hand: - `depth-anything/Depth-Anything-V2-Large-hf` — monocular depth - `facebook/sam2.1-hiera-base-plus` — segmentation - `facebook/detr-resnet-50` — automatic worker detection **fal.ai key** (only needed for the SAM 3 splat / scene mesh / body-skeleton passes; depth-only scenes work without it). Create `scripts/clip-to-3d/.env` — it is gitignored, **never commit a key**: ``` FAL_KEY=: ``` Ask Brad for the shared key, or use your own from . --- ## 2. Run the Studio ```bash cd demos/clip-to-3d && python3 studio_server.py 8790 ``` Then open ****. > Use `studio_server.py`, **not** `python3 -m http.server`. The plain static > server has no upload/generate API, so the ingest console can't work — the page > will tell you the engine is offline. **The flow:** drop a clip or image → draw the unsafe area on the frame (freehand, or tap-to-place points, then adjust) → set the scene name, detection moment, cage fit and breach buffer → **Generate**. The depth scene opens automatically in ~1 minute; SAM 3 splat, scene mesh and the body skeleton are added in the background and appear as they finish. Renders land in the 90-day archive on the Studio page. Each card's `⋮` menu can re-open the detection zone for editing, save the render permanently, or delete it. --- ## 3. Generate from the CLI (optional) Equivalent to the Studio, useful for batching: ```bash scripts/clip-to-3d/clip-to-3d.sh ~/Downloads/clip.mp4 \ --name belt-tail --safety --frame-time 10.5 --min-distance 0.15 ``` Add `--object-polygon '[[x,y],...]'` to constrain the hazard to a drawn zone, and `--cage-yaw` to rotate the cage along a fence/belt receding into depth. --- ## 4. Publishing a scene to the live site Full-resolution `.ply`/`.glb` files are **gitignored** (1.5 GB locally). Only `web/` ships, so a scene must be baked before it renders in production: ```bash # regenerate at a coarser stride straight into web/ (see bake_web.py header) scripts/clip-to-3d/reconstruct.py --image .jpg --out web/.ply \ --labels .labels.npz --scene-json web/.scene.json --layers \ --stride 3 --scale-mult 3.3 --jitter 0.4 --min-distance 0.15 ``` **Recipe:** `stride S` with `scale_mult × S/2` and `jitter 0.4`. Regenerate — don't post-hoc thin — and re-fit the skeleton against `web/.geom.npz` (the coarser build normalizes differently). Details in [`scripts/clip-to-3d/bake_web.py`](../../scripts/clip-to-3d/bake_web.py). Budget ~8–10 MB per base splat; the viewer prefers full-res locally and falls back to `web/` automatically. --- ## Layout | Path | What | |---|---| | `studio.html` | ingest + draw + archive | | `viewer.html` | depth scene (cage, layers, skeleton, live min-distance) | | `sam3view.html` / `object.html` | SAM 3 splat / scene mesh | | `studio_server.py` | static server + upload/generate/pin/delete API | | `realm-ui.css` / `.js` | Realm component layer (see `DESIGN_SPEC.md`) | | `scenes.json` | scene registry the viewers read | | `archive/manifest.json` | 90-day archive | | `web/` | committed, size-baked scenes for the static site | | `../../scripts/clip-to-3d/` | the engine (depth, segmentation, fal, skeleton, bake) | ## Consistency check ```bash python3 scripts/clip-to-3d/check.py ``` Runs automatically on any PR touching `demos/clip-to-3d/**` or `scripts/clip-to-3d/**`. Every check exists because the bug actually happened: | Check | Catches | |---|---| | `secrets` | a fal key or `.env` slipping into git | | `conflict` | committed merge-conflict markers | | `json` / `python` | unparsable registry/scene json, syntax errors | | `assetver` | `realm-ui.css/js` `?v=` drifting between pages, so a page silently loads stale CSS/JS | | `cssclass` | an `rds-*` class used in HTML that `realm-ui.css` never defines | | `registry` | a `scenes.json` entry whose frame / scene-json / declared metadata isn't tracked — a fresh clone would 404 | | `archive` | a card linking to a scene id the registry doesn't have, or an untracked thumbnail | Warnings (e.g. "no `web/` bake") are informational — local-only scenes are fine. ## Known limits - Monocular depth gives reliable **ranking** and geometry from one viewpoint, not metric distances. Fisheye and very dark scenes reconstruct rougher. - Authenticated production URLs (`app.realmalliance.com/...`) can't be ingested — the local engine has no signed-in session. Save the snapshot and upload it, or paste a direct image/video URL. - Editing a zone after generation means regenerating; the viewer's min-distance slider is a live preview over baked geometry.