init: media-acquisition pipeline scaffold
Self-hosted BitTorrent + arr-stack + catalog-update pipeline targeting
nullstone (Debian 13). Replaces the legacy onyx -> rsync -> import
round-trip.
Contents:
- README.md headline + ASCII architecture diagram + quickstart
- CLAUDE.md project rules (mirrors beta-flix style)
- .gitignore secrets dirs (.env, gluetun, qbt config, ssh keys)
- .gitleaksignore allowlist nullstone LAN addr + Tailscale CGNAT
- docs/architecture.md the plan in detail (gluetun + qbt + arr + catalog)
- docs/migration.md onyx-qbt -> nullstone-qbt runbook (3 phases)
- docs/trackers.md tracker schema + IP-pinning + ratio notes (user-curated)
- compose/docker-compose.yml gluetun v3.40 + qbt 5.0.5 (netns=gluetun) +
sonarr/radarr/prowlarr (hotio) + betaflix-catalog
- compose/.env.example documented env-var template (no secrets)
- compose/traefik/arr.yml file-provider for qbt/sonarr/radarr/prowlarr
.s8n.ru subdomains, LAN+TS only via
trusted-only@file + authentik-forwardauth@file
- catalog/catalog.py Flask service, ~340 LoC, /sonarr + /radarr +
/healthz; pulls beta-flix, inserts alphabetic
row into MEDIA-LIST.md, writes run log, commits
+ pushes as obsidian-ai. Idempotent via
payload-hash cache.
- catalog/Dockerfile python:3.12-slim + git + tini
- catalog/requirements.txt flask + jinja2 + requests + gitpython + pyyaml (pinned)
- catalog/templates/*.j2 run log + catalog row Jinja templates
- catalog/README.md service docs
- scripts/migrate-onyx.sh phase-2 helper (rsync + .torrent ship, dry-run by default)
- scripts/add-tracker.sh Prowlarr API helper
- scripts/killswitch-test.sh gluetun kill-switch verification (3 steps)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
commit
d300d83ce1
19 changed files with 1758 additions and 0 deletions
43
.gitignore
vendored
Normal file
43
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,43 @@
|
||||||
|
# --- Secrets ---
|
||||||
|
.env
|
||||||
|
.env.local
|
||||||
|
*.key
|
||||||
|
*.pem
|
||||||
|
*.crt
|
||||||
|
catalog/ssh/
|
||||||
|
compose/gluetun/
|
||||||
|
compose/qbittorrent/config/
|
||||||
|
compose/sonarr/
|
||||||
|
compose/radarr/
|
||||||
|
compose/prowlarr/
|
||||||
|
compose/catalog/ssh/
|
||||||
|
|
||||||
|
# --- Caches / runtime ---
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.pyo
|
||||||
|
.pytest_cache/
|
||||||
|
.mypy_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
*.sqlite
|
||||||
|
*.sqlite-journal
|
||||||
|
seen-imports.json
|
||||||
|
|
||||||
|
# --- Editor ---
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
*~
|
||||||
|
.DS_Store
|
||||||
|
|
||||||
|
# --- Build artefacts ---
|
||||||
|
*.tar
|
||||||
|
*.tar.gz
|
||||||
|
dist/
|
||||||
|
build/
|
||||||
|
*.egg-info/
|
||||||
|
|
||||||
|
# --- Logs ---
|
||||||
|
*.log
|
||||||
|
logs/
|
||||||
27
.gitleaksignore
Normal file
27
.gitleaksignore
Normal file
|
|
@ -0,0 +1,27 @@
|
||||||
|
# Allowlist false-positive LAN-IP / tailnet-IP hits in docs + compose.
|
||||||
|
# These are the documented nullstone LAN address, the LAN/CGNAT
|
||||||
|
# allowed-egress subnets baked into gluetun config, and Proton WG client
|
||||||
|
# addresses — all infrastructure facts, not credentials.
|
||||||
|
# The lan-ip-rfc1918 rule is low-confidence by design — see ~/.config/git/.gitleaks.toml
|
||||||
|
|
||||||
|
# CLAUDE.md — header references nullstone LAN IP.
|
||||||
|
CLAUDE.md:lan-ip-rfc1918:9
|
||||||
|
|
||||||
|
# docs/architecture.md — header + § "Current State" reference live nullstone host.
|
||||||
|
docs/architecture.md:lan-ip-rfc1918:3
|
||||||
|
docs/architecture.md:lan-ip-rfc1918:30
|
||||||
|
|
||||||
|
# docs/migration.md — ssh + rsync targets to nullstone.
|
||||||
|
docs/migration.md:lan-ip-rfc1918:22
|
||||||
|
docs/migration.md:lan-ip-rfc1918:32
|
||||||
|
docs/migration.md:lan-ip-rfc1918:81
|
||||||
|
|
||||||
|
# scripts/migrate-onyx.sh — default NULLSTONE_SSH and ssh target.
|
||||||
|
scripts/migrate-onyx.sh:lan-ip-rfc1918:27
|
||||||
|
scripts/migrate-onyx.sh:lan-ip-rfc1918:35
|
||||||
|
|
||||||
|
# compose/docker-compose.yml — FIREWALL_OUTBOUND_SUBNETS allows LAN +
|
||||||
|
# RFC1918 + the Tailscale CGNAT range for webui reachability from
|
||||||
|
# trusted networks. These are public, well-known subnet constants.
|
||||||
|
compose/docker-compose.yml:lan-ip-rfc1918:26
|
||||||
|
compose/docker-compose.yml:tailnet-ip:26
|
||||||
109
CLAUDE.md
Normal file
109
CLAUDE.md
Normal file
|
|
@ -0,0 +1,109 @@
|
||||||
|
# CLAUDE.md — media-acquisition
|
||||||
|
|
||||||
|
Read this at session start. Rules for managing the nullstone media-acquisition
|
||||||
|
pipeline.
|
||||||
|
|
||||||
|
## What this repo is
|
||||||
|
|
||||||
|
The BitTorrent + arr-stack + catalog-update pipeline that feeds the ARRFLIX
|
||||||
|
library on **nullstone** (Debian 13, `192.168.0.100`). Consumed by:
|
||||||
|
|
||||||
|
- **Jellyfin** at `tv.s8n.ru` (container `jellyfin-stock`).
|
||||||
|
- **Catalog** at `git.s8n.ru/s8n/beta-flix` → `playbooks/import-media/MEDIA-LIST.md`.
|
||||||
|
|
||||||
|
## Source map
|
||||||
|
|
||||||
|
```
|
||||||
|
docs/architecture.md Plan + reasoning. Read this BEFORE editing compose.
|
||||||
|
docs/migration.md onyx-qbt → nullstone-qbt migration runbook.
|
||||||
|
docs/trackers.md Tracker schema + IP-pinning risks (user-curated).
|
||||||
|
compose/docker-compose.yml gluetun + qbt + sonarr + radarr + prowlarr + catalog.
|
||||||
|
compose/.env.example Env template — secrets live in .env (gitignored).
|
||||||
|
compose/traefik/arr.yml File-provider routing for arr stack.
|
||||||
|
catalog/ betaflix-catalog Python service (Flask + webhooks).
|
||||||
|
scripts/ migrate-onyx.sh, add-tracker.sh, killswitch-test.sh.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deploy lifecycle
|
||||||
|
|
||||||
|
1. **Edit** files locally under `/home/admin/projects/media-acquisition/`.
|
||||||
|
2. **Push to Forgejo** (this repo's authoritative remote is
|
||||||
|
`git.s8n.ru/s8n/media-acquisition.git`).
|
||||||
|
3. **On nullstone**: `cd /opt/docker/media-acquisition && git pull && docker compose up -d`.
|
||||||
|
4. **CRITICAL — verify kill-switch after every gluetun change**:
|
||||||
|
`bash scripts/killswitch-test.sh`. If the second curl succeeds, you have a leak;
|
||||||
|
tear down before re-trying.
|
||||||
|
|
||||||
|
## Rules paid for in blood (mirrored from beta-flix where applicable)
|
||||||
|
|
||||||
|
### Rule 1 — SSH user
|
||||||
|
`user@nullstone`. **NOT** `admin@nullstone`. AllowUsers was tightened
|
||||||
|
2026-05-03; uid 1000 only. Memory: `[[feedback_nullstone_ssh_user]]`.
|
||||||
|
|
||||||
|
### Rule 2 — Commit + push to **my git**
|
||||||
|
Authoritative remote is `git.s8n.ru/s8n/media-acquisition.git` (Forgejo).
|
||||||
|
No GitHub mirror. Always `git remote -v` before push. Memory:
|
||||||
|
`[[feedback_always_commit_to_my_git]]`, `[[feedback_check_remote_before_push]]`,
|
||||||
|
`[[feedback_my_git_is_forgejo]]`.
|
||||||
|
|
||||||
|
### Rule 3 — Commit identity
|
||||||
|
- Human commits: `s8n <admin@s8n.ru>`.
|
||||||
|
- Bot/automation commits (e.g. catalog service, scripted edits): `obsidian-ai <obsidian-ai@s8n.ru>`.
|
||||||
|
|
||||||
|
Memory: `[[user_git_identity]]`.
|
||||||
|
|
||||||
|
### Rule 4 — Kill-switch is non-negotiable
|
||||||
|
Every change to `gluetun` service or VPN env vars MUST be followed by
|
||||||
|
`scripts/killswitch-test.sh`. A torrent client leaking outside the VPN is the
|
||||||
|
single failure mode that defines this project — do not "trust" the firewall
|
||||||
|
based on config inspection. Run the test.
|
||||||
|
|
||||||
|
### Rule 5 — No secrets in repo
|
||||||
|
`.env`, WireGuard keys, Forgejo PATs, deploy keys: all gitignored. Use
|
||||||
|
`.env.example` to document variable names with placeholders. If you commit a
|
||||||
|
secret by accident, rotate it (Proton WG: regenerate key, update gluetun;
|
||||||
|
Forgejo PAT: revoke at `git.s8n.ru/-/user/settings/applications`).
|
||||||
|
|
||||||
|
### Rule 6 — Tracker IP pinning
|
||||||
|
Private trackers may pin sessions to a single source IP. Switching from
|
||||||
|
onyx public IP → Proton exit IP will trip them. Before adding a new tracker
|
||||||
|
or migrating an old torrent, check `docs/trackers.md` for the per-tracker
|
||||||
|
policy. Update `docs/trackers.md` whenever a new tracker is on-boarded.
|
||||||
|
|
||||||
|
### Rule 7 — XFS reflinks / hardlinks
|
||||||
|
`/home/user/media` is XFS, single device. Sonarr/Radarr "Use Hardlinks
|
||||||
|
instead of Copy" = ON. Catalog service may use `cp --reflink=always` for
|
||||||
|
divergent-perm scenarios (free inodes, zero block cost). Never `cp` plain;
|
||||||
|
that doubles disk usage and breaks seeding atomicity.
|
||||||
|
|
||||||
|
## Canonical naming
|
||||||
|
|
||||||
|
Catalog rows pushed to `beta-flix/playbooks/import-media/MEDIA-LIST.md` follow
|
||||||
|
the ARRFLIX house style:
|
||||||
|
|
||||||
|
- TV: `Series Title (Year)` — alphabetic by title, year tiebreaker.
|
||||||
|
- Movies: `Movie Title (Year)` — alphabetic by title.
|
||||||
|
- "Source / Version" column = raw Sonarr/Radarr `sourceTitle` (release name).
|
||||||
|
Human edits to "Why on arrflix" stay; bot never overwrites that column.
|
||||||
|
|
||||||
|
The catalog service is **append + merge only** — never overwrites human-authored
|
||||||
|
notes.
|
||||||
|
|
||||||
|
## How to start a session
|
||||||
|
|
||||||
|
1. Read this file.
|
||||||
|
2. Read `docs/architecture.md` if working on compose or catalog code.
|
||||||
|
3. Check `git status` and `git remote -v` (must show
|
||||||
|
`git.s8n.ru/s8n/media-acquisition.git`).
|
||||||
|
4. Owner says what they want; ship + verify kill-switch + commit to Forgejo.
|
||||||
|
5. End every turn: commit + push to `git.s8n.ru/s8n/media-acquisition.git`.
|
||||||
|
|
||||||
|
## Glossary
|
||||||
|
|
||||||
|
| Term | Means |
|
||||||
|
|-----------------------|----------------------------------------------------------------------------------|
|
||||||
|
| **ship** / **deploy** | git push to Forgejo → on nullstone, `git pull && docker compose up -d`. Kill-switch test on any gluetun change. |
|
||||||
|
| **migrate** | Phase-2 onyx→nullstone runbook in `docs/migration.md`. Read `scripts/migrate-onyx.sh` first; dry-run mode mandatory. |
|
||||||
|
| **add tracker** | `scripts/add-tracker.sh <name> <url>`; then update `docs/trackers.md` with IP-pinning policy + ratio requirements. |
|
||||||
|
| **killswitch test** | `bash scripts/killswitch-test.sh`. NEVER claim "VPN works" without running this. |
|
||||||
|
| **owner** | P (xynki.dev@gmail.com). Final say. Executive-override pattern from `[[feedback_s8n_executive_override]]` applies. |
|
||||||
107
README.md
Normal file
107
README.md
Normal file
|
|
@ -0,0 +1,107 @@
|
||||||
|
# media-acquisition
|
||||||
|
|
||||||
|
Self-hosted BitTorrent + arr-stack + canonical-import pipeline that lands torrents
|
||||||
|
directly on **nullstone**, through a Proton WireGuard VPN with verified kill-switch,
|
||||||
|
hardlinks files into the existing ARRFLIX library, and auto-commits catalog rows
|
||||||
|
to `git.s8n.ru/s8n/beta-flix`.
|
||||||
|
|
||||||
|
Replaces the legacy `onyx → rsync → nullstone` round-trip.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
+-----------------+
|
||||||
|
| Proton VPN |
|
||||||
|
| (WireGuard) |
|
||||||
|
+--------+--------+
|
||||||
|
| wg0
|
||||||
|
v
|
||||||
|
+------------+ indexer queries +-------------------+ torrent traffic
|
||||||
|
| Prowlarr |-------------------->| gluetun |<------------------+
|
||||||
|
+-----+------+ (via netns) | kill-switch fw | |
|
||||||
|
| +-------------------+ |
|
||||||
|
| search ^ ^ ^ |
|
||||||
|
v | | | |
|
||||||
|
+------------+ grabs +----------+ | +----------+ |
|
||||||
|
| Sonarr/ |----------->| qBittorrent (netns=gluetun) |
|
||||||
|
| Radarr | | /home/user/media/_downloads/{incomplete,complete}
|
||||||
|
+-----+------+ +-------------------------------------------------+
|
||||||
|
|
|
||||||
|
| OnImport webhook (POST /sonarr or /radarr)
|
||||||
|
v
|
||||||
|
+--------------------+
|
||||||
|
| betaflix-catalog |--+ XFS reflink/hardlink into /home/user/media/{movies,tv}
|
||||||
|
| (Flask, Python) | |
|
||||||
|
+--------+-----------+ +--> Jellyfin (tv.s8n.ru) picks up new items
|
||||||
|
|
|
||||||
|
| git commit + push (obsidian-ai)
|
||||||
|
v
|
||||||
|
+-----------------------------+
|
||||||
|
| git.s8n.ru/s8n/beta-flix |
|
||||||
|
| playbooks/import-media/ |
|
||||||
|
| MEDIA-LIST.md (updated) |
|
||||||
|
| runs/<slug>.md (new) |
|
||||||
|
+-----------------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
Single XFS filesystem at `/home/user/media` → hardlinks / reflinks are free.
|
||||||
|
|
||||||
|
## Quickstart
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone on nullstone
|
||||||
|
ssh user@nullstone
|
||||||
|
git clone https://git.s8n.ru/s8n/media-acquisition.git /opt/docker/media-acquisition
|
||||||
|
cd /opt/docker/media-acquisition/compose
|
||||||
|
|
||||||
|
# Configure
|
||||||
|
cp .env.example .env
|
||||||
|
${EDITOR:-vi} .env # fill in PVPN_WG_PRIVKEY, PVPN_WG_ADDRESSES, FORGEJO_PUSH_TOKEN, etc.
|
||||||
|
|
||||||
|
# Bring up
|
||||||
|
docker compose up -d
|
||||||
|
|
||||||
|
# Verify VPN kill-switch (CRITICAL — do not skip)
|
||||||
|
bash ../scripts/killswitch-test.sh
|
||||||
|
|
||||||
|
# Sanity: pick a sacrificial legal torrent in qbt UI, confirm it lands in
|
||||||
|
# /home/user/media/_downloads/complete/ and arr stack hardlinks it.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
README.md This file.
|
||||||
|
CLAUDE.md Project rules for Claude Code.
|
||||||
|
docs/
|
||||||
|
architecture.md The plan in detail. Decision log + reasoning.
|
||||||
|
migration.md onyx-qbt → nullstone-qbt migration runbook.
|
||||||
|
trackers.md Tracker schema + IP-pinning notes (user fills in).
|
||||||
|
compose/
|
||||||
|
docker-compose.yml Full stack: gluetun + qbt + sonarr + radarr + prowlarr + catalog.
|
||||||
|
.env.example All env vars documented.
|
||||||
|
traefik/arr.yml Traefik file-provider for *.s8n.ru subdomains (LAN+TS only).
|
||||||
|
catalog/
|
||||||
|
catalog.py Flask webhook receiver → beta-flix catalog updater.
|
||||||
|
Dockerfile python:3.12-slim base.
|
||||||
|
requirements.txt Pinned versions.
|
||||||
|
templates/ Jinja2 templates for run logs and catalog rows.
|
||||||
|
README.md Catalog service docs.
|
||||||
|
scripts/
|
||||||
|
migrate-onyx.sh Phase-2 migration: rsync + .torrent mass-add.
|
||||||
|
add-tracker.sh Helper: add tracker to Prowlarr via API.
|
||||||
|
killswitch-test.sh Verify gluetun blocks traffic when VPN drops.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- Plan: `docs/architecture.md`
|
||||||
|
- Catalog target: `git.s8n.ru/s8n/beta-flix` (`playbooks/import-media/MEDIA-LIST.md`)
|
||||||
|
- Jellyfin (consumer): `tv.s8n.ru` (`jellyfin-stock` container on nullstone)
|
||||||
|
- Host docs: `ai-lab/SYSTEM.md`
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Scaffold. Live deploy pending VPN slot allocation + tracker IP-pinning review.
|
||||||
|
Next step: fill in `compose/.env` and bring up gluetun + qbt only (no arr yet)
|
||||||
|
to validate kill-switch.
|
||||||
28
catalog/Dockerfile
Normal file
28
catalog/Dockerfile
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
ENV PYTHONUNBUFFERED=1 \
|
||||||
|
PIP_NO_CACHE_DIR=1 \
|
||||||
|
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||||
|
|
||||||
|
# git is required for clone/pull/push; openssh-client for ssh remotes (future).
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends git openssh-client ca-certificates tini \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY requirements.txt /app/requirements.txt
|
||||||
|
RUN pip install -r requirements.txt
|
||||||
|
|
||||||
|
COPY catalog.py /app/catalog.py
|
||||||
|
COPY templates /app/templates
|
||||||
|
|
||||||
|
# Forge globally so the bot identity persists even if env vars get dropped.
|
||||||
|
RUN git config --global user.name "obsidian-ai" \
|
||||||
|
&& git config --global user.email "obsidian-ai@s8n.ru" \
|
||||||
|
&& git config --global pull.rebase true \
|
||||||
|
&& git config --global init.defaultBranch main
|
||||||
|
|
||||||
|
EXPOSE 5055
|
||||||
|
ENTRYPOINT ["/usr/bin/tini", "--"]
|
||||||
|
CMD ["python", "/app/catalog.py"]
|
||||||
73
catalog/README.md
Normal file
73
catalog/README.md
Normal file
|
|
@ -0,0 +1,73 @@
|
||||||
|
# betaflix-catalog
|
||||||
|
|
||||||
|
Flask service that receives Sonarr/Radarr **OnImport** webhooks and commits
|
||||||
|
catalog updates to `git.s8n.ru/s8n/beta-flix`.
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
For each `Import` event:
|
||||||
|
|
||||||
|
1. Pulls latest `main` of `beta-flix` (rebase).
|
||||||
|
2. Inserts a row into `playbooks/import-media/MEDIA-LIST.md`, alphabetic by
|
||||||
|
title. Dedupes — if the key (`Title (Year)`) already exists, it skips.
|
||||||
|
3. Writes a per-import run log to
|
||||||
|
`playbooks/import-media/runs/<slug>.md` using the Jinja template at
|
||||||
|
`templates/run.md.j2`.
|
||||||
|
4. Commits as `obsidian-ai <obsidian-ai@s8n.ru>`.
|
||||||
|
5. Pushes to Forgejo using `FORGEJO_PUSH_TOKEN` embedded in the URL.
|
||||||
|
|
||||||
|
Idempotency: payload-hash cache at `/state/seen-imports.json`. Sonarr/Radarr
|
||||||
|
retry transient failures; duplicates are no-ops.
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
- `POST /sonarr` — Sonarr Connect webhook target. Set Sonarr → Settings →
|
||||||
|
Connect → Webhook → URL `http://host.docker.internal:5055/sonarr`, method
|
||||||
|
POST, triggers: **On Import** only.
|
||||||
|
- `POST /radarr` — same shape for Radarr at `/radarr`.
|
||||||
|
- `GET /healthz` — liveness probe.
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd catalog/
|
||||||
|
docker build -t betaflix-catalog:local .
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use compose: the parent `compose/docker-compose.yml` defines the
|
||||||
|
`betaflix-catalog` service with `build:` set to this directory.
|
||||||
|
|
||||||
|
## Env vars
|
||||||
|
|
||||||
|
| Variable | Required | Default | What |
|
||||||
|
|----------------------|----------|--------------------------------------------|---------------------------------------|
|
||||||
|
| `FORGEJO_REMOTE` | yes | `https://git.s8n.ru/s8n/beta-flix.git` | Push target. |
|
||||||
|
| `FORGEJO_PUSH_TOKEN` | yes | (empty) | Forgejo PAT — scopes: repository RW. |
|
||||||
|
| `GIT_AUTHOR_NAME` | no | `obsidian-ai` | Commit author. |
|
||||||
|
| `GIT_AUTHOR_EMAIL` | no | `obsidian-ai@s8n.ru` | Commit author email. |
|
||||||
|
| `REPO_PATH` | no | `/repo` | Where beta-flix gets cloned. |
|
||||||
|
| `STATE_DIR` | no | `/state` | seen-imports.json lives here. |
|
||||||
|
| `LISTEN_PORT` | no | `5055` | Flask bind port. |
|
||||||
|
|
||||||
|
## Volumes
|
||||||
|
|
||||||
|
- `/repo` — beta-flix checkout. Bind-mounted persistent volume.
|
||||||
|
- `/state` — `seen-imports.json` cache.
|
||||||
|
- `/root/.ssh` (optional, read-only) — for SSH deploy key (currently uses
|
||||||
|
HTTPS+PAT; SSH path reserved for future).
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
Run locally without Docker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd catalog/
|
||||||
|
python -m venv .venv && . .venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
REPO_PATH=/tmp/beta-flix-test STATE_DIR=/tmp/catalog-state \
|
||||||
|
FORGEJO_PUSH_TOKEN=xxx python catalog.py
|
||||||
|
# In another shell:
|
||||||
|
curl -X POST http://localhost:5055/sonarr \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"eventType":"Test"}'
|
||||||
|
```
|
||||||
366
catalog/catalog.py
Normal file
366
catalog/catalog.py
Normal file
|
|
@ -0,0 +1,366 @@
|
||||||
|
"""betaflix-catalog — Sonarr/Radarr OnImport webhook receiver.
|
||||||
|
|
||||||
|
Listens for OnImport events from Sonarr and Radarr, edits
|
||||||
|
`playbooks/import-media/MEDIA-LIST.md` in the beta-flix Forgejo repo, writes
|
||||||
|
a per-import run log, and commits + pushes as `obsidian-ai`.
|
||||||
|
|
||||||
|
POST endpoints:
|
||||||
|
/sonarr Sonarr Connect webhook target.
|
||||||
|
/radarr Radarr Connect webhook target.
|
||||||
|
/healthz Liveness probe.
|
||||||
|
|
||||||
|
Idempotency: payload-hash cache at /state/seen-imports.json. Duplicates skipped.
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
FORGEJO_REMOTE e.g. https://git.s8n.ru/s8n/beta-flix.git
|
||||||
|
FORGEJO_PUSH_TOKEN PAT — embedded into the push URL.
|
||||||
|
GIT_AUTHOR_NAME obsidian-ai
|
||||||
|
GIT_AUTHOR_EMAIL obsidian-ai@s8n.ru
|
||||||
|
LISTEN_PORT 5055
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from flask import Flask, jsonify, request
|
||||||
|
from jinja2 import Environment, FileSystemLoader, select_autoescape
|
||||||
|
|
||||||
|
# --- Config -----------------------------------------------------------------
|
||||||
|
|
||||||
|
REPO_PATH = Path(os.environ.get("REPO_PATH", "/repo"))
|
||||||
|
STATE_DIR = Path(os.environ.get("STATE_DIR", "/state"))
|
||||||
|
TEMPLATES_DIR = Path(__file__).parent / "templates"
|
||||||
|
|
||||||
|
FORGEJO_REMOTE = os.environ.get("FORGEJO_REMOTE", "https://git.s8n.ru/s8n/beta-flix.git")
|
||||||
|
FORGEJO_TOKEN = os.environ.get("FORGEJO_PUSH_TOKEN", "")
|
||||||
|
GIT_AUTHOR_NAME = os.environ.get("GIT_AUTHOR_NAME", "obsidian-ai")
|
||||||
|
GIT_AUTHOR_EMAIL = os.environ.get("GIT_AUTHOR_EMAIL", "obsidian-ai@s8n.ru")
|
||||||
|
LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "5055"))
|
||||||
|
|
||||||
|
MEDIA_LIST = REPO_PATH / "playbooks" / "import-media" / "MEDIA-LIST.md"
|
||||||
|
RUNS_DIR = REPO_PATH / "playbooks" / "import-media" / "runs"
|
||||||
|
SEEN_PATH = STATE_DIR / "seen-imports.json"
|
||||||
|
|
||||||
|
# Section headers in MEDIA-LIST.md the bot will edit.
|
||||||
|
MOVIES_SECTION = "## Movies"
|
||||||
|
TV_SECTION = "## TV"
|
||||||
|
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
stream=sys.stdout,
|
||||||
|
)
|
||||||
|
log = logging.getLogger("catalog")
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
_lock = threading.Lock()
|
||||||
|
_jinja = Environment(
|
||||||
|
loader=FileSystemLoader(str(TEMPLATES_DIR)),
|
||||||
|
autoescape=select_autoescape(["html", "xml"]),
|
||||||
|
trim_blocks=True,
|
||||||
|
lstrip_blocks=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Idempotency ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _load_seen() -> set[str]:
|
||||||
|
if not SEEN_PATH.exists():
|
||||||
|
return set()
|
||||||
|
try:
|
||||||
|
return set(json.loads(SEEN_PATH.read_text()))
|
||||||
|
except Exception:
|
||||||
|
log.warning("seen-imports.json corrupt; resetting")
|
||||||
|
return set()
|
||||||
|
|
||||||
|
|
||||||
|
def _save_seen(seen: set[str]) -> None:
|
||||||
|
STATE_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
SEEN_PATH.write_text(json.dumps(sorted(seen)))
|
||||||
|
|
||||||
|
|
||||||
|
def _payload_hash(kind: str, payload: dict[str, Any]) -> str:
|
||||||
|
"""Stable hash for the import event — series:season:episode or movie:year."""
|
||||||
|
if kind == "sonarr":
|
||||||
|
sid = payload.get("series", {}).get("id", "?")
|
||||||
|
eps = payload.get("episodes", []) or [{}]
|
||||||
|
keys = sorted(f"{e.get('seasonNumber', '?')}x{e.get('episodeNumber', '?')}" for e in eps)
|
||||||
|
seed = f"sonarr:{sid}:{','.join(keys)}"
|
||||||
|
elif kind == "radarr":
|
||||||
|
mid = payload.get("movie", {}).get("id", "?")
|
||||||
|
seed = f"radarr:{mid}"
|
||||||
|
else:
|
||||||
|
seed = f"unknown:{json.dumps(payload, sort_keys=True)}"
|
||||||
|
return hashlib.sha256(seed.encode()).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
# --- Git helpers ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _git(*args: str, cwd: Path = REPO_PATH) -> subprocess.CompletedProcess:
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.setdefault("GIT_AUTHOR_NAME", GIT_AUTHOR_NAME)
|
||||||
|
env.setdefault("GIT_AUTHOR_EMAIL", GIT_AUTHOR_EMAIL)
|
||||||
|
env.setdefault("GIT_COMMITTER_NAME", GIT_AUTHOR_NAME)
|
||||||
|
env.setdefault("GIT_COMMITTER_EMAIL", GIT_AUTHOR_EMAIL)
|
||||||
|
return subprocess.run(
|
||||||
|
["git", *args],
|
||||||
|
cwd=cwd,
|
||||||
|
env=env,
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_repo() -> None:
|
||||||
|
"""Clone the repo if /repo is empty."""
|
||||||
|
if (REPO_PATH / ".git").is_dir():
|
||||||
|
return
|
||||||
|
REPO_PATH.mkdir(parents=True, exist_ok=True)
|
||||||
|
clone_url = _push_url()
|
||||||
|
subprocess.run(
|
||||||
|
["git", "clone", clone_url, str(REPO_PATH)],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _push_url() -> str:
|
||||||
|
if FORGEJO_TOKEN and FORGEJO_REMOTE.startswith("https://"):
|
||||||
|
return FORGEJO_REMOTE.replace("https://", f"https://{FORGEJO_TOKEN}@", 1)
|
||||||
|
return FORGEJO_REMOTE
|
||||||
|
|
||||||
|
|
||||||
|
def _pull_rebase() -> None:
|
||||||
|
_git("fetch", "origin")
|
||||||
|
_git("rebase", "origin/main")
|
||||||
|
|
||||||
|
|
||||||
|
def _commit_and_push(title: str) -> str:
|
||||||
|
_git("add", "playbooks/import-media/MEDIA-LIST.md", "playbooks/import-media/runs/")
|
||||||
|
status = _git("status", "--porcelain")
|
||||||
|
if not status.stdout.strip():
|
||||||
|
log.info("no changes to commit (%s)", title)
|
||||||
|
return ""
|
||||||
|
msg = f"catalog: add {title}"
|
||||||
|
_git("commit", "-m", msg, f"--author={GIT_AUTHOR_NAME} <{GIT_AUTHOR_EMAIL}>")
|
||||||
|
_git("push", _push_url(), "HEAD:main")
|
||||||
|
sha = _git("rev-parse", "HEAD").stdout.strip()
|
||||||
|
return sha
|
||||||
|
|
||||||
|
|
||||||
|
# --- MEDIA-LIST.md editing --------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _normalise_title(title: str) -> str:
|
||||||
|
return re.sub(r"\s+", " ", title.strip())
|
||||||
|
|
||||||
|
|
||||||
|
def _slugify(s: str) -> str:
|
||||||
|
s = re.sub(r"[^a-zA-Z0-9]+", "-", s.lower()).strip("-")
|
||||||
|
return s[:80] or "untitled"
|
||||||
|
|
||||||
|
|
||||||
|
def _row(kind: str, title: str, year: int | None, source: str) -> str:
|
||||||
|
year_s = f"({year})" if year else ""
|
||||||
|
if kind == "tv":
|
||||||
|
return f"| {title} {year_s} | TV | {source} | _todo_ | "
|
||||||
|
return f"| {title} {year_s} | Movie | {source} | _todo_ | "
|
||||||
|
|
||||||
|
|
||||||
|
def _insert_alphabetic(section_header: str, row: str, key: str) -> bool:
|
||||||
|
"""Insert `row` into the section under `section_header`, alphabetic by key.
|
||||||
|
|
||||||
|
Returns True if a new row was added, False if the key already existed
|
||||||
|
(caller handles merge/dedup separately).
|
||||||
|
"""
|
||||||
|
if not MEDIA_LIST.exists():
|
||||||
|
log.warning("MEDIA-LIST.md missing at %s — skipping insert", MEDIA_LIST)
|
||||||
|
return False
|
||||||
|
lines = MEDIA_LIST.read_text().splitlines()
|
||||||
|
try:
|
||||||
|
start = next(i for i, line in enumerate(lines) if line.strip() == section_header)
|
||||||
|
except StopIteration:
|
||||||
|
log.warning("section %r not found in MEDIA-LIST.md", section_header)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Find table boundaries.
|
||||||
|
i = start + 1
|
||||||
|
while i < len(lines) and not lines[i].lstrip().startswith("|"):
|
||||||
|
i += 1
|
||||||
|
if i >= len(lines):
|
||||||
|
log.warning("no table found under section %r", section_header)
|
||||||
|
return False
|
||||||
|
header_idx = i
|
||||||
|
# Skip header + separator rows.
|
||||||
|
i = header_idx + 2
|
||||||
|
section_rows_start = i
|
||||||
|
while i < len(lines) and lines[i].lstrip().startswith("|"):
|
||||||
|
if key.lower() in lines[i].lower():
|
||||||
|
log.info("row already present for key=%r — skipping", key)
|
||||||
|
return False
|
||||||
|
i += 1
|
||||||
|
section_rows_end = i
|
||||||
|
|
||||||
|
# Alphabetic insert by first column.
|
||||||
|
insert_at = section_rows_end
|
||||||
|
for j in range(section_rows_start, section_rows_end):
|
||||||
|
cell = lines[j].split("|")[1].strip() if "|" in lines[j] else ""
|
||||||
|
if key.lower() < cell.lower():
|
||||||
|
insert_at = j
|
||||||
|
break
|
||||||
|
|
||||||
|
lines.insert(insert_at, row)
|
||||||
|
MEDIA_LIST.write_text("\n".join(lines) + "\n")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _write_run_log(slug: str, ctx: dict[str, Any]) -> Path:
|
||||||
|
RUNS_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
template = _jinja.get_template("run.md.j2")
|
||||||
|
out = RUNS_DIR / f"{slug}.md"
|
||||||
|
out.write_text(template.render(**ctx))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# --- Webhook handlers -------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _handle_sonarr(payload: dict[str, Any]) -> tuple[str, bool]:
|
||||||
|
series = payload.get("series", {}) or {}
|
||||||
|
title = _normalise_title(series.get("title", "Unknown Series"))
|
||||||
|
year = series.get("year") or None
|
||||||
|
eps = payload.get("episodes", []) or []
|
||||||
|
files = payload.get("episodeFiles") or payload.get("episodeFile") or []
|
||||||
|
if isinstance(files, dict):
|
||||||
|
files = [files]
|
||||||
|
source = (files[0].get("sceneName") or files[0].get("relativePath") or "?") if files else "?"
|
||||||
|
key = f"{title} ({year})" if year else title
|
||||||
|
|
||||||
|
with _lock:
|
||||||
|
_ensure_repo()
|
||||||
|
_pull_rebase()
|
||||||
|
added = _insert_alphabetic(TV_SECTION, _row("tv", title, year, source), key)
|
||||||
|
slug = _slugify(f"{key}-S{eps[0].get('seasonNumber','?')}E{eps[0].get('episodeNumber','?')}" if eps else key)
|
||||||
|
_write_run_log(slug, {
|
||||||
|
"kind": "tv",
|
||||||
|
"title": title,
|
||||||
|
"year": year,
|
||||||
|
"source": source,
|
||||||
|
"episodes": eps,
|
||||||
|
"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||||
|
"row_added": added,
|
||||||
|
})
|
||||||
|
sha = _commit_and_push(f"{title} ({year})" if year else title)
|
||||||
|
return sha, added
|
||||||
|
|
||||||
|
|
||||||
|
def _handle_radarr(payload: dict[str, Any]) -> tuple[str, bool]:
|
||||||
|
movie = payload.get("movie", {}) or {}
|
||||||
|
title = _normalise_title(movie.get("title", "Unknown Movie"))
|
||||||
|
year = movie.get("year") or None
|
||||||
|
mfile = payload.get("movieFile") or {}
|
||||||
|
source = mfile.get("sceneName") or mfile.get("relativePath") or "?"
|
||||||
|
key = f"{title} ({year})" if year else title
|
||||||
|
|
||||||
|
with _lock:
|
||||||
|
_ensure_repo()
|
||||||
|
_pull_rebase()
|
||||||
|
added = _insert_alphabetic(MOVIES_SECTION, _row("movie", title, year, source), key)
|
||||||
|
slug = _slugify(key)
|
||||||
|
_write_run_log(slug, {
|
||||||
|
"kind": "movie",
|
||||||
|
"title": title,
|
||||||
|
"year": year,
|
||||||
|
"source": source,
|
||||||
|
"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||||
|
"row_added": added,
|
||||||
|
})
|
||||||
|
sha = _commit_and_push(f"{title} ({year})" if year else title)
|
||||||
|
return sha, added
|
||||||
|
|
||||||
|
|
||||||
|
# --- Flask routes -----------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/healthz")
|
||||||
|
def healthz():
|
||||||
|
return jsonify(ok=True), 200
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/sonarr")
|
||||||
|
def sonarr():
|
||||||
|
payload = request.get_json(silent=True) or {}
|
||||||
|
event = payload.get("eventType", "")
|
||||||
|
if event in ("Test", "ApplicationUpdate"):
|
||||||
|
log.info("sonarr probe event=%s — ack", event)
|
||||||
|
return jsonify(ok=True, ignored=event), 200
|
||||||
|
if event != "Import" and event != "Download":
|
||||||
|
log.info("sonarr event=%s — ignored", event)
|
||||||
|
return jsonify(ok=True, ignored=event), 200
|
||||||
|
|
||||||
|
h = _payload_hash("sonarr", payload)
|
||||||
|
seen = _load_seen()
|
||||||
|
if h in seen:
|
||||||
|
log.info("sonarr duplicate event hash=%s — skipping", h)
|
||||||
|
return jsonify(ok=True, duplicate=h), 200
|
||||||
|
try:
|
||||||
|
sha, added = _handle_sonarr(payload)
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
log.exception("git failed: %s", e.stderr)
|
||||||
|
return jsonify(ok=False, error=e.stderr), 500
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.exception("sonarr handler crashed")
|
||||||
|
return jsonify(ok=False, error=str(e)), 500
|
||||||
|
|
||||||
|
seen.add(h)
|
||||||
|
_save_seen(seen)
|
||||||
|
return jsonify(ok=True, sha=sha, row_added=added), 200
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/radarr")
|
||||||
|
def radarr():
|
||||||
|
payload = request.get_json(silent=True) or {}
|
||||||
|
event = payload.get("eventType", "")
|
||||||
|
if event in ("Test", "ApplicationUpdate"):
|
||||||
|
log.info("radarr probe event=%s — ack", event)
|
||||||
|
return jsonify(ok=True, ignored=event), 200
|
||||||
|
if event != "Import" and event != "Download":
|
||||||
|
log.info("radarr event=%s — ignored", event)
|
||||||
|
return jsonify(ok=True, ignored=event), 200
|
||||||
|
|
||||||
|
h = _payload_hash("radarr", payload)
|
||||||
|
seen = _load_seen()
|
||||||
|
if h in seen:
|
||||||
|
log.info("radarr duplicate event hash=%s — skipping", h)
|
||||||
|
return jsonify(ok=True, duplicate=h), 200
|
||||||
|
try:
|
||||||
|
sha, added = _handle_radarr(payload)
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
log.exception("git failed: %s", e.stderr)
|
||||||
|
return jsonify(ok=False, error=e.stderr), 500
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.exception("radarr handler crashed")
|
||||||
|
return jsonify(ok=False, error=str(e)), 500
|
||||||
|
|
||||||
|
seen.add(h)
|
||||||
|
_save_seen(seen)
|
||||||
|
return jsonify(ok=True, sha=sha, row_added=added), 200
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
log.info("betaflix-catalog listening on 0.0.0.0:%d", LISTEN_PORT)
|
||||||
|
app.run(host="0.0.0.0", port=LISTEN_PORT)
|
||||||
5
catalog/requirements.txt
Normal file
5
catalog/requirements.txt
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
flask==3.0.3
|
||||||
|
requests==2.32.3
|
||||||
|
jinja2==3.1.4
|
||||||
|
gitpython==3.1.43
|
||||||
|
pyyaml==6.0.2
|
||||||
4
catalog/templates/catalog_row.md.j2
Normal file
4
catalog/templates/catalog_row.md.j2
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
{# Reference template for MEDIA-LIST.md row generation.
|
||||||
|
Pipe-delimited; matches the existing beta-flix table schema.
|
||||||
|
Columns: Title | Kind | Source / Version | Why on arrflix | (trailing pipe) #}
|
||||||
|
| {{ title }}{% if year %} ({{ year }}){% endif %} | {{ kind }} | {{ source }} | {{ why | default("_todo_") }} |
|
||||||
24
catalog/templates/run.md.j2
Normal file
24
catalog/templates/run.md.j2
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
# {{ title }}{% if year %} ({{ year }}){% endif %} — import run
|
||||||
|
|
||||||
|
- **Date:** {{ ts }}
|
||||||
|
- **Kind:** {{ kind }}
|
||||||
|
- **Source / release name:** `{{ source }}`
|
||||||
|
- **Catalog row added:** {{ "yes" if row_added else "no (already present)" }}
|
||||||
|
|
||||||
|
{% if kind == "tv" and episodes %}
|
||||||
|
## Episodes imported
|
||||||
|
|
||||||
|
| Season | Episode | Title |
|
||||||
|
|--------|---------|-------|
|
||||||
|
{% for e in episodes %}
|
||||||
|
| {{ e.seasonNumber }} | {{ e.episodeNumber }} | {{ e.title | default("?") }} |
|
||||||
|
{% endfor %}
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
_(human-authored)_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Auto-generated by `betaflix-catalog` on Sonarr/Radarr OnImport webhook._
|
||||||
38
compose/.env.example
Normal file
38
compose/.env.example
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
# compose/.env.example
|
||||||
|
#
|
||||||
|
# Copy to .env (gitignored) and fill in real values.
|
||||||
|
#
|
||||||
|
# Never commit .env. Forgejo PAT + Proton WG key + arr API keys = secrets.
|
||||||
|
|
||||||
|
# --- Timezone (logs + scheduling) ---
|
||||||
|
TZ=Europe/London
|
||||||
|
|
||||||
|
# --- Proton VPN (gluetun) ---
|
||||||
|
# Generate a dedicated WireGuard key in the Proton dashboard:
|
||||||
|
# Account → WireGuard → New Configuration → name it "nullstone-gluetun-arr"
|
||||||
|
# Do NOT reuse the host's wg-pvpn-A/B keys.
|
||||||
|
PVPN_WG_PRIVKEY=REPLACE_WITH_PROTON_WG_PRIVATE_KEY
|
||||||
|
# The address Proton assigns to the new key (e.g. 10.2.0.3/32).
|
||||||
|
PVPN_WG_ADDRESSES=10.2.0.3/32
|
||||||
|
# Country (P2P-permitted). Comma-separated to let gluetun pick from a pool.
|
||||||
|
PVPN_SERVER_COUNTRIES=Netherlands
|
||||||
|
|
||||||
|
# --- Catalog service: Forgejo push ---
|
||||||
|
# https://git.s8n.ru → Settings → Applications → Generate New Token
|
||||||
|
# Scopes required: repository (read+write), user (read).
|
||||||
|
# Token is embedded in the remote URL inside the catalog container.
|
||||||
|
FORGEJO_PUSH_TOKEN=REPLACE_WITH_FORGEJO_PAT
|
||||||
|
# Remote URL — leave default unless beta-flix is moved.
|
||||||
|
FORGEJO_REMOTE=https://git.s8n.ru/s8n/beta-flix.git
|
||||||
|
|
||||||
|
# --- arr API keys ---
|
||||||
|
# Fetch from each app's Settings → General → Security after first launch.
|
||||||
|
# Used by catalog service to enrich the webhook payload via API calls.
|
||||||
|
SONARR_API_KEY=REPLACE_WITH_SONARR_API_KEY
|
||||||
|
RADARR_API_KEY=REPLACE_WITH_RADARR_API_KEY
|
||||||
|
PROWLARR_API_KEY=REPLACE_WITH_PROWLARR_API_KEY
|
||||||
|
|
||||||
|
# --- Optional: forwarded-port sync helper ---
|
||||||
|
# If you add caillef/qbittorrent-port-sync later for ratio-critical seeding,
|
||||||
|
# the qbt webui password goes here (used by that helper, not qbt itself).
|
||||||
|
QBT_WEBUI_PASSWORD=REPLACE_WITH_QBT_WEBUI_PASSWORD
|
||||||
141
compose/docker-compose.yml
Normal file
141
compose/docker-compose.yml
Normal file
|
|
@ -0,0 +1,141 @@
|
||||||
|
# nullstone media-acquisition stack
|
||||||
|
#
|
||||||
|
# Compose file for: gluetun (VPN + kill-switch) + qBittorrent + Sonarr +
|
||||||
|
# Radarr + Prowlarr + betaflix-catalog (Forgejo committer).
|
||||||
|
#
|
||||||
|
# Place this directory under /opt/docker/media-acquisition/ on nullstone.
|
||||||
|
# Run: docker compose up -d
|
||||||
|
# Verify kill-switch: bash ../scripts/killswitch-test.sh
|
||||||
|
|
||||||
|
services:
|
||||||
|
gluetun:
|
||||||
|
image: qmcgaw/gluetun:v3.40
|
||||||
|
container_name: gluetun
|
||||||
|
cap_add:
|
||||||
|
- NET_ADMIN
|
||||||
|
devices:
|
||||||
|
- /dev/net/tun:/dev/net/tun
|
||||||
|
environment:
|
||||||
|
- VPN_SERVICE_PROVIDER=protonvpn
|
||||||
|
- VPN_TYPE=wireguard
|
||||||
|
- WIREGUARD_PRIVATE_KEY=${PVPN_WG_PRIVKEY}
|
||||||
|
- WIREGUARD_ADDRESSES=${PVPN_WG_ADDRESSES}
|
||||||
|
- SERVER_COUNTRIES=${PVPN_SERVER_COUNTRIES:-Netherlands}
|
||||||
|
- VPN_PORT_FORWARDING=on
|
||||||
|
- VPN_PORT_FORWARDING_PROVIDER=protonvpn
|
||||||
|
- FIREWALL_OUTBOUND_SUBNETS=192.168.0.0/24,172.16.0.0/12,100.64.0.0/10
|
||||||
|
- DOT=off
|
||||||
|
- TZ=${TZ:-Europe/London}
|
||||||
|
ports:
|
||||||
|
# All published on 127.0.0.1 — Traefik file-provider picks them up.
|
||||||
|
- "127.0.0.1:8080:8080" # qBittorrent WebUI
|
||||||
|
- "127.0.0.1:9696:9696" # Prowlarr
|
||||||
|
- "127.0.0.1:8989:8989" # Sonarr
|
||||||
|
- "127.0.0.1:7878:7878" # Radarr
|
||||||
|
volumes:
|
||||||
|
- ./gluetun:/gluetun
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "wget", "-qO-", "--tries=1", "--timeout=10", "https://ipinfo.io"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 15s
|
||||||
|
retries: 3
|
||||||
|
start_period: 30s
|
||||||
|
|
||||||
|
qbittorrent:
|
||||||
|
image: qbittorrentofficial/qbittorrent-nox:5.0.5
|
||||||
|
container_name: qbittorrent
|
||||||
|
depends_on:
|
||||||
|
gluetun:
|
||||||
|
condition: service_healthy
|
||||||
|
network_mode: "service:gluetun"
|
||||||
|
user: "1000:1000"
|
||||||
|
environment:
|
||||||
|
- QBT_LEGAL_NOTICE=confirm
|
||||||
|
- QBT_WEBUI_PORT=8080
|
||||||
|
- UMASK=022
|
||||||
|
- TZ=${TZ:-Europe/London}
|
||||||
|
volumes:
|
||||||
|
- ./qbittorrent/config:/config
|
||||||
|
- /home/user/media/_downloads:/downloads
|
||||||
|
- /home/user/media:/media
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
prowlarr:
|
||||||
|
image: ghcr.io/hotio/prowlarr:release
|
||||||
|
container_name: prowlarr
|
||||||
|
depends_on:
|
||||||
|
gluetun:
|
||||||
|
condition: service_healthy
|
||||||
|
network_mode: "service:gluetun"
|
||||||
|
environment:
|
||||||
|
- PUID=1000
|
||||||
|
- PGID=1000
|
||||||
|
- UMASK=022
|
||||||
|
- TZ=${TZ:-Europe/London}
|
||||||
|
volumes:
|
||||||
|
- ./prowlarr:/config
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
sonarr:
|
||||||
|
image: ghcr.io/hotio/sonarr:release
|
||||||
|
container_name: sonarr
|
||||||
|
depends_on:
|
||||||
|
gluetun:
|
||||||
|
condition: service_healthy
|
||||||
|
network_mode: "service:gluetun"
|
||||||
|
environment:
|
||||||
|
- PUID=1000
|
||||||
|
- PGID=1000
|
||||||
|
- UMASK=022
|
||||||
|
- TZ=${TZ:-Europe/London}
|
||||||
|
volumes:
|
||||||
|
- ./sonarr:/config
|
||||||
|
- /home/user/media:/media
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
radarr:
|
||||||
|
image: ghcr.io/hotio/radarr:release
|
||||||
|
container_name: radarr
|
||||||
|
depends_on:
|
||||||
|
gluetun:
|
||||||
|
condition: service_healthy
|
||||||
|
network_mode: "service:gluetun"
|
||||||
|
environment:
|
||||||
|
- PUID=1000
|
||||||
|
- PGID=1000
|
||||||
|
- UMASK=022
|
||||||
|
- TZ=${TZ:-Europe/London}
|
||||||
|
volumes:
|
||||||
|
- ./radarr:/config
|
||||||
|
- /home/user/media:/media
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
betaflix-catalog:
|
||||||
|
image: betaflix-catalog:local
|
||||||
|
container_name: betaflix-catalog
|
||||||
|
build:
|
||||||
|
context: ../catalog
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
# NOT bound to gluetun — needs to reach Forgejo + Sonarr/Radarr
|
||||||
|
network_mode: bridge
|
||||||
|
extra_hosts:
|
||||||
|
- "host.docker.internal:host-gateway"
|
||||||
|
environment:
|
||||||
|
- FORGEJO_REMOTE=${FORGEJO_REMOTE:-https://git.s8n.ru/s8n/beta-flix.git}
|
||||||
|
- FORGEJO_PUSH_TOKEN=${FORGEJO_PUSH_TOKEN}
|
||||||
|
- GIT_AUTHOR_NAME=obsidian-ai
|
||||||
|
- GIT_AUTHOR_EMAIL=obsidian-ai@s8n.ru
|
||||||
|
- GIT_COMMITTER_NAME=obsidian-ai
|
||||||
|
- GIT_COMMITTER_EMAIL=obsidian-ai@s8n.ru
|
||||||
|
- SONARR_API_KEY=${SONARR_API_KEY}
|
||||||
|
- RADARR_API_KEY=${RADARR_API_KEY}
|
||||||
|
- TZ=${TZ:-Europe/London}
|
||||||
|
- LISTEN_PORT=5055
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:5055:5055"
|
||||||
|
volumes:
|
||||||
|
- ./catalog/repo:/repo
|
||||||
|
- ./catalog/ssh:/root/.ssh:ro
|
||||||
|
- ./catalog/state:/state
|
||||||
|
restart: unless-stopped
|
||||||
77
compose/traefik/arr.yml
Normal file
77
compose/traefik/arr.yml
Normal file
|
|
@ -0,0 +1,77 @@
|
||||||
|
# Traefik file-provider snippet for the media-acquisition stack.
|
||||||
|
#
|
||||||
|
# Symlink (or cp) this file into /opt/docker/traefik/config/arr.yml on
|
||||||
|
# nullstone. Traefik picks up file-provider configs without restart.
|
||||||
|
#
|
||||||
|
# All routes are LAN+Tailscale-only (trusted-only@file middleware) AND
|
||||||
|
# require Authentik forward-auth. Add the arr-stack Authentik group as
|
||||||
|
# needed.
|
||||||
|
#
|
||||||
|
# Backends are 127.0.0.1:<port> because gluetun publishes the qbt/prowlarr/
|
||||||
|
# sonarr/radarr ports on host loopback (network_mode: service:gluetun).
|
||||||
|
|
||||||
|
http:
|
||||||
|
routers:
|
||||||
|
qbt:
|
||||||
|
rule: "Host(`qbt.s8n.ru`)"
|
||||||
|
entryPoints: [websecure]
|
||||||
|
service: qbt
|
||||||
|
tls:
|
||||||
|
certResolver: gandi
|
||||||
|
middlewares:
|
||||||
|
- trusted-only@file
|
||||||
|
- authentik-forwardauth@file
|
||||||
|
|
||||||
|
prowlarr:
|
||||||
|
rule: "Host(`prowlarr.s8n.ru`)"
|
||||||
|
entryPoints: [websecure]
|
||||||
|
service: prowlarr
|
||||||
|
tls:
|
||||||
|
certResolver: gandi
|
||||||
|
middlewares:
|
||||||
|
- trusted-only@file
|
||||||
|
- authentik-forwardauth@file
|
||||||
|
|
||||||
|
sonarr:
|
||||||
|
rule: "Host(`sonarr.s8n.ru`)"
|
||||||
|
entryPoints: [websecure]
|
||||||
|
service: sonarr
|
||||||
|
tls:
|
||||||
|
certResolver: gandi
|
||||||
|
middlewares:
|
||||||
|
- trusted-only@file
|
||||||
|
- authentik-forwardauth@file
|
||||||
|
|
||||||
|
radarr:
|
||||||
|
rule: "Host(`radarr.s8n.ru`)"
|
||||||
|
entryPoints: [websecure]
|
||||||
|
service: radarr
|
||||||
|
tls:
|
||||||
|
certResolver: gandi
|
||||||
|
middlewares:
|
||||||
|
- trusted-only@file
|
||||||
|
- authentik-forwardauth@file
|
||||||
|
|
||||||
|
# Catalog service has no public route — Sonarr/Radarr hit it via
|
||||||
|
# host.docker.internal:5055 from inside their gluetun netns.
|
||||||
|
|
||||||
|
services:
|
||||||
|
qbt:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: "http://127.0.0.1:8080"
|
||||||
|
|
||||||
|
prowlarr:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: "http://127.0.0.1:9696"
|
||||||
|
|
||||||
|
sonarr:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: "http://127.0.0.1:8989"
|
||||||
|
|
||||||
|
radarr:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: "http://127.0.0.1:7878"
|
||||||
242
docs/architecture.md
Normal file
242
docs/architecture.md
Normal file
|
|
@ -0,0 +1,242 @@
|
||||||
|
# Architecture — nullstone BitTorrent + Import Pipeline
|
||||||
|
|
||||||
|
Last reviewed: 2026-05-20 against live state of `user@192.168.0.100`.
|
||||||
|
|
||||||
|
**Goal:** kill the `download-on-onyx → rsync → import` round-trip. Land torrents
|
||||||
|
directly on nullstone, through VPN, hardlink into the canonical ARRFLIX library,
|
||||||
|
auto-update the catalog in `git.s8n.ru/s8n/beta-flix`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TL;DR Decisions
|
||||||
|
|
||||||
|
| Question | Decision |
|
||||||
|
|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
|
||||||
|
| Client | `qbittorrentofficial/qbittorrent-nox:5.0.5` (single container, official build, slim) |
|
||||||
|
| VPN binding | **gluetun sidecar + `network_mode: service:gluetun`** (WireGuard, kill-switch built-in). Reuses Proton WG. Replaces `socks-pvpn` for BT only. |
|
||||||
|
| Folder layout | `/home/user/media/_downloads/{incomplete,complete}` (NOT scanned by JF) + hardlinks into existing `movies/tv/...` |
|
||||||
|
| Arr stack? | **Yes for Sonarr/Radarr/Prowlarr, NO for Bazarr/cross-seed (yet)**. Mature rename engine beats bespoke; manual selection still works. |
|
||||||
|
| FS atomic-import | XFS reflinks (`cp --reflink=always`) — same inode cost as hardlinks but allow free path/perm divergence. "Use Hardlinks" toggle works. |
|
||||||
|
| Catalog auto-update | sidecar Python service (`betaflix-catalog`) on Sonarr/Radarr webhooks → patches `MEDIA-LIST.md` → git commit+push to Forgejo. |
|
||||||
|
| GPU | untouched — qbt doesn't need it; Jellyfin keeps its existing passthrough (CPU-only post-driver-issue, separate concern). |
|
||||||
|
|
||||||
|
Override any of this if your gut disagrees — but record an ADR under
|
||||||
|
`docs/decisions/` first.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current State (verified live)
|
||||||
|
|
||||||
|
- `socks-pvpn` container: `serjs/go-socks5-proxy` on `socks-vpn` (172.31.0.0/24).
|
||||||
|
Already provides `socks5://socks-pvpn:1080` with `qbt` user. Egress via host's
|
||||||
|
`wg-pvpn-A` / `wg-pvpn-B` (policy-routed by fwmark `0x51820` / `0x51821`).
|
||||||
|
Proton WG is **host-side**, not in a container.
|
||||||
|
- `jellyfin-stock`: mounts `/home/user/media → /media` bind, in `proxy` network.
|
||||||
|
- xfs at `/dev/sda1 → /home/user/media`, 5.5T total / 3.9T free. Reflink-capable.
|
||||||
|
- No existing Sonarr / Radarr / Prowlarr / gluetun / qbt containers.
|
||||||
|
- Traefik on networks `proxy`, `socket-proxy-net`, `misskey-frontend`.
|
||||||
|
|
||||||
|
Two viable VPN strategies: keep using `socks-pvpn` SOCKS5 with qbt proxy, or
|
||||||
|
drop in a dedicated `gluetun` for the BT stack only. See § b.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## a) qBittorrent image
|
||||||
|
|
||||||
|
**Pick:** `qbittorrentofficial/qbittorrent-nox:5.0.5`.
|
||||||
|
|
||||||
|
- Official upstream build, signed, no LSIO PUID/PGID overhead.
|
||||||
|
- 5.0.x ships native WebUI v2 and modern logging.
|
||||||
|
- Single port: 8080 (WebUI) + chosen listen port (e.g. 51820+random for BT).
|
||||||
|
- Run as uid `1000:1000` (matches `user:user` on host) so anything qbt writes
|
||||||
|
to `/home/user/media/_downloads` already matches library ownership.
|
||||||
|
|
||||||
|
**Skip** `linuxserver/qbittorrent` — extra init scripts, slower updates, PUID
|
||||||
|
drift when paired with userns-remap.
|
||||||
|
|
||||||
|
**Skip** `qbittorrent-nox` bare on host — Docker buys VPN-namespace binding +
|
||||||
|
restart isolation. Cheap.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## b) VPN binding — pick `gluetun`
|
||||||
|
|
||||||
|
Three patterns considered:
|
||||||
|
|
||||||
|
| Pattern | Pro | Con |
|
||||||
|
|-------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
|
||||||
|
| qbt SOCKS5 → `socks-pvpn` | Zero new infra | qbt SOCKS support has historical leaks (UDP, trackers, DHT). Not a kill-switch — if SOCKS dies, qbt uses default route → clear-net leak |
|
||||||
|
| WireGuard inside qbt container | Tight blast radius | Bake wg into image; restart re-attach pain; upgrades painful |
|
||||||
|
| **`gluetun` sidecar, qbt joins its netns**| Mature kill-switch (iptables-enforced), port-forward helper, qbt unchanged | Adds one container; eats a Proton WG slot |
|
||||||
|
|
||||||
|
**Decision: gluetun.** It's a kill-switch by design — if WG drops, gluetun's
|
||||||
|
firewall blackholes traffic. Used by every torrent setup in /r/selfhosted for
|
||||||
|
that reason. Add a third Proton WG endpoint specifically for it (so it doesn't
|
||||||
|
collide with the existing wg-pvpn-A/B host-level policy routes).
|
||||||
|
|
||||||
|
Keep `socks-pvpn` running for other clients.
|
||||||
|
|
||||||
|
### Kill-switch verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec qbittorrent curl -sf --max-time 5 https://api.ipify.org # should return Proton exit IP
|
||||||
|
docker stop gluetun && docker exec qbittorrent curl -sf --max-time 5 https://api.ipify.org # MUST hang/fail
|
||||||
|
docker start gluetun
|
||||||
|
```
|
||||||
|
|
||||||
|
If the second command succeeds, you have a leak — **do not proceed**.
|
||||||
|
|
||||||
|
The wrapper script is at `scripts/killswitch-test.sh`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## c) Folder layout (xfs, single device)
|
||||||
|
|
||||||
|
```
|
||||||
|
/home/user/media/
|
||||||
|
├── _downloads/ # NOT in any JF library → JF can't see it
|
||||||
|
│ ├── incomplete/ # qbt's "temp path" — half-written files
|
||||||
|
│ ├── complete/ # qbt's "save path" — completed, still seeding
|
||||||
|
│ └── watch/ # drop .torrent files here for auto-add
|
||||||
|
├── movies/ # canonical, JF scans (existing)
|
||||||
|
├── tv/ # canonical, JF scans (existing)
|
||||||
|
├── education/ # YouTube creator, JF scans (existing)
|
||||||
|
├── music/ # (existing)
|
||||||
|
└── podcasts/ # (existing)
|
||||||
|
```
|
||||||
|
|
||||||
|
JF library paths are configured in dashboard and only point at the four
|
||||||
|
canonical roots. `_downloads/` is on the same xfs filesystem → hardlinks /
|
||||||
|
reflinks are free (zero extra blocks consumed) when sonarr/radarr import.
|
||||||
|
|
||||||
|
Sonarr/Radarr setting: **Use Hardlinks instead of Copy = yes**.
|
||||||
|
|
||||||
|
Permissions: qbt runs as `1000:1000`, files land 644/dirs 755 (image sets
|
||||||
|
`umask 022`). If defaults drift, force with `UMASK=022` in container env
|
||||||
|
(qbt 5 honors it).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## d, f) Arr stack vs custom watcher — pick Arr
|
||||||
|
|
||||||
|
For 20-40 items in pipeline the bespoke watcher is *tempting*. Pick
|
||||||
|
Sonarr/Radarr anyway:
|
||||||
|
|
||||||
|
**For Sonarr/Radarr (mature rename + import):**
|
||||||
|
|
||||||
|
- Their rename engine handles 100+ edge cases your bash will eventually trip:
|
||||||
|
multi-episode files, anime absolute numbering, special seasons,
|
||||||
|
daily-broadcast dates, year-disambiguated titles. You will hit these.
|
||||||
|
- "Interactive Search" gives manual selection — not forced into RSS auto-grab.
|
||||||
|
- Hardlink-on-import is a checkbox, not a function to debug.
|
||||||
|
- Webhook on import → ready-made trigger for catalog-update.
|
||||||
|
- Library "Scan after import" is built-in. Skip the cargo-cult JF scan task ID
|
||||||
|
dance (keep as manual escape hatch).
|
||||||
|
|
||||||
|
**For Prowlarr:**
|
||||||
|
|
||||||
|
- One-place indexer config. Even if you only use 3 trackers, having them
|
||||||
|
managed in Prowlarr and pushed to Sonarr+Radarr is less duplication.
|
||||||
|
- Categories + capabilities matter when manual-search returns results — you
|
||||||
|
want season-pack vs single-episode discrimination on the search UI.
|
||||||
|
|
||||||
|
**Against (kept honest):**
|
||||||
|
|
||||||
|
- Five extra containers (gluetun, qbt, sonarr, radarr, prowlarr). ~600 MB RAM
|
||||||
|
combined idle. nullstone has 31 G; rounding error.
|
||||||
|
- Sonarr database in SQLite — back up in `./backup.sh`.
|
||||||
|
- More UI surface. Two evenings.
|
||||||
|
|
||||||
|
**Hard NO for now:**
|
||||||
|
|
||||||
|
- **Bazarr** — subtitle pipeline is the WhisperX v4 build, not OpenSubtitles.
|
||||||
|
- **cross-seed** — only useful when seriously seeding to ratio. Defer.
|
||||||
|
- **Lidarr / Readarr** — out of scope (music + books not in this pipeline).
|
||||||
|
|
||||||
|
If after 2 weeks Sonarr's metadata picker is fighting you, **then** swap to
|
||||||
|
bespoke — files on disk are the same shape either way.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## e) Catalog-update service (mandatory regardless)
|
||||||
|
|
||||||
|
Even with Sonarr/Radarr, neither tool knows about
|
||||||
|
`/home/admin/projects/beta-flix/playbooks/import-media/MEDIA-LIST.md`. So:
|
||||||
|
|
||||||
|
`betaflix-catalog` (Python 3.12, Flask, ~200 LoC, in `catalog/`). Listens for
|
||||||
|
Sonarr/Radarr **"On Import"** webhooks. For each event:
|
||||||
|
|
||||||
|
1. Pull metadata from webhook payload (`series.title`, `series.year`,
|
||||||
|
`episodeFile.path`, or `movie.title` + `movie.year` + `movieFile.path`).
|
||||||
|
2. `git -C /repo pull --rebase origin main`.
|
||||||
|
3. Edit `playbooks/import-media/MEDIA-LIST.md`:
|
||||||
|
- Movies: insert into Movies table, alphabetic on title.
|
||||||
|
- TV: if series row exists, merge seasons into the `Seasons` column;
|
||||||
|
else insert new row.
|
||||||
|
- "Source / Version" column = parsed from filename release-group tokens
|
||||||
|
**before** Sonarr stripped them. The webhook gives `sourceTitle`
|
||||||
|
(original release name) — log it raw, you can edit later.
|
||||||
|
- "Why on arrflix" column stays blank — that's human-authored.
|
||||||
|
4. Write run log to `playbooks/import-media/runs/<slug>.md` using a Jinja
|
||||||
|
template (date, source path, target path, item count, ffprobe summary
|
||||||
|
from a `docker exec jellyfin-stock ffprobe` call — optional, deferred).
|
||||||
|
5. `git commit -m "catalog: add <title> (<year>)" --author "obsidian-ai <obsidian-ai@s8n.ru>"`.
|
||||||
|
6. `git push origin main`. Forgejo deploy key in `compose/catalog/ssh/`
|
||||||
|
(gitignored — placed by operator at deploy time).
|
||||||
|
|
||||||
|
Webhook config in Sonarr: Settings → Connect → Webhook → POST to
|
||||||
|
`http://host.docker.internal:5055/sonarr` on `OnImport` event only.
|
||||||
|
|
||||||
|
Idempotency: hash the payload (`{series_id}:{season}:{episode}`); skip if
|
||||||
|
seen in the last hour (Sonarr retries on transient failure). Cache lives at
|
||||||
|
`/tmp/seen-imports.json` (ephemeral; that's fine — duplicate commits are
|
||||||
|
benign-but-noisy, not destructive).
|
||||||
|
|
||||||
|
Skeleton lives at `catalog/catalog.py` in this repo. ~30 minutes to draft,
|
||||||
|
~2 hours to harden. The piece that bridges "files on nullstone" to "facts
|
||||||
|
in Forgejo".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## g) Migration from onyx-qbt → nullstone-qbt
|
||||||
|
|
||||||
|
State: 60+ active torrents on onyx, with download dirs on onyx local disk.
|
||||||
|
|
||||||
|
**Goal:** keep seeding (don't burn ratios) while shifting future downloads to
|
||||||
|
nullstone. Two-phase, no big-bang.
|
||||||
|
|
||||||
|
Full runbook: `docs/migration.md` (and the script `scripts/migrate-onyx.sh`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What this doesn't solve (be aware)
|
||||||
|
|
||||||
|
- **Tracker IP allowlists.** Some private trackers pin sessions to a single
|
||||||
|
IP. Switching from onyx public IP → Proton exit IP will trip them. Check
|
||||||
|
each tracker's rules before migrating — you may need an IP-update request
|
||||||
|
per private tracker. See `docs/trackers.md`.
|
||||||
|
- **Port forwarding via Proton.** gluetun's `VPN_PORT_FORWARDING=on` handles
|
||||||
|
this for Proton, but the forwarded port rotates. Set qbt to use the
|
||||||
|
gluetun-provided port via the gluetun control server (gluetun writes the
|
||||||
|
current port to `/tmp/gluetun/forwarded_port`; qbt's `qBittorrent.conf`
|
||||||
|
needs a wrapper script to read it on start). Known helper image:
|
||||||
|
`caillef/qbittorrent-port-sync` — drop in as a 6th container if seeding
|
||||||
|
ratio matters. Deferred until tracker ratio becomes a real concern.
|
||||||
|
- **Backup.** Add `/opt/docker/media-acquisition/compose/{sonarr,radarr,prowlarr,qbittorrent}/config`
|
||||||
|
to nullstone's `/opt/docker/backup.sh`. SQLite DBs — stop containers
|
||||||
|
briefly or use `sqlite3 .backup` semantics.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open decisions to confirm before implementing
|
||||||
|
|
||||||
|
1. Proton plan slot count — gluetun needs its own WG key. Free slot?
|
||||||
|
2. Which private trackers do you actually use? IP-pinning check.
|
||||||
|
3. Public hostnames for the arr-stack: confirm `qbt/sonarr/radarr/prowlarr.s8n.ru`
|
||||||
|
or pick a sub-zone (`arr.s8n.ru/qbt/`).
|
||||||
|
4. Authentik group for arr-stack access (LAN-only? or also from gravel via
|
||||||
|
Tailscale?).
|
||||||
|
5. Forgejo deploy key — generate now or reuse `obsidian-ai`'s existing key?
|
||||||
|
|
||||||
|
Answer those five and the implementation is ~1 evening of compose + ~2 hours
|
||||||
|
on the catalog service. Migration is a separate weekend.
|
||||||
136
docs/migration.md
Normal file
136
docs/migration.md
Normal file
|
|
@ -0,0 +1,136 @@
|
||||||
|
# Migration — onyx-qbt → nullstone-qbt
|
||||||
|
|
||||||
|
State at time of writing: 60+ active torrents on onyx with download dirs on
|
||||||
|
onyx local disk. **Goal:** keep seeding (don't burn ratios) while shifting
|
||||||
|
future downloads to nullstone. Two-phase, no big-bang.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — Stand up nullstone stack (no migration yet)
|
||||||
|
|
||||||
|
1. **Prep directory tree** on nullstone:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh user@nullstone
|
||||||
|
sudo mkdir -p /home/user/media/_downloads/{incomplete,complete,watch}
|
||||||
|
sudo chown -R user:user /home/user/media/_downloads
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Generate new Proton WG key + provisioning for gluetun.** Don't reuse
|
||||||
|
`wg-pvpn-A` keys (they're host-routed; conflict risk). Log into Proton
|
||||||
|
account → WireGuard → new key → name it `nullstone-gluetun-arr` → save
|
||||||
|
the privkey + assigned address (e.g. `10.2.0.3/32`).
|
||||||
|
|
||||||
|
3. **Drop the privkey + address into `compose/.env`:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/docker/media-acquisition/compose
|
||||||
|
cp .env.example .env
|
||||||
|
${EDITOR:-vi} .env
|
||||||
|
# Set:
|
||||||
|
# PVPN_WG_PRIVKEY=<the new privkey>
|
||||||
|
# PVPN_WG_ADDRESSES=10.2.0.3/32
|
||||||
|
# PVPN_SERVER_COUNTRIES=Netherlands
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Bring up the stack.** Start gluetun + qbt only first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d gluetun qbittorrent
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Kill-switch test (NON-NEGOTIABLE):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/killswitch-test.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
If second curl succeeds → leak. Tear down and debug. Do not proceed.
|
||||||
|
|
||||||
|
6. **Sacrificial torrent.** Pick something legal + big you don't care about
|
||||||
|
(e.g. a Linux distro ISO). Add it via qbt webui, watch it land in
|
||||||
|
`/home/user/media/_downloads/complete/`. Confirm it **does not** appear in JF.
|
||||||
|
|
||||||
|
7. **Bring up the rest of the stack.**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
Configure Prowlarr → Sonarr → Radarr (in that order — Prowlarr pushes
|
||||||
|
indexers downstream). Set "Use Hardlinks instead of Copy = yes" in
|
||||||
|
Sonarr/Radarr Media Management.
|
||||||
|
|
||||||
|
8. **Test arr → import path.** Sonarr Interactive Search → manual grab → import
|
||||||
|
into `/media/tv/...`. Verify catalog service commits to Forgejo.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — Migrate onyx torrents
|
||||||
|
|
||||||
|
For each active torrent on onyx that you want to keep seeding:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On onyx — export .torrent files + qbt's fastresume state
|
||||||
|
mkdir -p /tmp/qbt-migrate
|
||||||
|
cp ~/.local/share/qBittorrent/BT_backup/*.torrent /tmp/qbt-migrate/
|
||||||
|
cp ~/.local/share/qBittorrent/BT_backup/*.fastresume /tmp/qbt-migrate/
|
||||||
|
|
||||||
|
# rsync the actual data files to nullstone first (LAN gigabit)
|
||||||
|
rsync -av --info=progress2 ~/Downloads/qbt/ \
|
||||||
|
user@192.168.0.100:/home/user/media/_downloads/complete/
|
||||||
|
```
|
||||||
|
|
||||||
|
Then on nullstone qbt webui:
|
||||||
|
|
||||||
|
1. Add `.torrent` files in bulk via webui ("Add torrent files…"), save path =
|
||||||
|
`/downloads/complete/`, **uncheck "Start torrent"**.
|
||||||
|
2. Force-recheck each added torrent. qbt matches local files → `100%` → seeding.
|
||||||
|
3. Verify trackers respond. Private trackers may need source-IP rotation —
|
||||||
|
gluetun exit IP differs from onyx public IP. See `docs/trackers.md`.
|
||||||
|
4. On onyx: pause torrents one-by-one as nullstone takes over. Don't stop
|
||||||
|
onyx-qbt entirely until every torrent shows seeding on nullstone for 24h
|
||||||
|
with no tracker errors.
|
||||||
|
|
||||||
|
**Catalog backfill:** for torrents that correspond to already-imported
|
||||||
|
library items, **don't** trigger arr-import — they're already in canonical
|
||||||
|
locations. Just seed from `_downloads/complete/`. Catalog stays accurate.
|
||||||
|
|
||||||
|
For torrents that were mid-download on onyx but never made it into the
|
||||||
|
library: re-add on nullstone, let them complete via VPN, then sonarr/radarr
|
||||||
|
picks them up via the normal path.
|
||||||
|
|
||||||
|
**Estimated migration window:** 1 weekend. ~250 GB rsync over LAN gigabit ≈
|
||||||
|
~30 min wall clock for the data move, then a manual-but-tedious
|
||||||
|
add-and-recheck loop in qbt.
|
||||||
|
|
||||||
|
The wrapper script for steps 1-2 is at `scripts/migrate-onyx.sh`. It does
|
||||||
|
the rsync + builds a `.torrent` index for a follow-up bulk-add. The
|
||||||
|
fastresume-rewrite step is documented inline in the script.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — Decommission onyx-qbt
|
||||||
|
|
||||||
|
After 7 days clean on nullstone:
|
||||||
|
|
||||||
|
1. Stop qbt service on onyx (`systemctl --user stop qbittorrent-nox` or kill
|
||||||
|
the GUI; depends on how it was launched).
|
||||||
|
2. Delete `~/Downloads/qbt/` on onyx (only after confirming no in-flight
|
||||||
|
torrents reference it).
|
||||||
|
3. Update `ai-lab/CLAUDE.md` device registry note if onyx had a
|
||||||
|
"downloads role" annotation. (As of 2026-05-20 it does not — onyx has been
|
||||||
|
the staging host but is not formally documented as such.)
|
||||||
|
4. Optional: keep the `.torrent` files archive on onyx for 30 days as a
|
||||||
|
safety net.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
If nullstone stack starts failing during phase 2:
|
||||||
|
|
||||||
|
- `docker compose down` on nullstone.
|
||||||
|
- Re-enable onyx qbt (Phase 1's stack is non-destructive — onyx torrents still
|
||||||
|
have their data + fastresume).
|
||||||
|
- File an issue + revisit phase 1 step 5 (kill-switch test).
|
||||||
59
docs/trackers.md
Normal file
59
docs/trackers.md
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
# Trackers — schema, IP-pinning, ratio notes
|
||||||
|
|
||||||
|
Single source of truth for what trackers feed this pipeline, and what their
|
||||||
|
quirks are. Per-tracker entries get added by the operator; the schema is
|
||||||
|
below.
|
||||||
|
|
||||||
|
## IP-pinning risk
|
||||||
|
|
||||||
|
Many private trackers **pin sessions to a single source IP**. Switching
|
||||||
|
from onyx public IP → Proton exit IP (via gluetun) will trip them: tracker
|
||||||
|
returns `unauthorized: source IP mismatch` on announce, the torrent stops
|
||||||
|
announcing → seeding stats halt → ratio decays.
|
||||||
|
|
||||||
|
Mitigations, ordered cheapest → most invasive:
|
||||||
|
|
||||||
|
1. **Read the tracker's FAQ first.** Most private trackers have a documented
|
||||||
|
policy: "1 IP, change requires staff" / "rolling IP allowed, contact us
|
||||||
|
after change" / "IP locked to account, no exceptions".
|
||||||
|
2. **Request an IP update** from staff before migrating that torrent.
|
||||||
|
Provide the new Proton exit IP (gluetun reports current exit via
|
||||||
|
`docker exec gluetun cat /tmp/gluetun/ip`).
|
||||||
|
3. **Hot-swap manually:** announce on onyx, immediately re-add on nullstone,
|
||||||
|
force-announce. Some trackers' anti-abuse is rate-limited and won't catch
|
||||||
|
the swap.
|
||||||
|
4. **Multiple exit profiles.** Run two gluetun containers with different
|
||||||
|
Proton server selections (one for tracker A, one for tracker B). Heavy.
|
||||||
|
|
||||||
|
If a tracker rejects all of the above, **leave that torrent on onyx**. The
|
||||||
|
migration is not all-or-nothing; some seedboxes will live forever on the
|
||||||
|
old host. Document the exception in the table below.
|
||||||
|
|
||||||
|
## Per-tracker schema
|
||||||
|
|
||||||
|
Use this table format in this file. **Sort alphabetically by tracker name.**
|
||||||
|
|
||||||
|
| Tracker | URL | Type | IP-Pinning | Ratio Required | Notes |
|
||||||
|
|--------------------|------------------------------|---------|-----------------------|----------------|--------------------------------|
|
||||||
|
| _example.tracker_ | https://_example.tracker_/ | private | locked, request swap | 1.0 over 30d | Staff respond on IRC in < 24h. |
|
||||||
|
| _public.example_ | http://_public.example_/ | public | n/a | n/a | No account, no ratio. |
|
||||||
|
|
||||||
|
(Replace the example rows with real trackers as they are onboarded.)
|
||||||
|
|
||||||
|
## Onboarding a new tracker
|
||||||
|
|
||||||
|
When adding a new private tracker:
|
||||||
|
|
||||||
|
1. Read the tracker's FAQ / rules. Record IP-pinning + ratio policy in the
|
||||||
|
table above.
|
||||||
|
2. Run `scripts/add-tracker.sh <name> <url>` to push it into Prowlarr. The
|
||||||
|
script prompts for cookies / API key as needed.
|
||||||
|
3. Add a row to the per-tracker table above. Commit.
|
||||||
|
4. Monitor first 24h: check Prowlarr → Indexer → Stats for failed-query rate.
|
||||||
|
> 10% failures → recheck the IP-pinning column.
|
||||||
|
|
||||||
|
## Public trackers
|
||||||
|
|
||||||
|
Public trackers (e.g. open BitTorrent indexers) have no IP-pinning concerns
|
||||||
|
but generally bad quality + slow speeds. List them sparingly; prefer private
|
||||||
|
trackers for the long tail of niche media.
|
||||||
74
scripts/add-tracker.sh
Executable file
74
scripts/add-tracker.sh
Executable file
|
|
@ -0,0 +1,74 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/add-tracker.sh — register a tracker with Prowlarr via API.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# PROWLARR_API_KEY=xxx ./add-tracker.sh <indexer-name> <indexer-id>
|
||||||
|
#
|
||||||
|
# Where <indexer-id> is Prowlarr's internal ID for the indexer type (look it
|
||||||
|
# up via `curl /api/v1/indexer/schema` — see "Discovering indexer IDs" below).
|
||||||
|
#
|
||||||
|
# This script POSTs a minimal indexer config to Prowlarr. For trackers that
|
||||||
|
# need cookies / passkeys / 2FA, finish the setup in the Prowlarr webui
|
||||||
|
# afterwards.
|
||||||
|
#
|
||||||
|
# Pre-reqs:
|
||||||
|
# - Prowlarr container up.
|
||||||
|
# - PROWLARR_API_KEY exported (Prowlarr → Settings → General → Security).
|
||||||
|
# - PROWLARR_URL defaults to http://127.0.0.1:9696.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
NAME="${1:-}"
|
||||||
|
INDEXER_ID="${2:-}"
|
||||||
|
PROWLARR_URL="${PROWLARR_URL:-http://127.0.0.1:9696}"
|
||||||
|
PROWLARR_API_KEY="${PROWLARR_API_KEY:-}"
|
||||||
|
|
||||||
|
if [ -z "$NAME" ] || [ -z "$INDEXER_ID" ] || [ -z "$PROWLARR_API_KEY" ]; then
|
||||||
|
cat <<EOF >&2
|
||||||
|
Usage: PROWLARR_API_KEY=xxx $0 <indexer-name> <indexer-id>
|
||||||
|
|
||||||
|
Env:
|
||||||
|
PROWLARR_URL default: http://127.0.0.1:9696
|
||||||
|
PROWLARR_API_KEY required — Prowlarr → Settings → General → Security.
|
||||||
|
|
||||||
|
Discovering indexer IDs:
|
||||||
|
curl -s "\$PROWLARR_URL/api/v1/indexer/schema" \\
|
||||||
|
-H "X-Api-Key: \$PROWLARR_API_KEY" | \\
|
||||||
|
jq -r '.[] | "\\(.implementation)\\t\\(.implementationName)"' | sort -u
|
||||||
|
|
||||||
|
Find the row matching your tracker, then look up its integer id via the
|
||||||
|
full schema entry. Many private trackers use the "Cardigann" implementation
|
||||||
|
with a YAML config — see Prowlarr docs for the full attribute list.
|
||||||
|
EOF
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Querying schema for '$INDEXER_ID'..."
|
||||||
|
SCHEMA_JSON="$(curl -fsS \
|
||||||
|
-H "X-Api-Key: $PROWLARR_API_KEY" \
|
||||||
|
"$PROWLARR_URL/api/v1/indexer/schema" \
|
||||||
|
| jq --arg id "$INDEXER_ID" '.[] | select(.id == ($id | tonumber))')"
|
||||||
|
|
||||||
|
if [ -z "$SCHEMA_JSON" ]; then
|
||||||
|
echo "No schema entry for indexer id=$INDEXER_ID" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Override name with the user-provided one and keep all other fields as-is.
|
||||||
|
PAYLOAD="$(jq --arg name "$NAME" '. + {name: $name, enable: true}' <<<"$SCHEMA_JSON")"
|
||||||
|
|
||||||
|
echo "POSTing indexer config..."
|
||||||
|
RESPONSE="$(curl -fsS -X POST \
|
||||||
|
-H "X-Api-Key: $PROWLARR_API_KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "$PAYLOAD" \
|
||||||
|
"$PROWLARR_URL/api/v1/indexer")"
|
||||||
|
|
||||||
|
NEW_ID="$(jq -r '.id' <<<"$RESPONSE")"
|
||||||
|
echo "OK — indexer '$NAME' added with id=$NEW_ID"
|
||||||
|
echo
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Open Prowlarr → Indexers → $NAME → fill in cookies/passkey/API key."
|
||||||
|
echo " 2. Test indexer (Settings → Indexers → Test)."
|
||||||
|
echo " 3. Add a row to docs/trackers.md with IP-pinning + ratio notes."
|
||||||
|
echo " 4. Push to Sonarr/Radarr via Prowlarr's Apps → Sync."
|
||||||
96
scripts/killswitch-test.sh
Executable file
96
scripts/killswitch-test.sh
Executable file
|
|
@ -0,0 +1,96 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/killswitch-test.sh — verify gluetun blocks traffic when VPN drops.
|
||||||
|
#
|
||||||
|
# Test plan (per docs/architecture.md §b):
|
||||||
|
#
|
||||||
|
# 1. With gluetun UP, qbt MUST resolve api.ipify.org and return an IP
|
||||||
|
# that is NOT nullstone's WAN IP (i.e. Proton exit).
|
||||||
|
# 2. Stop gluetun. qbt's container MUST NOT be able to reach the internet
|
||||||
|
# (curl hangs / fails fast). If it succeeds → kill-switch leak → ABORT.
|
||||||
|
# 3. Restart gluetun, re-verify step 1.
|
||||||
|
#
|
||||||
|
# Run from anywhere with `docker` access on the host. Idempotent.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
GLUETUN="${GLUETUN_CONTAINER:-gluetun}"
|
||||||
|
QBT="${QBT_CONTAINER:-qbittorrent}"
|
||||||
|
TIMEOUT="${TIMEOUT:-8}"
|
||||||
|
|
||||||
|
red() { printf '\033[31m%s\033[0m\n' "$*"; }
|
||||||
|
green() { printf '\033[32m%s\033[0m\n' "$*"; }
|
||||||
|
yellow(){ printf '\033[33m%s\033[0m\n' "$*"; }
|
||||||
|
|
||||||
|
require_container() {
|
||||||
|
if ! docker inspect "$1" >/dev/null 2>&1; then
|
||||||
|
red "FAIL: container '$1' not found"; exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
require_container "$GLUETUN"
|
||||||
|
require_container "$QBT"
|
||||||
|
|
||||||
|
# --- Step 0: discover nullstone's WAN IP (so we can detect leaks).
|
||||||
|
WAN_IP="$(curl -sf --max-time "$TIMEOUT" https://api.ipify.org || true)"
|
||||||
|
if [ -z "$WAN_IP" ]; then
|
||||||
|
yellow "WARN: could not fetch host WAN IP — leak detection will be best-effort"
|
||||||
|
else
|
||||||
|
echo "Host WAN IP: $WAN_IP"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- Step 1: VPN-up check
|
||||||
|
echo
|
||||||
|
echo "Step 1: VPN-up — qbt should exit via Proton."
|
||||||
|
if ! docker inspect -f '{{.State.Running}}' "$GLUETUN" | grep -q true; then
|
||||||
|
yellow "gluetun not running — starting…"
|
||||||
|
docker start "$GLUETUN" >/dev/null
|
||||||
|
sleep 5
|
||||||
|
fi
|
||||||
|
VPN_IP="$(docker exec "$QBT" curl -sf --max-time "$TIMEOUT" https://api.ipify.org || true)"
|
||||||
|
if [ -z "$VPN_IP" ]; then
|
||||||
|
red "FAIL: qbt could not reach api.ipify.org even with gluetun up. Check VPN config."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "qbt sees IP: $VPN_IP"
|
||||||
|
if [ -n "$WAN_IP" ] && [ "$VPN_IP" = "$WAN_IP" ]; then
|
||||||
|
red "FAIL: qbt's IP == host WAN IP. Traffic is NOT going through VPN."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
green "OK: qbt egressing via VPN."
|
||||||
|
|
||||||
|
# --- Step 2: kill-switch check
|
||||||
|
echo
|
||||||
|
echo "Step 2: kill-switch — stop gluetun, qbt MUST fail to reach internet."
|
||||||
|
docker stop "$GLUETUN" >/dev/null
|
||||||
|
sleep 2
|
||||||
|
set +e
|
||||||
|
LEAK_IP="$(docker exec "$QBT" curl -sf --max-time "$TIMEOUT" https://api.ipify.org 2>/dev/null)"
|
||||||
|
RC=$?
|
||||||
|
set -e
|
||||||
|
docker start "$GLUETUN" >/dev/null
|
||||||
|
if [ $RC -eq 0 ] && [ -n "$LEAK_IP" ]; then
|
||||||
|
red "FAIL: kill-switch broken — qbt reached the internet with VPN down."
|
||||||
|
red " Leaked IP: $LEAK_IP"
|
||||||
|
red " Tear down the stack and investigate before adding any torrents."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
green "OK: qbt could not reach internet with gluetun stopped."
|
||||||
|
|
||||||
|
# --- Step 3: re-verify VPN comes back
|
||||||
|
echo
|
||||||
|
echo "Step 3: VPN restart — wait for gluetun to be healthy again."
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
if [ "$(docker inspect -f '{{.State.Health.Status}}' "$GLUETUN" 2>/dev/null || echo none)" = "healthy" ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
RECOVERY_IP="$(docker exec "$QBT" curl -sf --max-time "$TIMEOUT" https://api.ipify.org || true)"
|
||||||
|
if [ -z "$RECOVERY_IP" ]; then
|
||||||
|
yellow "WARN: gluetun did not recover within 60s. Check 'docker logs $GLUETUN'."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
green "OK: VPN recovered. qbt sees IP $RECOVERY_IP."
|
||||||
|
|
||||||
|
echo
|
||||||
|
green "Kill-switch test PASSED. Safe to seed."
|
||||||
109
scripts/migrate-onyx.sh
Executable file
109
scripts/migrate-onyx.sh
Executable file
|
|
@ -0,0 +1,109 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/migrate-onyx.sh — Phase 2 migration helper.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./migrate-onyx.sh <source-dir-on-onyx> <target-dir-on-nullstone>
|
||||||
|
#
|
||||||
|
# Example:
|
||||||
|
# ./migrate-onyx.sh "$HOME/Downloads/qbt/" \
|
||||||
|
# /home/user/media/_downloads/complete/
|
||||||
|
#
|
||||||
|
# What it does:
|
||||||
|
# 1. rsync <source-dir> → user@nullstone:<target-dir> over LAN.
|
||||||
|
# 2. Copies onyx's .torrent + .fastresume files to /tmp/qbt-migrate/
|
||||||
|
# on nullstone (you mass-add them via qbt webui afterwards).
|
||||||
|
# 3. Prints a checklist of remaining manual steps.
|
||||||
|
#
|
||||||
|
# Pre-reqs:
|
||||||
|
# - run from onyx (the SOURCE machine).
|
||||||
|
# - ssh user@nullstone reachable on LAN.
|
||||||
|
# - nullstone qbt stack already up (Phase 1 complete) — check with
|
||||||
|
# `bash killswitch-test.sh` first.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SRC="${1:-}"
|
||||||
|
DST="${2:-}"
|
||||||
|
NULLSTONE="${NULLSTONE_SSH:-user@192.168.0.100}"
|
||||||
|
DRY_RUN="${DRY_RUN:-1}"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<EOF
|
||||||
|
Usage: $0 <source-dir> <target-dir-on-nullstone>
|
||||||
|
|
||||||
|
Env:
|
||||||
|
NULLSTONE_SSH default: user@192.168.0.100
|
||||||
|
DRY_RUN default: 1 (rsync --dry-run). Set DRY_RUN=0 to actually copy.
|
||||||
|
|
||||||
|
Example (dry-run):
|
||||||
|
$0 "\$HOME/Downloads/qbt/" /home/user/media/_downloads/complete/
|
||||||
|
|
||||||
|
Example (real):
|
||||||
|
DRY_RUN=0 $0 "\$HOME/Downloads/qbt/" /home/user/media/_downloads/complete/
|
||||||
|
EOF
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
[ -z "$SRC" ] || [ -z "$DST" ] && usage
|
||||||
|
[ -d "$SRC" ] || { echo "Source dir not found: $SRC" >&2; exit 1; }
|
||||||
|
|
||||||
|
QBT_BACKUP="$HOME/.local/share/qBittorrent/BT_backup"
|
||||||
|
[ -d "$QBT_BACKUP" ] || { echo "qBittorrent BT_backup dir missing: $QBT_BACKUP" >&2; exit 1; }
|
||||||
|
|
||||||
|
RSYNC_FLAGS=(-av --info=progress2 --partial --human-readable)
|
||||||
|
if [ "$DRY_RUN" = "1" ]; then
|
||||||
|
RSYNC_FLAGS+=(--dry-run)
|
||||||
|
echo "=== DRY-RUN — no data will be copied. Set DRY_RUN=0 to run for real. ==="
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Step 1/3: rsync data files to nullstone ==="
|
||||||
|
rsync "${RSYNC_FLAGS[@]}" "$SRC" "$NULLSTONE:$DST"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== Step 2/3: ship .torrent + .fastresume to nullstone /tmp/qbt-migrate/ ==="
|
||||||
|
ssh "$NULLSTONE" "mkdir -p /tmp/qbt-migrate"
|
||||||
|
if [ "$DRY_RUN" = "1" ]; then
|
||||||
|
echo "(dry-run) would scp $QBT_BACKUP/*.torrent $QBT_BACKUP/*.fastresume → $NULLSTONE:/tmp/qbt-migrate/"
|
||||||
|
else
|
||||||
|
scp -q "$QBT_BACKUP"/*.torrent "$QBT_BACKUP"/*.fastresume "$NULLSTONE:/tmp/qbt-migrate/"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== Step 3/3: remaining manual steps ==="
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
Manual steps on the nullstone qbt webui (https://qbt.s8n.ru):
|
||||||
|
|
||||||
|
1. "Add Torrent" → multi-select all files in /tmp/qbt-migrate/*.torrent
|
||||||
|
Save path: $DST
|
||||||
|
[ ] UNCHECK "Start torrent" — we want them queued, not auto-resumed.
|
||||||
|
|
||||||
|
2. Select all newly-added torrents → right-click → "Force recheck"
|
||||||
|
qbt will hash-match files in $DST → mark each at 100% → start seeding.
|
||||||
|
|
||||||
|
3. Check tracker status per torrent. Private trackers may reject the new
|
||||||
|
source IP (Proton exit). See docs/trackers.md for the per-tracker
|
||||||
|
mitigation playbook.
|
||||||
|
|
||||||
|
4. On onyx (THIS machine): pause torrents one-by-one as nullstone takes
|
||||||
|
over each. Do NOT stop onyx-qbt entirely until every torrent shows
|
||||||
|
seeding on nullstone for 24h with zero tracker errors.
|
||||||
|
|
||||||
|
Fastresume path-rewrite (optional, only if save_path drift breaks recheck):
|
||||||
|
|
||||||
|
ssh $NULLSTONE 'python3 - <<PYEOF
|
||||||
|
import os, re, sys
|
||||||
|
from pathlib import Path
|
||||||
|
src_prefix = "/home/admin/Downloads/qbt" # onyx path
|
||||||
|
dst_prefix = "/downloads/complete" # nullstone path inside qbt container
|
||||||
|
for fr in Path("/tmp/qbt-migrate").glob("*.fastresume"):
|
||||||
|
data = fr.read_bytes()
|
||||||
|
if src_prefix.encode() in data:
|
||||||
|
new = data.replace(src_prefix.encode(), dst_prefix.encode())
|
||||||
|
fr.write_bytes(new)
|
||||||
|
print("rewrote:", fr.name)
|
||||||
|
PYEOF'
|
||||||
|
|
||||||
|
(Only run if force-recheck fails — qbt 5 usually handles re-pathing via
|
||||||
|
the UI's "Save path" field on add.)
|
||||||
|
EOF
|
||||||
Reference in a new issue