Import Media
Queue cloud-sync handover batches and monitor worker runs.
About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.
Purpose
Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).
Flow
- Scan: walks the mount for
.readysentinels and records newingest_batchrows. - Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
- Queue: auto-preflights if needed, then inserts an
import_jobrow. Live progress shows on Import Tasks. - Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.
Example
Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.
Consequences
- The library is site-agnostic. Sites surface library content via
asset_site_membership, gallery/collection membership, content allowlists, and access packages. - VIDEO-SIMPLE syncs import as stable
asset_type=videorows withasset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables. - Cancel is allowed only while a job is queued; once running, only
systemctl stop gpu-admin-panel-import-workerstops it. - Action
removeonly soft-deletes rows viadeleted_at; nothing is hard-deleted. - Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive
Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.
Worker contract
A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).
Sentinel protocol
A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.
Source-of-truth mapping
source/source.json->asset_source(cold provenance).source/data/photo.manifest.json+photo.analysis.json+photo.meaning.json+photo.family.json->asset_ai_runrows with explicitanalysis_kind; unknown filenames are captured withsource_data:<file>.- Bundled
cover_hash->asset_coverlinked to the parent asset. - Containers of type
gallery->asset_gallery+asset_gallery_itemjunctions.
Idempotency + integrity
Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.
Read-only mount + ledger boundary
The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.
Batch b8d72455-cab7-42f2-854f-1a9e54f04353
Status: imported; action: update
Type
asset
Action
update
Status
imported
Items
1
Bytes
582.0 MB
Source host
orbit-2
Manifest sha256
157c5b08a7b547c8...
Started
2026-06-01 22:25:04
Completed
2026-06-01 22:25:10
Manifest summary
- created_at: 2026-06-01T22:23:37Z
- completed_at: 2026-06-01T22:23:37Z
- verification.passed: true
- verification.total_files: 452
- prior_sync_id: f6d2ef02-c0a5-44d1-ab3d-db40bba8335c
Job history 1
Newest first. The active job (if any) is at the top.
| id | state | requested_by | queued | claimed | worker | progress | finished | last_error |
|---|---|---|---|---|---|---|---|---|
| 11 | done | cursor-agent-second-video | 2026-06-01 22:25:04 | 2026-06-01 22:25:05 | vultr/pid:1403025 | 1/1 | 2026-06-01 22:25:10 | - |
Items 1
Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.
| asset_type | asset_hash | cover_hash | decision | target ids | last_error | updated_at |
|---|---|---|---|---|---|---|
| video | f1e2b55cb3bab6dc... | - | updated | asset:114 | - | 2026-06-01 22:25:10 |
Recent events
Append-only audit ledger for this batch (latest 50).
| event_type | payload | created_at |
|---|---|---|
| scan | {"source": "filesystem", "manifest_sha256": "157c5b08a7b547c8e5bb2e0ae0a12ecb82ce4a1e1faa5be1cbac437c6cd3452c", "sync_type": "asset", "action": "update", "item_count": "1", "total_bytes": "610247135",... | 2026-06-27 11:50:00 |
| scan | {"source": "filesystem", "manifest_sha256": "157c5b08a7b547c8e5bb2e0ae0a12ecb82ce4a1e1faa5be1cbac437c6cd3452c", "sync_type": "asset", "action": "update", "item_count": "1", "total_bytes": "610247135",... | 2026-06-27 11:49:31 |
| job_finished | {"error":null,"job_id":"11","phase":"job_finish","summary":{"items_seen":1,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":0,"deferred":0,"failed":0,"... | 2026-06-01 22:25:10 |
| import_finished | {"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":0,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":1}} | 2026-06-01 22:25:10 |
| item_updated | {"cover_hash":null,"error":null,"hash":"f1e2b55cb3bab6dc02bd590196beacb7","phase":"item_import","target_ids":{"asset_id":"114"},"type":"video"} | 2026-06-01 22:25:10 |
| import_started | {"action":"update","already_terminal_count":0,"container_count":1,"item_count":1,"manifest_sha256":"157c5b08a7b547c8e5bb2e0ae0a12ecb82ce4a1e1faa5be1cbac437c6cd3452c","max_items":0,"phase":"import"} | 2026-06-01 22:25:05 |
| job_claimed | {"job_id":"11","phase":"job_claim","priority":100,"worker_id":"vultr/pid:1403025"} | 2026-06-01 22:25:05 |
| preflight_finished | {"counts":{"will_create":0,"will_fail":0,"will_link":0,"will_skip":0,"will_update":1},"failure_samples":[],"item_count":1,"item_type_counts":{"video":1},"ok":1,"phase":"preflight"} | 2026-06-01 22:25:04 |
| preflight_started | {"action":"update","container_count":1,"item_count":1,"item_type_counts":{"video":1},"manifest_sha256":"157c5b08a7b547c8e5bb2e0ae0a12ecb82ce4a1e1faa5be1cbac437c6cd3452c","phase":"preflight","shape_err... | 2026-06-01 22:25:04 |
| scan | {"source": "agent", "manifest_sha256": "157c5b08a7b547c8e5bb2e0ae0a12ecb82ce4a1e1faa5be1cbac437c6cd3452c", "sync_type": "asset", "action": "update", "item_count": "1", "total_bytes": "610247135", "man... | 2026-06-01 22:25:04 |