Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 10583d2e-7737-44e5-a0fa-fd9a71931deb

Status: imported; action: update

Type

asset

Action

update

Status

imported

Items

1

Bytes

364.3 MB

Source host

orbit-2

Manifest sha256

61345fa3c8e2a31e...

Started

2026-06-01 21:38:25

Completed

2026-06-01 21:57:19

Manifest summary

  • created_at: 2026-06-01T21:35:35Z
  • completed_at: 2026-06-01T21:35:35Z
  • verification.passed: true
  • verification.total_files: 241
  • prior_sync_id: 9c43d4e6-3e75-4b45-9ed3-3b55d2903882

Job history 3

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
10donecursor-agent-requeue2026-06-01 21:57:172026-06-01 21:57:17vultr/pid:14030251/12026-06-01 22:01:43-
9canceledcursor-agent-requeue2026-06-01 21:40:222026-06-01 21:40:25vultr/pid:14030251/12026-06-01 22:01:43superseded by deliverable refresh job 10
8failedcursor-agent2026-06-01 21:38:252026-06-01 21:38:30vultr/pid:13452810/12026-06-01 21:38:30manifest load failed: manifest missing: /mnt/s3-sync/assets/sync/10583d2e-7737-44e5-a0fa-fd9a71931deb.json

Items 1

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video66c6d940e2db2715...9a36d718dff64186...updatedasset:67-2026-06-01 21:57:19

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
scan{"source": "filesystem", "manifest_sha256": "61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a", "sync_type": "asset", "action": "update", "item_count": "1", "total_bytes": "381955222",...2026-06-27 11:50:00
scan{"source": "filesystem", "manifest_sha256": "61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a", "sync_type": "asset", "action": "update", "item_count": "1", "total_bytes": "381955222",...2026-06-27 11:49:32
job_finished{"phase": "job_finish_cleanup", "job_id": 10, "terminal": "done", "reason": "superseded earlier done row after deliverable refresh"}2026-06-01 22:01:43
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":0,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":1}}2026-06-01 21:57:19
item_updated{"cover_hash":"9a36d718dff64186b2037ea72b7a3ae3","error":null,"hash":"66c6d940e2db27158de3a5b6202d7470","phase":"item_import","target_ids":{"asset_id":"67"},"type":"video"}2026-06-01 21:57:19
import_started{"action":"update","already_terminal_count":0,"container_count":1,"item_count":1,"manifest_sha256":"61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a","max_items":0,"phase":"import"}2026-06-01 21:57:17
job_claimed{"job_id":"10","phase":"job_claim","priority":100,"worker_id":"vultr/pid:1403025"}2026-06-01 21:57:17
preflight_finished{"counts":{"will_create":0,"will_fail":0,"will_link":0,"will_skip":0,"will_update":1},"failure_samples":[],"item_count":1,"item_type_counts":{"video":1},"ok":1,"phase":"preflight"}2026-06-01 21:57:17
preflight_started{"action":"update","container_count":1,"item_count":1,"item_type_counts":{"video":1},"manifest_sha256":"61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a","phase":"preflight","shape_err...2026-06-01 21:57:17
job_finished{"error":null,"job_id":"9","phase":"job_finish","summary":{"items_seen":1,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":0,"deferred":0,"failed":0,"l...2026-06-01 21:40:26
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":0,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":1}}2026-06-01 21:40:26
item_updated{"cover_hash":"9a36d718dff64186b2037ea72b7a3ae3","error":null,"hash":"66c6d940e2db27158de3a5b6202d7470","phase":"item_import","target_ids":{"asset_id":"67"},"type":"video"}2026-06-01 21:40:26
import_started{"action":"update","already_terminal_count":0,"container_count":1,"item_count":1,"manifest_sha256":"61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a","max_items":0,"phase":"import"}2026-06-01 21:40:25
job_claimed{"job_id":"9","phase":"job_claim","priority":100,"worker_id":"vultr/pid:1403025"}2026-06-01 21:40:25
preflight_finished{"counts":{"will_create":0,"will_fail":0,"will_link":0,"will_skip":0,"will_update":1},"failure_samples":[],"item_count":1,"item_type_counts":{"video":1},"ok":1,"phase":"preflight"}2026-06-01 21:40:22
preflight_started{"action":"update","container_count":1,"item_count":1,"item_type_counts":{"video":1},"manifest_sha256":"61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a","phase":"preflight","shape_err...2026-06-01 21:40:21
job_finished{"error":"manifest load failed: manifest missing: /mnt/s3-sync/assets/sync/10583d2e-7737-44e5-a0fa-fd9a71931deb.json","job_id":"8","phase":"job_finish","summary":{"error":"manifest missing: /mnt/s3-sy...2026-06-01 21:38:30
job_claimed{"job_id":"8","phase":"job_claim","priority":100,"worker_id":"vultr/pid:1345281"}2026-06-01 21:38:30
preflight_finished{"counts":{"will_create":0,"will_fail":0,"will_link":0,"will_skip":0,"will_update":1},"failure_samples":[],"item_count":1,"item_type_counts":{"video":1},"ok":1,"phase":"preflight"}2026-06-01 21:38:25
preflight_started{"action":"update","container_count":1,"item_count":1,"item_type_counts":{"video":1},"manifest_sha256":"61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a","phase":"preflight","shape_err...2026-06-01 21:38:25
scan{"source": "agent", "manifest_sha256": "61345fa3c8e2a31e5c6eafdee6c9d243cd818b71603b45a67a2e65f7afcbed6a", "sync_type": "asset", "action": "update", "item_count": "1", "total_bytes": "381955222", "man...2026-06-01 21:38:25