Import Media

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/s3-sync/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.

Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

source/source.json -> asset_source (cold provenance).
source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
Bundled cover_hash -> asset_cover linked to the parent asset.
Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/s3-sync mount is read-only by configuration. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Type

asset

Action

import

Status

imported

Items

Bytes

12.9 MB

Source host

flux-1

Manifest sha256

e1e2b9355dba9f5a...

Started

2026-04-30 00:33:57

Completed

2026-04-30 00:34:02

Manifest summary

created_at: 2026-04-13T20:43:00Z
completed_at: 2026-04-13T20:43:26Z
verification.passed: true
verification.total_files: 69
prior_sync_id: none

id	state	requested_by	queued	claimed	worker	progress	finished	last_error
5	done	admin	2026-04-30 00:33:57	2026-04-30 00:34:01	`vultr/pid:355059`	2/2	2026-04-30 00:34:02	-

asset_type	asset_hash	cover_hash	decision	target ids	last_error	updated_at
photo	`c5f6c7f0cd4e87dd...`	-	created	asset:69	-	2026-04-30 00:34:02
cover	`636f1564ed2d49bc...`	-	created	asset:69	-	2026-04-30 00:34:02

event_type	payload	created_at
job_finished	`{"error":null,"job_id":"5","phase":"job_finish","summary":{"items_seen":2,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":2,"deferred":0,"failed":0,"l...`	2026-04-30 00:34:02
import_finished	`{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":2,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}`	2026-04-30 00:34:02
item_updated	`{"container":1,"gallery_hash":"55aa7e0e4f91b8a9c28573ccc8fce63f","phase":"container_link","skip_reason":"gallery_missing","skipped":1}`	2026-04-30 00:34:02
item_updated	`{"container":1,"gallery_hash":"7afebf9618e20f8533707c10c37de6c4","phase":"container_link","skip_reason":"gallery_missing","skipped":1}`	2026-04-30 00:34:02
item_created	`{"cover_hash":null,"error":null,"hash":"636f1564ed2d49bc9698881b7765dedf","phase":"item_import","target_ids":{"asset_id":"69","cover_hash":"636f1564ed2d49bc9698881b7765dedf","parent_hash":"c5f6c7f0cd4...`	2026-04-30 00:34:02
item_created	`{"cover_hash":null,"error":null,"hash":"c5f6c7f0cd4e87dd0d59464d08dc7e95","phase":"item_import","target_ids":{"asset_id":"69"},"type":"photo"}`	2026-04-30 00:34:02
import_started	`{"action":"import","already_terminal_count":0,"container_count":2,"item_count":2,"manifest_sha256":"e1e2b9355dba9f5a1083ac7d7f53c8b82b63a90f2d7c97370011507706db6df1","max_items":0,"phase":"import"}`	2026-04-30 00:34:01
job_claimed	`{"job_id":"5","phase":"job_claim","priority":100,"worker_id":"vultr/pid:355059"}`	2026-04-30 00:34:01
job_queued	`{"auto_preflighted":1,"items_total":"2","job_id":5,"phase":"job_queue","requested_by":"admin"}`	2026-04-30 00:33:57
preflight_finished	`{"counts":{"will_create":2,"will_fail":0,"will_link":0,"will_skip":0,"will_update":0},"failure_samples":[],"item_count":2,"item_type_counts":{"cover":1,"photo":1},"ok":1,"phase":"preflight"}`	2026-04-30 00:33:57
preflight_started	`{"action":"import","container_count":2,"item_count":2,"item_type_counts":{"cover":1,"photo":1},"manifest_sha256":"e1e2b9355dba9f5a1083ac7d7f53c8b82b63a90f2d7c97370011507706db6df1","phase":"preflight",...`	2026-04-30 00:33:57
scan	`{"source": "filesystem", "manifest_sha256": "e1e2b9355dba9f5a1083ac7d7f53c8b82b63a90f2d7c97370011507706db6df1", "sync_type": "asset", "action": "import", "item_count": "2", "total_bytes": "13531704", ...`	2026-04-30 00:29:19

Media Manager