Import Media
Queue cloud-sync handover batches; the background worker consumes them.
About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.
Purpose
Consume the upstream cloud-sync handover bundles from /mnt/s3-sync/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).
Flow
- Scan: walks the mount for
.readysentinels and records newingest_batchrows. - Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
- Queue: auto-preflights if needed, then inserts an
import_jobrow. Live progress shows on Import Tasks. - Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.
Example
Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.
Consequences
- The library is site-agnostic. Sites surface library content via
asset_site_membership, gallery/collection membership, content allowlists, and access packages. - Cancel is allowed only while a job is queued; once running, only
systemctl stop gpu-admin-panel-import-workerstops it. - Action
removeonly soft-deletes rows viadeleted_at; nothing is hard-deleted. - Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive
Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.
Worker contract
A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).
Sentinel protocol
A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.
Source-of-truth mapping
source/source.json->asset_source(cold provenance).source/data/photo.manifest.json+photo.analysis.json+photo.meaning.json+photo.family.json->asset_ai_runrows with explicitanalysis_kind; unknown filenames are captured withsource_data:<file>.- Bundled
cover_hash->asset_coverlinked to the parent asset. - Containers of type
gallery->asset_gallery+asset_gallery_itemjunctions.
Idempotency + integrity
Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.
Read-only mount + ledger boundary
The /mnt/s3-sync mount is read-only by configuration. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.