Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 8fccb37c-0689-4caa-857e-1e8f284fe2f6

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

25

Bytes

3.33 GB

Source host

orbit-2

Manifest sha256

89eaa0bc7638aed3...

Started

2026-06-27 18:00:11

Completed

2026-06-27 18:01:13

Manifest summary

  • created_at: 2026-06-27T17:53:06Z
  • completed_at: 2026-06-27T17:53:06Z
  • verification.passed: true
  • verification.total_files: 6058
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
12doneagent-e2e2026-06-27 18:00:142026-06-27 18:00:19vultr/pid:349192825/252026-06-27 18:01:13-

Items 25

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
videoc5bf9ae549af4e5a...99e73e6e5d44439d...createdasset:395-2026-06-27 18:00:26
reel6d1bcfe18f7259dd...f7dc88e84a154048...createdasset:396-2026-06-27 18:00:30
reelc5c7c52ca57ac72e...42262bf545a045e6...createdasset:397-2026-06-27 18:00:34
photo9a966fe0a853d078...41f60183537f4ca8...createdasset:398-2026-06-27 18:00:36
photo6ce31d467aa96b95...abc517177eb84823...createdasset:399-2026-06-27 18:00:37
photo4f36f836a6f1af42...a2453b3b69c949a4...createdasset:400-2026-06-27 18:00:39
photoffad87374ff59e3c...99722bb78bf44189...createdasset:401-2026-06-27 18:00:40
photoe689f77f61e03bb8...a8de705ca30748bd...createdasset:402-2026-06-27 18:00:42
photobd553eb27e99c3ed...278ece8bd9754956...createdasset:403-2026-06-27 18:00:44
photoae6467d076263013...ef0174589013486f...createdasset:404-2026-06-27 18:00:45
photod71384e8e5dca817...8e586fd79b7447df...createdasset:405-2026-06-27 18:00:47
photo79f1ad91a30d0a4f...9e43f8b7770949e7...createdasset:406-2026-06-27 18:00:49
photo2e1b5a3f7186cfcc...2345e6ebe42f4417...createdasset:407-2026-06-27 18:00:51
photo683eb48ab82afa0a...8189b2824a8c48f0...createdasset:408-2026-06-27 18:00:53
photo1845945ba1e41f4c...d22bcb779c13469d...createdasset:409-2026-06-27 18:00:55
photo61b1eada37171ac8...aa531a83b5ea4e83...createdasset:410-2026-06-27 18:00:57
photoe0d4eef7b27149d0...4a8c7f33978f4cf2...createdasset:411-2026-06-27 18:00:58
photob3a3b0af0b8cfd25...d08f760786474ca7...createdasset:412-2026-06-27 18:01:00
photo9be4b682329e1170...c0ff2d29f0e84f65...createdasset:413-2026-06-27 18:01:02
photoa5e6389b1d0eb33b...a9293f8070254482...createdasset:414-2026-06-27 18:01:04
photo33c637de4b615355...ab4e794dad5b4610...createdasset:415-2026-06-27 18:01:06
photoa13c13d88ea6d956...f4e5346287e14a3a...createdasset:416-2026-06-27 18:01:08
photo973316e7fe3021ce...15016c1cada04884...createdasset:417-2026-06-27 18:01:10
photoe714c82bfd0daea9...90925f4828f54276...createdasset:418-2026-06-27 18:01:12
photo60b94cd6c318fe5e...63436afc26e9452d...createdasset:419-2026-06-27 18:01:13

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"12","phase":"job_finish","summary":{"items_seen":25,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":25,"deferred":0,"failed":0...2026-06-27 18:01:13
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":25,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-27 18:01:13
item_updated{"container":1,"expected_member_count":22,"gallery_hash":"31b958c8ab9b6fc851f77f685b93ab79","gallery_id":"7","linked_member_count":22,"missing_member_count":0,"phase":"container_link","title":"AAAAAAA...2026-06-27 18:01:13
item_created{"cover_hash":"63436afc26e9452d9bf436211779ef11","error":null,"hash":"60b94cd6c318fe5e56b15435bdf7111e","phase":"item_import","target_ids":{"asset_id":"419"},"type":"photo"}2026-06-27 18:01:13
item_created{"cover_hash":"90925f4828f54276a6f215996fc2c92e","error":null,"hash":"e714c82bfd0daea9097a8f8e32cc2cc6","phase":"item_import","target_ids":{"asset_id":"418"},"type":"photo"}2026-06-27 18:01:12
item_created{"cover_hash":"15016c1cada048849284963730a413a9","error":null,"hash":"973316e7fe3021cebd6b08b6693131b0","phase":"item_import","target_ids":{"asset_id":"417"},"type":"photo"}2026-06-27 18:01:10
item_created{"cover_hash":"f4e5346287e14a3a8f496dccb41ff9cb","error":null,"hash":"a13c13d88ea6d956a430cc3ca6f311b7","phase":"item_import","target_ids":{"asset_id":"416"},"type":"photo"}2026-06-27 18:01:08
item_created{"cover_hash":"ab4e794dad5b4610bf49ac86cd0c6c0a","error":null,"hash":"33c637de4b615355069af5f61f12ea95","phase":"item_import","target_ids":{"asset_id":"415"},"type":"photo"}2026-06-27 18:01:06
item_created{"cover_hash":"a9293f8070254482b8c8fb0e14db701c","error":null,"hash":"a5e6389b1d0eb33b97f52dfaabf6cfbb","phase":"item_import","target_ids":{"asset_id":"414"},"type":"photo"}2026-06-27 18:01:04
item_created{"cover_hash":"c0ff2d29f0e84f659222f828c0b6869c","error":null,"hash":"9be4b682329e11707fbebc4791d14eb2","phase":"item_import","target_ids":{"asset_id":"413"},"type":"photo"}2026-06-27 18:01:02
item_created{"cover_hash":"d08f760786474ca7ae7862d57ac1f6a5","error":null,"hash":"b3a3b0af0b8cfd250823fa312ade6692","phase":"item_import","target_ids":{"asset_id":"412"},"type":"photo"}2026-06-27 18:01:00
item_created{"cover_hash":"4a8c7f33978f4cf28b5b6dd3d53a28c6","error":null,"hash":"e0d4eef7b27149d00f651aec0571070b","phase":"item_import","target_ids":{"asset_id":"411"},"type":"photo"}2026-06-27 18:00:58
item_created{"cover_hash":"aa531a83b5ea4e838d679b97ca45f917","error":null,"hash":"61b1eada37171ac83693b4fdcaa29f45","phase":"item_import","target_ids":{"asset_id":"410"},"type":"photo"}2026-06-27 18:00:57
item_created{"cover_hash":"d22bcb779c13469d97add0071c4989c0","error":null,"hash":"1845945ba1e41f4c44a2c2461ebf9511","phase":"item_import","target_ids":{"asset_id":"409"},"type":"photo"}2026-06-27 18:00:55
item_created{"cover_hash":"8189b2824a8c48f08b2ee8e93ea108f5","error":null,"hash":"683eb48ab82afa0a968700aa769bb1d3","phase":"item_import","target_ids":{"asset_id":"408"},"type":"photo"}2026-06-27 18:00:53
item_created{"cover_hash":"2345e6ebe42f44179ff4571fbc6e9b37","error":null,"hash":"2e1b5a3f7186cfcc1783383c0664bed0","phase":"item_import","target_ids":{"asset_id":"407"},"type":"photo"}2026-06-27 18:00:51
item_created{"cover_hash":"9e43f8b7770949e78f83daa7bfe50b3f","error":null,"hash":"79f1ad91a30d0a4fd2fa6737233ad045","phase":"item_import","target_ids":{"asset_id":"406"},"type":"photo"}2026-06-27 18:00:49
item_created{"cover_hash":"8e586fd79b7447dfa9e8fb5119f2b288","error":null,"hash":"d71384e8e5dca8174bb2548a569c28ab","phase":"item_import","target_ids":{"asset_id":"405"},"type":"photo"}2026-06-27 18:00:47
item_created{"cover_hash":"ef0174589013486fab8857438f3b264d","error":null,"hash":"ae6467d0762630136a419e71a01fa13a","phase":"item_import","target_ids":{"asset_id":"404"},"type":"photo"}2026-06-27 18:00:45
item_created{"cover_hash":"278ece8bd97549568dbb1abab03b9bcc","error":null,"hash":"bd553eb27e99c3ed4632d85ffb821c9b","phase":"item_import","target_ids":{"asset_id":"403"},"type":"photo"}2026-06-27 18:00:44
item_created{"cover_hash":"a8de705ca30748bd93e81d470f51bbea","error":null,"hash":"e689f77f61e03bb861c04a532a8175d8","phase":"item_import","target_ids":{"asset_id":"402"},"type":"photo"}2026-06-27 18:00:42
item_created{"cover_hash":"99722bb78bf44189b326124897bd2aaa","error":null,"hash":"ffad87374ff59e3cd8a7b77870132d75","phase":"item_import","target_ids":{"asset_id":"401"},"type":"photo"}2026-06-27 18:00:40
item_created{"cover_hash":"a2453b3b69c949a49e59db396de5b42a","error":null,"hash":"4f36f836a6f1af42b23c39ff57146fa6","phase":"item_import","target_ids":{"asset_id":"400"},"type":"photo"}2026-06-27 18:00:39
item_created{"cover_hash":"abc517177eb84823b9b9d53fa8c14029","error":null,"hash":"6ce31d467aa96b955a9893e8e74e0eb0","phase":"item_import","target_ids":{"asset_id":"399"},"type":"photo"}2026-06-27 18:00:37
item_created{"cover_hash":"41f60183537f4ca8afe8b2a4d423c524","error":null,"hash":"9a966fe0a853d078ce971626cfcfed8b","phase":"item_import","target_ids":{"asset_id":"398"},"type":"photo"}2026-06-27 18:00:36
item_created{"cover_hash":"42262bf545a045e6ac97b598af1d7dcd","error":null,"hash":"c5c7c52ca57ac72e2686e84a2e24a886","phase":"item_import","target_ids":{"asset_id":"397"},"type":"reel"}2026-06-27 18:00:34
item_created{"cover_hash":"f7dc88e84a15404890f0ce82972b35c2","error":null,"hash":"6d1bcfe18f7259dd338db8b141886ebc","phase":"item_import","target_ids":{"asset_id":"396"},"type":"reel"}2026-06-27 18:00:30
item_created{"cover_hash":"99e73e6e5d44439da9036d8e68b44f09","error":null,"hash":"c5bf9ae549af4e5a36fed5c94fd0a9d4","phase":"item_import","target_ids":{"asset_id":"395"},"type":"video"}2026-06-27 18:00:26
import_started{"action":"import","already_terminal_count":0,"container_count":2,"item_count":25,"manifest_sha256":"89eaa0bc7638aed3804308d1ab6fa5fdb69b4003557ba94618c8e1b1bc44b6c7","max_items":0,"phase":"import"}2026-06-27 18:00:19
job_claimed{"job_id":"12","phase":"job_claim","priority":100,"worker_id":"vultr/pid:3491928"}2026-06-27 18:00:19
job_queued{"job_id": "12", "requested_by": "agent-e2e", "items_total": "25"}2026-06-27 18:00:14
preflight_finished{"counts":{"will_create":25,"will_fail":0,"will_link":0,"will_skip":0,"will_update":0},"failure_samples":[],"item_count":25,"item_type_counts":{"photo":22,"reel":2,"video":1},"ok":1,"phase":"preflight...2026-06-27 18:00:14
preflight_started{"action":"import","container_count":2,"item_count":25,"item_type_counts":{"photo":22,"reel":2,"video":1},"manifest_sha256":"89eaa0bc7638aed3804308d1ab6fa5fdb69b4003557ba94618c8e1b1bc44b6c7","phase":"...2026-06-27 18:00:11
scan{"source": "agent-harness", "manifest_sha256": "89eaa0bc7638aed3804308d1ab6fa5fdb69b4003557ba94618c8e1b1bc44b6c7", "sync_type": "asset", "action": "import", "item_count": "25"}2026-06-27 18:00:11