Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch d69593c7-324c-48a2-900b-62248ff70426

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

43

Bytes

2.29 GB

Source host

orbit-2

Manifest sha256

46bd5f18edd16160...

Started

2026-06-28 22:48:21

Completed

2026-06-28 22:50:26

Manifest summary

  • created_at: 2026-06-28T22:47:25Z
  • completed_at: 2026-06-28T22:47:25Z
  • verification.passed: true
  • verification.total_files: 6322
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
20doneagent-e2e2026-06-28 22:48:262026-06-28 22:48:30vultr/pid:349192843/432026-06-28 22:50:26-

Items 43

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video68f9de8a46525d11...814b3f741b0c49e2...createdasset:2479-2026-06-28 22:48:38
reel26f6f30bff604f14...-createdasset:2480-2026-06-28 22:48:41
reelb517ac6cba31a680...-createdasset:2481-2026-06-28 22:48:43
reel30f05f589e13c140...-createdasset:2482-2026-06-28 22:48:46
reel8ec1330799b3e628...-createdasset:2483-2026-06-28 22:48:49
reel3b14a69ec8b415b1...-createdasset:2484-2026-06-28 22:48:51
reelb2a3cfb8189ac408...-createdasset:2485-2026-06-28 22:48:54
reeleac553c2ed46cd99...1fd2ce80f18f4af6...createdasset:2486-2026-06-28 22:48:57
reel75df5fbfa53bf7b4...5aaa68f3fb914203...createdasset:2487-2026-06-28 22:49:01
reelaa6366e4bae481af...29e6855a1cb94079...createdasset:2488-2026-06-28 22:49:04
reeld2034f673b68bec0...fe6bac1de1554f03...createdasset:2489-2026-06-28 22:49:08
reelaa91a3a0e2428134...da3ec3f3d33945ea...createdasset:2490-2026-06-28 22:49:12
reelb08849f3e27126f3...06148adacdf84099...createdasset:2491-2026-06-28 22:49:15
reelc1023153a8539b19...550db9bdd3dc4397...createdasset:2492-2026-06-28 22:49:19
reel8da6d87f8759b83a...fa3aad0dd299462f...createdasset:2493-2026-06-28 22:49:22
reelb66215776478ace1...4e9e65e6a4414d27...createdasset:2494-2026-06-28 22:49:26
reel7e1748d4422a18aa...0699302292494fdc...createdasset:2495-2026-06-28 22:49:29
reel8a28c5990ce3d4fa...-createdasset:2496-2026-06-28 22:49:32
reel1e3b0216567aea76...-createdasset:2497-2026-06-28 22:49:35
reelce1dc6e5ae435c6a...-createdasset:2498-2026-06-28 22:49:38
reelc41314a5e7fdc929...-createdasset:2499-2026-06-28 22:49:40
reel453e1de49378d57d...-createdasset:2500-2026-06-28 22:49:43
reelc43240047fc4fb87...-createdasset:2501-2026-06-28 22:49:46
photo2b230767918beb1c...c02d87d9095a4ac5...createdasset:2502-2026-06-28 22:49:48
photo1edf33295fe4a5e8...709991a9b3e4416b...createdasset:2503-2026-06-28 22:49:50
photod46152c1b405309d...de805eea08fc43cb...createdasset:2504-2026-06-28 22:49:52
photo52e8ed08aa690a83...76e89295d8d74ef0...createdasset:2505-2026-06-28 22:49:54
photo773194b6a4808da3...f83d105a56a741a8...createdasset:2506-2026-06-28 22:49:56
photoeb2010891f3c0497...aa788769a6374d7b...createdasset:2507-2026-06-28 22:49:58
photo00803b447e76c451...82460af6dfda4487...createdasset:2508-2026-06-28 22:50:00
photoc8d2bf761a00715f...91d003f4404b49a5...createdasset:2509-2026-06-28 22:50:02
photo95bcda473ea2f020...c6026ee9ea8e49d5...createdasset:2510-2026-06-28 22:50:04
photof9766a81d5b48826...6b319ede0781444e...createdasset:2511-2026-06-28 22:50:06
photo293182f3716bc75b...70c7db5553854ba1...createdasset:2512-2026-06-28 22:50:08
photo4037a4f13c0dc97b...2e41fc230e3343c6...createdasset:2513-2026-06-28 22:50:10
photobb28ddf13e135992...712a286459784026...createdasset:2514-2026-06-28 22:50:11
photo253ffb373ee3493a...330e8e51a9264c7e...createdasset:2515-2026-06-28 22:50:13
photofef01408f5ec5c17...7baff18e1bb541c1...createdasset:2516-2026-06-28 22:50:15
photobc0d29b3da3ce3e8...233e6a4cce554296...createdasset:2517-2026-06-28 22:50:18
photob63e4a7ec26bd41a...a650a42a78184457...createdasset:2518-2026-06-28 22:50:20
photo7a0908cdb915d379...31c988ebc73a4e54...createdasset:2519-2026-06-28 22:50:22
photo1ef2b5bac5906ad8...e97f871324a7471c...createdasset:2520-2026-06-28 22:50:24
photo02436a21e785ed29...e143fe9ba9974239...createdasset:2521-2026-06-28 22:50:25

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"20","phase":"job_finish","summary":{"items_seen":43,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":43,"deferred":0,"failed":0...2026-06-28 22:50:26
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":43,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-28 22:50:26
item_updated{"container":1,"expected_member_count":20,"gallery_hash":"7c8133013f972059ee1dc6826a605d4f","gallery_id":"15","linked_member_count":20,"missing_member_count":0,"phase":"container_link","title":"Its Co...2026-06-28 22:50:26
item_created{"cover_hash":"e143fe9ba9974239acfd00f3318451c5","error":null,"hash":"02436a21e785ed29255730105d7ab6f0","phase":"item_import","target_ids":{"asset_id":"2521"},"type":"photo"}2026-06-28 22:50:25
item_created{"cover_hash":"e97f871324a7471cac9cf74e1930fd14","error":null,"hash":"1ef2b5bac5906ad8f932e8ae93ca9bb8","phase":"item_import","target_ids":{"asset_id":"2520"},"type":"photo"}2026-06-28 22:50:24
item_created{"cover_hash":"31c988ebc73a4e54a7b199c98a5bc2a6","error":null,"hash":"7a0908cdb915d3799050599754b59dc4","phase":"item_import","target_ids":{"asset_id":"2519"},"type":"photo"}2026-06-28 22:50:22
item_created{"cover_hash":"a650a42a7818445784c95a19db6eb57a","error":null,"hash":"b63e4a7ec26bd41aab369a58b9da1b8d","phase":"item_import","target_ids":{"asset_id":"2518"},"type":"photo"}2026-06-28 22:50:20
item_created{"cover_hash":"233e6a4cce554296ad496f9e3ea0d362","error":null,"hash":"bc0d29b3da3ce3e87a9cc325222d3405","phase":"item_import","target_ids":{"asset_id":"2517"},"type":"photo"}2026-06-28 22:50:18
item_created{"cover_hash":"7baff18e1bb541c1a0266b6201dbe2e0","error":null,"hash":"fef01408f5ec5c179225add6cfa3767b","phase":"item_import","target_ids":{"asset_id":"2516"},"type":"photo"}2026-06-28 22:50:15
item_created{"cover_hash":"330e8e51a9264c7ea83d95a865fa0da6","error":null,"hash":"253ffb373ee3493ad7823db4711b4717","phase":"item_import","target_ids":{"asset_id":"2515"},"type":"photo"}2026-06-28 22:50:13
item_created{"cover_hash":"712a286459784026b7e3307dc934c78a","error":null,"hash":"bb28ddf13e135992f039c3330b6e83d4","phase":"item_import","target_ids":{"asset_id":"2514"},"type":"photo"}2026-06-28 22:50:11
item_created{"cover_hash":"2e41fc230e3343c68c9d006e977dffde","error":null,"hash":"4037a4f13c0dc97b02ec49ac41d9eb17","phase":"item_import","target_ids":{"asset_id":"2513"},"type":"photo"}2026-06-28 22:50:10
item_created{"cover_hash":"70c7db5553854ba1bec323c82af74fa7","error":null,"hash":"293182f3716bc75bb8bff542fb1923b7","phase":"item_import","target_ids":{"asset_id":"2512"},"type":"photo"}2026-06-28 22:50:08
item_created{"cover_hash":"6b319ede0781444e893bddc691e8c202","error":null,"hash":"f9766a81d5b48826cf63bbfc9033f1b4","phase":"item_import","target_ids":{"asset_id":"2511"},"type":"photo"}2026-06-28 22:50:06
item_created{"cover_hash":"c6026ee9ea8e49d5943f399d6b01df14","error":null,"hash":"95bcda473ea2f0207be6633a7941c2f0","phase":"item_import","target_ids":{"asset_id":"2510"},"type":"photo"}2026-06-28 22:50:04
item_created{"cover_hash":"91d003f4404b49a5899521db95dec162","error":null,"hash":"c8d2bf761a00715fb2d62d30abdd77d3","phase":"item_import","target_ids":{"asset_id":"2509"},"type":"photo"}2026-06-28 22:50:02
item_created{"cover_hash":"82460af6dfda4487934bf5e52b4fdf95","error":null,"hash":"00803b447e76c4511b1aa3babda8cb82","phase":"item_import","target_ids":{"asset_id":"2508"},"type":"photo"}2026-06-28 22:50:00
item_created{"cover_hash":"aa788769a6374d7b82d916ff8cd9d287","error":null,"hash":"eb2010891f3c0497ed4a07da8cb02f2f","phase":"item_import","target_ids":{"asset_id":"2507"},"type":"photo"}2026-06-28 22:49:58
item_created{"cover_hash":"f83d105a56a741a8a9cbb671684f2dfa","error":null,"hash":"773194b6a4808da388c1e130792773ce","phase":"item_import","target_ids":{"asset_id":"2506"},"type":"photo"}2026-06-28 22:49:56
item_created{"cover_hash":"76e89295d8d74ef0ba89f2c37ec30af8","error":null,"hash":"52e8ed08aa690a83b47af856515b6e52","phase":"item_import","target_ids":{"asset_id":"2505"},"type":"photo"}2026-06-28 22:49:54
item_created{"cover_hash":"de805eea08fc43cb99264e47c5e82446","error":null,"hash":"d46152c1b405309d28057a208ef1317e","phase":"item_import","target_ids":{"asset_id":"2504"},"type":"photo"}2026-06-28 22:49:52
item_created{"cover_hash":"709991a9b3e4416b9b29bb9340703218","error":null,"hash":"1edf33295fe4a5e8acdfa9b69ef9e297","phase":"item_import","target_ids":{"asset_id":"2503"},"type":"photo"}2026-06-28 22:49:50
item_created{"cover_hash":"c02d87d9095a4ac584b1e8f5574adc0b","error":null,"hash":"2b230767918beb1c9594077e90ffd6e7","phase":"item_import","target_ids":{"asset_id":"2502"},"type":"photo"}2026-06-28 22:49:48
item_created{"cover_hash":null,"error":null,"hash":"c43240047fc4fb87598e002139f0260a","phase":"item_import","target_ids":{"asset_id":"2501"},"type":"reel"}2026-06-28 22:49:46
item_created{"cover_hash":null,"error":null,"hash":"453e1de49378d57d94b41a3d020b54cb","phase":"item_import","target_ids":{"asset_id":"2500"},"type":"reel"}2026-06-28 22:49:43
item_created{"cover_hash":null,"error":null,"hash":"c41314a5e7fdc92900fdf64cd11a38db","phase":"item_import","target_ids":{"asset_id":"2499"},"type":"reel"}2026-06-28 22:49:40
item_created{"cover_hash":null,"error":null,"hash":"ce1dc6e5ae435c6ad7953bc1e76b6f24","phase":"item_import","target_ids":{"asset_id":"2498"},"type":"reel"}2026-06-28 22:49:38
item_created{"cover_hash":null,"error":null,"hash":"1e3b0216567aea76ca55a8a922a7a775","phase":"item_import","target_ids":{"asset_id":"2497"},"type":"reel"}2026-06-28 22:49:35
item_created{"cover_hash":null,"error":null,"hash":"8a28c5990ce3d4fa24e683b7747bcb84","phase":"item_import","target_ids":{"asset_id":"2496"},"type":"reel"}2026-06-28 22:49:32
item_created{"cover_hash":"0699302292494fdc883f914fc760c459","error":null,"hash":"7e1748d4422a18aac01d8ad205cf80c8","phase":"item_import","target_ids":{"asset_id":"2495"},"type":"reel"}2026-06-28 22:49:29
item_created{"cover_hash":"4e9e65e6a4414d278ee33117602658f9","error":null,"hash":"b66215776478ace177c34f8c1fd1fb98","phase":"item_import","target_ids":{"asset_id":"2494"},"type":"reel"}2026-06-28 22:49:26
item_created{"cover_hash":"fa3aad0dd299462f827ebbffe33e0257","error":null,"hash":"8da6d87f8759b83aac3166438e396836","phase":"item_import","target_ids":{"asset_id":"2493"},"type":"reel"}2026-06-28 22:49:22
item_created{"cover_hash":"550db9bdd3dc4397b9319a1ad281eb63","error":null,"hash":"c1023153a8539b192b5ae7cd7cf22cfa","phase":"item_import","target_ids":{"asset_id":"2492"},"type":"reel"}2026-06-28 22:49:19
item_created{"cover_hash":"06148adacdf84099abae5605be4da259","error":null,"hash":"b08849f3e27126f37c3fb7cfee6ec52a","phase":"item_import","target_ids":{"asset_id":"2491"},"type":"reel"}2026-06-28 22:49:15
item_created{"cover_hash":"da3ec3f3d33945eaa59c23bb56a95342","error":null,"hash":"aa91a3a0e2428134f0526bb875dcffcb","phase":"item_import","target_ids":{"asset_id":"2490"},"type":"reel"}2026-06-28 22:49:12
item_created{"cover_hash":"fe6bac1de1554f03b453d6b0de5009bf","error":null,"hash":"d2034f673b68bec083e6a79b994373e8","phase":"item_import","target_ids":{"asset_id":"2489"},"type":"reel"}2026-06-28 22:49:08
item_created{"cover_hash":"29e6855a1cb94079bd6336edd6c1946c","error":null,"hash":"aa6366e4bae481afb70c946777173b43","phase":"item_import","target_ids":{"asset_id":"2488"},"type":"reel"}2026-06-28 22:49:04
item_created{"cover_hash":"5aaa68f3fb914203a1da45d183a0a430","error":null,"hash":"75df5fbfa53bf7b4587f5261c32ffb99","phase":"item_import","target_ids":{"asset_id":"2487"},"type":"reel"}2026-06-28 22:49:01
item_created{"cover_hash":"1fd2ce80f18f4af6a2af5179ce14d4b1","error":null,"hash":"eac553c2ed46cd994aa4418bb75ea543","phase":"item_import","target_ids":{"asset_id":"2486"},"type":"reel"}2026-06-28 22:48:57
item_created{"cover_hash":null,"error":null,"hash":"b2a3cfb8189ac408b541160a8e8003c7","phase":"item_import","target_ids":{"asset_id":"2485"},"type":"reel"}2026-06-28 22:48:54
item_created{"cover_hash":null,"error":null,"hash":"3b14a69ec8b415b1b0f0015508aacdde","phase":"item_import","target_ids":{"asset_id":"2484"},"type":"reel"}2026-06-28 22:48:51
item_created{"cover_hash":null,"error":null,"hash":"8ec1330799b3e628b114589baa551ce4","phase":"item_import","target_ids":{"asset_id":"2483"},"type":"reel"}2026-06-28 22:48:49
item_created{"cover_hash":null,"error":null,"hash":"30f05f589e13c1409de6429fc1eb9752","phase":"item_import","target_ids":{"asset_id":"2482"},"type":"reel"}2026-06-28 22:48:46
item_created{"cover_hash":null,"error":null,"hash":"b517ac6cba31a6804357fbdedce02b7b","phase":"item_import","target_ids":{"asset_id":"2481"},"type":"reel"}2026-06-28 22:48:43
item_created{"cover_hash":null,"error":null,"hash":"26f6f30bff604f14b2f6b9721e29aa40","phase":"item_import","target_ids":{"asset_id":"2480"},"type":"reel"}2026-06-28 22:48:41
item_created{"cover_hash":"814b3f741b0c49e29bb0b2ed134e649e","error":null,"hash":"68f9de8a46525d11354280438abff090","phase":"item_import","target_ids":{"asset_id":"2479"},"type":"video"}2026-06-28 22:48:38
import_started{"action":"import","already_terminal_count":0,"container_count":2,"item_count":43,"manifest_sha256":"46bd5f18edd16160f66233cc6e46ecf0a2deff3e4138c9060d44fd932b0d284b","max_items":0,"phase":"import"}2026-06-28 22:48:31
job_claimed{"job_id":"20","phase":"job_claim","priority":100,"worker_id":"vultr/pid:3491928"}2026-06-28 22:48:30
job_queued{"job_id": "20", "requested_by": "agent-e2e", "items_total": "43"}2026-06-28 22:48:26
preflight_finished{"counts":{"will_create":43,"will_fail":0,"will_link":0,"will_skip":0,"will_update":0},"failure_samples":[],"item_count":43,"item_type_counts":{"photo":20,"reel":22,"video":1},"ok":1,"phase":"prefligh...2026-06-28 22:48:26