Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 21257013-de1c-4110-93e9-008729daedc2

Status: imported; action: import

Type

gallery

Action

import

Status

imported

Items

53

Bytes

2.71 GB

Source host

orbit-2

Manifest sha256

b7d2096880b895b6...

Started

2026-08-12 09:49:25

Completed

2026-08-12 11:59:16

Manifest summary

  • created_at: 2026-08-12T04:21:39Z
  • completed_at: 2026-08-12T04:21:39Z
  • verification.passed: true
  • verification.total_files: 10123
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
387doneimport-feed2026-08-12 09:49:302026-08-12 11:56:03vultr/pid:94621153/532026-08-12 11:59:16-

Items 53

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
photo1a2be43097e4add4...93254913dee24044...createdasset:117443-2026-08-12 11:56:08
photod3ce99176149169a...d81d100bbe4f4910...createdasset:117444-2026-08-12 11:56:11
photo431bc58d366699aa...a277d469522646e9...createdasset:117445-2026-08-12 11:56:15
photo0504c7d139741bbe...30658e28c0614fcd...createdasset:117446-2026-08-12 11:56:19
photod58e0ac5e430fe1b...ff5e4b786a0d4a52...createdasset:117447-2026-08-12 11:56:22
photof3a5cb4ef624ed25...4af3ab5f9a7a4015...createdasset:117448-2026-08-12 11:56:26
photo19d762e47c86e5af...bd0e3ac18c154ac0...createdasset:117449-2026-08-12 11:56:29
photo571c8b2b80b10f61...62f4863ee68d4e64...createdasset:117450-2026-08-12 11:56:33
photo7ac274b513b0e457...716f1fb584b84e0e...createdasset:117451-2026-08-12 11:56:37
photo7078876d2d4b22f4...dd9fd35a6f304f4d...createdasset:117452-2026-08-12 11:56:40
photo8326364c350aedb5...c6da8b22162a43a9...createdasset:117453-2026-08-12 11:56:44
photo03a13de3258cb4ad...763446d881c5429b...createdasset:117454-2026-08-12 11:56:48
photo3d4174a7d6ffb999...62d3b9cf42b347fc...createdasset:117455-2026-08-12 11:56:51
photo1c5db2e126aee3c3...f1181f40a8d745b8...createdasset:117456-2026-08-12 11:56:55
photofce428ed816c7506...b05f4c5e39fd4c5e...createdasset:117457-2026-08-12 11:56:58
photo7973e27f4c2545b8...cbe33a4fd4c04233...createdasset:117458-2026-08-12 11:57:02
photo898ae5b0ee93f55c...06f56ff0649d46bf...createdasset:117459-2026-08-12 11:57:05
photo0dc3e50e2aa1d4f9...0dad3e3726ec4b1b...createdasset:117460-2026-08-12 11:57:09
photob16e38ab48e20beb...6da3d771242440e7...createdasset:117461-2026-08-12 11:57:13
photo5c02678b624638fd...1f4c7f7d22664002...createdasset:117462-2026-08-12 11:57:16
photod52a12d3915d3b58...87c690499eed4ef3...createdasset:117463-2026-08-12 11:57:20
photo975683c2ecab2d76...86f715be83624d31...createdasset:117464-2026-08-12 11:57:23
photo4391c11fe5ad6a9a...6c0a286145d44b80...createdasset:117465-2026-08-12 11:57:27
photob1d23a35a9804117...6a8add173bc44310...createdasset:117466-2026-08-12 11:57:30
photo5494d56ebb68b391...1c403e13acb64df2...createdasset:117467-2026-08-12 11:57:34
photoa107f0482d11dde2...07fcb690d50845ad...createdasset:117468-2026-08-12 11:57:37
photoe22899e8db515bb1...b0aa2dbb9bf34faf...createdasset:117469-2026-08-12 11:57:41
photoa9e3c5a884158572...c8a31ac46d944d60...createdasset:117470-2026-08-12 11:57:45
photo2e6fd74b765ca9dd...983424b3602843f2...createdasset:117471-2026-08-12 11:57:49
photoa85d3dcb4a4e4b1b...94db96c4688244c4...createdasset:117472-2026-08-12 11:57:53
photoc39a8060894a0f16...1776d0ae4b4a4f22...createdasset:117473-2026-08-12 11:57:57
photo73a4a1abc76ae16c...e3e5a72b5fe64720...createdasset:117474-2026-08-12 11:58:00
photoe4a30306af421a62...525951e5d34f49e4...createdasset:117475-2026-08-12 11:58:03
photo920ffa76cc8d1291...e4de39f757fb4aee...createdasset:117476-2026-08-12 11:58:07
photo6d43f2897c04db32...075a013f7b8e477e...createdasset:117477-2026-08-12 11:58:11
photo6e028f734ec14e4f...ba52214ad2cb4ced...createdasset:117478-2026-08-12 11:58:14
photo70b01831f07e2b04...ae60621a052c4dee...createdasset:117479-2026-08-12 11:58:18
photoee93d81a90fabd3d...0cda4c80272a4156...createdasset:117480-2026-08-12 11:58:22
photoe12a1dc730cde8d5...ad55d911054a49c6...createdasset:117481-2026-08-12 11:58:25
photoa24bdbecb7792714...4cfb87930f844242...createdasset:117482-2026-08-12 11:58:29
photo71966cdaa2be888c...304bcdf617984a79...createdasset:117483-2026-08-12 11:58:32
photo74d3898f6a627b21...a2433407346640fa...createdasset:117484-2026-08-12 11:58:36
photo9d2076eb81b11489...2bc5cdae80964b69...createdasset:117485-2026-08-12 11:58:39
photoee18ef7fbcb68b36...06fc898aa1164b16...createdasset:117486-2026-08-12 11:58:43
photo7635103cf74b5696...262de8bc1cb848d7...createdasset:117487-2026-08-12 11:58:46
photo625b8696e61951e8...9408455e34e74764...createdasset:117488-2026-08-12 11:58:50
photo682ac53f8be857ea...397102f31a1c4760...createdasset:117489-2026-08-12 11:58:54
photo70bee9c784e4bed8...c1a040c9c3ab49aa...createdasset:117490-2026-08-12 11:58:57
photo50447e4442226c45...dbbc6701d89644ef...createdasset:117491-2026-08-12 11:59:02
photo627bedae0f200a48...2bff9a0c14094389...createdasset:117492-2026-08-12 11:59:06
photo56dcbdf392c3eac2...494d0777b8cc4ca2...createdasset:117493-2026-08-12 11:59:09
photo6cce3ada85ab1b39...2f441cfdbfff4ff5...createdasset:117494-2026-08-12 11:59:12
photoa4b1d456d69beb03...2fb463e945ab4cfa...createdasset:117495-2026-08-12 11:59:16

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"387","phase":"job_finish","summary":{"items_seen":53,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":53,"deferred":0,"failed":...2026-08-12 11:59:16
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":53,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-08-12 11:59:16
item_updated{"container":1,"expected_member_count":15,"gallery_hash":"17dae961829dd586d6da1b38a4175174","gallery_id":"3672","linked_member_count":15,"missing_member_count":0,"phase":"container_link","title":"Nude...2026-08-12 11:59:16
item_updated{"container":1,"expected_member_count":17,"gallery_hash":"b196ffbaf5c48cb0f514772c902a576d","gallery_id":"3671","linked_member_count":17,"missing_member_count":0,"phase":"container_link","title":"Pool...2026-08-12 11:59:16
item_updated{"container":1,"expected_member_count":21,"gallery_hash":"e8f0846a3388cafa0e8924e47e3e93b3","gallery_id":"3670","linked_member_count":21,"missing_member_count":0,"phase":"container_link","title":"Natu...2026-08-12 11:59:16
item_created{"cover_hash":"2fb463e945ab4cfaaf447dc552f1346f","error":null,"hash":"a4b1d456d69beb03bace428ad9a03f8c","phase":"item_import","target_ids":{"asset_id":"117495"},"type":"photo"}2026-08-12 11:59:16
item_created{"cover_hash":"2f441cfdbfff4ff5b45f1187b116b6bd","error":null,"hash":"6cce3ada85ab1b39565a222cabdec1c9","phase":"item_import","target_ids":{"asset_id":"117494"},"type":"photo"}2026-08-12 11:59:12
item_created{"cover_hash":"494d0777b8cc4ca2a9f48516f1fe1c39","error":null,"hash":"56dcbdf392c3eac2f512813b8501a36a","phase":"item_import","target_ids":{"asset_id":"117493"},"type":"photo"}2026-08-12 11:59:09
item_created{"cover_hash":"2bff9a0c140943898396608372b99465","error":null,"hash":"627bedae0f200a48a8c52b08d93e09ef","phase":"item_import","target_ids":{"asset_id":"117492"},"type":"photo"}2026-08-12 11:59:06
item_created{"cover_hash":"dbbc6701d89644efb8abc30652d2aaef","error":null,"hash":"50447e4442226c451880f6b2433be980","phase":"item_import","target_ids":{"asset_id":"117491"},"type":"photo"}2026-08-12 11:59:02
item_created{"cover_hash":"c1a040c9c3ab49aa8a93e75b731a9436","error":null,"hash":"70bee9c784e4bed85a3d6e7394e67768","phase":"item_import","target_ids":{"asset_id":"117490"},"type":"photo"}2026-08-12 11:58:57
item_created{"cover_hash":"397102f31a1c4760be2f8e913b74201a","error":null,"hash":"682ac53f8be857ea414409ce19181f5e","phase":"item_import","target_ids":{"asset_id":"117489"},"type":"photo"}2026-08-12 11:58:54
item_created{"cover_hash":"9408455e34e747649fb5acc58b71428b","error":null,"hash":"625b8696e61951e83a663eac4fa89a0c","phase":"item_import","target_ids":{"asset_id":"117488"},"type":"photo"}2026-08-12 11:58:50
item_created{"cover_hash":"262de8bc1cb848d78602374d0b7c0f9f","error":null,"hash":"7635103cf74b56964bdfea9c78d12289","phase":"item_import","target_ids":{"asset_id":"117487"},"type":"photo"}2026-08-12 11:58:46
item_created{"cover_hash":"06fc898aa1164b1686c258e5f3c2bf55","error":null,"hash":"ee18ef7fbcb68b3670ff7db7bf5b181b","phase":"item_import","target_ids":{"asset_id":"117486"},"type":"photo"}2026-08-12 11:58:43
item_created{"cover_hash":"2bc5cdae80964b6989bbe692d4477f6b","error":null,"hash":"9d2076eb81b1148972b3576c9df25da8","phase":"item_import","target_ids":{"asset_id":"117485"},"type":"photo"}2026-08-12 11:58:39
item_created{"cover_hash":"a2433407346640fa9134a62c450f83bd","error":null,"hash":"74d3898f6a627b21a1dcc0f9506d9c11","phase":"item_import","target_ids":{"asset_id":"117484"},"type":"photo"}2026-08-12 11:58:36
item_created{"cover_hash":"304bcdf617984a79abe2c50c05d2f107","error":null,"hash":"71966cdaa2be888cd470fffa77390d28","phase":"item_import","target_ids":{"asset_id":"117483"},"type":"photo"}2026-08-12 11:58:32
item_created{"cover_hash":"4cfb87930f8442429a0c9a73394e8c51","error":null,"hash":"a24bdbecb779271486e23c709e8d7b25","phase":"item_import","target_ids":{"asset_id":"117482"},"type":"photo"}2026-08-12 11:58:29
item_created{"cover_hash":"ad55d911054a49c696eb432011838464","error":null,"hash":"e12a1dc730cde8d5014f9885678f2377","phase":"item_import","target_ids":{"asset_id":"117481"},"type":"photo"}2026-08-12 11:58:25
item_created{"cover_hash":"0cda4c80272a415683a3114e57e0a48e","error":null,"hash":"ee93d81a90fabd3d0266ea2c17683afc","phase":"item_import","target_ids":{"asset_id":"117480"},"type":"photo"}2026-08-12 11:58:22
item_created{"cover_hash":"ae60621a052c4dee96b9e065714473d8","error":null,"hash":"70b01831f07e2b0447b172c08afc3164","phase":"item_import","target_ids":{"asset_id":"117479"},"type":"photo"}2026-08-12 11:58:18
item_created{"cover_hash":"ba52214ad2cb4cedb05d0e3b878c94c8","error":null,"hash":"6e028f734ec14e4f2c3049f41313e1e7","phase":"item_import","target_ids":{"asset_id":"117478"},"type":"photo"}2026-08-12 11:58:14
item_created{"cover_hash":"075a013f7b8e477ebbb895f2b6f76486","error":null,"hash":"6d43f2897c04db32b8ceb2188ae7ce9a","phase":"item_import","target_ids":{"asset_id":"117477"},"type":"photo"}2026-08-12 11:58:11
item_created{"cover_hash":"e4de39f757fb4aeea120dc48e6da309b","error":null,"hash":"920ffa76cc8d1291775e8e836cc2a502","phase":"item_import","target_ids":{"asset_id":"117476"},"type":"photo"}2026-08-12 11:58:07
item_created{"cover_hash":"525951e5d34f49e4b8454b88ee73d378","error":null,"hash":"e4a30306af421a62e1977554001e3f02","phase":"item_import","target_ids":{"asset_id":"117475"},"type":"photo"}2026-08-12 11:58:04
item_created{"cover_hash":"e3e5a72b5fe64720ace8d30f2d007be2","error":null,"hash":"73a4a1abc76ae16c8eed55001e500644","phase":"item_import","target_ids":{"asset_id":"117474"},"type":"photo"}2026-08-12 11:58:00
item_created{"cover_hash":"1776d0ae4b4a4f229428385ad2d29b12","error":null,"hash":"c39a8060894a0f16d079dccd8c846628","phase":"item_import","target_ids":{"asset_id":"117473"},"type":"photo"}2026-08-12 11:57:57
item_created{"cover_hash":"94db96c4688244c4b65312634b4febb3","error":null,"hash":"a85d3dcb4a4e4b1bcd8ee7586f158230","phase":"item_import","target_ids":{"asset_id":"117472"},"type":"photo"}2026-08-12 11:57:53
item_created{"cover_hash":"983424b3602843f2854486c3cfab4cc7","error":null,"hash":"2e6fd74b765ca9dd85b534399288a888","phase":"item_import","target_ids":{"asset_id":"117471"},"type":"photo"}2026-08-12 11:57:49
item_created{"cover_hash":"c8a31ac46d944d60b163c0e4ccbdc72b","error":null,"hash":"a9e3c5a8841585726eaebc423a958928","phase":"item_import","target_ids":{"asset_id":"117470"},"type":"photo"}2026-08-12 11:57:45
item_created{"cover_hash":"b0aa2dbb9bf34faf8a3b906d868f1341","error":null,"hash":"e22899e8db515bb1238c04203c3630e8","phase":"item_import","target_ids":{"asset_id":"117469"},"type":"photo"}2026-08-12 11:57:41
item_created{"cover_hash":"07fcb690d50845adad8bf2a7a8a21345","error":null,"hash":"a107f0482d11dde2a773238f98c878f7","phase":"item_import","target_ids":{"asset_id":"117468"},"type":"photo"}2026-08-12 11:57:37
item_created{"cover_hash":"1c403e13acb64df2a1da65419a9428ab","error":null,"hash":"5494d56ebb68b3910a2e176fd67ad442","phase":"item_import","target_ids":{"asset_id":"117467"},"type":"photo"}2026-08-12 11:57:34
item_created{"cover_hash":"6a8add173bc44310bcdbf68dc8487ab4","error":null,"hash":"b1d23a35a980411792ce78ebbd36e2e6","phase":"item_import","target_ids":{"asset_id":"117466"},"type":"photo"}2026-08-12 11:57:30
item_created{"cover_hash":"6c0a286145d44b80bec64556a08cdd29","error":null,"hash":"4391c11fe5ad6a9ab39467e18199e7b3","phase":"item_import","target_ids":{"asset_id":"117465"},"type":"photo"}2026-08-12 11:57:27
item_created{"cover_hash":"86f715be83624d31bf63af0461fc1d7d","error":null,"hash":"975683c2ecab2d7675a6b99c8b000b32","phase":"item_import","target_ids":{"asset_id":"117464"},"type":"photo"}2026-08-12 11:57:23
item_created{"cover_hash":"87c690499eed4ef381ec586b3fb31482","error":null,"hash":"d52a12d3915d3b586eab687b490f1737","phase":"item_import","target_ids":{"asset_id":"117463"},"type":"photo"}2026-08-12 11:57:20
item_created{"cover_hash":"1f4c7f7d226640028aa1ecd27d5d3dd6","error":null,"hash":"5c02678b624638fd254f4194b24d8c22","phase":"item_import","target_ids":{"asset_id":"117462"},"type":"photo"}2026-08-12 11:57:16
item_created{"cover_hash":"6da3d771242440e7be01656d6ebbe353","error":null,"hash":"b16e38ab48e20beb105bbac4858ba24f","phase":"item_import","target_ids":{"asset_id":"117461"},"type":"photo"}2026-08-12 11:57:13
item_created{"cover_hash":"0dad3e3726ec4b1bb8889e4b621e0e7d","error":null,"hash":"0dc3e50e2aa1d4f90ad7e68ea1f7af94","phase":"item_import","target_ids":{"asset_id":"117460"},"type":"photo"}2026-08-12 11:57:09
item_created{"cover_hash":"06f56ff0649d46bf88b26145d5dd5539","error":null,"hash":"898ae5b0ee93f55c349482a0908592df","phase":"item_import","target_ids":{"asset_id":"117459"},"type":"photo"}2026-08-12 11:57:05
item_created{"cover_hash":"cbe33a4fd4c0423399867cddbddd7127","error":null,"hash":"7973e27f4c2545b8434bc4fccd429430","phase":"item_import","target_ids":{"asset_id":"117458"},"type":"photo"}2026-08-12 11:57:02
item_created{"cover_hash":"b05f4c5e39fd4c5e9cbb714554c8ac1f","error":null,"hash":"fce428ed816c750660ad324e7a8bd1c4","phase":"item_import","target_ids":{"asset_id":"117457"},"type":"photo"}2026-08-12 11:56:58
item_created{"cover_hash":"f1181f40a8d745b895fc46400f724a5b","error":null,"hash":"1c5db2e126aee3c3697361e38add34d8","phase":"item_import","target_ids":{"asset_id":"117456"},"type":"photo"}2026-08-12 11:56:55
item_created{"cover_hash":"62d3b9cf42b347fcaf6a5712bf7c559a","error":null,"hash":"3d4174a7d6ffb999ff1d45f2f236ac3c","phase":"item_import","target_ids":{"asset_id":"117455"},"type":"photo"}2026-08-12 11:56:51
item_created{"cover_hash":"763446d881c5429b80ae636267e5a37d","error":null,"hash":"03a13de3258cb4add4daae266ac77ead","phase":"item_import","target_ids":{"asset_id":"117454"},"type":"photo"}2026-08-12 11:56:48
item_created{"cover_hash":"c6da8b22162a43a98809de5bde5ab318","error":null,"hash":"8326364c350aedb5b4bb1337b34c5ab9","phase":"item_import","target_ids":{"asset_id":"117453"},"type":"photo"}2026-08-12 11:56:44
item_created{"cover_hash":"dd9fd35a6f304f4dad300541954851c3","error":null,"hash":"7078876d2d4b22f445be8fb9a4590c67","phase":"item_import","target_ids":{"asset_id":"117452"},"type":"photo"}2026-08-12 11:56:40
item_created{"cover_hash":"716f1fb584b84e0eac87eb5f9178bddd","error":null,"hash":"7ac274b513b0e457c6d89ab1454b9386","phase":"item_import","target_ids":{"asset_id":"117451"},"type":"photo"}2026-08-12 11:56:37