Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 4118da38-a52c-4993-8741-3cd5694db707

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

53

Bytes

3.47 GB

Source host

orbit-2

Manifest sha256

e424b4000a5f5741...

Started

2026-06-29 09:32:17

Completed

2026-06-29 09:40:11

Manifest summary

  • created_at: 2026-06-29T09:31:15Z
  • completed_at: 2026-06-29T09:31:15Z
  • verification.passed: true
  • verification.total_files: 5676
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
32doneimport-feed2026-06-29 09:32:232026-06-29 09:38:06vultr/pid:349192853/532026-06-29 09:40:11-

Items 53

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
video0f9dddcc9cf8c148...bb66411648104e8e...createdasset:5113-2026-06-29 09:38:15
reel07d2df19a16ee469...6a61f74818af41cd...createdasset:5114-2026-06-29 09:38:17
reel26bdeaf51049d861...-createdasset:5115-2026-06-29 09:38:19
reel398ee786c97a54fc...de5d318eaa7b4786...createdasset:5116-2026-06-29 09:38:22
reel5713d672daa7007f...-createdasset:5117-2026-06-29 09:38:25
reelfa55bb5703886aa3...afe6dda5cd824a15...createdasset:5118-2026-06-29 09:38:28
reelb00af552f75f130e...-createdasset:5119-2026-06-29 09:38:29
reel6925ffd17cb4bf29...83af9969c17d4275...createdasset:5120-2026-06-29 09:38:32
reel2ab257bec4889672...-createdasset:5121-2026-06-29 09:38:34
reel92a0c24ffe59208f...11f9c5229fb74823...createdasset:5122-2026-06-29 09:38:37
reeld3f765c307af31cc...-createdasset:5123-2026-06-29 09:38:39
reeldaa42de77c9a4c39...8e17362b5c854201...createdasset:5124-2026-06-29 09:38:41
reele667a6d81093ee23...-createdasset:5125-2026-06-29 09:38:43
reel587f8274d7b04087...668a736abb3a447d...createdasset:5126-2026-06-29 09:38:46
reeld7421b3fbf6a485f...-createdasset:5127-2026-06-29 09:38:47
reel9166129db753978e...643b1be6d8634814...createdasset:5128-2026-06-29 09:38:50
reeldf1bd65026971929...-createdasset:5129-2026-06-29 09:38:51
reel978ae7f90a967443...75b98ce9cac646e8...createdasset:5130-2026-06-29 09:38:54
reel287270b7ad623909...-createdasset:5131-2026-06-29 09:38:56
reel15637bcfed1ef433...687808dfdcf44ee5...createdasset:5132-2026-06-29 09:38:59
reela415c64eb72885a0...-createdasset:5133-2026-06-29 09:39:01
reelc1b9e026f44d904c...c88c9323e52a4011...createdasset:5134-2026-06-29 09:39:03
reelb2c7ecf9227a2963...-createdasset:5135-2026-06-29 09:39:05
reel1dea5062014f146f...72048c85d70b4990...createdasset:5136-2026-06-29 09:39:07
reelec33a2c14b7d2f0e...20229873c6484058...createdasset:5137-2026-06-29 09:39:10
reelf64bbaa969a8bd05...66fcff3172d446ce...createdasset:5138-2026-06-29 09:39:12
reel502228862d2f1ad8...1f9a459f8cdf49da...createdasset:5139-2026-06-29 09:39:15
reel378577125d75cefc...fbe850e9f22047f0...createdasset:5140-2026-06-29 09:39:17
reele52a7447d6dc21e0...588253edd18f4c25...createdasset:5141-2026-06-29 09:39:20
reel8791e70cd1a06fce...bd5bbb8543404ed4...createdasset:5142-2026-06-29 09:39:24
reelf3c5357069fdc598...63e395c44b664b83...createdasset:5143-2026-06-29 09:39:27
photo4d02cd159c3d7856...ca4aa34afd2a4638...createdasset:5144-2026-06-29 09:39:29
photo37e0494586f6682b...99b8c94056a0490c...createdasset:5145-2026-06-29 09:39:31
photob9f8c23e2f69d9c4...8e231f81c3e54b4a...createdasset:5146-2026-06-29 09:39:33
photof9d0e5bbea24131e...13e63197de2e4ac3...createdasset:5147-2026-06-29 09:39:35
photoe40bde0cbdc38796...90060a902e6446da...createdasset:5148-2026-06-29 09:39:37
photof401ffde7ba5af2d...4e5c0dcb224d4874...createdasset:5149-2026-06-29 09:39:39
photo4e5f7bd3953bdabe...bcbe93709106456f...createdasset:5150-2026-06-29 09:39:41
photodd5c2fbb19dff4fb...1a48e7b7307e4f86...createdasset:5151-2026-06-29 09:39:43
photoc2d8a7514ab8d7d2...f8a81c0a79d64023...createdasset:5152-2026-06-29 09:39:45
photo7905f93ecb887464...cbf785661928422c...createdasset:5153-2026-06-29 09:39:47
photo54c811a84dc78f35...6c3c4081cd5c4dbf...createdasset:5154-2026-06-29 09:39:49
photo04756009a382637a...f07e209a395a42e2...createdasset:5155-2026-06-29 09:39:51
photo1e72c5b1bdd94226...4d63bc3406214073...createdasset:5156-2026-06-29 09:39:53
photoefa2b3c66321156f...df0b55146ff24b02...createdasset:5157-2026-06-29 09:39:55
photobce216addd57e8bf...704399ff1e5d4b13...createdasset:5158-2026-06-29 09:39:57
photocd9ad7e51dfab5a4...fe14207b375c4002...createdasset:5159-2026-06-29 09:39:59
photo1261523c11873c93...8e035cf9133e4317...createdasset:5160-2026-06-29 09:40:01
photo04fb431b7cf1f360...903bd63271ae4a59...createdasset:5161-2026-06-29 09:40:03
photo1a73e61b2ff52d65...0323863803874577...createdasset:5162-2026-06-29 09:40:05
photo9e6eff2f73c6703b...a696a19dacdd45a8...createdasset:5163-2026-06-29 09:40:07
photoac12befce0a8235b...46d3a243b16d4dad...createdasset:5164-2026-06-29 09:40:09
photo3d6319ed2e2ee39b...739bfc09b6144c17...createdasset:5165-2026-06-29 09:40:11

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"32","phase":"job_finish","summary":{"items_seen":53,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":53,"deferred":0,"failed":0...2026-06-29 09:40:11
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":53,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-29 09:40:11
item_updated{"container":1,"expected_member_count":22,"gallery_hash":"33e4f5cb7219f555d04efe407e31a714","gallery_id":"26","linked_member_count":22,"missing_member_count":0,"phase":"container_link","title":"beache...2026-06-29 09:40:11
item_created{"cover_hash":"739bfc09b6144c17b43e47aca07894b3","error":null,"hash":"3d6319ed2e2ee39b869d4e54daed5462","phase":"item_import","target_ids":{"asset_id":"5165"},"type":"photo"}2026-06-29 09:40:11
item_created{"cover_hash":"46d3a243b16d4dad82d7fab228b58a54","error":null,"hash":"ac12befce0a8235b5a7c79ff6cdf2646","phase":"item_import","target_ids":{"asset_id":"5164"},"type":"photo"}2026-06-29 09:40:09
item_created{"cover_hash":"a696a19dacdd45a8baca63c49115672e","error":null,"hash":"9e6eff2f73c6703b1db219d641215bcb","phase":"item_import","target_ids":{"asset_id":"5163"},"type":"photo"}2026-06-29 09:40:07
item_created{"cover_hash":"0323863803874577afe77c46490f8cf9","error":null,"hash":"1a73e61b2ff52d657629506510c6d204","phase":"item_import","target_ids":{"asset_id":"5162"},"type":"photo"}2026-06-29 09:40:05
item_created{"cover_hash":"903bd63271ae4a5997fc165643a8f272","error":null,"hash":"04fb431b7cf1f36056c1fa160a3a2b19","phase":"item_import","target_ids":{"asset_id":"5161"},"type":"photo"}2026-06-29 09:40:03
item_created{"cover_hash":"8e035cf9133e43179f7324d608504ae6","error":null,"hash":"1261523c11873c93f3e44a30a77f3ffe","phase":"item_import","target_ids":{"asset_id":"5160"},"type":"photo"}2026-06-29 09:40:01
item_created{"cover_hash":"fe14207b375c4002b98c1efefa916169","error":null,"hash":"cd9ad7e51dfab5a482b0b5c6f6abae52","phase":"item_import","target_ids":{"asset_id":"5159"},"type":"photo"}2026-06-29 09:39:59
item_created{"cover_hash":"704399ff1e5d4b13b1aa549faf40ec5e","error":null,"hash":"bce216addd57e8bf251c99998c78e18a","phase":"item_import","target_ids":{"asset_id":"5158"},"type":"photo"}2026-06-29 09:39:57
item_created{"cover_hash":"df0b55146ff24b028b7ab79465af49fb","error":null,"hash":"efa2b3c66321156fd4c6d6fe778a28ab","phase":"item_import","target_ids":{"asset_id":"5157"},"type":"photo"}2026-06-29 09:39:55
item_created{"cover_hash":"4d63bc34062140739ca1aacd53cde398","error":null,"hash":"1e72c5b1bdd94226574b8dfdc74d0286","phase":"item_import","target_ids":{"asset_id":"5156"},"type":"photo"}2026-06-29 09:39:53
item_created{"cover_hash":"f07e209a395a42e28a913fb2fe554e38","error":null,"hash":"04756009a382637ac04833493a79926d","phase":"item_import","target_ids":{"asset_id":"5155"},"type":"photo"}2026-06-29 09:39:51
item_created{"cover_hash":"6c3c4081cd5c4dbf9b8c28046458c658","error":null,"hash":"54c811a84dc78f35fb4f0c48d5da10ce","phase":"item_import","target_ids":{"asset_id":"5154"},"type":"photo"}2026-06-29 09:39:49
item_created{"cover_hash":"cbf785661928422ca0fa7257bee1866b","error":null,"hash":"7905f93ecb887464c5fb2892eec7f612","phase":"item_import","target_ids":{"asset_id":"5153"},"type":"photo"}2026-06-29 09:39:47
item_created{"cover_hash":"f8a81c0a79d64023ada8766da265eb78","error":null,"hash":"c2d8a7514ab8d7d20fede0fe44ea9c93","phase":"item_import","target_ids":{"asset_id":"5152"},"type":"photo"}2026-06-29 09:39:45
item_created{"cover_hash":"1a48e7b7307e4f86a263642ac1da0f64","error":null,"hash":"dd5c2fbb19dff4fbde6f0ea382127367","phase":"item_import","target_ids":{"asset_id":"5151"},"type":"photo"}2026-06-29 09:39:43
item_created{"cover_hash":"bcbe93709106456fb0a55682e55e9c79","error":null,"hash":"4e5f7bd3953bdabe228e09870ff89f8f","phase":"item_import","target_ids":{"asset_id":"5150"},"type":"photo"}2026-06-29 09:39:41
item_created{"cover_hash":"4e5c0dcb224d4874aa88d53f3711e626","error":null,"hash":"f401ffde7ba5af2d94365280c6239df3","phase":"item_import","target_ids":{"asset_id":"5149"},"type":"photo"}2026-06-29 09:39:39
item_created{"cover_hash":"90060a902e6446da922faf3fda753059","error":null,"hash":"e40bde0cbdc38796792700333c185a48","phase":"item_import","target_ids":{"asset_id":"5148"},"type":"photo"}2026-06-29 09:39:37
item_created{"cover_hash":"13e63197de2e4ac39f4890b0e754d996","error":null,"hash":"f9d0e5bbea24131ec572e34a7fe22692","phase":"item_import","target_ids":{"asset_id":"5147"},"type":"photo"}2026-06-29 09:39:35
item_created{"cover_hash":"8e231f81c3e54b4ab62e0ba4b6fffeb5","error":null,"hash":"b9f8c23e2f69d9c41505b62d654c3711","phase":"item_import","target_ids":{"asset_id":"5146"},"type":"photo"}2026-06-29 09:39:33
item_created{"cover_hash":"99b8c94056a0490c9c0595854210a8cf","error":null,"hash":"37e0494586f6682b17e23d94d979d9f7","phase":"item_import","target_ids":{"asset_id":"5145"},"type":"photo"}2026-06-29 09:39:31
item_created{"cover_hash":"ca4aa34afd2a46388a8d2d7559ac1b2f","error":null,"hash":"4d02cd159c3d785663540c2227427888","phase":"item_import","target_ids":{"asset_id":"5144"},"type":"photo"}2026-06-29 09:39:29
item_created{"cover_hash":"63e395c44b664b83803e0a530bf15743","error":null,"hash":"f3c5357069fdc5984c3c6c0a767207e8","phase":"item_import","target_ids":{"asset_id":"5143"},"type":"reel"}2026-06-29 09:39:27
item_created{"cover_hash":"bd5bbb8543404ed49c2bbfb7914e7a55","error":null,"hash":"8791e70cd1a06fce9283109f5bd541bf","phase":"item_import","target_ids":{"asset_id":"5142"},"type":"reel"}2026-06-29 09:39:24
item_created{"cover_hash":"588253edd18f4c25aa4bba7fb33751dc","error":null,"hash":"e52a7447d6dc21e0f64a01d5ba8f72cb","phase":"item_import","target_ids":{"asset_id":"5141"},"type":"reel"}2026-06-29 09:39:20
item_created{"cover_hash":"fbe850e9f22047f087e2ae3478113a31","error":null,"hash":"378577125d75cefc5beeedf5ce1793b3","phase":"item_import","target_ids":{"asset_id":"5140"},"type":"reel"}2026-06-29 09:39:17
item_created{"cover_hash":"1f9a459f8cdf49da96c3e2123fdc1c59","error":null,"hash":"502228862d2f1ad8f8388bc7c1d0da5d","phase":"item_import","target_ids":{"asset_id":"5139"},"type":"reel"}2026-06-29 09:39:15
item_created{"cover_hash":"66fcff3172d446ce91d03b3022cfa049","error":null,"hash":"f64bbaa969a8bd053cf575917f1b2d00","phase":"item_import","target_ids":{"asset_id":"5138"},"type":"reel"}2026-06-29 09:39:12
item_created{"cover_hash":"20229873c6484058817c654b568e2a3b","error":null,"hash":"ec33a2c14b7d2f0e69e3e867ae895a7e","phase":"item_import","target_ids":{"asset_id":"5137"},"type":"reel"}2026-06-29 09:39:10
item_created{"cover_hash":"72048c85d70b4990b81733e813b06c6a","error":null,"hash":"1dea5062014f146f6013626b874229ef","phase":"item_import","target_ids":{"asset_id":"5136"},"type":"reel"}2026-06-29 09:39:07
item_created{"cover_hash":null,"error":null,"hash":"b2c7ecf9227a2963b902e63f0f404761","phase":"item_import","target_ids":{"asset_id":"5135"},"type":"reel"}2026-06-29 09:39:05
item_created{"cover_hash":"c88c9323e52a40118e7b3980d43b44bb","error":null,"hash":"c1b9e026f44d904c7651d76717191374","phase":"item_import","target_ids":{"asset_id":"5134"},"type":"reel"}2026-06-29 09:39:03
item_created{"cover_hash":null,"error":null,"hash":"a415c64eb72885a088c305943ed2340a","phase":"item_import","target_ids":{"asset_id":"5133"},"type":"reel"}2026-06-29 09:39:01
item_created{"cover_hash":"687808dfdcf44ee580570aea713e324c","error":null,"hash":"15637bcfed1ef43368f778ab2238a6ff","phase":"item_import","target_ids":{"asset_id":"5132"},"type":"reel"}2026-06-29 09:38:59
item_created{"cover_hash":null,"error":null,"hash":"287270b7ad62390976564ad391e71c60","phase":"item_import","target_ids":{"asset_id":"5131"},"type":"reel"}2026-06-29 09:38:56
item_created{"cover_hash":"75b98ce9cac646e8b83ebad076c43e44","error":null,"hash":"978ae7f90a967443e3d5cabe72bf32ae","phase":"item_import","target_ids":{"asset_id":"5130"},"type":"reel"}2026-06-29 09:38:54
item_created{"cover_hash":null,"error":null,"hash":"df1bd6502697192965370d8bf2b863dd","phase":"item_import","target_ids":{"asset_id":"5129"},"type":"reel"}2026-06-29 09:38:51
item_created{"cover_hash":"643b1be6d8634814ac16a049ec2c1fce","error":null,"hash":"9166129db753978efbd243ae0dd8ddb0","phase":"item_import","target_ids":{"asset_id":"5128"},"type":"reel"}2026-06-29 09:38:50
item_created{"cover_hash":null,"error":null,"hash":"d7421b3fbf6a485fc332e869232414db","phase":"item_import","target_ids":{"asset_id":"5127"},"type":"reel"}2026-06-29 09:38:47
item_created{"cover_hash":"668a736abb3a447db21e2309fb7e1b95","error":null,"hash":"587f8274d7b04087893d2eab65a31ab4","phase":"item_import","target_ids":{"asset_id":"5126"},"type":"reel"}2026-06-29 09:38:46
item_created{"cover_hash":null,"error":null,"hash":"e667a6d81093ee23924d03caacc760c3","phase":"item_import","target_ids":{"asset_id":"5125"},"type":"reel"}2026-06-29 09:38:43
item_created{"cover_hash":"8e17362b5c8542018a47100003267599","error":null,"hash":"daa42de77c9a4c390a7f338e196ef845","phase":"item_import","target_ids":{"asset_id":"5124"},"type":"reel"}2026-06-29 09:38:41
item_created{"cover_hash":null,"error":null,"hash":"d3f765c307af31cc9a9782853e9c2053","phase":"item_import","target_ids":{"asset_id":"5123"},"type":"reel"}2026-06-29 09:38:39
item_created{"cover_hash":"11f9c5229fb748238b4d0bbbc56ae6b3","error":null,"hash":"92a0c24ffe59208f67ffb1748f4e9ffc","phase":"item_import","target_ids":{"asset_id":"5122"},"type":"reel"}2026-06-29 09:38:37
item_created{"cover_hash":null,"error":null,"hash":"2ab257bec4889672d8851831f9971032","phase":"item_import","target_ids":{"asset_id":"5121"},"type":"reel"}2026-06-29 09:38:34
item_created{"cover_hash":"83af9969c17d4275a73c15078f380c3b","error":null,"hash":"6925ffd17cb4bf299efa34b027030cb9","phase":"item_import","target_ids":{"asset_id":"5120"},"type":"reel"}2026-06-29 09:38:32
item_created{"cover_hash":null,"error":null,"hash":"b00af552f75f130ebd0975042416c346","phase":"item_import","target_ids":{"asset_id":"5119"},"type":"reel"}2026-06-29 09:38:29