Media Manager
No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Back to queueRuns

Batch 67e0bdf0-b755-4eed-80e5-857051d12e9d

Status: imported; action: import

Type

asset

Action

import

Status

imported

Items

50

Bytes

3.27 GB

Source host

orbit-2

Manifest sha256

649bf6a8f3b06750...

Started

2026-06-29 09:31:09

Completed

2026-06-29 09:38:06

Manifest summary

  • created_at: 2026-06-29T09:30:33Z
  • completed_at: 2026-06-29T09:30:33Z
  • verification.passed: true
  • verification.total_files: 5246
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
31doneimport-feed2026-06-29 09:31:152026-06-29 09:36:09vultr/pid:349192850/502026-06-29 09:38:06-

Items 50

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
videobbdfeb29d2eb9f04...faa5af00c9d74fa2...createdasset:5063-2026-06-29 09:36:17
reel1eddf10bececb22f...dc2573eebb35467b...createdasset:5064-2026-06-29 09:36:20
reel0fb195597bc5a733...-createdasset:5065-2026-06-29 09:36:22
reelc21cdd5b6601d6e1...e46e1c6d8a7d4870...createdasset:5066-2026-06-29 09:36:25
reel7583a77180d50cf5...-createdasset:5067-2026-06-29 09:36:27
reelc902c5b65aa3db92...b5cd9deea57e47e5...createdasset:5068-2026-06-29 09:36:29
reel988001e80c8f1f49...-createdasset:5069-2026-06-29 09:36:31
reele71ab36193bd024b...183f3df55cb54532...createdasset:5070-2026-06-29 09:36:34
reel33ac4a45748b643e...-createdasset:5071-2026-06-29 09:36:36
reel943846d415ff49ba...47e1be211914468c...createdasset:5072-2026-06-29 09:36:38
reelbdab198cedb66718...-createdasset:5073-2026-06-29 09:36:40
reelbba639592a39f4a2...299bfcfad8e54145...createdasset:5074-2026-06-29 09:36:43
reel349a490194176ff0...-createdasset:5075-2026-06-29 09:36:45
reelad2438505dd83255...22f7abff73904e2a...createdasset:5076-2026-06-29 09:36:47
reel03851d2f1228b0f3...-createdasset:5077-2026-06-29 09:36:49
reel7b1c9dcd5d2a1cf7...a41c6e9086564cde...createdasset:5078-2026-06-29 09:36:52
reelee3a0579a3ba4555...-createdasset:5079-2026-06-29 09:36:54
reel3cf546e81cd78161...4958e53dbc0a4028...createdasset:5080-2026-06-29 09:36:56
reel61ac7f062cced831...-createdasset:5081-2026-06-29 09:36:58
reel0dec4dbecb29c910...878b895deda34471...createdasset:5082-2026-06-29 09:37:01
reel0cb5f71dafc70eb3...-createdasset:5083-2026-06-29 09:37:02
reelb34645f63c0d6746...e0e941be4b134c77...createdasset:5084-2026-06-29 09:37:05
reel52283652f598fd2b...-createdasset:5085-2026-06-29 09:37:06
reelec3f2b39f5717509...d263391fffff4756...createdasset:5086-2026-06-29 09:37:09
reel03588e7c18fa668b...86f4c681c22a4d8c...createdasset:5087-2026-06-29 09:37:12
reel711ab1f7bfca26a1...c8f64403b48040a7...createdasset:5088-2026-06-29 09:37:14
reelc28edb2b5276e621...fab347fb15154104...createdasset:5089-2026-06-29 09:37:17
reel5fd8547edec0b7a5...817842dea9884374...createdasset:5090-2026-06-29 09:37:19
reela8b85cc6cd6d5e10...ceb17e826d5d4ab5...createdasset:5091-2026-06-29 09:37:23
reeld713d6083afd21ce...808c743d111e4601...createdasset:5092-2026-06-29 09:37:25
photo58761db2336717a3...d2dd68bd1aa44eae...createdasset:5093-2026-06-29 09:37:28
photod5e043f686e955d3...3ddf6aec25bd4e61...createdasset:5094-2026-06-29 09:37:30
photo26996dc371dd6dc0...ed484849e22649b3...createdasset:5095-2026-06-29 09:37:31
photo847c8c8619c90f83...e30cd6b03d2c4804...createdasset:5096-2026-06-29 09:37:34
photo2c0f12dca3873157...5c828b4d461949cf...createdasset:5097-2026-06-29 09:37:36
photo4e6bf76ba8028ed0...3273d353bb2e4873...createdasset:5098-2026-06-29 09:37:38
photo163f114f102b31e3...bc834be78ac24919...createdasset:5099-2026-06-29 09:37:40
photob7d9d4fae136e261...030ff7cca1434f25...createdasset:5100-2026-06-29 09:37:42
photode1b01e3910f2749...ff529bdf4044466d...createdasset:5101-2026-06-29 09:37:44
photob2840519a9cb0b2b...dce0e337c0ea4f2d...createdasset:5102-2026-06-29 09:37:46
photo72c49294504b839c...72d38a95cf5e47db...createdasset:5103-2026-06-29 09:37:48
photo49afbe702e88ecb7...39a585f6ef8348cc...createdasset:5104-2026-06-29 09:37:50
photodfef138540e8df9e...c8e0e88a86e4452b...createdasset:5105-2026-06-29 09:37:52
photof171b02674fe3062...35f80fa9edd148b2...createdasset:5106-2026-06-29 09:37:54
photo2b4d1f2fa512cb7c...7bd0c4716f4c41a7...createdasset:5107-2026-06-29 09:37:56
photo6d708383494985ac...e16c832f111f4f8f...createdasset:5108-2026-06-29 09:37:58
photo1fa36a87bdf6d266...6e4ed69631d94f65...createdasset:5109-2026-06-29 09:38:00
photo86c23fd99af80e3f...96f9e50aab8c4384...createdasset:5110-2026-06-29 09:38:02
photoc6adc7c57e20aa05...2c2d61dfe90946d5...createdasset:5111-2026-06-29 09:38:04
photo6d0c660e5ccbc53c...f40e9a10c4074fcd...createdasset:5112-2026-06-29 09:38:06

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
job_finished{"error":null,"job_id":"31","phase":"job_finish","summary":{"items_seen":50,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":50,"deferred":0,"failed":0...2026-06-29 09:38:06
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":50,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-06-29 09:38:06
item_updated{"container":1,"expected_member_count":20,"gallery_hash":"b7c1e14c9a707616aa2a50d78ad77db1","gallery_id":"25","linked_member_count":20,"missing_member_count":0,"phase":"container_link","title":"Contes...2026-06-29 09:38:06
item_created{"cover_hash":"f40e9a10c4074fcd834561f114a56566","error":null,"hash":"6d0c660e5ccbc53c96902ede657227cc","phase":"item_import","target_ids":{"asset_id":"5112"},"type":"photo"}2026-06-29 09:38:06
item_created{"cover_hash":"2c2d61dfe90946d59c0a586ca9069c30","error":null,"hash":"c6adc7c57e20aa05610a0f67779adacc","phase":"item_import","target_ids":{"asset_id":"5111"},"type":"photo"}2026-06-29 09:38:04
item_created{"cover_hash":"96f9e50aab8c4384967b1cf2416eb39a","error":null,"hash":"86c23fd99af80e3f3d6f2243f5f400cc","phase":"item_import","target_ids":{"asset_id":"5110"},"type":"photo"}2026-06-29 09:38:02
item_created{"cover_hash":"6e4ed69631d94f658329c2080e595fd5","error":null,"hash":"1fa36a87bdf6d2666296bce51e3175d4","phase":"item_import","target_ids":{"asset_id":"5109"},"type":"photo"}2026-06-29 09:38:00
item_created{"cover_hash":"e16c832f111f4f8f86f4e891fb310d3a","error":null,"hash":"6d708383494985ac2946a87749eb9545","phase":"item_import","target_ids":{"asset_id":"5108"},"type":"photo"}2026-06-29 09:37:58
item_created{"cover_hash":"7bd0c4716f4c41a7872bdfda344b4f50","error":null,"hash":"2b4d1f2fa512cb7cbcb52f517b154a7a","phase":"item_import","target_ids":{"asset_id":"5107"},"type":"photo"}2026-06-29 09:37:56
item_created{"cover_hash":"35f80fa9edd148b2a184dfd1265ed19a","error":null,"hash":"f171b02674fe3062802a94a64554193a","phase":"item_import","target_ids":{"asset_id":"5106"},"type":"photo"}2026-06-29 09:37:54
item_created{"cover_hash":"c8e0e88a86e4452b953261f11b8b98fd","error":null,"hash":"dfef138540e8df9e99589c397adc70ee","phase":"item_import","target_ids":{"asset_id":"5105"},"type":"photo"}2026-06-29 09:37:52
item_created{"cover_hash":"39a585f6ef8348ccb676217132bdfefd","error":null,"hash":"49afbe702e88ecb78f7364223763f330","phase":"item_import","target_ids":{"asset_id":"5104"},"type":"photo"}2026-06-29 09:37:50
item_created{"cover_hash":"72d38a95cf5e47db99f5c10fb26319c0","error":null,"hash":"72c49294504b839cc6099b97a7510b78","phase":"item_import","target_ids":{"asset_id":"5103"},"type":"photo"}2026-06-29 09:37:48
item_created{"cover_hash":"dce0e337c0ea4f2d86b21286b5798388","error":null,"hash":"b2840519a9cb0b2bf9f776e151dc4cac","phase":"item_import","target_ids":{"asset_id":"5102"},"type":"photo"}2026-06-29 09:37:46
item_created{"cover_hash":"ff529bdf4044466d85db981ee653905a","error":null,"hash":"de1b01e3910f2749a56cb349715bdaf4","phase":"item_import","target_ids":{"asset_id":"5101"},"type":"photo"}2026-06-29 09:37:44
item_created{"cover_hash":"030ff7cca1434f259a749bb512e64303","error":null,"hash":"b7d9d4fae136e261ec76cfe4b9aba26c","phase":"item_import","target_ids":{"asset_id":"5100"},"type":"photo"}2026-06-29 09:37:42
item_created{"cover_hash":"bc834be78ac24919bd55154e531344f9","error":null,"hash":"163f114f102b31e33f3c29b1590365e4","phase":"item_import","target_ids":{"asset_id":"5099"},"type":"photo"}2026-06-29 09:37:40
item_created{"cover_hash":"3273d353bb2e487385d37a73cdbb63f4","error":null,"hash":"4e6bf76ba8028ed0e402346393d2da46","phase":"item_import","target_ids":{"asset_id":"5098"},"type":"photo"}2026-06-29 09:37:38
item_created{"cover_hash":"5c828b4d461949cf8218e286eb2de8c8","error":null,"hash":"2c0f12dca387315758de45a7f1643d77","phase":"item_import","target_ids":{"asset_id":"5097"},"type":"photo"}2026-06-29 09:37:36
item_created{"cover_hash":"e30cd6b03d2c4804a42d870ea4122583","error":null,"hash":"847c8c8619c90f8316c33a57e382215f","phase":"item_import","target_ids":{"asset_id":"5096"},"type":"photo"}2026-06-29 09:37:34
item_created{"cover_hash":"ed484849e22649b39188d6a5b7f6278d","error":null,"hash":"26996dc371dd6dc04ef647bbec66dd5e","phase":"item_import","target_ids":{"asset_id":"5095"},"type":"photo"}2026-06-29 09:37:31
item_created{"cover_hash":"3ddf6aec25bd4e61b61d00ca52b0e5f1","error":null,"hash":"d5e043f686e955d3bdc8f4569b5608d4","phase":"item_import","target_ids":{"asset_id":"5094"},"type":"photo"}2026-06-29 09:37:30
item_created{"cover_hash":"d2dd68bd1aa44eae9e48e5e55e8f9585","error":null,"hash":"58761db2336717a3d8010e7445dda693","phase":"item_import","target_ids":{"asset_id":"5093"},"type":"photo"}2026-06-29 09:37:28
item_created{"cover_hash":"808c743d111e4601a8ca284484eb671a","error":null,"hash":"d713d6083afd21ceff80223f916b7517","phase":"item_import","target_ids":{"asset_id":"5092"},"type":"reel"}2026-06-29 09:37:25
item_created{"cover_hash":"ceb17e826d5d4ab5b1fcb98c931499c1","error":null,"hash":"a8b85cc6cd6d5e10118149f8e7774606","phase":"item_import","target_ids":{"asset_id":"5091"},"type":"reel"}2026-06-29 09:37:23
item_created{"cover_hash":"817842dea9884374b270ec53c1c3fb91","error":null,"hash":"5fd8547edec0b7a5cbd08201a9bc7e0c","phase":"item_import","target_ids":{"asset_id":"5090"},"type":"reel"}2026-06-29 09:37:19
item_created{"cover_hash":"fab347fb151541048e52abefa229edca","error":null,"hash":"c28edb2b5276e621f36f631bda7d3110","phase":"item_import","target_ids":{"asset_id":"5089"},"type":"reel"}2026-06-29 09:37:17
item_created{"cover_hash":"c8f64403b48040a7a59036085b60f667","error":null,"hash":"711ab1f7bfca26a1644b84c827a2e320","phase":"item_import","target_ids":{"asset_id":"5088"},"type":"reel"}2026-06-29 09:37:14
item_created{"cover_hash":"86f4c681c22a4d8c8516daf0d8695dc8","error":null,"hash":"03588e7c18fa668b0b643f8f35e3e9c1","phase":"item_import","target_ids":{"asset_id":"5087"},"type":"reel"}2026-06-29 09:37:12
item_created{"cover_hash":"d263391fffff47569865f85b9668160a","error":null,"hash":"ec3f2b39f5717509201f888f12aa869c","phase":"item_import","target_ids":{"asset_id":"5086"},"type":"reel"}2026-06-29 09:37:09
item_created{"cover_hash":null,"error":null,"hash":"52283652f598fd2be80d0f4607fe9443","phase":"item_import","target_ids":{"asset_id":"5085"},"type":"reel"}2026-06-29 09:37:06
item_created{"cover_hash":"e0e941be4b134c77b0789aef2f10881e","error":null,"hash":"b34645f63c0d6746ea81ee7079f97d03","phase":"item_import","target_ids":{"asset_id":"5084"},"type":"reel"}2026-06-29 09:37:05
item_created{"cover_hash":null,"error":null,"hash":"0cb5f71dafc70eb397fe35317b5eac30","phase":"item_import","target_ids":{"asset_id":"5083"},"type":"reel"}2026-06-29 09:37:02
item_created{"cover_hash":"878b895deda344719f7140fc0aee9490","error":null,"hash":"0dec4dbecb29c910436adcca67b987c0","phase":"item_import","target_ids":{"asset_id":"5082"},"type":"reel"}2026-06-29 09:37:01
item_created{"cover_hash":null,"error":null,"hash":"61ac7f062cced83158958c71fd5317c5","phase":"item_import","target_ids":{"asset_id":"5081"},"type":"reel"}2026-06-29 09:36:58
item_created{"cover_hash":"4958e53dbc0a4028915c2cad7fe07e9a","error":null,"hash":"3cf546e81cd78161d4a8a4c983642028","phase":"item_import","target_ids":{"asset_id":"5080"},"type":"reel"}2026-06-29 09:36:56
item_created{"cover_hash":null,"error":null,"hash":"ee3a0579a3ba45556a0b9c9bf729935c","phase":"item_import","target_ids":{"asset_id":"5079"},"type":"reel"}2026-06-29 09:36:54
item_created{"cover_hash":"a41c6e9086564cde907d1ba2e4929b89","error":null,"hash":"7b1c9dcd5d2a1cf71cd75edc162a3683","phase":"item_import","target_ids":{"asset_id":"5078"},"type":"reel"}2026-06-29 09:36:52
item_created{"cover_hash":null,"error":null,"hash":"03851d2f1228b0f3b8f0cd01b9f819d6","phase":"item_import","target_ids":{"asset_id":"5077"},"type":"reel"}2026-06-29 09:36:49
item_created{"cover_hash":"22f7abff73904e2ab1f5bff407af9a86","error":null,"hash":"ad2438505dd83255b2365e8b9be4a330","phase":"item_import","target_ids":{"asset_id":"5076"},"type":"reel"}2026-06-29 09:36:47
item_created{"cover_hash":null,"error":null,"hash":"349a490194176ff0148178ce012dafcc","phase":"item_import","target_ids":{"asset_id":"5075"},"type":"reel"}2026-06-29 09:36:45
item_created{"cover_hash":"299bfcfad8e541458b6eacba81700106","error":null,"hash":"bba639592a39f4a257da6a2d3d5a1785","phase":"item_import","target_ids":{"asset_id":"5074"},"type":"reel"}2026-06-29 09:36:43
item_created{"cover_hash":null,"error":null,"hash":"bdab198cedb6671875d183ad83432ef6","phase":"item_import","target_ids":{"asset_id":"5073"},"type":"reel"}2026-06-29 09:36:40
item_created{"cover_hash":"47e1be211914468c8c364d5a5b9b9c48","error":null,"hash":"943846d415ff49ba752c908de5ed77b5","phase":"item_import","target_ids":{"asset_id":"5072"},"type":"reel"}2026-06-29 09:36:38
item_created{"cover_hash":null,"error":null,"hash":"33ac4a45748b643e6219b45529aae0ac","phase":"item_import","target_ids":{"asset_id":"5071"},"type":"reel"}2026-06-29 09:36:36
item_created{"cover_hash":"183f3df55cb54532af09e4894ad30a65","error":null,"hash":"e71ab36193bd024b96d878c3fb935552","phase":"item_import","target_ids":{"asset_id":"5070"},"type":"reel"}2026-06-29 09:36:34
item_created{"cover_hash":null,"error":null,"hash":"988001e80c8f1f49b733efb16f560f07","phase":"item_import","target_ids":{"asset_id":"5069"},"type":"reel"}2026-06-29 09:36:31
item_created{"cover_hash":"b5cd9deea57e47e599c84d537004846c","error":null,"hash":"c902c5b65aa3db92093291b2870fa9e1","phase":"item_import","target_ids":{"asset_id":"5068"},"type":"reel"}2026-06-29 09:36:29
item_created{"cover_hash":null,"error":null,"hash":"7583a77180d50cf5ce1cac2c253ed91b","phase":"item_import","target_ids":{"asset_id":"5067"},"type":"reel"}2026-06-29 09:36:27
item_created{"cover_hash":"e46e1c6d8a7d487095884baa50de9c5a","error":null,"hash":"c21cdd5b6601d6e1fc9e119086f0f7d5","phase":"item_import","target_ids":{"asset_id":"5066"},"type":"reel"}2026-06-29 09:36:25