No site selected

Import Media

Queue cloud-sync handover batches and monitor worker runs.

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

  • Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
  • Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
  • Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
  • Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

  • The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
  • VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
  • Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
  • Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
  • Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.
Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

  • source/source.json -> asset_source (cold provenance).
  • source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
  • Bundled cover_hash -> asset_cover linked to the parent asset.
  • Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Batch 18b666e8-d48d-4dc8-8aeb-6002c8444595

Status: imported; action: import

Type

gallery

Action

import

Status

imported

Items

89

Bytes

14.58 GB

Source host

flux-1

Manifest sha256

814632d355162810...

Started

2026-05-22 01:18:20

Completed

2026-05-22 01:19:33

Manifest summary

  • created_at: 2026-05-22T00:03:12Z
  • completed_at: 2026-05-22T01:06:41Z
  • verification.passed: true
  • verification.total_files: 8370
  • prior_sync_id: none

Job history 1

Newest first. The active job (if any) is at the top.

idstaterequested_byqueuedclaimedworkerprogressfinishedlast_error
7doneadmin2026-05-22 01:18:262026-05-22 01:18:31vultr/pid:41088889/892026-05-22 01:19:33-

Items 89

Per-item decisions captured by preflight and updated by import. Re-running pre-flight is safe.

asset_typeasset_hashcover_hashdecisiontarget idslast_errorupdated_at
photo991584e182799730...-createdasset:70-2026-05-22 01:18:32
cover6fbe0523d2184f9f...-createdasset:70-2026-05-22 01:18:33
photocc3ad4a97b35bb2c...-createdasset:71-2026-05-22 01:18:34
cover1e2f594d75e244de...-createdasset:71-2026-05-22 01:18:35
photocfafd10ef63553a6...-createdasset:72-2026-05-22 01:18:36
cover753e7dd882a04140...-createdasset:72-2026-05-22 01:18:36
photo2fcabf6ef72e7849...-createdasset:73-2026-05-22 01:18:37
cover8ea0df5d558043ff...-createdasset:73-2026-05-22 01:18:37
photo486ccfe7695e9152...-createdasset:74-2026-05-22 01:18:38
cover04dde59ce5d342ad...-createdasset:74-2026-05-22 01:18:39
photo5e3d74a808a90cad...-createdasset:75-2026-05-22 01:18:40
cover7d94d024e0304102...-createdasset:75-2026-05-22 01:18:40
photo0b4e2eb855da158d...-createdasset:76-2026-05-22 01:18:41
cover2e962d817c1844e2...-createdasset:76-2026-05-22 01:18:41
photo6ec1534cbf7b0a13...-createdasset:77-2026-05-22 01:18:42
cover3778e195987b4779...-createdasset:77-2026-05-22 01:18:43
photof6919f4990a564cf...-createdasset:78-2026-05-22 01:18:44
coverd7798935e0ed430d...-createdasset:78-2026-05-22 01:18:44
photod8cba7b4b1d25896...-createdasset:79-2026-05-22 01:18:45
cover109a8219705f4283...-createdasset:79-2026-05-22 01:18:46
photoced2f89608361df1...-createdasset:80-2026-05-22 01:18:47
cover5175d3b4489748f5...-createdasset:80-2026-05-22 01:18:47
photo09c12421ab499d38...-createdasset:81-2026-05-22 01:18:48
coverdaa03320d2ff40ff...-createdasset:81-2026-05-22 01:18:49
photo78e20594888ddf4b...-createdasset:82-2026-05-22 01:18:50
coverf6714e55e9f94ea0...-createdasset:82-2026-05-22 01:18:50
photo9682e2d58e69d4f2...-createdasset:83-2026-05-22 01:18:51
coverc1575081eb0149b6...-createdasset:83-2026-05-22 01:18:51
gallery_legacy3a3d25a8db493586...-createdgallery:3-2026-05-22 01:18:51
photoc14a930afdc7e46d...-createdasset:84-2026-05-22 01:18:52
cover43320651066f4c34...-createdasset:84-2026-05-22 01:18:53
photoad0ff8b424069a0b...-createdasset:85-2026-05-22 01:18:54
cover86734eb29423489f...-createdasset:85-2026-05-22 01:18:54
photo48d80d51cf0f0941...-createdasset:86-2026-05-22 01:18:55
covere4959828c051406a...-createdasset:86-2026-05-22 01:18:56
photo3eb77c79b505a502...-createdasset:87-2026-05-22 01:18:57
cover1e2f160e96124869...-createdasset:87-2026-05-22 01:18:57
photo2222e1874fdc5d87...-createdasset:88-2026-05-22 01:18:58
cover225ce3eb1ecf4886...-createdasset:88-2026-05-22 01:18:59
photo27ad6a1bb1d94492...-createdasset:89-2026-05-22 01:19:00
coverb07099f525474677...-createdasset:89-2026-05-22 01:19:00
photo02bd169987a8ca89...-createdasset:90-2026-05-22 01:19:01
cover379a44ab39a942de...-createdasset:90-2026-05-22 01:19:02
photode36c24b43621e8f...-createdasset:91-2026-05-22 01:19:02
coverabe1c86d21ea4662...-createdasset:91-2026-05-22 01:19:03
photo5169370fbf9671e0...-createdasset:92-2026-05-22 01:19:04
cover5b2fd9b16e0545a7...-createdasset:92-2026-05-22 01:19:04
photo71c4e647c600fe33...-createdasset:93-2026-05-22 01:19:05
cover469bc0577e344c74...-createdasset:93-2026-05-22 01:19:06
photod0441f4209d84634...-createdasset:94-2026-05-22 01:19:07
cover988d1f1b9d994c0b...-createdasset:94-2026-05-22 01:19:07
photo9ecd01cdc9d5a61b...-createdasset:95-2026-05-22 01:19:08
coverb893eb519d1b4627...-createdasset:95-2026-05-22 01:19:08
photoa5bba31d75b8935d...-createdasset:96-2026-05-22 01:19:09
cover2eb14c27613e4694...-createdasset:96-2026-05-22 01:19:10
photo5c92cda582b40976...-createdasset:97-2026-05-22 01:19:11
cover4c0ae79ccce64ab8...-createdasset:97-2026-05-22 01:19:11
photoa3e99774194eb0d5...-createdasset:98-2026-05-22 01:19:12
cover82f75605064a42db...-createdasset:98-2026-05-22 01:19:13
gallery_legacy778548951971333d...-createdgallery:4-2026-05-22 01:19:13
photofd975e3923329e45...-createdasset:99-2026-05-22 01:19:14
coveraa0ceae9bc994647...-createdasset:99-2026-05-22 01:19:14
photo03976bc398bc54b2...-createdasset:100-2026-05-22 01:19:15
cover82083ce09e274748...-createdasset:100-2026-05-22 01:19:15
photoa29f3b3eab287518...-createdasset:101-2026-05-22 01:19:16
coverd1f2a8f573f14221...-createdasset:101-2026-05-22 01:19:17
photo7009228a2f0badd1...-createdasset:102-2026-05-22 01:19:18
cover81253116ac2d4d66...-createdasset:102-2026-05-22 01:19:18
photo3be0847676423875...-createdasset:103-2026-05-22 01:19:19
cover497655b336c047fe...-createdasset:103-2026-05-22 01:19:20
photo5909b54d32cbadda...-createdasset:104-2026-05-22 01:19:21
cover110d9ef75bea44b2...-createdasset:104-2026-05-22 01:19:21
photo329f0adfa485b142...-createdasset:105-2026-05-22 01:19:22
covera8e446b7678c4ef0...-createdasset:105-2026-05-22 01:19:22
photo9e95b5cc354ebf4e...-createdasset:106-2026-05-22 01:19:23
cover4083d074810d4113...-createdasset:106-2026-05-22 01:19:24
photobda842a1647dc3b9...-createdasset:107-2026-05-22 01:19:25
coverae61fe52c533456b...-createdasset:107-2026-05-22 01:19:25
photo396dd5945ce5c6c9...-createdasset:108-2026-05-22 01:19:26
cover60fe0641bb974aab...-createdasset:108-2026-05-22 01:19:26
photo954b3be4e774a8c2...-createdasset:109-2026-05-22 01:19:28
cover962900f5b59f42ad...-createdasset:109-2026-05-22 01:19:28
photo05904b17ee7abb7e...-createdasset:110-2026-05-22 01:19:29
cover6fbc48fe2da94f96...-createdasset:110-2026-05-22 01:19:30
photo81d7a1ec49311dcb...-createdasset:111-2026-05-22 01:19:31
cover5abd0059ce4544a5...-createdasset:111-2026-05-22 01:19:31
photod6c5f2c29b4f50e9...-createdasset:112-2026-05-22 01:19:32
cover2676f78f8611427e...-createdasset:112-2026-05-22 01:19:33
gallery_legacyc5fbb3309bbe7b70...-createdgallery:5-2026-05-22 01:19:33

Recent events

Append-only audit ledger for this batch (latest 50).

event_typepayloadcreated_at
scan{"source": "filesystem", "manifest_sha256": "814632d3551628108a8a4a86ddbdef5a83a845186af99c2b8e3319b4728a8562", "sync_type": "gallery", "action": "import", "item_count": "89", "total_bytes": "15655459...2026-06-27 11:50:01
scan{"source": "filesystem", "manifest_sha256": "814632d3551628108a8a4a86ddbdef5a83a845186af99c2b8e3319b4728a8562", "sync_type": "gallery", "action": "import", "item_count": "89", "total_bytes": "15655459...2026-06-27 11:49:36
job_finished{"error":null,"job_id":"7","phase":"job_finish","summary":{"items_seen":89,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":89,"deferred":0,"failed":0,...2026-05-22 01:19:33
import_finished{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":89,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}2026-05-22 01:19:33
item_updated{"container":1,"expected_member_count":14,"gallery_hash":"c5fbb3309bbe7b706f07a6218666d579","gallery_id":"5","linked_member_count":14,"missing_member_count":0,"phase":"container_link","title":"Captain...2026-05-22 01:19:33
item_updated{"container":1,"expected_member_count":15,"gallery_hash":"778548951971333dff84aa1b6713d132","gallery_id":"4","linked_member_count":15,"missing_member_count":0,"phase":"container_link","title":"Outdoor...2026-05-22 01:19:33
item_updated{"container":1,"expected_member_count":14,"gallery_hash":"3a3d25a8db49358615dc5cbc6ad51a97","gallery_id":"3","linked_member_count":14,"missing_member_count":0,"phase":"container_link","title":"Fruitie...2026-05-22 01:19:33
item_created{"cover_hash":null,"error":null,"hash":"c5fbb3309bbe7b706f07a6218666d579","phase":"item_import","target_ids":{"gallery_id":"5"},"type":"gallery_legacy"}2026-05-22 01:19:33
item_created{"cover_hash":null,"error":null,"hash":"2676f78f8611427e81cce5cc481a9263","phase":"item_import","target_ids":{"asset_id":"112","cover_hash":"2676f78f8611427e81cce5cc481a9263","parent_hash":"d6c5f2c29b...2026-05-22 01:19:33
item_created{"cover_hash":null,"error":null,"hash":"d6c5f2c29b4f50e9c36084fc849fbdeb","phase":"item_import","target_ids":{"asset_id":"112"},"type":"photo"}2026-05-22 01:19:32
item_created{"cover_hash":null,"error":null,"hash":"5abd0059ce4544a58dac1454bc1fa64d","phase":"item_import","target_ids":{"asset_id":"111","cover_hash":"5abd0059ce4544a58dac1454bc1fa64d","parent_hash":"81d7a1ec49...2026-05-22 01:19:31
item_created{"cover_hash":null,"error":null,"hash":"81d7a1ec49311dcbe55e6a9e40c55b2b","phase":"item_import","target_ids":{"asset_id":"111"},"type":"photo"}2026-05-22 01:19:31
item_created{"cover_hash":null,"error":null,"hash":"6fbc48fe2da94f96a0c980e73392176b","phase":"item_import","target_ids":{"asset_id":"110","cover_hash":"6fbc48fe2da94f96a0c980e73392176b","parent_hash":"05904b17ee...2026-05-22 01:19:30
item_created{"cover_hash":null,"error":null,"hash":"05904b17ee7abb7e1d340f328d113f7d","phase":"item_import","target_ids":{"asset_id":"110"},"type":"photo"}2026-05-22 01:19:29
item_created{"cover_hash":null,"error":null,"hash":"962900f5b59f42adb714cf2cc2cd7e7c","phase":"item_import","target_ids":{"asset_id":"109","cover_hash":"962900f5b59f42adb714cf2cc2cd7e7c","parent_hash":"954b3be4e7...2026-05-22 01:19:28
item_created{"cover_hash":null,"error":null,"hash":"954b3be4e774a8c278c315124694c96f","phase":"item_import","target_ids":{"asset_id":"109"},"type":"photo"}2026-05-22 01:19:28
item_created{"cover_hash":null,"error":null,"hash":"60fe0641bb974aaba64aeb3b58613f9f","phase":"item_import","target_ids":{"asset_id":"108","cover_hash":"60fe0641bb974aaba64aeb3b58613f9f","parent_hash":"396dd5945c...2026-05-22 01:19:26
item_created{"cover_hash":null,"error":null,"hash":"396dd5945ce5c6c9716b7efcd8154bf9","phase":"item_import","target_ids":{"asset_id":"108"},"type":"photo"}2026-05-22 01:19:26
item_created{"cover_hash":null,"error":null,"hash":"ae61fe52c533456baa8e3c562936c02d","phase":"item_import","target_ids":{"asset_id":"107","cover_hash":"ae61fe52c533456baa8e3c562936c02d","parent_hash":"bda842a164...2026-05-22 01:19:25
item_created{"cover_hash":null,"error":null,"hash":"bda842a1647dc3b9accd15e1aa17aaa5","phase":"item_import","target_ids":{"asset_id":"107"},"type":"photo"}2026-05-22 01:19:25
item_created{"cover_hash":null,"error":null,"hash":"4083d074810d4113b4c58fa2cd3656c6","phase":"item_import","target_ids":{"asset_id":"106","cover_hash":"4083d074810d4113b4c58fa2cd3656c6","parent_hash":"9e95b5cc35...2026-05-22 01:19:24
item_created{"cover_hash":null,"error":null,"hash":"9e95b5cc354ebf4eb223249c632b5da2","phase":"item_import","target_ids":{"asset_id":"106"},"type":"photo"}2026-05-22 01:19:23
item_created{"cover_hash":null,"error":null,"hash":"a8e446b7678c4ef0afe93deaa6d10c33","phase":"item_import","target_ids":{"asset_id":"105","cover_hash":"a8e446b7678c4ef0afe93deaa6d10c33","parent_hash":"329f0adfa4...2026-05-22 01:19:22
item_created{"cover_hash":null,"error":null,"hash":"329f0adfa485b14284f3da6642975d37","phase":"item_import","target_ids":{"asset_id":"105"},"type":"photo"}2026-05-22 01:19:22
item_created{"cover_hash":null,"error":null,"hash":"110d9ef75bea44b2808b084e2745671f","phase":"item_import","target_ids":{"asset_id":"104","cover_hash":"110d9ef75bea44b2808b084e2745671f","parent_hash":"5909b54d32...2026-05-22 01:19:21
item_created{"cover_hash":null,"error":null,"hash":"5909b54d32cbaddabcfa3760eca67f2e","phase":"item_import","target_ids":{"asset_id":"104"},"type":"photo"}2026-05-22 01:19:21
item_created{"cover_hash":null,"error":null,"hash":"497655b336c047fe8a02ef70e43fdb6c","phase":"item_import","target_ids":{"asset_id":"103","cover_hash":"497655b336c047fe8a02ef70e43fdb6c","parent_hash":"3be0847676...2026-05-22 01:19:20
item_created{"cover_hash":null,"error":null,"hash":"3be084767642387519b0c6afe0f6a3b5","phase":"item_import","target_ids":{"asset_id":"103"},"type":"photo"}2026-05-22 01:19:19
item_created{"cover_hash":null,"error":null,"hash":"81253116ac2d4d668251880d3f84963b","phase":"item_import","target_ids":{"asset_id":"102","cover_hash":"81253116ac2d4d668251880d3f84963b","parent_hash":"7009228a2f...2026-05-22 01:19:18
item_created{"cover_hash":null,"error":null,"hash":"7009228a2f0badd13669be043dcf475f","phase":"item_import","target_ids":{"asset_id":"102"},"type":"photo"}2026-05-22 01:19:18
item_created{"cover_hash":null,"error":null,"hash":"d1f2a8f573f14221b9ea3bec0bc92305","phase":"item_import","target_ids":{"asset_id":"101","cover_hash":"d1f2a8f573f14221b9ea3bec0bc92305","parent_hash":"a29f3b3eab...2026-05-22 01:19:17
item_created{"cover_hash":null,"error":null,"hash":"a29f3b3eab287518ebdfab08887fe88d","phase":"item_import","target_ids":{"asset_id":"101"},"type":"photo"}2026-05-22 01:19:16
item_created{"cover_hash":null,"error":null,"hash":"82083ce09e274748b9894a9a1c802893","phase":"item_import","target_ids":{"asset_id":"100","cover_hash":"82083ce09e274748b9894a9a1c802893","parent_hash":"03976bc398...2026-05-22 01:19:15
item_created{"cover_hash":null,"error":null,"hash":"03976bc398bc54b294eadadc75e92b91","phase":"item_import","target_ids":{"asset_id":"100"},"type":"photo"}2026-05-22 01:19:15
item_created{"cover_hash":null,"error":null,"hash":"aa0ceae9bc9946479e0a2c59765ce17e","phase":"item_import","target_ids":{"asset_id":"99","cover_hash":"aa0ceae9bc9946479e0a2c59765ce17e","parent_hash":"fd975e39233...2026-05-22 01:19:14
item_created{"cover_hash":null,"error":null,"hash":"fd975e3923329e451fbfabf902fca198","phase":"item_import","target_ids":{"asset_id":"99"},"type":"photo"}2026-05-22 01:19:14
item_created{"cover_hash":null,"error":null,"hash":"778548951971333dff84aa1b6713d132","phase":"item_import","target_ids":{"gallery_id":"4"},"type":"gallery_legacy"}2026-05-22 01:19:13
item_created{"cover_hash":null,"error":null,"hash":"82f75605064a42db91e238ea84138c17","phase":"item_import","target_ids":{"asset_id":"98","cover_hash":"82f75605064a42db91e238ea84138c17","parent_hash":"a3e99774194...2026-05-22 01:19:13
item_created{"cover_hash":null,"error":null,"hash":"a3e99774194eb0d5fcd4319e28af1b92","phase":"item_import","target_ids":{"asset_id":"98"},"type":"photo"}2026-05-22 01:19:12
item_created{"cover_hash":null,"error":null,"hash":"4c0ae79ccce64ab89bd217dedd8849ac","phase":"item_import","target_ids":{"asset_id":"97","cover_hash":"4c0ae79ccce64ab89bd217dedd8849ac","parent_hash":"5c92cda582b...2026-05-22 01:19:11
item_created{"cover_hash":null,"error":null,"hash":"5c92cda582b40976280f47a4cf4e494f","phase":"item_import","target_ids":{"asset_id":"97"},"type":"photo"}2026-05-22 01:19:11
item_created{"cover_hash":null,"error":null,"hash":"2eb14c27613e4694a6cb0a176a76701d","phase":"item_import","target_ids":{"asset_id":"96","cover_hash":"2eb14c27613e4694a6cb0a176a76701d","parent_hash":"a5bba31d75b...2026-05-22 01:19:10
item_created{"cover_hash":null,"error":null,"hash":"a5bba31d75b8935d23d5fc0e2525cc14","phase":"item_import","target_ids":{"asset_id":"96"},"type":"photo"}2026-05-22 01:19:09
item_created{"cover_hash":null,"error":null,"hash":"b893eb519d1b4627aa980ae8238bea04","phase":"item_import","target_ids":{"asset_id":"95","cover_hash":"b893eb519d1b4627aa980ae8238bea04","parent_hash":"9ecd01cdc9d...2026-05-22 01:19:08
item_created{"cover_hash":null,"error":null,"hash":"9ecd01cdc9d5a61b9ed5b889ff3a351a","phase":"item_import","target_ids":{"asset_id":"95"},"type":"photo"}2026-05-22 01:19:08
item_created{"cover_hash":null,"error":null,"hash":"988d1f1b9d994c0bbf42eb2874e9d6cd","phase":"item_import","target_ids":{"asset_id":"94","cover_hash":"988d1f1b9d994c0bbf42eb2874e9d6cd","parent_hash":"d0441f4209d...2026-05-22 01:19:07
item_created{"cover_hash":null,"error":null,"hash":"d0441f4209d846348b3d91870753568f","phase":"item_import","target_ids":{"asset_id":"94"},"type":"photo"}2026-05-22 01:19:07
item_created{"cover_hash":null,"error":null,"hash":"469bc0577e344c748fde5935fad0f9e6","phase":"item_import","target_ids":{"asset_id":"93","cover_hash":"469bc0577e344c748fde5935fad0f9e6","parent_hash":"71c4e647c60...2026-05-22 01:19:06
item_created{"cover_hash":null,"error":null,"hash":"71c4e647c600fe331f87da22da6a44ac","phase":"item_import","target_ids":{"asset_id":"93"},"type":"photo"}2026-05-22 01:19:05
item_created{"cover_hash":null,"error":null,"hash":"5b2fd9b16e0545a7b0f804b754b4b05f","phase":"item_import","target_ids":{"asset_id":"92","cover_hash":"5b2fd9b16e0545a7b0f804b754b4b05f","parent_hash":"5169370fbf9...2026-05-22 01:19:04