Import Media

About this section: Import MediaQueue cloud-sync handover bundles; the worker imports them in the background.

Purpose

Consume the upstream cloud-sync handover bundles from /mnt/ubuntu-nas/node2/assets/sync/ into the global library (asset, asset_gallery, asset_collection) per Decision 639. The CGI never imports inline; it queues batches into import_job for the background worker (Decision 640).

Flow

Scan: walks the mount for .ready sentinels and records new ingest_batch rows.
Pre-flight: validates the manifest shape, sentinel pair, and per-item decisions; no library writes.
Queue: auto-preflights if needed, then inserts an import_job row. Live progress shows on Import Tasks.
Bulk queue: select multiple batches and queue them in one click; the worker drains the queue in priority + FIFO order.

Example

Scan finds 37e428d2-a4f6-4a31-bacf-0c835fb24084.ready. Tick its checkbox (or several at once), click Queue selected. The worker claims the next queued job within seconds and writes job_* events as it progresses.

Consequences

The library is site-agnostic. Sites surface library content via asset_site_membership, gallery/collection membership, content allowlists, and access packages.
VIDEO-SIMPLE syncs import as stable asset_type=video rows with asset_subtype=video_simple, duration, HLS, transcript, cloud-package, and library-cover deliverables.
Cancel is allowed only while a job is queued; once running, only systemctl stop gpu-admin-panel-import-worker stops it.
Action remove only soft-deletes rows via deleted_at; nothing is hard-deleted.
Frosted, blurred, and vivid_blurred derivatives are intentionally not persisted.

Deeper dive

Sourced from gpu-floor/MIGRATION-INTRO.md and the queued-worker decision in DECISION_LOG.md.

Worker contract

A long-running import-worker.pl daemon claims queued jobs using SELECT ... FOR UPDATE SKIP LOCKED (MariaDB 10.6+). It updates heartbeat_at every few items; if a worker dies, its running jobs are reset to queued after the configured stale threshold (default 15 minutes).

Sentinel protocol

A bundle is consumable only when both <sync_id>.json and <sync_id>.ready exist and the JSON has status = completed and verification.passed = true. The importer recomputes manifest_sha256 at consumption time so audit history records what we actually ingested.

Source-of-truth mapping

source/source.json -> asset_source (cold provenance).
source/data/photo.manifest.json + photo.analysis.json + photo.meaning.json + photo.family.json -> asset_ai_run rows with explicit analysis_kind; unknown filenames are captured with source_data:<file>.
Bundled cover_hash -> asset_cover linked to the parent asset.
Containers of type gallery -> asset_gallery + asset_gallery_item junctions.

Idempotency + integrity

Every multi-row write is wrapped in AdminDB::transaction. Hash-based UNIQUE keys make re-running the same batch a no-op. Per-item failures isolate to ingest_batch_item.decision = failed with last_error; the batch becomes partial when any siblings still succeeded.

Read-only mount + ledger boundary

The /mnt/ubuntu-nas/node2/assets tree is read-only for admin consumers. The importer never writes to upstream or deletes .ready sentinels; all consumer state lives in ingest_batch / ingest_batch_item / ingest_event / import_job in admin_panel.

Type

gallery

Action

import

Status

imported

Items

Bytes

281.8 MB

Source host

flux-1

Manifest sha256

665465a7ef85f358...

Started

2026-08-12 19:35:20

Completed

2026-08-12 19:39:05

Manifest summary

created_at: 2026-04-14T19:49:31Z
completed_at: 2026-04-14T19:58:34Z
verification.passed: true
verification.total_files: 1720
prior_sync_id: none

id	state	requested_by	queued	claimed	worker	progress	finished	last_error
414	done	import-feed	2026-08-12 19:35:25	2026-08-12 19:37:25	`vultr/pid:946211`	47/47	2026-08-12 19:39:05	-

asset_type	asset_hash	cover_hash	decision	target ids	last_error	updated_at
photo	`b4c730cb3c3e957d...`	-	created	asset:121892	-	2026-08-12 19:37:28
cover	`9e05e107cca84328...`	-	created	asset:121892	-	2026-08-12 19:37:30
photo	`6cc454076105a185...`	-	created	asset:121893	-	2026-08-12 19:37:33
cover	`01ed4da8cec14c05...`	-	created	asset:121893	-	2026-08-12 19:37:34
photo	`66ea61689627cba7...`	-	created	asset:121894	-	2026-08-12 19:37:37
cover	`f780e6551adb4fef...`	-	created	asset:121894	-	2026-08-12 19:37:39
photo	`64a57708f723f071...`	-	created	asset:121895	-	2026-08-12 19:37:41
cover	`0977bbee76304bea...`	-	created	asset:121895	-	2026-08-12 19:37:43
photo	`ced9340bd71bbab9...`	-	created	asset:121896	-	2026-08-12 19:37:46
cover	`cd6677c5af824b35...`	-	created	asset:121896	-	2026-08-12 19:37:47
photo	`7a5cfef537b348cb...`	-	created	asset:121897	-	2026-08-12 19:37:50
cover	`9eefc289b56b4e84...`	-	created	asset:121897	-	2026-08-12 19:37:51
photo	`ab2ab7397718c5be...`	-	created	asset:121898	-	2026-08-12 19:37:54
cover	`96c4233add4d4c6d...`	-	created	asset:121898	-	2026-08-12 19:37:55
photo	`8f14929fb94506bb...`	-	created	asset:121899	-	2026-08-12 19:37:58
cover	`ced2c0e864bf4aaf...`	-	created	asset:121899	-	2026-08-12 19:38:00
photo	`1dada3fcbc99bb76...`	-	created	asset:121900	-	2026-08-12 19:38:02
cover	`fae2f16c59304b0e...`	-	created	asset:121900	-	2026-08-12 19:38:04
photo	`12c3d665a3ccabf9...`	-	created	asset:121901	-	2026-08-12 19:38:07
cover	`c078c77805874ce8...`	-	created	asset:121901	-	2026-08-12 19:38:08
photo	`ba1ba1da791c04bb...`	-	created	asset:121902	-	2026-08-12 19:38:12
cover	`f64873e480c94f77...`	-	created	asset:121902	-	2026-08-12 19:38:13
photo	`4fc5dcf5542f34e6...`	-	created	asset:121903	-	2026-08-12 19:38:16
cover	`618634d9ff754e01...`	-	created	asset:121903	-	2026-08-12 19:38:17
photo	`629c029eada4bf89...`	-	created	asset:121904	-	2026-08-12 19:38:20
cover	`bb4186021f8d4298...`	-	created	asset:121904	-	2026-08-12 19:38:22
photo	`cc0a6147dbf7c949...`	-	created	asset:121905	-	2026-08-12 19:38:24
cover	`feda1b63f5564f26...`	-	created	asset:121905	-	2026-08-12 19:38:26
photo	`ad6506b7c8c82dfd...`	-	created	asset:121906	-	2026-08-12 19:38:29
cover	`86efcc2127484d2f...`	-	created	asset:121906	-	2026-08-12 19:38:30
photo	`e4bd29a8b4cc1efb...`	-	created	asset:121907	-	2026-08-12 19:38:33
cover	`476f78291bf9458a...`	-	created	asset:121907	-	2026-08-12 19:38:34
photo	`0909df42076f68f4...`	-	created	asset:121908	-	2026-08-12 19:38:37
cover	`83c255c996f84ae2...`	-	created	asset:121908	-	2026-08-12 19:38:39
photo	`4665c5654008394a...`	-	created	asset:121909	-	2026-08-12 19:38:41
cover	`4f50f536633f4e2f...`	-	created	asset:121909	-	2026-08-12 19:38:43
photo	`53581a3ee963f14b...`	-	created	asset:121910	-	2026-08-12 19:38:46
cover	`601ada810bf14676...`	-	created	asset:121910	-	2026-08-12 19:38:47
photo	`b4a3b6f6a9b94ab7...`	-	created	asset:121911	-	2026-08-12 19:38:50
cover	`48a2b2e4d6164d3a...`	-	created	asset:121911	-	2026-08-12 19:38:51
photo	`8c52d47423c63c6f...`	-	created	asset:121912	-	2026-08-12 19:38:54
cover	`04404a380050404a...`	-	created	asset:121912	-	2026-08-12 19:38:56
photo	`68df0c5dbfd38e48...`	-	created	asset:121913	-	2026-08-12 19:38:59
cover	`cf59e7d08d034eca...`	-	created	asset:121913	-	2026-08-12 19:39:00
photo	`f66a69dbd35acc77...`	-	created	asset:121914	-	2026-08-12 19:39:03
cover	`136a2fab3c9349c1...`	-	created	asset:121914	-	2026-08-12 19:39:04
gallery_legacy	`3d8d0e597a9736dc...`	-	created	gallery:3931	-	2026-08-12 19:39:05

event_type	payload	created_at
job_finished	`{"error":null,"job_id":"414","phase":"job_finish","summary":{"items_seen":47,"ok":1,"status":"imported","summary":{"ok":1,"remaining":0,"status":"imported","tally":{"created":47,"deferred":0,"failed":...`	2026-08-12 19:39:05
import_finished	`{"phase":"import","remaining":0,"skipped_terminal":0,"status":"imported","tally":{"created":47,"deferred":0,"failed":0,"linked":0,"skipped":0,"updated":0}}`	2026-08-12 19:39:05
item_updated	`{"container":1,"expected_member_count":23,"gallery_hash":"3d8d0e597a9736dcbc44c6b4d71ba659","gallery_id":"3931","linked_member_count":23,"missing_member_count":0,"phase":"container_link","title":"Nept...`	2026-08-12 19:39:05
item_created	`{"cover_hash":null,"error":null,"hash":"3d8d0e597a9736dcbc44c6b4d71ba659","phase":"item_import","target_ids":{"gallery_id":"3931"},"type":"gallery_legacy"}`	2026-08-12 19:39:05
item_created	`{"cover_hash":null,"error":null,"hash":"136a2fab3c9349c1b29b4df55b19f972","phase":"item_import","target_ids":{"asset_id":"121914","cover_hash":"136a2fab3c9349c1b29b4df55b19f972","parent_hash":"f66a69d...`	2026-08-12 19:39:04
item_created	`{"cover_hash":null,"error":null,"hash":"f66a69dbd35acc77295a5165c3137374","phase":"item_import","target_ids":{"asset_id":"121914"},"type":"photo"}`	2026-08-12 19:39:03
item_created	`{"cover_hash":null,"error":null,"hash":"cf59e7d08d034ecaaab6b343e17ef982","phase":"item_import","target_ids":{"asset_id":"121913","cover_hash":"cf59e7d08d034ecaaab6b343e17ef982","parent_hash":"68df0c5...`	2026-08-12 19:39:00
item_created	`{"cover_hash":null,"error":null,"hash":"68df0c5dbfd38e4862735084d4c19be9","phase":"item_import","target_ids":{"asset_id":"121913"},"type":"photo"}`	2026-08-12 19:38:59
item_created	`{"cover_hash":null,"error":null,"hash":"04404a380050404a858267b8d91441b3","phase":"item_import","target_ids":{"asset_id":"121912","cover_hash":"04404a380050404a858267b8d91441b3","parent_hash":"8c52d47...`	2026-08-12 19:38:56
item_created	`{"cover_hash":null,"error":null,"hash":"8c52d47423c63c6f1b19f62b7b881b43","phase":"item_import","target_ids":{"asset_id":"121912"},"type":"photo"}`	2026-08-12 19:38:54
item_created	`{"cover_hash":null,"error":null,"hash":"48a2b2e4d6164d3a999eeebb9e44a893","phase":"item_import","target_ids":{"asset_id":"121911","cover_hash":"48a2b2e4d6164d3a999eeebb9e44a893","parent_hash":"b4a3b6f...`	2026-08-12 19:38:51
item_created	`{"cover_hash":null,"error":null,"hash":"b4a3b6f6a9b94ab74784ec22b5bc3644","phase":"item_import","target_ids":{"asset_id":"121911"},"type":"photo"}`	2026-08-12 19:38:50
item_created	`{"cover_hash":null,"error":null,"hash":"601ada810bf14676aaa7a43e821d3f98","phase":"item_import","target_ids":{"asset_id":"121910","cover_hash":"601ada810bf14676aaa7a43e821d3f98","parent_hash":"53581a3...`	2026-08-12 19:38:47
item_created	`{"cover_hash":null,"error":null,"hash":"53581a3ee963f14b7ff68520b0bfa6a6","phase":"item_import","target_ids":{"asset_id":"121910"},"type":"photo"}`	2026-08-12 19:38:46
item_created	`{"cover_hash":null,"error":null,"hash":"4f50f536633f4e2f95d8a78859723e28","phase":"item_import","target_ids":{"asset_id":"121909","cover_hash":"4f50f536633f4e2f95d8a78859723e28","parent_hash":"4665c56...`	2026-08-12 19:38:43
item_created	`{"cover_hash":null,"error":null,"hash":"4665c5654008394ab0243b5e7798c7d8","phase":"item_import","target_ids":{"asset_id":"121909"},"type":"photo"}`	2026-08-12 19:38:41
item_created	`{"cover_hash":null,"error":null,"hash":"83c255c996f84ae28e918d999fc44c9b","phase":"item_import","target_ids":{"asset_id":"121908","cover_hash":"83c255c996f84ae28e918d999fc44c9b","parent_hash":"0909df4...`	2026-08-12 19:38:39
item_created	`{"cover_hash":null,"error":null,"hash":"0909df42076f68f480dbd79cad5e2fa6","phase":"item_import","target_ids":{"asset_id":"121908"},"type":"photo"}`	2026-08-12 19:38:37
item_created	`{"cover_hash":null,"error":null,"hash":"476f78291bf9458ab2b4022a05c6fdda","phase":"item_import","target_ids":{"asset_id":"121907","cover_hash":"476f78291bf9458ab2b4022a05c6fdda","parent_hash":"e4bd29a...`	2026-08-12 19:38:34
item_created	`{"cover_hash":null,"error":null,"hash":"e4bd29a8b4cc1efb978668a91e318ed3","phase":"item_import","target_ids":{"asset_id":"121907"},"type":"photo"}`	2026-08-12 19:38:33
item_created	`{"cover_hash":null,"error":null,"hash":"86efcc2127484d2fbe1a13a464b77c86","phase":"item_import","target_ids":{"asset_id":"121906","cover_hash":"86efcc2127484d2fbe1a13a464b77c86","parent_hash":"ad6506b...`	2026-08-12 19:38:30
item_created	`{"cover_hash":null,"error":null,"hash":"ad6506b7c8c82dfd62a53e1f66834f37","phase":"item_import","target_ids":{"asset_id":"121906"},"type":"photo"}`	2026-08-12 19:38:29
item_created	`{"cover_hash":null,"error":null,"hash":"feda1b63f5564f26b5a17d5353016f58","phase":"item_import","target_ids":{"asset_id":"121905","cover_hash":"feda1b63f5564f26b5a17d5353016f58","parent_hash":"cc0a614...`	2026-08-12 19:38:26
item_created	`{"cover_hash":null,"error":null,"hash":"cc0a6147dbf7c9497d180aef55f13d6b","phase":"item_import","target_ids":{"asset_id":"121905"},"type":"photo"}`	2026-08-12 19:38:24
item_created	`{"cover_hash":null,"error":null,"hash":"bb4186021f8d429898be11309f119347","phase":"item_import","target_ids":{"asset_id":"121904","cover_hash":"bb4186021f8d429898be11309f119347","parent_hash":"629c029...`	2026-08-12 19:38:22
item_created	`{"cover_hash":null,"error":null,"hash":"629c029eada4bf890302bd3b5e754482","phase":"item_import","target_ids":{"asset_id":"121904"},"type":"photo"}`	2026-08-12 19:38:20
item_created	`{"cover_hash":null,"error":null,"hash":"618634d9ff754e0193725352eb80b8c1","phase":"item_import","target_ids":{"asset_id":"121903","cover_hash":"618634d9ff754e0193725352eb80b8c1","parent_hash":"4fc5dcf...`	2026-08-12 19:38:17
item_created	`{"cover_hash":null,"error":null,"hash":"4fc5dcf5542f34e623573e666a429822","phase":"item_import","target_ids":{"asset_id":"121903"},"type":"photo"}`	2026-08-12 19:38:16
item_created	`{"cover_hash":null,"error":null,"hash":"f64873e480c94f778480a2788858ea78","phase":"item_import","target_ids":{"asset_id":"121902","cover_hash":"f64873e480c94f778480a2788858ea78","parent_hash":"ba1ba1d...`	2026-08-12 19:38:13
item_created	`{"cover_hash":null,"error":null,"hash":"ba1ba1da791c04bbd1a01c5de7eaf31d","phase":"item_import","target_ids":{"asset_id":"121902"},"type":"photo"}`	2026-08-12 19:38:12
item_created	`{"cover_hash":null,"error":null,"hash":"c078c77805874ce893a02e820811789c","phase":"item_import","target_ids":{"asset_id":"121901","cover_hash":"c078c77805874ce893a02e820811789c","parent_hash":"12c3d66...`	2026-08-12 19:38:08
item_created	`{"cover_hash":null,"error":null,"hash":"12c3d665a3ccabf99612185a54ed61d7","phase":"item_import","target_ids":{"asset_id":"121901"},"type":"photo"}`	2026-08-12 19:38:07
item_created	`{"cover_hash":null,"error":null,"hash":"fae2f16c59304b0e80d804eb551f41b7","phase":"item_import","target_ids":{"asset_id":"121900","cover_hash":"fae2f16c59304b0e80d804eb551f41b7","parent_hash":"1dada3f...`	2026-08-12 19:38:04
item_created	`{"cover_hash":null,"error":null,"hash":"1dada3fcbc99bb76ef97f633569b5ac8","phase":"item_import","target_ids":{"asset_id":"121900"},"type":"photo"}`	2026-08-12 19:38:02
item_created	`{"cover_hash":null,"error":null,"hash":"ced2c0e864bf4aafb663f9caa6f1684f","phase":"item_import","target_ids":{"asset_id":"121899","cover_hash":"ced2c0e864bf4aafb663f9caa6f1684f","parent_hash":"8f14929...`	2026-08-12 19:38:00
item_created	`{"cover_hash":null,"error":null,"hash":"8f14929fb94506bb3dc6c23dbff9003e","phase":"item_import","target_ids":{"asset_id":"121899"},"type":"photo"}`	2026-08-12 19:37:58
item_created	`{"cover_hash":null,"error":null,"hash":"96c4233add4d4c6dad3e1d942cb332e4","phase":"item_import","target_ids":{"asset_id":"121898","cover_hash":"96c4233add4d4c6dad3e1d942cb332e4","parent_hash":"ab2ab73...`	2026-08-12 19:37:55
item_created	`{"cover_hash":null,"error":null,"hash":"ab2ab7397718c5beba6d0897e8c9416e","phase":"item_import","target_ids":{"asset_id":"121898"},"type":"photo"}`	2026-08-12 19:37:54
item_created	`{"cover_hash":null,"error":null,"hash":"9eefc289b56b4e848a19fa2a8e98c1b8","phase":"item_import","target_ids":{"asset_id":"121897","cover_hash":"9eefc289b56b4e848a19fa2a8e98c1b8","parent_hash":"7a5cfef...`	2026-08-12 19:37:51
item_created	`{"cover_hash":null,"error":null,"hash":"7a5cfef537b348cb3715b97b15b46975","phase":"item_import","target_ids":{"asset_id":"121897"},"type":"photo"}`	2026-08-12 19:37:50
item_created	`{"cover_hash":null,"error":null,"hash":"cd6677c5af824b35b292ceba1598f3e3","phase":"item_import","target_ids":{"asset_id":"121896","cover_hash":"cd6677c5af824b35b292ceba1598f3e3","parent_hash":"ced9340...`	2026-08-12 19:37:47
item_created	`{"cover_hash":null,"error":null,"hash":"ced9340bd71bbab945f115cb6294e585","phase":"item_import","target_ids":{"asset_id":"121896"},"type":"photo"}`	2026-08-12 19:37:46
item_created	`{"cover_hash":null,"error":null,"hash":"0977bbee76304bea81d67f93ebde2a77","phase":"item_import","target_ids":{"asset_id":"121895","cover_hash":"0977bbee76304bea81d67f93ebde2a77","parent_hash":"64a5770...`	2026-08-12 19:37:43
item_created	`{"cover_hash":null,"error":null,"hash":"64a57708f723f0718afda4c32f5136e7","phase":"item_import","target_ids":{"asset_id":"121895"},"type":"photo"}`	2026-08-12 19:37:41
item_created	`{"cover_hash":null,"error":null,"hash":"f780e6551adb4fefaba30a80dff6d8f3","phase":"item_import","target_ids":{"asset_id":"121894","cover_hash":"f780e6551adb4fefaba30a80dff6d8f3","parent_hash":"66ea616...`	2026-08-12 19:37:39
item_created	`{"cover_hash":null,"error":null,"hash":"66ea61689627cba709f920d9afeedfe3","phase":"item_import","target_ids":{"asset_id":"121894"},"type":"photo"}`	2026-08-12 19:37:37
item_created	`{"cover_hash":null,"error":null,"hash":"01ed4da8cec14c05b35d376373a785bc","phase":"item_import","target_ids":{"asset_id":"121893","cover_hash":"01ed4da8cec14c05b35d376373a785bc","parent_hash":"6cc4540...`	2026-08-12 19:37:34
item_created	`{"cover_hash":null,"error":null,"hash":"6cc454076105a18583e5226cde764c38","phase":"item_import","target_ids":{"asset_id":"121893"},"type":"photo"}`	2026-08-12 19:37:33
item_created	`{"cover_hash":null,"error":null,"hash":"9e05e107cca84328a1f6e82e7951818f","phase":"item_import","target_ids":{"asset_id":"121892","cover_hash":"9e05e107cca84328a1f6e82e7951818f","parent_hash":"b4c730c...`	2026-08-12 19:37:30
item_created	`{"cover_hash":null,"error":null,"hash":"b4c730cb3c3e957d304bdb7ab2d1fbb0","phase":"item_import","target_ids":{"asset_id":"121892"},"type":"photo"}`	2026-08-12 19:37:28

Media Manager