Skip to content

Generated Files Storage Code Guide (Developers)

This guide explains how generated files are saved, registered for quota tracking, served back to users, and cleaned up.

See also: - Storage API reference: Docs/API-related/Storage_API_Documentation.md:1 - Storage router: tldw_Server_API/app/api/v1/endpoints/storage.py:142

Scope and Design Intent

Generated files storage is responsible for: - Persisting file metadata in generated_files. - Enforcing quotas and surfacing soft-limit warnings. - Ensuring soft delete, restore, and hard delete keep usage counters correct. - Preventing path traversal on download and cleanup.

Key anchors: - Register/unregister: tldw_Server_API/app/services/storage_quota_service.py:760 - Cleanup worker: tldw_Server_API/app/services/storage_cleanup_service.py:176 - Helpers: tldw_Server_API/app/core/Storage/generated_file_helpers.py:128

Quick Map (Where Things Live)

  • API surface
  • Router: tldw_Server_API/app/api/v1/endpoints/storage.py:142
  • Schemas: tldw_Server_API/app/api/v1/schemas/storage_schemas.py:165
  • Core services and repos
  • Storage quota service: tldw_Server_API/app/services/storage_quota_service.py:760
  • Generated files repo: tldw_Server_API/app/core/AuthNZ/repos/generated_files_repo.py:848
  • Storage quotas repo: tldw_Server_API/app/core/AuthNZ/repos/storage_quotas_repo.py:511
  • Background cleanup
  • Cleanup service: tldw_Server_API/app/services/storage_cleanup_service.py:176
  • App lifecycle hooks: tldw_Server_API/app/main.py:2036

Base Directories and File Categories

Files are stored under per-user base directories resolved by category: - Default: outputs dir via DatabasePaths.get_user_outputs_dir. - Voice clones: voices dir via DatabasePaths.get_user_voices_dir.

Anchors: - Endpoint base-dir resolver: tldw_Server_API/app/api/v1/endpoints/storage.py:101 - Cleanup safe resolver: tldw_Server_API/app/services/storage_cleanup_service.py:32

Any code that computes a file path must resolve it and verify it stays inside the correct base directory.

Registering Files (Quota and Metadata)

The main entry point is StorageQuotaService.register_generated_file(...): - Anchor: tldw_Server_API/app/services/storage_quota_service.py:760 - Important behavior: - Enforces a hard per-file size limit of 10 GB. - Anchor: tldw_Server_API/app/services/storage_quota_service.py:809 - Performs combined quota checks across user/team/org scopes. - Persists a generated_files row and updates usage counters.

Use the generated-file helper functions when possible: - TTS audio: save_and_register_tts_audio(...) (tldw_Server_API/app/core/Storage/generated_file_helpers.py:128) - Images: save_and_register_image(...) (tldw_Server_API/app/core/Storage/generated_file_helpers.py:230) - Spreadsheets: save_and_register_spreadsheet(...) (tldw_Server_API/app/core/Storage/generated_file_helpers.py:392)

These helpers: - Save bytes into the correct per-user outputs directory. - Compute the relative storage_path. - Register the file with quota tracking. - Clean up the file if registration fails.

Deleting Files Without Double-Subtracting Usage

The deletion flow is intentionally careful about usage counters: - Anchor: tldw_Server_API/app/services/storage_quota_service.py:880

Key behavior: - Soft delete subtracts usage once when the file transitions from active to deleted. - Hard delete subtracts usage only if the file was not already soft-deleted. - Anchor: tldw_Server_API/app/services/storage_quota_service.py:901 - Trash purge and hard delete of already-deleted files should not subtract usage again.

Voice Clone Filesystem Cleanup

On hard delete, voice clone files are also removed from disk: - Anchor: tldw_Server_API/app/services/storage_quota_service.py:911

This hard-delete cleanup: - Uses the voices base directory. - Resolves and validates the path before unlinking. - Is best-effort and logs a debug message on failure.

API Behaviors Worth Knowing

Download Security and Access Tracking

Downloads: - Validate ownership. - Block deleted files with HTTP 410. - Resolve and validate the path to prevent traversal. - Update accessed_at on success.

Anchors: - Download endpoint: tldw_Server_API/app/api/v1/endpoints/storage.py:203 - Traversal guard: tldw_Server_API/app/api/v1/endpoints/storage.py:228 - Access updates: tldw_Server_API/app/core/AuthNZ/repos/generated_files_repo.py:887

Bulk Delete Tracks Usage Correctly

Bulk delete routes each file through unregister_generated_file so usage counters are updated correctly: - Anchor: tldw_Server_API/app/api/v1/endpoints/storage.py:318

Soft-Limit Warnings in Usage Responses

GET /api/v1/storage/usage now includes soft/hard limit flags and a warning message: - Endpoint calculation: tldw_Server_API/app/api/v1/endpoints/storage.py:478 - Schema fields: tldw_Server_API/app/api/v1/schemas/storage_schemas.py:172

Soft limit is 80% usage and hard limit is 100%.

Least-Accessed Listing for Cleanup Suggestions

A least-accessed endpoint and repo method support LRU-style cleanup suggestions: - Endpoint: tldw_Server_API/app/api/v1/endpoints/storage.py:420 - Repo method: tldw_Server_API/app/core/AuthNZ/repos/generated_files_repo.py:905

The sort order uses COALESCE(accessed_at, created_at).

Background Cleanup Worker

The cleanup worker handles three things per cycle: - Expired files: hard delete via the quota service (usage updates). - Anchor: tldw_Server_API/app/services/storage_cleanup_service.py:55 - Old trash: remove from disk (if present) and hard delete the row without touching usage. - Anchor: tldw_Server_API/app/services/storage_cleanup_service.py:104 - Temp directories: clean up old temporary files. - Anchor: tldw_Server_API/app/services/storage_cleanup_service.py:217

Worker lifecycle: - Started on app startup when STORAGE_CLEANUP_ENABLED is truthy. - Anchor: tldw_Server_API/app/main.py:2036 - Stopped on app shutdown. - Anchor: tldw_Server_API/app/main.py:2906

Environment variables: - STORAGE_CLEANUP_ENABLED - STORAGE_CLEANUP_INTERVAL_SEC - STORAGE_TRASH_RETENTION_DAYS - STORAGE_CLEANUP_BATCH_SIZE

Integrations Added in This Cycle

The TTS endpoint can save generated audio into storage and return a download path in headers: - Endpoint constraint: return_download_link requires stream=false. - Anchor: tldw_Server_API/app/api/v1/endpoints/audio.py:1045 - Headers on success: - X-Download-Path: /api/v1/storage/files/{id}/download - X-Generated-File-Id: generated file id - Anchor: tldw_Server_API/app/api/v1/endpoints/audio.py:1179 - Registration helper: save_and_register_tts_audio(...). - Anchor: tldw_Server_API/app/core/Storage/generated_file_helpers.py:128

Voice Studio: voice clone registration

Voice uploads are registered as voice_clone generated files for quota tracking: - Anchor: tldw_Server_API/app/core/TTS/voice_manager.py:640

File Artifacts: image and spreadsheet exports

Image and spreadsheet exports now register generated files and map storage errors to file-artifact error codes: - Export registration: tldw_Server_API/app/core/File_Artifacts/file_artifacts_service.py:374 - Error codes: storage_quota_exceeded and storage_persist_failed. - Anchor: tldw_Server_API/app/core/exceptions.py:158

Testing Anchors

Storage-specific tests live here: - tldw_Server_API/tests/Storage/

A good focused check after storage changes:

python -m pytest tldw_Server_API/tests/Storage -q