A focused set of capabilities for managing file backup to S3-compatible object storage. No proprietary formats. No lock-in. Designed to complement your existing infrastructure.
Each run walks one or more source directories and computes a CRC32 checksum combined with modification time and size for every file. By comparing against the last recorded state in MongoDB, only new or modified files are transferred. Unchanged files are skipped.
Arkhivio runs as a cron job, systemd timer, or Docker Compose service. Any scheduler that can invoke a command is sufficient. A dry-run mode previews changes without transferring any data.
Multi-threaded uploads with a configurable worker pool (default: 4 threads). A token-bucket bandwidth limiter is thread-safe and aggregated across all workers — supporting KB/s, MB/s, or GB/s limits without saturating the network. When S3 rate-limits are encountered, the framework backs off and retries automatically (30 s, up to 5 attempts).
Transparent gzip compression, configurable levels 1–9. A size threshold (default 500 MB) prevents
attempting to compress already-compressed content. Compression state is stored in the S3
object's metadata (x-amz-meta-compressed) so the restore process handles it automatically.
A CRC32 checksum is computed at scan time, stored in MongoDB, and written to the S3 object
as standard metadata (x-amz-meta-crc32). The checksum flows through every stage —
audit, restore — so drift is detectable at any point without requiring a full re-scan.
# CRC32 in standard S3 object metadata $ aws s3api head-object \ --bucket prod-backups \ --key webserver-01/data/records.db "x-amz-meta-crc32": "c0ffee12" "x-amz-meta-compressed": "false" # Readable by any S3 client — no Arkhivio needed
Every file is one S3 object at a deterministic path. No proprietary format. No index required. Accessible by any S3-compatible tool.
--endpoint
{hostname}/{absolute/path/to/file}
Deterministic and human-readable. Reconstruct any path by hand. No look-up table required.
Depending on the S3-compatible platform and bucket configuration, the storage layer can independently provide:
Available capabilities vary by provider and bucket configuration. Arkhivio does not implement proprietary versions of these features — it delegates to the storage layer by design.
Every run is recorded in MongoDB with a structured summary: files scanned, uploaded, skipped,
errors, bytes transferred, and timestamps. History is queryable by job and host, and a
compressed files.json.gz inventory is published to S3 after each run.
The audit command performs three independent checks: (1) verifies that a snapshot document
exists in MongoDB, (2) confirms that the S3 object's x-amz-meta-crc32 matches the
MongoDB record, and (3) downloads a file and recomputes the CRC32 from disk. Each check is
separately observable and separately alertable.
The reconciler compares MongoDB records against S3 object listings to detect drift: files marked as synced in MongoDB but absent from S3, or objects in S3 with no corresponding MongoDB record. Detected drift is logged and can be addressed interactively.
All 13 modules emit newline-delimited JSON events to stdout and an optional rotating log file. No parsing rules needed. Compatible with any modern log aggregator on day one.
Because every event is structured JSON with a named event field, any log aggregator
can generate alerts by filtering on event names. No custom agent or plugin required.
upload_error — file backup failuredelete_s3_error — S3 delete failurereconcile_missing_s3 — objects missingupload_slowdown — S3 rate limitingsecrets_load_failed — provider unreachableAlert destinations: Slack, Microsoft Teams, PagerDuty, OpsGenie, email, or SMS — using your existing alerting pipeline.
All restore paths verify CRC32 integrity on every file before writing to disk.
Restore a single file or a specific subtree by job, host, or path pattern. Useful for recovering an individual file or a specific directory without retrieving an entire backup set.
Restore all files for a job or host from the most recent recorded state. Configurable worker count for parallel download. Optional path remapping to restore to an alternative destination.
restore_onlys3.py works directly against the bucket — no MongoDB connection
required. It reads the files.json.gz inventory and x-amz-meta-crc32
metadata from S3 itself, then downloads and verifies each file.
Pull S3 credentials from your existing secrets infrastructure. No hardcoded keys anywhere in configuration files. Lazy-loaded so unused providers add zero overhead.
Arkhivio runs wherever Python 3.10+ runs. The OS version is detected and recorded on every run for compliance audit trails.
--host flag for stable identity across pod restartsS3 credentials stored locally are encrypted with Fernet (AES-128-CBC + HMAC). Key rotation is supported and documented.
No per-seat fees. No per-server royalties. No usage-based billing.
The source code is delivered when you sign the contract. Use it on as many servers, in as many environments, for as long as you need — with no recurring licensing cost tied to protected capacity.
Walk through the complete backup pipeline from scan to verified restore.