[{"data":1,"prerenderedAt":1720},["ShallowReactive",2],{"authors":3,"\u002Fblog\u002Fautomated-restic-backups":26,"\u002Fblog\u002Fautomated-restic-backups-surround":1698,"\u002Fblog\u002Fautomated-restic-backups-related":1705},[4,16],{"id":5,"avatar":6,"bio":7,"extension":8,"github":9,"meta":10,"name":11,"role":12,"stem":13,"website":14,"__hash__":15},"authors\u002Fauthors\u002Fsagar-kapoor.yml","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F69609200?v=4",null,"yml","sagar-kap",{},"Sagar Kapoor","Chief Executive Officer (CEO)","authors\u002Fsagar-kapoor","https:\u002F\u002Fsagarkapoor.eu","NsLNClfs_S-SNHFEnyR6Vq_7zEc1Xxl7QLoKrk_DB6s",{"id":17,"avatar":18,"bio":7,"extension":8,"github":19,"meta":20,"name":21,"role":22,"stem":23,"website":24,"__hash__":25},"authors\u002Fauthors\u002Fsomraj-saha.yml","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31373860?v=4","jarmos-san",{},"Somraj Saha","Chief Technology Officer (CTO)","authors\u002Fsomraj-saha","https:\u002F\u002Fjarmos.dev","mDAq8GgPZv9H8en0XHAraq3aYi_v8t2ULWGqp4cp3zY",{"id":27,"title":28,"author":29,"body":30,"category":1688,"cover":1689,"date":1690,"description":1691,"extension":1692,"meta":1693,"navigation":356,"path":1694,"seo":1695,"stem":1696,"__hash__":1697},"blog\u002Fblog\u002Fautomated-restic-backups.md","How We Automated Secure, Deduplicated Backups with Restic","somraj-saha",{"type":31,"value":32,"toc":1680},"minimark",[33,63,68,71,74,128,137,141,144,156,159,191,202,211,217,268,274,278,287,297,1384,1394,1522,1532,1612,1626,1630,1633,1667,1670,1673,1676],[34,35,36,37,44,45,50,51,56,57,62],"p",{},"At ",[38,39,43],"a",{"href":40,"rel":41},"https:\u002F\u002Fweburz.com",[42],"nofollow","Weburz",", we rely on a suite of self-hosted services that\npower our day-to-day operations — ",[38,46,49],{"href":47,"rel":48},"https:\u002F\u002Fpenpot.app",[42],"application"," for design\ncollaboration, ",[38,52,55],{"href":53,"rel":54},"https:\u002F\u002Fumami.is",[42],"Umami"," for analytics, and\n",[38,58,61],{"href":59,"rel":60},"https:\u002F\u002Fjs.wiki",[42],"Wiki.js"," for internal knowledge management, among others. Each\nof these tools stores data that is critical to how we work, which means losing\nit is not a hypothetical scenario but an existential risk. That reality made\nbuilding a robust, automated backup pipeline one of our top infrastructure\npriorities.",[64,65,67],"h2",{"id":66},"why-traditional-backup-strategies-failed","Why Traditional Backup Strategies Failed",[34,69,70],{},"When evaluating backup strategies for our infrastructure, we initially fell in\nto the same traps many growing engineering teams do. We relied on cloud-native\nconveniences and homegrown Bash scripts, only to hit a wall as our data\nfootprint and security requirements expanded.",[34,72,73],{},"Here is why we ultimately moved away from our legacy setup and standardised on\nRestic:",[75,76,77,97,112,118],"ol",{},[78,79,80,84,85,90,91,96],"li",{},[81,82,83],"strong",{},"The Cost and Lock-in of Native Cloud Snapshots",": Provider-managed volume\nsnapshots like (e.g.,\n",[38,86,89],{"href":87,"rel":88},"https:\u002F\u002Fdocs.vultr.com\u002Fproducts\u002Fstorage\u002Fsnapshots",[42],"Vultr Snapshots"," or\n",[38,92,95],{"href":93,"rel":94},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fvirtual-machines\u002Fsnapshot-copy-managed-disk",[42],"Azure managed disk snapshot",")\nare undeniably convenient for quick point-in-time recovery. However, they\nscale poorly, costs escalate rapidly as data grows and they locked us in to a\nsingle ecosystem, making cross-cloud or multi-region redundancy complex and\nexpensive to orchestrate natively.",[78,98,99,111],{},[81,100,101,102,106,107,110],{},"The Flaws of Basic ",[103,104,105],"code",{},"tar"," + ",[103,108,109],{},"rsync"," Scripts",": Custom scripts seem simple at\nfirst, but they quickly become maintenance nightmares. They lack native\ndeduplication, meaning we ended up uploading massive, redundant raw database\ndumps every single day, eating through bandwidth and storage. In the worst\ncase scenario, they forced us to manually handle encryption key rotations and\nintegrity verifications as well.",[78,113,114,117],{},[81,115,116],{},"Complications with the 3-2-1 Backup Strategy",": Adhering to the\n\"gold-standard 3-2-1 backup rule\" (3 copies of data, across 2 different media\ntypes, with 1 copy offsite) becomes an operational headache with fragmented\ntools. Trying to sync custom snapshots or raw archives across multiple\ndisparate cloud providers and local storage targets usually results in\nbrittle, custom-coded sync logic that is prone to silent failures.",[78,119,120,123,124,127],{},[81,121,122],{},"The Demands of Ransomware & Zero-Trust Requirements",": In a modern threat\nlandscape, a backup is only as good as its isolation. If a production server\nis compromised, any script of IAM role with read-write access to the storage\ncontainer can be used to wipe the backups right along with the application.\nWe needed a solution which enforced client-side encryption ",[81,125,126],{},"before"," the\ndata even touches the object storage, paired with an append-only repository\npermissions to prevent even a compromised root user from deleting historical\nsnapshots.",[34,129,130,131,136],{},"By addressing all of these pain points natively, ",[38,132,135],{"href":133,"rel":134},"https:\u002F\u002Frestic.net",[42],"Restic","\ngave us a predictable, secure and highly efficient backup pipeline.",[64,138,140],{"id":139},"why-we-chose-restic-instead","Why We Chose Restic Instead",[34,142,143],{},"We had mapped out our strict requirements and they were not a lot to ask for:",[145,146,147,150,153],"ul",{},[78,148,149],{},"Bulletproof security,",[78,151,152],{},"Efficiency at scale,",[78,154,155],{},"Adherence to modern redundancy standards",[34,157,158],{},"Based on those requirements, we needed a tool which would deliver on all fronts\nwithout adding operational overhead. So, here's why Restic became our go-to\nsolution:",[75,160,161,167,173],{},[78,162,163,166],{},[81,164,165],{},"Client-Side Encryption by Default",": Restic encrypts all data locally on\nthe host machine using robust AES-256 encryption before a single byte is ever\ntransmitted to a remote storage. Even if our object storage provider was\ncompromised, our data remains completely unreadable.",[78,168,169,172],{},[81,170,171],{},"Content-Defined Chunking (CDC) & Deduplication",": Restic uses smart\nchunking algorithms to split files in to dynamic blocks. This means duplicate\ndata across backup runs or even across multiple distinct server instances, is\nonly stored once. In practice, this optimisation cut our remote storage costs\ndown quite a lot.",[78,174,175,178,179,184,185,190],{},[81,176,177],{},"Backend Flexibility",": Deployment is remarkably straightforward thanks to a\nsingle, self-contained static binary. Whether we are pushing backups to a\nS3-compatible\n",[38,180,183],{"href":181,"rel":182},"https:\u002F\u002Fwww.vultr.com\u002Fproducts\u002Fobject-storage",[42],"Vultr Object Storage"," or\nperhaps an\n",[38,186,189],{"href":187,"rel":188},"https:\u002F\u002Fazure.microsoft.com\u002Fen-us\u002Fproducts\u002Fstorage\u002Fblobs",[42],"Azure Blob Storage","\ncontainer (which is our preferred choice), the configuration and workflow\nremains identical.",[34,192,193,194,197,198,201],{},"Restic transformed our backups in to an efficient, and cloud-agnostic pipeline.\nBut a great tool needs reliable execution as well, hence in the next section\nwe'll look in to how we replaced traditional ",[103,195,196],{},"cron"," jobs with ",[103,199,200],{},"systemd"," timers\nfor better logging and error control.",[64,203,205,206,208,209],{"id":204},"automation-architecture-systemd-timers-over-cron","Automation Architecture: ",[103,207,200],{}," Timers over ",[103,210,196],{},[34,212,213,214,216],{},"With Restic now standardised across our infrastructure, the next challenge was\nensuring every backup ran reliably and that we maintained redundant copies of\nour data — one stored in a cloud storage service and another transferred to a\nphysical offsite server via SFTP. Automating this dual-target pipeline required\nmoving beyond basic ",[103,215,196],{}," jobs in favour of something with stronger\nobservability and control. Here is how we built it:",[145,218,219,231,244],{},[78,220,221,222,197,224,226,227,230],{},"We replaced ",[103,223,196],{},[103,225,200],{}," timers since it gives us first-class\nlogging via ",[103,228,229],{},"journalctl",", built-in dependency management (guaranteeing that\nnetwork services are fully active before backup execution), and much cleaner\nfailure handling and alerting.",[78,232,233,234,237,238,243],{},"Rather than streaming dumps in real time, our pre- and post-backup hooks write\ntemporary database dumps (using ",[103,235,236],{},"pg_dump"," for our PostgreSQL database) that\nare encrypted and uploaded to Restic, then cleaned up once the backup\ncompletes. Since we deploy standardised scripts for this objective, failed\nbackup attempts are retried multiple times before escalating to a team\nnotification (usually using ",[38,239,242],{"href":240,"rel":241},"https:\u002F\u002Fntfy.sh",[42],"ntfy.sh","). Regardless of the\nbackup's execution state, these scripts always clean up the temporary dumps\nafterwards in an idempotent manner, ensuring no plaintext data lingers on\ndisk.",[78,245,246,247,264,267],{},"To prevent our repositories from bloating infinitely over time, we enforce a\nclean, grandfather-father-son retention policy paired with automatic cleanup:",[248,249,254],"pre",{"className":250,"code":251,"language":252,"meta":253,"style":253},"language-console shiki shiki-themes github-dark","restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune\n","console","",[103,255,256],{"__ignoreMap":253},[257,258,261],"span",{"class":259,"line":260},"line",1,[257,262,251],{"class":263},"sDLfK",[265,266],"br",{},"This retention policy is defined once in our standardised Restic configuration\nand applied uniformly across all repositories, so every backup target, whether\ncloud storage or the offsite SFTP server, all follows the same snapshot\nlifecycle. Combined with the automated cleanup in our standardised scripts,\nthis ensures no repository grows unbounded regardless of where it lives.",[34,269,270,271,273],{},"By combining ",[103,272,200],{},"'s robust orchestration with secure in-memory streaming\nand automated cleanup, our backup pipeline runs entirely hands-off while\nmaintaining strict security standards.",[64,275,277],{"id":276},"a-sample-implementation-for-reference","A Sample Implementation for Reference",[34,279,280,281,283,284,286],{},"For you reference, we are providing a simplified version of the standardised\nbackup script our ",[103,282,200],{}," service invokes, along with the service and timer\nunit files that schedule it. The full logic mirrors what we described above —\ndatabase dumping via ",[103,285,236],{},", Restic uploads with tags for filtering, offsite\nSFTP syncing, retention policy enforcement, and retry-based failure\nnotification.",[34,288,289,292,293,296],{},[81,290,291],{},"Backup script"," (",[103,294,295],{},"\u002Fusr\u002Flocal\u002Fbin\u002Fcreate-backup.sh","):",[248,298,302],{"className":299,"code":300,"language":301,"meta":253,"style":253},"language-bash shiki shiki-themes github-dark","#!\u002Fusr\u002Fbin\u002Fenv bash\n# ==============================================================================\n# Script Name: create-backup.sh\n# Description: Automates the backup of applications's PostgreSQL database and\n#              assets Docker volume, then uploads them using Restic with retry\n#              logic, dual-target sync (cloud + offsite SFTP), and failure\n#              notifications.\n# ==============================================================================\n\nset -euo pipefail\n\n# Configuration\nexport RESTIC_REPOSITORY=\"azure:container-name:\u002F\"\nexport RESTIC_PASSWORD_FILE=\"super-sensitive-password-which-should-be-secret\"\nexport AZURE_ACCOUNT_KEY=\"\"\nexport AZURE_ACCOUNT_NAME=\"\"\n\nOFFSITE_HOST=\"offsite.example.com\"\nOFFSITE_USER=\"backup\"\nOFFSITE_PATH=\"\u002Fbackups\u002Fapplication\"\nNOTIFY_URL=\"https:\u002F\u002Fntfy.sh\u002Fapplication-backups\"\n\nMAX_RETRIES=3\nRETRY_DELAY=30\n\nBACKUP_DIR=\"\u002Ftmp\u002Fapplication-backup-temp\"\nDB_DUMP_FILE=\"$BACKUP_DIR\u002Fapplication-db.sql\"\nASSETS_DIR=\"$BACKUP_DIR\u002Fassets\"\n\nnotify_failure() {\n  local attempt=\"$1\"\n  local message=\"$2\"\n  echo \"Backup failed on attempt ${attempt}: ${message}\"\n  curl -s -d \"application backup failed after ${attempt} attempt(s): ${message}\" \\\n    \"$NOTIFY_URL\" &>\u002Fdev\u002Fnull || true\n}\n\ncleanup() {\n  rm --recursive --force \"$BACKUP_DIR\" 2>\u002Fdev\u002Fnull || true\n}\ntrap cleanup EXIT\n\nmkdir --parents \"$ASSETS_DIR\"\n\ndump_and_upload() {\n  # Dump the PostgreSQL database to a temporary file\n  docker compose --project-name application \\\n    exec --no-tty application-postgres pg_dump --username=application --dbname=application |\n    tee \"$DB_DUMP_FILE\" >\u002Fdev\u002Fnull\n\n  if [ ! -s \"$DB_DUMP_FILE\" ]; then\n    echo \"Error: PostgreSQL dump failed or is empty.\"\n    return 1\n  fi\n\n  # Extract Docker assets\n  docker run --rm \\\n    --volume application_assets:\u002Fassets:ro \\\n    --volume \"$ASSETS_DIR\":\u002Fbackup \\\n    alpine cp --archive \u002Fassets\u002F. \u002Fbackup\u002F\n\n  # Upload both components to Restic\n  restic backup \\\n    --tag \"application-automated\" \\\n    \"$DB_DUMP_FILE\" \\\n    \"$ASSETS_DIR\"\n\n  # Mirror the latest snapshot to the offsite SFTP target\n  restic restore latest --target \u002Ftmp\u002Fapplication-restore-test --tag \"application-automated\"\n  rsync -az \u002Ftmp\u002Fapplication-restore-test\u002F \"${OFFSITE_USER}@${OFFSITE_HOST}:${OFFSITE_PATH}\u002F\"\n  rm --recursive --force \u002Ftmp\u002Fapplication-restore-test\n}\n\nprune_old() {\n  restic forget --tag \"application-automated\" --keep-daily 7 --keep-weekly 4 --prune\n}\n\nattempt=0\nwhile [ \"$attempt\" -lt \"$MAX_RETRIES\" ]; do\n  attempt=$((attempt + 1))\n  echo \"=== Backup attempt ${attempt} of ${MAX_RETRIES} ===\"\n\n  if dump_and_upload; then\n    prune_old\n    echo \"application backup completed successfully on attempt ${attempt}!\"\n    exit 0\n  fi\n\n  if [ \"$attempt\" -lt \"$MAX_RETRIES\" ]; then\n    echo \"Retrying in ${RETRY_DELAY} seconds...\"\n    sleep \"$RETRY_DELAY\"\n  fi\ndone\n\nnotify_failure \"$MAX_RETRIES\" \"All backup attempts exhausted.\"\nexit 1\n","bash",[103,303,304,310,316,322,328,334,340,346,351,358,371,376,382,399,412,425,437,442,453,464,475,486,491,502,513,518,529,546,561,566,576,595,612,633,661,687,693,698,706,736,741,753,758,774,779,787,793,810,834,853,858,884,893,902,908,913,919,932,943,957,975,980,986,997,1008,1019,1028,1033,1039,1062,1092,1104,1109,1114,1122,1149,1154,1159,1169,1199,1221,1239,1244,1257,1263,1276,1285,1290,1295,1320,1333,1346,1351,1357,1362,1376],{"__ignoreMap":253},[257,305,306],{"class":259,"line":260},[257,307,309],{"class":308},"sAwPA","#!\u002Fusr\u002Fbin\u002Fenv bash\n",[257,311,313],{"class":259,"line":312},2,[257,314,315],{"class":308},"# ==============================================================================\n",[257,317,319],{"class":259,"line":318},3,[257,320,321],{"class":308},"# Script Name: create-backup.sh\n",[257,323,325],{"class":259,"line":324},4,[257,326,327],{"class":308},"# Description: Automates the backup of applications's PostgreSQL database and\n",[257,329,331],{"class":259,"line":330},5,[257,332,333],{"class":308},"#              assets Docker volume, then uploads them using Restic with retry\n",[257,335,337],{"class":259,"line":336},6,[257,338,339],{"class":308},"#              logic, dual-target sync (cloud + offsite SFTP), and failure\n",[257,341,343],{"class":259,"line":342},7,[257,344,345],{"class":308},"#              notifications.\n",[257,347,349],{"class":259,"line":348},8,[257,350,315],{"class":308},[257,352,354],{"class":259,"line":353},9,[257,355,357],{"emptyLinePlaceholder":356},true,"\n",[257,359,361,364,367],{"class":259,"line":360},10,[257,362,363],{"class":263},"set",[257,365,366],{"class":263}," -euo",[257,368,370],{"class":369},"sU2Wk"," pipefail\n",[257,372,374],{"class":259,"line":373},11,[257,375,357],{"emptyLinePlaceholder":356},[257,377,379],{"class":259,"line":378},12,[257,380,381],{"class":308},"# Configuration\n",[257,383,385,389,393,396],{"class":259,"line":384},13,[257,386,388],{"class":387},"snl16","export",[257,390,392],{"class":391},"s95oV"," RESTIC_REPOSITORY",[257,394,395],{"class":387},"=",[257,397,398],{"class":369},"\"azure:container-name:\u002F\"\n",[257,400,402,404,407,409],{"class":259,"line":401},14,[257,403,388],{"class":387},[257,405,406],{"class":391}," RESTIC_PASSWORD_FILE",[257,408,395],{"class":387},[257,410,411],{"class":369},"\"super-sensitive-password-which-should-be-secret\"\n",[257,413,415,417,420,422],{"class":259,"line":414},15,[257,416,388],{"class":387},[257,418,419],{"class":391}," AZURE_ACCOUNT_KEY",[257,421,395],{"class":387},[257,423,424],{"class":369},"\"\"\n",[257,426,428,430,433,435],{"class":259,"line":427},16,[257,429,388],{"class":387},[257,431,432],{"class":391}," AZURE_ACCOUNT_NAME",[257,434,395],{"class":387},[257,436,424],{"class":369},[257,438,440],{"class":259,"line":439},17,[257,441,357],{"emptyLinePlaceholder":356},[257,443,445,448,450],{"class":259,"line":444},18,[257,446,447],{"class":391},"OFFSITE_HOST",[257,449,395],{"class":387},[257,451,452],{"class":369},"\"offsite.example.com\"\n",[257,454,456,459,461],{"class":259,"line":455},19,[257,457,458],{"class":391},"OFFSITE_USER",[257,460,395],{"class":387},[257,462,463],{"class":369},"\"backup\"\n",[257,465,467,470,472],{"class":259,"line":466},20,[257,468,469],{"class":391},"OFFSITE_PATH",[257,471,395],{"class":387},[257,473,474],{"class":369},"\"\u002Fbackups\u002Fapplication\"\n",[257,476,478,481,483],{"class":259,"line":477},21,[257,479,480],{"class":391},"NOTIFY_URL",[257,482,395],{"class":387},[257,484,485],{"class":369},"\"https:\u002F\u002Fntfy.sh\u002Fapplication-backups\"\n",[257,487,489],{"class":259,"line":488},22,[257,490,357],{"emptyLinePlaceholder":356},[257,492,494,497,499],{"class":259,"line":493},23,[257,495,496],{"class":391},"MAX_RETRIES",[257,498,395],{"class":387},[257,500,501],{"class":369},"3\n",[257,503,505,508,510],{"class":259,"line":504},24,[257,506,507],{"class":391},"RETRY_DELAY",[257,509,395],{"class":387},[257,511,512],{"class":369},"30\n",[257,514,516],{"class":259,"line":515},25,[257,517,357],{"emptyLinePlaceholder":356},[257,519,521,524,526],{"class":259,"line":520},26,[257,522,523],{"class":391},"BACKUP_DIR",[257,525,395],{"class":387},[257,527,528],{"class":369},"\"\u002Ftmp\u002Fapplication-backup-temp\"\n",[257,530,532,535,537,540,543],{"class":259,"line":531},27,[257,533,534],{"class":391},"DB_DUMP_FILE",[257,536,395],{"class":387},[257,538,539],{"class":369},"\"",[257,541,542],{"class":391},"$BACKUP_DIR",[257,544,545],{"class":369},"\u002Fapplication-db.sql\"\n",[257,547,549,552,554,556,558],{"class":259,"line":548},28,[257,550,551],{"class":391},"ASSETS_DIR",[257,553,395],{"class":387},[257,555,539],{"class":369},[257,557,542],{"class":391},[257,559,560],{"class":369},"\u002Fassets\"\n",[257,562,564],{"class":259,"line":563},29,[257,565,357],{"emptyLinePlaceholder":356},[257,567,569,573],{"class":259,"line":568},30,[257,570,572],{"class":571},"svObZ","notify_failure",[257,574,575],{"class":391},"() {\n",[257,577,579,582,585,587,589,592],{"class":259,"line":578},31,[257,580,581],{"class":387},"  local",[257,583,584],{"class":391}," attempt",[257,586,395],{"class":387},[257,588,539],{"class":369},[257,590,591],{"class":263},"$1",[257,593,594],{"class":369},"\"\n",[257,596,598,600,603,605,607,610],{"class":259,"line":597},32,[257,599,581],{"class":387},[257,601,602],{"class":391}," message",[257,604,395],{"class":387},[257,606,539],{"class":369},[257,608,609],{"class":263},"$2",[257,611,594],{"class":369},[257,613,615,618,621,624,627,630],{"class":259,"line":614},33,[257,616,617],{"class":263},"  echo",[257,619,620],{"class":369}," \"Backup failed on attempt ${",[257,622,623],{"class":391},"attempt",[257,625,626],{"class":369},"}: ${",[257,628,629],{"class":391},"message",[257,631,632],{"class":369},"}\"\n",[257,634,636,639,642,645,648,650,653,655,658],{"class":259,"line":635},34,[257,637,638],{"class":571},"  curl",[257,640,641],{"class":263}," -s",[257,643,644],{"class":263}," -d",[257,646,647],{"class":369}," \"application backup failed after ${",[257,649,623],{"class":391},[257,651,652],{"class":369},"} attempt(s): ${",[257,654,629],{"class":391},[257,656,657],{"class":369},"}\"",[257,659,660],{"class":263}," \\\n",[257,662,664,667,670,672,675,678,681,684],{"class":259,"line":663},35,[257,665,666],{"class":369},"    \"",[257,668,669],{"class":391},"$NOTIFY_URL",[257,671,539],{"class":369},[257,673,674],{"class":391}," &",[257,676,677],{"class":387},">",[257,679,680],{"class":391},"\u002Fdev\u002Fnull ",[257,682,683],{"class":387},"||",[257,685,686],{"class":263}," true\n",[257,688,690],{"class":259,"line":689},36,[257,691,692],{"class":391},"}\n",[257,694,696],{"class":259,"line":695},37,[257,697,357],{"emptyLinePlaceholder":356},[257,699,701,704],{"class":259,"line":700},38,[257,702,703],{"class":571},"cleanup",[257,705,575],{"class":391},[257,707,709,712,715,718,721,723,725,728,731,734],{"class":259,"line":708},39,[257,710,711],{"class":571},"  rm",[257,713,714],{"class":263}," --recursive",[257,716,717],{"class":263}," --force",[257,719,720],{"class":369}," \"",[257,722,542],{"class":391},[257,724,539],{"class":369},[257,726,727],{"class":387}," 2>",[257,729,730],{"class":369},"\u002Fdev\u002Fnull",[257,732,733],{"class":387}," ||",[257,735,686],{"class":263},[257,737,739],{"class":259,"line":738},40,[257,740,692],{"class":391},[257,742,744,747,750],{"class":259,"line":743},41,[257,745,746],{"class":263},"trap",[257,748,749],{"class":369}," cleanup",[257,751,752],{"class":369}," EXIT\n",[257,754,756],{"class":259,"line":755},42,[257,757,357],{"emptyLinePlaceholder":356},[257,759,761,764,767,769,772],{"class":259,"line":760},43,[257,762,763],{"class":571},"mkdir",[257,765,766],{"class":263}," --parents",[257,768,720],{"class":369},[257,770,771],{"class":391},"$ASSETS_DIR",[257,773,594],{"class":369},[257,775,777],{"class":259,"line":776},44,[257,778,357],{"emptyLinePlaceholder":356},[257,780,782,785],{"class":259,"line":781},45,[257,783,784],{"class":571},"dump_and_upload",[257,786,575],{"class":391},[257,788,790],{"class":259,"line":789},46,[257,791,792],{"class":308},"  # Dump the PostgreSQL database to a temporary file\n",[257,794,796,799,802,805,808],{"class":259,"line":795},47,[257,797,798],{"class":571},"  docker",[257,800,801],{"class":369}," compose",[257,803,804],{"class":263}," --project-name",[257,806,807],{"class":369}," application",[257,809,660],{"class":263},[257,811,813,816,819,822,825,828,831],{"class":259,"line":812},48,[257,814,815],{"class":369},"    exec",[257,817,818],{"class":263}," --no-tty",[257,820,821],{"class":369}," application-postgres",[257,823,824],{"class":369}," pg_dump",[257,826,827],{"class":263}," --username=application",[257,829,830],{"class":263}," --dbname=application",[257,832,833],{"class":387}," |\n",[257,835,837,840,842,845,847,850],{"class":259,"line":836},49,[257,838,839],{"class":571},"    tee",[257,841,720],{"class":369},[257,843,844],{"class":391},"$DB_DUMP_FILE",[257,846,539],{"class":369},[257,848,849],{"class":387}," >",[257,851,852],{"class":369},"\u002Fdev\u002Fnull\n",[257,854,856],{"class":259,"line":855},50,[257,857,357],{"emptyLinePlaceholder":356},[257,859,861,864,867,870,872,874,876,878,881],{"class":259,"line":860},51,[257,862,863],{"class":387},"  if",[257,865,866],{"class":391}," [ ",[257,868,869],{"class":387},"!",[257,871,641],{"class":387},[257,873,720],{"class":369},[257,875,844],{"class":391},[257,877,539],{"class":369},[257,879,880],{"class":391}," ]; ",[257,882,883],{"class":387},"then\n",[257,885,887,890],{"class":259,"line":886},52,[257,888,889],{"class":263},"    echo",[257,891,892],{"class":369}," \"Error: PostgreSQL dump failed or is empty.\"\n",[257,894,896,899],{"class":259,"line":895},53,[257,897,898],{"class":387},"    return",[257,900,901],{"class":263}," 1\n",[257,903,905],{"class":259,"line":904},54,[257,906,907],{"class":387},"  fi\n",[257,909,911],{"class":259,"line":910},55,[257,912,357],{"emptyLinePlaceholder":356},[257,914,916],{"class":259,"line":915},56,[257,917,918],{"class":308},"  # Extract Docker assets\n",[257,920,922,924,927,930],{"class":259,"line":921},57,[257,923,798],{"class":571},[257,925,926],{"class":369}," run",[257,928,929],{"class":263}," --rm",[257,931,660],{"class":263},[257,933,935,938,941],{"class":259,"line":934},58,[257,936,937],{"class":263},"    --volume",[257,939,940],{"class":369}," application_assets:\u002Fassets:ro",[257,942,660],{"class":263},[257,944,946,948,950,952,955],{"class":259,"line":945},59,[257,947,937],{"class":263},[257,949,720],{"class":369},[257,951,771],{"class":391},[257,953,954],{"class":369},"\":\u002Fbackup",[257,956,660],{"class":263},[257,958,960,963,966,969,972],{"class":259,"line":959},60,[257,961,962],{"class":369},"    alpine",[257,964,965],{"class":369}," cp",[257,967,968],{"class":263}," --archive",[257,970,971],{"class":369}," \u002Fassets\u002F.",[257,973,974],{"class":369}," \u002Fbackup\u002F\n",[257,976,978],{"class":259,"line":977},61,[257,979,357],{"emptyLinePlaceholder":356},[257,981,983],{"class":259,"line":982},62,[257,984,985],{"class":308},"  # Upload both components to Restic\n",[257,987,989,992,995],{"class":259,"line":988},63,[257,990,991],{"class":571},"  restic",[257,993,994],{"class":369}," backup",[257,996,660],{"class":263},[257,998,1000,1003,1006],{"class":259,"line":999},64,[257,1001,1002],{"class":263},"    --tag",[257,1004,1005],{"class":369}," \"application-automated\"",[257,1007,660],{"class":263},[257,1009,1011,1013,1015,1017],{"class":259,"line":1010},65,[257,1012,666],{"class":369},[257,1014,844],{"class":391},[257,1016,539],{"class":369},[257,1018,660],{"class":263},[257,1020,1022,1024,1026],{"class":259,"line":1021},66,[257,1023,666],{"class":369},[257,1025,771],{"class":391},[257,1027,594],{"class":369},[257,1029,1031],{"class":259,"line":1030},67,[257,1032,357],{"emptyLinePlaceholder":356},[257,1034,1036],{"class":259,"line":1035},68,[257,1037,1038],{"class":308},"  # Mirror the latest snapshot to the offsite SFTP target\n",[257,1040,1042,1044,1047,1050,1053,1056,1059],{"class":259,"line":1041},69,[257,1043,991],{"class":571},[257,1045,1046],{"class":369}," restore",[257,1048,1049],{"class":369}," latest",[257,1051,1052],{"class":263}," --target",[257,1054,1055],{"class":369}," \u002Ftmp\u002Fapplication-restore-test",[257,1057,1058],{"class":263}," --tag",[257,1060,1061],{"class":369}," \"application-automated\"\n",[257,1063,1065,1068,1071,1074,1077,1079,1082,1084,1087,1089],{"class":259,"line":1064},70,[257,1066,1067],{"class":571},"  rsync",[257,1069,1070],{"class":263}," -az",[257,1072,1073],{"class":369}," \u002Ftmp\u002Fapplication-restore-test\u002F",[257,1075,1076],{"class":369}," \"${",[257,1078,458],{"class":391},[257,1080,1081],{"class":369},"}@${",[257,1083,447],{"class":391},[257,1085,1086],{"class":369},"}:${",[257,1088,469],{"class":391},[257,1090,1091],{"class":369},"}\u002F\"\n",[257,1093,1095,1097,1099,1101],{"class":259,"line":1094},71,[257,1096,711],{"class":571},[257,1098,714],{"class":263},[257,1100,717],{"class":263},[257,1102,1103],{"class":369}," \u002Ftmp\u002Fapplication-restore-test\n",[257,1105,1107],{"class":259,"line":1106},72,[257,1108,692],{"class":391},[257,1110,1112],{"class":259,"line":1111},73,[257,1113,357],{"emptyLinePlaceholder":356},[257,1115,1117,1120],{"class":259,"line":1116},74,[257,1118,1119],{"class":571},"prune_old",[257,1121,575],{"class":391},[257,1123,1125,1127,1130,1132,1134,1137,1140,1143,1146],{"class":259,"line":1124},75,[257,1126,991],{"class":571},[257,1128,1129],{"class":369}," forget",[257,1131,1058],{"class":263},[257,1133,1005],{"class":369},[257,1135,1136],{"class":263}," --keep-daily",[257,1138,1139],{"class":263}," 7",[257,1141,1142],{"class":263}," --keep-weekly",[257,1144,1145],{"class":263}," 4",[257,1147,1148],{"class":263}," --prune\n",[257,1150,1152],{"class":259,"line":1151},76,[257,1153,692],{"class":391},[257,1155,1157],{"class":259,"line":1156},77,[257,1158,357],{"emptyLinePlaceholder":356},[257,1160,1162,1164,1166],{"class":259,"line":1161},78,[257,1163,623],{"class":391},[257,1165,395],{"class":387},[257,1167,1168],{"class":369},"0\n",[257,1170,1172,1175,1177,1179,1182,1184,1187,1189,1192,1194,1196],{"class":259,"line":1171},79,[257,1173,1174],{"class":387},"while",[257,1176,866],{"class":391},[257,1178,539],{"class":369},[257,1180,1181],{"class":391},"$attempt",[257,1183,539],{"class":369},[257,1185,1186],{"class":387}," -lt",[257,1188,720],{"class":369},[257,1190,1191],{"class":391},"$MAX_RETRIES",[257,1193,539],{"class":369},[257,1195,880],{"class":391},[257,1197,1198],{"class":387},"do\n",[257,1200,1202,1205,1207,1210,1212,1215,1218],{"class":259,"line":1201},80,[257,1203,1204],{"class":391},"  attempt",[257,1206,395],{"class":387},[257,1208,1209],{"class":391},"$((",[257,1211,623],{"class":571},[257,1213,1214],{"class":369}," +",[257,1216,1217],{"class":263}," 1",[257,1219,1220],{"class":391},"))\n",[257,1222,1224,1226,1229,1231,1234,1236],{"class":259,"line":1223},81,[257,1225,617],{"class":263},[257,1227,1228],{"class":369}," \"=== Backup attempt ${",[257,1230,623],{"class":391},[257,1232,1233],{"class":369},"} of ${",[257,1235,496],{"class":391},[257,1237,1238],{"class":369},"} ===\"\n",[257,1240,1242],{"class":259,"line":1241},82,[257,1243,357],{"emptyLinePlaceholder":356},[257,1245,1247,1249,1252,1255],{"class":259,"line":1246},83,[257,1248,863],{"class":387},[257,1250,1251],{"class":571}," dump_and_upload",[257,1253,1254],{"class":391},"; ",[257,1256,883],{"class":387},[257,1258,1260],{"class":259,"line":1259},84,[257,1261,1262],{"class":571},"    prune_old\n",[257,1264,1266,1268,1271,1273],{"class":259,"line":1265},85,[257,1267,889],{"class":263},[257,1269,1270],{"class":369}," \"application backup completed successfully on attempt ${",[257,1272,623],{"class":391},[257,1274,1275],{"class":369},"}!\"\n",[257,1277,1279,1282],{"class":259,"line":1278},86,[257,1280,1281],{"class":263},"    exit",[257,1283,1284],{"class":263}," 0\n",[257,1286,1288],{"class":259,"line":1287},87,[257,1289,907],{"class":387},[257,1291,1293],{"class":259,"line":1292},88,[257,1294,357],{"emptyLinePlaceholder":356},[257,1296,1298,1300,1302,1304,1306,1308,1310,1312,1314,1316,1318],{"class":259,"line":1297},89,[257,1299,863],{"class":387},[257,1301,866],{"class":391},[257,1303,539],{"class":369},[257,1305,1181],{"class":391},[257,1307,539],{"class":369},[257,1309,1186],{"class":387},[257,1311,720],{"class":369},[257,1313,1191],{"class":391},[257,1315,539],{"class":369},[257,1317,880],{"class":391},[257,1319,883],{"class":387},[257,1321,1323,1325,1328,1330],{"class":259,"line":1322},90,[257,1324,889],{"class":263},[257,1326,1327],{"class":369}," \"Retrying in ${",[257,1329,507],{"class":391},[257,1331,1332],{"class":369},"} seconds...\"\n",[257,1334,1336,1339,1341,1344],{"class":259,"line":1335},91,[257,1337,1338],{"class":571},"    sleep",[257,1340,720],{"class":369},[257,1342,1343],{"class":391},"$RETRY_DELAY",[257,1345,594],{"class":369},[257,1347,1349],{"class":259,"line":1348},92,[257,1350,907],{"class":387},[257,1352,1354],{"class":259,"line":1353},93,[257,1355,1356],{"class":387},"done\n",[257,1358,1360],{"class":259,"line":1359},94,[257,1361,357],{"emptyLinePlaceholder":356},[257,1363,1365,1367,1369,1371,1373],{"class":259,"line":1364},95,[257,1366,572],{"class":571},[257,1368,720],{"class":369},[257,1370,1191],{"class":391},[257,1372,539],{"class":369},[257,1374,1375],{"class":369}," \"All backup attempts exhausted.\"\n",[257,1377,1379,1382],{"class":259,"line":1378},96,[257,1380,1381],{"class":263},"exit",[257,1383,901],{"class":263},[34,1385,1386,292,1391,296],{},[81,1387,1388,1390],{},[103,1389,200],{}," service",[103,1392,1393],{},"\u002Fetc\u002Fsystemd\u002Fsystem\u002Fapplication-backup.service",[248,1395,1398],{"className":1396,"code":1397,"language":200,"meta":253,"style":253},"language-systemd shiki shiki-themes github-dark","[Unit]\nDescription=Application Automated Backup\nRequires=network-online.target\nAfter=network-online.target docker.service\nWants=network-online.target\n\n[Service]\nType=oneshot\nExecStart=\u002Fusr\u002Flocal\u002Fbin\u002Fapplication-backup.sh\nStandardOutput=journal\nStandardError=journal\nRestart=no\n\n[Install]\nWantedBy=multi-user.target\n",[103,1399,1400,1405,1416,1426,1436,1445,1449,1454,1464,1474,1484,1493,1503,1507,1512],{"__ignoreMap":253},[257,1401,1402],{"class":259,"line":260},[257,1403,1404],{"class":571},"[Unit]\n",[257,1406,1407,1411,1413],{"class":259,"line":312},[257,1408,1410],{"class":1409},"s4JwU","Description",[257,1412,395],{"class":387},[257,1414,1415],{"class":391},"Application Automated Backup\n",[257,1417,1418,1421,1423],{"class":259,"line":318},[257,1419,1420],{"class":1409},"Requires",[257,1422,395],{"class":387},[257,1424,1425],{"class":391},"network-online.target\n",[257,1427,1428,1431,1433],{"class":259,"line":324},[257,1429,1430],{"class":1409},"After",[257,1432,395],{"class":387},[257,1434,1435],{"class":391},"network-online.target docker.service\n",[257,1437,1438,1441,1443],{"class":259,"line":330},[257,1439,1440],{"class":1409},"Wants",[257,1442,395],{"class":387},[257,1444,1425],{"class":391},[257,1446,1447],{"class":259,"line":336},[257,1448,357],{"emptyLinePlaceholder":356},[257,1450,1451],{"class":259,"line":342},[257,1452,1453],{"class":571},"[Service]\n",[257,1455,1456,1459,1461],{"class":259,"line":348},[257,1457,1458],{"class":1409},"Type",[257,1460,395],{"class":387},[257,1462,1463],{"class":263},"oneshot\n",[257,1465,1466,1469,1471],{"class":259,"line":353},[257,1467,1468],{"class":1409},"ExecStart",[257,1470,395],{"class":387},[257,1472,1473],{"class":391},"\u002Fusr\u002Flocal\u002Fbin\u002Fapplication-backup.sh\n",[257,1475,1476,1479,1481],{"class":259,"line":360},[257,1477,1478],{"class":1409},"StandardOutput",[257,1480,395],{"class":387},[257,1482,1483],{"class":391},"journal\n",[257,1485,1486,1489,1491],{"class":259,"line":373},[257,1487,1488],{"class":1409},"StandardError",[257,1490,395],{"class":387},[257,1492,1483],{"class":391},[257,1494,1495,1498,1500],{"class":259,"line":378},[257,1496,1497],{"class":1409},"Restart",[257,1499,395],{"class":387},[257,1501,1502],{"class":263},"no\n",[257,1504,1505],{"class":259,"line":384},[257,1506,357],{"emptyLinePlaceholder":356},[257,1508,1509],{"class":259,"line":401},[257,1510,1511],{"class":571},"[Install]\n",[257,1513,1514,1517,1519],{"class":259,"line":414},[257,1515,1516],{"class":1409},"WantedBy",[257,1518,395],{"class":387},[257,1520,1521],{"class":391},"multi-user.target\n",[34,1523,1524,292,1529,296],{},[81,1525,1526,1528],{},[103,1527,200],{}," timer",[103,1530,1531],{},"\u002Fetc\u002Fsystemd\u002Fsystem\u002Fapplication-backup.timer",[248,1533,1535],{"className":1396,"code":1534,"language":200,"meta":253,"style":253},"[Unit]\nDescription=Run application backup daily at 02:00\n\n[Timer]\nOnCalendar=daily\nPersistent=true\nRandomizedDelaySec=15m\n\n[Install]\nWantedBy=timers.target\n",[103,1536,1537,1541,1556,1560,1565,1575,1585,1595,1599,1603],{"__ignoreMap":253},[257,1538,1539],{"class":259,"line":260},[257,1540,1404],{"class":571},[257,1542,1543,1545,1547,1550,1553],{"class":259,"line":312},[257,1544,1410],{"class":1409},[257,1546,395],{"class":387},[257,1548,1549],{"class":391},"Run application backup daily at ",[257,1551,1552],{"class":263},"02",[257,1554,1555],{"class":391},":00\n",[257,1557,1558],{"class":259,"line":318},[257,1559,357],{"emptyLinePlaceholder":356},[257,1561,1562],{"class":259,"line":324},[257,1563,1564],{"class":571},"[Timer]\n",[257,1566,1567,1570,1572],{"class":259,"line":330},[257,1568,1569],{"class":1409},"OnCalendar",[257,1571,395],{"class":387},[257,1573,1574],{"class":263},"daily\n",[257,1576,1577,1580,1582],{"class":259,"line":336},[257,1578,1579],{"class":1409},"Persistent",[257,1581,395],{"class":387},[257,1583,1584],{"class":263},"true\n",[257,1586,1587,1590,1592],{"class":259,"line":342},[257,1588,1589],{"class":1409},"RandomizedDelaySec",[257,1591,395],{"class":387},[257,1593,1594],{"class":263},"15m\n",[257,1596,1597],{"class":259,"line":348},[257,1598,357],{"emptyLinePlaceholder":356},[257,1600,1601],{"class":259,"line":353},[257,1602,1511],{"class":571},[257,1604,1605,1607,1609],{"class":259,"line":360},[257,1606,1516],{"class":1409},[257,1608,395],{"class":387},[257,1610,1611],{"class":391},"timers.target\n",[34,1613,1614,1615,1618,1619,1622,1623,1625],{},"With these files in place, enabling and starting the timer\n(",[103,1616,1617],{},"systemctl enable --now application-backup.timer",") schedules the backup to run\ndaily. The service captures full output in\n",[103,1620,1621],{},"journalctl -xeu application-backup.service --no-pager | less +G",", making\ntroubleshooting straightforward without relying on ",[103,1624,196],{},"'s limited mailing\ncapabilities.",[64,1627,1629],{"id":1628},"hardening-backup-security-integrity","Hardening Backup Security & Integrity",[34,1631,1632],{},"Even with automated scheduling and smooth streaming, a robust production backup\nstrategy must account for worst-case scenarios like compromised infrastructure\nand silent data corruption. Here is how we lock down our backups in Azure:",[145,1634,1635,1657],{},[78,1636,1637,1640,1641,1644,1645,1648,1649,1652,1653,1656],{},[81,1638,1639],{},"Least-Privilege Azure RBAC",": Instead of giving servers full control over\nour Azure Blob Storage, we restrict our backup service principal to write and\nread permissions\n(",[103,1642,1643],{},"Microsoft.Storage\u002FstorageAccounts\u002FblobServices\u002Fcontainers\u002Fblobs\u002Fread",",\n",[103,1646,1647],{},"write",", ",[103,1650,1651],{},"add",", and ",[103,1654,1655],{},"list","). We strip out delete permissions entirely so that\na compromised application server cannot wipe its own backups. Pruning and\nsnapshot expiration are handled exclusively by a separate, isolated\nmaintenance worker.",[78,1658,1659,1662,1663,1666],{},[81,1660,1661],{},"Automated Integrity Verification",": A backup which cannot be restored is\njust expensive garbage. To prevent silent data corruption (bit rot) or\nincomplete uploads from going unnoticed, we run scheduled ",[103,1664,1665],{},"restic check"," jobs\nto periodically scan the repository index, verify chunk checksums and ensure\nour recovery chain remain pristine.",[34,1668,1669],{},"By combining Azure's append-only guardrails with routine consistency checks, our\nbackups remain both tamper-proof and verified. However, locking down the\nrepository is only half the battle, we also need to know immediately if a backup\nfails and verify that we can actually recover from it.",[34,1671,1672],{},"To ensure our backups are healthy and recoverable, we run periodic manual\nrestoration exercises on a schedule (usually once or twice a year). This not\nonly provides us the confidence in our backup pipeline(s) but also provides\nvaluable experience to our engineering teams for disaster management and\nrecovery drills.",[34,1674,1675],{},"That said, we hope this article provided you with some knowledge and insight in\nto our infrastructure's backup management workflow. Since we're continuoulsy\nexperimenting and evolving our backup pipelines, we will keep this piece of\narticle updated as often as we can.",[1677,1678,1679],"style",{},"html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s4JwU, html code.shiki .s4JwU{--shiki-default:#85E89D}",{"title":253,"searchDepth":312,"depth":312,"links":1681},[1682,1683,1684,1686,1687],{"id":66,"depth":312,"text":67},{"id":139,"depth":312,"text":140},{"id":204,"depth":312,"text":1685},"Automation Architecture: systemd Timers over cron",{"id":276,"depth":312,"text":277},{"id":1628,"depth":312,"text":1629},"Infrastructure","\u002Fblog\u002Fautomated-restic-backups.webp","2026-09-29","Discover how our engineering team automated zero-trust backups using Restic, systemd, and Azure Blob Storage client-side encryption, deduplication, and fast restores.","md",{},"\u002Fblog\u002Fautomated-restic-backups",{"title":28,"description":1691},"blog\u002Fautomated-restic-backups","X-mLn96oz6FXiTyyz752_ZyTcVog9X59LD09ZQy0NAA",[7,1699],{"title":1700,"path":1701,"stem":1702,"description":1703,"date":1704,"children":-1},"Golden Images at Scale: Building Secure, Multi-Cloud VMs with Packer","\u002Fblog\u002Fgolden-images-with-packer","blog\u002Fgolden-images-with-packer","Learn how to build secure, immutable Debian 13 golden images for Microsoft Azure and Vultr simultaneously using HashiCorp Packer, Infrastructure-as-Code, and CI\u002FCD automation.","2026-09-11",[1706,1707,1711,1715],{"path":1701,"title":1700,"date":1704,"category":1688},{"path":1708,"title":1709,"date":1710,"category":1688},"\u002Fblog\u002Fmanaging-infrastructure-drift","Managing Infrastructure Drift: How Packer, Terraform, and Ansible Keep Enterprise Environments Compliant","2026-09-03",{"path":1712,"title":1713,"date":1714,"category":1688},"\u002Fblog\u002Fsecurely-self-hosting-postgresql","Securely Self-Hosting PostgreSQL: Configuration, Backups, and Best Practices","2026-08-10",{"path":1716,"title":1717,"date":1718,"category":1719},"\u002Fblog\u002Fhello-world","Hello, world","2026-05-22","People",1790685474706]