tl-dr.no

Introduction

These notes should not be to long, so i actually read them.

The current plan is to orgainze destinct projects or topics in their own "chapters" (since this tool is ment to create books). And sub-chapters for smaller parts, like a pice of the project, or an attempt at a solution, that did not work out.

Heimdall

My personal server is named Heimdall. It is a NixOs machine running services in containers in podman, configured as docker-compose files.

The OS is installed on an nvme drive, and the data is stored on a zfs pool, consisting of a few shucked drives.

NixOs notes

NixOs has given me an fast route to reinstall my server, and therfore lowering the time it takes to get back up and running after a failure.

Also the flow of using NixOs, "change the config, rebuild, test, commit" is a nice way to work with a server, as its quick to see the changes work, and easy to revert if something goes wrong.

Backups from portable devices

graph TD;
    Ntfy["ntfy.sh - notification service"]
    subgraph Tailscale network
        A[PC]     --> T
        C[phone]  --> T
        subgraph Home server
            subgraph Docker/Podman
                T[Traefik]  --> N[Syncthing]
            end
            B[Borg Backup nix service]
            N --> Bkp[Backup directory]
            Bkp --> B
            S>Sanoid snapshots] --> Bkp
        end
    end
    F[friends server]
    B --> R[Borgbase.net]
    R --if backups fail for
        24h, notify phone--> Ntfy
    Ntfy --> C
    Bkp --zfs send -w, via syncoid--> F
    
    click Ntfy "https://ntfy.sh"
    click R "https://borgbase.com"
    click T "https://traefik.io"
    click B "https://nixos.org/manual/nixos/stable/#module-borgbase"
    click N "https://syncthing.net"
    click S "https://github.com/jimsalterjrs/sanoid"
ℹ️zfs send, and sanoid are a work in progress

in addition to backing up regularly to Borgbase with the nix service, the ZFS dataset that contains the backup directory can use sanoid to snapshot regularly for local retention.

BorgBackup

points to write about

  • borg backup cannot backup to a repo that is currently mounted (e.g. fuse mount to get files from an old backup)
  • nix borgbackup service, and point out the doInit function.
  • notes on the append only key on borgbase and the drawbacks of borg serverside pruning. (timed attack can delete all backups)
  • keyfiles and how to use them with borgbackup, and that all keys have a passphrase, that may be null.

stuff learn about

  • secrets management with nix.
  • secrets management with podman. (is an env file ok? maybe use bitwarden?)

The borgbackup nix service

the options are well documented at search.nix.org, in the nix manual, and the wiki

an example:

  services.borgbackup.jobs."test-borgbase-backup" = {
    paths = [ 
      "test_backup/Camera"
      "test_backup/Music"
      "test_backup/Sync"
      "whatever..."
    ];
    startAt = "hourly";
    repo = "ssh://[email protected]/./repo";
    user = "user_name"; # a user with access to the files to backup, and the keys
    doInit = false;     # The remote repo on borgbase was already initialized.
    environment = {
      BORG_RSH = "ssh -i /keys/id_ed25519";
      BORG_KEY_FILE = "/keys/borgbase-borg-key"; # keyfile "exported" from the repo. 
                                                    # (it actually lives in ~/.config/borg/keys)
    };
    encryption = {
      mode = "keyfile";
      passCommand = "echo ''"; # The keyfile has no passphrase. (press enter when prompted)
    };
    prune.keep = {
      within = "1d"; # Keep all archives from the last day
      daily = 7;     # Keep daily archives for a week
      weekly = 4;    # Keep weekly archives for a month
      monthly = 6;   # Keep monthly archives for half a year
    };
  };

Attack recovery (Ransomware)

In the scenario where the home server has ransomware, and zfs snapshots cannot recover my data. The offsite Borgbase backup can be used to recover data. (hopefully verry unlikeley scenario)

The main steps are:

  1. Turn off borgbase server side compacting (if enabled), from a know safe PC (live disk?).
    This gives time to recover data before any data is actually deleted on the server.
  2. Clean up malware.
  3. Enable sftp on borgbase repo, and copy repo onto local system. (safer than editing repo directly.)
  4. Follow documentation to revert any delet transactions by attacker.
  5. Mount, and copy data from borg repo back to filesystem.

Manipulating transaction history.

This is following the steps from the Borg documentation, using examples from my own simulated test of this scenario.

you may want to start by getting repo info with borg info [repo-locaton]. Here the Repository ID may come in handy later, if your client has connected to multiple repos.

use less [repo-location]/transactions and look for transaction at odd times, or from knownn attack time:

transaction 1438   UTC time 2024-11-17T06:00:29.025851
transaction 1442   UTC time 2024-11-17T06:00:31.680865
transaction 1446   UTC time 2024-11-17T06:00:40.691154
transaction 1449   UTC time 2024-11-17T06:00:42.949990
transaction 1453   UTC time 2024-11-17T07:00:05.117055
transaction 1457   UTC time 2024-11-17T07:00:07.928139
transaction 1461   UTC time 2024-11-17T07:00:09.421660
transaction 1464   UTC time 2024-11-17T07:00:10.688233
transaction 1468   UTC time 2024-11-17T08:00:21.924977
transaction 1472   UTC time 2024-11-17T08:00:23.522328
transaction 1476   UTC time 2024-11-17T08:00:27.810536
transaction 1479   UTC time 2024-11-17T08:00:29.436371
transaction 1483   UTC time 2024-11-17T09:00:01.786366
transaction 1487   UTC time 2024-11-17T09:00:03.305158
transaction 1490   UTC time 2024-11-17T09:00:05.578356
transaction 1494   UTC time 2024-11-17T10:09:34.123150  # odd time, possible attack.
transaction 1498   UTC time 2024-11-17T10:11:08.537320  # odd time, possible attack.
transaction 1502   UTC time 2024-11-17T10:11:41.568902  # odd time, possible attack.
ℹ️this is a good time to take a zfs snapshot of the borg repo. in case of woopsie

Now manipulate the repository to delete the odd transactions:

for i in (seq 1491 1502)
    rm -r [repo-location]/data/**/$i 
end

The repo will now be in a state that the borg tool will not be happy with. So any command like borg list or borg info and such will fail. You will now need to clear the repo cache:

borg delete --cache-only [repo-location]

And then delete the timestamp from the borg client local config folder (repo-ID from the borg info you ran at the start):

rm ~/.config/borg/security/[repo-ID]/manifest-timestamp

Syncthing

syncthing service need to run as a user with access to the files you want to sync, and the same user as borgbackup.

resources and notes while investigating backup solutions

Server side sanitization (off-site)

To protect against an attacker compromizing the on-site backup server, and then the remote/off-site backup target, it is preferable that snapshots can only be deleted by the remote/offsite backup target. Configuration of this schedule should only be done with a strong login, that is seperate from the login used by the on-site backup server. Typically via a hardware key like a Fido2 key, either via a webauthn login or ssh-sk key.

So far only zfs seems to be capable of this due to the non encrypted metadata (snapshots are visible to the off-site target).

The only targets that support this natively are rsync.net and zfs.rent. (or vps hosting). Rsync.net is not the cheapest, and zfs.rent does not seem to be accepting signups.

only tool for ssh

MagmaSoft Software Wiki/ blog/ posts/ The Only Way For SSH Forced Commands

only is a bash script for restricting what commands a user can run over ssh. (source)

BorgBackup

BorgBackup

Currently BorgBackup is the best candidate for remote backup, due to its deduplication and encryption features, and due to the amount of services that support it, and the pricing of the remote options.

The one downside is the recovery steps in case of compromise of the on-site backup server, even with append-only mode. see: attach recovery.

Restic

TL;DR pros:

  • supports cheap S3 remote backups
  • can be run on may systems due to Golang's compilation support. cons:
  • cannot be configured to set server to handle snapshot sanitization.

Supports nearly all the same backup targets as Borg, in addition to S3 backend.

although according to the threat model it does not cover the case of a comopromise of the on-site backup server, just like Borg.

https://ruderich.org/simon/notes/append-only-backups-with-restic-and-rclone https://restic.readthedocs.io/en/stable/045_working_with_repos.html https://github.com/restic/restic/blob/60245970280c6e6442cc57777bd570cb49f66448/doc/design.rst#threat-model https://github.com/borgbackup/borg/issues/3504 https://github.com/borgbackup/borg/pull/4384 https://github.com/borgbackup/borg/issues/3579 https://search.nixos.org/options?channel=24.05&size=50&sort=relevance&type=packages&query=services.borgbackup.jobs.%3Cname%3E. https://nixos.org/manual/nixos/stable/#module-borgbase https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode-forbid-compaction https://wiki.nixos.org/wiki/Borg_backup https://borgwarehouse.com/docs/admin-manual/docker-installation/ https://stackoverflow.com/questions/33713680/ssh-force-command-execution-on-login-even-without-shell#33714333 https://at.magma-soft.at/sw/blog/posts/The_Only_Way_For_SSH_Forced_Commands/ https://docs.ntfy.sh/ https://docs.borgbase.com/setup/borg/ https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/ https://docs.oracle.com/cd/E19253-01/819-5461/gamnq/index.html https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ https://openzfs.github.io/openzfs-docs/man/master/8/zfs-send.8.html

Setup a user to allow a friend to use your server for backups

You and a friend want to "swap" some space on each others home servers, to use eachother as a backup target. But you do not want your friend's user to be able to do anything else then administer their backups (in case they get compromized etc.). This document outlines stategies to set up an environment to allow a friend to push new backups, list their backups, and recover from their backups using NixOs, and ZFS. (friends user is not intended to be able to delete their own backups, as a security measure for them. An attack on their end should not compromize their backup on your server.)

Resources

Restricted user chat on 2.5 Admins podcast @ 16m05s.
ZFS deligate dataset to user chat on 2.5 Admins podcast @ 22m25s.

Providers that allow ZFS send/recive:

  • rsync.net
  • zfs.rent