Disaster Recovery
VirtFusion 2.3+ supports DR (backup/restore) to minimize downtime and data loss in the event of a disaster.
Data can either be stored in S3 compatible object storage or on a local disk partition.
For a list of S3 compatible providers, see here.
This may look complicated, but it really isn't. VirtFusion will try it's very best to produce consistent backups that can be restored easily. You just need to configure it on the hypervisor the way you want.
Although disaster recovery is supported on hypervisors that utilize shared storage, it will only create backups of locally stored disks and configurations. All shared storage (i.e Ceph) disks will be ignored.
It's expected that later versions of VirtFusion will incorporate a GUI component for disaster recovery.
Hypervisor Preparation
S3
You only need to follow this step if you want to use object storage.
Installing the AWS CLI tools
To install the CLI tool, unzip is required on the hypervisor. It should be installed using your distro package
manager. i.e apt install unzip or dnf install unzip.
Once you have unzip, install the tool.
- x86_64
- AArch64
Run the following commands as a root SSH user.
cd /tmp
curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.22.35.zip -o awscliv2.zip
unzip awscliv2.zip
./aws/install
rm -rf /tmp/aws
rm -f /tmp/awscliv2.zip
Run the following commands as a root SSH user.
cd /tmp
curl https://awscli.amazonaws.com/awscli-exe-linux-aarch64-2.22.35.zip -o awscliv2.zip
unzip awscliv2.zip
./aws/install
rm -rf /tmp/aws
rm -f /tmp/awscliv2.zip
The official AWS instructions can be found here.
Configuring the AWS config profile
Create the file /root/.aws/config.
mkdir -p /root/.aws
nano -w /root/.aws/config
In that file you can configure your access credentials for your S3 compatible service.
[profile virtfusion]
aws_access_key_id=ACCESS_KEY
aws_secret_access_key=SECRET_ACCESS_KEY
retry_mode = standard
max_attempts = 3
s3 =
#max_concurrent_requests = 20
#max_bandwidth = 5MB/s
addressing_style = path
You should leave the profile name as virtfusion. If you would like to explore various other options, you can find them at https://docs.aws.amazon.com/cli/latest/topic/s3-config.html
Local Storage
Nothing extra needs to be configured for local storage.
Configuring VirtFusion
Create the file /home/vf-data/conf/dr.json.
nano -w /home/vf-data/conf/dr.json
In that file you can configure your S3 bucket, endpoint and region or/and your local storage.
{
"type": "S3",
"storeDirType": "once",
"tmpDir": "home/vf-data/tmp",
"threads": 1,
"S3": {
"endpoint": "https://s3.eu-west-1.wasabisys.com",
"region": "eu-west-1",
"bucket": "backups",
"compress": true,
"compressType": "lz4",
"compressLevel": 1,
"compressThreads": 1,
"prune": {
"enabled": true,
"hoursGrace": 168,
"keepLast": 30
}
},
"local": {
"path": "backups",
"compress": true,
"compressType": "lz4",
"compressLevel": 1,
"compressThreads": 1,
"prune": {
"enabled": true,
"hoursGrace": 168,
"keepLast": 30
}
}
}
typeshould be eitherS3orlocalstoreDirTypemay be one of the following:
| Type | Example Path | Description |
|---|---|---|
once | virtfusion_dr/[HV_ID]/once | The once type will create a static directory named once, meaning each time a backup runs, the files will be overwritten. |
Y-M-D-H | virtfusion_dr/[HV_ID]/y_m_d_h/[2023_06_01_04] | Typically this will create a new directory every hour. |
Y-M-D | virtfusion_dr/[HV_ID]/y_m_d/[2023_06_01] | This will create a new directory each day. |
Y-M | virtfusion_dr/[HV_ID]/y_m/[2023_06] | This will create a new directory each month. |
Y-W | virtfusion_dr/[HV_ID]/y_w/[2023_03] | This will create a directory weekly (Weeks starting on Monday). |
DW | virtfusion_dr/[HV_ID]/dw/[1-7] | This will create a directory for the day number of the week (1 for Monday, 7 for Sunday). |
All date style types are based on UTC.
-
tmpDiris where temporary snapshots will be stored. You don't need to include a/at the start or end. -
threadshow many backups can be running at any one time. The option is currently ignored and only 1 thread will be used until a future version. -
S3.endpointshould be the URL to your S3 service. -
S3.regionis the S3 region of your S3 service. -
S3.bucketshould be the name of the bucket you created to store the backups. -
S3.compresswill enable compression on server disks. Enabling compression will usually slow down the backup. -
S3.compressTypedefault islz4. You can choose betweenlz4andzstd. process, especially on a fully loaded system, but saves storage space. It's advised to keep this option enabled. -
S3.compressLevel(1 to 19) 1 = fast (default), 19 = slow. Be warned - 19 will be better compression, but it will be very very very very slow! It's advised to keep this option set to 1. -
s3.compressThreads(0 to 20). default = 1. 0 = core count. (only zstd). -
S3.prune.enabledcan be true or false. This option will enable or disabled backup pruning. -
S3.prune.hoursGraceis only used for the oncestoreDirTypesetting and will wait the specified number of hours before deleting files that no longer belong to the hypervisor. -
S3.prune.keepLastis used for all date stylestoreDirTypesettings and is used to specify how many of the newest folders should be kept on storage. -
local.pathshould be the local path where you want to store the backups. You don't need to include a/at the start or end. -
local.compresswill enable compression on server disks. Enabling compression will usually slow down the backup. process, especially on a fully loaded system, but saves storage space. It's advised to keep this option enabled. -
local.compressTypedefault islz4. You can choose betweenlz4andzstd. -
local.compressLevel(1 to 19) 1 = fast (default), 19 = slow. Be warned - 19 will be better compression, but it will be very very very very slow! It's advised to keep this option set to 1. -
local.compressThreads(0 to 20). default = 1. 0 = core count. (only zstd). -
local.prune.enabledcan be true or false. This option will enable or disabled backup pruning. -
local.prune.hoursGraceis only used for the oncestoreDirTypesetting and will wait the specified number of hours before deleting files that no longer belong to the hypervisor. -
local.prune.keepLastis used for all date stylestoreDirTypesettings and is used to specify how many of the newest folders should be kept on storage.
It's probably wise to backup both /root/.aws/config and /home/vf-data/conf/dr.json for easy access in case of a
fatal hypervisor failure.
Creating Backups
- If you run a backup manually, it's highly advised to run the command within a screen session.
- Even though signals are caught and the DR routine will attempt to exit cleanly, resist the urge to ctrl+c when a backup is running. Due to the nature of external snapshots, this may cause inconsistency on disk, which can be hard to repair.
- When a backup runs on a server disk, some of the server actions like reboot, reinstall, shutdown etc... will be blocked at hypervisor level. Once the backup process has been completed on that disk, the actions will be unlocked.
Generate a list of files that will be included in a backup (with filesize estimates)
vfcli-hv dr:backup --estimate-only
It will generate something similar to this.
+---------------------------------------------------------------+---------+-----------+
| Item | Type | Size |
+---------------------------------------------------------------+---------+-----------+
| /home/vf-data/disk/37fc81d8-402c-41a1-8f7b-5f421c7e2d20_1.img | file | 3.81 GB |
| /home/vf-data/disk/d33ab991-ec0e-491b-b38a-0999f01bbfc9_1.img | file | 6.76 GB |
| /home/vf-data/disk/6ba9b79c-0168-4988-ba6e-b7674c7c2c08_1.img | file | 1.8 GB |
| /home/vf-data/disk/6ba9b79c-0168-4988-ba6e-b7674c7c2c08_2.img | file | 6.69 MB |
| /home/vf-data/disk/6ba9b79c-0168-4988-ba6e-b7674c7c2c08_3.img | file | 6.69 MB |
| /home/vf-data/server | dir | 1.11 MB |
| /home/vf-data/stats | dir | 109.85 MB |
| /home/vf-data/events | dir | 0 B |
| /home/vf-data/conf | dir | 802 B |
| /opt/virtfusion/app/hypervisor/storage/stats | dir | 3.04 KB |
| /opt/virtfusion/app/hypervisor/storage/logs | dir | 314.9 KB |
| /opt/virtfusion/app/hypervisor/storage/nat | dir | 136 B |
| /opt/virtfusion/app/hypervisor/conf/auth.json | file | 339 B |
| /etc/haproxy/haproxy.cfg | file | 1.35 KB |
| /opt/virtfusion/app/hypervisor/database/database.sqlite | file | 12 KB |
| /opt/virtfusion/app/hypervisor/database/queue.sqlite | file | 16 KB |
|---------------------------------------------------------------|---------|-----------|
| Total (approx) | 12.49 GB |
+---------------------------------------------------------------+---------+-----------+
Backup everything
vfcli-hv dr:backup
Backup only specific servers
vfcli-hv dr:backup --only-servers=1754,1253,1002
Backup everything except disks that already exist on the backup storage
vfcli-hv dr:backup --only-missing-disks