# VirtFusion Docs > Documentation for VirtFusion, virtualization management software for the hosting industry. This file contains all documentation content in a single document following the llmstxt.org standard. ## Admin IP Restrict It's possible to restrict admin area access to a specific set of IP addresses. Administration of this feature is only available via CLI on the control server. ### Adding an IP address You can add as many addresses as you like. From CLI run the following command: ```shell vfcli-ctrl admin:ip-restrict addip IP_TO_ADD ``` Replace `IP_TO_ADD` with the address you would like to lock the admin area too. for example: ```shell vfcli-ctrl admin:ip-restrict addip 10.0.100.200 ``` You can also allow the whole range in the subnet by shortening the address: ```shell vfcli-ctrl admin:ip-restrict addip 10.0.100. ``` This functionality also applies to IPv6. ### Deleting an IP address From CLI run the following command: ```shell vfcli-ctrl admin:ip-restrict delip IP_TO_DELETE ``` Replace `IP_TO_DELETE` with the address you would like to delete. for example: ```shell vfcli-ctrl admin:ip-restrict delip 10.0.100.200 ``` ### Enable IP Restrict Once you have added the IP addresses using the commands above, you will need to enable restrictions. You can do that by running the following command: ```shell vfcli-ctrl admin:ip-restrict enable ``` ### Disable IP Restrict If you need to turn restrictions off for any reason, run the following command: ```shell vfcli-ctrl admin:ip-restrict disable ``` --- ## Backup Manager The Backup Manager provides automated and on-demand backups for virtual servers. It supports incremental, differential, and full backups with multiple storage backends, retention policies, and end-user self-service. ### Overview Backups are taken at the block level while the virtual server continues running, with minimal performance impact. Intelligent consistency detection ensures backups are application-safe without requiring downtime. The system is designed to be reliable and gentle on busy hypervisors. Backup concurrency, I/O priority, CPU priority, bandwidth limits, and disk read limits are all configurable per storage or per schedule. Resource controls ensure backups run in the background without impacting VM performance, even on heavily loaded hypervisors. Backups can be stored on SSH/SFTP servers, S3-compatible object storage (AWS, MinIO, Wasabi, R2, B2), FTP/FTPS servers, or any suitable provider supported by rclone. There is also a tight native integration with [Proxmox Backup Server](https://www.proxmox.com/en/proxmox-backup-server/overview), providing chunk-level deduplication and efficient storage without requiring any additional software on your hypervisors. #### Supported Storage Backends | Backend | Incremental | Encryption | Deduplication | |:--------|:-----------:|:----------:|:-------------:| | SSH/SFTP | | | | | S3-compatible (AWS, MinIO, Wasabi, R2, B2) | | | | | Proxmox Backup Server (PBS) | | | | | FTP/FTPS | | | | | Rclone | | | | :::info Local filesystem storage is supported by the backup engine but is not currently available in VirtFusion. Backups stored locally on the hypervisor do not provide protection against hardware failure and cannot be accessed from other hypervisors, limiting the ability to restore VMs after a migration or hardware loss. A remote storage backend is strongly recommended. ::: #### Backup Types | Type | Description | |:-----|:------------| | Full | Complete disk image. Forms the baseline for an incremental chain. | | Incremental | Only blocks changed since the last backup. Smallest and fastest. | | Differential | All blocks changed since the last full backup. | | Copy | Standalone full backup without affecting the incremental chain. | The system automatically manages backup chains. After a configurable interval (default: 7 days), a new full backup is taken to start a fresh chain. #### Supported Disk Types | Disk Format | Incremental | Notes | |:------------|:-----------:|:------| | qcow2 | | Recommended. Persistent dirty bitmaps survive VM reboots and live migrations. | | Raw (file-backed) | | Dirty bitmaps are non-persistent. A VM reboot or crash will reset the bitmap, causing the next backup to fall back to a full backup. Incremental chains work correctly as long as the VM remains running between backups. | | Ceph RBD | | Persistent change tracking. Survives reboots and migrations. | :::warning **Raw disk limitation:** If a VM using raw disk images is rebooted between backups, the incremental tracking bitmap is lost. The next scheduled backup will automatically detect this and perform a full backup to start a fresh chain. This is a QEMU limitation with raw format disks. If your VMs reboot frequently, consider using qcow2 format for more efficient incremental backups. ::: :::info **StorPool:** StorPool block storage is not currently supported. We are actively working with StorPool to find a resolution. ::: #### Access Levels Each virtual server has a backup access level that controls what operations are permitted: | Level | Included in Scheduled Backups | End User Can View | End User Can Restore | End User Can Backup | |:------|:---:|:---:|:---:|:---:| | Disabled | | | | | | Scheduled Only | | | | | | View & Restore | | | | | | Full Access | | | | | | Manual Only | | | | | Access levels are resolved in order: per-server override, then schedule default. If no override is set, the schedule's default applies. --- ### Getting Started Setting up backups requires three steps: 1. Add a storage backend 2. Create a schedule linking a hypervisor to that storage 3. Backups run automatically on the configured interval #### Step 1: Add Storage Navigate to **Backup Manager → Storage** and click **Add Storage**. ![Add Storage modal](img/backup-manager/add-storage.png) A modal will appear where you select a storage type and enter a name. After saving, you will be taken to the storage configuration page. #### Step 2: Configure Storage Each storage type has its own connection settings. Configure the connection details for your chosen backend. ![Storage configuration page](img/backup-manager/storage-manage.png) ##### SSH/SFTP | Field | Description | |:------|:------------| | Host | Hostname or IP address of the SSH server | | Username | SSH username | | Port | SSH port (default: 22) | | Path | Absolute path on the remote server for backup storage | | Private Key | PEM-formatted private key for authentication | ###### SSH Server Configuration For reliable backups and restores, especially for large disks that transfer for several hours, the remote SSH server should be configured to maintain long-lived connections. Add or verify the following settings in `/etc/ssh/sshd_config` on the storage server: ``` # Send keepalive probes every 60 seconds to detect dead clients ClientAliveInterval 60 # Allow 3 missed keepalives before disconnecting (3 minutes tolerance) ClientAliveCountMax 3 # Allow enough sessions for concurrent backup operations MaxSessions 20 ``` After editing, restart sshd: `systemctl restart sshd` If you are using a cloud firewall (e.g. Hetzner Cloud Firewall, AWS Security Groups with connection tracking), be aware that stateful firewalls have connection tracking timeouts that can silently drop long-lived TCP connections. If backups or restores fail consistently after several hours, try disabling the cloud firewall as a test, or ensure the connection tracking timeout exceeds your longest expected transfer time. You should also verify that no system-level TCP timeouts are too aggressive: ```bash # Check connection tracking timeout (if conntrack is active) sysctl net.netfilter.nf_conntrack_tcp_timeout_established # Should be at least 14400 (4 hours), default is 432000 (5 days) ``` ##### S3-Compatible Object Storage | Field | Description | |:------|:------------| | Bucket | S3 bucket name | | Region | AWS region (e.g. `us-east-1`) | | Endpoint | Custom endpoint URL for non-AWS providers (MinIO, Wasabi, R2, B2) | | Access Key | S3 access key ID | | Secret Key | S3 secret access key | | Key Prefix | Optional path prefix within the bucket | | Concurrency | Number of parallel upload threads (default: 8) | | Part Size (MB) | Minimum S3 multipart upload part size (default: auto, 10 MB) | Backups are uploaded to S3 using multipart uploads. The part size is automatically calculated based on disk size, targeting approximately 2,000 parts per upload. For most deployments the default is fine. If your S3 provider drops connections or times out during large uploads, increase the part size to reduce the number of parts and total upload time. This is most commonly needed with smaller or less mature S3-compatible providers. ##### Proxmox Backup Server VirtFusion integrates natively with Proxmox Backup Server using the PBS protocol directly. No `proxmox-backup-client` installation is required on your hypervisors. | Field | Description | |:------|:------------| | Host | PBS server hostname or IP | | Port | API port (default: 8007) | | Username | PBS username | | Realm | Authentication realm (e.g. `pbs`, `pam`) | | Token Name | API token name | | Token Secret | API token secret (UUID format) | | Datastore | PBS datastore name | | Namespace | Optional PBS namespace | | TLS Fingerprint | SHA-256 fingerprint for certificate pinning (optional) | | Skip TLS Verify | Bypass TLS certificate validation | | Skip Zero Chunks | Skip unallocated regions in full backups (reduces storage usage) | **Setting up PBS access:** 1. **Create an API token**: In the PBS web UI, go to **Configuration → Access Control → API Token**. Click **Add** and create a token for your backup user. Note the **Token Name** and **Token Secret** (the secret is only shown once). 2. **Datastore**: Go to **Administration → Storage / Disks** to see your available datastores. Enter the datastore name exactly as shown. 3. **Namespace** (optional): If you use PBS namespaces to organise backups, enter the namespace path (e.g. `virtfusion` or `dc1/virtfusion`). Leave blank to use the datastore root. 4. **TLS Fingerprint**: In the PBS web UI, the SHA-256 fingerprint is shown on the **Dashboard** in the top-right corner. Copy the full fingerprint string. This is recommended when the PBS server uses a self-signed certificate. 5. **Permissions**: The API token needs at minimum **DatastoreBackup** and **DatastoreReader** privileges on the target datastore/namespace. For full details, see the [Proxmox Backup Server documentation](https://pbs.proxmox.com/docs/). ###### PBS Encryption PBS supports client-side encryption where backup data is encrypted before being sent to the server. The PBS server never sees unencrypted data. To set up encryption: 1. **Generate an encryption key** by running the following on any Linux machine: ```bash echo '{"kdf":"none","data":"'$(openssl rand -base64 32)'"}' ``` This produces a JSON key like: ```json {"kdf":"none","data":"xY7f3kLm9QzB1vN4hR8wJ2pA5sT0uE6gC3iO7dF9aXk="} ``` Alternatively, if you have `proxmox-backup-client` installed, you can use: ```bash proxmox-backup-client key create --kdf none /tmp/backup-key.json cat /tmp/backup-key.json ``` 2. **Paste the JSON key** into the **Encryption Key** field in the PBS section of the storage configuration. 3. **Enable encryption** - toggle **Enable Encryption** on. 4. **Key Password** - leave blank when using a key generated with the commands above. This field is only needed if the key was created with password protection (e.g. `proxmox-backup-client key create` without `--kdf none`). :::warning Store your encryption key and password securely. If the key is lost, encrypted backups cannot be restored. Encryption settings cannot be changed after backups have been created - changing the key will make existing backups unrestorable. ::: :::info Enabling encryption disables PBS server-side deduplication. Each backup will consume its full size on the PBS server rather than only storing changed chunks. Consider whether encryption or deduplication is more important for your use case. ::: ##### FTP/FTPS | Field | Description | |:------|:------------| | Host | FTP server hostname or IP | | Port | FTP port (default: 21) | | Username | FTP username | | Password | FTP password | | Remote Path | Directory path on the FTP server | | TLS Max Version | TLS 1.2 (for vsFTPd compatibility) or TLS 1.3 | | Disable TLS | Use plain FTP (not recommended) | | Skip TLS Verify | Bypass TLS certificate validation | ##### Rclone | Field | Description | |:------|:------------| | Remote Name | Name of the configured rclone remote | | Remote Path | Path on the remote | | S3 Concurrency | Parallel upload threads for S3-compatible remotes | :::info The rclone configuration file must be placed at `/opt/virtfusion/rclone/rclone.conf` on each hypervisor. Configure your rclone remote on the hypervisor first using `rclone config`, then copy the resulting config to this path: ```bash mkdir -p /opt/virtfusion/rclone cp ~/.config/rclone/rclone.conf /opt/virtfusion/rclone/rclone.conf chown -R virtfusion:virtfusion /opt/virtfusion/rclone ``` The Backup Manager will use this file automatically. ::: Rclone supports a wide range of providers. The following are recommended for backup storage: | Provider | Rclone Type | Notes | |:---------|:------------|:------| | Any SFTP server | `sftp` | Excellent performance. The simplest and most reliable option. | | Backblaze B2 | `b2` | Cost-effective cloud storage with free egress to Cloudflare. | | Google Cloud Storage | `gcs` | Nearline/Coldline tiers suit backup workloads. | | Azure Blob Storage | `azureblob` | Cool/Archive tiers available for long-term retention. | | Hetzner Storage Box | `sftp` | Affordable European storage. Use SFTP (port 23), not WebDAV. | | OpenStack Swift | `swift` | Common in hosting environments (OVH, IONOS, etc). Files over 5 GiB use automatic chunking. | | Storj | `storj` or `s3` | Decentralised storage with erasure coding. | :::warning **MEGA** and **Dropbox** are not suitable for backup storage. MEGA does not support streaming uploads (the entire backup file is spooled to local disk before uploading). Dropbox is unreliable for large file uploads due to rate limiting and transfer stalls. ::: :::info For S3-compatible providers (AWS, MinIO, Wasabi, Cloudflare R2, Backblaze B2), the native **S3 storage backend** is recommended over rclone as it provides better performance with direct streaming uploads. ::: ##### Storage Status Each storage backend has two independent toggles: **Enabled** and **Read Only**. | State | Backups | Restores | Scheduled Batches | |:------|:-------:|:--------:|:-----------------:| | Enabled | | | | | Enabled + Read Only | | | | | Disabled | | | | - **Enabled**: normal operation. All backup and restore operations work. - **Enabled + Read Only**: no new backups are written, but restores from existing backups continue to work. Useful when decommissioning a storage backend or during maintenance. - **Disabled**: all operations are blocked. Neither backups nor restores can be performed. #### Common Storage Settings These settings apply to all storage types. ##### Network | Setting | Description | |:--------|:------------| | Bandwidth Limit | Maximum upload speed (unlimited by default) | | Read Limit | Maximum disk read speed (unlimited by default) | ##### Encryption :::warning Encryption settings cannot be changed after backups have been created. Changing the passphrase will make existing backups unrestorable. ::: | Setting | Description | |:--------|:------------| | Enable Encryption | Encrypt backup data with authenticated encryption | | Passphrase | Encryption passphrase. Store this securely as it cannot be recovered | For PBS storage, encryption is configured in the PBS-specific section using PBS-native encryption. ##### Performance | Setting | Description | Default | |:--------|:------------|:-------:| | Worker Threads | Number of parallel processing threads | 4 | | Backup Concurrency | Maximum concurrent VM backups per hypervisor | 1 | | I/O Priority | Disk I/O scheduling priority (Normal, Low, Idle) | Normal | | CPU Priority | Process CPU priority (Normal, Low, Idle) | Normal | | Compress | Enable zstd compression | | | Compression Level | zstd level 1 (fastest) to 19 (best ratio) | 3 | | Compression Threads | Threads for compression | 1 | :::info **Backup Concurrency** controls how many VMs are backed up simultaneously on a single hypervisor. Higher values complete batches faster but increase disk I/O load. Start with 1 and increase based on your storage performance. ::: ##### Backup Behaviour | Setting | Description | Default | |:--------|:------------|:--------| | Consistency Mode | How the VM is quiesced during backup | Freeze + Pause Fallback | | Full Backup Interval | Days between forced full backups (not applicable for PBS) | 7 | | Retention Policy | Retention strategy - age-based (keep all within N days) or GFS (calendar-based thinning) | None | | Keep Last | Always keep at least N most recent backups | 2 | :::info **PBS storage differences:** - **Full Backup Interval** is ignored. PBS snapshots are independently restorable regardless of type, so forced full backups are unnecessary. - **Skip Zero Chunks** (PBS-only setting) skips unallocated disk regions during full backups, significantly reducing storage usage for sparsely allocated disks. - **Keep Last** protects the N most recent snapshots per disk. For other backends, it protects the N most recent backup chains per disk. ::: **Consistency Modes:** | Mode | Description | |:-----|:------------| | Freeze + Pause Fallback | Try guest agent filesystem freeze first; if unavailable, briefly pause the VM | | Freeze Only | Only use guest agent freeze (fails if agent unavailable) | | Pause Only | Briefly pause the VM (no guest agent needed) | | None | No quiescing (crash-consistent only) | :::info **Freeze + Pause Fallback** is recommended for most deployments. It provides the best consistency when the guest agent is installed, and falls back to a brief pause (typically under 1 second) when it is not. Loop devices and Windows Reserved partitions are automatically excluded from the freeze. ::: **Retention Policies:** Two retention models are available: **age-based** and **GFS (Grandfather-Father-Son)**. Both can be combined with the **Keep Last** floor. **Age-Based Retention** Simple time-window retention: all backups within the window are kept, anything older is pruned. | Preset | Keep Within | |:-------|:-----------| | None | No auto-prune | | 1 Week | 7 days | | 2 Weeks | 14 days | | 3 Weeks | 21 days | | 4 Weeks | 28 days | | 2 Months | 60 days | | 3 Months | 90 days | | 6 Months | 180 days | | 9 Months | 270 days | | 1 Year | 365 days | | 2 Years | 730 days | | 3 Years | 1095 days | | 5 Years | 1825 days | | 7 Years | 2555 days | | 10 Years | 3650 days | For example, the "3 Months" preset keeps all backups from the last 90 days. **GFS (Grandfather-Father-Son) Retention** GFS retention uses calendar-based buckets to thin out older backups while preserving recent granularity. Instead of keeping *every* backup within a window, GFS keeps one backup per calendar period: - **Daily** - one backup per calendar day (the newest backup taken that day) - **Weekly** - one backup per calendar week (ISO week, the newest backup taken that week) - **Monthly** - one backup per calendar month (the newest backup taken that month) - **Yearly** - one backup per calendar year (the newest backup taken that year) Each bucket type keeps the N most recent buckets. A backup is kept if **any** bucket rule selects it (rules are a union, not a chain). **Example: GFS 1 Year (7 daily, 4 weekly, 12 monthly)** With daily backups, after several months you would have: - The last **7 days** of backups (one per day) - for recent recovery - The last **4 weeks** of weekly backups - for recent rollback - The last **12 months** of monthly backups - for long-term retention Older daily and weekly backups are pruned, but one backup per month is preserved. This gives you fine granularity for recent recovery and coarse granularity for older points, using far less storage than keeping every backup for a year. **GFS Presets:** | Preset | Daily | Weekly | Monthly | Yearly | |:-------|:-----:|:------:|:-------:|:------:| | GFS 1 Month | 7 | 4 | - | - | | GFS 2 Months | 7 | 4 | 2 | - | | GFS 3 Months | 7 | 4 | 3 | - | | GFS 6 Months | 7 | 4 | 6 | - | | GFS 9 Months | 7 | 4 | 9 | - | | GFS 1 Year | 7 | 4 | 12 | - | | GFS 2 Years | 7 | 4 | 12 | 2 | | GFS 3 Years | 7 | 4 | 12 | 3 | | GFS 5 Years | 7 | 4 | 12 | 5 | | GFS 7 Years | 7 | 4 | 12 | 7 | | GFS 10 Years | 7 | 4 | 12 | 10 | **Dense GFS Presets** (more daily/weekly granularity for environments with frequent backups): | Preset | Daily | Weekly | Monthly | Yearly | |:-------|:-----:|:------:|:-------:|:------:| | GFS 1 Month Dense | 14 | 4 | - | - | | GFS 2 Months Dense | 14 | 8 | 2 | - | | GFS 3 Months Dense | 14 | 8 | 3 | - | | GFS 6 Months Dense | 14 | 8 | 6 | - | | GFS 9 Months Dense | 14 | 8 | 9 | - | | GFS 1 Year Dense | 14 | 8 | 12 | - | | GFS 2 Years Dense | 14 | 8 | 12 | 2 | | GFS 3 Years Dense | 30 | 12 | 12 | 3 | | GFS 5 Years Dense | 30 | 12 | 12 | 5 | :::info **Choosing between age-based and GFS:** - **Age-based** is simpler - every backup within the window is kept. Best when you want guaranteed day-level granularity for the entire window and storage space is not a concern. - **GFS** is more storage-efficient for longer retention periods. A "GFS 1 Year" policy might keep ~23 backups (7 daily + 4 weekly + 12 monthly) instead of ~365 with an age-based "1 Year" policy. - **GFS works best with regular backup schedules.** If you back up daily, each daily bucket has one backup. If you back up weekly, most daily buckets will be empty - in that case, age-based retention is simpler and equally effective. ::: #### Test the Connection After saving, click **Save Settings & Test**. Select a hypervisor from the dropdown (one that will be assigned to this storage via a schedule) and click **Test**. This verifies the hypervisor can connect to the storage backend and write data. ![Test storage modal](img/backup-manager/test-storage.png) :::warning Always test the connection from at least one hypervisor before creating a schedule. A misconfigured storage will cause all backups to fail. ::: --- #### Step 3: Create a Schedule Navigate to **Backup Manager → Schedules** and click **Add Schedule**. ![Add Schedule modal](img/backup-manager/add-schedule.png) | Field | Description | |:------|:------------| | Storage | The storage backend to use | | Hypervisor | The hypervisor whose VMs will be backed up | | Interval | Hours between backup runs (e.g. 24 for daily). Set to 0 for manual-only | | Status | Enabled or Disabled | | Default Server Access | Default access level for VMs on this schedule | :::info Each schedule pairs one hypervisor with one storage backend. Each hypervisor can only have one schedule. If you have 3 hypervisors backing up to 1 storage, create 3 schedules. ::: After creating the schedule, backups will begin automatically at the configured interval. The first backup for each VM is always a full backup. ##### Cloning a Schedule To quickly apply the same configuration across multiple hypervisors, use the **Clone** option from a schedule's dropdown menu. Select the target hypervisors and click **Clone** - a new schedule is created for each one with identical settings (storage, interval, retention, resource controls, etc.). Hypervisors that already have a schedule are excluded automatically. ##### Schedule Overrides Click the schedule row to expand it and see the resolved settings. Each setting shows whether it inherits from the storage default or has been overridden. Click the **edit icon** to override any setting for this specific schedule: - Interval, consistency mode, full backup interval - Retention policy, keep last - Backup concurrency, worker threads - Bandwidth and read limits - I/O and CPU priority - Encryption (enable/disable/inherit) - Bind address (source IP for outbound connections, not supported for S3 storage) - Time window (restrict backups to specific hours) Overridden settings show an orange badge. Inherited settings show a grey badge with the resolved value. ![Schedule expanded with overrides](img/backup-manager/schedule-overrides.png) ##### Time Windows You can restrict a schedule to only run backups during specific hours. Set the **Start** and **End** times in the schedule edit modal. If the interval comes due outside the window, the backup waits until the next window opens. ##### Allowed Days Restrict a schedule to specific days of the week. Click the day buttons in the schedule edit modal to toggle individual days on or off. When all days are selected (or none are toggled), the schedule runs any day. Common examples: - **Weekdays only** - Mon, Tue, Wed, Thu, Fri - **Weekends only** - Sat, Sun - **Alternate days** - Mon, Wed, Fri Allowed days work alongside time windows. For example, a schedule set to Mon/Wed/Fri with a 05:00-07:00 window will only run during those hours on those three days. The **Next Runs Preview** at the bottom of the edit modal shows the combined effect of the interval, time window, and allowed days so you can verify the schedule before saving. ##### Bind Address If your hypervisors have multiple network interfaces (e.g. a public interface and a dedicated backup network), you can set a **Bind Address** to force all backup traffic through a specific IP address. Enter the local IP address on the hypervisor that should be used for connections to the storage backend. This applies to all operations - backups, restores, purges, and health checks. | Backend | Bind Address Support | |:--------|:--------------------:| | SSH/SFTP | | | Proxmox Backup Server | | | Rclone | | | FTP/FTPS | | | S3 | | :::warning **S3 and FTP do not support bind address.** For S3, use an rclone remote with an S3 backend instead (rclone supports bind), or configure OS-level routing as shown below. ::: If your storage backend does not support bind address, or you need to enforce routing at the OS level, you can use firewall rules to route backup traffic through a specific interface. **Example: route traffic to backup server `10.50.0.100` via interface `eth1` (IP `10.50.0.1`)** Using iptables: ```bash # Route outbound traffic to the backup server through the backup interface iptables -t nat -A POSTROUTING -d 10.50.0.100 -o eth1 -j SNAT --to-source 10.50.0.1 # Ensure traffic to the backup server uses the correct route ip route add 10.50.0.100/32 via 10.50.0.1 dev eth1 ``` Using UFW (add to `/etc/ufw/before.rules`, before the `*filter` section): ``` *nat :POSTROUTING ACCEPT [0:0] -A POSTROUTING -d 10.50.0.100 -o eth1 -j SNAT --to-source 10.50.0.1 COMMIT ``` Then reload UFW: ```bash ufw reload ``` :::info OS-level routing applies to all traffic destined for the backup server, not just backups. This is the recommended approach when using S3 storage with a dedicated backup network. ::: ##### Manual-Only Schedules Setting a schedule's interval to **0** creates a manual-only schedule. No automatic backups are dispatched for this schedule. VMs on the schedule can still be backed up and restored on demand (via the admin or end-user interface, depending on access level), and all other schedule settings (storage, encryption, retention, resource controls) still apply. This is useful when you want to: - Define storage and retention settings for a group of VMs without automated backups - Allow end users with **Full Access** or **Manual Only** access to trigger their own backups on their own schedule - Keep a schedule configured but temporarily stop automated runs without disabling it entirely ##### Managing Server Access Levels Click the **server count badge** on a schedule row to open the Manage Servers modal. This lists all commissioned VMs on the hypervisor with their current access level. ![Manage Servers modal](img/backup-manager/manage-servers.png) You can set per-server overrides or use the bulk action to change multiple VMs at once. The "Inherit" option uses the schedule's default access level. --- ### Overview The Overview (**Backup Manager → Overview**) provides an at-a-glance overview of your backup infrastructure. ![Overview page](img/backup-manager/overview.png) #### Status Cards | Card | Description | |:-----|:------------| | Active Now | Currently running batch or single backup jobs | | Servers Protected | Count of VMs with at least one backup | | Total Backup Data | Aggregate backup size across all storage | | Schedules | Enabled vs total schedule count | #### Recent Batches Shows the most recent scheduled backup runs with status, server count, and duration. #### Upcoming Batches Shows when the next scheduled run is due for each schedule, with overdue warnings. #### Recent Errors Lists recent failures across all jobs with the error message, making it easy to spot issues. #### VM Health | Card | Description | |:-----|:------------| | Failing Servers | VMs with consecutive backup failures | | Never Backed Up | VMs that have never been successfully backed up | | Pending Purge | Deleted VMs awaiting backup data cleanup | #### Storage Summary Overview of each storage backend showing health status, disk usage, backup count, and the time of the last backup. --- ### Scheduling and Resource Control The Backup Manager is designed to keep backup operations from impacting virtual server performance, even on heavily loaded hypervisors with hundreds of VMs. #### One Batch at a Time Each hypervisor runs at most **one batch backup at a time**. When a scheduled interval comes due, the system checks whether the hypervisor already has a running batch. If it does, the new batch is deferred until the current one completes. This prevents backup I/O from stacking up and overwhelming storage or network resources. Single-VM operations (on-demand backups, restores, purges) are also mutually exclusive with batch backups: | Running | Attempted | Result | |:--------|:----------|:-------| | Batch backup | Single backup/restore | Queued until batch completes | | Single backup/restore | Batch backup | Batch deferred to next cycle | | Single backup | Another single backup (same VM) | Rejected - one operation per VM | This means a user triggering a manual backup will never compete with a scheduled batch for the same hypervisor's resources, and two users cannot accidentally run simultaneous restores on the same VM. #### VM Concurrency Within a batch, the **Backup Concurrency** setting controls how many VMs are backed up in parallel on a single hypervisor. The default is 1 (sequential). Higher values complete batches faster but increase disk I/O and network load. For most deployments, a concurrency of 1-2 is recommended. On hypervisors with fast NVMe storage and dedicated backup network interfaces, higher values (3-4) can be used safely. #### I/O and CPU Priority Each storage backend (or schedule override) can set I/O and CPU priority levels for the backup process: | Priority | I/O Effect | CPU Effect | |:---------|:----------|:-----------| | Normal | Standard scheduling | Standard scheduling | | Low | Best-effort low priority | Nice +10 | | Idle | Only uses spare I/O bandwidth | Nice +19 (lowest) | Setting I/O priority to **Idle** ensures backups only consume disk bandwidth that VMs are not using. This is the safest option for production hypervisors under heavy load, though backups may take significantly longer. #### Bandwidth and Read Limits Two rate limits control how fast data moves during a backup: | Setting | Controls | Use Case | |:--------|:---------|:---------| | Bandwidth Limit | Upload speed to storage | Prevent saturating the network link to your backup server | | Read Limit | Disk read speed on the hypervisor | Prevent backup reads from starving VM disk I/O | Both limits can be set per storage or overridden per schedule. When both are set, the slower of the two is the effective bottleneck. #### Time Windows & Allowed Days Schedules can be restricted to only run during specific hours (e.g. 01:00-06:00). If a backup interval comes due outside the time window, it waits until the window opens. This allows you to confine backup I/O to off-peak hours. Schedules can also be restricted to specific days of the week (e.g. weekdays only, or Mon/Wed/Fri). When both a time window and allowed days are configured, the schedule only dispatches when both conditions are met. See [Allowed Days](#allowed-days) for configuration details. #### Stale Job Protection If a batch or single job stops responding (hypervisor reboot, network partition, storage outage), the system automatically detects and cleans it up: - **Batch jobs**: cancelled after 48 hours or 3× the schedule interval (whichever is shorter) - **Single jobs**: cancelled after 24 hours This prevents phantom "running" jobs from blocking future backups indefinitely. #### Performance Tuning Understanding how each setting affects hypervisor load and backup speed helps you find the right balance for your environment. ##### Worker Threads Worker threads control how many operations (network uploads, disk reads, frame processing) can execute in parallel within a single VM backup. | Value | Effect | |:------|:-------| | 1 | Single-threaded - minimal CPU usage, suitable for low-spec hypervisors | | 2-4 (default: 4) | Good balance for most deployments | | 8+ | Diminishing returns unless storage backend is very fast and disks are large | Worker threads are shared across all concurrent VM backups on the hypervisor. If you set **Backup Concurrency** to 3 and **Worker Threads** to 4, the hypervisor uses 4 threads total (not 12). Each concurrent VM shares the thread pool, so higher concurrency benefits from more workers. ##### Compression Compression is enabled by default using zstd. Two settings control its behaviour: **Compression Level** (1-19, default: 3) | Level | Speed | Ratio | Use Case | |:------|:------|:------|:---------| | 1 | Fastest | ~2-3x | Fast storage, bandwidth is not a concern | | 3 (default) | Fast | ~3-4x | Good balance for most setups | | 6-9 | Moderate | ~4-5x | Slow or metered network links | | 10-19 | Slow | ~5-6x | Rarely worthwhile - significant CPU cost for marginal gain | Higher levels use exponentially more CPU time per frame. Level 3 typically compresses at 500+ MB/s per thread, while level 9 may drop to 100-200 MB/s per thread. The compression ratio improvement above level 6 is marginal for most VM disk images. **Compression Threads** (default: 1) Controls how many frames are compressed in parallel. Each thread compresses one 4 MB frame independently. | Threads | Effect | |:--------|:------| | 1 (default) | Sequential compression - lowest CPU usage, simple pipeline | | 2-4 | Parallel compression - useful when compression is the bottleneck (high compression level or fast storage) | | 8+ | Only useful at high compression levels with very fast network and disk I/O | When **Compression Threads** is 1, the pipeline is: read frame, compress, upload (sequential). When greater than 1, frames are read ahead and compressed in parallel, then reordered for upload. This overlaps I/O with compression but increases CPU and memory usage proportionally. **When to increase compression threads**: If your backup speed is limited by CPU (check with `top` during a backup - if the backup process is near 100% CPU on one core), increasing threads will help. If your backup is limited by disk read speed or network upload, adding threads will not help and just wastes CPU. ##### Encryption Overhead When encryption is enabled (ChaCha20-Poly1305), each compressed frame is encrypted and authenticated before upload. The overhead is: - **CPU**: Minimal. ChaCha20 operates at 2-4 GB/s per core on modern x86 hardware - it will never be the bottleneck - **Size**: 28 bytes per frame (12-byte nonce + 16-byte authentication tag). Negligible for 4 MB frames - **Incompressibility**: Encrypted data cannot be deduplicated by the storage backend. This only matters for PBS, where server-side deduplication works on the raw chunk data. For all other backends, there is no difference :::caution Enabling encryption with PBS storage disables PBS server-side deduplication. Each backup will consume its full size on the PBS server rather than only storing changed chunks. Consider whether encryption or deduplication is more important for your use case. ::: ##### Backup Concurrency Controls how many VMs are backed up simultaneously on a single hypervisor during a batch. Each concurrent VM runs its own NBD export + read + compress + upload pipeline. | Concurrency | Disk I/O | Network | Memory | Recommended For | |:------------|:---------|:--------|:-------|:----------------| | 1 (default) | Low | Low | ~50 MB | Shared storage, busy hypervisors | | 2 | Moderate | Moderate | ~100 MB | Dedicated backup network, SSD storage | | 3-4 | High | High | ~150-200 MB | NVMe storage, dedicated backup network, fast storage backend | | 5+ | Very High | Very High | ~250+ MB | Not recommended for most deployments | Memory usage per VM depends on compression threads (each thread buffers ~4 MB frames) and the number of disks being backed up. ##### Practical Examples **Low-impact production hypervisor** (minimise disruption to VMs): - Backup Concurrency: 1 - Worker Threads: 2 - Compression Level: 3 - Compression Threads: 1 - I/O Priority: Idle - CPU Priority: Low - Read Limit: 50 MB/s **Fast overnight backups** (complete batches quickly during maintenance window): - Backup Concurrency: 3 - Worker Threads: 4 - Compression Level: 1 - Compression Threads: 2 - I/O Priority: Normal - CPU Priority: Normal - No read or bandwidth limits - Time Window: 01:00 - 06:00 **Metered or slow network link** (minimise bandwidth usage): - Backup Concurrency: 1 - Worker Threads: 4 - Compression Level: 6 - Compression Threads: 2 - Bandwidth Limit: 25 MB/s --- ### Monitoring #### Batches Navigate to **Backup Manager → Batch Jobs** to see the history of scheduled backup runs. ![Batches list](img/backup-manager/batches.png) Each batch represents one scheduled run across all VMs on a hypervisor. Click a row to expand it and see: - Per-VM job status (completed, failed, pending) - Progress bar for running batches - Error details for failed VMs - Full log output (click **View Log**) Filter by status, job ID, schedule, or date range. #### Jobs Navigate to **Backup Manager → Jobs** to see individual operations (backups, restores, and purges). ![Jobs list](img/backup-manager/jobs.png) Click a row to expand it and see: - Progress details for running jobs (stage, bytes transferred, per-disk status) - Per-disk results for completed backups (size, duration, chain) - Error details for failures - Log output Filter by status, type, server name, job ID, or date range. #### Cancelling Jobs Running backup and restore operations can be cancelled from the **Jobs**, **Batch Jobs**, or per-server **Backup Tree** pages. Click the **Cancel** button on a running job or batch and confirm. Cancelled jobs are marked with an orange "cancelled" badge. Cancellation sends a termination signal to the hypervisor - the backup process stops and any partially written data is cleaned up. #### Backups Navigate to **Backup Manager → Backups** to browse all backup records across every server and storage. ![Backups list](img/backup-manager/backups.png) Click a row to expand and see full record details: file path, virtual size, SHA-256 hash, compression, and encryption status. Filter by backup type, encryption status, server name, storage, or date range. #### System Logs Navigate to **Backup Manager → System Logs** to see internal dispatcher and subsystem log entries. Each log entry includes a timestamp, severity level, source subsystem, and message. Related entities (schedule, storage, hypervisor, server, batch) are linked when available. Filter by level (info, warn, error), source (dispatcher, inventory, purge, health, prune, heal), free-text search, or any combination of schedule/storage/hypervisor/server/batch ID and date range. :::info Info-level messages are only written when **Log Info Level** is enabled in [Global Settings](#logging). With it disabled, only warnings and errors appear. ::: --- ### Per-Server Management On a server's admin page, under the **Backup** tab, the Backup Manager section shows the backup tree for that VM. ![Backup tree for a server](img/backup-manager/backup-tree.png) #### Access Level Set the access level for this specific VM using the dropdown. "Inherit" uses the schedule's default. #### Backup Tree Backups are displayed in a hierarchical tree: - **Disk**: each virtual disk (disk 0, disk 1, etc.) - **Chain**: a sequence of backups starting from a full backup - **Backup records**: individual full, incremental, or differential backups Each chain shows the storage backend, backup count, total size, and start date. The most recent chain is marked as "current." #### Backup Now Click **Backup Now** to trigger an immediate backup of this VM. The backup type (full or incremental) is determined automatically based on the chain state. A progress banner appears showing the current stage, bytes transferred, and percentage complete. #### Restore from Backup Click **Restore from Backup** to open the restore modal. ![Restore modal](img/backup-manager/restore-modal.png) **Step 1: Selection** - Select which disks to restore using the checkboxes - Choose which chain to restore from (defaults to the latest) - Choose a restore point within the chain (defaults to the latest backup) **Step 2: Confirmation** - Review the selected disks, chains, and restore points - Confirm the operation :::warning Restoring a disk overwrites its current data. The VM will be stopped during the restore and restarted when complete. ::: #### Cross-Server Restore Cross-server restore allows you to restore a backup from one server onto a different server. This is useful for cloning a server or recovering data onto replacement hardware. To start a cross-server restore: 1. Navigate to **Backup Manager → Backups** 2. Find the backup record you want to restore from 3. Click the **Restore to Server** icon on the backup row ![Cross-server restore modal](img/backup-manager/cross-restore-modal.png) **Step 1: Select Target** - Search for the target server by name, ID, or UUID - Select the target disk. Only disks that match the backup's virtual size are shown - Toggle **Integrity Check** to run filesystem verification after the restore (recommended) **Step 2: Confirm** - Review the source backup, target server, and target disk - Confirm the operation The restore will: 1. Stop the target server 2. Download the backup data from storage 3. Write it to the target disk, replacing its contents 4. Run an integrity check if enabled 5. Leave the server offline :::warning The target server is **not** automatically started after a cross-server restore. Use a power action to boot it once the restore completes. This is intentional - the target server may need configuration changes (network, hostname, etc.) before being brought online. ::: #### Removed Disks If a virtual disk has been removed from a server but backups still exist for it, the disk is shown with a red **removed** badge. Removed disks are automatically excluded from restore operations (unchecked and disabled in the restore modal) but their backup data remains available for manual recovery. #### CLI Restore to File Click **CLI Restore to File** to generate a command for restoring a backup directly to a file on the hypervisor, rather than writing it back to the VM's disk. This is useful for manual recovery, forensic analysis, or extracting data from a backup without affecting the running VM. 1. Select the disk, chain, and restore point 2. Click **Generate Restore Command** 3. A config file is written to the hypervisor and a ready-to-run CLI command is displayed 4. Copy the command and run it via SSH on the hypervisor :::info The config file contains storage credentials. Delete it from the hypervisor after use. ::: #### Delete a Chain Click the trash icon on a chain header to delete all backups in that chain. This permanently removes the backup files from remote storage. :::warning **PBS storage:** PBS snapshots are shared across all disks. Deleting a chain on PBS storage will also delete backups for other disks at the same timestamps. The confirmation dialog shows the total number of affected records across all disks. ::: --- ### Storage Health The Backup Manager automatically checks storage health every 30 minutes. Unhealthy storage is rechecked every 15 minutes. Health status is visible in three places: - **Overview → Storage Summary**: health dot and disk usage bar - **Storage list**: health dot per storage - **Storage manage page**: full health card with status, last checked time, error message, and disk usage breakdown You can trigger an immediate health check from the storage manage page by clicking **Check Now**. Health checks verify connectivity and, where supported, report disk usage: | Backend | Connectivity | Disk Usage | |:--------|:---:|:---:| | SSH | | | | PBS | | | | Rclone | | | | FTP | | | | S3 | | | --- ### Retention and Pruning When a retention policy is configured, expired backups are automatically pruned. The system evaluates retention centrally and dispatches delete operations to hypervisors every 12 hours. The backup engine also prunes after each successful backup using the same rules. All retention rules form a **union** - a backup is kept if **any** rule selects it: 1. **Active chain** - for non-PBS backends, the latest incremental chain per disk per storage is always kept (pruning it would break the chain for the next incremental) 2. **Keep Last** - the N most recent backups per disk are always kept, regardless of age or bucket rules 3. **Age window** (age-based policies) - the backup is younger than the retention window (e.g. 90 days for "3 Months") 4. **GFS buckets** (GFS policies) - the backup is the newest in its calendar bucket (day, week, month, or year) and the bucket is within the configured count A backup is pruned only if **none** of these rules protect it. **How GFS pruning works:** When the prune cycle runs, each GFS bucket rule is evaluated independently: 1. All backups are grouped into calendar buckets (e.g. one bucket per calendar day) 2. Within each bucket, only the **newest** backup is considered (older ones in the same bucket are not protected by that rule) 3. The **N most recent** buckets are kept (e.g. `7 daily` keeps the 7 most recent calendar days that have a backup) 4. Each rule adds to the keep set independently - a backup kept by the weekly rule is protected even if the daily rule would not keep it **Example: GFS 3 Months (7 daily, 4 weekly, 3 monthly) with Keep Last 2** Assume daily backups have been running for 4 months: | Rule | Protects | Example | |:-----|:---------|:--------| | Active chain | 1 chain | Today's chain (always kept) | | Keep Last 2 | 2 newest inactive chains | Yesterday's and the day before | | 7 daily | Up to 7 chains | One per day for the last 7 calendar days | | 4 weekly | Up to 4 chains | One per week for the last 4 calendar weeks | | 3 monthly | Up to 3 chains | One per month for the last 3 calendar months | After pruning, you would keep roughly 10-12 unique chains (many overlap between rules) and all older chains would be removed. :::info The **Keep Last** setting on the storage is the global default. Schedules can override it. A value of 2 (the default) means at minimum the two most recent backups per disk are always kept, regardless of the retention policy. This applies to both age-based and GFS policies. ::: --- ### Auto-Purge of Deleted VMs When a virtual server is deleted in VirtFusion, its backups are not immediately removed. Instead, a purge is scheduled for a configurable number of days in the future. Pending purges are processed automatically. If a purge fails, it retries with a 48-hour backoff after 3 consecutive failures. To manually purge backups for deleted VMs on a schedule, click the **Purge Deleted VMs** icon on the schedule row. --- ### Schedule-Wide Purge To remove all backups for every VM on a schedule: 1. Go to **Backup Manager → Schedules** 2. Click the **Purge All Backups** icon on the schedule row 3. Review the count of backups and VMs that will be affected 4. Confirm the operation :::warning This permanently deletes all backup files from remote storage for all VMs on this schedule. This cannot be undone. ::: --- ### Deleted Schedules When a schedule is deleted, it is soft-deleted and can be restored. To view deleted schedules, use the **Deleted** filter on the Schedules page. Click the **restore icon** on a deleted schedule to bring it back in a disabled state, allowing you to review and re-enable it when ready. :::info Deleting a schedule does not remove existing backups. If you want to remove the backup data as well, use **Purge All Backups** before deleting the schedule. ::: --- ### Package Defaults You can set a default backup access level per server package. Navigate to the package editor and find the **Backups** section. ![Package backup settings](img/backup-manager/package-backups.png) | Setting | Description | |:--------|:------------| | Backup Manager Access | Default access level for servers on this package. "Inherit" defers to the schedule's default | This provides a convenient way to control backup access by plan tier. For example, give premium plans "Full Access" and basic plans "Scheduled Only." --- ### End-User Experience End users with access level **View & Restore** or higher see a simplified backup interface on their server management page. ![End-user backup view](img/backup-manager/user-backup.png) #### What Users See - **Last backup**: when the most recent backup completed - **Next scheduled backup**: when the next automatic backup is expected - **Retention summary**: how long backups are kept - **Disk list**: each virtual disk with its backup history Each backup record shows the date and an expiry indicator: | Indicator | Meaning | |:----------|:--------| | Currently Retained | No expiry configured, backup kept indefinitely | | Date with days remaining | Backup will expire on this date | | Expired | Past retention, will be removed on the next pruning cycle | #### User Actions **Backup Now** (Full Access and Manual Only): Users can trigger an immediate backup. A progress bar shows the operation status. **Restore** (View & Restore, Full Access, and Manual Only): Users select which disks to restore and optionally pick a specific restore point. The interface uses drive letters (Drive A, Drive B). :::info Users see a simplified view with no chain IDs, storage details, or technical metadata. Error messages are generic and suggest contacting support for persistent issues. ::: --- ### Global Settings Navigate to **Backup Manager → Settings** (button in the top-right of the dashboard) to tune runtime behaviour without code changes. ![Global settings](img/backup-manager/settings.png) Settings take effect on the next cron tick. All values have sensible defaults and most installations will not need to change them. #### Dispatcher | Setting | Default | Description | |:--------|:-------:|:------------| | Paused | Off | When enabled, no new backup batches are dispatched. Running batches, retention pruning, auto-purge, and health checks continue normally. Force dispatch is also blocked while paused. | | Concurrency | 50 | Maximum concurrent hypervisor connections during dispatch. Increase for large fleets (500+ hypervisors). | | Max Concurrent Batches | Unlimited | Global limit on simultaneously running batches across all schedules. Leave empty for unlimited. Useful for limiting load on shared storage. | #### Stale Detection | Setting | Default | Description | |:--------|:-------:|:------------| | Max Batch Hours | 48 | Batches running longer than this are automatically cancelled and marked failed. | | Max Single Job Hours | 24 | Individual backup or restore jobs running longer than this are cancelled. | #### Auto-Purge | Setting | Default | Description | |:--------|:-------:|:------------| | Max Purges Per Tick | 5 | Maximum deleted-VM purge operations dispatched per cron tick. Purges run asynchronously and are tracked as jobs. | | Purge Backoff Hours | 48 | After 3 consecutive purge failures for a VM, wait this many hours before retrying. | #### Retention Pruning | Setting | Default | Description | |:--------|:-------:|:------------| | Prune Interval Hours | 12 | How often each storage's retention policy is re-evaluated. | | Max Prune Storages Per Tick | 3 | Number of storages to evaluate per cron tick. | | Max Prune Dispatches Per Tick | 20 | Maximum delete-backups jobs dispatched per tick across all storages. | | Warning Window Days | 7 | Days ahead to show expiration warnings in the backup tree. | #### Health Checks | Setting | Default | Description | |:--------|:-------:|:------------| | Check Interval (minutes) | 30 | Re-check healthy storages every N minutes. | | Unhealthy Re-check (minutes) | 15 | Re-check failing storages more frequently. | #### Housekeeping | Setting | Default | Description | |:--------|:-------:|:------------| | Job Retention Days | 120 | Days to keep completed/failed job and batch records. | | Log Retention Days | 30 | Days to keep system log entries. | #### Logging | Setting | Default | Description | |:--------|:-------:|:------------| | Log Info Level | Off | When enabled, info-level messages are written to the system logs table. Useful for debugging dispatch behaviour. Increases log volume significantly. | #### User Controls | Setting | Default | Description | |:--------|:-------:|:------------| | Manual Backups | Enabled | When disabled, end users cannot trigger manual backups. Admin-initiated and scheduled backups are unaffected. | | Manual Restores | Enabled | When disabled, end users cannot trigger restores. Admin restores are unaffected. | | Backup Cooldown (minutes) | 60 | Minimum wait time between successful manual backups per server. Only successful completions trigger the cooldown - failed or cancelled jobs do not. Set to 0 to disable. | | Restore Cooldown (minutes) | 60 | Minimum wait time between successful manual restores per server. Same rules as backup cooldown. Set to 0 to disable. | :::tip Use the **Paused** toggle for planned maintenance windows. All monitoring and cleanup continues - only new batch dispatches are held. ::: --- ### Disabling the Legacy Backup System If you are migrating to the Backup Manager, you should disable the legacy backup system. Navigate to **Settings → General** and find the **Legacy Backup System** setting, then disable it. This will: - **Stop all legacy backup schedules** from running - **Remove legacy backup options** from the end-user interface - **Allow in-progress backups** to finish normally - **Continue deleting expired backups** that are already in a deleting state This setting only affects the legacy system. The Backup Manager operates independently and is unaffected. :::caution Before disabling the legacy system, ensure all servers have been migrated to Backup Manager schedules and that any legacy backups you need to retain have been accounted for. ::: --- ### Advanced CLI The Backup Manager includes a powerful command-line tool installed on each hypervisor. The CLI provides advanced capabilities beyond what is available in the web interface: | Feature | Description | |:--------|:------------| | Mount | Mount a backup as a read-only filesystem for file-level recovery. Supports local and remote backups (SSH, S3), with optional writable overlay | | Verify | Deep integrity verification of backup files with per-frame SHA-256 checksums | | Reindex | Rebuild backup metadata by scanning storage (disaster recovery) | | Estimate | Query dirty data size per disk without running a backup | | Prune | Run retention pruning manually with custom rules | | Download | Download a raw disk image directly from PBS storage | | Cleanup | Remove stale bitmaps, metadata, and temporary files from a hypervisor | | Cancel | Cancel a running backup or restore operation | | Inventory | List all backups for a server from local metadata | | S3 Cleanup | List and abort stale S3 multipart uploads | | Self-heal | Automatic recovery from interrupted backups (stale locks, frozen VMs, orphaned processes) | The CLI also supports granular control over individual backup and restore operations, including per-disk backup type selection, mixed backup types in a single run, point-in-time restore, multi-disk restore, and resumable uploads. :::info Full CLI documentation will be available with the stable release. ::: --- ### Troubleshooting #### Backups are not running 1. Check the **Overview** for recent errors 2. Verify the schedule is **Enabled** and the interval is > 0 3. Verify the storage is **Enabled** (not disabled or read-only) 4. Check the storage health status and run a manual health check 5. Check that the VM's access level is not **Disabled** #### Backup fails with "hypervisor unreachable" VirtFusion cannot reach the hypervisor's API. Verify: - The hypervisor is online and the VirtFusion agent is running - Network connectivity between VirtFusion and the hypervisor - The hypervisor's API port is accessible #### Backup fails with storage connection errors 1. Go to the storage manage page and click **Check Now** 2. Review the error message in the health status 3. Verify credentials and connection details 4. Test from the hypervisor directly (SSH, S3 credentials, PBS token, etc.) #### Restore fails 1. Check the job log for the specific error (Jobs → click the job → View Log) 2. Ensure the storage is not in read-only mode 3. Verify the VM is not in rescue mode, migrating, or otherwise locked #### SSH backup or restore fails after several hours If backups or restores to SSH storage fail consistently after running for a long time (typically with exit status 255 in the log), the SSH connection is being dropped. This most commonly affects large disks (hundreds of GB) where the transfer takes several hours. **Check the remote SSH server:** 1. Verify `ClientAliveInterval` and `ClientAliveCountMax` are set in `/etc/ssh/sshd_config` (see [SSH Server Configuration](#ssh-server-configuration) above) 2. Check sshd logs for disconnect events: `journalctl -u sshd` around the time of the failure 3. Check for OOM kills: `dmesg | grep -i "oom\|kill\|memory"` **Check the network path:** 1. If using a cloud firewall (Hetzner, AWS, etc.), connection tracking timeouts may be dropping long-lived connections. Try disabling the firewall as a test 2. Check `sysctl net.netfilter.nf_conntrack_tcp_timeout_established` on both servers 3. NAT gateways and load balancers in the network path may also have idle connection timeouts #### "A backup is already running" Only one backup or restore operation can run per VM at a time. Wait for the current operation to complete, or check the Jobs page for stuck jobs. #### Stale or stuck jobs Occasionally a job may become stuck, for example if a hypervisor reboots mid-backup or a storage backend becomes unreachable during a transfer. Stuck jobs are automatically detected and cancelled: - Batch jobs stuck for more than 48 hours (or 3x the schedule interval) - Single jobs stuck for more than 24 hours If a job appears stuck, it will be automatically cleaned up on the next dispatch cycle. --- ## Quick Admin Setup ## Quick admin setup To get you setup quickly, the following steps are required. ### Hypervisor #### Connecting a hypervisor to the system :::info You must have installed at least one hypervisor as [documented here](/installation/hypervisor) before continuing. ::: From the admin menu, select `Hypervisors`. Click on the Add Hypervisor button on the top right. Fill in a name for the hypervisor (It can be changed at any time) and the ip address (IPv4 or IPv6). ![hypervisor](img/hypervisor1.png) Once added you will be asked to commission the hypervisor. Click the start button. ![hypervisor](img/hypervisor2.png) The hypervisor will now set up authentication automatically and finish the commissioning process. ![hypervisor](img/hypervisor3.png) If all goes well (If an error occurs it will print the error and give you the opportunity to configure again) the hypervisor will now be connected and ready to configure. #### Configuring the hypervisor When configuring a hypervisor it's vital you set limits on the resources available. ![hypervisor](img/hypervisor4.png) - `Servers` should be set to the maximum amount of servers you would like on the hypervisor. This can be left at 0 (unlimited) if you are not sure. - `Virtual CPU Cores` should be set to the maximum amount of cores. It's not easy to calculate, so this setting can be set at a high value. - `Memory` should be set to the maximum amount of memory available for servers. You should take the amount of memory in the hypervisor and deduct a few GB for hypervisor usage. Once you have set the values, `Enable` the hypervisor and click the update button. Lower down the page you will see the storage section. You will need to set the `Maximum Disk` value. This should be set to the available disk space you have on the hypervisor but allowing space for cached operating system templates and temporary files. ![hypervisor](img/hypervisor5.png) Make sure its `Enabled` and click the Update button. Select the correct network type for the primary network and update it. ![hypervisor](img/hypervisor8.png) #### Creating a hypervisor group A hypervisor needs to be placed in a group. Doing this means you can cluster hypervisors and distribute servers across the cluster. Click on `Hypervisor Groups` from the main menu. You will see a pre-added group called Default. You can either edit and rename this group or create your own. Let's just edit the default group. ![hypervisor](img/hypervisor6.png) - `Name` can be set to anything you like. The most common method is countries and cites. For example `United Kingdom (London)`. - `Server Distribution` is how the servers are placed across all hypervisors in the group. Once edited, click the Update button. As you can see the hypervisor that we added earlier is already in this group. If you created a new group, you would need to edit the hypervisor and select the group you created. ![hypervisor](img/hypervisor7.png) ### IP Addressing You don't strictly need IP addresses, but it's unlikely the use case you require. #### Creating an IP address block Select `IP Address Blocks` from the main menu and click the `Create IP Block` button. ![ip](img/ip1.png) - `Name` can be set to anything you like and can be changed later. - `Type` IPv4 or IPv6. Once created you will land on the configuration screen. ##### IPv4 configuration ![ip](img/ip2.png) Fill in all the details for your IP block, Enable it and click `Save Settings`. Once save a new box will be displayed underneath. ![ip](img/ip3.png) Enter the start IP and the end IP then click the `Add` button. IP's can be reserved so full ranges can be added. The last thing to do is aggin the IP block to the hypervisor we added. Select it and click `Save Hypervisors`. ![ip](img/ip4.png) #### IPv6 configuration IPv6 configuration is a little more complicated as it requires a master subnet that the system can break down into smaller subnets. The fields should be self-explanatory though. ![ip](img/ip5.png) Once configured click `Save Settings`. Information about the subnet will be shown at the top of the screen. ![ip](img/ip6.png) You may also blacklist subnets you don't want to be made available to servers at the bottom of the page. ![ip](img/ip7.png) The final step is to enable this block for the hypervisor like we did with IPv4. ![ip](img/ip4.png) ### Packages Each server will be built from a package. A package is a collection of resources to be applied to a server when it's built. #### Creating a package Select `Packages` from the main menu and click the `Create Package` button. ![package](img/package1.png) Give it a `Name` which can always be changed later. Once created you configure the package. Packages are limited at the moment but will be expanded on as development progresses. ![package](img/package2.png) Again, the settings should be self-explanatory. Fill in the fields, Enable and click `Update`. That's it. You have now done the minimal configuration to build servers! --- ## Reverse DNS VirtFusion has two modes for reverse DNS. The first is **Manual Mode** and the second is **Automation Mode**. Both modes give the same end user experience, however, manual mode will be a request to an administrator to update the reverse DNS entries manually. ![rdns](img/rdns1.png) ### Manual Mode The idea behind manual mode is that sometimes it may not be possible to automate reverse DNS. i.e your service provider may only take manual requests for the changes, or they can't delegate the zone to you. To use manual mode, it must be enabled on the IP Address Block. ![rdns](img/rdns2.png) Once it's enabled, an end user can make requests for IPs within the block you enabled manual mode on. After the user has made a request, An email will be sent to administrators on the system alerting them of a pending request. Within the admin area navigate to `Reverse DNS Requests` from the main menu. You can view the request and run actions against it. If you don't approve the request, click the `Cancel` button. If you've approved the request and set up the reverse DNS entry, click `Complete`. ![rdns](img/rdns3.png) Once you've selected your action, the buttons will change to the following. ![rdns](img/rdns4.png) You have the option to send the end user an email that the request has either been approved or not and an option to set the request back to pending if you have made a mistake. Be aware that requests in the complete state will not be shown in the list after 2 hours have passed. That's it - There is nothing more for you to do after you've approved or cancelled the request. ### Automation Mode Automation is a little more complicated to setup and configure. Currently, automation supports `ClouDNS`, `PowerDNS` and `Bunny DNS`. #### Automation with ClouDNS [ClouDNS](https://www.cloudns.net/) is a low-cost premium DNS hosting service with a global Anycast DNS network. **You must have an active paid account at [ClouDNS](https://www.cloudns.net/) to continue.** Login to your account at [ClouDNS](https://www.cloudns.net/) and generate an API user (`Resellers` menu option). You should end up with an `auth id` and `password`. Save them for later. Once you have your API details, Navigate to `DNS Services` in the VirtFusion admin area and click the `Add Service` button. ![rdns](img/rdns5.png) Fill in all the details and add the service. After adding the service you may test the connection to make sure that all details are correct, and you have access to the ClouDNS API. ![rdns](img/rdns6.png) Next we need to add a reverse DNS zone (in VirtFusion). Select `Reverse DNS Zones` from the main menu and click the `Add Zone` button. ![rdns](img/rdns7.png) - Select the `IP Version` - Fill in the `Zone` details - `Default Entry` is only used for IPv4 and is optional. Leave blank to disable it - Select the `DNS Service` you just created Once added, Navigate to the IP Address Block that you have created this zone for. ![rdns](img/rdns8.png) Click `Automated (DNS Serivce)` and select the zone you just created. Click the `Update` button. That's it, Automation is now setup. The first DNS entry will create the zone in ClouDNS or if you navigate back to the `Reverse DNS Zone` you created and click the `Rebuild` button. If you would like to manually insert reverse DNS for an IP, you may do so by visiting the IPv4 or IPv6 Addresses menu item and editing the reverse DNS (+RDNS in the list). **You will need to rebuild the zone after making these manual changes**. #### Automation with PowerDNS You are responsible for setting up your own PowerDNS cluster with API access. VirtFusion will communicate with PowerDNS via API and requires a URL, port and API token. Navigate to `DNS Services` in the VirtFusion admin area and click the `Add Service` button. ![rdns](img/rdns9.png) Fill in all the details and add the service. After adding the service you may test the connection to make sure that all details are correct, and you have access to the PowerDNS API. ![rdns](img/rdns6.png) Next we need to add a reverse DNS zone. Select `Reverse DNS Zones` from the main menu and click the `Add Zone` button. ![rdns](img/rdns10.png) - Select the `IP Version` - Fill in the `Zone` details - `Default Entry` is only used for IPv4 and is optional. Leave blank to disable it - Select the `DNS Service` you just created - Enter a `Hostmaster Email` - Enter at least two `Nameservers` Once added, Navigate to the IP Address Block that you have created this zone for. ![rdns](img/rdns8.png) Click `Automated (DNS Serivce)` and select the zone you just created. Click the `Update` button. That's it, Automation is now setup. The first DNS entry will create the zone in PowerDNS or if you navigate back to the `Reverse DNS Zone` you created and click the `Rebuild` button. If you would like to manually insert reverse DNS for an IP, you may do so by visiting the IPv4 or IPv6 Addresses menu item and editing the reverse DNS (+RDNS in the list). **You will need to rebuild the zone after making these manual changes**. #### Automation with Bunny.net **Important Information** - You do not need to create the RDNS zones in the Bunny.net control panel. ### Zoning IP Blocks You can create an IP block of any compliant size; however, the largest size supported for an RDNS zone is a /24. VirtFusion allows you to subdivide your IP blocks and assign individual zone files to each sub-block. #### Create a Sub Block 1. Navigate to your IPv4 block and select the `General` tab. 2. At the bottom of the page you will see an option called `Sub Blocks`. Click the `Add` button. 3. In the pop up, enter a `Name`, select the `Reverse DNS Type` and select the relevant `Reverse DNS Zone` for the portion of the IP block you would like to assign. 4. Click the `Add` button to save it. #### Adding Addresses to a Sub Block 1. Click the `Addresses` tab at the top of the page. 2. Select (tick) all the addresses in the table that you would like included in the Sub Block. 3. At the bottom of the screen, click the `Assign to Sub Block` button. 4. In the pop up, select the Sub Block and click `Assign` --- ## Server Backups VirtFusion supports user level backups. The current implementation supports on demand and scheduled. ### Configuring a backup server in VirtFusion From the admin area navigate to `Backups -> Backup Storage` then click the `Add Storage` button. ![backups](img/backup1.png) Give the backup server a name (it can be changed later) and click `Add`. There may be a slight delay while the system generates a unique key pair. ![backups](img/backup2.png) Fill in all the information on the next screen and make a copy of the public key. Then click `update`. On the same page, Configure the hypervisor groups or hypervisors that can use this backup server to store backups. Then click `update`. ![backups](img/backup3.png) It's also worth noting that hypervisors also have a setting that will define how they choose backup servers to use. The default is `Directly assigned storage only`. ![backups](img/backup4.png) ### Configuring the backup server Now you've added a backup server in VirtFusion, you will need to make sure the following tasks are completed on the remote backup server: - Make sure the specified unix user is created - Add the pre-generated public key for that user (authorized keys) - **Make sure rsync is installed** ### Adding a backup plan From the admin area navigate to `Backups -> Server Backup Plan` then click the `Create Plan` button. ![backups](img/backup5.png) Give the plan a name (it can be changed later) and click `Create`. ![backups](img/backup6.png) Fill in all the information on the next screen. Again, don't forget to enable it. ### Assign a backup plan to a server package Edit your server packages (`Resources -> Server Packages`) and assign the backup plan. ![backups](img/backup7.png) ### Assign a backup plan to server You may need to assign a backup plan to a server or edit the backup plan it uses. You can do this by selecting the server in the admin area and viewing the `Backups` pane. ![backups](img/backup8.png) ### Debugging #### Error -> Failed to login on the SSH2 check This error indicates that either the control server or the hypervisor is unable to complete a basic SSH connectivity check to the backup server. Please check firewall rules, routing, and general network connectivity between the relevant host and the backup server. --- ## Trusted Hosts Setting your trusted hosts is a great security measure because it helps prevent cache poisoning and phishing attacks. By explicitly defining which Host headers VirtFusion will accept, you reduce the risk of attackers injecting malicious or unauthorized hostnames into requests. :::info Hosts can only be managed via CLI on the control server using the `vfcli-ctrl trusted-hosts` command. ::: ### Adding a Host You can add as many hosts as you like. From CLI, run the following command: ```shell vfcli-ctrl trusted-hosts add ``` Replace `HOST|IP` with the domain of your control server. You can also specify IPv4 and IPv6 addresses. > **Example** > > ```shell > vfcli-ctrl trusted-hosts add cp.control.com > ``` ### Deleting a Host From CLI, run the following command: ```shell vfcli-ctrl trusted-hosts del ``` ### Resetting Hosts :::warning This command will remove all the hosts you have defined and will set the system to allow all. ::: From CLI, run the following command: ```shell vfcli-ctrl trusted-hosts reset ``` Replace `HOST|IP` with the host you would like to delete. > **Example** > > ```shell > vfcli-ctrl trusted-hosts del cp.control.com > ``` ### Listing Hosts ```shell vfcli-ctrl trusted-hosts list ``` --- ## FAQ ### The firewall feature blocks connectivity to other VMS This issue is usually caused by the UFW firewall on the hypervisor. The fix is pretty simple with a one time only command. ```shell ufw default allow routed ``` ### Re-commissioned but receiving the error `commission status done` This happens when the hypervisor has already been locked to a control server. Usually the hypervisor should be re-installed as it may contain files or even virtual servers which may conflict if you add it to a new control server. If you are sure the hypervisor is clean, you can remove the file `/opt/virtfusion/app/hypervisor/conf/auth.json` to allow it to be re-linked. ### Hybrid Migration has Stalled Sometimes you may run in to a situation where connectivity is lost between hypervisors when migrating a server. In this situation, you can manually restart the migration. * Navigate to the list of migrations (`Server -> Migrations`). * Find the migration that has stalled and click on the log icon at the end of the row. * Scroll down to the section called `Commands`. * Take the first command listed and run it via SSH on the control server. it will look similar to `/opt/virtfusion/php/bin/php /opt/virtfusion/app/control/artisan server:migrate-v2 virtfusion-migrate-v2-id=1234` * Wait for it to complete. It may take a while. ### I receive error: Unable to get index for interface eth0: No such device when booting a server You will receive this error if you are attempting to use MacVTap networking and the interface is specified incorrectly under the hypervisor network settings. Find the name of your primary interface using `ip a` on the hypervisor and update Hypervisor -> Networks -> Primary Network -> Interface Name with the correct name. ### Updating Supervisor on the Control Server Follow the instructions below for you control server Linux distribution. All commands should be run in SSH as root. :::caution Important Before reloading Supervisor. Make sure there are no server tasks running. ::: ##### Debian/Ubuntu ```shell wget https://install.virtfusion.net/debian-supervisord.conf -O /etc/supervisor/supervisord.conf supervisorctl reload ``` ##### RHEL ```shell wget https://install.virtfusion.net/rhel-supervisord.conf -O /etc/supervisord.conf supervisorctl reload ``` ### priority.xml found You may encounter this error during certain actions if a disaster recovery session is currently running. Additionally, this error can also appear if a disaster recovery process has failed on a specific server. If you are certain that the disaster recovery process is no longer running, you should attempt to restart it for the specific server. You can do this using the following command. ```shell vfcli-hv dr:backup --only-servers=SERVER_ID ``` You should replace `SERVER_ID` with the appropriate numeric ID for the server. ### How do i enable nested virtualization Nested virtualization is enabled by default if a virtual server uses the `host-passthrough` CPU model. You may disable nested virtualization by toggling the appropriate extra CPU flags (`svm`/`vmx`) to disable. ### Disabling QEMU Sandbox Why you should not disable this: * Protects the host from potentially compromised VMs. * Adds an additional layer of defense in multi-tenant environments. * Helps meet security/compliance standards in datacenters. If you still want to disable sandboxing, run the following commands on the hypervisor: ```shell sed -i "s/#seccomp_sandbox = 1/seccomp_sandbox = 0/g" /etc/libvirt/qemu.conf systemctl restart libvirtd ``` ### You receive a 500 internal server error after updating MariaDB Sometimes, MariaDB release breaking changes. You can install our latest configurations by running the following commands on your control server: #### Debian & Ubuntu ```shell wget https://install.virtfusion.net/my.cnf -O /etc/my.cnf systemctl restart mariadb vfcli app:restart --services=fpm,web,queue-hard ``` #### RHEL ```shell wget https://install.virtfusion.net/my-rhel.cnf -O /etc/my.cnf systemctl restart mariadb vfcli app:restart --services=fpm,web,queue-hard ``` ### I receive a 401 error when connecting to a hypervisor A 401 Unauthorized error indicates an authentication issue between the control server and the hypervisor. One common reason for this is that the IP address of the control server has changed. The hypervisor uses this as part of the authentication process. Edit `/opt/virtfusion/app/hypervisor/conf/auth.json` on the hypervisor and edit the `"ip"` field to reflect the new IP address of the control server. Example: ```json { "ip": "NEW_IP_HERE", "token": "CqlsVq6zS4vwLQEZqlVo5tHmkgBzvdZYQ2NJnr5q...", "hash": "bc1a53dd974e54c903399fcbe8d05550b35842ca052bc5...", "id": 6 } ``` :::important Do not edit or make any changes to the `token`, `hash`, or `id` fields as this will invalidate all the authentication and a re-commission will be required. ::: ### Server console previews are not working Console previews require ImageMagick installed on a hypervisor. With older installations of VirtFusion this package may be missing and will need installing. RHEL ```shell dnf install ImageMagick ``` Debian/Ubuntu ```shell apt install imagemagick ``` --- ## Fix Outdated Installation import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; :::info Keeping your systems up to date is essential for security, performance, and compatibility. The instructions below are provided to guide you through fixing an outdated installation. You can find more information on service updates [here](releases/services). ::: ### Control Server The following commands should be run as the root user on the control server only. **Debian/Ubuntu** ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 -y wget -qO- https://install.virtfusion.net/fix_outdated_control.sh | bash vfcli-ctrl app:restart --services=fpm,web,queue-hard ``` **RHEL** ```shell dnf makecache -y && dnf update vf-nginx vf-php8 -y wget -qO- https://install.virtfusion.net/fix_outdated_control.sh | bash vfcli-ctrl app:restart --services=fpm,web,queue-hard ``` ### Hypervisor The following commands should be run as the root user on a hypervisor only. **Debian/Ubuntu** ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 vf-system-php8 -y wget -qO- https://install.virtfusion.net/fix_outdated_hypervisor.sh | bash vfcli-hv app:restart --services=system-fpm,fpm,web,queue-hard ``` **RHEL** ```shell dnf makecache -y && dnf update vf-nginx vf-php8 vf-system-php8 -y wget -qO- https://install.virtfusion.net/fix_outdated_hypervisor.sh | bash vfcli-hv app:restart --services=system-fpm,fpm,web,queue-hard ``` --- ## Change the Control Server Domain Changing your control server domain is straightforward. You'll need SSH access to your control server to proceed. ### 1. Update the ACME Client Ensure your ACME client is up to date by running: ```shell acme.sh --upgrade ``` ### 2. Remove the Old SSL Certificate Replace OLD_DOMAIN.COM with your current domain name and execute: ```shell acme.sh --remove -d OLD_DOMAIN.COM ``` ### 3. Request a New SSL Certificate Replace DOMAIN.COM with your new domain name and run: ```shell acme.sh --issue \ --webroot /opt/virtfusion/nginx/acme \ --keylength 4096 \ --server letsencrypt \ -d DOMAIN.COM ``` ### 4. Install the New SSL Certificate Again, replace DOMAIN.COM with your new domain and execute: ```shell acme.sh --installcert \ --keypath /opt/virtfusion/ssl/ssl.key \ --fullchainpath /opt/virtfusion/ssl/ssl.crt \ --reloadcmd "systemctl reload vf-nginx" \ -d DOMAIN.COM ``` ### 5. Verify the New Domain Visit your control server using the new domain. If you are already logged in, log out and then log back in to confirm everything is functioning correctly. --- ## Cloning a Server (VM) VirtFusion is designed primarily for service providers, where server cloning is not typically part of regular workflows. As such, cloning a server is a **manual process** and requires careful attention. This guide walks you through the steps required to manually clone a server. :::warning The original server must remain offline during the entire disk copy process to avoid data corruption or inconsistencies. ::: ### Prepare a New Server - Create a new server and configure it to mirror the original server. While you may choose different CPU and memory allocations, the **storage size must be identical to that of the original server**. - Build the server from the user area and select **Self Install** as the install method. VirtFusion will now build the new server and create a blank disk. :::info Keep in mind that the self install option automatically enables VNC. If you don't need it, be sure to disable VNC after the build is complete. ::: ### Identify Disk Paths Once the new server has built, you will need to locate the disk paths. - Example original server disk path: ``` /home/vf-data/disk/0c2c37b3-6bf4-47d8-9238-2549d23ac09f_1.img ``` - Example new server disk path: ``` /home/vf-data/disk/622d09c8-11c9-4522-a471-db341e2b19ea_1.img ``` To find the actual paths, go to each server's configuration by selecting them in the admin area. You will see a section called `####### STORAGE #######` and something along the lines of: ``` ``` In this case `/home/vf-data/disk/622d09c8-11c9-4522-a471-db341e2b19ea_1.img` is the path you require. ### Power Off Both Servers :::warning Before copying the disk, you will need to shutdown both servers. You can do this from within VirtFusion. ::: Confirm that the servers are fully stopped before proceeding. ### Copy the Disk Now copy the original server’s disk over the new one using `cp` or `rsync`. This will overwrite the new server’s disk with the original server's data. Example: ``` cp -f /home/vf-data/disk/0c2c37b3-6bf4-47d8-9238-2549d23ac09f_1.img /home/vf-data/disk/622d09c8-11c9-4522-a471-db341e2b19ea_1.img ``` Or, for progress tracking: ``` rsync -a --progress /home/vf-data/disk/0c2c37b3-6bf4-47d8-9238-2549d23ac09f_1.img /home/vf-data/disk/622d09c8-11c9-4522-a471-db341e2b19ea_1.img ``` :::info If the new server is on a different hypervisor, you'll need to transfer the disk over the network or manually download it from the original hypervisor and upload it to the new one. ::: ### Boot the Cloned Server After the disk copy is complete: 1. Start the **new server** from VirtFusion. 2. Confirm it boots correctly and functions as expected. 3. Boot the original server if required. --- ## Protecting the Control Server with Cloudflare You may use Cloudflare to protect your control server. You should set Cloudflare up like any other website. ### Things to Note Make sure VirtFusion has a valid SSL certificate per the installation instructions and enable the `Full (strict)` option for SSL/TLS in Cloudflare. ![cloudflare](img/cf1.png) Do **NOT** enable `Rocket Loader™` under `Speed > Optimization`. It will cause full page loads. ![cloudflare](img/cf2.png) ### Detecting the Users Real IP in VirtFusion All connections to VirtFusion will originate one of the Cloudflare IP ranges, and it will make it difficult to detect the real IP of the connecting user. You can overcome this issue by modifying the control servers NGINX configuration to include the Cloudflare IP ranges. On the control server via SSH: ```shell nano -w /opt/virtfusion/nginx/conf/nginx.conf ``` Add the following in the `http {` section: ``` set_real_ip_from 173.245.48.0/20; set_real_ip_from 103.21.244.0/22; set_real_ip_from 103.22.200.0/22; set_real_ip_from 103.31.4.0/22; set_real_ip_from 141.101.64.0/18; set_real_ip_from 108.162.192.0/18; set_real_ip_from 190.93.240.0/20; set_real_ip_from 188.114.96.0/20; set_real_ip_from 197.234.240.0/22; set_real_ip_from 198.41.128.0/17; set_real_ip_from 162.158.0.0/15; set_real_ip_from 104.16.0.0/13; set_real_ip_from 104.24.0.0/14; set_real_ip_from 172.64.0.0/13; set_real_ip_from 131.0.72.0/22; set_real_ip_from 2400:cb00::/32; set_real_ip_from 2606:4700::/32; set_real_ip_from 2803:f800::/32; set_real_ip_from 2405:b500::/32; set_real_ip_from 2405:8100::/32; set_real_ip_from 2a06:98c0::/29; set_real_ip_from 2c0f:f248::/32; real_ip_header X-Forwarded-For; ``` Then restart NGINX: ```shell systemctl restart vf-nginx ``` The IP ranges above may change from time to time. All the latest Cloudflare ranges can be referenced [here](https://www.cloudflare.com/ips/). --- ## Multiple Control Server Domains ### Generate an SSL Certificate for the Additional Domain Replace `DOMAIN2.COM` with your domain. ```shell acme.sh --issue \ --webroot /opt/virtfusion/nginx/acme \ --keylength 4096 \ --server letsencrypt \ -d DOMAIN2.COM ``` Install the certificate. ```shell acme.sh --installcert \ --keypath /opt/virtfusion/ssl/ssl-DOMAIN2.COM.key \ --fullchainpath /opt/virtfusion/ssl/ssl-DOMAIN2.COM.crt \ --reloadcmd "systemctl reload vf-nginx" \ -d DOMAIN2.COM ``` ### NGINX Configuration Create a new configuration in `/opt/virtfusion/nginx/conf/conf.d/` ```shell nano -w /opt/virtfusion/nginx/conf/conf.d/DOMAIN2.COM.conf ``` With the contents (don't forget to replace the DOMAIN2.COM references): ```shell server { server_name DOMAIN2.COM; listen 443 ssl; listen [::]:443 ssl; http2 on; client_max_body_size 0; index index.php; root /opt/virtfusion/app/control/public; ssl_stapling on; ssl_stapling_verify on; ssl_certificate /opt/virtfusion/ssl/ssl-DOMAIN2.COM.crt; ssl_certificate_key /opt/virtfusion/ssl/ssl-DOMAIN2.COM.key; ssl_dhparam /opt/virtfusion/ssl/dhparam.pem; ssl_session_timeout 10m; ssl_session_cache shared:SSL:10m; ssl_session_tickets off; resolver_timeout 10s; add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"; add_header X-Frame-Options DENY; add_header X-Content-Type-Options nosniff; add_header X-XSS-Protection "1; mode=block"; ssl_ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA512:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:ECDH+AESGCM:ECDH+AES256:DH+AESGCM:DH+AES256:RSA+AESGCM:!aNULL:!eNULL:!LOW:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS; ssl_ecdh_curve secp384r1; ssl_prefer_server_ciphers on; ssl_protocols TLSv1.2 TLSv1.3; ssl_buffer_size 1369; include /opt/virtfusion/nginx/conf/gzip.conf; include /opt/virtfusion/nginx/conf/error.conf; add_header X-Frame-Options "SAMEORIGIN"; add_header X-Content-Type-Options "nosniff"; charset utf-8; location = /favicon.ico { log_not_found off; access_log off; } location = /vnc/ { proxy_pass http://vnc-backend; proxy_read_timeout 360; proxy_connect_timeout 60; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache_bypass $http_upgrade; } location = /ws/ { proxy_pass http://ws-backend; proxy_read_timeout 360; proxy_connect_timeout 60; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache_bypass $http_upgrade; } location /admin { error_page 418 /error.html; fastcgi_buffer_size 128k; fastcgi_buffers 256 4k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_pass unix:/opt/virtfusion/php8/socket/control_admin.sock; fastcgi_intercept_errors on; include /opt/virtfusion/nginx/conf/fastcgi.conf; fastcgi_param SCRIPT_FILENAME $document_root/index.php; } location / { error_page 418 /error.html; fastcgi_buffer_size 128k; fastcgi_buffers 256 4k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_pass unix:/opt/virtfusion/php8/socket/control_front.sock; fastcgi_intercept_errors on; include /opt/virtfusion/nginx/conf/fastcgi.conf; fastcgi_param SCRIPT_FILENAME $document_root/index.php; } location ~* \.(php)$ { return 404; } location ~* \.(ico|gif|svg|jpg|jpeg|png|js|css|woff|woff2|webmanifest|txt)$ { try_files $uri =404; expires max; log_not_found off; add_header Pragma public; add_header Cache-Control "public, must-revalidate, proxy-revalidate"; } } ``` Reload NGINX: ```shell systemctl reload vf-nginx ``` ### Add the Trusted Host ```shell vfcli-ctrl trusted-hosts add DOMAIN2.COM ``` ### Additional Notes If this is the first additional domain you have added. You will need to edit `/opt/virtfusion/nginx/conf/conf.d/ui.conf` and make a small change. Find the section that starts with: ```shell server { server_name _; listen 443 ssl; listen [::]:443 ssl; ........ ``` And replace the underscore in `server_name _` to your main domain and reload vf-nginx. --- ## Custom ISO VirtFusion 3.1+ supports custom ISO URLs for end users. Custom ISO's will be streamed from a remote server (not downloaded on the host). :::warning Important Custom ISO support is disabled by default. Every effort has been made to make sure this feature is as secure as possible, however, the ISO is in the control of the end user. There is no guarantee that the ISO loaded will be a legitimate ISO image. ::: ![iso](img/iso3.png) ### Enabling Custom ISO support :::info If any of your hypervisors were installed before Friday 24th November 2023, they may be missing an important package that is required for this feature to function correctly. **Debian/Ubuntu** ```shell apt-get install qemu-block-extra ``` **RHEL** ```shell dnf install qemu-kvm-block-curl ``` ::: #### Globally You can enable Custom ISO support globally for all servers that inherit this global setting (servers inherit by default) by enabling `Settings -> Virtualization -> Custom ISO`. ![iso](img/iso1.png) #### Per Server You can enable (or disable) Custom ISO on a per server basis `Server -> Settings -> General -> OS -> Custom ISO` ![iso](img/iso2.png) --- ## Custom Icons ## Adding Custom OS Template/Group Icons You have the ability to add up to 20 of your own icons (logos) for use in operating system templates and groups. Custom icons should be uploaded to `/opt/virtfusion/app/control/public/img/logo` on the control server and should have the following file name structure: ``` custom1_logo.png custom2_logo.png ... custom19_logo.png custom20_logo.png ``` All icons must be `.png` and should be kept fairly small in dimension (512px is plenty). They should also be of a uniform size, equal dimensions work best. Once you've uploaded the icons you need, you can set them under the template or template groups settings. ![logo](img/logo1.png) --- ## Debian 11 Backports ### Install the latest version of Libvirt & QEMU from backports ```shell echo "deb http://deb.debian.org/debian bullseye-backports main" > /etc/apt/sources.list.d/bullseye-backports.list apt update apt install -t bullseye-backports --no-install-recommends qemu-system qemu libvirt-clients libvirt-daemon-system ``` --- ## DHCP VirtFusion has the ability to run a DHCP server on each hypervisor in a cluster to serve the primary IPv4 of virtual servers. This is ideal if you boot from a live CD or do an install from ISO. Previously [Kea](https://www.isc.org/kea/) was used to serve requests until April 2023, however [Dnsmasq](https://thekelleys.org.uk/dnsmasq/doc.html) is now the new default. ### Dnsmasq Setup (for those using Kea) Stop Kea on the hypervisor: ```shell systemctl stop vf-kea-dhcpd4 systemctl disable vf-kea-dhcpd4 ``` If the file `/home/vf-data/conf/config.json` doesn't exist on the hypervisor, you may do the following: ```shell mkdir -p /home/vf-data/conf echo '{"dhcp4":{"service":"dnsmasq"}}' > /home/vf-data/conf/config.json ``` If the config file already exists, edit and add: ```json "dhcp4": { "service": "dnsmasq" } ``` Install Dnsmasq for Debian/Ubuntu hypervisors: ```shell apt install dnsmasq ``` Install Dnsmasq for RHEL hypervisors: ```shell dnf install dnsmasq ``` Edit `/etc/dnsmasq.conf` and add the following: ``` conf-dir=/home/vf-data/conf/dnsmasq/,*.conf ``` Make sure the config directory exists: ```shell mkdir -p /home/vf-data/conf/dnsmasq ``` Then start Dnsmasq. ```shell systemctl enable dnsmasq systemctl restart dnsmasq ``` ### ISC DHCP Setup As an alternative, [ISC DHCP](https://www.isc.org/dhcp/) is also supported. Stop Kea on the hypervisor: ```shell systemctl stop vf-kea-dhcpd4 systemctl disable vf-kea-dhcpd4 ``` If the file `/home/vf-data/conf/config.json` doesn't exist on the hypervisor, you may do the following: ```shell mkdir -p /home/vf-data/conf echo '{"dhcp4":{"service":"isc-dhcp"}}' > /home/vf-data/conf/config.json ``` If the config file already exists, edit and add: ```json "dhcp4": { "service": "isc-dhcp" } ``` Install ISC DHCP for Debian/Ubuntu hypervisors: ```shell apt install isc-dhcp-server systemctl enable isc-dhcp-server systemctl restart isc-dhcp-server ``` Install ISC DHCP for RHEL hypervisors: ```shell dnf install dhcp-server systemctl enable --now dhcpd systemctl restart dhcpd ``` --- ## Disaster Recovery import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; VirtFusion 2.3+ supports DR (backup/restore) to minimize downtime and data loss in the event of a disaster. Data can either be stored in S3 compatible object storage or on a local disk partition. For a list of S3 compatible providers, [see here](#s3-compatible-storage-providers). This may look complicated, but it really isn't. VirtFusion will try it's very best to produce consistent backups that can be restored easily. You just need to **configure it on the hypervisor** the way you want. :::caution Although disaster recovery is supported on hypervisors that utilize shared storage, it will only create backups of locally stored disks and configurations. All shared storage (i.e Ceph) disks will be ignored. ::: It's expected that later versions of VirtFusion will incorporate a GUI component for disaster recovery. ### Hypervisor Preparation #### S3 You only need to follow this step if you want to use object storage. ##### Installing the AWS CLI tools To install the CLI tool, `unzip` is required on the hypervisor. It should be installed using your distro package manager. i.e `apt install unzip` or `dnf install unzip`. Once you have unzip, install the tool. **x86_64** Run the following commands as a root SSH user. ```shell cd /tmp curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.22.35.zip -o awscliv2.zip unzip awscliv2.zip ./aws/install rm -rf /tmp/aws rm -f /tmp/awscliv2.zip ``` **AArch64** Run the following commands as a root SSH user. ```shell cd /tmp curl https://awscli.amazonaws.com/awscli-exe-linux-aarch64-2.22.35.zip -o awscliv2.zip unzip awscliv2.zip ./aws/install rm -rf /tmp/aws rm -f /tmp/awscliv2.zip ``` The official AWS instructions can be found [here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). ##### Configuring the AWS config profile Create the file `/root/.aws/config`. ```shell mkdir -p /root/.aws nano -w /root/.aws/config ``` In that file you can configure your access credentials for your S3 compatible service. ```shell [profile virtfusion] aws_access_key_id=ACCESS_KEY aws_secret_access_key=SECRET_ACCESS_KEY retry_mode = standard max_attempts = 3 s3 = #max_concurrent_requests = 20 #max_bandwidth = 5MB/s addressing_style = path ``` You should leave the profile name as `virtfusion`. If you would like to explore various other options, you can find them at https://docs.aws.amazon.com/cli/latest/topic/s3-config.html #### Local Storage Nothing extra needs to be configured for local storage. #### Configuring VirtFusion Create the file `/home/vf-data/conf/dr.json`. ```shell nano -w /home/vf-data/conf/dr.json ``` In that file you can configure your S3 bucket, endpoint and region or/and your local storage. ```json { "type": "S3", "storeDirType": "once", "tmpDir": "home/vf-data/tmp", "threads": 1, "S3": { "endpoint": "https://s3.eu-west-1.wasabisys.com", "region": "eu-west-1", "bucket": "backups", "compress": true, "compressType": "lz4", "compressLevel": 1, "compressThreads": 1, "prune": { "enabled": true, "hoursGrace": 168, "keepLast": 30 } }, "local": { "path": "backups", "compress": true, "compressType": "lz4", "compressLevel": 1, "compressThreads": 1, "prune": { "enabled": true, "hoursGrace": 168, "keepLast": 30 } } } ``` - `type` should be either `S3` or `local` - `storeDirType` may be one of the following: | Type | Example Path | Description | |:----------|:------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------| | `once` | `virtfusion_dr/[HV_ID]/once` | The once type will create a static directory named once, meaning each time a backup runs, the files will be overwritten. | | `Y-M-D-H` | `virtfusion_dr/[HV_ID]/y_m_d_h/[2023_06_01_04]` | Typically this will create a new directory every hour. | | `Y-M-D` | `virtfusion_dr/[HV_ID]/y_m_d/[2023_06_01]` | This will create a new directory each day. | | `Y-M` | `virtfusion_dr/[HV_ID]/y_m/[2023_06]` | This will create a new directory each month. | | `Y-W` | `virtfusion_dr/[HV_ID]/y_w/[2023_03]` | This will create a directory weekly (Weeks starting on Monday). | | `DW` | `virtfusion_dr/[HV_ID]/dw/[1-7]` | This will create a directory for the day number of the week (1 for Monday, 7 for Sunday). | All date style types are based on UTC. - `tmpDir` is where temporary snapshots will be stored. You don't need to include a `/` at the start or end. - `threads` how many backups can be running at any one time. The option is currently ignored and only 1 thread will be used until a future version. - `S3.endpoint` should be the URL to your S3 service. - `S3.region` is the S3 region of your S3 service. - `S3.bucket` should be the name of the bucket you created to store the backups. - `S3.compress` will enable compression on server disks. Enabling compression will usually slow down the backup. - `S3.compressType` default is `lz4`. You can choose between `lz4` and `zstd`. process, especially on a fully loaded system, but saves storage space. **It's advised to keep this option enabled**. - `S3.compressLevel` (1 to 19) 1 = fast (default), 19 = slow. Be warned - 19 will be better compression, but it will be very very very very slow! **It's advised to keep this option set to 1**. - `s3.compressThreads` (0 to 20). default = 1. 0 = core count. (only zstd). - `S3.prune.enabled` can be true or false. This option will enable or disabled backup pruning. - `S3.prune.hoursGrace` is only used for the once `storeDirType` setting and will wait the specified number of hours before deleting files that no longer belong to the hypervisor. - `S3.prune.keepLast` is used for all date style `storeDirType` settings and is used to specify how many of the newest folders should be kept on storage. - `local.path` should be the local path where you want to store the backups. You don't need to include a `/` at the start or end. - `local.compress` will enable compression on server disks. Enabling compression will usually slow down the backup. process, especially on a fully loaded system, but saves storage space. **It's advised to keep this option enabled**. - `local.compressType` default is `lz4`. You can choose between `lz4` and `zstd`. - `local.compressLevel` (1 to 19) 1 = fast (default), 19 = slow. Be warned - 19 will be better compression, but it will be very very very very slow! **It's advised to keep this option set to 1**. - `local.compressThreads` (0 to 20). default = 1. 0 = core count. (only zstd). - `local.prune.enabled` can be true or false. This option will enable or disabled backup pruning. - `local.prune.hoursGrace` is only used for the once `storeDirType` setting and will wait the specified number of hours before deleting files that no longer belong to the hypervisor. - `local.prune.keepLast` is used for all date style `storeDirType` settings and is used to specify how many of the newest folders should be kept on storage. :::info It's probably wise to backup both `/root/.aws/config` and `/home/vf-data/conf/dr.json` for easy access in case of a fatal hypervisor failure. ::: ### Creating Backups :::caution Important - If you run a backup manually, it's highly advised to run the command within a screen session. - Even though signals are caught and the DR routine will attempt to exit cleanly, resist the urge to ctrl+c when a backup is running. Due to the nature of external snapshots, this may cause inconsistency on disk, which can be hard to repair. - When a backup runs on a server disk, some of the server actions like reboot, reinstall, shutdown etc... will be blocked at hypervisor level. Once the backup process has been completed on that disk, the actions will be unlocked. ::: #### Generate a list of files that will be included in a backup (with filesize estimates) ```shell vfcli-hv dr:backup --estimate-only ``` It will generate something similar to this. ``` +---------------------------------------------------------------+---------+-----------+ | Item | Type | Size | +---------------------------------------------------------------+---------+-----------+ | /home/vf-data/disk/37fc81d8-402c-41a1-8f7b-5f421c7e2d20_1.img | file | 3.81 GB | | /home/vf-data/disk/d33ab991-ec0e-491b-b38a-0999f01bbfc9_1.img | file | 6.76 GB | | /home/vf-data/disk/6ba9b79c-0168-4988-ba6e-b7674c7c2c08_1.img | file | 1.8 GB | | /home/vf-data/disk/6ba9b79c-0168-4988-ba6e-b7674c7c2c08_2.img | file | 6.69 MB | | /home/vf-data/disk/6ba9b79c-0168-4988-ba6e-b7674c7c2c08_3.img | file | 6.69 MB | | /home/vf-data/server | dir | 1.11 MB | | /home/vf-data/stats | dir | 109.85 MB | | /home/vf-data/events | dir | 0 B | | /home/vf-data/conf | dir | 802 B | | /opt/virtfusion/app/hypervisor/storage/stats | dir | 3.04 KB | | /opt/virtfusion/app/hypervisor/storage/logs | dir | 314.9 KB | | /opt/virtfusion/app/hypervisor/storage/nat | dir | 136 B | | /opt/virtfusion/app/hypervisor/conf/auth.json | file | 339 B | | /etc/haproxy/haproxy.cfg | file | 1.35 KB | | /opt/virtfusion/app/hypervisor/database/database.sqlite | file | 12 KB | | /opt/virtfusion/app/hypervisor/database/queue.sqlite | file | 16 KB | |---------------------------------------------------------------|---------|-----------| | Total (approx) | 12.49 GB | +---------------------------------------------------------------+---------+-----------+ ``` #### Backup everything ```shell vfcli-hv dr:backup ``` #### Backup only specific servers ```shell vfcli-hv dr:backup --only-servers=1754,1253,1002 ``` #### Backup everything except disks that already exist on the backup storage ```shell vfcli-hv dr:backup --only-missing-disks ``` #### Backup everything but exclude specific servers ```shell vfcli-hv dr:backup --exclude-servers=1754,1253,1002 ``` `--exclude-servers=` should be a comma seperated list of server ids. #### Backup everything except the system data (only server disks) ```shell vfcli-hv dr:backup --exclude-system-data ``` ### Automating Backups Obviously you will want to automate your backups. You can do this using a cronjob or a systemd timer. #### Cronjob Create a file in `/etc/cron.d/` called `virtfusion_dr` and use any of the following, or add your own. ##### Run daily at 2am ```text PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0 2 * * * root /usr/bin/vfcli-hv dr:backup >/dev/null 2>&1 ``` ##### Run every Friday at 2am ```text PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0 2 * * FRI root /usr/bin/vfcli-hv dr:backup >/dev/null 2>&1 ``` ##### Run on the first day of the month at 2am ```text PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0 2 1 * * root /usr/bin/vfcli-hv dr:backup >/dev/null 2>&1 ``` ##### Run every day of week from Monday through Friday at 2am ```text PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0 2 * * 1-5 root /usr/bin/vfcli-hv dr:backup >/dev/null 2>&1 ``` ### Restoring Backups `SOURCE_PATH` should be the path after the hypervisor ID. For example if you use `Y-M-D` for the `storeDirType`, the path would be `y_m_d/2023_06_01` (swap out the date you want) or if you use `once` it would be `once`. #### Restoring system files ```shell vfcli-hv dr:restore --mode=system --source=once --system-opts=haproxy,eventhooks,config ``` `--system-opts=` should be a comma seperated string of any of the following options. | Option | Description | |:-----------|:-------------------------------------------------| | haproxy | Restore the HAProxy configuration files. | | eventhooks | Restore all custom event hooks. | | config | Restore all system configuration files and logs. | #### Restoring all servers ```shell vfcli-hv dr:restore --mode=server --servers=all --source=[SOURCE_PATH] --power-control=true ``` #### Restoring specific servers You may restore specific servers based on their ID using the `--servers=` argument. ```shell vfcli-hv dr:restore --mode=server --servers=1,5,7,45,193 --source=[SOURCE_PATH] --power-control=true ``` You can also use the following command to fetch a pre-populated `--servers` argument. ```shell vfcli-hv dr:restore --mode=servers-arg ``` Which would output something like the following. ``` --servers=1387,1390,1393,1465,1698,1794 ``` #### Restoring after a disaster (Replacement hypervisor) If the dreaded does happen, you can use DR to restore your backups on to the replacement hypervisor. :::caution Important You should NOT remove the failed hypervisor from VirtFusion. It MUST remain, as this is where your virtual servers are linked. ::: ##### Hypervisor preparation You've now got to the stage where you have a new hypervisor, and it has the base OS installed. The next step is to run the [VirtFusion installer](../installation/hypervisor) as you normally would when installing a new hypervisor. This also includes setting up your network. ##### Configuring access to your backups Once you have VirtFusion installed and a working bridge (or whatever you use for networking), you need to link the new hypervisor to your control server. This can be done using a special DR mode called `get-auth`. Before you can run this mode, you need to setup access to your backup storage. You should follow [this previous step](#hypervisor-preparation). ##### Re-connecting the hypervisor to the control server You should now have working access to your storage. You can run the following command to download the auth.json file that is used to connect the control server to the hypervisor. ```shell vfcli-hv dr:restore --mode=get-auth --source=[SOURCE_PATH] ``` `SOURCE_PATH` must include the hypervisor ID as listed in VirtFusion. For example: `--source=26/y_m_d/2023_06_06` If all goes well, you should see something like the following. ``` root@test:~# vfcli-hv dr:restore --mode=get-auth --source=26/y_m_d/2023_06_06 S3 connection check success! Found backup storage source: virtfusion_dr/26/y_m_d/2023_06_06 Are you sure you would like to continue? (yes/no) [no]: > y download: s3://backup-vf/virtfusion_dr/26/y_m_d/2023_06_06/system/auth.json to ../opt/virtfusion/app/hypervisor/conf/auth.json Downloaded successfully root@test:~# ``` The hypervisor should now be linked again and the test connection button in VirtFusion should report success. :::info Worse case, you can manually download the `auth.json` and save it as `/opt/virtfusion/app/hypervisor/conf/auth.json`. ::: ##### Restoring the system files The following command will restore the VirtFusion system files. ```shell vfcli-hv dr:restore --mode=system --system-opts=all --source=[SOURCE_PATH] ``` `SOURCE_PATH` should NOT include the hypervisor ID. Now the control server is linked, It already knows the ID. For example: `--source=y_m_d/2023_06_06` ##### Restoring the servers The following command will restore all servers and configure them. ```shell vfcli-hv dr:restore --mode=server --servers=all --power-control=true --source=[SOURCE_PATH] ``` That should be it. The `--power-control=true` option should have taken care of booting all the servers (or keeping them turned off if suspended) and they should now be online. ### Pruning Backups You can remove any expired backups using the following command. This command will use the `prune.[value]` settings that were configured in [this section](#configuring-virtfusion). You may also set this command as a cronjob. ```shell vfcli-hv dr:prune ``` ### Custom Scripts You have the option to run custom scripts at specific stages of the backup process. Currently, scripts can be executed at two key moments: either right at the beginning of the backup or immediately after it completes. :::caution Important All scripts must be owned by root; they won't run if owned by any other user. All scripts must terminate with a valid exit code. If a script exits with any code other than 0 (zero), the backup process will be stopped. ::: #### Pre scripts ##### S3 Storage If you're using S3 storage, the following script will be triggered as soon as the disaster recovery process begins. ```text /home/vf-data/conf/dr-backup-s3-pre.sh ``` ##### Local Storage If you're using local storage, the following script will be triggered as soon as the disaster recovery process begins. ```text /home/vf-data/conf/dr-backup-local-pre.sh ``` #### Post scripts ##### S3 Storage If you're using S3 storage, the following script will be triggered just before the disaster recovery process ends. ```text /home/vf-data/conf/dr-backup-s3-post.sh ``` ##### Local Storage If you're using local storage, the following script will be triggered just before the disaster recovery process ends. ```text /home/vf-data/conf/dr-backup-local-post.sh ``` ### Known Issues | Issue | Description | Action/Fix/Workaround | |---------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------| | CentOS/CloudLinux system freeze | The QEMU guest agent freezes the file systems so no single change will be made during the backup, but the guest agent does not respect the loop* devices in freezing order which leads to a hung task and kernel crash. **This is a bug in the guest agent, not VirtFusion**. | Disable `qemu-guest-agent` | ### S3 Compatible Storage Providers A non-exhaustive list of providers. | Provider | America | Europe (EMEA) | Asia (APAC) | |:---------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [iDrive e2](https://www.idrive.com/object-storage-e2/) | Oregon, Los Angeles, Virginia, Chicago, Miami, Dallas, San Jose, Phoenix, Montreal (Canada) | Ireland, London, Madrid, Paris, Frankfurt | Singapore | | [Wasabi](https://wasabi.com/s3-compatible-cloud-storage/) | Oregon, Virginia, Plano (TX), Toronto (Canada) | London, Paris, Amsterdam, Frankfurt | Tokyo, Osaka, Sydney, Singapore | | [Cloudflare R2](https://www.cloudflare.com/lp/pg-r2/) | Western North America, Eastern North America | Western Europe, Eastern Europe | Asia-Pacific | | [Backblaze B2](https://www.backblaze.com/b2/cloud-storage.html) | US East, US West | EU Central | - | | [Contabo](https://contabo.com/en/object-storage/) | United States | Germany | Singapore | | [Scaleway](https://www.scaleway.com/en/object-storage/) | - | Amsterdam, Paris, Warsaw | - | | [DigitalOcean](https://www.digitalocean.com/products/spaces) | New York City, San Francisco | Amsterdam, Frankfurt | Singapore, Sydney | | [Huawei OBS](https://www.huaweicloud.com/intl/en-us/product/obs.html) | Mexico City, Santiago, Sao Paulo | Johannesburg, Istanbul | Bangkok, Singapore, Shanghai, Beijing, Guangzhou, Hong Kong | | [Vultr](https://www.vultr.com/products/object-storage/) | New Jersey, Silicon Vally | Amsterdam | Delhi, Bangalore, Singapore | | [OVH](https://www.ovhcloud.com/en-gb/public-cloud/object-storage/) | - | Gravelines, Strasbourg, Frankfurt, Beauharnois, Roubaix, Warsaw, London | - | | [IONOS](https://cloud.ionos.com/storage/object-storage) | - | Frankfurt, Berlin, Logrono (Spain) | - | | [Alibaba OSS](https://www.alibabacloud.com/product/object-storage-service) | Silicon Valley, Virginia, | Frankfurt, London, Dubai | Hangzhou, Shanghai, Nanjing, Qingdao, Beijing, Zhangjiakou, Hohhot, Ulanqab, (Shenzhen, Heyuan, Guangzhou, Chengdu, Hong Kong, Tokyo, Seoul, Singapore, Sydney, Kuala Lumpur, Jakarta, Manila, Bangkok, Mumbai | | [DreamHost DreamObjects](https://www.dreamhost.com/cloud/storage/) | US East | - | - | | [Amazon S3](https://aws.amazon.com/s3/) | Ohio, N. Virginia, N. California, Oregon, Canada, São Paulo | Cape Town, Frankfurt, Ireland, London, Milan, Paris, Stockholm, Spain, Zurich, Bahrain, UAE | Hong Kong, Hyderabad, Jakarta, Melbourne, Mumbai, Osaka, Seoul, Singapore, Sydney, Tokyo, Beijing, Ningxia | You can also host your own with solutions like [MinIO](https://min.io/). --- ## Elevate VirtFusion 4.1+ supports the new Elevate functionality. When you elevate a server, it will allow you to make changes to the server without making them final. If the changes are successful, you commit those changes. If they don't work as expected, you discard the changes and the server is reverted back to the point of elevation. ### Elevating a server Navigate to the server you would like to elevate in the administration area. You will see a button on the power panel. ![elevate](img/elevate1.png) Once you have clicked that button, you will be presented with a popup. ![elevate](img/elevate2.png) Select the drives you would like to elevate and click the `Elevate` button. A task will be added to the queue and the server will be elevated. Once elevated, the elevate control area will display. ![elevate](img/elevate3.png) You can now make changes to the server. ### Committing changes Once you have finished making your changes to the server and are happy that everything is how you want it, you can commit the changes. Click the `Commit` button for each drive to make the changes complete. Once all drives have been committed, the elevation is finished. ### Discarding changes If your changes did not go to plan, you can discard the changes and revert back to the state the server was at the point of elevation. Click the `Discard` button to discard all changes for all disks. --- ## Server Event Hook Examples ### Mitigating a Libvirt Bug RHEL introduced a UEFI bug in one of their Libvirt builds. We can use a hooks to fix this issue. **Hook Filename**: `server-before-boot_php_0001_fix-rhel-libvirt.event` **Hook Contents**: ```php settings->uefi) { if (!file_exists('/home/vf-data/server/' . $data->uuid . '/OVMF_VARS.fd')) { copy('/home/vf-data/lib/uefi/ovmf/OVMF_VARS.fd', '/home/vf-data/server/' . $data->uuid . '/OVMF_VARS.fd'); } } } ``` --- ## Server Event Hooks ## Server event hooks All server hook points have access to the full server object. The object may differ between servers, so it's wise to check a property exists before trying to access it. ### Example server object ```json { "id": 499, "ownerId": 1, "hypervisorId": 26, "name": "My Server 1", "hostname": null, "commissionStatus": 3, "uuid": "c2ee9c72-6245-4f11-a0ae-e6c847110ff1", "state": "complete", "migrateLevel": 0, "deleteLevel": 0, "configLevel": 0, "backupLevel": 0, "rebuild": false, "suspended": false, "protected": false, "buildFailed": false, "primaryNetworkDhcp4": false, "primaryNetworkDhcp6": false, "built": "2022-06-25T22:00:39+00:00", "created": "2022-06-25T21:59:53+00:00", "updated": "2022-07-18T14:33:05+00:00", "traffic": { "public": { "countMethod": 1, "currentPeriod": { "start": "2022-06-25T00:00:00.000000Z", "end": "2022-07-24T23:59:59.999999Z", "limit": 10 } } }, "settings": { "osTemplateInstall": true, "osTemplateInstallId": 3, "encryptedPassword": "eyJpdiI6IkVXM3ZVTWlObTRaQ1Nvb3RzclY3dlE9PSIsInZhbHVlIjoiNFBRdU04ZWtVOGloVFpYQXo5NHlEdmdrdjlxZGhzQU9xbFozY2V0VmN0Yz0iLCJtYWMiOiJmZTc5NDYzMjZhMDM4ZTBjOWM3ZGU3MDQ3YzJhMzU0ZGEyNTA4MTEyMmU5YTlmYmIzMTEyZjc3MmFiYzFjNjJmIiwidGFnIjoiIn0=", "backupPlan": 1, "uefi": false, "cloudInit": true, "cloudInitType": 1, "config": { "cloud.init": { "on.all": { "user.data": { "package_update": true, "package_upgrade": true, "mounts": [ [ "vdb", "/data1", "auto", "defaults,nofail", "0", "0" ], [ "vdc", "/data2", "auto", "defaults,nofail", "0", "0" ] ] } }, "on.password": { "user.data": [] }, "on.sshkey": { "user.data": [] }, "on.network": [], "on.network.libvirtrouted": [] } }, "userConfig": [], "bootOrder": [ "hd", "cdrom" ], "resources": { "memory": 1024, "storage": 10, "traffic": 10, "cpuCores": 1 } }, "media": { "isoMounted": false, "isoName": "", "isoFilename": "", "isoUrl": "", "isoDownload": false }, "vnc": { "ip": "10.0.0.10", "encryptedPassword": "eyJpdiI6Imhtd1dGQlJWaDZvUXN5bkU0emRib2c9PSIsInZhbHVlIjoiS1Y5TzdqSEFCRmtqSjVaKzF4eEdldz09IiwibWFjIjoiNzNlZjJjYjMxNjAwYzNiYWE5MDRlMGU5MmJiN2ZiZGVkMWJlNTQyZmVlODA2ZjM4MTQwNjI0NTc4MjcyZjkxNyIsInRhZyI6IiJ9", "port": 5906, "enabled": false }, "network": { "interfaces": [ { "order": 1, "enabled": true, "tag": 8527570150, "name": "eth0", "type": "public", "driver": null, "mac": "00:F5:5C:5E:26:76", "ipv4ToMac": null, "ipv6ToMac": null, "inTrafficCount": true, "outTrafficCount": false, "inAverage": 0, "inPeak": 0, "inBurst": 0, "outAverage": 0, "outPeak": 0, "outBurst": 0, "ipFilter": true, "ipFilterType": "4", "portIsolated": true, "ipv4_resolver_1": 1, "ipv4_resolver_2": 2, "ipv6_resolver_1": 1, "ipv6_resolver_2": 2, "networkProfile": 0, "dhcpV4": 0, "dhcpV6": 0, "hypervisorNetwork": 1, "hypervisorConnectivity": { "id": 1, "type": "simpleBridge", "bridge": "br0", "mtu": null, "primary": true, "default": true }, "ipWhitelist": [], "actions": [], "ipv4": [ { "id": 81, "order": 1, "enabled": true, "address": "10.0.1.23", "gateway": "10.0.1.1", "netmask": "255.255.255.0", "resolver1": "8.8.8.8", "resolver2": "8.8.4.4", "rdns": null, "mac": null } ], "ipv6": [] } ], "secondaryInterfaces": [ { "order": 1, "enabled": true, "tag": 6384380578, "name": "eth1", "type": "private", "driver": null, "mac": "00:0C:F6:FB:59:AC", "ipv4ToMac": null, "ipv6ToMac": null, "inTrafficCount": true, "outTrafficCount": false, "inAverage": 0, "inPeak": 0, "inBurst": 0, "outAverage": 0, "outPeak": 0, "outBurst": 0, "ipFilter": true, "ipFilterType": "6", "portIsolated": true, "ipv4_resolver_1": 1, "ipv4_resolver_2": 2, "ipv6_resolver_1": 1, "ipv6_resolver_2": 2, "networkProfile": 0, "dhcpV4": 0, "dhcpV6": 0, "hypervisorNetwork": 1, "hypervisorConnectivity": { "id": 1, "type": "simpleBridge", "bridge": "br0", "mtu": null, "primary": true, "default": true }, "ipWhitelist": [], "actions": [], "ipv4": [], "ipv6": [ { "id": 1906, "order": 1, "enabled": true, "addresses": [ "2a0f:93c1:26:2fb:5::a" ], "addressesDetailed": [ { "address": "2a0f:93c1:26:2fb:5::a", "rdns": null } ], "subnet": "2a0f:93c1:26:2fb:5::", "cidr": 80, "gateway": "2a0f:93c1:26::1", "resolver1": "2001:4860:4860::8888", "resolver2": "2001:4860:4860::8844" } ] } ] }, "storage": [ { "_id": 528, "id": 1, "cache": null, "bus": null, "capacity": 10, "drive": "a", "filesystem": null, "iops": { "read": null, "write": null }, "bytes": { "read": null, "write": null }, "type": "qcow2", "profile": 0, "status": 3, "enabled": true, "primary": true, "created": "2022-06-25T21:59:53+00:00", "updated": "2022-07-04T17:27:36+00:00", "name": "c2ee9c72-6245-4f11-a0ae-e6c847110ff1_1", "filename": "c2ee9c72-6245-4f11-a0ae-e6c847110ff1_1.img", "hypervisorStorageId": 11, "local": true, "locationType": "mountpoint", "path": "/data1/disk1" } ], "hypervisor": { "id": 26, "ip": "10.0.0.10", "hostname": null, "maintenance": false, "groupId": 1, "backupStorageType": 2, "defaultDiskType": "inherit", "defaultDiskCacheType": "none", "created": "2021-05-21T10:31:19+00:00", "updated": "2022-05-23T08:12:39+00:00", "name": "Test Hypervisor 1", "dataDir": "/home/vf-data", "encryptedToken": "eyJpdiI6InpvT0JaUWJyK3kxY3pXQ1VuYjM5dkE9PSIsInZhbHVlIjoialBiNnVxM1cvQncwSW42eHlPNEtxZzVyM2FhSzl6TEtiRXdYdFBmMnU1U2RqZmhEdktWVFJlbktPYU5RVnZ5MHBKWHNxNEhTeExxWThUZzl4V1dHc1hBMDVWNHAvNXp0T1lSYmowSHBPdnpGVDY1WFkxSUpmQWN2bEltT2hORU9hbnBlSDRDT1pkTTFBa2Z4TThOWmQ2cGo2QTh6clZsb2M4VDJ1Y2xhZXVUTzRybzIvai9IanI1bEZPU296T1REaHQvb1Jjc05zb21LVWZtYWdkT2ZXUnNFMkUrSm1pNGc3eTF4SHNyQUpiZGx3Tko5Mk9ZeG5Sc3VKOWpKVWxUTVp0enlrc3FkS0x0ZkU4R2w0UUJJbDdjbW9qbkpUc1YzeHoyWEJXTkJEcFdjUHUxSThWa2xhczhNMGhMbTZCaWVQZUNSNm51ZGQ3SFd5TEZGK3ZFZXV5YTVPS3FBZTFMRTIxYzdJand0ZnVlWU9BYzQvM20yQzdUODJKeVFQZEdpdzM0Q1NjQzFvNkRLZ05iUDZpOW5wRXU2L28zTXRhalQ1S1NLUURuK1Q1MTZHazR4Wk5Qc1d0Ym9WaWRSLzl5akFrTm5XaEVZSjcvN09tZW1wVVhHVkNZTGhCUHREbStCeVJ4Q3R4N3NGUXkySVNnU2R5cHMzK0ZmVjlOSDdUcjNnL3Q5bGZ3TTZjUGRkVStiN3ByTVZGWkVXRmFYRk8zVVpQOWdXT1hGVVJ0aGdQS25iYm5BNk1HNVNzSysweG0zVFZxWEIxMGJUN015Q1pNWlR5L0pkWk91ZkJmKzVNSkdpMlVKbWVlWHhMeEtlbGZYUkswSy9qVjhmc0Fzd2F3TFVoVkR1QUR1dmU0MC9CTVplbkM4QVhiUzBHOTNlZ0RDYXdmaTNkL1NFbVU9IiwibWFjIjoiNWIwNDZiNTAxMzE4YzY1YzlhZGU5YTIwOGYxODViMjgxN2Q1ZTI1NjM4ZjRiOTdmZjc3MjJkZDEyZDZlOGQ5ZCIsInRhZyI6IiJ9" }, "owner": { "id": 1, "admin": true, "extRelationId": null, "name": "Harry Potter", "email": "harry.potter@gmail.com", "timezone": "Europe/London", "suspended": false, "twoFactorAuth": false, "created": "2020-12-11T19:54:21+00:00", "updated": "2022-07-12T13:32:03+00:00" }, "sshKeys": [], "xmlConfig": "\n\n\n c2ee9c72-6245-4f11-a0ae-e6c847110ff1\n c2ee9c72-6245-4f11-a0ae-e6c847110ff1\n 1048576\n 1048576\n \n 128\n \n \n \n hvm\n \n \n \n \n \n \n \n \n \n \n \n \n \n /usr/bin/qemu-system-x86_64\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n destroy\n restart\n restart\n\n" } ``` ### Example hook All PHP based hook points must have an `event($data)` function and must NOT contain a closing tag (?>). ```php server-before-config-write | Triggered just before the server XML configuration is written to disk. | 1.7.2 | | 
server-after-config-write
| Triggered just after the server XML configuration has been written to disk. | 1.7.2 | | 
server-before-boot
| Triggered just before the server is booted. | 1.7.2 | | 
server-after-boot
| Triggered just after the server has been booted. | 1.7.2 | | 
server-before-poweroff
| Triggered just before the server is powered off. | 1.7.2 | | 
server-after-poweroff
| Triggered just after the server has been powered off. | 1.7.2 | | 
server-before-shutdown
| Triggered just before the server is shutdown. | 1.7.2 | | 
server-after-shutdown
| Triggered just after the server has been shutdown. | 1.7.2 | | 
server-before-suspend
| Triggered just before the server is suspended. | 1.7.2 | | 
server-after-suspend
| Triggered just after the server has been suspended. | 1.7.2 | | 
server-before-unsuspend
| Triggered just before the server is unsuspended. | 1.7.2 | | 
server-after-unsuspend
| Triggered just after the server has been unsuspended. | 1.7.2 | | 
server-before-cloudinit-config
| Triggered just before the cloud-init config is written to disk. | 1.7.2 | | 
server-after-cloudinit-config
| Triggered just after the cloud-init config has been written to disk. | 1.7.2 | | 
server-before-cloudinit-attach
| Triggered just before the cloud-init drive is attached. | 1.7.2 | | 
server-after-cloudinit-attach
| Triggered just after the cloud-init drive has been attached. | 1.7.2 | | 
server-before-traffic-throttle-primary
| Triggered just before the server primary interface is throttled. | 1.7.2 | | 
server-after-traffic-throttle-primary
| Triggered just after the server primary interface has been throttled. | 1.7.2 | | 
server-before-traffic-unthrottle-primary
| Triggered just before the server primary interface is unthrottled. | 1.7.2 | | 
server-after-traffic-unthrottle-primary
| Triggered just after the server primary interface has been unthrottled. | 1.7.2 | | 
server-before-insert-iso
| Triggered just before an ISO is inserted. | 1.7.2 | | 
server-after-insert-iso
| Triggered just after an ISO has been inserted. | 1.7.2 | | 
server-before-eject-iso
| Triggered just before an ISO is ejected. | 1.7.2 | | 
server-after-eject-iso
| Triggered just after an ISO has been ejected. | 1.7.2 | | 
server-before-enable-vnc
| Triggered just before VNC is enabled. | 1.7.2 | | 
server-after-enable-vnc
| Triggered just after VNC has been enabled. | 1.7.2 | | 
server-before-disable-vnc
| Triggered just before VNC is disabled. | 1.7.2 | | 
server-after-disable-vnc
| Triggered just after VNC has been disabled. | 1.7.2 | | 
server-before-disk-create-primary
| Triggered just before the primary disk is created (build/rebuild). | 1.7.2 | | 
server-after-disk-create-primary
| Triggered just after the primary disk has been created (build/rebuild). | 1.7.2 | | 
server-before-reset-password
| Triggered just before the password is reset. | 1.7.2 | | 
server-after-reset-password
| Triggered just after the password has been reset. | 1.7.2 | | 
server-before-destroy
| Triggered just before the server is destroyed. | 1.7.2 | | 
server-after-destroy
| Triggered just after the server has been destroyed. | 1.7.2 | | 
server-before-enable-rescue
| Triggered just before rescue mode is enabled. | 6.0.0 | | 
server-after-enable-rescue
| Triggered just after rescue mode is enabled. | 6.0.0 | --- ## About Event hooks Event hooks allow you to run custom code or actions when events occur in VirtFusion. All hooks are specified in files with specific filenames either in `/home/vf-data/events/control` for a control server event or `/home/vf-data/events/hypervisor` for a hypervisor event. ### Hook naming Hooks should be named in a specific format. ```shell {hook name}_{hook langauge}_{hook number}_{hook description}.event ``` `hook name` must be one of the listed hook names. `hook language` is limited to php at this current time. `hook number` should be between 0001 & 9999 and is used to order hooks of the same type in ascending order. `hook description` should be a short description of what the hook does. ### Event hook types #### Hypervisor | Hook Type | |-----------------------------------------| | [Server](./hypervisor/server) | ### Event hook logs Logs for all events can be found in `/opt/virtfusion/app/hypervisor/storage/logs/event-{date}.log` for a hypervisor event or `/opt/virtfusion/app/control/storage/logs/event-{date}.log` for a control server event. Event hooks may also log errors to `/opt/virtfusion/app/hypervisor/storage/logs/app-{date}.log` on a hypervisor under some circumstances. --- ## Custom Favicon ## Replacing the default favicon with your own if you have an unbranded license replacing the default favicon is easy, but you can't just replace the existing one, You are required to create a custom HTML snippet so VirtFusion can find the replacement icons. We recommend you generate your favicon images and code using [https://realfavicongenerator.net](https://realfavicongenerator.net/) Once you've generated your package you can place the custom HTML in `/opt/virtfusion/app/control/public/custom/favicon.html`. Your code should look similar to this. ```html ``` Upload the rest of the icons to `/opt/virtfusion/app/control/public/custom`. Your custom favicon should now show in all areas of VirtFusion. --- ## Hypervisor Assets Starting from version 5.1, VirtFusion allows the configuration of asset sets on a per-hypervisor basis. This feature is especially useful for offering additional resources such as GPUs, USB access, or even basic CPU pinning, providing greater flexibility in resource allocation. ### Configuration In most cases, assets are hardware devices at host level, this means they will need to be configured on a per hypervisor basis. To configure an asset for a particular hypervisor, navigate to `Compute Resources -> Hypervisors -> [Selected Hypervisor] -> Assets`. ![asset](img/asset1.png) To create an asset, click the `Create Asset` button. ![asset](img/asset2.png) Give the asset a name. In this case we will be adding a GPU so we will need to select `Device`. Then click `Create`. Once created the asset will show in the list. ![asset](img/asset3.png) Click the little edit/note icon at the end to configure it. ![asset](img/asset4.png) Fill in all the details and add the correct device XML. In this case we used the `GPU` example and added the correct bus. Click `Update`. :::warning Important If you are adding GPUs, we would expect all your IOMMU grouping to be completed beforehand and know your device IDs. A simple search for `GPU passthrough with libvirt` will yield plenty of results. ::: ### Using an Asset :::info Please Note With assets being at hypervisor level, this will obviously disable the ability to migrate a server while an asset is attached. Just keep this in mind. ::: To add an asset to a server, Select the server (it obviously must be on the hypervisor you added the asset for) and click on the `Other` tab. ![asset](img/asset5.png) Click the `Add` button in the `Hypervisor Assets` section to reveal the list of available assets. ![asset](img/asset6.png) Select the asset and click assign. ![asset](img/asset7.png) The GPU has now been added to the server. A reboot will be required. :::info Please Note In a lot of cases, live inserting a device (especially a GPU) could crash the server. Live functionality will be added at a later date as it requires more testing. ::: ### Automation Assets can be automatically attached to a server at the point of it being created. This is done within a package or directly using the API. **Before an asset can be assigned to a package, it must be grouped.** Navigate back to the list of assets (`Compute Resources -> Hypervisors -> [Selected Hypervisor] -> Assets`) and click the `Groups` tab. ![asset](img/asset8.png) Click the `Create Group` button and give it a name. Because we want to group devices, you should select `Device` ![asset](img/asset9.png) Now we have a group, we need to add the GPU asset we created earlier in to the group. Click the `Assets` tab and edit the GPU. ![asset](img/asset10.png) Select the group that was created and click update. The asset is now in the group. ![asset](img/asset11.png) Now you have a group, you can add that group to a package. Edit a package and scroll to `Hypervisor Asset Groups`. ![asset](img/asset12.png) Select the group you created and click assign. It is also possible to assign multiple instances of the same group. Don't forget to click the Update button at the bottom of the page to save the settings. The group has now been assigned to the package. ![asset](img/asset14.png) Thats it. Each time a server is created with the package, it will search for a hypervisor that contains a free asset in each of the specified group(s). Please see the [API documentation](https://docs.virtfusion.com/api/#api-Servers-Create) for instructions on how to override the default groups on a package or define your own on the server create method. ### Displaying to End Users It is possible to display the attached hardware to the end user. All settings are configurable when editing an asset. The following two fields should be filled in correctly. ![asset](img/asset15.png) And the following two toggles should be set like below. ![asset](img/asset16.png) When the asset is added to a server, it should now show like this. ![asset](img/asset17.png) #### Optional Icons/Logos You can also add an icon if you like. You can have up to 250 icons but you will need to add them to the control server in a specific location and format. In our GPU example we can add the Nvidia logo as follows. Make sure `/opt/virtfusion/app/control/public/custom/img/assets` exists. ```shell mkdir -p /opt/virtfusion/app/control/public/custom/img/assets ``` Inside the assets folder, add a small (250x250(ish)) PNG named `img_1.png`. Once added, you should end up with `/opt/virtfusion/app/control/public/custom/img/assets/img_1.png` Edit the asset in VirtFusion and select `Image 1` from the dropdown and click `Update`. ![asset](img/asset18.png) The end user should now see a logo! ![asset](img/asset19.png) :::info If you need to add more images, you can use the same method and increment the numbering (`img_2.png`, `img_3.png`, `img_4.png` etc...) and selecting the correct Icon/Logo (`Image 2`, `Image 3`, `Image 4` etc...). ::: --- ## Importing KVM Servers ### Introduction Although no import methods exist yet from competing virtualization solutions, it's still possible to migrate data in to VirtFusion. VirtFusion uses the industry standard [cloud-init](https://cloudinit.readthedocs.io/en/latest/) and [guest agent](https://wiki.qemu.org/Features/GuestAgent) (if available) to provision and manage virtual servers, however a large portion of competing products use invasive methods at hypervisor level to achieve the same result. This means virtual servers imported in to VirtFusion will need extra packages if you want to take full control. This guide will help you install those dependencies and move virtual disks in to VirtFusion. :::info This guide presumes you are migrating virtual machines from an existing hypervisor with virtual machines to a newly installed VirtFusion hypervisor. ::: ### Disk Preparation :::caution Disks come in all shapes, sizes and locations and differ in type (qcow2, raw and LVM). Dependant on the software you are using, the location of the disk could be anywhere on the hypervisor filesystem, you'll need to find the location of the disk before you can continue. ::: :::danger ALL DISK ACTIONS SHOULD BE COMPLETED WITH THE VIRTUAL SERVER IN AN OFFLINE STATE. ::: One of the most important parts of migrating a disk, is making sure you know the size and type of the disk you are going to import. If you don't do this, it may cause data corruption and inaccurate statistics in Virtfusion. #### LVM If your disks are stored in logical volumes, you will need to convert them to images. You can convert to disk images easily using the following commands: :::note `/dev/vg_name/lv_name` should be the path to the volume that contains the virtual server data and `/path/to/disk.img` will be the output disk image. ::: ##### Convert to Qcow2 Image ```shell qemu-img convert -O qcow2 /dev/vg_name/lv_name /path/to/disk.img ``` ##### Convert to Raw Image ```shell qemu-img convert -O raw /dev/vg_name/lv_name /path/to/disk.img ``` #### Disk Images ##### Find the Type and Size If the disk is an image (qcow2/raw) or you have converted from LVM, you can easily reference the size and type using the following command: ```shell qemu-img info /path/to/disk.img ``` Example output for a `raw` disk: ``` image: disk.img file format: raw virtual size: 10 GiB (10737418240 bytes) disk size: 4 KiB ``` Example output for a `qcow2` disk: ``` image: disk.img file format: qcow2 virtual size: 10 GiB (10737418240 bytes) disk size: 2.12 GiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false extended l2: false ``` The two important fields are `file format` (disk type) and `virtual size` (disk size). ##### Convert VirtFusion defaults to `qcow2` as it's storage type but also supports `raw` images. If the disk is not in the correct format, you can convert it easily. :::note The following commands will produce a copy of the original disk called `new_disk.img`. ::: ###### Raw to Qcow2 ```shell qemu-img convert -f raw -O qcow2 /path/to/disk.img /path/to/new_disk.img ``` ###### Qcow2 to Raw ```shell qemu-img convert -f qcow2 -O raw /path/to/disk.img /path/to/new_disk.img ```` #### Networking This part can get a little messy due to how some competing products create network configurations within a virtual server. If you would like VirtFusion to take full control of networking, you will need to make sure network configurations are correct. Some OS images ship with a static network configuration baked into the image instead of relying on cloud-init. VirtFusion uses cloud-init to inject the correct network settings (IP addresses, gateway, and DNS) at boot, writing its generated configuration into a drop-in directory. If an imported image already contains a hard-coded network definition it will conflict with or override the cloud-init configuration, and the server may boot with the wrong network settings or no connectivity at all. :::note Check the image's network configuration before importing. A hard-coded interface definition is one of the most common reasons an imported server comes up without its assigned networking. ::: ##### Interface Naming `Predictable Network Interface Device Names` may have been disabled, i.e `net.ifnames=0` and `biosdevname=0` on the kernel command line in `/etc/grub.conf`. This can cause issues with cloud-init configuration where it expects predictable naming. This can mostly be resolved by setting an interface name in VirtFusion `Server -> Network -> Interface Name` and is explained in a later step. ##### Debian Interface File Debian uses `/etc/network/interfaces` for network configuration. An imported image may define its interface directly in this file, for example: ``` auto eth0 iface eth0 inet static address 192.0.2.10/24 gateway 192.0.2.1 ``` Because VirtFusion writes the cloud-init generated configuration into `/etc/network/interfaces.d/`, the main `/etc/network/interfaces` file must be the stock cloud-init compatible version that only sources that directory: ``` source /etc/network/interfaces.d/* auto lo iface lo inet loopback ``` Remove any hard-coded physical interface stanzas so that only the loopback interface and the `source` line remain. Cloud-init then applies its generated configuration cleanly on boot. ##### Other Distributions The same principle applies beyond Debian. Check for and remove hard-coded addresses in whichever network stack the image uses, for example: - Netplan YAML in `/etc/netplan/*.yaml` (Ubuntu) - `ifcfg-*` scripts in `/etc/sysconfig/network-scripts/` (RHEL-based images such as Rocky Linux and AlmaLinux) - NetworkManager connection profiles in `/etc/NetworkManager/system-connections/` In each case, leave network configuration to cloud-init rather than pinning it in the image. #### Install Cloud-Init & Guest Agent It's wise to install cloud-init and the guest agent if possible. The following commands may not work on all disks and depend on the operating system installed on the disk. :::caution Important Once you have completed this step, you should not boot the disk image until it has been imported into VirtFusion. ::: ```shell virt-customize -a /path/to/disk.img --install cloud-init,qemu-guest-agent ``` If you would prefer to install both these applications manually, you may do so using the stock distribution tools (apt, yum, dnf etc...). :::note Windows Windows does not use the Cloud-Init package directly, and uses a solution called [Cloudbase-Init](https://cloudbase.it/cloudbase-init/) which can be downloaded directly from [here](https://cloudbase.it/downloads/CloudbaseInitSetup_Stable_x64.msi) in MSI format. For guest agent installation, you can download the latest version [here](https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-qemu-ga/) in MSI format. ::: ### VirtFusion Preparation #### Create Server To import a virtual server disk in to VirtFusion, you will need to create a new server with the same disk size as the disk you want to import. It's fine to specify `0` IPv4 as they can be added later (steps below) and is the recommended process. #### Build Server Install an operating system on the VM as close as you can to the operating system on the disk you plan to import. Pre-installing a similar operating system will make sure the internal configurations that VirtFusion makes will be a close match as possible. **Once it's built, shut the server down.** #### IP Configuration :::caution Important If you have installed cloud-init in the previous steps, it's important you complete the IP Configuration process. ::: When building the server you may have specified how many IPs you wanted, and they will have been assigned. **The important part here is to trigger cloud-init to only update the IP configuration**. To make cloud-init update the IP configuration you must make an IP change, even if the IPs are correct. Either add a new IP (`Server -> Network -> Assigned IPv4`) or remove an IP and re-add it. This will trigger VirtFusion to run an IP reconfiguration when the VM is started. :::note If your server needs a specific interface name, it can be configured in the network settings (`Server -> Network -> Interface Name`). ::: #### Import Disk Disks are created in `/home/vf-data/disk/` on a hypervisor if you are using the default location and the disk filename is in the format `[UUID]_[DISK_ID].img`. The `UUID` can be obtained from the admin area and the `DISK_ID` should be set to `1` for the primary disk. Well use the UUID `043035a0-5887-4b05-888d-e83ff17f8174` in this example so the disk image path will be `/home/vf-data/disk/043035a0-5887-4b05-888d-e83ff17f8174_1.img`. :::note VirtFusion will always use the `.img` file extension for both `qcow2` and `raw` images and can identify formats internally. ::: Copy the virtual server disk prepared earlier from the old hypervisor to the new VirtFusion hypervisor. You can do this via `SCP`, `rsync` or any other method suitable for transferring data to and from remote locations. The examples below copy the disk into `/home/vf-data/disk/` on the new hypervisor, ready to be renamed in the next step. Adjust the paths, IP address, and SSH user to match your environment. Using `scp`, run from the old hypervisor: ```shell scp /path/to/disk.img root@NEW_HYPERVISOR_IP:/home/vf-data/disk/ ``` Using `rsync`, run from the old hypervisor. `rsync` is recommended for large disks because `-P` shows progress and lets an interrupted transfer resume where it left off: ```shell rsync -avP /path/to/disk.img root@NEW_HYPERVISOR_IP:/home/vf-data/disk/ ``` If the old hypervisor cannot reach the new one directly, run the transfer from the new hypervisor to pull the disk instead: ```shell rsync -avP root@OLD_HYPERVISOR_IP:/path/to/disk.img /home/vf-data/disk/ ``` :::note For a large disk, run the transfer inside `screen` or `tmux` (or with `nohup`) so it continues if your SSH session drops. After the copy completes, you can confirm both files are identical by comparing their checksums with `sha256sum /path/to/disk.img` on each hypervisor. ::: Once the disk is on the new hypervisor, you will need to overwrite the disk created by VirtFusion: ```shell mv /path/to/original/disk.img /home/vf-data/disk/043035a0-5887-4b05-888d-e83ff17f8174_1.img ``` Once copied, you should now be able to boot the server from within VirtFusion. If all goes well and you have cloud-init installed, the networking should be configured automatically and the server will be online. --- ## IPv6 Route Blocks IPv6 Route Blocks allow the assignment of a /128, /127 or /126 subnet with a larger IPv6 subnet. The main use case for this is to allow routing the larger block through the route block on network switches to prevent cache exhaustion. This feature can also be used in conjunction with the `server.assign.ipv6` and `server.unassign.ipv6` webhooks. --- ## Keeping VirtFusion Up to Date ### Services When VirtFusion is initially installed, several important services are installed alongside the main software. It's important to keep these services up to date. :::caution Important Due to how VirtFusion is built and functions, it's not always possible to automate service updates at root level. It is your responsibility to make sure services stay up to date. **When updating any of your hypervisors or control server, it's important to make sure there are no tasks running. Use Maintenance Mode.** ::: #### Control Server The following command will update the most important services required for VirtFusion to run on a control server. ##### Debian/Ubuntu Install updates: ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 vf-control-wss -y ``` Restart services: :::info The following command will restart all the services. If only one service was updated, you only need to restart that service. ::: ```shell systemctl restart vf-php8-fpm vf-control-wss vf-nginx ``` ##### RHEL Install updates: ```shell dnf makecache -y && dnf update vf-nginx vf-php8 vf-control-wss -y ``` Restart services: :::info The following command will restart all the services. If only one service was updated, you only need to restart that service. ::: ```shell systemctl restart vf-php8-fpm vf-control-wss vf-nginx ``` #### Hypervisors The following command will update the most important services required for VirtFusion to run on hypervisors. ##### Debian/Ubuntu Install updates: ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 vf-system-php8 -y ``` Restart services: :::info The following command will restart all the services. If only one service was updated, you only need to restart that service. ::: ```shell systemctl restart vf-php8-fpm vf-system-php8-fpm vf-nginx ``` ##### RHEL Install updates: ```shell dnf makecache -y && dnf update vf-nginx vf-php8 vf-system-php8 -y ``` Restart services: :::info The following command will restart all the services. If only one service was updated, you only need to restart that service. ::: ```shell systemctl restart vf-php8-fpm vf-system-php8-fpm vf-nginx ``` --- ## Localization VirtFusion 2.3 introduces partial support for various languages. All languages apart from english are machine translated using services such as DeepL, Google Translate and Amazon Translate. If you would like to help improve these files, you may fork the latest language files from https://github.com/VirtFusion/app-localization and create a pull request. We are currently evaluating other services to make this task easier. ### Status Language support is only available in the following areas. More sections will be translated in future versions. | | Status | Notes | |:---------|:---------|-------------------------------| | Guest | Complete | Initially machine translated. | | End User | Complete | Initially machine translated. | | Admin | - | | | Email | - | | ### Supported Languages | Code | Name | |:------|:--------------------------------------------------| | ar | العربية (Arabic) | | bg | български (Bulgarian) | | bn | বাংলা (Bangla) | | da | Dansk (Danish) | | de | Deutsch (German) | | el | Ελληνικά (Greek) | | en | English (English) | | en_US | English (United States) (English (United States)) | | es | Español (Spanish) | | fi | Suomi (Finnish) | | fr | Français (French) | | hi | हिन्दी (Hindi) | | hr | Hrvatski (Croatian) | | hu | Magyar (Hungarian) | | id | Indonesia (Indonesian) | | it | Italiano (Italian) | | ja | 日本語 (Japanese) | | ko | 한국인 (Korean) | | nl | Nederlands (Dutch) | | pl | Polski (Polish) | | pt_BR | Português (Portuguese) | | ro | Română (Romanian) | | ru | Pусский (Russian) | | sk | Slovenský (Slovak) | | sl | Slovenščina (Slovenian) | | sr | Српски (Serbian) | | sv | Svenska (Swedish) | | th | ไทย (Thai) | | tr | Türkçe (Turkish) | | uk | Українська (Ukrainian) | | ur | اردو (Urdu) | | vi | Tiếng Việt (Vietnamese) | | zh | 中文 (Chinese) | | zh_TW | 中文 (Chinese (Traditional)) | ### Setup Language support is not enabled by default. It can be enabled under `Settings -> General`. ![](img/lang1.png) Selecting `Enable Localization` will allow end users to select their preferred language on the login form and under their account settings. `Default Langauge` does not need Localization enabled. This setting will set one of the supported languages as the default for the currently supported areas. ### Modifying a Language Although you can't add your own language, you may modify the supported ones. You can create custom overrides and place them in `/opt/virtfusion/app/custom/control/language` on the control server. ( This folder may not already exist and will need to be created). ```shell mkdir -p /opt/virtfusion/app/custom/control/language chown virtfusion. /opt/virtfusion/app/custom/control/language ``` For example. If you would like to override `Welcome Back` and the slogan underneath it (`Login to your dashboard.`) on the login page for the English language. Find the correct `.json` in the GitHub repo. https://github.com/VirtFusion/app-localization/blob/main/en/guest.json Create `/opt/virtfusion/app/custom/control/language/en/guest.json` :::caution If any of the custom language files contain invalid JSON code, they will be ignored and the system defaults will be used. ::: In your custom `guest.json` add the following: ```json { "1": "Hello!", "2": "Dashboard Login" } ``` Those two entries will override the keys that are in the default `guest.json` and will continue to be used even after further updates to the default files. --- ## Mailouts A Mailout allows you to send an email message to multiple recipients at once. It’s commonly used for maintenance notifications, or any other communication that needs to reach your users quickly and efficiently. ### Creating a Mailout Navigate to Mailouts in the admin menu (`System -> Mailouts`) and click the `Create` tab to reveal the main mailout form. ![mailouts](img/mailout1.png) :::info VirtFusion includes several ready to use templates. You can easily select one from the `Predefined Templates` dropdown menu. ::: The following options are available when creating a Mailout: * **`Subject`** This field is required and should contain the subject line of the email. You can use any available variables in the subject, but please avoid using the `*_server_list` variable. * **`Preview Text`** This field is optional and should contain a short snippet of text. Most email clients use this as a glimpse of the email's content before it's opened. * **`HTML`** This field holds the HTML body of the email. You should avoid entering raw HTML directly, as it uses * *Markdown** syntax to automatically generate the HTML when the email is sent. * **`Text`** This field is used for plain text email delivery. It also serves as the text only version of the email when sending HTML emails (optional), ensuring compatibility with email clients that do not support HTML. * **`Predefined Templates`** All your predefined templates and example templates. * **`Recipients`** This will open a popup where you can choose the recipients for the mailout. * **`Reference ID`** An optional reference string that can help recipients identify related Mailouts, especially when multiple messages are sent as part of the same incident. * **`Date #1 (UTC)`** An optional date that will be passed to the `{{ utc_date_1 }}` variable. * **`Date #2 (UTC)`** An optional date that will be passed to the `{{ utc_date_2 }}` variable. * **`Custom #1`** An optional value for the `{{ custom_1 }}` variable. * **`Custom #2`** An optional value for the `{{ custom_2 }}` variable. * **`Custom #3`** An optional value for the `{{ custom_3 }}` variable. Fill in all the relevant fields for your Mailout or select a predefined template. For more information on variables, please see the [Variable Reference](#variable-reference) ![mailouts](img/mailout2.png) Once you have filled in all the fields, you can either save it as a template for later (`Save as Template` button in the bottom bar) or preview it before sending by clicking the `Preview` button in the top bar. It will render the email and pre-fill the variables. :::warning Important Tip! It's always a good idea to preview a mailout before sending. While you can stop a mailout after it starts, some emails may already be delivered before you realise you have made a mistake. ::: #### Selecting Recipients To select recipients, click the orange outlined `Select Recipients` button to open the popup. ![mailouts](img/mailout6.png) The following Recipient Types are available: * **`Hypervisors & Groups`** The mailout will be sent to all users who have at least one server on the selected hypervisors and groups. If a user has multiple servers across the selected targets, they will receive only one email. * **`All Users`** The mailout will be sent to all users on the system based on a criteria selection. * **`Selected Users`** The mailout will be sent to all selected users. Most of the Recipient Types will allow you to select options. For example the `Hypervisors & Groups` will allow you to select multiple hypervisors and hypervisor groups. ![mailouts](img/mailout7.png) After selecting your options, a button will appear allowing you to calculate the number of recipients the mailout will reach. ![mailouts](img/mailout8.png) When you are happy with the selection. Click the `Done` button. #### Sending the Mailout **Review your selected recipients and preview the mailout one final time**. When you're ready, click the `Send` button in the bottom bar. A confirmation popup will appear to confirm that you want to proceed. :::danger Don't see a Send Button? If the `Send` button is not visible, it’s likely that you haven’t selected any recipients or email has not been configured in general settings. ::: ### Managing Mailouts #### Stopping a Mailout If you need to stop a Mailout in progress, click the pause icon in the Mailouts list. This option is only visible when a Mailout is actively being sent. ![mailouts](img/mailout3.png) #### Starting a Previously Stalled Mailout If a Mailout has been stopped, stalled, or encountered failed email deliveries, you can restart it by clicking the play icon in the Mailouts list. ![mailouts](img/mailout4.png) :::info The system ensures that each email is only sent once per Mailout. If any emails fail to send, only those specific ones will be retried. ::: #### Viewing the Mailout Progress Each Mailout shows its progress with a percentage bar while sending. To view the status of individual emails, click the log icon in the Mailouts list. ![mailouts](img/mailout5.png) ### Mailout Settings Mailouts have a limited number of configurable settings via the `Settings` tab, each of which is explained in detail below. * **`Enable Logging`** All emails sent through VirtFusion, including Mailouts, are logged under `Logging -> Email`. If you prefer not to log Mailouts, you can disable this by toggling the setting off. * **`Send Throttling`** The system will pause for the specified duration between sending each email. This helps accommodate SMTP services with strict sending limits. ### Variable Reference The following variables are available in all Mailout types. | Variable | Description | |:------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `{{ html_server_list }}` | List of servers associated to a user based on the recipient selection. This variable should only be used in the HTML section. | | `{{ html_server_list_ip }}` | List of servers (including the main IP) associated to a user based on the recipient selection. This variable should only be used in the HTML section. | | `{{ html_server_list_ul }}` | Unordered (ul) list of servers associated to a user based on the recipient selection. This variable should only be used in the HTML section. | | `{{ html_server_list_ip_ul }}` | Unordered (ul) list of servers (including the main IP) associated to a user based on the recipient selection. This variable should only be used in the HTML section. | | `{{ text_server_list }}` | List of servers associated to a user based on the recipient selection. This variable should only be used in the TEXT section. | | `{{ text_server_list_ip }}` | List of servers associated to a user (including the main IP) based on the recipient selection. This variable should only be used in the TEXT section. | | `{{ reference_id }}` | Value of the `Reference ID` field. | | `{{ custom_1 }}` | Value of the `Custom #1` field. | | `{{ custom_2 }}` | Value of the `Custom #2` field. | | `{{ custom_3 }}` | Value of the `Custom #3` field. | | `{{ utc_date_1 }}` | Value of the `Date #1 (UTC)` field. | | `{{ utc_date_2 }}` | Value of the `Date #2 (UTC)` field. | | `{{ utc_date_1_to_user_timezone }}` | Value of the `Date #1 (UTC)` field converted to the recipients timezone. | | `{{ utc_date_2_to_user_timezone }}` | Value of the `Date #2 (UTC)` field converted to the recipients timezone. | | `{{ user_name }}` | Recipients full name. | | `{{ user_email }}` | Recipients email address. | | `{{ user_time_now_timezone }}` | Time now based on users specified timezone. i.e `Sunday 6 April 21:38:13 BST 2025`. | | `{{ system_company_name }}` | Company name as specified in general settings. | | `{{ system_domain }}` | Domain of the control server. | | `{{ system_url }}` | URL of the control server. | | `{{ system_company_web_url }}` | Website link as specified in general settings. | | `{{ system_company_tos_url }}` | Terms of service link as specified in general settings. | ### Webhooks There are two available Webhook Events for Mailouts. `system.mailout` & `system.mailout.recipient` For more information, see the [Webhook documentation](webhooks). --- ## Moving the Control Server ### Existing control server preparation and backup :::warning The existing control server must be on the latest version. You should fully update before proceeding. ::: #### Stop the scheduler process ```shell systemctl stop vf-schedule.timer ``` #### Make a copy of the main configuration file ```shell cat /opt/virtfusion/app/control/.env ``` Which should output something similar to this: ```shell APP_KEY=base64:vBJSynHOLyOwTVP+MGk7wy0E4CVM/5do/UDB5Ij0qtk= DB_HOST=127.0.0.1 DB_PORT=3306 DB_DATABASE="vf_d_H2aJvrdwS6A67g62cGRw" DB_USERNAME="vf_u_SAZJ8WOZt5cZpy3VdTtN" DB_PASSWORD="PB4oFYFc7ucs07I7bwXTi" WSS_STORE_TYPE=push JWT_SIGNING_TYPE=RS256 JWT_EXPIRE_SECONDS=60 JWT_SECRET_KEY="vtw5pfOZfU2O1ljukJ31oK8yVxh7SC" JWT_PRIVATE_KEY=/opt/virtfusion/app/control/store/jwt_rs256_signing.key JWT_PUBLIC_KEY=/opt/virtfusion/app/control/store/jwt_rs256_signing.key.pub WSS_LISTEN_HOST=127.0.0.1 WSS_LISTEN_PORT=6001 WSS_SECRET_KEY="P4wTuVC8BOYeOgLjcJ5FPAV4wmys8M" WSS_REDIS_HOST=127.0.0.1 WSS_REDIS_PASSWORD=null WSS_REDIS_PORT=6379 ``` The `APP_KEY` is the most vital part of the configuration file. Without the correct key, the database will be of no use. #### Backup the database ```shell mariadb-dump --skip-ssl --user=root `cat /opt/virtfusion/app/control/.env | grep 'DB_DATABASE' | cut -d'"' -f2` > /tmp/database.sql ``` Once the command has completed, a database backup called `database.sql` will be placed in the `/tmp` folder. #### Put the system into maintenance mode ```shell vfcli-ctrl down ``` ### New control server preparation and restore #### Installation Install the control software on the new server as normal but **don't log in** and configure it. :::warning The new installation must be running the exact same version as the old installation. If you were running the testing version on the old installation, you should update the new installation to match. ::: #### Replace the app key Edit `/opt/virtfusion/app/control/.env`: ```shell nano /opt/virtfusion/app/control/.env ``` Replace `APP_KEY` with the `APP_KEY` from the old installation and save the configuration. #### Restore the database backup Copy the database backup you made earlier to the new server and place it in `/tmp`. ```shell mariadb --user=root `cat /opt/virtfusion/app/control/.env | grep 'DB_DATABASE' | cut -d'"' -f2` < /tmp/database.sql ``` #### Restart VirtFusion ```shell vfcli-ctrl app:restart ``` #### SSL certificate A stock installation will generate a self-signed SSL certificate but this is probably not what you need. You can generate a signed set of certificates for your domain at no cost using a free SSL service or use your own custom certificate. ##### Let's Encrypt ###### acme.sh :::caution You must have an A record pointing to the **new** control server IP in your DNS records before proceeding to setup a certificate. As you are most likely reusing the same domain, update the existing record to point to the new server's IP. ::: Install acme.sh ([Official Website](https://github.com/acmesh-official/acme.sh)) ```shell wget -O - https://get.acme.sh | sh . ~/.bashrc ``` The installer will perform 3 actions: 1. Create and copy acme.sh to your home dir `($HOME): ~/.acme.sh/`. All certs will be placed in this folder too. 2. Create alias for: `acme.sh=~/.acme.sh/acme.sh`. 3. Create daily cron job to check and renew the certs if needed. Generate a certificate (replace `DOMAIN.COM` with your own domain). ```shell acme.sh --issue \ --webroot /opt/virtfusion/nginx/acme \ --keylength 4096 \ --server letsencrypt \ -d DOMAIN.COM ``` Install the certificate (replace `DOMAIN.COM` with your own domain). ```shell acme.sh --installcert \ --keypath /opt/virtfusion/ssl/ssl.key \ --fullchainpath /opt/virtfusion/ssl/ssl.crt \ --reloadcmd "systemctl reload vf-nginx" \ -d DOMAIN.COM ``` ##### Custom Certificate The location of the key and certificate should be the following. ``` /opt/virtfusion/ssl/ssl.key /opt/virtfusion/ssl/ssl.crt ``` Once your key and certificate are in place, you simply reload the web server. ``` systemctl reload vf-nginx ``` #### Log in :::info Trusted Hosts If you receive an error stating that your host is not allowed, you may need to reconfigure your trusted hosts. Instructions for editing the allowed hosts can be found [here](../administration/trusted-hosts). ::: Once logged in, you may receive a license error. You must re-issue the license to work on the new server. You may do this by logging in at [account.virtfusion.com](https://account.virtfusion.com/client/login/) and ticking the re-issue box under your license. Finally, rebuild the VNC configuration for the new control server. Navigate to **Settings > General > Misc > VNC Configuration** and click the **Rebuild** button. ### Hypervisors Communication between the control server and hypervisors will be lost due to the ip change from the old server to the new one. You will need to update the authentication file on each hypervisor. Edit the file `/opt/virtfusion/app/hypervisor/conf/auth.json`: ```shell nano -w /opt/virtfusion/app/hypervisor/conf/auth.json ``` It will look similar to this: ```shell { "ip": "192.168.100.19", "token": "CqlsVq6zS4vwLQEZqlVo5tHmkgBzvdZYQ2NJnr5qyB1s2JjgYjSqR72wUYmjGcQbnTf8EWQgHnsqaUU8d8TxhPvaad2fz0zyKMxb5wmfJ9IulHQARYvqHFAabL9I56q7eMQtYSFhtRvNp4K4AmEXYUfW7HiD41fQSHgyIFpaAwP8n1TZVX59AwvIxVAvmlGYnghvvCrt", "hash": "bc1a53dd974e54c903399fcbe8d05550b35842ca052bc583ecb7b0fe4cf64b86" } ``` Replace the `ip` with the new ip of the control server and save the file. --- ## Optimizing Windows Deployment VirtFusion v1.9+ supports Hyper-V Enlightenments for Windows servers. These features make Windows servers think they’re running on top of a Hyper-V compatible hypervisor and use Hyper-V specific features. ### Configuration Optimization settings are located in several areas of the admin interface and work with hypervisor level inheritance. The first place to configure settings is on a per hypervisor basis. #### Hypervisor Level Settings ![windows](img/windows1.png) These hypervisor level settings will be used by a Windows template when a Windows operating system is deployed. #### Template Level Settings ![windows](img/windows2.png) Template level settings can either be inherited from the hypervisor settings or configured independently. #### Server Level Settings ![windows](img/windows3.png) All settings can be configured at server level. Server level settings also contain a description of what each setting does. ### Deploying RDP Ready Servers VirtFusion contains a mechanism to deploy optimized RDP ready servers. #### Configure Template. You can use any of the stock Windows templates by changing the Deployment Profile under the template settings to `Windows Direct RDP` ![windows](img/windows4.png) You may also want to change the description and email excerpt as the new deployment profile will set an Administrator password automatically and email the credentials directly to the user once ready. ![windows](img/windows5.png) You can now deploy servers as normal and the user will be emailed connection details when the build completes. >Your server 'Super Dooper' has been installed with a fresh copy of Windows Server 2022 and is now ready for use. > >You can access your server via Remote Desktop with the following credentials: >Address: 10.100.10.10 > >Port: 3389 > >User: Administrator > >Password: gV6R4N8cMkVNEJjHutRG > >To improve security, we recommend that you change the password on first log in. > >Please note it may take a short while before the above credentials are fully active. > >Thanks, >Demo Cloud Ltd --- ## NAT import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; VirtFusion supports NAT IPv4 addressing in conjunction with native IPv6. This allows you to have a single IPv4 address per hypervisor and manage port and domain forwarding on a per VM basis within VirtFusion. :::info Current Limitations - Can't be used on a control/hypervisor combo due to proxy access on ports 80 & 443 - IPv4 NAT will only function correctly using a VM primary interface. - Migrations are not currently supported. ::: ### Hypervisor Setup #### Prerequisites **Debian** **Install Dnsmasq** ```shell apt install dnsmasq -y ``` **Install HAProxy** ```shell apt install haproxy -y ``` **RHEL** **Install Dnsmasq** ```shell dnf install dnsmasq -y ``` **Install HAProxy** ```shell dnf install haproxy -y systemctl enable haproxy systemctl start haproxy ``` **Ubuntu** **Install Dnsmasq** ```shell apt install dnsmasq -y ``` **Install HAProxy** ```shell apt install haproxy -y ``` #### Configure Network Create a file named network.xml with the following contents. You may change the IP ranges defined, but it's recommended to specify them in /24 ranges for maximum VM OS compatibility. You should also change the device name `dev='eth0'` to suit your hypervisor primary network name. ```xml natbr0 ``` If you would also like native IPv6 on the primary interface you will need to specify it within the configuration. It should be a valid IPv6 address and specified as a `/64` if possible. This will be used as the gateway for your IPv6 range. ```xml ``` Once you have your configuration, you should define the network, set it to autostart and start it. ```shell virsh net-define --file network.xml virsh net-autostart natbr0 virsh net-start natbr0 ``` ### VirtFusion Setup #### IP Blocks Configure the local IP blocks as normal ip blocks. For example `192.168.200.1/24` with a gateway of `192.168.200.1`, netmask of `255.255.255.0` and add the ips as a `/24` subnet. :::info Don't forget to reserve the usual non-usable ips from the range for compliance (.0, .1, .255). ::: #### Hypervisor Network Either create a new network or edit the primary network under the hypervisor settings and set the Type to `NAT v1` and the Interface Name to `natbr0` (or whichever name you used for the network in the XML configuration earlier). ![nat](img/nat-1.png) --- ## NAT Provider Native IPv6 Setup ### Hetzner Hetzner supply a `/64` IPv6 subnet as default with their servers, but you will need to split it into smaller subnets before you can assign IPv6 to virtual servers. We'll use `/80` in this example. Take `2a01:4f9:1a:9f6c::/64` as the example IPv6 subnet. Create a new IPv6 block in VirtFusion with the following settings: `Name` 2a01:4f9:1a:9f6c::/64 `Gateway` 2a01:4f9:1a:9f6c:2::2 `Primary Resolver` 2001:4860:4860::8888 `Secondary Resolver` 2001:4860:4860::8844 `Source Subnet` 2a01:4f9:1a:9f6c:: `Source Subnet CIDR` /64 `Output Subnet CIDR` /80 Once you've saved those settings, add the following `/80` subnets (the first two `/80`s from the `/64`) to the blacklist at the bottom of the page: `2a01:4f9:1a:9f6c:1::` (Used on the hypervisor main interface). `2a01:4f9:1a:9f6c:2::` (Used on the NAT interface). On the hypervisor create a file named network.xml with the following contents. You may change the IPv4 ranges defined, but it's recommended to specify them in /24 ranges for maximum VM OS compatibility. ```xml natbr0 ``` Notice we used the address `2a01:4f9:1a:9f6c:2::2` from the second `/80` subnet, and we will be using it as our gateway. Replace `MAIN_NETWORK_NAME` with the name of the hypervisor main network. You can find it with `ip a`. Still on the hypervisor, you will need to assign the first `/80` to the main interface replacing the `/64` already defined. For example on Debian 11: ``` iface enp41s0 inet6 static #address 2a01:4f9:1a:9f6c::1 address 2a01:4f9:1a:9f6c:1::2 netmask 80 gateway fe80::1 ``` Notice we used the address `2a01:4f9:1a:9f6c:1::2` from the first `/80` subnet. Once you have made the changes, restart the networking. All we need to do now is bring up the NAT network: ```shell virsh net-define --file network.xml virsh net-autostart natbr0 virsh net-start natbr0 ``` ### Misc Notes #### NDPPD Configuration ``` route-ttl 30000 proxy eth0 { router yes timeout 500 ttl 30000 rule xxxx:xxxx:xxxx:xxxx::/XX { static } } ``` #### sysctl.conf ``` net.ipv6.conf.all.forwarding=1 net.ipv6.conf.all.proxy_ndp=1 net.ipv6.conf.default.proxy_ndp=1 net.ipv6.conf.default.autoconf=0 net.ipv6.conf.default.accept_dad=0 net.ipv6.conf.default.accept_ra=0 net.ipv6.conf.default.accept_ra_defrtr=0 net.ipv6.conf.default.accept_ra_rtr_pref=0 net.ipv6.conf.default.accept_ra_pinfo=0 net.ipv6.conf.default.accept_source_route=0 net.ipv6.conf.default.accept_redirects=0 net.ipv6.conf.all.autoconf=0 net.ipv6.conf.all.accept_dad=0 net.ipv6.conf.all.accept_ra=0 net.ipv6.conf.all.accept_ra_defrtr=0 net.ipv6.conf.all.accept_ra_rtr_pref=0 net.ipv6.conf.all.accept_ra_pinfo=0 net.ipv6.conf.all.accept_source_route=0 net.ipv6.conf.all.accept_redirects=0 ``` --- ## Network Filtering (IP Anti-Hijacking) Since version 3.0, VirtFusion has built in native bridge network filtering. The following filters are available: - **`None`** No filters will be applied to any virtual servers on the hypervisor. - **`Libvirt`** A standard set of filters to prevent MAC spoofing, IP stealing and various other filters to clean traffic. - **`Native Standard`** This filter applies the same rule sets as **`Libvirt`**. - **`Native Performance (IPv4)`** A minimal rule set to prevent IPv4 MAC spoofing and IP stealing. IPv6 is not filtered. - **`Native Performance (IPv4 + IPv6)`** A minimal rule set to prevent IPv4 and IPv6 MAC spoofing, IP stealing and various ICMPv6 codes ( including router advertisement). Existing hypervisors (added pre v2.4) will continue to use Libvirt filters until native filters are enabled on a per hypervisor basis. Any hypervisors added post v2.4 will have `Native Performance (IPv4 + IPv6)` filters enabled by default. If you would like to switch your existing hypervisors to native filtering (or vice versa), you may change The `Network Filtering` type under the hypervisor settings. ![network](img/net1.png) Changing this setting will only enable the selected filter type for new servers or servers that have had an action run against them. You can force a configuration and filter update with [this control server CLI command](#rewrite-all-server-filters-by-hypervisor) to force a full change and update. ### Creating a Custom Filter :::info You most likely don't need to do this. VirtFusion by default has an adequate filter for most use cases. ::: At the moment, only a single filter is supported. This means all your rules must be included in a single file and in the correct order. You may override any of the native filters. You can do so by copying the original filter to a custom location. | Filter | Location | |:---------------------------------|:---------------------------------------------------------------------------------------------| | Native Standard | `/opt/virtfusion/app/hypervisor/app/Configurations/NetworkFilter/Native/standard.json` | | Native Performance (IPv4) | `/opt/virtfusion/app/hypervisor/app/Configurations/NetworkFilter/Native/performance.json` | | Native Performance (IPv4 + IPv6) | `/opt/virtfusion/app/hypervisor/app/Configurations/NetworkFilter/Native/performance-v6.json` | To copy the Standard filter. ```shell cp /opt/virtfusion/app/hypervisor/app/Configurations/NetworkFilter/Native/standard.json /home/vf-data/conf/netfilters/standard.json ``` To copy the Native Performance (IPv4) filter. ```shell cp /opt/virtfusion/app/hypervisor/app/Configurations/NetworkFilter/Native/performance.json /home/vf-data/conf/netfilters/performance.json ``` To copy the Native Performance (IPv4 + IPv6) filter. ```shell cp /opt/virtfusion/app/hypervisor/app/Configurations/NetworkFilter/Native/performance-v6.json /home/vf-data/conf/netfilters/performance-v6.json ``` Now that your custom filter exists, the original filter will be ignored. All content in the filter should be valid JSON and contain a `set` and `unset` parameter. Each rule should be specified as follows. ```json { "service": "ebtables", "table": "nat", "type": "rule", "cmd": ["-X", "vf-pre-[[VF_INTERFACE]]"] } ``` - `service` should be `ebtables` - `table` should be `nat` or `filter`. If omitted, `nat` will be used. - `type` can be either `rule`, `rule-repeat-ipv4` or `rule-repeat-ipv6` - `cmd` should be an array of the command to be executed. The following global placeholders are available. - `[[VF_INTERFACE]]` is the interface name. - `[[VF_MAC]]` is the MAC address for the interface. Additionally the following placeholders are available in `rule-repeat-ipv4` or `rule-repeat-ipv6`. - `[[VF_IP]]` an ipaddress. - `[[VF_CIDR]]` the cidr of the ipaddress. ### Control Server CLI Commands #### Rewrite all server filters by hypervisor ```shell vfcli-ctrl server:configurations HYPERVISOR_ID --options=network ``` ### Hypervisor CLI Commands These commands are generally not required, but useful for testing. #### Rewrite all server filters ```shell vfcli-hv server:filters set ``` #### Rewrite filters for a specific server ```shell vfcli-hv server:filters set UUID ``` UUID should be replaced with the UUID of the server. #### Rewrite all server filters based on the last known server configuration ```shell vfcli-hv server:filters set-build-force ``` #### Rewrite filters for a specific server based on the last known server configuration ```shell vfcli-hv server:filters set-build-force UUID ``` UUID should be replaced with the UUID of the server. #### Unset (remove) all filters ```shell vfcli-hv server:filters unset ``` #### Unset (remove) filters for a specific server ```shell vfcli-hv server:filters unset UUID ``` UUID should be replaced with the UUID of the server. ### Other Important Notes If you use ebtables outside of VirtFusion (custom scripts) we highly suggest to use the following execution command. ```text /usr/bin/flock /var/tmp/vf-ebtables /usr/sbin/ebtables ``` This will allow all commands to run concurrently and interference free. --- ## PHP Upgrade :::caution Important Prior to VirtFusion v1.9, the upgrade process contained no PHP version checks and some systems that used our early package repos may receive an error on upgrade. To fix any errors, you must update to our newer repo to install a compatible PHP version. ::: ### Check your control server PHP version From SSH as root on the control server run: ```shell /opt/virtfusion/php8/bin/php -v ``` The following is an example output: ``` PHP 8.0.13 (cli) (built: Nov 23 2021 13:12:18) ( NTS ) Copyright (c) The PHP Group Zend Engine v4.0.13, Copyright (c) Zend Technologies with Zend OPcache v8.0.13, Copyright (c), by Zend Technologies with SourceGuardian v12.1.2, Copyright (c) 2000-2021, by SourceGuardian Ltd. ``` As you can see this is PHP 8.0.13 and does require upgrading. If your PHP version is 8.1.x you don't need to continue any further. ### Upgrading to the new repo and PHP version Only RHEL 8 and Debian 11 are affected. The commands below should be run via SSH on the control server. #### RHEL 8 ```shell dnf config-manager --add-repo https://packages.virtfusion.net/rhel/virtfusion.repo dnf update vf-php8 systemctl restart vf-php8-fpm ``` #### Debian 11 ```shell apt-get install curl -y echo "deb http://packages.virtfusion.net/debian bullseye main" > /etc/apt/sources.list.d/virtfusion.list curl -s https://security.virtfusion.net/gpg.asc | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/virtfusion.gpg --import chmod 644 /etc/apt/trusted.gpg.d/virtfusion.gpg apt update -y apt-get remove vf-php8 -y apt-get install vf-php8 -y systemctl restart vf-php8-fpm ``` That's it. You now have the new packages installed. --- ## Server Provisioners A Server Provisioner is a script that is executed during the initial setup of a virtual server. Provisioners can be written in a variety of programming or scripting languages, such as `sh`, `Bash`, `zsh`, `Python`, `PowerShell`, `Batch`, or `clout-init yaml`, depending on the operating system and use case. ### Creating a Provisioner Navigate to `Server -> Provisioners` and click the `Create Provisioner` button. ![provisioner](img/prov1.png) Select the provisioner `Type` and give it a `Name`. Then click the `Create` button. On the next screen you can enter your script in the `Script` field. In this example (Bash) we will update the MOTD when a server is built. ![provisioner](img/prov2.png) Here is a copy + pastable version of the example. ```shell /usr/bin/cat > /etc/update-motd.d/99-virtfusion-info < Templates` and select the template you would like to attach the provisioner to. You will find a selectable list of provisioners in several sections. The most common choice is just after the` Cloud Config (All)` field. ![provisioner](img/prov3.png) In this example we have attached the MOTD provisioner to the Debian 12 template. You can however clone a template and rename it. Click the `Update` button to save your changes. ### Using a Provisioner With a provisioner being attached to a template, there is nothing more to do. When a server is built with a template that has a provisioner attached, it will run automatically. Based on our example provisioner. This is what we see when the server is built. ![provisioner](img/prov4.png) ### Debugging Provisioners Provisioners are executed inside a server during its initial setup. Because of this, debugging provisioners can only be done from within the server after it has been built. To troubleshoot or verify what happened during the provisioning process, you’ll need to access the server and review the following log files: **Linux & Unix** - **`/var/log/cloud-init.log`** – Contains detailed logs of the provisioning process and any issues encountered. - **`/var/log/cloud-init-output.log`** – Captures the output of the scripts, which is useful for identifying script level errors or confirming successful execution. **Windows** - **`C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init`** These logs provide insight into how the server was configured and can help identify problems with the provisioner script itself. --- ## Self Service Self Service allows end users to build and destroy virtual servers within a set of defined limits. There are two variations to self service: - **Hourly Billing** - VirtFusion will track server usage from start to end and supports credit balance billing (top up). - **Resource Packs** - Packs of resources (memory, disk etc...) are allocated to a user and servers can be built within those limits. User may also have a mixture of traditional and self service virtual servers. A user cannot delete a traditional virtual server. :::info Current Limitations - No integrations are currently available for hourly billing purposes (resource packs only WHMCS is supported at the moment), although all required API endpoints exists. - For hourly billing, Tax cannot be shown correctly for package pricing. Prices can only state inclusive or exclusive of tax. This is to be addressed in future versions. - Token value changes will alter the current billing period. ::: ### Configuration #### General System Setup To use hourly billing, specifically the credit balance system, VirtFusion must be configured to monitor balances and credits. This is done by activating schedulers **on the control server via SSH**: ```shell vfcli-ctrl app:schedule-manager self-service start ``` :::caution This scheduler should not be disabled for a prolonged period of time if any users are configured with credit balance billing. ::: #### Hourly Pricing You only need to add pricing to your packages for hourly billing. Resource Packs are not tracked and won't be displayed in any usage statistics. Select `Settings` under the `Self Service` category from the admin menu. Most of the settings are self-explanatory, it's the currencies section (at the bottom of the page) that needs to be configured. Click the `Add Currency` button. In the popup, select the currency code you would like. We will use USD as an example. Enter a `Token Value` of `0.01` and a `Prefix` of `$`, select `Default` and Save. ![Self Service](img/ss1.png) It should look like this once added. ![Self Service](img/ss2.png) :::info You can repeat the process for other currencies, but it's probably worth coming back later once you understand the token concept... ::: #### Packages The standard VirtFusion server packages are used for Self Service, but require a token value setting. Select a server package or create a new one and edit the package. You will see section at the bottom called `Self Service` an input field call `Token Value`. ![Self Service](img/ss3.png) The value entered here is important. Remember we set a `Token Value` of `0.01` for `USD` (equates to 1 cent) which means if we set a value of `199` in the package field, the price will be `$1.99/month`. Enter the value you would like and save the settings. Continue the process for the other packages you would like to use as self service packages. :::info Take note of the other self service options in the package settings. Most of these settings will help refine how the package is displayed to the user. ::: #### Self Service Package Groups Package Groups are essentially groups of server packages. You can name them what you like. Select `Package Groups` under the `Self Service` category from the admin menu. Click `Create Group` and enter a name in the popup. ![Self Service](img/ss6.png) Once created, you can add the packages you created or modified for self service to this group. They will be displayed to the user in the order they are added. ![Self Service](img/ss7.png) Don't forget to click the `Update` button once you've finished configuring the group. #### Self Service Groups You need atleast one self service group. Groups are essentially groups of hypervisor groups. You can name them what you like. Select `Groups` under the `Self Service` category from the admin menu. Click `Create Group` and enter a name in the popup. ![Self Service](img/ss4.png) Once created, you have the option to enable the group as a `Resource Group`. If this is enabled, prices won't show to the end user and/or the packages won't show up in hourly billed groups. It's optional to add a `Description` and it will only be shown if an end user has access to more than one group. The next step is to add your hypervisor groups to this group and add the package groups you created to each of the hypervisor groups. Note that the hypervisors and package groups will be displayed in the order they are added. ![Self Service](img/ss5.png) :::info Take note of the other options in the group settings. Most of these settings will help refine how the package is displayed to the user. ::: #### Self Service Group Profiles You need atleast one self service group profile. Group profiles are groups of self service groups (like you just created in the previous step). You can name them what you like. Select `Group Proofiles` under the `Self Service` category from the admin menu. Click `Create Profile` and enter a name in the popup. ![Self Service](img/ss16.png) Once created, you have the option to enable the profile as a `Resource Group` and enable/disable access. The next step is to add the self service groups you created previously to this profile. ![Self Service](img/ss17.png) #### Resource Packs Resource Packs make it easy to give end users a set of resource limits that can be changed globally at any point. You will need to create a resource pack for hourly billing. Select `Resource Packs` under the `Self Service` category from the admin menu. Click `Create Pack` and enter a name in the popup. You should select `Hourly Only` if this is to be used for hourly billing. ![Self Service](img/ss18.png) On the next page you can enter the limits for the profile. If a limit is left blank, it will default to unlimited. ![Self Service](img/ss15.png) #### Users End users don't have access to self service by default. It will need to be enabled and configured on a per user basis. When editing a user from the admin area, you have several options. ![Self Service](img/ss9.png) - **Group Profiles - Hourly** - You can select the Group Profiles that were created for hourly billing. - **Group Profiles - Resource** - You can select the Group Profiles that were created for resource billing. - **Resource Pack - Hourly** - The resource pack for hourly billing. - **Access** - Enable/disable access to the specific self service options. - **Hourly Credit Balance Billing** - Select this to add the user to balance based billing (top up). Once configured, the user will have the ability to create servers within their hourly limits and/or resource packs. ![Self Service](img/ss11.png) #### Addons Addons are used for hourly billing only. Addons allow you to define a set of extras that can be assigned to a server. i.e IP addresses or backups. To create an addon, select `Addons` under the `Self Service` category from the admin menu. Click `Create Addon` and enter a name in the popup. ![Self Service](img/ss19.png) On the next page you can enter the token value for the addon, enable it and add a description. The token value is the same system we used when configuring a package. A value of `100` will translate to `$1.00` if using the `USD` currency set at `0.01`. ![Self Service](img/ss20.png) ### End User Build Forms All build forms are automatically created based on the group assigned to a user and the resources added within that group. Here's an example of a fully loaded form that supports hourly billing and resource packs. ![Self Service](img/ss12.png) ### Addons All addons will be listed in the usage reports and are billed accordingly. ![Self Service](img/ss24.png) #### Assigning Multiple addons can be assigned to a server. To assign an addon, navigate to the server and click the `Self Service` tab. ![Self Service](img/ss21.png) Click the `Add` button on the right of the `Addons` section. and select one of the defined addons. ![Self Service](img/ss22.png) #### Cancelling To cancel an addon, click the yellow cross button at the end of the table for the assigned addon. Cancelled addons are billed up to the date they are cancelled. ![Self Service](img/ss23.png) #### Deleting Addons should not be deleted unless they were added by mistake. :::warning If an addon is deleted, all accounting for the addon is lost. ::: ### Custom Token Values By default, pricing is defined by a package or addon. However, token values can be defined on a per server or server addon basis. #### Server To define a token value, navigate to the server and click the `Self Service` tab. ![Self Service](img/ss25.png) Enter a custom value in the `Tokens` field and tick `Override Package Token` then click the `Update` button. #### Addon To define a token value, navigate to the server and click the `Self Service` tab. Click the edit icon (pencil in a box) from the list of assigned addons. ![Self Service](img/ss26.png) In the popup, enter a custom value in the `Custom Token Value` field and tick `Enable Custom Token` then click the `Save` button. ### Usage Accounting Usage accounting is only performed on hourly billing. End users have access their usage via a widget on the dashboard or a dedicated usage page. ![Self Service](img/ss27.png) ![Self Service](img/ss28.png) Administrators can also see (and download JSON) reports per user by selecting `Reports` under the `Self Service` category from the admin menu. ### Credit Balance Billing The recommended way to bill your users for hourly servers is via the credit balance system. Credits are added to the user account and the user may use their credit. :::info Current Limitations - Top up credits have to be added manually by administrators until billing modules become available. This can be done via the UI or API. - No actions are taken on a negative credit balance. ::: #### Enabling Credit balance billing can be enabled via the self service settings (Admin -> Self Service -> Settings). ![Self Service](img/ss30.png) It must also be enabled for each user you would like to use the credit system (Admin -> Users -> [selected user] -> Self Service). ![Self Service](img/ss31.png) #### Adding Credits As explained above, No billing modules currently exist. This means top ups for users must be added via the admin UI or API. To add credits for a user, navigate to `Admin -> Self Service -> Balance & Credits` and select the `Credits` tab. Click the `Add Credit` button in the top bar. ![Self Service](img/ss32.png) Select the user and enter a `Token Amount`. It will translate under the field to currencies added on the system. You don't need to add any references, both are optional and will be used at a later date for automation. Click `Apply Credit` and the credit will be added to the user. #### Removing Credits :::caution Removing a credit may result in a negative balance. ::: To remove (unapply) a credit, click the yellow cross icon at the end of the credit entry. ![Self Service](img/ss34.png) It will be removed from the users balance but will remain in the list as `unapplied`. ### API Self service can be integrated with external billing systems for hourly billing and resource packs. Endpoints are available in the API based on user relational ID's. [https://docs.virtfusion.com/api/#api-Self_Service](https://docs.virtfusion.com/api/#api-Self_Service) --- ## Server Migration VirtFusion 3.0+ supports two types of server migration, **Live** and **hybrid**. - **Hybrid migration** uses a system where the server can be offline or online. If the server is online, local storage will be transferred in two stages (once online and again with any changes with the server suspended). It will then be booted on the destination hypervisor once complete. - **Live migration** uses Libvirt's (QEMU) inbuilt live migration and can migrate servers with local and shared (Ceph) storage. This type of migration relies on many factors, mainly hardware. [See the live migration section for more detailed information](#live-migration). :::info Servers that have NAT interfaces can't currently be migrated by either migration system. ::: ### Hybrid Migration Hybrid migration can be performed from within the control UI from either the table of servers (two arrow icon at the end of each table row) or by viewing the server and clicking the two arrow button at the top of the screen. ### Live Migration Live migration is currently managed via **CLI on the control server** with the exception of a live migration status log (Admin > Server > Live Migrations). :::caution For live migration to work correctly: - The server must be online (running or suspended) or awaiting setup (internal transfer). It's also possible to migrate a server if it's offline, but you must supply the `--force-offline` parameter. - The destination hypervisor must have SSH enabled (root) and the correct SSH port specified under the hypervisor settings. ::: #### Requirements - All IP addresses must be usable on the destination hypervisor. No address changes are made (for obvious reasons). - All storage mount points must exist on the destination hypervisor. - Servers must have all storage enabled. - The destination hypervisor CPU vendor must match the source hypervisor. This is only the basics where the CPU is concerned. If CPU flags are missing on the destination hypervisor, the migration will fail. There are many more requirements for live migration. The rule of thumb is to make sure your hypervisors are as identical as possible. This includes the hypervisor operating system and version and also Libvirt and QEMU versions. #### Start a Migration ```shell vfcli-ctrl server:migrate-live start --dst=DESTINATION_HYPERVISOR_ID --server=SERVER_ID ``` :::info A live migration can't be started if a server is offline. You can use the `--force-offline` argument to temporarily start the server in a paused mode so it can be transferred. It will then be shutdown once complete. ::: | Argument | Required | Description | |:--------------------------------------------------------------------------------|:--------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--dst=` | Yes | ID of the destination hypervisor. | | `--server=` | Yes | ID of the server. | | `--force-offline` | No | If a server is not running. Attempt to start it in a paused mode. | | `--dst-alt-ip` | No | Use the alternative IP defined under the destination hypervisor settings. | | `--dst-user=` | No | An alternative user to root. This user must have full root access and a home directory in `/home` | | `--xfer-speed=` | No | Set the max speed of the data transfer in Mib/sec (mebibyte/second). | | `--max-downtime=` | No | Set the max downtime in milliseconds (default 300). | | `--abort-after=` | No | Automatically abort the migration after the specified amount of time in seconds. | | `--disable-compression` | No | By default compression is used. This argument will disable compression. | | `--disable-auto-converge` | No | Auto converge is enabled by default. This argument will disable it. | | `--auto-converge-initial=<%>` | No | if auto converge is enabled, reduce the CPU frequency by the specified percentage (1 - 70) when the migration starts. | | `--auto-converge-increment=<%>` | No | if auto converge is enabled and the initial throttling rate is not enough to ensure convergence, the rate is periodically increased by the specified percentage (1 - 15). | | `--auto-post-copy` | No | Enables post-copy logic and automatically switch to post-copy mode after the first pass of pre-copy is finished (disabled by default). | | `--allow-unsafe` | No | In some cases the system may refuse to migrate the server because doing so may lead to potential problems such as data corruption, and thus the migration is considered unsafe. This may happen if the server uses disks without cache mode "none". Migrating such servers is unsafe unless the disk images are stored on coherent clustered filesystems. | | `--disable-filter-check` | No | By default, the source filters are checked and updated before transfer. This argument will disable the filter check. | | `--detach` | No | Once the migration has started, detach from the migration process. This is useful if you would like to start more than one migration in the same session. | | `--dst-iso` | No | Re-attach the ISO if it was mounted before the migration. This option only covers system defined ISO's, not custom. | | `--json` | No | Output all information in JSON format (non interactive). | If all pre tests pass. The migration will begin. ![live migrate](img/lm1.png) Each step will be shown in the console. The possible statuses are `complete`, `in-progress` or `failed`. If the migration fails, you can attempt it again using the same command used to start the migration or simply `server:migrate-live start --server=SERVER_ID`. ![live migrate](img/lm3.png) Once the migration is complete. All steps will show as `complete` and no further action needs to be taken. ![live migrate](img/lm2.png) :::info Using a remote console session can sometimes be troublesome. Disconnects are very common. If you lose connection while a migration is ongoing, you can re-attach to the session by using the same command you started the migration. If the migration has already completed by the time you re-attach, you will receive an error that the server is already on the destination. In this case, no further action is required. You may also use `server:migrate-live start --server=SERVER_ID` to re-attach and if the migration has already completed by the time you re-attach, you will receive the error `Specify a destination hypervisor ID`. In this case, no further action is required. ::: You may also use `--detach` if you don't want to monitor the migration. The `--json` argument is also available, but does not offer interactivity, just a hard status in JSON format. #### Cancel a Migration If a migration fails, it can be cancelled up to the point of a successful data transfer. ```shell vfcli-ctrl server:migrate-live cancel --server=SERVER_ID ``` #### Status of a Migration You can retrieve the last status of a migration. The possible statuses are `complete`, `in-progress` or `failed` ```shell vfcli-ctrl server:migrate-live last-status --server=SERVER_ID ``` #### Debug Information for a Migration You can retrieve the debug information of the last migration. ```shell vfcli-ctrl server:migrate-live last-debug --server=SERVER_ID ``` #### Abort a Migration You can abort a migration when its running a data transfer. If you try to abort a migration before it starts transferring data, you will receive an error. The same applies if you run this command after the data transfer. ```shell vfcli-ctrl server:migrate-live abort --server=SERVER_ID ``` :::info If an abort is successful, the migration will enter a failed status. To fully exit the migration process, you will need to run the cancel command. ::: #### Retrieve a List of Servers for a Specific Hypervisor ```shell vfcli-ctrl server:migrate-live servers --src=SOURCE_HYPERVISOR_ID --json ``` :::info This command is useful for scripting the migration process. For example [https://github.com/chunguscodes/virtfusion-live-migrate](https://github.com/chunguscodes/virtfusion-live-migrate) ::: Expected output using the `--json` option. ```json { "servers": [ { "id": 1902, "name": "Limited Arm", "uuid": "807127b4-dffb-4794-baa6-faf42028901f" }, { "id": 1903, "name": "Immaterial Slide", "uuid": "265b131c-5f3f-4a6c-a961-364e969feedc" }, ... ] } ``` And without `--json` ``` 1902,1903 ``` #### Force Change Data Status :::danger Important This command is potentially dangerous and may result in data loss if used incorrectly. ::: This command can be used to set the data transfer process as complete. This would be used if all data tracking has been lost (unlikely situation) and **you are sure the data transfer process has completed**. If this command is successful, You are required to run the **`start`** command to complete the migration. ```shell vfcli-ctrl server:migrate-live data-success-force --server=SERVER_ID ``` #### Force Cancel :::danger Important This command is potentially dangerous and may result in data loss if used incorrectly. ::: This command is similar to the `cancel` command, however it skips several of the migration checks (source server online, migration still running). This would be used if all data tracking has been lost (unlikely situation) and **you are sure the data transfer process has failed**. ```shell vfcli-ctrl server:migrate-live cancel-force --server=SERVER_ID ``` --- ## System Backup :::caution VERY IMPORTANT! A database backup is of no use unless you have a backup of the system encryption key. To retrieve the encryption key, see the section [Encryption Key Backup](#encryption-key-backup). **It will not be included in any of your backups**. ::: ### Backup #### Automated Backup VirtFusion supports automated backups of its database using either local storage, SCP or a combination of both. You can find the settings in the admin area `System -> System Backup`. All the options should be self explanatory. ![](img/sb1.png "System Backup") #### Manual Backup We know some people like to be in full control of their backups are prefer to offload backups using different methods. Below you will find a few examples on how to dump the database. ##### Dumping the Database Simple dump: ```shell mariadb-dump --skip-ssl --user=root `cat /opt/virtfusion/app/control/.env | grep 'DB_DATABASE' | cut -d'"' -f2` > /tmp/database.sql ``` Single transaction using `gzip` compression: ```shell mariadb-dump --skip-ssl --single-transaction --opt --user=root `cat /opt/virtfusion/app/control/.env | grep 'DB_DATABASE' | cut -d'"' -f2` | gzip > /tmp/database.gz ``` #### Encryption Key Backup :::caution IMPORTANT! The system encryption key must be securely stored in a safe location. Without this key, certain parts of the database will be inaccessible and unusable. The encryption key is not included in backups. ::: To retrieve the encryption key, run the following command via SSH on the control server as a privileged user: ```shell vfcli-ctrl app:retrieve-key ``` ### Restore #### Restoring on a New Control Server ##### Database For this guide, we’ll assume your database backup is saved in the `/tmp` folder and called either `database.sql` or `database.gz`. If your file has a different name, just replace it with the correct one when following the steps. If the database is not compressed: ```shell mariadb --user=root `cat /opt/virtfusion/app/control/.env | grep 'DB_DATABASE' | cut -d'"' -f2` < /tmp/database.sql ``` If the database is `gzip` compressed: ```shell gzip -d /tmp/database.gz mariadb --user=root `cat /opt/virtfusion/app/control/.env | grep 'DB_DATABASE' | cut -d'"' -f2` < /tmp/database ``` ##### Encryption Key Edit `/opt/virtfusion/app/control/.env`: ```shell nano /opt/virtfusion/app/control/.env ``` Replace the `APP_KEY` value and save the configuration. _Example:_ ```shell APP_KEY=base64:/Ph6OtUdimmo2EiYlUbd9zx+BKM72NNRWnVAd3pE498= ``` --- ## Task Queue Configuration VirtFusion uses background tasks to process server actions such as installs, reboots, backups etc... The default queue system uses the local relational database to store pending tasks where background workers will collect the tasks and process them. On busier systems, it makes sense to use something a little more robust -- Redis. ### Redis #### Redis Installation By default, the VirtFusion control server installer has always installed Redis as a requirement. This means Redis should already be up and running. If you are unsure if Redis is already running, you can check the status on the control server. ```shell systemctl status redis ``` If Redis is not running, you can enable and start it. ```shell systemctl enable redis systemctl start redis ``` #### Configuring VirtFusion to use Redis You need to make a simple configuration change to tell VirtFusion to use Redis. On your control server, edit `/opt/virtfusion/app/control/.env` and add the following at the bottom. ```text QUEUE_CONNECTION=redis ``` Once you have made the change to `/opt/virtfusion/app/control/.env` you will need to gracefully restart the queue workers. You can do this using the following command. ```shell vfcli-ctrl app:restart --services=queue ``` ### Database :::info This is the default driver and you will only need to follow these steps if you would like to revert from using Redis. ::: On your control server, edit `/opt/virtfusion/app/control/.env` and add the following at the bottom. ```text QUEUE_CONNECTION=database ``` Once you have made the change to `/opt/virtfusion/app/control/.env` you will need to gracefully restart the queue workers. You can do this using the following command. ```shell vfcli-ctrl app:restart --services=queue ``` --- ## Webhooks VirtFusion webhooks can be used to automatically send data in real-time when certain events occur. Webhooks require a HTTP endpoint that accepts `POST` requests. Webhooks are commonly used for a variety of purposes, such as: - Automated notifications and alerts - Real-time data synchronization - Triggering automated actions or workflows - Integrating with third-party services and APIs ### Authentication Webhooks optionally support authentication using a `Authorization: Bearer ` header. The token will always be sent base64 encoded. ### Requests #### Headers All webhook requests will include the following headers: | Header | Description | |:----------------------------|:--------------------------------------------------------| | `X-VirtFusion-Hook` | The ID of the webhook. | | `X-VirtFusion-Created` | ISO8601 zulu date string. `2023-04-07T07:18:15.316192Z` | | `X-VirtFusion-Hook-Event` | The ID of the webhook event. | | `X-VirtFusion-Event` | The event name. `server.create` etc... | | `X-VirtFusion-Event-Status` | The event status. `success`, `fail` etc... | | `User-Agent` | `VirtFusion-Hookshot/57HE28D3RT` | | `Content-Type` | `application/json` | | `Accept` | `application/json` | If authentication is enabled on the webhook, the following extra headers are included: | Header | Description | |:----------------|:---------------------------------------------------------------------------------| | `Authorization` | `Bearer ` The Token will be automatically base64 encoded on each request. | | #### Body All requests will contain at least the following in the JSON body: ```json { "event": "", "created": "", "hookId": "", "hookEventId": "", "eventStatus": "", "controlDomain": "", "controlName": "", "controlSupportLink": "", "data": { "errors": [], ... } } ``` The `data` property will always contain data specific to the type of event triggered. ### Events #### Catch All | ID | Event | Status Triggers | Data Body Includes | |:-----|:----------|:-----------------------|:-------------------| | 1000 | catch.all | `success` `fail` `any` | - | #### Server | ID | Event | Status Triggers | Data Body Includes | |:---|:---------------------------|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 | server.create | `success` `fail` `any` | `server object/false` `args object` `allocation object/false` | | 2 | server.destroy | `success` `fail` `any` | `server object/false` | | 3 | server.build | `success` `fail` `any` | `server object/false` | | 4 | server.assign.ipv6 | `success` `fail` `any` | `server object` `ipv6.block object/false` `ipv6.subnet object/false` `ipv6.ip object/false` `ipv6.routeBlock object/false` `ipv6.routeSubnet object/false` | | 5 | server.unassign.ipv6 | `success` `fail` `any` | `server object` `ipv6.block object/false` `ipv6.subnet object/false` `ipv6.ip object/false` `ipv6.routeBlock object/false` `ipv6.routeSubnet object/false` | | 6 | server.poweroff | `success` `fail` `any` | `server object/false` | | 7 | server.boot | `success` `fail` `any` | `server object/false` | | 8 | server.restart | `success` `fail` `any` | `server object/false` | | 9 | server.shutdown | `success` `fail` `any` | `server object/false` | | 10 | server.suspend | `success` `fail` `any` | `server object/false` | | 11 | server.unsuspend | `success` `fail` `any` | `server object/false` | | 12 | server.insert.cd | `success` `fail` `any` | `server object/false` | | 13 | server.eject.cd | `success` `fail` `any` | `server object/false` | | 14 | server.change.boot.order | `success` `fail` `any` | `server object/false` | | 15 | server.enable.vnc | `success` `fail` `any` | `server object/false` | | 16 | server.disable.vnc | `success` `fail` `any` | `server object/false` | | 17 | server.reset.password | `success` `fail` `any` | `server object/false` | | 18 | sserver.disk.trim | `success` `fail` `any` | `server object/false` | | 19 | server.assign.ipv4 | `success` `fail` `any` | `server object/false` `ipv4.block object/false` `ipv4.ip object/false` | | 20 | server.unassign.ipv4 | `success` `fail` `any` | `server object/false` `ipv4.block object/false` `ipv4.ip object/false` | | 21 | server.migrate | `success` `fail` `any` | `server object/false` `hypervisor.source object` `hypervisor.destination object` | | 22 | server.migrate.complete | `success` `fail` `any` | `server object/false` `hypervisor.source object` `hypervisor.destination object` | | 23 | server.migrate.revert | `success` `fail` `any` | `server object/false` `hypervisor.source object` `hypervisor.destination object` | | 31 | server.migrate.live | `success` `fail` `any` | `server object/false` `hypervisor.source object` `hypervisor.destination object` | | 32 | server.migrate.live.cancel | `success` `fail` `any` | `server object/false` `hypervisor.source object` `hypervisor.destination object` | | 33 | server.migrate.live.abort | `success` `fail` `any` | `server object/false` `hypervisor.source object` `hypervisor.destination object` | ### User | ID | Event | Status Triggers | Data Body Includes | |:---|:------------|:----------------|:-----------------------------------------------------| | 25 | user.create | `success` `any` | `user object/false` | | 26 | user.modify | `success` `any` | `user.before object/false` `user.after object/false` | | 27 | user.delete | `success` `any` | `user object/false` | #### System | ID | Event | Status Triggers | Data Body Includes | |:---|:--------------------------|:-----------------------|:------------------------------------------------| | 28 | system.backup.local | `success` `fail` `any` | `log object/false` | | 29 | system.backup.scp | `success` `fail` `any` | `log object/false` | | 30 | system.monitor.hypervisor | `success` `fail` `any` | `type string` `hypervisor object/false` | | 34 | system.mailout | `success` `fail` `any` | `mailout object/false` | | 35 | system.mailout.recipient | `success` `fail` `any` | `mailout object/false` `recipient object/false` | ### Backup Manager | ID | Event | Status Triggers | Data Body Includes | |:---|:-----------------------|:-----------------------|:-------------------| | 36 | backup.manager.batch | `success` `fail` `any` | `data object` | | 37 | backup.manager.restore | `success` `fail` `any` | `data object` | ## ### Integrations A simple [PHP application can be found here](../integrations/webhook-proxy.md) that supports `Slack`, `Discord`, `Google Chat`, `Pushover` and `Telegram` notifications. --- ## Control Server import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Control Server Installation :::caution Warning We do not recommend installing the VirtFusion control server directly on a hypervisor. While it may seem convenient, this setup comes with several limitations and potential issues: - **Resource Contention**: The control server will consume CPU, memory, and disk resources that would otherwise be available for virtual servers. - **Scalability Limits**: Hosting the control server on a hypervisor can significantly limit the number of virtual servers (100) you can create and manage efficiently on the hypervisor that contains the control server. - **Migration Challenges**: Moving the control server later (e.g., for scaling or redundancy) is more complex when it's installed directly on a hypervisor. For best performance and long-term flexibility, we strongly recommend installing the control server on a dedicated or virtual server, separate from your hypervisors. ::: ### Requirements #### Resources - A minimum of `2GB` memory (`4GB+` recommended) - A minimum of `20GB` disk space It's always good practice to make sure you can increase the resources of the control server. As the system grows and more users are interacting at the same time, resource usage will increase. #### Operating Systems The following operating systems are supported. | Operating System | x86_64 | AArch64 | Status | Notes | |-----------------------------------------------------------------------------------|:------:|:-------:|:---------------:|:--------------------------------------------------------------------------------------------------------------------| | Debian 12 (Bookworm) | ✅ | ❌ | Fully Supported | VirtFusion is built and tested primarily on Debian based systems. This is our preferred choice of operating system. | | Debian 13 (Trixie) | ✅ | ❌ | Fully Supported | VirtFusion is built and tested primarily on Debian based systems. This is our preferred choice of operating system. | | AlmaLinux 9 (RHEL 9) | ✅ | ❌ | Fully Supported | | | AlmaLinux 10 (including v2) (RHEL 10) | ✅ | ❌ | Fully Supported | | | Rocky Linux 9 (RHEL 9) | ✅ | ❌ | Fully Supported | | | Rocky Linux 10 (RHEL 10) | ✅ | ❌ | Fully Supported | | | Ubuntu 22.04 | ✅ | ❌ | Fully Supported | **We don't recommend** Ubuntu unless it supports something that Debian does not (drivers etc...). | | Ubuntu 24.04 | ✅ | ❌ | Fully Supported | **We don't recommend** Ubuntu unless it supports something that Debian does not (drivers etc...). | :::danger Important VirtFusion should be the only application installed on the server. Installing any other applications (i.e. Pterodactyl, cPanel etc...) at control server level is not advised, and may invalidate any support contracts included with your VirtFusion license. It is permitted to install monitoring agents (Netdata, Hetrix Tools etc...). VirtFusion does not include an uninstaller. If you make a mistake or decide not to use it, you must re-install the operating system to remove it. ::: ### Installation Select your operating system. **Debian** Run the following commands as a root SSH user. **Debian 12 (Bookworm)** _x86_64_ ```shell apt install curl -y curl https://install.virtfusion.net/install-control-debian-12.sh | sh -s -- --verbose ``` **Debian 13 (Trixie)** _x86_64_ ```shell apt install curl -y curl https://install.virtfusion.net/install-control-debian-13.sh | sh -s -- --verbose ``` **RHEL** Run the following commands as a root SSH user. **RHEL 9** _x86_64_ ```shell dnf install curl -y curl https://install.virtfusion.net/install-control-rhel-9.sh | sh -s -- --verbose ``` **RHEL 10** _x86_64_ ```shell dnf install curl -y curl https://install.virtfusion.net/install-control-rhel-10.sh | sh -s -- --verbose ``` **Ubuntu** Run the following commands as a root SSH user. **Ubuntu 22.04** _x86_64_ ```shell apt install curl -y curl https://install.virtfusion.net/install-control-ubuntu-22.04.sh | sh -s -- --verbose ``` **Ubuntu 24.04** _x86_64_ ```shell apt install curl -y curl https://install.virtfusion.net/install-control-ubuntu-24.04.sh | sh -s -- --verbose ``` Once the installer has completed the installation process, a URL and temporary username & password will be given. ```text +-------------+--------------------------------------+ | URL | https://10.0.0.10 | | Username | admin@admin.com | | Password | sQpfY1p1nhsxqXt | +-------------+--------------------------------------+ ``` ### SSL setup A stock installation will generate a self-signed SSL certificate but this is probably not what you need. You can generate a signed set of certificates for your domain at no cost using a free SSL service or use your own custom certificate. #### Let's Encrypt ##### acme.sh :::caution You must have an A record pointing to the control server IP in your DNS records before proceeding to setup a certificate. ::: Install acme.sh ([Official Website](https://github.com/acmesh-official/acme.sh)) ```shell wget -O - https://get.acme.sh | sh . ~/.bashrc ``` The installer will perform 3 actions: 1. Create and copy acme.sh to your home dir `($HOME): ~/.acme.sh/`. All certs will be placed in this folder too. 2. Create alias for: `acme.sh=~/.acme.sh/acme.sh`. 3. Create daily cron job to check and renew the certs if needed. Generate a certificate (replace `DOMAIN.COM` with your own domain). ```shell acme.sh --issue \ --webroot /opt/virtfusion/nginx/acme \ --keylength 4096 \ --server letsencrypt \ -d DOMAIN.COM ``` Install the certificate (replace `DOMAIN.COM` with your own domain). ```shell acme.sh --installcert \ --keypath /opt/virtfusion/ssl/ssl.key \ --fullchainpath /opt/virtfusion/ssl/ssl.crt \ --reloadcmd "systemctl reload vf-nginx" \ -d DOMAIN.COM ``` #### Custom Certificate The location of the key and certificate should be the following. ``` /opt/virtfusion/ssl/ssl.key /opt/virtfusion/ssl/ssl.crt ``` Once your key and certificate are in place, you simply reload the web server. ``` systemctl reload vf-nginx ``` ### Activating a license Login to VirtFusion with the new login details that were generated on installation. You will be given the options to either generate an evaluation license or enter an existing license key. --- ## Hypervisor import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Hypervisor Installation ## Requirements #### Resources - A large /home partition (This is the default location for virtual server data) - It is recommended that the filesystem where the VM data resides (/home) is formatted in ext4 #### Operating Systems The following operating systems are supported. | Operating System | x86_64 | AArch64 | Status | Notes | |-----------------------------------------------------------------------------------|:------:|:-------:|:---------------:|:--------------------------------------------------------------------------------------------------------------------------------------------| | Debian 12 (Bookworm) | ✅ | ✅ | Fully Supported | VirtFusion is optimised for Debian based systems. This is our preferred choice of hypervisor. | | Debian 13 (Trixie) | ✅ | ✅ | Fully Supported | VirtFusion is optimised for Debian based systems. This is our preferred choice of hypervisor. | | AlmaLinux 9 (RHEL 9) | ✅ | ❌ | Fully Supported | | | AlmaLinux 10 (including v2) (RHEL 10) | ✅ | ✅ | Fully Supported | | | Rocky Linux 9 (RHEL 9) | ✅ | ❌ | Fully Supported | | | Rocky Linux 10 (RHEL 10) | ✅ | ✅ | Fully Supported | | | Ubuntu 22.04 | ✅ | ❌ | Fully Supported | **We don't recommend** Ubuntu unless it supports something that Debian does not (drivers etc...). _"I don't know Debian" is not an excuse._ | | Ubuntu 24.04 | ✅ | ❌ | Fully Supported | **We don't recommend** Ubuntu unless it supports something that Debian does not (drivers etc...). _"I don't know Debian" is not an excuse._ | | Fedora 44 | ✅ | ❌ | Testing Only | **For testing and evaluation only. Do not use in production.** | :::danger Important VirtFusion should be the only application installed on the server. Installing any other applications (i.e. Pterodactyl, cPanel etc...) at hypervisor level is not advised, and may invalidate any support contracts included with your VirtFusion license. It is permitted to install monitoring agents (Netdata, Hetrix Tools etc...). VirtFusion does not include an uninstaller. If you make a mistake or decide not to use it, you must re-install the operating system to remove it. ::: ### Installation The following command supports all operating systems listed above and should be executed over SSH on the hypervisor: ```shell ( set -euo pipefail; (command -v apt-get >/dev/null && apt-get update && apt-get install -y curl) || (command -v dnf >/dev/null && dnf -y install curl); curl -fsSL https://install.virtfusion.net/hypervisor-install.sh | bash ) ``` Once the installer has completed, you may add the hypervisor to the control server. ### Network setup VirtFusion supports `Direct`, `Bridged`, `Routed`, `NAT`, `Open vSwitch Bridged`, and `Isolated` networking. :::info Important MacVTap is the default networking used by VirtFusion since v1.7.2 and usually does not require further network setup, however, if you need the ability to filter traffic (IP Spoofing) or firewalls, you should use a bridged or routed network. ::: :::caution Very Important If your intention is to use NAT networking, It's not mandatory to setup any of the below networks. * *[Continue the installation process here](../guides/nat-setup.md)** ::: #### MacVTap (direct) networking MacVTap (direct) is a device driver meant to simplify virtualized bridged networking. It replaces the combination of the tun/tap and bridge drivers with a single module based on the macvlan device driver. This option has a limitation that you cannot filter traffic and you will lose IP Anti-Hijacking and firewall functionality. **VirtFusion will attempt to detect the primary interface naming, however, it may not be possible. If you have a problem booting servers, it may mean your interface name is incorrect under the hypervisor settings (Network) in VirtFusion. Find `eth0` and replace it with correct name for your primary interface.** **No further configuration required.** #### Standard bridge networking **Debian** On Debian a standard /etc/network/interfaces file will look similar to this. ```shell auto lo iface lo inet loopback auto eth0 iface eth0 inet static address gateway netmask dns-nameservers ``` This will need to be modified to the following. ```shell auto lo iface lo inet loopback auto eth0 iface eth0 inet manual auto br0 iface br0 inet static bridge_ports eth0 address gateway netmask dns-nameservers bridge_stp off bridge_waitport 0 bridge_fd 0 ``` Once the bridge is set up, networking will need to be restarted. ```shell systemctl restart networking ``` **RHEL** **Configuration via NetworkManager** Get the main interface name: ```shell nmcli con show --active ``` We will use `eth0` and the main interface name. ```shell nmcli con mod eth0 ipv4.method auto nmcli con mod eth0 ipv4.gateway "" nmcli con mod eth0 ipv4.address "" nmcli conn add type bridge con-name br0 ifname br0 nmcli conn mod br0 ipv4.addresses 'xxx.xxx.xxx.xxx/24' nmcli conn mod br0 ipv4.gateway 'xxx.xxx.xxx.xxx' nmcli conn mod br0 ipv4.dns '8.8.8.8' nmcli con mod br0 ipv4.method manual nmcli conn add type ethernet slave-type bridge con-name bridge-br0 ifname eth0 master br0 nmcli con modify br0 bridge.stp no ``` **Configuration via network-scripts** On RHEL a standard /etc/sysconfig/network-scripts/ifcfg-eth0 file will look similar to this. ```shell TYPE="Ethernet" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" IPADDR="xxx.xxx.xxx.xxx" GATEWAY="xxx.xxx.xxx.xxx" NETMASK="255.255.255.0" DNS1="8.8.8.8" IPV6_PRIVACY="no" ``` Make a copy of the original file. ```shell cp /etc/sysconfig/network-scripts/ifcfg-eth0 /root/ifcfg-eth0 ``` Create a new file for the bridge. ```shell vi /etc/sysconfig/network-scripts/ifcfg-br0 ``` And copy the networking parts from /etc/sysconfig/network-scripts/ifcfg-eth0 to /etc/sysconfig/network-scripts/ifcfg-br0. ```shell DEVICE=br0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no IPADDR=xxx.xxx.xxx.xxx NETMASK=255.255.255.0 GATEWAY=xxx.xxx.xxx.xxx ``` Next, remove the networking parts from /etc/sysconfig/network-scripts/ifcfg-eth0. ```shell BRIDGE=br0 TYPE="Ethernet" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" DNS1="8.8.8.8" IPV6_PRIVACY="no" ``` Once the bridge is set up, networking will need to be restarted. ```shell systemctl restart NetworkManager.service ``` **Ubuntu** On Ubuntu a standard NetPlan configuration file will look similar to this. Your configuration can be found in `/etc/netplan/`. ```yaml network: version: 2 ethernets: ens3: addresses: - gateway4: nameservers: addresses: - - ``` This will need to be modified to the following. ```yaml network: version: 2 ethernets: ens3: dhcp4: false bridges: br0: interfaces: - ens3 addresses: - gateway4: nameservers: addresses: - - parameters: stp: true forward-delay: 0 dhcp4: false ``` Once the bridge is set up, networking will need to be tested and restarted. ```shell netplan try ``` ```shell netplan apply ``` #### Libvirt routed networking No modification of the main networking configuration is required. ##### Network configuration Defining a Libvirt network is reasonably straightforward and should be low maintenance. Create a file named network.xml with the following contents. ```xml br0 ``` - If you don't require IPv6 you may remove the `family="ipv6"` line. - Replace `eth0` with the device name of the main network. - Replace `10.0.0.1` with your own subnet that is to be assigned to the server and the corresponding netmask. - If you require IPv6, replace the `address` with your own subnet and the corresponding `prefix`. You may need to install dnsmasq if it's not installed already. You can do so by running apt -y install dnsmasq. Define the network, set it to autostart and start it. ```shell virsh net-define --file network.xml virsh net-autostart br0 virsh net-start br0 ``` Network setup is now complete. You should use the main IP of the hypervisor as the gateway for the defined subnet. #### Open vSwitch Bridged Open vSwitch supports multiple VLANS and trunking per virtual server out of the box. :::caution Important This option has a limitation that you cannot filter traffic and you will lose IP Anti-Hijacking and firewall functionality. ::: There is currently limited documentation for this network type. [See here](networking/ovs.md) ### Notes Although it's not advised, you may install a hypervisor directly on the control server. This allows for a single server setup but will restrict the hypervisor to a total of 99 active servers. --- ## Bridged Hypervisor Mac Restricted ## Configuring a Bridged Network with Mac Address Restrictions Providers like Hetzner and OVH are known to enforce MAC address restrictions, and other providers may do the same. By default, a virtual machine’s network interface is assigned a random MAC address, which can lead to connectivity issues with these providers, as they require all traffic to originate from a predefined MAC address. :::info **This guide exclusively covers IPv4 configuration on a Debian based hypervisor. We do not supply documentation for any other distribution.** ::: #### Install `bridge-utils` Before creating a bridge, make sure you install `bridge-utils`. If you don't, you will lose network connectivity. _By default, the VirtFusion installer will install it. This step is just in case you are setting up the bridge before installing VirtFusion._ ``` sh apt install bridge-utils -y ``` #### Bridge Setup To set up the persistent network (including the bridge) the `/etc/network/interfaces` file should be as follows (it should not contain `source-directory /etc/network/interfaces.d`): ```shell auto lo iface lo inet loopback auto iface inet manual auto br0 iface br0 inet static bridge_ports address gateway netmask dns-nameservers bridge_hw bridge_stp off bridge_waitport 0 bridge_fd 0 ``` `` should be replaced with the public interface name of the server. It may be `eno1`, it may be `eth0` or even named something different, but to be sure run `ip a` to list the interfaces on the server. `` should be the main ipv4 address of the server. which can also be double-checked using `ip a`. `` should be set to the correct gateway. `` should be set to the correct netmask for the main IPv4 subnet. ` ` can be set to your favorite public nameserver. `8.8.8.8` & `8.8.4.4` for Google. Once you have it configured. It should look something along the lines of this: ```shell auto lo iface lo inet loopback auto eno1 iface eno1 inet manual auto br0 iface br0 inet static bridge_ports eno1 address 100.100.100.25 gateway 100.100.100.1 netmask 255.255.255.0 dns-nameservers 8.8.8.8 8.8.4.4 bridge_hw eno1 bridge_stp off bridge_waitport 0 bridge_fd 0 ``` :::warning Double check your network configuration before restarting the network. If you have made a mistake, you may lose connectivity. ::: You can now restart the networking. It's also wise to reboot the server at this point. ```shell systemctl restart networking ``` ### VirtFusion Configuration #### IP Addresses :::info Your provider will give you a list of IPv4 addresses along with their corresponding MAC addresses. If you haven't received this information, please contact your provider. ::: IPv4 addresses need to be configured correctly in VirtFusion for them to work. You can either do this from `Admin -> Connectivity -> IPv4 Addresses` or `Admin -> Connectivity -> IP Block -> (select the IPlock) -> Addresses (tab)`. On the list of IPv4, you will see a column in the table called `MAC`. ![mac](../img/mac1.png) Click `+MAC` in the correct row for your IP. A popup will open for you to enter the correct MAC. ![mac](../img/mac2.png) Enter the MAC that has been provided by you provider in to the field and click `Update`. The MAC will now be assigned to the IP. ![mac](../img/mac3.png) #### Hypervisor Network Navigate to `Compute Resources -> Hypervisors -> (select your hypervisor) -> Network (tab)` and set the primary network as shown below. ![mac](../img/mac4.png) :::warning Important If you're experiencing connectivity issues after following these instructions, please double-check all your settings. Our support team is unable to assist with network debugging in environments with such restrictive configurations. ::: --- ## Open vSwitch ## What is Open vSwitch? Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag). ### Debian 11/12 Example Hypervisor Setup ```shell apt install openvswitch-switch systemctl enable openvswitch-switch systemctl start openvswitch-switch ``` ```shell ovs-vsctl add-br br0 ovs-vsctl add-port br0 HYPERVISOR_INTERFACE ovs-vsctl set port HYPERVISOR_INTERFACE vlan_mode=native-untagged ovs-vsctl set port HYPERVISOR_INTERFACE tag=1 ovs-vsctl add-port br0 mgmt -- set Interface mgmt type=internal ovs-vsctl set port mgmt tag=1 ``` `/etc/network/interfaces` ```shell auto lo iface lo inet loopback post-up ip link set br0 up post-up ip link set HYPERVISOR_INTERFACE up post-up ip addr add HYPERVISOR_IP/HYPERVISOR_CIDR dev mgmt post-up ip link set mgmt up post-up ip route add HYPERVISOR_GATEWAY dev mgmt post-up ip route add default via HYPERVISOR_GATEWAY dev mgmt ``` --- ## OVH Debian 11 Bridged Hypervisor ## Configuring a bridged network on Debian for OVH & SYS hypervisors [OVH](https://www.ovhcloud.com/en-gb/bare-metal/prices/) and [So You Start](https://www.soyoustart.com/en/) are extremely popular choices for hypervisors due to cost, speed and availability of IPs. However, these providers run strict networking and require explicit MAC address assignment (with the exception of vrack subnets) when bridging networks. At the time of writing, OVH and SYS offer a Debian 11 template and that's what this guide is based off. One thing that must be installed to configure the bridge is `brctl` which is part of the `bridge-utils` package. Make sure its installed. ``` sh apt install bridge-utils -y ``` ### Bridge Setup On the OVH Debian 11 template install a standard `/etc/network/interfaces` file will look similar to this. ```shell source-directory /etc/network/interfaces.d ``` And will have a corresponding cloud-init configuration in `/etc/network/interfaces.d`. The templates used by OVH are provisioned using cloud-init which is not really required once the hypervisor has been set up. It's wise to remove cloud-init and set a persistent network. To remove cloud-init run the following: ```shell apt-get purge cloud-init rm -rf /etc/cloud/ rm -rf /var/lib/cloud/ ``` To set up the persistent network (including the bridge) the `/etc/network/interfaces` file should be as follows (it should not contain `source-directory /etc/network/interfaces.d`): ```shell auto lo iface lo inet loopback auto iface inet manual auto br0 iface br0 inet static bridge_ports address gateway netmask dns-nameservers hwaddress ether bridge_stp off bridge_waitport 0 bridge_fd 0 ``` `` should be replaced with the main interface name of the server. It's most likely `eno1` but to be sure run `ip a` to list the interfaces on the server. `` should be the main ipv4 address of the server. which can also be double-checked using `ip a`. `` should be set to the correct gateway. This is usually the last ip in the primary ips subnet (`.254`). `` can be set to `255.255.255.0` or `255.255.255.255`. ` ` can be set to your favorite public nameserver. `8.8.8.8` & `8.8.4.4` for Google. `` MUST be set to the MAC address of the primary network interface which again can also be found using `ip a`. Once you have it configured. It should look something along the lines of this: ```shell auto lo iface lo inet loopback auto eno1 iface eno1 inet manual auto br0 iface br0 inet static bridge_ports eno1 address 100.100.100.25 gateway 100.100.100.254 netmask 255.255.255.0 dns-nameservers 8.8.8.8 8.8.4.4 hwaddress ether 00:25:90:ed:c1:d0 bridge_stp off bridge_waitport 0 bridge_fd 0 ``` You can now restart the networking. It's also wise to reboot the server at this point. ```shell systemctl restart networking ``` ### VirtFusion Configuration IPv4 addresses need to be configured correctly in VirtFusion for them to work. You must generate MAC addresses for each of your IPs in the OVH/SYS control panel and update the IPs in VirtFusion to match (Admin -> IP Addresses -> + MAC). IPv6 does not require any special treatment. The supplied /64 that OVH offer will subnet down correctly. --- ## Hetzner Routed Network ## Hetzner Routed Network (Subnets Only) [Hetzner](https://www.hetzner.com/) is an extremely popular choice for hypervisors due to cost, speed and availability of IPs. However, Hetzner runs a strict network and their IPv4 subnets generally won't work with a standard network bridge. In our example we will use a /29 subnet. You can find the IP pricing [here](https://docs.hetzner.com/general/others/ipv4-pricing/). ![hz1](../img/hz1.png) Hetzner state that there are 6 usable IPs with a /29, however, when routing the subnet you can usually use both the network addresses. This means you will have the use of all 8 IPs. In the Hetzner Robot control panel, it will list all your IPs. For example, our /29 subnet. ![hz2](../img/hz2.png) All the addresses can be used for virtual servers. ### Hypervisor Network Configuration In `/root` on the hypervisor create a file called `network.xml`. ```sh nano -w network.xml ``` In that file you should configure your network. ```xml BRIDGE_NAME ``` - `BRIDGE_NAME` should be the name of your network. `br0` is a good example. - `PRIMARY_INTERFACE` should be the name of your primary interface. You can find it with the `ip a` command. - `IP_RANGE` should be your Hetzner subnet. Try to use `xxx.xxx.xxx.1` if possible. It doesnt matter if you don't own that IP. It will be used as the gateway. - `IP_NETMASK` in most cases should be `255.255.255.0` and you should ignore the netmask Hetzner give you. For example using our subnet, the configuration would be the following. ```xml br0 ``` You should now save the file (`ctrl+x`). You can now define the network, set it to autostart and start it. ```shell virsh net-define --file network.xml virsh net-autostart br0 virsh net-start br0 ``` `br0` should be the name of the network you used in the configuration. ### VirtFusion Network Configuration Now that you have your routed network setup on the hypervisor, you can configure it inside the control panel. Modify your hypervisor network and make sure its set to `Libvirt Routed` and you specify the correct network name. ![hz5](../img/hz5.png) Create a new IPv4 Block and fill in the details. The following is based on our example /29 block. ![hz3](../img/hz3.png) :::info You may also use the hypervisor main IP address as the gateway, although some server operating systems don't play nice when the gateway is out of subnet range (looking at you, Windows). ::: Once you have saved the block settings, assign the block to the correct hypervisor and interface. Then add your IPs. ![hz4](../img/hz4.png) That's it. You should now have a working configuration and are ready to build servers with your routed IPs. ### Other Useful Information #### Modifying the Hypervisor Network Configuration At some point you may need to make changes to the network configuration. You can edit the live configuration using the following command. ```shell virsh net-edit br0 ``` You would then need to stop, then start the network after making the changes. :::caution Warning Running the following commands will disconnect all existing servers from the network. You will either need to reboot each server or restart the hypervisor, or re-link the interfaces using virsh. ::: ```shell virsh net-destroy br0 virsh net-start br0 ``` ##### Re-linking the virtual server interfaces ```shell virsh list --name | xargs -i virsh domiflist {} | grep -w BRIDGE_NAME | awk '{print $1}' | xargs -i ip link set {} master BRIDGE_NAME ``` - `BRIDGE_NAME` should be the name of your network. --- ## Ceph RBD Since VirtFusion v3.1 Ceph block devices (RADOS) are supported. Ceph RBD (RADOS Block Device) is a component of the Ceph distributed storage system. Ceph is an open-source software-defined storage platform that provides scalable and reliable storage. :::info Important There are a few limitations you should be aware of. - Offline migrations are not supported (Live migration is fully supported. Even with a mix of local and remote disks). - Backups will be disabled for remote disks. - Disaster Recovery will also ignore remote disks. Only the local configurations and assets will be stored. ::: :::warning Very Important VirtFusion will not set up and manage your storage cluster. It is expected that you already have your cluster in place and managed by experienced administrators. We will not be able to offer support for any of your attached storage. ::: ### Configuration #### Creating a Ceph user You may already have your Ceph access setup, however these are simple instructions to create a Ceph user with the correct pool permissions. On your Ceph cluster this command will create a user called `virtfusion` with access to the `vps_data` RBD pool. ```shell ceph auth get-or-create client.virtfusion mon 'profile rbd' osd 'profile rbd pool=vps_data' ``` Expected output: ```shell [client.virtfusion] key = AQD5kFNl2LgUCBAANc2gVXBw4WJDhmIlV8Zrbg== ``` The key should be saved for future use. #### Install required tools The following packages will need to be installed on each hypervisor. ``` qemu-block-extra ceph-common ``` #### Configuring access for hypervisors On your first hypervisor, you need to create a secret for the Libvirt tool stack. ```shell cat > secret.xml < client.virtfusion secret EOF ``` ```shell virsh secret-define --file secret.xml ``` Expected output: ```shell Secret 4b5f6d26-81c3-42b1-a196-dd668854ce3b created ``` The UUID should be saved for future use. ```shell virsh secret-set-value --secret ``` `UUID` should be the UUID generated for the secret and the `USER_KEY` should be the key that was generated for the user. Expected output: ```shell Secret value set ``` :::info Important This secret must be the same across all of your hypervisors. You don't need to generate a new one for subsequent hypervisors. You can copy the one you just generated. The secret files are stored in `/etc/libvirt/secrets` and will be in the form of `.base64` and `.xml`. These can be directly copied to the rest of your hypervisors. ::: Edit `/etc/ceph/ceph.conf` ``` [global] mon_host = ... [client.virtfusion] key = AQD5kFNl2LgUCBAANc2gVXBw4WJDhmIlV8Zrbg== ``` Define your list of monitor host IPs and the user that was created earlier. **Just note this is a bare minimum and you will most likely have your own advanced configuration**. Restart the Ceph monitor. ```shell systemctl restart ceph-mon ``` If all goes well, you should be able to successfully connect. ```shell ceph version --name client.virtfusion ``` Expected output: ```shell ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) ``` ### VirtFusion Configuration #### Storage From the Admin menu, select `Compute Resources` -> `Datastores` Click the `Add Storage` button at the top right of the screen. ![ceph](../img/ceph1.png) Select the correct `Type` and give it a `Name`. Then click `Add`. On the next screen you can configure certain cluster settings. ![ceph](../img/ceph2.png) Fill in the pool `Capacity`, the `Username` generated in the previous step, the `Pool` name and the `Libvirt Secret UUID` that was also in the previous step. Click the `Update` button to save the settings. #### Hypervisor Edit the hypervisor you would like to add the storage to and click the `Add Storage` button. ![ceph](../img/ceph3.png) Give it a `Name`, select `Type` Datastore, select the `Datastore` you created, leave `Configuration` blank and then click the `Add` button. You should now be able to create servers using the new datastore. --- ## Lightbits Since VirtFusion v3.1 Lightbits NVMe/TCP storage is supported. :::warning Important This feature is currently restricted. It should not be used in production. ::: --- ## Remote Filesystem Since VirtFusion v3.1 remote filesystems are supported. Remote filesystems will need to be mounted locally on each hypervisor. This can be achieved by using software such as CephFS, CIFS, SMB, iSCSI etc. :::info Important There are a few limitations you should be aware of. - Offline migrations are not supported (Live migration is fully supported. Even with a mix of local and remote disks). - Backups will be disabled for remote disks. - Disaster Recovery will also ignore remote disks. Only the local configurations and assets will be stored. ::: ### VirtFusion Configuration #### Storage From the Admin menu, select `Compute Resources` -> `Datastores` Click the `Add Storage` button at the top right of the screen. ![filesystem](../img/filesystem1.png) Select the correct `Type` and give it a `Name`. Then click `Add`. On the next screen you can configure certain cluster settings. ![filesystem](../img/filesystem2.png) Fill in the `Capacity` and the `Path` of the local mountpoint. The remote filesystem must already be mounted to this point). Click the `Update` button to save the settings. #### Hypervisor Edit the hypervisor you would like to add the storage to and click the `Add Storage` button. ![filesystem](../img/filesystem3.png) Give it a `Name`, select `Type` Datastore, select the `Datastore` you created, leave `Configuration` blank and then click the `Add` button. You should now be able to create servers using the new datastore. --- ## StorPool Since VirtFusion v4.0, [StorPool distributed storage](https://storpool.com/) is supported. :::info Important There are a few limitations you should be aware of. - Offline migrations are not supported (Live migration is fully supported. Even with a mix of local and remote disks). - Backups will be disabled for remote disks. - Disaster Recovery will also ignore remote disks. Only the local configurations and assets will be stored. ::: ### StorPool Configuration The StorPool CLI must be available on each hypervisor (`/usr/sbin/storpool`). ### VirtFusion Configuration #### Storage From the Admin menu, select `Compute Resources` -> `Datastores` Click the `Add Storage` button at the top right of the screen. ![filesystem](../img/storpool1.png) Select the correct `Type` and give it a `Name`. Then click `Add`. On the next screen you can configure certain cluster settings. ![filesystem](../img/storpool2.png) Fill in the `Default Template` field with the default template used in StorPool (usually `nvme`). Click the `Update` button to save the settings. #### Hypervisor Edit the hypervisor you would like to add the storage to and click the `Add Storage` button. ![filesystem](../img/filesystem3.png) Give it a `Name`, select `Type` Datastore, select the `Datastore` you created, leave `Configuration` blank and then click the `Add` button. You should now be able to create servers using the new datastore. --- ## BillingServ ## BillingServ Provisioning The billingServ module is a simple module that can `create`, `terminate`, `suspend` and `unsuspend` servers with a direct login bridge between BillingServ and VirtFusion. ### How it works When a customer orders a server, they use the standard ordering system of BillingServ but don't have all the confusing options of choosing a root password, operating system etc... they just order and make payment. Once payment is complete, they have an option to directly login to VirtFusion to set up their server. Which takes them straight to the setup page. ![api](img/whmcs_works2.png) ### Download & Install This is a module built by the BillingServ team and is included in the product. --- ## Blesta ## Blesta Direct Provisioning The Blesta Direct Provisioning module is a simple module that can `create`, `terminate`, `suspend` and `unsuspend` servers with a direct login bridge between Blesta and VirtFusion. ### How it works When a customer orders a server, they use the standard ordering system of Blesta but don't have all the confusing options of choosing a root password, operating system etc... they just order and make payment. Once payment is complete, they have an option to directly login to VirtFusion to set up their server. Which takes them straight to the setup page. ![api](img/whmcs_works2.png) ### Download & Install Download the latest version of the module. | Version | Released | Link | |---------|------------|:-----------:| | 1.0.1 | 2022-01-25 | [download](https://software.virtfusion.net/integrations/virtfusion-blesta-direct-provisioning-module-1.0.1.zip) | | 1.0.0 | 2022-01-19 | obsolete | Once downloaded, extract the archive and upload the contents of the `upload` directory to the root of your Blesta installation. ### Configuration #### VirtFusion API credentials As a VirtFusion administrator, create a new global API token (Configuration > Global API > Create Token) ![api](img/api4.png) Fill in the `Name` field and enter the IP of your Blesta installation in the `Access` field (This is optional, entering an IP will lock requests to the specified IP). Then click `Create`. ![api](img/api3.png) Copy the generated API key to a safe place. We will need it later. #### Blesta setup ##### Activating the module As a Blesta admin, navigate to Settings > Company > Modules. Click the `Available` tab and scroll to the VirtFusion module. Click the Install button. ![api](img/blesta1.png) ##### Adding the connection credentials Once installed, click `Add Server` to add your VirtFusion control server. ![api](img/blesta2.png) Fill in all the details for your VirtFusion control server and save the settings. ![api](img/blesta3.png) ##### Creating a package Navigate to `Packages` and add a package. Fill in all the basic package details as normal, then click the Module tab. Select `VirtFusion Direct Provisioning` and select the server you configured. ![api](img/blesta4.png) - `Hypervisor Group ID` can be found in VirtFusion (Compute Resources > Hypervisor Groups > ID (table column)) - `Package ID` can be found in VirtFusion (Packages > ID (table column)) - `Default IPv4` the number of IPv4 that should be allocated Save the package, and you're done with the configuration. --- ## Clientexec ## Clientexec Direct Provisioning The Clientexec Direct Provisioning module is a simple module that can `create`, `terminate`, `suspend` and `unsuspend` servers with a direct login bridge between Clientexec and VirtFusion. ### How it works When a customer orders a server, they use the standard ordering system of Clientexec but don't have all the confusing options of choosing a root password, operating system etc... they just order and make payment. Once payment is complete, they have an option to directly login to VirtFusion to set up their server. Which takes them straight to the setup page. ![api](img/whmcs_works2.png) ### Download & Install This is a module built by the Clientexec team and is included in the product. --- ## HostBill ## HostBill Provisioning Module ### Download & Install This module is built and maintained by the HostBill team. Module information: [https://hostbillapp.com/feature/virtfusion/](https://hostbillapp.com/feature/virtfusion/) --- ## Paymenter ## Paymenter Provisioning Module ### Download & Install This module is built and maintained by the Paymenter team. Module information: [https://paymenter.org/features/virtfusion/](https://paymenter.org/features/virtfusion/) --- ## Upmind ## Upmind Provisioning ### Download & Install This module is built and maintained by the Upmind team. Setup information can be found here: [https://docs.upmind.com/docs/how-to-add-virtfusion-products](https://docs.upmind.com/docs/how-to-add-virtfusion-products) --- ## Webhook Proxy The webhook proxy is a simple PHP application that will allow you to send notifications to `Slack`, `Discord`, `Google Chat`, `Pushover` and `Telegram`. It's also possible to create your own providers for other services. ### Install locally on your control server You may install the proxy locally on your control server. This is the easiest option and will allow for communication internally. To install the proxy, run the following commands on your control server. ```shell mkdir -p /opt/virtfusion/extras cd /opt/virtfusion/extras wget https://software.virtfusion.net/integrations/virtfusion-webhook-proxy.tar.gz tar -xzf virtfusion-webhook-proxy.tar.gz rm virtfusion-webhook-proxy.tar.gz cd webhook-proxy cp config-example.php config.php chown virtfusion. -R /opt/virtfusion/extras/webhook-proxy cp /opt/virtfusion/extras/webhook-proxy/extras/control_nginx.conf /opt/virtfusion/nginx/conf/conf.d/webhook_proxy.conf systemctl reload vf-nginx ``` After it's installed, you may use the following webhook URL without authentication or encryption. `http://127.0.0.1:7001` #### Examples **Slack Notifications** ``` http://127.0.0.1:7001?provider=slack_notifications&url=SLACK_WEBHOOK_URL ``` **Discord Notifications** ``` http://127.0.0.1:7001?provider=discord_notifications&url=DISCORD_WEBHOOK_URL ``` **Google Chat Notifications** ``` http://127.0.0.1:7001?provider=google_chat_notifications&url=GOOGLE_WEBHOOK_URL ``` Other providers require a more complex configuration. ### Install remotely The proxy can be installed anywhere you like. Its just a simple PHP application that will run on shared hosting. The only requirement is PHP 8.0+ and the `public` directory as your web root. ### Download Download the latest version of the module. | Version | Released | Link | |---------|------------|:-----------------------------------------------------------------------------------------------------------------:| | 1.0.0 | 2023-11-09 | [download](https://software.virtfusion.net/integrations/virtfusion-webhook-proxy.tar.gz) | ### Configuration The configuration file (`config.php`) is located in the root folder and contains the provider configuration and credentials plus settings for a bearer token and IP restrictions. ### Creating your own provider To create your own provider, you need to add it to the providers array in the configuration file like so. ```php 'providers' => [ /* | | Custom notifications. | */ 'custom_notifications' => [ 'url' => '', ], ... ``` You then need to create a matching provider in the `providers` directory with the same naming, `custom_notifications.php`. Feel free to use the existing providers as a base for your own. If you just want to modify an existing provider, it's wise to duplicate it and add it as a new provider. I.e `slack_notifications` --> `slack_notifications_custom`. ### Provider Configuration #### Slack Creating a Slack incoming webhook involves a few steps. **Log in to Slack** - Open your web browser and go to the Slack workspace where you want to create the incoming webhook. - Log in with your Slack credentials. **Go to App Directory** - In the left sidebar, click on `Apps` or `App Directory`, depending on your Slack workspace version. Search for Incoming WebHooks: - In the App Directory search bar, type "Incoming WebHooks" and press Enter. **Select Incoming WebHooks** - Click on the "Incoming WebHooks" app from the search results. **Add to Slack** - On the Incoming WebHooks app page, click the `Add to Slack` button. **Choose a Channel** - Select the Slack channel where you want the incoming webhook to post messages. - Click on `Add Incoming WebHooks Integration` **Copy the Webhook URL** - After adding the integration, you'll be provided with a Webhook URL. This URL is unique to your Slack channel. - Copy the Webhook URL. #### Discord Creating a Discord incoming webhook involves a few steps. **Create a Discord Server** - If you don't have a Discord server, create one. If you already have a server, you can use an existing one. **Create a Text Channel** - Choose or create a text channel where you want the incoming webhook messages to appear. **Integrate an Incoming Webhook** - Go to your Discord server. - Click on the settings icon (gear icon) next to the channel name. - Choose "Integrations." - Click on the `Create Webhook` button. **Set up Webhook** - `Name:` Give your webhook a name. This is the name that will be displayed when it sends messages. - `Channel:` Choose the text channel where you want the messages to appear. - `Avatar:` You can optionally set a custom avatar for the webhook. **Copy Webhook URL** - Once you've filled in the details, click on the `Copy Webhook URL` button. #### Google Chat Creating a Google Chat incoming webhook involves a few steps. **Create a Google Chat Room** - Open Google Chat in your browser: https://chat.google.com/ - Create a new room or select an existing room where you want the incoming webhook messages to appear. **Configure Room Settings** - Click on the room name at the top of the screen. - In the room details panel, click on the settings icon (gear icon) to access room settings. - Enable "Allow external people" if you want external services to post messages to this room. **Create an Incoming Webhook** - Go to the Google Chat API page: https://developers.google.com/hangouts/chat/how-tos/webhooks - Scroll down to the `Incoming webhooks` section. - Click on the link that says `Create an incoming webhook`. **Choose a Google Account** - Select the Google account that you want to associate with the incoming webhook. **Select a Room** - Choose the room where you want the incoming webhook messages to appear. **Configure the Webhook** - Give your webhook a name. - Optionally, you can upload an image for the webhook. **Copy the Webhook URL** - After configuring the webhook, click on the `Save` button. - Copy the generated webhook URL. #### Pushover Obtain your API credentials from [https://pushover.net/](https://pushover.net/) Edit `config.php` and amend your credentials. ```php 'pushover_notifications' => [ 'url' => 'https://api.pushover.net/1/messages.json', 'token' => '', 'user' => '', 'successPriority' => 0, 'failPriority' => 1, ] ``` #### Telegram Creating a Telegram bot involves several steps, including creating a Telegram account, creating a new bot on the Telegram BotFather platform and obtaining the API token. **Create a Telegram Account** - If you don't already have a Telegram account, download the Telegram app on your mobile device or use the web version and create an account. **Create a New Bot on BotFather** - Open the Telegram app and search for `BotFather`. - Start a chat with BotFather and use the `/newbot` command to create a new bot. - Follow the instructions provided by BotFather to set a name and username for your bot. - Once the bot is created, BotFather will provide you with an API token. Save this token; you'll need it to authenticate your bot. **Get Your Chat ID** - If your bot needs to send messages to a specific user or group, you'll need to obtain the chat ID. You can use the `getUpdates` method after sending a message to your bot. Alternatively, there are online tools that can help you retrieve your chat ID. **Construct the URL** ``` https://api.telegram.org/bot/sendMessage?chat_id=- ``` --- ## WHMCS Resource Packs The WHMCS Resource Pack Provisioning module is a simple module that can `create`, `terminate`, `suspend` and `unsuspend` packs of resources with a direct login bridge between WHMCS and VirtFusion. ### How it works When a customer orders a resource pack, they use the standard ordering system of WHMCS and make payment. Once payment is complete, the pack is created and the resources are allocated to their VirtFusion account. They have an option to directly login to VirtFusion to create, manage and destroy servers. ![api](img/whmcs_rs1.png) ### Download & Install Download the latest version of the module. | Version | Released | Link | Notes | |---------|------------|:-------------------------------------------------------------------------------------------------------:|--------------------------------------| | 1.0 | 2023-02-11 | [download](https://software.virtfusion.net/integrations/virtfusion-whmcs-resource-pack-module-1.0.zip) | Requires VirtFusion v2.0.1 or higher | Once downloaded, extract the archive and upload the contents of the `upload` directory to the root of your WHMCS installation. ### Configuration #### VirtFusion API credentials As a VirtFusion administrator, create a new global API token (System > API > Create Token) ![api](img/api1.png) Fill in the `Name` field and enter the IP of your WHMCS installation in the `Access` field (This is optional, entering an IP will lock requests to the specified IP). Then click `Create`. ![api](img/api2.png) Copy the generated API key to a safe place. We will need it later. #### WHMCS setup ##### Create a server As a WHMCS administrator, add a new server (System settings > Servers). ![servers](img/whmcs_rs_server.png) - Give it a `Name` - Enter the `hostname` of your VirtFusion management UI - Select `Module` VirtFusion Resource Packs - Enter the VirtFusion API token in the `Password` field - Tick `Secure` - Save ##### Create a server group Create a new Server Group and add the previously created server to it. ![servers](img/whmcs2.png) ##### Create a product group Create a new Product Group (System settings > Products/Services). ![servers](img/whmcs_rs_3.png) ##### Create a product Create a new Product. ![servers](img/whmcs_rs_4.png) - `Product Type` Other - `Product Group` should be the one you just created - `Product Name` - `Module` VirtFusion Resource Packs Once the product is created, click the `Module Settings` tab. ![servers](img/whmcs_rs_5.png) - `Server Group` select the server group you created earlier - `Resource Pack ID` can be found in VirtFusion (Self Service -> Resource Packs -> ID (table column)) - `Deletion Delay` How long in minutes should the system wait to delete servers when a pack is terminated - `Termination Type` When a pack is cancelled (terminated) should the servers attached to the pack be deleted or just suspended (Need to be deleted manually) - Continue to configure the product as normal. --- ## WHMCS ## WHMCS Direct Provisioning The WHMCS Direct Provisioning module is a simple module that can `create`, `terminate`, `suspend` and `unsuspend` servers with a direct login bridge between WHMCS and VirtFusion. ### How it works When a customer orders a server, they use the standard ordering system of WHMCS but don't have all the confusing options of choosing a root password, operating system etc... they just order and make payment. Once payment is complete, they have an option to directly login to VirtFusion to set up their server. ![api](img/whmcs_works.png) Which takes them straight to the setup page. ![api](img/whmcs_works2.png) ### Download & Install Download the latest version of the module. | Version | Released | Link | Notes | |---------|------------|:-------------------------------------------------------------------------------------------------------:|--------------------------------------| | 2.5 | 2026-08-08 | [download](https://software.virtfusion.net/integrations/virtfusion-whmcs-direct-billing-module-2.5.zip) | Requires VirtFusion v6.0 or higher · [Release notes](#25) | | 2.4 | 2024-06-12 | [download](https://software.virtfusion.net/integrations/virtfusion-whmcs-direct-billing-module-2.4.zip) | Requires VirtFusion v3.0 or higher | | 2.3 | 2022-08-18 | obsolete | Requires VirtFusion v1.7.3 or higher | | 2.2 | 2022-07-01 | obsolete | | | 2.0 | 2022-06-05 | obsolete | Requires VirtFusion v1.6.1 or higher | | 1.4 | 2022-04-06 | obsolete | | | 1.3 | 2022-02-02 | obsolete | | | 1.2 | 2022-01-03 | obsolete | | | 1.1 | 2021-12-16 | obsolete | | | 1.0 | 2021-11-29 | obsolete | | Once downloaded, extract the archive and upload the contents of the `upload` directory to the root of your WHMCS installation. #### Release notes ##### 2.5 - Adds [multi-instance support](#multiple-whmcs-instances-optional) for running multiple WHMCS installations against a single control server - Minor text fixes ### Configuration #### VirtFusion API credentials As a VirtFusion administrator, create a new global API token (System > API > Create Token) ![api](img/api1.png) Fill in the `Name` field and enter the IP of your WHMCS installation in the `Access` field (This is optional, entering an IP will lock requests to the specified IP). Then click `Create`. ![api](img/api2.png) Copy the generated API key to a safe place. We will need it later. #### WHMCS setup ##### Create a server As a WHMCS administrator, add a new server (System settings > Servers). ![servers](img/whmcs1.png) - Give it a `Name` - Enter the `hostname` of your VirtFusion management UI - Select `Module` VirtFusion Direct Provisioning - Enter the VirtFusion API token in the `Password` field - Tick `Secure` - Save ##### Create a server group Create a new Server Group and add the previously created server to it. ![servers](img/whmcs2.png) ##### Create a product group Create a new Product Group (System settings > Products/Services). ![servers](img/whmcs3.png) ##### Create a product Create a new Product. ![servers](img/whmcs4.png) - `Product Type` Other - `Product Group` should be the one you just created - `Product Name` - `Module` VirtFusion Direct Provisioning Once the product is created, click the `Module Settings` tab. ![servers](img/whmcs5.png) - `Server Group` select the server group you created earlier - `Hypervisor Group ID` can be found in VirtFusion (Compute Resources > Hypervisor Groups > ID (table column)) - `Package ID` can be found in VirtFusion (Packages > ID (table column)) Continue to configure the product as normal. ##### Configurable Options The following configurable options are available. | Name | Description | |----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **IPv4** | The amount of IPv4 address to be added to the server on creation. This will override the default IPv4 specified in the product module settings. | | **Package** | The ID of a package in VirtFusion. This will override the default package specified in the product module settings. | | **Location** | The ID of a hypervisor group in VirtFusion. This will override the default hypervisor group specified in the product module settings. | | **Storage** | An amount of storage in GB for the primary disk. This will override the storage value for the specified package. | | **Memory** | An amount of memory in MB. This will override the memory value for the specified package. | | **Bandwidth** | An amount of traffic in GB. This will override the traffic value for the specified package. | | **Inbound Network Speed** | The inbound network speed in kB/s in the range of 0 - 9999999999 (0 = unlimited). This will override the inbound network speed value for the specified package. | | **Outbound Network Speed** | The outbound network speed in kB/s in the range of 0 - 9999999999 (0 = unlimited). This will override the outbound network speed value for the specified package. | | **CPU Cores** | An amount of CPU cores. This will override the CPU cores value for the specified package. | | **Network Type** | The ID of a network profile. This will override the network profile for the specified package. | | **Storage Type** | The ID of a storage profile. This will override the storage profile for the specified package. | You must use the exact name specified for a configurable option to work, however, the names can be overridden using configurable option mapping. To use config option mapping you may rename `ConfigOptionMapping-example.php` in the `config` folder to `ConfigOptionMapping.php` (it won't be overwritten on updates) and edit the values in the array to match your configurable option names. ```php return [ 'ipv4' => 'IPv4', 'packageId' => 'Package', 'hypervisorId' => 'Location', 'storage' => 'Storage', 'memory' => 'Memory', 'traffic' => 'Bandwidth', 'networkSpeedInbound' => 'Inbound Network Speed', 'networkSpeedOutbound' => 'Outbound Network Speed', 'cpuCores' => 'CPU Cores', 'networkProfile' => 'Network Type', 'storageProfile' => 'Storage Type', ]; ``` **Do not edit the array keys, these must remain as specified in the script.** ###### Examples ###### Location selector (optional) You may use configurable options to enable a location selector for your order forms. A Location configurable option will override the default `Hypervisor Group ID` configured in the package. ![options](img/whmcs7.png) The option name must be `Location` (if you are not overriding the default name) and options must use the format `Hypervisor Group ID|Friendly Name`. ###### IPv4 selector (optional) You may use configurable options to enable a IPv4 selector for your order forms. A IPv4 configurable option will override the `Default IPv4` configured in the package. ![options](img/whmcs8.png) The option name must be `IPv4`. #### Multiple WHMCS instances (optional) By default, the module links a WHMCS client to a VirtFusion user using the WHMCS client ID (stored in VirtFusion as the `External Relational Identifier`). If more than one WHMCS installation provisions to the same VirtFusion control server, client IDs will collide, e.g. client `123` on one installation would match client `123` on another. To run multiple WHMCS installations against a single VirtFusion control server, you can define a unique prefix for each installation. Users are then identified by a relational string consisting of the prefix followed by the WHMCS client ID (e.g. `VF123`) instead of the plain client ID. This requires module version 2.5 or later. To enable this, rename `MultiInstance-example.php` in the `config` folder to `MultiInstance.php` (it won't be overwritten on updates) and set your prefix. ```php return [ 'prefix' => 'VF', ]; ``` The prefix must be 1-5 characters, letters and digits only, starting with a letter (e.g. `VF`, `AB`, `VF2`). Give each WHMCS installation its own unique prefix. :::warning - The prefix must be in place **before** any users are provisioned and must not be changed afterwards. Users provisioned without a prefix (or with a different one) will not be found once it is enabled. - If `MultiInstance.php` exists with an invalid prefix, all user-related actions (provisioning, control panel login, password reset) will fail with an error rather than falling back to non-prefixed IDs. ::: ### Emails It's a good idea to assign a welcome email to each of your products so customers are informed when their server is ready. Here are a few examples (and yes, we know—AI at its finest!). #### Example 1 ```text Hello {$client_name}, Your server has been successfully created and is ready for you to configure. You can manage all aspects of your server using our best-in-class, fully featured control panel. To access the control panel, please visit {$whmcs_url}, select your server from the list of services, and click the "Open Control Panel" button. Enjoy your new server from all of us at {$company_name}! ``` #### Example 2 ```text Hello {$client_name}, Your server has been successfully created and is now ready for you to configure. You can manage every aspect of your server using our best-in-class, fully featured control panel. Here’s what you can do: - Choose from a comprehensive range of operating systems. - Use a Custom ISO for your server. - Monitor your server with detailed statistics. - Access power functions to control your server effortlessly. To get started, visit {$whmcs_url}, select your server from the list of services, and click the "Open Control Panel" button. Enjoy your new server from all of us at {$company_name}! ``` #### Example 3 ```text Hey {$client_name}, Guess what? Your server’s finally been created. Took long enough, right? Anyway, it’s sitting there, waiting for you to get off your butt and configure it. You’ve got this fancy-schmancy control panel at your disposal—yeah, we know, we’re the best. To get to it, drag yourself over to {$whmcs_url}, pick your server from the list (you do remember which one it is, right?), and hit the "Open Control Panel" button. It’s not rocket science. So, go ahead and enjoy your shiny new server. And if you mess it up, well, that’s on you. Cheers from the sarcastic bunch at {$company_name}! ``` #### Example 4 ```text Dear {$client_name}, We are pleased to inform you that your server has been successfully created and is now ready for configuration. You may manage every aspect of your server through our state-of-the-art, fully featured control panel. To access this control panel, kindly visit {$whmcs_url}. Upon logging in, please select your server from the list of available services and click the "Open Control Panel" button. We trust that this server will meet your expectations and serve your needs effectively. Should you require any assistance, our team is at your service. Warm regards, The Team at {$company_name} ``` ### Troubleshooting #### You receive the error _**Oops! Something went wrong. Try again later**_ when clicking the Open Control Panel button This issue can occur when there is a communication problem between WHMCS and VirtFusion or modifications on your WHMCS installation. There are several checks you can perform: - Make sure the WHMCS System URL is correct and using `https` under the WHMCS general settings. - If the user is a VirtFusion administrator, the functionally will not work. It only allows authentication for normal users. - The user was created in VirtFusion manually and not by the module. If this is the case, you will need to take the `WHMCS user ID` and input the ID in the `External Relational Identifier` field under the user settings in VirtFusion. If you are using [multi-instance mode](#multiple-whmcs-instances-optional), input the prefixed relational string (e.g. `VF123`) in the `Relational String` field instead. - The WHMCS client area template is custom and is causing javascript errors. Check the browser console to see any errors generated. If all else fails, enable module logging in WHMCS and check the logs generated for errors. --- ## Introduction ## VirtFusion Documentation VirtFusion is Affordable, Reliable virtualization management software for the hosting industry. It's built to reduce the daily administration burden and allow your customers the experience they deserve. :::warning Important VirtFusion is a B2B platform designed specifically for service providers and hosting professionals. As such, we expect users to have a reasonable level of Linux expertise, including advanced networking and system administration skills. If you're not confident in these areas, we recommend engaging with an experienced system administrator to assist with the deployment and ongoing management of your environment. ::: ### Product Status | Version | Status | |---------|------------------------| | 7.x | Current Stable Version | | 6.x | Old Stable Version | | 5.x | Legacy Version | | 4.x | Legacy Version | | 3.x | Legacy Version | | 2.x | Legacy Version | | 1.x | Legacy Version | ### Billing System Integration | System | Documentation | Website | |-------------|:---------------------------------:|------------------------------------------------------------| | Blesta | [link](/integrations/blesta) | [https://www.blesta.com](https://www.blesta.com) | | Clientexec | [link](/integrations/clientexec) | [https://www.clientexec.com](https://www.clientexec.com) | | WHMCS | [link](/integrations/whmcs) | [https://www.whmcs.com](https://www.whmcs.com) | | BillingServ | [link](/integrations/billingserv) | [https://www.billingserv.com](https://www.billingserv.com) | | HostBill | [link](/integrations/hostbill) | [https://hostbillapp.com](https://hostbillapp.com) | | Paymenter | [link](/integrations/paymenter) | [https://paymenter.org](https://paymenter.org) | | Upmind | [link](/integrations/upmind) | [https://upmind.com](https://upmind.com) | ### Support Our product is fully supported within the license fee. For official support [follow this link](https://account.virtfusion.com/client/plugin/support_manager/client_tickets/add/1/). ### Licensing A fully featured 30 day evaluation license is activated on first install. [Production licenses can be obtained here](https://account.virtfusion.com). --- ## Licensing Explained ### Purchasing a License VirtFusion is licensed on a per cluster basis and any hypervisors within your cluster should be configured within your license. This means you will only require one license key. #### Options Depending on the license type, the following license options are: - **Hypervisors with UNLIMITED servers** - This option allows as many VMs as you would like on the hypervisor. - **Hypervisors with 35 servers** - This options allows a maximum of 35 VMs on the hypervisor. - **Hypervisors with 5 servers** - This option allows a maximum of 5 VMs on the hypervisor. - **Hypervisors with 1 server** - This option allows a maximum of 1 VM on the hypervisor. - **Branding Removal** - This option removes the VirtFusion branding from the control server. #### Example This is an example if you have 5 hypervisors: ![license](img/license-structure.png) 2 hypervisors can host unlimited VMs, 1 a maximum of 35 VMs, 1 a maximum of 5 VMs and 1 a maximum of 1 VM. And the license order form for the above example would be: ![license](img/license1.png) ### Upgrading or Downgrading a License You may upgrade or downgrade your license at any time (as long as no invoices are pending payment). To modify your license, you will need to [login to your account](https://account.virtfusion.com). Once logged in, navigate to your license and click the `Change Configurable Options` button. On the next screen you will see your current options and also options that are not currently set on your license. ![license](img/license3.png) You can now change the options to fit your new requirements. Once you are happy with your changes, click the `Review` button to view the cost of the changes before you commit. ### Re-issuing a License Due to licenses being locked to your control server, we have made it possible to unlock the license if you need to move your control server to a new location. To unlock your license, you will need to [login to your account](https://account.virtfusion.com). Once logged in, navigate to your license and click `Manage License`. You will see an `Option` to Reissue. Just tick that and press `Save`. ![license](img/license2.png) :::caution Be aware that a limited number of re-issues can be performed per license. If your re-issue option is locked, you should contact support to have your license reset. ::: ### Account Limits We actively monitor account creations and cross-check license information across multiple accounts. If a match is detected, licenses may be automatically cancelled without notice. To avoid issues or disruptions, please do not create multiple accounts. Maintaining a single account helps ensure compliance with our licensing policies and smooth operation of your services. We may impose a limit on the number of licenses allowed under a single account. If you reach this limit, you will need to submit a billing support ticket so we can review your situation and work toward a resolution. ### My License Won't Validate License validation requests are rejected if the control server clock is out of sync by more than 60 seconds. #### RHEL / Rocky / Alma **Check current status** ``` timedatectl ``` **Enable NTP sync** ``` timedatectl set-ntp true ``` **Force immediate sync** ``` chronyc makestep ``` **Or if using ntpd** ``` systemctl stop ntpd ntpdate pool.ntp.org systemctl start ntpd ``` **Verify** ``` timedatectl ``` #### Debian / Ubuntu **Check current status** ``` timedatectl ``` **Enable NTP sync** ```timedatectl set-ntp true ``` **Force immediate sync** ``` systemctl restart systemd-timesyncd ``` **Or if using chrony** ``` chronyc makestep ``` **Or if using ntp** ``` systemctl stop ntp ntpdate pool.ntp.org systemctl start ntp ``` **Verify** ``` timedatectl ``` --- ## CLI ## CLI Reference ### Reset the Traffic Accounting Statistics To reset all local traffic accounting statistics, use the following command on the control server. ```shell vfcli-ctrl server:traffic-statistics reset-all ``` To reset ALL traffic accounting statistics (including the datastore), use the following command on the control server. ```shell vfcli-ctrl server:traffic-statistics reset-all --include-datastore ``` ### Cancel a Task You may cancel a task if its run time is more than 10 minutes. ```shell vfcli-ctrl task:cancel ``` ### Force Remove a Hypervisor and its Servers ```shell vfcli-ctrl hypervisor:force-remove ``` ### Set a Hypervisor to Re-Commission State ```shell vfcli-ctrl hypervisor:re-commission ``` ### Reset an Administrator Password ```shell vfcli-ctrl user:reset-admin-password ``` ### Reset System Access Credentials ```shell vfcli-ctrl system:reset-access-credentials ``` ### Force Flush Backups based on Storage ID ```shell vfcli-ctrl backups:force-flush ``` ### Regenerate Backup Storage Keys ```shell vfcli-ctrl backup-storage:reset-key ``` ### Disable Captcha ```shell vfcli-ctrl captcha-off ``` ### Disable forcing admins and end users to activate TFA ```shell vfcli-ctrl tfa-force-off ``` ### Change Server Creation Date :::warning Important Changing a server creation date will remove the monthly historical traffic statistics. ::: ```shell vfcli-ctrl server:creation-date SERVER_ID 'YYYY-MM-DD HH:MM:SS' ``` ### Retrieve Last Used Password for a Server ```shell vfcli-ctrl server:password show ``` ### Admin IP Restrict #### Enable ```shell vfcli-ctrl admin:ip-restrict enable ``` #### Disable ```shell vfcli-ctrl admin:ip-restrict disable ``` #### Add address ```shell vfcli-ctrl admin:ip-restrict addip IP_TO_ADD ``` #### Delete address ```shell vfcli-ctrl admin:ip-restrict delip IP_TO_DELETE ``` #### List addresses ```shell vfcli-ctrl admin:ip-restrict list ``` ### Server Do Not Destroy (DND) When DND is enabled for a server, it prevents it from being deleted, re-installed, ISO mounts and password resets. This includes end users, admins, and API. The only way to delete the server is to disable DND. #### Enable ```shell vfcli-ctrl server:dnd enable SERVER_ID ``` #### Disable ```shell vfcli-ctrl server:dnd disable SERVER_ID ``` ### Apply storage QoS to all servers by hypervisor Regenerates the server configurations and live-applies the disk I/O limits currently stored to every disk on each server. ```shell vfcli-ctrl server:configurations HYPERVISOR_ID --options=storage-qos ``` #### Set storage QoS values and apply them Sets the disk I/O limits on the every disk on the hypervisor, then live-applies them. Values are passed as `column:value` pairs. ```shell vfcli-ctrl server:configurations HYPERVISOR_ID --options=storage-qos --qos=read_iops_sec:500,write_iops_sec:500,read_bytes_sec:104857600,write_bytes_sec:104857600 ``` Available QoS columns: - `read_iops_sec` — read operations per second - `write_iops_sec` — write operations per second - `read_bytes_sec` — read throughput in bytes per second - `write_bytes_sec` — write throughput in bytes per second --- ## Cloud-init Reference ### Ubuntu `accept-ra: false` This will set `accept-ra` to false on all Ubuntu templates using the `ens3` interface naming. ```yaml write_files: - encoding: b64 content: bmV0d29yazoKICB2ZXJzaW9uOiAyCiAgZXRoZXJuZXRzOgogICAgZW5zMzoKICAgICAgYWNjZXB0LXJhOiBmYWxzZQ== owner: 'root:root' path: /etc/netplan/99-user-conf.yaml permissions: '0600' ``` --- ## Required Ports ## Ports used by VirtFusion ### Control server (UI) | Port | Type | Description | | ------ | ---------- |-------------------------------------| | 80 | http | Used for acquiring SSL certificates | | 443 | https | Public access to the user interface | ### Hypervisor | Port | Type | Description | |-------- | ----------|------------------------------------| | 8892 | https | Control & Hypervisor communication | | 5900+ | VNC | Used for VNC | ** UFW is installed and enabled on VirtFusion hypervisors by default. It's advised to use UFW if you would like to manage firewall rules.** [UFW man pages](http://manpages.ubuntu.com/manpages/jammy/man8/ufw.8.html) --- ## Updating ## Updating VirtFusion It's very important to keep your installation up to date, especially now while major features are still being added and refined. ### Updating the whole cluster To update the whole cluster to the latest version run the following command from the control server via SSH. ```shell vfcli-ctrl update ``` ### Updating a single hypervisor To update a single hypervisor to the latest version run the following command from the hypervisor via SSH. ```shell vfcli-hv update ``` --- ## Services import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Package & Service Updates ### `December 27, 2025` #### Updated Packages | Package | Version | Systems | Distribution | Arch | Security | Notes | |:-----------------------------------------------------------|:---------------------------------------------------|:--------------------|:----------------------------------------------------------------------------|:-------|:--------:|:-------------------------------------------| | vf-php8 | 8.3.29 | Control, Hypervisor | Debian 13, Debian 12, Debian 11, Ubuntu 24.04, Ubuntu 22.04, RHEL 9, RHEL 8 | x86_64 | 🚨 | https://www.php.net/ChangeLog-8.php#8.3.29 | | vf-system-php8 | 8.3.29 | Hypervisor | Debian 13, Debian 12, Debian 11, Ubuntu 24.04, Ubuntu 22.04, RHEL 9, RHEL 8 | x86_64 | 🚨 | https://www.php.net/ChangeLog-8.php#8.3.29 | ### `June 10, 2025` #### Updated Packages | Package | Version | Systems | Distribution | Arch | Security | Notes | |:-----------------------------------------------------------|:---------------------------------------------------|:--------------------|:-------------------------------------------------------------------------------------|:--------|:--------:|:-------------------------------------------| | vf-php8 | 8.2.28 | Control, Hypervisor | Debian 12, Debian 11, Ubuntu 24.04, Ubuntu 22.04, RHEL 9, RHEL 8 | x86_64 | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.28 | | vf-system-php8 | 8.2.28 | Hypervisor | Debian 12, Debian 11, Ubuntu 24.04, Ubuntu 22.04, RHEL 9, RHEL 8 | x86_64 | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.28 | | vf-php8 | 8.2.28 | Hypervisor | Debian 12 | AArch64 | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.28 | | vf-system-php8 | 8.2.28 | Hypervisor | Debian 12 | AArch64 | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.28 | | vf-nginx | 1.28.0 | Control, Hypervisor | Debian 12 | AArch64 | | https://nginx.org/en/CHANGES-1.28 | | vf-nginx | 1.28.0 | Control, Hypervisor | Debian 13, Debian 12, Debian 11, Ubuntu 24.04, Ubuntu 22.04, RHEL 10, RHEL 9, RHEL 8 | x86_64 | | https://nginx.org/en/CHANGES-1.28 | | vf-php8 | 8.3.22 | Control, Hypervisor | Debian 13, RHEL 10 | x86_64 | | https://www.php.net/ChangeLog-8.php#8.3.22 | | vf-system-php8 | 8.3.22 | Hypervisor | Debian 13, RHEL 10 | x86_64 | | https://www.php.net/ChangeLog-8.php#8.3.22 | ### `January 6, 2025` :::danger Security Release This release addresses several vulnerabilities that could potentially impact the security and performance of your system. ::: #### Updated Packages | Package | Version | Systems | Security | Notes | |:---------------|:---------|:--------------------|:--------:|:-------------------------------------------| | vf-php8 | 8.2.27-1 | Control, Hypervisor | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.27 | | vf-system-php8 | 8.2.27-1 | Hypervisor | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.27 | | vf-nginx | 1.26.2-1 | Control, Hypervisor | | https://nginx.org/en/CHANGES-1.26 | #### How to Update The following commands which should be run as **root** in **SSH** have been pre-prepared for your base distribution. **Debian/Ubuntu** **Control** ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 -y vfcli-ctrl app:restart --services=queue,fpm,web ``` **Hypervisor** ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 vf-system-php8 -y vfcli-hv app:restart --services=queue,system-fpm,fpm,web ``` **RHEL** **Control** ```shell dnf makecache -y && dnf update vf-nginx vf-php8 -y vfcli-ctrl app:restart --services=queue,fpm,web ``` **Hypervisor** ```shell dnf makecache -y && dnf update vf-nginx vf-php8 vf-system-php8 -y vfcli-hv app:restart --services=queue,system-fpm,fpm,web ``` [Learn more about how to update your system packages](/guides/keeping-up-to-date#services) ### `June 7, 2024` :::danger Security Release This release addresses several high-priority vulnerabilities that could potentially impact the security and performance of your system. ::: #### Updated Packages | Package | Version | Systems | Security | Notes | |:---------------|:---------|:--------------------|:--------:|:-------------------------------------------| | vf-php8 | 8.2.20-1 | Control, Hypervisor | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.20 | | vf-system-php8 | 8.2.20-1 | Hypervisor | 🚨 | https://www.php.net/ChangeLog-8.php#8.2.20 | | vf-nginx | 1.26.1-1 | Control, Hypervisor | 🚨 | https://nginx.org/en/CHANGES-1.26 | #### How to Update The following commands which should be run as **root** in **SSH** have been pre-prepared for your base distribution. **Debian/Ubuntu** **Control** ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 -y vfcli-ctrl app:restart --services=queue,fpm,web ``` **Hypervisor** ```shell apt-get update -y && apt-get install --only-upgrade vf-nginx vf-php8 vf-system-php8 -y vfcli-hv app:restart --services=queue,system-fpm,fpm,web ``` **RHEL** **Control** ```shell dnf makecache -y && dnf update vf-nginx vf-php8 -y vfcli-ctrl app:restart --services=queue,fpm,web ``` **Hypervisor** ```shell dnf makecache -y && dnf update vf-nginx vf-php8 vf-system-php8 -y vfcli-hv app:restart --services=queue,system-fpm,fpm,web ``` [Learn more about how to update your system packages](/guides/keeping-up-to-date#services) --- ## Version 1.x ### 1.9.x #### 1.9.2 `November 19, 2022` - Fixed issue where setting no logo for the misc OS template group would throw an error in the end user area. - Fixed several display bugs in the server firewall list. - Added option to supply a TPM (1.2 or 2.0) device to a VM. - Added option to enable/disable UEFI on a per VM basis. - Added option to enable/disable the VM boot menu. #### 1.9.1 `November 17, 2022` - Improved server creation with permissive locking. - Improved SMTP SSL auto negotiation. - Fixed issue with object parsing in task debug output. - Added maintenance mode ACL to Admin Access Groups. - Added simple host level firewall on a per VM interface basis. This is an experimental feature. #### 1.9.0 `November 11, 2022` `IMPORTANT NOTICE` This version contains a major framework update. As stated in previous communications, some early environments may not be compatible with this update. Please see [https://docs.virtfusion.com/php-upgrade](https://docs.virtfusion.com/php-upgrade) for more information. - Major framework update (v8 -> v9). - Added option to set the timezone of a VM on build. - Added ability to set a default timezone for hypervisors. (used when building a VM). - Added software compatibility checker to software updater. - Fixed issues with some toggle buttons not working correctly in admin area. - Added memory balloon options for VMs (enable/disable, auto deflate, free page reporting). - Added advanced settings for VMs running Microsoft Windows. Admin -> VM -> Settings -> Microsoft Windows. - Added modal alerts when installing/re-install a VM from user area. - Added CPU (priority, topology, model), machine type and PCI ports to VM packages. - Added Microsoft Windows options to OS templates. - Added ability to disable memory ballooning on a per OS template basis. - Added new Windows Direct RDP deployment profile to OS template settings (used for new Windows templates). - Added Microsoft Windows settings to hypervisor settings (for OS template inheritance). - Added optimized Windows templates to template manager. - Fixed issue where an unsuspend action could be initiated before a scheduled suspend action completes. - Fixed issue with email local domain. - Added no-store cache control headers to all web requests. - Removed unused broadcasting routes. - Added the ability for end users to add more addresses from their assigned IPv6 subnets. - Improved cloud-init configuration for VM migrations and IP changes. - Reserved IPs will now sync to RDNS zones. - Multiple general admin UI improvements. - Added YubiKey OTP as a two-factor authentication option. - Added the option to set the admin menu header item color. - Removed the requirement for an IP block to have a gateway and resolvers defined. - Numerous layout, javascript and CSS improvements. - Added user session log (System -> User Sessions). - Added Hypervisor groups filter and column to hypervisors table. - Fixed issue where the IPv6 status was being called twice in the end user area. - Added the ability to force IPv6 assignment when a VM is built (Hypervisor -> Force IPv6). - Added option to change the name of a VM primary or secondary network interface. - Added IPv6 subnet notes. - Fixed issue with IPv6 address allocation routine. - Fixed issue where the task queue would show tasks for other hypervisors when viewing a specific hypervisor. - Added the ability to disable specific emails. - Improved traffic limit notifications for administrators. - Fixed issue where an operating system logo would show a broken link if no ID was specified. - Improved task handling when the system is in maintenance mode. - Added the option to trim (sparsify) VM disks. - Added support for Google reCAPTCHA v2 "I'm not a robot", Cloudflare Turnstile, hCaptcha and MTCaptcha (Settings -> General). - Added `vfcli-ctrl captcha-off` CLI command. - Email variables are now passed to and parsed in operating system email excerpts. - Added API method to retrieve a server package by ID. [https://docs.virtfusion.com/api/#api-Packages-Retrieve_a_server_package_by_id](https://docs.virtfusion.com/api/#api-Packages-Retrieve_a_server_package_by_id) - Added the ability to include custom XML in a server configuration file. - Add IPv4 Sub Blocks with reverse DNS support. - Added API method to add ip addresses to a server interface whitelist. [https://docs.virtfusion.com/api/#api-Servers-Add_a_IPv4_or_IPv6_address_to_the_server_interface_whitelist](https://docs.virtfusion.com/api/#api-Servers-Add_a_IPv4_or_IPv6_address_to_the_server_interface_whitelist) - Added API method to remove ip addresses from a server interface whitelist. [https://docs.virtfusion.com/api/#api-Servers-Delete_a_IPv4_or_IPv6_address_from_the_server_interface_whitelist](https://docs.virtfusion.com/api/#api-Servers-Delete_a_IPv4_or_IPv6_address_from_the_server_interface_whitelist) - Added IP Block Groups with the ability to assign a group to a hypervisor network interface. - Improved CLI task cancelling functionality. - Updated server operating system logos (all known variants). - Fixed issue where the admin server table filters would reset when starting a migration. - Added VNC option to admin VM management area. - Sectioned admin VM management area (tabs). - User impersonation will now return to the last known admin url when exiting. - Added basic sorting to the admin server table. - Added filters to hypervisor table. - Fixed issue where network types were not being honoured when building a VM. - Improved VM backup error checking and scheduling. - VM backup and restore history is now shown for VMs that are pending deletion. - Added API method to reset a VM password. [https://docs.virtfusion.com/api/#api-Servers-Reset_Password](https://docs.virtfusion.com/api/#api-Servers-Reset_Password) #### 1.9.0 `Testing Build 5` `November 4, 2022` - Added PHP compatibility checker to software updater. - Fixed issues with some toggle buttons not working correctly in admin area. - Added clock field to VM custom XML elements. - Added memory balloon options for VMs (enable/disable, auto deflate, free page reporting). - Added CPU tune field to VM custom XML elements (can be used for pinning). - Added advanced settings for VMs running Microsoft Windows. Admin -> VM -> Settings -> Microsoft Windows. - Added modal alerts when installing/re-install a VM from user area. - Added CPU (priority, topology, model), machine type and PCI ports to VM packages. - Added Microsoft Windows options to OS templates. - Added ability to disable memory ballooning on a per OS template basis. - Added new Windows Direct RDP deployment profile to OS template settings (used for new Windows templates). - Added Microsoft Windows settings to hypervisor settings (for OS template inheritance). - Added optimized Windows templates to template manager. #### 1.9.0 `Testing Build 4` `October 25, 2022` - Major framework update (v8 -> v9). - Fixed issue where an unsuspend action could be initiated before a scheduled suspend action completes. - Fixed issue with email local domain. - Fixed enable/disable switch for VM disks. - Added no-store cache control headers to all web requests. - Removed unused broadcasting routes. - Added the ability for end users to add more addresses from their assigned IPv6 subnets. - Improved cloud-init configuration for VM migrations and IP changes. - Reserved IPs will now sync to RDNS zones. - Several admin UI improvements. #### 1.9.0 `Testing Build 3` `October 18, 2022` - Added hCaptcha and MTCaptcha as supported captchas. - Added YubiKey OTP as a two-factor authentication option. - Fixed CSS issues caused by the latest framework update. - Added the option to set the admin menu header item color. - Removed the requirement for an IP block to have a gateway and resolvers defined. - Numerous layout, javascript and CSS improvements. #### 1.9.0 `Testing Build 2` `October 14, 2022` - Added user session log (System -> User Sessions). - Added Hypervisor groups filter and column to hypervisors table. - Fixed issue where the IPv6 status was being called twice in the end user area. - Added the ability to force IPv6 assignment when a VM is built (Hypervisor -> Force IPv6). - Added option to change the name of a VM primary or secondary network interface. - Added IPv6 subnet notes. - Fixed issue with IPv6 address allocation routine. - Fixed issue where the task queue would show tasks for other hypervisors when viewing a specific hypervisor. - Added the ability to disable specific emails. - Improved traffic limit notifications for administrators. - Fixed issue where an operating system logo would show a broken link if no ID was specified. - Improved task handling when the system is in maintenance mode. - Added the option to trim (sparsify) VM disks. - Added support for Google reCAPTCHA v2 "I'm not a robot" and Cloudflare Turnstile (Settings -> General). - Added `vfcli-ctrl captcha-off` CLI command. #### 1.9.0 `Testing Build 1` `October 11, 2022` - Email variables are now passed to and parsed in operating system email excerpts. - Added API method to retrieve a server package by ID. https://docs.virtfusion.com/api/#api-Packages-Retrieve_a_server_package_by_id - Added the ability to include custom XML in a server configuration file. - Add IPv4 Sub Blocks with reverse DNS support. - Added API method to add ip addresses to a server interface whitelist. https://docs.virtfusion.com/api/#api-Servers-Add_a_IPv4_or_IPv6_address_to_the_server_interface_whitelist - Added API method to remove ip addresses from a server interface whitelist. https://docs.virtfusion.com/api/#api-Servers-Delete_a_IPv4_or_IPv6_address_from_the_server_interface_whitelist - Added IP Block Groups with the ability to assign a group to a hypervisor network interface. - Improved CLI task cancelling functionality. - Updated server operating system logos (all known variants). - Fixed issue where the admin server table filters would reset when starting a migration. - Added VNC option to admin VM management area. - Sectioned admin VM management area (tabs). - User impersonation will now return to the last known admin url when exiting. - Added basic sorting to the admin server table. - Added filters to hypervisor table. - Fixed issue where network types were not being honoured when building a VM. - Improved VM backup error checking and scheduling. - VM backup and restore history is now shown for VMs that are pending deletion. - Added API method to reset a VM password. https://docs.virtfusion.com/api/#api-Servers-Reset_Password ### 1.8.x #### 1.8.0 `September 30, 2022` - Added the ability to restrict administrators to specific roles through access groups. - Added server packages API endpoint. https://docs.virtfusion.com/api/#api-Packages-Retrieve_server_packages - Fixed issue where IPs were not showing correctly when viewing an IP block. - Fixed issue with traffic collections when a hypervisor has no VMs. - Added ability for super admins to impersonate admins. - Improved hypervisor ISO downloader. - CPU model can now be set on a global, hypervisor or VM level. - VM CPUs can now be prioritized and throttled. - Added API endpoint to add IPv4 addresses to a VM. https://docs.virtfusion.com/api/#api-Servers-Add_IPv4 - Added system schedule log. - Added global API log. - Added user authentication log. - Added global search by VM interface ID. - Optimized adding a specific IPv6 subnet to a VM. - Added debugging methods to traffic collectors. - Operating system template downloads now check for an existing download process. - Server builds now do a connection check and verification of the hypervisor software version before proceeding with the build process. - Added the option to rebuild hypervisor configurations (Hypervisors -> Selected hypervisors -> Update configuration). - Added the ability to rebuild the Local VNC configuration file. (Settings -> General -> VNC configuration). - Added `vfcli-ctrl update-testing` & `vfcli-hv update-testing` CLI commands. - Added Bleeding software update channel (Only use if asked). - Fixed an issue where a CD/DVD was listed twice in the user area if global access was enabled and users were assigned. - Added API endpoint to add a quantity of IPv4 to a VM. https://docs.virtfusion.com/api/#api-Servers-Add_a_quantity_of_IPv4 - Added API endpoint to remove IPv4 from a VM. https://docs.virtfusion.com/api/#api-Servers-Delete_IPv4 - Added API endpoint to retrieve available operating system template for a package. https://docs.virtfusion.com/api/#api-Media-Retrieve_grouped_OS_templates_available_for_server_package - Added CLI command to cancel an active task `vfcli-ctrl task:cancel`. - Added the ability to define QEMU command line parameters on a per VM basis. - Added global, hypervisor and VM level option to set the machine type of a VM (pc/q35 + PCI slots). - Major hypervisor application framework update. - Added sticky search filters to server, user & ipv4 table in admin UI. - Added option to set the density of the admin ui. - Added the ability to apply package settings to an existing VM (upgrade/downgrade). - Added change package API endpoint https://docs.virtfusion.com/api/#api-Servers-Change_package - Added VM power control API endpoint https://docs.virtfusion.com/api/#api-Servers-power - Added the ability to set the VM machine type (q35 support) and quantity of PCI ports. - Fixed issue where some OS templates would show to end users when disabled. - Added VM CPU topology (Sockets/Cores/Threads). - VM operating systems (logo, distro & kernel) are now shown in the end user and admin areas if detected via the guest agent. - Added the option to set the amount of packet processing queues from a VM network. - Template download and VM build system has been rebuilt with progress support. #### 1.8.0 `Testing Build 6` `September 22, 2022` - Added sticky search filters to server, user & ipv4 table in admin UI. - Added option to set the density of the admin ui. - Added the ability to apply package settings to an existing VM (upgrade/downgrade). - Added change package API endpoint https://docs.virtfusion.com/api/#api-Servers-Change_package - Added VM power control API endpoint https://docs.virtfusion.com/api/#api-Servers-power - Added option to supply qemu command line parameters directly to a VM. - Added the ability to set the VM machine type (q35 support) and quantity of PCI ports. - Fixed issue where some OS templates would show to end users when disabled. - Added VM CPU topology (Sockets/Cores/Threads). - VM operating systems (logo, distro & kernel) are now shown in the end user and admin areas if detected via the guest agent. - Added the option to set the amount of packet processing queues from a VM network. - Template download and VM build system has been rebuilt with progress support. #### 1.8.0 `Testing Build 5` `September 13, 2022` - Added Bleeding software update channel (Only use if asked). - Fixed an issue where a CD/DVD was listed twice in the user area if global access was enabled and users were assigned. - Added API endpoint to add a quantity of IPv4 to a VM. https://docs.virtfusion.com/api/#api-Servers-Add_a_quantity_of_IPv4_to_a_server - Added API endpoint to remove IPv4 from a VM. https://docs.virtfusion.com/api/#api-Servers-Delete_IPv4_from_a_server - Added API endpoint to retrieve available operating system template for a package. https://docs.virtfusion.com/api/#api-Media-Retrieve_grouped_OS_templates_available_for_server_package - Added CLI command to cancel an active task `vfcli-ctrl task:cancel`. - Added the ability to define QEMU command line parameters on a per VM basis. - Added global, hypervisor and VM level option to set the machine type of a VM (pc/q35 + PCI slots). - Major hypervisor application framework update. #### 1.8.0 `Testing Build 4` `September 5, 2022` - Added system schedule log. - Added global API log. - Added user authentication log. - Added global search by VM interface ID. - Optimized adding a specific IPv6 subnet to a VM. - Added debugging methods to traffic collectors. - Operating system template downloads now check for an existing download process. - Server builds now do a connection check and verification of the hypervisor software version before proceeding with the build process. - Added the option to rebuild hypervisor configurations (Hypervisors -> Selected hypervisors -> Update configuration). - Added the ability to rebuild the Local VNC configuration file. (Settings -> General -> VNC configuration). - Added `vfcli-ctrl update-testing` & `vfcli-hv update-testing` CLI commands. - Application dependencies have been updated to their latest versions (framework). #### 1.8.0 `Testing Build 3` `August 17, 2022` - Added ability for super admins to impersonate admins. - Improved hypervisor ISO and template downloader. - CPU model can now be set on a global, hypervisor or VM level. - VM CPUs can now be prioritized and throttled. - Added API endpoint to add IPv4 addresses to a VM. #### 1.8.0 `Testing Build 2` `August 13, 2022` - Fixed an issue where the default sound card would sometimes prevent a VM from booting. #### 1.8.0 `Testing Build 1` `August 12, 2022` - Added the ability to restrict administrators to specific roles through access groups. - Added server packages API endpoint. - Fixed issue where IPs were not showing correctly when viewing an IP block. - Fixed issue with traffic collections when a hypervisor has no VMs. ### 1.7.x #### 1.7.3 `August 13, 2022` - Fixed an issue where the default sound card would sometimes prevent a VM from booting. #### 1.7.2 `August 11, 2022` - Fixed an issue where operating system templates were not showing the full name in the admin UI. - Used CPU cores now show correctly when viewing grouped hypervisor resources. - Bulk admin notification emails are now sent individually. - Fixed issue where a gateway4 element would remain in the cloud-init network configuration even when no IPv4 addresses were specified. - Added support for new hypervisor licensing options. - Improved interaction with the QEMU guest agent. - Added support for MacVTap hypervisor networking (default on new installations). - It's now possible to edit the primary network interface name for hypervisors. - When commissioning a new hypervisor, the system will attempt to auto-populate resource limits (memory, cpu). - Added hypervisor server events (hooks). https://docs.virtfusion.com/administration/event-hooks/overview - Fixed several UI typos. - Added package overrides for storage, traffic, memory, cpu cores, network inbound speed, network outbound speed, storage profile and network profile to the API create server endpoint. - Backend changes for OVMF firmware paths. - Added API endpoint to build a VM after it's been created. - Improved reliability when adding and removing additional VM disks. - Added API endpoint to manage SSH keys. - Fixed an issue where invalid routes were being added to VM secondary interface. - Fixed an issue where clicking the server icon on the main dashboard would load the user profile. - Added a delay query parameter to the API server destroy method. - Fixed an issue when removing a secondary interface from a VM would fail trigger a cloud-init action. - VMs that use host-passthrough now use the passthrough cache mode. - Boot menu is now enabled by default for a VM. #### 1.7.2 `Testing Build 4` `August 7, 2022` - Added API endpoint to build a VM after it's been created. - Improved reliability when adding and removing additional VM disks. - Added API endpoint to manage SSH keys. - Fixed an issue where invalid routes were being added to VM secondary interface. - Fixed an issue where clicking the server icon on the main dashboard would load the user profile. - Added a delay query parameter to the API server destroy method. - Fixed an issue when removing a secondary interface from a VM would fail trigger a cloud-init action. #### 1.7.2 `Testing Build 3` `July 31, 2022` - Fixed several UI typos. - Added package overrides for storage, traffic, memory, cpu cores, network inbound speed, network outbound speed, storage profile and network profile to the API create server endpoint. - Backend changes for OVMF firmware paths. #### 1.7.2 `Testing Build 2` `July 26, 2022` - Added support for MacVTap network settings per operating system template. #### 1.7.2 `Testing Build 1` `July 25, 2022` - Fixed an issue where operating system templates were not showing the full name in the admin UI. - Used CPU cores now show correctly when viewing grouped hypervisor resources. - Bulk admin notification emails are now sent individually. - Fixed issue where a gateway4 element would remain in the cloud-init network configuration even when no IPv4 addresses were specified. - Added support for new hypervisor licensing options. - Improved interaction with the QEMU guest agent. - Added support for MacVTap hypervisor networking (default on new installations). - It's now possible to edit the primary network interface name for hypervisors. - When commissioning a new hypervisor, the system will attempt to auto-populate resource limits (memory, cpu). - Added hypervisor server events (hooks). https://docs.virtfusion.com/administration/event-hooks/overview #### 1.7.1 `July 13, 2022` This version ships with bugfixes, features and improvements. - Fixed issue with VM statistic charts not displaying if the supplied datasets have missing data. - Improved end user area breakpoints, layout and CSS. - Added several database column constraints to improve data integrity. - Optimized IPv4 address selection in the server create functionality. - Added search to VM transfer ownership user list. - Added relational id (without server id) authentication endpoint to the API. - Added the ability for end users to bookmark servers. - Javascript bundles have been updated to their latest versions. #### 1.7.0 `July 5, 2022` This version ships with bugfixes, features and improvements. - Added in panel software updater with optional automatic updates and new version alerts. - System backup now uses a single transaction when performing a database backup to prevent table locking. - A failed VM sub action can now be deleted from the UI. - Fixed issue where additional VM disk formatting would randomly fail on large disks. - Added secondary system schedulers (reserved for future use). - Queue monitoring has been improved with support for dynamic timeouts. - Rebuilt the VM migrator to support additional disks, data compression modes and multiple passes to decrease the overall downtime of a VM. - All application dependencies have been updated to their latest versions (framework). - Optimized the hypervisor codebase and removed deprecations. - Queue workers now auto spawn for heavy workloads. - VM backup processes are now exit code and error aware giving better stability to the backup system. - Data passed over the UI websocket has been optimized for faster delivery. - Windows datacenter templates have been updated and are available from the template manager. ### 1.6.x #### 1.6.3 `June 23, 2022` This version ships with bugfixes and improvements. - Fixed an issue where all Ubuntu templates would reject the cloud-init configuration if a secondary network was enabled. - Windows secondary networks are now configured through cloud-init. This will only work with build/rebuild options. - Fixed several typos in the admin area. - Fixed an issue where hypervisors could not be deleted from the admin UI. #### 1.6.2 `June 20, 2022` This version ships with bugfixes and improvements. - Fixed issue where cloud-init would not set a valid ip configuration for a Windows VM after a migration. - Removed MongoDB dependencies on hypervisors. - Secondary Network traffic chart now shows if a VM has a secondary network. - VMs and hypervisors can now be configured if a VirtFusion license mismatch is detected. - Various changes to the end user interface CSS to improve visibility in dark mode. - Software updater rebuilt to allow realtime output and per version migrations. - Changes to VM network filtering to allow for live firewall rules. (Ready for firewall feature in v1.7). - Added scheduler to monitor build logs. This version contains major changes to the statistics collectors. Updating to this version may take longer than previous versions due to changes being made in the background. Historical statistics will be purged (CPU, memory etc..) except for bandwidth tracking data. This version also adds support for AlmaLinux 9 and Ubuntu 22.04 hypervisors. #### 1.6.1 `June 7, 2022` This version ships with bugfixes and improvements. - Fixed an issue in server scheduled backups where invalid timestamps would cause a backup to randomly fail. - An IPv6 address can now be used as the hypervisor IP. - Added CLI command to reset the system credentials used for data transfers. https://docs.virtfusion.com/administration/cli/#reset-system-access-credentials - Admin server migration queue now updates correctly on data changes. - Exposed the `/users/:extRelationId/byExtRelation/resetPassword` API route to reset a user password (required by the latest WHMCS module). - Various enhancements to the admin UI, including bulk actions for most tables. - All application dependencies have been updated to their latest versions (framework). This version is also compatible with the latest WHMCS module (v2.0) that supports configurable options for a Location selector and IPv4 selector. https://docs.virtfusion.com/integrations/whmcs/ #### 1.6.0 `May 29, 2022` This version ships with bugfixes, features and improvements. - Improved IPv4 list filtering when manually adding an address to a VM. - Fixed sorting order of users table. - Additional storage (hard disks) can now be added to a VM with the following temporary limitations: - Adding or removing a disk requires a VM restart. - Disks will not be included in user backups. - The VM can't be migrated. - Secondary network interfaces can now be added to a VM with the following temporary limitations: - Secondary interfaces are not considered when migrating a VM. Manual intervention may be required if the destination does not have a matching network configuration. - When adding a server package, the OS template collection is automatically set to Allow All if no other collections are selected. - Improved the processing of VM resource changes (now instant) if the server is awaiting build. - Sub actions can now run on a boot process. Previously only power down actions were available. - Server passwords (root/Administrator) can now be reset from the user interface. - Added CLI command to force remove a hypervisor from the system. https://docs.virtfusion.com/administration/cli/#force-remove-a-hypervisor-and-its-servers - Added CLI command to recommission a hypervisor. https://docs.virtfusion.com/administration/cli/#set-a-hypervisor-to-re-commission-state - Added CLI command to reset an administrator password. https://docs.virtfusion.com/administration/cli/#reset-an-administrator-password ### 1.5.x #### 1.5.1 `May 21, 2022` This version ships with bugfixes, features and improvements. - Fixed terms of service link in initial setup process. - The two-factor authentication input has been converted to a single input field. - Added support for DHCP only networks. - A specific hypervisor can now be selected when building a VM from the admin UI. - If a VM only has an IPv6 address, this will now be shown as the primary IP in the user area. - Added server name search option in the global search. - Updated admin and end user javascript bundles. #### 1.5.0 `May 18, 2022` This version ships with bugfixes, features and improvements. - Fixed issue where a server migration would occasionally throw an exception before completion. - Admin area news and version information now uses an improved caching system. - VNC passwords are now compatible with Libvirt 8+. - Added low level communication logging between control and hypervisors. - IPv6 auto allocation now honours network profiles. - Fixed issue where an operating system template would show in the Other group if the assigned group was disabled. - Operating system template table now links to assigned group and displays the group status. - Added Operating System Template Collections for server packages/vms (Admin -> Media -> Template Collections). - Added kvm64 & qemu64 as extra Default CPU Mode settings. - Fixed an issue where a domain would not be accepted for PowerDNS resource settings. - The default miscellaneous template group name and status can now be modified (Admin -> Media -> Template Groups). - Various admin UI improvements to dropdowns/sliders. - IPv6 block configuration now supports all CIDRs. - Fixed an issue with IP block & hypervisor network database mapping. - Added IPv6 allocation logging to the resource allocation log. Operating System Template Collections allow you to create multiple collections of templates and groups to be assigned to a Package or/and VM. This feature is ideal if you would like to exclude specific templates for specific packages. There is no limit to the amount of collections that can be assigned. ### 1.4.x #### 1.4.2 `May 11, 2022` This version ships with bugfixes, features and improvements. - Fixed issue where server disk cache settings (in some circumstances) were not being set correctly. - Added Network Profiles allowing specific IP blocks/IPs to be used when a server is built. Profiles can bet set within packages and IP blocks. - Operating system Cloud-init config Yaml now represents multi-line content as literal blocks. - Added filters to IPv4, server and user tables. - IPv4 addresses are now listed under their corresponding IP block. - User table now displays the number of accessible servers and last login date. - IP block table now displays the number of hypervisors and IPv4 assigned. - Added build/migrate resource allocation log (Admin -> Logging -> Resource Allocation). - Fixed issue when creating a server from the admin interface, an exception was being thrown if a package was not selected. - Fixed issue where a server network interface would not adopt the correct default settings. - Operating system template updates can now be delivered without relying on a software update. - Fixed issue when adding a license key branding removal settings were not honoured until it was next updated. - Various enhancements to the admin UI, including an improved sidebar that now shows system alerts (pending/failed tasks, migrations and os template updates) and updated dark modes. - VNC and UI websockets now run under the standard 443 port. This only applies to new installations. #### 1.4.1 `May 3, 2022` This version ships with bugfixes and improvements. - The server build/rebuild email now contains connection information when servers have an IPv6 only address. - Increased the verbosity of the task logs. Allowing for better debugging. - Improved the user impersonation functionality. Multiple browser tabs can now be used without causing session issues. - When migrating a server, the destination hypervisor resource checks would report a not enough resources error for CPU cores when limits were set to 0 (unlimited). All resource checks now honour unlimited resources. - The timeout for server builds has been increased for large operating system templates. - Power controls could not be used (in admin area) when a server was migrated and was in testing mode. All power controls can now be used once a server has migrated and is pending completion. - Windows deployments now use OpenStack network metadata allowing for flexible network configurations. #### 1.4.0 `April 29, 2022` This version ships with bugfixes, features and improvements. - Fixed an issue with the default favicon showing when branding removal is active. - Added Ubuntu 22.04 LTS (Jammy Jellyfish) template. (installable via the template manager). - Added a VirtFusion beta version updater (should not be used unless instructed). - QEMU guest agent statistics can be viewed from the admin area on a per VM basis (only if the VM has a running agent). - Admin Template and Template Group pages have been rewritten in Vue. - Added option to a templates settings to automatically enable VNC when a VM is built from template. - Rebuilt cloud-init module to support NoCloud & Cloud Drive v2 (Openstack/EC2) configurations. - Added support for Windows operating system templates. In this version, VirtFusion allows you to deploy Windows VMs from prebuilt templates. At this point in time we have supplied the following templates for testing: - Windows Server 2019 Standard - Windows Server 2019 Datacenter - Windows Server 2022 Standard - Windows Server 2022 Datacenter Templates are available for install via the Template Manager. Please note that the first install of a VM using any of the templates will take longer than normal. This is due to the template download procedure on the hypervisor. Current deployment features include: - Automatic static IP assignment of the primary IPv4 address. - Automatic filesystem resizing. - Cloud-init scripting. (runcmd etc..) - Force setting of the Administrator password. Please report any issues you may encounter with the Windows templates via a support ticket or Discord. ### 1.3.x #### 1.3.0 `April 17, 2022` This version ships with bugfixes, features and improvements. - Added the ability to specify multiple hypervisor local storage locations with profiling. - Added the option to show the CPU model to the end user (CPU model field under hypervisor settings). - Added an internal system cleanup scheduler. - All application dependencies have been updated to their latest versions (framework). - Fixed an issue with the end user validation CSS. - Added an optional hostname field in the server build process. ### 1.2.x #### 1.2.6 `April 5, 2022` This version ships with bugfixes, features and improvements. - Fixed an issue where saving invalid cloud-init data for a template would throw an exception. - Hypervisor group information can now be shown to the end user (group name, an optional label and icon (currently only flags)). - The HTML 5 VNC console now has paste functionality. - VNC connection details can now be hidden from the end user (Settings --> Virtualization --> Default VNC Details). - Fixed an issue where a hypervisor could not be removed from an IP block. - IPv4 addresses can now be added to an IP block in subnet form. Currently supports /16 -> /30. - Added CentOS Stream 9 template. #### 1.2.5 `March 31, 2022` This version ships with bugfixes, features and improvements. - Improvements to the hypervisor dhcp service adding stability for virtual servers that rely on dynamically assigned IPv4 addresses. - It's now possible to change the default path and port used by the UI/VNC websocket services (Settings --> Other). - Fixed an issue where a virtual server build would fail if no public resolvers were selected. - Fixed an issue where a virtual server would fail to delete if an RDNS service was unavailable. - Increased the max limit for the Memory value on a server package. - Added a slide-out window to the admin UI that shows the latest system scheduled tasks, authentication attempts and emails. (Message icon at the top of the admin sidebar). - Improved the virtual server IP Anti-Hijacking functionality. Each virtual server now has its own filter allowing firewall rules to be added on the fly (upcoming feature). This feature also allows whitelisting of IP addresses that are not assigned to the virtual server (VM --> Primary Network --> Configure --> Whitelist). All updates are currently performed from CLI on the UI server. This update has an optional one time command that is advised to be run after the standard update has finished. Due to the changes made in the new filtering system, existing virtual servers will continue to use the legacy filters until they have rebooted, powered down etc... This optional command will configure all the new filters before any actions are run for a virtual server. This will guarantee the new filters will be used when a server is power cycled. After running ` vfcli-ctrl update ` run `vfcli-ctrl server:network-filters rebuild` #### 1.2.4 `March 22, 2022` This version ships with bugfixes, features and improvements. - Added an operating system template manager that allows the installation of new templates and updates of current templates easily without overwriting your cloud-init and other settings. - Added a new scheduler that will clean up unused operating system templates and ISO in the hypervisor cache. - Fixed an issue that would sometimes cause the remaining hypervisors resources to display minus values when editing a virtual servers resources. - A default for virtual server Port Isolation can now be set under the global virtualization settings. This option is only applied when a virtual server is built. - Added on_crash element to virtual server configuration. If a virtual server crashes, it will now reboot. - Fixed issue that would display a blank alert on the virtual server creation screen if no suitable hypervisor groups were found. - Hypervisor resource allocation can now be viewed on a per hypervisor group basis. (Compute Resources --> Hypervisor Resources). - Added an option to force data transfer of virtual server backups through localhost. Ideal for hypervisors that have a local mountpoint. #### 1.2.3 `March 18, 2022` This version ships with bugfixes and improvements. - Added the option to specify the MTU that should be applied to a virtual server network interface on a per hypervisor bridge basis. - Style and layout enhancements to the user and admin UI. #### 1.2.2 `March 6, 2022` This version ships with bugfixes, features and improvements. - Added the ability to insert custom scripts into the end user area header and body (Admin -> Configuration -> Settings -> Look & Feel). - MAC addresses are now set correctly in the hypervisor DHCP configuration when using custom MAC per IP. - Fixed issue with footer links in the end user area not showing a valid TOS and website URL if they were specified. - End users can now schedule backups on a daily or weekly basis if enabled under a backup plan. #### 1.2.1 `March 2, 2022` This version ships with bugfixes and improvements. - Fixed an issue where a pre-defined MAC address was not being assigned correctly in the cloud-init configuration. - Basic hypervisor system information can now be selected and displayed on the main admin dashboard (previously only the control server). - Many styling tweaks and fixes to both the end user and admin UI. - Improved toast alerts for the end user and admin UI. #### 1.2.0 `February 24, 2022` This version ships with bugfixes, features and improvements. - PowerDNS (reverse DNS) now supports self-signed SSL certificates when using HTTPS. - IPv4 addresses can now (optionally) have MAC addresses assigned which will override the assigned interface MAC. - Fixed a bug that caused connection errors when specifying the hostname for a hypervisor. - Additional networks can now be defined for hypervisors and assigned to IP blocks. - Introduced native dark mode for admin/user UI. ### 1.1.x #### 1.1.5 `February 16, 2022` This version ships with features and improvements. - Added automated PowerDNS reverse DNS support. https://docs.virtfusion.com/administration/reverse-dns/ - Users can now select the public (IPv4) resolvers they would like to use when building and managing a VM. #### 1.1.4 `February 13, 2022` This version ships with bugfixes, features and improvements. - Fixed an issue where backup pre-checks may give false positives when testing connectivity to the remote backup server. - Added Reverse DNS support. Manual and a single automation option is included in this release. https://docs.virtfusion.com/administration/reverse-dns/ - Backups pending deletion now show how long left (retention time) before they will be deleted. #### 1.1.3 `February 8, 2022` This version ships with bugfixes, features and improvements. - Added actions on server traffic limit. The action can be configured under Configuration -> Settings -> Virtualization -> Server Traffic Management with the ability to configure the directions of traffic to be counted towards the limit (also configurable per hypervisor) and the actions that should be taken when the limit has been reached. Currently, none or throttle network. - Added server traffic blocks. This allows an admin to give a server extra traffic on a monthly term basis (also covers future months). Ideal for when a server reaches its traffic limit. - The end user timezone is now automatically detected. - Simplified the server task list shown to an end user. - Added the ability to set read I/O, write I/O, read bytes, write bytes for a server disk under Server Packages and a server itself. - Custom OS template icons (logos) can now be added. https://docs.virtfusion.com/configuration/custom-template-icons #### 1.1.2 `February 3, 2022` This version ships with bugfixes, features and improvements. - Added the ability to restrict the admin area to a specific set of IP addresses. https://docs.virtfusion.com/administration/admin-ip-restrict - The network driver and IP anti-hijacking can now be modified on a per server basis (Server -> Primary HDD -> Configure). - It's now possible to stop a server seeing or communicating with other servers on the same logical network using the port isolation feature (Server -> Primary Network -> Configure -> Options). - The primary hard disk bus and cache can now be set on a per server basis. - Admin area general tidy up. #### 1.1.1 `February 1, 2022` This version ships with bugfixes, features and improvements. - A warning message is now shown to a user when trying to build a server that has lower resources than any available templates. - The admin area is easier to navigate due to the introduction of breadcrumbs in the header. - A server package could not be created with an unlimited traffic allowance. This is now possible. - User area notices can now be created and configured from the admin area. Configuration -> User Notices. #### 1.1.0 `January 31, 2022` This version ships with improvements and a major feature. - Removed compression by default for server migrations. - Users can now configure a login alert under their profile. - Added user level backups [https://docs.virtfusion.com/administration/server-backups](https://docs.virtfusion.com/administration/server-backups/). ### 1.0.x #### 1.0.11 `January 24, 2022` This version ships with bugfixes, features and improvements. - When creating a server, the reason is now shown when not enough resources are available in the selected hypervisor group. - Some OS types installed on a server were showing incorrect disk statistics. - Fixed an issue where a users' relation id could not be removed. - Email templates are now editable (Configuration -> Email Templates). - Emails are now logged (Logs -> Email). - A global email signature can now be set (Configuration -> Settings -> Email). - Fixed an issue where server migrations failed to select the correct hypervisor primary network interface adaptor. #### 1.0.10 `January 17, 2022` This version ships with bugfixes, features and improvements. - Fixed an issue where server resource usage would not show if the statistics storage agent was offline. - Improved time series storage for server statistics. - Server statistics can now be displayed over different time periods. - Added support for routed networking. - New network configuration options added to template settings. - Custom favicons can now be added. [https://docs.virtfusion.com/configuration/favicon](https://docs.virtfusion.com/configuration/favicon) - Admin & end user colour selection is no longer limited to specific colours. - Fixed an issue where server monthly statistics were prematurely showing the next month. #### 1.0.9 `January 5, 2022` This version ships with a few bugfixes and improvements. - Default disk bus settings are now available under Configuration > Settings > Virtualization. (virtio, virtio-scsi, sata). - Default CPU mode settings are now available under Configuration > Settings > Virtualization. (host-passthrough, host-model). - Added the ability to add specific IPv4/IPv6 to a virtual server. - System backup logs now clean themselves up automatically. - Added system backup retentions so any backups older than the specified period will be deleted automatically. #### 1.0.8 `January 4, 2022` This version ships with a few bugfixes and improvements. - Templates now have network configuration settings. Imported templates are pre-configured (if required). These settings are mainly are for internal use at this current time. - The admin sidebar now keeps its scroll position between page transitions. - When a virtual server is configured via the build wizard, the form scrolls back to any errors on the page (usually the server name field). - Logos for template groups and templates can now be set when adding/editing a template or group. - Fixed a bug in system backup when testing SCP settings (caused a 500 error under private key option). - CD/DVD (ISOs) can now be assigned globally (as standard) or assigned to specific users on the system. #### 1.0.7 `December 30, 2021` This version ships with a few bugfixes and improvements. - Added various improvements to the cloud-init provisioning system - Fixed a bug while trying to enable a hypervisor with disk cache type inherit - QEMU guest agent support to improve memory statistics for admin and end users #### 1.0.6 `December 22, 2021` This version brings the option to specify the disk cache type (none/writethrough/writeback/directsync/unsafe). A global setting under Settings > Virtualization and override settings for hypervisors. #### 1.0.5 `December 20, 2021` This version ships with few new features and improvements. - Added the ability to disable sharing of Servers & SSH keys (Settings > General) - Added the ability to perform simple searches on Server mac, id, uuid, ipv4, ipv6 and User name, email #### 1.0.4 `December 18, 2021` This version brings the option to specify the disk image type (Qcow2/Raw) used when a server is built. A global setting under Settings > Virtualization and override settings for hypervisors and templates. #### 1.0.3 `December 17, 2021` This version improves the way the cloud-init configurations are managed and triggered for server resource changes. #### 1.0.2 `December 16, 2021` This version contains a fix for server traffic accounting. #### 1.0.1 `December 15, 2021` This version ships with a multitude of fixes and a few new features. - Offline migrations are now supported - The ability to test email settings - When an IPv6 block is assigned to a server, a single address will also be generated #### 1.0.0 `December 1, 2021` The first public release of VirtFusion. --- ## Version 2.x ### 2.3.x #### 2.3.1 `August 3, 2023` - Added Korean language (client area). https://docs.virtfusion.com/guides/localization - Improved error logging for disaster recovery pruning. - Fixed issue where high value prices were not displaying correctly for self service packages. - Improved admin API rate limiting. - Server statistics timespan selector now shows on small screen devices. - Fixed issue where the server suspended rate limits were not showing correctly for users. - Fixed issue where text email variables were not being translated. - Added the ability for end users to disable IPv6 addresses. - Various layout fixes and improvements to the end user area. #### 2.3.0 `July 12, 2023` - Added hypervisor Disaster Recovery with support for S3 compatible object storage & local storage. See https://docs.virtfusion.com/guides/disaster-recovery for full setup instructions. - Added Gravatar support for user avatars. - Added localization for the end user portal. See https://docs.virtfusion.com/guides/localization for the current status and setup instructions. - Major hypervisor framework update. - Removed support for PHP 8.0 hypervisors (very few installations will still be using this version). See https://docs.virtfusion.com/troubleshooting/php#hypervisor-specific for instructions on how to keep your system up to date. - Cloud-init will no longer run on a VM migration if no IP changes were made. - IP block list now shows the number of block groups assigned. - Third-party control server frontend assets have been updated to the latest versions. - Improved task scheduler delegation on large clusters. - Added slow shutdown routines for fast activating task workers. - Added basic task error info for end users when a task fails due to a temporary error. - Improved VM destroy functionality. - Improved network filter application functionality. - Improved VM build task functionality. - VM Migrations now do a version check before starting. - If 2FA is enabled for an account, the VNC window will now reject connections with a 401 HTTP code. - IPv4 addresses can now be reserved if already added to a VM. - Hypervisor event hooks (server-before-boot, server-after-boot) now run when VMs start after a hypervisor reboot. - System backup and backup storage can now use a domain, IPv4 or IPv6. - Added VM traffic collection support for network namespaces. - Added the ability to use Redis as a task queue driver. See https://docs.virtfusion.com/guides/task-queue for setup instructions. - Added support for Libvirt domain power management statuses. (If a VM operating system sleeps, it will show in the UI as `Sleeping`). - It is now possible to view the QEMU, Libvirt, CPU and PHP versions from the hypervisor settings page in the admin UI. - Improved task queue error reporting. Most boot errors will now show at the top of the task items list (big red box). - Improved task status checks based on VM states. - Added hypervisor app:restart cli command. See https://docs.virtfusion.com/troubleshooting/restarting-services#hypervisor - Added the ability to specify up to four BCC email addresses for system emails. - Improved email error logging. #### 2.3.0 `Testing Build 3` `July 4, 2023` - Added the ability to use Redis as the task queue driver. See https://docs.virtfusion.com/guides/task-queue for setup instructions. - Added support for Libvirt domain power management statuses. (If a VM operating system sleeps, it will show in the UI as `Sleeping`). - It's now possible to view the QEMU, Libvirt, CPU and PHP versions from the hypervisor settings page in the admin UI. - Improved task queue error reporting. - Improved task status checks based on VM states. #### 2.3.0 `Testing Build 2` `June 26, 2023` - Added prune support and compression levels to disaster recovery. See https://docs.virtfusion.com/guides/disaster-recovery for setup instructions. `Important` The backup layout has changed since build 1. We suggest you empty your bucket/local storage and run a new backup. - Added basic task error info for end users when a task fails due to a temporary error. - Improved VM destroy functionality. - Improved network filter application functionality. - Improved VM build task functionality. - VM Migrations now do a quick version check before starting. - If 2FA is enabled for an account, the VNC window will now reject connections with a 401 HTTP code. - Added traditional Chinese language files. - IPv4 addresses can now be reserved if already added to a VM. - Hypervisor event hooks (server-before-boot, server-after-boot) now run when VMs start after a hypervisor reboot. - System backup and backup storage can now use a domain, IPv4 or IPv6. - Added VM traffic collection support for network namespaces. #### 2.3.0 `Testing Build 1` `June 13, 2023` - Added hypervisor Disaster Recovery, supporting S3 compatible object storage & local storage. See https://docs.virtfusion.com/guides/disaster-recovery for setup instructions. - Added Gravatar support. - Added localization for the end user area. See https://docs.virtfusion.com/guides/localization for the current status and setup instructions. - Major hypervisor framework update. - Removed support for PHP 8.0 hypervisors. - Cloud-init will no longer run on a VM migration if no IP changes were made. - IP block list now shows the number of groups assigned. - Third-party control server frontend assets have been updated to the latest versions. - Improved task scheduler delegation on large clusters. - Added slow shutdown routines for fast activating task workers. ### 2.2.x #### 2.2.1 `May 20, 2023` - Added option to show the name/label of a hypervisor a VM resides on to the end user. (Hypervisor -> Display Name) - Improved VM power-off functionality. It will now ignore the domain status if the VM has crashed. - Increased the amount of queues that can be set on a VM network interface to 40. - Fixed issue where the available IPv4 amount would display incorrectly when viewing Hypervisor Resources by group. - Added a CLI Server Do Not Destroy (DND) mode that prevents a VM from being deleted, re-installed, ISO mounts and password resets. (https://docs.virtfusion.com/reference/cli/#server-do-not-destroy-dnd) - Fixed issue where a backup plan would not be set when a new package is applied to a VM. - Added the ability to duplicate VM packages. - IPv6 reverse DNS zones now support nibble formats starting from a /32. - Added new Options button on a hypervisor settings page with the ability to restart libvirtd and flush ebtables. - Added hypervisor internal callbacks. #### 2.2.0 `May 2, 2023` - Improved VM network filter detection and management. - Improved post cloud-init module execution on network change and migrations. - Added option to send an account creation email when a user is created. - Added id, email and relational id filters to users table. - Increased timeouts for VM actions that could potentially run slow on larger deployments. - Fixed issue where available IPv4 would show incorrectly when viewing hypervisor resources by group if IP block groups were in use. - Major improvements to the email system - including: - Sectional HTML layouts. - Pre-built layout components. - Text versions of each email. - Preview text (pre-header). - Multi-level parsing to mitigate potential layout issues due to user input. - HTML+Text emails (now default), Text only or HTML only. - Logo header to replace the default company name. - Footer disclaimer. - Fixed issue where the build email would not include a root password if no IP was set. - License type can now be set from the hypervisor list. - IPv4 addresses can now be ordered by priority. - Kea DHCPv4 configurations no longer uses a global subnet. All subnets are now configured independently. - Fixed issue with failed backup emails not sending. - Improved the power control functionality to enable actions if a hypervisor fails to respond with a status. - Added cloud-init user_data support to server provisioners. - IP changes are now automated on Windows VMs through cloud-init. - Added quick links to user servers from the users table and management page. - /126 - /128 ranges can now be generated from a source IPv6 subnet. - Fixed issue where a secondary network filter may not delete if a VM was running while the interface was being deleted. - Added the ability to live update network devices. - NAT domain TLD verification files are now automatically downloaded if they don't exist. - Improved additional disk creation and formatting functionality. - Added webhook support. https://docs.virtfusion.com/guides/webhooks - Added support for extra CPU flags (including topoext) at hypervisor, operating system template and VM levels. - Improved the display of network speeds to end users. - Added forwardfor to the HAProxy configuration. - Dnsmasq and ISC are now supported for DHCP. https://docs.virtfusion.com/guides/dhcp - OVMF is now bundled to better handle UEFI and custom NVRAM configurations. - Cloud-init `on.sshkey` and `on.password` run levels have been optimized to speed up authentication activation on operating system templates with long deployment times. - Added Hook points to VM XML configuration files. - Added the ability to pre-generate IPv6 subnets. - Added the ability to manage & provision IPv6 routing subnets alongside a IPv6 subnet. https://docs.virtfusion.com/guides/ipv6-route-blocks - Added subnet filter to IPv6 subnets table. - Added send-proxy & send-proxy-v2 support to NAT domains. - Added VNC API endpoint. https://docs.virtfusion.com/api/#api-Servers-VNC - Fixed issue where the Manage as Owner button would not work on some mobile browsers. - Added logs to server migration queue. - Added the ability to migrate a VM directly from the VM management page. - Added `netplan_accept_ra: true|false` flags to cloud-init config. - Added the option to change VM network interface profiles. - Multiple CSS and Javascript improvements to user/admin interfaces. - Improved various database indexes. - Frontend and backend framework updates. - Added support for Debian 12 (Bookworm). #### 2.2.0 `Testing Build 4` `April 25, 2023` - NAT domain TLD verification files are now automatically downloaded if they don't exist. - Improved additional disk creation and formatting functionality. - Added webhook support. https://docs.virtfusion.com/guides/webhooks - Added support for extra CPU flags (including topoext) at hypervisor, operating system template and VM levels. - Improved the display of network speeds to end users. - Added forwardfor to the HAProxy configuration. - Dnsmasq and ISC are now supported for DHCP. https://docs.virtfusion.com/guides/dhcp - OVMF is now bundled to better handle UEFI and custom NVRAM configurations. - Cloud-init `on.sshkey` and `on.password` run levels have been optimized to speed up authentication activation on operating system templates with long deployment times. - Several CSS and Javascript improvements to user/admin interfaces. - Added Hook points to VM XML configuration files. #### 2.2.0 `Testing Build 3` `April 4, 2023` - Added cloud-init user data support to server provisioners. - IP changes are now automated on Windows VMs through cloud-init. - Added quick links to user servers from the users table and management page. - Improved hypervisor level DHCP support. - /126 - /128 ranges can now be generated from a source IPv6 subnet. - Fixed issue where a secondary network filter may not delete if a VM was running while the interface was being deleted. - Improved various database indexes. - Added the ability to live update network devices. #### 2.2.0 `Testing Build 2` `March 28, 2023` - Fixed issue where the build email would not include a root password if no IP was set. - License type can now be set from the hypervisor list. - IPv4 addresses can now be ordered by priority. - DHCP configuration no longer uses a global subnet. All subnets are now configured independently. - Fixed issue with failed backup emails not sending. - Improved the power control functionality to enable actions if a hypervisor fails to respond with a status. #### 2.2.0 `Testing Build 1` `March 8, 2023` - Improved VM network filter detection and management. - Improved post cloud-init module execution on network change and migrations. - Added option to send an account creation email when a user is created. - Added id, email and relational id filters to users table. - Increased timeouts for VM actions that could potentially run slow on larger deployments. - Fixed issue where available IPv4 would show incorrectly when viewing hypervisor resources by group if IP block groups were in use. - Major improvements to the email system - including: - Sectional HTML layouts. - Pre-built layout components. - Text versions of each email. - Preview text (pre-header). - Multi-level parsing to mitigate potential layout issues due to user input. - HTML+Text emails (now default), Text only or HTML only. - Logo header to replace the default company name. - Footer disclaimer. - Frontend and backend framework updates. ### 2.1.x #### 2.1.0 `February 17, 2023` - Added support for hourly Self Service credit balances. Documentation can be found [here](https://docs.virtfusion.com/guides/self-service). - Fixed display bugs in admin logs. - Fixed an issue with UEFI ROM auto detection. - Added the option to enable network boot on a per server basis (Server -> Settings -> Network Boot). - Increased server power function timeouts. - Moved slow loading assets to the application vendor bundles. ### 2.0.x #### 2.0.0 `February 8, 2023` - Added support for end user Self Service. Documentation can be found [here](https://docs.virtfusion.com/guides/self-service) and this feature comes in two variations, both of which can be used in conjunction with each other. - **Hourly Accounting**. VirtFusion tracks the server usage from start to finish and logs the duration and addons. Consult the documentation for configuration and limitations. - **Resource Packs**. Collections of resources (memory, storage, cpu etc...) allocated to a user. Consult the documentation for configuration and limitations. - Added support for NAT based networking ([docs](https://docs.virtfusion.com/guides/nat-setup)) with the following features. - User custom port configuration (20 + 3 service ports). - User domain forwarding configuration over ports 80 & 443. - Native IPv6. - Added the ability to customise the login screen with logo, background and text (Admin -> Settings -> Look & Feel). - Added firewall rulesets that can be assigned to server packages and applied on deployment or directly to a server. - Added support for Shell, Powershell, Batch and Python Provisioner scripts (Admin -> Server -> Provisioners). Provisioners are assigned to operating system templates at multiple points in the cloud-init configuration. - Added force IPv6 option to packages. This option will force IPv6 on the server instead of giving the user an option to choose if they deploy it or not. - Added the option to prevent users from changing their name and email address (Settings -> General -> Misc -> Allow Users to Change Account Details). - Added support for auto-detection of a suitable UEFI ROM (through Libvirt capabilities). This should solve the issue of failed server boots if the selected ROM is missing. - Added the option to disable cloud-init on a per server basis (Server -> Settings -> Cloud-Init). - Added server audit logs to track the user, ip address, endpoint and date/time of each action performed. Logs are not currently available to end users (Admin -> Servers -> Audit & Admin -> Server -> Audit). - Added the option to disable the forgot password option on the login page. This option is enabled by default (Admin -> Settings -> General -> Misc -> Allow Forgot Password). - Added a CLI command to restart application services on the control server. (`vfcli-ctrl app:restart`) - Added internal server action throttling. This allows fine grained abuse prevention of reboots, password resets etc... User definable options will be available in a future release. - Added search by user ID/email and self service resource pack to servers table. - Improved server migration integrity with write lock support for the metrics datastore. - Improved the end user IPv6 address allocation routine allowing faster generation of addresses from large subnets. - Improved VNC port allocation when server migrate completions are in a pending state. - Updates will now attempt to flush the previous version of the application from memory. - Fixed various issues with server package upgrades. (CPU topology now applies correctly). - Fixed many minor (insignificant) backend bugs and annoyances (control & hypervisor). - Multiple improvements to the admin user interface including fixes for previous display bugs. - Multiple improvements to the end user interface including fixes for previous display bugs. - All application dependencies have been updated to their latest versions (framework). #### 2.0.0 `Testing Build 5` `February 1, 2023` - Fixed issue where CPU topology was not being altered on VM package changes. - Added Self Service (hourly) per user reports. (Admin -> Self Service -> Reports) - Self Service token values are now inherited from packages by default (with per VM overrides). - Added Self Service (hourly) addons. - Improved VNC port allocation when migration completions are pending. - Added search by user ID/email and resource pack to servers tables. - Added Self Service group profiles. **This is a breaking change for hourly services**. Please see https://docs.virtfusion.com/2.0/guides/self-service for the latest configuration information. #### 2.0.0 `Testing Build 4` `January 20, 2023` - Added option to prevent users from changing their name and email address (Settings -> General -> Misc -> Allow Users to Change Account Details). - Added support for auto-detection of UEFI ROM (Libvirt capabilities). - Added option to disable cloud-init on a per VM basis (Server -> Settings -> Cloud-Init). - Added VM audit logs. (Admin -> Servers -> Audit & Server -> Audit). - Added server action throttling. - Added Self Service widget to End User dashboard. - Fixed various UI elements (Admin & End User). - Updated vendor bundles to latest versions. #### 2.0.0 `Testing Build 3` `January 10, 2023` - Added support for Provisioner scripts (Shell, Powershell, Batch, Python). - Added write lock support to metrics datastore on VM migrations. - Improved Self Service package display settings with overrides. - Added Self Service package option to specify the number of IPv4 to be issued. - Fixed display issue with User Authentication Logs. - Added force IPv6 option to packages. - Added Name and Label overrides to Self Service hypervisor groups. #### 2.0.0 `Testing Build 2` `December 20, 2022` - Fixed issue with saving server package settings. - Added token & hour totals to self service usage API endpoint. - Added option for end users to view a basic monthly usage breakdown of self service VMs. #### 2.0.0 `Testing Build 1` `December 17, 2022` - Added support for end user Self Service. https://docs.virtfusion.com/2.0/guides/self-service - Added support for NAT networking (including custom port & domain forwarding). https://docs.virtfusion.com/2.0/guides/nat-setup - Added ability to customise the login screen with logo/background/text. - Added firewall rulesets (can be assigned to VM packages or directly to a VM). - Fixed multiple display bugs in the end user and admin UI (missing labels, un-flushed reactive data, object conversion...). - Fixed issue where end users couldn't assign multiple IPv6 addresses from certain subnet sizes. --- ## Version 3.x ### 3.1.x #### 3.1.0 `December 5, 2023` - Added support for Ceph RBD centralized storage. https://docs.virtfusion.com/next/installation/storage/ceph-rbd - Added support for Remote Filesystem storage. https://docs.virtfusion.com/next/installation/storage/remote-filesystem - Added the ability to allow users to specify a custom ISO https://docs.virtfusion.com/next/guides/custom-iso - Added the option to specify a list of servers for backup via Disaster Recovery https://docs.virtfusion.com/next/guides/disaster-recovery#backup-only-specific-servers - Added `controlDomain` and `controlName` to the webhook request body (https://docs.virtfusion.com/next/guides/webhooks#body) for use with https://docs.virtfusion.com/next/integrations/webhook-proxy - Added CLI tool to force flush backups from storage. https://docs.virtfusion.com/next/reference/cli#force-flush-backups-based-on-storage-id - Added simple application information and cache statistics (System -> Health -> Application). - Added an optional intermediate subnet CIDR to IPv6 route blocks. - Added the option to set the gateway and first IPv6 on intermediate /126 route subnets. - Added DNS log (Logging -> DNS). - Added the ability to flush a server backup entry. - Added default options for Memory Ballooning Free Page Reporting and Auto Deflate (Settings -> Virtualization). - The number of assigned, reserved, free and total IPv4 are now shown on the IP block list. - Live migrations now run a full check for disk device and path/name mismatches. - All migration tasks are now shown to end users as `System Maintenance` and will never show a failed state. - Fixed an issue with the self service usage display when a package has been deleted. - Fixed issue when adding additional disks the buttons would show as disabled. - Rebuilding a DNS zone will now aggregate data from all assigned IP blocks. - IPv6 route block details are now shown to the end user for /126 intermediate allocations. - Several new columns added (RDNS, IPv6 route block) to IP tables (Server -> Network). - Health monitors now use a grace period before alerting selected administrators of potential issues. - Fixed an issue with Native Filters when using custom MAC addressing. #### 3.1.0 `Testing Build 3` `November 25, 2023` - Added the ability to allow users to specify a custom ISO https://docs.virtfusion.com/next/guides/custom-iso - Added the option to specify a list of servers to be backed up via Disaster Recovery https://docs.virtfusion.com/next/guides/disaster-recovery#backup-only-specific-servers - Fixed an issue with Native Filters when using custom MAC addressing. #### 3.1.0 `Testing Build 2` `November 20, 2023` - Added CLI tool to force flush backups from storage. https://docs.virtfusion.com/next/reference/cli#force-flush-backups-based-on-storage-id - All migration tasks are now shown to end users as `System Maintenance` and will never show a failed state. - Fixed an issue with the self service usage display when a package has been deleted. - Fixed issue when adding additional disks the buttons would show as disabled. - Rebuilding a DNS zone will now aggregate data from all assigned IP blocks. - IPv6 route block details are now shown to the end user for /126 intermediate allocations. - Several new columns added (RDNS, IPv6 route block) to IP tables (Server -> Network). #### 3.1.0 `Testing Build 1` `November 17, 2023` - Health monitors now use a grace period before alerting selected administrators of potential issues. - Added simple application information and cache statistics (System -> Health -> Application). - Added an optional intermediate subnet CIDR to IPv6 route blocks. - Added the option to set the gateway and first IPv6 on intermediate /126 route subnets. - Added default options for Memory Ballooning Free Page Reporting and Auto Deflate (Settings -> Virtualization). - Added support for Ceph RBD centralized storage. https://docs.virtfusion.com/next/installation/storage/ceph-rbd - Added support for Remote Filesystem storage. https://docs.virtfusion.com/next/installation/storage/remote-filesystem - The number of assigned, reserved, free and total IPv4 are now shown on the IP block list. - Live migrations now run a full check for disk device and path/name mismatches. - Added `controlDomain` and `controlName` to the webhook request body (https://docs.virtfusion.com/next/guides/webhooks#body) for use with https://docs.virtfusion.com/next/integrations/webhook-proxy ### 3.0.x #### 3.0.0 `November 1, 2023` - Added native network filtering. https://docs.virtfusion.com/guides/network-filtering - Added the ability to switch a VM disk to a different storage location. - Added hypervisor CLI power commands. `vfcli-hv server:start`, `vfcli-hv server:stop`, `vfcli-hv server:kill` - Added searchable filters to the IPv4 table (admin UI). - Added the ability to disable a secondary network. - Added new auto start routines for VMs after a hypervisor restart. - Added the option to switch between IPv4 and IPv6 for VNC connections (Hypervisor settings -> VNC Listen Type). - Fixed several end user UI translations. - Fixed an issue with disaster recovery pruning that uses S3 CommonPrefixes. - Added functionality to prevent a hypervisor being removed until any migrations are complete. - Fixed an issue with server package duplication not copying the correct template collections. - Fixed an issue with template groups and templates not assigning template collections correctly. - Updated frontend javascript framework and third party vendors. - Updated hypervisor backend framework. - Windows Administrator passwords are now pre-generated when a VM is built/re-built. - Improved pre-migration IP and VNC port checks. - Added the ability to live migrate VMs. https://docs.virtfusion.com/guides/server-migration - Added support for Libvirt Open vSwitch bridged networks (simple tagged + untagged vlans). - Improved Windows network configuration when a VM is hybrid migrated. - Updated all CSS assets. - Removed the use of google fonts. - Added the option to disable Gravatar. - Javascript charting libraries are now distributed via the control server. - Added Slovak and Hungarian languages. - Fixed issue where VNC mappings would be incorrect if a hybrid migration was reverted. - Added support for Bunny.net reverse DNS (IPv4 only). - Raw disk image usage statistics now show correctly in the admin area. - Templates can now have a minimum memory requirement as low as 64MB. - Added the option to enable the trimming of VirtIO Block type disks (Settings -> Virtualization -> VirtIO Block + Discard). - Added schedule and basic hypervisor connection & app version monitoring (System -> Health). - Fixed issue with hypervisor unsuspend event hook. - Admin email alerts can now be manged with ACL. - IPv6 subnets can now be reserved while in use by a VM. #### 3.0.0 `Testing Build 5` `October 24, 2023` - Live migration now updates filters on the source VM before attempting to migrate. - Added several new command line `start` arguments/options to live migrate. https://docs.virtfusion.com/next/guides/server-migration#start-a-migration - Templates can now have a minimum memory requirement as low as 64MB. - Added live migration `abort`, `data-success-force` and `cancel-force` commands. https://docs.virtfusion.com/next/guides/server-migration#abort-a-migration #### 3.0.0 `Testing Build 4` `October 18, 2023` - Removed the use of google fonts. - Added the option to disable Gravatar. - Javascript charting libraries are now distributed from the control server. - Fixed several translation errors. - Added Slovak and Hungarian languages. - Fixed issue where VNC mappings would be incorrect if a hybrid migration was reverted. - Added support for Bunny.net reverse DNS (IPv4 only). - Raw disk image usage statistics now show correctly in the admin area. - Added live migration support for offline and pre-built VMs. - Live migration now gives a basic overview of the data transfer progress. #### 3.0.0 `Testing Build 3` `October 3, 2023` - Added the ability to live migrate VMs. https://docs.virtfusion.com/next/guides/server-migration - Added support for Libvirt Open vSwitch bridged networks (simple tagged + untagged vlans). - Improved Windows network configuration when a VM is hybrid migrated. - Updated all javascript and CSS assets. #### 3.0.0 `Testing Build 2` `September 3, 2023` - Added the ability to disable network filtering on a per hypervisor basis. - Added multiple network filter types. https://docs.virtfusion.com/next/guides/network-filtering - Windows Administrator passwords are now pre-generated when a VM is built/re-built. - Improved pre-migration IP and VNC port checks. #### 3.0.0 `Testing Build 1` `August 21, 2023` - Added native network filtering. https://docs.virtfusion.com/next/guides/network-filtering - Added the ability to switch a VM disk to a different storage location. - Added hypervisor CLI power commands. `vfcli-hv server:start`, `vfcli-hv server:stop`, `vfcli-hv server:kill` - Added searchable filters to the IPv4 table (admin UI). - Added the ability to disable a secondary network. - Added new auto start routines for VMs after a hypervisor restart. - Added the option to switch between IPv4 and IPv6 for VNC connections (Hypervisor settings > VNC Listen Type). - Fixed several end user UI translations. - Fixed an issue with disaster recovery pruning that uses S3 CommonPrefixes. - Added functionality to prevent a hypervisor being removed until any migrations are complete. - Fixed an issue with server package duplication not copying the correct template collections. - Fixed an issue with template groups and templates not assigning template collections correctly. - Updated frontend javascript framework and third party vendors. - Updated hypervisor backend framework. --- ## Version 4.x ### 4.3.x #### 4.3.0 `Testing Build 4` `August 11, 2024` - Added several API endpoints to api/v1. - https://docs.virtfusion.com/api/#api-IP_Blocks-List - https://docs.virtfusion.com/api/#api-IP_Blocks-Add_IPv4 - https://docs.virtfusion.com/api/#api-Servers-List - Added the ability to enforce two-factor authentication for admins, end users or both. - Fixed issue where the traffic override when creating a server would convert incorrectly. - Added CLI command to disable two-factor enforcement (`vfcli-ctrl tfa-force-off`). - Control and hypervisor framework updates. #### 4.3.0 `Testing Build 3` `July 19, 2024` - Improved server boot functionality. - Added the option for end users to enable or disable backup notifications. - An email is now sent when a user changes an account password. - Added the option to email traffic limit warnings (Settings -> Virtualization -> Server Traffic Management -> Traffic Limit Notification 1/2). - An email is now sent (to the old address) when a user changes an account email address. - Added the option to verify/unverify account email addresses. - Added release notes to the admin area updates page. - Added basic server and IPv4 information to hypervisor settings. - User area account settings have been moved to their own section/page. - The throttle CPU API method now accepts a server ID or UUID. https://docs.virtfusion.com/api/#api-Servers-Throttle_CPU #### 4.3.0 `Testing Build 2` `July 8, 2024` - Fixed issue with additional disk type inheritance. - Improved layout and small screen view of the client server management area. #### 4.3.0 `Testing Build 1` `July 1, 2024` - Added support for secondary storage in server packages with API overrides for capacity and profile in the API create method. https://docs.virtfusion.com/api/#api-Servers-Create - Added the following API methods: - https://docs.virtfusion.com/api/#api-Servers-Traffic_Block_Add - https://docs.virtfusion.com/api/#api-Servers-Traffic_Block_Remove - https://docs.virtfusion.com/api/#api-Servers-Traffic_Blocks_Retrieve - https://docs.virtfusion.com/api/#api-Servers-Traffic_Statistics - https://docs.virtfusion.com/api/#api-Servers-Throttle_CPU - https://docs.virtfusion.com/api/#api-Servers-Backup_Plan_Add_or_Remove - https://docs.virtfusion.com/api/#api-Servers-Modify_Name - https://docs.virtfusion.com/api/#api-Servers-Firewall_Disable - https://docs.virtfusion.com/api/#api-Servers-Firewall_Enable - https://docs.virtfusion.com/api/#api-Servers-Firewall_Retrieve - https://docs.virtfusion.com/api/#api-Servers-Firewall_Rules_Apply - Fixed issue where TPM certificates would not persist across restarts. - Fixed issue where CPU topology would auto enable on server creation via API. - Removed depreciated `priority` element from powerDNS API methods. - Fixed issue with session remember me token generation. - Added support for FreeBSD memory usage display. - Fixed several client side localization errors. ### 4.2.x #### 4.2.3 `June 13, 2024` - Added `compressThreads` option to Disaster Recovery for zstd compression. https://docs.virtfusion.com/guides/disaster-recovery#configuring-virtfusion - Fixed issue with Disaster Recovery `compressLevel`. - Server remote state can now be returned via the `/servers/:serverId` API endpoint. https://docs.virtfusion.com/api/#api-Servers-Retrieve - Improved Disaster Recovery performance and removed unnecessary mount checks. - Added `Awaiting Setup` filter to server list. - Users are now instantly logged out and sessions removed when disabled. - Fixed issue where `delay=0` in the `DELETE /servers/:serverId` API endpoint would be ignored and default to 5. - Fixed several admin UI layout issues. #### 4.2.2 `May 23, 2024` - Fixed several admin UI layout issues. - Added the option to live update server disk QoS (IOPS/BPS). - Improved destination checks for hybrid migrations. - Added the option to force change a server VNC port. - Added resource package overrides on server creation. #### 4.2.1 `May 10, 2024` - Fixed several admin UI layout issues. - Reduced latency in server state collection. - Added CPU usage to server administration. - Added filters on IP block table. - Added option to pause server creation on a hypervisor (via API or Self Service). #### 4.2.0 `May 3, 2024` - Improved admin dashboard layout, including widgets and color schemes. - Added basic hypervisor monitoring & dashboard widget. - Added ARM (AArch64) VM support (Debian hypervisors only). - Updated media template manager to support ARM and to display warnings for templates that are EOL. - Added support for `zstd` compression in Disaster Recovery. https://docs.virtfusion.com/guides/disaster-recovery#configuring-virtfusion - Added `firewallRulesets` to the server create API method. https://docs.virtfusion.com/api/#api-Servers-Create - Failed task backend log now purges old entries. - Fixed issue where monitoring would give a false positive alert on an app version mismatch. - VM system info will no longer abort the whole process on a collection timeout. - Fixed issue where the Elevate button would display in the admin area even if a VM had not been commissioned. - Control server backend and frontend framework updates. - Hypervisor framework update. - Added the option to change the internal port used for control -> hypervisor communication. - Statistical charts now show times based on users timezone. #### 4.2.0 `Testing Build 3` `April 23, 2024` This release focuses on the final changes to the admin area interface refresh and framework updates. #### 4.2.0 `Testing Build 2` `April 15, 2024` This release focuses mainly on refreshing the admin area's interface. We've dedicated considerable effort to ensuring that no detail goes unnoticed. However, if you come across anything that seems off, let us know. #### 4.2.0 `Testing Build 1` `April 8, 2024` - Added experimental ARM (AArch64) VM support (Debian hypervisors only). - Updated media template manager to support ARM and to display warnings for templates that are EOL. - Added support for `zstd` compression in Disaster Recovery. - Added `firewallRulesets` to the server create API method. https://docs.virtfusion.com/api/#api-Servers-Create - Failed task backend log now purges old entries. - Fixed an issue where monitoring would give a false positive alert on an app version mismatch. - VM system info will no longer abort the whole process on a collection timeout. - Fixed issue where the Elevate button would display in the admin area even if a VM had not been commissioned. - Control server backend and frontend framework updates. - Hypervisor framework update. ### 4.1.x #### 4.1.0 `March 12, 2024` - Added server Elevate functionality. See https://docs.virtfusion.com/guides/elevate for more details. - Improved Windows network configuration using predictable interface naming. - Improved server build progress monitoring and removed double exceptions. - Fixed javascript issue when previewing email templates. - Added `expectedPassword` to admin API server reset password response body. https://docs.virtfusion.com/api/#api-Servers-Reset_Password - Removed the 20 IPv4 limit on server creation. - Improved template manager caching. - Extended backup job timeouts and improved error collection. - Updated hypervisor framework. ### 4.0.x #### 4.0.1 `February 24, 2024` - Fixed issue where maintenance mode would fail to disable via the admin interface. - Fixed issue where webhooks could not be enabled/disabled. - Added defaults for Network Packet Processing Queues (Settings -> Virtualization). #### 4.0.0 `February 22, 2024` **Major framework upgrade**. Although relatively small in feature set, this release contains a large number of core changes. Every care has been taken to make sure it is production ready and will work as a seamless update as long as the control and hypervisor operating system has been regularly updated. Please see https://docs.virtfusion.com/guides/keeping-up-to-date. - Added the ability for an administrator to capture VM console screenshots and update the operating system information. - Added the option for an administrator to add shortcuts to the menu and main dashboard. - Added the option to use the source subnet CIDR with assigned IPv6 route block addresses. - Added the ability to add notes to a Server, User, Hypervisor, IP Block, Server Package, Hypervisor Group, Media Template, System Notes (All admins), and private notes. - Added support for StorPool distributed storage https://docs.virtfusion.com/installation/storage/storpool - Added an option to automatically send an IP change email to the user when a hybrid migration completes. - Added an option to ignore unavailable hypervisors when completing a hybrid migration. - Added an option to enable/disable the security drivers for a VM. - Multiple UI performance improvements. - Improved several aspects of Disaster Recovery and how it handles failed block commits. - Hypervisor metric collectors now run parallel processes for more accurate results. - Disaster Recovery will now proactively check the operating system installed inside a VM and will abort a filesystem freeze if CentOS or CloudLinux is detected. See https://docs.virtfusion.com/next/guides/disaster-recovery#known-issues - End users will now see an alert in the control panel if any disk, ram, cpu up/downgrades etc... are pending. #### 4.0.0 `Testing Build 3` `February 7, 2024` - Added an option to automatically send an IP change email to the user when a hybrid migration completes. - Added an option to ignore unavailable hypervisors when completing a hybrid migration. - Improved several aspects of Disaster Recovery and how it handles failed block commits. - End users will now see an alert in the control panel if any disk, ram, cpu up/downgrades etc... are pending. - Added an option to enable/disable the security drivers for a VM. - Final asset cleanup ready for stable release. #### 4.0.0 `Testing Build 2` `January 22, 2024` - Added the option to use the source subnet CIDR with assigned IPv6 route block addresses. - Added the ability to add notes to a Server, User, Hypervisor, IP Block, Server Package, Hypervisor Group, Media Template, System Notes (All admins), and private notes. - Added support for StorPool distributed storage https://docs.virtfusion.com/next/installation/storage/storpool - Multiple UI performance improvements. #### 4.0.0 `Testing Build 1` `January 1, 2024` - **Major framework upgrade**. This release contains over 16k code changes. Every care has been taken to make sure it is production ready and will work as a seamless update. - Added the ability for an administrator to capture VM console screenshots. - Disaster Recovery will now proactively check the operating system installed inside a VM and will abort a filesystem freeze if CentOS or CloudLinux is detected. See https://docs.virtfusion.com/next/guides/disaster-recovery#known-issues - Added the option for an administrator to add shortcuts to the menu and main dashboard. - Hypervisor metric collectors now run parallel processes for more accurate results. --- ## Version 5.x ### 5.3.x #### 5.3.0 `May 6, 2025` - Added support for end users to assign specific addresses from an IPv6 subnet. - Added the ability for end users to enable/disable auto configuration (cloud-init drive). Especially useful when installing via ISO. - Added the option for end users to switch between UEFI and legacy bios boot. - Added several options to the live migrate CLI. https://docs.virtfusion.com/guides/server-migration#start-a-migration - `--dst-iso` Re-attach the ISO if it was mounted before the migration. - `--dst-user=` An alternative user to root. - `--allow-unsafe` Allow unsafe storage migrations. - Added swap space selection for Linux templates on the server build wizard. Enabled by default and can be disabled via Settings -> Virtualization -> Allow Swap Selection. - Added the ability to view user specific emails when managing a user (Emails tab). - Added support for mass emailing users (Mailouts). https://docs.virtfusion.com/guides/mailouts - Added support for `mariadb-dump` via system backup. - Added Mailout webhooks. - Added backup logs. - Added server change owner API endpoint https://docs.virtfusion.com/api/#tag/servers/PUT/servers/{serverId}/owner/{newOwnerId} - Added trusted hosts functionality https://docs.virtfusion.com/administration/trusted-hosts - Improved VNC port mapping functionality. - Traffic limiting has been improved and is now significantly stricter. - Improved two factor authentication when impersonating a user. - Improved user session validation. - Administrators can no longer use the lost password feature on the login pages. Passwords must be reset from CLI. - Removed support for unlimited (Remember Me) user sessions. - Major hypervisor framework update. #### 5.3.0 `Testing Build 3` `May 1, 2025` - Added backup logs. - Administrators can no longer use the lost password feature on the login pages. Passwords must be reset from CLI. - Improved swap size selection and options are now based on disk and memory size of a VM. - Added trusted hosts functionality. - Added support for `mariadb-dump` in system backup. - Removed support for unlimited (Remember Me) user sessions. #### 5.3.0 `Testing Build 2` `April 16, 2025` - Added Mailout webhooks. - Added server change owner API endpoint https://docs.virtfusion.com/api/#tag/servers/PUT/servers/{serverId}/owner/{newOwnerId} - Added several `server_list_*` variables for Mailouts. - Swap selection can now be globally disabled (Settings -> Virtualization -> Allow Swap Selection). - Added several Mailout recipient options. - Improved two factor authentication when impersonating a user. - Improved user session validation. #### 5.3.0 `Testing Build 1` `April 7, 2025` - Added support for end users to assign specific addresses from an IPv6 subnet. - Added the ability for end users to enable/disable auto configuration (cloud-init drive). - Added the option for end users to switch between UEFI and legacy bios boot. - Added several options to the live migrate CLI. https://docs.virtfusion.com/next/guides/server-migration#start-a-migration - `--dst-iso` Re-attach the ISO if it was mounted before the migration. - `--dst-user=` An alternative user to root. - `--allow-unsafe` Allow unsafe storage migrations. - Added swap space selection for Linux templates on the server build wizard. - Added the ability to view user specific emails when managing a user (Emails tab). - Added support for mass emailing users (Mailouts). https://docs.virtfusion.com/next/guides/mailouts - Improved VNC port mapping functionality. - Traffic limiting has been improved and is now significantly stricter. - Major hypervisor framework update. ### 5.2.x #### 5.2.0 `March 7, 2025` - Removed application support for PHP 8.1 - Make sure your system services are up to date https://docs.virtfusion.com/releases/services#how-to-update - Added PHP 8.3 application support. - Added Cloud-init network version 1 support. - Added the ability to select the Cloud-init network version on a per template basis. - Added the ability to change a server creation date. https://docs.virtfusion.com/reference/cli#change-server-creation-date - Added `dryRun` query parameter to the server create API method. https://docs.virtfusion.com/api/#tag/servers/POST/servers - Added the ability to specify cpuset parameters at a hypervisor level. - Added API method to list hypervisors https://docs.virtfusion.com/api/#tag/hypervisors/GET/compute/hypervisors - Added delete user by external ID API endpoint. https://docs.virtfusion.com/api/#tag/usersexternal-relational-id/DELETE/users/{extRelationId}/byExtRelation - Added the ability to use markdown in an OS template Email Excerpt. - Added API method to retrieve hypervisor group resources https://docs.virtfusion.com/api/#tag/hypervisor-groups/GET/compute/hypervisors/groups/{hypervisorGroupId}/resources - Added API method to retrieve hypervisor groups https://docs.virtfusion.com/api/#tag/hypervisor-groups/GET/compute/hypervisors/groups - Added API method to retrieve a hypervisor group https://docs.virtfusion.com/api/#tag/hypervisor-groups/GET/compute/hypervisors/groups/{hypervisorGroupId} - Added local IP support for backup storage. - Added server disk IO driver support (native, threads, io_uring). Available globally, hypervisor and server level. - Improved Cloud-init configuration on VM migration. - Updated control server framework. - Updated hypervisor backend framework. - Updated IPv6 core packages. - SSH public keys are now in decrypted form when retrieved via API. - Several small bug fixes and performance improvements - Fixed an issue where the hypervisor selection would not show when attempting a VM migration. - Supervisor status and configurations are now available in the admin UI (System -> Health -> Workers). - Various layout fixes and improvements to admin and user UI. #### 5.2.0 `Testing Build 2` `January 17, 2025` - Fixed an issue where the hypervisor selection would now show when attempting a VM migration. - Exposed the delete user by external ID API endpoint. https://docs.virtfusion.com/api/#api-Users-Delete_by_extRelationId - Supervisor status and configurations are now available in the admin UI (System -> Health -> Workers). - Various layout fixes and improvements to admin and user UI. - Added the ability to use markdown in an OS template Email Excerpt. #### 5.2.0 `Testing Build 1` `January 6, 2025` - Removed application support for PHP 8.1 - Added PHP 8.3 application support - Added Cloud-init network version 1 support - Added the ability to select the Cloud-init network version on a per template basis - Improved Cloud-init configuration on VM migration - Added the ability to change a server creation date. https://docs.virtfusion.com/next/reference/cli#change-server-creation-date - Added `dryRun` query parameter to the server create API method. https://docs.virtfusion.com/api/#api-Servers-Create - Added the ability to specify cpuset parameters at a hypervisor level. - Added API method to list hypervisors https://docs.virtfusion.com/api/#api-Hypervisors-List - Updated control server framework - Updated hypervisor backend framework - Updated IPv6 core package (major change) - Several small bug fixes and performance improvements ### 5.1.x #### 5.1.0 `November 8, 2024` - Added hypervisor asset support (GPU/USB/PCI passthrough). See https://docs.virtfusion.com/next/guides/hypervisor-assets for more information. - Introduced new server distribution settings for hypervisor groups: - Lowest available unallocated storage - Lowest available unallocated CPU cores - Lowest available unallocated memory - Lowest available unallocated server slots - Hypervisor with the shortest time since a server was built - Newest available hypervisor - Fixed an issue where Ubuntu templates failed to download correctly from official mirrors. - Resolved a bug that prevented additional storage creation when the type was set to anything other than "inherit." - Added the ability to filter servers by hypervisor ID on the https://docs.virtfusion.com/api/#api-Servers-List API method. - API tokens now track the last successful usage data & time. - Added IP allocation auditing. - Admin sidebar icons now show a description on hover. - Added the option for end users to locally change a server hostname. - Hybrid and live migration now offer the option to transfer disk data over an alternative IP allocated to a hypervisor. - Updated the experimental Native Performance Clean filters to reduce ARP traffic. - Added the ability to disable API logging on a per token basis. - Implemented basic ACLs (restricting dangerous methods) for API tokens. - All migration webhooks now include details about both the source and destination hypervisors. - Frontend assets have been updated to their latest versions. - Hypervisor backend framework has been updated to the latest dependency versions. #### 5.1.0 `Testing Build 1` `October 9, 2024` - Added hypervisor asset support (GPU/USB/PCI passthrough). See https://docs.virtfusion.com/next/guides/hypervisor-assets for more information. - Introduced new server distribution settings for hypervisor groups: - Lowest available unallocated storage - Lowest available unallocated CPU cores - Lowest available unallocated memory - Lowest available unallocated server slots - Hypervisor with the shortest time since a server was built - Newest available hypervisor - Fixed an issue where Ubuntu templates failed to download correctly from official mirrors. - Resolved a bug that prevented additional storage creation when the type was set to anything other than "inherit." - Added the ability to filter servers by hypervisor ID on the https://docs.virtfusion.com/api/#api-Servers-List API method. - API tokens now track the last successful usage data & time. - Hybrid and live migration now offer the option to transfer disk data over an alternative IP allocated to a hypervisor. - Updated the experimental Native Performance Clean filters to reduce ARP traffic. - Added the ability to disable API logging on a per token basis. - Implemented basic ACLs (restricting dangerous methods) for API tokens. - All migration webhooks now include details about both the source and destination hypervisors. - Frontend assets have been updated to their latest versions. - The hypervisor backend framework has been updated to the latest dependency versions. ### 5.0.x #### 5.0.0 `August 29, 2024` - Added support for additional storage configuration in server packages. - Added API overrides for additional storage capacity and profile in the create method. https://docs.virtfusion.com/api/#api-Servers-Create - Added support for FreeBSD memory usage display. - Added the option to verify/unverify account email addresses. - Added release notes to the admin area updates page. - Added basic server and IPv4 information to hypervisor settings. - Added the ability to enforce two-factor authentication for admins, end users or both. - Added the following API methods: - https://docs.virtfusion.com/api/#api-Servers-Traffic_Block_Add - https://docs.virtfusion.com/api/#api-Servers-Traffic_Block_Remove - https://docs.virtfusion.com/api/#api-Servers-Traffic_Blocks_Retrieve - https://docs.virtfusion.com/api/#api-Servers-Traffic_Statistics - https://docs.virtfusion.com/api/#api-Servers-Throttle_CPU - https://docs.virtfusion.com/api/#api-Servers-Backup_Plan_Add_or_Remove - https://docs.virtfusion.com/api/#api-Servers-Modify_Name - https://docs.virtfusion.com/api/#api-Servers-Firewall_Disable - https://docs.virtfusion.com/api/#api-Servers-Firewall_Enable - https://docs.virtfusion.com/api/#api-Servers-Firewall_Retrieve - https://docs.virtfusion.com/api/#api-Servers-Firewall_Rules_Apply - https://docs.virtfusion.com/api/#api-IP_Blocks-List - https://docs.virtfusion.com/api/#api-IP_Blocks-Add_IPv4 - https://docs.virtfusion.com/api/#api-Servers-List - Fixed issue where TPM certificates would not persist across restarts. - Fixed issue where CPU topology would auto enable on server creation via API. - Removed depreciated `priority` element from powerDNS API methods. - Fixed issue with session remember me token generation. - Fixed several client side localization errors. - Fixed issue where several server related database entries were not being purged when force deleting a hypervisor. - User area account settings have been moved to their own section/page. - Improved layout and small screen view of the client server management area. - Improved server boot functionality. - Added the ability to assign up to 600 CPU cores to a server. - Added the option for end users to enable or disable backup notifications. - An email is now sent when a user changes an account password. - Added the option to email traffic limit warnings to end users (Settings -> Virtualization -> Server Traffic Management -> Traffic Limit Notification 1/2). - An email is now sent (to the old and new address) when a user changes an account email address. - Fixed issue where the traffic override when creating a server would convert values incorrectly. - Added CLI command to disable two-factor enforcement (`vfcli-ctrl tfa-force-off`). - Improved system backup deletion routines. - Added Disaster recovery pre and post scripts. https://docs.virtfusion.com/guides//disaster-recovery#custom-scripts - Added various data exporters (System -> Export). - Improved create server distribution calculation routine. - Added `sendMail` field to server reset password method. https://docs.virtfusion.com/api/#api-Servers-Reset_Password - Disaster Recovery will now check the agent before freezing the VM operating system filesystem. - Added VirtIO SCSI Address Type settings under virtualization settings. - Control and hypervisor framework updates. --- ## Version 6.x ### 6.2.x #### 6.2.0 `January 2, 2026` - Added VirtIO SCSI (Threaded) disk bus option. - Added two-factor authentication trusted device (browser) support for all users. - Added global API endpoint to modify server memory [[link]](https://docs.virtfusion.com/api/#tag/servers/PUT/servers/{serverId}/modify/memory). - Added global API endpoint to modify server CPU cores [[link]](https://docs.virtfusion.com/api/#tag/servers/PUT/servers/{serverId}/modify/cpuCores). - Added global API endpoint to modify custom XML [[link]](https://docs.virtfusion.com/api/#tag/servers/POST/servers/{serverId}/customXML). - Added CLI command to regenerate backup storage keys [[link]](https://docs.virtfusion.com/next/reference/cli/#regenerate-backup-storage-keys). - Added option to sparsify user generated backups on transfer. - Added new intelligent backend scheduler that allows scaling across large deployments. - Added support for Bunny.net IPv6 reverse DNS. - Fixed an issue where remote storage disks would fail to resize. - Fixed an issue where secondary server network interfaces could fail to display after a hybrid migration. - Fixed an issue where IPv6 route blocks would not display correctly to end users. - Fixed an issue where the admin hypervisor list would lose sorting order. - User area interface refresh and legacy code cleanup. #### 6.2.0 `Testing Build 2` `December 16, 2025` - Added CLI command to regenerate backup storage keys [[link]](https://docs.virtfusion.com/next/reference/cli/#regenerate-backup-storage-keys). - Added option to sparsify user generated backups on transfer. - Scheduler optimisations and updated crontab management. #### 6.2.0 `Testing Build 1` `December 10, 2025` - Added VirtIO SCSI (Threaded) disk bus option. - Added 2fa trusted device (browser) support for all users. - Added global API endpoint to modify server memory [[link]](https://docs.virtfusion.com/api/#tag/servers/PUT/servers/{serverId}/modify/memory). - Added global API endpoint to modify server CPU cores [[link]](https://docs.virtfusion.com/api/#tag/servers/PUT/servers/{serverId}/modify/cpuCores). - Added global API endpoint to modify custom XML [[link]](https://docs.virtfusion.com/api/#tag/servers/POST/servers/{serverId}/customXML). - Added new intelligent backend scheduler that allows scaling across large deployments. - Fixed an issue where remote storage disks would fail to resize. - Fixed an issue where secondary server network interfaces would fail to display after a hybrid migration. ### 6.1.x #### 6.1.0 `November 20, 2025` - Added segregated end user API. Documentation is available directly from each user account under the Account section, where they can also create and manage API keys. Administrators can disable API access at any time from General Settings. This feature is enabled by default. - Added the ability to set the order IP blocks are used when assigning to servers at hypervisor level (Hypervisor -> IP Blocks). - Added Admin API endpoint to retrieve VNC details https://docs.virtfusion.com/api/#tag/servers/get/servers/%7BserverId%7D/vnc - Added support for API relational identifier strings. - Added support for VLANs on Linux bridges. Requires Libvirt 11+ (Debian 13, RHEL 10). - Added the ability to resize (increase) all server disks. - IPv6 /128 route information is now shown to the end user. - The number of servers is now shown on the hypervisors table. - Improved request throttling on 2FA and reset password functionality (reported by Abdullah Alotaibi). - Fixed an issue where hourly billing may display negative values to end users. - Self-Service Resource Pack enhancements - Traffic for all servers within a Resource Pack is now aggregated and managed at the Resource Pack level. - The direction of traffic counting (inbound, outbound, or both) can be configured within the Resource Pack settings. - Notifications for traffic allowance thresholds can be customised within the Resource Pack settings. - Traffic throttling limits can now be configured at the Resource Pack level through the Resource Pack settings. - Resource Pack limits can now be overridden at the user level. - Server packages can be configured with variable resource limits and defaults, allowing flexible allocation of memory, storage, and CPU for end users. - Compute metrics (the widget on the main admin page) now links to a separate longer list. - Updated control server Javascript dependencies. - Updated control server and hypervisor backend dependencies. #### 6.1.0 `Testing Build 2` `November 7, 2025` - Self-Service Resource Pack enhancements - Traffic for all servers within a Resource Pack is now aggregated and managed at the Resource Pack level. - The direction of traffic counting (inbound, outbound, or both) can be configured within the Resource Pack settings. - Notifications for traffic allowance thresholds can be customised within the Resource Pack settings. - Traffic throttling limits can now be configured at the Resource Pack level through the Resource Pack settings. - Resource Pack limits can now be overridden at the user level. - Server packages can be configured with variable resource limits and defaults, allowing flexible allocation of memory, storage, and CPU for end users. #### 6.1.0 `Testing Build 1` `October 28, 2025` - Added segregated end user API. Documentation is available directly from each user account under the Account section, where they can also create and manage API keys. Administrators can disable API access at any time from General Settings. - Added the ability to set the order IP blocks are used when assigning to servers at hypervisor level (Hypervisor -> IP Blocks). - Added Admin API endpoint to retrieve VNC details https://docs.virtfusion.com/api/#tag/servers/get/servers/%7BserverId%7D/vnc - IPv6 /128 route information is now shown to the end user. - The number of servers is now shown on the hypervisors table. - Improved throttling on 2FA and reset password functionality. - Fixed an issue where hourly billing may display negative values to end users. - Updated control server Javascript dependencies. - Updated control server and hypervisor backend dependencies. ### 6.0.x #### 6.0.2 `August 17, 2025` - Fixed an issue where server deletions via the UI would fail for any period longer than ASAP. - Fixed an issue where TFA could not be enabled by users. - Added operating system template sorting. (Media -> Template Groups). - Minor admin UI tweaks. #### 6.0.1 `August 8, 2025` - Fixed an issue where rescue mode would fail on RHEL based hypervisors due to a bug in the current Libvirt package. - The Native Privacy network filter now drops ICMPv6 code 134 (Router Advertisement). #### 6.0.0 `August 4, 2025` This version brings a complete upgrade to both the frontend and backend underlying frameworks serving as the foundation for VirtFusion v6. - Major control and hypervisor framework updates. - Added UTC to available timezones when creating a server. - Added control & hypervisor support for RHEL 10 (AlmaLinux v2/v3) and Debian 13 (Trixie). - Added the option to lock sessions (admin/end user) to their login IP and/or user agent. - Added the ability to specify a negative credit for a user in self service. https://docs.virtfusion.com/api/#tag/self-serviceexternal-relational-id/post/selfService/credit/byUserExtRelationId/{extRelationId} - Added new CPU models `AMD EPYC Milan v2`, `AMD EPYC Genoa`, `AMD EPYC v2, v3, v4`, `AMD EPYC Rome v2, v3, v4`, `Hygon Dhyana v2`, `Intel Sapphire Rapids v1, v2, v3`, `Intel Granite Rapids v1, v2`, `Intel Sierra Forest`, `Intel Denverton v1 ,v2 ,v3`, `Intel Knights Mill`, `Intel Ice Lake Server v2, v3, v4, v5, v6, v7`. - Added customer facing Rescue Mode for x86 servers. - Added a root CLI command to retrieve the last used password for a server (`vfcli-ctrl server:password show `). - Added noindex,nofollow robots meta tags to guest headers. - Added API endpoint to modify server traffic allowance. https://docs.virtfusion.com/api/#tag/servers/put/servers/{serverId}/modify/traffic - Added the option to specify a reply-to name and address for all emails sent out by the system. - Added a self service negative credit email template. - Fixed issue where traffic accounting periods would show incorrectly if a server creation date was near the end of a month. - Fixed issue when attempting to start an end user backup failed and no error message was being displayed. - Fixed issue where traffic block dates were showing incorrectly via the https://docs.virtfusion.com/api/#tag/serverstraffic/get/servers/{serverId}/traffic/blocks API method. - Last login date now shows correctly when authentication is initiated via API token authentication keys. - Fixed issue where control server health would not display correctly when caching was disabled. - Passwords are now masked in the admin email log. This only applies to logs generated after this update. - The system encryption key is no longer displayed in the UI to administrators. It now must be retrieved using the `vfcli-ctrl app:retrieve-key` CLI command. - The token authentication endpoint is now rate limited to match the standard UI login limits. - Various fixes and enhancements for updated framework compatibility. - Fixed issue where a 2FA code could be used multiple times within its validity timeframe. - Fixed issue when trying to run a hybrid migration on an IPv6 only system would fail. - Fixed display issue where it was possible for CPU usage to show more than 100%. - Tabbed content in customer area now persists over page reloads. - Fixed an issue when adding a token value for self service via the UI would fail. - Fixed admin header layout in Safari browsers. - Increased the amount of characters (300) allowed in the hypervisor CPU Set field. - The last known operating system template used is now shown to administrators. - Fixed an issue with specific date ranges when displaying traffic usage. - Improved the distribution of the required UEFI vars file (fix for RHEL hypervisors). - Fixed display errors in server audit log. - Reduced the frequency that charting data is collected on a hypervisor. #### 6.0.0 `Testing Build 3` `July 14, 2025` - Fixed an issue when adding a token value for self service via the UI would fail. - Fixed admin header layout in Safari browsers. - Added API endpoint to modify server traffic allowance. https://docs.virtfusion.com/api/#tag/servers/put/servers/{serverId}/modify/traffic - Increased the amount of characters allowed in the hypervisor CPU Set field. - The last known operating system template used is now shown to admins. - Fixed an issue with specific date ranges when displaying traffic usage. - Improved the distribution of the required UEFI vars file (fix for RHEL hypervisors). - Added the option to specify a reply-to name and address for all emails sent out by the system. - Final fixes and tweaks to the underlying control and hypervisor frameworks. #### 6.0.0 `Testing Build 2` `June 27, 2025` - Added new CPU models `AMD EPYC Milan v2`, `AMD EPYC Genoa`, `AMD EPYC v2, v3, v4`, `AMD EPYC Rome v2, v3, v4`, `Hygon Dhyana v2`, `Intel Sapphire Rapids v1, v2, v3`, `Intel Granite Rapids v1, v2`, `Intel Sierra Forest`, `Intel Denverton v1 ,v2 ,v3`, `Intel Knights Mill`, `Intel Ice Lake Server v2, v3, v4, v5, v6, v7`. - Added customer facing Rescue Mode for x86 VMs. - Added a root CLI command to retrieve the last used password for a VM (`vfcli-ctrl server:password show `). - Added noindex,nofollow robots meta tags to guest headers. - Fixed issue where a 2FA code could be used multiple times within its validity timeframe. - Fixed issue when trying to run a hybrid migration on an IPv6 only system would fail. - Fixed display issue where it was possible for CPU usage to show more than 100%. - Tabbed content in customer area is now persistent. #### 6.0.0 `Testing Build 1` `June 11, 2025` This version brings a complete upgrade to both the frontend and backend underlying frameworks serving as the foundation for VirtFusion v6. - Major control and hypervisor framework updates. - Added UTC to available timezones when creating a VM. - Added control & hypervisor support for RHEL 10 (AlmaLinux v2/v3) and Debian 13 (Trixie). - Added the option to lock sessions (admin/end user) to their login IP and/or user agent. - Added the ability to specify a negative credit for a user in self service. https://docs.virtfusion.com/api/#tag/self-serviceexternal-relational-id/post/selfService/credit/byUserExtRelationId/{extRelationId} - A new self service negative credit email template has been added. - Fixed issue where traffic accounting periods would show incorrectly if a VM creation date was near the end of a month. - Fixed issue when attempting to start an end user backup failed and no error message was being displayed. - Fixed issue where traffic block dates were showing incorrectly via the https://docs.virtfusion.com/api/#tag/serverstraffic/get/servers/{serverId}/traffic/blocks API method. - Last login date now shows correctly when authentication is initiated via API token authentication keys. - Fixed issue where control server health would not display correctly when caching was disabled. - Passwords are now masked in the admin email log. This only applies to logs generated after this update. - The system encryption key is no longer displayed in the UI to administrators. It now must be retrieved using the `vfcli-ctrl app:retrieve-key` CLI command. - The token authentication endpoint is now rate limited to match the standard UI login limits. - Various fixes and enhancements for updated framework compatibility. --- ## Version 7.x ### 7.0.x #### 7.0.6 `September 9, 2026` **Security** - **This release resolves several security vulnerabilities, including one high-priority issue. All installations should be updated to 7.0.6 as soon as possible. Full details will be published in a coordinated disclosure once 7.0.6 has been widely applied.** **Improvements** - **API**: The admin API now rate limits repeated failed authentication attempts per IP address. - **API**: Admin API tokens now support a configurable per-token request rate limit (default 5000 requests per minute, or unlimited), returning standard rate-limit response headers. - **API**: Admin API tokens can now be given an optional expiry date and time, after which they stop working. Tokens without an expiry never expire. - **Hypervisor**: The CPU Set field now accepts a larger range of values, allowing servers to be pinned to a wider set of host CPU cores. **Fixes** - **Backup Manager**: Fixed cross-server restores using the source backup's disk format instead of the target server's configured format, which could leave the restored VM unable to boot when the two differed (for example, restoring a qcow2 backup onto a raw disk). The target server's disk format now takes priority. - **Backup Manager**: Fixed S3 restores failing when the storage provider returned an HTTP 429 (Too Many Requests), 500, or 503 response, which were previously treated as fatal and could abort a large restore partway through. These responses are now retried with a longer backoff than connection-level errors to respect the provider's rate limit. #### 7.0.5 `August 13, 2026` **New Features** - **Backup Manager**: Backups can now be restored to a different server. Cross-server restore lets you take a backup made on one server and restore it onto another, rather than being limited to the server the backup was originally created from. See the [cross-server restore documentation](https://docs.virtfusion.com/administration/backup-manager#cross-server-restore) for details. **Fixes** - **Backup Manager**: Fixed pagination in the batch detail view so all jobs in a batch can be browsed correctly. - **Backup Manager**: Fixed zero-chunk detection for encrypted backups stored on a Proxmox Backup Server. Empty (all-zero) chunks in encrypted backups were not being recognised correctly. #### 7.0.4 `July 21, 2026` **Improvements** - **Backup Manager**: Suspended VMs are now automatically skipped during scheduled backup runs, saving storage and avoiding unnecessary work. Backups resume on the next scheduled run once the VM is unsuspended. - **Search**: Reduced search query times on large clusters. - **Server Management**: User selection when creating a VM or changing a VM's owner now loads results as you search, instead of loading every user at once. This improves performance on installations with large numbers of users. **Fixes** - **Backup Manager**: Fixed an error that stopped the Backups page in the admin area from loading when a daily, weekly, monthly, or yearly retention policy was in use. Other retention settings were not affected. - **Backup Manager**: Fixed VMs becoming temporarily unresponsive during backup when a slow VM agent caused the filesystem freeze command to time out. The VM agent timeout has been increased from 5 to 30 seconds, and a precautionary thaw is now sent after resuming a paused VM so any lingering freeze is cleared immediately. - **Backup Manager**: Fixed the admin dashboard throwing an error instead of loading when recent activity contained both a failed batch run and a failed individual job. - **Backup Manager**: Purges from a Proxmox Backup Server no longer report as "completed" when the deletion actually failed (typically due to insufficient PBS permissions). Failed purges now report correctly and the backups remain visible so they can be retried. **Security** - **Email**: Fixed an issue where the full email body was written to the logs when a message failed to send. For emails that included credentials, such as a build or password reset email, this could expose a password in plain text (administrators who have access). Failed email sends are now logged with masked passwords. #### 7.0.3 `June 19, 2026` - **Fix** - **Backup Manager** - Fixed auto-prune phase starvation where storages could be skipped indefinitely if a previous prune cycle was interrupted. - **Improvement** - **Backup Manager** - Added per-storage I/O timeout override. Configurable from the storage Performance settings to prevent timeout failures on slow or overloaded storage backends. #### 7.0.2 `June 11, 2026` - **Fix** - **Backup Manager** - Fixed backup restores failing on Cloudflare R2 and other strict S3-compatible storage providers. - **Fix** - **Backup Manager** - Fixed a timeout that could occur when finalising backups on slower PBS storage. - **Fix** - **Backup Manager** - Fixed backup jobs remaining in a "running" state when a hypervisor unexpectedly loses connectivity during a backup. - **Improvement** - **Backup Manager** - Improved error messages when S3 restores fail. The actual cause is now shown instead of a generic error. - **Improvement** - **Backup Manager** - Added automatic cleanup of stale jobs that were left behind by incomplete batch runs. #### 7.0.1 `May 28, 2026` - **Fix** - **Backup Manager** - When a VM's disk was resized between backups, the next incremental PBS backup would fail with "no such chunk" errors. The backup manager now detects the size change and automatically falls back to a full backup, then resumes incremental backups from the next run onwards. - **Fix** - **Backup Manager** - In some cases, errors that should have triggered an automatic fallback to full backup were being wrapped in a generic error, preventing the fallback from activating. These errors are now correctly preserved so the retry logic works as expected. - **Fix** - **Backup Manager** - Fixed PBS encryption key handling so that keys configured in the storage settings are correctly loaded. Previously, test-connection would fail with "unknown error" when encryption was enabled, and backups would fail to read the key. Encryption keys can be generated with a simple openssl command and configured directly in the storage settings. See the documentation for setup instructions. #### 7.0.0 `May 21, 2026` - **Feature** - **Backup Manager** - A new integrated backup system with support for multiple storage backends and flexible retention policies. [docs](https://docs.virtfusion.com/administration/backup-manager) - Native Proxmox Backup Server integration with server-side deduplication - Support for SSH/SFTP, S3, Rclone, and FTP/FTPS storage backends - Incremental, differential, and full backup types with a custom sparse-aware binary format - Per-frame compression and optional encryption - GFS, time and count-based retention policies with automatic pruning - Scheduled batch backups with concurrent VM processing - Schedule day restrictions to limit backups to specific days of the week - Support for Ceph RBD backed disks with native snapshot-based full and incremental backups and direct block device restore - Administrative dashboard with real-time backup health overview, recent errors, and storage utilisation - User-facing backup tree with on-demand backup and point-in-time restore - Live progress tracking during backup and restore operations - Post-restore filesystem repair automatically runs filesystem integrity checks - Per-VM access controls with package-level defaults - Advanced CLI for file-level recovery (mount), integrity verification, metadata reindex, dirty data estimation, manual pruning, and self-healing from interrupted operations - Granular control over individual operations including per-disk backup type selection, mixed backup types in a single run, point-in-time restore, and resumable uploads - **Feature** - **Step Up Authentication** - Additional authentication challenges for sensitive operations. - **Feature** - **Arm, Debian 13 & RHEL 10** - Support for ARM hypervisor architecture along with Debian 13 and RHEL 10 operating systems. **Feature** Support for native WebSocket Secure (wss) connections. Run apt update && apt install vf-control-wss or dnf install vf-control-wss on the control server to gain full benefits. - **Feature** - **Modernised VNC Console** - Complete rewrite of the browser-based VNC client: - Auto-connect with automatic reconnection on disconnect - Live status badge showing server name and connection state, with frame-rate, Caps Lock, and sticky-modifier indicators - Three quality presets (Fast/Balanced/Best) switchable during a live session - Clipboard panel with "Send" and "Type" modes - Type simulates keystrokes for servers without a clipboard manager - Saved snippets - save up to 50 reusable commands or credentials for quick access - Session recording - record the console to WebM video (up to 30 minutes) with one-click download - Picture-in-Picture support (Chromium) with full mouse and keyboard forwarding - View-only mode, pinch-to-zoom on touch devices, and virtual keyboard support - Auto-hiding toolbar with pin option, extra keys panel (function keys, modifiers, Ctrl+Alt+Del), power controls, and screenshot - Audible bell with on-screen notification when the remote rings - Quality preset, view-only choice, and toolbar pin state remembered between sessions - Adapts to system light/dark theme - Mobile responsive layout with collapsible toolbar on narrow screens - **Improvement** - **Filesystem Freeze for Disaster Recovery** - Option to define filesystem freeze behaviour for DR configurations. - **Improvement** - **End User API Custom RPM Per User** - Custom requests-per-minute limits can now be set on a per-user basis for the end user API. - **Improvement** - **Ceph Image Create and Deploy** - Improved Ceph image creation and deployment process. - **Improvement** - End users can now create and delete resource pack based VMs via the API. - **Fix** - Fixed Cron Manager incorrectly showing as not running on RHEL. - **Fix** - Fixed admin ACL for manage hypervisor assets being incorrect. - **Fix** - Fixed network tab not showing IPv6 status in the client area. - **Fix** - Fixed sharing tab not refreshing correctly. - **Fix** - API auth and white screen when Advanced Session Validation is enabled. - **Fix** - Race condition and stale file descriptors in traffic monitoring script. - **Maintenance** Updated hypervisor framework to support the latest KVM and QEMU releases. - **Maintenance** Control framework and dependency updates for improved stability and security. #### 7.0.0 `Testing Build 11` `May 9, 2026` - **Fix: Backup Manager** - SSH restores of large disks no longer incorrectly report failure when the remote TCP connection drops after all backup data has already been successfully transferred. Previously, the SSH exit status was treated as fatal even though the restore data was complete. - **Fix: Backup Manager** - Job status tracking now works correctly for VMs that have been migrated between hypervisors. Previously, restores on migrated VMs could show exit code -1 and incorrect failure status because the job progress file could not be located on the new hypervisor. #### 7.0.0 `Testing Build 10` `May 7, 2026` - **Fix: Backup Manager** - Restore no longer reports "completed" while the post-restore integrity check is still running. - **Improvement: Backup Manager** - Integrity check timeout now scales with disk size and monitors process group activity. - **Fix: VNC Console** - Clipboard now correctly handles uppercase letters and shifted symbols. #### 7.0.0 `Testing Build 9` `April 27, 2026` - **Feature: Modernised VNC Console** - Complete rewrite of the browser-based VNC client: - Auto-connect with automatic reconnection on disconnect - Three quality presets (Fast/Balanced/Best) switchable during a live session - Clipboard panel with "Send" and "Type" modes - Type simulates keystrokes for servers without a clipboard manager - Saved snippets - save up to 50 reusable commands or credentials for quick access - Session recording - record the console to WebM video (up to 30 minutes) with one-click download - Picture-in-Picture support (Chromium) with full mouse and keyboard forwarding - View-only mode, pinch-to-zoom on touch devices, and virtual keyboard support - Auto-hiding toolbar with pin option, extra keys panel (function keys, modifiers, Ctrl+Alt+Del), power controls, and screenshot - Mobile responsive layout with collapsible toolbar on narrow screens - **Fix: Backup Manager** - Inventory catchup sync no longer retries indefinitely for backups that have been pruned by retention policy. #### 7.0.0 `Testing Build 8` `April 24, 2026` - **Feature: Backup Manager** - Post-restore integrity checks now run in full isolation mode using a lightweight appliance. Falls back to sandboxed mode if unavailable. Integrity checks are now enabled by default for all restores (admin and user). - **Feature: Backup Manager** - Integrity checks automatically detect and skip filesystems with features newer than the installed repair tools, reporting them as skipped rather than failed. - **Feature: Backup Manager** - Option to disable the legacy backup system when the new Backup Manager is active. - **Feature:** Support for native WebSocket Secure (wss) connections. Run apt update && apt install vf-control-wss or dnf install vf-control-wss on the control server to gain full benefits. - **Fix:** API auth and white screen when Advanced Session Validation is enabled. - **Fix:** Race condition and stale file descriptors in traffic monitoring script. - **Fix:** Backup Manager - Retention policy display in batch history now shows human-readable labels instead of raw keys. #### 7.0.0 `Testing Build 7` `April 20, 2026` - **Feature: Backup Manager** - GFS (Grandfather-Father-Son) retention. A new retention model alongside the existing age-based presets. GFS keeps daily, weekly, monthly, and yearly snapshots on independent schedules. Configurable per-storage with per-schedule overrides. Works as a union with keep_last. - **Feature: Backup Manager** - User cooldown. Two new global settings (Backup Cooldown, Restore Cooldown) enforce a per-server waiting period between successful manual operations. Failed or cancelled jobs do not trigger the cooldown. Scheduled batch backups are excluded. Default: 60 minutes. Set to 0 to disable. - **Fix: Backup Manager** - Fixed batch backups failing silently when the target storage is unreachable. The error is now properly surfaced. - **Fix: Backup Manager** - Fixed S3 stale record check incorrectly deleting valid records on non-standard S3 providers. #### 7.0.0 `Testing Build 6` `April 14, 2026` - **Feature: Backup Manager** - CLI restore to file. A new button on the per-server backup tree generates a ready-to-run CLI command for restoring a backup directly to a file on the hypervisor via SSH, without affecting the running VM. - **Improvement: Backup Manager** - Async auto-purge. Deleted VM backup cleanup is now non-blocking. Each server+storage pair is dispatched as an independent tracked job instead of running synchronously. - **Improvement: Backup Manager** - PBS chain delete warning. Deleting a backup chain on PBS storage now warns that PBS snapshots are shared across all disks and shows the total affected record count. - **Improvement: Backup Manager** - Removed disk detection. Disks that no longer exist on a server are shown with a "removed" badge in the backup tree and are automatically excluded from restore operations. - **Improvement: Backup Manager** - Inventory sync catchup. Completed backup jobs missing inventory records are automatically retried with backoff. Log messages now include job, server, and hypervisor IDs for easier diagnosis. - **Improvement: Backup Manager** - Log size limits. Hypervisor log endpoints now cap output (2 MB plain text, 5,000 lines) to prevent large batch logs from causing memory issues. - **Fix: Backup Manager** - Fixed pagination losing filters. All admin pages now preserve filters and per-page settings when navigating between pages. - **Fix: Backup Manager** - 11 minor binary fixes including safe purge cleanup, scoped backup deletion, soft-delete schedule lookup, and restore error check. **Fix: Backup Manager** - 11 minor binary fixes including safe purge cleanup, scoped backup deletion, soft-delete schedule lookup, and restore error check. #### 7.0.0 `Testing Build 5` `April 12, 2026` - **Improvement: Backup Manager** - Allowed days scheduling. Schedules can now be restricted to specific days of the week (e.g. weekdays only, Mon/Wed/Fri). Configured via toggle buttons in the schedule edit modal. All next-due calculations, the dashboard, and clone reflect the setting. - **Improvement: Backup Manager** - Next runs preview. The schedule edit modal now shows a live preview of the next 8 scheduled run times based on the current interval, time window, and allowed days settings. - **Improvement: Backup Manager** - Richer batch job details. Batch results now include per-disk data (backup type, size, virtual size, duration, chain ID), giving the panel immediate access to full job details without re-reading log files. Completed/cancelled jobs no longer show a spurious "No progress data available" error. - **Fix: Backup Manager** - Fixed time window drift. Schedules with an interval longer than the time window (e.g. 24h interval, 2h window) could gradually drift toward the end of the window. The interval is now anchored to the window start time. - **Fix: Backup Manager** - Incremental and differential PBS backups now correctly report the disk type (inc/diff) in job details. - **Fix: Backup Manager** - Fixed stale error on inventory sync. The panel-to-hypervisor sync client now correctly handles stale response errors instead of treating them as failures. - **Fix: Backup Manager** - Fixed purge not finding deleted VMs. #### 7.0.0 `Testing Build 4` `April 11, 2026` - **Improvement: Backup Manager** - Post-restore filesystem repair. After restoring a VM, the backup manager now automatically runs filesystem integrity checks (xfs_repair, e2fsck, btrfs check, ntfsfix) on restored disks before starting the VM. All repair errors are non-fatal warnings. Configurable via the "Verify Filesystem Integrity" toggle in the restore modal. - **Improvement: Backup Manager** - Automated retention pruning. Expired backups are now automatically pruned based on the configured retention policy (keep_last, daily, weekly, monthly, yearly). - **Improvement: Backup Manager** - System logs. Adds a dedicated filterable and paginated log viewer for all dispatcher events (dispatches, completions, failures, prune actions, purge operations, health checks). - **Improvement: Backup Manager** - Global settings. Includes dispatcher pause (stops new scheduled batches while existing jobs finish), global concurrent batch limit, stale detection thresholds, purge/prune intervals, health check timing, job retention period, info-level logging, and the ability to globally disable user-initiated manual backups or restores. - **Fix: Backup Manager** - Fixed chain ID collisions when a VM has backups across multiple storage backends. - **Fix: Backup Manager** - Fixed inventory sync for completed backup jobs. Jobs that complete successfully but fail the initial inventory sync (e.g. due to a network timeout) are now automatically retried with a backoff, ensuring backup records are not permanently lost. - **Fix: Backup Manager** - Fixed purge deleting backup records from all storages. When purging a deleted VM's backups, only records for the specific storage being purged are now removed, preserving records on other storages until they are successfully purged too. - **Fix: Backup Manager** - Fixed restore jobs not detecting hypervisor error responses. Restore jobs now correctly detect and report errors returned by the hypervisor (e.g. process already running). - **Fix: Backup Manager** - Multiple stability and correctness fixes. Cancelled batches now work when the schedule has been deleted, prune job IDs use collision-resistant generation, purge failure counting no longer triggers premature backoff on multi-storage servers, and date range filters are resilient to malformed input. #### 7.0.0 `Testing Build 3` `April 8, 2026` - **Improvement #3680: Backup Manager** - Duplicate a schedule's settings to multiple hypervisors. - **Improvement #3690: Backup Manager** - S3 multipart upload minimum part size is now configurable. - **Improvement #3700: Backup Manager** - Last 20 jobs for the VM with expandable detail rows, log viewer, per-disk results in admin per VM view. - **Improvement #3710: Backup Manager** - Show schedule status (scheduled/overdue/manual/disabled), interval, storage, next due, last backup time on admin VM view. - **Improvement #3720: Backup Manager** - Backup size totals, per-storage breakdown when multi-storage on per vm admin view. - **Improvement #3730: Backup Manager** - Healthy/failure alert with consecutive failure count, last error, last backup type and time on per vm admin view. - **Improvement #3740: Backup Manager** - Admin API route to modify per server access to backups. [doc](https://docs.virtfusion.com/api/#tag/serversbackup-manager/PUT/servers/{serverId}/backupManager/access) - **Improvement #3750: Backup Manager** - Admin can cancel running backup jobs, restore jobs, and entire batches from the UI. - **Improvement #3760: Backup Manager** - Per-VM progress tracking during batch backups with automatic cleanup of old job data. - **Improvement #3770: Backup Manager** - Stale batch and job detection — orphaned jobs are automatically cancelled and marked as failed. - **Fix #3780: Backup Manager** - Fixed issue where online backups could crash VMs on hypervisors running QEMU 7.2. - **Fix #3790: Backup Manager** - Restores now correctly skip disk size validation. #### 7.0.0 `Testing Build 2` `April 6, 2026` **Fix: Possible online backup crashes VMs on QEMU < 9.1** (`7250b75`) A bug in the backup cleanup process could cause the VM's QEMU process to terminate unexpectedly during an online backup (CVE-2024-7409). This affected all QEMU versions up to 9.1 and required a manual VM restart. **Fix: Offline backups no longer hang before failing** (`4b562f7`) Offline backups would hang for 120 seconds then fail with a timeout error. An internal connection was not being released properly, preventing the backup from proceeding. **Fix: Offline backup progress now reports correctly** (`f4d7754`, `f4a2309`) Offline backups previously showed 0% progress throughout the entire operation. Progress now updates in near real-time. **Fix: Incorrect hypervisor license count calculation** (`a83f1d2`) Hypervisor count was being calculated incorrectly, which could cause license validation to fail or report inaccurate usage. #### 7.0.0 `Testing Build 1` `April 6, 2026` - **Feature #3590** - **Backup Manager** - A new integrated backup system with support for multiple storage backends and flexible retention policies. [docs](https://docs.virtfusion.com/next/administration/backup-manager) - Native Proxmox Backup Server integration with server-side deduplication - Support for SSH/SFTP, S3, Rclone, and FTP/FTPS storage backends - Incremental, differential, and full backup types with a custom sparse-aware binary format - Per-frame compression and optional encryption at rest - GFS and count-based retention policies with automatic pruning - Scheduled batch backups with concurrent VM processing - User-facing backup tree with on-demand backup and point-in-time restore - Live progress tracking during backup and restore operations - Per-VM access controls with package-level defaults - Advanced CLI for file-level recovery (mount), integrity verification, metadata reindex, dirty data estimation, manual pruning, and self-healing from interrupted operations - Granular control over individual operations including per-disk backup type selection, mixed backup types in a single run, point-in-time restore, and resumable uploads - **Feature #3470** - **Step Up Authentication** - Additional authentication challenges for sensitive operations, adding an extra layer of security. - **Feature #3530** - **Arm, Debian 13 & RHEL 10** - Support for ARM hypervisor architecture along with Debian 13 and RHEL 10 operating systems. - **Improvement #3550** - **Filesystem Freeze for Disaster Recovery** - Option to define filesystem freeze behaviour for DR configurations. - **Improvement #3450** - **End User API Custom RPM Per User** - Custom requests-per-minute limits can now be set on a per-user basis for the end user API. - **Improvement #3600** - **Ceph Image Create and Deploy** - Improved Ceph image creation and deployment process. - **Improvement #3440** - End users can now create and delete resource pack based VMs via the API. - **Fix #3510** - Fixed Cron Manager incorrectly showing as not running on RHEL. - **Fix #3540** - Fixed admin ACL for manage hypervisor assets being incorrect. - **Fix #3560** - Fixed network tab not showing IPv6 status in the client area. - **Fix #3570** - Fixed sharing tab not refreshing correctly. - **Maintenance #3640** Updated hypervisor framework to support the latest KVM and QEMU releases. - **Maintenance #3650** Control framework and dependency updates for improved stability and security. --- ## Security Advisories This section documents security issues that affect VirtFusion hypervisors, with patch status for the Linux distributions VirtFusion supports (Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 9/10, and Rocky Linux 9/10). Each advisory explains what the issue is, whether your hosts are affected, how to patch, and how to mitigate in the meantime. ### The KVM Escape Trilogy During 2026, security researcher Hyunwoo Kim (V4bel) disclosed three related VM-to-host escape vulnerabilities in KVM, collectively called "The KVM Escape Trilogy". All three can affect VirtFusion hypervisors, split by CPU architecture: - **[Januscape (CVE-2026-53359)](januscape.md)**: x86 (Intel and AMD). A use-after-free in the KVM shadow MMU, present since 2010. - **[Zapscape (CVE-2026-64561)](zapscape.md)**: x86 (AMD, and Intel under a specific EPT condition). A use-after-free in the KVM shadow MMU. - **[ITScape (CVE-2026-46316)](itscape.md)**: arm64 only. A double-free use-after-free in KVM's vGIC-ITS emulation. VirtFusion runs arm64 hypervisors on Debian 12/13, AlmaLinux 9/10, and Rocky Linux 9/10. Debian 12 is not affected (the flaw was introduced after its kernel branched); Debian 13, AlmaLinux, and Rocky Linux are affected and fixed (with one gap: Rocky Linux 10 has no ITScape-specific erratum yet). Which advisories apply to a given hypervisor depends on its CPU: x86_64 hosts are exposed to Januscape and Zapscape, while arm64 hosts are exposed to ITScape. ### Timeline Each event below is dated from the upstream commit, the public disclosure, or the vendor advisory. AlmaLinux and Rocky Linux ship the same kernel builds but publish separate advisories on separate dates. As of **2026-08-24**, a few gaps remain: Ubuntu 22.04 and 24.04 are fixed for Januscape but not yet for Zapscape; Debian 12's default 6.1 kernel is not yet patched for Zapscape (its `linux-6.12` backport is); and Rocky Linux 10 still has no ITScape erratum (see the [ITScape advisory](itscape.md)). | Date | Event | | --- | --- | | 2010-08-01 | Januscape flaw introduced in KVM/x86 (roughly 16 years before disclosure) | | 2024-04-25 | ITScape flaw introduced in KVM/arm64 | | 2026-06-05 | ITScape fixed in the mainline Linux kernel (CVE-2026-46316) | | 2026-06-10 | ITScape publicly disclosed | | 2026-06-16 | Januscape fixed in the mainline Linux kernel (CVE-2026-53359) | | 2026-06-21 | Debian ships the ITScape fix (DSA-6355-1, trixie `6.12.94-1`) | | 2026-07-02 | AlmaLinux 10 ships the ITScape fix (ALSA-2026:34911) | | 2026-07-05 | Debian ships the Januscape fix (DSA-6381-1, bookworm `6.1.180-1` / trixie `6.12.95-1`) | | 2026-07-06 | Januscape publicly disclosed; AlmaLinux 9 ships the ITScape fix (ALSA-2026:36018) | | 2026-07-09 | AlmaLinux ships the Januscape fix (ALSA-2026:36957 for 9, ALSA-2026:36956 for 10) | | 2026-07-11 | Rocky Linux 9 ships the ITScape fix (RLSA-2026:36018); Rocky Linux 10 ships the Januscape fix (RLSA-2026:36956) | | 2026-07-13 | Rocky Linux 9 ships the Januscape fix (RLSA-2026:36957) | | 2026-07-21 | Zapscape fixed in the mainline Linux kernel (CVE-2026-64561) | | 2026-07-24 | AlmaLinux ships the Zapscape fix (ALSA-2026:45192 for 9, ALSA-2026:45114 for 10) | | 2026-07-27 | Rocky Linux ships the Zapscape fix (RLSA-2026:45192 for 9, RLSA-2026:45114 for 10) | | 2026-08-04 | Zapscape publicly disclosed | | 2026-08-06 | Debian ships the Zapscape fix (DSA-6415-1, trixie `6.12.101-1`) | ### What to do In every case the fix is the same: **patch the host kernel and reboot**. See the per-distribution fixed versions in each advisory. The interim options differ by architecture: - **x86_64 hosts (Januscape, Zapscape)**: if a fixed kernel is not yet available, disable nested virtualization as an interim mitigation. The full procedure is in each advisory. - **arm64 hosts (ITScape)**: there is no practical mitigation short of patching, so affected hosts (Debian 13, AlmaLinux 9/10, and Rocky Linux 9/10) should be updated as a priority. Start with the advisory for the issue you are checking: - [Januscape (CVE-2026-53359)](januscape.md) (x86) - [Zapscape (CVE-2026-64561)](zapscape.md) (x86) - [ITScape (CVE-2026-46316)](itscape.md) (arm64) --- ## ITScape (CVE-2026-46316) import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; :::danger VM-to-host escape (arm64 only) ITScape is a **VM-to-host escape** in the Linux kernel's KVM/arm64 subsystem. A tenant with **root inside their own VM** can corrupt the host kernel and run code as root on the hypervisor, breaking isolation from every other VM on that host. This flaw is **arm64 only**. It does not affect x86_64 hosts. ::: :::info Applies to VirtFusion arm64 hypervisors only This advisory is relevant only to **arm64** hypervisors. Any x86_64 hypervisor is unaffected. If all of your hypervisors are x86_64, this advisory does not apply to you; see [Januscape](januscape.md) and [Zapscape](zapscape.md) instead. ::: ### Summary | Field | Detail | | --- | --- | | **CVE** | CVE-2026-46316 | | **Nickname** | ITScape (part of "The KVM Escape Trilogy", alongside Januscape and Zapscape) | | **Component** | Linux kernel, KVM/arm64 vGIC-ITS emulation (`arch/arm64/kvm/vgic/`) | | **Class** | Double-free use-after-free (UAF) | | **Impact** | VM-to-host escape, leading to arbitrary code execution as root on the host | | **Architecture** | **arm64 only.** x86_64 hosts are not affected. | | **Introduced** | Commit `8201d1028caa` (2024-04-25) | | **Upstream fix** | Commit `13031fb6b835` (merged mainline 2026-06-05) | | **Where it applies** | VirtFusion arm64 hypervisors. This does **not** affect the VirtFusion control server. | ### Am I affected? ITScape affects the **host kernel** on **arm64 hypervisors** only. It is not a VirtFusion application flaw and does not affect the control server or any x86_64 host. You are potentially exposed if all of the following are true: - The hypervisor is **arm64** and runs an **unpatched Linux kernel** that includes the vulnerable vGIC-ITS code, introduced in April 2024. - A VM is controlled by an **untrusted party with root inside the VM**, which is the normal situation in a multi-tenant hosting environment. The vulnerable code arrived in 2024, so kernels that branched earlier do not contain it. **Debian 12 (bookworm) is not affected** because its 6.1 kernel predates the flaw. The AlmaLinux and Rocky Linux kernels include the affected code and are affected. See the table below for the exact status of each arm64 platform. ### Technical details ITScape is a double-free use-after-free in KVM's virtual GIC Interrupt Translation Service (vGIC-ITS) emulation. The function `vgic_its_invalidate_cache()` performs a double-put on a reference count, which frees an object that is still in use and lets a VM drive the host kernel into reusing freed memory. The bug lives entirely inside in-kernel KVM rather than QEMU user space, so a successful exploit yields host kernel privilege rather than a user-process compromise. Triggering it requires root inside the VM to drive GIC/ITS MMIO operations, a VM with at least two vCPUs, and precise timing to win the race. It was disclosed as the first VM-to-host escape research targeting KVM/arm64. The upstream fix (`13031fb6b835`) corrects the reference counting so the object can no longer be freed while still referenced. The disclosure also points to a related fix (CVE-2026-46317); a current distribution kernel includes both. ### Patch status by distribution VirtFusion runs arm64 hypervisors on Debian 12/13, AlmaLinux 9/10, and Rocky Linux 9/10, so those are the platforms tracked here. :::caution This is a point-in-time snapshot The state below reflects **{`2026-08-24`}**. Distribution patch status changes as errata ship. **Always confirm against the vendor's live security tracker** (linked in [References](#references)) before deciding a host is safe. ::: | Distribution (arm64) | Status | Fixed kernel / advisory | | --- | --- | --- | | **Debian 12 (bookworm)** | Not affected | Vulnerable code not present in the 6.1 kernel | | **Debian 13 (trixie)** | Patched | `linux` **6.12.94-1** (DSA-6355-1) | | **AlmaLinux 9** | Patched | `kernel-5.14.0-687.22.1.el9_8` or newer (ALSA-2026:36018) | | **AlmaLinux 10** | Patched | `kernel-6.12.0-211.30.1.el10_2` or newer (ALSA-2026:34911) | | **Rocky Linux 9** | Patched | `kernel-5.14.0-687.22.1.el9_8` or newer (RLSA-2026:36018) | | **Rocky Linux 10** | No dedicated advisory | Fix included in `kernel-6.12.0-211.32.1.el10_2` or newer (via RLSA-2026:36956); see the note below | #### Debian - **Debian 12 (bookworm)**: The vulnerable vGIC-ITS code was introduced after bookworm's kernel branched, so bookworm is not affected and no action is required for ITScape. - **Debian 13 (trixie)**: Fixed in `linux` **6.12.94-1** (DSA-6355-1). Update and reboot. #### AlmaLinux - **AlmaLinux 9**: Fixed in `kernel-5.14.0-687.22.1.el9_8` or newer (ALSA-2026:36018). Update and reboot. - **AlmaLinux 10**: Fixed in `kernel-6.12.0-211.30.1.el10_2` or newer (ALSA-2026:34911). Update and reboot. #### Rocky Linux - **Rocky Linux 9**: Fixed in `kernel-5.14.0-687.22.1.el9_8` or newer (RLSA-2026:36018). Update and reboot. - **Rocky Linux 10**: No ITScape-specific erratum has been published. See the caution below for how to get the fix. :::caution Rocky Linux 10 has no ITScape-specific advisory As of this snapshot, Rocky Linux has **not** published an erratum for CVE-2026-46316 on Rocky 10 (the expected RLSA-2026:34911 does not exist), so vulnerability scanners keyed on Rocky errata may report Rocky 10 as uncovered for ITScape. The code fix is nonetheless present in later Rocky 10 kernels: `kernel-6.12.0-211.32.1.el10_2` (shipped 2026-07-11 as RLSA-2026:36956, the Januscape erratum) is a newer build than the `211.30.1` that first carried the ITScape fix. Update Rocky 10 hosts to the latest available kernel and reboot. A fully patched Rocky 10 host carries the ITScape fix even though no ITScape erratum names it. ::: ### Remediation The fix is a **host kernel update followed by a reboot** on every affected arm64 hypervisor. A kernel update does not take effect until the host is running the new kernel, so the reboot is required. **Debian** ```shell apt-get update -y && apt-get install --only-upgrade linux-image-arm64 -y # Reboot into the new kernel once VMs are migrated/stopped: reboot ``` **AlmaLinux / Rocky Linux** ```shell dnf makecache -y && dnf update kernel -y # Reboot into the new kernel once VMs are migrated/stopped: reboot ``` After rebooting, confirm the host is running a fixed kernel: ```shell uname -r ``` Compare the output against the fixed version in the [table above](#patch-status-by-distribution). #### No interim mitigation Unlike the x86 issues in this series, ITScape has **no practical mitigation short of patching**. It does not depend on nested virtualization, and the vGIC-ITS cannot simply be disabled because it provides MSI delivery that arm64 VMs rely on. Affected arm64 hosts should therefore be patched and rebooted as a priority. ### References - [ITScape write-up and PoC (V4bel/ITScape)](https://github.com/V4bel/ITScape) - [oss-security disclosure, ITScape (CVE-2026-46316)](https://seclists.org/oss-sec/2026/q2/877) - [NVD, CVE-2026-46316](https://nvd.nist.gov/vuln/detail/CVE-2026-46316) - [Debian security tracker, CVE-2026-46316](https://security-tracker.debian.org/tracker/CVE-2026-46316) - [AlmaLinux errata, ALSA-2026:36018](https://errata.almalinux.org/9/ALSA-2026-36018.html) - [Rocky Linux errata, RLSA-2026:36018](https://errata.rockylinux.org/RLSA-2026:36018) --- ## Januscape (CVE-2026-53359) import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; :::danger VM-to-host escape Januscape is a **VM-to-host escape** in the Linux kernel's KVM/x86 subsystem. A tenant who has **root inside their own VM** can corrupt the host kernel and, in the worst case, run code as root on the hypervisor, breaking isolation from every other VM on that host. Because VirtFusion hypervisors run tenant workloads on KVM, any affected hypervisor should be patched and rebooted as a priority. ::: ### Summary | Field | Detail | | --- | --- | | **CVE** | CVE-2026-53359 | | **Nickname** | Januscape (part of "The KVM Escape Trilogy", alongside ITScape and Zapscape) | | **Component** | Linux kernel, KVM/x86 shadow MMU | | **Class** | Use-after-free (UAF) | | **Impact** | VM-to-host escape. The public proof-of-concept panics the host; the researcher reports a private variant that achieves code execution as root on the host. | | **Severity** | Important | | **Age** | Introduced by commit `2032a93d66fa` (2010-08-01). Roughly 16 years of exposure. | | **Upstream fix** | Commit `81ccda30b4e8` (merged mainline 2026-06-16) | | **Architectures** | Both Intel and AMD. Unlike Zapscape, there is no additional Intel-only EPT condition. | | **Where it applies** | Hypervisors running KVM VMs. This does **not** affect the VirtFusion control server. | ### Am I affected? The vulnerability lives in the **host kernel** on your **hypervisors**. It is not a VirtFusion application flaw and does not affect the control server. You are potentially exposed if all of the following are true: - A hypervisor runs an **unpatched Linux kernel** that includes the vulnerable shadow-MMU code, which has been present since 2010. - VMs are able to use **nested virtualization**. - A VM is controlled by an **untrusted party with root inside the VM**, which is the normal situation in a multi-tenant hosting environment. Januscape is triggerable on both Intel and AMD hosts with no extra CPU condition, which makes it broader in reach than [Zapscape](zapscape.md). ### Technical details Januscape is a use-after-free in KVM's shadow MMU caused by a shadow-page role mismatch. When a page directory entry (PDE) that maps a large 2 MB page is changed to point to a non-leaf 4 KB page, KVM reuses an existing `kvm_mmu_page` with `direct=1` instead of creating one with `direct=0`. The resulting rmap entry survives memslot deletion, so KVM later dereferences shadow-page memory that has already been freed. VM-side actions are enough to drive this and corrupt the host kernel's shadow page state, which is what breaks the isolation boundary between a VM and its host. The bug was serious enough to be used as a working zero-day against Google's kvmCTF, earning the top $250,000 bounty tier reserved for a complete VM-to-host escape. The upstream fix (`81ccda30b4e8`) corrects the shadow-page handling so the stale entry can no longer be reused after it is freed. :::caution Patch the companion CVE too Fully closing the KVM shadow-paging hole requires both **CVE-2026-53359** and its companion **CVE-2026-46113**. Patching only one leaves shadow paging exposed. Confirm that **both** CVE IDs appear in your updated kernel package changelog, not just that the package version increased. ::: ### Patch status by distribution VirtFusion is supported on Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 9/10, and Rocky Linux 9/10. The table below tracks the fix for those distributions. :::caution This is a point-in-time snapshot The state below reflects **{`2026-08-24`}**. Distribution patch status changes as errata ship. **Always confirm against the vendor's live security tracker** (linked in [References](#references)) before deciding a host is safe. ::: | Distribution | Status | Fixed kernel / advisory | | --- | --- | --- | | **Debian 13 (trixie)** | Patched | `linux` **6.12.95-1** (DSA-6381-1) | | **Debian 12 (bookworm)** | Patched | `linux` **6.1.180-1** (DSA-6381-1) | | **Ubuntu 24.04 LTS (noble)** | Patched | `linux` **6.8.0-137.137** or newer | | **Ubuntu 22.04 LTS (jammy)** | Patched | `linux` **5.15.0-187.197** or newer | | **AlmaLinux 9** | Patched | `kernel-5.14.0-687.24.1.el9_8` or newer (ALSA-2026:36957) | | **AlmaLinux 10** | Patched | `kernel-6.12.0-211.32.1.el10_2` or newer (ALSA-2026:36956) | | **Rocky Linux 9** | Patched | `kernel-5.14.0-687.24.1.el9_8` or newer (RLSA-2026:36957) | | **Rocky Linux 10** | Patched | `kernel-6.12.0-211.32.1.el10_2` or newer (RLSA-2026:36956) | #### Debian - **Debian 13 (trixie)**: Fixed in `linux` **6.12.95-1** (DSA-6381-1). Update and reboot. - **Debian 12 (bookworm)**: Fixed in `linux` **6.1.180-1** (DSA-6381-1). Update and reboot. #### Ubuntu - **Ubuntu 24.04 LTS (noble)**: Fixed in `linux` **6.8.0-137.137** or newer. Update and reboot. - **Ubuntu 22.04 LTS (jammy)**: Fixed in `linux` **5.15.0-187.197** or newer. Update and reboot. Hardware-enablement (HWE) and some derivative kernels (`linux-aws`, `linux-azure`, `linux-gcp`, and so on) may lag the general-availability kernel, so confirm the fixed version for the exact flavour you run. #### AlmaLinux - **AlmaLinux 9**: Fixed in `kernel-5.14.0-687.24.1.el9_8` or newer (ALSA-2026:36957). Update and reboot. - **AlmaLinux 10**: Fixed in `kernel-6.12.0-211.32.1.el10_2` or newer (ALSA-2026:36956). Update and reboot. #### Rocky Linux - **Rocky Linux 9**: Fixed in `kernel-5.14.0-687.24.1.el9_8` or newer (RLSA-2026:36957). Update and reboot. - **Rocky Linux 10**: Fixed in `kernel-6.12.0-211.32.1.el10_2` or newer (RLSA-2026:36956). Update and reboot. Before treating an AlmaLinux or Rocky Linux host as fixed, confirm both this CVE and its companion CVE-2026-46113 (see the [companion-CVE note](#technical-details)) appear in the kernel changelog. ### Remediation The fix is a **host kernel update followed by a reboot** on every hypervisor. A kernel update does not take effect until the host is running the new kernel, so the reboot is required. **Debian/Ubuntu** ```shell apt-get update -y && apt-get install --only-upgrade linux-image-amd64 -y # Reboot into the new kernel once VMs are migrated/stopped: reboot ``` **AlmaLinux / Rocky Linux** ```shell dnf makecache -y && dnf update kernel -y # Reboot into the new kernel once VMs are migrated/stopped: reboot ``` After rebooting, confirm the host is running a fixed kernel: ```shell uname -r ``` Compare the output against the fixed version in the [table above](#patch-status-by-distribution), and check the changelog for both CVE-2026-53359 and CVE-2026-46113. #### Mitigations if you cannot patch yet If a fixed kernel is not yet available for a host, reduce exposure until you can update. **Disable nested virtualization** on hypervisors that do not need it. This removes the primary exploitation path. VirtFusion uses the stock distribution libvirt/KVM, and on current kernels the `nested` module parameter defaults to **enabled** for both `kvm_intel` and `kvm_amd`, so hosts are exposed by default. The `nested` parameter cannot be changed while `kvm_intel` / `kvm_amd` is loaded, and the module cannot be unloaded while any VM is running. The reliable sequence is therefore to set the value in the `modprobe.d` config, then **reboot** the host so the module reloads with `nested=0`. Nested can be enabled two ways: by an active `options ... nested=1` line in a `modprobe.d` file, or, when nothing sets it, by the kernel module's own default (which is `nested=1` on current kernels). Handle whichever applies. When you do set it in a file, correct the value at its source rather than adding a second override file: `modprobe` reads every `.conf` file in the `modprobe.d` directories in lexicographic filename order and the last value for a repeated parameter wins, and a plain name like `kvm.conf` sorts after any numeric-prefixed file, so a separate override is easily outranked. First, find any **active** (uncommented) line that sets `nested`. Commented example lines, such as those AlmaLinux ships in `/etc/modprobe.d/kvm.conf`, do not count: ```shell grep -RHn '^[^#]*nested' /etc/modprobe.d/ /run/modprobe.d/ /usr/lib/modprobe.d/ /lib/modprobe.d/ 2>/dev/null ``` If an active line sets `nested=1` (some VirtFusion hosts ship `/etc/modprobe.d/kvm.conf` this way), change its value to `0`: ```shell sed -i 's/nested=1/nested=0/' /etc/modprobe.d/kvm.conf ``` If the command returns nothing, nested is on by the kernel default and no file is setting it. Add an active line that disables it, matching your host's CPU vendor: **Intel** ```shell echo "options kvm_intel nested=0" >> /etc/modprobe.d/kvm.conf ``` **AMD** ```shell echo "options kvm_amd nested=0" >> /etc/modprobe.d/kvm.conf ``` The change only takes effect once the module reloads, which means a reboot: ```shell reboot ``` After the host is back up, confirm nested virtualization is off. This sysfs value is the effective setting regardless of what produced it, so it is the definitive check. `0` means disabled: **Intel** ```shell cat /sys/module/kvm_intel/parameters/nested ``` **AMD** ```shell cat /sys/module/kvm_amd/parameters/nested ``` If it still reads `1` after a reboot, the value is being forced on the kernel boot command line, which the `modprobe.d` config cannot override. Remove any `kvm_intel.nested=1` / `kvm_amd.nested=1` from your bootloader config (`/etc/default/grub`), regenerate the config, and reboot. :::note Disabling nested virtualization requires a host reboot, which is also what applies a kernel update. If a fixed kernel has already shipped for your distribution, it is simpler to patch and reboot once (see [Remediation](#remediation)) instead. Turning off nested is intended for hosts where no fixed kernel is available yet. It lowers the likelihood of exploitation but does **not** remove the vulnerability. ::: ### References - [Januscape write-up and PoC (V4bel/Januscape)](https://github.com/V4bel/Januscape) - [NVD, CVE-2026-53359](https://nvd.nist.gov/vuln/detail/CVE-2026-53359) - [Debian security tracker, CVE-2026-53359](https://security-tracker.debian.org/tracker/CVE-2026-53359) - [Ubuntu security, CVE-2026-53359](https://ubuntu.com/security/CVE-2026-53359) - [AlmaLinux errata, ALSA-2026:36957](https://errata.almalinux.org/9/ALSA-2026-36957.html) - [Rocky Linux errata, RLSA-2026:36957](https://errata.rockylinux.org/RLSA-2026:36957) --- ## Zapscape (CVE-2026-64561) import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; :::danger VM-to-host escape Zapscape is a **VM-to-host escape** in the Linux kernel's KVM/x86 subsystem. A tenant who has **root inside their own VM** can, under the right conditions, gain **root on the hypervisor** and therefore over every other VM on that host. Because VirtFusion hypervisors run tenant workloads on KVM, any affected hypervisor should be patched and rebooted as a priority. ::: ### Summary | Field | Detail | | --- | --- | | **CVE** | CVE-2026-64561 | | **Nickname** | Zapscape (part of "The KVM Escape Trilogy", alongside ITScape and Januscape) | | **Component** | Linux kernel, KVM/x86 shadow MMU | | **Class** | Use-after-free (UAF) | | **Impact** | VM-to-host escape, leading to arbitrary code execution as root on the host | | **Severity** | Important. CVSS 3.1 base score 7.0 | | **Upstream fix** | Commit `2abd5287f083` (merged mainline 2026-07-21) | | **Where it applies** | Hypervisors running KVM VMs. This does **not** affect the VirtFusion control server. | ### Am I affected? The vulnerability lives in the **host kernel** on your **hypervisors**. It is not a VirtFusion application flaw and does not affect the control server. You are potentially exposed if all of the following are true: - A hypervisor runs an **unpatched Linux kernel** that includes the vulnerable shadow-MMU code (introduced in the 5.9 era via commit `f95eec9bed76`). - VMs are able to use **nested virtualization** (nested VMX/EPT on Intel, or nested SVM/NPT on AMD). - The VM is controlled by an **untrusted party with root inside the VM**, which is the normal situation in a multi-tenant hosting environment. On **Intel** hosts an extra condition applies: exploitation requires both 4-level and 5-level EPT page-walk support to be exposed to the L1 VM. **AMD** hosts have no such extra condition. ### Technical details The bug is a use-after-free in KVM's shadow MMU, on the recursive "zap" path used during shadow-page reclamation. Two page-fault paths validate the shadow root **before** reclaiming shadow pages instead of **after**. When a single `kvm_mmu_page` is aliased as both a **child** of one nested page table and the **pinned root** of another, quota-driven reclamation can recursively prepare (and free) that page through its parent without the `root_count` protection applying. KVM then keeps building mappings beneath a root that reclamation has already invalidated, leaving dangling pointers that allow writes to freed memory. From there the public proof-of-concept chains a KASLR leak and a usermode-helper callback to run arbitrary code as root on the host. The upstream fix (`2abd5287f083`) moves the stale-root check to **after** `make_mmu_pages_available()`, so KVM restarts the fault if the root became invalid during reclamation rather than continuing to use freed structures. ### Patch status by distribution VirtFusion is supported on Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 9/10, and Rocky Linux 9/10. The table below tracks the fix for those distributions. :::caution This is a point-in-time snapshot The state below reflects **{`2026-08-24`}**. Distribution patch status changes daily as errata ship. **Always confirm against the vendor's live security tracker** (linked in [References](#references)) before deciding a host is safe. ::: | Distribution | Status | Fixed kernel / advisory | | --- | --- | --- | | **Debian 13 (trixie)** | Patched | `linux` **6.12.101-1** (DSA-6415-1) | | **Debian 12 (bookworm)** | Partially fixed | Default `linux` (6.1) still vulnerable; `linux-6.12` backport fixed in **6.12.101-1~deb12u1** (DLA-4745-1) | | **Ubuntu 24.04 LTS (noble)** | Not yet fixed | Patch prepared, in validation | | **Ubuntu 22.04 LTS (jammy)** | Not yet fixed | Patch prepared, in validation | | **AlmaLinux 9** | Patched | `kernel-5.14.0-687.30.1.el9_8` or newer (ALSA-2026:45192) | | **AlmaLinux 10** | Patched | `kernel-6.12.0-211.39.1.el10_2` or newer (ALSA-2026:45114) | | **Rocky Linux 9** | Patched | `kernel-5.14.0-687.30.1.el9_8` or newer (RLSA-2026:45192) | | **Rocky Linux 10** | Patched | `kernel-6.12.0-211.39.1.el10_2` or newer (RLSA-2026:45114) | #### Debian - **Debian 13 (trixie)**: Fixed in `linux` **6.12.101-1** (DSA-6415-1). Update and reboot. - **Debian 12 (bookworm)**: The default `linux` kernel (6.1.x) is still vulnerable at the time of writing. The newer `linux-6.12` backport kernel is fixed in **6.12.101-1~deb12u1** (DLA-4745-1), so a bookworm host can either move to that 6.12 kernel or apply the [interim mitigations](#mitigations-if-you-cannot-patch-yet) until the 6.1 line is patched. #### Ubuntu - **Ubuntu 24.04 LTS (noble)** and **22.04 LTS (jammy)**: No fixed `linux` package released yet at the time of writing; patches were prepared and in validation. This covers the cloud and derivative kernels too (`linux-kvm`, `linux-aws`, `linux-azure`, `linux-gcp`, `linux-oracle`, FIPS variants, and so on). Apply the [interim mitigations](#mitigations-if-you-cannot-patch-yet) and update once the USN is published. #### AlmaLinux - **AlmaLinux 9**: Fixed in `kernel-5.14.0-687.30.1.el9_8` or newer (ALSA-2026:45192). Update and reboot. - **AlmaLinux 10**: Fixed in `kernel-6.12.0-211.39.1.el10_2` or newer (ALSA-2026:45114). Update and reboot. #### Rocky Linux - **Rocky Linux 9**: Fixed in `kernel-5.14.0-687.30.1.el9_8` or newer (RLSA-2026:45192). Update and reboot. - **Rocky Linux 10**: Fixed in `kernel-6.12.0-211.39.1.el10_2` or newer (RLSA-2026:45114). Update and reboot. ### Remediation The fix is a **host kernel update followed by a reboot** on every hypervisor. A kernel update does not take effect until the host is running the new kernel, so the reboot is required. **Debian/Ubuntu** ```shell apt-get update -y && apt-get install --only-upgrade linux-image-amd64 -y # Reboot into the new kernel once VMs are migrated/stopped: reboot ``` **AlmaLinux / Rocky Linux** ```shell dnf makecache -y && dnf update kernel -y # Reboot into the new kernel once VMs are migrated/stopped: reboot ``` After rebooting, confirm the host is running a fixed kernel: ```shell uname -r ``` Compare the output against the fixed version in the [table above](#patch-status-by-distribution). #### Mitigations if you cannot patch yet If a fixed kernel is not yet available for a host, reduce exposure until you can update. **Disable nested virtualization** on hypervisors that do not need it. This removes the primary exploitation path. VirtFusion uses the stock distribution libvirt/KVM, and on current kernels the `nested` module parameter defaults to **enabled** for both `kvm_intel` and `kvm_amd`, so hosts are exposed by default. The `nested` parameter cannot be changed while `kvm_intel` / `kvm_amd` is loaded, and the module cannot be unloaded while any VM is running. The reliable sequence is therefore to set the value in the `modprobe.d` config, then **reboot** the host so the module reloads with `nested=0`. Nested can be enabled two ways: by an active `options ... nested=1` line in a `modprobe.d` file, or, when nothing sets it, by the kernel module's own default (which is `nested=1` on current kernels). Handle whichever applies. When you do set it in a file, correct the value at its source rather than adding a second override file: `modprobe` reads every `.conf` file in the `modprobe.d` directories in lexicographic filename order and the last value for a repeated parameter wins, and a plain name like `kvm.conf` sorts after any numeric-prefixed file, so a separate override is easily outranked. First, find any **active** (uncommented) line that sets `nested`. Commented example lines, such as those AlmaLinux ships in `/etc/modprobe.d/kvm.conf`, do not count: ```shell grep -RHn '^[^#]*nested' /etc/modprobe.d/ /run/modprobe.d/ /usr/lib/modprobe.d/ /lib/modprobe.d/ 2>/dev/null ``` If an active line sets `nested=1` (some VirtFusion hosts ship `/etc/modprobe.d/kvm.conf` this way), change its value to `0`: ```shell sed -i 's/nested=1/nested=0/' /etc/modprobe.d/kvm.conf ``` If the command returns nothing, nested is on by the kernel default and no file is setting it. Add an active line that disables it, matching your host's CPU vendor: **Intel** ```shell echo "options kvm_intel nested=0" >> /etc/modprobe.d/kvm.conf ``` **AMD** ```shell echo "options kvm_amd nested=0" >> /etc/modprobe.d/kvm.conf ``` The change only takes effect once the module reloads, which means a reboot: ```shell reboot ``` After the host is back up, confirm nested virtualization is off. This sysfs value is the effective setting regardless of what produced it, so it is the definitive check. `0` means disabled: **Intel** ```shell cat /sys/module/kvm_intel/parameters/nested ``` **AMD** ```shell cat /sys/module/kvm_amd/parameters/nested ``` If it still reads `1` after a reboot, the value is being forced on the kernel boot command line, which the `modprobe.d` config cannot override. Remove any `kvm_intel.nested=1` / `kvm_amd.nested=1` from your bootloader config (`/etc/default/grub`), regenerate the config, and reboot. :::note Disabling nested virtualization requires a host reboot, which is also what applies a kernel update. If a fixed kernel has already shipped for your distribution, it is simpler to patch and reboot once (see [Remediation](#remediation)) instead. Turning off nested is intended for hosts where no fixed kernel is available yet. It lowers the likelihood of exploitation but does **not** remove the vulnerability. ::: ### References - [Zapscape write-up and PoC (V4bel/Zapscape)](https://github.com/V4bel/Zapscape) - [NVD, CVE-2026-64561](https://nvd.nist.gov/vuln/detail/CVE-2026-64561) - [Debian security tracker, CVE-2026-64561](https://security-tracker.debian.org/tracker/CVE-2026-64561) - [Ubuntu security, CVE-2026-64561](https://ubuntu.com/security/CVE-2026-64561) - [AlmaLinux errata, ALSA-2026:45192](https://errata.almalinux.org/9/ALSA-2026-45192.html) - [Rocky Linux errata, RLSA-2026:45192](https://errata.rockylinux.org/RLSA-2026:45192) --- ## Support Access ## Support Access Key If the VirtFusion team require access to your servers, you may use the following commands to install, check and uninstall our SSH access key. ### Adding the support key ```shell wget -qO- https://security.virtfusion.net/vf_support_key.sh | bash -s -- add ``` ### Removing the support key ```shell wget -qO- https://security.virtfusion.net/vf_support_key.sh | bash -s -- remove ``` ### Check if the support key is installed ```shell wget -qO- https://security.virtfusion.net/vf_support_key.sh | bash -s -- check ``` --- ## Requesting Support import BrowserWindow from '@site/src/components/BrowserWindow'; ### Importance of Detailed Issue Reporting **1. Accurate Problem Diagnosis** - Providing a detailed description of the issue helps our support team accurately diagnose the problem. The more information you provide, the easier it is for us to understand what’s going wrong and identify the root cause. **2. Efficient Resolution** - When we have a clear understanding of the problem, we can resolve it more efficiently. Detailed reports enable us to skip over unnecessary troubleshooting steps and go straight to the solution, saving you time and minimizing disruption. #### What to Include in Your Issue Report **1. Detailed Description** - Explain the issue in as much detail as possible. - Include information about what you were doing when the problem occurred, any error messages received, and how frequently the issue occurs. **2. Screenshots** - Provide screenshots that capture the error messages, unexpected behaviors, or any relevant screens. - Screenshots give us a visual context, which can often reveal issues that might not be evident from a text description alone. **3. Diagnostic Outputs** - Run any diagnostic tools and include the output logs. - These outputs should contain technical details about your system and the issue, which are crucial for a thorough analysis. #### Example of a Detailed Issue Report **Description:** "I encountered an issue while trying to start a new virtual machine. The VM fails to boot and displays an error message saying, 'Failed to initialize network interface.' This started happening after the last software update. It happens every time I try to start a new VM." **Screenshots:** Screenshot of the error message on the VM startup screen. Screenshot of the network configuration settings in VirtFusion. **Diagnostic Output:** Attached log files from the diagnostic tools (ex log file name: diagnostic_log_20240704.txt). :::info Helpful Hint If our support team requests additional information, please provide it as soon as possible to ensure a swift resolution. By thoroughly explaining the issue and providing concrete examples, you help us help you more effectively. Detailed reports lead to quicker, more accurate solutions, ensuring your VirtFusion experience is as smooth as possible. ::: --- ## Support Access(2) ## Support Access Key If the VirtFusion team require access to your servers, you may use the following commands to install, check and uninstall our SSH access key. ### Adding the support key ```shell wget -qO- https://security.virtfusion.net/vf_support_key.sh | bash -s -- add ``` ### Removing the support key ```shell wget -qO- https://security.virtfusion.net/vf_support_key.sh | bash -s -- remove ``` ### Check if the support key is installed ```shell wget -qO- https://security.virtfusion.net/vf_support_key.sh | bash -s -- check ``` --- ## PHP Compatibility ### Hypervisor Specific From time to time we release new versions of PHP for your hypervisor. This allows us to keep VirtFusion up to date. If you don't update your hypervisor level software often enough (DNF/APT) you may run into issues with VirtFusion. To update to the latest version of PHP you can run the following commands. :::caution Important Running the following commands may disrupt the connection between the control server and hypervisor. Make sure no server tasks are running. Ideally put the control server into maintenance mode. ::: #### Debian/Ubuntu ```shell apt update apt install vf-php8 vf-system-php8 systemctl restart vf-php8-fpm systemctl restart vf-system-php8-fpm ``` #### RHEL ```shell dnf clean metadata dnf update vf-php8 vf-system-php8 systemctl restart vf-php8-fpm systemctl restart vf-system-php8-fpm ``` Your hypervisor should now be on the latest PHP version. #### (Re) Installing the VirtFusion Package Repository :::caution Important **You don't need to take this step unless you are still having issues with PHP.** **You also don't need to take this step if you are not using RHEL 8 or Debian 11** ::: Very early versions of VirtFusion used a different package repository and your installation may still be using it. You can add our newer repository using the following commands. ##### RHEL 8 Only ```shell dnf config-manager --add-repo https://packages.virtfusion.net/rhel/virtfusion.repo dnf clean metadata dnf update vf-php8 vf-system-php8 systemctl restart vf-php8-fpm systemctl restart vf-system-php8-fpm ``` ##### Debian 11 Only ```shell echo "deb http://packages.virtfusion.net/debian bullseye main" > /etc/apt/sources.list.d/virtfusion.list curl -s https://security.virtfusion.net/gpg.asc | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/virtfusion.gpg --import chmod 644 /etc/apt/trusted.gpg.d/virtfusion.gpg apt update apt remove vf-php8 vf-system-php8 apt install vf-php8 vf-system-php8 systemctl restart vf-php8-fpm systemctl restart vf-system-php8-fpm ``` :::info Still having issues? If you are still having issues with PHP. You can [submit a support ticket](https://account.virtfusion.com/plugin/support_manager/client_tickets/departments/) for further assistance. ::: --- ## Priority XML Found ```shell export VFUUID=UUID ``` 0c2c37b3-6bf4-47d8-9238-2549d23ac09f ```shell export VFUUID=0c2c37b3-6bf4-47d8-9238-2549d23ac09f ``` ```shell virsh domblklist $VFUUID ``` ```shell Target Source ------------------------------------------------------------------------------------- vda /home/vf-data/tmp/0c2c37b3-6bf4-47d8-9238-2549d23ac09f_1.img_dr-overlay sdy - sdx /home/vf-data/server/0c2c37b3-6bf4-47d8-9238-2549d23ac09f/cloud-drive.img ``` ```shell sed -i '/deny "\/home\/vf-data\/disk\/'"${VFUUID}"'_1.img" w,/g' '/etc/apparmor.d/libvirt/libvirt-'"${VFUUID}"'.files' virsh blockcommit ${VFUUID} vda --active --verbose --pivot ``` ```shell rm -f /home/vf-data/tmp/${VFUUID}_1.img_dr-overlay rm -f /home/vf-data/server/${VFUUID}/server-priority.xml ``` --- ## Restarting Services You can restart some of the system services from the commandline. Commands may differ between a control server and a hypervisor. ### Control Server To restart the basic services on the control server (fpm,web,queue). You can run the following command. ```shell vfcli-ctrl app:restart ``` If you would like to restart specific services, you can specify the required services using the `--services` argument. The following services are available. | Flag | Description | |------------|---------------------------------------| | fpm | Restart PHP FPM. | | web | Restart the webserver. | | wss | Restart the websocket server. | | vnc | Restart the VNC proxy. | | queue | Gracefully restart the queue workers. | | queue-hard | Hard restart the queue workers. | For example, to restart the queue and websocket server. ```shell vfcli-ctrl app:restart --services=queue,wss ``` ### Hypervisor :::caution Caution Restarting services on a hypervisor may cause tasks to fail. It's advised to put the control server into maintenance mode before running any restart commands. ::: To restart the basic services on the hypervisor (fpm,system-fpm,web,queue). You can run the following command. ```shell vfcli-hv app:restart ``` If you would like to restart specific services, you can specify the required services using the `--services` argument. The following services are available. | Flag | Description | |------------|---------------------------------------| | fpm | Restart PHP FPM. | | system-fpm | Restart system PHP FPM. | | web | Restart the webserver. | | queue | Gracefully restart the queue workers. | | queue-hard | Hard restart the queue workers. | For example, to restart system-fpm and webserver. ```shell vfcli-hv app:restart --services=system-fpm,web ``` --- ## SSL ### Removing Old & Generating New **Make sure the acme client is up to date** ```shell acme.sh --upgrade ``` :::info replace DOMAIN.COM with your own domain. ::: **Remove the old certificate** ```shell acme.sh --remove -d DOMAIN.COM ``` **Issue a new certificate** ```shell acme.sh --issue \ --webroot /opt/virtfusion/nginx/acme \ --keylength 4096 \ --server letsencrypt \ -d DOMAIN.COM ``` ```shell acme.sh --installcert \ --keypath /opt/virtfusion/ssl/ssl.key \ --fullchainpath /opt/virtfusion/ssl/ssl.crt \ --reloadcmd "systemctl reload vf-nginx" \ -d DOMAIN.COM ```