Repository navigation
Description
Description
This bug occurs when executing an incremental backup job with a Restic datastore. I use the KEEP_LAST variable in the backup job to define how many increments I want to keep. If the increments the backup job has made is equal to KEEP_LAST and I execute the backup job once again, I get an error that the Restic repository is locked by another PID. During this process Opennebula also uses the method def pull_disks in /var/lib/one/remotes/datastore/restic/restic.rb. This method doesn't have any handling for exclusive locks, while there can still be exclusive locks caused by other methods.
After we had fixed bug #7403 in our test environment, we ran into this bug with the following error message:
Mon Dec 8 21:39:53 2025 : Error flattening backup increments: Command failed:export LANG=Cexport LC_ALL=Cset -e -o pipefail; shopt -qs failglobexport RESTIC_BIN="/var/tmp/one/datastore/restic/restic"export RESTIC_PASSWORD='REDACTED'install -d '/var/lib/one/datastores/104/87/backup/'"$RESTIC_BIN" '--quiet' '--repo=/var/lib/one/datastores/111/job_10' dump '837b4b54' '/var/lib/one/datastores/104/87/backup/disk.0.4' > '/var/lib/one/datastores/104/87/backup/disk.0.4'install -d '/var/lib/one/datastores/104/87/backup/'"$RESTIC_BIN" '--quiet' '--repo=/var/lib/one/datastores/111/job_10' dump 'a14c747d' '/var/lib/one/datastores/104/87/backup/disk.0.5' > '/var/lib/one/datastores/104/87/backup/disk.0.5'/var/lib/one/remotes/datastore/restic/restic.rb:440:in `pull_disks': Unable to pull disks: unable to create lock in backend: repository is already locked exclusively by PID 2075 on backup1 by oneadmin (UID 1005, GID 1005) (StandardError)lock was created at 2025-12-08 21:39:52 (349.599036ms ago)storage ID 16ca7d33the `unlock` command can be used to remove stale locksfrom /var/lib/one/remotes/datastore/restic/restic.rb:256:in `pull_snapshots'from /var/lib/one/remotes/datastore/restic/increment_flatten:154:in `<main>'
To fix this problem with Restic 0.17.0 and up and it's updated behaviour on error handling of an exclusively locked repository, we've changed the method def pull_disks to include handling of a locked repository like in def remove_snapshots:
def pull_disks(disks, rhost, wdir = nil)
options = {
:retries => 60,
:delay => 15 # seconds
}
script = [<<~EOS]
set -e -o pipefail; shopt -qs failglob
#{resticenv_sh}
EOS
disks[:by_snap].each do |snap, disk_paths|
disk_paths.each do |path|
wdir = File.dirname(path) if wdir.nil?
file = File.basename(path)
rcmd = restic("dump '#{snap}' '#{path}'", 'quiet' => nil)
script << "install -d '#{wdir}/'"
script << "#{rcmd} > '#{wdir}/#{file}'"
end
end
options[:retries].to_i.times do
rc = run_action 'pull_disks', script.join("\n"), rhost
return if rc.code == 0
if rc.code == 11
sleep options[:delay].to_i
else
break
end
end
raise StandardError, "Unable to pull disks: #{rc.stderr}" if rc.code != 0
endTo Reproduce
- Install and configure a separate backup server with Debian 12 or 13
- Install Restic 0.16.5, 0.17.3 or 0.18.0 on the server (Restic 0.17.3 is supported by Opennebula: https://docs.opennebula.io/7.0/software/release_information/release_notes_70/platform_notes/#monitoring-and-backups
- Configure a Restic datastore according to: https://docs.opennebula.io/7.0/product/cluster_configuration/backup_system/restic/
- Configure a backup job that's incremental and has
KEEP_LASTvalue of at least 2 - Execute the backup job and let it go over the
KEEP_LASTvalue to trigger flattening the snapshots
Expected behavior
Removal of the oldest snapshot without an error when the repository is still exclusively locked by another process.
Details
- Affected Component: Sunstone - Incremental Backup Job with a limit on the amount of snapshots kept with the Restic repository
- Hypervisor: KVM
- Version: Opennebula Sunstone 7.0.1 tested with Restic 0.16.5, 0.17.3, and 0.18.0
Additional context
Add any other context about the problem here.
Progress Status
- Code committed
- Testing - QA
- Documentation (Release notes - resolved issues, compatibility, known issues)
Activity
B OpenNebula/one#7404: update resolved issues (#656)
B #7404: Continue #7403 => pull_disks (#4217)
B #7404: Coerce Restic retry options
B #7404: Continue #7403 => pull_disks (#4217)
B #7404: Coerce Restic retry options