Repository navigation
Description
Description
This bug occurs when executing an incremental backup job with a Restic datastore. I use the KEEP_LAST variable in the backup job to define how many increments I want to keep. If the increments the backup job has made is equal to KEEP_LAST and I execute the backup job once again, I get an error that the Restic repository is locked by another PID. The restic.rb file from Opennebula that handles this, should have error handling for it, but the Error handling doesn't work with recent Restic versions.
When sticking to version 0.17.3 that's mentioned as supported in the platform notes, it also doesn't detect an exclusive lock correctly and throws the following error:
Wed Nov 26 16:16:51 2025 : Error flattening backup increments: Command failed:export LANG=Cexport LC_ALL=Cset -e -o pipefail; shopt -qs failglobexport RESTIC_BIN="/var/tmp/one/datastore/restic/restic"export RESTIC_PASSWORD='REDACTED'export GOMAXPROCS="1""$RESTIC_BIN" '--prune' '--max-unused=0' '--repo=/var/lib/one/datastores/108/job_5' forget 9714ce12 0d2b4990/var/lib/one/remotes/datastore/restic/restic.rb:308:in `remove_snapshots': Unable to remove snapshots: repo already locked, waiting up to 0s for the lock (StandardError)unable to create lock in backend: repository is already locked by PID 13485 on backup1 by oneadmin (UID 1005, GID 1005)lock was created at 2025-11-26 16:16:51 (268.316798ms ago)storage ID dd0ce970the `unlock` command can be used to remove stale locksfrom /var/lib/one/remotes/datastore/restic/increment_flatten:163:in `<main>'
This error leads us to the method def remove_snapshots on line 269 in /var/lib/one/remotes/datastore/restic/restic.rb. This method removes snapshots when the amount of incremental backups goes over the value of KEEP_LAST. This method has functionality to retry removing old snapshots when the repository is locked:
options[:retries].to_i.times do
# NOTE: Running this operation on the Restic server (@sftp) can
# improve performance significantly.
rc = run_action 'remove_snapshots', script, rhost
return if rc.code == 0
# https://github.com/restic/restic/issues/2492#issuecomment-1126910866
break if rc.stderr.match(/unable to create lock.+locked exclusively/).nil?
sleep options[:delay].to_i
endThe problem is that Restic doesn't return this message when the repository is exclusively locked. With Restic 0.16.5, 0.17.3, and 0.18.0 we get the following message returned from Restic:
unable to create lock in backend: repository is already locked by PID 13485 on backup1 by oneadmin
As one can see, the "locked exclusively" part isn't returned anymore. That's why it breaks on the match and doesn't get to the sleep below it.
On top of that, Restic has new behaviour since version 0.17.0 that returns an error code 11 when a repository is exclusively locked. Because of this, we've tested a fix in our Opennebula test cluster that executes the sleep command when error code 11 is returned:
options[:retries].to_i.times do
# NOTE: Running this operation on the Restic server (@sftp) can
# improve performance significantly.
rc = run_action 'remove_snapshots', script, rhost
return if rc.code == 0
# https://github.com/restic/restic/issues/2492#issuecomment-1126910866
# break if rc.stderr.match(/unable to create lock.+locked exclusively/).nil?
# sleep options[:delay].to_i
if rc.code == 11
sleep options[:delay].to_i
else
break
end
endThis works with Restic 0.17.0 and higher, because of the aforementioned change.
To Reproduce
- Install and configure a separate backup server with Debian 12 or 13
- Install Restic 0.16.5, 0.17.3 or 0.18.0 on the server (Restic 0.17.3 is supported by Opennebula: https://docs.opennebula.io/7.0/software/release_information/release_notes_70/platform_notes/#monitoring-and-backups
- Configure a Restic datastore according to: https://docs.opennebula.io/7.0/product/cluster_configuration/backup_system/restic/
- Configure a backup job that's incremental and has
KEEP_LASTvalue of at least 2 - Execute the backup job and let it go over the
KEEP_LASTvalue to trigger flattening the snapshots
Expected behavior
Removal of the oldest snapshot without an error when the repository is still exclusively locked by another process.
Details
- Affected Component: Sunstone - Incremental Backup Job with a limit on the amount of snapshots kept with the Restic repository
- Hypervisor: KVM
- Version: Opennebula Sunstone 7.0.1 tested with Restic 0.16.5, 0.17.3, and 0.18.0
Additional context
Add any other context about the problem here.
Progress Status
- Code committed
- Testing - QA
- Documentation (Release notes - resolved issues, compatibility, known issues)
Activity
[-]Restic exclusive repository locks handling doens't work in method remove_snapshots in restic.rb[/-][+]Restic exclusive repository locks handling doesn't work in method remove_snapshots in restic.rb[/+]B #7403: Fix Restic exclusive lock handling (#4211)
B #7403: Fix Restic exclusive lock handling (#4211)
B OpenNebula/one#7403: Fix Restic exclusive lock handling
B OpenNebula/one#7403: Add to resolved issues
rsmontero commentedon May 13, 2026
Thanks @ScalableSec , fixed as proposed!
B OpenNebula/one#7404: update resolved issues (#656)
B #7404: Continue #7403 => pull_disks (#4217)
B #7404: Continue #7403 => pull_disks (#4217)