Skip to content

Restic exclusive repository locks handling doesn't work in method remove_snapshots in restic.rb #7403

Closed

Description

@ScalableSec

Description
This bug occurs when executing an incremental backup job with a Restic datastore. I use the KEEP_LAST variable in the backup job to define how many increments I want to keep. If the increments the backup job has made is equal to KEEP_LAST and I execute the backup job once again, I get an error that the Restic repository is locked by another PID. The restic.rb file from Opennebula that handles this, should have error handling for it, but the Error handling doesn't work with recent Restic versions.

When sticking to version 0.17.3 that's mentioned as supported in the platform notes, it also doesn't detect an exclusive lock correctly and throws the following error:

Wed Nov 26 16:16:51 2025 : Error flattening backup increments: Command failed:export LANG=Cexport LC_ALL=Cset -e -o pipefail; shopt -qs failglobexport RESTIC_BIN="/var/tmp/one/datastore/restic/restic"export RESTIC_PASSWORD='REDACTED'export GOMAXPROCS="1""$RESTIC_BIN" '--prune' '--max-unused=0' '--repo=/var/lib/one/datastores/108/job_5' forget 9714ce12 0d2b4990/var/lib/one/remotes/datastore/restic/restic.rb:308:in `remove_snapshots': Unable to remove snapshots: repo already locked, waiting up to 0s for the lock (StandardError)unable to create lock in backend: repository is already locked by PID 13485 on backup1 by oneadmin (UID 1005, GID 1005)lock was created at 2025-11-26 16:16:51 (268.316798ms ago)storage ID dd0ce970the `unlock` command can be used to remove stale locksfrom /var/lib/one/remotes/datastore/restic/increment_flatten:163:in `<main>'

This error leads us to the method def remove_snapshots on line 269 in /var/lib/one/remotes/datastore/restic/restic.rb. This method removes snapshots when the amount of incremental backups goes over the value of KEEP_LAST. This method has functionality to retry removing old snapshots when the repository is locked:

        options[:retries].to_i.times do
            # NOTE: Running this operation on the Restic server (@sftp) can
            # improve performance significantly.
            rc = run_action 'remove_snapshots', script, rhost

            return if rc.code == 0

            # https://github.com/restic/restic/issues/2492#issuecomment-1126910866

            break if rc.stderr.match(/unable to create lock.+locked exclusively/).nil?

            sleep options[:delay].to_i
        end

The problem is that Restic doesn't return this message when the repository is exclusively locked. With Restic 0.16.5, 0.17.3, and 0.18.0 we get the following message returned from Restic:

unable to create lock in backend: repository is already locked by PID 13485 on backup1 by oneadmin

As one can see, the "locked exclusively" part isn't returned anymore. That's why it breaks on the match and doesn't get to the sleep below it.

On top of that, Restic has new behaviour since version 0.17.0 that returns an error code 11 when a repository is exclusively locked. Because of this, we've tested a fix in our Opennebula test cluster that executes the sleep command when error code 11 is returned:

        options[:retries].to_i.times do
            # NOTE: Running this operation on the Restic server (@sftp) can
            # improve performance significantly.
            rc = run_action 'remove_snapshots', script, rhost

            return if rc.code == 0

            # https://github.com/restic/restic/issues/2492#issuecomment-1126910866

            # break if rc.stderr.match(/unable to create lock.+locked exclusively/).nil?

            # sleep options[:delay].to_i

            if rc.code == 11
                sleep options[:delay].to_i
            else
                break
            end
        end

This works with Restic 0.17.0 and higher, because of the aforementioned change.

To Reproduce

  1. Install and configure a separate backup server with Debian 12 or 13
  2. Install Restic 0.16.5, 0.17.3 or 0.18.0 on the server (Restic 0.17.3 is supported by Opennebula: https://docs.opennebula.io/7.0/software/release_information/release_notes_70/platform_notes/#monitoring-and-backups
  3. Configure a Restic datastore according to: https://docs.opennebula.io/7.0/product/cluster_configuration/backup_system/restic/
  4. Configure a backup job that's incremental and has KEEP_LAST value of at least 2
  5. Execute the backup job and let it go over the KEEP_LAST value to trigger flattening the snapshots

Expected behavior
Removal of the oldest snapshot without an error when the repository is still exclusively locked by another process.

Details

  • Affected Component: Sunstone - Incremental Backup Job with a limit on the amount of snapshots kept with the Restic repository
  • Hypervisor: KVM
  • Version: Opennebula Sunstone 7.0.1 tested with Restic 0.16.5, 0.17.3, and 0.18.0

Additional context
Add any other context about the problem here.

Progress Status

  • Code committed
  • Testing - QA
  • Documentation (Release notes - resolved issues, compatibility, known issues)

Activity

  1. added theissue type on Dec 9, 2025
  2. changed the title [-]Restic exclusive repository locks handling doens't work in method remove_snapshots in restic.rb[/-] [+]Restic exclusive repository locks handling doesn't work in method remove_snapshots in restic.rb[/+] on Dec 9, 2025
  3. added this to the Release 7.2.1 milestone on Apr 9, 2026
  4. self-assigned this
    on May 13, 2026
  5. added 2 commits that reference this issue on May 13, 2026
  6. added a commit that references this issue on May 13, 2026
  7. added a commit that references this issue on May 13, 2026
  8. rsmontero commented on May 13, 2026

    @rsmontero
    Member

    Thanks @ScalableSec , fixed as proposed!

  9. added 3 commits that reference this issue on May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions