Skip to content

Restic exclusive repository locks aren't handled in method pull_disks in restic.rb #7404

Closed

Description

@ScalableSec

Description
This bug occurs when executing an incremental backup job with a Restic datastore. I use the KEEP_LAST variable in the backup job to define how many increments I want to keep. If the increments the backup job has made is equal to KEEP_LAST and I execute the backup job once again, I get an error that the Restic repository is locked by another PID. During this process Opennebula also uses the method def pull_disks in /var/lib/one/remotes/datastore/restic/restic.rb. This method doesn't have any handling for exclusive locks, while there can still be exclusive locks caused by other methods.

After we had fixed bug #7403 in our test environment, we ran into this bug with the following error message:

Mon Dec 8 21:39:53 2025 : Error flattening backup increments: Command failed:export LANG=Cexport LC_ALL=Cset -e -o pipefail; shopt -qs failglobexport RESTIC_BIN="/var/tmp/one/datastore/restic/restic"export RESTIC_PASSWORD='REDACTED'install -d '/var/lib/one/datastores/104/87/backup/'"$RESTIC_BIN" '--quiet' '--repo=/var/lib/one/datastores/111/job_10' dump '837b4b54' '/var/lib/one/datastores/104/87/backup/disk.0.4' > '/var/lib/one/datastores/104/87/backup/disk.0.4'install -d '/var/lib/one/datastores/104/87/backup/'"$RESTIC_BIN" '--quiet' '--repo=/var/lib/one/datastores/111/job_10' dump 'a14c747d' '/var/lib/one/datastores/104/87/backup/disk.0.5' > '/var/lib/one/datastores/104/87/backup/disk.0.5'/var/lib/one/remotes/datastore/restic/restic.rb:440:in `pull_disks': Unable to pull disks: unable to create lock in backend: repository is already locked exclusively by PID 2075 on backup1 by oneadmin (UID 1005, GID 1005) (StandardError)lock was created at 2025-12-08 21:39:52 (349.599036ms ago)storage ID 16ca7d33the `unlock` command can be used to remove stale locksfrom /var/lib/one/remotes/datastore/restic/restic.rb:256:in `pull_snapshots'from /var/lib/one/remotes/datastore/restic/increment_flatten:154:in `<main>'

To fix this problem with Restic 0.17.0 and up and it's updated behaviour on error handling of an exclusively locked repository, we've changed the method def pull_disks to include handling of a locked repository like in def remove_snapshots:

def pull_disks(disks, rhost, wdir = nil)
        options = {
            :retries => 60,
            :delay   => 15 # seconds
        }

        script = [<<~EOS]
            set -e -o pipefail; shopt -qs failglob
            #{resticenv_sh}
        EOS

        disks[:by_snap].each do |snap, disk_paths|
            disk_paths.each do |path|
                wdir = File.dirname(path) if wdir.nil?
                file = File.basename(path)

                rcmd = restic("dump '#{snap}' '#{path}'", 'quiet' => nil)

                script << "install -d '#{wdir}/'"
                script << "#{rcmd} > '#{wdir}/#{file}'"
            end
        end

        options[:retries].to_i.times do
            rc = run_action 'pull_disks', script.join("\n"), rhost

            return if rc.code == 0

            if rc.code == 11
                sleep options[:delay].to_i
            else
                break
            end
        end

        raise StandardError, "Unable to pull disks: #{rc.stderr}" if rc.code != 0
    end

To Reproduce

  1. Install and configure a separate backup server with Debian 12 or 13
  2. Install Restic 0.16.5, 0.17.3 or 0.18.0 on the server (Restic 0.17.3 is supported by Opennebula: https://docs.opennebula.io/7.0/software/release_information/release_notes_70/platform_notes/#monitoring-and-backups
  3. Configure a Restic datastore according to: https://docs.opennebula.io/7.0/product/cluster_configuration/backup_system/restic/
  4. Configure a backup job that's incremental and has KEEP_LAST value of at least 2
  5. Execute the backup job and let it go over the KEEP_LAST value to trigger flattening the snapshots

Expected behavior
Removal of the oldest snapshot without an error when the repository is still exclusively locked by another process.

Details

  • Affected Component: Sunstone - Incremental Backup Job with a limit on the amount of snapshots kept with the Restic repository
  • Hypervisor: KVM
  • Version: Opennebula Sunstone 7.0.1 tested with Restic 0.16.5, 0.17.3, and 0.18.0

Additional context
Add any other context about the problem here.

Progress Status

  • Code committed
  • Testing - QA
  • Documentation (Release notes - resolved issues, compatibility, known issues)

Activity

  1. added theissue type on Dec 9, 2025
  2. added this to the Release 7.2.1 milestone on Apr 9, 2026
  3. self-assigned this
    on May 14, 2026
  4. added a commit that references this issue on May 14, 2026
  5. added 4 commits that reference this issue on May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions