Skip to content

transdecoder/predict cannot be resumed: it writes checkpoints into its staged input, then fails on re-execution #12799

Closed

Description

@erikrikarddaniel

Summary

TransDecoder.Predict writes its checkpoint directory into the LongOrfs directory it receives as a staged input.
That directory belongs to the TRANSDECODER_LONGORF task, so two things follow, and both are reproducible from a clean run:

  1. The task never resumes. Because it modifies its own input, a following -resume re-executes it even when nothing has changed.
  2. The re-execution then fails. Nextflow wipes the task directory before re-running, but not the LongOrfs directory the symlink points at, so the checkpoints outlive the outputs. TransDecoder.Predict sees a complete checkpoint set, skips every step including the ones that create the outputs, and exits 0 having written nothing.

The failure surfaces as Missing output file(s) '*.transdecoder.pep' with Command exit status: 0, which points at the module rather than at the checkpoints, and it is permanent: every retry repeats it.

Reproduction

With nf-core/metatdenovo's smallest test profile, about two minutes per run:

nextflow run nf-core/metatdenovo -profile test,docker --outdir out --orf_caller transdecoder
# [SUCCESS] completed=63 failed=0 cached=0

nextflow run nf-core/metatdenovo -profile test,docker --outdir out --orf_caller transdecoder -resume
# TRANSDECODER_LONGORF   CACHED
# TRANSDECODER_PREDICT   FAILED
# [FAILED] completed=1 failed=1 cached=39

Nothing changed between the two runs.
TRANSDECODER_LONGORF is cached as expected; TRANSDECODER_PREDICT is not, and its re-execution fails.

From the failing task's .command.err, every step is skipped, including the final copies:

-- Skipping CMD: cp .../longest_orfs.cds.best_candidates.gff3.revised_starts.gff3
                    .../megahit_assembly.contigs.fa.transdecoder.gff3,
   checkpoint [.../megahit_assembly.contigs.fa.transdecoder_dir/__checkpoints_TDpredict/copy_to_final_outputfile.974da7....ok] exists.
transdecoder is finished.

and in the task directory the checkpoints resolve outside it:

megahit_assembly.contigs.fa.transdecoder_dir ->
  .../work/6a/36d125.../megahit.transdecoder/megahit_assembly.contigs.fa.transdecoder_dir

Versions: transdecoder 5.7.1--pl5321hdfd78af_0, Nextflow 26.04.6, local executor with Docker.

Why the module cannot just pass this through

This is how the tool is meant to work -- TransDecoder.Predict's own help says --output_dir | -O <string> same output directory from the TransDecoder.LongOrfs, so writing into the LongOrfs directory is by design.
What is specific to the module is that the directory is a staged input rather than a working copy, so the tool's writes land in another task's directory and Nextflow's caching sees an input that changes after use.

Possible fixes

In rough order of preference:

  1. Give Predict a private, writable view of the LongOrfs directory.
    Building a directory of symlinks to each file inside the staged one costs no extra disk: reads follow the symlinks, and everything Predict creates -- the checkpoint directory and its intermediates -- is local to the task.
    Worth verifying first that Predict only ever adds files there and never overwrites a LongOrfs output.

  2. Copy the directory (cp -rL) into the task directory before running.
    Simplest and certainly correct, but it duplicates the LongOrfs output, which is large for metagenome assemblies -- longest_orfs.cds alone can reach several GB.

  3. At minimum, remove a stale __checkpoints_TDpredict before running, so a re-executed task redoes the work instead of failing.
    One line, and it fixes the hard failure, but it writes into another task's directory and leaves the self-invalidation in place, so it is a fallback rather than a fix.

Independently of which is chosen, the module could check that its declared outputs exist and fail with a message naming the checkpoints, instead of exiting 0 and leaving Nextflow to report a missing file.

Impact

Any interrupted run, any errorStrategy 'retry', and any -resume can hit this.
With a retry policy it converts a transient failure into a permanent one, and on large assemblies the wasted work is substantial: LongOrfs plus Predict over a multi-sample metagenome assembly is hours.


Reported with help from Claude Code.

Activity

  1. added a commit that references this issue on Aug 27, 2026
    a990e41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions