Skip to content

multisem-inpainting-20261111 - #264

Open
trautmane wants to merge 8 commits into
newsolverfrom
multisem-inpainting-20261111
Open

trautmane wants to merge 8 commits into
newsolverfrom
multisem-inpainting-20261111

Conversation

@trautmane

@trautmane trautmane commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

add createNoPaintDatasets method and run on driver to avoid "exceeded the rate limit for object mutation operations ... see https://cloud.google.com/storage/docs/gcs429" exceptions

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

Approval is blocked because there are still open review findings in Wafer6061Inpainter.java, specifically nit-level logging issues that duplicate identifiers and omit dataset-path context, making the logs less actionable.

2 open issues
What changed in this PR

Reduces GCS 429 “object mutation rate limit” errors during MultiSEM inpainting by moving N5 “no paint” dataset creation to the Spark driver and adding retry/backoff around dataset creation.

Changes:

  • Pre-create required “no paint” N5 pyramid datasets serially on the driver before executor work starts.
  • Add rate-limit-aware retry/backoff logic for dataset creation to handle transient GCS throttling.
  • Update CLI parameter help text for intensity-adjust settings.
File Description
render-ws-spark-client/​src/​main/​java/​org/​janelia/​render/​client/​spark/​multisem/​Wafer6061Inpainter.java Moves “no paint” dataset creation to a driver-side pre-step and adds rate-limit retry/backoff plus minor logging adjustments to reduce GCS 429 failures.
render-ws-java-client/​src/​main/​java/​org/​janelia/​render/​client/​parameter/​AlgorithmicIntensityAdjustParameters.java Refines JCommander CLI parameter descriptions for algorithmic intensity adjustment options.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

The change is not approval-ready because there are still documented inconsistencies in user-facing documentation (CLI help text and Javadoc) that diverge from the actual default and retry/backoff behavior, requiring updates to keep operational guidance accurate.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Low severity Help omits default value for CLI flag

render-ws-java-client/​src/​main/​java/​org/​janelia/​render/​client/​parameter/​AlgorithmicIntensityAdjustParameters.java:111

The CLI help for this flag no longer indicates the default value, even though the default is still set in code (8000). Keeping the default in the description makes --help output self-contained and matches other parameters in this class (e.g. --maxPixelCacheGb, --renderScale).

Low severity Retry Javadoc understates worst-case wait time

render-ws-spark-client/​src/​main/​java/​org/​janelia/​render/​client/​spark/​multisem/​Wafer6061Inpainter.java:977

The Javadoc says backoff is capped at 64s and ~4 minutes total, but with maxRetries=7 and delay doubling from 2s the largest delay is 128s (2,4,8,16,32,64,128) and total wait is ~254s. Please update the comment (or the retry parameters) so operators can reason about worst-case driver stall time.

2 resolved since last review

Review effort: Balanced

@trautmane

Copy link
Copy Markdown
Collaborator Author

With regard to "Help omits default value" issue flagged by Copilot, the com.beust.jcommander library automatically includes default values in it's help so there is no need to add them to parameter descriptions. Interestingly, Claude also wanted to include the redundant default values.

@trautmane
trautmane requested a review from minnerbe November 12, 2026 15:56
…t is easier to check locations after processing. In most cases there will only be a small number of locations, so they should not cause too much bloat in the attributes.json file. A maximum of 1000 locations will be written to attributes.json to prevent problems. All locations are logged.
this.inpaintedLocations = this.inpaintedLocationsTruncated ?
new ArrayList<>(inpaintedLocations.subList(0, MAX_RECORDED_LOCATIONS)) :
inpaintedLocations;
this.inpaintedLocationCount = inpaintedLocations.size();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it would be a good idea to sort the locations by size so that it's more likely that we capture the most important inpainting events?

…tions by changed pixels with the most changed pixel locations listed first. This ensures that in the rare case where the inpaint locations list needs to be truncated (because there are too many of them), the largest changes will be written to attributes.json.
…n be viewed in neuroglancer with multiple scales
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants