Skip to content

GROOVY-12711: publish in-memory class output per compilation unit - #3242

Closed
daniellansun wants to merge 2 commits into
apache:masterfrom
daniellansun:GROOVY-12711-mem-class-output
Closed

daniellansun wants to merge 2 commits into
apache:masterfrom
daniellansun:GROOVY-12711-mem-class-output

Conversation

@daniellansun

@daniellansun daniellansun commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Joint compilation can keep javac CLASS_OUTPUT in memory when memStub is enabled (GROOVY-12711).

With memStub and memClassOutput (CompilerConfiguration.MEM_CLASS_OUTPUT), JavaAwareCompilationUnit does not require stubDir or a filesystem target directory: Groovy resolves the javac bytes from memory, and Groovy's own class files are published to the same class loader instead of being written to disk. The in-memory store is owned by that compilation unit. A failed compile drops the stage; a later unit does not share it. Compiling the same classes again uses a new GroovyClassLoader, matching groovyc / Ant.

There is no loader-to-store cache across units in this PR. Callers that reuse a loader (parseClass with its short-lived InnerLoader, groovy-lsp overlapping recompiles) are left for a follow-up issue once this lands.

Disk stubDir and target-directory output stay the default when the option is off. groovy.mem.stub still enables in-memory stubs only.

New API is marked @since 7.0.0, matching this line (7.0.0-SNAPSHOT).

Test plan

  • ./gradlew :lintGate
  • ./gradlew :test --tests org.codehaus.groovy.tools.javac.MemClassOutputTest --tests org.codehaus.groovy.tools.javac.InMemoryJavaFileManagerTest
  • ./gradlew :test

@daniellansun daniellansun left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build-and-test is red with the two ParserNegativeSyntaxTest failures that were broken on master after GROOVY-12641 (GEP-19). Those fixtures have been fixed on current master (see #3247).

Please rebase onto the latest master and push so CI can re-run against that tip — that should clear this noise. If anything is still red after the rebase, we can dig into it then.

@daniellansun
daniellansun force-pushed the GROOVY-12711-mem-class-output branch from a7fc949 to 7ece7be Compare November 6, 2026 19:50
@codecov-commenter

codecov-commenter commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.79070% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.4554%. Comparing base (cb1b208) to head (1a509dc).
⚠️ Report is 25 commits behind head on master.

Files with missing lines Patch % Lines
...us/groovy/tools/javac/InMemoryJavaFileManager.java 85.4839% 5 Missing and 13 partials ⚠️
...codehaus/groovy/tools/javac/JavacJavaCompiler.java 73.3333% 0 Missing and 8 partials ⚠️
...s/groovy/tools/javac/JavaAwareCompilationUnit.java 89.7959% 2 Missing and 3 partials ⚠️
...us/groovy/tools/javac/MemClassOutputClasspath.java 94.2029% 2 Missing and 2 partials ⚠️
...e/src/main/groovy/groovy/console/ui/Console.groovy 0.0000% 4 Missing ⚠️
...rg/codehaus/groovy/control/ModuleImportHelper.java 33.3333% 1 Missing and 1 partial ⚠️
...rg/apache/groovy/groovysh/jline/PackageHelper.java 0.0000% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                Coverage Diff                 @@
##               master      #3242        +/-   ##
==================================================
+ Coverage     74.4126%   74.4554%   +0.0428%     
- Complexity      44372      44864       +492     
==================================================
  Files            1699       1704         +5     
  Lines          151051     152169      +1118     
  Branches        28783      29155       +372     
==================================================
+ Hits           112401     113298       +897     
- Misses          28376      28438        +62     
- Partials        10274      10433       +159     
Files with missing lines Coverage Δ
.../codehaus/groovy/ast/decompiled/AsmDecompiler.java 96.7391% <100.0000%> (+0.2275%) ⬆️
...a/org/codehaus/groovy/control/CompilationUnit.java 81.5939% <100.0000%> (+0.0701%) ⬆️
...codehaus/groovy/control/CompilerConfiguration.java 75.4821% <100.0000%> (+0.0631%) ⬆️
...odehaus/groovy/tools/javac/MemClassFileObject.java 100.0000% <100.0000%> (ø)
...rg/apache/groovy/groovysh/jline/PackageHelper.java 45.2174% <0.0000%> (+0.3898%) ⬆️
...rg/codehaus/groovy/control/ModuleImportHelper.java 73.8095% <33.3333%> (-1.1905%) ⬇️
...us/groovy/tools/javac/MemClassOutputClasspath.java 94.2029% <94.2029%> (ø)
...e/src/main/groovy/groovy/console/ui/Console.groovy 28.0335% <0.0000%> (-0.0706%) ⬇️
...s/groovy/tools/javac/JavaAwareCompilationUnit.java 91.1290% <89.7959%> (-0.9762%) ⬇️
...codehaus/groovy/tools/javac/JavacJavaCompiler.java 71.4286% <73.3333%> (+0.9022%) ⬆️
... and 1 more

... and 47 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

* @return the mutable resource map, keyed by {@code com/example/Type.class}
*/
@SuppressWarnings("deprecation") // custom handler; URL.of(URI, handler) is not on the Java 17 baseline
static Map<String, byte[]> install(final GroovyClassLoader loader) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what situation exactly will there be a reuse of the GroovyClassLoader?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The install cache is hit only when another JavaAwareCompilationUnit is constructed with the same GroovyClassLoader instance. compile() on one unit does not call install again; that unit already holds the map and clears it.

URLClassLoader cannot drop a URL. A second addURL would leave the first one in front, and getResource returns the first URL that still opens. The second unit therefore writes into the map behind the URL already installed.

groovyc and the Ant task build one unit per run, so they do not take this path. It is the embedder case: one long-lived loader, a new compilation unit per compile. testSecondCompileOnTheSameLoaderSeesTheNewJavaApi does that, and calls loadClass only after the second compile. A class the loader has already defined is unchanged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a new compilation unit per compile tends to involve a new class loader as well. There has to be a new class loader if the same class is supposed to be recompiled. Which means you are talking about a situation where we know beforehand that the second compile will not overlap with the first compile. Is that realistic? testSecondCompileOnTheSameLoaderSeesTheNewJavaApi does not help me here. It forces the situation, instead of going through API that would normally be used. Even if there is a long lived loader, that does not exclude short lived children for the actual compilation like GroovyClassLoader#parseClass for example does. In fact I could argue that the idea of that map is flawed in that it does not install the cache where it would actually matter.

My suggestion would be to leave it out of this PR and instead make this its own issue. Then we can look at the case in isolation and with this PR already committed. And then I would also expect some of the "normal" API to be used and the structure been proven to actually exist and doing the caching.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at the callers again instead of defending the test.

groovyc (FileSystemCompiler) does new JavaAwareCompilationUnit(configuration), which builds a new GroovyClassLoader for that unit. The Ant task does the same in Groovyc#buildClassLoaderFor: a new GroovyClassLoader on every run, then one unit. Neither reuses a loader across units.

GroovyClassLoader#parseClass does reuse the outer loader, but not in the way the map assumed. createCompilationUnit builds a plain CompilationUnit on that loader, not a JavaAwareCompilationUnit. The classes are defined on a new InnerLoader per parseClass. That inner loader delegates getResource and addURL to the outer loader. Joint compilation never goes through this path, so the map was not installed for it.

The one production place that builds a new JavaAwareCompilationUnit on an existing loader is groovy-lsp (GroovyCompiler#loaderFor). It reuses the loader while compiler settings are unchanged. It sets memStub and still writes javac output to a temp directory. It does not set memClassOutput, and a typical recompile overlaps the previous one.

So there is no current caller for which a loader-to-store cache is the thing that makes memClassOutput work. The test testSecondCompileOnTheSameLoaderSeesTheNewJavaApi was the situation. I removed the cache and that test. A recompile of the same classes in this PR is a new GroovyClassLoader, which is what those entry points already do. The same unit can still compile again, including after compile(Phases.SEMANTIC_ANALYSIS) and after addSources, because the store is held by the unit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I took the map out of this PR.

A second unit on the same loader only does something useful when the class names do not overlap, or when nothing has been defined yet. I could not find a normal API that is that situation. parseClass is the long-lived loader you described, and it already uses a short-lived InnerLoader for the definitions. A map keyed by the compilation unit's loader does not follow that split: parseClass does not build a JavaAwareCompilationUnit, and InnerLoader#addURL would publish on the parent anyway. The language server does reuse a loader across units, but it recompiles overlapping files and does not use in-memory class output. Caching on that loader would be the wrong structure even there.

What remains is the part this issue is actually about. With memStub and memClassOutput, javac CLASS_OUTPUT stays in memory for that compilation unit, the unit's loader can resolve it, and no stub directory or target directory is required. The store lives on the unit. A failed compile drops the stage. A later unit does not share it. Compiling the same classes again is a new loader.

I would like to file the cross-unit question separately once this is in, and look at it against parseClass / a real embedder rather than a test that passes the same loader into two units. The outline I have in mind:

  • Title along the lines of "GROOVY-12711 follow-up: in-memory class output across compilation units".
  • Show, through an existing API, which loader is long-lived and which loader defines classes.
  • Respect that a URL cannot be removed and that a defined class cannot be replaced, so appending another mem URL on a long-lived loader is not an update mechanism.
  • Do not turn memClassOutput on inside parseClass or the language server as part of that issue; those are consumers of whatever rule we settle.

No such cache is in this PR anymore.

@daniellansun
daniellansun force-pushed the GROOVY-12711-mem-class-output branch from 7ece7be to ee8f27f Compare November 8, 2026 12:06
@daniellansun
daniellansun force-pushed the GROOVY-12711-mem-class-output branch from bae09d8 to d413c51 Compare November 8, 2026 13:42
A compilation unit opens one in-memory classpath on its own loader.
groovyc and the Ant task already use a new loader per run, and
compiling the same classes again needs a new loader. The loader-keyed
store cache is left for a follow-up.
@testlens-app

testlens-app Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

✅ All tests passed ✅

🏷️ Commit: 1a509dc
▶️ Tests: 26268 executed
🟡 Checks: 25/26 completed


Learn more about TestLens at testlens.app/docs.

@daniellansun daniellansun changed the title GROOVY-12711: keep javac CLASS_OUTPUT in memory when memStub is enabled GROOVY-12711: publish in-memory class output per compilation unit Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants