Repository navigation
GROOVY-12711: publish in-memory class output per compilation unit - #3242
daniellansun wants to merge 2 commits into
Conversation
daniellansun
left a comment
There was a problem hiding this comment.
Build-and-test is red with the two ParserNegativeSyntaxTest failures that were broken on master after GROOVY-12641 (GEP-19). Those fixtures have been fixed on current master (see #3247).
Please rebase onto the latest master and push so CI can re-run against that tip — that should clear this noise. If anything is still red after the rebase, we can dig into it then.
a7fc949 to
7ece7be
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3242 +/- ##
==================================================
+ Coverage 74.4126% 74.4554% +0.0428%
- Complexity 44372 44864 +492
==================================================
Files 1699 1704 +5
Lines 151051 152169 +1118
Branches 28783 29155 +372
==================================================
+ Hits 112401 113298 +897
- Misses 28376 28438 +62
- Partials 10274 10433 +159
🚀 New features to boost your workflow:
|
| * @return the mutable resource map, keyed by {@code com/example/Type.class} | ||
| */ | ||
| @SuppressWarnings("deprecation") // custom handler; URL.of(URI, handler) is not on the Java 17 baseline | ||
| static Map<String, byte[]> install(final GroovyClassLoader loader) { |
There was a problem hiding this comment.
In what situation exactly will there be a reuse of the GroovyClassLoader?
There was a problem hiding this comment.
The install cache is hit only when another JavaAwareCompilationUnit is constructed with the same GroovyClassLoader instance. compile() on one unit does not call install again; that unit already holds the map and clears it.
URLClassLoader cannot drop a URL. A second addURL would leave the first one in front, and getResource returns the first URL that still opens. The second unit therefore writes into the map behind the URL already installed.
groovyc and the Ant task build one unit per run, so they do not take this path. It is the embedder case: one long-lived loader, a new compilation unit per compile. testSecondCompileOnTheSameLoaderSeesTheNewJavaApi does that, and calls loadClass only after the second compile. A class the loader has already defined is unchanged.
There was a problem hiding this comment.
a new compilation unit per compile tends to involve a new class loader as well. There has to be a new class loader if the same class is supposed to be recompiled. Which means you are talking about a situation where we know beforehand that the second compile will not overlap with the first compile. Is that realistic? testSecondCompileOnTheSameLoaderSeesTheNewJavaApi does not help me here. It forces the situation, instead of going through API that would normally be used. Even if there is a long lived loader, that does not exclude short lived children for the actual compilation like GroovyClassLoader#parseClass for example does. In fact I could argue that the idea of that map is flawed in that it does not install the cache where it would actually matter.
My suggestion would be to leave it out of this PR and instead make this its own issue. Then we can look at the case in isolation and with this PR already committed. And then I would also expect some of the "normal" API to be used and the structure been proven to actually exist and doing the caching.
There was a problem hiding this comment.
I looked at the callers again instead of defending the test.
groovyc (FileSystemCompiler) does new JavaAwareCompilationUnit(configuration), which builds a new GroovyClassLoader for that unit. The Ant task does the same in Groovyc#buildClassLoaderFor: a new GroovyClassLoader on every run, then one unit. Neither reuses a loader across units.
GroovyClassLoader#parseClass does reuse the outer loader, but not in the way the map assumed. createCompilationUnit builds a plain CompilationUnit on that loader, not a JavaAwareCompilationUnit. The classes are defined on a new InnerLoader per parseClass. That inner loader delegates getResource and addURL to the outer loader. Joint compilation never goes through this path, so the map was not installed for it.
The one production place that builds a new JavaAwareCompilationUnit on an existing loader is groovy-lsp (GroovyCompiler#loaderFor). It reuses the loader while compiler settings are unchanged. It sets memStub and still writes javac output to a temp directory. It does not set memClassOutput, and a typical recompile overlaps the previous one.
So there is no current caller for which a loader-to-store cache is the thing that makes memClassOutput work. The test testSecondCompileOnTheSameLoaderSeesTheNewJavaApi was the situation. I removed the cache and that test. A recompile of the same classes in this PR is a new GroovyClassLoader, which is what those entry points already do. The same unit can still compile again, including after compile(Phases.SEMANTIC_ANALYSIS) and after addSources, because the store is held by the unit.
There was a problem hiding this comment.
Agreed. I took the map out of this PR.
A second unit on the same loader only does something useful when the class names do not overlap, or when nothing has been defined yet. I could not find a normal API that is that situation. parseClass is the long-lived loader you described, and it already uses a short-lived InnerLoader for the definitions. A map keyed by the compilation unit's loader does not follow that split: parseClass does not build a JavaAwareCompilationUnit, and InnerLoader#addURL would publish on the parent anyway. The language server does reuse a loader across units, but it recompiles overlapping files and does not use in-memory class output. Caching on that loader would be the wrong structure even there.
What remains is the part this issue is actually about. With memStub and memClassOutput, javac CLASS_OUTPUT stays in memory for that compilation unit, the unit's loader can resolve it, and no stub directory or target directory is required. The store lives on the unit. A failed compile drops the stage. A later unit does not share it. Compiling the same classes again is a new loader.
I would like to file the cross-unit question separately once this is in, and look at it against parseClass / a real embedder rather than a test that passes the same loader into two units. The outline I have in mind:
- Title along the lines of "GROOVY-12711 follow-up: in-memory class output across compilation units".
- Show, through an existing API, which loader is long-lived and which loader defines classes.
- Respect that a URL cannot be removed and that a defined class cannot be replaced, so appending another mem URL on a long-lived loader is not an update mechanism.
- Do not turn
memClassOutputon insideparseClassor the language server as part of that issue; those are consumers of whatever rule we settle.
No such cache is in this PR anymore.
7ece7be to
ee8f27f
Compare
bae09d8 to
d413c51
Compare
A compilation unit opens one in-memory classpath on its own loader. groovyc and the Ant task already use a new loader per run, and compiling the same classes again needs a new loader. The loader-keyed store cache is left for a follow-up.
✅ All tests passed ✅🏷️ Commit: 1a509dc Learn more about TestLens at testlens.app/docs. |
Summary
Joint compilation can keep javac
CLASS_OUTPUTin memory whenmemStubis enabled (GROOVY-12711).With
memStubandmemClassOutput(CompilerConfiguration.MEM_CLASS_OUTPUT),JavaAwareCompilationUnitdoes not requirestubDiror a filesystem target directory: Groovy resolves the javac bytes from memory, and Groovy's own class files are published to the same class loader instead of being written to disk. The in-memory store is owned by that compilation unit. A failed compile drops the stage; a later unit does not share it. Compiling the same classes again uses a newGroovyClassLoader, matchinggroovyc/ Ant.There is no loader-to-store cache across units in this PR. Callers that reuse a loader (
parseClasswith its short-livedInnerLoader, groovy-lsp overlapping recompiles) are left for a follow-up issue once this lands.Disk
stubDirand target-directory output stay the default when the option is off.groovy.mem.stubstill enables in-memory stubs only.New API is marked
@since 7.0.0, matching this line (7.0.0-SNAPSHOT).Test plan
./gradlew :lintGate./gradlew :test --tests org.codehaus.groovy.tools.javac.MemClassOutputTest --tests org.codehaus.groovy.tools.javac.InMemoryJavaFileManagerTest./gradlew :test