Repository navigation
wac_types::Package not having its own wac_types::Types requires re-parsing packages on every composition #85
Description
In my usecase, I need to perform several compositions that each have a different "main" component but several util components that are the same (they implement WASI, similar to what wasi-virt does but even more constrained). Previously, I was only parsing these util packages once (using Package::from_bytes, storing them in a std::sync::OnceLock, and then using them in many compositions.
I just updated my code to the latest version of wac on git. Now that the wac_types::Types are inside the wac_graph::CompositionGraph, I can no longer cache (or at least I don't know how to) the util packages but need to re-parse them on every composition.
Perhaps there could be an extra utility function, e.g. wac_graph::CompositionGraph::register_package_with_types, to take in another wac_types::Types instance and to add all of these types to the composition graph's types and to remap the package's types into the composition graph's ones (I assume something like that was done before)?
Thank you for your help and all of your amazing work on these crates!
Activity
peterhuene commentedon Apr 16, 2024
Would a
clear_nodesmethod onCompositionGraphthat keeps the registered packages (and thus nothing to reparse), but deletes all nodes and edges help with your usage pattern?juntyr commentedon Apr 16, 2024
Mmmm good question. There is always one different package, so just clearing one composition graph wouldn't help. But perhaps I could pre-generate a composition graph with all the utils packages pre-registered, store that in a
OnceLock, and then clone it for every new composition (and the new package would just be added on top of that). I'll try that out and report back :)peterhuene commentedon Apr 16, 2024
While that might work just fine, it does seem like a less than ideal API experience.
Let me think more on this and see if I can come up with a solution where packages can be shared well between composition graphs, eliminating the need for reparses and, ideally, a lot of remapping as it can be a bunch of unnecessary allocations.
peterhuene commentedon Apr 16, 2024
I won't hold up a
0.1.0release for this, but I think we can follow-up quickly with a0.2.0once we get the API around this right.juntyr commentedon Apr 16, 2024
The approach does work, also on v0.1.0 :)
Congratulations for publishing the first release!
peterhuene commentedon Apr 18, 2024
So here's what I'm currently thinking for addressing this:
Typescollection back toPackage; remove thetypesparameter fromPackage::from_fileandPackage::from_bytes.CompositionGraph::register_packagetakeInto<Arc<Package>>for the package, which you can pass anArc<Package>if you want to share packages between graphs (note:Packageshould already beSendandSync).define_typeandimportmethods onCompositionGraphwill translate, as necessary, a type coming from a currently registered package to its own remapped copy; thus, from users of the graph's point of view, there's still a singularTypescollection (the graph's).peterhuene commentedon Apr 18, 2024
I'm also toying with ways we can prevent remapping entirely.