Repository navigation
Description
Hello all,
I want to open up the discussion about how Backpack can be integrated with Stack; in particular, I'd be willing to work on and submit a patchset adding support for it in Stack. But before I do that, I want to hash out some general architectural concerns. The specification for the GHC and Cabal support for Backpack is here: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst ; please read it for more motivation on why you might like to support it.
The design process over two years lead us to a design where Backpack requires the package manager (Stack) to know how to instantiate packages that have required signatures in them (I spent the better part of six months trying to figure out how to do it all in GHC; if you are interested in the sordid details I can let you know). Things seem to work most smoothly if:
- The package manager has a component oriented build model; that is, the unit of work in the install plan is a component rather than a package. (I don't believe Stack is architected in this way, although issues like stack 0.1.8 lost ability to build individual executables or library #1406 suggest it might be a good architecture for Stack anyway.) See also https://github.com/ezyang/ghc-proposals/blob/master/proposals/0000-componentized-cabal.rst for the proposal that added per-component configure to Cabal
- The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a
PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.
I'll note that private use of Backpack (i.e., the public library does not have any signatures) should be supported out-of-the-box, as there is no change to the outward-facing API of Setup scripts in this case (Stack will need to understand how to register multiple libraries produced by a Setup script, however). In the meantime, public libraries which have signatures would not be supported.
One of the major applications of Backpack I would like to see in the near future is the Backpack-ification of GHC's boot libraries to solve the string problem. Backpack would make it easy to write modules which are parametric over a particular desired string representation; the original package names would continue to export String-based interfaces, but versions using ByteString etc could be made available, and the availability of a package abstracted over holes means that an end-user could specify their own. Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.
Please let me know if you have any questions or concerns; I'd be happy to answer them.
CC @mgsloan, @snoyberg, @Blaisorblade , @borsboom , @harendra-kumar
Activity
aljce commentedon Aug 29, 2016
This would be awesome.
Blaisorblade commentedon Aug 29, 2016
@ezyang First let me thank you for all your work on this.
TL;DR. Does stack need this shipped tomorrow, in a month or in a year?
Before I can look into details and ask more substantive questions: is there a timeline for Backpack being merged/released/used, and an estimate how much work was required on cabal-install?
I agree this is probably essential. Having feature parity also matters—having to choose between stack and Backpack, or stack and sane strings, would be pretty bad.
borsboom commentedon Aug 29, 2016
@mgsloan, I think you know the most about the parts of Stack that @ezyang is talking about. Once you're back from BM, can you weigh in?
@ezyang: how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.
ezyang commentedon Aug 29, 2016
Definitely not tomorrow or a month. If all goes well Backpack will get merged to GHC and Cabal, and people can start experimenting with it, but there were certainly be bugs and I won't claim that anyone should start using it for real immediately. Backpack'ing base is at least blocked on teaching GHC's build system how to deal, which will take some time and I am not planning for 8.2. On the other hand, base is probably the most plausible, widely used package that will get Backpack'ed, because (1) the potential upside is so great, and (2) it closely tracks GHC versions, so we don't have to worry about supporting old GHCs.
There are some key library functions for configuring and instantiating packages, and cabal-install (new-build only; the legacy codepath doesn't support it) just calls out to them. At present, there are some things that are duplicated because I couldn't figure out an API that could be reused in both cases, but this is definitely a case where I can try harder.
The two key functions have these signatures:
and a
LinkedComponenthas all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)chrisdone commentedon Sep 5, 2016
Stack does this. We use
Cabalto parse the .cabal file but then resolve theGenericPackageDescriptionmanually. Here is an example: https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L765 The whole of Stack.Package is pretty much dedicated to resolving package info. See also https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L198ezyang commentedon Mar 1, 2017
Great. Another question I have (since I am looking into this) is how difficult it would be to get Stack to support a per-component build model. I'm hoping that I can take anywhere we talk about 'Package' and actually it's talking about a 'Component', but I am not familiar enough with the code to say if this is the case or not.
mgsloan commentedon Mar 3, 2017
@ezyang Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:
And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.
ezyang commentedon Mar 7, 2017
OK. So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package (another thing where I'm not sure how well Stack will deal with this change.)
ezyang commentedon Mar 10, 2017
I've been fishing around the codebase, and here are some initial architectural thoughts on what Backpack needs.
Stack defines a simplified
Packagedata type, which it reads out the contents ofPackageDescriptioninto. Backpack introduces some new fields, most notablymixins, which need to be preserved so that Backpack can figure out how packages are being instantiated. At the moment,Packagedoesn't really preserve any information like exposed modules which we would need to do this. It seems the most convenient way of keeping all the information Backpack needs is to keepPackageDescriptionverbatim inPackage? This will also make it easier to call into Cabal to do mix-in linking.Once we have all the packages, we need to do a "mix-in" linking step to figure out exactly how packages have been instantiated. For example, if I have
build-depends: regex-indef, str-string, I need to figure out thatregex-indef's signature namedStrwas instantiated with theStrfromstr-string.It is an interesting question on how to integrate this with Stack. Stack's model is that it goes ahead and reads in all of the local source packages, but then lazily reads in the external package information using the accessor function created by
withLoadPackage. The idea is that if a package is already installed, then we can avoid traversing its dependency graph. This should integrate reasonably well with Backpack: if a package is installed, we can read off the module shape (that's the metadata you need to mix-in link) from the installed package info; if it is not, we can compute it when weinstallPackage.Mix-in linking is not enough: after we figure out how everything is linked, we have to instantiate, creating a build task for every configuration by which a package was instantiated. In Stack, a build
Planconsists of some number ofTasks to be done. I'm not sure how dependencies between tasks are computed, but each task is uniquely identified by thePackageIdentifierit provides (taskProvides). ATaskroughly corresponds to a configure/build pair on Cabal. So, we need to modifytaskProvidesto be a bit more flexible: rather than only build aPackageIdentifier, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier aUnitIdentifier, following the terminology I've used in Cabal and my thesis. Roughly speaking, these data structures should look like this:If Stack was modified in this way, would you take the patch? How would you like the design to be different? What is unclear?
mgsloan commentedon Jul 27, 2017
Hey @ezyang , sorry for letting this languish, got caught up in other things. Passing this over to @snoyberg
snoyberg commentedon Jul 27, 2017
I'm just coming up to speed on this now Edward. It seems relevant to work I'm already doing upgrading to Cabal 2.0. I'm also in favor of pushing Stack towards a component base build system. My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.
I realize this issue is a bit dated now, and perhaps you have different thoughts. Maybe we could jump on a text chat and synch up?
ezyang commentedon Aug 2, 2017
@snoyberg Sure thing; ping me on IRC some time? As far as I am concerned, the thinking on this ticket hasn't changed much (although maybe Stack's internals have!)
Yeah, so in cabal new-build, this is done by treating these packages as non-per-component, single packages in the plan. It's a little bit of complexity but not too bad, and keeps BC.
snoyberg commentedon Aug 2, 2017
Just chatted with @ezyang on IRC about this. Here's my summary:
stack testinside thebytestringpackage.)I'll ping this issue with updates when Cabal 2.0 is merged in
snoyberg commentedon Aug 17, 2017
Pinging appropriately: Cabal 2.0 support has just been merged to master :)
mgsloan commentedon Aug 18, 2017
One thing for switching stack to a per-component build plan will also be adding support for internal libraries - #3361
Also support for foreign libraries, not sure if that's as relevant - #3364
56 remaining items
flip111 commentedon Feb 4, 2020
@ezyang
Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?
flip111 commentedon Feb 4, 2020
On how to blend backpack code with stack code ...
Searching in cabal's codebase for
toLinkedComponentbrings up thebuildComponentfunction. Even if this function was on top level and being exported on first glance it seems there is too much cabal specific stuff in there.On the one hand for stack's purposes it seems useful to do the expansion of tasks to build that is needed for backpack as late as possible. So after tasks are made expand them in yet even more tasks.
stack/src/Stack/Build/ConstructPlan.hs
Line 206 in a7a6c35
On the other hand toConfiguredComponent and toLinkedComponent need information that will only be available earlier than
toTaskso this information needs to be carried through anyway.First step seems to be to figure out where the arguments from
toConfiguredComponentcome from and if they can be constructed (have data constructors exported) and where in the stackconstructPlanthis information is available. After that the same fortoLinkedComponent.After that see if the return value of
toConfiguredComponentcorrespond to a type Stack has, and wether this return type (LogProgress ConfiguredComponent) can be embedded in a Stack type.bbarker commentedon Feb 4, 2020
As a very new Haskell user, I was exposed to backpack (while still getting my bearings straight with regard to Haskell build tools), only to realize it would be a no-go if I wanted to use it with Stack. I've since been tempted to use it again, but realized the issue was still open. Had I been a user of backpack all along (which has only been a little over a year), I'm sure I'd have thought of many more places I could have used it. Just my 2c - I understand all too well that development time is limited.
ezyang commentedon Feb 14, 2020
Sorry about the delay, I saw this in my GH notifications but sat on it for a week.
@flip111
referring to:
My recollection from the last time I looked at the Stack code, was that everything in Stack revolves around installing packages (there's a dep graph of packages, you install them one by one). The comment on private Backpack use is simply pointing out that sometimes installing one package results in multiple libraries getting installed, and as long as Stack can deal with this situation, all is good.
Yeah, they're not the same, precisely because the proposed type here refers to packages rather than components.
Components are a pretty big deal for usability of Backpack, because a lot of things you want to do involve having to write many small libraries and then recombining them together. If you'd have to make (and distribute) a separate package for each of these libraries, it would be a serious source of friction.
It's fine to do it after depsolving; that's how Cabal-install does it. Depsolve the package structure, and then elaborate it into a more fine-grained component structure.
As for the last few comments, the kit in the Cabal library is what stack can usefully use, the stuff in cabal-install is a useful reference for how to use the APIs but I expect none of it is salvageable. That's OK; I have a "mini" implementation of the Cabal-install logic for Setup.hs and it was mostly fine.
To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end.
cartazio commentedon Feb 28, 2020
@snoyberg is this on the todo queue of y’all ?
theobat commentedon Aug 16, 2020
Following:
I think for now, it's better to follow the progress here, for anyone interested:
#4745
Also, given that component-centric builds seem to require significant changes, my guess is backpack support in stack is unlikely to land before some time.
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack
Add Backpack (cross-package) support to Stack