Making it possible to vendor an opam package inside a dune project

I’ve recently been doing some work to benchmark dune building a large monorepo. This has involved constructing a large monorepo by vendoring as many packages from the opam repository as I can. In this process I have come across many packages which build fine on their own, but not when vendored inside another project.

I’ve fixed all the simple issues I’ve come across that prevent vendoring. If you maintain a package and have seen a PR from me in the past month or so it was likely part of this work. In retrospect it would have been clearer for me to have created this topic before embarking on my quest to fix vendoring so I could have linked to it in PRs for extra context.

The two most common problems I’ve seen which prevent a package from being vendored:

• use of the dune variable %{workspace_root} instead of %{project_root}. When building a package on its own these both resolve to the root directory of the project but when a package is vendored the former resolves to the root of the entire project while the latter refers to the directory of the package’s source code.
• depending on the library ppx_deriving. ppx_deriving is an opam package which defines a number of libraries such as ppx_deriving.eq and ppx_deriving.iter, but there is no dune library with (public_name ppx_deriving). Depending on ppx_deriving works when ppx_deriving is installed with opam because at the moment dune’s method of searching for libraries is different depending on whether a library is part of a package installed with opam, and whether the library is part of a vendored package. If ppx_deriving is vendored rather than installed with opam, including the library ppx_deriving in a (libraries ...) field of a dune file doesn’t work. The general problem (I think it’s fair to call this a bug in dune) is that “metapackages” which define libraries <pkg>.*, but no library simply named <pkg> can only appear in the (libraries ...) field if they are installed globally - not if they are vendored.

These issues both point to changes we could make to dune to improve vendoring workflows. We (dune devs) are discussing making %{workspace_root} produce a warning as its use is almost always in error.

I also want to plug vendredi which is a tool I made to help debug issue with vendoring packages! It’s a command-line tool which creates trivial dune projects which contain vendored source code from specified opam packages, git repos, and local directories.

1 post - 1 participant

Read full topic