Dune: serious bug found in the design of library variants

@diml wrote:

Dear all,

Dune 1.9.0 which was released about 3 weeks ago introduced a new feature: library variants. However, we recently realised that there is a serious flaw with the current design of this feature: some of the choices Dune is making depend on the set of opam packages installed. This is a serious reproducibility issue. This post describes the problem, explains what immediate actions we are taking to mitigate it, and what it means if you were already using this feature or were planning to use it.

The problem

When you write:

(executable
 (name prog)
 (libraries a b c)
 (variants x y z))

If a, b, c or one of their transitive dependency is a virtual library and no concrete implementation was found in the list of transitide dependencies, then dune will look at the variants attached to all installed libraries in order to automatically select a concrete implementation. As you might have guessed, the result depends on what is installed on the system. In particular, reinstalling a previously installed package could lead to a different result.

Who is affected?

Anyone who is using the new library variants feature, i.e. any project with at least one of these fields in their dune files: variant, variants, default_implementation.

I grepped the dune-universe repository to see if this feature was already used by released packages. Fortunately, it doesn’t seem to be the case.

For the record, dune-universe is a repository that embed a snapshot of the latest version of every package released in opam and using dune or jbuilder.

Resolution

We decided to take the two following immediate actions:

1. we are narrowing the scope of library variants to something we are more confident can be safely supported
2. we are putting back the feature into development mode

We will soon release Dune 1.9.2 with these changes and mark Dune 1.9.0 and 1.9.1 as unvavaiable in the opam repository to prevent these versions from being used by newly released packages.

Limiting the scope of variants

It will now be forbidden to attach a variant to a library that implements a virtual library from another project. More precisely, if you write:

(library
 (public_name foo)
 (implements bar)
 (variant blah))

Then foo and bar must be part of the same Dune project. Put it another way, one must declare all the variants of a virtual library upfront.

Putting the feature back in development mode

In order to give us a bit more time to think about the design and come up with a strong final one, we are putting back the feature into development mode. Technically, this means that to be able to use the variant, variants and default_implementation fields in your dune files you will need to add the following line in your dune-project file:

(using library_variants 0.1)

The 0.x version indicates that the design of this feature is not finalized and might change at any point. Once the design of this feature is finalized, this line will need to be removed from dune-project files and the feature will be part of the vanilla dune language again.

It is OK to release packages in opam using this feature while it is still in development mode. However, it means that your package will get a upper bound on its dune dependency in a few weeks.

Future plans

We are planning to brainstorm more about library variants to come up with a more robust design. Once we find a more satisfying one, we will implement it, test it and finally integrate it into the vanilla dune language. In the meantime, feedback on library variants is warmly welcome!

Posts: 27

Participants: 7

Read full topic