This was supposed to be a follow-up to #4787. But I realized, after careful consideration of the various dependency scenarios, that the presence of scalajs-scalalib will always imply the presence of an appropriate scalajs-library. Therefore, making scalajs-scalalib not refer to anything in scala.scalajs is not such a big deal. It becomes something akin to a "provided" dependency.

This allows us to setup our artifacts in a way that will allow forward-binary-incompatible changes to scala-library.jar (as per scala/improvement-proposals#54) without compromising on the optimizations we are currently applying inside our scalalib.

@gzm0 Does that seem like a reasonable solution?

FYI, I'll be on vacation for two weeks starting Saturday. Do we need this to be looked at before that or can it wait? (I might have time, but I can't commit).

It can wait.

If you can provide your opinion on the cyclic-but-not-cyclic dependency between scalalib and library on principle before you leave, that helps. That way I can plan ahead whether I need to invest more into #4787 or not.

I think overall such a cyclic-but-not-cyclic approach is acceptable.

What I'm wondering is if it wouldn't make sense to have the dependency the other way around:

• scalajs-library (or scalajs-scalalib)
  • versioned with Scala.js and patch Scala version
  • contains the scalalib
  • depends on:
• scalajs-library-internal (or scalajs-library, but internal makes it clear it's not for direct consumption)
  • versioned with Scala.js and minor Scala version
  • depends on:
• scalajs-javalib
  • versioned with Scala.js version

With the proposed approach, it is not clear to me which Scala version of scalajs-scalalib, scalajs-library depends on.

What I'm unsure about with this approach is how it looks in eviction scenarios across the version boundary where we split the artifacts.

What I'm unsure about with this approach is how it looks in eviction scenarios across the version boundary where we split the artifacts.

For that, I think whatever the approach we take, we must have that scalajs-library_2.13 is the name of the artifact that transitively depends on the other two (no matter what it contains). Moreover, its version number must be of a form that is considered bigger than 1.14.x (this is fine whether it's 1.15.0 or 2.13.12+1.15.0, but just making it clear).

If it contains the scalalib, then its version number must be 2.13.12+1.15.0 (that's the only scheme I can think of that will resolve to the right binary API of the scalalib contents). Although it technically works, it seems weird to me that we would jump from scalajs-library_2.13 v1.14.0 to v2.13.12+1.15.0.

Other than for the transition, as long as the thing that contains the scalalib is versioned v2.13.12+1.15.0 and the thing that contains scala.scalajs.* is versioned v1.15.0, I think resolution will always do the right thing (irrespective of the order of the dependency between the two).

So that leaves me to the question: why do you think scalalib-depends-on-scalajs makes more sense? To me it seems that the converse is less weird. For example scala.scalajs.* contains classes that are subclasses of scala.collection.*. IMO that's a stronger dependency (interface dependency) than the other way around, where scala.collection.* only uses scala.scalajs.* in its implementation (implementation dependency).

IIUC it's because of your comment:

With the proposed approach, it is not clear to me which Scala version of scalajs-scalalib, scalajs-library depends on.

which is of course not obvious from the public version numbers, but that's not different from any other Scala library: it does not expose in its version number (only in its POM, which we would do) the specific patch version of scala-library.jar that it depends on.

Ping @gzm0 :)

Eviction Scenarios

Consider the following project and dependency versions:

This makes the assumption that compiling with an older Scala patch version than an upstream dependency is allowed. Note that the equivalent for Scala.js versions (compile & link with 1.15.0 using lib built with 1.15.1) is allowed.

IIUC, this will not resolve scalajs-scalalib correctly (should be 2.13.5+1.15.1). Or is the + somehow special in the version numbers? (I'm interpreting this as SemVer here).

Or is the intent that we allow this and ensure that scalajs-library can work with older scalajs-scalalib versions (IMO that's quite prohibitive).

This also somewhat raises the question: Why do we even need to separate the scalajs-library / scalajs-scalalib artifacts? What happens if we just version scalajs-library with the new versioning scheme?

Dependency order

From an actual dependency POV, you are 100% right. And this is probably the stronger argument: If we ever intend to go through with #4787, we need it in that order.

My concern was about scalajs-scalalib artifacts that are "internal" but not being depended upon by anything: For a specific Scala.js / Scala minor version, I assume the scalajs-library will depend on the scalajs-scalalib it was compiled with.

In graphs, for the current scheme this would look like this:

graph LR
  jl["scalajs-javalib@1.15.0"]
  sjs["scalajs-library_2.13@1.15.0"]
  s11["scalajs-scalalib_2.13@2.13.11+1.15.0"]
  s12["scalajs-scalalib_2.13@2.13.12+1.15.0"]
  s13["scalajs-scalalib_2.13@2.13.13+1.15.0"]

  m11["?"] -.-> s11 --> jl

  sjs --> s12 --> jl
  sjs --> jl

  m13["?"] -.-> s13 --> jl

If we inverted scalajs-library and scalajs-scalalib, it would look like this:

graph LR
  jl["scalajs-javalib@1.15.0"]
  sjs["scalajs-library_2.13@1.15.0"]
  s11["scalajs-scalalib_2.13@2.13.11+1.15.0"]
  s12["scalajs-scalalib_2.13@2.13.12+1.15.0"]
  s13["scalajs-scalalib_2.13@2.13.13+1.15.0"]

  sjs --> jl

  s11 --> sjs
  s11 --> jl

  s12 --> sjs
  s12 --> jl

  s13 --> sjs
  s13 --> jl

IIUC, this will not resolve scalajs-scalalib correctly (should be 2.13.5+1.15.1). Or is the + somehow special in the version numbers? (I'm interpreting this as SemVer here).

Or is the intent that we allow this and ensure that scalajs-library can work with older scalajs-scalalib versions (IMO that's quite prohibitive).

That is the intent, yes. But it's not an older scalajs-scalalib. It's a scalajs-scalalib for a newer Scala version built by an older Scala.js. That is perfectly fine and not constraining: the API that we can use is anyway that advertised by scala-library.jar; the newer Scala.js could not have added in its scalajs-scalalib something that its scalajs-library would technically be able to rely on. This amount of "limitation" is even already true today, since we compile them separately. The proposed system does not create any new constraint.

(the + is not magical)

This also somewhat raises the question: Why do we even need to separate the scalajs-library / scalajs-scalalib artifacts? What happens if we just version scalajs-library with the new versioning scheme?

That would be problematic for a third-party library that was built with 2.13.4+1.16.0, relies on an API added and provided by 1.16.0 in scala.scalajs.*. If its dependency gets evicted by one on 2.13.5+1.15.0, it will link against an artifact that does not offer in scala.scalajs.* the things it needs.

My concern was about scalajs-scalalib artifacts that are "internal" but not being depended upon by anything: For a specific Scala.js / Scala minor version, I assume the scalajs-library will depend on the scalajs-scalalib it was compiled with.

In graphs, for the current scheme this would look like this:

I'm afraid I fail to see what's problematic about that. If nothing depends on them, they won't even appear in the resolution, will they? I think I don't understand what you mean by "internal" here.

If we inverted scalajs-library and scalajs-scalalib, it would look like this:

Just to be clear: AFAICT that specific variant with the artifacts named like that is not realistic from a 1.14->1.15 transition point of view. We need the "outermost" artifacts to be named scalajs-library_2.13, no matter what they contain. But your graph remains valid if we take the variant where scalajs-library_2.13 actually contains the scalalib (scala.collection.* for example) and we have a scalajs-sjslib_2.13 in the middle.

That said, couldn't you make the same argument that you're doing for the first graph, about a ? -> scalajs-library_2.13 v1.16.0 added to the second graph? Is the difference related to the "internal" thing I don't understand above?

It's a scalajs-scalalib for a newer Scala version built by an older Scala.js.

OK. I understand that this will not cause any issues for interface compatibility. But what about code in overrides? If the "Project" in the above example depends on a fix to an override that was made in Scala.js 1.15.1, that's a problem, no?

IIUC pretty much the sole purpose of having the Scala.js version in the scalajs-scalalib artifact is so that we can publish a new one if there is an issue with the Scala.js side (override or compiler bug)?

I'm afraid I fail to see what's problematic about that. If nothing depends on them, they won't even appear in the resolution, will they? I think I don't understand what you mean by "internal" here.

There is no issue for dependency resolution. My initial feeling was just that the global dependency graph is "odd" because there are artifacts that will only be depended upon when evictions happen (in the example graph, when a SJS 1.15.0 user targets Scala 2.13.13, in fact, IIUC the 2.13.11 artifact seems pointless?).

OK. I understand that this will not cause any issues for interface compatibility. But what about code in overrides? If the "Project" in the above example depends on a fix to an override that was made in Scala.js 1.15.1, that's a problem, no?

Yes, that could be a problem. Not sure if there is a way out of that.

At least if it happens, it's always fixable by using a recent enough Scala.js in the final project (the one that we actually link).

IIUC pretty much the sole purpose of having the Scala.js version in the scalajs-scalalib artifact is so that we can publish a new one if there is an issue with the Scala.js side (override or compiler bug)?

It's more important than that. We need that to be able to make any kind of change in scalalib after the release of 1.15.0. For example, if the latest Scala version at the time of 1.15.0 is 2.13.14. Then we will publish a given scalajs-scalalib v2.13.14. Some time later, 2.13.15 gets published. We need to quickly (same as what I've been doing with compiler plugins) back-publish scalajs-scalalib 2.13.15 out of a released Scala.js, which means 1.15.0. Even if in main we make changes to scalalib, by the time we release 1.15.1, there is already a scalajs-scalalib with version 2.13.15 out there, which we cannot override. So the changes we've made cannot be published. The cycle repeats, of course, and so in fact we never get to publish any newer changes.

There is no issue for dependency resolution. My initial feeling was just that the global dependency graph is "odd" because there are artifacts that will only be depended upon when evictions happen (in the example graph, when a SJS 1.15.0 user targets Scala 2.13.13, in fact, IIUC the 2.13.11 artifact seems pointless?).

Ah yes, I see. It's not that some can only surface through evictions (that's not possible by Ivy rules), but there are some that will always be evicted by something else. That might be a bit odd, yes. I would still consider it an improvement over the current status quo where this always happens, just silently without any graph to make it explicit (-ly odd), no?

Yes, that could be a problem. Not sure if there is a way out of that.

It seems like there won't, unless the dependency resolution system is able to calculate "new" artifact versions. (combine the bigger Scala version with the bigger Scala.js version). This is likely not something we can expect from it.

Sorry I didn't catch on to this earlier, somehow I thought this is possible (when we had the original discussions in the SIP).

At least if it happens, it's always fixable by using a recent enough Scala.js in the final project (the one that we actually link).

You mean Scala, not Scala.js? In the example above, the final project uses the latest Scala.js version, but not the latest Scala version.

Does the Scala library offer backwards source compatibility? If no (which I'd expect), this can be a real problem, no? We essentially force downstream projects to update for versions where we promise(d) that it is not necessary.

You mean Scala, not Scala.js? In the example above, the final project uses the latest Scala.js version, but not the latest Scala version.

Actually both Scala and Scala.js. So that we introduce a dependency on scalajs-scalalib_2.13 that has both a recent Scala version and Scala.js version.

An alternative is to directly add it as a libraryDependencies += "org.scala-js" %% "scalajs-scalalib" % "2.13.14+1.15.1". That's a bit ugly and inconvenient, though.

Does the Scala library offer backwards source compatibility? If no (which I'd expect), this can be a real problem, no? We essentially force downstream projects to update for versions where we promise(d) that it is not necessary.

The Scala library won't promise backwards source compatibility given SIP-51, no. However, on that particular point, the most recent analysis of SIP-51 for the JVM suggests that even Scala/JVM will require this. To be more precise, if a project transitively depends on scala-library v2.x.y, it will enforce that its scalaVersion >= 2.x.y. So we won't be imposing more constraints than Scala/JVM itself, I believe.

OK, in the case where we require the project to have a Scala version that is at least as new as the transitive dependencies, the above scenario cannot happen.

What can happen is this:

Basically, a Scala version upgrade can break something in a dependency that was relying on a bugfix introduced in a later Scala.js version. But I think at that point, this is indistinguishable from a bug introduced in the new Scala library. So we can probably live with it.

So for me the only question remains how do we ensure the Scala library version dependency is enforced / checked?

So for me the only question remains how do we ensure the Scala library version dependency is enforced / checked?

The relationship between scalaVersion and the highest transitive scala-library will be checked by the build tools. This will be done independently of Scala.js.

Our sbt plugin, with this PR, adds scalajs-scalalib with a version based on scalaVersion. We'll ask the other build tools to do the same.

Combining the above two means that we get the enforcement that scalajs-scalalib is, for its Scala version number part, >= scalaVersion and >= scala-library.

Is that what you were asking?

Is that what you were asking?

Yes.

So at this point, I'm convinced that going the way of this PR is OK. What I'm wondering is if we should actually merge / release this now (when the build tools do not enforce the scala version thingy yet). Then again, a build tool author will say the same about Scala.js publishing the relevant artifacts 🤷

So probably we should just go for it. Maybe, as a stop-gap, we can add the check for the version ourselves, but doing this correctly (considering older versions in a backwards compatible manner) is likely not trivial.

I'll give this another thorough review for good measure, but I don't expect any issues.

Thanks for the thorough analysis and questioning. This stuff is hard.