Several years ago I worked for a company that was developing a distributed front-end application using Angular. We had issues getting everything to work together and I worked out some sort of solution to that problem using library versioning that was not semver. I'm currently working on a very similar problem (a distributed UI application built with Angular) and decided to try to figure out an actual good way of doing what I was trying to do back then. And I think I got it!
For starters, here's what I wrote up about that failed attempt all those years ago:
I was a contributor on a large application that used Angular for the front end. We decided that each piece would be developed separately by individual teams and then brought together as one massive monolithic application. One of the problems we encountered with this process was how we tested those individual packages prior to making them available for consumption in the application. We had more than one case where everything worked locally, but upon publication the whole application broke because of a small defect in one package. I designed a solution using VSTS (our build tool) to publish beta packages to our internal npm feed, then trigger a custom build of the application that consumed the newly published beta package. Using this new solution developers were able to test their published packages on a deployed test environment without changing the production ready packages and potentially crashing the application. Although this seems like something Semver could have handled (and was intended to handle), Semver was unfortunately not an option in our environment so we had to find a different solution.
Today's problem is pretty similar. We have multiple libraries that fully contain dedicated functionality and then a single presentation application that brings those libraries together. (I owe myself and you a separate blog post on creating a library in Angular that just works and is easy to change and validate before publishing, but this isn't that post.) For the sake of this article we're going to call the libraries tundra-ui-core, tundra-ui-payment, and tundra-ui-presentation.
tundra-ui-core
This library contains anything that is shared across two or more other libraries and/or tundra-ui-presentation.
tundra-ui-payment
This library contains all of the components, services, etc. responsible for accepting payments.
tundra-ui-presentation
This is the actual application, which will consume tundra-ui-core and tundra-ui-payment.
The Problem
As we work on tundra-ui-payment we're probably going to publish multiple versions of it to our internal feed to test it out within tundra-ui-presentation, but we don't want to waste a bunch of real version numbers doing that. We've decided we'll publish alpha versions for our developer testing and beta versions for our QA process. These will coincide with the branching strategy we already have in place for tundra-ui-presentation so that when a dev version of that project is published it automatically installs alpha versions of all libraries and when a QA version is published it automatically installs beta versions of all libraries.
Although this works, it created a major pain point because we had to manually increment the version of the library so that it included an alpha version, then change it to a beta version when we merged into the QA environment (I'll try to remember to document the branching strategy another time). It caused lots of extra commits with comments like "forgot to increment version" or "forgot to remove beta".
The Solution
It turns out that it's now possible to update the version of the library in the package.json file during the build process, then commit that change back into git without triggering another CI build. It took some doing to get things just right, but early testing is very promising and I couldn't wait to document it here.
I got started by following this guide, but I had to make some pretty significant changes so I'm going to document my whole process here. First off, we use Azure DevOps for our repositories, build and release pipelines, and to host our internal npm feed. We do not host our own instance of Azure DevOps so we do have access to the latest features.
Permissions
We're going to need two permissions setup in our Azure DevOps instance for each project that's going to host a repository that uses this method. If our Azure DevOps project is called Tundra then we'll need to make sure the associated user account for the Tundra project has these permissions. We'll also need to configure one of these permissions for each repository in each project. I know that's a pain, but the good news you only have to do it once per repository.
Open your project in Azure DevOps and navigate to Project Settings:
In the Project Settings pane, scroll down and select Repositories:
Choose the repository you want to add permissions for and navigate to the Security tab:
Scroll down to the Users section (expand it if necessary) and select the <Project> Build Service <Organization> user. For example, if our Tundra project was in the ArcticSoftware Organization we'd be looking for "Tundra Build Service (ArcticSoftware). Set the permissions below to "Allow" for this user.
- Contribute
If you have branch policies active on any of the branches you'll be committing back to during the build process, you'll also need to enable these permissions:
- Force push (rewrite history, delete branches and tags)
- Bypass policies when pushing
The next step is granting permissions to the Artifacts feed to that same user account. In the left pane, choose Artifacts:
Click the settings cog near the upper right corner of the screen:
Navigate to the Permissions tab:
If you do not see the same user as above (Tundra Build Service (ArcticSoftware) in our example), click the "Add users/groups" button, select "Contributor", search for and select the user, click the "Save" button.
We now have the necessary permissions configured for this to work. We just have to create the build pipeline. I did this with yaml (I'm learning to love it) so I'm going to go through each step and explain what's happening and why I did it the way I did.
Build Pipeline
We're setting up some variables we'll use in conditions later. The really basic explanation of our branching strategy is that we have main, ready-for-deployment (which is a holding branch for the period between the end of our sprint and the time the code gets deployed to production), release/* and hotfix/* branches for things that are in QA, and then everything else (story and task branches basically).
We have to checkout the repository. This is a key step that I initially overlooked and it caused me some problems later. You can find out more about the options specified here on your own. This is what I needed.
We have an .npmrc file setup for our internal feed that's in the same director as this .yaml file so we don't need to specify any credentials or anything here to install dependencies (remember that we setup permissions on our internal feed on an earlier step).
This was another step that caused a headache for me. The native npm install command will authenticate internally, but in order to access the npm feed through Powershell (which we'll do in the next step) we have to authenticate explicitly, which modifies the .npmrc file.
This is where the heavy lifting starts. This is a pretty big Powershell script (obviously) that's doing a lot of stuff. I'm going to break this one down pretty much line-by-line so it makes more sense.
The built-in variables Azure DevOps gives us are great, but they weren't quite what we needed for a future step. We want to checkout the branch that triggered this build, but if that branch is in a folder we need to specify the folder and the branch name. If we use $(Build.SourceBranchName) we don't get the folder and if we use $(Build.SourceBranch) we get the folder and branch name, but also "refs/heads/". All we're doing here is getting the folder and branch name and saving it for later.
We're using a couple of cmdlets available to Powershell to store the contents of the package.json file in a variable so we can find the version number and manipulate it.
We want to isolate the base package version and this line does that. We use Semver so by "base package version" I mean if the value in the version field is "1.0.1-alpha.1" we just want the "1.0.1" part saved for later. By splitting the entire value on the "-" and taking the first part, we get just that.
This is possibly the coolest part of the whole thing. We're going to our internal feed and retrieving every version that's ever been published of this specific package (tundra-ui-payment). The --json switch saves the value as JSON and the --silent switch just prevents the pipeline from displaying error messages if the package doesn't exist at all.
The view versions command appears to return the packages in publication order, which is exactly what we want. However, I can't find any confirmation that will always be the case so this is a potential spot for an issue to arise.
Once we have all of the versions, we need to filter that list to get only the alpha versions of this base version. If our base version is 1.0.1 then we want to get all of the versions that are "1.0.1-alpha*" where the * is a wildcard standing in for any number of characters. This will return "1.0.1-alpha.1", "1.0.1-alpha.sigma" or anything else that matches the pattern (though we expect it to always be in the format "1.0.1-alpha.<number>").
The next several steps all go together and are pretty dumb to have to do, honestly. First we're creating a new variable ($newSubversion) and initializing it to 1. Then we're going to try to figure out what the last alpha version was that was published to our feed. Unfortunately, the where cmdlet of Powershell sometimes returns an array and sometimes returns a single value. If there are multiple matches we'll get back an array containing all matches, but if there's only one match then we get back just that value as a string (which is also an array of characters).
If we try to split on the "." character and there was only one result we'll end up with either an error or useless data. I don't remember which, but it wasn't right. That's where the second if statement comes into play. (The first if statement just checks whether we got any matches; that is: whether there are currently any alpha versions that match this semver version.) We're checking the length of the second index in the array. If the "array" is an array of characters then the length of the second index will be one (because a character has a length of one), but if the "array" is actually an array of versions then the length of the second index will be longer. Either way we want to get the very last part of the version so we know what the next subversion should be. If the last version published was 1.0.1-alpha.3 then we want to isolate the 3 so we can increment it to 4 and make the next version 1.0.1-alpha.4.
We're combining the base version with "-alpha." and the new subversion number to get the full new alpha version number.
We actually have to check whether the version has changed because there's a possibility it hasn't. For example, if the previous build didn't publish the package to the feed successfully then the version won't change this time around. If we just proceed as though the version has changed we actually end up with an error and our build fails.
This is changing the value in the JSON representation of the package.json that we still have saved in memory (and which we'll write back to disk in a future step).
Remember that branch variable from earlier? This is where we use it. Even though this pipeline was triggered by a specific branch, we have to checkout the branch so we can commit the modified package.json file back to it.
git needs to know a little about "us" in order to allow us to commit our changes. You can use whatever values you want right here, but you have to do this.
Just like we used cmdlets to get the contents of the package.json file into memory, we're dumping the updated JSON back into the file. I don't know what the -Depth option does. I just know that 2 works for us.
We want to revert the changes to the .npmrc file because it was modified earlier to include a key that we don't want to commit to git. This is a weird way to do this, but it's the only way that worked. I originally tried to just stage package.json, but that command failed when it ran in Azure DevOps. It was weird. This works, though. The only files that should have changed are .npmrc and package.json so just restoring .npmrc means we can commit everything else.
This is where that happens. We're staging and committing all of the remaining modified files in one command. The key part of this comment is "[skip ci]" which tells Azure DevOps not to trigger a build pipeline for this commit. If we don't include that, our changes will just keep triggering new builds forever (or until Azure DevOps gets tired of our shenanigans and quits).
Finally, we have to exit the Powershell script with a success code (0) if the version hasn't changed. If we don't do this, the whole build will fail.
This condition just specifies that this Powershell script should only run when the source (triggering) branch is not main, ready-for-deployment, release/*, or hotfix/*. In other words, only do this step for story or task branches.
Man, that was a lot. It took me a while to get all the nuances figured out. There's a nearly duplicate script that runs for the beta builds. I'm not going to put it here because it really is nearly identical to this version. Just picture all the references to "alpha" changed to "beta" and the condition requiring the branch to be release/* or hotfix/*.
If the source branch is ready-for-deployment, however, we want to strip off the "alpha" and "beta" parts of the package, which does require a different Powershell script.
A lot of this is the same so it doesn't warrant a line-by-line explanation. The big difference here is that instead of building a new version/subversion number, we're just writing the base version as the version. If the merge had "1.0.1-alpha.19" as the version, this will replace that with just "1.0.1".
Before we publish the package to the internal feed (I've omitted the build step, but you'll see it in the full file I post at the end) we want to push our changes back to the source branch of the triggering repository. That's one more Powershell script.
We have to get the source branch again and check it out again. That's just to make sure we're on the branch in case the earlier step didn't end up changing the version. The key piece is the git push line, which uses another built-in variable ($(System.AccessToken)) to authenticate to the repository. This is why we had to add the Contribute permission earlier. I've specified the -f (force) flag here because we have branch policies on some of these branches that we need to override.
This is really the last relevant step. There are three of these tasks, each with different conditions and tags. We're publishing our previously built package to our internal feed. Once again we have a condition to execute this step for all branches that are not main, ready-for-deployment, release/*, or hotfix/*. You can see on the customCommand line that we're using the --tag switch of the npm publish command to tag this package with "alpha". This enables us to install this version of the package in tundra-ui-presentation without overwriting the production-ready tundra-ui-payment package. The other two blocks replace "alpha" with "beta" and "latest", which is a keyword used by npm to identify the latest live version of the package.
The other half of this is that our tundra-ui-presentation build pipeline installs packages tagged as alpha whenever a task or story branch is built, installs packages tagged as beta whenever a release/* or hotfix/* branch is built. Otherwise, it installs the latest packages. As long as we're using semver correctly and allowing npm to get updated minor or patch versions, we shouldn't have to modify tundra-ui-presentation just to get an updated version of a library.
Here's the whole file. I did all the work so you (or future me) don't have to!
