Building a full MSP automation workflow in Azure DevOps

/

This week we had a hackaton at my employer OGD ICT-diensten. The goal of this hackaton was to get some more work done on our automation efforts. I’ve been very hands on with this project from the start in both a developer as more of a consulting role. In this blog I want to tell about how we are using Azure DevOps to facilitate a complete automation workflow to serve different customers.

As an MSP we are delivering different services to customers. To make sure we can do our work as quick as possible while making as few errors as possible we use automation to do most of the changes to the environments. In the past many teams had their own way of doing this automation and we had multiple different collections of scripts to run these automations. So we decided to centralize this all into what we’ve called “Project Omnibus”.

The Goals of Project Omnibus is to deliver many standardized automations for our customers while making sure the platform is reliable. To achieve this we’ve incorporated many of the features of Azure DevOps to reach our goal. A simplified version of our setup is as follows:

The goals we set out to accomplish where:

  • All effort which is put into automation has to be done in a way where as many customers can profit from it.

  • The solution needs to be scalable

  • Changes to automation need to be controlled so they wont bring down automation

  • Everything has to be set up in a way where as many people can easily contribute

The webportal

The web portal is a custom made solution by our inhouse developers. They made a front-end interface for the azure DevOps pipelines. It will read the YAML files and generate a form based on this. By using extra parameters we are able to also make fields like “devicepickers” or “userpickers” which are tied to the environments and can be used to make sure the input is valid. When a user has entered the right information and submits the form it will be send to azure DevOps and queued as a pipeline run.

The pipelines

In the picture above you’ll see a simplified view of our setup. In reality we have several different azure DevOps projects which are being synced by an inhouse tool to make sure that all the pipelines in a main repository are synced to the different projects for our customers. The pipelines will be automatically set up so they can be ran. But for the sake of this blogpost we could just see it as one big project containing some pipelines for a customer. The pipelines are written in YAML and stored in a repository so version control can be applied. We also use templates in our pipelines to make sure there is not so much code reuse. One of the templates you’ll see is the invoke-PowerShell template. This is a template made by us to run PowerShell scripts while the modules we need are loaded up already.

The problem we had was that many of our scripts required different PowerShell modules. Some where our own modules and some where Microsoft managed PowerShell modules. We didn’t want to have an install and import section at the start of every script which was ran in the pipelines. So we’ve made a pipeline template. We could have made our own pipeline task, but to make sure it was easier for other engineers to maintain it we opted to write this as a YAML template. In the main pipeline the template is called with the script you want to run and an array of modules to install. The template will then go through these different steps:

  1. It will do some validation on the given parameters to check if they are valid

  2. It will use iterative insertion to create a loop for every module in the given array

  3. It will first check on the agent if the required module is already installed

  4. It will check if the module is available in the pipeline cache and if so it will download it from there

  5. It compares the name with a list of internal PowerShell modules and if it’s in this list it will download it from the internal artifact feed.

  6. If step 4 and 5 both don’t result in a download it will download the module from the PSGallery.

  7. It will check if the module has any dependencies and if so it will start at step 3 again for this dependency

  8. It will loop through all modules downloaded and add them to the PSModulePath.

  9. It will now run the provided script by passing the PSModulePath as an environmental variable to the task.

This way developers of the pipelines can easily list all modules required. All modules need to be version pinned in the pipeline and the different download tasks will keep versions in mind and only download this version. This way we know for sure the automation will keep working. Once a while we will upgrade the version in the pipelines and run it through all the tests to see if it still works and if so push this to production again.

The CI/CD Pipelines

Our project consist of pipelines which have small scripts and internal PowerShell modules. The PowerShell modules are build by the CI/CD pipelines every time a change is done. I’ve written in the past about testing the code quality by using azure DevOps pipelines. The CI/CD pipelines used in this project are more complex, next to testing for code quality they also have unit testing for all functions and different aspects (like testing if all necessary files are present etc.). The pipelines will eventually create a NuGet package which will be uploaded to the internal Azure DevOps Artifact Feed. During the hackaton I was mostly working on two different things we are now improving on these CI/CD pipelines.

Using Git Submodules to have less code reuse

Most of our modules had scripts which where present in the different modules. These scripts where common tests and scripts to start the testing etc. During the hackaton we implemented the use of git submodules. By creating a repository which housed the common scripts we could refer to this repository in the module repositories and have it use these files instead. This way we know for sure they will always be the same over all the modules. In the azure devops pipelines we did need to change our checkout steps to include the submodules because else they weren’t checked out during the ci/cd pipelines. Also we had to change our work instructions to make sure every contributor does an init of the submodules after cloning the repository.

Creating a test environment for integration testing

One of the challenges we are still facing is how to do proper integration testing. At the moment most of our automation is being tested by unit tests. We are trying to keep our code coverage as high as possible but this doesn’t guarantee everything will work. So during the hacking I’ve wrote up a proposal where we will set up a representative testing environment to actually do integration testing. I’ve written about using integration testing before in policies. This wouldn’t be much different. The tests would see the pipelines being triggered to an testing environment and then other code would check if the changes are being done correct.

Tying it all together

With the modules being deployed to Azure DevOps Artifact Feeds and the template installing these modules automatically we can have our simple PowerShell scripts which perform tasks. We try to make sure that as many as the tasks call our internal modules. We sometimes still call external modules but this should change in the future. Our end goal would be to have a abstraction layer between the scripts which define what to do and the code which actually executes it. This way if any module or api gets deprecated we only have to change our internal modules to look to the new modules or api’s. This would make developing for other engineers a lot easier.

In the end everything will run on dedicated azure DevOps agents which run in the environment of the customer. From there it can access the different resources needed and perform the changes which need to be made.

Conclusion

Setting up an automation workflow is always hard. Especially when many different developers are working on it and many different customers need to be taken into account. Also lately with all the news about supply chain attacks it becomes even more important to keep tight grips on your automation workflows. One of the improvements we still want to include is to get our modules from an intermediary artifact feed instead of directly from the PSGallery. I have been experimenting with this but encountered some errors which are outlined here. Hopefully soon I can write a post on how to get that to work too!

I hoped you enjoyed this little glimpse into our way of automation. Sorry that things are kept a bit vague, if you would love to hear more detailed explanations on certain aspects contact me on twitter or comment on this post and I will see what I can do about that!