Test Powershell code quality automatically
/Do you recognize the situation where a new team member joins the team and writes a script. During the pull request you notice they don’t follow the coding standards you where following and you have to make a lot of comments to explain all the different standards you are using?
There are several tools available for us to fix this challenge and in this blogpost I’m going to explain how you can use these tools to create a automated workflow which will test code that is written and make sure it follows some basic guidelines and best practices. All files referred to in this post can be found on github. But first let’s take a quick look at the tools we will be using (if you know these tools already you can skip to here).
PSScriptAnalyzer
PSScriptAnalyzer is a tool you can use to check your code quality. It has several rules which can also be enhanced by user or community created rules. PSScriptanalyzer is in essence a powershell module so it can be installed from the Gallery. Now say we have a small Powershell function like this.
Param(
[Parameter(Mandatory = $true)]
[string]$Name
)
Write-Host "Hello $Name"
We can use Invoke-ScriptAnalyzer -Path <path to the file> to check the file and get a overview of the rules which are not compliant. This will results in an overview like this:
Here we can see there are two rules which where not compliant. For both rules it shows which file and which line there issue occurs. It also gives a message on how to fix it. If this looks familiar to you that is possible. Programs like VSCode also have PSScriptAnalyzer build into them when the Powershell extension is installed. The ruleset used by VSCode is slightly different from the default ruleset you get from the powershell module, but often it will show the same issues in VSCode too.
PSScriptAnalyzer has a lot of customization options where some rules can be ignored. We will mostly use the default settings in this blogpost.
Pester
Pester is a Powershell Module you can use to test your code. It has several different features to help you test your code. The scripts used to test your code don’t follow the powershell syntax completly, they follow the syntax used by most testing frameworks. So if there is a function like this:
[CmdLetBinding()]
Param(
[Parameter(Mandatory = $true)]
[string]$Type
)
$year = (Get-Date).Year
Write-Output "Data $Type $year"
}
The test script can look like this:
BeforeAll {
. .\Get-TestData.ps1
}
It 'Returns a string' {
(Get-TestData -Type "Important").GetType().Name | Should -Be "String"
}
It 'Returns the right value' {
Mock 'Get-Date' {@{'Year' = '2000'}}
Get-TestData -Type "Important" | Should -Be "Data Important 2000"
}
}
It consists of several codeblocks. The main block is “Describe”, this will be a collection of tests. In this block there is a “BeforeAll” block. This runs a specific piece of code before the tests are done, in this case this is used to dotsource the function. After it there are two “It” blocks. These blocks contain the tests that need to be done. The tests contain of Powershell code that need to be executed. a value or result of this code is piped into an assertion (in these cases that’s “-Be”) which will evaluate it. And depending on the result of the evaluation the test will return successful or failed.
Pester tests can be started by running the command Invoke-Pester this will look for files with end on .tests.ps1 and process these. So running the tests above would create an output like:
Here you can see that all test where successful.
Pester has many ways to configure it to handle different situations. In this blogpost we will use some of these functions.
Combining PSScriptAnalyzer and Pester
Pester and PSScriptAnalyzer are both very powerful tools, but if we combine them we can start working the quality control into the normal testing workflows. Often testing is part of the normal workflow of writing scripts already so by adding PSScriptanalyzer into your tests you can easily enhance this. A simple way of adding PSScriptAnalyzer to your tests is by using this test:
Param(
[Parameter(Mandatory = $true)]
[string]$TestLocation
)
Describe "Testing PSSA Rules" {
It "ScriptAnalyzer" {
# Test scripts
(Invoke-ScriptAnalyzer -Path $TestLocation -Recurse).count | Should -Be 0
}
}
First you’ll notice that the test now has a parameter. In the examples provided in this blog I’ve used a parameter to make the test as easily applicable in different situations. This parameter is used to specify which files need to be checked. Adding a parameter to your tests does complicate the way Pester needs to be called. To call Pester the following code is used:
$container.Data = @{
TestLocation = $TestLocation
}
$configuration = New-PesterConfiguration
$configuration.Run.PassThru = $true
$configuration.Run.Container = $container
Invoke-Pester -Configuration $configuration
The first part creates a container. Containers describe a set of files that contain tests and specify the data associated with it. This is used to pass parameters to tests. The next part creates a configuration for pester and adds the container to it. The last line Pester is started. Since Pester 5 it’s advised to call Pester by creating a pesterconfiguration instead of using different parameters in the Invoke-Pester function.
Going back to the test you’ll see it consists of a single tests which just calls Invoke-ScriptAnalyzer and there is a count to see how many things where found by ScriptAnalyzer. If the amount is zero the test will be successful.
This test will allow you to have a fail when rules aren’t followed but it doesn’t provide much feedback so if the test fail developers often still need to run the scriptanalyzer commands to figure out what is wrong. Also it’s an all or nothing test while PSScriptAnalyzer has different severities for different rules. So it might be nice if some severities could be ignored or be set up in a way that they are reported but don’t fail the rest of the workflow.
So to tackle this I propose the following test:
Param(
[Parameter(Mandatory = $true)]
[string]$TestLocation
)
# Get all PSScript Analyzer Rules and save them in an array
$scriptAnalyzerRules = Get-ScriptAnalyzerRule
$Rules = @()
$scriptAnalyzerRules | Foreach-Object { $Rules += @{"RuleName" = $_.RuleName; "Severity" = $_.Severity } }
# Create an array of the types of rules
$Severities = @("Information", "Warning", "Error")
foreach ($Severity in $Severities) {
Describe "Testing PSSA $Severity Rules" -Tag $Severity {
It "<RuleName>" -TestCases ($Rules | Where-Object Severity -eq $Severity) {
param ($RuleName)
#Test all scripts for the given rule and if there is a problem display this problem in a nice an reabable format in the debug message and let the test fail
Invoke-ScriptAnalyzer -Path $TestLocation -IncludeRule $RuleName -Recurse |
Foreach-Object {"Problem in $($_.ScriptName) at line $($_.Line) with message: $($_.Message)" } |
Should -BeNullOrEmpty
}
}
}
This test is a bit more complex. Let’s break it down. First of all there is the same parameter as in the other test. After this the rules in PSScriptAnalyzer are retrieved and stored in an array with the severities linked to them. After this an array is defined which contains the severities known in PSScriptAnalyzer.
Now a loop is created for every severity. This loop will create a describe block for each severity. It also adds a tag to it. Tags can be used to filter specific tests when calling Pester. So this same test script can be used to either test for all test or only a specific severity.
Inside the describe block a test is defined with the -testcases parameter. This causes the test to be copied for every testcase. You notice that the name of the it block is “<RuleName>”, this is a way in Pester to use testcases in the name of the it block. So every test will be named after the rule in PSScriptanalyzer. Inside the it block the parameter is defined which contains the value form the testcase.
Now the invoke-scriptanalyzer function is called again with only a specific rule used. The output of this is piped into a loop which creates a string containing a nice readable description of the problems found by PSScriptAnalyzer. This output is checked to be null or empty. If this is the case the test was successful. But if it isn’t it will show the value of the string we created, so we can instantly see what is wrong. Calling the test will output a result like this:
If you are using the provided test.ps1 script to call pester in the github repositoy you’ll notice there is more output. Because of the usage of containers Pester will also show a summary for the container.
If we look at the output we’ve got the problems identified with the simple implementation are gone. This version of the test has the possibility to use tags to filter on a specific severity. And a developer can fix the problems purely based on the output of this test.
Control exemptions to the rules
We all know that rules and standards are nice but sometimes they just don’t fit our script exactly. For example there is a rule which states that functions should always use singular nouns in their names. But when you are working with producs like Azure Log Analytics and you want to write a function for this it’s possible PSScriptAnalyzer will detect the “Analytics” part of the name and say it’s a plural noun and therefore shouldn’t be used.
To combat this problem PSScriptAnalyzer has an option to ignore some rules. By adding a specific line of code at the top of your script you can suppress a specific rule. The line looks like this:
The first parameter in this line is the rule which is suppressed, so in the example that is “PSAvoidTrailingWhitespace”. The third parameter is the reason for suppressing it. It’s important that this is entered so other developers also know why this rule is being suppressed.
Adding the tests to your workflow
There is now a nice script to test if the code is of high quality but we want to make sure that these tests are always run. There are several ways of achieving this but the way I propose to do it is via a branch policy with build validation.
If you are using Azure DevOps you can set up a branch with policies, one of these policies is called “build validation”, this will run a pipeline, and only if the pipeline was successful will it allow for a pull request to be merged with this branch. So to use this we need to create a pipeline. This pipeline will test the code and report if a test failed.
- none
pool:
vmImage: 'windows-latest'
steps:
- task: PowerShell@2
inputs:
filePath: 'test.ps1'
arguments: "-Type advanced -OutputResults"
pwsh: true
failOnStderr: false
continueOnError: true
- task: PublishTestResults@2
inputs:
testResultsFormat: 'NUnit'
testResultsFiles: 'pssa.testresults.xml'
mergeTestResults: true
failTaskOnFailedTests: true
testRunTitle: 'PSScriptAnalyzer tests'
condition: in(variables['Agent.JobStatus'], 'Succeeded', 'SucceededWithIssues', 'Failed')
The pipeline consists of two tasks. The first task if a simple powershell script which is run. The script will call pester with the right arguments. It will also generate an xml file which contains the test result. The whole test script looks like this:
[Parameter(Mandatory = $false)]
[ValidateSet('simple','advanced')]
[string]$Type = "simple",
[Parameter(Mandatory = $false)]
[string]$TestLocation = ".\src\default",
[Parameter(Mandatory = $false)]
[switch]$OutputResults
)
# initialize
.\tests\init\init.ps1
# test
$pestercommand = Get-Command Invoke-Pester
"`n`tSTATUS: Testing with PowerShell $($PSVersionTable.PSVersion.ToString())"
"`tSTATUS: Testing with Pester $($pestercommand.version)`n"
$container = New-PesterContainer -Path (Join-Path -Path $PSScriptRoot -ChildPath "tests/$Type")
$container.Data = @{
TestLocation = $TestLocation
}
$configuration = New-PesterConfiguration
$configuration.Run.PassThru = $true
$configuration.Run.Container = $container
if ($PSBoundParameters.OutputResults.IsPresent) {
# Outputting to file when running in a pipeline
$configuration.TestResult.Enabled = $true
$configuration.TestResult.OutputPath = "pssa.testresults.xml"
}
Invoke-Pester -Configuration $configuration
In the init script it will check if the required modules are installed and if not it will install them. After that there is a bit of output to show which version of powershell and pester are being used. Then there is the part we looked at before to call pester with the only exception that it now also shows an if statement where the outputpath for the testresults is defined. This will generate the xml file.
In the yaml pipeline the second task will take this xml file which is generated and create a nice report from it, this report is then uploaded to azure devops to show in the pipeline. It looks like this:
In this test report the same information is shown was when we where using the terminal. If you click on the rules which failed it will show in which file(s) on which line(s) the error(s) are.
The pipeline step has also an attribute called “failTaskOnFailedTests”, if this is set to true it will fail the pipeline once it sees that there are failed tests. In the example this is the case but as discussed before you can also add tags to the pester call, and if you do you could make a situation where you check for the different severity levels and publish them all separately. And for example you could have the pipeline only fail if there where warning and error problems found by PSScriptAnalyzer while information problems would still be logged but not breaking the flow.
To set up the build validation you go to the branches in azure devops and select the branch policies, here you add a build validation.
Now if someone makes a pull request to your branch it will trigger the pipeline to run and only if the pipeline is successful will they be able to continue. So this achieves the goal we set out to begin with. Now there is a automated workflow to check the quality of the code written. With the help of custom PSScriptAnalyzer rules and Pester Tags can this solution be even more enhanced.
Hopefully this will help you with having less stress over code reviews, and I hope to see you in my next blogpost!