When working on your precious pipeline templates that you write in YAML, it can be a burdensome to document each parameter individually. Documentation is also overlooked, as you always seem to be busy with other things, but eventually you’ll be glad you did it. What if you can make it more fun and easier to do?
In this blog, you will learn to automatically generate your documentation with a simple technique and dump it out in Markdown.
Pre-requisites
In this blog, you will only need the following Powershell module available:
- The InvokeBuild Powershell module v5.9+
You’ll see me use Visual Studio Code as editor. It is not required to have it, but some extensions on writing Markdown help out in formatting.
Parsing YAML
Let’s assume that you’ve already have your repository checked out. Inside the repository, you have a folder called src that stores your YAML files. You have structured it nicely, so your jobs and task are separated. A good example would be:
Now, you want to read each parameter that is defined in your YAML files, so you can document each parameter. You are going to setup a task that is responsible for fetching the YamlDotnet library that is responsible for parsing the YAML. Let’s get started.
- Create a file .build.ps1 in the root of the directory
- Add the following content to the file
[CmdletBinding()]
param (
$LibPath = "$BuildRoot\lib",
$DocsDirectory = "$BuildRoot\docs",
$TemplateDirectory = "$BuildRoot\src"
)
task Init {
if (-not (Test-Path $LibPath)) {
New-Item -Path $LibPath -ItemType Directory
}
$PackagePath = Join-Path $LibPath -ChildPath 'YamlDotNet.11.2.1.nupkg'
if (-not (Test-Path $PackagePath)) {
$Uri = 'https://www.nuget.org/api/v2/package/YamlDotNet/11.2.1'
Write-Build Yellow "Downloading YamlDotnet library"
try {
Invoke-WebRequest -Uri $Uri -OutFile $PackagePath
}
catch {
Write-Error "Error occurred: $_"
}
Write-Build Yellow "Extracting YamlDotNet package"
# load ZIP methods
Add-Type -Assembly System.IO.Compression.FileSystem
# open ZIP archive for reading
$ZipFile = [IO.Compression.ZipFile]::OpenRead($PackagePath)
#Find all files in ZIP that match the filter (i.e. file extension)
#Use the Entries property to retrieve the entire collection of entries
$Entry = $ZipFile.Entries | Where-Object { $_.FullName -eq 'lib/net45/YamlDotNet.dll' }
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($Entry, (Join-Path $LibPath -ChildPath 'YamlDotNet.dll'))
# Dispose
$ZipFile.Dispose()
# Make sure the dll is loaded in session
Add-Type -Path "$LibPath\YamlDotNet.dll"
}
}- Run Invoke-Build in your terminal
You should now see an extra folder called lib that contains the library.
The Add-Type -Path "$LibPath\YamlDotNet.dll" line is responsible for loading the library into your Powershell session. Sweet, with the library now in-place, you can start parsing the YAML files. Let’s create another task to generate the docs
Generating documentation
- Add the following content under the Init task
task GenerateDocs {
function ConvertFrom-Yaml {
[CmdletBinding()]
param
(
[parameter(Position = 0, Mandatory = $false, ValueFromPipeline = $true)]
$YamlString,
[parameter(Position = 1, Mandatory = $false, ValueFromPipeline = $false)]
$Path
)
BEGIN {
function ConvertFrom-YAMLDocument {
[CmdletBinding()]
param (
[object]$TheNode #you pass in a node that, when you call it, will be the root node.
)
#initialise variables that are needed for providing the correct powershell data type for a string-based value.
[bool]$ABool = $false
[int]$AnInt = $null
[long]$ALong = $null
[decimal]$adecimal = $null
[single]$ASingle = $null
[double]$ADouble = $null
[datetime]$ADatetime = '1/1/2000'
$TheTypeOfNode = $TheNode.GetType().Name # determine this
Write-Verbose "$TheTypeOfNode = $($theNode)" #just so see what is going on
$Style = $TheNode.Style
$Tag = $TheNode.Tag
$Anchor = $TheNode.Anchor
Write-Verbose "Tag=$tag, Style=$style, Anchor=$anchor"
#if it is the document, then call recursively with the root node
if ($TheTypeOfNode -eq 'YamlDocument') {
$TheObject = ConvertFrom-YAMLDocument $TheNode.RootNode
}
elseif ($TheTypeOfNode -eq 'YamlMappingNode') {
#ah mapping nodes
$TheObject = [ordered]@{ }
$theNode | ForEach-Object {
$TheObject.($_.Key.Value) = ConvertFrom-YAMLDocument $_.Value
}
}
elseif ($TheTypeOfNode -eq 'YamlScalarNode' -or $TheTypeOfNode -eq 'Object[]') {
$value = "$($theNode)"
if (! $tag) {
$value = switch -Regex ($value) {
# if it is one of the allowed boolean values
'(?i)\A(?:on|yes)\z' {
'true'
break
} #Deal with all the possible YAML booleans
'(?i)\A(?:off|no)\z' {
'false'
break
}
default { $value }
}
}
$TheObject = if ($tag -ieq 'tag:yaml.org,2002:str') { [string]$Value } #it is specified as a string
elseif ($tag -ieq 'tag:yaml.org,2002:bool') { [bool]$Value } #it is specified as a boolean
elseif ($tag -ieq 'tag:yaml.org,2002:float') { [double]$Value } #it is specified as adouble
elseif ($tag -ieq 'tag:yaml.org,2002:int') { [int]$Value } #it is specified as a int
elseif ($tag -ieq 'tag:yaml.org,2002:null') { $null } #it is specified as a null
elseif ($tag -ieq 'tag:yaml.org,2002:timestamp') { [datetime]$Value } #it is date/timestamp
elseif ($tag -ieq 'tag:yaml.org,2002:binary') { [System.Convert]::FromBase64String($Value) }
elseif ([int]::TryParse($Value, [ref]$AnInt)) { $AnInt } #is it a short integer
elseif ([bool]::TryParse($Value, [ref]$ABool)) { $ABool } #is it a boolean
elseif ([long]::TryParse($Value, [ref]$ALong)) { $ALong } #is it a long integer
elseif ([decimal]::TryParse($Value, [ref]$ADecimal)) { $ADecimal } #is it a decimal
elseif ([single]::TryParse($Value, [ref]$ASingle)) { $ASingle } #is it a single float
elseif ([double]::TryParse($Value, [ref]$ADouble)) { $ADouble } #is it a double float
elseif ([datetime]::TryParse($Value, [ref]$ADatetime)) { $ADatetime } #is it a datetime
else { [string]$Value }
}
elseif ($TheTypeOfNode -eq 'Object[]') {
#sometimes you just get a raw object, not a node
$TheObject = $theNode.Value #so you return its value
}
elseif ($TheTypeOfNode -eq 'YamlSequenceNode') {
#in which case you
$TheObject = @()
$theNode | ForEach-Object {
$TheObject += ConvertFrom-YAMLDocument $_
}
return , $TheObject
}
else {
Write-Verbose "Unrecognized token $TheTypeOfNode"
}
Return $TheObject
}
}
PROCESS {
try {
If ($Path) {
$streamReader = [System.IO.File]::OpenText($Path)
}
Else {
$streamReader = new-object System.IO.StringReader([string]$yamlString)
}
$yamlStream = New-Object YamlDotNet.RepresentationModel.YamlStream
$yamlStream.Load([System.IO.TextReader]$streamReader)
ConvertFrom-YAMLDocument ($yamlStream.Documents[0])
}
Catch {
Write-Error $_
}
Finally {
if ($streamReader.Basestream -ne $null) {
$streamReader.Close()
}
}
}
END {}
}
try {
Write-Build Yellow "Start documentation generation for folder $DocsDirectory"
if (!(Test-Path $DocsDirectory)) {
Write-Build Yellow "Output path does not exists creating the folder: $($DocsDirectory)"
New-Item -ItemType Directory -Force -Path $DocsDirectory
}
$Templates = Get-ChildItem -Path $TemplateDirectory -Filter "*.yml" -Recurse
$InputObject = @()
foreach ($Template in $Templates) {
Write-Build Yellow "Generating documentation for $($Template.FullName)"
$TemplateContent = Get-Content $Template.FullName -Raw -Force
$TemplateYaml = ConvertFrom-Yaml -YamlString $TemplateContent
if (!$TemplateYaml) {
Write-Error "YAML is invalid, please specify the correct YAML content"
}
$Header = $Template.DirectoryName.Substring($TemplateDirectory.Length) + "\$($Template.BaseName).yml"
# Create a Parameter List Table
$parameterHeader = "| Parameter Name | Parameter Type | Parameter description | Default Value | Example |"
$parameterHeaderDivider = "| --- | --- | --- | --- | --- | "
$parameterRow = " | {0}| {1} | {2} | {3} | | "
$StringBuilderParameter = @()
$StringBuilderParameter += "## $Header"
$StringBuilderParameter += [System.Environment]::NewLine
$StringBuilderParameter += $parameterHeader
$StringBuilderParameter += $parameterHeaderDivider
$StringBuilderParameter += $TemplateYaml.parameters | ForEach-Object {
$parameterRow -f $_.Name, $_.Type, $_.DisplayName, $_.Default
}
$StringBuilderParameter += [System.Environment]::NewLine
$InputObject += $StringBuilderParameter
}
$InputObject | Out-File -FilePath $DocsDirectory\README.md -Append
}
catch {
Write-Error $_.Exception.Message
}
}- Under the last task, add the following line
task . Init, GenerateDocsto default both task whenInvoke-Buildis called
Inside the task itself, you’ve a function that is responsible for the parsing of the YAML. The New-Object YamlDotNet.RepresentationModel.YamlStream streams all the data accordingly and you’ll get back a nice hashtable representing the YAML. With the hashtable available, it is easy to create the documentation, so let’s test it out.
- Create file copy-files-task.yml in the task directory if not done already
- Add the following parameters to the file and save it
parameters:
- name: content
type: string
default: **
displayName: 'The content to be copied'
- name: sourceFolder
type: string
default: $(System.DefaultWorkingDirectory)
displayName: 'Folder that contains the files you want to copy'
- name: targetFolder
type: string
default: $(Build.ArtifactStagingDirectory)
displayName: Target folder or UNC path files will copy to- Run
Invoke-Buildin your terminal
The documentation directory is created and the copy-files-task.yml is documented nicely into a table.
Very cool. Now for each file that is introduced into the src directory, when running the Invoke-Build cmdlet, the documentation is generated automatically. You have to keep in mind, that all parameters are defined correctly to reflect it back into the documentation.
Related
Conclusion
It was a small blog post, but extremely useful to keep your documentation up to date. Is it possible for you to integrate this in an Azure Pipeline and post it back? A nice challenge to do!