Building Azure DevOps REST API extension

Pre-requisites
Preparing folder structure
Creating the extension
- Adding VstsTaskSdk module
Subscribe Google Translate API
Building Google Translator Powershell module
Create task Powershell script
Finishing the extension
- Creating publisher
Create and publish the extension task
Installing the extension task
Consuming the extension
Conclusion

Before starting the tutorial, I want to give a huge thanks to Jaap Brasser for showing me this technique during the Powershell DevOps Global Summit 2022. I would highly encourage you to have a look into his video on “abstracting your Powershell code” to gain more in-depth knowledge on using the technique itself. Most code in this tutorial is modified that comes from his Rubrik SDK. Again, thank you for showing this technique.

Let’s start the tutorial by going through the pre-requisites as the title says enough!

Pre-requisites

An Azure DevOps account
A code editor like Visual Studio (VSCode)
A Azure (Git) Repo, google-api-extension repository is used in this tutorial
A RapidApi account, you are going to use the Google Translate API to translate languages
NodeJS installed

Preparing folder structure

Before you begin, you’ll need a set of files and folders that form the basis of your extension. Let’s create them manually. Alternatively, if you want, you can also use the tfx build tasks create command to create the same structure.

Open your repository
Create folder cicd and add a file azure-pipelines.yml to the folder
Create folder Extensions that will house your extensions, in this case the Google Translator task
Inside the Extension folder, create folder GoogleTranslatorTask
Create folder images which stores the images to be displayed for your extension on the marketplace
Create a file vss-extension.json in the root of the directory

The vss-extension.json defines the basic information about the extension. Later in the tutorial, you are going to define this information. Let’s start by defining the extension itself in the GoogleTranslatorTask folder.

Creating the extension

In this section, you’ll be adding the task.json file that contains the metadata of the task itself and will reflect on the UI of Azure DevOps. The information that is contained in here, can be used to retrieve the values you are going to need to invoke the REST API. You’re also going to add the VstsTaskSdk for Powershell. Let’s move!

In the GoogleTranslatorTask folder, create a file task.json
Add the content in the file

{
  "id": "",
  "name": "google-translator-api-extension",
  "friendlyName": "Google Translator API extension",
  "description": "This task calls the Google Translator API extension",
  "author": "",
  "category": "Utility",
  "visibility": ["Build", "Release"],
  "runsOn": ["Agent"],
  "demands": [],
  "version": {
    "Major": "0",
    "Minor": "1",
    "Patch": "0"
  },
  "minimumAgentVersion": "1.95.0",
  "instanceNameFormat": "Google Translator API extension",
  "groups": [
    {
      "name": "translateOptions",
      "displayName": "Translate options",
      "isExpanded": false
    }
  ],
  "inputs": [
    {
      "name": "url",
      "type": "string",
      "label": "Endpoint address",
      "defaultValue": "",
      "required": true,
      "helpMarkDown": "Specify the endpoint URL to Google Translator API"
    },
    {
      "name": "token",
      "type": "string",
      "label": "Token",
      "required": true,
      "defaultValue": "",
      "helpMarkDown": "Provide the token to connect Google Translator API"
    },
    {
      "name": "target",
      "type": "string",
      "label": "Target language",
      "required": true,
      "defaultValue": "",
      "helpMarkDown": "Provide the target language to translate to",
      "groupName": "translateOptions"
    },
    {
      "name": "query",
      "type": "string",
      "label": "Text translation",
      "required": true,
      "defaultValue": "",
      "helpMarkDown": "Provide the text to translate",
      "groupName": "translateOptions"
    },
    {
      "name": "source",
      "type": "string",
      "label": "Source language",
      "required": false,
      "defaultValue": "",
      "helpMarkDown": "Provide the source language",
      "groupName": "translateOptions"
    }
  ],
  "execution": {
    "PowerShell3": {
      "target": "task.ps1"
    }
  }
}

Fill in the id with a valid GUID

You can create a new GUID simply in a Powershell terminal by using ([guid]::newguid()).Guid

Make sure you have the author filled in representing you

You are going to upload the extension later in the marketplace privately, but to give you a sneak peak, it will look in the UI as followed:

Adding VstsTaskSdk module

As mentioned earlier, you’ll be needing the VstsTaskSdk module in your project. You are going to add this in the ps_modules folder inside the GoogleTranslatorTask folder.

Open a Powershell terminal
Save the module by entering Save-Module –Name VstsTaskSdk –Path .\Extensions\GoogleTranslatorTask\ps_modules –Force in the root of the project

You notice that there is a version number included. Make sure that you remove this subfolder, and copy over all the content directly under the VstsTaskSdk folder. A picture always shows more.

Make sure you do this, else the SDK will not load appropriately when the task is executing.

You are going to need an API to test against in your hosted agent. In this tutorial, the Google Translate API is a going to be used, as this API is free for the first 500 calls on RapidAPI. Let’s subscribe to the API, so you can later build your module around it.

Login to RapidAPI.com
Open Google Translate API
In the top-corner, press subscribe

If you’ve successfully subscribed, you’ll notice that you now have the X-RapidKey-Key header value available to you.

You are going to need this key to add to the calls you are going to make through Azure DevOps. Make note of it and move a long to the next section where you’re going to create a Powershell module to be used in the hosted agent.

Building Google Translator Powershell module

It is time to create the Google Translator Powershell module and include it in the ps_modules folder. The module will be responsible for calling the Google Translator API. In the RapidAPI page, you’ve seen that there are 3 endpoints exposed. In this tutorial, you are going to focus on the translate endpoint to translate languages.

Create file GoogleTranslator.psm1 in the root of the ps_modules folder

Add the Connect-GoogleTranslator function

Function Connect-GoogleTranslator
{
    [CmdletBinding()]
    Param (
        [Parameter(Mandatory = $true, Position = 0)]
        [ValidateNotNullorEmpty()]
        [String]$Server,

        [Parameter(Mandatory = $true, Position = 1)]
        [ValidateNotNullOrEmpty()]
        [String]$Token
    )

    Begin
    {
        try
        {
            if ([Net.ServicePointManager]::SecurityProtocol -notlike '*Tls12*')
            {
                Write-Verbose -Message 'Adding TLS 1.2'
                [Net.ServicePointManager]::SecurityProtocol = ([Net.ServicePointManager]::SecurityProtocol).tostring() + ', Tls12'
            }
        }
        catch
        {
            Write-Verbose -Message $_
            Write-Verbose -Message $_.Exception.InnerException.Message
        }

        $function = $MyInvocation.MyCommand.Name

        Write-Verbose -Message "Gather API Data for $function"
    }

    Process
    {
        $head = @{'X-RapidAPI-Key' = $Token; 'X-RapidAPI-Host' = $Server; }
        Write-Verbose -Message 'Storing all connection details into $global:GoogleApiTranslater'
        $global:GoogleApiTranslater = @{
            server   = $Server
            header   = $head
            time     = (Get-Date)
            authType = 'Token'
        }
    }
}

The Google Translator API does not have an endpoint exposed to test out the authentication, thus store it in a variable for later usage

Add the Get-GoogleLanguageTranslate function

Function Get-GoogleLanguageTranslate
{
    [CmdletBinding(SupportsShouldProcess)]
    Param (
        [Parameter(Mandatory = true, Position = 0)]
        [string]$Target,

        [Parameter(Mandatory = true, Position = 1)]
        [Alias("Query")]
        [string]$Q,

        [Parameter(Mandatory = $false, Position = 1)]
        [string]$Source,

        [Parameter(Mandatory = $false)]
        $Server = $Global:GoogleApiTranslater.server
    )

    Begin
    {
        Test-GoogleTranslater

        $function = $MyInvocation.MyCommand.Name

        Write-Verbose -Message "Gather API Data for $function"
        $resources = Get-GoogleTranslatorApiData -endpoint $function
        Write-Verbose -Message "Load API data for $($resources.Function)"
        Write-Verbose -Message "Description: $($resources.Description)"
    } 
    Process
    {
        $uri = New-URIString -server $Server -endpoint ($resources.URI) -id $id
        $body = Test-QueryParam -querykeys ($resources.Query.Keys) -parameters ((Get-Command $function).Parameters.Values)
        $body = $body.Replace($($body[0]).ToString(), '')
        $result = Submit-Request -uri $uri -header $Header -method $($resources.Method) -body $body

        return ($result.data.translations.translatedText)
    }
}

If you have noticed, there first 3 parameters have the reflection of parameters as requested body on the RapidAPI page. While the target and q parameters are mandatory, you can optionally also give the source language parameter to translate from.

Let’s add the abstract helper functions in the module. If you want to learn more about it, definitely look into the introduction credits.

Add the following helper functions

Function Test-GoogleTranslater()
{
    Write-Verbose -Message 'Validate the Google Translator token exist'
    if (-not $global:GoogleApiTranslater.header)
    {
        Write-Warning -Message 'Please connect to only one Google Translator before running this command.'
        throw 'A single connection with Connect-GoogleTranslator is required.'
    }
    Write-Verbose -Message 'Found a Google Translator token for authentication'
    $script:Header = $global:GoogleApiTranslater.header
}

Function Get-GoogleTranslatorApiData
{
    [CmdletBinding()]
    Param (
        $Endpoint
    )

    $Api = @{
        'Get-GoogleLanguageTranslate' = @{
            '1.0' = @{
                Description = 'Retrieve translation from Google Translate API'
                URI         = '/language/translate/v2'
                Method      = 'Post'
                Body        = ''
                Query       = @{
                    target = 'target'
                    q      = 'q'
                    source = 'source'
                }
                Result      = ''
                Filter      = ''
                Success     = '200'
            }
        }
    }

    $key = '1.0'

    Write-Verbose -Message "Selected $key API Data for $endpoint"
    $api.$endpoint.$key.Add('Function', $endpoint)
    return $api.$endpoint.$key
}

Function New-URIString($server, $endpoint)
{
    Write-Verbose -Message 'Build the URI'
    $uri = 'https://' + $server + $endpoint
    Write-Verbose -Message "URI = $uri"

    return $uri
}

Function Test-QueryParam($querykeys, $parameters, $uri)
{
    Write-Verbose -Message "Build the query parameters for $(if ($querykeys){$querykeys -join ','}else{'<null>'})"
    $querystring = @()
    # Walk through all of the available query options presented by the endpoint
    # Note: Keys are used to search in case the value changes in the future across different API versions
    foreach ($query in $querykeys)
    {
        # Walk through all of the parameters defined in the function
        # Both the parameter name and parameter alias are used to match against a query option
        # It is suggested to make the parameter name "human friendly" and set an alias corresponding to the query option name
        foreach ($param in $parameters)
        {
            # If the parameter name matches the query option name, build a query string
            if ($param.Name -eq $query)
            {
                $querystring += Test-QueryObject -object (Get-Variable -Name $param.Name).Value -location $resources.Query[$param.Name] -params $querystring
            }
            # If the parameter alias matches the query option name, build a query string
            elseif ($param.Aliases -eq $query)
            {
                $querystring += Test-QueryObject -object (Get-Variable -Name $param.Name).Value -location $resources.Query[$param.Aliases] -params $querystring
            }
        }
    }
    # After all query options are exhausted, build a new URI with all defined query options

    $uri = New-QueryString -query $querystring -uri $uri
    Write-Verbose -Message "URI = $uri"

    return $uri
}
Function Test-QueryObject($object, $location, $query)
{
    <#
      .SYNOPSIS
      Builds a query string for an endpoint
      .DESCRIPTION
      The Test-QueryObject function is used to build a custom query string for supported endpoints
      .PARAMETER object
      The parent function's variable holding the user generated query data

      .PARAMETER location
      The key/value pair that contains the correct query name value

      .PARAMETER params
      An array of query values that are added based on which $objects have been passed by the user
    #>

    Write-Debug -Message ($PSBoundParameters | Out-String)

    if ((-not [string]::IsNullOrWhiteSpace($object)) -and ($location))
    {
        return "$location=$object"
    }
}

Function New-QueryString($query, $uri, $nolimit)
{
    # TODO: It seems like there's a more elegant way to do this logic, but this code is stable and functional.
    foreach ($_ in $query)
    {
        # The query begins with a "?" character, which is appended to the $uri after determining that at least one $params was collected
        if ($_ -eq $query[0])
        {
            $uri += '?' + $_
        }
        # Subsequent queries are separated by a "&" character
        else
        {
            $uri += '&' + $_
        }
    }
    return $uri
}

Function New-BodyString($bodykeys, $parameters)
{
    <#
      .SYNOPSIS
      Function to create the body payload for an API request
      .DESCRIPTION
      This function compares the defined body parameters within Get-RubrikAPIData with any parameters set within the invocation process.
      If matches are found, a properly formatted and valid body payload is created and returned.
      .PARAMETER bodykeys
      All of the body options available to the endpoint

      .PARAMETER parameters
      All of the parameter options available within the parent function
    #>

    # If sending a GET request, no body is needed
    if ($resources.Method -eq 'Get')
    {
        return $null
    }

    # Look at the list of parameters that were set by the invocation process
    # This is how we know which params were actually set by the call, versus defaulting to some zero, null, or false value
    # We can also add any custom variables here, such as SLAID which is populated after the invocation resolves the name
    if ($slaid -and $PSCmdlet.MyInvocation.BoundParameters.ContainsKey('SLAID'))
    {
        $PSCmdlet.MyInvocation.BoundParameters.SLAID = $slaid
    }
    elseif ($slaid)
    {
        $PSCmdlet.MyInvocation.BoundParameters.Add('SLAID', $slaid)
    }

    # Now that custom params are added, let's inventory all invoked params
    $setParameters = $pscmdlet.MyInvocation.BoundParameters
    Write-Verbose -Message "List of set parameters: $($setParameters.GetEnumerator())"

    Write-Verbose -Message 'Build the body parameters'
    $bodystring = @{ }
    # Walk through all of the available body options presented by the endpoint
    # Note: Keys are used to search in case the value changes in the future across different API versions
    foreach ($body in $bodykeys)
    {
        Write-Verbose "Adding $body..."
        # Array Object
        if ($resources.Body.$body.GetType().BaseType.Name -eq 'Array')
        {
            $bodyarray = $resources.Body.$body.Keys
            $arraystring = @{ }
            foreach ($arrayitem in $bodyarray)
            {
                # Walk through all of the parameters defined in the function
                # Both the parameter name and parameter alias are used to match against a body option
                # It is suggested to make the parameter name "human friendly" and set an alias corresponding to the body option name
                foreach ($param in $parameters)
                {
                    # If the parameter name or alias matches the body option name, build a body string

                    if ($param.Name -eq $arrayitem -or $param.Aliases -eq $arrayitem)
                    {
                        # Switch variable types
                        if ((Get-Variable -Name $param.Name).Value.GetType().Name -eq 'SwitchParameter')
                        {
                            $arraystring.Add($arrayitem, (Get-Variable -Name $param.Name).Value.IsPresent)
                        }
                        # All other variable types
                        elseif ($null -ne (Get-Variable -Name $param.Name).Value)
                        {
                            $arraystring.Add($arrayitem, (Get-Variable -Name $param.Name).Value)
                        }
                    }
                }
            }
            $bodystring.Add($body, @($arraystring))
        }

        # Non-Array Object
        else
        {
            # Walk through all of the parameters defined in the function
            # Both the parameter name and parameter alias are used to match against a body option
            # It is suggested to make the parameter name "human friendly" and set an alias corresponding to the body option name
            foreach ($param in $parameters)
            {
                # If the parameter name or alias matches the body option name, build a body string
                if (($param.Name -eq $body -or $param.Aliases -eq $body) -and $setParameters.ContainsKey($param.Name))
                {
                    # Switch variable types
                    if ((Get-Variable -Name $param.Name).Value.GetType().Name -eq 'SwitchParameter')
                    {
                        $bodystring.Add($body, (Get-Variable -Name $param.Name).Value.IsPresent)
                    }
                    # All other variable types
                    elseif ($null -ne (Get-Variable -Name $param.Name).Value -and (Get-Variable -Name $param.Name).Value.Length -gt 0)
                    {
                        # These variables will be cast to upper or lower, depending on what the API endpoint expects
                        $ToUpperVariable = @('Protocol')
                        $ToLowerVariable = @('')

                        if ($body -in $ToUpperVariable)
                        {
                            $bodystring.Add($body, (Get-Variable -Name $param.Name).Value.ToUpper())
                        }
                        elseif ($body -in $ToLowerVariable)
                        {
                            $bodystring.Add($body, (Get-Variable -Name $param.Name).Value.ToLower())
                        }
                        else
                        {
                            $bodystring.Add($body, (Get-Variable -Name $param.Name).Value)
                        }
                    }
                }
            }
        }
    }

    # Store the results into a JSON string
    if (0 -ne $bodystring.count)
    {
        $bodystring = ConvertTo-Json -InputObject $bodystring
        Write-Verbose -Message "Body = $bodystring"
    }
    else
    {
        Write-Verbose -Message 'No body for this request'
    }
    return $bodystring
}

function Submit-Request
{
    <#
        .SYNOPSIS
        Sends data to an endpoint and formats the response
        .DESCRIPTION
        This is function is used by nearly every cmdlet in order to form and send the request off to an API endpoint.
        The results are then formated for further use and returned.
        .PARAMETER uri
        The endpoint's URI

        .PARAMETER header
        The header containing authentication details

        .PARAMETER method
        The action (method) to perform on the endpoint
        .PARAMETER body
        Any optional body data being submitted to the endpoint
    #>


    [cmdletbinding(SupportsShouldProcess = $true)]
    param(
        $uri,
        $header,
        $method = $($resources.Method),
        $body
    )

    if ($PSCmdlet.ShouldProcess($id, $resources.Description))
    {
        try
        {
            Write-Verbose -Message 'Submitting the request'
            $result = ConvertFrom-Json -InputObject (Invoke-GoogleTranslatorWebRequest -Uri $uri -Headers $header -Method $method -Body $body)
        }
        catch
        {
            Throw $_
        }

        return $result

    }
}

Function Invoke-GoogleTranslatorWebRequest
{
    [cmdletbinding(SupportsShouldProcess)]
    param(
        $Uri,
        $Headers,
        $Method,
        $Body
    )
    $result = Invoke-WebRequest -UseBasicParsing @PSBoundParameters

    Write-Verbose -Message "Received HTTP Status $($result.StatusCode)"

    return $result

}

These functions are responsible for testing if the header is present, retrieving the configuration data, building up the correct URI to sent and building up the body. Let’s quickly test it out how it would works.

In your terminal, use Import-Module <pathtomodule>
Use Connect-GoogleTranslator -Server <servername> -Token <token> to connect to the API

Fill in the servername and token from the RapidAPI UI

Enter Get-GoogleLanguageTranslate -Target en -Query 'Hallo wereld!', it should return Hello World!

Turn on -Verbose parameter to see verbosity messages

You have now build your abstracted Powershell module that is going to be used in the task execution runner of Azure DevOps.

Create task Powershell script

In the task.json file, you’ve specified to search for the task.ps1 script. In this section, you’ll be adding the script to retrieve the input values and call the Google Translate API from the task execution runner itself.

In the GoogleTranslatorTask folder, create task.ps1 file
Add the script content below

[CmdletBinding()]
param ()

Trace-VstsEnteringInvocation $MyInvocation

try
{
    $endpoint = Get-VstsInput -Name "url" -Require
    $token = Get-VstsInput -Name "token" -Require
    $target = Get-VstsInput -Name "target" -Require
    $query = Get-VstsInput -Name "query" -Require
    $source = Get-VstsInput -Name "source"

    Import-Module -Name $PSScriptRoot\ps_modules\GoogleTranslator.psm1
    
    Connect-GoogleTranslator -Server $endpoint -Token $token

    $Params = @{
        Target = $target 
        Query  = $query 
    }

    if ($source)
    {
        Write-VstsTaskVerbose -Message "Adding source parameter to call"
        $Params.Add('Source', $source)
    }

    Get-GoogleLanguageTranslate @Params
}
catch
{
    $ErrorMessage = $_.Exception.Message
    Throw $ErrorMessage
}

The Get-VstsInput is responsible for retrieving the value from the specified input name. This was created in the task.json file. GoogleTranslator.psm1 is imported into the session when running the task itself, where you can call the functions accordingly from the runner. Sweet, let’s finish the extension.

Finishing the extension

Most of the parts are now in-place, but let’s finish up the extension for publishing.

Save the icon below in the root of GoogleTranslatorTask

Add the index picture in the images folder

Add json content to vss-extension.json

{
  "manifestVersion": 1,
  "id": "google-translator-api",
  "name": "Google Translator API extension",
  "version": "0.1.0",
  "publisher": "",
  "targets": [
    {
      "id": "Microsoft.VisualStudio.Services"
    }
  ],
  "description": "A task that invokes Google Translator API",
  "categories": ["Azure Pipelines"],
  "icons": {
    "default": "images/index.png"
  },
  "files": [
    {
      "path": "Extensions"
    }
  ],
  "contributions": [
    {
      "id": "GoogleTranslatorTask",
      "type": "ms.vss-distributed-task.task",
      "targets": ["ms.vss-distributed-task.tasks"],
      "properties": {
        "name": "Extensions/GoogleTranslatorTask"
      }
    }
  ]
}

Make sure that you fill in your own publisher. If you don’t have a publisher, check out the optional section

A quick summary of important properties to mention:

targets: supports the products and services to your extension
files: includes any files in the extension
contributions: defines a contribution entry by it’s properties

Creating publisher

To publish to the marketplace, you need a profile where you will be able to manage your extensions. Check out the documentation how to do this step by step in the Microsoft Docs. Make sure that the publisher is reflected in the vss-extension.json

Create and publish the extension task

After the publisher property is updated, you are now set to create and upload the task extension to the marketplace.

In the terminal enter tfx extension create --manifest-globs vss-extension.json

Open the marketplace
Click New extension and select Azure DevOps

Drag and drop or upload the extension created in step 1

When the task is verified, you can share the extension with your Azure DevOps organization by clicking on the three dots and press Share/Unshare. Make sure you do so.

Installing the extension task

If you have shared your extension with your own organization, you can come back to the marketplace and view the extension in the marketplace.

From here, install the extension. When it is installed, you’ll be able to view it in the assistant.

Consuming the extension

Open the azure-pipelines.yml file
Add the content below

jobs:
  - job: Google
    pool:
      vmImage: windows-latest
    steps:
      - task: google-translator-api-extension@0
        displayName: "Run translate"
        inputs:
          url: "google-translate1.p.rapidapi.com"
          token: "$(token)"
          target: "nl"
          query: "Hello World!"

Import the pipeline and make sure you add the token as secret variable

Run the pipeline and verify the output

Conclusion

Awesome, you have covered a lot of material in this tutorial, but you should have been able to create your first extension in Azure DevOps. In this tutorial, you’ve also seen the usages of abstraction which is a wonderful technique to use. If you played around with it, you should know by now, that it is going to be easy for you to introduce maybe the other two endpoints. Challenge yourself at least to see if you can do so! Good luck, and see you next time.

Published Jun 18, 2022

Writing about Azure DevOps, Powershell and AutomationThe Lonely Engineer on Twitter