Skip to content

Latest commit

 

History

History
290 lines (219 loc) · 19 KB

invoke-checks.md

File metadata and controls

290 lines (219 loc) · 19 KB
title description ms.topic ms.author author ms.date monikerRange
Using Invoke Azure Function / REST API checks
Use Invoke Azure Function / REST API checks to determine when a deployment stage can run
conceptual
sandrica
silviuandrica
07/13/2022
>= azure-devops-2020

Invoke Azure Function / REST API checks

The Invoke Azure Function / REST API Checks allow you to write code to decide if a specific pipeline stage is allowed to access a protected resource or not. These checks can run in two modes:

  • Asynchronous (Recommended): push mode, in which Azure DevOps awaits for the Azure Function / REST API implementation to call back into Azure DevOps with a stage access decision
  • Synchronous: poll mode, in which Azure DevOps periodically calls the Azure Function / REST API to get a stage access decision

In the rest of this guide, we'll refer to Azure Function / REST API Checks simply as checks.

The recommended way to use checks is in asynchronous mode. This mode offers you the highest level of control over the check logic, makes it easy to reason about what state the system is in, and decouples Azure Pipelines from your checks implementation, providing the best scalability. All synchronous checks can be implemented using the asynchronous checks mode.

Asynchronous checks

In asynchronous mode, Azure DevOps makes a call to the Azure Function / REST API check and awaits a callback with the resource access decision. There's no open HTTP connection between Azure DevOps and your check implementation during the waiting period.

The rest of this section talks about Azure Function checks, but unless otherwise noted, the guidance applies to Invoke REST API checks as well.

Recommended implementation of asynchronous checks

The recommended asynchronous mode has two communication steps:

  1. Deliver the check payload. Azure Pipelines makes an HTTP call to your Azure Function / REST API only to deliver the check payload, and not to receive a decision on the spot. Your function should just acknowledge receipt of the information and terminate the HTTP connection with Azure DevOps. Your check implementation should process the HTTP request within 3 seconds.
  2. Deliver access decision through a callback. Your check should run asynchronously, evaluate the condition necessary for the pipeline to access the protected resource (possibly doing multiple evaluations at different points in time). Once it reaches a final decision, your Azure Function makes an HTTP REST call into Azure DevOps to communicate the decision. At any point in time, there should be a single open HTTP connection between Azure DevOps and your check implementation. Doing so saves resources on both your Azure Function side and on Azure Pipelines's side.

If a check passes, then the pipeline is allowed access to a protected resource and stage deployment can proceed. If a check fails, then the stage fails. If there are multiple checks in a single stage, all need to pass before access to protected resources is allowed, but a single failure is enough to fail the stage.

The recommended implementation of the async mode for a single Azure Function check is depicted in the following diagram.

:::image type="content" source="media/checks/async-checks.png" alt-text="Diagram that shows the recommended implementation of the async mode for a single Azure Function check.":::

The steps in the diagram are:

  1. Check Delivery. Azure Pipelines prepares to deploy a pipeline stage and requires access to a protected resource. It invokes the corresponding Azure Function check and expects receipt confirmation, by the call ending with an HTTP 200 status code. Stage deployment is paused pending a decision.
  2. Check Evaluation. This step happens inside your Azure Function implementation, which runs on your own Azure resources and the code of which is completely under your control. We recommend your Azure Function follow these steps:
    • 2.1 Start an async task and return HTTP status code 200
    • 2.2 Enter an inner loop, in which it can do multiple condition evaluations
    • 2.3 Evaluate the access conditions
    • 2.4 If it can't reach a final decision, reschedule a reevaluation of the conditions for a later point, then go to step 2.3
  3. Decision Communication. The Azure function calls back into Azure Pipelines with the access decision. Stage deployment can proceed

Recommended configuration for asynchronous checks

In the Azure Function / REST API check configuration panel, make sure you:

  • Selected Callback for the Completion event
  • Set Time between evaluations (minutes) to 0

Setting the Time between evaluations to a non-zero value means the check decision (pass / fail) isn't final. The check will be reevaluated until all other Approvals & Checks reach a final state.

Pass pipeline run information to checks

When configuring the check, you can specify the pipeline run information you wish to send to your check. At a minimum, you should send:

  • "PlanUrl": "$(system.CollectionUri)"
  • "ProjectId": "$(system.TeamProjectId)"
  • "HubName": "$(system.HostType)"
  • "PlanId": "$(system.PlanId)"
  • "JobId": "$(system.JobId)"
  • "TaskInstanceId": "$(system.TaskInstanceId)"
  • "AuthToken": "$(system.AccessToken)"

These key-value pairs are set, by default, in the Headers of the REST call made by Azure Pipelines.

You can use AuthToken to make calls into Azure DevOps, such as when your check will call back with a decision. The AuthToken is restricted to the scope of the pipeline run from which the check call was made.

Send a decision back to Azure DevOps

Your check implementation must use the Post Event REST API call to communicate a decision back to Azure Pipelines. Make sure you specify the following properties:

  • Headers: Basic: {AuthToken}
  • Body:
{
    "name": "TaskCompleted",
    "taskId": "{TaskInstanceId}",
    "jobId": "{JobId}",
    "result": "succeeded|failed"
}

Send status updates to Azure DevOps from checks

You can provide status updates to Azure Pipelines users from within your checks using Azure Pipelines REST APIs. This functionality is useful, for example, if you wish to let users know the check is waiting on an external action, such as someone needs to approve a ServiceNow ticket.

The steps to send status updates are:

  1. Create a task log
  2. Append to the task log
  3. Update timeline record

All REST API calls need to be authenticated.

Examples

Basic Azure Function check

In this basic example, the Azure Function checks that the invoking pipeline run executed a CmdLine task, prior to granting it access to a protected resource.

The Azure Function goes through the following steps:

  1. Confirms the receipt of the check payload
  2. Sends a status update to Azure Pipelines that the check started
  3. Uses {AuthToken} to make a callback into Azure Pipelines to retrieve the pipeline run's Timeline entry
  4. Checks if the Timeline contains a task with "id": "D9BAFED4-0B18-4F58-968D-86655B4D2CE9" (the ID of the CmdLine task)
  5. Sends a status update with the result of the search
  6. Sends a check decision to Azure Pipelines

You can download this example from GitHub.

To use this Azure Function check, you need to specify the following Headers when configuring the check:

{
    "Content-Type":"application/json", 
    "PlanUrl": "$(system.CollectionUri)", 
    "ProjectId": "$(system.TeamProjectId)", 
    "HubName": "$(system.HostType)", 
    "PlanId": "$(system.PlanId)", 
    "JobId": "$(system.JobId)", 
    "TimelineId": "$(system.TimelineId)", 
    "TaskInstanceId": "$(system.TaskInstanceId)", 
    "AuthToken": "$(system.AccessToken)",
    "BuildId": "$(build.BuildId)"
}

Advanced Azure Function check

In this advanced example, the Azure Function checks that the Azure Boards work item referenced in the commit message that triggered the pipeline run is in the correct state.

The Azure Function goes through the following steps:

  1. Confirms the receipt of the check payload
  2. Sends a status update to Azure Pipelines that the check started
  3. Uses {AuthToken} to make a callback into Azure Pipelines to retrieve the state of the Azure Boards work item referenced in the commit message that triggered the pipeline run
  4. Checks if the work item is in the Completed state
  5. Sends a status update with the result of the check
  6. If the work item isn't in the Completed state, it reschedules another evaluation in 1 minute
  7. Once the work item is in the correct state, it sends a positive decision to Azure Pipelines

You can download this example from GitHub.

To use this Azure Function check, you need to specify the following Headers when configuring the check:

{
    "Content-Type":"application/json", 
    "PlanUrl": "$(system.CollectionUri)", 
    "ProjectId": "$(system.TeamProjectId)", 
    "HubName": "$(system.HostType)", 
    "PlanId": "$(system.PlanId)", 
    "JobId": "$(system.JobId)", 
    "TimelineId": "$(system.TimelineId)", 
    "TaskInstanceId": "$(system.TaskInstanceId)", 
    "AuthToken": "$(system.AccessToken)",
    "BuildId": "$(build.BuildId)"
}

Error handling

Currently, Azure Pipelines evaluates a single check instance at most 2,000 times.

If your check doesn't call back into Azure Pipelines within the configured timeout, the associated stage will be skipped. Stages depending on it will be skipped as well.

Synchronous checks

In synchronous mode, Azure DevOps makes a call to the Azure Function / REST API check to get an immediate decision whether access to a protected resource is permitted or not.

The implementation of the sync mode for a single Azure Function check is depicted in the following diagram.

:::image type="content" source="media/checks/sync-checks.png" alt-text="Diagram that shows the implementation of the sync mode for a single Azure Function check.":::

The steps are:

  1. Azure Pipelines prepares to deploy a pipeline stage and requires access to a protected resource
  2. It enters an inner loop in which:
  • 2.1. Azure Pipelines invokes the corresponding Azure Function check and waits for a decision
  • 2.2. Your Azure Function evaluates the conditions necessary to permit access and returns a decision
  • 2.3. If the Azure Function response body doesn't satisfy the Success criteria you defined and Time between evaluations is non-zero, Azure Pipelines reschedules another check evaluation after Time between evaluations

Configure synchronous checks

To use the synchronous mode for the Azure Function / REST API, in the check configuration panel, make sure you:

  • Selected ApiResponse for the Completion event under Advanced
  • Specified the Success criteria that define when to pass the check based on the check's response body
  • Set Time between evaluations to 0 under Control options
  • Set Timeout to how long you wish to wait for a check to succeed. If there's no positive decision and Timeout is reached, the corresponding pipeline stage will be skipped

The Time between evaluations setting defines how long the check's decision is valid. A value of 0 means the decision is final. A non-zero value means the check will be retried after the configured interval, when its decision is negative. When multiple Approvals and Checks are running, the check will be retried regardless of decision.

The maximum number of evaluations is defined by the ratio between the Timeout and Time between evaluations values. We recommend you ensure this ratio is at most 10.

Pass pipeline run information to checks

When configuring the check, you can specify the pipeline run information you wish to send to your Azure Function / REST API check. By default, Azure Pipeline adds the following information in the Headers of the HTTP call it makes.

  • "PlanUrl": "$(system.CollectionUri)"
  • "ProjectId": "$(system.TeamProjectId)"
  • "HubName": "$(system.HostType)"
  • "PlanId": "$(system.PlanId)"
  • "JobId": "$(system.JobId)"
  • "TaskInstanceId": "$(system.TaskInstanceId)"
  • "AuthToken": "$(system.AccessToken)"

We don't recommend making calls into Azure DevOps in synchronous mode, because it will most likely cause your check to take more than 3 seconds to reply, so the check will fail.

Error handling

Currently, Azure Pipelines evaluates a single check instance at most 2,000 times.

When to use asynchronous vs synchronous checks

Let's look at some example use cases and what are the recommended type of checks to use.

External information must be correct

Say you have a Service Connection to a production resource, and you wish to ensure that access to it's permitted only if the information in a ServiceNow ticket is correct. In this case, the flow would be as follows:

  • You add an asynchronous Azure Function check that verifies the correctness of the ServiceNow ticket
  • When a pipeline that wants to use the Service Connection runs:
    • Azure Pipelines calls your check function
    • If the information is incorrect, the check returns a negative decision. Assume this outcome
    • The pipeline stage fails
    • You update the information in the ServiceNow ticket
    • You restart the failed stage
    • The check runs again and this time it succeeds
    • The pipeline run continues

External approval must be granted

Say you have a Service Connection to a production resource, and you wish to ensure that access to it's permitted only after an administrator approved a ServiceNow ticket. In this case, the flow would be as follows:

  • You add an asynchronous Azure Function check that verifies the ServiceNow ticket has been approved
  • When a pipeline that wants to use the Service Connection runs:
    • Azure Pipelines calls your check function.
    • If the ServiceNow ticket isn't approved, the Azure Function sends an update to Azure Pipelines, and reschedules itself to check the state of the ticket in 15 minutes
    • Once the ticket is approved, the check calls back into Azure Pipelines with a positive decision
    • The pipeline run continues

Development process was followed

Say you have a Service Connection to a production resource, and you wish to ensure that access to it's permitted only if the code coverage is above 80%. In this case, the flow would be as follows:

  • You write your pipeline in such a way that stage failures cause the build to fail
  • You add an asynchronous Azure Function check that verifies the code coverage for the associated pipeline run
  • When a pipeline that wants to use the Service Connection runs:
    • Azure Pipelines calls your check function
    • If the code coverage condition isn't met, the check returns a negative decision. Assume this outcome
    • The check failure causes your stage to fail, which causes your pipeline run to fail
  • The engineering team adds the necessary unit tests to reach 80% code coverage
  • A new pipeline run is triggered, and this time, the check passes
    • The pipeline run continues

Performance criteria must be met

Say you deploy new versions of your system in multiple steps, starting with a canary deployment. You wish to ensure your canary deployment's performance is adequate. In this case, the flow would be as follows:

  • You add an asynchronous Azure Function check
  • When a pipeline that wants to use the Service Connection runs:
    • Azure Pipelines calls your check function
    • The check starts a monitor of the canary deployment's performance
    • The check schedules multiple evaluation checkpoints, to see how the performance evolved
    • Once you gain enough confidence in the canary deployment's performance, your Azure Function calls back into Azure Pipelines with a positive decision
    • The pipeline run continues

Deployment reason must be valid

Say you have a Service Connection to a production environment resource, and you wish to ensure that access to it happens only for manually queued builds. In this case, the flow would be as follows:

  • You add a synchronous Azure Function check that verifies that Build.Reason for the pipeline run is Manual
  • You configure the Azure Function check to pass Build.Reason in its Headers
  • You set the check's Time between evaluations to 0, so failure or pass is final and no reevaluation is necessary
  • When a pipeline that wants to use the Service Connection runs:
    • Azure Pipelines calls your check function
    • If the reason is other than Manual, the check fails, and the pipeline run fails

Multiple checks

Before Azure Pipelines deploys a stage in a pipeline run, multiple checks may need to pass. A protected resource may have one or more Checks associated to it. A stage may use multiple protected resources. Azure Pipelines collects all the checks associated to each protected resource used in a stage and evaluates them concurrently.

A pipeline run is allowed to deploy to a stage only when all checks pass at the same time. A single final negative decision causes the pipeline to be denied access and the stage to fail.

When you use checks in the recommended way (asynchronous, with final states) makes their access decisions final, and eases understanding the state of the system.

Check out the Multiple Approvals and Checks section for examples.

Learn more