Running functional tests in an Azure DevOps Pipeline

After you create the tests in HCL DevOps Test UI (Test UI) for the application that you are testing, and after you install the Test Workbench extension in your organization, you can run the tests in Azure DevOps pipelines.

Before you begin

You must have completed the following tasks:

About this task

After you add the Test Workbench extension in your Azure DevOps organization, you can use an existing pipeline or create a new one to add Test UI test tasks. You can install an agent or use the one that you installed in your default agent pool. You can add the Test UI tests to your task for the agent job, configure the task, and then run the task in the Azure DevOps pipeline.

If you have created test cases under test plans in your Azure DevOps project, you can provide the details of the Azure DevOps URL, test plan, test case, and your personal access token (PAT) while you configure the test job in a pipeline so that you can view the results of the test run on your Test Plan dashboard.

Procedure

  1. Open your Organization page in Azure DevOps and perform the following steps:
    1. Click the project you want to use.
    2. Initialize the repository by performing the following steps:
      1. Click Repos from the left pane.
      2. Click Initialize from the Initialize with a README or gitignore section.
        Note: Select the Add a README check box if it is not selected.
    3. Click Pipelines from the left pane.
    4. Click Create Pipeline.
    5. Click Use the classic editor to create a pipeline without YAML.
    6. Verify the project, repository, and branch for manual and scheduled builds, and then click Continue.
    7. Click Empty job.
  2. Select Pipeline and complete the following steps:
    1. Change the name for the build pipeline if required.
    2. Select the Agent pool for your build pipeline.

      You can use the agent from the default agent pool or use the one you have installed.

    3. Select the Agent Specification for the agent if required.
  3. Add a task to the agent job by completing the following steps:
    1. Click the Add Task icon Add Task icon in Azure DevOps pipeline for the agent job.

      The Add tasks pane is displayed.

    2. Search for the HCL tasks defined in the Test Workbench extension.

      The tasks that you can select are displayed.

      HCL Task selection in Azure DevOps

      Depending on the type of test that you have created in Test UI, you can select the type of task. You must use the following table to identify the task you must select:

      Type of test Task to select

      Functional test scripts

      		 DevOps Test UI - Functional Test Task
    3. Select the DevOps Test UI - Functional Test Task option, and then click Add to add the task to the agent job.

      The selected task is added to the agent job and it is displayed with a warning that some settings require attention. You must configure the settings mentioned in Step 4.

      You can also remove the tasks that are not required in your job. Select the tasks in the list that you want to remove. You can then right-click the tasks, and click Remove selected task(s) to remove them.

  4. Configure the settings by performing the following steps:
    1. Select the task version from the list if required.
    2. Follow the action for the functional tester task by referring to the following table:
      Note: All mandatory fields are marked with an asterisk (*) in the UI.
      Field Description Action
      Display name Displays the name of the selected task. Enter the name of the task.
      Testcase Type The type of test to execute. Select Traditional (Java) from the Testcase Type list.
      Product Path

      The fully qualified path to the Test UI. This path must exist on the agent computer.

      Enter the complete path of Test UI.
      Project Name The name of the project containing the test. Enter the name of the project containing the test.

      Project Directory

      The path where the project containing the test is located on your computer.

      Enter the complete path to the location of the project containing the test. For example, D:\Projects\
      Test Suite Name

      The name of the functional test script that you execute without the file extension.

      Enter the name of the script that you want to run. For example, script1.
      Export Report The option to export the unified report to other file formats such as PDF, XML, and HTML. Enter the following details in the text field:

      type=<report type>;format=<file type>;folder=<destination folder path>;filename=<name of the exported file>

      For example, to export the report to only the pdf format, you can use

      type=unified;format=pdf;folder=Expreport1;filename=testreport .

      If you want to export the report to multiple formats, you can specify the file formats as comma-separated values. The file type value can be in uppercase or lowercase.

      For example, to export the report to all the supported formats, you can use

      type=unified;format=pdf,xml,html;folder=Expreport1;filename=testreport.

      Log Format

      The format of the script logs.
      Select the format of the script log from the list.
      Note: By default, Default is selected.
      Iteration Count

      The number of datasets iterations to be run.

      Enter the number that you want for the dataset iterations.
      User Arguments

      You can specify additional arguments that you want to run in the test.

      Specify any arguments for the test run. For example, you can specify the playback arguments, if applicable in your test.
      Azure DevOps Project URL

      The URL of the test project in the organization on the Azure DevOps server.

      Note: You must enter the details for this option if you have created test cases under Test plans in your Azure project and want to view the test results on the Test Plans Dashboard.
      		 Enter the URL of the test project in the organization on the Azure DevOps server in the following format:

      https://<host>/<orgname>/<projectName>

      You must use this option with the following options:
      • Azure DevOps PAT
      • Test Plan Name
      • Test Case Name
      Azure DevOps PAT
      The personal access token (PAT) of the Azure server where the test project that contains the test plan is hosted. You can also enter the variable for the encrypted PAT that you create in Azure DevOps.
      Note: The token must have read and write access.
      		Enter your personal access token.
      You must use this option with the following options:
      • Azure DevOps Project URL
      • Test Plan Name
      • Test Case Name
      Test Plan Name The name of the test plan in the Azure server. Enter the name of the test plan.
      You must use this option with the following options:
      • Azure DevOps Project URL
      • Azure DevOps PAT
      • Test Case Name
      Test Case Name The name of the test case in the Azure server. Enter the name of the test case.
      You must use this option with the following options:
      • Azure DevOps Project URL
      • Azure DevOps PAT
      • Test Plan Name
      Disconnected Mode The option to run the test through an agent in the background. Optional.
      Select one of the following options:
      • True: Runs the test through an agent, and the test run continues even when the screen is locked or the agent is disconnected.
      • False: Runs the test, but the test run stops when the screen is locked or the agent is disconnected.
    3. Expand Control Options and configure the settings for your task if required.
    4. Expand Output Variables and configure the settings for your task if required.
  5. Select the following options:
    1. Click Save to save the configured settings for the task.
      Note: The task is not queued for a run.

      You can save the task to a build pipeline and opt to run the build at a later time.

    2. Click Save & queue to save the configurations and queue the run in the pipeline.

      The Run pipeline is displayed.

  6. Complete the following steps:
    1. Enter a comment for the test in the Save comment field.
    2. Select the agent that you configured for the test from the Agent pool list.
    3. Select the agent specification from the Agent Specification list for the agent if required.
    4. Select the branch from the Branch/tag list.
    5. Add the variables and demands for the task run from the Advanced Options pane if required.
    6. Select the Enable system diagnostics check box for a detailed log view.
    7. Click Save and run.

      The pipeline summary page displays the progress of the job run.

Results

You have run the tests for the application you are testing, in the Azure DevOps pipeline.

What to do next

  • You can open the job to view the task logs from the pipeline summary page.
  • You can click Test Plans to view the test result if you specified the values for Azure DevOps Project URL, Azure DevOps PAT, Test Plan Name, and Test Case Name.

  • You must click the task to open the Task page to view the test results.

  • You can access the report URLs to view the test execution information at any point of time. The report URLs are the HCL DevOps Test Hub URLs where the reports are stored.

    In Test UI, if the DevOps Test Hub URL is configured in Window > Preferences > Test > DevOps Test Hub and Publish result after execution is set as Always in Window > Preferences > Test > DevOps Test Hub > Results, then the logs in the Task page also displays the names of the published report along with its corresponding URLs.