3.24.5. Automatically reporting to external repositories after test runs

You can automatically update tasks in an external repository when a test has run. You can report the test results by writing a comment on a task in the external repository with a link to the test result in the ITE. You can also update fields / attributes on tasks based on information you specify in your Project.

3.24.5.1. Configuring a task repository for your Project

Once you have configured one or more repositories for your workspace (Section 3.24.2, “Configuring task repositories in your workspace”), you can select one of these to be the test-relevant repository for your Project.

This will let you:

  • Add a task ID from this repository to Test Cases and Test Suites in the Project to signify that this item is the test for this task (Section 3.24.5.3, “Adding task IDs to Test Suites and Test Cases”).

  • Automatically report test results to the task defined when a test runs.

  • View the test results for the relevant item in the ITE as a link from the task repository.

To configure a task repository for your Project:

  1. In the Project Properties, select Mylyn ALM from the tree on the left (Figure 3.31, “ALM Settings”).

  2. In the page that appears, you can select a repository from the combo-box. You can validate the repository settings using the button.

  3. You can (optionally) enter the URL of the ITE that is configured to use the correct database for your test results. If you enter its URL here, then you will be able to click on test result links from your task repository to open the test result in the ITE automatically.

Figure 3.31. ALM Settings

3.24.5.2. Configuring reporting to tasks

When a test has run, you can update relevant tasks in your repository with information:

  • A comment that contains the status of the test run, and a link to the results for that test.

  • You can update fields / attributes for a task based on whether the test succeeded or failed.

    Tests that have a status other than passed (e.g. failed, stopped, still testing) are considered as failed

For each item (Test Case, Test Suite) in your test run that is linked to a task (Section 3.24.5.3, “Adding task IDs to Test Suites and Test Cases”) the rules you specify in the Project properties will be carried out to report test results to that task.

  1. In the Project Properties, select Mylyn ALM from the tree on the left.

  2. In the page that appears, you can configure how to report results. You have the same options for what to do on success and on failure.

    • You can activate the checkbox to add a comment to each linked task. The comment contains information about the test run (date, status) and also a link to the test results. The test results can be seen in the specified ITE (Section 3.24.5.1, “Configuring a task repository for your Project”).

    • You can add rules in the table to update attributes for the task. Enter a descriptive name (this is only used for display purposes in the Properties), the ID for the attribute you want to update and the value that the attribute should be updated with. As values for the update, you can enter:

      • plain text, e.g. "passed"

      • a value from a column in the Test Result Summary View for this run. The value for a column is gained by entering the column name as a variable. Some examples are:

        $summary_autHostname, $summary_testRunState (gives e.g. "OK" or "FAILED"), $summary_testsuiteName, $summary_testJobName, $summary_testsuiteEndTime, $summary_testsuiteDate (for full date format), $summary_statusString (gives e.g. "Successfully tested" or "Error in Children"), $summary_testsuiteDuration, $summary_testsuiteStartTime, $summary_date (for DD.MM.YYYY), $summary_autOS

        but you can use any column value by entering the column name as a variable preceded by "summary_".

      • a value from the executed node. The value is entered as a variable, e.g.

        $node_taskId, $node_timeStamp, $node_typeOfNode (e.g. Test Case), $node_name, $node_parameterDescription, $node_date (for DD.MM.YYYY), $node_url (see below)

        Use the $node_url parameter to receive the URL that will link to this node (Test Case or Test Suite) in the ITE (xSection 3.24.5, “Automatically reporting to external repositories after test runs”).

  3. To see which attributes are available in your ALM repository, you can view the information for individual tasks from the Task View (Section 3.24.5.2.1, “Viewing task attributes”)

3.24.5.2.1. Viewing task attributes

In order to define rules for reporting to ALM systems, you will need to enter the ID for any attributes you wish to update (Section 3.24.5.2, “Configuring reporting to tasks”).

You can see which attributes a task has by using the Show attributes option from the Task View.

  1. Open the Task View if it is not already open by selecting:

    Window --> Show View --> Task View.

  2. If you have not already configured your repository for your workspace, do so in the Task Repositories View (Section 3.24.2, “Configuring task repositories in your workspace”).

  3. Navigate to the task whose attributes you want to see and select:

    Show attributes from the context menu.

  4. The Task Attribute Viewer will appear. In this dialog, you can see the IDs for the attributes for this task, which you can use to define rules for test result reporting in the Project Properties (Section 3.24.5.2, “Configuring reporting to tasks”). You can copy attributes using the keyboard.

3.24.5.3. Adding task IDs to Test Suites and Test Cases

You can add a task ID to Test Cases and Test Suites in your Project.

The task ID should be a valid ID in the repository that you have specified as the repository for this Project (Section 3.24.5.1, “Configuring a task repository for your Project”). Adding the task ID to an item in your Project means that this item is the relevant test for that task in your repository. You will be able to report test results for this item.

To add a task ID to a Test Case or Test Suite:

  1. Open the item in the editor by double-clicking it.

  2. In the Properties View, in the cell for Task ID, enter the task ID from the external repository. You can only enter task IDs at the place of specification – you cannot overwrite them when you reuse the item.

  3. Save the editor.

  4. When you have added a task ID to a node, you can open the task for this node from the browser by selecting:

    Open with-->Mylyn Task Editor

You should ensure that you add task IDs to the right node-level to provide you with the relevant amount of information for the tasks in your repository. This will usually be at the level of Use Cases within a Test Suite.

3.24.5.4. Test execution with reporting to external repositories

If your Project is joined to an external repository (Section 3.24.5.1, “Configuring a task repository for your Project”), and you have added task IDs to one or more items in your Project (Section 3.24.5.3, “Adding task IDs to Test Suites and Test Cases”), then you will be able to report the test results to the repository after the test has run. The test run can be either via the ITE or via the command line testexec.

If you run a "relevant" test via the ITE which should report to ALM systems, the reporting is triggered automatically once the test has run. If the reporting for the test run encounters an error, then you will be able to manually trigger the reporting later Section 3.24.5.4, “Test execution with reporting to external repositories”. To report to ALM systems after running a test via testexec, you must also manually trigger the recording (Section 3.24.5.4, “Test execution with reporting to external repositories”).

When reporting is triggered, the following happens:

  • A connection is made to the ALM system configured in the Project properties.

  • The test results are then analyzed to see if any comments need to be added in the external repository.

  • If there is reporting to be done, the ITE reports to each necessary task as batch jobs. These batch jobs contain the desired field updates and comments. Although this is displayed as one operation in the console view you will get N transactions for N comments/field updates. If multiple items in one test run reference the same task ID, or if an item with a task ID is executed multiple times in the same test run, then only one comment will be written, but this one will contain information about all executed items with this ID. However, field updates will occur for each item linked to the task ID.

    Tests that have a status other than passed (e.g. failed, stopped, still testing) are considered as failed.

  • You can see the status of the ALM reporting in the console view.

  • Once reporting has been done, you can see the changes in the external repository. If you added a comment, you can click on the link provided to open the test result report in the ITE that you specified for your Project. The ITE must be already running for this action to succeed. The test results must also still be present in the database (Section 3.22.3.1, “Re-opening the test result view for a test run”).

  • The test result report is opened at the node which referenced the current task ID.

  • You can see the status of the reporting in the Test Result Summary View (Section 3.24.5.4, “Test execution with reporting to external repositories”).

    The integration for writing to external repositories can be used with Bugzilla 3.6+, JIRA 5.0+ and HP ALM 11+. Other repositories for which there are Mylyn connectors may also work, but these have not been tested.

Reporting to ALM repositories after a test started via testexec For tests started via testexec, the reporting does not happen automatically. Instead, you must choose to report to ALM from the Test Result Summary View.

  1. In a Project that is configured to report to an ALM repository, when a test has run via testexec, you can open the Reporting Perspective and locate the test run in the Test Result Summary View.

  2. In the ALM Report Status column, you will see whether there are any reports pending. You can see various statuses in this column:

    Not configured:

    if you have not configured ALM reporting for the Project, or if the relevance of the Test Suite was set to false when the test was started.

    Not yet reported:

    if reporting was configured for the Project, and there is information available to be reported that has not yet been reported, e.g. from a test run via testexec. You will also see this status if you have run a test via the ITE that could not report to the ALM system due to e.g. a connection problem. This allows you to report test results later, once the problem has been resolved.

    Reported:

    this status will be shown if you have run a test with reporting configured via the ITE, which reports automatically. It will also be shown if you have chosen to report to the ALM system after running a test via testexec. This status will also be shown even if the test run was not linked to any tasks, but reporting is configured for the Project. It can be understood as any reporting that was necessary has been carried out. If an error occurs while reporting to one of the tasks, the status will still show reported even though not all tasks were correctly updated

    Report discarded:

    if reporting information was available, but the information was discarded and therefore not reported. You can do this by selecting the option to discard a pending report from the context menu. This status will also be shown for any test runs in a Project that you import that had the status Not yet reported when the Project was exported.

    Marked as non-relevant:

    you will see this status for any test runs that were marked as relevant when the test was started and which have information to report (status: Not yet reported),but which you then mark as non-relevant by toggling the relevance in the Test Result Summary View. This status means that you could report or discard the information, but only if you re-toggle the test run to be relevant.

  3. Select any test runs whose results you want to report to ALM, and click on the ”Report to ALM” button on the toolbar. You can also select this option from the context-sensitive menu. If you select multiple test runs, the reporting will take place sequentially for each test run.

  4. When you select this option, the reporting will be triggered as described above. If the reporting is successful, the status in the ALM Report Status column changes from Not yet reported to Reported. If an error occurs, then the status remains as Not yet reported.

  5. If you do not want to report the test results to an ALM tool, you can discard the reporting information by selecting the option to discard a pending report from the context menu. This changes the status in the ALM Report Status column from Not yet reported to Report discarded.

3.24.5.5. Specific information for HP ALM users

  • To use the HP ALM integration, you must use a separate connector for HP ALM which may incur license costs. Visit the Tasktop website for more details http://www.tasktop.com.

  • If you have changed the attribute IDs in your HP ALM installation for defects and / or requirements from their relative defaults (BG_DEV_COMMENTS and REQ_DEV_COMMENTS), or if you have changed the ID for the connector, then you must modify these properties in the almAccess.properties file contained in the jar:

    org.eclipse.jubula.client.alm.mylyn.core

  • Since tasks in HP ALM only have one comment field, the result information is appended to the comment field instead of being added as a new comment each time.

  • To write to requirements in a default HP ALM installation, you must use the prefix REQ, e.g. REQ100. To write to defects in a default HP ALM installation, you must use the prefix DEF, e.g. DEF42.



Copyright BREDEX GmbH 2015. Made available under the Eclipse Public License v1.0.