File structure¶

There’s no filesystem organization structure where test plans, requirements and test cases are discovered by testcases.library.Library. All the structures should be then accessed through the testcases.library.Library object.

Structures are destinguished by their filename suffix (described below) and are available in testcases.library.Library after successful load. If a file doesn’t contain correct structure, there will be error raised during discovery.

Test plan¶

suffix

.plan.yaml

class

tplib.structures.testplan.TestPlan

library attribute

testplans

yaml structure

name (str, unique) – Test plan name. Used as identifier that can be used in parent_plan field of other test plans.
description (str, optional) – Description of the testplan providing information about the purpose of the test plan, possibly specifying who the stakeholders are, turnaround times and other possibly useful information.
point_person (str, inherited) – Identifier of person responsible for maintenance of this test plan.
tags (set, inherited, optional) – Set of strings that can be used for categorization purposes.
artifact_type (str, inherited) – Artifact which is handled by the test plan. This can be any string. The idea is to use dots as separators allowing fine grained specification for which artifacts and under which circumstances should this test plan be started. For example, there could be an artifact called package which can be e.g. built, deleted or accepted and the package itself could be e.g. production or scratch. If one would like to have the test plan executed for anything that happens with the package, then artifact_type would be just plain package. If one would like to execute the test plan only for scratch packages at the moment they were built, then artifact_type would be package.built.scratch. Please consult with the execution tooling (pipeline) that uses tplib for storing test plans data.
components (set, inherited, optional) – Set of component names related to the testplan
execute_on (list, inherited) – Define additional conditions for the test plan execution.
- type (str) –
- filter (str) – Jinja2 expression, for more information consult with the execution tooling (pipeline) that uses tplib for storing test plans data.
parent_plan (str, optional) – name of other plan which should be used as parent for this plan allowing inheritance of certain fields. Circular inheritance (A -> B -> A -> …) is not allowed and will result in error as well as referencing non-existing test plan.
verified_by (dict - tplib.structures.testplan.Selection, optional)
- test_cases (dict - tplib.structures.testplan.QueryObject, optional)
  - direct_list (list) – List of testcase names
  - query (str) – Jinja2 expression, see more here: TBD
  - named_query (str) – Reference to query, not implemented
- requirements (dict - tplib.structures.testplan.QueryObject, optional)
  - direct_list (list) – List of requirement names
  - query (str) – Jinja2 expression, see more here: TBD
  - named_query (str) – Reference to query, not implemented
acceptance_criteria (dict - tplib.structures.testplan.Selection, optional) – Same as verified_by
reporting (list, inherited)
- type (str) – Name of reporting should be used while executing the test plan.
- condition (str) – Jinja2 expression used to determine when this reporting definition should be used. One may limit using the reporting only in cases of some events.
- group_by (list) – List of testcases configuration keys based on which the reporting should be grouped.
- submit_issues (bool) – Tells issue analyzers to submit found issues during reporting.
- data – Data consumed by automation for purposes of reporting done by the used reporting type. There’s no type restriction for this structure.
configurations (list, inherited, optional) –
document (str, optional) –

Examples¶

name: testplan main 1
description: |
  Main testplan containg other testplans
point_person: zveleba@redhat.com
tags:
  - foo_tag
  - bar_tag
artifact_type: compose
components:
  - anaconda
  - python-blivet
execute_on:
  - filter: "event.status == built"
parent_plan:
verified_by:
  test_cases:
    direct_list:
      - testing case (automated)
  requirements:
    direct_list:
      - First requirement
reporting:
  - type: email
    condition:
    data:
      template: |
        email template
acceptance_criteria:
  test_cases:
    direct_list:
      - testing case (automated)
    query: tc.name == 'testing case (disabled)'
configurations:
  - architecture: x86_64
  - architecrure: ppc64le
document: |
  reference to a test plan document template
  the content used to generate a test plan document
  *requirement:* automatically insert information about tested requirement

name: testplan sub 1
description: |
  Simple testplan
point_person: zveleba@redhat.com
tags:
  - foo_tag
  - bar_tag
artifact_type: compose
parent_plan: testplan main 1
verified_by:
  test_cases:
    direct_list:
      - testing case (automated)
  requirements:
    direct_list:
      - First requirement
reporting:
  - type: email
    condition:
    group_by:
      - architecture
    data:
      template: |
        email template
acceptance_criteria:
  test_cases:
    direct_list:
      - testing case (automated)
  requirements:
    direct_list:
      - First requirement
document: |
  reference to a test plan document template
  the content used to generate a test plan document
  *requirement:* automatically insert information about tested requirement

Requirement¶

suffix

.req.yaml

class

tplib.structures.requirement.Requirement

library attribute

requirements

yaml structure

name (str, unique) –
description (str, optional) –
tags (list) –
verified_by (dict - tplib.structures.requirement.QueryObject, optional)
- direct_list (list) – List of testcase names
- query (str) – Jinja2 expression, see more here: TBD
- named_query (str) – Reference to query, not implemented
acceptance_criteria (dict - tplib.structures.requirement.QueryObject, optional) – Same as verified_by

Examples¶

name: First requirement
description: |
  This requirement serves only for testing purposes for git repository -> tcms
  synchronization.
tags:
  - foo_tag
  - bar_tag
verified_by:
  direct_list:
    - testing case (disabled)
    - testing case (automated)
    - testing case (manual)
acceptance_criteria:
  direct_list:
    - testing case (manual)

name: Priority requirement
description: |
  This requirement serves only for testing purposes for git repository -> tcms
  synchronization.
tags:
  - foo_tag
  - bar_tag
verified_by:
  query: "tc.priority >= 9"

Test case¶

suffix

.tc.yaml

class

tplib.structures.testcase.TestCase

library attribute

testcases

yaml structure

name (str, unique) – Test case name used as identifier inside library
description (str) – TODO
priority (int) – This can be theoretically any number. Convention is to use values in range <1,10> where higher number means higher priority.
components (list, optional) – List of components covered by the test case.
execution (dict) –
- type (str) – This field is used by automation of testcase execution (pipeline) where the value signals what kind of execution is used for this test case. Special word manual is used to signal that manual (human) execution is required and no automation is available. The manual word is also interpreted when the testcase is exported e.g. to Polarion.
- automation_data (any type, optional) – The data type depends on the automation of testcase execution (pipeline) and can be used as machine-readable execution instructions. This field can be used even in case of manual for purposes of some upcoming automation.
filter (list, optional) – TBD
instructions (dict - tplib.structures.testcase.Instructions)
- setup (list, optional) – list of tplib.structures.testcase.Instruction
  variable item (full)
  - step (str) –
  - result (str, optional) – (default: ‘Success’)
  variable item (short)
  - str – Used as step with default result
- steps (list) – same as setup
- teardown (list, optional) – same as setup
configurations (list, optional) –
author: (str, optional) – Name of test case author
tags: (list, optional) – List of strings used as tags for queries.

Examples¶

name: testing case (manual)
description: |
  This test case serves only for testing purposes for git repository -> tcms
  synchronization.
author: pholica@redhat.com
tags:
  - foo_tag
  - bar_tag
priority: 7
execution:
  type: manual
configurations:
  - architecture: x86_64
    variant: BaseOS
  - architecture: ppc64le
    variant: BaseOS
instructions:
  setup:
    - Find this test case
  steps:
    - step: Read the description
      result: Description is correct
    - step: Check all other fields
      result: The field values correspond with data in git
    - Drink cup of coffee
  teardown:
    - step: Delete this test case
      result: Test case cannot be found in this tcms anymore

name: testing case (automated)
description: |
  This test case serves only for testing purposes for git repository -> tcms
  synchronization. This is variant of automated test case.
author: pholica@redhat.com
tags:
  - autofoo_tag
  - autobar_tag
priority: 10
execution:
  type: automated-executable
  automation_data: |
    #!/bin/bash
    exit 0
configurations:
  - architecture: x86_64
    variant: BaseOS
  - architecture: ppc64le
    variant: BaseOS
instructions:
  setup:
    - Create executable file from this test case using automation_data
  steps:
    - step: Run the executable
      result: |
        * Executable finishes with return code 0
        * Nothing is printed
  teardown:
    - step: Delete the executable created in setup
      result: The executable have ceased to exist
    - step: Delete this test case
      result: Test case cannot be found in this tcms anymore

name: testing case (disabled)
x-polarion-status: inactive
description: |
  This test case serves only for testing purposes for git repository -> tcms
  synchronization.

  Note that such testcases is disabled only in Polarion but is still used
  by all the tooling which doesn't reflect the polarion extensions.
author: pholica@redhat.com
priority: 3
execution:
  type: manual
configurations:
  - architecture: x86_64
    variant: BaseOS
instructions:
  setup:
    - Don't find this test case
  steps:
    - step: Don't read the description
      result: Description is not visible
    - step: Don't check any other field
      result: The field values are gone
  teardown:
    - step: Keep this test case