Content Comparison

...

The Validation Scripts have historically produced a number of reports geared toward human consumption, but no report has existed that provided a full accounting of results in a machine readable format. A file named failures was produced in the output directory, but it contained only a subset of information about the execution run. The following proposal is to create a new file ( report.json ) that contains a full account of all available information. It would be produced on every run of the validation scriptswhen the --report-format json option is passed to the pytest command, and would be suitable for consumption for tools such as VNFSDK or Dovetail as part of the certification program.

...

Code Block

language	js
theme	DJango
title	Sample - Test Result
linenumbers	true

{
  "files": [
    "/Users/username/Desktop/stark_template2/STARKDB-nested-1.yaml",
    "/Users/username/Desktop/stark_template2/base_starkdb.yaml",
 ],
  "test_module": "test_resource_indices",
  "test_case": "test_indices_start_at_0_increment",
  "result": "FAIL",
  "error": " Index values associated with resource ID prefix STARKDB_server_ do not start at 0\n",
  "requirements": [
    {
      "id": "R-11690",
      "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*.  While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
      "keyword": "MUST"
    }
  ]
}

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description
`files`	Required	list of string		List of files that were passed to the test case. NOTE: If `result` is `ERROR` than this may be an empty list.
test_module	Required	string		Name of the module/file name that contains the test case
test_case	Required	string		Name of the test case
result	Required	string	`PASS` `SKIP` `FAIL` `ERROR`	One of the valid values. `PASS` - The test case passed with no violations or issues `SKIP` - The test case was skipped because it was deemed not applicable `FAIL` - The test case completed, but it found a violation `ERROR` - An unexpected error was found while setting up the test case
error	Required	string		If the test failed or encountered an error, then this will be an message summarizing the error. If the test passed or was skipped, then this will be an empty string
`requirements`	Required	List of Requirement Metadata		A list of Requirement Metadata for each requirement this test case validates. While uncommon, this can be an empty list in some circumstances.

Requirement Metadata

For each test case, the following requirement metadata will be collected. The validation scripts will attempt to download the latest version of the requirements from the internet, but if it is not available, then a cached version of the requirement will be used.

Code Block

language	js
theme	DJango
title	Sample - Requirement Metadata
linenumbers	true

{
  "id": "R-11690",
  "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*.  While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
  "keyword": "MUST"
}

Field Name

Required/

Optional/

Conditional

Datatype

Valid Values

Description

id Required string Requirement ID from the VNFRQTS project

text

Required

string

Full text of the requirement. This text may be multi-line, and it will

keyword

Required

string

MUST

MUST NOT

MAY

SHOULD

SHOULD NOT

RFC 2119 keyword associated with the requirement

Requirement Result

The file also includes an aggregated view of adherence to the VNF Requirements validated by the validation scripts. Since some requirements have multiple test cases, these results roll-up the result to an aggregated result for each requirement. This section does not include detailed test results. If you require detailed error information, then refer to the tests section of the results.

Code Block

language	js
theme	DJango
title	Sample - Requirement Result
linenumbers	true

{
  "id": "R-16447",
  "text": "A VNF's <resource ID> **MUST** be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF.",
  "keyword": "MUST",
  "result": "FAIL"
  "errors": [
     "The error message"
  ]
}

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description
`id`	Required	string		Requirement ID from the VNFRQTS project
text	Required	string		Full text of the requirement. This text may be multi-line, and it will
keyword	Required	string	`MUST` `MUST NOT` `MAY` `SHOULD` `SHOULD NOT`	RFC 2119 keyword associated with the requirement
result	Required	string	`PASS` `SKIP` `FAIL` `ERROR`	One of the valid values. `PASS` - The test case passed with no violations or issues `SKIP` - The test case was skipped because it was deemed not applicable `FAIL` - The test case completed, but it found a violation `ERROR` - An unexpected error was found while setting up the test case
errors	Required	list of string		Error messages associated with this requirement. This will be an empty string if the result is `PASS` or `SKIP`

Version	Old Version 3	New Version 4
Changes made by	Trevor Lovett	Trevor Lovett
Saved on	Jan 20, 2019	Jan 20, 2019

Versions Compared

Key

Requirement Metadata

Requirement Result