Updated JSON Report Output for Validation Scripts

Proposal

The Validation Scripts have historically produced a number of reports geared toward human consumption, but no report has existed that provided a full accounting of results in a machine readable format. A file named failures was produced in the output directory, but it contained only a subset of information about the execution run. The following proposal is to create a new file ( report.json ) that contains a full account of all available information. It would be produced when the --report-format json option is passed to the pytest command, and would be suitable for consumption for tools such as VNFSDK or Dovetail as part of the certification program.

File Specification

File Header/Top Level

The top level will include a summary of available execution metadata.

NOTE: The tests and requirements entries are elided in the example below.

Sample - Top Level

{
  "version": "dublin",
  "template_directory": "C:\\Users\\tl2972\\Desktop\\stark_template2",
  "timestamp": "2019-01-21T02:11:07.305000",
  "checksum": "6296aa211870634f9b4a23477c5eab28",
  "profile": "",
  "outcome": "FAIL",
  "tests": [],
  "requirements": [],
}

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description
^version	Required	string		Version of the validation scripts that produced the report. For now, this is set to the ONAP release name, but it may change to a semantic version number in the future.
^{template_directory}	Required	string		Absolute path of the directory containing the Heat templates that were validated
^timestamp	Required	string		ISO 8601 Timestamp in UTC
^checksum	Required	string		MD5 hash of all file contents in the `template_directory`
^profile	Required	string		Validation profile used to validate the templates. In the default ONAP code base, this is always the empty string
^outcome	Required	string	`PASS` `FAIL` `ERROR`	One of the valid values. `PASS` - All tests passed successfully (some may have been skipped as not applicable based on the contents of the template). `FAIL` - At least one test failed a validation. In this scenario the templates will need to be corrected to comply with the requirements. `ERROR` - An unexpected error occurred during test setup. Some or all tests may have not executed. Issue should be referred to the VNF Validation Program team for investigation.
^tests	Required	list of Test Result		List of Test Result
requirements	Required	list of Requirement Result		List of Requirement Result

Test Result

For each test result a JSON object will be provided that informs the consumer what tests was run, its result, and the requirements it validated.

Sample - Test Result

{
  "files": [
    "/Users/username/Desktop/stark_template2/STARKDB-nested-1.yaml",
    "/Users/username/Desktop/stark_template2/base_starkdb.yaml",
 ],
  "test_module": "test_resource_indices",
  "test_case": "test_indices_start_at_0_increment",
  "result": "FAIL",
  "error": " Index values associated with resource ID prefix STARKDB_server_ do not start at 0\n",
  "requirements": [
    {
      "id": "R-11690",
      "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*.  While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
      "keyword": "MUST"
    }
  ]
}

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description
`files`	Required	list of string		List of files that were passed to the test case. NOTE: If `result` is `ERROR` than this may be an empty list.
^test_module	Required	string		Name of the module/file name that contains the test case
^test_case	Required	string		Name of the test case
^result	Required	string	`PASS` `SKIP` `FAIL` `ERROR`	One of the valid values. `PASS` - The test case passed with no violations or issues `SKIP` - The test case was skipped because it was deemed not applicable `FAIL` - The test case completed, but it found a violation `ERROR` - An unexpected error was found while setting up the test case
^error	Required	string		If the test failed or encountered an error, then this will be an message summarizing the error. If the test passed or was skipped, then this will be an empty string
`requirements`	Required	List of Requirement Metadata		A list of Requirement Metadata for each requirement this test case validates. While uncommon, this can be an empty list in some circumstances.

Requirement Metadata

For each test case, the following requirement metadata will be collected. The validation scripts will attempt to download the latest version of the requirements from the internet, but if it is not available, then a cached version of the requirement will be used.

Sample - Requirement Metadata

{
  "id": "R-11690",
  "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*.  While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
  "keyword": "MUST"
}

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description

Field Name

Required/

Optional/

Conditional

Datatype

Valid Values

Description

id

Required

string

Requirement ID from the VNFRQTS project

^text

Required

string

Full text of the requirement. This text may be multi-line, and it will

^keyword

Required

string

MUST

MUST NOT

MAY

SHOULD

SHOULD NOT

RFC 2119 keyword associated with the requirement

Requirement Result

The file also includes an aggregated view of adherence to the VNF Requirements validated by the validation scripts. Since some requirements have multiple test cases, these results roll-up the result to an aggregated result for each requirement. This section does not include detailed test results. If you require detailed error information, then refer to the tests section of the results.

Sample - Requirement Result

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description
`id`	Required	string		Requirement ID from the VNFRQTS project NOTE: a requirement ID of "Unmapped" may be included if one or more tests are not mapped to a requirement. In this case, all test cases not mapped to are requirement will be rolled up to a single "Unmapped" requirement.
^text	Required	string		Full text of the requirement. This text may be multi-line, and it will
^keyword	Required	string	`MUST` `MUST NOT` `MAY` `SHOULD` `SHOULD NOT`	RFC 2119 keyword associated with the requirement
^result	Required	string	`PASS` `SKIP` `FAIL` `ERROR`	One of the valid values. `PASS` - The test case passed with no violations or issues `SKIP` - The test case was skipped because it was deemed not applicable `FAIL` - The test case completed, but it found a violation `ERROR` - An unexpected error was found while setting up the test case
^errors	Required	list of string		Error messages associated with this requirement. This will be an empty string if the result is `PASS` or `SKIP`