Updated JSON Report Output for Validation Scripts

Proposal

The Validation Scripts have historically produced a number of reports geared toward human consumption, but no report has existed that provided a full accounting of results in a machine readable format. A file named failures was produced in the output directory, but it contained only a subset of information about the execution run. The following proposal is to create a new file ( report.json ) that contains a full account of all available information. It would be produced on every run of the validation scripts, and would be suitable for consumption for tools such as VNFSDK or Dovetail as part of the certification program.

File Specification

File Header/Top Level

The top level will include a summary of available execution metadata.

NOTE: The tests and requirements entries are elided in the example below.

Sample - Top Level

{
  "version": "dublin",
  "template_directory": "C:\\Users\\tl2972\\Desktop\\stark_template2",
  "timestamp": "2019-01-21T02:11:07.305000",
  "checksum": "6296aa211870634f9b4a23477c5eab28",
  "profile": "",
  "outcome": "FAIL",
  "tests": [],
  "requirements": [],
}

Field Name	Required/ Optional/ Conditional	Datatype	Valid Values	Description
version	Required	string		Version of the validation scripts that produced the report. For now, this is set to the ONAP release name, but it may change to a semantic version number in the future.
template_directory	Required	string		Absolute path of the directory containing the Heat templates that were validated
timestamp	Required	string		ISO 8601 Timestamp in UTC
checksum	Required	string		MD5 hash of all file contents in the `template_directory`
profile	Required	string		Validation profile used to validate the templates. In the default ONAP code base, this is always the empty string
outcome	Required	string	`PASS` `FAIL` `ERROR`	One of the valid values. `PASS` - All tests passed successfully (some may have been skipped as not applicable based on the contents of the template). `FAIL` - At least one test failed a validation. In this scenario the templates will need to be corrected to comply with the requirements. `ERROR` - An unexpected error occurred during test setup. Some or all tests may have not executed. Issue should be referred to the VNF Validation Program team for investigation.
tests	Required	list of Test Result		List of Test Result

Test Result

For each test result a JSON object will be provided that informs the consumer what tests was run, its result, and the requirements it validated.

Sample - Test Result

{
  "files": [
    "/Users/username/Desktop/stark_template2/STARKDB-nested-1.yaml",
    "/Users/username/Desktop/stark_template2/base_starkdb.yaml",
 ],
  "test_module": "test_resource_indices",
  "test_case": "test_indices_start_at_0_increment",
  "result": "FAIL",
  "error": " Index values associated with resource ID prefix STARKDB_server_ do not start at 0\n",
  "requirements": [
    {
      "id": "R-11690",
      "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*.  While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
      "keyword": "MUST"
    }
  ]
}