Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Version History

« Previous Version 34 Next »

Contributors

Keong Lim, Tian Lee

References

2018-10-03 AAI Meeting Notes

2018-09-26 AAI Meeting Notes

PTL 2018-11-05

Project Recommendations for Package Upgrades 

AAI R3 Security/Vulnerability Threat Analysis

Faster XML Jackson usage in Portal Code and replacing it with Gson

AAI-628 - Getting issue details... STATUS

AAI-908 - Getting issue details... STATUS

AAI-910 - Getting issue details... STATUS

AAI-928 - Getting issue details... STATUS

AAI-1218 - Getting issue details... STATUS

Jackson Replacement

Security subcommittee has recommended teams move away from jackson, and will be presenting alternatives and asking for an assessment from each project. Our team will need to do an analysis - this would not be trivial, especially given how many of our repos are impacted. As of now, this would be a very high LOE for the team, we need to understand what the recommendation from the SECCOM is before we can provide better details on what the LOE would be.

Vulnerable Packages per ONAP project

Three Areas of Concern

  1. Direct usage of Jackson by ONAP code
  2. Frameworks configured with Jackson like Spring Boot
  3. Usage of Jackson by third-party tools like Cassandra

Survey of Replacement Options

Articles with comparisons and benchmarks:

Rationale for eliminating some options from the articles above (about 20 libraries in total):

  • Related to or derived from Jackson code
  • Requires change to compilers and compile-time processes
  • Counter-productive to CII Badging criteria, see also https://github.com/coreinfrastructure/best-practices-badge
    • Unmaintained in recent years
    • Vulnerabilities not addressed
    • "Bus factor" too low
    • Number of contributors and reviewers too low

Short-list of libraries as reasonable options to be explored, including:

Quick CVE comparison:

Code Analysis

Search on AAI source code shows:

  • approx 611 hits in 227 files for "fasterxml", which includes pom.xml and Java imports
  • approx 978 hits in 215 files for "gson", which includes pom.xml and Java imports and initialising Java object
  • zero hits for "fastjson"
  • zero hits for "moshi"
  • zero hits for "genson"

  File Modified

Code Examples

  • aai\aai-common\aai-auth\src\main\java\org\onap\aaiauth\auth\AuthCore.java
  • aai\aai-common\aai-core\src\main\java\org\onap\aai\auth\AAIAuthCore.java
FasterXML Jackson example 
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;


    public synchronized void loadUsers(String authFilename) throws Exception {
        users = new HashMap<>();

        mapper = new ObjectMapper(); // can reuse, share globally
        JsonNode rootNode = mapper.readTree(new File(authFilename));
        JsonNode rolesNode = rootNode.path(AuthConstants.ROLES_NODE_PATH);

        for (JsonNode roleNode : rolesNode) {
            String roleName = roleNode.path(AuthConstants.ROLE_NAME_PATH).asText();

            AuthRole role = new AuthRole();
            JsonNode usersNode = roleNode.path(AuthConstants.USERS_NODE_PATH);
            JsonNode functionsNode = roleNode.path(AuthConstants.FUNCTIONS_NODE_PATH);
            for (JsonNode functionNode : functionsNode) {
                String function = functionNode.path(AuthConstants.FUNCTION_NAME_PATH).asText();
                JsonNode methodsNode = functionNode.path(AuthConstants.METHODS_NODE_PATH);
                boolean hasMethods = handleMethodNode(methodsNode, role, function);

                if (!hasMethods) {
                    // iterate the list from HTTP_METHODS
                    for (HTTP_METHODS meth : HTTP_METHODS.values()) {
                        String thisFunction = meth.toString() + ":" + function;

                        role.addAllowedFunction(thisFunction);
                    }
                }
            }

            handleUserNode(usersNode, roleName, role);
        }

        usersInitialized = true;
    }
gson example 
import com.fasterxml.jackson.core.JsonProcessingException;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;


	private synchronized void reloadUsers() {

		Map<String, AAIUser> tempUsers = new HashMap<>();

		try {
			LOGGER.debug("Reading from " + globalAuthFileName);
			String authFile = new String(Files.readAllBytes(Paths.get(globalAuthFileName)));
			
			JsonParser parser = new JsonParser();
			JsonObject authObject = parser.parse(authFile).getAsJsonObject();
			if (authObject.has("roles")) {
				JsonArray roles = authObject.getAsJsonArray("roles");
				for (JsonElement role : roles) {
					if (role.isJsonObject()) {
						JsonObject roleObject = role.getAsJsonObject();
						String roleName = roleObject.get("name").getAsString();
						Map<String, Boolean> usrs = this.getUsernamesFromRole(roleObject);
						List<String> aaiFunctions = this.getAAIFunctions(roleObject);
						
						usrs.forEach((key, value) -> {
							final AAIUser au = tempUsers.getOrDefault(key, new AAIUser(key, value));
							au.addRole(roleName);
								aaiFunctions.forEach(f -> {
								List<String> httpMethods = this.getRoleHttpMethods(f, roleObject);
								httpMethods.forEach(hm -> au.setUserAccess(f, hm));
								this.validFunctions.add(f);
							});
								
							tempUsers.put(key, au);
							
						});
					}
				}
				if (!tempUsers.isEmpty()) {
					users = tempUsers;
				}	
			}
		} catch (FileNotFoundException e) {
			ErrorLogHelper.logError("AAI_4001", globalAuthFileName + ". Exception: " + e);
		} catch (JsonProcessingException e) {
			ErrorLogHelper.logError("AAI_4001", globalAuthFileName + ". Not valid JSON: " + e);
		} catch (Exception e) {
			ErrorLogHelper.logError("AAI_4001", globalAuthFileName + ". Exception caught: " + e);
		}
	}

Side-by-side comparison

FasterXML Jackson versionGoogle gson versionComments
mapper = new ObjectMapper();
JsonParser parser = new JsonParser();

JsonNode rootNode = mapper.readTree(new File(authFilename));
JsonNode rolesNode = rootNode.path(AuthConstants.ROLES_NODE_PATH);
JsonObject authObject = parser.parse(authFile).getAsJsonObject();
JsonArray roles = authObject.getAsJsonArray("roles");

Jackson's JsonNode is a more abstract data structure, compared with Gson's more concrete data structures JsonObject and JsonArray.

String function = functionNode.path(AuthConstants.FUNCTION_NAME_PATH).asText();
String roleName = roleObject.get("name").getAsString();
Code structure differs at this point (function name vs role name) but the general intent of the code is equivalent (get the element name as a string).
public synchronized void loadUsers(String authFilename) throws Exception


(no exception handling in this method)
} catch (JsonProcessingException e) {
			ErrorLogHelper.logError("AAI_4001", globalAuthFileName + ". Not valid JSON: " + e);

For some reason, this version still catches com.fasterxml.jackson.core.JsonProcessingException even though it uses Google gson for parsing.

Not a good idea to defer exception handling to the caller since the caller has no idea why/how/when/where the parsing failed and might be left with an invalid data structure as well.

boolean hasMethods = handleMethodNode(methodsNode, role, function);
usrs.forEach((key, value) -> {
...
});
Method call vs Java lambda call is not really relevant to the Jackson replacement, but consistency of style could be an overall goal if the code is being re-factored anyway.

POC: Replacing default Spring Boot Jackson dependencies with Gson - Tian Lee

I have investigated the feasibility of replacing all Jackson Spring Boot dependencies with Gson, by converting the two AAF security microservices, aaf-fproxy and aaf-rproxy to use Gson only.

The basic method I followed is detailed here: https://www.callicoder.com/configuring-spring-boot-to-use-gson-instead-of-jackson/

What I did:

  • Exclude all transitive Jackson dependencies being pulled in from the Spring Boot dependencies.
    • Exclude spring-boot-starter-json from spring-boot-starter-web
    • Remove any spring-boot-starter-actuator dependencies, as they can only work with Jackson.
  • Add a dependency to Gson (2.8.5), if one does not exist already.

At this point, if the application is only using Jackson for automatically serializing and deserializing request and response objects in its REST APIs, the conversion should be complete. Your Spring Boot application should now be switched to using the Gson implementation, and function as before.

Notes:

  • I did not need to add the "preferred-json-mapper" property to my application.properties as stated in the link above. Spring Boot 2.0.3 seems to be capable of detecting and using the Gson dependencies on its classpath automatically.
  • Additional complications and code changes may arise if you are explicitly using any of the Jackson library classes in your code. These will need to be manually converted to use the equivalent Gson classes instead.
  • Jackson dependencies may be pulled in transitively from other AAI modules (such as aai-common). Excluding these manually in your own pom may be risky, so ideally they need to be fixed at the source.

Example an example of the changes I made to the pom.xml can be found in this changelist for the AAF rProxy project: https://gerrit.onap.org/r/gitweb?p=aaf/cadi.git;a=commitdiff;h=0d9b3896ad594816b1eb7048949114e6a18c4bd4
(Note that this changelist contains other code changes but only the pom.xml changes are required for switching to Gson)

Discussion from Seccom meeting 10th Oct


Actions:

  • Maybe approach Cassandra team about also replacing Jackson?
  • Share this with PTLs and aim for implementation in Dublin release?

Cassandra Usage of Jackson


  • From: Greg Matza [mailto:greg@scylladb.com]
    Sent: Friday, 12 October 2018 12:02
    To: Keong Lim <Keong.Lim@huawei.com>
    Subject: Re: ScyllaDB


    Sounds great! 


    I know that time is very short, but we are holding the Scylla Summit in 4 weeks - 6th November and 7th November. We will be near San Francisco. http://www.scylladb.com/scylla-summit-2018/  We welcome you and your team to attend the Summit, in order to better understand the technology and integrate with the user community. 


    There is also a full day of training on 5th November that would likely be valuable.


    If it is possible for you and your team to come, I will be happy to waive the $500 fee for Training and Summit. Let me know, and I can arrange for the free registration. If necessary, I can also  get you a letter of invitation, which may help with obtaining a visa.


    Greg


  • From: Greg Matza [mailto:greg@scylladb.com]
    Sent: Wednesday, 24 October 2018 03:58
    To: Keong Lim <Keong.Lim@huawei.com>
    Subject: Re: ScyllaDB


    Keong,


    Google Alert notified me of your posting on the ONAP site. From that notification, I saw your notes on our conversation and learned a bit more about your project.


    We are excited by the prospect of inclusion in the ONAP AAI project. Our co-founders - Dor Laor and Avi Kivity are well experienced with Linux Foundation projects, as they are the creator and early manager of the KVM Hypervisor. So working with OSS Foundations is in our company's DNA. In addition, telecom are among the early adopters of Scylla - AT&T, Verizon, Huawei, Comcast, T-mobile are all either running Scylla in Production or are in POC. 


    As you continue to research the viability of Scylla for AAI, I'd like to make our technical and executive resources available to you. From a technical perspective, we are available to help with any POC or evaluation. (Installation, hardware recommendations, monitoring setup, or other practical questions). From an executive perspective, we'd be happy to discuss any potential questions with roadmap, licensing or other 'vision'-type questions. 


    Let me know if/when you are ready to engage with our technical or executive resources.


    Greg

ScyllaDB as replacement for Cassandra

Quick CVE comparison:

Discussion from PTL meeting 15th Oct


Actions:

  • Should ONAP pay for Jackson project to fix the vulnerabilities?
  • Could ScyllaDB be used to replace Cassandra?

Conclusion So Far

For AAI project:

  • code already uses Google gson, so
    • gson has already been scanned for vulnerabilities
    • gson does not appear on Seccom lists for package upgrade or replacement
  • currently no usage for the alternative Json libraries, so
    • introducing the new libraries may also bring in new vulnerabilities and problems
  • there is already at least one worked example for translating from Jackson usage to gson usage, facilitating further conversions to gson
  • the POC shows that transitive dependencies on Jackson could also be eliminated in some cases
  • there are nearly 30 AAI repositories and over 200 files that need to be updated
  • fully eliminating Jackson may not be possible due to other tools, such as Cassandra
    • could Cassandra be replaced by using ScyllaDB?


  • No labels