Resiliency Levels (New)

Owned by Chaker Al-Hakim
Last updated: Aug 24, 2020
1 min read

Level Definitions

  •  

    • Level 0: no redundancy

    • Level 1: support manual failure detection & rerouting or recovery within a single site; tested to complete in 30 minutes

    • Level 2: support automated failure detection & rerouting 

      • within a single geographic site

      • stateless components: establish baseline measure of failed requests for a component failure within a site 

      • stateful components: establish baseline of data loss for a component failure within a site

    • Level 3: support automated failover detection & rerouting 

      • across multiple sites 

      • stateless components 

        • improve on # of failed requests for component failure within a site 

        • establish baseline for failed requests for site failure 

      • stateful components 

        • improve on data loss metrics for component failure within a site 

        • establish baseline for data loss for site failure

Minimum Levels (Dublin)

  • Runtime Projects: Level 2 (stretch goal Level 3)

    • NOTE: For Dublin, the building blocks will be put in place for Level 3 geo-redundancy, and a few projects will pilot it

  • All other Projects: Level 1 (stretch goal Level 2)

Guidance for Implementation

  • Level 2 resiliency within a single site can be easily implemented by project teams using OOM, Kubernetes clusters, and health checks.

  • CNI - OOM is introducing CNI which will allow for multi-site Kubernetes clusters (VxLAN or BGP).  One can deploy pods and label them by geo location, which will be scheduled to corresponding labeled nodes.  Labels would need to be defined in the Helm charts.  See OOM-1506

  • MUSIC - MUSIC supports geo-redundancy, particularly for stateful components where traditional clustering techniques are insufficient (not performant, for example).  Pilot projects: Portal, OOF, and SDC.

  • Integration testing details TBD.

Contacts

OOM, MUSIC, and Integration teams.