Multi-cluster support

Proposal: MultiClusterSupport.pptx

Thoughts:

Common DB shared across clusters
At this point, as PAP is not a bottleneck for event processing, do not need more than one PAP per cluster
Run PAP on each cluster in active-active vs active-hot vs active-cold
Do PAPs manage PDPs across clusters?
- Would require cross-cluster DMaaP
Is there a way to trigger PAPs, in other clusters, to examine the DB? (DB triggers maybe?)
Separate PDP Groups for each cluster?
- And possibly multiple groups within a cluster, to support multi-tenancy
- Maybe don't separate them
Would clamp talk to all of the PAPs across the clusters?
- What can kubernetes do to support this?
PDPs can be active-active
- Can deploy same policy across clusters
Prefer a single point for configuring policies
- Implies a shared DB to store all policies
- Transactions would be required to prevent conflicting updates by multiple policy-api components
How should the consolidated health check work with multiple clusters? Query a PAP in each cluster? Query one single PAP?
- The additional services (e.g., A&AI, DMaaP) may be available in one cluster, but not the other. How would that be reported?
What about pdp-policy deployment status? Should one query report status for PDPs on all clusters?
- If so, then that implies that the deployment status is kept in a shared DB
Are PAPs aware of PDPs across all clusters?
- If not, then need a flag in the DB to indicate which PDPs are in which cluster so that PAP doesn't remove PDPs from other clusters
- If not, then need a way to trigger the PAPs in the other clusters to deploy/undeploy policies to/from their respective PDPs
- Can we use a shared DMaaP for POLICY-PDP-PAP topic? Or configure the PAPs in each cluster so they can communicate with the DMaaPs in the other clusters?