Multi Cluster Application Scheduler HPA Placement Controller
- Marcus Williams
- 1 Placement Controllers
- 1.1 HPA Placement Controller
- 1.1.1 HPA Placement Controller Design
- 1.1.1.1 HPC Registration
- 1.1.1.2 K8s Cluster Registration
- 1.1.1.3 HPA Instantiation Flow
- 1.1.2 HPA Placement Controller Northbound APIs
- 1.1.2.1 OpenAPI File: openapi.yaml
- 1.1.2.2 HPA Intent API
- 1.1.2.2.1 HPA Placement Intent
- 1.1.2.2.1.1 POST
- 1.1.2.2.2 POST
- 1.1.2.2.2.1 GET, DELETE
- 1.1.2.2.3 POST
- 1.1.2.2.1 HPA Placement Intent
- 1.1.2.3 Sites API
- 1.1.2.4 Feature API
- 1.1.2.4.1 POST
- 1.1.2.4.2 POST
- 1.1.2.4.3 GET, DELETE
- 1.1.2.4.4 POST
- 1.1.3 HPA Placement Controller gRPC APIs
- 1.1.4 HPA Placement Controller Targeted Hardware Features
- 1.1.1 HPA Placement Controller Design
- 1.1 HPA Placement Controller
- 2 Controllers gRPC Interface
- 2.1 gRPC Code
- 2.2 gRPC Messages
- 2.3 gRPC Server
- 2.4 gRPC Client
- 2.5 gRPC Controller Registration
- 3 ETCD Client
- 4 Backup
Placement Controllers
HPA Placement Controller
HPA Placement Controller is a stand-alone microservice that returns available cluster sites for placing applications that require specific hardware acceleration features.
HPA Placement Controller Design
HPA Placement Controller (HPC) is a gRPC Microservice written in Go Lang that stores data in mongo DB and etcd.
-- Store cluster features in unique sets of whats available in the cluster
Feature Retrieval Design
HPA Placement Controller on prompt from scheduler or northbound API goes to a k8s cluster and retrieves feature labels
requires auth/kubectl file for cluster access - some security implications
allows use of placement controller independently of scheduler
HPC Registration
K8s Cluster Registration
HPA Instantiation Flow
HPA Placement Controller Northbound APIs
OpenAPI File: openapi.yaml
HPA Intent API
HPA Placement Intent
https://NODE_IP:30280/api/multicloud-k8s/v2/project/{project-name}/rb/{rb-name}/{rb-version}/hpa-placement-intent
Use HPA API expressed here Policy Specification and Retrieval for OOF as a base for hardware features.
POST
POST
URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/hpa-placement-intents
{
"metadata": {
"name": "Edge-1-intent",
"description": "Edge Intent for Sec. and Load Balancing",
"status": "active",
"userdata1": {
"external-proxy-ip": "10.7.100.4"
},
"userdata2": ""
}
}
URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/hpa-placement-intents/{hpa-intent-name}/workload-placements/
{
"spec": [
{
"application-name": "Edge-Security",
"workload-resource": "Firewall",
"platform-features": [
{
"platform-feature-name": "pci-0300_102b.present",
"mandatory": "True",
"score": 5,
"platform-feature-attribute": [
{
"feature-attribute-key": "qatPresence",
"feature-attribute-value": "True",
"operator": "ALL",
"unit": ""
}
]
},
{
"platform-feature-name": "memory-numa",
"mandatory": "True",
"score": 2,
"platform-feature-attribute": [
{
"feature-attribute-key": "numaNodes",
"feature-attribute-value": "2",
"operator": "=",
"unit": ""
},
{
"feature-attribute-key": "numaNodes",
"feature-attribute-value": "4",
"operator": "=",
"unit": ""
}
]
}
]
}
]
}
RETURN STATUS: 201GET, DELETE
POST
GET
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/hpa-placement-intent/{intent-name}
RETURN STATUS: 200
RETURN BODY:
{
"metadata": {
"name": "Edge-1-intent",
"description": "Edge Intent for Sec. and Load Balancing",
"status": "active",
"userdata1": {
"external-proxy-ip": "10.7.100.4"
},
"userdata2": ""
},
"spec": [
{
"application-name": "Edge-Security",
"workload-resource": "Firewall",
"platform-features": [
{
"platform-feature-name": "pci-0300_102b.present",
"mandatory": "True",
"score": 5,
"platform-feature-attribute": [
{
"feature-attribute-key": "qatPresence",
"feature-attribute-value": "True",
"operator": "ALL",
"unit": ""
}
]
},
{
"platform-feature-name": "memory-numa",
"mandatory": "True",
"score": 2,
"platform-feature-attribute": [
{
"feature-attribute-key": "numaNodes",
"feature-attribute-value": "2",
"operator": "=",
"unit": ""
},
{
"feature-attribute-key": "numaNodes",
"feature-attribute-value": "4",
"operator": "=",
"unit": ""
}
]
}
]
}
]
}
DELETE
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/hpa-placement-intent/{intent-name}
RETURN STATUS: 204
Sites API
https://NODE_IP:30280/api/multicloud-k8s/v2/placement/hpa/update-sites
Note: if this were to be uncoupled with scheduler it would need to have body for POST containing the following info and it would need to return app-context-name in return body:
candidate sites, apps, sub-apps
hpa-intent-name
Could we create another api endpoint "update-sites-standalone" that contains above to uncouple?
POST
POST
URL: /v2/placement/hpa/update-sites
POST BODY:
{
"intent-group":"intent-group-1",
"app-context":"app-context-1"
}
RETURN STATUS: 201
RETURN BODY:
{
"app-context-updated": "true"
}
Feature API
https://NODE_IP:30280/api/multicloud-k8s/v2/placement/hpa/features
POST
POST
GET, DELETE
POST
HPA Placement Controller gRPC APIs
see gRPC messages above.
HPA Placement Controller Targeted Hardware Features
Controllers gRPC Interface
Each controller will communicate with the Multi Cluster Application Scheduler through gRPC.
gRPC Code
gRPC patch: https://gerrit.onap.org/r/c/multicloud/k8s/+/100148
gRPC Messages
HPA Placement Controller
Traffic Controller
Register Controller
gRPC Server
gRPC Client
gRPC Controller Registration
?
ETCD Client
Generic ETCD Client module should be created in the scheduler/orchestrator and imported and used in scheduler and controllers.
Backup
gRPC getSites using profile
old
feature backup
Feature API
https://NODE_IP:30280/api/multicloud-k8s/v2/placement/hpa/features
POST
POST
GET, DELETE