Background
The DCM is one of the components of ONAP4K8s. It will run as a microservice exposing Rest APIs, external components will use REST to communicate with the DCM while other microservices will use gRPC. The DCM will perform the following functions;
- User creation (currently one user per logical cloud)
- Namespace creation (currently one namespace per logical cloud)
- Generate intermediate CA key for each edge which is signed by a root or intermediate key
- Logical Cloud creation - Create Istio control planes for the logical clouds.
Different components (microservice) work together with the DCM to make the above possible, the components are;
- Main DCM Microservice (contains the service mesh Module(formally Logical Cloud Controller), User Module and Namespace Module, Quota Module(Limits resources available to each logical cloud))
- CA Key Distribution Controller ( Generate intermediate CA key for each edge which is signed by an root or intermediate key)
Design Overview
Fig 1: DCM Components
Sample DCM DB tables
Logical Cloud | User | Namespace | Cluster |
---|---|---|---|
logical-cloud-1 | user1 | ns1 | c1 |
User | Permissions | user-key | user-csr |
---|---|---|---|
user1 | permission-1 | ajjfoiogwf | ekfapoekp |
DCM Source Code Directory Structure
dcm
├── core
│ └── main.go
├── namespace-controller
│ └── namespace.go
├── quota-controller
│ └── quota.go
├── service-mesh-controller
│ └── service-mesh.go
└── user-controller
└── user.go
GO API
func createNamespace(logicalCloudName Namespace string) error //Stores the namespace for the logical cloud in the database func createUser(user logicalCloudName string permissions []map[string]map[string][]string) error //Stores the user details for the logical cloud in the database func createKVpair(name description string userData map[string]string kvPair []map[string]string) error //Stores a new key value pair in the database func addCluster(cluster logicalCloudName string) error //Associates a new cluster with the logical cloud func addUserPermissions(user permissionName string apiGroups resources verbs []string) error func applyConfig(logicalCloudName string) error //Talks to the Resource Synchronizer to start the actual creation of all the resources for the logical cloud func getKVPair(name string) ([]map[string]string error) func getClusterConfig(cluster logicalCloudName string) ([]byte, error) //Returns Kubeconfig for the cluster in JSON format func getNamespace(logicalCloudName string) (string error) func getUser(logicalCloudName string) (string error) func getClusters(logicalCloudName string) ([]string error) func getUserPermissions(user string) (([]map[string]map[string][]string) error) //Sample output [{"permission-1": {"apiGroups": ["stable.example.com"], "resources" : ["secrets", "pods"], "verbs" : ["get", "watch", "list", "create"] }}, {"permission-2": {"apiGroups": [""], "resources" : ["configmaps"], "verbs" : ["*"] }}]
REST API
1. Create Logical Cloud
URL: /v2/projects/<project-name>/logical-clouds POST BODY: { "metadata" : { "name": "lc-1", //unique name for the record "description": "logical cloud for walmart finance department", //description for the logical cloud "userData1":"<user data>", "userData2":"<user data>" }, "spec" : { "namespace" : "ns-1", // one namespace per logical cloud "user" : { "user-name" : "user-1", //name of user for this cloud (username and logical cloud name would be used as subject for the user key) "type" : "certificate", //type of authentication credentials used by user (certificate, Token, UNPW) "user-permissions" : [ { "permission-name" : "permission-1", "apiGroups" : ["stable.example.com"], "resources" : ["secrets", "pods"], "verbs" : ["get", "watch", "list", "create"] }, { "permission-name" : "permission-2", "apiGroups" : [""], "resources" : ["configmaps"], "verbs" : ["*"] } ] } } } Return Status: 201 Return Body: { "name" : "logical-cloud-1", "logical-cloud-name" : "logical-cloud-1", "namespace" : "ns-1", "user-name" : "user-1" }
Important points to Note
- cluster CA and cluster CRT will be gotten when a cluster is registered and this will be used to create the user crt after the user csr and user key are created
Kubeconfig will be put in the mongoDB
2. Get Logical Cloud kubeconfig
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/kubeconfig?cluster-reference=cluster-1 GET Return Status: 201 Return Body : { "apiVersion": "v1", "clusters": [ { "cluster": { "certificate-authority-data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ERXhOakl5TlRReU1sb1hEVE13TURFeE16SXlOVFF5TWxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTXZ5ClpSMmtHZ201OHJtRUYxZ0VVNDlwR281TjR5WlZzbUtsemJxeitHcmp3MWY4ZHBaa01JN1RYbm1xaXdjbmpiZksKdlZDYmFKblBwRm9Wc0gyMTFMRVYxa1pqQ2RZakgrQnA4VUNadFpOZnJha1o1TU91RW40MXlpbDRxVTFxRnBYcgpvMnAvTTJNSTc0bzdYSis5V2VUNmZ1MFJ0RjRjK1p6K3IzbWY3YWFnem9weEo1TzcyN010WkN5bzJHaWNJdzgyCk1uSmUrbHBnNDdEdTNwK0JzOVZ4cENNVjhUTFBDWWFxVUZHZWZ1U25zTHpCOWFHUGJaMi9kWlMrQll6VGJ1dWIKZ0pzbmxKd1o2Z1orVkJKWGtxcFN4ZmJTWFE3V2VLR1BkMkptYk04THFtd2UxcEtIMnNnVEs0cnBuM3dKdzk1Uwp5c01LZWp5aS9TcmZWci9ZdmRNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIRkJ3VUJrRE56MDV3cGlVVkhaeld3Z0JWWTgKTFRIV2RZYUliaTZzeCthTUhIOFF3cERhSUIvME9KUlZmc01XNXd2eldTWE05d28zR2ltMlYxNnBsN0E3ZXRkLwo3OWZiT0FaTTh0bUFHMVlraFlJbjc0NzRvaE5GZjhLdjFqY3ZIUStIREZZRTRHdTBXUXhBQU9sRmh2SUNKc1VDCmRrN25mREpMRTIwa1E0M1ZIMnc3Ukg3clFVcUVNVnhVU2VTTWdid0xEQms5bWFQNm83RjdsT0ZqQnJibmhaVlgKNDA3U3Z2aTFRM0x6eCtubklhY3RidkZaUGFDRlpPMlkvS0dpcFpYM3o0R3hiR0sxM3VweExuSllHZEI4eFBrRApmK3FISFYvMHVVV2Yzb2JpSUNTT09qUjF5VnB3eXdIVFcrTHhyM1BZcXQ5b0NTRXErYitPUDE2SVV5az0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "server": "https://1.2.3.4:6443" }, "name": "kubernetes" } ], "contexts": [ { "context": { "cluster": "cluster-1", "user": "user-1" }, "name": "user-1-context" } ], "current-context": "user-1-context", "kind": "Config", "users": [ { "name": "user-1", "user": { "client-certificate-data" : "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJZVovY05tQVE5NGd3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TURBeE1UWXlNalUwTWpKYUZ3MHlNVEF4TVRVeU1qVTBNamhhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW0zblA0eTdURjNrZldaZFoKaFdaV2RTSWZlNlpkbTdWK1BpcER6UWFHMTVuU0ZNSVRSeFVyYkhHdWlzakZQRlAzbUIyT09yN3BSQjJab3VDegppOFlYS21iYjJ6K2tjeWZxT1drcHhmTzlHQlV6SlYxL1BoUGU2dGRaSEp3c3FtNlhYZ2xkcTEvNjBSTWNwUVUxCi9LOXNZNHhWQ1djSkN4SEkvTnp4VDY0TU5zQlF3VldONXZWTTJOUDJtZDFOa2x2S3J2bnFRUERXTGxVWEx2THIKK2NESk50VytxcFc4dzVreXF5YWp1ZHQ4ZGw0dzZSY3FnL3VnbXRVMHRnVEdxcFdSYm5yZlFMSzBsaGJKejVMTgpmK1pNTjRCYllxWGRBZ2hFMTNEeHhYd2tHUHdnL3h0aFhManBaQzhjeTNlV0hCenV2cWY1aWJ2S0hRQ20zRmFjCjhBTlVpUUlEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFDV3BuY1RhTWowOWZDL25CQTF1NWhkbFQrQmdhc3NZSFVEeQplM2tQUXJlUXdseUhYTGtWdDdiSkIxT0l6Y1V3K2M5MVF6Mm9lRFBaNzZGNGlQMTd5RUgrUFZrMVVUSzBLRU9jCjM2cVpXTUdMK0ptZy9wTnFBNXRsNG1EUTVneFhHTENpa2JiYzRTM0oxL0FicmFVakRtM1FEOTd6UEhSUkZnN2oKN2VXMnB2V3ZEakRTWDZGejY0dEorRHB2NUpGZGRHNU5lQVErZ0hNOWFPVUdCVG1oZlYzZnl1NzkzV0cyUGlxMgpMMlZQU0YycU5DRG96Y3Z3am84VHkxbUpXSzIvTkVjN2ZMd24wbml3UTd3aXpMWHU0N1hvL3Frb2pBMUN6MW9YCkhid1JQMjZXdVNDTGpnNnpHVUh3VnBZWmV4Z3pkY05CRERQTnlPem94RTFwUVlXRXkrZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "client-key-data" : "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBbTNuUDR5N1RGM2tmV1pkWmhXWldkU0lmZTZaZG03VitQaXBEelFhRzE1blNGTUlUClJ4VXJiSEd1aXNqRlBGUDNtQjJPT3I3cFJCMlpvdUN6aThZWEttYmIyeitrY3lmcU9Xa3B4Zk85R0JVekpWMS8KUGhQZTZ0ZFpISndzcW02WFhnbGRxMS82MFJNY3BRVTEvSzlzWTR4VkNXY0pDeEhJL056eFQ2NE1Oc0JRd1ZXTgo1dlZNMk5QMm1kMU5rbHZLcnZucVFQRFdMbFVYTHZMcitjREpOdFcrcXBXOHc1a3lxeWFqdWR0OGRsNHc2UmNxCmcvdWdtdFUwdGdUR3FwV1JibnJmUUxLMGxoYkp6NUxOZitaTU40QmJZcVhkQWdoRTEzRHh4WHdrR1B3Zy94dGgKWExqcFpDOGN5M2VXSEJ6dXZxZjVpYnZLSFFDbTNGYWM4QU5VaVFJREFRQUJBb0lCQURSeCs3RTd5MU1ndFhXSQpPMWRuZFFTZ0ZSU2x3dS9TWWhwZ01XeklwZFcyZW9vc0NVcXlGbXJIVWtSWWcwZmRYeWk5MTR0emVNWlVZYzN5CmxENHkvUDk5b080dFlyREJweDNrbm9XNnVXK1ZQeUo4am42SFAydmlacG5qQ0tJWkdoQkxnb0JicVFTN3VLN2wKdWhkWnFXdFBIQ1JHMEdNZWhiamVZcndwRHMrc3V5Y0tUNlh2OGR0am9PRGlSenpodTBBWjErSDBMbC9MaUp5NgpubUF1M3IzUDN2dGt1ZndoOXB2SjA2TjhHVWhRZFlvVENBMHQ4c0lEN0Q0ZUZGcXdjM2FVTmlyVGdtcllmcVFkCnJQMnRUMkYxbEE5bEJ1bUJmNDBCUFMrejR0MS92Tk5XSEtseFFzaDRkTnNJczlFbHB1YkVTMlovbmJkUHpEOXUKUHlmWmNCMENnWUVBemZEZ21TVVZ2REpJMEZWWnNENWszNWxRWFpmQ3ZLZWpaaU5ScTZzaDlROUhBcTdaN3dESwo3RmNHSGZFQVhNZ1YyQ2VibUtkbFZ4aFJMQUVQb0todzNCbG94RlhiNC9ESytYdmV0ek4vUnQzQ0VGN29jZTFECko0eXVTS0pYR25RVW82NXV6RVBJMmVVdi81Njh5dEdDdXlNRmdIZDNTRHBxeUZqSjhlVXp3RGNDZ1lFQXdVU2kKVVpkMTEzVnFWeEpUQytSeTlsRzNIeGZFekhBbUdkMXNQNFdmYm9nZ3hjbmZrZk9ndU1ua0dzVVNnNGZjN3BKNworVDd2dFg2QVRXQkUzeUV1SHdMcytJd2VQT2RFeDNBZytSYnBKU25pTEc2WmUxdER0K2NtdTdObGZGRWRuR0l2CkUvTkt1c3FLdmduN3FzK216OHR5UmwxZ0pYMGdjT3J1TFVnbnNUOENnWUVBaGlUdUY3TnBXZ0lqSGRsS3A1dXMKMTEwbFZTR2lqb0pmMUFzVGlzL1pPYWh1NTlkL1M4aG5aZFUxdmRFYkhGU1VyZ3oydEZQdGxmTFlCT0xZREIxTQpEb0phbFBFY1gzaWNyaSs2bmZqa1lnUFhBaFRnTWoyTExicmNWNkd2UFNMNXdyaS9vVHhTRzJUSGhDa2c3cmZVCkFSUEo1S2xzd0ZhVThkV3NEVzN2N0xjQ2dZQlhlZGc2TStLcmpjSisvSlZJR2JPTEY3dFp3R2xiMnhyenRBdk4KeUk0NytqTlRNcWNWcVg3Q2hPYlEwd2dwTG5KcUxUVWR3RVhCRVN2RFdlSnlWOU5IU0F5NEJydWM5MVJqTExaUAo1L1hJMDJkQ2t5QzIrN3p2M1JqajlqUG1DOVRxTm1wMmpqVHh6TUQxZVJGRzQ4dnQyM2l5cm9yWkRRU0U5MkNzCmNDOC9Bd0tCZ1FDamRrVTNOcWFPZHUzMlpkWm5IK0cxY0NwbGtRK1VFZXNmTlcrWjdiWWtocng2Z2l6eCtjY0UKS3FhNTRxTmJkQVdLQXRzSU5lZ3pQU0toV0g0eXFnaWYrNmQ1aVVZUlFoUUhNVG0yTHpEQ2xGQ001N2FPdmQ3YQpNUEZxR3BNV2tqVmlnZjlwdHV3YW1qWkYwbm5MUVE5dWIzMlB3Zjk0Yzl4NFBEdkpmdTQrUHc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=" } } ] }
3. PUT (Change logical cloud contents)
URL: /v2/projects/<project-name>/logical-clouds/<name> PUT BODY: { "metadata" : { "description": "logical cloud for walmart finance department", //description for the logical cloud "userData1":"<user data>", "userData2":"<user data>" }, "spec" : { "namespace" : "ns-1", // one namespace per logical cloud "user" : { "user-name" : "user-1", //name of user for this cloud (username and logical cloud name would be used as subject for the user key) "type" : "certificate", //type of authentication credentials used by user (certificate, Token, UNPW) "user-permissions" : [ { "permission-name" : "permission-1", "apiGroups" : ["stable.example.com"], "resources" : ["secrets", "pods"], "verbs" : ["get", "watch", "list", "create"] }, { "permission-name" : "permission-2", "apiGroups" : [""], "resources" : ["configmaps"], "verbs" : ["*"] } ] } } } Return Status: 200 (OK) Return Body: { "name" : "logical-cloud-1", "logical-cloud-name" : "logical-cloud-1", "namespace" : "ns-1", "user" : "user-1" }
4. POST (Associate cluster with logical cloud )
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/cluster-references/ POST BODY: { "cluster-reference": "cluster-name" //name of the cluster } Return Status: 200 (OK) Return Body: { "name" : "cluster-reference-1", "cluster-name" : "cluster-1" }
5. GET Logical Cloud
GET URL: /v2/projects/<project-name>/logical-clouds/<name> RESPONSE BODY: { "metadata" : { "name": "lc-1", //unique name for the record "description": "logical cloud for walmart finance department", //description for the logical cloud "userData1":"<user data>", "userData2":"<user data>" }, "spec" : { "namespace" : "ns-1", // one namespace per logical cloud "user" : { "user-name" : "user-1", //name of user for this cloud (username and logical cloud name would be used as subject for the user key) "type" : "certificate", //type of authentication credentials used by user (certificate, Token, UNPW) "user-permissions" : [ { "permission-name" : "permission-1", "apiGroups" : ["stable.example.com"], "resources" : ["secrets", "pods"], "verbs" : ["get", "watch", "list", "create"] }, { "permission-name" : "permission-2", "apiGroups" : [""], "resources" : ["configmaps"], "verbs" : ["*"] } ], "clusters" : ["cluster-1", "cluster-2", "cluster-3] } } }
6. DELETE Logical Cloud
DELETE URL: /v2/projects/<project-name>/logical-clouds/<name> RESPONSE STATUS: 204
7. Add user permissions
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/user-permissions POST BODY: { "name" : "permission-3", "apiGroups" : [""], "resources" : ["jobs"], "verbs" : ["list"] } Return Status: 200 (OK) Return Body: { "name" : "permission-3", "apiGroups" : [""], "resources" : ["jobs"], "verbs" : ["list"] }
8. GET User Permissions
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/user-permissions GET: Return Body: {"permissions" : [ { "name" : "permission-1", "apiGroups" : ["stable.example.com"], "resources" : ["secrets", "pods"], "verbs" : ["get", "watch", "list", "create"] }, { "name" : "permission-2", "apiGroups" : [""], "resources" : ["configmaps"], "verbs" : ["*"] }, { "name" : "permission-3", "apiGroups" : [""], "resources" : ["jobs"], "verbs" : ["list"] } ] }
9. PUT (Update User permissions)
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/user-permissions/permission-3 POST BODY: { "apiGroups" : [""], "resources" : ["pvc"], "verbs" : ["list"] } Return Status: 200 (OK) Return Body: { "name" : "permission-3", "apiGroups" : [""], "resources" : ["pvc"], "verbs" : ["list"] }
10. Delete Permissions
DELETE URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/user-permissions/<permission-name> RETURN STATUS: 204
11. Create logical cloud Quota ( quota will be applied to each cluster in the logical cloud)
This allows resources to be tuned for the logical cloud
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/cluster-quotas POST BODY: { "metadata" : { "name" : "quota-1", "description": "desc" }, "spec" : { "limits.cpu": "400", "limits.memory": "1000Gi", "requests.cpu": "300", "requests.memory": "900Gi", "requests.storage" : "500Gi", "requests.ephemeral-storage": "", "limits.ephemeral-storage": "", "persistentvolumeclaims" : " ", "pods": "500", "configmaps" : "", "replicationcontrollers": "", "resourcequotas" : "", "services": "", "services.loadbalancers" : "", "services.nodeports" : "", "secrets" : "", "count/replicationcontrollers" : "", "count/deployments.apps" : "", "count/replicasets.apps" : "", "count/statefulsets.apps" : "", "count/jobs.batch" : "", "count/cronjobs.batch" : "", "count/deployments.extensions" : "" } } RETURN STATUS: 201
12. GET logical cloud Quota
GET URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/cluster-quotas/<quota-name> RETURN BODY: { "metadata" : { "name" : "quota-1", "description": "desc" }, "spec" : { "cpu": "400", "memory": "1000Gi", "pods": "500" } }
13. Update Logical Cloud Quota
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/cluster-quotas/<quota-name> PUT BODY: { "metadata" : { "description": "desc" }, "spec" : { "cpu": "400", "memory": "1000Gi", "pods": "500" } } RETURN STATUS: 201
14. POST (Apply all the created configuration, this creates the K8s resources)
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/apply Return Status: 200 (OK) Return Body: { "logical-cloud-name" : "logical-cloud-1", "namespace" : "ns-1", // one namespace per logical cloud "description": "logical cloud for walmart finance department", //description for the logical cloud "user" : "user-1", "clusters" : ["cluster1", "cluster2", "cluster3"] "quota-name" : "quota-1" }
15. GET (Check status of operation)
GET URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/status GET BODY: Return Body : { "metadata" : { "name" : "logical-cloud-1" "description" : "<description>", } "clusters" : [ "cluster-1" : { "namespace-status" : "<status>", "role-status" : "<status>", "role-binding-status" : "<status>" } "cluster-2" : { "namespace-status" : "<status>", "role-status" : "<status>", "role-binding-status" : "<status>" } ], "status": "Creation in Progress " //Created, Creation Failed }
16. Add Key Value pair to logical cloud database
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/kv-pairs POST BODY { "metadata":{ "name":"<name>", "description":"<description>", "userData1":"<user data>", "userData2":"<user data>" }, "spec":{ "kv":[ { "key1":"val1" }, { "key2":"val2" } ] } } RETURN STATUS: 201 RETURN BODY: { "metadata":{ "name":"<name>", "description":"<description>", "userData1":"<user data>", "userData2":"<user data>" }, "spec":{ "kv":[ { "key1":"val1" }, { "key2":"val2" } ] } }
17. PUT (Update kv pair)
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/kv-pairs/<name> PUT BODY { "metadata":{ "description":"<description>", "userData1":"<user data>", "userData2":"<user data>" }, "spec":{ "kv":[ { "key1":"val3" }, { "key2":"val4" } ] } } RETURN STATUS: 201 RETURN BODY: { "metadata":{ "description":"<description>", "userData1":"<user data>", "userData2":"<user data>" }, "spec":{ "kv":[ { "key1":"val3" }, { "key2":"val4" } ] } }
18. GET KV pair
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/kv-pairs/<name> RETURN STATUS: 200 RETURN BODY: { "metadata":{ "name":"<name>", "description":"<description>", "userData1":"<user data>", "userData2":"<user data>" }, "spec":{ "kv":[ { "key1":"val1" }, { "key2":"val2" } ] } }
19. DELETE KV pair
URL: /v2/projects/<project-name>/logical-clouds/<logical-cloud-name>/kv-pairs/<name> RETURN STATUS: 204