{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Background","type":"text"}]},{"type":"paragraph","content":[{"text":"FPGA and GPUs are becoming common place to take away the load from cores. Some of the use cases for FPGA and GPU include","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Machine Learning and Deep Learning inference/training offloads","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Crypto and compression offloads","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Protocol (such as IPsec and PDCP) offloads","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Though fixed function accelerators do provide better performance, multiple accelerators are required in a given compute node if there are workloads of different types.","type":"text"}]},{"type":"paragraph","content":[{"text":"Being programmable, acceleration functions can be dynamically changed in FPGA/GPUs based on the need and hence FPGAs and GPUs are considered to be generic accelerators.","type":"text"}]},{"type":"paragraph","content":[{"text":"Openstack and Kubernetes orchestrators are adding support for FPGAs and GPUs.","type":"text"}]},{"type":"paragraph","content":[{"text":"ONAP being a service orchestrator (end-to-end), makes placement decisions for VNFs/workloads and hence the awareness of FPGA/GPU is required in ONAP like any other HPA feature.","type":"text"}]},{"type":"paragraph","content":[{"text":"Since Openstack is in advanced stages (in Rocky release) of providing FPGA support (via Cyborg project), initial support in ONAP would support Openstack based cloud regions. K8S support will be added later.","type":"text"}]},{"type":"paragraph","content":[{"text":"FGPAs can be used in following ways:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Pre-programmed with acceleration functions ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Orchestrator (Cyborg) programmed acceleration functions","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Supports dynamic programming - Based on workload being brought up, Openstack programs the acceleration function in the compute node where workload is about to be brought up.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Workload programmed","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In this case, workload programs the FPGA.","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"From ONAP perspective, there is not much difference between pre programmed and orchestrator programmed. These two are commonly known as AFaaS (Acceleration Function as a Service). In case of workload programmed, it is called FPGAaaS (FPGA as a Service). ","type":"text"}]},{"type":"paragraph","content":[{"text":"Since workload programmed FPGA is yet to be matured in Industry, AFaaS is considered initially. Rest of design note assumes the support for AFaaS.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Note on Openstack FPGA support","type":"text"}]},{"type":"paragraph","content":[{"text":"One good thing that is happening in Rocky release is that all resources, whether it is CPU, disk, memory, PCIe SRIOV VF or even FPGA/GPU, are represented in uniform way. And hence the request for resources via flavor is also becomes uniform.There are no longer unique way of representing flavor attributes. Please see this design note here : ","type":"text"},{"text":"https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/granular-resource-requests.html","type":"text","marks":[{"type":"link","attrs":{"href":"https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/granular-resource-requests.html"}}]}]},{"type":"paragraph","content":[{"text":"In summary: Any resource request in flavor always starts with property resource::. Any associated trait for the resource or traits by themselves can be represented as trait:=required.","type":"text"}]},{"type":"paragraph","content":[{"text":"In case of FPGA and GPU, there is additional generic property is getting added, whose syntax is function:=required. Note that this keyword is not interpreted by NOVA. It is only interpreted by Cyborg.","type":"text"}]},{"type":"paragraph","content":[{"text":"One example:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Say that Openstack instance supports ARRIA10 FPGAs and functions such as IPSEC_ACCELERATION, PUBLIC_CRYPTO_ACCELERATION, TENSORFLOW_ACCELERATION, COMPRESSION_ACCELERATION. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Say that compute nodes have 3 FPGA cards in each of them.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Say that Openstack operator thinks that there would be","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Some workload types that require IPSEC_ACCLERATION and COMPRESSION_ACCELERATION together.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Some workload types that require TENSORFLOW_ACCELERATION (2 of them) and COMPRESSION_ACCELERATION together","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Some workload types that require IPSEC_ACCELERATION, PUBLIC_CRYPTO_ACCLERATION and COMPRESSION_ACCELERATION together","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Openstack operator creates following flavors ","type":"text"}]},{"type":"paragraph","content":[{"text":"Flavor 1:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource1:CUSTOM_ACCELERATOR_FPGA=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait1:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function1:IPSEC_ACCLERATION_AF=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource2:CUSTOM_ACCELERATOR_FPGA=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait2:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function2:COMPRESSON_ACCLERATION_AF=required","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Flavor 2:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource1:CUSTOM_ACCELERATOR_FPGA=2","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait1:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function1:TENSORFLOW_ACCLERATION_AF=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource2:CUSTOM_ACCELERATOR_FPGA=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait2:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function2:COMPRESSON_ACCLERATION_AF=required","type":"text"}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Flavor 3:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource1:CUSTOM_ACCELERATOR_FPGA=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait1:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function1:IPSEC_ACCLERATION_AF=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource2:CUSTOM_ACCELERATOR_FPGA=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait2:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function2:PUBLIC_CRYPTO_ACCLERATION_AF=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resource3:CUSTOM_ACCELERATOR_FPGA=1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"trait3:FPGA_INTEL_ARRIA10=required","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"function3:COMPRESSON_ACCLERATION_AF=required","type":"text"},{"type":"hardBreak"},{"type":"hardBreak"}]}]}]},{"type":"paragraph","content":[{"text":"Some points to note are:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Grouping of various properties together is achieved by in resource, trait and function keywords.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Even though FPGA accelerators typically are PCIe cards, PCIe vendor ID/device ID and VFs are hidden. They don't need to be specified. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note that in above examples, only the FPGA accelerators are taken as an example. In reality, flavors will have other resources too.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"ONAP design considerations","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Leverage HPA functionality added in Multi-Cloud, OOF and A&AI.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add new HPA feature called \"FPGA-AFaaS-Acceleration\"","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Have following HPA feature attributes:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"fpga_device. As per above examples, this attributes takes value \"FPGA_INTEL_ARRIA10\".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"fpga_AF_function: As per above examples, this attributes takes values such as IPSEC_ACCELERATION_AF, TENSORFLOW_ACCELERATION_AF, PUBLIC_CRYPTO_ACCELERATION_AF or COMPRESSION_ACCELERATION_AF","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"fpga_AF_count: This holds the number of AFs supported by the site for each workload","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Changes in R4:","type":"text"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Component","type":"text"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Changes","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"OOF","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"No changes are expected in OOF. Our design goal for HPA is to ensure that there are no changes to the OOF for every new HPA feature.","type":"text"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"A&AI","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"No changes are expected","type":"text"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Multi-Cloud","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"No changes in existing Openstack plugins","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce new Openstack plugin for Rocky","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Flavor discovery and flavor discovery in each flavor to follow new method introduced in Rocky release (resource, trait and function tuples)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Support for following features","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Continue to support following features","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"VCPUS","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Memory","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Disk","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"NUMA","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Huge pages","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"NI acceleration","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PCIe passthrough (Normal and SR-IOV)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"OVS-DPDK","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Core affinity and thread affinity policies","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Additional support","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"FPGA","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Stretch","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"GPU","type":"text"}]}]}]}]}]}]}]}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Discovery by Multi-Cloud:","type":"text"}]},{"type":"paragraph","content":[{"text":"","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"HPA policies","type":"text"}]},{"type":"paragraph","content":[{"text":"","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"TOSCA compute requirements","type":"text"}]},{"type":"paragraph","content":[{"text":"","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}],"version":1}