Intel SGX at a glance and its role in the cloud
Intel Software Guard Extensions (Intel SGX) are a set of CPU instructions that create private protected areas (enclaves) in the address space of an application where user-level code is located. The technology ensures the confidentiality and integrity of sensitive data. By isolating them in an enclave, they gain additional protection from both unauthorized external access, including from the cloud service provider, and from internal threats, including attacks from privileged software.
Work principles.Intel SGX technology allocates a Processor Reserved Memory (PRM) to store code and enclave data. The CPU protects it from all external calls, including kernel and hypervisor access. The PRM contains an Enclave Page Cache (EPC) of 4 KiB page blocks, each page must be owned by only one enclave, and their state is captured in the Enclave Page Cache Metadata (EPCM) and monitored by the CPU.
EPC security is ensured by the Memory Encryption Engine (MEE), which generates encryption keys stored on the CPU. It is assumed that pages can only be decrypted inside the physical processor core.
Benefits.Intel SGX helps increase the level of trust in the public cloud on the part of organizations using sensitive data in their work (passwords, encryption keys, identification, biometric, medical data, as well as information related to intellectual property). We are talking about representatives of a wide variety of industries - the financial sector, medicine and health care, retail, game development, telecom, media.
Our Approach to Implementing Intel SGX
To make it possible to allocate virtual machines with Intel SGX enclaves in the public cloud of G-Core Labs, we had to go from compiling the patched KVM and QEMU kernel to writing Python scripts in OpenStack Nova services. We decided to define the compute nodes that were planned to be used to allocate high-security virtual machines into a separate aggregator - a type of computational resources that requires additional configuration. On such nodes it was necessary:
- Enable Intel SGX BIOS support.
- Install patched QEMU / KVM.
Initially, we did not have an understanding of how this should work and what in the end we have to screw in order to get the VM of the desired configuration. Part of the Intel Developer Guide helped to sort this out. With its help, we learned how to prepare a computational node to work with SGX and what additional parameters the XML configuration file of a virtual machine should have. Here we found comprehensive information on how to create a guest machine using Intel SGX using KVM virtualization. To ensure that we were able to provide support for this technology, we used two methods:
- Checked the / dev / sgx / virt_epc section in the / proc / $ PID / smaps file:
[root@compute-sgx ~]# grep -A22 epc /proc/$PID/smaps 7f797affe000-7f797b7fe000 rw-s 00000000 00:97 57466526 /dev/sgx/virt_epc Size: 8192 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr sh mr mw me ms pf io dc dd hg
- And we used this shell script, after installing the SGX driver (all actions were carried out inside the VM):
[root@sgx-vm ~]# cat check_sgx.sh #!/bin/bash METRICS="sgx_nr_total_epc_pages \ sgx_nr_free_pages \ sgx_nr_low_pages \ sgx_nr_high_pages \ sgx_nr_marked_old \ sgx_nr_evicted \ sgx_nr_alloc_pages \ " MODPATH="/sys/module/isgx/parameters/" for metric in $METRICS ; do echo "$metric= `cat $MODPATH/$metric`" done [root@sgx-vm ~]# curl -fsSL https://raw.githubusercontent.com/scontain/SH/master/install_sgx_driver.sh | bash -s - install -p metrics -p page0 [root@sgx-vm ~]# ./check_sgx.sh sgx_nr_total_epc_pages= 2048 sgx_nr_free_pages= 2048 sgx_nr_low_pages= 32 sgx_nr_high_pages= 64 sgx_nr_marked_old= 0 sgx_nr_evicted= 0 sgx_nr_alloc_pages= 0
It should be borne in mind that if one page is 4 KiB, then 2048 pages require 8 MiB (2048 x 4 = 8192).
Development difficulties and overcoming them
The lack of any technical documentation for integrating Intel SGX into OpenStack was our main difficulty at the time of implementation. The search led us to an article by the SecureCloud project, which presented a way to manage virtual machines with SGX enclaves.
The information found helped to understand what exactly we have to work on. As a result, we have formed the following tasks:
- Get the OpenStack Nova service to generate an XML file with additional parameters for virtual machines with Intel SGX support.
- Write an OpenStack Nova scheduler filter to determine the available memory for enclaves on compute nodes and perform some other checks.
Their execution was enough to integrate Intel SGX into our public cloud.
In addition, we have added the collection of statistics taking into account the EPC:
# openstack usage show
Usage from 2020-11-04 to 2020-12-03 on project a968da75bcab4943a7beb4009b8ccb4a:
+---------------+--------------+
| Field | Value |
+---------------+--------------+
| CPU Hours | 47157.6 |
| Disk GB-Hours | 251328.19 |
| EPC MB-Hours | 26880.02 |
| RAM MB-Hours | 117222622.62 |
| Servers | 23 |
+---------------+--------------+
Safe environment for running containerized applications
After learning to provision Intel SGX-enabled VMs, we used Scontain's SCONE platform to ensure that containerized applications can safely run in the event of privileged software threats. For Docker, Kubernetes, and Rancher environments, this solution requires an Intel SGX-enabled processor and Linux SGX driver to transparently secure file systems.
The launch of each of the containers is possible only if there is a configuration file created by the client extension of the SCONE platform. It contains encryption keys, application arguments, and environment variables. Files, network traffic, and standard I / O streams (stdin / stdout) are transparently encrypted and inaccessible even to root users.
The SCONE platform is equipped with a built-in attestation and configuration service that validates applications against accepted security policies. It generates private keys and certificates that should only be available within the enclave. The confidentiality and integrity of data in the process of their transfer is ensured by the TLS cryptographic protocol.
The SGX driver reserves up to 64 GB of memory for each enclave in the virtual address space. The SCONE platform supports C / C ++ / C # / Rust / Go / Python / Java programming languages. Due to the special compiler, the source code is automatically (without the need for additional modifications) prepared for use in conjunction with Intel SGX.
Aggregion case
Having completed all the necessary work on the integration of Intel SGX, we connected the Aggregion distributed data management platform to our public cloud.
It is intended for the implementation of joint marketing projects by representatives of various industries - financial and insurance services, government, telecommunications, retail. Partners analyze consumer behavior, develop targeted promotion of goods and services, develop demanded loyalty programs, exchange and process anonymized data sets on the Aggregion platform. Since the leakage of confidential information is highly undesirable and threatens with serious reputational risks, the company pays special attention to security issues.
The Aggregion software is entirely integrated into the data provider, which means that it has an infrastructure with Intel SGX support at its disposal. Now the company's customers can consider connecting to our public cloud as an alternative to renting or buying physical servers.
Principles of secure work on the Aggregion platform. In the loop of each supplier, sensitive data is isolated into Intel SGX enclaves, which are actually black boxes: what happens inside is not available to anyone, including the cloud infrastructure provider. Verification of the initial state of the enclave and the possibility of its use for storing confidential information is carried out through remote attestation, when MrEnclavedefines a hash value.
Potential customer benefit. Combining databases from several vendors can improve the effectiveness of joint advertising campaigns. When the target audience is allocated according to the specified parameters, the matching (matching) of segments is performed directly inside containers with support for Intel SGX enclaves. Only the end result is displayed out of bounds: for example, the number of users matching the selected attributes. The effectiveness of campaigns is assessed in a similar way: data on ad impressions and completed sales are uploaded to enclaves to calculate the increase in purchases of the target group relative to the control group, which is then sent out for further use.
conclusions
We understand that Intel SGX is not a panacea for data protection and you can find a number of articles condemning this technology, including on Habré. Periodically, there are reports of attacks capable of extracting sensitive data from enclaves: for example, Meltdown and Specter in 2018 opened SGX holes, and SGAxe and CrossTalk in 2020. In turn, Intel is addressing the identified vulnerabilities through processor microcode updates.
Why did we decide to implement this technology? We see the use of Intel SGX as an opportunity to reduce the potential area of cyberattacks by creating an additional protection loop for the cloud infrastructure of G-Core Labs along with the already used information security technologies and thereby increasing the confidence of our users in the storage and processing of confidential data. We hope that in the future we have yet to share with you successful client cases, although we do not undertake to claim that our articles will not be based on stories of discovering and eliminating new vulnerabilities. In the meantime, we invite you to share your methods for protecting sensitive data in the comments.