Last week we presented a webinar with our partner Zettaset about containerized data encryption and why it’s important. Here are the answers to your questions, provided by Bryan Langston and Uday Shetty of Mirantis and Tim Reilly and Maksim Yankovskiy of Zettaset.
Why is encryption so important if we wrap our existing security around all workloads?
It goes back to why encrypting a containerized environment is different than with a legacy environment. The single word answer would be multi-tenancy. Multi-tenant datacenters usually depend on the segmentation of the hardware. Multi-tenant containerized environments are entirely in the software stack. So we cannot just take an existing, legacy encryption technology or security technology and apply it to the entire software stack of containers and just call it a day. We have to do something that’s specific to containers. We have to do something that integrates directly with containers and works seamlessly in the containerized environment.
What is the performance impact of encrypting data in containers? How do you ensure that encryption does not introduce latency when dealing with very large buckets? How can we reduce processing time?
With the Zettaset solution, we’re running in containers, but we’re running within the kernel and working at the block level using AES_NI instructions for fast cryptographic performance. We measure about a 3% performance hit for read and write using file system benchmark tests on the underlying encryption scheme. It performs just as well on large file systems as it does on small ones. You can reduce that performance hit by splitting your application across more containers on more CPUs.
Performance is critical because you don’t want your encryption solution to slow down your analytics system, bringing it down to its knees. It happened before, that’s why people are rightfully concerned. Minimizing performance overhead is one of the fundamentals of our solution.
What is the most common security incident reported for containers?
Bryan: Like Tim and Maksim have been talking about, the data breaches, storming the castle. Storming the castle is made possible by not locking down your environment in terms of network access policies, the RBAC, lack of implementation of least privilege. Think of a top secret security clearance in the government: information is granted on a need-to-know basis. Explicitly defining that will avoid a lot of problems. The most common security incident is access to data by someone who shouldn’t have it.
Maksim: The most common security incidents stem from improperly configured containerized environments that allow attackers to install malicious software on a single container and then from that single container distribute that malicious software to all other containers within the infrastructure. Malicious software then takes over the entire container infrastructure and has unrestricted access to containers and data. While tools such as intrusion detection and container image integrity scanners would help alert the admins of the breach, these tools would not protect the data from compromise. This emphasizes the need for data at rest encryption.
How big of a component of DevSecOps is encryption? If you take advantage of a solution like Zettaset, what else remains to say we have a “DevSecOps Practice?”
Bryan: Good question, it’s kind of like, “How do I know if I’m doing DevSecOps right?” I would phrase that as: Encryption needs to be as big as needed to satisfy your company’s risk management policy. Every company has a different level of risk. Every company has different subsets of controls within a security framework, like PCI for example, to which they have to comply. Just because you sell stuff over the Internet doesn’t mean you’re handling payment, for example. Your subset of controls for PCI might look different from other players in the same online reselling space. So it’s as big as needed to satisfy your company’s risk management policy. What your DevSecOps team has to enforce will vary.
One other thing to keep in mind is that the implementation of DevSecOps is a combination of both industry best practices and the layer of security that pertains to your company’s business model, like how I was just talking about risk management. There are very well defined industry best practices for many components — I’m talking about CIS benchmarks, stuff you can easily download and run and give a quick assessment to how you are doing compared to those benchmarks — but then you also have to define the layer that pertains to you.
In order to say “I have a DevSecOps practice,” we’re talking about having a team that focuses on understanding the attack vectors, and then identifying the controls that are relevant to your workload and your business, and then having the means to implement those controls from a technology perspective, whether that’s encryption, like we’re talking about here, or RBAC, or network policies or some of the other things that we discussed earlier, or all of them.
Nick: Is it fair to say that you’re better off with too much security than not enough security?
Bryan: Yeah, you want to stay off the front page of the headlines. I’m sure Marriott, in light of the current environment we’re in with coronavirus, did not need to exacerbate their problem with a security breach. So definitely, err on the side of caution.
Maksim: By the way, I don’t think there’s such a thing as too much security. There are things that you may choose not to implement, but there’s no such thing as too much security.
My containers are running on SEDs (Self Encrypting Drives). Do I still need Zettaset Container Encryption?
SEDs offer a simple approach to encrypting data. SEDs, however, are not suitable for properly securing container environments.
SEDs do not offer key granularity to address the fluid topology of container environments – containers will share SED partitions and data from different containers will be encrypted with the same key. In the event of a compromise, one bad actor container compromises all other containers that share the same SED.
Also, SEDs do not offer enterprise grade key management; some store keys on the drive, some rely on the OS to store the master key. Both of these approaches are not scalable and not secure.
Zettaset Container Encryption ensures that:
Each container is allocated its own storage volume mapped to a unique storage volume group that is encrypted with unique encryption key.
The container volume is only available when a container mounts it; the volume is automatically unmounted when the container exits.
The KMIP-compatible enterprise key manager running natively in a container provides secure key management infrastructure.
I am curious, I should be able to encrypt the disks before I make it an image, so before I run “docker build”, shouldn’t I be able to run the application when I run “docker run”?
The Docker build command builds the container image, which includes unmodifiable layers of OS, software, and data. This is not the data that Zettaset encrypts. Zettaset encrypts data that containers use and generate at runtime, the data that persists, essentially – production data.
How do you handle encryption when containers move from one host to another host?
When containers move between hosts, the data doesn’t necessarily move between the hosts, because the storage should not be associated with the hosts. In a typical distributed environment, when you have processes running on different hosts, this is addressed by having shared storage, but shared storage is still managed by Docker daemons, which are on hosts. So we are able to handle the storage allocation to the container on the shared volumes just like we are able to handle storage allocation to the containers on the volumes that are hardwired to the hosts.
So, we can use storage that is tied to the host, or we can use the storage that is shared between the Docker hosts. in addition, with centralized virtual key manager, we are able to provide access to container data regardless of which host container runs on. This goes hand-in-hand with the shared storage approach.
Does Docker Enterprise integrate with any Key Manager/ HSM?
Docker Enterprise doesn’t have any specific integrations with any HSM currently. UCP provides a certificate authority (for TLS, client bundles, node joining, API certs, etc) and DTR provides the notary (for image content trust). The cert authority can be a 3rd party.
Can you clarify the ephemeral nature of the key manager container, and how it securely accesses the keys (and how those are stored securely?) Can the system leverage key storage like KMS in AWS?
That question speaks to several important things. Yes, a key manager in a containerized environment should run in a container. There has been a lot of work on key managers over the last 20 years. We started with proprietary key managers, then we started moving to key managers supporting what is known as KMIP (Key Manager Interoperability Protocol), which is essentially a common language that every key manager out there speaks. The key manager running in a container is part of our solution. It is fully KMIP compatible, and integration has been tested with other encryption solutions, and of course ours.
Understanding that enterprise security requirements may be different, we provide you with a software-based key manager, but we also understand that you may already have investments in existing Gemalto/Safenet or Thales/Vormetric security key management infrastructure. We allow you to connect very simply and very easily to any existing key management infrastructure that you may have. We have a KMIP compatible key manager that stores the keys in a key database. The key database is secured with a hierarchy of master keys and hashed with appropriate hash keys. Those keys are stored in the software security module, which is essentially a software implementation of your typical Hardware Security Module (HSM). We’re also (as any solution should be) PKCS11 compliant, which means it can talk to HSM’s if required, which are essentially hardware devices that store master keys securely.
AWS KMS does not support the industry-standard Key Management Interoperability Protocol (KMIP) and because KMS is owned by the cloud provider, Zettaset recommends using a third-party key manager that allows the customer to own their keys. AWS’s policy is that “security is a shared responsibility” and this is the optimal way to implement it.
The reason why I’m saying all this and giving such a detailed and somewhat technical answer is that when you look at an encryption solution, you should make sure that it does a good job with housekeeping. If encryption is an afterthought, then believe me when I tell you, key management is not even on the map, but it should be.
How does your “driver” know what size storage to create if the developer isn’t doing anything different in docker. It seems that the developer would have to specify that size.
We talk about transparency, and now we’re saying, when you specify a volume, we need to know the size of the volume you allocate. So it’s an additional burden of the development to ask for a certain amount of storage.
There are three options that our solution is providing. One option is: Storage volumes can be preallocated by administrators, and that’s very well integrated with the Docker Create Volume command. So an administrator can specify a volume for, let’s say, a MySQL database and specify size there.
The second option is that an administrator, at the time of installation of our software, can specify the default size of the volume that the container will get, and then a developer simply runs the Docker Run command and doesn’t have to specify the size of the volume and the volume gets allocated.
The third options is that nothing gets specified, and we just specify a default value of the volume size, which is specified within the installation of the software.
So between the three, transparency should be addressed.
In a cloud-based Kubernetes offering such as AKS/EKS, most operational deployments provision separate block storage disks (EBS/Azure Disk) per container. Since there is a logical separation here, and containers aren’t sharing a disk, what does a product like Zettaset bring to the table?
While storage provisioning can provision separate AWS EBS devices per container, Zettaset allows for partitioning one (or small number) of storage devices for use by multiple containers, therefore dramatically reducing the number of EBS volumes required. In addition, Zettaset provides automated encryption and key management of those storage devices without the need to deploy encrypted EBS volumes.
How is RBAC here different than in K8s?
Zettaset XCrypt relies on K8s RBAC for access and permission management. In addition, XCrypt provides ability to securely decommission a K8s worker node completely, if a node is compromised. This is done with a single admin command without the need to have access to the node.
Can we use Azure key vault with this to store encryption keys for containers, and if yes, then how will it be communicating ?
We don’t support Azure key vaults at this time. Only KMIP compatible key managers.
As a Docker Enterprise Partner, can we offer Zettaset as an OEM product?
Mirantis values our partners and will consider what’s best for our customers and mutual business benefits. Please contact us with specific questions.
How is Zettaset integrated with Docker containers? Is it a layer above Docker Enterprise or something which is integrated within every Docker container? Can it be used with Docker Enterprise orchestration tools ?
Zettaset XCrypt integrates with Docker Enterprise by providing “XCrypt Volume Driver”, a Docker Volume Driver API-compatible driver that transparently integrates into the Docker storage management stack. It is fully usable with Docker Enterprise orchestration tools.