Securing Containers & Pods

Often people who are new to container and orchestration technologies are under the impression that these products are inherently secure and suitable for production use. While that might not be true, these solutions offer a lot of configuration options for strengthening the security of the containerized environment in order to get protections similar to that of a virtual machine. This way you can ensure that, even if an attacker manages to break into an application, they will have a hard time pivoting to other machines on the network or the host system.

In the below, you find some strategies to improve the security of your Kubernets and Docker environments.

Kubernetes Environments

RequirementDescriptionAdd'l info
Use central services instead of spinning up your own clusters or containers. Using central services allows you to delegate the responsibility for maintaining a secure configuration. CERN Openshift
CERN OpenStack
Enable CERN SSO for any websites that will be exposed to the Internet. CERN SSO allows users to authenticate to your application using their CERN credentials. Manage your SSO applications
Authentication documentation
Enable HTTPS on your web applications. Encryption makes sure somebody who is listening to your communications can't see sensitive data that is transmitted. In Openshift you can set the route that exposes your application to have Edge termination
In OpenStack you can define an ingress controller with SSL termination
Run trusted, official images Official images are maintained and are usually better secured than alternatives, even if you might find an unofficial image that fits your purposes better you should make the effort to use an official one. CERN runs its own container registry where you can find trusted images.
Also, we suggest that you base your images off of CERN's cc7-base image.
CERN Gitlab
cc7-base image
Run kube-bench for checking your Kubernetes deployment kube-bench checks your setup against the CIS Kubernetes Benchmark. This tool is already ran by the OpenStack team, so there is no need to run it if your cluster resides on OpenStack. kube-bench
Set up Pod Security Policies Pod Security Policies allows you to specify rules about sensitive aspects of your pods. Any pods that do not follow those rules are not allowed into the system. PodSecurityPolicy specification
Do not expose your pods to the network if not necessary Running a container in the host network or exposing ports without the need allows anyone in the network to connect to them. PodSecurityPolicy - Host namespaces options can help ensure that you don't accidentally expose something that you don't want to.
Do not run pods as the root user An attacker who is running as root inside a pod is in a much better position to attach the network. PodSecurityPolicy - Users and groups options can help ensure that all your pods run as allowed users.
Scan images before deploying them. We can provide a Clair instance, I've created a test instance but it doesn't seem to fetch CVEs properly. Alternatively OpenSCAP can be integrated into Openshift repository but it's only for RHEL based images. TODO
Do not hard-code secrets (e.g. login details) in your images. Sensitive data such as username-password combos should not be stored in an image, as then anybody with access to the image will be able to read it. Kubernetes secrets. If running on OpenStack Magnum, you can use Helm managed Secrets with Barbican
Delete secrets after reading them. You can delete the files containing your secrets to prevent a possible attacker from reading them and using them to access other CERN services that your application uses, if they manage to break into your application. However your secrets will still possibly be found in memory so a skilled attacker can circumvent this mitigation. In Kubernetes, you can have an Init Container which runs before your main container runs. You can also create an emptyDir volume which is initially empty when created a pod is created. You can combine the two by reading your secret files in the initContainer and copying them over to your volume. Then, in your main container, you can mount the same volume, read your secrets in your application and delete the files from the volume.
Restrict outgoing connections to only the necessary hosts. This way, if your application is broken into, an attacker cannot use it to pivot to the rest of the CERN network. Create an egress NetworkPolicy to only allow connections from your pods to the services that they need. You can specify either pods or CIDRs to allow for egress traffic, as well as ports.
For this to work in OpenStack pods, the cluster has to be using a network driver that allows this, such as calico. To use that, you first have to create a cluster template (e.g. with coe cluster template create --floating-ip-disabled --master-flavor m2.medium --server-type vm --external-network CERN_NETWORK --image 2886e9bf-7702-4c06-99c1-39a9e7bd5951 --docker-storage-driver overlay2 --network-driver calico --fixed-network CERN_NETWORK --coe kubernetes --flavor m2.medium --dns-nameserver 137.138.17.5 --labels="calico_backend=vxlan" calico-kubernetes) and then create your cluster using that template. When doing this procedure please use the parameters from an existing template to ensure there are no issues, changing only the labels and network-driver parameters.
It should also work in Openshift without any changes when Openshift 4 is deployed (est. end of 2019).
Use role-based access control (RBAC) RBAC allows you to manage which users or applications can access which resources in what ways. By creating roles and assigning them to users or applications, you can control which namespaces they can access. You can also specify the types of access in the form of verbs (e.g. list, create, update). Roles and Role Bindings reference

Docker Environments

RequirementDescriptionAdd'l info
Do not expose your containers (forward ports) to the network if not necessary Running a container in the host network or exposing ports without the need allows anyone in the network to connect to them.
Do not run containers as the root user An attacker who is running as root inside the container is in a much better position to escape the container and attack the host. Docker USER directive
Run trusted, official images Official images are maintained and are usually better secured than alternatives, even if you might find an unofficial image that fits your purposes better you should make the effort to use an official one. CERN runs its own container registry where you can find trusted images.
Also, we suggest that you base your images off of CERN's cc7-base image.
CERN Gitlab
cc7-base image
Run Docker Bench for Security for ensuring your Docker service runs with a good security setup Docker Bench for Security checks your Docker daemon configuration for common best practices for deploying containers. Docker Bench for Security
See here for solutions to possible warnings.
Isolate containers with User namespaces User namespaces limit privilege-escalation attacks to the host from within the container, if an attacker manages to get root access inside the container. User namespaces
Use the host firewall to restrict outgoing connections You can use iptables to restrict the hosts a container can connect to by creating a network interface for it and applying rules to it. This way, an attacker cannot easily use a vulnerability to pivot to the rest of the CERN network. For example:
dk network create -o com.docker.network.bridge.name=testbridge testbridge
sudo iptables -N BEFORE_DOCKER
sudo iptables -I BEFORE_DOCKER -d 172.217.168.4 -j RETURN
sudo iptables -I BEFORE_DOCKER -d 104.27.202.92 -j RETURN
sudo iptables -I FORWARD -i testbridge -m state --state NEW -j BEFORE_DOCKER
dk run --rm -it --net testbridge centos:7
Create a seccomp profile for your image A seccomp profile can restrict the system calls a container has access to to only the necessary, reducing the attack surface. This is not trivial to enable, as it sometimes leads to crashes. syscall2seccomp
Do not expose your Docker daemon socket to the network The Docker daemon can be configured to listen to the network for commands. By default, if enabled, Docker listens to port 2375 and allows users to run containers without any authentication. This is very dangerous and might be abused to gain root privileged on the host.