This post chronicles our adventure of implementing (and migrating) our Kubernetes storage provisioning from FlexVolumes to Third Party Resources to Storage Classes.
Some quick notes before we begin:
- We are running on AWS in the Asia Pacific region.
- We have a high level API called “Skipper” which we use as a “man in the middle” for easing developer usage on the cluster.
- We are an Ops team of 3.
When we first started hosting our sites on Kubernetes we did not have access to ideal “managed storage” options. The only options we
had available to us were:
- EBS - Could be used for a single Pod deployment, locking us into a non high availability solution.
- S3 - This is ok if you have 1 or 2 applications per dev team, but the effort required to migrate a large amount of applications was very daunting.
- Roll our own - We are a small team and didn’t want to have to maintain our own solution.
Around this time FlexVolumes were added to Kubernetes.
FlexVolumes are a very low level entrypoint which allows for a large amount of control over the storage which is mounted on a node.
The options that FlexVolumes opened up for us were very exciting, so we decided to go with a sneaky fourth option, “fuse filesystems”.
We had a choice between 3 fuse filesystems:
- S3FS - Stable, but very slow for our needs.
- RioFS - Faster, less support.
- Goofys - Fastest, but at the time it was experimental.
We chose RioFS. At the time it seemed like a good compromise.
The FlexVolume we implemented was awesome! We wrote it to auto provision S3 buckets and then shell out to RioFS to mount the volume.
Given we run our high-level "Skipper" API over the top of Kubernetes, this made rolling out the storage very easy.
Here is an example of a Deployment definition using our FlexVolume.
apiVersion: apps/v1beta1 kind: Deployment metadata: namespace: test-project name: production spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: web image: nginx:latest volumeMounts: - mountPath: /var/www/files name: files volumes: - name: files flexVolume: driver: "pnx/volume" fsType: "fuse" options: name: "previousnext-test-project-files"
Over time we found that this solution was not viable, we ran into multiple random issues with replication across nodes and the dev team started to lose faith in the solution.
While I really liked the idea of a fuse mounted filesystem, we saw this as a means to an end, we were waiting for AWS EFS to launch in Australia.
If you are interested in the FlexVolume code, you can find it here:
Third Party Resources
I now reference my “Kubernetes life” as “before” and “after” the AWS EFS launch in Sydney.
The first implementation we wrote for provisioning EFS volumes was done via a ThirdPartyResource which was comprised of 3 components:
- Provisioner - Daemon to provision new AWS EFS resources.
- Status - Daemon to monitor the AWS EFS API for changes to the state of the volumes.
- CLI - Simple command line client for listing all the EFS Third Party Resources (cluster admin only)
With these components installed, admins were able to provision new AWS EFS resources with the following definition:
apiVersion: skpr.io/v1 kind: Efs metadata: name: public-files namespace: test-project spec: performance: "generalPurpose" region: ap-southeast-2 securityGroup: sg-xxxxxxxx subnets: - subnet-xxxxxxxx - subnet-xxxxxxxx
We then integrated this ThirdPartyResource into our Skipper API, allowing our developers to automatically have EFS backed deployments.
At the time of writing this post we are managing 100 AWS EFS volumes with this implementation.
While this has worked great for us, we acknowledged that this approach would not be ideal for the Kubernetes community as developers would be required to have knowledge of the infrastructure the cluster is running on, such as:
- Security Group
- VPC Subnets
We have now marked this implementation as v1.
After reading through the Kubernetes Blog post, Dynamic Provisioning and Storage Classes in Kubernetes, we knew that this architecture was for us.
This is how I think of Storage Classes:
- Developer submits a PersistentVolumeClaim which contains a StorageClass reference
- StorageClass references a Provisioner
- Provisioner creates our SourceVolumes and returns PersistentVolumeSource information (how to mount)
- Developer references PersistentVolumeClaim in Pod definition
Not only has this approach allowed us to decouple our applications from the storage layer, but it also allowed us to move away from our ThirdPartyResource definition and command-line client, meaning less code to maintain!
So, how do you use our AWS EFS Storage Class?
First, we declare our Storage Classes. I think of these as aliases for our storage, allowing us to decouple the hosting provider from the deployment.
For example, a Storage Class named “general” could be provisioned by “EFS: General Purpose” on AWS or “Azure File Storage” on Microsoft Azure.
kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: general provisioner: efs.aws.skpr.io/generalPurpose --- kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: fast provisioner: efs.aws.skpr.io/maxIO
Now that we have our Storage Classes declared, we need provisioners to do the work.
The provisioner was very easy to implement with the help of the Kubernetes incubator project External Storage.
To implement a provisioner with this library all you need to do is satisfy the interface functions
The examples I used to bootstrap our provisioner can be found here:
Our implementation can be found here:
To deploy the provisioners we used the following manifest file:
kind: Deployment apiVersion: extensions/v1beta1 metadata: name: aws-efs-gp namespace: kube-system spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: aws-efs-gp spec: containers: - name: provisioner image: previousnext/k8s-aws-efs:2.0.0 env: - name: EFS_PERFORMANCE value: "generalPurpose" - name: AWS_REGION value: "ap-southeast-2" - name: AWS_SECURITY_GROUP value: "sg-xxxxxxxxx" - name: AWS_SUBNETS value: "subnet-xxxxxx,subnet-xxxxxx" --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: aws-efs-max-io namespace: kube-system spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: aws-efs-max-io spec: containers: - name: provisioner image: previousnext/k8s-aws-efs:2.0.0 env: - name: EFS_PERFORMANCE value: "maxIO" - name: AWS_REGION value: "ap-southeast-2" - name: AWS_SECURITY_GROUP value: "sg-xxxxxxxxx" - name: AWS_SUBNETS value: "subnet-xxxxxx,subnet-xxxxxx"
Now we can provision some storage!
In the following definition, we are requesting one of each Storage Class type.
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: files namespace: test-project annotations: volume.beta.kubernetes.io/storage-class: "general" spec: accessModes: - ReadWriteMany resources: requests: storage: 1Mi --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: files-fast namespace: test-project annotations: volume.beta.kubernetes.io/storage-class: "fast" spec: accessModes: - ReadWriteMany resources: requests: storage: 1Mi
We can inspect the status of these PersistentVolumeClaim objects with the following command:
$ kubectl -n test-project get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE files Bound fs-xxxxxxxx 8E RWX general 5m files-fast Bound fs-xxxxxxxx 8E RWX fast 5m
Consuming a PersistentVolumeClaim is super easy, we now only need to reference what storage we want, not how we mount it (eg. nfs mount details).
apiVersion: apps/v1beta1 kind: Deployment metadata: namespace: test-project name: production spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: web image: nginx:latest volumeMounts: - mountPath: /var/www/files name: files volumes: - name: files persistentVolumeClaim: claimName: files
Each of these APIs are for different use cases:
- FlexVolumes are for how a volume is mounted, in this case, we already had access to the NFS volume mount in Kubernetes.
- Storage Classes are for the what do you want eg. give me some storage please with X speed and X size.
- Third Party Resources allowed us to prototype early and we still use this for other custom API definitions on our clusters.
I am very grateful for the contributors working on these APIs. What we have been able to achieve with them is a testament to their excellent design.
Any feedback or contributions on the AWS EFS project are most welcome.
Further discussion is also welcome on this site and on this Hacker News discussion.
Hi Nick, very nice and detailed post, thanks. Were you able to run attach/detach modes from kube-controller-manager? This is enabled by using "--enable-controller-attach-detach" flag
Thanks, I'm also looking for a Kubernetes storage solution to fit my needs and this post clarified things a good deal.