In a previous post , I set up a Ceph cluster as a storage backend and default storage class for my Kubernetes cluster. It works great for workloads such as databases and object storage, but not ideal for:
shared storage for multiple applications
accessing files from outside the cluster
Using CephFS as a shared filesystem solves both of these problems. The trade-off is that it is not as performant as using Ceph RBD, so it will be only for certain situations where I absolutely need shared storage. The default storage class is still RBD.
Overview
With a Ceph cluster, CephFS, and a Kubernetes cluster running, in this post, I will set up a nginx pod that serves files from a CephFS volume.
As an example, I am serving my resume here: https://share.junyi.me/resume_en.pdf for public access, which is also linked in my portfolio website . The nice thing about this setup is that each time I update my resume, I simply need to put the new one in the CephFS volume, without needing to rebuilt/restart any application.
I also set up a NFS server pod to expose that CephFS volume to my home network so that it’s more convenient to access files from my devices.
Install CephFS CSI
CephFS CSI can be installed with a helm chart. Following is the configuration I used:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# values.yaml
csiConfig :
- clusterID : "<cluster_id>"
monitors :
- <node_1_ip>:6789
- <node_2_ip>:6789
- <node_3_ip>:6789
# ...
storageClass :
create : true
name : csi-cephfs-sc
clusterID : "<cluster_id>"
fsName : "<cephfs_name>"
provisionerSecret : csi-cephfs-secret
controllerExpandSecret : csi-cephfs-secret
nodeStageSecret : csi-cephfs-secret
reclaimPolicy : Retain
allowVolumeExpansion : true
secret :
create : true
name : csi-cephfs-secret
annotations : {}
adminID : <admin_id>
adminKey : <admin_key>
userID : <user_id>
userKey : <user_key>
Cluster ID of ceph cluster can be retrieved by:
Admin key can be retrieved by:
1
ceph auth get-key client.admin
For the user id and key, you can create a new user with the following command:
1
ceph auth get-or-create-key client.<user_id> mon 'allow r' osd 'allow rwx pool=<pool_name>'
Install the helm chart by:
1
2
3
helm repo add ceph-csi https://ceph.github.io/csi-charts
helm repo update
helm install csi-cephfs ceph-csi/csi-cephfs -f values.yaml
For more information about ceph-csi-cephfs and helm values, see the official GitHub repo .
After installing, confirm that the storage class is created:
nginx pod
This is the nginx pod that I’m using to serve files from the CephFS volume to the internet:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
apiVersion : v1
kind : PersistentVolume
metadata :
name : public
namespace : network
spec :
storageClassName : csi-cephfs-sc
accessModes :
- ReadWriteMany
capacity :
storage : 1Gi
csi :
driver : cephfs.csi.ceph.com
nodeStageSecretRef :
name : csi-cephfs-secret
namespace : ceph-csi-cephfs
volumeAttributes :
"fsName": "<cephfs_name>"
"clusterID": "<cluster_id>"
"staticVolume": "true"
"rootPath": /public # only this subdirectory will be mounted, and thus served
volumeHandle : public-handle
persistentVolumeReclaimPolicy : Retain
volumeMode : Filesystem
---
apiVersion : v1
kind : PersistentVolumeClaim
metadata :
name : public
namespace : network
spec :
accessModes :
- ReadWriteMany
resources :
requests :
storage : 1Gi
storageClassName : "csi-cephfs-sc"
volumeMode : Filesystem
volumeName : public
---
apiVersion : v1
kind : ConfigMap
metadata :
name : nginx-public-cm
namespace : network
data :
nginx.conf : |
server {
listen 80;
listen [::]:80;
server_name _;
root /mnt/data;
add_header 'Access-Control-Allow-Origin' '*.junyi.me' always;
add_header 'Access-Control-Allow-Methods' 'GET' always;
limit_except GET { deny all; }
location = /healthz {
access_log off;
return 200 'OK';
add_header Content-Type text/plain;
}
}
---
apiVersion : apps/v1
kind : Deployment
metadata :
name : nginx-public
namespace : network
labels :
app : nginx-public
spec :
replicas : 1
selector :
matchLabels :
app : nginx-public
template :
metadata :
labels :
app : nginx-public
spec :
containers :
- name : nginx-public
image : nginx:bookworm
ports :
- containerPort : 80
volumeMounts :
- mountPath : /mnt/data
name : data
- mountPath : /etc/nginx/conf.d/default.conf
name : nginx-config
subPath : nginx.conf
livenessProbe :
httpGet :
path : /healthz
port : 80
httpHeaders :
initialDelaySeconds : 10
periodSeconds : 30
volumes :
- name : data
persistentVolumeClaim :
claimName : public
- name : nginx-config
configMap :
name : nginx-public-cm
---
apiVersion : v1
kind : Service
metadata :
name : nginx-public
namespace : network
spec :
selector :
app : nginx-public
ports :
- protocol : TCP
port : 80
targetPort : 80
Apply the above along with an ingress pointing to the service to expose the nginx pod to the internet.
NFS pod
This is the NFS server pod that I’m using to provide access to my home network:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
apiVersion : v1
kind : PersistentVolume
metadata :
name : nfs-pv
spec :
storageClassName : csi-cephfs-sc
accessModes :
- ReadWriteMany
capacity :
storage : 1Gi # doesn't matter
csi :
driver : cephfs.csi.ceph.com
nodeStageSecretRef :
name : csi-cephfs-secret
namespace : ceph-csi-cephfs
volumeAttributes :
"fsName": "<cephfs_name>"
"clusterID": "<cluster_id>"
"staticVolume": "true"
"rootPath": / # entire cephfs volume will be mounted
volumeHandle : nfs-pvc
persistentVolumeReclaimPolicy : Retain
volumeMode : Filesystem
---
apiVersion : v1
kind : PersistentVolumeClaim
metadata :
name : nfs-pvc
namespace : admin
spec :
accessModes :
- ReadWriteMany
resources :
requests :
storage : 1Gi
storageClassName : "csi-cephfs-sc"
volumeMode : Filesystem
volumeName : nfs-pv
---
apiVersion : apps/v1
kind : Deployment
metadata :
name : nfs-server
namespace : admin
spec :
replicas : 1
selector :
matchLabels :
app : nfs-server
template :
metadata :
labels :
app : nfs-server
spec :
containers :
- name : nfs-server
image : itsthenetwork/nfs-server-alpine
ports :
- name : nfs
containerPort : 2049
- name : rpcbind
containerPort : 111
securityContext :
privileged : true
env :
- name : SHARED_DIRECTORY
value : "/cephfs"
volumeMounts :
- name : cephfs-storage
mountPath : "/cephfs"
volumes :
- name : cephfs-storage
persistentVolumeClaim :
claimName : nfs-pvc
---
apiVersion : v1
kind : Service
metadata :
name : nfs-service
namespace : admin
spec :
selector :
app : nfs-server
ports :
- name : nfs
protocol : TCP
port : 2049
targetPort : 2049
- name : rpcbind
protocol : TCP
port : 111
targetPort : 111
type : LoadBalancer
loadBalancerIP : <load_balancer_ip>
The volumeHandle in the PV spec can be anything, but must be unique across all PVs in the cluster.
Now the NFS share can be mounted like any other NFS share with the load balancer IP.
Conclusion
I’m happy that now I have a convenient way to serve static files from my CephFS volume, and also share some files across myself and my pods.
Currently I only have one CephFS configured, and everything inside is managed by subdirectories. So far I hadn’t had any problem with this setup, but I assume if it’s for a corporate environment, it would be better to have some kind of more fine-grained access control.
