Azure

How I have moved my Ghost blog to Azure Kubernetes

Curia Damiano

• 7 min read
How I have moved my Ghost blog to Azure Kubernetes

I have written already in this blog How I have moved my blog to Ghost.
That solution worked well until the beginning of this year, when there has been a change in Azure that prevented to write the SQLite database hosted in an Azure Storage Blob because it was made now readonly. This is documented in Serve content from Azure Storage in App Service on Linux - Limitations of Azure Storage with App Service. This was already a workaround, because we have already found issues hosting the database in Azure Files, like all the rest of the content.

Currently there is no way to mount a file in Azure File specifying the Unix lock mask and the only available solution is to serve the Ghost docker image from a Kubernetes cluster.
As I am not skilled on this argument, my three wonderful colleagues Gareth Emslie, Andre Bossard and Raghuram Tandra have helped me in this migration.

I have used as much as possible the Azure Cloud Shell, that you can access from the top right corner via the icon representing a prompt.

Creation of the Kubernetes cluster

As I have recreated all my resources from scratch, I have started with a new Resource Group called RG_CuriaKube hosted in West Europe. Feel free to adapt the scripts below to adjust to your needs.

az group create --name RG_CuriaKube --location westeurope

Now let's create the Kubernetes cluster called CuriaKube:

az aks create -g RG_CuriaKube -n CuriaKube --location westeurope --node-vm-size Standard_DS2_v2 --node-count 1 --enable-addons monitoring --generate-ssh-keys

Creation of the Azure Container Registry

I already had an Azure Container Registry from my previous installation, but I have created a new one to have everything in this new Resource Group. I have named it CuriaKubeACR.

az acr create --resource-group RG_CuriaKube --name CuriaKubeACR --sku Basic --admin-enabled true

From the previous episode: I have created a custom Docker file, fork of Gareth's, to add Application Insights logging and monitoring.
Now I create the task to pull images from my custom docker file (remember: you get the Git access token from github.com, your profile, Settings, Developer Settings, Personal Access Tokens, create a token with repo rights):

  az acr task create \
   --registry CuriaKubeACR \
   --name docker-ghost-ai \
   --image ghost:3-alpine-ai \
   --context https://github.com/curia-damiano/docker-ghost-ai.git \
   --file 3/alpine/ai/DOCKERFILE \
   --git-access-token 93************************************f3

Run the task:

az acr task run --registry CuriaKubeACR --name docker-ghost-ai

Verify that the task has completed:

az acr task list-runs --registry CuriaKubeACR --output table

Now we connect the Azure Container Registry to the Kubernetes cluster:

AKS_SP_ID=$(az aks show --resource-group RG_CuriaKube --name CuriaKube --query "servicePrincipalProfile.clientId" -o tsv)
ACR_RESOURCE_ID=$(az acr show --resource-group RG_CuriaKube --name CuriaKubeACR --query "id" -o tsv)
az role assignment create --assignee $AKS_SP_ID --scope $ACR_RESOURCE_ID --role contributor

And now we connect to the Kubernetes cluster:

az aks get-credentials --resource-group RG_CuriaKube --name CuriaKube

Creation of the Storage account

We create the storage account with name curiakube to save our files and database:

az storage account create -n curiakube -g RG_CuriaKube -l westeurope --sku Standard_RAGRS

From the portal, under Azure File Shares, we create a new share called contentfiles.
Now, if we have existing files, we copy them in this share, for example using Azure Storage Explorer.

Now we need to connect the Storage account to the Kubernetes cluster.
From the portal, get the access key of the Storage account, then execute:

kubectl create secret generic secret-pv-curiakube --from-literal=azurestorageaccountname=curiakube --from-literal=azurestorageaccountkey=+ir*********************************************************************************7g==

Creation of the ingress controller

Following the NGINX Installation Guide, we need to run the following scripts to install the ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/cloud-generic.yaml

Verify it with:

kubectl describe ingress

Installing cert-manager

To install the Cert manager, we need to run the following scripts:

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.1/cert-manager.yaml

Label the ingress-basic namespace to disable resource validation:

kubectl label namespace default certmanager.k8s.io/disable-validation=true

Creation of the Application Insights resource

Create the Application Insights resource named CuriaKube:

az extension add -n application-insights
az monitor app-insights component create --app CuriaKube --location westeurope --kind web -g RG_CuriaKube --application-type web

The last command will give in its output the instrumentation key. Take note of it, because we will use it later when connecting it to our application.

Creation of the CA cluster issuer

Save locally the file named cluster-issuer.yaml, replacing the email with yours:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: damiano.curia@gmail.com
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx

You can create it running from the Azure Console:

code cluster-issuer.yaml

pasting the file content above, save and close the code editor.

Then apply it with:

kubectl apply -f cluster-issuer.yaml --namespace default

You should see it here:

kubectl get clusterissuers

Creation of the pod for your application

You can copy and save the file curiakube.yaml copying the following content and:

  • replace the domain name (2 times);
  • add the Application Insights instrumentation key you have got before.

Note also from the file below that:

  • the persistent volume pv-curiakube, that is used to map the contentfiles folder of the storage account, is not marked in read-only;
  • the deployment curiakube has the readOnlyRootFilesystem property set to true;
  • sec-ctx-vol, on which the folder /tmp is mapped, is a non persistent volume, so that it is not in read-only (this is required because when uploading a picture, Ghost uses this folder to create temporary files before writing to the content storage).
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-curiakube
spec:
  capacity:
    storage: 5Ti
  accessModes:
    - ReadWriteOnce
  storageClassName: azurefile
  azureFile:
    secretName: secret-pv-curiakube
    shareName: contentfiles
    readOnly: false
  mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=1000
  - gid=1000
  - mfsymlinks
  - nobrl
  - cache=none
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-curiakube
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: azurefile
  resources:
    requests:
      storage: 5Ti
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: curiakube
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: curiakube
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 2000
      containers:
      - name: curiame
        image: curiakubeacr.azurecr.io/ghost:3-alpine-ai
        imagePullPolicy: Always
        ports:
        - containerPort: 2368
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        volumeMounts:
          - name: content
            mountPath: /var/lib/ghost/content/
          - name: sec-ctx-vol
            mountPath: /tmp
        env:
        - name: privacy__useUpdateCheck
          value: "false"
        - name: APPINSIGHTS_INSTRUMENTATIONKEY
          value: ""
        - name: NODE_ENV
          value: "production"
        - name: url
          value: "https://curia.me"
        securityContext:
          readOnlyRootFilesystem: true
          runAsUser: 1000
          runAsGroup: 1000
      volumes:
      - name: content
        persistentVolumeClaim:
          claimName: pvc-curiakube
      - name: sec-ctx-vol
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: curiakube
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 2368
  selector:
    app: curiakube
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: curiakube-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: curia.me
      http:
        paths:
          - backend:
              serviceName: curiakube
              servicePort: 80
            path: /

Apply it with:

kubectl apply -f curiakube.yaml
kubectl get service
kubectl get all

Creation of the ingress route

Edit the following file curiakube_ingress.yaml, where again you replace the domain name (2 times):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: curiakube-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - curia.me
    secretName: tls-secret
  rules:
  - host: curia.me
    http:
      paths:
      - backend:
          serviceName: curiakube
          servicePort: 80
        path: /

Apply it with:

kubectl apply -f curiakube_ingress.yaml --namespace default

Determine your public IP, and update your DNS accordingly:

kubectl get ingress

Verify that a secret has been produced:

kubectl describe certificates

Wait for the certificate to be produced, when the output of the next command will give Ready=True (it took me 6 minutes to see that it has been done).

kubectl get certificate --namespace default

Now, if you flush your local DNS cache and open the browser pointing to your domain, you should see everything running, in https.

Shut down the site

Kubernetes is not like an App Service, that you can shut it down when you don't need it.
Note that if you need to take a backup of your site, you don't need to do it!
But, in case you want for other reasons, you need to undeploy the pod of your application:

kubectl delete deployment curiakube

When you want to put it back, I have verified that you have to reapply both the application and the ingress controller (otherwise you don't get the SSL certificate):

kubectl apply -f curiakube.yaml --namespace default
kubectl apply -f curiakube_ingress.yaml --namespace default

Other troubleshooting

During my experiments, I have done multiple troubleshooting. I have taken many commands from the Kubernetes cheat-sheet.

Here I list the main one:

kybectl get all
kubectl get certificate --namespace default
kubectl describe certificates
kubectl get ingress
kubectl describe ingress
kubectl get service
kubectl get pods
kubectl get clusterissuers

In particular to access logs and log in into the pod:

kubectl get all
kubectl logs pod/curiakube-**************vh
kubectl exec -it pod/curiakube-**************vh -- /bin/bash

Conclusions

First of all, I need again to thank my colleagues for their invaluable support. Without the help of Gareth in doing the initial setup in http, and the help of Andre in configuring Let's Encrypt and removing helm, I would have not been absolutely able to do it. Finally we needed a support engineer, Raghuram, to fix the permissions on the tmp folder.

The site works very well, I am already editing it now, draft are saved after every few seconds, and I verify that the SQLite database is correctly updated in real time! Also uploading of pictures works well.

But that is the point: I have to do this very complex migration just because in Azure I can't use a storage blob read-write, and with Azure Files I can't specify an access mask.

So all of this is a huge workaround because of technical limitations in the access to storage. We have built a incredible complex system, full of scripts and yaml files, that I don't know for how long they will continue to work or be reproducible in the (near ?) future.

This is my last worry: all the Kubernetes world is complex and moving so fast, week after week, that I don't see it sustainable even in the near future. I hope that it will be replaced by something simpler, easier and more stable. Or better, that Azure App Services could mount blobs and/or files specifying the access mask.

Sign up for more like this.

Enter your email
Subscribe