Use Horizontal Node Autoscaling

MetaKube Kubernetes clusters support horizontal node autoscaling out of the box. This tutorial shows how
you can configure and activate it and how it plays together nicely with horizontal pod autoscaling.

Currently the node autoscaler only automatically creates nodes, removing nodes automatically to scale a cluster down is on our roadmap and will be released in the near future.

Scaling up a MachineDeployment from 0 replicas currently only works with clusters running on OpenStack.


Deploy an application

For easy cleanups we create a new namespace for our tutorial:

$ kubectl create namespace hna-tutorial
namespace/hna-tutorial created

For our tutorial we will just deploy an NGINX Hello World app, run:

$ kubectl run hello-app --image=nginxdemos/hello --port=80 --namespace hna-tutorial --requests="cpu=500m,memory=700Mi"
deployment.apps/hello-app created

Check that the pod of the new application was created successfully and is running:

$ kubectl get pods --namespace hna-tutorial
NAME                           READY     STATUS    RESTARTS   AGE
hello-app-5c7477d7b7-n44wq     1/1       Running   0          9s

Not that we defined fairly high CPU and memory requests for our pod. This way we will quickly get to the point where the
scheduler can not schedule new pods due to insufficient resources, when scaling the deployment up.

Now let's manually scale the deployment, in real life this would likely be done by the horizontal pod autoscaler:

$ kubectl scale deployment/hello-app --replicas 15 --namespace hna-tutorial
deployment.extensions/hello-app scaled

If you list all pods, there should be several pods stuck in "pending" state:

$ kubectl get pods --namespace hna-tutorial
NAME                           READY     STATUS    RESTARTS   AGE
hello-app-6f488fcdfc-6pj74   1/1     Running   0          13s
hello-app-6f488fcdfc-9bb8h   1/1     Running   0          13s
hello-app-6f488fcdfc-9bn8l   1/1     Running   0          13s
hello-app-6f488fcdfc-n44wq   1/1     Running   0          3m47s
hello-app-6f488fcdfc-dbhq9   0/1     Pending   0          13s
hello-app-6f488fcdfc-dvnsd   0/1     Pending   0          13s
hello-app-6f488fcdfc-g2m8n   0/1     Pending   0          13s
hello-app-6f488fcdfc-hm8kp   0/1     Pending   0          13s
hello-app-6f488fcdfc-kfx95   0/1     Pending   0          13s
hello-app-6f488fcdfc-m2n8x   0/1     Pending   0          13s
hello-app-6f488fcdfc-mcnbc   0/1     Pending   0          13s
hello-app-6f488fcdfc-q5fmn   0/1     Pending   0          13s
hello-app-6f488fcdfc-spk6p   0/1     Pending   0          13s
hello-app-6f488fcdfc-xr6r8   0/1     Pending   0          13s
hello-app-6f488fcdfc-zx8st   0/1     Pending   0          13s

If you describe one of these pending pods, you can see that the scheduling failed because of insufficient memory and CPU

$ kubectl describe pod hello-app-6f488fcdfc-m2n8x --namespace hna-tutorial
Name:               hello-app-6f488fcdfc-m2n8x
Namespace:          hna-tutorial
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             pod-template-hash=6f488fcdfc
Annotations:        <none>
Status:             Pending
Controlled By:      ReplicaSet/hello-app-6f488fcdfc
    Image:      nginxdemos/hello
    Port:       80/TCP
    Host Port:  0/TCP
      cpu:        500m
      memory:     700Mi
    Environment:  <none>
      /var/run/secrets/ from default-token-6d8qc (ro)
  Type           Status
  PodScheduled   False
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6d8qc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations: for 300s
        for 300s
  Type     Reason            Age                 From                Message
  ----     ------            ----                ----                -------
  Warning  FailedScheduling  36s (x25 over 81s)  default-scheduler   0/5 nodes are available: 1 Insufficient memory, 5 Insufficient cpu.

This means we need to increase the capacity of our cluster by adding more nodes. Of course this could be done manually through
the MetaKube dashboard (see Manage number of worker nodes) or over CLI (see Managing worker nodes over CLI),
but for this tutorial, let's use the cluster autoscaler.

For this we have to create a scalable MachineDeployment in our cluster. You can either do it via the MetaKube dashboard by creating a new NodeDeployment and activating autoscaling:

Configure Horizontal Node Autoscaler

or with the CLI and kubectl:

# Choose the public key you want to deploy on the node
SSH_PUBLIC_KEY=$(cat ~/.ssh/

# Getting the cluster name like this only works if you did not rename the
# context in the downloaded kubeconfig. If you did, choose the original name of
# the cluster (10 character alphanumeric string).
CLUSTER_NAME=$(kubectl config current-context)

# Set up the flavor for the node

# Set up the correct region and availability zone

IMAGE_NAME="Ubuntu Bionic 18.04 (2020-01-25)"

cat <<EOF | kubectl apply -f -
kind: MachineDeployment
    cluster-autoscaler/min-size: "0"
    cluster-autoscaler/max-size: "15"
  name: scalable-machine-deployment
  namespace: kube-system
  replicas: 0
  minReadySeconds: 0
      deployment: scalable-machine-deployment
    type: RollingUpdate
      maxUnavailable: 0
      maxSurge: 1
  paused: false
        deployment: scalable-machine-deployment
          cloudProvider: openstack
            availabilityZone: ${AVAILABILITY_ZONE}
            domainName: ""
            flavor: ${FLAVOR}
            floatingIpPool: ${FLOATING_IP_POOL}
            identityEndpoint: "https://api.${REGION}"
            image: "${IMAGE_NAME}"
            network: metakube-${CLUSTER_NAME}
            password: ""
            region: ${REGION}
            - metakube-${CLUSTER_NAME}
            tenantName: ""
            tokenId: ""
            username: ""
          operatingSystem: ${OPERATING_SYSTEM}
            distUpgradeOnBoot: false
          - "${SSH_PUBLIC_KEY}"
        kubelet: "${K8S_VERSION}"

This will create a MachineDeployment that initially contains 0 machines, but can be scaled up by the cluster autoscaler to a maximum
of 15 machines, if you have enough quota of course. For every machine a VM will be automatically created in OpenStack, provisioned and joined as a node to the
Kubernetes cluster.

For more details on MachineDeployments see Cluster Management API.

If you list all available machines, you can see that a few new machines have been automatically created by the autoscaler:

$ kubectl get machines --namespace kube-system
NAME                                           AGE
machine-metakube-fhgbvx65xg-7flj7              8d
machine-metakube-fhgbvx65xg-hmgd4              8d
machine-metakube-fhgbvx65xg-q287t              8d
scalable-machine-deployment-5c4cbbc47b-62wsd   4m
scalable-machine-deployment-5c4cbbc47b-zwrts   4m

And the replica count of the MachineDeployment has been updated:

$ kubectl get machinedeployment --namespace kube-system scalable-machine-deployment --output jsonpath="{.spec.replicas}"

After a few minutes, once the VMs are started and provisioned, new nodes will appear in the cluster as well:

$ kubectl get nodes
NAME                                           STATUS     ROLES    AGE     VERSION
metakube-fhgbvx65xg-7flj7                      Ready      <none>   8d      v1.12.2
metakube-fhgbvx65xg-hmgd4                      Ready      <none>   8d      v1.12.2
metakube-fhgbvx65xg-q287t                      Ready      <none>   8d      v1.12.2
scalable-machine-deployment-5c4cbbc47b-62wsd   Ready      <none>   2m58s   v1.12.2
scalable-machine-deployment-5c4cbbc47b-zwrts   Ready      <none>   2m30s   v1.12.2

And the previously pending pods are now successfully scheduled and running:

$ kubectl get pods --namespace hna-tutorial
NAME                           READY     STATUS    RESTARTS   AGE
hello-app-6f488fcdfc-6pj74   1/1     Running   0          13s
hello-app-6f488fcdfc-9bb8h   1/1     Running   0          13s
hello-app-6f488fcdfc-9bn8l   1/1     Running   0          13s
hello-app-6f488fcdfc-n44wq   1/1     Running   0          3m47s
hello-app-6f488fcdfc-dbhq9   1/1     Running   0          13s
hello-app-6f488fcdfc-dvnsd   1/1     Running   0          13s
hello-app-6f488fcdfc-g2m8n   1/1     Running   0          13s
hello-app-6f488fcdfc-hm8kp   1/1     Running   0          13s
hello-app-6f488fcdfc-kfx95   1/1     Running   0          13s
hello-app-6f488fcdfc-m2n8x   1/1     Running   0          13s
hello-app-6f488fcdfc-mcnbc   1/1     Running   0          13s
hello-app-6f488fcdfc-q5fmn   1/1     Running   0          13s
hello-app-6f488fcdfc-spk6p   1/1     Running   0          13s
hello-app-6f488fcdfc-xr6r8   1/1     Running   0          13s
hello-app-6f488fcdfc-zx8st   1/1     Running   0          13s

Clean up

Delete the MachineDeployment to remove the created machines and VMs:

$ kubectl delete machinedeployment --namespace kube-system scalable-machine-deployment "scalable-machine-deployment" deleted

Delete the namespace:

$ kubectl delete namespace hna-tutorial
namespace "hna-tutorial" deleted