Written by Vignesh Ragupathy
on November 24, 2019

[ kubernetes linux opensource ]

Pod scheduling in kubernetes - detailed step by step

We can assign the pod to node based on various methods. Lets discuss all the below methods in the post

Using nodeName
Using labels in nodeSelector
Node Affinity/Anti Affinity
Pod Affinity/Anti Affinity
Taints and tolerations

Setup

I am using the Virtualbox(running in Ubuntu 18.04 physical machine) for this entire setup . The physical machine is Dell inspiron laptop with 12GB RAM , Intel® Core™ i7-6500U CPU @ 2.50GHz × 4 and 512GB SSD hardisk.

Using nodeName

Step 1: Create a pod and assign using nodeName

vikki@kubernetes1:~$ vim pod_node_name.yaml

vikki@kubernetes1:~$ kubectl apply -f pod_node_name.yaml 
pod/nginx-pod-nodename created

vikki@kubernetes1:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 4 5h47m 192.168.249.141 kubernetes2 <none> <none>
nginx-pod-nodename 1/1 Running 0 8s 192.168.80.199 kubernetes3 <none> <none>
web-0 1/1 Running 2 5h21m 192.168.249.139 kubernetes2 <none> <none>
web-1 1/1 Running 2 5h21m 192.168.249.140 kubernetes2 <none> <none>
web-2 1/1 Running 2 5h22m 192.168.249.138 kubernetes2 <none> <none>

Now we can see the pod is created in the node “kubernetes3” conifgued in nodeName option

Using labels in nodeSelector

Step 1: Add a new label to the node

Check the current lables using the below command

vikki@kubernetes1:~$ kubectl get nodes --show-labels 
NAME STATUS ROLES AGE VERSION LABELS
kubernetes1 Ready master 20d v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes1,kubernetes.io/os=linux,node-role.kubernetes.io/master=
kubernetes2 Ready <none> 20d v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes2,kubernetes.io/os=linux
kubernetes3 Ready <none> 117m v1.16.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes3,kubernetes.io/os=linux,namee=node3

Add a new lable “disktype=vhd” to the kubernetes3 node

vikki@kubernetes1:~$ kubectl label nodes kubernetes3 disktype=vhd
node/kubernetes3 labeled

vikki@kubernetes1:~$ kubectl get nodes --show-labels 
NAME STATUS ROLES AGE VERSION LABELS
kubernetes1 Ready master 20d v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes1,kubernetes.io/os=linux,node-role.kubernetes.io/master=
kubernetes2 Ready <none> 20d v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes2,kubernetes.io/os=linux
kubernetes3 Ready <none> 118m v1.16.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=vhd,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes3,kubernetes.io/os=linux,namee=node3

Step 2: Create a pod and assign using nodeSelector

vikki@kubernetes1:~$ vim pod_label.yaml

vikki@kubernetes1:~$ kubectl create -f pod_label.yaml 
pod/nginx-pod-label created

vikki@kubernetes1:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 3 4h50m 192.168.249.141 kubernetes2 <none> <none>
nginx-pod-label 1/1 Running 0 4m14s 192.168.80.195 kubernetes3 <none> <none>
web-0 1/1 Running 2 4h25m 192.168.249.139 kubernetes2 <none> <none>
web-1 1/1 Running 2 4h25m 192.168.249.140 kubernetes2 <none> <none>
web-2 1/1 Running 2 4h25m 192.168.249.138 kubernetes2 <none> <none>
vikki@kubernetes1:~$

Now we can see the new pod is assinged to kubernetes3 based on nodeSelector label option

Advance pod scheduling

We can also assing the pod to a specific node using the Node/pod affinity and anti affinity rules.

Affinity types:

requiredDuringSchedulingRequiredDuringExecution
requiredDuringSchedulingIgnoredDuringExecution
preferredDuringSchedulingIgnoredDuringExecution

Affinity operators:

In
NotIn
Exists
DoesNotExist
Gt
Lt

Node Affinity/Anti Affinity

Step 1: Create a pod with node affinity

Create a pod with node affinity specs and match the lable using matchExpressions spec

vikki@kubernetes1:~$ vim pod_node_affinity.yaml

vikki@kubernetes1:~$ kubectl create -f pod_node_affinity.yaml 
pod/nginx-pod-nodeaffinity created

vikki@kubernetes1:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 3 5h17m
nginx-pod-label 1/1 Running 0 31m
nginx-pod-nodeaffinity 0/1 Pending 0 4s
web-0 1/1 Running 2 4h52m
web-1 1/1 Running 2 4h52m
web-2 1/1 Running 2 4h52m

vikki@kubernetes1:~$ kubectl describe pods nginx-pod-nodeaffinity |grep Events: -A 3
Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 3 node(s) didn't match node selector.

Now the pod is failing because there is no node has the match lables

Lets add a label bandwidth to the kubernetes3 node

vikki@kubernetes1:~$ kubectl label nodes kubernetes3 bandwidth=100GB
node/kubernetes3 labeled

vikki@kubernetes1:~$ kubectl describe pods nginx-pod-nodeaffinity |grep Events: -A 5
Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 3 node(s) didn't match node selector.
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 3 node(s) didn't match node selector.
    Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-pod-nodeaffinity to kubernetes3

vikki@kubernetes1:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 3 5h23m 192.168.249.141 kubernetes2 <none> <none>
nginx-pod-label 1/1 Running 0 37m 192.168.80.195 kubernetes3 <none> <none>
nginx-pod-nodeaffinity 1/1 Running 0 6m16s 192.168.80.196 kubernetes3 <none> <none>
web-0 1/1 Running 2 4h58m 192.168.249.139 kubernetes2 <none> <none>
web-1 1/1 Running 2 4h58m 192.168.249.140 kubernetes2 <none> <none>
web-2 1/1 Running 2 4h59m 192.168.249.138 kubernetes2 <none> <none>

Now we can see the node pod is successully assinged to the kubernetes3 node after adding the label

Pod Affinity/Anti Affinity

We can also assing the pod to a specific node using the pod affinity and anti affinity rules.

Step 1: Create a pod with pod affinity

Create a pod with pod affinity specs and match the lable using matchExpressions spec

vikki@kubernetes1:~$ vim pod_pod_affinity.yaml

vikki@kubernetes1:~$ kubectl create -f pod_pod_affinity.yaml 
pod/nginx-pod-podaffinity created

vikki@kubernetes1:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 4 5h36m
nginx-pod-label 1/1 Running 0 49m
nginx-pod-nodeaffinity 1/1 Running 0 18m
nginx-pod-podaffinity 0/1 Pending 0 4s
web-0 1/1 Running 2 5h10m
web-1 1/1 Running 2 5h10m
web-2 1/1 Running 2 5h11m

vikki@kubernetes1:~$ kubectl describe pods nginx-pod-podaffinity |grep Events: -A 5 
Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity.
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity.

Now the pod is failing because there is no pod running in any nodes that has the match lables

Lets create a new pod with the label configued previously

vikki@kubernetes1:~$ vim pod_label_podaffinity.yaml

vikki@kubernetes1:~$ kubectl apply -f pod_label_podaffinity.yaml 
pod/nginx-pod-web-nginx-backend created

vikki@kubernetes1:~$ kubectl describe pods nginx-pod-podaffinity |grep Events: -A 5 
Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity.
    Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity.
    Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-pod-podaffinity to kubernetes3

vikki@kubernetes1:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 4 5h39m 192.168.249.141 kubernetes2 <none> <none>
nginx-pod-label 1/1 Running 0 52m 192.168.80.195 kubernetes3 <none> <none>
nginx-pod-nodeaffinity 1/1 Running 0 21m 192.168.80.196 kubernetes3 <none> <none>
nginx-pod-podaffinity 1/1 Running 0 3m2s 192.168.80.197 kubernetes3 <none> <none>
nginx-pod-web-nginx-backend 1/1 Running 0 14s 192.168.80.198 kubernetes3 <none> <none>
web-0 1/1 Running 2 5h13m 192.168.249.139 kubernetes2 <none> <none>
web-1 1/1 Running 2 5h13m 192.168.249.140 kubernetes2 <none> <none>
web-2 1/1 Running 2 5h14m 192.168.249.138 kubernetes2 <none> <none>

Now we can see only the pod with lable app: web-nginx-backend is created, the previous pod is also created in the same node

Taints and tolerations

Node affinity is a property of pods that attracts them to a set of nodes. Taints are the opposite, they allow a node to repel a set of pods.

Step 1: Create a label to node and assign pod using nodeSelector

vikki@kubernetes1:~$ kubectl label nodes kubernetes4 app=highperformance
node/kubernetes4 labeled
vikki@kubernetes1:~$ kubectl get nodes kubernetes4 --show-labels 
NAME STATUS ROLES AGE VERSION LABELS
kubernetes4 Ready <none> 2m55s v1.16.3 app=highperformance,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes4,kubernetes.io/os=linux

    vikki@kubernetes1:~$ vim pod_label_1.yaml

vikki@kubernetes1:~$ kubectl create -f pod_label_1.yaml 
pod/nginx-pod-taint created

vikki@kubernetes1:~$ kubectl get pod nginx-pod-taint -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-pod-taint 1/1 Running 0 10m 192.168.48.129 kubernetes4 <none> <none>

We can see the new pod nginx-pod-taint is created and assigned to kubernetes4 node

Step 2: Create a taint

Now lets create a taint “NoSchedule” and add to the kubernetes4 node.

vikki@kubernetes1:~$ kubectl taint nodes kubernetes4 key1=value1:NoSchedule
node/kubernetes4 tainted

vikki@kubernetes1:~$ kubectl get nodes kubernetes4 -o yaml |grep -i taint -A 3
    taints:
    - effect: NoSchedule
    key: key1
    value: value1

Step 3: Create a new pod and try assign to kubernetes4

Now create a new pod and use nodeSelector to assign to kubernetes4

vikki@kubernetes1:~$ vim pod_label_2.yaml

vikki@kubernetes1:~$ kubectl create -f pod_label_2.yaml 
pod/nginx-pod-taint-2 created

vikki@kubernetes1:~$ kubectl get pods nginx-pod-taint-2 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-pod-taint-2 0/1 Pending 0 7s <none> <none> <none> <none>

vikki@kubernetes1:~$ kubectl describe pod nginx-pod-taint-2 |grep -i events: -A 5
Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 3 node(s) didn't match node selector.
    Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 3 node(s) didn't match node selector.

We can see the pod creatation if failing due to the taints setting.

Step 4: Update the pod with tolerations

Now lets add toleration to the same pod for “NoSchedule” and apply the changes.

vikki@kubernetes1:~$ vim pod_label_3.yaml

vikki@kubernetes1:~$ kubectl apply -f pod_label_3.yaml 
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
pod/nginx-pod-taint-2 configured

vikki@kubernetes1:~$ kubectl describe pod nginx-pod-taint-2 |grep -i events: -A 5
Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 3 node(s) didn't match node selector.
    Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 3 node(s) didn't match node selector.
    Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-pod-taint-2 to kubernetes4

vikki@kubernetes1:~$ kubectl get pod nginx-pod-taint-2 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-pod-taint-2 1/1 Running 0 11m 192.168.48.130 kubernetes4 <none> <none>

Now we can see the pod changed from failed to success state and assinged to kubernetes4 node according to the nodeSelector.

Share on:

Discussion and feedback

← → Top