Running kubernetes custom scheduler

Kubernetes cluster have a default scheduler kube-scheduler. If the default scheduler does not suits our requirement we can also create our own scheduler. In the post we will discus how to create multiple scheduler and schedule pods based on different scheduler.

Setup

I am using the Virtualbox(running in Ubuntu 18.04 physical machine) for this entire setup . The physical machine is Dell inspiron laptop with 12GB RAM , Intel® Core™ i7-6500U CPU @ 2.50GHz × 4 and 512GB SSD hardisk.

Step 1: Copy the default scheduler yaml and modify

Copy the default scheduler yaml from master node and modify the name.

In this below example i am naming the custom schedule are my-scheduler. Add a ServiceAccount and ClusterRoleBinding to the yaml file as given below.

The key changes made are

line 27: name: my-scheduler
line 30: serviceAccountName: my-scheduler
line 38: - –leader-elect=false
line 39: - –scheduler-name=my-scheduler

vikki@kubernetes1:~$ sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml custom-scheduler.yaml
vikki@kubernetes1:~$ sudo chmod 777 custom-scheduler.yaml 
vikki@kubernetes1:~$ vim custom-scheduler.yaml

	apiVersion: v1
	kind: ServiceAccount
	metadata:
	name: my-scheduler
	namespace: kube-system
	---
	apiVersion: rbac.authorization.k8s.io/v1
	kind: ClusterRoleBinding
	metadata:
	name: my-scheduler-as-kube-scheduler
	subjects:
	- kind: ServiceAccount
	name: my-scheduler
	namespace: kube-system
	roleRef:
	kind: ClusterRole
	name: system:kube-scheduler
	apiGroup: rbac.authorization.k8s.io
	---
	apiVersion: v1
	kind: Pod
	metadata:
	creationTimestamp: null
	labels:
	component: kube-scheduler
	tier: control-plane
	name: my-scheduler
	namespace: kube-system
	spec:
	serviceAccountName: my-scheduler
	containers:
	- command:
	- kube-scheduler
	- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
	- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
	- --bind-address=127.0.0.1
	- --kubeconfig=/etc/kubernetes/scheduler.conf
	- --leader-elect=false
	- --scheduler-name=my-scheduler
	image: k8s.gcr.io/kube-scheduler:v1.16.2
	imagePullPolicy: IfNotPresent
	livenessProbe:
	failureThreshold: 8
	httpGet:
	host: 127.0.0.1
	path: /healthz
	port: 10251
	scheme: HTTP
	initialDelaySeconds: 15
	timeoutSeconds: 15
	name: kube-scheduler
	resources:
	requests:
	cpu: 100m
	volumeMounts:
	- mountPath: /etc/kubernetes/scheduler.conf
	name: kubeconfig
	readOnly: true
	hostNetwork: true
	priorityClassName: system-cluster-critical
	volumes:
	- hostPath:
	path: /etc/kubernetes/scheduler.conf
	type: FileOrCreate
	name: kubeconfig
	status: {}

view raw custom_scheduler.yaml hosted with ❤ by GitHub

Step 2: Create the custom scheduler

vikki@kubernetes1:~$ kubectl create -f custom-scheduler.yaml 
serviceaccount/my-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler created
pod/my-scheduler created

vikki@kubernetes1:~$ kubectl get pods -n=kube-system 
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-55754f75c-fgs8g 1/1 Running 7 22d
calico-node-4h72l 1/1 Running 7 22d
calico-node-ld84s 1/1 Running 7 22d
calico-node-lrfz9 1/1 Running 2 31h
calico-node-ws576 1/1 Running 1 27h
coredns-5644d7b6d9-2g6rs 1/1 Running 7 22d
coredns-5644d7b6d9-ccxsg 1/1 Running 7 22d
etcd-kubernetes1 1/1 Running 8 22d
kube-apiserver-kubernetes1 1/1 Running 8 22d
kube-controller-manager-kubernetes1 1/1 Running 8 22d
kube-proxy-6xd8l 1/1 Running 2 31h
kube-proxy-96q5x 1/1 Running 7 22d
kube-proxy-njl6r 1/1 Running 7 22d
kube-proxy-whlhw 1/1 Running 1 27h
kube-scheduler-kubernetes1 1/1 Running 8 22d
my-scheduler 1/1 Running 0 6s

Now we can see a new custom scheduler my-scheduler is running along with defautl scheduler kube-scheduler-kubernetes1

Step 3: Edit the custerrole for kube-scheduler

If RBAC is enabled on your cluster, you must update the system:kube-scheduler cluster role. Edit the clusterrole for kube-scheduler and modify.

Below are the key changes made in orignal file

line 33: - my-scheduler
line 131: - storageclasses

vikki@kubernetes1:~$ kubectl edit clusterrole system:kube-scheduler

	apiVersion: rbac.authorization.k8s.io/v1
	kind: ClusterRole
	metadata:
	annotations:
	rbac.authorization.kubernetes.io/autoupdate: "true"
	creationTimestamp: "2019-11-03T13:47:53Z"
	labels:
	kubernetes.io/bootstrapping: rbac-defaults
	name: system:kube-scheduler
	resourceVersion: "108110"
	selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Akube-scheduler
	uid: 7a542070-5e51-46c1-9336-1bb9d20f166b
	rules:
	- apiGroups:
	- ""
	- events.k8s.io
	resources:
	- events
	verbs:
	- create
	- patch
	- update
	- apiGroups:
	- ""
	resources:
	- endpoints
	verbs:
	- create
	- apiGroups:
	- ""
	resourceNames:
	- kube-scheduler
	- my-scheduler
	resources:
	- endpoints
	verbs:
	- delete
	- get
	- patch
	- update
	- apiGroups:
	- ""
	resources:
	- nodes
	verbs:
	- get
	- list
	- watch
	- apiGroups:
	- ""
	resources:
	- pods
	verbs:
	- delete
	- get
	- list
	- watch
	- apiGroups:
	- ""
	resources:
	- bindings
	- pods/binding
	verbs:
	- create
	- apiGroups:
	- ""
	resources:
	- pods/status
	verbs:
	- patch
	- update
	- apiGroups:
	- ""
	resources:
	- replicationcontrollers
	- services
	verbs:
	- get
	- list
	- watch
	- apiGroups:
	- apps
	- extensions
	resources:
	- replicasets
	verbs:
	- get
	- list
	- watch
	- apiGroups:
	- apps
	resources:
	- statefulsets
	verbs:
	- get
	- list
	- watch
	- apiGroups:
	- policy
	resources:
	- poddisruptionbudgets
	verbs:
	- get
	- list
	- watch
	- apiGroups:
	- ""
	resources:
	- persistentvolumeclaims
	- persistentvolumes
	verbs:
	- get
	- list
	- watch
	- apiGroups:
	- authentication.k8s.io
	resources:
	- tokenreviews
	verbs:
	- create
	- apiGroups:
	- authorization.k8s.io
	resources:
	- subjectaccessreviews
	verbs:
	- create
	- apiGroups:
	- storage.k8s.io
	resources:
	- csinodes
	- storageclasses
	verbs:
	- get
	- list
	- watch

view raw custerrole.yaml hosted with ❤ by GitHub

Step 4: Create pods with different types of scheduler

Create a pod without explicitly mentioning any scheuler name. This pod should be scheduled by default scheduler kube-scheduler

vikki@kubernetes1:~$ vim pod_default_scheduler.yaml

	apiVersion: v1
	kind: Pod
	metadata:
	name: nginx-pod-default-scheduler
	spec:
	containers:
	- name: nginx-pod-default-scheduler
	image: nginx

view raw pod_default_scheduler.yaml hosted with ❤ by GitHub

Create a pod by explicitly mentioning custom scheuler name my-scheduler.

vikki@kubernetes1:~$ vim pod_custom_scheduler.yaml

	apiVersion: v1
	kind: Pod
	metadata:
	name: nginx-pod-custom-scheduler
	spec:
	schedulerName: my-scheduler
	containers:
	- name: nginx-pod-custom-scheduler
	image: nginx

view raw pod_custom_scheduler.yaml hosted with ❤ by GitHub

vikki@kubernetes1:~$ kubectl create -f pod_default_scheduler.yaml 
pod/nginx-pod-default-scheduler created
vikki@kubernetes1:~$ kubectl create -f pod_custom_scheduler.yaml 
pod/nginx-pod-custom-scheduler created

vikki@kubernetes1:~$ kubectl get pods nginx-pod-custom-scheduler nginx-pod-default-scheduler
NAME READY STATUS RESTARTS AGE
nginx-pod-custom-scheduler 1/1 Running 0 9m9s
nginx-pod-default-scheduler 1/1 Running 0 9m13s
vikki@kubernetes1:~$

Now we can see both the pods are created and running

vikki@kubernetes1:~$ kubectl get events -o wide |grep nginx-pod-custom-scheduler |head -1
<unknown> Normal Scheduled pod/nginx-pod-custom-scheduler my-scheduler Successfully assigned default/nginx-pod-custom-scheduler to kubernetes3 <unknown> 0 nginx-pod-custom-scheduler.15da76383c2c2859

vikki@kubernetes1:~$ kubectl get events -o wide |grep nginx-pod-default-scheduler |head -1
<unknown> Normal Scheduled pod/nginx-pod-default-scheduler default-scheduler Successfully assigned default/nginx-pod-default-scheduler to kubernetes3 <unknown> 0 nginx-pod-default-scheduler.15da76372f4444f7

From the events, we can see pod nginx-pod-custom-scheduler is scheduled by my-scheduler and nginx-pod-default-scheduler by default-scheduler.

Share on:

Discussion and feedback

← → Top