Kubernetes cluster on AWS EC2 instances

All credits goes to the author of the original post "12 steps to setup Kubernetes Cluster on AWS EC2".

In my testbed, I've checked the post and successfully created Kubernetes cluster. There were some caveats related to the updated versions of the software, so my post introduces some improvements to the original post in the section on cluster kubelet error troubleshooting and network CNI plugin installation.

Installation notes:

In my installation I have several AWS profiles. To specify the required profile for usage in AWS CLI use: --profile=[profile_name]
My region is: eu-central-1. To specify region in AWS CLI use: --region=eu-central-1
Instance type is: t2.medium. To save money you can use: t2.micro. But there may be issues related to the performance.
SSH key pair name is: makbanov-aws-ec2
If you have several AWS CLI profiles: When using setup.sh script for instance profiles, manually change the script by adding --profile=[profile_name] --region=[region_name] to the commands. Then run the script.
For CNI install check the following links:
- https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.3.4
- https://github.com/aws/amazon-vpc-cni-k8s/tree/v1.3.4/config/v1.3

Installation scheme

(credits: www.golinuxcloud.com): scheme

Instruction on how to install Kubernetes cluster on AWS

Create SSH key-pair and import into AWS region

Choose region in AWS console (Frankfurt eu-central-1)

On your local host generate the key pairt

cd ~/.ssh
ssh-keygen -t rsa -P "" -f [key_pair_name]

Copy the content of the generated public key and import into the AWS region on EC2 section

Create VPC

VPC_ID=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --region=eu-central-1 --query "Vpc.VpcId" --output text --profile=work)

Show VPC ID:
```
echo $VPC_ID
```

Enable DNS support

DNS support:

aws ec2 modify-vpc-attribute --enable-dns-support --vpc-id $VPC_ID --profile=work --region=eu-central-1

Enable DNS hostname support:

aws ec2 modify-vpc-attribute --enable-dns-hostnames --vpc-id $VPC_ID --profile=work --region=eu-central-1

Add tags to the VPC and subnet

aws ec2 create-tags --resources $VPC_ID --tags Key=Name,Value=monitoring-stand Key=kubernetes.io/cluster/monitoring-stand,Value=shared --profile=work --region=eu-central-1

Get Private route table ID

Get ID:

PRIVATE_ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=$VPC_ID --query "RouteTables[0].RouteTableId" --output=text --profile=work --region=eu-central-1)

Show ID
```
echo $PRIVATE_ROUTE_TABLE_ID
```

Add second route table to manage public subnets in VPC

Get Public route ID:

PUBLIC_ROUTE_TABLE_ID=$(aws ec2 create-route-table --vpc-id $VPC_ID --query "RouteTable.RouteTableId" --output text --profile=work --region=eu-central-1)

Show ID:
```
echo $PUBLIC_ROUTE_TABLE_ID
```

Give the route tables names

Name public subnet:

aws ec2 create-tags --resources $PUBLIC_ROUTE_TABLE_ID --tags Key=Name,Value=monitoring-stand-public --profile=work --region=eu-central-1

Name private subnet:

aws ec2 create-tags --resources $PRIVATE_ROUTE_TABLE_ID --tags Key=Name,Value=monitoring-stand-private --profile=work --region=eu-central-1

Create private and public subnets for cluster

Get all available AZs in your region:

aws ec2 describe-availability-zones --region=eu-central-1

Create private subnet with CIDR /24 == 256 IP in eu-central-1b:

PRIVATE_SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID --availability-zone eu-central-1b --cidr-block 10.0.0.0/24 --query "Subnet.SubnetId" --output text --profile=work --region=eu-central-1)

Show private subnet ID:
```
echo $PRIVATE_SUBNET_ID
```

Create tags for subnet:

aws ec2 create-tags --resources $PRIVATE_SUBNET_ID --tags Key=Name,Value=monitoring-stand-private-1b Key=kubernetes.io/cluster/monitoring-stand,Value=owned Key=kubernetes.io/role/internal-elb,Value=1 --profile=work --region=eu-central-1

Create public subnet in the same AZ:

PUBLIC_SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID --availability-zone eu-central-1b --cidr-block 10.0.16.0/24 --query "Subnet.SubnetId" --output text --profile=work --region=eu-central-1)

Show public subnet ID:
```
echo $PUBLIC_SUBNET_ID
```

Create tags for public subnet:

aws ec2 create-tags --resources $PUBLIC_SUBNET_ID --tags Key=Name,Value=monitoring-stand-public-1b Key=kubernetes.io/cluster/monitoring-stand,Value=owned Key=kubernetes.io/role/elb,Value=1 --profile=work --region=eu-central-1

Associate public subnet with the public route table:

ws ec2 associate-route-table --subnet-id $PUBLIC_SUBNET_ID --route-table-id $PUBLIC_ROUTE_TABLE_ID --profile=work --region=eu-central-1

Create Internet Gateway

In order for the instances in our public subnet to communicate with the internet, we will create an internet gateway, attach it to our VPC, and then add a route to the route table, routing traffic bound for the internet to the gateway
```
INTERNET_GATEWAY_ID=$(aws ec2 create-internet-gateway --query "InternetGateway.InternetGatewayId" --output text --profile=work --region=eu-central-1)
```
Show IG ID:
```
echo $INTERNET_GATEWAY_ID
```

Attach internet gateway to VPC:

aws ec2 attach-internet-gateway --internet-gateway-id $INTERNET_GATEWAY_ID --vpc-id $VPC_ID --profile=work --region=eu-central-1

Create public route table:

aws ec2 create-route --route-table-id $PUBLIC_ROUTE_TABLE_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $INTERNET_GATEWAY_ID --profile=work --region=eu-central-1

Create NAT gateway

In order to configure the instances in the private subnet, we will need them to be able to make outbound connections to the internet in order to install software packages and so on.

To make this possible, we will add a NAT gateway to the public subnet and then add a route to the private route table for internet-bound traffic

Allocate address for NAT gateway:

NAT_GATEWAY_ALLOCATION_ID=$(aws ec2 allocate-address --domain vpc --query AllocationId --output text --profile=work --region=eu-central-1)

Show NAT Gateway ID:
```
echo $NAT_GATEWAY_ALLOCATION_ID
```

Create NAT gateway:

NAT_GATEWAY_ID=$(aws ec2 create-nat-gateway --subnet-id $PUBLIC_SUBNET_ID --allocation-id $NAT_GATEWAY_ALLOCATION_ID --query NatGateway.NatGatewayId --output text --profile=work --region=eu-central-1)

Show NAT gateway ID:
```
echo $NAT_GATEWAY_ID
```

Create route:

At this stage, you may have to wait a few moments for the NAT gateway to be created before creating the route.

Create route:

aws ec2 create-route --route-table-id $PRIVATE_ROUTE_TABLE_ID --destination-cidr-block 0.0.0.0/0 --nat-gateway-id $NAT_GATEWAY_ID --profile=work --region=eu-central-1

Set bastion host:

We will use the first host we are going to launch as a bastion host that will allow us to connect to other servers that are only accessible from within the private side of our VPC network.

Create security group:

Create a security group to allow SSH traffic to the bastion host:

BASTION_SG_ID=$(aws ec2 create-security-group --group-name ssh-bastion --description "SSH Bastion Hosts" --vpc-id $VPC_ID --query GroupId --output text --profile=work --region=eu-central-1)

Show Bastion host SG ID:
```
echo $BASTION_SG_ID
```

Allow SSH ingress on port 22 of the Bastion Host. For all Internet traffic (insecure) use 0.0.0.0/0. Recommended way to use your own stable IP address:

aws ec2 authorize-security-group-ingress --group-id $BASTION_SG_ID --protocol tcp --port 22 --cidr 109.233.108.6/32 --profile=work --region=eu-central-1

Create EC2 Compute Instance for bastion host

Secect official Ubuntu AMI ID for your EC2 instances:

UBUNTU_AMI_ID=$(aws ec2 describe-images --owners 099720109477 --filters Name=root-device-type,Values=ebs Name=architecture,Values=x86_64 Name=name,Values='*hvm-ssd/ubuntu-xenial-16.04*' --query "sort_by(Images, &Name)[-1].ImageId" --output text --profile=work --region=eu-central-1)

Show Ubuntu AMI ID:
```
echo $UBUNTU_AMI_ID
```

Run bastion host based on EC2 t2.micro instance with SSH key pairt makbanov-aws-ec2

BASTION_ID=$(aws ec2 run-instances --image-id $UBUNTU_AMI_ID --instance-type t2.micro --key-name makbanov-aws-ec2 --security-group-ids $BASTION_SG_ID  --subnet-id $PUBLIC_SUBNET_ID --associate-public-ip-address --query "Instances[0].InstanceId" --output text --profile=work --region=eu-central-1)

Show Bastion ID:
```
echo $BASTION_ID
```

Update the Bastion instance with the Name tag to recognize it in the AWS EC2 dashboard:

aws ec2 create-tags --resources $BASTION_ID --tags Key=Name,Value=ssh-bastion --profile=work --region=eu-central-1

Access bastion host

Once the instance has launched, you should be able to run the aws ec2 describe-instances command to discover the public IP address of your new instance:

BASTION_IP=$(aws ec2 describe-instances --instance-ids $BASTION_ID --query "Reservations[0].Instances[0].PublicIpAddress" --output text --profile=work --region=eu-central-1)

Show Bastion IP:
```
echo $BASTION_IP
```
Check SSH connection to the Bastion host. You should now be able to access the instance using the private key from the same key pair as used to create the instane:
```
ssh -i ~/.ssh/work-makbanov-aws-ec2 ubuntu@$BASTION_IP
```

Install sshutle to configure proxy

It is possible to forward traffic from your workstation to the private network by just using SSH port forwarding. However, we can make accessing servers via the bastion instance much more convenient by using the sshuttle tool.

Install sshuttle:

pip install sshuttle

or for Ubuntu:

sudo apt-get install sshuttle

To transparently proxy traffic to the instances inside the private network, we can run the following command: In the separate terminal run:

export BASTION_IP=[bastion_ip_address]
export BASTION_IP=3.120.98.213

run sshuttle:

sshuttle -r ubuntu@$BASTION_IP 10.0.0.0/16 --dns  --ssh-cmd 'ssh -i ~/.ssh/work-makbanov-aws-ec2'

On another terminal, we can validate that this setup is working correctly by trying to log in to our instance through its private DNS name:

ws ec2 describe-instances --profile=work --region=eu-central-1 --instance-ids $BASTION_ID --query "Reservations[0].Instances[0].PrivateDnsName"

Now that we have the DNS name, try to connect to the instance using the DNS name:
```
ssh -i ~/.ssh/work-makbanov-aws-ec2 ubuntu@ip-10-0-16-105.eu-central-1.compute.internal
```
This tests whether you can resolve a DNS entry from the private DNS provided by AWS to instances running within your VPC, and whether the private IP address now returned by that query is reachable.

Create Instance Profiles

In order for Kubernetes to make use of its integrations with the AWS cloud APIs, we need to set up IAM instance profiles. An instance profile is a way for the Kubernetes software to authenticate with the AWS API, and for us to assign fine-grained permissions on the actions that Kubernetes can take.
```
curl https://raw.githubusercontent.com/errm/k8s-iam-policies/master/setup.sh -o setup.sh
```
Execute this script:
```
sh -e setup.sh
```

Create AMI - Install Kubernetes Software

Now we will create one EC2 instance to setup our Kubernetes Cluster. We will use this as a AMI to create EC2 instances for our Kubernetes Cluster on AWS.

Create security group

Create SG for this intsance:

K8S_AMI_SG_ID=$(aws ec2 create-security-group --group-name k8s-ami --description "Kubernetes AMI Instances" --vpc-id $VPC_ID --query GroupId --output text --profile=work --region=eu-central-1)

We will need to be able to access this instance from our bastion host in order to log in and install software, so let's add a rule to allow SSH traffic on port 22 from instances in the ssh-bastion security group, as follows:
```
aws ec2 authorize-security-group-ingress --group-id $K8S_AMI_SG_ID --protocol tcp --port 22 --source-group $BASTION_SG_ID --profile=work --region=eu-central-1
```

Create EC2 instance

We're using t2.medium type:

K8S_AMI_INSTANCE_ID=$(aws ec2 run-instances --subnet-id $PRIVATE_SUBNET_ID --image-id $UBUNTU_AMI_ID --instance-type t2.medium --key-name makbanov-aws-ec2 --security-group-ids $K8S_AMI_SG_ID  --query "Instances[0].InstanceId" --output text --profile=work --region=eu-central-1)

Show instance ID:
```
echo $K8S_AMI_INSTANCE_ID
```

Add Name tag for the instance:

aws ec2 create-tags --resources $K8S_AMI_INSTANCE_ID --tags Key=Name,Value=kubernetes-node-ami --profile=work --region=eu-central-1

Grab the IP address of the instance:

K8S_AMI_IP=$(aws ec2 describe-instances --instance-ids $K8S_AMI_INSTANCE_ID --query "Reservations[0].Instances[0].PrivateIpAddress" --output text --profile=work --region=eu-central-1)

Show IP address of the instance:
```
echo $K8S_AMI_IP
```

ssh -i ~/.ssh/work-makbanov-aws-ec2 ubuntu@$K8S_AMI_IP

Now we are ready to start configuring the instance with the software and configuration that all of the nodes in our cluster will need. Start by synchronizing the apt repositories:
```
sudo apt-get update
```
Install container runtime (Docker):
Change to root:
```
sudo su -
```
Kubernetes will work well with the version of Docker that is included in the Ubuntu repositories, so we can install it simply by installing the docker.io package, as follows:
```
apt-get install -y docker.io
```
Check that Docker is installed:
```
docker version
```

Install Kubernetes packages

Next, we will install the packages that we need to set up a Kubernetes control plane on this host. These packages are described in the following list:
- kubelet: The node agent that Kubernetes uses to control the container runtime. This is used to run all the other components of the control plane within Docker containers.
- kubeadm: This utility is responsible for bootstrapping a Kubernetes cluster.
- kubectl: The Kubernetes command-line client, which will allow us to interact with the Kubernetes API server.

Add the signing key for the apt repository that hosts the Kubernetes packages:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Add the Kubernetes apt repository:

apt-add-repository 'deb http://apt.kubernetes.io/ kubernetes-xenial main'

Resynchronize the package indexes
```
apt-get update
```

Install the required packages:

apt-get install -y kubelet kubeadm kubectl

Shutdown the instance:
```
shutdown -h now
```

Create an AMI

We can use the create-image command to instruct AWS to snapshot the root volume of our instance and use it to produce an AMI.

K8S_AMI_ID=$(aws ec2 create-image --name k8s-1.10.3-001 --instance-id $K8S_AMI_INSTANCE_ID --description "Kubernetes v1.10.3" --query ImageId --output text --profile=work --region=eu-central-1)

Check the status with the describe-images command:

aws ec2 describe-images --profile=work --region=eu-central-1 --image-ids $K8S_AMI_ID --query "Images[0].State"

Setup Kubernetes Cluster on AWS

Now we can launch an instance for Kubernetes control plane components.

Create security group

Create a security group for this new instance:

K8S_MASTER_SG_ID=$(aws ec2 create-security-group --group-name k8s-master --description "Kubernetes Master Hosts" --vpc-id $VPC_ID --query GroupId --output text --profile=work --region=eu-central-1)

Show SG ID:
```
echo $K8S_MASTER_SG_ID
```
We will need to be able to access this instance from our bastion host in order to log in and configure the cluster. We will add a rule to allow SSH traffic on port 22 from instances in the ssh-bastion security group:
```
aws ec2 authorize-security-group-ingress --group-id $K8S_MASTER_SG_ID --protocol tcp --port 22 --source-group $BASTION_SG_ID --profile=work --region=eu-central-1
```

Launch EC2 instance using AMI:

Now we can launch the instance using the AMI image we created earlier which contains all the Kubernetes packages and docker as container runtime:

K8S_MASTER_INSTANCE_ID=$(aws ec2 run-instances --private-ip-address 10.0.0.11 --subnet-id $PRIVATE_SUBNET_ID --image-id $K8S_AMI_ID --instance-type t2.medium --key-name makbanov-aws-ec2 --security-group-ids $K8S_MASTER_SG_ID --credit-specification CpuCredits=unlimited --iam-instance-profile Name=K8sMaster --query "Instances[0].InstanceId" --output text --profile=work --region=eu-central-1)

Show Master instance ID:
```
echo $K8S_MASTER_INSTANCE_ID
```

Give an instance name:

aws ec2 create-tags --resources $K8S_MASTER_INSTANCE_ID --tags Key=Name,Value=monitoring-stand-k8s-master Key=kubernetes.io/cluster/monitoring-stand,Value=owned --profile=work --region=eu-central-1

Connect to this instance:

ssh -i ~/.ssh/work-makbanov-aws-ec2 ubuntu@10.0.0.11

Pre-requisite configuration of controller node

To ensure that all the Kubernetes components use the same name, we should set the hostname to match the name given by the AWS metadata service. This is because the name from the metadata service is used by components that have the AWS cloud provider enabled. The 169.254. 169.254 IP address is a “magic” IP in the cloud world, in AWS it used to retrieve user data and instance metadata specific to a instance. In Ubuntu instance:
```
sudo hostnamectl set-hostname $(curl http://169.254.169.254/latest/meta-data/hostname)
```
Check hostname:
```
hostname
```
To correctly configure the kubelet to use the AWS cloud provider, we create a systemd drop-in file to pass some extra arguments to the kubelet:
```
printf '[Service]\nEnvironment="KUBELET_EXTRA_ARGS=--node-ip=10.0.0.11"' | sudo tee /etc/systemd/system/kubelet.service.d/20-aws.conf
```

Check:

cat /etc/systemd/system/kubelet.service.d/20-aws.conf

Reload the configuration file and restart kubelet service:

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Initialize controller node

We need to provide kubeadm with --token-ttl 0, this means that the token that is issued to allow worker nodes to join the cluster won't expire. Now initialize the controller node:
```
sudo su -
kubeadm init --token-ttl 0 --ignore-preflight-errors=NumCPU --ignore-preflight-errors=Mem
```

If you will encounter error related to kubelet then:

kubeadm reset

cat <<EOF | sudo tee /etc/docker/daemon.json
{
 "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

cat /etc/docker/daemon.json
systemctl daemon-reload
systemctl restart docker

kubeadm init --token-ttl 0 --ignore-preflight-errors=NumCPU --ignore-preflight-errors=Mem

You must save the kubeadm join command as highlighted above as we will use this to join worker node to our controller node.:

Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.11:6443 --token kp9gfv.5g1x40k2dgz6oobx \
 --discovery-token-ca-cert-hash sha256:c045d5deaddc565839e1046f084dbd969aa5b2c74774a92c02e4c260402731a7

We can check that the API server is functioning correctly by following the instructions given by kubeadm to set up kubectl on the host:

ubuntu@ip-10-0-0-11:~$ mkdir -p $HOME/.kube
ubuntu@ip-10-0-0-11:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
ubuntu@ip-10-0-0-11:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

If you're planning to use root user then (not recommended):

ubuntu@ip-10-0-0-11:~$ sudo su -
root@ip-10-0-0-11:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@ip-10-0-0-11:~# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

Check the version of kubectl:

kubectl version --short

OUTPUT:

Client Version: v1.23.5
Server Version: v1.23.5

Check the status of cluster nodes:
```
kubectl get nodes
```
Currently the status of the controller node is NotReady because the network plugin is still not installed.

Install Container Network Interface (CNI) Plugin

We will be deploying a CNI plugin called amazon-vpc-cni-k8s that integrates Kubernetes with the native networking capabilities of the AWS VPC network.

This plugin works by attaching secondary private IP addresses to the elastic network interfaces of the EC2 instances that form the nodes of our cluster, and then assigning them to pods as they are scheduled by Kubernetes to go into each node. Traffic is then routed directly to the correct node by the AWS VPC network fabric.

touch aws-k8s-cni.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
# kubernetes versions before 1.8.0 should use rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: aws-node
rules:
- apiGroups:
  - crd.k8s.amazonaws.com
  resources:
  - "*"
  - namespaces
  verbs:
  - "*"
- apiGroups: [""]
  resources:
  - pods
  - nodes
  - namespaces
  verbs: ["list", "watch", "get"]
- apiGroups: ["extensions"]
  resources:
  - daemonsets
  verbs: ["list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-node
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
# kubernetes versions before 1.8.0 should use rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: aws-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: aws-node
subjects:
- kind: ServiceAccount
  name: aws-node
  namespace: kube-system
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: aws-node
  namespace: kube-system
  labels:
    k8s-app: aws-node
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      k8s-app: aws-node
  template:
    metadata:
      labels:
        k8s-app: aws-node
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: aws-node
      hostNetwork: true
      tolerations:
      - operator: Exists
      containers:
      - image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.3.4
        imagePullPolicy: Always
        ports:
        - containerPort: 61678
          name: metrics
        name: aws-node
        env:
          - name: AWS_VPC_K8S_CNI_LOGLEVEL
            value: DEBUG
          - name: MY_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: WATCH_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        resources:
          requests:
            cpu: 10m
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /host/opt/cni/bin
          name: cni-bin-dir
        - mountPath: /host/etc/cni/net.d
          name: cni-net-dir
        - mountPath: /host/var/log
          name: log-dir
        - mountPath: /var/run/docker.sock
          name: dockersock
      volumes:
      - name: cni-bin-dir
        hostPath:
          path: /opt/cni/bin
      - name: cni-net-dir
        hostPath:
          path: /etc/cni/net.d
      - name: log-dir
        hostPath:
          path: /var/log
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: eniconfigs.crd.k8s.amazonaws.com
spec:
  scope: Cluster
  group: crd.k8s.amazonaws.com
  versions:
  - name: v1beta1
    # Each version can be enabled/disabled by Served flag.
    served: true
    # One and only one version must be marked as the storage version.
    storage: true
    # A schema is required
    schema:
      openAPIV3Schema:
        type: object
        properties:
          host:
            type: string
          port:
            type: string
  - name: v1
    served: true
    storage: false
    schema:
      openAPIV3Schema:
        type: object
        properties:
          host:
            type: string
          port:
            type: string
  names:
    plural: eniconfigs
    singular: eniconfig
    kind: ENIConfig

Apply aws-k8s-cni.yaml file:
```
kubectl apply -f aws-k8s-cni.yaml
```
You can monitor the networking plugin that is being installed and started by running the following:
```
root@ip-10-0-0-11:~# kubectl -n kube-system get pods
```
Now you can check the status of your controller node and it should be in Ready state:
```
kubectl get nodes
```

Create worker nodes

Create security group (on local machine):

Create a new security group for the worker nodes

K8S_NODES_SG_ID=$(aws ec2 create-security-group --group-name k8s-nodes --description "Kubernetes Nodes" --vpc-id $VPC_ID --query GroupId --output text --profile=work --region=eu-central-1)

Show SG ID:
```
echo $K8S_NODES_SG_ID
```

We will allow access to the worker nodes via the bastion host in order for us to log in for debugging purposes:

aws ec2 authorize-security-group-ingress --group-id $K8S_NODES_SG_ID --protocol tcp --port 22 --source-group $BASTION_SG_ID --profile=work --region=eu-central-1

We want to allow the kubelet and other processes running on the worker nodes to be able to connect to the API server on the master node. We do this using the following command:
```
aws ec2 authorize-security-group-ingress --group-id $K8S_MASTER_SG_ID --protocol tcp --port 6443 --source-group $K8S_NODES_SG_ID --profile=work --region=eu-central-1
```

Since the kube-dns add-on may run on the master node, let's allow this traffic from the nodes security group, as follows:

aws ec2 authorize-security-group-ingress --group-id $K8S_MASTER_SG_ID --protocol all --port 53 --source-group $K8S_NODES_SG_ID --profile=work --region=eu-central-1

We also need the master node to be able to connect to the APIs that are exposed by the kubelet in order to stream logs and other metrics. We enable this by entering the following command:

ws ec2 authorize-security-group-ingress --group-id $K8S_NODES_SG_ID --protocol tcp --port 10250 --source-group $K8S_MASTER_SG_ID --profile=work --region=eu-central-1

aws ec2 authorize-security-group-ingress --group-id $K8S_NODES_SG_ID --protocol tcp --port 10255 --source-group $K8S_MASTER_SG_ID --profile=work --region=eu-central-1

Finally, we need to allow any pod on any node to be able to connect to any other pod. We do this using the following command:

aws ec2 authorize-security-group-ingress --group-id $K8S_NODES_SG_ID  --protocol all --port -1 --source-group $K8S_NODES_SG_ID --profile=work --region=eu-central-1

Create user-data script (on local machine):

In order to have the worker node(s) register themselves with the master when they start up, we will create a startup script. These are the user-data script which are executed immediately after an instance is started.

I have enabled logging for troubleshooting, systemd configuration to update hostname and connect master node. Lastly the script contains kubeadm join command which will be used to join the worker node to the cluster. This command was printed at the end of kubeadm init stage which we executed earlier.

#!/bin/bash
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
echo END

sudo hostnamectl set-hostname $(curl http://169.254.169.254/latest/meta-data/hostname)
sudo mkdir -p /etc/systemd/system/kubelet.service.d
cat << EOF >/etc/systemd/system/kubelet.service.d/20-aws.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--node-ip=$(curl http://169.254.169.254/latest/meta-data/local-ipv4) --node-labels=node.kubernetes.io/node="
EOF

sudo systemctl daemon-reload
sudo systemctl restart kubelet

sudo kubeadm join 10.0.0.11:6443 --token kp9gfv.5g1x40k2dgz6oobx --discovery-token-ca-cert-hash sha256:c045d5deaddc565839e1046f084dbd969aa5b2c74774a92c02e4c260402731a7

Create AWS::AutoScaling::LaunchConfiguration

The AWS::AutoScaling::LaunchConfiguration resource specifies the launch configuration that can be used by an Auto Scaling group to configure Amazon EC2 instances.

First, we create a launch configuration using the following command. This is like a template of the configuration that the autoscaling group will use to launch our worker nodes. Many of the arguments are similar to those that we would have passed to the EC2 run-instances command:

aws autoscaling create-launch-configuration --launch-configuration-name k8s-node-1.10.3-t2-medium-001 --image-id $K8S_AMI_ID --key-name makbanov-aws-ec2 --security-groups $K8S_NODES_SG_ID --user-data file://~/startup.sh --instance-type t2.medium --iam-instance-profile K8sNode --no-associate-public-ip-address --profile=work --region=eu-central-1

Create AWS::AutoScaling::AutoScalingGroup

The AWS::AutoScaling::AutoScalingGroup resource defines an Amazon EC2 Auto Scaling group, which is a collection of Amazon EC2 instances that are treated as a logical grouping for the purposes of automatic scaling and management.

Once we have created the launch configuration, we can create an autoscaling group, as follows:

aws autoscaling create-auto-scaling-group --auto-scaling-group-name monitoring-stand-t2-medium-nodes --launch-configuration-name k8s-node-1.10.3-t2-medium-001 --min-size 1 --max-size 1 --vpc-zone-identifier $PRIVATE_SUBNET_ID --tags Key=Name,Value=monitoring-stand-k8s-node Key=kubernetes.io/cluster/monitoring-stand,Value=owned Key=k8s.io/cluster-autoscaler/enabled,Value=1 --profile=work --region=eu-central-1

This step will automatically create a new AWS EC2 instance which will act as our worker node. Since we have defined a user data script, so that script will be executed immediately after the launch of the instance and join it to the controller node.

Verify worker node status

Next you can connect to your master node and check the status of available nodes. You should see your worker node in a few minutes once kubeadm is initialized:

root@ip-10-0-0-11:~# kubectl get nodes

Troubleshoot worked node startup

If your worker node is not showing up in the previous command, to troubleshoot you need to ssh into your worker node and check the following logs:
```
/var/log/user-data.log
/var/log/cloud-init-output.log
```

If error related to not starting up and running kubelet process then you need to run the following commands:

cat <<EOF | sudo tee /etc/docker/daemon.json
{
 "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

cat /etc/docker/daemon.json
systemctl daemon-reload
systemctl restart docker

Create a Pod to verify cluster (on Master node)

Create simple web nginx web server deployment on cluster:

apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
 image: nginx
 ports:
 - containerPort: 80

Apply file:

root@ip-10-0-0-11:~# kubectl create -f nginx.yaml

Check the status of the Pod:
```
root@ip-10-0-0-11:~# kubectl get pods
```

Try to connect to the container inside the Pod:

root@ip-10-0-0-11:~# kubectl exec -it nginx -- nginx -v

How to manually install Kubernetes cluster on AWS EC2 instances

Table of contents

Kubernetes cluster on AWS EC2 instances

Links

Installation notes:

Installation scheme

Instruction on how to install Kubernetes cluster on AWS

Create SSH key-pair and import into AWS region

Create VPC

Enable DNS support

Add tags to the VPC and subnet

Get Private route table ID

Add second route table to manage public subnets in VPC

Give the route tables names

Create private and public subnets for cluster

Create Internet Gateway

Create NAT gateway

Create route:

Set bastion host:

Create security group:

Create EC2 Compute Instance for bastion host

Access bastion host

Install sshutle to configure proxy

Create Instance Profiles

Create AMI - Install Kubernetes Software

Create security group

Create EC2 instance

Install Kubernetes packages

Create an AMI

Setup Kubernetes Cluster on AWS

Create security group

Launch EC2 instance using AMI:

Pre-requisite configuration of controller node

Initialize controller node

Install Container Network Interface (CNI) Plugin

Create worker nodes

Create security group (on local machine):

Create user-data script (on local machine):

Create AWS::AutoScaling::LaunchConfiguration

Create AWS::AutoScaling::AutoScalingGroup

Verify worker node status

Troubleshoot worked node startup

Create a Pod to verify cluster (on Master node)