How to add Jenkins remote build agent based on Ubuntu 18.04

SSH into your system
Create Jenkins home directory:
```
sudo mkdir /var/lib/jenkins
```
Check if jenkins folder was created:
```
ls -l /var/lib/ | grep jenkins
```
Create jenkins username for build agent with home directory /var/lib/jenkins. Master node will use this directory to write data into. jenkins user will be used to access Jenkins server:
```
sudo useradd -d /var/lib/jenkins jenkins
```
Make jenkins user to have permissions to operate on /var/lib/jenkins directory:
```
sudo chown jenkins:jenkins /var/lib/jenkins
```
Check if user jenkins has permissions on the directory:
```
ls -l /var/lib/ | grep jenkins
```
Generate a SSH key pair:
```
ssh-keygen
```
Create .ssh folder to store the SSH key pairs:
```
sudo mkdir /var/lib/jenkins/.ssh
```
Copy the contents of the generated public key (e.g. id_rsa.pub):
```
cat ./.ssh/id_rsa.pub
```
Create authorized_keys file under jenkins home folder and store the generated public key there:
```
sudo vi /var/lib/jenkins/.ssh/authorized_keys
```
Install Java environment on the build agent machine:
```
sudo apt install -y openjdk-8-jre-headless
```
Check environment setup:
```
java -version
```
Copy the contents of the private key:
```
cat ./.ssh/id_rsa
```

Go to Manage Jenkins --> Manage nodes and clouds --> New Node
Specify the worker node name and select Permanent Agent (physical server):
In remote home directory box specify the /var/lib/jenkins folder created on build agent machine. For launching method select Launch via SSH. In the Host specify the IP or DNS address of the build agent machine.
Under credentials add build agent credentials for access. Select SSH username with private key:
Under Host verification strategy select Known hosts file Verification Strategy
After saving configuration you will see your worker node:

SSH into Jenkins master node machine
In order to make your build agent connect with Jenkins master node we need to add build agent entry into known_hosts file. After logging into your master node, SSH into your build agent machine to add entry into known_hosts file:
```
ssh [username]@[ip_address]
```
Back on your master node machine create .ssh directory on Jenkins home library if it doesn't exist:
```
sudo mkdir /var/lib/jenkins/.ssh
```
If no configuration is presented currently then (otherwise it will override the configuration, you can manually add it if needed) copy the known_hosts file into Jenkins .ssh folder:
```
sudo cp ./.ssh/known_hosts /var/lib/jenkins/.ssh
```
In the UI dashboard in the agent logs section you will find a successful connection message:

In order to make default builds in Jenkins to run on remote agent, we need to globally configure master build runs by using labels. To do this go to Manage Jenkins --> Configure system --> Choose label for master (e.g master) --> In Usage form select "Only build jobs with label expressions matching this node" --> Save changes:
This change will make sure that if you want to run build on master node, you need to explicitly define this in job configuration by choosing "master" label:
All the rest of jobs by default will run on remote build agent

The authorized_keys file in SSH specifies the SSH keys that can be used for logging into the user account for which the file is configured. It is a highly important configuration file, as it configures permanent access using SSH keys and needs proper management. It stores public keys. Link
The known_hosts File is a client file containing all remotely connected known hosts, and the ssh client uses this file. This file authenticates for the client to the server they are connecting to. The known_hosts file contains the host public key for all known hosts. The use of this file is optional, but, if used then it is prepared by the system administrator. Link
What is actually in known_hosts?
Linux : SSH known_hosts