Creating a mini Kubernetes cluster

In this post, we will set up a Kubernetes cluster to start tinkering with. In addition to the Kubernetes cluster, we will add FreeIPA and Keycloak to handle IAM services and help with future application development. The added benefit of using FreeIPA is the centralized login services through LDAP/Kerberos and the PKI setup to help automate the distribution of TLS/SSL certs. Keycloak is used to help offer OpenID-Connect services for now.

In a previous post, we went over setting up a single node virtualization server. In this post, we will use that virtualization server to help spin up a FreeIPA, Keycloak and a 3-node Kubernetes cluster. If you are wondering why we aren’t using minikube, the reason is we wanted to explore near production-ready setups and strategically break and tinker with various elements of a full Kubernetes cluster.

Configure VMs

The first step is to set up the 5 VMs on the virtualization server. This can be done by logging onto that server and running the following commands:

$ sudo virt-clone -o CentOS-Stock -n k8s-master --auto-clone
$ sudo virt-clone -o CentOS-Stock -n k8s-node1 –-auto-clone
$ sudo virt-clone -o CentOS-Stock -n k8s-node2 --auto-clone

For FreeIPA and Keycloak, we had installed a CentOS-8 stock image via VNC and the typical GUI install of the OS. In the following post, I will go over source-to-image to show how we can create these images via code. For now, we clone the CentOS-8 base image.

$ sudo virt-clone -o CentOS-8-Stock -n FreeIPA --auto-clone
$ sudo virt-clone -o CentOS-8-Stock -n Keycloak --auto-clone

Once you have cloned all the stock images to their rightful named images we can start them up via virsh.

$ sudo virsh
virsh # start k8s-master
virsh # start k8s-node1
virsh # start k8s-node2
virsh # start FreeIPA
virsh # start Keycloak

If you have a real cloud environment then you can set up a fixed IP for your VMs via the DHCP server. You can get the MAC address from each of the VM by running:

virsh # dumpxml FreeIPA
…
<interface type=’bridge’>
  <mac address=’52:54:00:ee:94:4c’/>
  <source bridge=’ovs-br0'/>
  <virtualport type=’openvswitch’>
    <parameters interfaceid=’eddeb9c5–23ad-46d0–8b3c-10121ee0291e’/>
  </virtualport>
  <target dev=’vnet5'/>
  <model type=’rtl8139'/>
  <alias name=’net0'/>
  <address type=’pci’ domain=’0x0000' bus=’0x00' slot=’0x03' function=’0x0'/>
</interface>
…

Note the MAC Address in the interface section and record that in your DHCP server.

If you don’t have a way to configure your DHCP server then you will have to edit /etc/sysconfig/network-scripts/ifcfg-eth0 with the following settings

DEVICE=”ens3"
BOOTPROTO=”none”
ONBOOT=”yes”
TYPE=”Ethernet”
USERCTL=”yes”
PEERDNS=”yes”
IPV6INIT=”no”
PERSISTENT_DHCLIENT=”1"
DNS1=”192.168.5.50"
IPADDR=”192.168.5.107"
NETMASK=”255.255.255.0"
GATEWAY=”192.168.5.1"

Note: DEVICE needs to be set to your interface listed on the command `ip a`

You will need to do this for all 5 servers. The easiest way is to log in through the console on virsh and make the edits.

virsh # console FreeIPA

Log in as a standard user and run sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0 and make the appropriate edit.

At this point, I like to take a snapshot of the VMs so I can quickly revert to this state you can run the following commands to take a snapshot.

virsh # shutdown FreeIPA
virsh # snapshot-create-as — domain FreeIPA –name FreeIPA_pre
virsh # start FreeIPA

You need to do this for the other VMs. To save your sanity, you can use the shell script below to automate this part.

Install FreeIPA/Keycloak

There is an ansible playbook that has been created to help install FreeIPA and Keycloak on the CentOS-8 servers. You can get the playbook from the Git repo https://github.com/salimwp/freeipa-ansible.

If you are using a Mac or Linux, just install Python 3 with the Virtual Environment package and run the following commands:

$ git clone https://github.com/salimwp/freeipa-ansible
$ cd freeipa-ansible
$ python3 -m venv .
$ source bin/activate

You will need to edit a few files to meet your site’s needs.

Once the settings are correct. We can run the ansible-playbook:

$ ansible-playbook -i hosts -k -K site.yaml

This will take some time.

In the event, something breaks we can run the following script to revert the VMs to the state before running Ansible. This is the awesome-sauce of virtualization and DevOps. We have the safety to screw up and revert in a snap. You can also feel free to experiment with various settings or rename your DNS if you made a mistake.

When everything has been installed and configured. We can run the following ansible-playbook as a test and create some test accounts.

$ ansible-playbook -i hosts -k -K init.yaml

This will create accounts in both FreeIPA and Keycloak and an example realm in Keycloak.

Now the tricky part is this setup is designed to work with FQDNs. So you will need to set up your DNS on your host machine to use the coredns on the FreeIPA server. If you are using a Mac/Linux, there is a quick way by using SSH to create a SOCKS proxy to a Linux server using coredns as the resolving nameserver. To do this run the following command:

$ ssh -D 8800 -f -C -q -N cloud@192.168.5.50

Then open Mozilla Firefox and go to the Preferences. In the search field, type in proxy.

Click on ‘Settings…’

Select ‘Manual proxy configuration. Make sure SOCKS Host is set to ‘localhost’ and Port ‘8800’. Scroll to the bottom and check ‘Proxy DNS when using SOCKS v5’ and then click on ‘OK’

With your web browser, enter the URL to the ipa01 server. In our example the domain is dc1.sddclabs.com. So we would enter https://ipa01.dc1.sddclabs.com

When you first arrive on the page, you will be notified that this page is a security risk. This is because we don’t have the FreeIPA’s CA key loaded into the browser. Once we get the configuration sorted out and move FreeIPA into production, we can retrieve the CA cert and load it into the browser. For now, click on ‘Advanced…’ and then click on ‘Accept the Risk and Continue’

Now, you should be at the main login page to FreeIPA. The login is ‘admin’ and the password can be found in the secrets directory and the file adminpwd. When you have successfully logged in you will see a list of all the users in the system.

Now we have FreeIPA set up. The next step is to configure Keycloak to pull the login creds from FreeIPA. Log in to the Keycloak dashboard, https://keycloak.dc1.sddclabs.com/auth/admin. Change dc1.sddclabs.com to your defined domain. The login is admin and the password is found in secrets/keycloakadminpassword in the root folder you used to launch ansible from. Once you have logged in as admin, you will be taken to the ‘Example’ realm with the General settings tab open.

Click on ‘User Federation’ and select ‘LDAP’ as the provider. In the next screen enter the following configuration.

Required Settings:
Edit Mode: READ_ONLY
Vendor: ‘Red Hat Directory Server’
Connection URL: ldaps://ipa01.dc1.sddclabs.com #note change dc1.sddclabs.com to match your domain
Users DN: cn=users,cn=accounts,dc=dc1,dc=sddclabs,dc=com
Bind DN: uid=keycloakbind,cn=users,cn=accounts,dc=dc1,dc=sddclabs,dc=com
Bind Credential: (This is found in secrets/user-keycloakbind)

Sync Settings:
Periodic Full Sync: On
Full Sync Period: 180 # Since we are testing, for now, we can have a short duration. In production choose a longer time.
Periodic Changed Users Sync: On
Changed Users Sync Period: 180 # Since we are testing, for now, we can have a short duration. In production choose a longer time.

Click ‘Save’ on the bottom. Once the configurations are saved, click on ‘Synchronize all users’

Click on the ‘Users’ on the left pane under Manage and then ‘View all users’ to see the list of all imported users.

We can configure a better group in FreeIPA and add users to that group to filter out which accounts can be synced to Keycloak. But we will leave that for another day.

Installing Kubernetes

Now that Keycloak and FreeIPA are working together, we can start deploying Kubernetes. With the next 3 CentOS 7.7 VMs up and running we can grab the ansible-playbook to perform this deployment. Run the following commands to deploy Kubernetes.

$ git clone https://github.com/salimwp/k8s-ansible
$ cd k8s-ansible
$ python3 -m venv .
$ source bin/activate
$ pip3 install -r requirements.txt

Verify that the mutlihosts file matches your setup. Then run the following command.

$ ansible-playbook -i multihosts -k -K k8s.yaml

To configure the DNS to resolve with Keycloak and FreeIPA, edit DNSSERVER and DOMAIN in group_vars/all to the appropriate settings and run:

$ ansible-playbook -i multihosts -k -K domain.yaml

At this point, we can only interact with our Kubernetes cluster by logging onto the master node and running kubectl commands via sudo. This will read the configuration file in /root/.kube/config, which contains the credentials to communicate with the Kubernetes cluster.

Log onto the master node and run the following commands:

$ sudo kubectl get services
$ sudo kubectl get nodes

You should see the following results shown in the screenshot below.

At this point, we will take a break for this week. In the following post, we will show a couple of authentication setups to communicate with Kubernetes, one involving Keycloak.

I would like to give a shout-out to one of my early mentors in this area Brian Blakesley. Many useful tips he passed onto me, especially navigating through FreeIPA/Keycloak.