k8s.rottigni.tech

Minikube

Ports

LoadBalancer port

• run "minikube tunnel" to allow LoadBalancer to receive an external IP
• run yaml fle containing LoadBalancer port specified

NodePort

• run yaml file containing NodePort
• run yaml service (--url)

ClusterIP

• run yaml file contaning ClusterIP port

Ingress

• run "minikube tunnel" to create a tunnel that binds the minikube VM to the localhost interface.
• run yaml file containing Ingress configuration
• edit /etc/hosts adding: 127.0.0.1 <host-from-ingress-config.info>
• curl hello-world.info

Ingress + Registry

mkdir /opt/ecr
mkdir /opt/ecr/auth
mkdir /opt/ecr/data

htpasswd -Bbn lorenzo passwd > /opt/ecr/auth/htpasswd

minikube tunnel

kubectl apply -R -f k8s

nano /etc/hosts -> 127.0.0.1 registry.rottigni.tech

curl registry.rottigni.tech:443

Secrets & TLS Certs

localhost

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout registry.rottigni.tech.key -out registry.rottigni.tech.crt -subj "/CN=registry.rottigni.tech" -addext "subjectAltName=DNS:registry.rottigni.tech"

cp registry.rottigni.tech.key /opt/ssl/registry.rottigni.tech.key
cp registry.rottigni.tech.crt /opt/ssl/registry.rottigni.tech.crt

kubectl create secret tls registry-ssl --cert=/opt/ssl/registry.rottigni.tech.crt --key=/opt/ssl/registry.rottigni.tech.key -n lr-labs

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout portfolio.rottigni.tech.key -out portfolio.rottigni.tech.crt -subj "/CN=portfolio.rottigni.tech" -addext "subjectAltName=DNS:portfolio.rottigni.tech"

cp portfolio.rottigni.tech.key /opt/ssl/portfolio.rottigni.tech.key
cp portfolio.rottigni.tech.crt /opt/ssl/portfolio.rottigni.tech.crt

kubectl create secret tls portfolio-ssl --cert=/opt/ssl/portfolio.rottigni.tech.crt --key=/opt/ssl/portfolio.rottigni.tech.key -n lr-labs

Production

sudo certbot certonly --standalone -d registry.rottigni.tech

kubectl create secret tls registry-ssl --cert=/etc/letsencrypt/live/registry.rottigni.tech/fullchain.pem --key=/etc/letsencrypt/live/registry.rottigni.tech/privkey.pem -n lr-labs

Production K8s setup

Control Plane setup

sh sh/k8s-ubuntu.sh
sh sh/k8s-control-plane.sh

# labelling worker node
kubectl label node zeta-rottigni-tech node-role.kubernetes.io/worker=true

Worker Node setup

sh sh/k8s-ubuntu.sh

kubeadm join <control-plane-ip>:6443 --token <token> \
	--discovery-token-ca-cert-hash <sha256-token>

	kubeadm join 149.62.186.95:6443 --token l60hpw.155wv7bokkugntgf \
	--discovery-token-ca-cert-hash sha256:bceeda74a963d2d9d2cef1e25c223111a4278304512e644e110de734e0f0df56

Debugging

Useful commands:

# get nginx logs
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --follow --tail=200

# restart nginx
kubectl rollout restart -n=ingress-nginx deployment ingress-nginx-controller

# get into nginx controller pod
kubectl exec -it -n ingress-nginx deploy/ingress-nginx-controller -- /bin/sh


# debug cert-manager
kubectl get certificate wildcard-tls-secret -n lr-labs
kubectl get certificate -n lr-labs
kubectl get certificaterequest -n lr-labs
kubectl describe certificate -n lr-labs wildcard-tls-secret
describe certificaterequest -n lr-labs wildcard-tls-secret-cw88l
kubectl logs -n cert-manager -l app=cert-manager

# cleanup
kubectl delete deployment,service,ingress,pods,endpoints --all -n lr-labs
kubectl delete deployment,service,ingress,pods,endpoints --all -n cert-manager
kubectl delete deployment,service,ingress,pods,endpoints --all -n ingress-nginx

kubectl delete all --all -n lr-labs
kubectl delete all --all -n cert-manager
kubectl delete all --all -n ingress-nginx
kubectl delete all --all -n default

kubectl patch pvc -n=lr-labs registry-pvc -p '{"metadata":{"finalizers":null}}'
kubectl patch pv portfolio-pv -p '{"metadata": {"finalizers": null}}'

kubectl exec -it deploy/portfolio -n lr-labs -- /bin/bash
kubectl exec -it pod/prisma-seeder-28415050-q4r87 -n lr-labs -- /bin/bash

Docs:

• https://www.techrepublic.com/article/install-containerd-ubuntu/
• https://kubernetes.io/docs/setup/production-environment/container-runtimes/
• https://kubernetes.io/docs/reference/setup-tools/kubeadm/
• https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
• https://medium.com/@olorunfemikawonise_56441/simplifying-kubernetes-installation-on-ubuntu-using-a-bash-shell-script-d75fed68a31
• https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/baremetal.md

Clean up the registry

# delete untagged images older then 14 days
kubectl exec -n lr-labs  -it deploy/registry -- sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;'

# delete unreferenced blob revisions
kubectl exec -n lr-labs -it deploy/registry -- sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/revisions/sha256/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;'