My Experiments for Truth.

First login to the Control-plane node where you can list the nodes. 1) Drain the node (Remove from cluster) # kubectl drain <node name> You might have to ignore daemonsets and local-data in the machine # kubectl drain --ignore-daemonsets --delete-local-data 2) Now delete the node # kubectl delete node <node name> Now if you want to add the node again, you might want to re-generate the token of kubeadm join command for adding the worker. Always use the --v=5 along with the kubeadm join command which will show the error description. # kubeadm token create Now use this token for the kubeadm join command , so that you can join the worker node.

Setting up Alertmanager and Rules --------------------------------- Now that you have prometheus set up, you need to specify some instructions. The next step is to create a values.yaml file that specifies 1) what the alert rules are, 2) what the Prometheus targets are (i.e the definition of what to scrape and how) and any jobs for Prometheus, and 3) where alerts should be routed to (in this case, Slack). Alert Rules ------------ vi prometheus.values ## Prometheus server ConfigMap entries ## serverFiles:   ## Alerts configuration   ## Ref: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/   alerts:     groups:       - name: Instances         rules:           - alert: InstanceDown             expr: up == 0             for: 5m             labels:               severity: page             annotations:               description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'          

Prometheus setup ################ Prerequisites: -------------- > PV & PVC setup with NFS. >> https://jinojoseph.blogspot.com/2019/11/persistent-volume-in-k8s-multinode.html > helm setup with tiller. >> https://jinojoseph.blogspot.com/2019/10/setting-up-helm-chart-for-k8s-cluster.html git clone https://github.com/jinojosep/k8s.git cd k8s/prometheus vi 1.6-deployments.yaml Replace the values of NFS_SERVER & NFS_PATH NFS_SERVER : 10.0.1.9 NFS_PATH : /srv/nfs/k8sdata vi 1.6-class.yaml Add the below line in the metadata:   annotations:     storageclass.kubernetes.io/is-default-class: "true"   # ubuntu@namenode:~/myk8syamls/nfs-provisioner$ kubectl create -f rbac.yaml -f 1.6-class.yaml -f 1.6-deployment.yaml serviceaccount/nfs-client-provisioner created clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created role.rbac.auth

FROM openjdk:8 # Add Maintainer Info LABEL maintainer="jinojoseph@ndimensions.com" # Add a volume pointing to /tmp VOLUME /tmp # The application's jar file ARG JAR_FILE=/home/ubuntu/YourJarFileNOV7.jar # Add the application's jar to the container COPY ${JAR_FILE} app.jar # Run the jar file ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"] COPY --from=python:3.6 / / ENV APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn RUN set -e; \      apt-get update; \      apt-get install -y --no-install-recommends \         software-properties-common \     ; \      apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0xB1998361219BD9C9; \      apt-add-repository 'deb http://repos.azulsystems.com/debian stable main'; \      apt-get update; \      apt-get install -y --no-install-recommends \         zulu-8 \     ; \      apt-get clean; \     rm -rf /var/tmp/* /tm

Nagios Setup ############# Pre-requisites Setup: Apache ------ # sudo yum install httpd # sudo systemctl start httpd.service # sudo systemctl status httpd.service # sudo systemctl enable httpd.service Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service. Take your servers public ip in the browser, and it will show the apache default page: PHP ---- # sudo yum install php php-mysql # sudo systemctl restart httpd.service Now create a file in /var/www/html/info.php vi info.php phpinfo(); ?> http://ip/info.php This will load the php configuration details Nagios ------ Install Build Dependencies in nagios server # sudo yum install gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip Create Nagios User and Group in nagios server --------------------------------------------- # sudo useradd nagios # sudo groupadd nagcmd # sudo usermod -a -G nagcmd nagios

In K8s inorder to create the HPA we just need to give a below kubectl command like below: # kubectl autoscale deployment myapp  --min=1 --max=4 --cpu-percent=80 But this will give a series of errors like below if metrics API is not registered. ######################################## ubuntu@namenode:~$ kubectl get hpa NAME           REFERENCE                 TARGETS         MINPODS   MAXPODS   REPLICAS   AGE myapp   Deployment/myapp   < unknown> /80%   1         4         2          43h ubuntu@namenode:~$ kubectl describe horizontalpodautoscaler.autoscaling/myapp | grep Warning   Warning  FailedComputeMetricsReplicas  32m (x10333 over 43h)   horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)   Warning  FailedGetResourceMetric       2m7s (x1

Login to the Master Node: ##################### sudo kubeadm reset sudo systemctl stop docker && sudo systemctl stop kubelet sudo rm -rf /etc/kubernetes/ sudo rm -rf .kube/ sudo rm -rf /var/lib/kubelet/ sudo rm -rf /var/lib/cni/ sudo rm -rf /etc/cni/ sudo rm -rf /var/lib/etcd/ ifconfig cni0 down ifconfig flannel.1 down ifconfig docker0 down sudo ip link delete cni0 sudo ip link delete flannel.1 kubeadm reset  does not delete  any  of the iptables rules it originally created. In other words, if you try to bootstrap your cluster with a different pod networking CIDR range or different networking options, you might run into trouble. Please note if you are using a firewall configuration tool like  ufw , which uses iptables as system-of-record, the commands below might render your system inaccessible. Because of this, we recommend that you flush all iptables rules: iptables -F && iptables -t nat -F && iptables -t mangle -F &

Pre-requisites: >> k8s cluster setup with 1 Master and 2 Worker Nodes. >> docker is install in all the nodes. >> static ips for each nodes. Add the blow entries in the /etc/hosts file of all the nodes. 10.0.1.13 registryserver.mydomain.com 10.0.1.12 registryclient01.mydomain.com 10.0.1.14 registryclient02.mydomain.com Then in the Registry server node, issue the below command: Install Docker Registry ####################### Before starting, you will need a Docker private Registry on registry-server instance. First, download the registry image from the Docker Hub using the following command: # docker pull registry:2 Once the registry image downloaded, you will need to generate a self-signed certificate for securing Docker Registry. Because, Docker node uses a secure connection over TLS to upload or download images to or from the private registry. Go to the registry-server and run the following command to generate certificate: # mkdir /etc/certs

https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md Pre-requisites: ############### >> K8s cluster setup with 1 Master and 2 Worker nodes. >> Deployed an application with Deployment name "client-sb" >> Also you need to create an HA-proxy server by spinning an Ec2 instance. After login the Ha-proxy server. # yum install haproxy # vi /etc/haproxy/haproxy.cfg delete everything after the global and defaults starting from "Main front-end which proxys to the backend" paste the below code in the end of the file: --------------------- frontend http_front   bind *:80   stats uri /haproxy?stats   default_backend http_back backend http_back   balance roundrobin   server kube 10.0.1.14:80   server kube 10.0.1.12:80 --------------------- # systemctl status haproxy # systemctl enable haproxy # systemctl start haproxy 1. Create a Namespace, a SA, the Default Secret, the Customization Config Map, an

nfs - server (10.0.1.9) ############ Amazon machine linux # yum update # yum install nfs-utils -y # systemctl enable nfs-server # systemctl start nfs-server # mkdir -p /srv/nfs/k8sdata # chmod -R 777 /srv/nfs/k8sdata # vi /etc/exports /srv/nfs/k8sdata *(rw,no_subtree_check,no_root_squash,insecure) :wq! # exportfs -rav # exportfs -v /srv/nfs/k8sdata (rw,sync,wdelay,hide,no_subtree_check,sec=sys,insecure,no_root_squash,no_all_squash) Now in NFS- Client: ################# Ubuntu 16.04 # sudo apt-get update # sudo apt-get install nfs-common # showmount -e 10.0.1.9 Export list for 10.0.1.9: /srv/nfs/k8sdata * Testing ------- # sudo mount -t nfs 10.0.1.9:/srv/nfs/k8sdata /mnt # root@dn1:~# df -h | grep nfs 10.0.1.9:/srv/nfs/k8sdata  8.0G  1.8G  6.3G  22% /mnt # umount /mnt Now in the kubectl terminal issue the pv and pvc create with yamls cat > 4-pv-nfs.yaml apiVersion: v1 kind: PersistentVolume metadata:   name: pv-nfs-pv1   labels:

If you try to install certbot you will get a message like below in Amazon machine image EC2 instance. ------------------------------------------------------------ Sorry, I don't know how to bootstrap Certbot on your operating system! You will need to install OS dependencies, configure virtualenv, and run pip install manually. Please see https://letsencrypt.readthedocs.org/en/latest/contributing.html#prerequisites ------------------------------------------------------------ Fix: ###  Amazon Linux 2 doesn't have  epel-release  in its repositories, but I've found you can install the EPEL RPM package itself, and then you'll be able to install  certbot  or  certbot-nginx  from there. Download the RPM curl -O http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm Then install it sudo yum install epel-release-latest-7.noarch.rpm Now you can install certbot sudo yum install certbot And then run it as usual sudo certbot Now give t