Skip to main content

Prometheus Alertmanager integration by routing to slack using helm

Setting up Alertmanager and Rules
---------------------------------


Now that you have prometheus set up, you need to specify some instructions. The next step is to create a values.yaml file that specifies

1) what the alert rules are,
2) what the Prometheus targets are (i.e the definition of what to scrape and how) and any jobs for Prometheus, and
3) where alerts should be routed to (in this case, Slack).

Alert Rules
------------

vi prometheus.values

## Prometheus server ConfigMap entries
##
serverFiles:

  ## Alerts configuration
  ## Ref: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
  alerts:
    groups:
      - name: Instances
        rules:
          - alert: InstanceDown
            expr: up == 0
            for: 5m
            labels:
              severity: page
            annotations:
              description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
              summary: 'Instance {{ $labels.instance }} down'
             
             
Prometheus Targets
------------------

Next, you will set up the Prometheus targets and specify the jobs. Fortunately, Kube scraping is already set up out of the box, so for our purposes, no additional action is required for this step


Alert Routing
-------------

Here I am routing the alerts to slack api.

api_url / WebHOOK_URL : https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxx/xxxxxxxxxxxxx

you can test if the webhook url is working or not by sending a message to slack using the below curl command.

For getting your webhook url take the below url after login to your slack:

https://curai.slack.com/apps/A0F7XDUAZ-incoming-webhooks?next_id=0

curl -X POST --data-urlencode "payload={\"channel\": \"#MYCHANNELNAME\", \"username\": \"sanitybot\", \"text\": \"Just a sanity check that slack webhook is working.\", \"icon_emoji\": \":ghost:\"}" MY_WEBHOOK_URL

curl -X POST --data-urlencode "payload={\"channel\": \"#devops\", \"jino\": \"webhookbot\", \"text\": \"This is posted to #devops and comes from a bot named webhookbot.\", \"icon_emoji\": \":ghost:\"}" https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxx/xxxxxxxxxxxxx



vi prometheus.values

## alertmanager ConfigMap entries
##
alertmanagerFiles:
  alertmanager.yml:
    global: {}
      # slack_api_url: ''

    receivers:
      - name: default-receiver
        slack_configs:
         - channel: "#devops"
           send_resolved: true
           api_url: 'https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxx/xxxxxxxxxxxxx'
           text: "description: {{ .CommonAnnotations.description }}\nsummary: {{ .CommonAnnotations.summary }}"
    route:
      group_by: [cluster]
      receiver: default-receiver
      routes:
        - match:
            severity: critical
          receiver: default-receiver
          repeat_interval: 1m
          group_wait: 10s
          group_interval: 5m

Once your values.yaml file is prepared, you’re ready to upgrade.

# helm upgrade -f prometheus.values prometheus stable/prometheus

Now check your prometheus alert url :

https://monitoring.abtest.tk/alerts

You can also confirm that the settings that you given are correct using the below command:


# kubectl describe configmap prometheus-alertmanager -n prometheus
Name:         prometheus-alertmanager
Namespace:    prometheus
Labels:       app=prometheus
              chart=prometheus-9.3.1
              component=alertmanager
              heritage=Tiller
              release=prometheus
Annotations: 

Data
====
alertmanager.yml:
----
global: {}
receivers:
- name: default-receiver
  slack_configs:
  - api_url: https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxx/xxxxxxxxxxxxx
    channel: '#devops'
    send_resolved: true
    text: |-
      description: {{ .CommonAnnotations.description }}
      summary: {{ .CommonAnnotations.summary }}
route:
  group_by:
  - cluster
  group_interval: 5m
  group_wait: 10s
  receiver: default-receiver
  repeat_interval: 3h
  routes:
  - group_interval: 5m
    group_wait: 10s
    match:
      severity: critical
    receiver: default-receiver
    repeat_interval: 1m

Events: 


That is all!!!! cheers :-)

Comments

Post a Comment

Popular posts from this blog

K8s External Secrets integration between AWS EKS and Secrets Manager(SM) using IAM Role.

What is K8s External Secrets and how it will make your life easier? Before saying about External Secrets we will say about k8s secrets and how it will work. In k8s secrets we will create key value pairs of the secrets and set this as either pod env variables or mount them as volumes to pods. For more details about k8s secrets you can check my blog http://jinojoseph.blogspot.com/2020/08/k8s-secrets-explained.html   So in this case if developers wants to change the ENV variables , then we have to edit the k8s manifest yaml file, then we have to apply the new files to the deployment. This is a tiresome process and also chances of applying to the wrong context is high if you have multiple k8s clusters for dev / stage and Prod deployments. So in-order to make this easy , we can add all the secrets that is needed in the deployment, in the AWS Secret Manager and with the help of External secrets we can fetch and create those secrets in the k8s cluster. So what is K8s external Secret? It i...
Post a Comment
Read more

Password reset too simplistic/systematic issue

Some time when we try to reset the password of our user in linux it will show as simple and systematic as below: BAD PASSWORD: it is too simplistic/systematic no matter how hard password you give it will show the same. Solution: ######### Check if your password is Ok with the below command, jino@ndz~$ echo 'D7y8HK#56r89lj&8*&^%&^%#56rlKJ!789l' | cracklib-check D7y8HK#56r89lj&8*&^%&^%#56rlKJ!789l: it is too simplistic/systematic Now Create a password with the below command : jino@ndz~$ echo $(tr -dc '[:graph:]' 7\xi%!W[y*S}g-H7W~gbEB4cv,9:E:K; You can see that this password will be ok with the cracklib-check. jino@ndz~$ echo '7\xi%!W[y*S}g-H7W~gbEB4cv,9:E:K;' | cracklib-check                 7\xi%!W[y*S}g-H7W~gbEB4cv,9:E:K;: OK Thats all, Thanks.
Post a Comment
Read more

Setting /etc/hosts entries during the initial deployment of an Application using k8s yaml file

Some times we have to enter specific hosts file entries to the container running inside the POD of a kubernetes deployment during the initial deployment stage itself. If these entries are not in place, the application env variables mentioned in the yaml file , as hostnames , will not resolve to the IP address and the application will not start properly. So to make sure the /etc/hosts file entries are already there after the spin up of the POD you can add the below entries in your yaml file. cat > api-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: spec:   template:     metadata:     spec:       volumes:       containers:       - image: registryserver.jinojoseph.com:5000/jinojosephimage:v1.13         lifecycle:           postStart:             exec:               command:...
Post a Comment
Read more