#90DaysOfDevOps Challenge - Day 75 - Monitor Docker Containers with Grafana

#90DaysOfDevOps Challenge - Day 75 - Monitor Docker Containers with Grafana

Welcome back to the #90DaysOfDevOps Challenge. On Day 75, we'll explore how to send Docker logs to Grafana for real-time monitoring and analysis. By integrating Docker containers with Grafana, you can gain valuable insights into your containerized applications and ensure their smooth operation. Let's get started!

Monitoring Container Performance with cAdvisor and Prometheus

cAdvisor (Container Advisor) is an open-source tool designed for monitoring and analyzing container performance at the individual container level. It provides valuable insights into resource usage, performance metrics, and health statistics of containers running on a host or within a container orchestration platform like Kubernetes. cAdvisor enables DevOps engineers and system administrators to efficiently track the behaviour of containers, troubleshoot issues, and optimize resource allocation for better overall system performance.

Key Features of cAdvisor:

  1. Container-Level Metrics: cAdvisor collects various metrics at the container level, including CPU usage, memory consumption, file system usage, network statistics, and more.

  2. Real-Time Monitoring: It provides real-time monitoring of containers, allowing users to observe changes in resource utilization and performance metrics as containers operate.

  3. Resource Utilization Analysis: cAdvisor offers comprehensive insights into the resource utilization of containers, helping identify bottlenecks and inefficiencies.

  4. Container Health Analysis: It provides health checks and analysis for containers, indicating the overall health status of containers.

Integrating cAdvisor with Prometheus enhances container monitoring capabilities, providing valuable insights into container performance and resource utilization. By leveraging the powerful features of cAdvisor and Prometheus, DevOps teams can optimize containerized applications, troubleshoot performance issues, and ensure the overall health and efficiency of their container infrastructure.

Task: Sending Docker Logs to Grafana

Step 1: Install Docker and Start Docker Service on Ubuntu EC2 through USER Data

  1. We'll use the below script through USER DATA to install Docker and start and enable the service.

     sudo apt update
     sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
     curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
     echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
     sudo apt update
     sudo apt install -y docker-ce docker-ce-cli containerd.io
     sudo systemctl start docker
     sudo systemctl enable docker
     sudo usermod -aG docker $USER
     sudo reboot

Step 2: Create a docker-compose.yml file with all the required containers

  1. Connect to your Ubuntu EC2 instance

  2. I've chosen to create two containers. Code-Server, which is a containerized version of VS Code and Jenkins. The docker compose file also includes Grafana, Prometheus and cAdvisor. Create a docker-compose.yml file with the following content.

     version: "3"
         driver: local
         driver: local
       # Grafana service
         image: grafana/grafana-oss:latest
         container_name: grafana
           - "3000:3000"
           - grafana-data:/var/lib/grafana
         restart: unless-stopped
       # Prometheus service
         image: prom/prometheus:latest
         container_name: prometheus
           - "9090:9090"
           - /etc/prometheus:/etc/prometheus
           - prometheus-data:/prometheus
         command: "--config.file=/etc/prometheus/prometheus.yml"
         restart: unless-stopped
       # Code-Server service
         image: lscr.io/linuxserver/code-server:latest
         container_name: code-server
           - PUID=1000
           - PGID=1000
           - TZ=Etc/UTC
           - /etc/opt/docker/code-server/config:/config
           - 8443:8443
         restart: unless-stopped
       # Jenkins service
         image: jenkins/jenkins:lts
         container_name: jenkins
         privileged: true
         user: root
           - /etc/opt/docker/jenkins/config:/var/jenkins_home
           - /var/run/docker.sock:/var/run/docker.sock
           - /usr/local/bin/docker:/usr/local/bin/docker
           - 8081:8080
           - 50000:50000
         restart: unless-stopped
       # cAdvisor service
         image: gcr.io/cadvisor/cadvisor:v0.47.0
         container_name: cadvisor
           - 8080:8080
         network_mode: host
           - /:/rootfs:ro
           - /var/run:/var/run:ro
           - /sys:/sys:ro
           - /var/lib/docker/:/var/lib/docker:ro
           - /dev/disk/:/dev/disk:ro
           - /dev/kmsg
         privileged: true
         restart: unless-stopped
  3. Start the Docker containers by running docker compose up -d.

Step 3: Configure Prometheus to Scrap Metrics

  1. Ensure your Ubuntu EC2 instance is connected to Grafana with the Prometheus data source configured as described in the previous article (#90DaysOfDevOps - Day 74: Connecting EC2 with Grafana).

  2. Amend the prometheus.yml file as per the below:

       scrape_interval:     15s # By default, scrape targets every 15 seconds.
       # Attach these labels to any time series or alerts when communicating with
       # external systems (federation, remote storage, Alertmanager).
       # external_labels:
       #  monitor: 'codelab-monitor'
     # A scrape configuration containing exactly one endpoint to scrape:
     # Here it's Prometheus itself.
       # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
       - job_name: 'prometheus'
         # Override the global default and scrape targets from this job every 5 seconds.
         scrape_interval: 5s
           - targets: ['']
       # Example job for node_exporter
     #  - job_name: 'node_exporter'
     #    static_configs:
     #      - targets: ['', '']
       # Example job for cadvisor
       - job_name: 'cadvisor'
           - targets: ['']
  3. Restart Prometheus and access its dashboard to check if it can connect to the target nodes

  4. Access your Grafana dashboard using the public IP address or DNS name of your EC2 instance and port 3000.

  5. Select "Dashboards", click on the plus sign at the top of the menu and hit "import dashboard"

  6. Go to the Grafana Dashboards Library and search for cAdvisor. Choose your preferred dashboard and copy the ID Number. In my case, I'll use the Cadvisor exporter dashboard.

  7. Paste the above ID number in the Grafana "import dashboard" and hit load to create the chosen dashboard in our Grafana app. Select your Prometheus Server and select import.

Step 4: Check the Logs or Docker Container Name on Grafana UI

  1. Go to your Grafana dashboard and check the real-time logs from your Docker containers.

Congratulations! You've successfully sent Docker logs to Grafana, allowing you to monitor your containerized applications in real time. By integrating Docker containers with Grafana, you can gain valuable insights into your application's performance and troubleshoot any issues effectively.

Stay tuned for Day 76 of the #90daysofdevops challenge, where we'll explore how to build a Grafana Dashboard.

Did you find this article valuable?

Support Esteban Moreno by becoming a sponsor. Any amount is appreciated!