Nginx Logs to AWS CloudWatch: A Complete How-To Guide

Introduction:

When it comes to managing your website or application, understanding server logs is crucial. These logs contain valuable information about visitor activity, errors, and performance metrics. If you’re using Nginx as your web server, integrating its logs with AWS CloudWatch can streamline monitoring and analysis processes, providing you with actionable insights in real time.

In this guide, we’ll walk you through the steps to seamlessly transfer your Nginx logs to AWS CloudWatch. By the end, you’ll have a clear understanding of how to set up this integration and leverage CloudWatch’s powerful features for effective log management.

Prerequisites:

Basic understanding of AWS services and nginx.

Step 1: Configure AWS IAM permissions.

Create an IAM role and attach it CloudWatchFullAccessV2 policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchFullAccessPermissions",
            "Effect": "Allow",
            "Action": [
                "application-autoscaling:DescribeScalingPolicies",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribePolicies",
                "cloudwatch:*",
                "logs:*",
                "sns:CreateTopic",
                "sns:ListSubscriptions",
                "sns:ListSubscriptionsByTopic",
                "sns:ListTopics",
                "sns:Subscribe",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRole",
                "oam:ListSinks",
                "rum:*",
                "synthetics:*",
                "xray:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CloudWatchApplicationSignalsServiceLinkedRolePermissions",
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/application-signals.cloudwatch.amazonaws.com/AWSServiceRoleForCloudWatchApplicationSignals",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "application-signals.cloudwatch.amazonaws.com"
                }
            }
        },
        {
            "Sid": "EventsServicePermissions",
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents*",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "events.amazonaws.com"
                }
            }
        },
        {
            "Sid": "OAMReadPermissions",
            "Effect": "Allow",
            "Action": [
                "oam:ListAttachedLinks"
            ],
            "Resource": "arn:aws:oam:*:*:sink/*"
        }
    ]
}

You can adjust the policy permissions based on your requirements.

Step 2: Assign the Created role to the Ec2 Instance where nginx is Installed.

Select the Ec2 instance to which you want to attach the IAM role, Navigate to the “Actions” dropdown menu at the top of the EC2 dashboard. Under the “Security” Section Choose “Modify IAM Role”.

Choose the role that you had created in step 1 and click on Update IAM role.

Step 3: Look for the logs file location in the nginx conf file

cat /etc/nginx/nginx.conf

Step 4: Ensure that the log files have the necessary permissions.

sudo chmod o+r /var/log/nginx/access.log
sudo chmod o+r /var/log/nginx/error.log

Step 5: Download and install the CloudWatch agent package

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

sudo dpkg -i amazon-cloudwatch-agent.deb

Step 6: Configure the CloudWatch Agent

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Choose the appropriate answers from the prompts to configure the Amazon CloudWatch Agent.
Enter for default option.

Step 7: Install Collectd monitoring daemon service

sudo apt-get install collectd
sudo service collectd enable
sudo service collectd start
sudo systemctl status collectd.service

Step 8: Fetch the latest configuration and apply it to the current environment.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

Step 9: Start and verify CloudWatch Agent

Start the CloudWatch Agent service:

sudo systemctl start amazon-cloudwatch-agent

Verify that the agent is running without errors:

sudo systemctl status amazon-cloudwatch-agent

Step 10: Create CloudWatch Dashboard

Navigate to the CloudWatch service on your AWS console.
In the CloudWatch dashboard, navigate to the left-hand menu and click on “Logs groups” under the “Logs” section.

You should now be in the “Log groups” section, where you can view and manage your log groups.

You will find your Created log groups. Click on the LG that you want to view.

Click on the instance ID in the Log stream and you will be able to view your logs.