Monitor services

X-ray

These are the points for exam

• Application Load balancers do not send data to X-Ray - Load balancers add a trace ID to incoming HTTP requests in a header named X-Amzn-Trace-Id. Load balancers do not send data to X-Ray and do not appear as a node on your service map.

Use X-ray to debug microservices specific issues

Imagine a company uses microservices-based infrastructure to process the API calls from clients, perform request filtering and cache requests using the AWS API Gateway. Users report receiving 501 error code and you have been contacted to find out what is failing.

You may use X-Ray to debug the issue.

• CloudWatch can collect numbers and respond to AWS service-related events, but it can't help you debug microservices specific issues on AWS.
• X-Ray cannot be used to capture metrics and set up alarms as per the given use-case, so this option is incorrect.

The AWS X-Ray SDK needs permission to run in resources (like Lambda)

Create an IAM role with write permissions and assign it to the resources running your application. You can use AWS Identity and Access Management (IAM) to grant X-Ray permissions to users and compute resources in your account. This should be one of the first places you start by checking that your permissions are properly configured before exploring other troubleshooting options.

Here is an example of X-Ray Read-Only permissions via an IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "xray:GetSamplingRules",
                "xray:GetSamplingTargets",
                "xray:GetSamplingStatisticSummaries",
                "xray:BatchGetTraces",
                "xray:GetServiceGraph",
                "xray:GetTraceGraph",
                "xray:GetTraceSummaries",
                "xray:GetGroups",
                "xray:GetGroup"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Another example of write permissions for using X-Ray via an IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords",
                "xray:GetSamplingRules",
                "xray:GetSamplingTargets",
                "xray:GetSamplingStatisticSummaries"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

CloudTail

Source: Field Notes: Analyze Cross-Account AWS KMS Call Usage with AWS CloudTrail and Amazon Athena

AWS CloudTrail is a service that records actions as events taken by a user, role, or an AWS service. Events include actions taken in the AWS Management Console, AWS CLI, and AWS SDKs and APIs. For an ongoing record of events in your AWS account, you create a trail. A trail enables CloudTrail to deliver log files of events to an Amazon S3 bucket.

Organization

If you have created an organization in AWS Organizations, you can also create a trail that will log all events for all AWS accounts in that organization (need root account permission). This is referred to as an organization trail.

• S3 related
  • By default, CloudTrail traczks only bucket-level actions. To track object-level actions, you need to enable Amazon S3 data events
  • A bucket owner enabled CloudTrail. It doesn’t mean he can see the object access logs. The bucket owner also needs to be object owner to get the object access logs. Otherwise, the bucket owner must get permissions, through the object ACL, for the same object API to get the same object-access API logs.
• Member accounts will be able to see the organization trail, but cannot modify or delete it. (By default, member accounts will not have access to the log files for the organization trail in the Amazon S3 bucket.)
• Organization trails must be created in the master account, and when specified as applying to an organization, are automatically applied to all member accounts in the organization.

Events

• Management Events: These events track management operations performed on AWS resources, such as creating, modifying, or deleting resources. Examples include EC2 instance launches, S3 bucket creations, IAM policy updates, etc.
• Data Events: These events capture data-related operations performed on AWS resources. Examples include S3 object access, AWS Lambda function invocations, and Amazon RDS database queries.
• CloudTrail Insights Events: CloudTrail Insights helps identify unusual or potentially harmful activity in your AWS account. It generates events for specific patterns identified as potentially high-risk actions.

Log file integrity validation(digest file)

TL;DR - Log file integrity validation is for auditing integrity of the CloudTrail log files by using digest file

Tips

You can use Amazon S3 MFA Delete on the S3 bucket that holds CloudTrail logs and digest files to secure the CloudTrail logs

When you enable log file integrity validation, CloudTrail creates a hash for every log file that it delivers. Every hour, CloudTrail also creates and delivers a file that references the log files for the last hour and contains a hash of each. This file is called a digest file. CloudTrail signs each digest file using the private key of a public and private key pair. After delivery, you can use the public key to validate the digest file. CloudTrail uses different key pairs for each AWS region.

These digest files are stored in the same Amazon S3 bucket as the log files but in a separate folder. This separation allows for granular security policies and ensures compatibility with existing log processing solutions. Each digest file also includes the digital signature of the previous digest file (if available) in its metadata properties.

Source: CloudTrail log file Integrity Validation

The image above illustrate - if we trust the latest digest file that we have, we can identify if any of the digests are missing or invalid, and this gives us a chain of digest file trust. If we trust the latest, then we trust them all.

IAM & STS integration

CloudTrail logs all authenticated API requests (made with credentials) to IAM and AWS STS API operations. CloudTrail also logs unauthenticated requests to the AWS STS actions, AssumeRoleWithSAML and AssumeRoleWithWebIdentity, and logs information provided by the identity provider.

• To track the original federated user who made the API call
  • You can use this information to map calls made by a federated user with an assumed role back to the originating external federated caller.
• To track the original user who used AssumeRole to made the API call
  • You can map calls back to the originating AWS service or to the account of the originating user. The userIdentity section of the JSON data in the CloudTrail log entry contains the information that you need to map the AssumeRole* request with a specific federated user.

Notes

• CloudTrail can deliver log files from multiple regions to a single Amazon S3 bucket. As long as CloudTrail has permissions to write to an S3 bucket, the bucket for a multi-Region trail does not have to be in the trail's home Region.
  • You can configure CloudTrail to deliver log files from multiple Regions to a single S3 bucket for a single account. For example, you have a trail in the US West (Oregon) Region that is configured to deliver log files to a S3 bucket, and a CloudWatch Logs log group. When you change an existing single-Region trail to log all Regions, CloudTrail logs events from all Regions that are in a single AWS partition in your account. CloudTrail delivers log files to the same S3 bucket and CloudWatch Logs log group. ref: Receiving CloudTrail log files from multiple Regions
• Digest files can only be encrypted with Amazon S3-managed encryption keys (SSE-S3).
• Log files is encrypted by SSE-S3 by default but you can change it to use SSE-KMS.

Amazon Inspector

TL;DR- It uses security assessments to identify security issues in the application layer, rather than the operating system or infrastructure layer.

Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. It detects software vulnerabilities and unintended network exposure in near real time in AWS workloads such as Amazon EC2, AWS Lambda functions, and Amazon ECR. You can create an assessment template and launch a security assessment run of this target. ref

Summery dashboard:

Assessment:

During the assessment run, the network, file system, and process activity within the specified target are monitored, and a wide set of activity and configuration data is collected. The collected data is correlated, analyzed, and compared to a set of security rules specified in the assessment template. A completed assessment run produces a list of findings - potential security problems of various severity.

Amazon Inspector vs AWS Trusted Advisor

Trusted Advisor provides advice about your AWS Account in the areas of Cost Optimization, Fault Tolerance, Performance, Service Limits, Security.

Amazon Inspector checks the configuration of EC2 instances. An agent runs on EC2 instances and checks operating system patches, known vulnerabilities, and common issues.

AWS Config

AWS config sequential flow:

AWS Config does this through the use of rules that define the desired configuration state of your AWS resources. AWS Config provides a number of AWS managed rules that address a wide range of security concerns such as..

• checking if you encrypted your Amazon EBS volumes
• tagged your resources appropriately
• enabled multi-factor authentication (MFA) for root accounts

IAM monitoring Example Another example is you can view the IAM policy that was assigned to an IAM user, group, or role at any time in which AWS Config was recording. This information can help you determine the permissions that belonged to a user at a specific time: for example, you can view whether the user Sai Tai had permission to modify Amazon VPC settings on Jan 1, 2023.

Config rule with SSM Automation document In order to detect any activity that exposes the application externally and restricts the action right away. You can create a custom AWS Config rule that continuously evaluates whether a security group allows inbound requests outside of the corporate CIDR range. Then, configure an AWS Systems Manager Automation document to automatically remove any noncorporate CIDR range from the security groups of the application.

Amazon Inspector vs AWS Config

Amazon Inspector is a security assessment service that helps you to identify potential security issues and compliance of applications deployed on AWS but it may not be the most efficient way to address the specific security concern.

For example, if you want to check a specific security concern whether a public SSH is always disabled for the private servers. It is more efficient to use AWS Config instead of Amazon Inspector

Config rule page:

Notification

AWS Config sends notifications for the following events:

• Configuration item change for a resource.
• Configuration history for a resource was delivered for your account.
• Configuration snapshot for recorded resources was started and delivered for your account.
• Compliance state of your resources and whether they are compliant with your rules.
• Evaluation started for a rule against your resources.
• AWS Config failed to deliver the notification to your account.

Triggers

When you add a rule to your account, you can specify when you want AWS Config to run the rule; this is called a trigger. AWS Config evaluates your resource configurations against the rule when the trigger occurs. There are two types of triggers:

• Configuration changes: AWS Config runs evaluations for the rule when certain types of resources are created, changed, or deleted.
• Periodic: AWS Config runs evaluations for the rule at a frequency that you choose (for example, every 24 hours).

You can use both types in one rule

If you choose both configuration changes and periodic to the same rule, AWS Config invokes your Lambda function when it detects a configuration change and also at the frequency that you specify.

Config Aggregator

TL;DR - It aggregates the config data from different AWS accounts in multiple AWS regions.

An aggregator is an AWS Config resource type that collects AWS Config configuration and compliance data from the following:

• Multiple accounts and multiple regions.
• Single account and multiple regions.
• An organization in AWS Organizations and all the accounts in that organization which have AWS Config enabled.

Use an aggregator to view the resource configuration and compliance data recorded in AWS Config.

In the navigation pane, choose Aggregators, and then review the configuration data of your AWS resources and compliance state of your rules.

Source: Set up an organization-wide aggregator in AWS Config using a delegated administrator account

Remediation

AWS Config uses

• AWS Lambda functions perform compliance evaluations, and
• System Manager automation documents perform auto-remediation to automatically correct non-compliant resources.

Some rules for S3

AWS Config includes auto-remediation for specific non-compliant S3 activities through rules like s3-bucket-logging-enabled and s3-bucket-server-side-encryption-enabled.

Source: Remediate noncompliant AWS Config rules with AWS Systems Manager Automation runbooks

Some practices

• Use EventBridge to detect NON_COMPLIANT in AWS Config and send message via SNS
  • You can use an EventBridge rule with a custom event pattern and an input transformer to match an AWS Config evaluation rule output as NON_COMPLIANT. Then, route the response to an Amazon SNS topic.
• AWS Config sends notifications only when the compliance status changes. If a resource was previously non-compliant and is still non-compliant, Config will not send a new notification.
• You can leverage the require-tags managed rule in AWS Config to check if a resource contains the tags that you specify.