To learn more watch the following video:

The following list summarizes the prerequisites steps to be completed in your AWS account before you can create a Densify connection to collect CloudWatch utilization data.

• In each linked and payer account, create an AWS role. Optionally, use the same name and external ID. You must enter the following Densify account that will assume the IAM role:

036437403198

This is the AWS account that becomes the trusted entity.

• Select and assign AWS's predefined, ReadOnlyAccess policy. This policy is an easy option for non-sensitive accounts. In most cases, you will need to create a permission policy to grant minimum access.
• Once the AWS role has been created, copy the ARN and the External ID to create the Densify connection.
• Define your resource tags. The value of the resource tag can be used for filtering within Densify. You must tag your resources appropriately and then map your AWS Resource tags to Densify attributes so the tags will be included in the analyses. Contact [email protected] for details. See the video: Tagging for Cloud Optimization.

When collecting billing data you must also complete the following additional steps:

• Configuring or selecting a Cost and Usage Report;
• Granting the IAM Role Access to the S3 Bucket

Creating Connections from the Payer Account

In addition to creating connections from your linked accounts, you also need to create a Resource Utilization Metrics (CloudWatch) connection for your payer account. The payer account connection provides additional data when working with reserved instances and savings plans.

Working with Reserved Instances

When working with Reserved Instances you must collect both CloudWatch and billing data from all accounts in which you are using Reserved Instances. Since RIs can be shared between linked accounts they are often purchased through the payer account, so in order to see how the RI's are utilized, you need to collect both CloudWatch and billing data from the payer account.

To learn more watch the video:

Creating the IAM Role and Attaching a Permission Policy to Collect CloudWatch Data

This role allows you to collect CloudWatch data for the selected account. You need to attach a policy that allows the role to collect the required CloudWatch resource utilization metrics.

1. Log into the AWS Management Console and navigate to Services > Security, Identity & Compliance > IAM. In the navigation tree on the left, click Roles.
2. Click Create Role in the Roles dashboard.
3. Select the Another AWS account type of trusted entity.
4. Enter an Account ID. This is the Densify account that will assume the role. Enter the following Densify account ID: 036437403198.
5. Select Require external ID and enter your external ID. This value is similar to a password and should be unique and difficult to guess. Densify recommends using a password generator to create a random, alphanumeric string (e.g. ae91ccf4) for the external ID.

You will need this external ID later when creating the cloud connection from Densify.

6. Click Next: Permissions.
Figure: Creating a Role



7. Attach the appropriate permission policy to the role. Select AWS's predefined ReadOnlyAccess policy. Use the filter to find the ReadOnlyAccess policy.

Note:  The ReadOnlyAccess policy is provided here as an easy option for non-sensitive accounts. In most cases, you will need to create a custom permission policy to grant Densify, the minimum permissions to collect only the required CloudWatch data . Refer to Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection for details.

Figure: Selecting the ReadOnlyAccess Policy



For an IAM role that will be collecting both CloudWatch and billing data, you need to attach 2 policies:

• ReadOnlyAccess or a customized minimum requirement policy that allows this role to collect CloudWatch data.
• A policy granting the role access to the S3 bucket containing the cost and usage report. See Granting the IAM Role Access to the S3 Bucket for details on creating a custom policy for accessing billing data. This policy is in addition to the ReadOnlyAccess policy listed above.
8. After selecting the permission policies for the role, click Next: Review.
9. In the Review page, specify the Role name* and Role description. The role name can be any string used to identify and describe the role within the AWS account (e.g. DensifyCrossAccountRole).
10. Click Create role. The new role is created.
11. From the Roles page, click on the role name that you have just created, to view the role summary.
Figure: Role Summary



12. Copy and save the Role ARN as you will need to paste this string into the Densify Cloud Connection wizard to create the connection.
13. Continue with one of the following options:
• If you are collecting only the resource utilization metrics (CloudWatch), then your AWS configuration is complete. You can now create an AWS connection through the Densify Public Cloud Connection wizard. See Using the Public Cloud Connections Wizard.
• If you are collecting billing data you now need to create a cost and usage report and an S3 bucket. See Granting the IAM Role Access to the S3 Bucket.

To learn more watch the video:

Granting the IAM Role Access to the S3 Bucket

For AWS billing connections, you need to configure an IAM role in the payer account with a policy that grants the IAM role access to the S3 bucket to which the billing report is being stored. This policy is in addition to the policy appended to the bucket, enabling the CUR to be added to the bucket.

When working with savings plans you need to add an additional permission to the policy to enable Densify to collect the associated billing data:

• ce:GetSavingsPlansUtilizationDetails

The following steps assume that have created the role and are on the Summary page.

1. In the navigation tree on the left, click Policies
2. Click Create policy. The Create Policy page opens.
3. Click the JSON tab and enter the permission policy. You can copy this sample into the JSON tab and replace the highlighted string with your S3 bucket name.

Note:  If you are using this sample to create your policy, you must provide the name of your specific bucket, as the resource. In the example below it is indicated as, "billing-report-for-mycompany".

The sample policy below contains sufficient permissions for collecting the cost and usage report from your S3 bucket. The additional Cost Explorer permission, is required for collecting savings plan cost data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                    "iam:GetUser",
                    "ce:GetSavingsPlansUtilizationDetails",
                    "cur:DescribeReportDefinitions",
                    "organizations:DescribeOrganization",
                    "organizations:ListAccounts"
            ],
            "Resource": [
                    "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action":[
                    "s3:Get*",
                    "s3:List*"
            ],
            "Resource":[
                    "arn:aws:s3:::billing-report-for-mycompany*"
            ]
        }
    ]
}

4. Click Review Policy.
5. Enter a name and description for the policy. i.e. "Grant_Role_Permission_to_Bucket"
6. Click Create policy. You will be returned to the Policies page.
Figure: Attaching Permission Policies



7. In the navigation tree on the left, click Roles and then click on your newly created role.
8. Click Attach policy.
9. Use the filter to find the policy you created above. i.e. "Grant_Role_Permission_to_Bucket".
10. Click Attach policy. You are returned to the Roles page.
11. Click on your role and validate there are now two polices attached.
Figure: Attaching Permission Policies



12. You can now create AWS billing connection through the Densify Public Cloud Connection wizard. See Using the Public Cloud Connections Wizard.

Optional Configuration

The following sections contain detailed instructions for optional configuration. Some of this configuration is referenced in the procedures above.

• Resource Tagging
• Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection
• Creating a Cost and Usage Report for Billing Data Collection
• Enabling Collection of Memory Usage Metrics

Resource Tagging

You must define your resource tags for the grouping and filtering options to be used effectively. The value of the resource tag can be used for grouping and filtering your AWS instances within Densify. See the following video: Tagging for Cloud Optimization

You must tag your resources appropriately and then map your AWS Resource tags to Densify attributes so the tags will be included in the analyses. Contact [email protected] for details.

When collecting billing data you must enable the collection of Resource IDs when configuring the cost and usage report. See Configuring a Cost and Usage Report.

Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection

To simplify setup and maintenance of either an IAM user account or an IAM role for performing the CloudWatch audit, Densify recommends attaching the AWS-managed “ReadOnlyAccess” policy to the user or role. This policy provides read-only access to your AWS services and resources and supports the requirements of the Densify CloudWatch audit. As the Densify CloudWatch audit continues to evolve and expand, you do not need to update permission policy to include newly added services and features.

Alternatively, if you must restrict the IAM user or role with the minimum permissions to perform the CloudWatch audit, you can create a custom policy with only the required permissions, as shown below.

Note:  This custom policy must be updated periodically as Densify’s standard audit requirements are updated to support additional AWS services and features.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
{
            "Version": "2012-10-17",    
            "Statement": [
                    {
                        "Sid": "Stmt1499171905000",
                        "Effect": "Allow",    
                        "Action": [
                            "autoscaling:DescribeAutoScalingGroups",
                            "autoscaling:DescribeLaunchConfigurations",
                            "cloudformation:DescribeStackResource",
                            "cloudformation:DescribeStackResources",
                            "cloudformation:ListStackInstances",
                            "cloudformation:ListStackResources",
                            "cloudwatch:GetMetricData",
                            "cloudwatch:GetMetricStatistics",    
                            "cloudwatch:ListMetrics",
                            "ec2:DescribeHosts",
                            "ec2:DescribeImages",
                            "ec2:DescribeInstances",
                            "ec2:DescribeLaunchTemplateVersions",
                            "ec2:DescribeRegions",
                            "ec2:DescribeSnapshots",
                            "ec2:DescribeVolumes",
                            "ec2:DescribeSubnets",
                            "ec2:DescribeSecurityGroupRules",
                            "ec2:DescribeSecurityGroups",                    
                            "ec2:DescribeVpcs",
                            "ecs:DescribeCapacityProviders",
                            "ecs:DescribeClusters",    
                            "ecs:DescribeContainerInstances",    
                            "ecs:DescribeServices",    
                            "ecs:DescribeTaskDefinition",    
                            "ecs:ListClusters",    
                            "ecs:ListContainerInstances",    
                            "ecs:ListServices",
                            "ecs:ListTagsForResource",    
                            "ecs:ListTaskDefinitions",    
                            "eks:DescribeCluster",
                            "eks:ListClusters",    
                            "elasticache:DescribeCacheClusters",
                            "elasticache:DescribeReplicationGroups",
                            "elasticache:ListTagsForResource",
                            "iam:ListAccountAliases",
                            "organizations:DescribeOrganization",
                            "organizations:ListAccounts",
                            "rds:DescribeDBInstances",    
                            "rds:DescribeReservedDBInstances",    
                            "rds:DescribeDBClusters",    
                            "rds:ListTagsForResource"
                            
                        ],
                        
                        "Resource": "*"
                     }
            ]
}

Note:  The permissions related to CloudFormation are used for linking ASGs with ECS clusters. If these permissions are not included and the data is not available, linking the ASGs to ECS clusters may be done based on existing container instances. If the permissions are missing, Densify may not link some ASGs to their ECS clusters,

1. Log into the AWS management console and navigate to Services > IAM.
2. Select Policies and click Create policy.
3. Click the JSON tab and enter the policy from the example above.
4. Review the policy and enter a policy name (e.g. DensifyMinimumReadAccess) and a description (e.g. Minimum permissions required for Densify standard audit).

Creating a Cost and Usage Report for Billing Data Collection

Use this procedure to create a cost and usage report. You can either reference an existing S3 bucket or create a new one.

To create an AWS Cost and Usage report:

1. Login to the AWS management console of the payer AWS account. Billing data must be collected from the payer account.
2. On the Billing & Cost Management Dashboard select Cost & Usage Reports from the sidebar.
3. Click Create report. You can only have 10 different cost and usage reports. If the Create button is not available (grayed out) then you will need to delete one of the existing reports.

Note:  Note: Consult your AWS administrator before deleting anything!

4. Enter a valid name and select the following content and then click Next.
• Include resource IDs—This setting is optional.
• Data refresh settings—Leave the default setting to automatically update the report with any changes to past reports.
5. Define the Delivery options for this report:
• Configure S3 Bucket—See Step 6 below.
• Report path prefix—This setting is optional. The default prefix is the name that you specify for the report and the date range for the report, in the following format: /report-name/date-range/.
• Time granularity—Select "Daily".
• Report versioning—Select "Overwrite existing report", to minimize the size of the bucket. Both options are supported.
• Enable report data integration for—This is optional. Do not select Amazon QuickSight or Athena, as these formats are not supported. RedShift is a data warehouse. Densify still only picks up the .CSV from the bucket.
• Compression type—This is optional. Do not select Parquet.
Figure: Creating the S3 Bucket



6. Click Configure to define an S3 bucket for your report:
• If you have a valid S3 bucket, then select an existing bucket from the list. Review and verify the policy. Click Next to return to the Delivery Options page.
• If you need to create the bucket then enter a bucket name. Review the note at the bottom of the dialog box.
1. Select another region, if necessary and then click Next.
2. Review and verify the policy. Click Next to return to the Delivery Options page.
Figure: Viewing Sample Bucket Policy



7. On the Delivery options page, your S3 bucket will be listed and verified. Click Next.
8. Review you settings and then click Review and Complete.

Enabling Collection of Memory Usage Metrics

Memory metrics are not collected by default, and they are not required to complete the Densify analyses. You can manually enable collection of memory and disk metrics.

Note:  The CloudWatch Agent must be installed and configured on each instance for which you want to obtain memory and/or disk metrics. Refer to the AWS user documentation for details. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

Once the CloudWatch Agent is installed and configured, Densify uses the default, CWAgent as the namespace for metrics collected by the CloudWatch agent.

Additionally, when working with ASGs, ASG EC2 members will have memory utilization data using the basic memory settings, but you need to specify "aggregation_dimensions" to collect memory, aggregated at the ASG level.

Use the following information to configure the CloudWatch Agent (CWagent) via config.json to collect the metrics that Densify can use for analyses. Instructions are provided for both Linux and Windows instances.

If you are using a third-party application to collect memory metrics, the collected data can be loaded using the Receive Metrics API endpoint. See Importing Metrics for Existing Services in the API documentation.

Refer to the AWS user documentation for details on using the CloudWatch Agent to collect memory metrics.