<section class="blog-post-content"><p><em>This blog post was co-written by Linh Lam, Solution Architect, Kubecost</em></p><p>Customers can track their <a href="https://kubernetes.io/">Kubernetes</a> control plane and <a href="https://aws.amazon.com/ec2/">Amazon Elastic Compute Cloud (Amazon EC2)</a> costs using <a href="https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html">AWS Cost and Usage Reports</a>. However, they often need deeper insights to accurately track Kubernetes costs across namespaces, clusters, pods, and more. We recently announced that <a href="https://aws.amazon.com/blogs/containers/aws-and-kubecost-collaborate-to-deliver-cost-monitoring-for-eks-customers/">AWS and Kubecost collaborated to deliver cost monitoring for EKS customers</a>. Today, in partnership with <a href="https://www.kubecost.com/">Kubecost</a>, we are launching a streamlined integration between Kubecost and <a href="https://aws.amazon.com/prometheus/">Amazon Managed Service for Prometheus</a>. This approach provides cluster operators with <a href="https://aws.amazon.com/eks/">Amazon Elastic Kubernetes Service (Amazon EKS)</a> cost insights powered by Kubecost for a single Amazon EKS cluster and backed by a scalable Amazon Managed Service for Prometheus workspace.</p><p>Amazon Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that makes it easy to monitor containerized applications and infrastructure at scale. You can use the open-source Prometheus query language to monitor and alert for the performance of containerized workloads without having to worry about scaling the underlying monitoring infrastructure. The service automatically scales the ingestion, storage, alerting, and querying of operational metrics as workloads grow or shrink. Furthermore, it’s integrated with AWS security services to enable fast and secure access to data. This lets you concentrate on your workloads instead of having to manage your monitoring stack.</p><p>Kubecost is built on <a href="https://www.opencost.io/">OpenCost</a>, which was recently accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project, and is actively supported by AWS. Kubecost provides fine-grained visibility into your cluster, letting you break down costs by Kubernetes resources, such as pods, nodes, namespaces, and labels. This costs-visibility allows teams to have transparent and accurate cost data based on their actual AWS bill.</p><p>Now let’s configure Kubecost to be backed by an Amazon Managed Service for Prometheus workspace.</p><h2>Prerequisites</h2><p>To get started with this post, you should have the following:</p><h2>Installing Kubecost</h2><p>Begin by installing Kubecost on the Amazon EKS cluster by running the following command:</p><pre class="lang-bash">helm upgrade -i kubecost \oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> --namespace kubecost --create-namespace -f https://tinyurl.com/kubecost-amazon-eks</pre><p>Where <code><VERSION></code> is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.97.0. You can find all available versions of the EKS optimized Kubecost bundle <a href="https://gallery.ecr.aws/kubecost/cost-analyzer">here</a>. We recommend finding and installing the latest available Kubecost cost analyzer chart version.</p><p>Next, you must set up <a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">IAM roles for Kubecost service accounts (IRSA)</a>. Using the OIDC provider for the cluster, you grant IAM permissions to your cluster’s service accounts. You must grant appropriate permissions to the <code>kubecost-cost-analyzer</code> and <code>kubecost-prometheus-server</code> service accounts. These will be used to send and retrieve metrics from the workspace. Run the following commands on the command line:</p><pre class="lang-bash">eksctl create iamserviceaccount \ --name kubecost-cost-analyzer \ --namespace kubecost \ --cluster <CLUSTER_NAME> --region <REGION> \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \ --override-existing-serviceaccounts \ --approveeksctl create iamserviceaccount \ --name kubecost-prometheus-server \ --namespace kubecost \ --cluster <CLUSTER_NAME> --region <REGION> \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \ --override-existing-serviceaccounts \ --approve</pre><p><code><CLUSTER_NAME></code> is the name of the Amazon EKS cluster where you want to install Kubecost and <code><REGION></code> is the region of the Amazon EKS cluster. Note that these commands each generate an <a href="https://aws.amazon.com/cloudformation/">AWS CloudFormation</a> stack that creates a new IAM role, assigns the <code>AmazonPrometheusQueryAccess</code> and <code>AmazonPrometheusRemoteWriteAccess</code> managed policies, and configures a trust relationship with the OIDC provider for the cluster. This allows the service account to assume the IAM role.</p><p>If you don’t yet have an Amazon Managed Service for Prometheus workspace, then the following <a href="https://aws.amazon.com/cli/">AWS Command Line Interface (AWS CLI)</a> command will create a workspace called <code>kubecost-amp</code> in the current region. You may skip this step if you have a pre-existing workspace.</p><pre class="lang-bash">aws amp create-workspace --alias kubecost-amp --region <REGION></pre><p><code><REGION></code> is the region where you want the workspace to be created. Now you can update the Kubecost configuration so that it uses your Amazon Managed Service for Prometheus workspace as the source for your cluster metrics. Run the following command from the command line:</p><pre class="lang-bash">export AWS_REGION=<REGION>export AMP_WORKSPACE_ID=<WORKSPACE-ID></pre><p>Where <code><WORKSPACE-ID></code> is the Workspace ID of the cluster and <code><REGION></code> is the current region of the workload. You can get the Workspace ID of the workspace from the Summary section of the Amazon Managed Service for Prometheus console. See the following figure.</p><div id="attachment_32956" class="wp-caption aligncenter c4"><img aria-describedby="caption-attachment-32956" class="wp-image-32956" src="https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_1.png" alt="Figure 1: The Amazon Managed Service for Prometheus workspace detail, which lists the workspaces ARN, Workspace ID, Endpoint – remote write URL, and the Endpoint – query URL" width="700" height="517" /><p id="caption-attachment-32956" class="wp-caption-text">Figure 1: The Amazon Managed Service for Prometheus workspace detail, which lists the workspaces ARN, Workspace ID, Endpoint – remote write URL, and the Endpoint – query URL</p></div><p>Run the following command to create a file called <code>config-values.yaml</code>, which contains the defaults that Kubecost will use for connecting to your Amazon Managed Service for Prometheus workspace.</p><pre class="lang-bash">cat << EOF > config-values.yamlglobal: amp: enabled: true prometheusServerEndpoint: http://localhost:8005/workspaces/${AMP_WORKSPACE_ID} remoteWriteService: https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${AMP_WORKSPACE_ID}/api/v1/remote_write sigv4: region: ${AWS_REGION}sigV4Proxy: region: ${AWS_REGION} host: aps-workspaces.${AWS_REGION}.amazonaws.comEOF</pre><p>Now, run the following command to configure Kubecost to begin using your workspace:</p><pre class="lang-bash">helm upgrade -i kubecost \oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> --namespace kubecost --create-namespace -f https://tinyurl.com/kubecost-amazon-eks -f config-values.yaml</pre><p><code><VERSION></code> is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.97.0.</p><p>Finally, restart the Prometheus deployment, which reloads the service account configuration.</p><pre class="lang-bash">kubectl rollout restart deployment/kubecost-prometheus-server -n kubecost</pre><p>After a few minutes, your Kubecost cluster should be available and ready to view.</p><p>You can enable port-forwarding to expose the Kubecost dashboard:</p><pre class="lang-bash">kubectl port-forward deployment.apps/kubecost-cost-analyzer 9090:9090 -n kubecost</pre><p>The dashboard will be available to view <a href="http://localhost:9090/">here</a>.</p><div id="attachment_32957" class="wp-caption aligncenter c4"><img aria-describedby="caption-attachment-32957" class="wp-image-32957" src="https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_3.png" alt=": The Kubecost dashboard, which shows monthly savings of $1,058.56, monthly Kubernetes costs of $1,627.16, and a 3.9% cost efficiency. The dashboard shows how costs are allocated across various Kubernetes resources.]" width="700" height="517" /><p id="caption-attachment-32957" class="wp-caption-text">Figure 2: The Kubecost dashboard, backed by an Amazon Managed Service for Prometheus workspace</p></div><h2>Conclusion</h2><p>We’re excited that this release allows customers to store and analyze their Kubecost metrics using Amazon Managed Service for Prometheus as a backend. This release helps customers better support cost monitoring for their Amazon EKS workloads without having to deal with managing the monitoring stack.</p><p>Using Kubecost in your Amazon EKS workloads lets you better monitor costs associated with containerized workloads. Your Amazon EKS cost insights are powered by Kubecost and backed by a scalable Amazon Managed Service for Prometheus workspace.</p><p>If you need support, you can submit a support request via <a href="https://aws.amazon.com/contact-us/">AWS Support</a>.</p><p>If you would like to learn more from the Kubecost team, contact them <a href="https://www.kubecost.com/aws-eks-cost-monitoring/">here</a>.</p><p>Get started today by <a href="https://docs.aws.amazon.com/prometheus/latest/userguide/integrating-kubecost.html">integrating your workload with Amazon EKS cost monitoring</a>.</p><p><strong>About the authors:</strong></p></section>
![https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_1.png"](https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_1.png"){kind=link}
![https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_3.png"](https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_3.png"){kind=link}