Set up enterprise-level cost allocation for ML environments and workloads using resource tagging in Amazon SageMaker

<p>As businesses and IT leaders look to accelerate the adoption of machine learning (ML), there is a growing need to understand spend and cost allocation for your ML environment to meet enterprise requirements. Without proper cost management and governance, your ML spend may lead to surprises in your monthly AWS bill. <a href="https://aws.amazon.com/sagemaker/" target="_blank" rel="noopener noreferrer">Amazon SageMaker</a> is a fully managed ML platform in the cloud that equips our enterprise customers with tools and resources to establish cost allocation measures and improve visibility into detailed cost and usage by your teams, business units, products, and more.</p><p>In this post, we share tips and best practices regarding cost allocation for your SageMaker environment and workloads. Across almost all AWS services, SageMaker included, applying tags to resources is a standard way to track costs. These tags can help you track, report, and monitor your ML spend through out-the-box solutions like <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/" target="_blank" rel="noopener noreferrer">AWS Cost Explorer</a> and <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/" target="_blank" rel="noopener noreferrer">AWS Budgets</a>, as well as custom solutions built on the data from <a href="https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting/" target="_blank" rel="noopener noreferrer">AWS Cost and Usage Reports</a> (CURs).</p><h2>Cost allocation tagging</h2><p>Cost allocation on AWS is a three-step process:</p><ol><li>Attach <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html" target="_blank" rel="noopener noreferrer">cost allocation tags</a> to your resources.</li><li>Activate your tags in the <strong>Cost allocation tags</strong> section of the AWS Billing console.</li><li>Use the tags to track and filter for cost allocation reporting.</li></ol><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image001.jpg"><img class="alignnone size-full wp-image-42872 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image001.jpg" alt="" width="1288" height="315" /></a></p><p>After you create and attach tags to resources, they appear in the AWS Billing console’s <strong>Cost allocation tags</strong> section under <strong>User-defined cost allocation tags</strong>. It can take up to 24 hours for tags to appear after they’re created. You then need to activate these tags for AWS to start tracking them for your resources. Typically, after a tag is activated, it takes about 24–48 hours for the tags to show up in Cost Explorer. The easiest way to check if your tags are working is to look for your new tag in the tags filter in Cost Explorer. If it’s there, then you’re ready to use the tags for your cost allocation reporting. You can then choose to group your results by tag keys or filter by tag values, as shown in the following screenshot.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image003.png"><img class="alignnone size-full wp-image-42873 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image003.png" alt="" width="1523" height="449" /></a></p><p>One thing to note: if you use <a href="https://aws.amazon.com/organizations/" target="_blank" rel="noopener noreferrer">AWS Organizations</a> and have linked AWS accounts, tags can only be activated in the primary payer account. Optionally, you can also activate CURs for the AWS accounts that enable cost allocation reports as a CSV file with your usage and costs grouped by your active tags. This gives you more detailed tracking of your costs and makes it easier to set up your own custom reporting solutions.</p><h2>Tagging in SageMaker</h2><p>At a high level, tagging SageMaker resources can be grouped into two buckets:</p><ul><li>Tagging the SageMaker notebook environment, either <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/studio.html" target="_blank" rel="noopener noreferrer">Amazon SageMaker Studio</a> domains and domain users, or SageMaker notebook instances</li><li>Tagging SageMaker-managed jobs (labeling, processing, training, hyperparameter tuning, batch transform, and more) and resources (such as models, work teams, endpoint configurations, and endpoints)</li></ul><p>We cover these in more detail in this post and provide some solutions on how to apply governance control to ensure good tagging hygiene.</p><h3>Tagging SageMaker Studio domains and users</h3><p>Studio is a web-based, integrated development environment (IDE) for ML that lets you build, train, debug, deploy, and monitor your ML models. You can launch Studio notebooks quickly, and dynamically dial up or down the underlying compute resources without interrupting your work.</p><p>To automatically tag these dynamic resources, you need to assign tags to SageMaker domain and domain users who are provisioned access to those resources. You can specify these tags in the tags parameter of <a href="https://awscli.amazonaws.com/v2/documentation/api/2.0.33/reference/sagemaker/create-domain.html" target="_blank" rel="noopener noreferrer">create-domain</a> or <a href="https://docs.aws.amazon.com/cli/latest/reference/sagemaker/create-user-profile.html" target="_blank" rel="noopener noreferrer">create-user-profile</a> during profile or domain creation, or you can add them later using the <a href="https://docs.aws.amazon.com/cli/latest/reference/sagemaker/add-tags.html" target="_blank" rel="noopener noreferrer">add-tags</a> API. Studio automatically copies and assigns these tags to the Studio notebooks created in the domain or by the specific users. You can also add tags to SageMaker domains by editing the domain settings in the Studio Control Panel.</p><p>The following is an example of assigning tags to the profile during creation.</p><p>To tag existing domains and users, use the <code>add-tags</code> API. The tags are then applied to any new notebooks. To have these tags applied to your existing notebooks, you need to restart the Studio app (Kernel Gateway and Jupyter Server) belonging to that user profile. This won’t cause any loss in notebook data. Refer to this <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-update.html" target="_blank" rel="noopener noreferrer">Shut Down and Update SageMaker Studio and Studio Apps</a> to learn how to delete and restart your Studio apps.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image007.jpg"><img class="alignnone size-full wp-image-42875 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image007.jpg" alt="" width="1288" height="535" /></a></p><h3>Tagging SageMaker notebook instances</h3><p>In the case of a SageMaker notebook instance, tagging is applied to the instance itself. The tags are assigned to all resources running in the same instance. You can specify tags programmatically using the tags parameter in the <a href="https://docs.aws.amazon.com/cli/latest/reference/sagemaker/create-notebook-instance.html" target="_blank" rel="noopener noreferrer">create-notebook-instance</a> API or add them via the SageMaker console during instance creation. You can also add or update tags anytime using the <a href="https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker/add-tags.html" target="_blank" rel="noopener noreferrer">add-tags</a> API or via the SageMaker console.</p><p>Note that this excludes SageMaker managed jobs and resources such as training and processing jobs because they’re in the service environment rather than on the instance. In the next section, we go over how to apply tagging to these resources in greater detail.</p><h3>Tagging SageMaker managed jobs and resources</h3><p>For SageMaker managed jobs and resources, tagging must be applied to the <code>tags</code> attribute as part of each API request. An <code>SKLearnProcessor</code> example is illustrated in the following code. You can find more examples of how to assign tags to other SageMaker managed jobs and resources on the <a href="https://github.com/aws-samples/amazon-sagemaker-script-mode/blob/master/tf-2-workflow-smpipelines/tf-2-workflow-smpipelines.ipynb" target="_blank" rel="noopener noreferrer">GitHub repo</a>.</p><h3>Tagging SageMaker pipelines</h3><p>In the case of SageMaker pipelines, you can tag the entire pipeline as a whole instead of each individual step. The SageMaker pipeline automatically propagates the tags to each pipeline step. You still have the option to add additional, separate tags to individual steps if needed. In the Studio UI, the pipeline tags appear in the metadata section.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image011.jpg"><img class="alignnone size-full wp-image-42877 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image011.jpg" alt="" width="1394" height="242" /></a></p><p>To apply tags to a pipeline, use the SageMaker Python SDK:</p><h3>Enforce tagging using IAM policies</h3><p>Although tagging is an effective mechanism for implementing cloud management and governance strategies, enforcing the right tagging behavior can be challenging if you just leave it to the end-users. How do you prevent ML resource creation if a specific tag is missing, how do you ensure the right tags are applied, and how do you prevent users from deleting existing tags?</p><p>You can accomplish this using <a href="http://aws.amazon.com/iam" target="_blank" rel="noopener noreferrer">AWS Identity and Access Management</a> (IAM) policies. The following code is an example of a policy that prevents SageMaker actions such as <code>CreateDomain</code> or <code>CreateNotebookInstance</code> if the request doesn’t contain the environment key and one of the list values. The <code>ForAllValues</code> modifier with the <code>aws:TagKeys</code> condition key indicates that only the key <code>environment</code> is allowed in the request. This stops users from including other keys, such as accidentally using <code>Environment</code> instead of <code>environment</code>.</p><p><a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies.html" target="_blank" rel="noopener noreferrer">Tag policies</a> and <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html?icmpid=docs_orgs_console" target="_blank" rel="noopener noreferrer">service control policies</a> (SCPs) can also be a good way to standardize creation and labeling of your ML resources. For more information about how to implement a tagging strategy that enforces and validates tagging at the organization level, refer to <a href="https://aws.amazon.com/blogs/aws-cloud-financial-management/cost-allocation-blog-series-3-enforce-and-validate-aws-resource-tags/" target="_blank" rel="noopener noreferrer">Cost Allocation Blog Series #3: Enforce and Validate AWS Resource Tags</a>.</p><h2>Cost allocation reporting</h2><p>You can view the tags by filtering the views on Cost Explorer, viewing a <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/configurecostallocreport.html#allocation-viewing" target="_blank" rel="noopener noreferrer">monthly cost allocation report</a>, or by examining the CUR.</p><h3>Visualizing tags in Cost Explorer</h3><p>Cost Explorer is a tool that enables you to view and analyze your costs and usage. You can explore your usage and costs using the main graph: the Cost Explorer cost and usage reports. For a quick video on how to use Cost Explorer, check out <a href="https://www.youtube.com/watch?v=3peNAKB3VxA" target="_blank" rel="noopener noreferrer">How can I use Cost Explorer to analyze my spending and usage?</a></p><p>With Cost Explorer, you can filter how you view your AWS costs by tags. <strong>Group by</strong> allows us to filter out results by tag keys such as <code>Environment</code>, <code>Deployment</code>, or <code>Cost Center</code>. The tag filter helps us select the value we desire regardless of the key. Examples include <code>Production</code> and <code>Staging</code>. Keep in mind that you must run the resources after adding and activating tags; otherwise, Cost Explorer won’t have any usage data and the tag value won’t be displayed as a filter or group by option.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image015.png"><img class="alignnone size-full wp-image-42879 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image015.png" alt="" width="1394" height="376" /></a></p><p>The following screenshot is an example of filtering by all values of the <code>BusinessUnit</code> tag.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image017.png"><img class="alignnone size-full wp-image-42880 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image017.png" alt="" width="1350" height="681" /></a></p><h3>Examining tags in the CUR</h3><p>The Cost and Usage Rreport contains the most comprehensive set of cost and usage data available. The report contains line items for each unique combination of AWS product, usage type, and operation that your AWS account uses. You can customize the CUR to aggregate the information either by the hour or by the day. A monthly cost allocation report is one way to set up cost allocation reporting. You can set up a <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/configurecostallocreport.html" target="_blank" rel="noopener noreferrer">monthly cost allocation report</a> that lists the AWS usage for your account by product category and linked account user. The report contains the same line items as the <a href="https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html" target="_blank" rel="noopener noreferrer">detailed billing report</a> and additional columns for your tag keys. You can set it up and download your report by following the steps in <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/configurecostallocreport.html" target="_blank" rel="noopener noreferrer">Monthly cost allocation report</a>.</p><p>The following screenshot shows how user-defined tag keys show up in the CUR. User-defined tag keys have the prefix <code>user</code>, such as <code>user:Department</code> and <code>user:CostCenter</code>. AWS-generated tag keys have the prefix <code>aws</code>.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image019.jpg"><img class="alignnone size-full wp-image-42881 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image019.jpg" alt="" width="1076" height="196" /></a></p><h3>Visualize the CUR using Amazon Athena and Amazon QuickSight</h3><p><a href="http://aws.amazon.com/athena" target="_blank" rel="noopener noreferrer">Amazon Athena</a> is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run. To integrate Athena with CURs, refer to <a href="https://docs.aws.amazon.com/cur/latest/userguide/cur-query-athena.html" target="_blank" rel="noopener noreferrer">Querying Cost and Usage Reports using Amazon Athena</a>. You can then build custom queries to query CUR data using standard SQL. The following screenshot is an example of a query to filter all resources that have the value TF2WorkflowTraining for the <code>cost-center</code> tag.</p><p>In the following example, we’re trying to figure out which resources are missing values under the <code>cost-center</code> tag.</p><p>More information and example queries can be found in the <a href="https://www.wellarchitectedlabs.com/cost/300_labs/300_cur_queries/" target="_blank" rel="noopener noreferrer">AWS CUR Query Library</a>.</p><p>You can also feed CUR data into <a href="https://aws.amazon.com/quicksight" target="_blank" rel="noopener noreferrer">Amazon QuickSight</a>, where you can slice and dice it any way you’d like for reporting or visualization purposes. For instructions on ingesting CUR data into QuickSight, see <a href="https://aws.amazon.com/premiumsupport/knowledge-center/quicksight-cost-usage-report/" target="_blank" rel="noopener noreferrer">How do I ingest and visualize the AWS Cost and Usage Report (CUR) into Amazon QuickSight</a>.</p><h2>Budget monitoring using tags</h2><p>AWS Budgets is an excellent way to provide an early warning if spend spikes unexpectedly. You can create custom budgets that alert you when your ML costs and usage exceed (or are forecasted to exceed) your user-defined thresholds. With AWS Budgets, you can monitor your total monthly ML costs or filter your budgets to track costs associated with specific usage dimensions. For example, you can set the budget scope to include SageMaker resource costs tagged as <code>cost-center: ML-Marketing</code>, as shown in the following screenshot. For additional dimensions and detailed instructions on how to set up AWS Budgets, refer to <a href="https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html" target="_blank" rel="noopener noreferrer">here</a>.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image025.jpg"><img class="alignnone size-full wp-image-42885 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image025.jpg" alt="" width="1458" height="630" /></a></p><p>With <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/budgets-sns-policy.html" target="_blank" rel="noopener noreferrer">budget alerts</a>, you can send notifications when your budget limits are (or are about to be) exceeded. These alerts can also be posted to an <a href="http://aws.amazon.com/sns" target="_blank" rel="noopener noreferrer">Amazon Simple Notification Service</a> (Amazon SNS) topic. An <a href="http://aws.amazon.com/lambda" target="_blank" rel="noopener noreferrer">AWS Lambda</a> function that subscribes to the SNS topic is then invoked, and any programmatically implementable actions can be taken.</p><p>AWS Budgets also lets you configure <a href="https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-controls.html" target="_blank" rel="noopener noreferrer">budget actions</a>, which are steps that you can take when a budget threshold is exceeded (actual or forecasted amounts). This level of control allows you to reduce unintentional overspending in your account. You can configure specific responses to cost and usage in your account that will be applied automatically or through a workflow approval process when a budget target has been exceeded. This is a really powerful solution to ensure that your ML spend is consistent with the goals of the business. You can select what type of action to take. For example, when a budget threshold is crossed, you can move specific IAM users from admin permissions to read-only. For customers using Organizations, you can apply actions to an entire organizational unit by moving them from admin to read-only. For more details on how to manage cost using budget actions, refer to <a href="https://aws.amazon.com/blogs/mt/manage-cost-overruns-part-1/" target="_blank" rel="noopener noreferrer">How to manage cost overruns in your AWS multi-account environment – Part 1</a>.</p><p>You can also set up a report to monitor the performance of your existing budgets on a daily, weekly, or monthly cadence and deliver that report to up to 50 email addresses. With <a href="https://docs.aws.amazon.com/cost-management/latest/userguide/create-budget-report.html" target="_blank" rel="noopener noreferrer">AWS Budgets reports</a>, you can combine all SageMaker-related budgets into a single report. This feature enables you to track your SageMaker footprint from a single location, as shown in the following screenshot. You can opt to receive these reports on a daily, weekly, or monthly cadence (I’ve chosen <strong>Weekly</strong> for this example), and choose the day of week when you want to receive them.</p><p>This feature is useful to keep your stakeholders up to date with your SageMaker costs and usage, and help them see when spend isn’t trending as expected.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image029.jpg"><img class="alignnone size-full wp-image-42887 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image029.jpg" alt="" width="1596" height="844" /></a></p><p>After you set up this configuration, you should receive an email similar to the following.</p><p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image031.jpg"><img class="alignnone size-full wp-image-42888 c4" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/ML-10691-image031.jpg" alt="" width="1287" height="412" /></a></p><h2>Conclusion</h2><p>In this post, we showed how you can set up cost allocation tagging for SageMaker and shared tips on tagging best practices for your SageMaker environment and workloads. We then discussed different reporting options like Cost Explorer and the CUR to help you improve visibility into your ML spend. Lastly, we demonstrated AWS Budgets and the budget summary report to help you monitor the ML spend of your organization.</p><p>For more information about applying and activating cost allocation tags, see <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/custom-tags.html" target="_blank" rel="noopener noreferrer">User-Defined Cost Allocation Tags</a>.</p><h3>About the authors</h3><p class="c5"><strong><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2021/01/07/Sean-Morgan.jpg"><img class="size-full wp-image-20390 alignleft" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2021/01/07/Sean-Morgan.jpg" alt="Sean Morgan" width="100" height="117" /></a>Sean Morgan</strong> is an AI/ML Solutions Architect at AWS. He has experience in the semiconductor and academic research fields, and uses his experience to help customers reach their goals on AWS. In his free time, Sean is an active open-source contributor and maintainer, and is the special interest group lead for TensorFlow Add-ons.</p><p class="c5"><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2019/08/28/brent-rabowsky-100.gif"><img class="size-full wp-image-9588 alignleft" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2019/08/28/brent-rabowsky-100.gif" alt="" width="100" height="131" /></a><strong>Brent Rabowsky</strong> focuses on data science at AWS, and leverages his expertise to help AWS customers with their own data science projects.</p><p class="c5"><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/nileshet.png"><img class="size-full wp-image-42891 alignleft" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/09/20/nileshet.png" alt="" width="100" height="133" /></a><strong>Nilesh Shetty</strong> is as a Senior Technical Account Manager at AWS, where he helps enterprise support customers streamline their cloud operations on AWS. He is passionate about machine learning and has experience working as a consultant, architect, and developer. Outside of work, he enjoys listening to music and watching sports.</p><p class="c5"><strong><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/06/20/IMG_20200901_092324.jpg"><img class="wp-image-38360 size-full alignleft" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2022/06/20/IMG_20200901_092324.jpg" alt="" width="100" height="133" /></a>James Wu</strong> is a Senior AI/ML Specialist Solution Architect at AWS. helping customers design and build AI/ML solutions. James’s work covers a wide range of ML use cases, with a primary interest in computer vision, deep learning, and scaling ML across the enterprise. Prior to joining AWS, James was an architect, developer, and technology leader for over 10 years, including 6 years in engineering and 4 years in marketing & advertising industries.</p>

Fish AI Reader

FishAI

联系邮箱 441953276@qq.com

相关标签