本文介绍了如何利用Amazon Bedrock的多Agent协作能力，构建一个智能的Amazon EKS集群故障排除系统。通过整合K8sGPT和ArgoCD，该系统能够自动识别、分析并解决集群问题，从而减少人工干预，提高运维效率。文章详细阐述了系统架构、先决条件以及部署步骤，为用户提供了实践指南。

💡该方案的核心在于利用Amazon Bedrock的多Agent协作能力，构建一个专门用于EKS集群故障排除的系统，它通过整合K8sGPT和ArgoCD来实现自动化运维。

🤖系统由三个主要组件构成：Amazon Bedrock协调器Agent、用于K8sGPT的Amazon Bedrock Agent和用于ArgoCD的Amazon Bedrock Agent。每个Agent都扮演着特定的角色，共同完成集群问题的识别、分析和解决。

⚙️部署该系统需要AWS CLI、EKS集群、Helm、Kubectl，以及对Amazon Bedrock模型的访问权限。此外，文章提供了详细的安装和配置K8sGPT operator和ArgoCD控制器的步骤，以及如何配置Bedrock Agent。

✅通过使用K8sGPT分析集群事件，并由Bedrock Agent协调ArgoCD执行修复操作，该系统能够自动处理诸如配置错误、资源限制等问题，从而减少MTTI和MTTR。

As organizations scale their Amazon Elastic Kubernetes Service (Amazon EKS) deployments, platform administrators face increasing challenges in efficiently managing multi-tenant clusters. Tasks such as investigating pod failures, addressing resource constraints, and resolving misconfiguration can consume significant time and effort. Instead of spending valuable engineering hours manually parsing logs, tracking metrics, and implementing fixes, teams should focus on driving innovation. Now, with the power of generative AI, you can transform your Kubernetes operations. By implementing intelligent cluster monitoring, pattern analysis, and automated remediation, you can dramatically reduce both mean time to identify (MTTI) and mean time to resolve (MTTR) for common cluster issues.

At AWS re:Invent 2024, we announced the multi-agent collaboration capability for Amazon Bedrock (preview). With multi-agent collaboration, you can build, deploy, and manage multiple AI agents working together on complex multistep tasks that require specialized skills. Because troubleshooting an EKS cluster involves deriving insights from multiple observability signals and applying fixes using a continuous integration and deployment (CI/CD) pipeline, a multi-agent workflow can help an operations team streamline the management of EKS clusters. The workflow manager agent can integrate with individual agents that interface with individual observability signals and a CI/CD workflow to orchestrate and perform tasks based on user prompt.

In this post, we demonstrate how to orchestrate multiple Amazon Bedrock agents to create a sophisticated Amazon EKS troubleshooting system. By enabling collaboration between specialized agents—deriving insights from K8sGPT and performing actions through the ArgoCD framework—you can build a comprehensive automation that identifies, analyzes, and resolves cluster issues with minimal human intervention.

Solution overview

The architecture consists of the following core components:

Amazon Bedrock collaborator agent – Orchestrates the workflow and maintains context while routing user prompts to specialized agents, managing multistep operations and agent interactions Amazon Bedrock agent for K8sGPT – Evaluates cluster and pod events through K8sGPT’s Analyze API for security issues, misconfigurations, and performance problems, providing remediation suggestions in natural language Amazon Bedrock agent for ArgoCD – Manages GitOps-based remediation through ArgoCD, handling rollbacks, resource optimization, and configuration updates

The following diagram illustrates the solution architecture.

Prerequisites

You need to have the following prerequisites in place:

The AWS Command Line Interface (AWS CLI) version 2. For installation instructions, refer to Installing or updating to the latest version of the AWS CLI. An EKS cluster. helm. Kubectl. Amazon Bedrock model access (In this post, we used Anthropic Claude 3.5 Sonnet v1) in the AWS Region of deployment. Download the accompanying AWS CloudFormation template. The template is dependent on downloading resources from an Amazon Simple Storage Service (Amazon S3) bucket provisioned in the US East (N. Virginia) us-east-1 AWS Region. Hence, it’s restricted to running in the us-east-1 Region only.

Set up the Amazon EKS cluster with K8sGPT and ArgoCD

We start with installing and configuring the K8sGPT operator and ArgoCD controller on the EKS cluster.

The K8sGPT operator will help with enabling AI-powered analysis and troubleshooting of cluster issues. For example, it can automatically detect and suggest fixes for misconfigured deployments, such as identifying and resolving resource constraint problems in pods.

ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes that automates the deployment of applications by keeping the desired application state in sync with what’s defined in a Git repository.

The Amazon Bedrock agent serves as the intelligent decision-maker in our architecture, analyzing cluster issues detected by K8sGPT. After the root cause is identified, the agent orchestrates corrective actions through ArgoCD’s GitOps engine. This powerful integration means that when problems are detected (whether it’s a misconfigured deployment, resource constraints, or scaling issue), the agent can automatically integrate with ArgoCD to provide the necessary fixes. ArgoCD then picks up these changes and synchronizes them with your EKS cluster, creating a truly self-healing infrastructure.

Create the necessary namespaces in Amazon EKS:

kubectl create ns helm-guestbookkubectl create ns k8sgpt-operator-system

Add the k8sgpt Helm repository and install the operator:

helm repo add k8sgpt https://charts.k8sgpt.ai/helm repo updatehelm install k8sgpt-operator k8sgpt/k8sgpt-operator \  --namespace k8sgpt-operator-system

You can verify the installation by entering the following command:

kubectl get pods -n k8sgpt-operator-systemNAME                                                          READY   STATUS    RESTARTS  AGErelease-k8sgpt-operator-controller-manager-5b749ffd7f-7sgnd   2/2     Running   0         1d

After the operator is deployed, you can configure a K8sGPT resource. This Custom Resource Definition(CRD) will have the large language model (LLM) configuration that will aid in AI-powered analysis and troubleshooting of cluster issues. K8sGPT supports various backends to help in AI-powered analysis. For this post, we use Amazon Bedrock as the backend and Anthropic’s Claude V3 as the LLM.

You need to create the pod identity for providing the EKS cluster access to other AWS services with Amazon Bedrock:

eksctl create podidentityassociation  --cluster PetSite --namespace k8sgpt-operator-system --service-account-name k8sgpt  --role-name k8sgpt-app-eks-pod-identity-role --permission-policy-arns arn:aws:iam::aws:policy/AmazonBedrockFullAccess  --region $AWS_REGION

Configure the K8sGPT CRD:

cat << EOF > k8sgpt.yamlapiVersion: core.k8sgpt.ai/v1alpha1kind: K8sGPTmetadata:  name: k8sgpt-bedrock  namespace: k8sgpt-operator-systemspec:  ai:    enabled: true    model: anthropic.claude-v3    backend: amazonbedrock    region: us-east-1    credentials:      secretRef:        name: k8sgpt-secret        namespace: k8sgpt-operator-system  noCache: false  repository: ghcr.io/k8sgpt-ai/k8sgpt  version: v0.3.48EOFkubectl apply -f k8sgpt.yaml

Validate the settings to confirm the k8sgpt-bedrock pod is running successfully:

kubectl get pods -n k8sgpt-operator-systemNAME                                                          READY   STATUS    RESTARTS      AGEk8sgpt-bedrock-5b655cbb9b-sn897                               1/1     Running   9 (22d ago)   22drelease-k8sgpt-operator-controller-manager-5b749ffd7f-7sgnd   2/2     Running   3 (10h ago)   22d

Now you can configure the ArgoCD controller:

helm repo add argo https://argoproj.github.io/argo-helmhelm repo updatekubectl create namespace argocdhelm install argocd argo/argo-cd \  --namespace argocd \  --create-namespace

Verify the ArgoCD installation:

kubectl get pods -n argocdNAME                                                READY   STATUS    RESTARTS   AGEargocd-application-controller-0                     1/1     Running   0          43dargocd-applicationset-controller-5c787df94f-7jpvp   1/1     Running   0          43dargocd-dex-server-55d5769f46-58dwx                  1/1     Running   0          43dargocd-notifications-controller-7ccbd7fb6-9pptz     1/1     Running   0          43dargocd-redis-587d59bbc-rndkp                        1/1     Running   0          43dargocd-repo-server-76f6c7686b-rhjkg                 1/1     Running   0          43dargocd-server-64fcc786c-bd2t8                       1/1     Running   0          43d

Patch the argocd service to have an external load balancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

You can now access the ArgoCD UI with the following load balancer endpoint and the credentials for the admin user:

kubectl get svc argocd-server -n argocdNAME            TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                      AGEargocd-server   LoadBalancer   10.100.168.229   a91a6fd4292ed420d92a1a5c748f43bc-653186012.us-east-1.elb.amazonaws.com   80:32334/TCP,443:32261/TCP   43d

Retrieve the credentials for the ArgoCD UI:

export argocdpassword=`kubectl -n argocd get secret argocd-initial-admin-secret \-o jsonpath="{.data.password}" | base64 -d`echo ArgoCD admin password - $argocdpassword

Push the credentials to AWS Secrets Manager:

aws secretsmanager create-secret \--name argocdcreds \--description "Credentials for argocd" \--secret-string "{\"USERNAME\":\"admin\",\"PASSWORD\":\"$argocdpassword\"}"

Configure a sample application in ArgoCD:

cat << EOF > argocd-application.yamlapiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:name: helm-guestbooknamespace: argocdspec:project: defaultsource:repoURL: https://github.com/awsvikram/argocd-example-appstargetRevision: HEADpath: helm-guestbookdestination:server: https://kubernetes.default.svcnamespace: helm-guestbooksyncPolicy:automated:prune: trueselfHeal: trueEOF

Apply the configuration and verify it from the ArgoCD UI by logging in as the admin user:

kubectl apply -f argocd-application.yaml



It takes some time for K8sGPT to analyze the newly created pods. To make that immediate, restart the pods created in the k8sgpt-operator-system namespace. The pods can be restarted by entering the following command:

kubectl -n k8sgpt-operator-system rollout restart deploydeployment.apps/k8sgpt-bedrock restarteddeployment.apps/k8sgpt-operator-controller-manager restarted

Set up the Amazon Bedrock agents for K8sGPT and ArgoCD

We use a CloudFormation stack to deploy the individual agents into the US East (N. Virginia) Region. When you deploy the CloudFormation template, you deploy several resources (costs will be incurred for the AWS resources used).

Use the following parameters for the CloudFormation template:

EnvironmentName: The name for the deployment (EKSBlogSetup) ArgoCD_LoadBalancer_URL: Extracting the ArgoCD LoadBalancer URL:

kubectl  get service argocd-server -n argocd -ojsonpath="{.status.loadBalancer.ingress[0].hostname}"

AWSSecretName: The Secrets Manager secret name that was created to store ArgoCD credentials

The stack creates the following AWS Lambda functions:

<Stack name>-LambdaK8sGPTAgent-<auto-generated> <Stack name>-RestartRollBackApplicationArgoCD-<auto-generated> <Stack name>-ArgocdIncreaseMemory-<auto-generated>

The stack creates the following Amazon Bedrock agents:

ArgoCDAgent, with the following action groups:
argocd-rollback argocd-restart argocd-memory-management

K8sGPTAgent, with the following action group:
k8s-cluster-operations

CollaboratorAgent

The stack outputs the following, with the following agents associated to it:

ArgoCDAgent K8sGPTAgent

LambdaK8sGPTAgentRole, AWS Identity and Access Management (IAM) role Amazon Resource Name (ARN) associated to the Lambda function handing interactions with the K8sGPT agent on the EKS cluster. This role ARN will be needed at a later stage of the configuration process. K8sGPTAgentAliasId, ID of the K8sGPT Amazon Bedrock agent alias ArgoCDAgentAliasId, ID of the ArgoCD Amazon Bedrock Agent alias CollaboratorAgentAliasId, ID of the collaborator Amazon Bedrock agent alias

Assign appropriate permissions to enable K8sGPT Amazon Bedrock agent to access the EKS cluster

To enable the K8sGPT Amazon Bedrock agent to access the EKS cluster, you need to configure the appropriate IAM permissions using Amazon EKS access management APIs. This is a two-step process: first, you create an access entry for the Lambda function’s execution role (which you can find in the CloudFormation template output section), and then you associate the AmazonEKSViewPolicy to grant read-only access to the cluster. This configuration makes sure that the K8sGPT agent has the necessary permissions to monitor and analyze the EKS cluster resources while maintaining the principle of least privilege.

Create an access entry for the Lambda function’s execution role

export CFN_STACK_NAME=EKS-Troubleshooter       export EKS_CLUSTER=PetSiteexport K8SGPT_LAMBDA_ROLE=`aws cloudformation describe-stacks --stack-name $CFN_STACK_NAME --query "Stacks[0].Outputs[?OutputKey=='LambdaK8sGPTAgentRole'].OutputValue" --output text`aws eks create-access-entry \    --cluster-name $EKS_CLUSTER \    --principal-arn $K8SGPT_LAMBDA_ROLE

Associate the EKS view policy with the access entry

aws eks associate-access-policy \    --cluster-name $EKS_CLUSTER \    --principal-arn  $K8SGPT_LAMBDA_ROLE\    --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \    --access-scope type=cluster

Verify the Amazon Bedrock agents. The CloudFormation template adds all three required agents. To view the agents, on the Amazon Bedrock console, under Builder tools in the navigation pane, select Agents, as shown in the following screenshot.

Perform Amazon EKS troubleshooting using the Amazon Bedrock agentic workflow

Now, test the solution. We explore the following two scenarios:

The agent coordinates with the K8sGPT agent to provide insights into the root cause of a pod failure The collaborator agent coordinates with the ArgoCD agent to provide a response

Agent coordinates with K8sGPT agent to provide insights into the root cause of a pod failure

In this section, we examine a down alert for a sample application called memory-demo. We’re interested in the root cause of the issue. We use the following prompt: “We got a down alert for the memory-demo app. Help us with the root cause of the issue.”

The agent not only stated the root cause, but went one step further to potentially fix the error, which in this case is increasing memory resources to the application.

Collaborator agent coordinates with ArgoCD agent to provide a response

For this scenario, we continue from the previous prompt. We feel the application wasn’t provided enough memory, and it should be increased to permanently fix the issue. We can also tell the application is in an unhealthy state in the ArgoCD UI, as shown in the following screenshot.

Let’s now proceed to increase the memory, as shown in the following screenshot.

The agent interacted with the argocd_operations Amazon Bedrock agent and was able to successfully increase the memory. The same can be inferred in the ArgoCD UI.

Cleanup

If you decide to stop using the solution, complete the following steps:

To delete the associated resources deployed using AWS CloudFormation:
On the AWS CloudFormation console, choose Stacks in the navigation pane. Locate the stack you created during the deployment process (you assigned a name to it). Select the stack and choose Delete.
Delete the EKS cluster if you created one specifically for this implementation.

Conclusion

By orchestrating multiple Amazon Bedrock agents, we’ve demonstrated how to build an AI-powered Amazon EKS troubleshooting system that simplifies Kubernetes operations. This integration of K8sGPT analysis and ArgoCD deployment automation showcases the powerful possibilities when combining specialized AI agents with existing DevOps tools. Although this solution represents advancement in automated Kubernetes operations, it’s important to remember that human oversight remains valuable, particularly for complex scenarios and strategic decisions.

As Amazon Bedrock and its agent capabilities continue to evolve, we can expect even more sophisticated orchestration possibilities. You can extend this solution to incorporate additional tools, metrics, and automation workflows to meet your organization’s specific needs.

To learn more about Amazon Bedrock, refer to the following resources:

GitHub repo: Amazon Bedrock Workshop Amazon Bedrock User Guide Workshop: GenAI for AWS Cloud Operations Workshop: Using generative AI on AWS for diverse content types Getting insight from Amazon Managed Service for Prometheus using natural language powered by Amazon Bedrock

---

About the authors

Vikram Venkataraman is a Principal Specialist Solutions Architect at Amazon Web Services (AWS). He helps customers modernize, scale, and adopt best practices for their containerized workloads. With the emergence of Generative AI, Vikram has been actively working with customers to leverage AWS’s AI/ML services to solve complex operational challenges, streamline monitoring workflows, and enhance incident response through intelligent automation.

Puneeth Ranjan Komaragiri is a Principal Technical Account Manager at Amazon Web Services (AWS). He is particularly passionate about monitoring and observability, cloud financial management, and generative AI domains. In his current role, Puneeth enjoys collaborating closely with customers, leveraging his expertise to help them design and architect their cloud workloads for optimal scale and resilience.

Sudheer Sangunni is a Senior Technical Account Manager at AWS Enterprise Support. With his extensive expertise in the AWS Cloud and big data, Sudheer plays a pivotal role in assisting customers with enhancing their monitoring and observability capabilities within AWS offerings.

Vikrant Choudhary is a Senior Technical Account Manager at Amazon Web Services (AWS), specializing in healthcare and life sciences. With over 15 years of experience in cloud solutions and enterprise architecture, he helps businesses accelerate their digital transformation initiatives. In his current role, Vikrant partners with customers to architect and implement innovative solutions, from cloud migrations and application modernization to emerging technologies such as generative AI, driving successful business outcomes through cloud adoption.