随着大型语言模型（LLM）的兴起，人工智能领域取得了长足进步，其中智能体（Agent）作为LLM的一种高级应用，能够模拟人类的推理能力，执行复杂任务。文章介绍了四种评估AI智能体有效性和可靠性的前沿方法：‘智能体作为评判者’、‘智能体应用评估框架’(AAEF)、‘MOSAIC AI’以及‘WORFEVAL’。这些方法分别从不同角度评估智能体，例如推理能力、工具利用效率、记忆检索、计划能力等，旨在帮助开发者测试智能体的性能，找到最佳配置，从而提升LLM的性能。文章也探讨了每种方法的优缺点，例如‘智能体作为评判者’在复杂任务中可能存在局限性，而WORFEVAL则更适用于处理复杂数据结构。

🤔 **‘智能体作为评判者’**：该方法利用AI来评估AI，LLM扮演评判者、监考者和考生角色，通过评估准确性、完整性、相关性、及时性和成本效率等指标来判断智能体的性能。这种方法在OpenHands评估中表现出色，比LLM评估效果提升了30%。

💡 **‘智能体应用评估框架’(AAEF)**：该框架通过四个关键组件（工具利用效率、记忆一致性和检索、战略计划指数、组件协同得分）来评估智能体在特定任务上的表现，例如工具选择、记忆能力、计划执行能力和协同工作能力等。

🚀 **‘MOSAIC AI’**：Databricks提出的MOSAIC AI智能体评估框架，提供了一套统一的指标，例如准确率、精确率、召回率和F1分数，简化了评估过程，并结合了人工审查和反馈来定义高质量的响应。同时，它还集成了MLFlow，方便模型从开发到生产的部署和改进。

⚙️ **‘WORFEVAL’**：该方法通过基于高级子序列和子图匹配的定量算法，评估LLM智能体的工作流程能力。它将预测的节点链和工作流程图与正确的流程进行比较，适用于复杂数据结构，例如有向无环图。

⚠️ **方法局限性**：文章指出了每种方法的局限性，例如‘智能体作为评判者’在复杂任务中可能存在局限性，AAEF在复杂动态任务中也存在挑战，MOSAIC AI在数据规模和多样性增加时可信度下降，WORFEVAL的性能依赖于正确的工作流程定义，而这本身是一个随机变量。

The advent of LLMs has propelled advancements in AI for decades. One such advanced application of LLMs is Agents, which replicate human reasoning remarkably. An agent is a system that can perform complicated tasks by following a reasoning process similar to humans: think (solution to the problem), collect (context from past information), analyze(the situations and data), and adapt (based on the style and feedback). Agents encourage the system through dynamic and intelligent activities, including planning, data analysis, data retrieval, and utilizing the model’s past experiences. 

A typical agent has four components:

Brain: An LLM with advanced processing capabilities, such as prompts.Memory: For storing and recalling information.Planning: Decomposing tasks into sub-sequences and creating plans for each.Tools: Connectors that integrate LLMs with the external environment, akin to joining two LEGO pieces. Tools allow agents to perform unique tasks by combining LLMs with databases, calculators, or APIs.

Now that we have established the wonders of agents in transforming an ordinary LLM into a specialized and intelligent tool, it is necessary to assess the effectiveness and reliability of an agent. Agent evaluation not only ascertains the quality of the framework in question but also identifies the best processes and reduces inefficiencies and bottlenecks. This article discusses four ways to gauge the effectiveness of an agent.

Agent as Judge: It is the assessment of AI by AI and for AI. LLMs take on the roles of judge, invigilator, and examinee in this arrangement. The judge scrutinizes the examinee’s response and gives its ruling based on accuracy, completeness, relevance, timeliness, and cost efficiency. The examiner coordinates between the judge and examinee by providing the target tasks and retrieving the response from the judge. The examiner also offers descriptions and clarifications to the examinee LLM. The “Agent as Judge” framework has eight interacting modules. Agents perform the role of judge much better than LLMs, and this approach has a high alignment rate with human evaluation. One such instance is the OpenHands evaluation, where Agent Evaluation performed 30% better than LLM judgment.

Agentic Application Evaluation Framework (AAEF) assesses agents’ performance on specific tasks. Qualitative outcomes such as effectiveness, efficiency, and adaptability are measured for agents through four components: Tool Utilization Efficacy (TUE), Memory Coherence and Retrieval (MCR), Strategic Planning Index (SPI), and Component Synergy Score (CSS). Each of these specializes in different assessment criteria, from the selection of appropriate tools to the measurement of memory, the ability to plan and execute, and the ability to work coherently.MOSAIC AI: The Mosaic AI Agent Framework for evaluation, announced by Databricks, solves multiple challenges simultaneously. It offers a unified set of metrics, including but not limited to accuracy, precision, recall, and F1 score, to ease the process of choosing the right metrics for evaluation. It further integrates human review and feedback to define high-quality responses. Besides furnishing a solid pipeline for evaluation, Mosaic AI also has MLFlow integration to take the model from development to production while improving it. Mosaic AI also provides a simplified SDK for app lifecycle management.WORFEVAL: It is a systematic protocol that helps assess an LLM agent’s workflow capabilities through quantitative algorithms based on advanced subsequence and subgraph matching. This evaluation technique compares predicted node chains and workflow graphs with correct flows. WORFEVAL comes on the advanced end of the spectrum, where agent application is done on complex structures like Directed Acyclic Graphs in a multi-faceted scenario.

Each of the above methods helps developers test if their agent is performing satisfactorily and find the optimal configuration, but they have their demerits. Discussing Agent Judgment first could be questioned in complex tasks that require deep knowledge. One could always ask about the competence of the teacher! Even agents trained on specific data may have biases that hinder generalization. AAEF faces a similar fate in complex and dynamic tasks. MOSAIC AI is good, but its credibility decreases as the scale and diversity of data increase. At the highest end of the spectrum, WORFEVAL performs well even on complex data, but its performance depends on the correct workflow, which is a random variable—the definition of the correct workflow changes from computer to computer.

Conclusion: Agents are an attempt to make LLMs more human-like with reasoning capabilities and intelligent decision-making. The evaluation of agents is thus imperative to ensure their claims and quality. Agents as Judge, the Agentic Application Evaluation Framework, Mosaic AI, and WORFEVAL are the current top evaluation techniques. While Agents as Judge starts with the basic intuitive idea of peer review, WORFEVAL deals with complex data. Although these evaluation methods perform well in their respective contexts, they face difficulties as tasks become more intricate with complicated structures.

The post Four Cutting-Edge Methods for Evaluating AI Agents and Enhancing LLM Performance appeared first on MarkTechPost.