Databricks在2024年数据+AI峰会上发布了Mosaic AI代理框架和代理评估的公开预览版。这些创新工具旨在帮助开发人员在Databricks数据智能平台上构建和部署高质量的代理和检索增强生成（RAG）应用程序。

🤔 **人机反馈整合:** 代理评估允许开发人员通过邀请组织中的主题专家（即使他们不是Databricks用户）来审查并提供反馈，从而为其生成式AI应用程序定义高质量的响应。此过程有助于收集不同的观点和见解，以改进应用程序。

📊 **全面评估指标:** 代理评估提供了一套指标来衡量应用程序质量，这些指标是与Mosaic Research合作开发的。这些指标包括准确性、幻觉、有害性和帮助性。该系统会自动将响应和反馈记录到评估表中，方便快速分析并识别潜在的质量问题。AI评判员通过专家反馈进行校准，评估响应以查明问题的根本原因。

🚀 **端到端开发工作流程:** 与MLflow集成，代理框架允许开发人员使用标准MLflow API记录和评估生成式AI应用程序。这种集成支持从开发到生产的无缝过渡，并具有持续的反馈循环，以提高应用程序质量。

⚙️ **应用程序生命周期管理:** 代理框架提供了一个简化的SDK，用于管理代理应用程序的生命周期，从权限管理到使用Mosaic AI模型服务进行部署。这个全面的管理系统确保应用程序在整个生命周期内保持可扩展性和高质量。

💡 **构建高质量的RAG代理:** Databricks提供了一个构建高质量RAG应用程序的示例，以说明Mosaic AI代理框架的功能。此示例涉及创建一个简单的RAG应用程序，该应用程序从预先创建的向量索引中检索相关片段，并根据查询对它们进行总结。此工作流程演示了开发人员使用Mosaic AI工具轻松构建、评估和改进生成式AI应用程序的方式。

🏢 **实际应用和证明:** 多家公司已成功实施Mosaic AI代理框架，以增强其生成式AI解决方案。例如，康宁使用该框架构建了一个AI研究助理，该助理索引了数十万份文档，显着提高了检索速度、响应质量和准确性。利珀特利用该框架来评估其生成式AI应用程序的结果，确保数据准确性和控制。福特直销公司将该框架集成到一起，为其经销商创建了一个统一的聊天机器人，促进了更好的性能评估和客户互动。

💰 **定价和下一步:** 代理评估的定价基于评判请求，而Mosaic AI模型服务的定价根据Mosaic AI模型服务费率。Databricks鼓励客户通过访问各种资源（例如代理框架文档、演示笔记本和生成式AI食谱）来尝试Mosaic AI代理框架和代理评估。这些资源提供了有关从概念验证到部署构建生产级生成式AI应用程序的详细指南。

Databricks announced the public preview of the Mosaic AI Agent Framework and Agent Evaluation during the Data + AI Summit 2024. These innovative tools aim to assist developers in building and deploying high-quality Agentic and Retrieval Augmented Generation (RAG) applications on the Databricks Data Intelligence Platform.

Challenges in Building High-Quality Generative AI Applications

Creating a proof of concept for generative AI applications is relatively straightforward. However, delivering a high-quality application that meets the rigorous standards required for customer-facing solutions takes time and effort. Developers often struggle with:

Choosing the right metrics to evaluate application quality.Efficiently collecting human feedback to measure quality.Identifying the root causes of quality issues.Rapidly iterating to improve application quality before deploying to production.

Introducing Mosaic AI Agent Framework and Agent Evaluation

The Mosaic AI Agent Framework and Agent Evaluation address these challenges through several key capabilities:

Human Feedback Integration: Agent Evaluation allows developers to define high-quality responses for their generative AI applications by inviting subject matter experts across their organization to review and provide feedback, even if they are not Databricks users. This process helps in gathering diverse perspectives and insights to refine the application.Comprehensive Evaluation Metrics: Developed in collaboration with Mosaic Research, Agent Evaluation offers a suite of metrics to measure application quality. These metrics include accuracy, hallucination, harmfulness, and helpfulness. The system automatically logs responses and feedback to an evaluation table, facilitating quick analysis and identifying potential quality issues. AI judges, calibrated using expert feedback, evaluate responses to pinpoint the root causes of problems.End-to-End Development Workflow: Integrated with MLflow, the Agent Framework allows developers to log and evaluate generative AI applications using standard MLflow APIs. This integration supports seamless transitions from development to production, with continuous feedback loops to enhance application quality.App Lifecycle Management: The Agent Framework provides a simplified SDK for managing the lifecycle of agentic applications, from permissions management to deployment with Mosaic AI Model Serving. This comprehensive management system ensures that applications remain scalable and maintain high quality throughout their lifecycle.

Building a High-Quality RAG Agent

To illustrate the capabilities of the Mosaic AI Agent Framework, Databricks provided an example of building a high-quality RAG application. This example involves creating a simple RAG application that retrieves relevant chunks from a pre-created vector index and summarizes them in response to queries. The process includes connecting to the vector search index, setting the index into a LangChain retriever, and leveraging MLflow to enable traces and deploy the application. This workflow demonstrates the ease with which developers can build, evaluate, and improve generative AI applications using the Mosaic AI tools.

Real-World Applications and Testimonials

Several companies have successfully implemented the Mosaic AI Agent Framework to enhance their generative AI solutions. For instance, Corning used the framework to build an AI research assistant that indexes hundreds of thousands of documents, significantly improving retrieval speed, response quality, and accuracy. Lippert leveraged the framework to evaluate the results of their generative AI applications, ensuring data accuracy and control. FordDirect integrated the framework to create a unified chatbot for their dealerships, facilitating better performance assessment and customer engagement.

Pricing and Next Steps

The pricing for Agent Evaluation is based on judge requests, while Mosaic AI Model Serving is priced according to Mosaic AI Model Serving rates. Databricks encourages customers to try the Mosaic AI Agent Framework and Agent Evaluation by accessing various resources such as the Agent Framework documentation, demo notebooks, and the Generative AI Cookbook. These resources provide detailed guidance on building production-quality generative AI applications from proof of concept to deployment.

In conclusion, Databricks’ announcement of the Mosaic AI Agent Framework and Agent Evaluation represents a significant advancement in generative AI. These tools provide developers with the necessary capabilities to efficiently build, evaluate, and deploy high-quality generative AI applications. By addressing common challenges and offering comprehensive support, Databricks empowers developers to create innovative solutions that meet the highest quality and performance standards.

The post Databricks Announced the Public Preview of Mosaic AI Agent Framework and Agent Evaluation  appeared first on MarkTechPost.