NewsJamba 1.5 报道了人工智能领域最新的进展，包括混合模型 SSM+Transformer 在长文本处理上的效率提升、扩散模型在实时游戏引擎中的应用、xAI 团队使用 SGLang、Cartesia AI 推出的设备端 SSM 模型等。此外，文章还介绍了 Llamafile 1.58 位 LLM 支持、LinkedIn 工程团队的 Liger-Kernel Medusa 头部训练、Salesforce 发布的更多大型行动模型，以及关于混合模型蒸馏的最新研究成果。

😊 **混合模型 SSM+Transformer 效率提升:** NewsJamba 1.5 报道了混合模型 SSM+Transformer 在长文本处理上的效率提升，表明了这种混合架构在处理长文本时能够取得显著的效果。这对于需要处理大量文本数据的应用场景，例如问答系统、机器翻译和文本摘要等，具有重要的意义。 例如，该报道中提到了 Liger-Kernel Medusa 头部训练，它是 LinkedIn 工程团队开发的混合模型，利用了 SSM 和 Transformer 的优势，在长文本处理方面表现出色。

🤖 **扩散模型应用于实时游戏引擎:** 文章还报道了扩散模型在实时游戏引擎中的应用，这表明了扩散模型在游戏开发领域的潜力。扩散模型可以通过生成逼真的图像和视频，为游戏玩家提供更身临其境的体验。 例如，文章提到扩散模型可以用于生成游戏中的角色、场景和特效，从而提高游戏的视觉效果和沉浸感。

📚 **开源社区的贡献:** NewsJamba 1.5 还介绍了开源社区在人工智能领域的重要贡献，例如 Mozilla/Justine Tunney 发布的 Llamafile 1.58 位 LLM 支持，以及 OpenDevin 重命名为 OpenHands 等。这些开源项目为研究人员和开发者提供了宝贵的资源和工具，推动了人工智能领域的发展。 例如，Llamafile 1.58 位 LLM 支持为开发者提供了在 CPU 上运行大型语言模型的能力，降低了人工智能开发的门槛。

🌐 **分布式训练的探索:** 文章还探讨了分布式训练的最新进展，例如 Nous Research 的 DisTrO（分布式训练 Over-The-Internet）项目，这是一个很有趣的尝试，旨在将分布式模型训练扩展到“Folding@Home”风格的互联网规模。 DisTrO 项目的目标是利用全球用户的计算资源，共同训练大型语言模型，这将有助于克服单一服务器资源有限的限制，推动人工智能模型的规模化训练。

🧠 **混合模型蒸馏:** 文章重点介绍了混合模型蒸馏领域的最新研究成果，包括来自 CMU 的 MOHAWK 和来自 Cornell/ex-Stanford/Princeton 和日内瓦大学的 Mamba 等。这些研究表明，通过蒸馏技术，可以将大型语言模型的知识转移到更小、更高效的模型中，从而提高模型的效率和部署能力。 例如，MOHAWK 框架提出了一种将 Transformer 模型蒸馏到 SSM 模型的方法，该方法能够有效地保留 Transformer 模型的性能，同时降低模型的大小和计算成本。

? News

Jamba 1.5: hybrid SSM+Transformer Efficient Long Context Models

Diffusion Models Are Real-Time Game Engines

This is a crazy result, I’m curious to see what this implies in the limit. Yann, is this our world model?

SGLang in use by xAI team

Rene: on-device SSM model by Cartesia AI

Mozilla/Justine Tunney releases 1.58 bit LLM support in llamafile (CPU inference)

? Repos

Liger-Kernel Medusa head training

Created by the LinkedIn engineering team.

Salesforces releases more "Large Action Models" on HF

MoE, 32k context, why these are not called function calling models beats me. Still doesn't allow commercial use.

? Papers

BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

Scaling Law with Learning Rate Annealing

Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler

Generative Verifiers: Reward Modeling as Next-Token Prediction
I like the idea of simplifying reward models.

EAGLE: Exploring The Design Space for MultimodalLLMs with Mixture of Encoders

Not One Vision Encoder to Rule Them All I guess! Work by NVIDIA et al.

Critique-out-Loud Reward Models

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
DeepSeek strikes again! Stacked with GPU-poor-ish innovations to scale up large-scale model training.

The Mamba in the Llama: Distilling and Accelerating Hybrid Models

Two SSM distilling papers this week, MOHAWK from CMU and this from Cornell/ex-Stanford/Princeton and University of Geneva.

Nous Research: DisTrO (Distributed Training Over-The-Internet)

Interesting attempt at “folding at home style” distributed model training. Code is on the way I’ve heard.

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

? Resources

Ape - your first prompt engineer
Anthropic is leaning into prompt generation support in their console, this might be the open source version of that.

Berkeley Function Calling Leaderboard V2
Looks promising! I’m taking a closer look next week, more to follow…

Llama 3.1 8B (quantized) with 128k context in 18.9GB VRAM

[Video] MLSys conference talk about AI systems by Jeff Dean

Cross-Architecture Distillation Part I - The MOHAWK Framework (Transformer->SSM)

[Video] Interview with Jürgen Schmidhuber – the father of generative AI

Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)

Hotshot: 4 person team creates Sora/Luma alternative

Anthropic adds system prompts to docs

Love how they’re like “you can prompt jailbreak extract them anyway, let’s own that they are public” instead of pretending they’re invisible to users.

GPUd by Lepton AI

OpenDevin rebrands to OpenHands

The Devin competitor (the OSS team is now a company too).

Llama 3.1 405b bf16 base model

Some AI sommeliers apparently really dig what you can get a bf16 precision Llama 3.1 405b base model to do. If you find out cool examples please share (as GitHub gist) and I'll promote! Hosting curtesy of Hyperbolic.

Local Speech-to-Speech models

By the excellent Trelis Research

Want more? Follow me on X! @ricklamers