语言模型中存在两种逊于指令调优却仍能实现指令遵循的适应形式，即隐式指令调优。仅训练模型对响应的学习，或在窄域数据上的指令响应训练，都能产生广泛的指令遵循行为。为解释此现象，提出简单改变语言模型分布可实现指令遵循的假设，并通过手写基于规则的语言模型加以支持。

🎯语言模型的指令调优通常是在指令 - 响应对上进行微调，但研究发现仅对响应进行训练，无需相应指令，也能产生指令遵循。这表明预训练模型具有指令 - 响应映射，可通过教授模型期望的响应分布来揭示。

💡进一步发现，无需专门教授期望的响应分布，在如诗歌等窄域数据上进行指令 - 响应训练，仍能导致如食谱生成等广泛的指令遵循行为。特别是当指令与窄微调域中的指令差异很大时，模型的响应不会遵循微调域的风格。

🤔为解释隐式指令调优，假设对语言模型的分布进行非常简单的更改就能产生指令遵循。通过手写一个基于规则的语言模型，在与预训练模型的专家乘积中产生指令遵循来支持这一假设。规则包括缓慢增加序列结束的概率、惩罚重复以及统一改变 15 个单词的概率。

Published on September 24, 2024 1:49 PM GMT

Authors: John Hewitt, Nelson F. Liu, Percy Liang, Christopher D. Manning.

Abstract:

Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.

Seems like some amount of evidence that getting models to follow instructions might be surprisingly easy.

Discuss