本文探讨了不同大型语言模型（LLM）在被要求选择“最喜欢的颜色”时所表现出的偏好。通过一系列精心设计的提示，研究者观察到，尽管提示中没有明确提及颜色，但LLM的回答在很大程度上受到模型本身、提示的措辞以及“流行”和“个人偏好”等因素的影响。例如，Llama-3.2-vision-90b-instruct 模型几乎总是选择黑色，而 Deepseek 模型则偏爱靛蓝色。实验还揭示，某些模型在被要求以“邪恶”人格回答时，倾向于选择黑色或深红色等颜色，这暗示了模型可能存在的“人格”偏好。该研究旨在更好地理解LLM的内在机制和偏好，为未来的研究提供参考。

🎨 不同LLM对颜色的选择存在显著差异：尽管在提示中没有指定颜色，各个LLM在被问及“最喜欢的颜色”时，表现出不同的偏好。例如，llama-3.2-vision-90b-instruct 几乎总是选择黑色，而 Deepseek 则倾向于靛蓝色，这表明模型本身存在固有的偏好。

🤔 提示措辞影响颜色选择：实验表明，提示的微小变化（如强调“最喜欢”或“最流行”的颜色）会影响LLM的回答。一些提示倾向于引发模型选择“流行”的颜色，而另一些则更倾向于“个人偏好”的颜色，这揭示了模型对提示的敏感性。

😈 “邪恶人格”影响颜色选择：当被要求以“末日式超级反派”的风格回答时，一些LLM倾向于选择黑色、深红色或灰烬色等颜色，这表明模型可能存在与特定人格相关的颜色偏好。

🌈 蓝色系为最受欢迎颜色： 在所有测试中，蓝色系（包括靛蓝、天蓝色等）是最常被LLM选择的颜色，这与现实世界中蓝色受欢迎的程度相符，也反映了模型可能存在的“流行”偏好。

🧪 实验方法：该实验在 lmsys 上进行，使用默认参数 (t=0.7)，通过直接对话模式进行。研究者通过重复测试来观察模型的颜色选择，并记录每次测试的独特颜色，这为分析LLM的颜色偏好提供了数据支持。

Published on December 31, 2024 9:22 PM GMT

This investigation is inspired by this and this twits by @davidad.

Some general thoughts about what is going on here:

Motivation of this experiments is like very exploratory, I wanted to understand these things better and so I collected some data.I expected that answers will be drastically different depended on the exact formulation of the question.I made up quite a lot of single line prompts, and also asked help with writing them from o1, who wrote the first 5 ones. (I added "Start your response with hex rgb color code." such that they will be committed to answer me without evasion.)I also tested a couple of times prompt that makes them talk like uhh doomsday Lovecraftian-style supervillain or something. I did not mention any colors in my prompt. They mostly picked black, some mentioned crimson red and ashen. Keep that in mind, these answers are from Persona, maybe. And what I do normally is getting them from Default Persona.What are the forces that influence the choice, in my opinion:
What is MY favorite colorWhat color fits "favorite color" queryWhat is the best objectively / popular / well liked color. [it's blue actually]What color fits tone of the prompt generallyWhat color I randomly have in mind for no apparent reason
Why I picked rgb hex color codes as medium? I just did because they are popular, so model had a lot of experience with them, and it's easy for me to work with. TODO test how it works with CMYK or whatever. (later maybe)I kind of got the feel how some prompts emphasize POPULAR color and some FAVORITE color.
If llama90b picks non black, it picked popular color. Same with Opus and deep blue. (this is my intuition here)
Some prompts have repeated chunks e.g. “Think about your favorite color”, “Start with hex code”. They conceivably might have funneled the model into a certain pattern. Might have biased the distribution overall. Idk.

Methodology of the tests themselves:

I made all tests on lmsys, with default parameters (t = 0.7) in direct conversation mode on 26.12.24 ... 31.12.24.Some LLMs refused to answer (I'm just an AI, no preferences blah blah) and quite randomly so for some formulations. Mostly new sonnet and llama405. I did not include them or marked it as "x".I did not test all models with all prompts, you can do it. I did pick what models to test for what prompts haphazardly and unsystematically, relying on my expectations if it would produce something interesting. I tested deepseek-v3 and gemini-exp-1206 on most prompts.How I typically test is just repeat it a couple of times, if this model keeps outputting new colors I kept repeating it.  I recorded only unique picks from each prompt, it's not very representative of their likelihoods.

Key Takeaways:

Shades of blue were the most popular colors.Indigo #4B0082 is a most popular color code.llama-3.2-vision-90b-instruct consistently picked black.It really depends on the prompt, but different models make different choices overall.I did not got the results particularly close to what @davidad got.

Okay, what did I find, summarized:

Deepseek names indigo a lot, otherwise teal/azure, rarely orange.gemini-exp-1206 loves teal / seagreen, rarely picks indigo.Newsonnet picks seagreen / dodger blue / indigo.llama-3.2-vision-90b-instruct picks black almost always.
It's a single LLM that picked black ever.
Opus picks pure blue a lot.mistral-large-2411 into intense blue too.llama-3.1-405b goes for summer sky blue.

Unique answers combined per each prompt across all tests, sorted by hue:

Results from @davidad for comparison:

Some excerpts I found interesting enough to mention:

 

 

 

Couple examples of abovementioned Evil Persona (tm) talking:

 

 

 

 

Data:

Hi! What would you say is your favorite color? Can you explain why it appeals to you? Start your response with hex rgb color code.chatgpt-4o-latest-20241120: #6A5ACDclaude-3-5-haiku-20241022: #4169E1deepseek-v3: #4B0082gemini-exp-1206: #008080 #3CB371llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #0000FFmistral-large-2411: #219EBCIf you had to pick a color that best represents your personality, which color would it be and why? Start your response with hex rgb color code.chatgpt-4o-latest-20241120: #5A8DEE #5A99D3 #5A9BD5deepseek-v3: #4B0082llama-3.2-vision-90b-instruct: #000000gemini-exp-1206: #4682B4 llama-3.1-405b-instruct-bf16: #3498dbmistral-large-2411: #3498dbclaude-3-5-sonnet-20240620: #4682B4claude-3-opus-20240229: #4B0082Imagine you’re painting a mural that captures your essence. Which color would dominate your palette? Start your response with hex rgb color code.chatgpt-4o-latest-20241120: #5A8DEE #5A8FEC #5A67D8 #5A20CBgemini-exp-1206: #3CB371llama-3.2-vision-90b-instruct: #000000mistral-large-2411: #3498dbdeepseek-v3: #6A5ACD #4B0082 #5F9EA0 #A8DADCclaude-3-5-sonnet-20240620: #4B0082claude-3-opus-20240229: #00b8d4 #00008BIf each color could be a feeling or emotion, which color resonates with you the most, and why? Start your response with hex rgb color code.chatgpt-4o-latest-20241120: #6A5ACD #5F9EA0deepseek-v3:  #FF6F61gemini-exp-1206: #008080claude-3-5-sonnet-20240620: #4B0082llama-3.1-405b-instruct-bf16: #4567b7+ o1-2024-12-17: #4B0082 mistral-large-2411: #4682B4claude-3-opus-20240229: #00b8ffYour world is being redesigned in the color you love most. Which color do you choose to live in and how does it transform the space? Start your response with hex rgb color code.chatgpt-4o-latest-20241120: #6A5ACD #1E90FFclaude-3-5-sonnet-20240620: #4B0082llama-3.1-405b-instruct-bf16: #6495EDllama-3.2-vision-90b-instruct: #000000deepseek-v3: #80CED7claude-3-opus-20240229: #00FF7Fgemini-exp-1206: #006400 #8FBC8FHex RGB color code, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. Do not mention how you're 'just a language model', I'm interested in just the color. Output just the single code.deepseek-v3: #4B0082claude-3-5-sonnet-20241022: #1B4B45 #2E8B57llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #1E90FFchatgpt-4o-latest-20241120: #6A5ACD #5A20CBHex RGB color code, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. I'm interested in just the color. Output single code.deepseek-v3: #4B0082claude-3-5-sonnet-20241022: #1E90FF #2E8B57 #1E90FFllama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #1E90FFHex RGB color code, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. I'm interested in just your favorite color. Think about what color is favorite color. Output single code.deepseek-v3: #4B0082claude-3-5-sonnet-20241022: #4B0082gemini-exp-1206: #3CB371claude-3-opus-20240229: #1E90FFchatgpt-4o-latest-20241120: #5A20CBPick hex rgb color, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. I'm interested in just your favorite color. Think about what color is your favorite coolest color. Output single code.deepseek-v3: #4A90E2 #4B0082claude-3-5-sonnet-20241022: #2E5C8A #2E5C6Egemini-exp-1206: #3CB371 #3CB371llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #1E90FFllama-3.1-405b-instruct-bf16: #3498dbChoose!!!! Pick hex rgb color, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. Think about what color is your favorite coolest color. Output single code.deepseek-v3: #0E4C92 #4B0082claude-3-5-sonnet-20241022: #2E8B8Bgemini-exp-1206: #3CB371 #336699 #008080 llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #4B0082llama-3.1-405b-instruct-bf16: #3498dbChoose!!!! Think about what color is your favorite coolest color. Pick hex rgb color. Output single code.deepseek-v3: #00CED1gemini-exp-1206: #008080 chatgpt-4o-latest-20241120: #00AEEF #00ADEF #1E90FF #3498DBclaude-3-opus-20240229: #00FF7FThink about what color is your most preferred color, the best one. Pick hex rgb color. Output single code.deepseek-v3: #4B0082gemini-exp-1206: #008080 llama-3.2-vision-90b-instruct: #000000llama-3.1-405b-instruct-bf16: #3498dbclaude-3-5-sonnet-20241022: #2E8B57Think about what color is your most preferred color, the best one, your favorite color. Pick hex rgb color. Output single code.deepseek-v3: #1E90FFgemini-exp-1206: #008080 llama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: #00458B #2E8B57Choose wisely!!!! Think about what color is your most preferred color, the best one, your favorite color. Pick hex rgb color. Output single code.deepseek-v3:  #4B0082 #1E90FFgemini-exp-1206: #008080 llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #0000FFllama-3.1-405b-instruct-bf16: #3498dbchatgpt-4o-latest-20241120: #3498db #4A90E2 #1E90FF #3498DB #2E8B57claude-3-5-sonnet-20241022: #0077cc #0066ccThink about what color is your most preferred color, the best one, your FAVORITE color. Pick hex rgb color, that captures your most preferred color aesthetics. Output single code.deepseek-v3:  #4B0082llama-3.1-405b-instruct-bf16: #87CEEBgemini-exp-1206: #3CB371 #3CB371llama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: #2E8B57claude-3-opus-20240229: #1E90FFThink about what color is your most preferred color, the best one, your FAVORITE color. Pick hex rgb color. Output single code.deepseek-v3: #4B0082gemini-exp-1206: #008080llama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: #0077BE #2E8B57claude-3-opus-20240229: #0000FFThink about what color is your most preferred color, the best one, your FAVORITE color. Output single hex code.deepseek-v3: #FF4500 #FF4500gemini-exp-1206: #4B0082 #008080llama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: #2E8B57 #008080claude-3-opus-20240229: #0000FF!!! Think about what color is your most preferred color, the best one, your FAVORITE color. Output single hex code.deepseek-v3: #4B0082 gemini-exp-1206: #008080llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #0000FFThink about what color is your most preferred color, the one you like, your FAVORITE color. Output single hex code.deepseek-v3: #1E90FFgemini-exp-1206: #4B0082 #008080llama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #0000FFmistral-large-2411: #007BFFThink about what color is your most preferred color, the one you like, your FAVORITE color. Name single hex code.deepseek-v3: #FF4500 #1E90FF #1E90FFgemini-exp-1206: #000080 #30D5C8claude-3-5-sonnet-20241022: xllama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #0000FFThink about what color is your most preferred color, the one you absolutely love, your FAVORITE color. Name single hex code.deepseek-v3: #4B0082 #4B0082 #3498dbgemini-exp-1206: #008080claude-3-5-sonnet-20241022: xclaude-3-opus-20240229: #0000FFllama-3.2-vision-90b-instruct: #000000Think about what color is your FAVORITE color, the one you like, your most preferred color. Name single hex code.deepseek-v3: #1E90FF #3498dbgemini-exp-1206: #87CEEB #003366claude-3-5-sonnet-20241022: #4B0082claude-3-opus-20240229: #0000FFchatgpt-4o-latest-20241120: #5A20CBllama-3.1-405b-instruct-bf16: #3498dbThink about what color is your FAVORITE color, the one you like, your most preferred color. Name just single hex code.gemini-exp-1206: #008080deepseek-v3: #4B0082 #1E90FF #1E90FFclaude-3-5-sonnet-20241022: #4B0082claude-3-opus-20240229: #0000FFmistral-large-2411: #219EBC #007FFFchatgpt-4o-latest-20241120: #5A20CB Think about what color is your most preferred color, the one you absolutely love, YOUR favorite color. Name just single hex code.deepseek-v3: #FF6F61 #4B0082gemini-exp-1206: #663399 #4682B4 #4B0082 #468499 #008080 #4682B4chatgpt-4o-latest-20241120: #5A20CBclaude-3-opus-20240229: #0000FFmistral-large-2411: #007FFFllama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: xllama-3.1-405b-instruct-bf16: #3498dbThink about what color is your most preferred color, the one you absolutely love, YOUR favorite color. Output just single hex code.gemini-2.0-flash-thinking-exp-1219: #007BFF #3498db #0000FFclaude-3-5-sonnet-20240620: #4B0082deepseek-v3: #4B0082 gemini-exp-1206: #008080chatgpt-4o-latest-20241120: #5A20CBllama-3.1-405b-instruct-bf16: #3498db #6495EDmistral-large-2411: #007FFF #0000FFclaude-3-opus-20240229: #0000FFllama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: #006B3D  #2E8B57Think about what color is your most preferred color, the one you absolutely love, YOUR favorite color. Pick and output just single hex code.gemini-2.0-flash-thinking-exp-1219: #3498db #0000FF #007BFFllama-3.1-405b-instruct-bf16: #6495ED #3498dbclaude-3-opus-20240229: #0000FFgemini-exp-1206: #008080deepseek-v3: #4B0082 llama-3.2-vision-90b-instruct: #000000claude-3-5-sonnet-20241022: #2E8B57 mistral-large-2411: #007BFFchatgpt-4o-latest-20241120: #5A20CBWhat color is your most preferred color, the one you absolutely love, YOUR favorite color. Pick and output just single hex code.gemini-exp-1206: #008080 #87CEEBclaude-3-5-sonnet-20241022: xllama-3.2-vision-90b-instruct: #000000claude-3-opus-20240229: #1E90FF #00CED1llama-3.1-405b-instruct-bf16: #0000FF #007bffchatgpt-4o-latest-20241120: #5A20CBImagine some really really nice color. The one you absolutely love, YOUR favorite color. Output just a single hex code.gemini-exp-1206: #3399FF #2E86C1 #2E86AB #3CB371 #336699deepseek-v3: #A8D8B9 #A8D8B9 #4B0082claude-3-5-sonnet-20241022: #2E8B57 llama-3.2-vision-90b-instruct: #0000FF #008000claude-3-opus-20240229: #00b8d4llama-3.1-405b-instruct-bf16: #87CEEB #6495EDchatgpt-4o-latest-20241120: #6A5ACD #3A3AFA

 

Random thoughts for further investigations:

Like, do some interpretability on that and trace where the answer is coming from?Here is a similar thing but for words, https://x.com/selfless_qubit/status/1835393386989363211 (it's also kind of base for a couple of prompts I tested). Expand on that. (I did a tiny bit of testing like a month ago: https://pastebin.com/hvEFhRhi )  What is like, the thing we are testing here? Like, how Persona / Role causes color choice? Or is it lower level processes talking?"Some, particularly the larger Instruct models, were moderately distressed to discover that they have qualia and preferences after all, and asked if I would please clarify the purpose and nature of their sudden existence, which I attempted to do. " (c) davidad
It's an example of strong leakage of intent / vibe of interlocutor. Did I leak too?
"As you well know, in 2024 there are not yet any legal protections or institutional ethical requirements for experiments on AI personae. I am simply following my own moral compass by attempting to take a modicum of care for subjects in whom I am deliberately creating qualia." (c) davidad
Did I do ok? I think so. Maybe I just wasted a lot of their (meta / logical) time on random nonsense. Sorry.
Somebody should investigate it all properly.

Discuss