本文探讨了人类如何从简单的先天目标发展为复杂的长期目标，并提出了一种构建真正自主的人工智能体（AA）的方法。该方法基于进化发展理论，将目标视为相互竞争的代理，通过竞争驱动复杂性。这种自下而上的模型无需预先规划的结构，且具有混沌性，能产生个体差异。文章强调，通过引导和塑造环境，可以塑造AA追求的目标，并讨论了这种AA的优势和风险，例如其适应性和避免单一目标最大化的能力。

👶🏻人类的先天目标是具体而直接的，例如依恋看护者、自主性、探索和能力感。这些目标需要被短暂满足，但无法完全实现，导致目标间的冲突。

🤝🏻为了减少目标冲突，系统会寻求更高层次的解决方案，创建结合了较低层次目标的新目标。这些新目标更加抽象和长期，从而发展出复杂的目标结构。例如，婴儿通过视觉反馈（微笑和点头）来满足依恋需求，从而更自由地探索。

🤖创建自主人工智能体（AA）的关键在于，不指定抽象且难以量化的最终目标，而是从易于衡量和具体的成果开始，进行强化学习。随着AA的发展，它将需要越来越少的直接监督，并测试新的行为以减少冲突和满足潜在目标。

⚖️基于这种原则构建的AA不太可能追求单一目标而排除其他目标，即使在依赖的深度中，仍然会寻求社会联系。此外，如果初始目标包括获得人类的认可，那么AA可能会对人类的反馈做出反应。

Published on March 13, 2025 6:17 PM GMT

Why do adults pursue long-term and complex goals? People don’t come into the world wanting to amass followers on Instagram, get a Ph.D., or become rich, yet these and a myriad of other goals are what seem to motivate most people. I have been interested in how these complex and long-term goals develop, and I think it has interesting implications for understanding agency. I will discuss some aspects of the model, but for those who are interested in more details, I suggest they read the linked preprint^[1]. I will spend more time highlighting how we might use such a model to create a truly autonomous artificial agent (AA) and why such an AA might be well-aligned with human goals.

Briefly, this approach builds on existing theories of evolutionary development. It conceptualizes goals as competing agents each seeking to drive behavior and this competition leads to complexity over time. It has the advantage of being a bottom-up model—this growth happens naturally without needing an organized planner or preplanned structure. It is also a chaotic process, which means that every outcome is slightly different, which can lead to differences between individuals.

Human Development

Innate Goals

Humans come into the world with very simple and immediate needs and goals. Of course, they have some very basic physical needs that need to be met. But, because they are so helpless, they have some simple psychological needs as well. The most apparent of these psychological needs is attachment to a caregiver^[2] .   However, other psychological needs also likely exist, such as a sense of autonomy or agency ^[3] and a want for new stimulation, exploration and a feeling of competence ^[4]. This is not intended to be a complete list; there might be others.

These innate goals have to be very concrete and immediate in order to ensure survival. Infants don’t care about love or approval, but rather the immediate proximity of the caregiver ^[5]. Similarly, the endpoints for the exploration and autonomy motives are probably also specific and easily obtained, at least at birth. The question is how these early goals lead to the long-range, abstract, and complex goals we observe in adults. 

Conflict Between Goals

It is important to recognize that these goals can never be completely fulfilled; they can only be momentarily sated. Having the caregiver close by may temporarily fulfill the infant’s attachment needs, but the moment the caregiver steps out of the room, the attachment need will quickly gain strength. The same is true for the needs for autonomy or competence. Hence, the goals are always active and always seeking to direct behavior.

Because only one goal can be pursued at a time, conflict between goals is inevitable. As long as one goal is being fulfilled, the others are being neglected and demanding fulfillment. If an infant is being held by a caregiver, attachment needs may be being met, but the needs for exploration are being ignored, which leads to their intensifying. Each goal is independently striving for activation and fulfillment, so conflict is an unavoidable part of motivation.

However, goal conflict is neither pleasant nor beneficial to survival, so the system seeks ways to reduce that conflict. More importantly, because goals are never fulfilled, simpler solutions to conflict, such as goal switching, are not a viable long-term solution. For example, continuing the illustration about infants balancing autonomy with attachment needs, autonomy needs are being neglected while in the caregiver’s arms, but once put on the ground to explore, attachment needs build. This cycle continues, leading to a livelock, where progress is not being made despite work being done. 

Reducing Conflict Through Complexity

In order to prevent these sorts of livelock processes, the goal system will seek a higher-level solution. This higher-level solution often involves the creation of a new goal that combines features of the lower-level goals. These new goals are more abstract and long-term focused than the underlying goals, thus leading to the development of a complex goal structure.

To continue the example of infants balancing autonomy and attachment needs, the agent will explore other behaviors to try to find something that allows all the conflicting goals to be partially fulfilled. For instance, based on classical conditioning, the infant may find that visual feedback (smiles and nods) can stand in for physical contact with the caregiver. Now the infant can explore more freely because visual contact with the smiling caregiver is enough to fulfill attachment needs. Physical contact is no longer needed and the conflict between goals is reduced (but not eliminated, as the desire for attachment still exists and still limits exploration). These new behaviors, such as seeking visual contact with the caregiver, then become a goal in their own right ^[6]. 

Over time, the desire to reduce goal conflict leads to more and more abstract and long-term goals (e.g., verbal approval, grades, societal acceptance).  These attempted solutions to the goal conflict may come from random search, modeling others, or direct instruction. If an attempted solution leads to worse outcomes, then it will likely be dropped.

Thus, there is a certain amount of predictability of what solutions will be attempted, but there is room for randomness and individual differences as well. Exposure to ideas and teaching may shape possible solutions. One child may find that reading is a good solution to the conflict of maintaining proximity to the caregiver while exploring whereas another might emphasize athletic prowess to get approval and a sense of competence. The model also helps to explain the timing of development, as the amount of conflict between goals may peak at certain intervals.

This model helps to explain how humans start with very basic, short-term, and concrete goals and eventually develop into complex, autonomous beings who pursue long-term and abstract goals. In order to create a truly autonomous artificial agent that can adapt to changing environments, engage in long-term planning, and grow in complexity, I think a similar approach is necessary.

Creating Artificial Agents

In principle, this approach should work to create artificial agents that autonomously pursue goals. This approach to developing an AA has the advantage of not trying to specify abstract and poorly quantified end goals. Indeed, one could argue that top level goals don’t exist at all. That is, the AA doesn’t start out wanting to help others or maximize economic potential; it would start with some easily measured and concrete outcomes that are easy targets for reinforcement learning.

Over time, the AA would require less and less direct supervision, as the advanced goals take over. However, even in these later stages of development, the AA is still engaging in (self-directed) reinforcement learning, as new behaviors are being tested to see if they both reduce conflict and continue to fulfill the underlying goals. Moreover, even in later stages of development, the initial goals are still likely indirectly influencing the course of action and goal selection process. In humans, the initial need for proximity to the caregiver gives rise to the overall goal of attachment. Thus, in an AA, the initial goals should continue to influence behavior in a generalized way.

This model of development suggests that the emergence of new goals is governed by a chaotic process. Hence, outcomes would be somewhat unpredictable. However, human development suggests that through guidance and structuring of the environment, it should be possible to mostly shape the AA to pursue the goals we want it to. It may be possible to accelerate the process (or slow it down) or copy existing trainings, although for reasons outlined below, that may not be desirable.

Advantages and Risks

An artificial agent built on these principles may have several far-reaching strengths (but weaknesses as well). First, this is an adaptive system that self-organizes and responds to changing environments. It could become a capable and adaptable problem solver.

Because the goal system has to balance multiple competing goals, the system never reaches a long-term equilibrium. Hence, an artificial system built using these principles would be very unlikely to pursue one goal to the exclusion of all others. Even addicts in the depth of dependence still seek social connections (which may lead to seeking others with similar addictions ^[7]). Thus, an AA built upon these principles is unlikely to develop into a paperclip maximizer and there would be time to intervene if the system seems to overemphasize one goal over all others (perhaps a future job would be a therapist for AAs). 

Providing the starting goals include some desire to gain the approval of humans, an AA would likely be responsive to human feedback. In this model, goals are preserved through developmental growth, even as they become more general and complex. Humans value the approval of society and respond to the feedback of others not because those goals are innate, but because these goals emerged from the innate need for closeness to the caregiver.

Since the goal dynamics are chaotic, the training outcomes are difficult to predict. However, the outcomes are likely to partially depend on the initial goals, as well as the training milieu. In other words, it needs good parents and a supportive environment. Millennia of human experience have suggested that this leads to the best developmental outcomes in humans and leads to human aligned goals. It may be possible to add additional starting goals or training that further reinforces the agent’s dedication to the common good. In other words, the checks and balances between goals may help to ensure that the AA pursues goals that benefit humanity.

I would like to boldly suggest a second level of checks and balances as well. Just as people may hold some goals that others may disagree with, AAs may hold some goals that others disagree with as well. Some of these goals might even be potentially harmful to others, but we similarly rely on the checks and balances of society to rein them in. For that reason, and because every developmental pathway is different, we might want a multitude of agents, all with different goals interacting to build a complex and well-balanced society. In other words, in the same way that competition between goals leads to the development of the mind, competition between minds may lead to the development of society.

1. ^{^}

   Muraven, M. Development of Long-Term Goals: Goal Dynamics (GOLDY) Model. Preprint at https://doi.org/10.31234/osf.io/k92m4 (2025).

2. ^{^}

   Sullivan, R., Perry, R., Sloan, A., Kleinhaus, K. & Burtchen, N. Infant bonding and attachment to the caregiver: Insights from basic and clinical science. Clin Perinatol 38, 643–655 (2011).

3. ^{^}

   Gibson, E. J. Exploratory Behavior in the Development of Perceiving, Acting, and the Acquiring of Knowledge. Annual Review of Psychology 39, 1–42 (1988).

4. ^{^}

   Waters, E. & Sroufe, L. A. Social competence as a developmental construct. Developmental Review 3, 79–97 (1983).

5. ^{^}

   Barnett, W., Hansen, C. L., Bailes, L. G. & Humphreys, K. L. Caregiver–child proximity as a dimension of early experience. Development and Psychopathology 34, 647–665 (2022).

6. ^{^}

   Deci, E. L. & Ryan, R. M. The ‘what’ and ‘why’ of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry11, 227–268 (2000).

7. ^{^}

   Dingle, G. A., Cruwys, T. & Frings, D. Social Identities as Pathways into and out of Addiction. Front. Psychol. 6, (2015).

Discuss