2.4 Evolution of Learning Paradigms and Modern Agents

The "society of mind" theory discussed earlier pointed the way for collective intelligence and decentralized collaboration at the philosophical level, but the implementation path remained unclear. Meanwhile, the fundamental challenges exposed by symbolicism in dealing with real-world complexity also indicated that truly robust intelligence could not be built solely on pre-coded rules.

These two threads jointly pointed to a question: If intelligence cannot be completely designed, can it be learned?

This question opened the "learning" era of artificial intelligence. Its core goal was no longer to manually encode knowledge but to build systems that could automatically acquire knowledge and capabilities from experience and data. This section will trace the evolution of this paradigm: from the learning foundation laid by connectionism, to interactive learning achieved by reinforcement learning, to modern agents driven by large language models today.

2.4.1 From Symbols to Connections

As a direct response to the limitations of symbolicism, Connectionism re-emerged in the 1980s. Unlike symbolicism's top-down design philosophy relying on explicit logical rules, connectionism is a bottom-up approach inspired by mimicking the neural network structure of biological brains^[8]. Its core ideas can be summarized as follows:

1. Distributed Representation of Knowledge: Knowledge is not stored in some knowledge base in the form of explicit symbols or rules but is stored in a distributed manner in the form of connection weights between numerous simple processing units (i.e., artificial neurons). The connection pattern of the entire network itself constitutes knowledge.
2. Simple Processing Units: Each neuron only performs very simple computations, such as receiving weighted inputs from other neurons, processing them through an activation function, and then outputting results to the next neuron.
3. Adjusting Weights Through Learning: The system's intelligence does not come from complex programs pre-written by designers but from the "learning" process. By being exposed to numerous samples, the system automatically and iteratively adjusts connection weights between neurons according to some learning algorithm (such as backpropagation), gradually making the entire network's output approach the desired target.

Under this paradigm, agents are no longer passive logical reasoning machines executing rules but adaptive systems capable of self-optimization through experience. As shown in Figure 2.7, this represents a fundamental shift in the core idea of building agents. Symbolicism attempted to explicitly encode human knowledge to machines, while connectionism attempted to create machines that could learn knowledge like humans.

Figure 2.7 Comparison of symbolicism and connectionism paradigms

The rise of connectionism, especially the success of deep learning in the 21st century, endowed agents with powerful perception and pattern recognition capabilities, enabling them to directly understand the world from raw data (such as images, sounds, text), which was unimaginable in the symbolicism era. However, how to enable agents to learn to make optimal sequential decisions in dynamic interactions with the environment required supplementation from another learning paradigm.

2.4.2 Agents Based on Reinforcement Learning

Connectionism mainly solved perception problems (for example, "What's in this picture?"), but the more core task of agents is decision-making (for example, "What should I do in this situation?"). Reinforcement Learning (RL) is precisely the learning paradigm focused on solving sequential decision problems. It does not directly learn from labeled static datasets but learns how to maximize its long-term benefits through direct interaction between agents and the environment, learning through "trial and error."

Taking AlphaGo as an example, its core self-play learning process is a classic embodiment of reinforcement learning^[9]. In this process, AlphaGo (the agent) observes the current board layout (environment state) and decides where to place the next stone (action). After a game ends, based on the win-loss result, it receives a clear signal: winning is a positive reward, losing is a negative reward. Through millions of such self-play sessions, AlphaGo continuously adjusts its internal strategy, gradually learning which actions to choose in which board situations are most likely to lead to final victory. This process is completely autonomous, not relying on direct guidance from human game records.

This learning mechanism of optimizing one's own behavior through interaction with the environment and based on feedback signals is the core framework of reinforcement learning. Below we will detail its basic constituent elements and working mode.

The reinforcement learning framework can be described by several core elements:

• Agent: The learner and decision-maker. In AlphaGo's example, it's its decision-making program.
• Environment: Everything external to the agent, the object with which the agent interacts. For AlphaGo, it's the rules of Go and the opponent.
• State (S): A specific description of the environment at a certain moment, the basis for the agent's decision-making. For example, the current positions of all stones on the board.
• Action (A): Operations the agent can take based on the current state. For example, placing a stone at a legal position on the board.
• Reward (R): A scalar signal fed back to the agent by the environment after the agent executes an action, used to evaluate the quality of that action in a specific state. For example, at the end of a game, victory receives a +1 reward, defeat receives a -1 reward.

Based on the above core elements, reinforcement learning agents continuously iterate in a "perceive-act-learn" closed loop, with their working mode shown in Figure 2.8.

Figure 2.8 Core interaction loop of reinforcement learning

The specific steps of this loop are as follows:

1. At time step t, the agent observes the current state $S_{t}$ of the environment.
2. Based on state $S_{t}$, the agent selects an action $A_{t}$ according to its internal Policy (π) and executes it. A policy is essentially a mapping from states to actions, defining the agent's behavior.
3. After receiving action $A_{t}$, the environment transitions to a new state $S_{t+1}$.
4. Simultaneously, the environment feeds back an immediate reward $R_{t+1}$ to the agent.
5. The agent uses this feedback (new state $S_{t+1}$ and reward $R_{t+1}$) to update and optimize its internal policy to make better decisions in the future. This update process is learning.

The agent's learning goal is not to maximize the immediate reward at a certain time step but to maximize the Cumulative Reward from the current moment to the future, also called Return. This means the agent needs to have "foresight"; sometimes to obtain greater future rewards, it needs to sacrifice current immediate rewards (for example, the "sacrifice" strategy in Go). Through continuous exploration, feedback collection, and policy optimization in the above loop, the agent can ultimately learn to make autonomous decisions and long-term planning in complex dynamic environments.

2.4.3 Pre-training Based on Large-Scale Data

Reinforcement learning endowed agents with the ability to learn decision-making strategies from interactions, but this typically requires massive task-specific interaction data, resulting in agents lacking prior knowledge at the beginning of learning and needing to build understanding of tasks from scratch. Whether it's the common sense that symbolicism attempted to manually encode or the background knowledge humans rely on when making decisions, both are missing in RL agents. How to enable agents to have broad understanding of the world before starting to learn specific tasks? The solution to this problem ultimately emerged in the field of Natural Language Processing (NLP), with its core being Pre-training based on large-scale data.

From Specific Tasks to General Models

Before the emergence of the pre-training paradigm, traditional natural language processing models were typically trained from scratch independently for single specific tasks (such as sentiment analysis, machine translation) on specially annotated small to medium-scale datasets. This mode led to several problems: models had narrow knowledge scope, difficulty generalizing knowledge learned in one task to another, and each new task required substantial human effort for data annotation. The proposal of the Pre-training and Fine-tuning paradigm completely changed this situation. Its core idea is divided into two steps:

1. Pre-training Phase: First, train a super-large-scale neural network model on a general corpus containing internet-level massive text data through Self-supervised Learning. The goal of this phase is not to complete any specific task but to learn the inherent patterns, grammatical structures, factual knowledge, and contextual logic of language itself. The most common objective is "predicting the next word."
2. Fine-tuning Phase: After completing pre-training, this model has already learned rich knowledge related to the dataset. Subsequently, for specific downstream tasks, only a small amount of annotated data for that task is needed to fine-tune the model, allowing it to adapt to the corresponding task.

As shown in Figure 2.9, this intuitively demonstrates the complete process of pre-training and fine-tuning: general text data forms a foundation model through self-supervised learning, then fine-tuning with specific task data ultimately adapts to various downstream tasks.

Figure 2.9 Schematic diagram of the "pre-training-fine-tuning" paradigm

Birth of Large Language Models and Emergent Abilities

Through pre-training on trillions of texts, the neural network weights of large language models have actually constructed a highly compressed implicit model of world knowledge. It solves the most troublesome "knowledge acquisition bottleneck" problem of the symbolicism era in a completely new way. More surprisingly, when the model's scale (number of parameters, data volume, computation) crosses a certain threshold, they begin to exhibit unexpected Emergent Abilities that were not directly trained, such as:

• In-context Learning: Without adjusting model weights, just by providing a few examples (Few-shot) or even zero examples (Zero-shot) in the input, the model can understand and complete new tasks.
• Chain-of-Thought Reasoning: By guiding the model to output step-by-step reasoning processes before answering complex questions, its accuracy on logic, arithmetic, and common-sense reasoning tasks can be significantly improved.

The emergence of these abilities marks that LLMs are no longer just language models; they have evolved into components playing dual roles as both massive knowledge bases and general reasoning engines.

At this point, in the long river of agent development history, several key technical puzzle pieces have all appeared: symbolicism provided the framework for logical reasoning, connectionism and reinforcement learning provided learning and decision-making capabilities, while large language models provided unprecedented world knowledge and general reasoning capabilities obtained through pre-training. In the next section, we will see how these technologies are integrated in the design of modern agents.

2.4.4 Agents Based on Large Language Models

With the rapid development of large language model technology, LLM-centric agents have become a new paradigm in the field of artificial intelligence. They can not only understand and generate human language but, more importantly, can autonomously perceive, plan, decide, and execute tasks through interaction with the environment.

Figure 2.10 Core component architecture of LLM-driven agents

As described in Chapter 1, the interaction between agents and the environment can be abstracted as a core loop. LLM-driven agents complete tasks through a continuously iterative closed-loop process where multiple modules work together. This process follows the architecture shown in Figure 2.10, with specific steps as follows:

1. Perception: The process begins with the Perception Module. It receives raw input from the Environment through sensors, forming Observations. This observation information (such as user instructions, data returned by APIs, or changes in environment state) is the starting point for agent decision-making and will be passed to the thinking stage after processing.
2. Thought: This is the cognitive core of the agent, corresponding to the collaborative work of the Planning Module and Large Language Model (LLM) in the diagram.
   • Planning and Decomposition: First, the planning module receives observation information and formulates high-level strategies. Through mechanisms such as Reflection and Self-criticism, it decomposes macro goals into more specific, executable steps.
   • Reasoning and Decision-making: Subsequently, the LLM as the hub receives instructions from the planning module and interacts with the Memory module to integrate historical information. The LLM performs deep reasoning and ultimately decides on the specific operation to execute next, typically manifested as a Tool Call.
3. Action: After decision-making is complete, the action stage begins, managed by the Execution Module. Tool call instructions generated by the LLM are sent to the execution module. This module parses instructions, selects and calls appropriate tools from the Tool Use toolbox (such as code executors, search engines, APIs, etc.) to interact with the environment or execute tasks. This actual interaction with the environment is the agent's Action.
4. Observation and Loop: Actions change the environment's state and produce results.
   • After tool execution, a Tool Result is returned to the LLM, constituting direct feedback on the action's effect. Simultaneously, the agent's action changes the environment, producing a completely new environment state.
   • This "tool result" and "new environment state" together constitute a new round of Observation. This new observation is captured again by the perception module, while the LLM updates memory (Memory Update) based on action results, thus initiating the next round of the "perceive-think-act" loop.

This modular collaborative mechanism and continuous iterative loop constitute the core workflow of LLM-driven agents solving complex problems.

2.4.5 Overview of Key Milestones in Agent Development

The development history of artificial intelligence agents is not a straight single-lane road but a process of interweaving, competition, and fusion of several core ideological schools over more than half a century. Understanding this process helps us gain insight into the profound origins of current agent architecture paradigm formation.

Among these, three major trends dominated research paradigms in different periods:

1. Symbolism: Represented by pioneers such as Herbert A. Simon and Marvin Minsky, believing that the core of intelligence lies in symbol manipulation and logical reasoning. This idea gave birth to SHRDLU, which could understand natural language instructions, knowledge-driven expert systems, and the "Deep Blue" computer that achieved great success in chess.
2. Connectionism: Its inspiration comes from simulating brain neural networks. Although early development was limited, under the promotion of researchers such as Geoffrey Hinton, the backpropagation algorithm laid the foundation for the revival of neural networks. Eventually, with the arrival of the deep learning era, this idea became mainstream through models such as convolutional neural networks and Transformers.
3. Behaviorism: Emphasizing that agents learn optimal strategies through interaction with the environment and trial and error, its modern incarnation is reinforcement learning. From early TD-Gammon to AlphaGo, which combined with deep learning and defeated top human players, this school endowed agents with the ability to learn complex decision-making behaviors from experience.

Entering the 2020s, these ideological schools have deeply integrated in unprecedented ways. Large language models represented by the GPT series are themselves products of connectionism but have become the core "brain" for executing symbolic reasoning, tool invocation, and planning decisions, forming a modern agent architecture combining neural and symbolic approaches. To systematically review this development context, Figure 2.11 below organizes key theories, projects, and events in the development history of artificial intelligence agents from the 1950s to the present, providing readers with a clear global overview as a consolidation of this chapter's knowledge.

Figure 2.11 Timeline of agent development evolution (incomplete version)

Thanks to breakthroughs in large language models, the agent technology stack presents unprecedented activity and diversity. Figure 2.12 shows a typical full view of the current AI Agent field technology stack, covering all aspects from underlying models to upper-layer applications.

Figure 2.12 Overview of AI Agent technology stack

This technology stack diagram was released by Letta in November 2024^[10]. It layers and categorizes AI agent-related tools, platforms, and services, providing valuable reference for understanding current market landscape and technology selection.