1. Text Tasks (Text / NLP / LLM)

Among AI capabilities, text tasks are the most fundamental. Whether we ultimately want to do content moderation, search and recommendation, knowledge Q&A, or writing assistants and code copilots, it all essentially comes down to one question: how to make machines truly understand text.

1.1 Foundational Language Modeling and Representation

Let's start from the most fundamental layer: foundational language modeling and representation. Its role is to first familiarize the machine with language in a statistical sense and, on that basis, find a stable vector/matrix representation for words, sentences, and documents, so as to facilitate downstream tasks such as classification, matching, extraction, and generation. No matter what text-related task you want to do in the future, you will more or less need to answer the same question first: how do I represent this passage with a string of numbers?

We can look at this topic from three angles: scenarios, principles, and models:

• Scenarios
  • Retrieval and search related
    • General search engines: users casually type a sentence and get documents relevant in meaning, rather than just exact keyword matches.
    • On-site search / e-commerce search: users use colloquial descriptions (e.g., "a white shirt suitable for summer commuting") and find products matching the meaning.
    • Document library / knowledge base retrieval: in technical documentation, policy regulations, or enterprise knowledge bases, directly enter a sentence to retrieve relevant entries.
  • Recommendation and ranking related
    • Information feed / content recommendation: based on content users have recently viewed or clicked, automatically find other content with similar meaning to continue recommending, rather than relying solely on manual rules or tags.
    • E-commerce / product recommendation: based on product descriptions users have viewed, purchased, or favorited, find products with similar style or use to provide personalized recommendations.
    • User interest modeling: based on titles users have viewed, search terms they've used, etc., summarize several main interest directions to improve recommendation and ranking effectiveness.
  • Q&A assistant related
    • FAQ Q&A: users ask the same question in different ways ("How do I get an invoice?" vs "Where do I issue an invoice?"), and the system can jump to the same answer.
    • Knowledge base Q&A / enterprise assistant: users ask questions in natural language, and the system matches by meaning within internal documents to find the most relevant passage to answer.
  • Text understanding and analysis related
    • Comment sentiment analysis: roughly classify a large volume of comments and posts by "what they're about / what the sentiment is."
    • Text deduplication / similarity detection: used to discover rewritten articles and pseudo-original content.
    • Document clustering / grouping: group many articles and reports by content similarity into several clusters, facilitating navigation, recommendation, or sampling checks.
  • Serving as general-purpose features for downstream tasks (downstream tasks refer to using the model's foundational capabilities to implement more specific text processing tasks)
    • Text classification: downstream models for sentiment classification, intent recognition, spam identification, etc., directly reuse the representations from this layer.
    • Information extraction: entity recognition and relation extraction are fine-tuned on top of word/sentence representations rather than trained from scratch.
    • Text generation: provide semantic representation input for generation tasks such as summarization, rewriting, and continuation, improving generation quality and controllability.
• Principles Learning representations of words, sentences, and documents as the foundation for subsequent, more complex tasks.
  • Language modeling
    • Autoregressive language models: predict the next token (GPT series, LLaMA, Qwen, etc.)
    • Masked language models (Masked LM): predict masked tokens (BERT, RoBERTa, ERNIE)
  • Word / sentence / paragraph representation
    • Static word vectors: Word2Vec, GloVe, FastText
    • Contextual representations: BERT embedding, Sentence‑BERT, etc.
    • Document-level vectors: used for semantic retrieval and similarity matching
• Models BERT / RoBERTa / ERNIE, the GPT family, LLaMA / Qwen / Yi and other LLMs; various Embedding models (OpenAI text‑embedding‑3 series, bge, E5, SimCSE, etc.).

1.1.1 Language Modeling: Learning Language by "Guessing the Next Word"

The first step at this layer is to let the model become familiar with linguistic patterns through massive amounts of text. The approach can be simply understood as: give the model countless "fill-in-the-blank" exercises — after seeing the context of a passage, have it fill in the most reasonable word (token). With enough practice problems and sufficiently broad corpora, the model gradually learns: what a natural sentence looks like, which words frequently appear together, and what expressions sound awkward. This process is called "language modeling," which is essentially a unified word-guessing training mechanism.

There are two common ways of posing the question, each illustrated with a simple one-sentence example:

1. Continue forward (autoregressive): only give the preceding content and have the model guess "what comes next."
2. Input prefix: It's raining today, so I
3. Model task: guess the next word, such as "brought (an umbrella)," "didn't (go out)," "plan to (stay home)," etc., and then continue generating further. This approach primarily trains the model's grasp of continuation, coherence, and common expressions.
4. Fill in the blank (masked): remove a word from the middle and have the model use both preceding and following context to fill in the blank.
5. Original sentence: It's raining today, so I brought an umbrella
6. Training sentence: Today [MASK] raining, so I brought an umbrella
7. Model task: fill [MASK] with a reasonable word like "is." Here the model must simultaneously look at the left context "Today" and the right context "so I brought an umbrella" to decide what to fill in, which is more conducive to learning whole-sentence semantics.

By repeatedly doing these two types of "word-guessing exercises" on massive corpora, the model gradually accumulates linguistic intuition and statistical common sense. On this foundation, the next step is to explicitly transform this capability into vector representations of words, sentences, and documents, laying the groundwork for subsequent tasks such as retrieval, recommendation, and Q&A.

1.1.2 Word, Sentence, and Document Representation: Mapping Discrete Symbols into Semantic Space

The earliest generation of methods for constructing text vectors was static word vectors: assigning each word a fixed vector that, once trained, does not change with context — intuitive and simple, but unable to distinguish the meanings of polysemous words in different contexts. To solve this problem, context-based dynamic representation methods later emerged: the same word in different sentences generates different vectors, entirely determined by its surrounding context. For example, "apple" in "Apple released a new phone" would lean toward the semantic direction of "tech company," while in "apples are rich in vitamins" it would be closer to the "fruit" concept.

This mechanism not only enhances representational capacity at the word level but also paves the way for vectorizing sentences and documents. For sentences, sentence vectors can be generated; for documents, the entire text can be encoded as input (if length permits), or encoded segment by segment and then aggregated into a global vector through attention mechanisms, hierarchical pooling, contrastive learning, or other methods. In recent years, dedicated embedding models (such as bge, E5, and the text-embedding series) have been continuously optimized around the goal of "making semantically similar texts closer in vector space," performing particularly well on tasks such as semantic retrieval and similarity matching.

This pipeline from contextual modeling to sentence/document vector generation has become the core infrastructure behind systems for search, recommendation, Q&A, and more, bringing us back to the various scenarios mentioned earlier:

• Retrieval and search scenarios (general search, e-commerce search, knowledge base retrieval) all require encoding both user input and candidate documents into vectors, then performing similarity matching in vector space to find the semantically closest results, rather than relying solely on exact keyword matching.
• Recommendation and ranking scenarios (information feed recommendation, product recommendation, user interest modeling) require converting the content corresponding to user historical behavior into vectors, then finding new content with similar vectors to recommend to the user, achieving the personalized effect of "viewed A, recommended B."
• Q&A assistant scenarios (FAQ Q&A, knowledge base Q&A) require encoding both user questions and the questions or passages in the knowledge base into vectors, then finding the best-matching answer through vector similarity.
• Text understanding and analysis scenarios (comment sentiment, deduplication, clustering) require first converting each piece of text into a vector, then performing clustering, similarity calculation, or classification based on the vectors.
• Downstream task scenarios (text classification, information extraction, text generation) directly use the vector representations from this layer as input features, feeding them to subsequent classifiers, extractors, or generators, avoiding the need to learn semantics from scratch.

In engineering terms, the common practice is to encapsulate this into a unified "text vector service": input any piece of text, output a fixed-dimension vector, shared across multiple systems such as search, recommendation, and Q&A. At the product level, the capabilities of this layer are mainly reflected in: semantic recall in search and recommendation (no longer relying solely on keywords, but recalling content that is "worded differently but similar in meaning" through vector similarity), as well as unified embedding/vector retrieval services for enterprise knowledge bases, FAQs, and case libraries.

1.2 Text Classification and Text Matching (Classification & Matching)

In the previous section, we found the "coordinates" of each piece of text in semantic space through foundational language modeling and representation. But coordinates alone are not enough — the questions that businesses truly care about are often: What category does this text belong to? Is it about the same thing as another piece of text? Do two sentences logically support or contradict each other? You can think of it this way: use the two capabilities of classification and matching to transform the underlying vector representations into labels and relevance signals that can directly drive business decisions. We'll again examine this layer from three angles: scenarios, principles, and models:

• Scenarios
  • Content understanding and moderation: tag comments, posts, and articles with labels for topic, sentiment, risk, etc., for use in moderation, recommendation, and statistical analysis.
  • Recommendation and ranking: decide what content to display and how high to rank it based on the degree of match between "user interest tags" and "content tags."
  • Search and FAQ: users casually enter a natural language question, and the system can automatically find the most relevant question-answer pair or document fragment.
  • Similar content identification: find "semantically similar" entries among large volumes of text for deduplication, merged statistics, and recommending "related content."
  • Logical relationship judgment: determine whether two sentences support each other, contradict each other, or are unrelated, for use in fact-checking, multi-turn conversation consistency checks, etc.
• Principles On top of semantic representations, make holistic judgments about an entire piece of text or text pairs:
  • Text classification: assign labels to individual pieces of text (such as sentiment, topic, risk type, etc.);
  • Text matching: determine the similarity or relevance between two pieces of text, or whether a "question–answer" pair matches;
• Models Based on pre-trained encoders, with simple classification / matching structures attached:
  • Single text classification: BERT / RoBERTa / DeBERTa + fully connected classification layer;
  • Text matching: Sentence‑BERT, SimCSE, two-tower (Bi‑Encoder), Cross‑Encoder;
  • Complex judgment: instruction fine-tune on LLMs to have the model directly output labels or logical relationships.

1.2.1 Text Classification: From "Understanding Content" to "Characterizing Content"

Leveraging the semantic representations from the previous layer, we can very naturally attach a simple classification head on top and, with a small amount of labeled data, have the model learn to answer one question: "What category does this text belong to?"

The most classic example is sentiment classification. A user's review might be praise, a complaint, or simply a statement of fact. After obtaining the vector representation of the sentence, the model only needs to attach a softmax classification layer to output the probabilities of "positive / negative / neutral." This capability is already very mature in scenarios such as e-commerce, social platforms, and app marketplaces.

Another major category is topic / industry classification. In news recommendation, we want to know whether an article is about sports, finance, or entertainment; in an enterprise's internal customer service / ticketing system, the concern is more about whether it's a product inquiry, a functional anomaly, or a complaint/suggestion. These labels can help route content more precisely to the appropriate workflow and can also serve as important features in the recommendation and ranking stage.

Going further, risk / compliance classification is directly tied to platform safety. We set up dedicated classification models for categories such as ad trafficking, abusive attacks, politically sensitive content, and vulgar/pornographic material, working in conjunction with human review to intercept or downgrade high-risk content. It can be said that the first gate of the vast majority of content safety strategies is built from these types of classifiers.

As we can see, by this layer we are already able to transform "abstract semantic representations" into several business-usable labels. Next, we will discuss: when relationships arise between pieces of text, how do we perform matching and inference.

1.2.2 Text Matching: "Finding the Most Suitable Match" for a Sentence

Unlike classification, which "characterizes individual pieces of text," text matching focuses on "the relevance between two pieces of text." In many products, this is often the key link in achieving "intelligence": whether the system can find the most suitable response in the knowledge base when a user says something depends entirely on matching quality.

The most fundamental aspect is semantic similarity calculation. We first use the embedding model from the previous layer to encode two sentences into vectors, then determine their distance in semantic space through cosine similarity, dot product, or other methods. Models like SimCSE and Sentence‑BERT are specifically designed, through contrastive learning, to pull "similar sentence pairs" closer and push "dissimilar sentence pairs" farther apart.

Building on this, paraphrase detection and plagiarism detection are simply matching tasks for specific application scenarios. The former is used for content deduplication, preventing platforms from being flooded with repetitive expressions; the latter is used in scenarios such as education and knowledge communities to identify highly similar answers or articles. Technically, both essentially involve binary classification or ranking based on text similarity.

A very important downstream application is Q&A matching. When a user poses a natural language question, we don't directly use keywords to match FAQs; instead, we first perform recall through semantic vectors, then use a finer-grained matching model (such as a Cross‑Encoder) to re-rank several candidates and select the most likely corresponding one. This pipeline forms the foundation of FAQ bots and document Q&A systems.

At this layer, we already have the ability to classify and judge relationships for "entire pieces of text." But in many scenarios, businesses are not satisfied with this alone — they further want to know: what specific entities are mentioned in this text, and what events occurred. This naturally leads to the topic of the next section — sequence labeling and information extraction.

1.3 Sequence Labeling and Information Extraction (Sequence Labeling & Information Extraction)

After completing the classification and matching of entire texts, we often encounter a more granular requirement: not only knowing "what this article is about and how risky it is," but also knowing "who it specifically mentions, where, when, and what the amount is." This section is a key step beyond holistic judgment toward "fine-grained structuring." You can think of it as: on the premise of already knowing "which category of text to look at and roughly what it's about," we mine entities, relations, events, and various fields from within the text, enabling unstructured text to be directly consumed by business systems. We'll again look at this layer from four aspects: scenarios, principles, models, and products:

• Scenarios
  • Industry text structuring: extract key fields such as person names, organizations, amounts, dates, and clauses from documents like contracts, reports, announcements, medical records, and policies, for storage and retrieval.
  • Knowledge graphs and relationship networks: identify entities and their relationships from news, papers, and Q&A, building graphs of "who has what relationship with whom" for search, recommendation, and analysis.
  • Invoice and document processing: automatically extract fields such as title, tax ID, amount, and date from invoices, reconciliation statements, and expense reports, reducing manual data entry.
  • Public opinion and event analysis: extract "who did what, when, and where" from massive volumes of text for event tracking, risk alerts, and statistical reporting.
  • Log and ticket structuring: extract key information from unstructured text such as customer service conversations, tickets, and system logs, facilitating statistics, monitoring, and automated processing.
• Principles Perform fine-grained labeling and structuring of text at the token / phrase level:
  • Sequence labeling: tag each token (e.g., person name, location, organization, product name, etc.) to achieve named entity recognition, part-of-speech tagging, phrase segmentation, etc.;
  • Relation and event extraction: on top of entities, identify relationships between "entity–entity" pairs and the event structure of "who did what, when, and where";
  • Business field extraction: around specific business schemas (e.g., contract fields, invoice fields), convert long documents into standardized key‑value pairs or record tables.
• Models Based on pre-trained representations, perform information extraction through structures such as sequence labeling or span extraction:
  • Sequence labeling models: BiLSTM‑CRF, BERT + CRF / Softmax, etc.;
  • Span‑based extraction: directly predict the start and end positions of entity / relation spans;
  • Document-level extraction: DocIE-type models that incorporate layout and formatting;
  • LLM-based extraction: through Prompt / Few‑shot, have large models extract required fields in a specified format.

1.3.1 Sequence Labeling: Attaching Semantic "Tags" to Each Token and Phrase

In the text classification stage, we only cared about what category an entire piece of text belongs to; in the sequence labeling stage, we need to label every token and every phrase in the text. The most typical task is Named Entity Recognition (NER): identifying specific types of entities such as person names, organization names, location names, product names, and disease names.

• For example, in the sentence "Zhang San joined a certain tech company in Beijing," label "Zhang San" as a person name, "Beijing" as a location, and "a certain tech company" as an organization.

In terms of modeling approaches, the traditional method uses sequence labeling structures like BiLSTM + CRF, while later approaches more commonly adopt BERT + CRF or BERT + Softmax, leveraging the contextual representation capability of pre-trained encoders to determine each token's label (e.g., B‑ORG, I‑ORG, O, etc.). In practice, NER models are often the first "preprocessing" step for subsequent knowledge graphs and relation extraction.

Beyond NER, part-of-speech tagging and phrase segmentation are also typical sequence labeling tasks. They serve more for underlying linguistic analysis, providing basic structures for subsequent, more complex syntactic / semantic tasks.

• For example, for "quickly improve model performance," tag "quickly" as an adverb, "improve" as a verb, and "performance" as a noun, for downstream analysis.

1.3.2 Relation and Event Extraction: Connecting "Points" into "Lines" and "Stories"

Once we have identified entities in the text through sequence labeling, a natural next question is: what exactly are the relationships between these entities, and what kind of events do they collectively constitute?

Relation extraction focuses on "entity pairs + relation types." For example, in the sentence "Zhang San joined a certain tech company as CTO in 2024," we need to not only identify the two entities "Zhang San" and "a certain tech company" but also extract the "employed at" relationship between them.

• Simply put, it means attaching a relation label like "employment" to the entity pair "Zhang San – a certain tech company."

Building on relations, event extraction attempts to reconstruct "who did what, when, and where." Taking a news article as an example, a standard event template might include multiple slots: event type (acquisition, partnership, accident), time, location, participants, amount, consequences, etc. Event extraction models need to automatically fill these slots from lengthy text, thereby constructing an "event table" that can be retrieved, statistically analyzed, and reasoned about.

• For example, from "Company A acquired another company for 500 million RMB," extract: event type = acquisition, amount = 500 million RMB, participants = two companies.

In terms of modeling methods, beyond traditional sequence-labeling-style extraction, we also use Span‑based IE (directly predicting the start and end positions of entity / relation spans) as well as the more recently emerged Prompt‑based IE and LLM-based Few‑shot extraction. The advantage of the latter is that new schemas can be quickly adapted through natural language prompts, significantly reducing the cost of extensive re-labeling and retraining.

From an engineering perspective, a mature extraction system typically forms a pipeline:

• Upstream: NER / sequence labeling identifies entities;
• Middle layer: models relations and event structures;
• Downstream: writes results into databases or knowledge graphs for consumption by search, analysis, and risk control systems.

1.4 Text Generation and Editing (Text Generation & Editing)

In the preceding sections, we sequentially built the understanding pipeline of "representation → classification and matching → sequence labeling and extraction": the model can not only map text into semantic space but also make judgments about entire pieces of text and extract structured information from them. What this section aims to do is "reverse" that understanding pipeline: on the basis of sufficient understanding, have the model proactively produce, rewrite, compress, and polish text. You can think of it as: performing "reverse encoding" in semantic space, turning internal representations back into high-quality natural language output — the layer closest to user perception in the entire text modality capability chain. We'll again break it down from four dimensions: scenarios, principles, models, and products:

• Scenarios
  • Daily writing and office work: generate emails, notifications, draft proposals, or expand, rewrite, and polish existing text.
  • Knowledge management and summarization: automatically summarize long documents, reports, and meeting minutes to help quickly grasp key points.
  • Customer service and Q&A: automatically generate clearly structured, consistently toned responses based on user questions and retrieved materials.
  • Marketing and creative content: generate ad copy, social media posts, event descriptions, scripts, etc.
  • Multilingual scenarios: complete translation and localization rewriting while preserving original meaning, adapting to different languages and contexts.
• Principles On the foundation of language modeling, perform "creation from scratch" and "modification based on existing content" on text:
  • Free generation: generate a complete piece of text from scratch based on intent, prompts, or outlines;
  • Controlled rewriting: adjust style, length, and structure while keeping core information unchanged (e.g., summarization, expansion, style transfer);
  • Error correction and polishing: correct typos and grammar issues, optimize expression order and logical structure.
• Models Primarily large-scale pre-trained + instruction fine-tuned generative models:
  • Instruction fine-tuned LLMs: GPT series, LLaMA / Qwen / GLM, etc., for general-purpose generation and editing;
  • Seq2Seq models: T5, BART, mT5, etc., for tasks such as summarization, translation, and format conversion;
  • Alignment and safety: through methods like RLHF / RLAIF, make generated content more compliant with instructions and safety requirements.

Since this section is essentially equivalent to prompt engineering, it will not be elaborated further here; you can refer to the prompt engineering tutorial section on your own.