2. Image Modality (Image / Vision)

In AI capabilities, the image modality is responsible for "understanding the world through vision." Whether the goal is security surveillance, autonomous driving, short video effects, e-commerce intelligent retouching, multimodal Q&A, or AI painting, it all essentially follows one path: starting from raw pixels and progressively obtaining structured understanding and controllable generation of images.

2.1 Low-Level Vision

In the previous section, we introduced the role of the vision modality in multimodal systems and how it interfaces with language and speech. But before diving into "high-level semantic tasks" like object detection, image understanding, and visual question answering, there is a foundational capability layer that is often overlooked yet critically important — low-level vision. You can think of it this way: before "understanding what is in the image," the system needs to first address two questions: "how good is the quality of this image itself" and "what stable local structures can be reused by upper layers." Through a layer of universal restoration, enhancement, and structure extraction, raw pixels are transformed into cleaner, more stable image representations.

From an engineering perspective, low-level vision directly affects both the "image quality experience" perceived by users and the health of input distributions for downstream tasks like detection, recognition, and segmentation. If this layer is not done well, all subsequent models have to struggle in environments with "heavy noise, severe distortion, and extreme lighting." Conversely, if images are repaired as much as possible and structural information is well extracted at this layer, high-level tasks can perform on a much friendlier foundation. Below, we organize this layer from three angles: scenarios, principles, and models.

• Scenarios
  • Cameras and capture devices: automatic denoising, HDR, night mode, stabilization on phones/cameras; multi-frame fusion to enhance detail and dynamic range.
  • Content platforms and short videos: one-click quality enhancement for uploaded images/videos — removing compression artifacts, improving sharpness and contrast, and enhancing subjective visual perception.
  • Old photo and document restoration: denoising, colorization, and super-resolution of old photos; automatically straightening and enhancing crooked or dark receipts, contracts, and book pages to facilitate OCR.
  • Surveillance and security: denoising, dehazing, rain removal, and resolution enhancement of low-light surveillance footage, laying the groundwork for subsequent face/license plate recognition.
  • AR/VR and 3D reconstruction: providing stable corners, edges, and local descriptors for SLAM, panorama stitching, and 3D reconstruction, ensuring robustness in tracking and registration.
• Principles Centered on two core objectives — "image quality" and "local structure" — physical and statistical modeling is applied to pixel-level information:
  • Image restoration and enhancement: assuming the observed image is a degraded version of the ideal image (through noise, blur kernels, compression, and imaging nonlinearities), perform denoising, deblurring, compression artifact removal, low-light enhancement, and super-resolution reconstruction, making the output closer to real scene imaging while aligning with human visual perception habits.
  • Structural feature extraction: without introducing specific semantic labels, extract features such as edges, corners, local textures, and salient regions from pixel gradients and texture statistics, providing a "geometric skeleton" for subsequent detection, registration, tracking, and segmentation.
  • Geometric and illumination preprocessing: based on camera models and simple geometric cues (straight lines, vanishing points, symmetry, etc.), estimate distortion and perspective relationships; through operations like undistortion, straightening, contrast and illumination normalization, align the raw image into a more standardized and stable input space.
• Models A combination of classical image processing methods and deep learning models, balancing efficiency and effectiveness:
  • Traditional image processing: bilateral filtering, non-local means, guided filtering, Retinex, histogram equalization, Canny/LoG edge detection, Harris/FAST corners, SIFT/SURF/ORB descriptors, Hough transform, camera calibration and geometric correction, etc.
  • Deep restoration and enhancement models: CNN-based or Vision Transformer-based denoising, deblurring, super-resolution, deraining/dehazing/compression artifact removal models (e.g., EDSR, RCAN, SwinIR, ESRGAN), as well as multi-frame/video enhancement networks that learn the mapping from degraded images to high-quality images in an end-to-end manner, or use modern image editing models such as Jimeng and Qwen editing models.

2.1.1 Image Restoration and Enhancement: From "Visible" to "Clearly Visible"

In low-level vision, image restoration and enhancement first confront various types of degradation: noise, blur, compression artifacts, low light, insufficient dynamic range, etc. Raw images in many real-world scenarios are not "clean": night scenes and indoor low light fill the frame with grain and color noise; snapshots and surveillance footage often appear blurry due to motion or poor focus; video compression introduces blocky artifacts. The goal of restoration and enhancement is to restore clear details and natural visual perception as much as possible without altering the semantic content of the image — turning "blurry, dark, dirty" inputs into "clear, bright, comfortable" ones.

Typical tasks include denoising, deblurring, low-light enhancement, and super-resolution. Denoising and deblurring require a trade-off between local texture and global structure: suppressing high-frequency noise and deconvolving blur kernels without also smoothing away real details. Low-light enhancement must boost brightness and contrast while avoiding amplifying dark-area noise, correcting color casts, and controlling overexposed regions. Super-resolution focuses on adding reasonable high-frequency information during upscaling, so the enlarged image neither looks "blurry and plasticky" nor excessively "fabricates" details. Modern approaches mostly use deep networks (CNN or Vision Transformer), learning the mapping from observed image y to ideal image x on large-scale "degraded–clean" paired data, with composite objectives including pixel error, perceptual loss, and adversarial loss to strike a balance between "good metrics" and "pleasing to the human eye."

These capabilities often manifest implicitly in products: the night mode and HDR capture on phone cameras, one-click quality enhancement on short video platforms, old photo restoration tools, and cloud-based enhancement services for surveillance systems all fundamentally rely on this layer's restoration and enhancement modules. For businesses, they directly affect users' subjective perception of "image quality" and indirectly determine the input quality for downstream detection, recognition, and segmentation algorithms. One could say that the more complex the upper-level vision tasks, the more they depend on a high-quality, distributionally stable "image foundation" at the bottom layer.

2.1.2 Structural Features and Preprocessing: Building the "Scaffolding" for High-Level Understanding

Once image quality has been restored to a usable level, the second key task of low-level vision is to extract features from pixels that are temporarily unrelated to specific semantics but are very important for geometric structure and visual perception, and to unify geometry and illumination. This step won't directly tell you "this is a car" or "this is someone's face," but it will answer questions like "where are the clear contours and corners," "which regions have significant texture structure," and "is the image distorted or tilted," providing reliable structural input for upper-level models.

In terms of feature extraction, edges and corners are the most fundamental elements. Using operators like Canny and Sobel, the system can mark the "edges" where grayscale or color changes most sharply across the entire image — these often correspond to object contours, component boundaries, and texture directions. Corner detection (e.g., Harris, FAST) finds "corners" where local gradients change significantly in multiple directions, typically appearing at object corners and line intersections. Further, local descriptors like SIFT, SURF, and ORB encode the texture pattern of a small region around these keypoints, enabling the same physical point to be matched across different viewpoints, scales, and certain illumination changes. This provides foundational support for image registration, panorama stitching, SLAM, AR tracking, and 3D reconstruction.

In parallel with feature extraction are various geometric and illumination preprocessing operations. Barrel/pincushion distortion from wide-angle lenses, and skew and perspective stretching when photographing documents, are identified through low-level geometric cues like line detection and vanishing point estimation, and are "pulled back to normal" through undistortion, straightening, and perspective correction. Global or adaptive histogram equalization, contrast stretching, and illumination normalization enhance local contrast and reduce the impact of uneven lighting and shadows while preserving details. Color space transformations (RGB → HSV/Lab) and color histogram statistics provide directly usable input for simple color-based segmentation, saliency region detection, and color cast correction.

After end-to-end deep learning became mainstream, some of these structural features and preprocessing steps have been "internalized" into the convolutional kernels and normalization strategies of the first few network layers, no longer appearing as explicit operators in system architecture diagrams. But functionally, they still play the same role: first using a relatively universal, class-agnostic low-level processing layer to organize raw pixels into representations that are more stable in terms of geometric shape, lighting conditions, and local structure, then handing them off to upper-level classification, detection, segmentation, and multimodal modules to complete the task of "understanding what this is." Without this layer of "scaffolding," upper-level models would have to struggle on raw images with heavy noise, severe distortion, and blurred structures, and the overall system's robustness and generalization ability would significantly degrade.

2.2 Image Classification & Recognition

In most image tasks, the questions businesses truly care about are: What category does this entire image belong to? Who is the person in this image? Is this pedestrian the same person across different cameras? You can think of this layer as: on a unified, clean input space, assigning "category labels" or "identity labels" to entire images or entire persons/objects, converting visual signals into the most directly usable recognition results.

From a product perspective, image classification and recognition were among the first vision capabilities to be deployed at scale and serve as the "entry module" for many upper-level applications. E-commerce and content platforms use it to automatically tag images and identify main product categories; security and access control systems use it to confirm "is this the same person"; person re-identification systems sift through feeds from multiple cameras to find the cross-scene trajectory of the same target. Below, we organize this layer from three angles: scenarios, principles, and models.

• Scenarios
  • General image understanding: automatically tagging user-uploaded images with theme labels like "landscape / food / pet / document" for retrieval, recommendation, and content moderation.
  • Face recognition and access control: in face-based access control and attendance systems, identifying individuals from face images to enable "face-scan entry" and "face-scan clock-in."
  • Pedestrian/person re-identification: determining whether the same pedestrian or person appears in different camera feeds, used for security retrieval and trajectory analysis.
  • Human attribute recognition: without directly confirming identity, recognizing attributes such as gender, age group, whether wearing a hat/backpack/uniform, etc., providing clues for retrieval and behavior analysis.
• Principles Discriminative modeling of entire images or entire persons/objects in a unified visual feature space:
  • Image classification: taking the entire image as input, extracting global features through convolutional networks or Vision Transformers, and attaching a classification head on top of the features to output single-label or multi-label category probabilities, answering "what type of image is this."
  • Identity/instance recognition: transforming the "who is this" question into a metric learning problem in feature space — learning an embedding space where features of the same identity are close to each other and features of different identities are far apart, then using nearest neighbor search or clustering to complete recognition and retrieval.
  • Attribute recognition: on top of shared pedestrian/human features, adding multi-task output heads to predict attribute labels such as gender, age group, clothing color, whether carrying items, etc., so that the same features can serve multiple downstream retrieval and analysis needs.
• Models Deep convolutional networks and Vision Transformers serve as the backbone, combined with classification heads or metric learning heads to implement different types of recognition tasks:
  • Image classification backbones: ResNet, DenseNet, EfficientNet, ConvNeXt, Vision Transformer (ViT), Swin Transformer, etc., typically pre-trained on large-scale datasets like ImageNet and then fine-tuned on specific business data.
  • General classification structures: Backbone + fully connected classification layer (Softmax / Sigmoid) for single-label or multi-label image classification tasks; class re-weighting and focal loss can be used to handle long-tail distributions.
  • Identity/instance recognition: on top of backbone feature outputs, using angular-margin loss functions like ArcFace, CosFace, SphereFace to explicitly widen inter-class margins between different identities, improving separability in feature space, and completing large-scale gallery matching via vector retrieval (ANN).
  • Pedestrian/attribute recognition structures: for person Re-ID and human attribute recognition, a common approach is to use a shared backbone to extract pedestrian features, then branch out "identity branch" and "attribute branch" at the top, optimizing both cross-camera identity discrimination and multi-attribute prediction.

Corresponding to specific product forms, the capabilities of this layer are often delivered as "image content recognition / classification APIs," "face recognition SDK / SaaS," "person re-identification platforms," etc. They often directly drive business decisions (e.g., access control clearance, content tag writing) and also serve as upstream, providing structured labels and stable identity representations for subsequent retrieval, recommendation, behavior analysis, and multimodal understanding. Below, we expand on two directions: image classification and identity/attribute recognition.

2.2.1 Image Classification: Answering "What Kind of Image Overview of This"

In the most basic image classification task, the system faces an entire image and the goal is to assign it one or several semantic category labels. The most common form is single-label classification — for example, in datasets like ImageNet, each image is labeled with one primary category such as "dog," "cat," "car," "airplane." In business scenarios, this capability is widely used to tag user-uploaded images with theme labels like "landscape / food / pet / portrait / document" to support retrieval, recommendation, and content moderation. Similar to text classification, the model attaches a fully connected + Softmax layer on top of global visual features extracted by a pre-trained backbone, outputting a probability distribution over all candidate categories.

In many real-world applications, an image often belongs to multiple categories simultaneously. For example, a "seaside sunset selfie" image could be "landscape," "portrait," and also tagged as "travel" or "seaside." This calls for multi-label classification: the model still starts from whole-image features, but the output layer is no longer a mutually exclusive Softmax — instead, it predicts the presence/absence probability (Sigmoid) for each label independently, trained with a multi-label loss function. To handle the large number of "long-tail categories" (rare labels with very few samples) in real-world data, multi-label classification models often incorporate class re-weighting, hard example mining, or label structure modeling to improve recall for niche categories.

At the human-machine interface level, image classification is typically provided as an "image content recognition API." Upstream businesses only need to upload an image to receive a set of category labels and their confidence scores for subsequent decision-making: for example, an ad delivery system can restrict certain sensitive categories based on image content; an e-commerce platform can use image classification to assist with product category correction; a content platform can use it to enrich recommendation features and moderation signals. Although technically relatively mature, this capability remains the cornerstone for more complex downstream capabilities like object detection, instance segmentation, and visual question answering.

2.2.2 Image Recognition and Attribute Recognition: Answering "Who Overview of This / What Instance Is This"

Unlike "what type of image is this," image recognition is more concerned with "who is the person/object in this image" — that is, identity-level, instance-level differentiation. Typical representatives are face recognition and person re-identification: the former determines "which identity in the gallery is closest to the current face" in scenarios like access control, attendance, and payment; the latter searches across multiple camera feeds and time periods to find whether the same pedestrian appears, assisting with case review and trajectory analysis. The core of these tasks is no longer simple multi-class classification, but rather learning an embedding in feature space that is "compact within classes and separated between classes," so that images of the same identity captured under different poses, lighting, and cameras can still be clustered together.

In model design, face recognition and person re-identification typically adopt a similar paradigm: first use backbones like ResNet, ConvNeXt, ViT, Swin to extract face/person-centered features, then apply loss functions specifically designed for metric learning, such as ArcFace, CosFace, etc. Unlike ordinary classification losses, these losses directly constrain inter-class boundaries in angular space or feature space, explicitly widening the margin between features of different identities, so that the trained features can be used for large-scale vector retrieval without being limited to the fixed classes seen during training. During online serving, the system pre-computes and indexes features for each identity in the gallery, then performs approximate nearest neighbor search on the query face/person features to find the most similar candidates, making final decisions based on business thresholds and multimodal information.

Complementing "direct identity recognition" is attribute recognition, which does not point to a specific person. In many security and retail scenarios, the system only needs to know attributes like "male or female," "approximate age group," "whether wearing a hat/mask," "clothing color and style," "whether carrying a backpack/luggage," etc., for quick target filtering — without, and often without being appropriate to, directly outputting personal identity. These tasks typically attach multiple parallel attribute heads on top of shared pedestrian/human features (a "head" here means the position where probabilities are output; multiple probability output positions can be used to determine categories), with each head responsible for predicting one or a group of attribute labels, forming a multi-task learning framework. On one hand, multi-task training can make features richer and better generalized; on the other hand, attributes themselves can serve as auxiliary conditions for Re-ID or retrieval, improving system usability in complex scenarios.

In product form, these capabilities are typically packaged as "face recognition SDK / cloud service," "person re-identification platform," "human attribute recognition API," etc., and integrated into access control gates, attendance machines, security platforms, and video structuring systems. Compared to general image classification, they have higher requirements for data security and privacy protection, and are more sensitive to the trade-off between false recognition rate and recall rate. Therefore, beyond algorithms, they are supplemented with mechanisms such as quality detection (e.g., whether it is a real person, whether occluded/recaptured), liveness detection, and multimodal cross-verification, forming a more complete and responsible identity recognition solution.

2.3 Object Detection

In the previous image classification and recognition sections, we only assigned an overall label to "the entire image" or "the entire person," ignoring where and at what size the object appears in the image. However, a more common question in real business is: What objects are in this image? Where is each one located? For example, in a street scene image, we want to simultaneously mark all pedestrians, vehicles, and traffic signs; on an industrial production line, we need to mark all defect areas and part positions in the same frame. Object detection is designed for these needs: in a single image or video frame, it simultaneously predicts the position (bounding box) and category of every object, serving as the foundational capability for many downstream vision tasks (tracking, segmentation, behavior analysis, multi-object counting, etc.).

From an engineering usage perspective, object detection is the "first step of structuring" for many vision systems — decomposing a raw image into several labeled rectangular boxes, each of which can be further sent to other modules for recognition, tracking, attribute analysis, and even semantic generation. Pedestrian/vehicle detection in surveillance cameras, product detection on unmanned retail shelves, defect/foreign object detection in industrial quality inspection, and the "object detection" APIs provided by cloud vendors all fundamentally rely on this layer of capability. Below, we organize object detection from three angles — scenarios, principles, and models — and expand on key directions in subsequent subsections.

• Scenarios
  • Security and traffic surveillance: real-time detection of pedestrians, vehicles, non-motor vehicles, traffic signs, wrong-way/encroaching objects in camera feeds, providing the basis for subsequent behavior analysis and alerts.
  • Industrial quality inspection and manufacturing: detecting product defects (scratches, breakage, foreign objects), part positions, and assembly completeness on production lines, supporting automatic rejection and robot positioning.
  • Retail and logistics: product detection on unmanned retail shelves and at checkout; target detection and positioning of packages, pallets, and stacks in warehousing, assisting inventory counting and robotic grasping.
  • Content understanding and moderation: detecting people, logos, weapons, sensitive items, etc. in images/videos, providing structured signals for content moderation, ad compliance, and brand recognition.
• Principles The core of object detection is building a dense prediction mechanism on the image:
  • Extract multi-scale feature maps from the input image through a backbone; on these feature maps, simultaneously predict "is there an object," "what category," and "the corresponding bbox parameters" for each "position" (or candidate region).
  • By architecture, there are two-stage detectors that first generate candidate boxes and then refine them, and one-stage detectors that directly perform classification + regression on feature maps in a unified manner; each has different emphases on accuracy vs. speed.
  • By candidate box design, there are anchor-based methods that rely on predefined anchor boxes, and anchor-free methods that directly predict center points/boundaries, as well as the DETR family based on set matching.
  • To handle small objects, dense objects, occlusion, and scale variation in real-world data, detectors typically combine multi-scale features (FPN), higher-resolution inputs, specific loss functions, and post-processing strategies (e.g., NMS variants, multi-scale testing) for optimization.
• Models Detection models are generally composed of three parts: backbone network + feature pyramid / head structure + loss and post-processing:
  • Classic two-stage detectors: Faster R-CNN, Mask R-CNN, etc., which first generate candidate boxes via RPN, then perform fine classification and regression on each candidate region — high accuracy, clear structure, suitable for scenarios with extremely high accuracy requirements.
  • One-stage detectors: SSD, RetinaNet, YOLO series (YOLOv5/6/7/8, YOLOX, YOLOv10, etc.), which complete detection in a unified network — compact structure, low latency, the workhorse of real-time detection in industry.
  • Anchor-free / Transformer detectors: FCOS, CenterNet, ATSS, etc., which predict boxes directly centered on pixels; DETR / Deformable DETR, etc., which treat detection as "generating a set of objects from a set of queries" through Transformers and set matching, simplifying many hand-crafted designs.
  • Video detection and tracking: building on image detectors, introducing temporal information and association strategies (e.g., tracking heads, optical flow, trajectory matching) to form a unified Detection + Tracking framework, supporting long-duration, multi-object behavior analysis.

Overall, object detection sits at the "hub position" of the vision capability spectrum — on one hand, it receives clean image inputs provided by low-level vision; on the other hand, it decomposes images into "object-level" elements that can be used for recognition, tracking, segmentation, and multimodal understanding. Below, we expand on three directions: single/two-stage detection architectures, anchor-based / anchor-free / Transformer detection, and small object and video detection.

2.3.1 Single-Stage and Two-Stage Detection: The Accuracy–Speed Architectural Trade-off

From an architectural perspective, the most classic division in object detection is two-stage vs. one-stage. The main difference between the two is: whether to first "roughly select a batch of candidate boxes and then refine them," or to "predict all boxes and categories in one go on the feature map."

Two-stage detection is represented by Faster R-CNN. It first generates a batch of candidate boxes with "high probability of containing an object" through an RPN (Region Proposal Network) on the backbone feature maps (first stage), then performs RoI alignment and feature extraction for each candidate region, followed by finer classification and bounding box regression (second stage). The advantage of this design is that a large number of negative samples are filtered out at the RPN stage, and the second stage can focus on high-quality discrimination on a small number of candidate regions, thus often having an edge in accuracy and being easier to extend to tasks like instance segmentation (Mask R-CNN) and keypoint detection (Keypoint R-CNN). However, the multi-stage structure brings relatively higher computational and implementation complexity, making it more suitable for offline or near-real-time scenarios where real-time requirements are less stringent but accuracy and extensibility are emphasized.

One-stage detection aims to streamline the entire pipeline, completing both category classification and bounding box regression in a unified network. Representative models include SSD, RetinaNet, and the YOLO series: they directly predict "foreground/background + category + bbox" for several candidate boxes at each position on multi-scale feature maps, omitting the explicit proposal stage and being more suitable for end-to-end acceleration and deployment. Early one-stage detectors had a certain accuracy gap compared to two-stage ones, but with their simple structure and fast speed, they quickly dominated industry. With the introduction of FPN, focal loss, IoU-aware loss, and stronger backbones and necks, newer models like RetinaNet, YOLOX, YOLOv7/8/10 have achieved accuracy–speed balances that "approach or even surpass two-stage" on many tasks.

At the application level, engineering choices between these two architectures are typically made based on requirements: for cloud-based batch offline analysis tasks requiring high accuracy and extensibility (e.g., simultaneous detection + segmentation + keypoints), two-stage detection remains a stable and reliable choice; for latency-sensitive scenarios like edge devices, mobile applications, and real-time camera detection, one-stage detectors like the YOLO series are almost the default go-to, often combined with techniques like quantization, pruning, and distillation to further compress models and increase throughput.

2.3.2 Anchor-Based and Anchor-Free: From Hand-Crafted Design to End-to-End Learning

On the question of how to define "candidate boxes," detection methods can be divided into two major categories: anchor-based and anchor-free. Early mainstream methods (e.g., Faster R-CNN, SSD, RetinaNet, YOLOv3/v4/v5) adopted the anchor-based approach: predefining several anchor boxes with different scales and aspect ratios at each position on the feature map, then learning the foreground probability and bbox offset for each anchor. This approach is simple to implement and effective, but requires substantial manual tuning of anchor sizes and ratios, and in small-object or dense-object scenarios, can lead to an enormous number of anchors and extreme positive-negative sample imbalance.

Anchor-free methods attempt to break free from dependence on predefined anchors. Represented by FCOS, CenterNet, ATSS, etc., they typically directly predict at each pixel of the feature map "whether this is the center of some object (or belongs to that object)" and the corresponding boundary distances, thus completely avoiding the complexity of preset anchors. The benefits are: simpler model structure, more natural training sample assignment strategies, and better generalization and scalability, especially when facing real-world scenes with large scale variations and complex object shapes. At the same time, anchor-free detectors have also driven more pixel/point-based unified frameworks, making it easier to jointly model detection with keypoints, segmentation, and other tasks.

Going further, Transformer-based detectors like DETR / Deformable DETR rethink the detection problem from another dimension: instead of densely laying anchors on feature maps, they introduce a fixed number of "object queries" and, through the Transformer's self-attention and cross-attention mechanisms, "generate" a set of object predictions from global features, achieving one-to-one alignment via Hungarian matching. This set prediction approach completely eliminates traditional components like NMS and manual sample assignment — conceptually very clean, but early implementations suffered from slow convergence and being unfriendly to small objects. Subsequent work like Deformable DETR, by introducing deformable attention and multi-scale mechanisms, has significantly improved both convergence speed and performance, gradually gaining more applications in detection and multi-task scenarios.

For engineering practice, anchor-based, anchor-free, and Transformer detection are not mutually exclusive choices but rather an evolutionary chain: from heavily engineered anchor design, to more end-to-end point/center prediction, to fully set-prediction and attention-based unified frameworks. In current industrial deployment, mature anchor-based models like the YOLO series remain the workhorses, while anchor-free and the DETR family appear more in systems with higher demands for structural simplicity, multi-task unification, and scalability.

2.3.3 Small Object and Video Detection: Robustness for Real-World Scenarios

Object detection on public datasets often gives the impression that "the problem is basically solved," but once entering real-world scenarios, two types of challenging problems immediately arise: small/dense objects and robust detection and tracking in video.

In small object detection, the target often occupies only a tiny pixel area in the original image — for example, distant pedestrians, faraway vehicles, aerial drones, or minute defects on high-resolution industrial images. As the backbone downsamples and feature map resolution decreases, these small objects can easily be "drowned out" in high-level features, leading to missed detections. To address this, detectors typically employ multi-scale feature pyramids (FPN/PAFPN, etc.), increase input resolution, add detection heads on shallow feature maps, and even design dedicated branches and loss weighting strategies for small objects. At the data level, techniques like cropping, zooming, and small-object resampling are also needed to improve the model's perception and memory of small-scale targets.

Dense objects (e.g., crowded crowds, dense parking lots, tightly packed products/parts) expose problems like anchor box overlap, NMS false suppression, and severe occlusion. Improvement strategies include finer label assignment (e.g., adaptive assignment methods like ATSS), soft NMS or learning-based deduplication strategies, and mitigating inter-box competition through center-point/density-map modeling. In industrial quality inspection, many systems also combine detection with pixel-level segmentation for more precise defect localization, facilitating subsequent automated processing.

When detection extends from single frames to video, another challenge is temporal continuity and target stability. Single-frame detectors make independent predictions on each frame, making it difficult to avoid short-term missed detections, ID jitter, and false alarms — yet real-world applications for alerting, counting, and trajectory analysis often require consistent cross-frame target trajectories. For this, video object detection typically overlays a tracking module, connecting "detection + object tracking": the classic approach uses an image detector as the frontend, and in the backend uses Kalman filtering, Hungarian matching, and appearance feature similarity for multi-object tracking (e.g., SORT, DeepSORT). More advanced approaches integrate tracking heads directly into the detection network, jointly learning detection and cross-frame association to improve robustness in scenarios with short-term occlusion and fast motion.

In real systems, small objects, dense objects, and video detection are often not isolated problems but appear simultaneously: for example, distant pedestrians/vehicles in urban road surveillance, dense crowds in station squares, and high-speed moving parts in production line video. This also means that high-quality object detection modules, beyond having impressive metrics on standard benchmarks, need to withstand the test of various complex factors under real-world conditions like multi-scale, multi-density, and long-duration video, in order to truly support upper-level behavior analysis, intelligent alerting, and multimodal understanding.

2.4 Image Segmentation

With object detection, we already know "what objects are in the image and roughly where they are," but many tasks require even finer structured understanding: precisely determining, for every pixel, which category it belongs to and which instance it belongs to. For example, in autonomous driving, we need to know which pixels are road, which are people, and which are cars; cutout tools need to cleanly separate hair strands from the background; medical images require precise delineation of tumor and organ boundaries. These tasks are collectively called image segmentation, which directly outputs semantic or instance labels at the pixel level, providing finer-grained spatial structure information compared to detection.

From a product perspective, image segmentation is the core capability for "pixel-level structuring": cutout and background replacement tools rely on it to decide which pixels to keep; autonomous driving perception modules rely on it to build fine-grained "drivable area + obstacle" maps; medical imaging software relies on it to measure lesion size, shape, and volume; remote sensing platforms rely on it to distinguish farmland, water bodies, buildings, roads, and other land features. Below, we organize image segmentation from three angles — scenarios, principles, and models — and expand on directions like semantic/instance/panoptic/large-model segmentation in subsequent subsections.

• Scenarios
  • Content editing and cutout: portrait cutout, hair-level background replacement, object extraction and layered editing for image beautification, short video effects, and advertising creative production.
  • Autonomous driving and robotics: labeling each pixel as road surface, lane line, pedestrian, vehicle, guardrail, building, sky, etc., for path planning, collision warning, and environment modeling.
  • Medical image analysis: precisely segmenting organs, tumors, and lesion regions in CT, MRI, ultrasound, and other images, supporting assisted diagnosis, surgical planning, and treatment evaluation.
  • Remote sensing and geographic information: segmenting farmland, water bodies, roads, buildings, forests, and other land features in satellite/aerial imagery, supporting land planning, land use monitoring, and disaster assessment.
• Principles Image segmentation is essentially "dense prediction": extracting multi-scale features from the input image through an encoder (backbone), then progressively restoring the feature maps to a segmentation map of the same size as the input through a decoder or upsampling modules, outputting a semantic or instance label at each pixel position.
  • Semantic Segmentation: assigns a semantic category to each pixel (e.g., road, person, car, sky), without distinguishing different individuals of the same class — suitable for describing "scene composition."
  • Instance Segmentation: further distinguishes different instances of the same class on top of semantic information, generating independent masks for "each car, each person" — a combination of detection and segmentation.
  • Panoptic Segmentation: uniformly handles "countable things (e.g., people, cars)" and "uncountable stuff (e.g., road, sky)," providing both a semantic label and an instance ID for each pixel. Compared to detection, segmentation is more sensitive to spatial details and boundary quality, requiring richer multi-scale contextual information and finer upsampling/fusion strategies.
• Models Classic to latest segmentation models have roughly evolved along the path of "FCN → encoder–decoder → multi-scale context → detection+segmentation unification → large-model segmentation":
  • Semantic segmentation: FCN, U-Net and variants, DeepLab series (DeepLabv3/v3+), PSPNet, etc., obtaining multi-scale context and fine boundaries through dilated convolutions, pyramid pooling, skip connections, etc.
  • Instance/panoptic segmentation: Mask R-CNN, Panoptic FPN, Mask2Former, etc., combining detection heads with segmentation heads to achieve object-level segmentation and panoptic segmentation.
  • Large models and general segmentation: foundation segmentation models like Segment Anything Model (SAM), elevating segmentation from "training separately for each task" to "one model adaptable to most segmentation scenarios," supporting interactive, prompt-based segmentation.

Overall, image segmentation provides finer spatial structure representation compared to object detection, and is an indispensable link in building highly reliable perception systems and advanced editing tools. Below, we expand on three directions: semantic segmentation and instance segmentation, panoptic segmentation and detection unification, and general segmentation, large models, and unsupervised segmentation.

2.4.1 Semantic Segmentation and Instance Segmentation: From "Pixel Category" to "Pixel Instance"

The goal of semantic segmentation is to assign a semantic category to every pixel in the image, so that the network learns "this region is road, that region is car, here is person, there is sky and building." Classic approaches typically use an encoder–decoder structure: the encoder (e.g., ResNet, EfficientNet, Swin Transformer) extracts progressively downsampled high-level features, and the decoder restores resolution to the original through upsampling, skip connections, and multi-scale fusion, combining coarse high-level semantic features with low-level details. FCN was the first to systematize this dense prediction form; U-Net achieved great success in medical imaging through its symmetric U-shaped structure and abundant skip connections; the DeepLab series expanded the receptive field without reducing resolution through dilated convolutions and ASPP (Atrous Spatial Pyramid Pooling); PSPNet obtained global context information through pyramid pooling. These models collectively drove large-scale applications in road scenes, remote sensing, medical imaging, and other domains.

Instance segmentation further distinguishes different individuals of the same class on top of pixel semantic labels: not only knowing which pixels are "car," but also knowing which pixels belong to which specific car. The most representative model is Mask R-CNN, which adds a parallel segmentation branch to the Faster R-CNN detection framework: the detection head first predicts the category and position of each candidate box, then generates a binary mask within each box, yielding "box + mask" object-level segmentation results. Compared to pure semantic segmentation, this approach handles object overlap and occlusion well, and serves as the foundation for tasks like portrait/product cutout, multi-object counting, and fine-grained editing. Subsequent instance segmentation methods have continuously improved in mask quality, multi-scale handling, and speed, with new anchor-free and Transformer-based architectures also emerging, but the "detection + local segmentation" approach remains very mainstream.

At the product level, semantic segmentation typically appears in "scene-level" applications such as autonomous driving road segmentation, remote sensing land feature recognition, and medical organ segmentation; instance segmentation is more commonly used for "object-level" cutout, counting, and editing, such as one-click selection and separation of each car, each person, and each product. The combination of both provides fine-grained yet structured spatial information for upper-level tasks.

Doing only semantic segmentation mixes same-class objects together (all "car" pixels belong to the same class); doing only instance segmentation often focuses only on countable "things" (e.g., people, cars, animals) while ignoring large areas of uncountable "stuff" (e.g., road, grass, sky). In many scenarios, we need both instance-level masks for each object and an understanding of the overall scene composition. This gave rise to panoptic segmentation: simultaneously providing a semantic class and instance ID for every pixel, achieving unified modeling of things + stuff.

Early panoptic segmentation systems were typically implemented through "semantic segmentation model + instance segmentation model + post-processing fusion": first using one network to predict the semantic category of each pixel, then using another network to output masks and categories for each instance, and finally merging the two into a consistent panoptic segmentation result through a set of rules (e.g., priority, overlap handling). Panoptic FPN represents a more elegant engineering path: on a shared backbone and feature pyramid (FPN), separately attaching semantic segmentation heads and instance segmentation heads, obtaining both outputs simultaneously through joint training and feature sharing, then fusing them through lightweight post-processing. This not only improves efficiency but also enhances consistency between semantics and instances.

At the model level, with the development of detection/segmentation unification and Transformer architectures, unified panoptic segmentation frameworks like Mask2Former have emerged: they tend to use a universal "query + mask decoder" structure, simultaneously predicting masks for semantics, instances, and even other downstream tasks within the same network, thereby greatly simplifying the system architecture and facilitating multi-task extension. For complex tasks like autonomous driving, robot navigation, and AR scene understanding, panoptic segmentation provides a more complete scene description closer to "human subjective perception," allowing upper-level decision-making and planning to operate on more accurate spatial semantics.

In product form, panoptic segmentation is often embedded within autonomous driving, robotics systems, and high-end visual analysis platforms. Users may not directly perceive the concept of "panoptic segmentation," but they genuinely benefit from more robust scene understanding and more natural interactive experiences.

2.4.2 General Segmentation and Unsupervised Segmentation: From Task-Specific to "Segment Anything"

Traditional segmentation models are often trained around specific datasets and tasks: for example, "19-class semantic segmentation of road scenes," "segmentation of a certain type of tumor," "segmentation of certain product categories," etc. Each new task requires re-annotation and re-training. In real business, this approach of heavily relying on precisely labeled data is extremely costly and struggles to cover long-tail categories and constantly emerging new scenarios. In recent years, with the development of large-scale pre-trained vision models and prompt-based paradigms, general segmentation large models represented by the Segment Anything Model (SAM) have emerged, attempting to elevate segmentation capability from "task-specific customization" to "infrastructure."

Taking SAM as an example, it learns universal features of the entire image through a powerful image encoder (typically a large-scale pre-trained ViT), and then converts user-provided points, boxes, text prompts, etc. into segmentation results through a lightweight prompt encoder and mask decoder. During training, SAM leverages massive, multi-source, multi-task mask annotations, so that the model learns a "generalized segmentation capability" rather than rote memorization of a specific dataset's labels. During use, users only need to provide very few prompts (a single point or a rough box) to obtain high-quality masks on various unseen image types and object categories. This paradigm greatly lowers the barrier to building new segmentation applications and provides a powerful tool for unsupervised/weakly-supervised scenarios.

Related to this is the broader direction of unsupervised / self-supervised segmentation: without relying on or minimally relying on human-annotated masks, automatically dividing images into several meaningful regions through signals such as intra-image similarity, temporal consistency, and multi-view constraints. Early work focused more on "visual clustering" and region proposal generation, while nowadays it is increasingly internalized by large models as a form of representation learning, providing good initialization for downstream segmentation tasks. Combined with text–image contrastive learning models like CLIP, an increasing number of methods can perform zero-shot or few-shot segmentation under the condition of "only providing text category names, without providing mask annotations," offering new solutions for cold-start scenarios and long-tail categories.

In actual products, general segmentation large models often appear in forms like "interactive cutout tools," "smart selection," and "one-click background removal," and are gradually being integrated into professional software in fields like medicine, remote sensing, and industry, serving as accelerators for semi-automatic annotation and assisted segmentation. Compared to traditional custom models, they may not achieve the ultimate performance on a specific task, but they have a significant advantage in "being able to do a bit of everything and quickly landing in multiple scenarios," laying the groundwork for building truly multimodal foundation vision models.

2.5 Keypoint Detection & Action Recognition

After classification, detection, and segmentation, we already know "what is in the image, where it is, and what each pixel belongs to." But in many real-world tasks, businesses care not only about "object presence and position" but also about pose and action: Is a person walking or running? Is this hand raised or making a certain gesture? Is the worker correctly wearing safety equipment and performing standardized actions? Is the athlete's technique correct? These questions require us to further understand the internal structure and temporal changes of objects.

Keypoint detection and action recognition are two layers of capability designed for this need:

• Keypoint Detection: on an image or video frame, predict several "skeleton points" (e.g., joints, fingertips, facial features) of a target (typically a human body, hand, face, or specific mechanical structure), obtaining a fine-grained structured pose representation.
• Action Recognition: analyze how these keypoints or appearance features change over time in the temporal dimension, determining "what action or behavior this person/group is performing."

From a product perspective, this capability widely serves: human-computer interaction (gesture control), sports analysis (technique evaluation), security (fall detection, fighting/running and other abnormal behavior recognition), industrial safety (violation action detection), virtual human driving (driving 3D skeletons and animations via body/facial keypoints), and other scenarios. Below, we organize this layer of capability from three angles — scenarios, principles, and models — and expand on keypoint detection and action recognition in subsections.

• Scenarios
  • Human-computer interaction and AR/VR: through gesture recognition and body pose detection, enabling natural interaction where "a gesture is enough to control," or driving virtual avatars in real-time in AR/VR.
  • Sports training and motion analysis: tracking keypoints and analyzing angles for actions like running, high jump, shooting, weightlifting, providing technique evaluation and correction suggestions.
  • Security and public safety: detecting abnormal behaviors like falling, fighting, violent running, climbing over barriers for timely alerts; identifying whether operations are standardized on construction sites and in factory areas.
  • Industry and human-robot collaboration: detecting whether workers operate in standard postures, safe distances during robot collaboration, and whether dangerous actions occur.
  • Facial/expression driving and virtual humans: capturing expression details through facial keypoints for expression transfer, digital human driving, video conference avatars, etc.
• Principles The two types of tasks respectively emphasize spatial structure and temporal change, but both are essentially structured predictions in high-dimensional feature space:
  • Keypoint detection: locating a set of predefined keypoints on an image (e.g., 17/25 body joints, 21 hand joints, 68/106 facial keypoints). A common approach is predicting a heatmap for each keypoint on feature maps, then deriving coordinates through peak position back-projection; in multi-person scenarios, "joint-to-person assembly" is also needed.
  • Single-frame/short-term action recognition: based on a single image or short time window, using human pose (keypoints) and appearance features to determine the action category occurring in that frame/clip (e.g., walking, running, raising hand, waving, sitting).
  • Temporal action recognition: on longer time scales, analyzing feature sequences (image features, keypoint sequences, or optical flow, etc.), modeling the start, duration, and end of actions, recognizing complex behaviors like "making a phone call," "doing push-ups," "two people shoving each other."
  • Structured representation: keypoint sequences provide a more compact and stable structured representation than raw pixels, making it easier to handle viewpoint changes, background interference, and appearance differences in action recognition.
• Models Common models have roughly developed along the unified paradigm of "convolutional/Transformer feature extraction + keypoint/temporal heads":
  • Keypoint detection: OpenPose series, Hourglass Network, HRNet, branching into top-down (detect person first then estimate pose) and bottom-up (detect joints first then assemble) approaches; in recent years, Transformer-based pose estimators have also emerged.
  • Video action recognition: 2D/3D CNN-based video models (I3D, SlowFast, etc.), skeleton-based GCN models (ST-GCN, etc., directly modeling spatiotemporal relationships on keypoint graphs), and end-to-end solutions based on video Transformers (Video Swin, TimeSformer, etc.).
  • Unified multi-task and large models: simultaneously outputting detection, segmentation, keypoint, and action labels on a general vision backbone, or using multimodal large models to directly understand "what action this person is doing" through text prompts, connecting structured prediction with semantic understanding.

Below, we expand on two directions: keypoint detection and pose estimation and action recognition and behavior understanding.

2.5.1 Keypoint Detection and Pose Estimation: Drawing the "Skeleton" for People and Objects

Keypoint detection (also often called pose estimation) focuses on the spatial structure in a single frame or single image: finding a set of semantically meaningful keypoints in a 2D image and connecting them into a skeleton. For example, in human pose estimation, we typically need to detect joints like the head, shoulders, elbows, wrists, hips, knees, and ankles; in facial pose, it's the eye corners, mouth corners, nose tip, face contour, etc.; in hand pose, it's the finger roots, knuckles, and fingertips. For non-human objects like robotic arms and articulated structural components, a keypoint system can also be defined similarly.

In model design, keypoint detection commonly uses the "feature extraction + heatmap prediction" paradigm:

• First, use a CNN or Vision Transformer (e.g., ResNet, HRNet, Swin) to extract multi-scale features from the input image.
• Then, through a decoding head or multi-layer convolution, output a heatmap for each keypoint type, where each pixel value represents "the likelihood that this position is that keypoint."
• During inference, the peak position of each heatmap is typically taken as the keypoint coordinate, with sub-pixel optimization through bilinear interpolation, local fitting, etc.

For multi-person scenarios, pose estimation methods roughly divide into two paths:

• Top-down: first use a person detector to find each person's bounding box in the image, then perform single-person pose estimation on the image within each box. This approach has high per-person accuracy and a simple framework, but incurs high computational cost in dense multi-person scenarios and is sensitive to detection quality. Representative systems include many combinations based on Faster R-CNN/YOLO + Hourglass/HRNet.
• Bottom-up: instead of distinguishing each person first, directly predict all potential keypoints (and their types) on the entire image, while also predicting connection relationships or affinity fields between keypoints (e.g., OpenPose's PAF). Then, through graph matching/clustering algorithms, assemble keypoints into multiple independent human skeletons. These methods are more efficient in dense multi-person scenarios and more robust to crowd size, but the assembly process is complex and sensitive to connection quality.

In recent years, Transformer-based pose estimation models have also gradually emerged, treating keypoint detection as a set of "query–response" tasks, similar to DETR, which can unify object detection and pose estimation architecturally. In engineering applications, keypoint detection capability is typically packaged as "human body/gesture/facial keypoint SDK or API," where upstream applications only need to input images or video frames to obtain structured skeleton coordinates for subsequent action recognition, interaction control, or animation driving.

2.5.2 Action Recognition and Behavior Understanding: Making the "Skeleton" Move

After obtaining keypoints or high-level visual features, the next step is to understand changes in the temporal dimension — that is, action recognition and behavior analysis. Unlike keypoint detection, action recognition is no longer confined to single frames; it is concerned with the evolution patterns of features over a period of time: from "raising a hand" to "waving," from "walking" to "running," from "standing" to "falling."

In terms of input representation, there are roughly three routes:

• Based on raw video frames / optical flow: directly modeling video frame sequences, or additionally introducing optical flow (a field describing local motion velocity) as input, allowing the model to jointly learn from appearance + motion information.
• Based on skeleton/keypoint sequences: first using pose estimation to obtain human keypoint coordinate sequences, then modeling on a "spatiotemporal skeleton graph," weakening background and lighting interference while focusing more on human body structure and motion patterns.
• Multimodal fusion: incorporating video features, keypoint sequences, and even audio, text, and other modalities together to handle complex behavior scenarios (e.g., multi-person interaction, event-level actions).

Correspondingly, model structures have also shown diversified development:

• Early action recognition mainly relied on 2D CNN + temporal pooling or 3D CNN (e.g., I3D, C3D): the former extracts features for each frame and then pools or applies RNN in the temporal dimension; the latter directly performs 3D convolutions in both space and time to capture short-term motion patterns.
• For skeleton sequences, the typical method is spatial-temporal graph convolutional networks (ST-GCN): treating human keypoints as graph structure nodes, with edges between joint connections and also edges in the temporal dimension, propagating information on the spatiotemporal graph through graph convolutions to learn action patterns. These methods are lightweight, robust to background, and suitable for deployment on resource-constrained devices.
• In recent years, video Transformers (e.g., TimeSformer, Video Swin) have performed outstandingly in action recognition, segmenting video into spatiotemporal patches and modeling long-range dependencies through self-attention mechanisms, better capturing complex actions and multi-object interactions.

On the business side, action recognition is often combined with detection, tracking, and keypoint detection to form end-to-end behavior analysis systems:

• In security, first detect and track people, then classify actions on the keypoint sequences of each trajectory to achieve fall detection, fighting/running recognition, etc.;
• In sports and fitness applications, analyze whether actions are standard and whether the range of motion is appropriate through keypoint sequences, and provide correction suggestions;
• In human-computer interaction scenarios, perform lightweight action classification on real-time pose streams to achieve interactions like waving, making a heart gesture, gesture commands, etc.;
• In industrial safety, continuously monitor worker operations to identify dangerous postures (e.g., leaning into danger zones, crossing safety lines, etc.).

Looking ahead, multimodal large models are elevating "action recognition" to a higher level of "event and intent understanding": models can not only label "walking, running, making a phone call" but also answer descriptions closer to everyday language like "this person seems to be signaling someone" or "these two people are having an argument." Keypoint detection and action recognition, as important structured motion cues, together with appearance features and language prompts, jointly support more complex spatiotemporal understanding capabilities.

2.6 Open-Vocabulary / Open-World / Open-Domain Detection

The detection and segmentation capabilities discussed earlier all basically default to one premise: the set of categories during training and inference is fixed. That is, the model has fully seen "all categories to be recognized" during the training phase, and during inference, it only needs to choose from this closed set of labels. But the real world is far more complex than datasets: new products, new brands, new signs, new species, and new scenarios appear at any time, and it is impossible to prepare sufficient annotated data for each new class and retrain the detector. This gave rise to open-vocabulary / open-world / open-domain detection: under the condition that training data only covers a limited set of "known classes," enabling the model to still perceive, locate, and recognize unseen new classes during inference, while maintaining robustness under changes in visual style and capture domain.

You can think of this layer as: adding "alignment and generalization capability to the language space and open world" on top of traditional detection. The model no longer only says "this is one of the 80 COCO classes," but can understand and retrieve objects in the space of arbitrary text descriptions — for example, "detect all 'red sports shoes' in the image," "mark all 'suspected small aircraft'," even if these fine-grained categories never explicitly appeared in the training set. Below, we organize this layer from three angles — scenarios, principles, and models — and expand on open-vocabulary detection, open-world detection, and open-domain generalization in subsections.

• Scenarios
  • General scene understanding API: users provide arbitrary natural language descriptions (category words or short phrases), and the system returns detection boxes or segmentation masks for corresponding objects in images of any style — e.g., "all hard hats in the image," "all suspected brand logos," "all objects with wheels."
  • Large-scale product / species recognition: constantly emerging long-tail products in e-commerce, the enormous number of animal and plant species in nature — training data can only cover a portion of known classes, but the system needs to locate and roughly recognize massive new classes, and support retrieval via text or images.
  • Cross-domain security / autonomous driving perception: training data mostly comes from daytime urban roads / a few camera angles, but actual deployment faces different cities, rural areas, highways, extreme weather, infrared/fisheye cameras, and other "new domains," where new types of objects never annotated in the training set may also appear (new car models, new traffic facilities, new types of obstacles).
• Principles The core of these methods is to replace the traditional "fixed one-hot category head" with a vision–language aligned embedding space, and handle "unseen classes" and "new domains" through various mechanisms:
  • Open-vocabulary detection: during training, leverage large-scale image–text pairs to pre-train an aligned space similar to CLIP, so that image regions and text embeddings can be directly matched for similarity in the same semantic space; the detection head no longer outputs fixed category logits, but outputs a region feature vector that can be compared with arbitrary text description vectors, thus supporting "training only sees some categories, inference can specify arbitrary text categories."
  • Open-world detection: further handles "new classes completely unannotated in the training set," requiring the model to detect such objects as "unknown" and, through subsequent interactive annotation or continual learning, gradually incorporate these unknown classes into the known class set, forming an online learning system that can continuously expand categories.
  • Open-domain / cross-domain detection: facing significant changes in image style, imaging equipment, environmental conditions, etc. (domain shift), through techniques like domain adaptation and domain generalization, enabling the detector to maintain stable detection performance in unseen new domains; common approaches include adversarial domain alignment, multi-domain training, style randomization, meta-learning, etc.
  • Open-vocabulary segmentation and detection unification: extending the above ideas to the pixel level, generating segmentation masks for arbitrary text descriptions (open-vocabulary segmentation), achieving "describe a region/object in natural language and get the corresponding mask or box" through Region–Word or Mask–Word alignment losses.
• Models Current mainstream technical routes for open-vocabulary / open-world / open-domain detection basically revolve around "large-scale vision–language pre-training + detection head adaptation + domain generalization mechanisms":
  • CLIP-based detectors: using CLIP-style image encoders and text encoders as the foundation, applying contrastive learning and Region–Word alignment losses between region-level features (ROI, feature map patches, mask regions) and text embeddings; typical implementations include replacing or extending classification heads on architectures like Faster R-CNN / RetinaNet / YOLO / DETR to output category scores via "cosine similarity + text embeddings."
  • Caption-driven / prompt-based detection: leveraging large-scale image caption data to automatically generate text descriptions for regions or masks in images, then aligning these auto-generated texts with detection/segmentation regions for training, thereby reducing dependence on manual category labels; during inference, natural language prompts (e.g., "all people wearing red clothes," "all electric vehicles") drive detection/segmentation.
  • Open-world detection series: explicitly introducing "unknown class" modeling, progressive category expansion, and incremental learning mechanisms in traditional detection frameworks; some methods determine "whether it is an unknown class" through distance and uncertainty estimation in metric space, while others introduce memory banks and online retraining, enabling the system to accumulate new category knowledge over time.
  • Domain adaptation / domain generalization detection: adding domain discriminators, adversarial losses, multi-domain batch normalization, style randomization augmentation, and other modules at the backbone and detection head levels, enabling the detector to learn more domain-invariant representations across different domains; some work also introduces multi-source domain training and meta-learning strategies on Transformer detection frameworks (e.g., Deformable DETR) to improve cross-domain generalization.
  • General / foundation detection models: elevating the detection problem to the "foundation model" level, pre-training a detection foundation model that is as general as possible in both categories and domains, then adapting to specific scenarios through lightweight fine-tuning or text prompts; these models typically combine large-scale detection annotations, multi-source image–text pairs, and even video data, with the goal of making "arbitrary text + arbitrary style image" general understanding possible.

In specific product forms, open-vocabulary/open-world/open-domain detection often manifests as "more natural, less restricted" visual interfaces: users don't need to pre-agree on a small set of fixed labels, but can describe what they want to find in natural language; the system also doesn't need to retrain detectors from scratch for each business scenario, but can quickly adapt through prompts or few-shot samples based on a unified general model. For large-scale product/species recognition and globally deployed security and autonomous driving perception systems, this layer of capability is becoming the key springboard from "closed dataset performance" to "real open-world usability."

2.6.1 Open-Vocabulary Detection: From Fixed Category Heads to Text-Driven Category Spaces

The starting point of open-vocabulary detection is to break through the limitation of "fixed category heads" in traditional detection. Previous detectors attached a fixed-size classification layer at the top (corresponding to the N categories in the training set), and after training, could only choose among these N categories. Open-vocabulary detection, by introducing text encoders and a shared semantic embedding space, allows the region features output by the detection head to be compared for similarity with arbitrary text descriptions, thereby accepting unseen new categories during inference.

A typical approach uses CLIP-like vision–language pre-trained models:

• Text side: encode category names or natural language descriptions (e.g., "person," "red sports car," "yellow construction helmet") to obtain text vectors.
• Vision side: in the detection framework (Faster R-CNN, RetinaNet, YOLO, DETR, etc.), extract region feature vectors for each candidate region or feature point.
• Alignment training: through contrastive loss and Region–Word alignment loss, make text and region features of the same semantics close in the embedding space, and vectors of different semantics far apart. Even if explicit box annotations are only provided for some categories during training, semantic coverage can be expanded using image–text pairs or image captions.

During inference, the system no longer relies on a fixed set of class names from training, but allows users to provide arbitrary category words or natural language descriptions online, convert them to embeddings via the text encoder, and then perform similarity matching with region features. This enables the detector to support flexible requirements like "detect all skateboards," "detect all green plants," "detect all safety-related equipment" without retraining — even if certain specific categories never had complete annotations in the training set, as long as they semantically overlap with the pre-trained image–text space, they can be recognized and located to some degree.

In engineering practice, open-vocabulary detection needs to balance effectiveness and efficiency: on one hand, maintaining semantic alignment with the large-scale pre-trained vision–language backbone; on the other hand, meeting the detection task's requirements for multi-scale and real-time performance. Mainstream CLIP-based detectors often adopt the approach of "pre-computed text embeddings + efficient vector similarity computation" to avoid repeatedly encoding text during online serving, while also quantizing or distilling region features to balance accuracy and inference speed.

2.6.2 Open-World Detection: From "Unseen Classes" to "Learnable Unknowns"

Open-world detection, building on open-vocabulary, further requires the model to explicitly handle "unknown classes": the training data only annotates some categories, while other objects are either unannotated or collectively labeled as background. During inference, these "unannotated real objects" should neither be simply treated as background nor incorrectly classified into known categories, but should be detected as "unknown" and have the potential to be subsequently converted into "new known classes."

In modeling, open-world detection typically needs to address three problems:

1. Unknown class perception: how to avoid learning all unannotated objects as "background" during training? Common approaches include: introducing explicit "unknown class" slots, using negative example mining and uncertainty modeling to make the model output "unknown" in low-confidence regions; or using unannotated data and self-supervised mechanisms to cluster and generate pseudo-labels for high-confidence potential object regions.
2. Misclassification control: the model needs to balance "preferring to judge as unknown rather than incorrectly classifying into wrong known classes," which involves loss design (e.g., margin, open-set discrimination), decision thresholds, and post-processing strategies.
3. Progressive category expansion: when business teams manually annotate new categories for a batch of "unknown" objects, the model should be able to incorporate these new categories into the "known class" set through incremental learning without significantly forgetting old classes. To this end, many works introduce memory banks, distillation losses, parameter isolation, or replay mechanisms to achieve stable absorption of new categories.

From a product perspective, open-world detection is particularly suitable for scenarios where categories are constantly growing and long-tail is extremely severe, such as natural species recognition, product recognition for rapidly emerging new items, and abnormal object detection in complex security scenarios. The system can first use open-world detection to mark "any non-background suspicious objects," and gradually upgrade valuable clusters to formal categories through manual or semi-automatic annotation, thus forming a detection system where "categories can sustainably grow," rather than being constrained by fixed datasets.

2.6.3 Open-Domain / Open-Distribution Detection: Robustness Across Styles, Devices, and Scenarios

Even if the category set remains unchanged, detectors still encounter severe domain shift in real deployment: training data may come from high-definition daytime cameras in a few cities, while deployment environments include different countries, rural areas, highways, tunnels, nighttime, rain/snow, low-resolution cameras, fisheye lenses, and even infrared imaging. Huge differences also exist between e-commerce product photography and user-taken photos, or between ad images/illustrations/anime styles. Open-domain detection focuses precisely on: maintaining stable and reliable detection performance under conditions of significantly changed image distributions.

Typical technical paths include:

• Domain adaptation: with access to unlabeled or a small amount of labeled data from the target domain, using adversarial domain alignment (confusing source/target domains in feature space), multi-level domain alignment (image style, features, detection head outputs), style transfer (e.g., transferring source domain image style to target domain), etc., to make the model learn domain-insensitive features.
• Domain generalization: with only multiple source domain data and no target domain data, using multi-domain training, style randomization, feature perturbation, meta-learning, etc., to expose the model to diverse distributions as much as possible during training, improving generalization to unknown new domains.
• General / foundation detection models: by pre-training detection backbones and head structures on extremely large-scale, multi-source, multi-style data (including natural images, video frames, synthetic data, cross-modal data, etc.), then lightly fine-tuning for specific business scenarios, thereby achieving stronger open-domain robustness than "single-domain training."

These open-domain mechanisms are often superimposed with open-vocabulary/open-world capabilities: a general detection system for the real world needs to both understand users' natural language category descriptions (open-vocabulary), give reasonable "unknown" judgments and progressive absorption for newly appearing objects (open-world), and maintain performance across different countries, devices, weather conditions, and styles (open-domain). In engineering deployment, these three are not isolated research directions but together form the key capability combination for moving from "closed benchmarks" to "open-world usability."

2.7 Vision–Language Tasks

The previous sections mainly focused on "single-modality vision": the input is an image, and the output is detection boxes, segmentation masks, category labels, or quality scores. But in many real applications, visual information does not exist in isolation — an image often comes with captions, descriptive text, dialogue, or search queries. Users want to ask "what is happening in this image" or "does this image match this sentence." Vision–language tasks are precisely designed to solve these problems: they take image + text as input or output, and through cross-modal alignment and joint modeling, enable the system to "describe images in words," "answer questions about images," and "find images by text / find text by image."

From a product perspective, vision–language models (VLMs) are the hub capability of multimodal systems: search engines rely on them for "text-to-image / image-to-text search"; content platforms use them for smart image matching, ad moderation, and image–text consistency checking; multimodal assistants use them as foundational capabilities for "chatting about images" and "asking questions about documents/screenshots." Below, we organize this layer from three angles — scenarios, principles, and models — and expand on image captioning, visual question answering, and cross-modal retrieval in subsequent subsections.

• Scenarios
  • Image captioning: automatically generating one or two sentences of natural language description for an image, used for accessibility-assisted reading, smart album captions, and search index enrichment.
  • Visual question answering (VQA): users ask natural language questions about an image ("What is this person holding?" "What is the license plate number?"), and the system provides precise answers, applicable to education, assisted decision-making, and multimodal assistants.
  • Cross-modal retrieval: retrieving relevant images by text (Text-to-Image) and retrieving relevant text by image (Image-to-Text), supporting "text-to-image / image-to-text" search, creative image selection, and ad delivery moderation.
  • Image–text consistency and moderation: determining whether an image matches its caption/ad copy, whether there are risks like "image–text mismatch" or "misleading descriptions," used for content moderation and brand safety.
• Principles The core problem is: how to map images and text into the same semantic space and perform alignment and reasoning within this space:
  • Cross-modal alignment: through jointly trained image encoders and text encoders, making corresponding "image–text pairs" close to each other in the representation space and unrelated pairs far apart (typified by CLIP); this provides the foundation for retrieval and matching.
  • Joint understanding and generation: on top of aligned representations, introducing cross-modal attention so that the language model can generate text (image captioning), reason, and answer questions (VQA) while "looking at image features."
  • Prompting and instruction-based: using natural language instructions to uniformly describe various vision–language tasks ("write a caption for this image," "answer a question about this image," "determine whether this text describes the image"), allowing one model to complete multiple tasks through different prompts.
• Models Mainstream vision–language models have roughly evolved into two categories: contrastive learning VLMs and generative multimodal large models:
  • Contrastive learning type: CLIP, ALIGN, etc., encoding images and text separately into vectors, trained on large-scale image–text pairs, performing excellently on retrieval and matching tasks — the foundation for "text-to-image / image-to-text search."
  • Vision–language generative models: BLIP / BLIP-2, Flamingo, Kosmos, LLaVA, etc., connecting vision encoders with large language models (LLMs), supporting complex tasks like image captioning, VQA, and multi-turn dialogue through cross-modal attention and instruction fine-tuning.
  • General multimodal large models: GPT-4.1 with Vision, Gemini 1.5, etc., further unifying vision with more modalities (speech, code, etc.) in a single large model, completing retrieval, Q&A, reasoning, and generation through a unified interface.

Overall, vision–language tasks mark the point where "vision is no longer a separate perception channel" but participates together with language in higher-level knowledge representation and reasoning. Below, we expand on two directions: image captioning and visual question answering, and cross-modal retrieval and cross-modal alignment (merged into two subsections here based on content).

2.7.1 Image Captioning and Visual Question Answering: From "Describing Images" to "Reasoning About Images"

The goal of image captioning is to take an image as input and output a natural language description, such as "a little girl flying a kite on the grass." Traditional approaches typically used a "CNN + RNN" structure: extracting whole-image features with a convolutional network, then generating descriptions word by word with LSTM/GRU. With the emergence of Transformers and pre-trained VLMs, the mainstream paradigm has gradually shifted to an "image encoder + text decoder" structure, such as BLIP / BLIP-2, ViT + GPT, etc. For training, models are typically trained autoregressively on large-scale image–text pairs, sometimes also using reinforcement learning or contrastive losses to optimize description diversity and correctness. At the product level, image captioning is widely used for accessibility reading (generating image descriptions for screen reader software), automatically adding captions to smart albums, and providing more text indices for search systems.

Visual question answering (VQA) further brings human interaction into the picture: the model's input is no longer "image + blank prompt" but "image + question," and the output is a short answer or natural language explanation. Compared to image captioning, VQA emphasizes controllability and reasoning ability more: questions can focus on local details ("What color is the man's hat?"), relationships ("Which car is closer to the intersection?"), counting ("How many dogs are there?"), and even require external knowledge ("Which cuisine does this dish belong to?"). Early VQA models typically used an image encoder + question encoder + fusion module (e.g., bilinear pooling, attention) + classification head, outputting an answer from a limited vocabulary. Modern multimodal large models directly use an image encoder + LLM, performing natural language generation while "looking at the image," with clear advantages in open-ended answers and multi-turn dialogue.

Both can be viewed as different "prompt templates" under a unified VLM framework:

• Captioning: <image> + "Describe this image in one sentence." → text;
• VQA: <image> + "Q: ... A:" → text.

Through instruction tuning, the same multimodal large model can be compatible with multiple tasks like captioning, Q&A, explanation, and tagging — this is also the foundational engineering approach for modern VLM products (multimodal assistants, image Q&A bots, etc.).

2.7.2 Cross-Modal Retrieval and Cross-Modal Alignment: Text-to-Image & Image-to-Text Search

Cross-modal retrieval addresses another high-frequency need: given a piece of text, find matching images (Text-to-Image Retrieval); or given an image, find related text descriptions, product information, news reports, etc. (Image-to-Text Retrieval). These capabilities form the core of products like "text-to-image / image-to-text search," "find products by image," and "match images to news articles."

The core technology is cross-modal alignment: models represented by CLIP use separate encoders for images and text (e.g., ViT and a Transformer text encoder), trained with contrastive learning on large-scale image–text pair data:

• For the same (image, text) pair, make their vectors close to each other in the embedding space;
• For mismatched image–text pairs, push their vectors apart.

After training, simply encode all images and texts into vectors, and fast matching can be performed in the shared space through vector retrieval (nearest neighbor search):

• Text-to-Image: text → text vector → nearest image vectors;
• Image-to-Text: image → image vector → nearest text vectors.

In engineering practice, these models typically use a two-stage structure:

• The first stage uses a lightweight, fast bi-encoder (e.g., CLIP) for coarse retrieval, quickly filtering a small set of candidates from a billion-scale image library;
• The second stage can optionally use a stronger cross-encoder or multimodal large model for fine-ranking and re-ranking of candidates to improve relevance and robustness.

On the product side, cross-modal retrieval and alignment are widely used in: image search, ad retrieval (finding suitable images based on ad copy), compliance moderation (checking ad image–text consistency), content recommendation (recommending relevant images/videos based on users' reading text history), etc. With the rise of multimodal large models, these retrieval capabilities are gradually being unified into larger multimodal frameworks, provided as unified interfaces in the form of "natural language instructions + multimodal memory/vector database."

2.8 Optical Character Recognition (OCR)

In many businesses, the most important information is neither reflected in "objects and scenes in the image" nor in natural language descriptions of the image, but is directly written as text on the image: contract clauses, invoice amounts, street sign names, meter readings, error messages on screenshots, etc. Optical character recognition (OCR) is precisely the structured understanding task centered on "image + document layout": automatically detecting and recognizing text content from complex visual inputs, understanding document layout and structure, and further supporting search, statistics, automatic data entry, and intelligent Q&A.

From a product perspective, OCR is the key bridge for "turning paper/image information into computable text" and is the infrastructure for digitization, automation, and intelligent office work: contract review, invoice entry, government and enterprise archive digitization, PDF-to-Word in office software, document Q&A assistants, etc., are all built on OCR capabilities. Below, we organize the OCR system from three angles — scenarios, principles, and models — and expand on core directions in subsequent subsections.

• Scenarios
  • Scene text recognition: store signs, street signs, billboards, packaging text in street scenes, used for navigation, search, retail insights, and compliance moderation.
  • Document OCR: text recognition and structuring of scanned documents, faxes, PDFs, photos of contracts/invoices/reports, etc., restoring them to editable text.
  • Specialized scenarios: license plate recognition, meter reading (electricity, water, gas meters), screenshot text extraction, exam paper/form recognition, etc.
  • Document understanding: in long documents with complex layouts, extracting structures like titles, paragraphs, tables, and annotations, laying the foundation for search, summarization, and Q&A.
• Principles The OCR system is typically divided into several key steps:
  • Text detection: detecting all text regions (text lines or text blocks) on the image, outputting positioning boxes (horizontal or quadrilateral polygons), which serve as input for subsequent recognition.
  • Text recognition: performing sequence recognition on each detected text region, converting pixel sequences into character sequences (e.g., Chinese, English, numbers, symbols).
  • Layout analysis: in document scenarios, identifying the roles of various regions (title, body text, image, table, header, footer, etc.), recovering reading order and hierarchical structure.
  • Table structure recognition: performing row/column division, cell boundary parsing, and merged cell recovery on table regions, reconstructing logical table structures.
  • Document visual question answering (DocVQA): building on OCR and layout understanding, enabling the model to answer questions like "What is the payment date of this contract?" or "What is the amount on this invoice?" — questions requiring cross-region, multi-step reasoning.
• Models In engineering, a common combination is "specialized OCR modules + document understanding models + multimodal large models":
  • Text detection and recognition:
    • Detection: EAST, DBNet/DBNet++, etc., segmentation-based or edge-learning-based methods, adept at handling curved text and complex backgrounds;
    • Recognition: CRNN, RARE, SAR, etc., sequence models (CNN + RNN/Attention + CTC or autoregressive decoding), supporting multiple languages and fonts.
  • Document layout and structure understanding:
    • LayoutLM / LayoutLMv2/v3, DocFormer, etc., jointly encoding text content (token), position information (bounding box), and visual features;
    • Donut and other "end-to-end document understanding" models, going directly from image to structured output (e.g., JSON / Markdown), weakening the boundaries of traditional OCR.
  • Document Q&A and multimodal understanding:
    • Stacking task heads on top of layout models for DocVQA;
    • Or directly using multimodal large models (VLMs) to read document images, completing Q&A and summarization at the natural language level while implicitly leveraging OCR capabilities.

Overall, OCR has evolved from early "simple character recognition" to a comprehensive document understanding system covering text + layout + structure + Q&A, and is a key pillar of enterprise digitization, government archive management, and intelligent office work. Below, we expand on three directions: text detection and recognition, document layout and table structure analysis, and document Q&A and multimodal DocVQA.

2.8.1 Text Detection and Recognition: From Pixels to Usable Text

The first step of OCR is text detection: finding all text-containing regions in the input image. Street/scene text faces challenges like diverse fonts, skew and distortion, complex lighting, and severe background interference; document scenarios emphasize robust support for dense text and multi-column layouts. Methods like EAST, DBNet, etc., transform the detection problem into "pixel-level segmentation + edge learning," predicting text probability and geometric parameters on feature maps, then obtaining precise text boxes (which can be horizontal boxes or arbitrary quadrilaterals/polygons) through post-processing, balancing accuracy and speed.

Text recognition then crops each detected text region and converts it into a character sequence. The classic approach is represented by CRNN: first extracting features with CNN, then performing sequence modeling with RNN or Transformer, and finally outputting character sequences using CTC or attention decoding. For variable-length text, curved text, and complex languages (Chinese-English mixed, multilingual), recognition models need to simultaneously excel at visual feature modeling and character language modeling. Methods like RARE, SAR, etc., introduce spatial transformer networks (STN) or attention alignment mechanisms to correct geometric distortions and improve adaptability to complex layouts.

In engineering systems, detection and recognition typically serve as two decoupled services forming an OCR pipeline: the frontend detection breaks the image into several text lines/blocks, and the backend recognition performs character recognition on each block, optionally overlaying a language model for error correction (e.g., spell correction, number/amount validation). For specific scenarios like license plates and meter readings, specially fine-tuned detection/recognition models are used to leverage scene priors (fixed fonts, limited character sets) for higher accuracy and lower latency.

2.8.2 Document Layout and Table Structure Analysis: Restoring the "Shape of the Document"

Simply recognizing the text is not enough, especially in scenarios like long documents, reports, contracts, and invoices, where layout structure often determines the meaning and importance of information: the hierarchical relationship between titles and body text, the positioning of figures and captions, the role of headers and footers, the logical order of text inside and outside tables, etc. The goal of document layout analysis is to identify the roles and boundaries of different regions on a two-dimensional page and recover a reasonable reading order and hierarchical structure.

Models like LayoutLM / LayoutLMv2/v3, DocFormer, etc., jointly encode the content of each text token (text embedding), spatial position (bounding box coordinates), and local visual features (from CNN/ViT), modeling semantic–spatial relationships between tokens through Transformers. By training on datasets with layout annotations, the model can learn to distinguish various region types such as "title/paragraph/list/table/figure caption/header/footer" and output corresponding labels and hierarchies. These models typically serve as a "middle layer," providing structured document skeletons for contract review systems, report parsing, and archive digitization platforms.

Table structure recognition is a particularly critical branch of layout analysis: it not only needs to detect table regions but also further parse row/column boundaries, cell coordinates, and merged cells, ultimately reconstructing a logical table (typically represented as HTML, Markdown tables, or structured JSON with coordinates). Implementation methods include:

• Rule/vision-based: using line detection, segmentation networks, object detection, etc., to extract table lines and cell regions, then performing topological graph construction;
• Transformer-based: encoding text blocks and geometric information of table regions into sequences, directly predicting cell structures and association relationships.

In products, these capabilities support high-value scenarios like "PDF to Word/Excel," "structured entry of receipts/invoices," and "report parsing and metric extraction," and are key components of government and enterprise office automation.

2.8.3 Document Q&A and DocVQA: From "Reading Documents" to "Asking Documents"

When OCR and layout analysis capabilities are strong enough, the next natural demand is: no longer having people flip through documents themselves, but directly "asking the document." This is document visual question answering (DocVQA): the model answers questions on complex documents like contracts, reports, receipts, and manuals — for example, "What is the effective date of this contract?" "What is the net profit for Q4 2023 on this report page?" "Who is the buyer on this invoice?"

Traditional DocVQA systems are typically built as "OCR + layout model + QA head":

• First, use OCR to extract text and coordinates;
• Use LayoutLM / DocFormer, etc., to model the text–layout–visual tri-modal relationships;
• Finally, stack a task head (classification / extraction / span prediction) on this representation to locate answers or relevant passages in the document based on the question.

With the development of multimodal large models, more and more systems are starting to directly use "document image + question" as input, letting a VLM or multimodal LLM directly generate answers or cited explanations. Under this architecture, OCR, layout, semantic understanding, and reasoning capabilities work together end-to-end within the model: the model can both see the original layout and visual cues and leverage linguistic world knowledge and reasoning patterns to complete complex question answering.

In product form, DocVQA typically appears as "contract review assistant," "invoice/report Q&A," "long document intelligent Q&A," helping users quickly locate key information from large volumes of documents, automatically generate summaries, perform clause comparison, etc., significantly reducing the burden of manual review and information retrieval.

2.9 Image Generation & Editing

Most of the vision capabilities introduced earlier are "discriminative": input an image, output labels, boxes, masks, or text. Another main line that has rapidly developed in recent years is generative vision: models no longer just understand images, but create or modify images, generating high-quality, multi-style visual content given text/image conditions. Image generation and editing is precisely the core capability in this direction, supporting a large number of products from AIGC drawing platforms to intelligent retouching/effects tools.

From a business perspective, generative vision has evolved from "technical demonstrations" to genuinely usable productivity tools: designers use it for inspiration sketches and refined drafts; marketing teams use it to batch-generate posters and ad materials; ordinary users use it to create avatars, illustrations, and wallpapers; video creators use it for cutout, background replacement, and effects. Below, we organize this layer from three angles — scenarios, principles, and models — and expand on text-to-image generation, image-to-image, and editing capabilities in subsequent subsections.

• Scenarios
  • Text-to-image generation: users input a description ("a cyberpunk-style night city"), and the system automatically generates multiple images matching the description, supporting image selection and iterative refinement.
  • Style transfer and image translation: converting real photos to anime/sketch/oil painting/watercolor styles, or mapping between different domains (day ↔ night, summer ↔ winter).
  • Conditional inpainting and expansion: locally repainting parts of the original image (inpainting), expanding beyond the frame (outpainting), for fixing defects, removing/adding objects, and expanding composition.
  • Text-driven editing: modifying images with natural language instructions ("change the sky to sunset," "make this car a red sports car"), so users don't need to master complex image editing software.
• Principles Generative vision models primarily complete generation and editing by learning "image distributions" and "conditional control":
  • Distribution modeling: GANs, diffusion models, Flow Matching, etc., learn high-dimensional distributions from large volumes of images, enabling models to gradually "sample" realistic images from random noise.
  • Conditional generation: on top of pure image distribution modeling, introducing conditions like text/sketch/segmentation map/keypoints/depth map, so that the generation process is constrained by external signals (Text-to-Image, Image-to-Image, ControlNet, etc.).
  • Controllable editing: in the latent space of existing images, guiding and modifying local features through text or local masks to achieve local inpainting, style changes, composition adjustments, etc.
• Models Current mainstream image generation and editing models are primarily diffusion models + conditional control:
  • GAN series: StyleGAN, etc., perform outstandingly in high-resolution faces and style control; but training is unstable and difficult to cover complex multimodal distributions.
  • Diffusion models: Stable Diffusion, Imagen, DALL·E series, etc., sample through a "forward noising + reverse denoising" process, combining quality and diversity — the current main direction for Text-to-Image.
  • Controllable generation and editing: ControlNet, T2I-Adapter, etc., stack conditional channels (edges, poses, segmentation, etc.) on top of base diffusion models for precise control; combined with text-guided inpainting/outpainting for local editing and frame expansion.
  • Flow Matching and next-generation generative models: transforming noise distributions to image distributions by learning continuous flow fields, exploring new balances in efficiency, controllability, and stability.

At the product level, these technologies are presented to users in forms like Jimeng, Alibaba Qwen image models, FLUX, OpenAI or Gemini NanoBanana, Stable Diffusion ecosystem, Photoshop Generative Fill, Canva AI, Jianying/CapCut smart cutout and effects, gradually evolving from "toys" to formal links in the content production chain. Below, we expand on three directions: text-to-image generation, image-to-image translation, and text-driven editing.

2.9.1 Text-to-Image Generation: From a Sentence to a Picture

The core task of text-to-image generation is: given a natural language description, generate an image that matches its semantics and style as closely as possible. Modern text-to-image models are primarily based on diffusion architectures:

• First, use a text encoder (e.g., CLIP Text Encoder or T5/LLM) to encode the input text into a conditional vector;
• Then, in the image latent space, starting from a high-noise state, perform multi-step reverse denoising sampling, using the text condition to guide the generation direction at each step;
• Finally, obtain a high-resolution image matching the description, which can be further upscaled or post-processed.

Stable Diffusion, Imagen, DALL·E series, and other methods are trained on large-scale image–text pairs, enabling the model to both master the visual spectrum (shapes, textures, composition, lighting) and acquire a certain degree of language–vision alignment capability (understanding complex descriptions like "style," "material," "composition"). At the product level, this capability allows "people who can't draw to create images": users simply describe their ideas in natural language, and the system provides multiple visual implementations, supporting iterative exploration and refinement.

Text-to-image models typically support multi-style, multi-resolution output simultaneously: by incorporating style tokens, size conditions, etc., during training or inference, the same model can switch between different styles like "photorealistic, flat illustration, 3D render." Common engineering techniques include:

• Text prompt engineering for refining and stabilizing output style;
• LoRA / DreamBooth and other lightweight fine-tuning techniques, quickly adapting general models to specific characters, IPs, or brand styles.

2.9.2 Image-to-Image: Translation, Style Transfer, and Local Inpainting

Image-to-Image tasks, given an input image, generate another image version "constrained by it": preserving the overall structure or content of the original while achieving some transformation or enhancement. Typical forms include:

• Image translation / style transfer: mapping between different visual domains, such as "photo → anime," "summer → winter," "day → night," "sketch → color image." Early work was mostly based on GANs (CycleGAN, Pix2Pix, etc.), but diffusion models can now also accomplish this under conditional control.
• Conditional generation: using sketches, segmentation maps, depth maps, edge maps, etc., as conditions, guiding the diffusion process through modules like ControlNet, T2I-Adapter, so that the generated image strictly follows geometric/layout conditions while freely expressing texture, lighting, and style.
• Inpainting / outpainting: delineating a region on the original image to be treated as the area to repaint (inpainting), or generating new content extending beyond the frame (outpainting), achieving operations like "filling holes" and "expanding the image."

The key to these tasks is creating new content while preserving constraints. Diffusion models perform outstandingly here: in inpainting, the model only samples the masked region while keeping the original image unchanged in unoccluded areas, using semantic understanding and contextual information to naturally blend new content with surrounding areas in style and lighting. For style transfer, the model preserves the input structure while sampling textures and colors from the target style distribution, achieving "changing the shell without changing the bones."

In products, image-to-image capabilities support a large number of creative tools: style filters, comic conversion, one-click sky replacement, automatic beautification, old photo restoration, local retouching, etc., typically presented to users through highly visual interfaces.

2.9.3 Text-Driven Image Editing: Natural Language as the "Brush"

In traditional image editing software, users need to master a whole set of professional concepts like layers, masks, selections, and filters. Text-driven image editing attempts to replace most professional operations with natural language:

• "Change the background to a night city skyline";
• "Make this person wear a black suit";
• "Turn this car into a blue sports car, add motion blur effect."

Technically, text-driven editing is typically built on top of text-to-image diffusion models, implemented through several approaches:

• Searching or sampling in the latent space near the original image, so that the edited image maintains high similarity with the original, changing only in local areas affected by the text;
• Using explicit masks (user-delineated regions) to limit the editing scope to specific areas (this is the "select area then input text instruction" in many tools);
• Introducing "instruction control" modules (e.g., ControlNet, learnable control tokens) to enhance the model's controllability and stability for editing requests.

Jimeng, FLUX, Alibaba Qwen image models, the Stable Diffusion ecosystem, Canva AI, and other products all provide similar capabilities: users can complete complex edits through simple text and minimal interaction. For professional users, this becomes a "smart assistant" that accelerates the creative workflow; for ordinary users, it greatly lowers the barrier to image editing.

2.10 Image Quality Assessment (IQA)

In tasks like low-level vision enhancement, compression encoding, and image generation and editing, we often need to answer a seemingly subjective question: "Does this image look good?" Manual inspection clearly cannot scale, and traditional metrics like PSNR often do not align with human subjective perception. The goal of image quality assessment (IQA) is to establish an automated mechanism for scoring or ranking the subjective/objective quality of images, serving as the key link between "low-level algorithm output" and "real user experience."

From a system perspective, IQA is the "gatekeeper" and "tuning reference" in many pipelines: e-commerce/content platforms use it to filter out blurry, noisy, heavily compressed uploaded images; phone cameras/albums use it to pick the "best shot" from a burst; cloud-based enhancement and compression services use it for before-and-after comparison evaluation to guide model iteration. Below, we organize IQA from three dimensions — scenarios, principles, and models — and expand on evaluation types and metrics/learning paradigms in subsequent subsections.

• Scenarios
  • Upload quality inspection and moderation: scoring the quality of user-uploaded images/videos, filtering out content with severe blur, exposure anomalies, obvious noise, and heavy compression artifacts.
  • Smart photo selection and deduplication: in phone albums and camera apps, selecting versions with better sharpness, expression, and composition from multiple similar photos, while identifying low-quality or redundant images for cleanup.
  • Enhancement/compression algorithm evaluation: in A/B testing of algorithms like image enhancement, denoising, super-resolution, and encoding/decoding, using IQA metrics to objectively measure "which strategy is better," assisting parameter search and model selection.
  • Poster/thumbnail auto-selection: automatically selecting frames with higher visual quality and appeal from videos or multi-image collections as cover or poster candidates.
• Principles The core of IQA is to characterize image quality from two dimensions: the degree of distortion relative to a reference image and the goodness of human subjective perception:
  • Full-reference IQA (FR-IQA): with a high-quality reference image available, comparing the image under evaluation with the reference pixel-by-pixel or feature-by-feature to measure the degree of distortion, used for algorithm development and experimental evaluation.
  • No-reference IQA (NR-IQA / Blind IQA): more common in real scenarios, with no reference image available — quality can only be inferred from the statistical features or deep features of a single image, requiring the model to learn "what kind of images the human eye likes" from large volumes of images and subjective ratings.
  • Pseudo-reference / reduced-reference IQA: in some scenarios, using an obtainable approximate version (e.g., pre-compression low-resolution image, model-predicted "clean image") as a reference to estimate the degree of degradation, balancing feasibility and evaluation accuracy.
• Models IQA models are roughly divided into two categories: traditional hand-crafted feature metrics and deep learning-based quality prediction:
  • Traditional metrics:
    • FR-IQA: PSNR, SSIM, MS-SSIM, FSIM, etc., emphasizing structure, contrast, and phase information, more sensitive to simple degradations (e.g., noise, blur).
    • Perceptual metrics: LPIPS, DISTS, etc., measuring perceptual differences between images in deep feature space, with higher correlation to human subjective perception.
  • No-reference / learning-based IQA:
    • Early methods: BRISQUE, NIQE, BLIINDS series, etc., starting from natural scene statistics (NSS) and hand-crafted features, training shallow models to predict quality scores.
    • Deep NR-IQA: RankIQA, DBCNN, HyperIQA, MUSIQ, etc., directly extracting features from images using CNN / ViT, and training with supervision on MOS (Mean Opinion Score) data to make output quality scores fit human eye evaluations as closely as possible.
    • Pre-trained representations: using features from large models like CLIP, ViT as input or backbone for quality prediction networks, fine-tuning on limited MOS data to improve generalization across complex distortion types.

Overall, IQA is not a single metric where "higher is always better," but an evaluation system related to specific business objectives: in some scenarios (e.g., surveillance enhancement), preserving detail and recognizability is more important than visual naturalness; in content creation platforms, subjective perception and aesthetic standards dominate. Therefore, common industry practice is: building on top of general IQA models, fine-tuning or learning weights with a small amount of business data to construct "task-aware" quality evaluators.

2.10.1 Evaluation Types: Full-Reference, No-Reference, and Pseudo-Reference

Based on whether a high-quality reference image exists, IQA can be divided into three categories: full-reference (FR-IQA), no-reference (NR-IQA), and pseudo-reference.

In full-reference IQA, we assume the existence of an ideal high-quality reference image, and the image under evaluation is its degraded version after compression, transmission, or processing. The model quantifies the degree of distortion by comparing the two pixel-by-pixel or at the feature level. PSNR is the simplest metric (based on mean squared error); SSIM/MS-SSIM/FSIM, etc., further consider brightness, contrast, structure, or phase information, coming closer to human visual perception to some extent. These metrics are very suitable for evaluating methods like encoding/decoding, super-resolution, and denoising during the algorithm development phase, but in real business, reference images are often unavailable, limiting application scenarios.

No-reference IQA (Blind IQA) is the more common setting in real systems: only the image under evaluation itself is available, with no reference at all. Early no-reference methods (e.g., BRISQUE, NIQE, BLIINDS, etc.) were mainly based on natural scene statistics: assuming that high-quality natural images have stable forms in certain statistical distributions, and that distortion causes changes in statistical features, allowing models to be trained to predict quality scores based on these features. In the deep learning era, NR-IQA models typically directly use CNN / ViT to extract features and regress quality scores or learn ranking relationships on datasets with human subjective ratings (MOS), enabling them to cover various distortion types like noise, blur, compression artifacts, and exposure anomalies.

Pseudo-reference / reduced-reference IQA falls between the two: without a truly high-quality reference, using some obtainable approximate version (e.g., pre-compression low-resolution image, model-predicted "clean image") as a reference to estimate the degree of degradation. This approach is common in online video quality monitoring and encoding/decoding optimization tasks, striking a balance between cost and accuracy.

2.10.2 Metrics and Learning Paradigms: From PSNR to Perceptual Quality Prediction

At the specific implementation level, IQA employs various metrics and learning paradigms to approximate human subjective perception.

Traditional metrics:

• PSNR is directly based on pixel-level error — simple and efficient, but it also heavily penalizes changes that are imperceptible to the human eye (e.g., slight translation, structure-preserving filtering);
• SSIM, MS-SSIM, FSIM, etc., model image similarity from multiple dimensions like brightness, contrast, structure, and phase, being more sensitive to structural distortions and also reflecting to some degree the human eye's preference for structural information.

Perceptual metrics: LPIPS, DISTS, etc., compute vector differences in the internal feature layers of pre-trained deep networks (VGG, AlexNet, ViT, etc.), weighted by the importance of different layers, yielding a "distance in feature space" that has higher correlation with subjective perceptual similarity. They are particularly suitable as training objectives or evaluation metrics for generative tasks (super-resolution, generation, editing), used to measure "how similar it looks."

Learning-based quality prediction: deep NR-IQA models (e.g., RankIQA, DBCNN, HyperIQA, MUSIQ, etc.) directly score or rank images:

• In training data, each image comes with a set of subjective ratings (MOS), and the model is trained with this supervision for quality regression or ranking networks;
• In model structure, CNN/ViT + global pooling + MLP are commonly used to output quality scores, or output a quality distribution and take the expectation;
• Some methods also use contrastive learning or pairwise ranking to make the model focus more on "relatively good/bad" relationships rather than absolute scores.

With the widespread adoption of large-scale pre-trained vision models, more and more IQA methods adopt the "pre-trained backbone + lightweight head" paradigm: leveraging rich visual representations from CLIP, ViT, etc., and fine-tuning on relatively little MOS data, thereby maintaining good generalization across distortion types and scenarios.

In engineering deployment, multiple of the above metrics are typically combined: for example, FR-IQA metrics for evaluating algorithm improvements during the experimental phase; deep NR-IQA models for online real-time quality inspection; perceptual metrics for internal optimization of generative tasks. Through A/B experiments, these automatic metrics are aligned with real user data (click-through rate, completion rate, complaint rate, etc.), gradually building a "perceptual quality measurement system" highly correlated with business objectives.