5. Video

In the multimodal AI system, the video modality is responsible for understanding and generating "visual signals that change over time." Compared to single-frame images, video not only contains spatial information such as texture, shape, and layout, but also carries rich temporal cues: the rise and fall of actions, the motion trajectories of objects, the rhythm of shot transitions, and so on. Whether it's behavior recognition in surveillance, motion analysis in sports training, one-click editing on short-video platforms, or intelligent parsing of long videos, they all fundamentally rely on a comprehensive set of understanding and generation capabilities built around "frame sequences."

From an engineering perspective, video capabilities can be roughly divided into several layers: low-level video enhancement and restoration ensures that content is "clear enough to see"; video understanding and structural analysis answers the question of "what is happening"; building on that, video + language multimodal tasks convert video content into structured descriptions and retrieval interfaces usable by text; further, video generation and editing works in reverse, starting from text or example videos to generate or reassemble video content in a controllable manner; and a class of applications represented by digital humans / virtual avatars integrates speech, language, motion, and video rendering together, forming a new paradigm for interaction and content production.

Below, we similarly start from layered capabilities to organize video-related capabilities.

5.1 Traditional Video Processing: From "Playable" to "Good-Looking and Useful"

At the most fundamental level of video technology, the first concern is not "who is in the frame" or "what event is happening," but whether the video itself is stable, clear, and comfortable: whether the picture shakes, is blurry, has excessive noise, or whether the aspect ratio suits the target playback device. The traditional video processing layer primarily works at the level of frame sequences and spatiotemporal pixels, using operations such as enhancement, restoration, super-resolution, frame interpolation, and reframing to convert noisy, shaky, low-resolution, or improperly proportioned raw video into "high-quality temporal signals" that are more suitable for viewing and subsequent analysis. It can be analogized to "image restoration and enhancement + geometric correction" in the image modality, except that here, smoothness and consistency across the time dimension are additionally introduced.

From a product perspective, this layer of capabilities is almost "invisible" behind all video products: one-click quality enhancement in editing software, automatic quality upgrades on short-video platforms, intelligent super-resolution and frame interpolation in TV boxes and players, film restoration services, and multi-frame preprocessing for upstream detection/recognition models are all direct manifestations of traditional video processing. Below, we continue to organize from three angles — scenarios, principles, and models — and expand on key directions such as video enhancement and restoration, super-resolution, and frame interpolation in subsequent subsections.

• Scenarios In online video platforms, editing tools, surveillance systems, and terminal devices, traditional video processing mainly appears in the following typical scenarios:
  • Content platforms and editing tools: When short videos or long videos are uploaded or edited, one-click quality enhancement, image stabilization, anti-shake, and denoising allow users to "pick up a phone and shoot, then use it right away"; when old video footage is imported into an editing project, restoration and frame supplementation make it more visually consistent with new material.
  • Film and old footage restoration: Digital restoration of historical film, early TV programs, and standard-definition material — removing scratches, noise, and jitter, restoring color and detail — to provide higher-quality versions for re-release, redistribution, and digital archiving.
  • Video surveillance and dashcams: Denoising, dehazing, contrast enhancement, and image stabilization for surveillance footage shot in low light, rain, fog, or heavy compression, improving the robustness of downstream detection and recognition modules and facilitating forensics and traceability.
  • Terminal playback and device-side enhancement: TVs, set-top boxes, and mobile players locally integrate super-resolution and frame interpolation to "upgrade" existing 720p/1080p, 24/30fps content to visual effects approximating 4K, 60/120fps at playback time.
  • Multi-terminal adaptation and distribution: To cover mobile portrait, tablet landscape, and large-screen TV simultaneously, the same video undergoes portrait/landscape adaptation, intelligent cropping, and multi-aspect-ratio reframing, reducing the cost of manual editing and multi-version maintenance.
• Principles Traditional video processing typically does not directly understand semantic categories; instead, it models and optimizes around quality, stability, and temporal consistency at the spatiotemporal signal level:
  • Joint spatiotemporal modeling: Building on single-frame image enhancement, temporal information is introduced. Through optical flow estimation, camera motion modeling, or spatiotemporal convolution, preceding and following frames serve as additional "observations," enabling multi-frame fusion and noise suppression along the time axis.
  • Image stabilization and anti-shake: Camera shake is modeled as a sequence of geometric transformations (translation, rotation, scaling, etc.) over a period of time. By estimating global or local motion trajectories and smoothing them before reprojecting onto the output video, shake is removed and stability is achieved.
  • Video super-resolution and frame interpolation: Video super-resolution uses multi-frame alignment and detail reconstruction to improve spatial resolution while maintaining temporal consistency; frame interpolation uses optical flow estimation or spatiotemporal generation networks to synthesize intermediate frames between two frames, presenting motion at a higher frame rate for improved smoothness.
  • Reframing and automatic composition: By detecting and tracking subjects (people, objects) in the video, estimating subject trajectories along the time axis, and combining with the target resolution's aspect ratio, an appropriate cropping window is selected for each frame, and the cropping window's motion is temporally smoothed to ensure a natural viewing experience.
  • Quality vs. efficiency trade-offs: Cloud offline processing can pursue optimal quality and complex models, while mobile, player, and real-time scenarios require controlling model parameter counts, computational complexity, and latency, making fine trade-offs in algorithm architecture and inference frameworks.
• Models In concrete implementation, traditional video processing comprehensively uses classical video signal processing methods and deep learning models, striking a balance between effectiveness, efficiency, and deployment form:
  • Classical video processing methods: Optical-flow-based stabilization and frame interpolation, temporal filtering and multi-frame fusion, block-matching-based denoising and compression artifact removal, etc., are still widely used in scenarios with limited compute or where interpretability is required.
  • Deep video restoration and enhancement models: Multi-frame super-resolution and enhancement networks represented by EDVR, BasicVSR / BasicVSR++, and the video version of Real‑ESRGAN, through alignment and spatiotemporal feature aggregation, significantly outperform traditional methods in denoising, deblurring, detail recovery, and compression artifact removal.
  • Deep frame interpolation models: Frame interpolation networks such as DAIN, RIFE, and FILM generate intermediate frames through explicit or implicit optical flow estimation and intermediate feature fusion, and are more stable than traditional optical flow + resampling methods in complex motion and occlusion scenarios.
  • Transformer-based video restoration: Using spatiotemporal attention to uniformly process spatial textures and temporal dependencies, these models offer stronger modeling capability in complex camera motion and multi-object scenes, while controlling computation at inference time through mechanisms such as sparse attention and sliding windows.
  • Real products and systems: The intelligent enhancement in Jianying / CapCut, commercial enhancement software such as Topaz Video Enhance, the quality enhancement pipelines of Bilibili and various short-video platforms, and old film restoration SaaS services typically cascade multiple models and strategies, dynamically selecting the optimal processing path based on material type and terminal conditions.

Taken together, this layer primarily lays the physical and perceptual foundation for video "before semantics": it helps users obtain a more comfortable viewing experience and also provides cleaner, more stable input for upstream detection, recognition, and generation models. Below, we expand on the sub-directions of video enhancement and restoration and super-resolution and frame interpolation.

5.1.1 Video Enhancement and Restoration: Polishing "Watchable" to "Good-Looking"

Under real shooting conditions, video is often not "clean": severe shaking from handheld devices, high noise and a smeared look in low light, block artifacts and color banding from network compression, and fading and scratches from old equipment all make video quality noticeably below the ideal. The goal of video enhancement and restoration is to restore a stable, clear, and natural viewing experience to the greatest extent possible without changing the semantic content of the video — polishing material that is "barely watchable" to a level that is "pleasing or even good-looking."

In the temporal domain, enhancement and restoration must first address the problem of stability. By performing feature matching or optical flow estimation on consecutive frames, global camera motion and local object motion can be separated, and the smoothed camera trajectory is then used to re-render output frames, thereby suppressing rapid jitter and subtle shaking and preventing viewers from experiencing dizziness during viewing. On this foundation, frame-level denoising, deblurring, and artifact removal focus more on joint spatial–temporal modeling: multi-frame joint denoising uses redundant information from preceding and following frames, performing processing similar to "multi-exposure fusion" in the temporal direction, effectively suppressing high-ISO noise and compression noise while preserving detail texture; for mild motion blur, deconvolution-style sharpening is performed on the frame sequence by estimating the blur kernel or using an end-to-end deep network, making both static backgrounds and moving subjects sharper.

For old films and low-quality material, restoration also involves "reconstruction" at the color and structure level. Film aging causes yellowing, reduced contrast, and noticeable local scratches and blemishes; early digital video commonly suffers from low resolution, heavy compression, and edge aliasing. Modern restoration workflows often use multi-step collaboration: first, detection and segmentation models locate locally damaged areas such as scratches and blemishes; then, spatiotemporal inpainting networks "borrow material to fill holes" from neighboring frames and neighboring spatial pixels; simultaneously, color restoration and contrast reshaping bring the overall tone close to the original shooting or intended style reference. For heavily compressed video, dedicated artifact-removal networks targeting blocking artifacts and ringing artifacts are also introduced, improving edges and details without excessive smoothing.

These enhancement and restoration capabilities are often presented as "one-click" in products: the user simply checks "stabilization," "quality enhancement," or "old video restoration," and the system automatically selects the appropriate model and parameter combination in the background, performing multi-stage processing on the video frame sequence. For the business, this layer directly determines the audience's subjective evaluation of image quality and indirectly affects the performance of upstream analysis models: cleaner, more stable video input often means more reliable face/license plate recognition, more accurate behavior detection, and fewer false alarms.

5.1.2 Super-Resolution and Frame Interpolation: From "Barely Clear" to "Smoother"

Against the backdrop of ever-upgrading display devices and increasing user demands for detail and smoothness, a large amount of existing video content appears "innately inadequate" in resolution and frame rate: 1080p looks insufficiently sharp on 4K screens, and 24/30fps tends to show ghosting or stuttering on large screens and in fast-motion scenes. Super-resolution and frame interpolation technologies are designed to solve these two problems: the former "fills in details" in the spatial dimension, and the latter "fills in the process" in the temporal dimension, together elevating video that is "barely clear" to a viewing experience that is "rich in detail and smoothly playing."

Video super-resolution adds one key dimension compared to single-frame image super-resolution: time. Simple frame-by-frame upscaling tends to cause inconsistency in details between adjacent frames, resulting in flickering and texture jitter. Therefore, mainstream methods all leverage information from multiple surrounding frames, using optical flow estimation or feature-level alignment to align details from neighboring frames to the target frame, and then performing detail reconstruction after alignment. Models such as EDVR, BasicVSR / BasicVSR++, and the video version of Real‑ESRGAN first align and aggregate multiple frames in feature space, then use deep networks to infer high-resolution details, avoiding the "blur" and "plastic feel" that come with simple interpolation. In this process, balancing "physical plausibility" and "perceptual attractiveness" is the core of loss design and training strategy: both objective metrics (such as PSNR, SSIM) must be improved, and subjective viewing must feel natural, without over-sharpening or false details.

Frame interpolation focuses on "filling frames" along the time axis. Traditional methods rely on optical flow estimation, first predicting the motion of each pixel between two frames, then interpolating at an intermediate position according to certain rules to generate a new frame. However, in areas of fast motion, multi-object occlusion, or complex texture, optical flow is often not accurate enough, easily producing ghosting, double images, or local deformation. Deep frame interpolation models such as DAIN, RIFE, and FILM simultaneously learn fusion strategies for optical flow, depth, or intermediate features through end-to-end networks, directly outputting interpolated frames, with noticeably improved stability and visual quality in complex scenes. For sports events, action game recordings, and slow-motion creation, frame interpolation can smoothly boost 24/30fps original video to 60/120fps, preserving motion detail while reducing stutter and ghosting.

In engineering practice, super-resolution and frame interpolation are often used together: for low-resolution, low-frame-rate existing content, temporal frame interpolation is performed first, followed by spatial super-resolution, or both are implemented in a unified spatiotemporal network. In terms of deployment, cloud offline processing is suitable for film restoration and platform-level "quality upgrade" services that demand the highest image quality, while device-side real-time inference is more commonly seen in TV boxes, player apps, and gaming/action cameras, requiring low latency through model compression and hardware acceleration. Regardless of the form they take, super-resolution and frame interpolation have become essential infrastructure for the "HD/UHD experience," giving old content a "second life" on new terminals.

5.2 Video Understanding and Structural Analysis

If traditional video processing stays more at the level of "image quality and stability," then video understanding and structural analysis begins to answer semantic questions of the type "what is happening in the video": who is doing what, where, for how long, and whether there is abnormal behavior. The goal here is to structurally decompose the video along the time axis: recognize actions and behaviors, detect and track targets, segment foreground and background, partition scenes and shots, and extract high-level semantic signals usable for downstream decision-making, retrieval, and alerting.

From a product perspective, this layer of capabilities has already penetrated various intelligent security platforms, sports training analysis systems, smart dashcams, and industrial quality inspection video analysis systems: identifying fighting, falling, loitering, and other anomalies in surveillance; analyzing movement standardization and technical details in sports and fitness scenarios; tracking vehicle and person trajectories and monitoring whether production processes are normal in traffic and industrial environments. Below, we continue to organize these capabilities from three angles — scenarios, principles, and models — and focus on several representative directions in subsequent subsections.

• Scenarios
  • Security and public safety: In urban surveillance, campuses, and buildings, identify behaviors such as fighting, falling, gathering, running, and climbing over fences, and provide early warnings for abnormal patterns such as loitering and late-night lingering.
  • Traffic and travel: Detect and track the trajectories of pedestrians, vehicles, and bicycles at intersections, in tunnels, and on highways; analyze behaviors such as running red lights, driving against traffic, occupying lanes, and speeding, providing a basis for traffic management and accident tracing.
  • Sports and athletic training: Analyze the key phases and posture quality of actions such as basketball shooting, tennis serving, and yoga poses, providing technical analysis and corrective suggestions for athletes and general users.
  • Industrial production and quality inspection: Monitor whether operational steps on the production line are standardized, detect missing or incorrect assembly and abnormal movements during the assembly process, and provide foundational data for safe production and yield improvement.
  • Content structuring and retrieval: Perform shot segmentation, scene classification, and key moment marking on long videos, providing structured indices for subsequent retrieval, recommendation, and editing.
• Principles The key to video understanding and structural analysis is joint modeling of spatial targets and semantics in the temporal dimension:
  • Action recognition and behavior analysis: Based on 2D/3D convolution, temporal pooling, or Transformers, an entire video clip is encoded to recognize the action category occurring within it; advanced methods combine human keypoint sequences with skeletal topology for more fine-grained analysis of action quality and patterns.
  • Object detection and tracking: While performing detection on each frame, cross-frame association mechanisms (appearance features, motion trajectories, etc.) are introduced to connect the detection boxes of the same target at different times into continuous trajectories, yielding multi-object tracking results.
  • Video semantic segmentation and scene analysis: Semantic segmentation or instance segmentation is performed on each frame of the video at the pixel level, and temporal continuity is used to smooth predictions; simultaneously, shot transitions and scene boundaries are detected to achieve structural decomposition of long videos.
  • High-level event and anomaly detection: On top of basic action and trajectory features, temporal modeling and pattern recognition methods are used to detect rare events and abnormal patterns, often combined with unsupervised or weakly supervised learning to alleviate the problem of scarce annotations.
• Models In terms of model selection, video understanding and structural analysis typically adopt a combined architecture of "spatial features + temporal modeling":
  • Classical models based on 3D convolution and Two‑Stream, such as I3D, perform end-to-end action recognition on short video clips by convolving simultaneously in spatial and temporal dimensions.
  • SlowFast series models based on multi-path and multi-temporal scales capture semantics through a slow pathway and motion details through a fast pathway, achieving a better balance between computation and accuracy.
  • Transformer-based video models, such as TimeSformer and Video Swin Transformer, use spatiotemporal attention mechanisms to model videos over long time ranges, and are better suited to capturing complex events and multi-agent interactions.
  • Tube‑based detectors and spatiotemporal convolution / Transformer models extend detection boxes temporally into "tubes," performing action detection and spatiotemporal segmentation on joint spatial–temporal features.
  • Multi-object tracking (MOT) methods, such as DeepSORT, combine frame-level detection results with appearance embeddings and motion prediction to stably associate target identities across the video.

Overall, this layer of capabilities further abstracts video from a "high-quality pixel stream" into a "behavior and event stream," laying the structural foundation for upstream multimodal understanding, retrieval, and decision-making. Below, we expand on three directions: action recognition and behavior analysis, object detection and tracking, and event and anomaly detection.

5.2.1 Action Recognition and Behavior Analysis: From Frame Sequences to "Who Is Doing What"

Action recognition and behavior analysis is concerned with "what the subject is doing within a time window." In security scenarios, this means recognizing behaviors such as "walking, running, falling, fighting" from video; in sports and fitness, it corresponds to more fine-grained actions such as "whether the basketball shot, tennis serve, or squat is standard" and "whether the yoga pose is correct." Technically, early methods mainly relied on 2D convolution + optical flow or handcrafted features, stacking several frames and classifying them as a whole; modern methods more commonly use 3D convolution (I3D, a series of 3D ResNet variants), multi-temporal-scale structures such as SlowFast, or spatiotemporal-attention-based models such as TimeSformer and Video Swin Transformer to jointly model spatial textures and temporal changes.

In many scenarios requiring high-precision pose analysis, directly classifying RGB clips is insufficient; human pose estimation and skeletal sequence modeling are also incorporated: 2D/3D keypoints are first extracted from each frame, and the keypoint sequence is then fed into RNNs, temporal convolutions, or GCN/Transformer networks to analyze the temporal structure and spatial coordination of the action. This "pose prior + temporal modeling" approach is more robust to changes in background, lighting, and clothing, and is suitable for applications with high demands on action detail, such as yoga, fitness, and industrial operation compliance assessment.

5.2.2 Object Detection and Tracking: From "Where in This Frame" to "The Entire Trajectory"

Single-frame object detection can tell us "what targets are in this frame and where," but many real-world tasks require knowing "where this car / person came from, where they went, and what they did in between." The object detection and tracking module exists precisely to string frame-level detections into continuous temporal trajectories: on one hand, a detector runs on each frame, producing candidate bounding boxes; on the other hand, based on cues such as appearance features (ReID embeddings), motion prediction (Kalman filtering), and spatial overlap, boxes on adjacent frames are matched and associated, yielding multi-object tracking (MOT) results.

In engineering practice, a typical pipeline is: "robust pedestrian / vehicle detection + an association algorithm such as DeepSORT," deployed on surveillance cameras or dashcams, outputting the motion trajectory of each ID in real time. In more complex systems, these trajectories are further combined with regional semantics (lanes, zone divisions) and business logic rules to infer higher-level behavior patterns such as driving against traffic, prolonged loitering, and frequent entry/exit, providing continuous temporal signals for upstream security, traffic flow analysis, and industrial process monitoring.

5.2.3 Event and Anomaly Detection: Finding "What's Off" from "Normal Patterns"

In most business scenarios, what truly requires focused attention is often the "minority of anomalies" and "key events": for example, fighting, falling, and gathering in security; abnormal shutdowns or non-compliant operations in industrial production; dangerous driving behavior in traffic. Such events are relatively rare, with high annotation costs and extremely imbalanced samples, posing additional challenges for model construction.

A common approach is to build a temporal anomaly detection module on top of basic action recognition, object tracking, and scene segmentation: either directly learning from a small number of labeled anomaly samples through supervised methods; or using unsupervised/weakly supervised methods to model the motion and behavior distribution of "normal patterns," issuing an alert whenever a new observation significantly deviates from the historical distribution. At the model level, temporal autoencoders, contrastive learning, graph neural networks, or temporal Transformers are combined to uniformly encode spatial relationships and temporal dependencies, thereby capturing more complex group behavior patterns and long-range dependencies.

5.3 Video + Language Multimodal Tasks (Video‑Language)

If video understanding solves the problem of "understanding the video itself clearly," then video + language multimodal tasks focus on "how to use natural language to describe, answer questions about, and retrieve video content," as well as "how to quickly locate key information along the long video timeline based on textual needs." Such tasks require simultaneously processing visual, speech, and text signals: on one hand, extracting visual and audio features from the video; on the other, connecting to the reasoning and generation capabilities of language models, compressing spatiotemporal content into text summaries, Q&A results, and semantic indices suitable for human consumption and machine invocation.

From a product perspective, this layer of capabilities has already penetrated scenarios such as automatic subtitle and timeline generation for long videos, "smart marking / key clip extraction" on short-video editing platforms, and Q&A assistants for corporate training and meeting videos: users no longer need to "watch from beginning to end," but can directly search, ask questions about, and reorganize video content through natural language. Below, we continue to expand from three angles — scenarios, principles, and models.

• Scenarios
  • Subtitle and summary generation: Automatically generate multilingual subtitles for courses, lectures, meetings, and long-form video content, and on that basis generate chapter-level summaries, highlight lists, and timelines.
  • Video Q&A and knowledge access: Build "video Q&A assistants" for instructional videos, operation demonstrations, and corporate training content, supporting users in asking questions in natural language, such as "how is this step done" or "where did this person put the phone in the end."
  • Video content retrieval and clip localization: Support precise "text → video clip" retrieval in large-scale video libraries, for example, "find the part that mentions the price" or "find the clip that explains a certain formula"; automatically mark and annotate highlights and key information within a single long video.
  • Content production and editing assistance: Combine video content understanding with language generation to automatically generate titles, copy, and storyboard scripts, assisting creators in quickly editing and reorganizing material.
• Principles The core of video–language multimodal systems is to align temporal visual features with text representations in a unified embedding space, and to perform retrieval, generation, and reasoning on this foundation:
  • Multimodal feature extraction and alignment: Extract spatiotemporal features (CNN/ViT/Video Transformer) from video frames/clips and language embeddings (pretrained LLM or text encoder) from text, and align the two modalities through contrastive learning or multimodal pretraining.
  • Speech and text pipeline: For content containing speech, ASR is typically used first to generate timestamp-aligned transcription text, which is then jointly modeled with visual features — text can directly drive retrieval and also enable cross-modal comparison and error correction.
  • Temporal modeling and clip localization: For long videos, "clip-level" representations must be learned along the time axis, dynamically switching between local clips and global context through attention or temporal RAG to achieve precise localization of query-relevant intervals.
  • Generation and reasoning: Connect a large language model to the aligned multimodal representations to perform natural language generation (subtitles, summaries, explanations) or multi-turn Q&A and logical reasoning.
• Models In terms of model form, video–language multimodal tasks have undergone an evolution from "dedicated encoder + simple head" to "unified multimodal large models":
  • Early video–language models: Such as VideoBERT, which jointly model visual and text tokens during the pretraining phase, obtaining transferable video–language representations through masked prediction and contrastive learning.
  • All‑in‑One Video‑Language Models: Unify video, text (and speech) into a single multimodal Transformer, achieving unified processing of multiple tasks such as description generation, retrieval, and QA through shared or partially shared parameters.
  • Long-video multimodal models: Such as Gemini, Claude, and GPT with video capabilities, which use long contexts and hierarchical temporal modeling to holistically understand videos ranging from tens of minutes to hours, supporting timeline-level summarization and Q&A.
  • Temporal RAG + VLM: Build a "temporal vector index" on the video, first using a VLM to encode video clips and build a database, then retrieving relevant clips at query time, and combining with an LLM for answer synthesis and explainable reasoning.

Overall, this layer elevates video from "machine understanding" further to the level of "human–machine dialogue and collaboration": users can ask questions about a video as if asking a person, while the system performs complex visual, speech, and language alignment and reasoning behind the scenes.

5.3.1 Subtitles, Summaries, and Timelines: Compressing Long Videos into Browsable Text

For courses, lectures, meetings, and long-form content videos, the most urgent need is often to "quickly know what was said and where the key points are," rather than watching the entire thing from beginning to end. Automatic subtitle and summary systems use a combination of "ASR + text processing + visual assistance" to transcribe audio content into timestamp-aligned text, and then generate structured outlines and concise summaries on that basis, achieving information compression from "hour-long video" to "minute-level reading."

At the implementation level, the ASR module is responsible for stably and reliably producing multilingual transcription and timeline alignment; the text side uses a large language model to correct errors, segment sentences, and semantically reorganize the raw transcription, extracting chapter titles, key information, and question–answer pairs. In some scenarios, visual cues (such as PPT page changes, scene transitions) are also incorporated to assist in demarcating chapter boundaries and key clips, ensuring that the summary structure is more consistent with the rhythm of the actual content.

5.3.2 Video Q&A and Semantic Retrieval: "Manipulating" Video with Natural Language

Building on subtitles and summaries, a further need is the ability to perform Q&A and retrieval on specific video content: for example, "where did this person put the phone in the end," "which part talks about the pricing strategy," "at what minute is this step demonstrated." Such tasks require semantic localization of the query along the time axis: understanding the people, objects, and actions involved in the query, and finding the corresponding clip in the video's temporal representation.

In concrete terms, a multi-granularity index is typically built offline for the video: multimodal representations (visual + text/speech) are extracted for fixed-length clips, and a vector index or graph structure is established. During online interaction, the user's question is encoded as a text vector and matched against the clip representations in the index to find the most relevant time intervals; then, the content of these clips (keyframe screenshot descriptions, transcription text, etc.) is fed into an LLM together with the question, and the model generates a natural language answer or returns the corresponding time points. For large-scale video libraries, "cross-video retrieval" can be supported under the same mechanism, for example, searching across collections in corporate training knowledge bases or e-commerce product videos.

5.3.3 Multimodal Editing Assistance: From Understanding to "Helping You Edit"

Once the system can stably understand the content and semantic structure of a video, the natural next step is to reversely use these understanding results to assist in creation and editing. Video–language multimodal models can automatically select clips that match the semantics from existing material based on a script or prompt provided by the creator, generating a rough-cut timeline; they can also automatically generate titles, cover copy, and chapter labels based on the video content, and even suggest shot rhythm and soundtrack choices.

In workflows, such capabilities typically appear in the form of "intelligent recommendations" and "automatic rough cuts": after the creator uploads material, the system automatically completes analysis, storyboarding, and marking, and provides several candidate versions (such as editing plans with different rhythms and durations); the creator can fine-tune on this basis without needing to screen frame by frame from scratch. For enterprise applications, the system can also combine knowledge bases and brand guidelines to ensure that the generated copy, subtitles, and editing style comply with established business requirements and compliance standards.

5.4 Video Generation & Editing

After possessing stable understanding and structural analysis capabilities, video generation and editing steps into the phase of "actively creating content": no longer just improving image quality or performing structural analysis, but generating entirely new shots based on text scripts, reference images, or existing videos, or performing structural editing and reorganization of original videos. This includes both text-to-video generation from scratch, as well as style transfer, extension, and rearrangement based on existing images/videos, and fine-grained object-level editing and replacement.

In terms of products, this layer of capabilities has already entered the content creation mainstream through a series of products such as Jimeng Video, MiniMax Video, Sora, Runway Gen‑2, Pika, and Kling: advertisements, concept films, animations, and storyboard sequences can be quickly generated without relying on large filming crews and complex post-production; creators can drive shots and styles through natural language scripts; traditional video editing workflows are beginning to deeply integrate with structured generation tools. Below, we continue to organize from the angles of scenarios, principles, and models.

• Scenarios
  • Copy and script to short video: Brand advertisements, mini-dramas, plot clips, and concept animations — automatically or semi-automatically generate playable video drafts based on scripts.
  • Image / video to video: Generate animated versions for illustrations or character designs, perform style transfer on real-world footage (real → anime / illustration), or extend/reorganize existing video in time and space.
  • Structured editing and post-production: Without changing the overall content semantics, achieve fine operations such as face swapping, lip-sync, object erasure and replacement, and text-driven clip rearrangement.
• Principles Current mainstream video generation and editing methods mostly use diffusion models or their variants as the core, gradually "denoising" to generate video in a high-dimensional spatiotemporal latent space:
  • Text-conditioned modeling: A text encoder (such as T5/CLIP text tower or a dedicated language model) maps the script into a condition vector, guiding the video decoder to align with the text description in terms of style, content, and motion patterns.
  • Spatiotemporal consistency and motion control: Spatiotemporal convolutions, temporal attention, or 4D representations (NeRF/GS, etc.) are incorporated into the diffusion process or posterior optimization to ensure the video's coherence and physical plausibility along the time axis.
  • Image / video conditioned generation: The diffusion process is initiated in the feature space of the input image or video, and by controlling noise injection, mask regions, and condition channels, controlled editing or extension is achieved that "preserves the given parts + generates new content."
  • Structured control signals: Structural information such as pose skeletons, segmentation masks, depth maps, and camera trajectories is incorporated to make the generated video more controllable in terms of subject motion and viewpoint changes.
• Models Representative models and directions include:
  • Diffusion‑based Text‑to‑Video models (Sora, Runway Gen‑2, Pika, Kling, etc.), pretrained on large-scale video–text pairs, with strong generation capabilities in complex scenes, multi-shot motion, and diverse styles.
  • Image‑to‑Video diffusion models: Using a single-frame image as a condition to predict the dynamic evolution of subsequent frames, achieving "single image → animation / motion effect"; or performing operations such as continuation, extension, and viewpoint rotation on short videos.
  • NeRF / 4D representations and keyframe + interpolation methods: Using 3D scene representations or keyframes + temporal interpolation to combine generation with geometric and consistency modeling, achieving more stable viewpoint roaming and complex motion.

These capabilities do not exist in isolation but are gradually permeating editing and post-production pipelines: from copy to storyboard, storyboard to rough cut, rough cut to stylization and local editing — more and more stages are being driven by "text + structured control."

5.4.1 Text-to-Video: From Script to "Watchable" Shot Sequences

What text-to-video (Text‑to‑Video) aims to achieve is: the user describes a scene, shot, or story fragment in natural language, and the system automatically generates a coherent video. Compared to image generation, text-to-video adds the challenge of the temporal dimension: not only must image quality and style consistency be maintained at the single-frame level, but the coherence of subject identity, lighting, background, and motion trajectories across frames must also be ensured.

Typical diffusion-based text-to-video models are first pretrained on large-scale video–text paired data: a text encoder extracts semantic conditions, and a video decoder repeatedly denoises a "noisy video" in latent space, gradually converging to spatiotemporal signals consistent with the text. In this process, structures such as temporal attention, 3D convolution, or 4D representations explicitly build temporal dependencies into the network to avoid problems such as "inter-frame jumping" and "character resetting." Some systems also support control over shot motion (push, pull, pan, tilt) and composition rhythm, making the generated results closer to real cinematographic language.

5.4.2 Image / Video to Video: "Growing" and "Morphing" on Existing Content

Another important route is generation and editing based on existing images or videos: for example, "bringing to life" an illustration or concept design, stylizing real-person video into anime, or changing the background, adjusting weather and time while keeping the structure unchanged. Technically, such methods often add a "reference channel" to the diffusion process: the input image or video is encoded as features, participating in denoising as a condition or initial state, while mechanisms such as masks and explicit geometric constraints control "which regions can be changed and which must be preserved."

For style transfer scenarios, the model redraws textures and lighting while preserving the original motion and composition, matching the target style; for video extension and reorganization, new frames are "continued" at the temporal ends or in the middle, achieving horizontal/vertical scene expansion, viewpoint orbiting, or plot supplementation. Such capabilities are very suitable for integration with traditional editing workflows: the editor first provides key shots and rhythm, and the model then automatically generates transitions and variations between these "anchor points."

5.4.3 Structured Video Editing: Object-Level Fine Control

In many business scenarios, fully regenerating video is not a hard requirement; what is more critical is performing fine, controllable structured editing on existing footage: for example, face swapping, modifying mouth shapes, erasing unwanted objects, replacing advertising content, or rearranging shot order based on a text script. Structured video editing develops along this line of thinking: building on video understanding, object-level segmentation, tracking, and parametric representations are introduced, so that editing operations can be stably bound to specific targets and time periods.

Face swapping and lip-sync are the most typical applications in this direction: the model needs to map the target person's identity onto the original video's performance while ensuring natural and coherent head pose and overall expression, and precisely control mouth shape movements based on the new speech signal. Object erasure / replacement relies on high-quality segmentation and spatiotemporal inpainting: first segment and remove the target object in each frame, then fill the hole using neighboring frames and contextual texture, avoiding obvious "patching" traces. Text-driven editing aligns the "script structure" with the video timeline, automatically selecting and stitching clips that match the script semantics, achieving higher-level automated editing.

5.5 Digital Human / Avatar

Digital Human / Avatar can be seen as a "system-level integration" of video generation, speech synthesis, multimodal understanding, and graphics rendering: it is not just about generating a segment of video, but about continuously and controllably driving a virtual figure to "speak, make expressions, and gesture" based on text or speech input, and in more and more scenarios, achieving near-real-time or even real-time interaction. Compared to general video generation, digital humans emphasize three points more: long-term consistency of identity and appearance, fine alignment of speech–expression–motion, and the real-time performance and stability of the end-to-end system.

From a product perspective, digital humans have already widely appeared in scenarios such as content production platforms, virtual customer service / smart reception / virtual guided tours, education and training and online classrooms, brand virtual IP / virtual idols, and virtual streamer / digital avatar tools for creators: enterprises can batch-produce video content with fixed appearances and styles, government and enterprise services can use virtual receptionists to serve users 24/7, and individual creators can consistently produce "person-on-camera" videos without ever showing their face. Below, we continue to organize from three dimensions — scenarios, principles, and models — and expand on three directions in subsequent subsections: driving and expression, appearance and video generation, and real-time interaction and system integration.

• Scenarios
  • Content production and online distribution: Corporate promotional videos, product feature explanations, course recordings, and news broadcasts — using digital humans instead of real people on camera, significantly reducing the costs of shooting venues, lighting equipment, and human resources.
  • Virtual customer service and guided tours: In bank branches, government service halls, scenic spots, museums, and other venues, using digital humans to handle greeting, inquiries, business consulting, and route guidance, balancing consistent image with 24/7 service.
  • Brand virtual IP / virtual idols: Operating short videos, live streams, and e-commerce content around a virtual figure over the long term, maintaining a unified persona and visual style across different platforms.
  • Virtual streamers and digital avatars: For creators who prefer not to appear on camera or need to operate multiple identities, providing configurable virtual streamers / digital avatars bound to real or synthesized voices, enabling "stable on-camera presence just by speaking or typing."
• Principles A digital human system is essentially a multimodal pipeline of "speech / text driving + appearance modeling + video / rendering output," with slight differences between offline and real-time scenarios but similar core components:
  • Speech and language driving: Use TTS to synthesize speech directly from a script, or connect ASR + LLM to generate reply text from user speech/text and then use TTS to output speech; speech features (such as mel spectrograms) serve as driving signals to control mouth shape and expression timelines.
  • Appearance and motion space modeling: Construct a controllable geometric and appearance representation for the virtual figure, such as a 2D portrait/illustration, a skeleton- and Blendshape-based 3D Avatar, or a renderable volumetric representation based on NeRF / 4D Gaussians; and define a set of "driving parameters" (such as keypoints, pose skeletons, Blendshape coefficients) to encode expressions and poses.
  • Speech → expression / motion mapping: Through a dedicated "speech-driven" model, map speech features to driving parameters for the face and upper body, achieving lip-sync, expression details, and head-and-shoulder movements; real-time digital humans require this mapping to be end-to-end low-latency and stable.
  • Rendering and compositing: Based on the current frame's driving parameters, perform image or 3D rendering of the virtual figure, outputting a continuous video stream or real-time display; backgrounds, props, subtitles, and other elements can be overlaid, integrating with traditional video editing workflows.
• Models In terms of specific models, digital human systems often comprehensively use multiple types of specialized models and general multimodal models:
  • Audio‑driven Talking Head models: Lip-sync models such as Wav2Lip learn the alignment between speech and mouth-region pixels/geometry, generating natural mouth movements while maintaining identity consistency.
  • Real-time / lightweight digital human models: Such as Ultralight‑Digital‑Human and lightweight Talking Head models, which significantly compress parameter counts and computation in their architecture, enabling near-real-time driving and rendering even on CPU / mobile / WebGPU.
  • NeRF / 4D representation models: Such as ER‑NeRF (digital human NeRF approaches in the Explicit / Efficient / Editable direction), which model the figure's appearance and expression changes in 3D space, making viewpoint, lighting, and motion more natural and coherent — suitable for high-fidelity and multi-camera scenarios.
  • Speech-driven and multimodal alignment models: Such as MuseTalk-type "speech → facial expression / talking head" models, which align audio features and visual features, achieving realistic speaking expressions and head movements without relying on large amounts of 3D annotations.
  • Speech and dialogue models: High-naturalness multi-speaker TTS, end-to-end speech dialogue models (ASR + LLM + TTS integrated), providing digital humans with multi-style, multilingual voice and dialogue capabilities.

Taken together, digital humans are both a set of models and a complete system: they integrate language understanding, speech, visual generation, and real-time inference to present an interactive virtual character "on screen." Below, we expand on three directions: driving and expression, appearance and video generation, and real-time interaction and system integration.

5.5.1 Driving and Expression: From Script / Speech to a Person Who "Describes and Expresses"

In the digital human pipeline, driving and expression is responsible for answering a core question: given a script or speech, what mouth shape, expression, and head-and-shoulder movements should the virtual figure present in each frame. This includes both offline batch production scenarios and responses to real-time dialogue.

In offline content production, a common pipeline is "text script → TTS → speech-driven": the business side provides the narration script, the TTS module generates speech in the target timbre (such as a brand's virtual spokesperson), and the speech features are then fed into a "speech → motion" model. Wav2Lip-type models are an important representative of this stage:

• They take a reference portrait frame and the corresponding speech segment as input, and through a convolutional / attention network predict a mouth region finely aligned with the speech, which is then fused with the original portrait, thereby precisely modifying the mouth shape while preserving identity and most expressions.
• During training, speech–video alignment data supervises the network to learn the oral cavity shapes corresponding to different phonemes, maintaining temporal continuity to avoid mouth-shape jumping or delay.

Compared to earlier pure lip-sync approaches, the new generation of speech-driven models (such as MuseTalk-type methods) further extend to full-face expressions and head pose:

• Such models typically map speech features to a low-dimensional "emotion / expression latent space," then generate keypoints, Blendshape coefficients, or directly generate image features through a decoder, driving subtle changes in areas such as eyebrows, eyes, and cheeks, making "speaking expressions" more vivid.
• Some models also encode the semantic information of the speech content (such as questions, emphasis, exclamations), combining syntactic/pragmatic signals analyzed by an LLM to add nodding, frowning, gesturing, and other movements at points of intonation change, enhancing the naturalness and appeal of the expression.

At a higher dimension, driving and expression can also incorporate external control signals: for example, using pose skeletons, gesture trajectories, and gaze direction as additional inputs, enabling the digital human to imitate the style of a specific speaker, or to execute predefined action templates based on "directive actions" in the script (such as "point to the screen," "open both hands"). Whether it is local mouth-shape driving like Wav2Lip, or more full-body expression modeling like MuseTalk / real-time skeleton driving, they together achieve continuous mapping from speech/text to facial and upper-body movements, and are the key link that makes a digital human "look like it's seriously speaking."

5.5.2 Appearance and Video Generation: From "a Model" to "a Malleable Character"

The driving pipeline solves "how to move," while appearance and video generation determines "who is moving, where they are moving, and in what style." This includes both high-fidelity photorealistic digital humans and stylized figures such as 2D anime, cartoon, and low-poly avatars, as well as different technical choices for real-time and offline rendering.

In 2D portrait and illustration scenarios, the typical approach is to train a Talking Head generation model based on a small number of reference images and short videos:

• The model encodes the person's identity information as an "appearance vector" or style feature, and takes driving parameters (such as speech latent vectors, keypoints, expression encodings) as conditional inputs to synthesize new frames in image space.
• Unlike pure Wav2Lip which only modifies the mouth shape, such models can make small-amplitude pose swings and superimpose emotional changes on expressions, making the digital human look less "stiff."

In scenarios pursuing higher realism, freer viewpoints, and multi-camera switching, an increasing number of solutions adopt digital human modeling based on NeRF / 4D representations (such as ER‑NeRF-type methods):

• Through multi-view capture or video, the 3D volume or Gaussian field of the person's head / upper body is first reconstructed, encoding the states corresponding to different expressions and mouth shapes as an interpolatable latent space;
• During driving, speech/expression parameters are mapped to this latent space, and volumetric rendering or Gaussian rendering is performed in 3D, then projected onto the screen.
• The advantage of this approach is that viewpoint, lighting, and background are more natural, supporting "surround viewpoint" and "virtual camera" movements, and are particularly friendly to VR/AR, virtual live streaming rooms, and high-end advertising production.

In businesses emphasizing cross-platform deployment and real-time performance, lightweight solutions such as Ultralight‑Digital‑Human are also adopted:

• Through structural pruning, operator restructuring, and model distillation, the Talking Head or Avatar rendering network is compressed to a scale that can run on mobile / WebGPU;
• Generation from driving parameters to a single frame image is completed within milliseconds, aligned with real-time speech streams or control signals, achieving "low-latency digital humans" suitable for interactive terminals, self-service kiosks, and web frontend applications.

At the level of complete video production, appearance and video generation must also be combined with backgrounds, props, and cinematic language. A common workflow is:

• First, customize a digital human appearance (2D or 3D) for a brand or individual;
• Preset several virtual scenes (studio, office, classroom, exhibition hall, etc.);
• When producing content, the system automatically selects the appropriate scene and camera position based on the script, generates the digital human footage, and arranges it in multi-picture compositions with PPTs, demo videos, and product shots. This makes the digital human not just a "talking head," but a "character" that can naturally blend into various program and content formats.

5.5.3 Real-Time Digital Humans and System Integration: From Offline Video to "a Colleague on Screen"

As ASR, TTS, LLM, and lightweight video generation models mature, more and more digital human systems are moving from offline batch video production to real-time interaction: the user speaks or types at the terminal, and within a few hundred milliseconds to a few seconds, the digital human on screen "understands — thinks — responds — speaks," creating an experience similar to a real customer service agent / guide / host. The key here is not just the models themselves, but also how to compress the multimodal pipeline to acceptable end-to-end latency.

In a typical real-time digital human closed loop:

• Frontend input: The ASR module converts user speech to text in real time, or directly receives user text input.
• Semantic understanding and decision-making: The LLM combines the business knowledge base and tools (RAG, database queries, workflow orchestration) to generate reply text, along with necessary structured instructions (such as which PPT page to show, which video clip to play).
• Speech and driving: TTS converts the reply text into speech in the target timbre; the speech stream is generated and simultaneously consumed by the Wav2Lip / MuseTalk / real-time skeleton driving model, outputting corresponding mouth-shape and expression parameters segment by segment.
• Rendering output: A lightweight rendering network of the Ultralight‑Digital‑Human type or a GPU-based NeRF / Avatar rendering engine converts driving parameters into video frames in real time, outputting directly to the screen via WebRTC, RTMP, or local rendering.

To provide a consistent experience across multiple terminals, the system also needs to make careful trade-offs among latency, bandwidth, and compute:

• In cloud rendering schemes, the vast majority of computation (LLM, TTS, driving, and rendering) is completed on the server, and the terminal only plays the video stream — suitable for web/app and offline large screens with limited compute, but dependent on network stability;
• In "cloud + edge hybrid" schemes, ASR and part of the LLM inference are completed in the cloud, while lightweight driving and rendering are performed locally, significantly reducing audio-visual interaction latency — suitable for mobile devices and self-service kiosks;
• On high-compute terminals (such as high-performance PCs, dedicated workstations), most of the pipeline can even be offloaded locally, achieving stable interaction in weak-network environments.

On the model side, real-time digital humans also impose additional requirements on architectural design:

• Speech-driven models need to support streaming inference, capable of producing mouth-shape and expression predictions after receiving a small segment of speech, rather than waiting for the entire sentence to finish;
• Rendering networks need to minimize reliance on large convolution kernels and global attention, using structures such as local convolutions, lightweight self-attention, and resolution pyramids to control computation;
• For high-fidelity solutions based on NeRF / 4D, techniques such as mesh caching, frustum culling, sparse volumes, and GPU optimization are needed to keep per-frame rendering within a few milliseconds to tens of milliseconds.

At the system integration level, real-time digital humans often also need to be tightly bound to business knowledge, persona settings, and dialogue strategies:

• Manage industry knowledge, business processes, and FAQs through knowledge bases and RAG, ensuring "what is said is correct and complete";
• Control speaking style and expression boundaries through persona configuration and script templates, ensuring "what is said sounds like this person (or this brand)";
• Through multi-turn dialogue strategies and conversation state management, enable the digital human to remember user context, confirm and follow up at appropriate times, presenting an interaction feel "like a real colleague / guide / instructor."

Overall, with the addition of models such as Wav2Lip, MuseTalk, ER‑NeRF, and Ultralight‑Digital‑Human specifically designed for lip-sync, expression driving, and real-time rendering, digital humans are accelerating their evolution from "offline video template tools" into virtual entities that can respond in real time, have stable personalities and professional knowledge, becoming the most comprehensive and application-rich component of the video technology system.