4. Audio / Speech

In the overall technology stack, "audio" corresponds to the perception and generation of acoustic signals: it includes not only the processing of raw waveforms and spectra, but also converting speech to text, understanding "who is speaking" and "what is being said," as well as further creating and synthesizing sound and music. Similar to vision, audio can also be broken down into multiple layers: the bottom layer of waveform and spectrum processing is responsible for "hearing clearly"; the middle layer of speech recognition and speaker technology is responsible for "understanding who is saying what"; above that are more abstract layers of audio/music understanding and speech and music generation. This entire set of capabilities collectively supports products such as real-time meeting captions, voice assistants, podcast post-production tuning, smart speakers, acoustic security monitoring, and music recommendation and generation.

4.1 Waveform-Level Audio Processing: Starting from "Hearing Clearly"

At the lowest level of audio technology, what we first care about is not "what is being said," "who is speaking," or "what style of music this is," but rather whether the sound itself is clean and clear enough to hear. This layer primarily works at the waveform and spectrum level, using operations such as resampling, enhancement, noise reduction, and separation to process noisy, distorted, and mixed raw audio into "clean signals" that are more suitable for subsequent recognition, analysis, and generation. It can be likened to "image enhancement + denoising + foreground/background separation" in vision — it is more about performing acoustic-level cleanup rather than directly processing semantics.

From a product perspective, this layer is almost "invisible" behind every audio product: real-time noise reduction in meeting software, post-production tuning for podcasts and short videos, the "voice enhancement mode" in voice recorders and phones, the "beautify voice" toggle on live streaming platforms, and the front-end preprocessing for ASR/speaker verification models — all of these are direct manifestations of waveform-level audio processing. Below, we continue to organize this from three angles — scenarios, principles, and models — and in subsequent subsections we will specifically expand on three key directions: preprocessing & feature extraction, enhancement & noise reduction, and sound source separation.

• Scenarios
  • Online communication and meetings: Zoom, Tencent Meeting, etc., in noisy offices, open workstations, and home environments, suppress keyboard sounds, tapping, street noise, and echo in real time to make speech clearer.
  • Content creation and post-production tuning: In podcast, short video, and live streaming post-production, automatically eliminate background noise, electrical hum, and room reverberation, repair recording pops and frequency band gaps, and improve overall listening quality.
  • Recording and transcription front-end: Voice recorders, smart captions, and meeting transcription services perform VAD, noise reduction, and loudness normalization before entering ASR, improving back-end recognition robustness.
  • Terminals and IoT: "Far-field pickup" and "noise reduction modes" on devices such as smart speakers, in-vehicle systems, and cameras capture the primary speaker or key sound sources as much as possible in complex acoustic fields.
• Principles Waveform-level processing usually does not directly understand semantics; instead, it performs signal optimization around spectral structure and statistical characteristics:
  • Transform back and forth between the time domain and frequency domain (e.g., STFT → spectrum/Mel-spectrogram → iSTFT), suppressing or modeling noise bands, reverberation characteristics, or background sounds.
  • Use VAD and energy/spectral features to distinguish "segments with speech" from "silence/noise segments," reducing the impact of invalid segments on the back end.
  • Use deep learning or classical filtering methods to estimate masks or gain functions for "clean speech spectrum" and "noise spectrum," weighting the spectrum to achieve enhancement and noise reduction.
  • In multi-source mixing scenarios, use end-to-end separation networks or sparse representations to unmix different speakers, vocals and accompaniment, and foreground and background ambient sounds into independent tracks.
• Models Models at the waveform/spectrum level can be roughly divided into two categories: spectral-domain models and time-domain end-to-end models:
  • U-Net series on spectrograms/Mel-spectrograms: Spectrogram-based U-Net, DCCRN, etc., performing "image-style" convolution and encoding–decoding on the time–frequency plane — a common approach for tasks such as speech enhancement and singing voice separation.
  • Waveform end-to-end models: Wave-U-Net, Conv-TasNet, Demucs, etc., modeling directly on time-domain waveforms, avoiding explicit STFT/ISTFT, and often achieving better results in subjective listening quality and time-domain fidelity.
  • Classical signal processing methods: Traditional frequency-domain methods such as spectral subtraction and Wiener filtering are still widely used in lightweight devices or latency-sensitive scenarios, often combined with deep enhancement networks to form "hybrid solutions."

4.1.1 Preprocessing & Feature Extraction: "Clearing the Stage" for the Back End

Any subsequent ASR, speaker verification, event detection, TTS, and other models require audio input that is as unified, clean, and structured as possible — this is the responsibility of the preprocessing and feature extraction layer. It handles the most basic yet critically important tasks of "clearing the stage" and "format unification," setting the stage for upstream audio models.

In the preprocessing stage, the collected audio first undergoes sample rate conversion and channel conversion: for example, converting 48kHz stereo to 16kHz mono to meet the input specifications of downstream models and reduce computational cost. Subsequently, loudness normalization, DC offset removal, simple filtering, etc., are performed to make audio recorded from different devices and scenarios more consistent on the energy scale.

Voice Activity Detection (VAD) is another key component of preprocessing. It attempts to automatically segment "speech segments" and "silence/pure noise segments" in the audio stream, often based on frame energy, spectral entropy, zero-crossing rate, or small neural network discrimination. The benefit of VAD is that it can significantly reduce the amount of invalid data fed into ASR/speaker verification models, lowering computational load while preventing silence segments from interfering with recognition (e.g., being misrecognized as long strings of spaces or strange characters). In real-time communication, VAD can also drive the "voice activity indicator" and automatic mute logic.

At the feature extraction level, the most common approach is to convert the time-domain waveform into a spectrum or Mel-spectrogram. Through the Short-Time Fourier Transform (STFT), audio is decomposed into a frequency distribution that varies over time; through Mel filter banks, more perceptually relevant Mel-spectrograms or Mel cepstral features (such as log Mel-spectrogram, MFCC) can be obtained. These time–frequency features provide a "two-dimensional representation" for subsequent recognition, separation, and generation, analogous to grayscale images or multi-channel feature maps in vision, making them convenient for convolution, attention, and other architectures to process. With the development of end-to-end modeling, more and more models are learning features directly from waveforms (e.g., Wav2Vec 2.0), but in engineering practice, the combination of STFT + Mel features remains the most common and reliable front-end.

4.1.2 Enhancement & Noise Reduction: Restoring "Muddy Sound" to "Dry Sound"

In real environments, sound almost always propagates through noise and reverberation: air conditioning hum, keyboard clicks, road noise, crowd chatter, and room echo all degrade the intelligibility and subjective quality of speech and music to varying degrees. The goal of speech enhancement and noise reduction is to suppress these background interferences while preserving the naturalness and completeness of speech as much as possible, restoring "muddy" sound to "clean" sound.

In traditional methods, this task is primarily achieved through frequency-domain techniques such as spectral subtraction and Wiener filtering: first estimate the noise spectrum, then "subtract" the noise or adjust frequency band gains on the spectrum according to certain rules. While simple to implement and offering good real-time performance, these methods tend to produce noticeable "musical noise" and artifacts in scenarios with strong noise, non-stationary noise, and complex reverberation.

Deep learning methods, on the other hand, learn a mapping on the spectrum or waveform: given noisy speech, predict a time–frequency mask or directly predict the clean waveform. Common approaches include using Spectrogram-based U-Net, DCCRN, and other encoder–decoder architectures on Mel/linear spectrograms to carefully repair the spectrum of each frame; there are also end-to-end waveform enhancement methods using models such as Conv-TasNet, Demucs, Wave-U-Net directly on time-domain waveforms. These methods can significantly improve speech clarity and subjective listening quality in scenarios such as voice calls, online meetings, and recording restoration.

In content creation and post-production, "recording restoration" often also involves reducing plosives, cutting sibilance, compensating for frequency band gaps, as well as equalization (EQ) and dynamic processing (compressor/limiter) — operations that are more "audio engineer" in nature. An increasing number of tools combine these traditional processes with deep models to provide one-click "audio repair" and "audio beautification" capabilities, serving podcasters, video creators, and live streaming platforms.

4.1.3 Sound Source Separation: Unmixing the "Mix"

If enhancement and noise reduction are about "making the main sound more prominent and the background quieter," then sound source separation goes a step further by attempting to completely split multiple mixed sound sources into independent tracks. For example: multiple speakers talking simultaneously in a meeting recording; vocals and accompaniment mixed together in music; key events (such as alarms, shouting) buried in background noise in environmental recordings. The goal of sound source separation is to recover the waveform or spectrum of each individual sound source from a single or multiple mixed signals.

In the speech domain, multi-speaker separation is a core application: the model needs to separate multiple overlapping voices into different channels based on speaker embeddings, time–frequency structure, and speaker characteristics, without separate microphone tracks. This capability not only improves the performance of multi-speaker ASR but also provides cleaner input for speaker diarization. In the music domain, vocals/accompaniment separation (singing voice separation) can extract clear vocal tracks and pure accompaniment tracks from a mixed song, used for covers, remixes, karaoke, music analysis, and more. Similarly, ambient sound/foreground sound separation can be used in security and IoT scenarios to extract key event sounds (such as glass breaking, conflict sounds) from complex backgrounds.

At the model level, sound source separation typically employs stronger modeling capabilities and more complex architectures than ordinary enhancement. End-to-end networks such as Conv-TasNet, Demucs, Wave-U-Net can directly perform multi-source decomposition in the time domain; in the spectral domain, multi-branch U-Net, attention, mask estimation, and other architectures are common, predicting specialized masks or spectra for different sound sources. With the growth of training data and computational resources, modern sound source separation models can already output high-quality separated tracks usable for practical creation and analysis in fairly complex reverberation and noise environments, providing a solid foundation for live voice beautification, multi-speaker meetings, music production, and audio retrieval.

4.2 Speech Recognition & Describeer Technology (ASR & Describeer)

After completing preprocessing, enhancement, and separation at the waveform level, we can finally begin to ask higher-level questions: "What is being said in the audio?" "Who is speaking?" "When is who speaking?" This layer focuses on various "understanding and annotation" tasks centered around speech itself: Automatic Speech Recognition (ASR), speaker recognition and verification, speaker diarization, and hotword and keyword detection (KWS) for interaction.

From a product perspective, this layer is the core of the vast majority of "voice products": voice input methods, meeting transcription, customer service recording analysis, intelligent customer service quality inspection, smart speaker and in-vehicle voice interaction, phone robots, and voiceprint verification in financial scenarios — almost all directly depend on these technologies. They transform the "clean sound" from the previous layer into text sequences, speaker labels, or keyword events, serving as one of the most important bridges from audio to the semantic world.

• Scenarios
  • Automatic Speech Recognition (ASR): Real-time captions, voice input methods, meeting and classroom recording, customer service call transcription, providing users with an immediate "hearing to text" channel.
  • Speaker Recognition and Verification: "Voiceprint unlock" and "voiceprint verification" in phones/banking/call centers, as well as searching for a specific speaker in massive amounts of recordings.
  • Speaker Diarization: In meetings, interviews, and roundtable discussions, automatically answering "who is speaking when," enabling "speaker-attributed transcription."
  • Hotword and Keyword Detection (KWS): Wake word detection in smart speakers/in-vehicle systems ("Hey Siri," "OK Google"), as well as capturing key phrases (such as "complaint," "refund," "upgrade," etc.) in customer service recordings and quality inspection.
• Principles Most tasks in this layer can be uniformly viewed as performing time alignment and sequence labeling on audio sequences:
  • ASR: Given a segment of speech, learn the mapping from acoustic features to text sequences, often using CTC, RNN-Transducer (RNN-T), or attention-based end-to-end architectures; modern models typically use large-scale pre-training (e.g., Wav2Vec 2.0, Whisper, etc.) followed by fine-tuning.
  • Speaker Recognition: Extract a fixed-dimensional speaker embedding (e.g., x-vector, ECAPA-TDNN) from audio. In this embedding space, voices from the same person are close to each other, and voices from different people are far apart, which is then combined with metric or classification models for recognition and verification.
  • Speaker Diarization: Comprehensively use speaker embeddings, VAD, segment clustering, or end-to-end networks (EEND) to assign speaker labels to each time slice, thereby piecing together a "multi-speaker timeline on the time axis."
  • KWS: Perform low-latency small-model detection on continuous audio streams, conducting local pattern matching and confidence evaluation for predefined wake words or keywords, balancing low computational cost with high recall.
• Models The model landscape for ASR and speaker technology includes both end-to-end architectures and specialized embedding models and clustering methods:
  • ASR: Wav2Vec 2.0, Conformer, Whisper, RNN-T, Citrinet, etc., mostly using convolution + self-attention or pure self-attention architectures, supporting multilingual, large-vocabulary, and long-context scenarios.
  • Speaker Embedding: ECAPA-TDNN, x-vector, i-vector, etc., trained through classification or metric learning on large amounts of speaker data to obtain robust speaker feature spaces.
  • Diarization: From the traditional pipeline of VAD + segmentation + clustering, to end-to-end methods such as End-to-End Diarization (EEND) that directly output a "time × speaker" matrix.
  • Hotword/Keyword Detection: Lightweight CNN/RNN/Transformer front-ends combined with CTC or gating mechanisms, embedded locally on devices for always-on listening at ultra-low computational cost and latency.

4.2.1 Automatic Speech Recognition (ASR): Turning "Sound" into "Text"

Automatic Speech Recognition (ASR) is the main "audio → text" pathway: whether it's voice input methods, meeting transcription, smart captions, or customer service recording analysis, the first step is always to accurately convert what the user says into text. Modern ASR systems mostly adopt end-to-end architectures: starting from acoustic features (such as Mel-spectrograms or raw waveforms), going through a series of deep networks (such as Conformer, Citrinet, Transformer-based Encoders), and directly outputting text sequences or corresponding token sequences.

In terms of modeling, the main challenges of ASR include long-range dependencies, multilingualism and dialects, accent variations, overlapping speech, background noise, and domain-specific terminology. To address these, the current mainstream direction is to use large-scale unlabeled audio for self-supervised pre-training (such as Wav2Vec 2.0, HuBERT), or to perform large-scale supervised training on multilingual and multi-task data (such as Whisper), and then fine-tune with relatively small amounts of domain-specific data, thereby achieving good robustness across different languages, accents, and scenarios.

At the product level, ASR is typically packaged as capabilities such as "voice input SDK," "cloud speech recognition API," and "meeting transcription service": the front-end can be real-time streaming recognition (RNN-T, streaming Transformer, etc.), while the back-end can enhance recognition of specific person names, place names, brand names, and business terminology through hotword injection, custom vocabularies, and contextual constraints. These recognition results often serve as the foundation for subsequent NLP, dialogue systems, and data analysis.

4.2.2 Describeer Recognition & Diarization: Answering "Who" and "When"

Compared to "what is being said," "who is speaking" is equally important in many applications: scenarios such as finance, government, customer service, and security require voiceprint recognition to verify identity or investigate risks; while meeting and interview scenarios need to know "who said each sentence" to support speaker-attributed transcription, speaking statistics, and behavioral analysis.

In the Speaker Recognition/Verification task, the system's goal is: given a segment of speech, determine who the speaker is, or determine whether it matches a registered speaker. Modern systems typically use models such as ECAPA-TDNN and x-vector to extract a fixed-dimensional speaker embedding vector from the speech segment. During the training phase, a combination of speaker classification and metric learning ensures that embeddings of the same person are more clustered and the distance between embeddings of different people is larger; during the inference phase, nearest neighbor or back-end discriminators (such as PLDA, Cosine scoring with margin) are used for verification and recognition. In this way, the system can answer "is it the same person" with a certain confidence level across phone, microphone, and noisy environments.

Speaker Diarization further answers "who is speaking when." The traditional approach typically involves three steps: first use VAD to find segments with speech, then cut the long audio into short segments, extract speaker embeddings for each segment, and finally perform clustering and temporal stitching in the embedding space to obtain a multi-speaker timeline. More advanced End-to-End Diarization (EEND) methods attempt to directly output a "time × speaker" boolean matrix from audio features, learning complex patterns such as overlapping speech and speaker changes in an end-to-end manner. Diarization is highly valuable in scenarios such as meetings, interview programs, court records, and phone customer service, often combined with ASR to produce "text records with speaker labels."

4.2.3 Hotword & Keyword Detection: The "Ears" for Interaction and Monitoring

In a continuous audio stream, not every second is worth being fully recognized and stored. The role of hotword and keyword detection (KWS) is that of an always-on "gatekeeper":

• In smart speakers, in-vehicle systems, and phone assistants, the KWS module is responsible for detecting wake words (such as "Hey Siri," "OK Google," "Xiao Ai Tong Xue"). Once a wake word is detected, the audio stream is handed off to more expensive ASR and dialogue systems for processing.
• In intelligent customer service, quality inspection, and compliance scenarios, KWS marks and alerts on key phrases (such as "complaint," "return," "rights protection," "fraud") appearing in recordings or real-time calls, providing trigger points for back-end analysis and quality inspection strategies.

In terms of technical implementation, KWS typically needs to operate under constraints of extremely low computational cost and low latency, especially for wake word detection on local devices: the model is often a small CNN/RNN/Transformer front-end connected to a CTC or gated discrimination head, detecting acoustic patterns of specific words, and using sliding windows and confidence smoothing to avoid false wakes. For keyword quality inspection scenarios, stronger ASR + keyword matching/regex + statistical analysis can be used, or end-to-end keyword tagging models can be trained directly. Regardless of the form, KWS essentially adds a layer of "event-level" semantic filtering on the speech stream, serving as an important interface connecting the audio world with interaction logic.

4.3 Audio/Music Understanding (Audio Event & Music Understanding)

Not all audio is centered on "speech." In reality, there are many scenarios related to environmental sounds, event sounds, and music, which are more concerned with: "What sound event occurred?" "What is the current acoustic scene?" "What style is this song, what instruments are used, what are the rhythm and key?" This set of capabilities is collectively referred to as audio/music understanding, primarily revolving around sound event detection, environmental/scene classification, and music attribute understanding.

From a product perspective, audio understanding technology supports a wide range of applications such as acoustic security monitoring, IoT acoustic sensors, environmental adaptation for smart devices, music recommendation and classification, music copyright identification, music retrieval, and creative assistance. Similar to "image classification + fine-grained classification" in vision, this layer structures the originally continuous and complex sound space into discrete event labels, multi-dimensional attribute vectors, and style descriptions.

• Scenarios
  • Sound Event Detection: Detecting alarm sounds, glass breaking, baby crying, impact sounds, etc., for security monitoring, smart buildings, vehicle safety systems, and industrial alerts.
  • Environmental/Scene Classification: Identifying acoustic scenes such as "indoor/outdoor," "office/car/street/subway," providing a basis for noise reduction strategies, adaptive gain, and mode switching in smart devices.
  • Music Understanding and Music Information Retrieval (MIR): Genre classification, instrument recognition, rhythm and key analysis, supporting music recommendation, playlist generation, music retrieval, copyright identification, and creative assistants.
• Principles Audio/music understanding is mostly based on time–frequency features + deep neural networks for classification or multi-label annotation:
  • Using features such as log Mel-spectrogram to convert audio into "acoustic images," then using CNN, CRNN, or Transformer architectures for time–frequency pattern recognition.
  • For sound event detection, multi-label, multi-temporal output is often used, predicting the presence of each event on the time axis, sometimes combined with weak supervision labels and multi-instance learning.
  • For environmental/scene classification, more emphasis is placed on long-term statistical features and background patterns, often requiring modeling over longer windows.
  • Music understanding tasks combine music theory knowledge to model tempo (BPM), beat positions, key, chords, and structure, with some tasks using self-supervised or contrastive learning to pre-train music embeddings, followed by downstream fine-tuning.
• Models Common audio understanding models are mostly pre-trained on public datasets (such as AudioSet) and then transferred to specific tasks:
  • CNN/CRNN models such as VGGish, YAMNet, PANNs, pre-trained on large-scale audio data, can be used for various audio event and acoustic scene tasks.
  • Transformer-based models such as AST (Audio Spectrogram Transformer) directly use self-attention on spectrograms, achieving stronger global time–frequency modeling capabilities.
  • Music-oriented MusicTagging/MIR models are pre-trained on millions of songs to learn tag models or embedding models for style/mood/instrument tagging, music retrieval, and recommendation.

4.3.1 Sound Events & Acoustic Scenes: Letting Devices "Understand the Environment"

In security, IoT, smart cities, and in-vehicle systems, cameras alone are not sufficient to fully understand environmental states. The goal of sound event detection is to enable systems to "hear and understand" key events: when glass breaks, an alarm sounds, a baby cries, a collision, scream, fight, or destructive behavior occurs, the system can identify and alert from the audio signal. Unlike speech recognition, such events are often short, non-verbal, with varying frequency ranges and energy patterns, and may be highly overlapped with background noise.

Environmental/scene classification focuses more on persistent acoustic scenes: is it a quiet office, a bustling street, inside a car, a high-speed rail station, or a café? The system can automatically adjust noise reduction intensity, echo cancellation parameters, microphone array beam steering based on the acoustic scene, and even change interaction strategies (e.g., using shorter feedback interactions in a car, increasing output volume on noisy streets). In IoT scenarios, an "acoustic network" composed of multiple sound sensors can be used for long-term monitoring and statistical analysis of environmental states.

In terms of technical implementation, both types of tasks mostly adopt multi-label classification + temporal modeling approaches: convert audio to Mel-spectrograms, use VGGish, PANNs, AST, or similar models for feature extraction, and then use temporal pooling or sequence models to output the activation of each label on the time axis. Since many datasets only provide "clip-level labels" (weak labels), models often need to learn the temporal localization of events under weak supervision through methods such as multi-instance learning and self-attention pooling.

4.3.2 Music Understanding & Tagging: From "Playlist Tags" to "Structural Analysis"

In the music domain, the goal of audio understanding is not merely "what song is this," but also to answer: "What style is this song? What instruments are used? How fast or slow is the tempo? What is the key and general harmonic structure?" This information supports music recommendation and playlist curation on the one hand, and on the other hand provides structured "music metadata" for creators and generative models.

The genre classification task categorizes songs into different styles such as pop, rock, classical, hip-hop, electronic, Lo-Fi based on their overall acoustic features and structure; instrument recognition distinguishes the acoustic fingerprints of different instruments such as drums, bass, guitar, piano, and strings on time–frequency features, which can be used for instrument statistics, music retrieval, and mixing analysis. Tempo/key analysis estimates BPM, beat positions, time signature, and key, providing a foundation for tasks such as beat matching, automatic harmonization, DJ mixing, and game soundtrack synchronization.

In terms of models, music understanding largely follows general audio models (such as PANNs, AST), but there are also many models and pre-trained embeddings specifically oriented toward Music Information Retrieval (MIR). A typical approach is to perform multi-label music tag learning (genre, mood, instrument, era, etc.) on large-scale music datasets to obtain a music embedding space, and then fine-tune or perform zero-shot inference on the specific tasks mentioned above. By combining these models, music platforms can more intelligently complete music classification and recommendation, copyright platforms can enhance music fingerprinting and similarity retrieval, and creative tools can leverage these understanding capabilities to recommend suitable accompaniment to users, extend similar styles, or automatically generate musical structures.

4.4 Speech & Audio Generation (TTS / VC / Music Generation)

After completing the "cleanup," "recognition," and "understanding" of audio, the natural next-layer question is: "Can we directly make machines 'speak,' 'sing,' or even 'compose'?" This is the world of speech and audio generation: from Text-to-Speech (TTS), from one voice to another (VC / Voice Cloning), to broader music and sound effect generation, and further to singing voice synthesis that can sing lyrics and melodies. Similar to image generation, this layer is no longer just about labeling or extracting structure from existing data but actively "creating" new sound content.

At the product level, this layer's capabilities have already permeated various applications: voice product lines such as OpenAI TTS, ElevenLabs, ByteDance Volcano Engine, and MiniMax provide high-quality synthesized speech for applications; music generation platforms such as Suno and Udio provide creators and even ordinary users with the ability to go from text prompts to complete music; games, videos, virtual streamers, and digital humans rely on these models for dubbing and singing, greatly lowering the barrier to content production.

• Scenarios
  • Text-to-Speech (TTS): News broadcasting, navigation announcements, intelligent customer service voice responses, content reading in learning apps, accessibility screen reading, etc., requiring the conversion of arbitrary text into natural, clear, and controllable speech.
  • Voice Conversion / Voice Cloning (VC / Voice Cloning): Changing the speaker's timbre while preserving semantics and prosody, enabling "voice swapping" or "few-shot voiceprint cloning" (under strict compliance conditions).
  • Music and Sound Effect Generation: Generating suitable background music and sound effects (ambient sounds, UI sound effects, transition sounds) for short videos, games, advertisements, podcasts, etc.
  • Singing Voice Synthesis and Cover Songs: Given a melody and lyrics, having a virtual singer perform, or generating a cover version in a certain style/timbre under compliant conditions.
• Principles Speech and audio generation typically adopts a layered modeling approach of "high-level representation → low-level waveform":
  • In TTS, text is first converted into phoneme/syllable/character-level sequences, then through sequence-to-acoustic-feature (e.g., Mel-spectrogram) models (Tacotron, FastSpeech, VITS, etc.), and finally neural vocoders (WaveNet, WaveRNN, HiFi-GAN, etc.) generate high-fidelity waveforms from the features.
  • In Voice Conversion, by decoupling "what is said (content)" from "who is speaking (timbre)," content representations are extracted from the source speech and then combined with the target speaker embedding or vocoder conditions to generate new speech waveforms.
  • Music and sound effect generation can be based on tokenized representations (such as notes, MIDI, encoded spectrograms/codec tokens), using autoregressive, diffusion, or neural codec generative models to sample new audio from text, reference audio, or structural parameters.
  • Singing voice synthesis introduces finer prosody, pitch trajectories, and singing control on top of TTS, typically modeling pitch, duration, legato, vibrato, etc., explicitly or implicitly.
• Models The current mainstream technical approaches for speech and audio generation include:
  • TTS: Tacotron/Tacotron2, FastSpeech series (non-autoregressive TTS), VITS, etc., handle text-to-Mel-spectrogram or codec tokens; WaveNet, WaveRNN, HiFi-GAN, WaveGlow, etc., serve as vocoders or decoders handling features-to-waveform. Recent Diffusion-based TTS and Neural Codec models have further improved naturalness and diversity.
  • Voice Conversion / Cloning: VC frameworks based on speaker embedding + content encoder, as well as voice conversion models using neural codecs, supporting few-shot timbre cloning and cross-lingual speaker transfer. This type of technology has been commercially deployed by multiple platforms, providing convenient voice cloning API services. Common domestic platforms include ByteDance Volcano Engine, MiniMax, iFlytek Open Platform, Baidu AI Cloud Qianfan Large Model Platform, and Alibaba Cloud Intelligent Speech Interaction Platform; overseas mainstream platforms include ElevenLabs, Resemble.ai, Play.ht, and others. Among them, ByteDance Volcano Engine's voice cloning capability supports rapid training with a small number of audio samples, adapting to commercial use across multiple scenarios such as intelligent customer service and audiobooks; MiniMax leverages its large model technology advantages to achieve natural alignment between cloned timbre and text content, while also supporting cross-lingual speaker timbre transfer; iFlytek Open Platform's voice cloning has significant advantages in the clarity and emotional expressiveness of Chinese pronunciation, widely serving fields such as education and broadcasting.
  • Music and Sound Effect Generation: MusicLM, MusicGen, and Suno/Udio-type models typically generate long-form audio on discrete codec tokens using autoregressive or diffusion architectures based on text and/or reference audio conditions.

4.4.1 Text-to-Speech (TTS): Making Machines "Communicate Naturally"

Text-to-Speech (TTS) is the most intuitive speech generation task: input a piece of text, output a segment of natural and fluent speech that ideally is nearly indistinguishable from a human voice. Modern TTS systems are typically divided into two main stages: text to acoustic features (such as Mel-spectrogram), and acoustic features to waveform.

In the first stage, the model needs to handle problems such as tokenization, phonemization, polyphone disambiguation, punctuation and pausing, and prosody prediction. Typical models include the attention-based Tacotron series and the length-prediction-based FastSpeech series, with the latter significantly accelerating synthesis and improving stability through a non-autoregressive architecture. In recent years, end-to-end models such as VITS have merged acoustic modeling and the vocoder into a unified framework, further simplifying the system.

In the second stage, Neural Vocoders such as WaveNet, WaveRNN, HiFi-GAN, WaveGlow, etc., are responsible for converting Mel-spectrograms or other intermediate representations into high-fidelity waveforms. A well-trained vocoder can not only generate natural and clear speech but also faithfully reproduce different timbres, emotions, and styles. Modern TTS systems also support multi-speaker modeling (via speaker embedding), timbre/speed/emotion control (such as "excited," "calm," "broadcasting style"), and cross-lingual TTS, providing highly customizable voice capabilities for various applications.

4.4.2 Voice Conversion & Voice Cloning: Changing "Who Is Communicating"

In many creative and assistive scenarios, we want to change the speaker's timbre or style without altering the content and prosody — this is the task of Voice Conversion (VC) and Voice Cloning. The former primarily addresses "turning person A's speech into person B's voice"; the latter further emphasizes "learning a new timbre from just a few samples or even a few seconds of speech."

Technically, VC typically adopts a "content–timbre decoupling" approach: a content encoder extracts speech content and prosody information (which can be discrete units based on ASR or continuous representations from self-supervised learning), and then a conditional generator combines the target speaker embedding or codec conditions to generate new speech with the target timbre but largely unchanged semantics and rhythm. With the introduction of neural codecs, speech can be directly edited in the codec space to achieve high-fidelity conversion.

Voice Cloning builds on VC with an emphasis on few-shot and generalization capabilities: the model needs to extract stable speaker representations from a few samples or even a few seconds of audio, and based on this, generate synthesized speech with consistent style and similar timbre. This capability is very useful in virtual personas, personalized assistants, game character customization, dubbing acceleration, and more, but it must also strictly comply with legal and ethical norms, ensuring use only under conditions of compliant authorization, full informed consent, and security controls, to avoid risks of misuse or identity impersonation.

4.4.3 Music & Sound Effect Generation: From Prompts to Complete Soundscapes

Compared to speech generation, music and sound effect generation is more complex in structure and time scale: music often lasts longer with richer internal structure (sections, melody, harmony, rhythm); sound effects are diverse, ranging from natural environments (rain, wind, ocean waves) to foley sounds (UI clicks, notification sounds, game skill effects), each with their own patterns. In recent years, models based on neural codecs, sequence modeling, and diffusion have made "generating complete music/sound effects from text" a reality.

In music generation, models like MusicLM, MusicGen, Suno, and Udio typically encode audio into discrete codec token sequences and then train text-conditioned or multimodal-conditioned generative models in this discrete space. Users only need to provide a text description (such as "moderate tempo, warm and healing Lo-Fi background music, suitable for studying and focusing," "tense electronic orchestral score, suitable for a sci-fi trailer"), or upload a reference music clip, and the model can generate high-quality music lasting tens of seconds or even minutes. For creators, this is both a source of inspiration and a powerful tool for rapid prototyping and background music generation.

In sound effect generation, similar techniques can generate UI sound effects, notification sounds, game ambient sounds, etc., from text prompts, helping product and game teams rapidly iterate on sound design. Combined with the audio understanding capabilities of the previous layer, style alignment and scene adaptation can also be achieved, such as automatically matching sound effect styles based on visuals or game levels.

Whether it is speech or music and sound effect generation, this layer of capability is rapidly evolving: from the early days of heavily synthesized machine sounds to today's high-fidelity content that is nearly indistinguishable from human voices and professional music. At the same time, issues surrounding copyright, compliance, traceability, and controllability have become increasingly important — how to provide powerful creative tools while protecting the legitimate rights and interests of creators and users will remain a key topic that this layer of technology must continuously address.