PATRYK KULPOK
BACK TO SYSTEM MAP

SYSTEM 05

DUBFORGE

AUTOMATED DUBBING R&D PIPELINE

Research prototype exploring automated audiovisual dubbing through speech processing, speaker analysis, translation, voice generation and media reconstruction.

PYTHONSPEECHAUDIODUBBING

SYSTEM IDENTITY

SYSTEM IDDUBFORGE-05

STATUSR&D

TYPERESEARCH PROTOTYPE

R&D / experimental workflow for studying automation, speaker identity, timing and expression in audiovisual dubbing.

01 / OVERVIEW

PROJECT OVERVIEW

DubForge is an R&D project exploring how much of the audiovisual dubbing pipeline can be automated with speech processing, speaker analysis, translation, voice generation and media reconstruction.

The work focuses on building and evaluating individual pipeline stages rather than presenting the current output as production-ready dubbing.

02 / EXPLORATION

WHAT THE PROJECT EXPLORES

EXP-01

SOURCE SEPARATION

Separate dialogue from music and effects for downstream processing.

EXP-02

SPEAKER ANALYSIS

Detect and group speaker identities from extracted dialogue.

EXP-03

DIALOGUE RECONSTRUCTION

Rebuild timestamp-aware dialogue segments from speech recognition output.

EXP-04

TRANSLATION

Generate duration-aware Polish dialogue while preserving context and meaning.

EXP-05

PERFORMANCE ANALYSIS

Use acoustic features and context to estimate delivery, energy and pacing.

EXP-06

VOICE GENERATION

Experiment with generated Polish speech and speaker-reference conditioning.

EXP-07

TIMELINE RECONSTRUCTION

Place generated dialogue back onto the original episode timeline.

EXP-08

AUDIO MIXING & MEDIA ASSEMBLY

Recombine dialogue, background audio and video into an experimental output.

03 / PIPELINE

EXPERIMENTAL DUBBING PIPELINE

Audiovisual dubbing research path

  1. SOURCE VIDEO
  2. AUDIO SEPARATION
  3. ASR + SPEAKER ANALYSIS
  4. DIALOGUE RECONSTRUCTION
  5. POLISH TRANSLATION
  6. PERFORMANCE ANALYSIS
  7. VOICE GENERATION
  8. TIMELINE ASSEMBLY
  9. AUDIO MIX
  10. VIDEO OUTPUT

These stages are implemented or actively experimented with as pipeline components. Their combined output remains experimental and requires manual review.

04 / R&D FOCUS

R&D FOCUS

RND-01

AUTOMATION

How much of a traditionally manual dubbing workflow can be automated?

RND-02

SPEAKER CONSISTENCY

Can speaker identity be preserved reliably across many short dialogue segments?

RND-03

TIMING

How closely can generated Polish speech fit the timing constraints of the original material?

RND-04

PERFORMANCE

How much of the original delivery can be inferred from acoustic measurements and transferred into generated speech?

RND-05

MEDIA RECONSTRUCTION

Can generated dialogue be reconstructed into a usable audiovisual timeline?

05 / VALIDATION

TECHNICAL VS PERCEPTUAL QUALITY

Technical validation confirms that pipeline artifacts can be generated and assembled correctly. Perceptual dubbing quality remains an active research problem.

Producing technically valid intermediate artifacts does not imply professional dubbing quality. Voice quality, speaker consistency, translation naturalness, timing and final perceptual quality still require manual evaluation.

06 / ENGINEERING

VOICE PROCESSING DECISIONS

E-01

TIMESTAMPED UTTERANCES

Use faster-whisper large-v3 output and word timestamps to create atomic speech units.

E-02

SPEAKER PROFILES

Use ECAPA-TDNN embeddings and canonical references to retain speaker context.

E-03

PERFORMANCE SIGNAL

Treat expression and delivery as part of the voice problem, not just translation text.

07 / TECH STACK

TECHNICAL LAYER

  • Python
  • Faster-Whisper large-v3
  • ECAPA-TDNN
  • Audio Separation
  • Speaker Embeddings

08 / STATUS

CURRENT R&D STATUS

The pipeline has successfully produced complete intermediate artifacts, including separated audio stems, reconstructed dialogue, Polish translations, speaker-aware voice references and generated Polish dialogue tracks.

The current output remains experimental. Voice quality, speaker consistency, translation naturalness, timing and final perceptual quality still require manual evaluation and further iteration.

PIPELINE ARCHITECTURE IMPLEMENTED

DIALOGUE PROCESSING IMPLEMENTED

POLISH VOICE GENERATION EXPERIMENTAL

SPEAKER CONSISTENCY UNDER EVALUATION

PERCEPTUAL QUALITY R&D

PRODUCTION READINESS NOT TARGETED YET