wavenode

WaveNODE: A Continuous Normalizing Flow for Speech Synthesis

Hyeongju Kim Hyeonseung Lee Woo Hyun Kang Sung Jun Cheon Byoung Jin Choi Nam Soo Kim

Abstract

In recent years, various flow-based generative models have been proposed to generate high-fidelity waveforms in real-time. However, these models require either a well-trained teacher network or a number of flow steps making them memory-inefficient. In this paper, we propose a novel generative model called WaveNODE which exploits a continuous normalizing flow for speech synthesis. Unlike the conventional models, WaveNODE places no constraint on the function used for flow operation, thus allowing the usage of more flexible and complex functions. Moreover, WaveNODE can be optimized to maximize the likelihood without requiring any teacher network or auxiliary loss terms. We experimentally show that WaveNODE achieves comparable performance in terms of mean opinion score (MOS) and conditional log-likelihood (CLL) with fewer parameters compared to the existing models.

Audio Examples

Example 1

Ground Truth

Griffin-Lim

WaveNet

WaveGlow

WaveGlow (17.1M)

FloWaveNet

FloWaveNet (18.6M)

WaveNODE

Example 2

Ground Truth

Griffin-Lim

WaveNet

WaveGlow

WaveGlow (17.1M)

FloWaveNet

FloWaveNet (18.6M)

WaveNODE

Example 3

Ground Truth

Griffin-Lim

WaveNet

WaveGlow

WaveGlow (17.1M)

FloWaveNet

FloWaveNet (18.6M)

WaveNODE

Example 4

Ground Truth

Griffin-Lim

WaveNet

WaveGlow

WaveGlow (17.1M)

FloWaveNet

FloWaveNet (18.6M)

WaveNODE

Ablation Studies

1. Test Tolerance

Example 1

test_tol: 1E-5

test_tol: 1E-4

test_tol: 1E-3

test_tol: 5E-3

test_tol: 1E-2

Example 2

test_tol: 1E-5

test_tol: 1E-4

test_tol: 1E-3

test_tol: 5E-3

test_tol: 1E-2

Example 3

test_tol: 1E-5

test_tol: 1E-4

test_tol: 1E-3

test_tol: 5E-3

test_tol: 1E-2

Example 4

test_tol: 1E-5

test_tol: 1E-4

test_tol: 1E-3

test_tol: 5E-3

test_tol: 1E-2

2. Norm Layer

Example 1

Actnorm

MBN

None

Example 2

Actnorm

MBN

None

Example 3

Actnorm

MBN

None

Example 4

Actnorm

MBN

None

3. Batch Size

Example 1

batch_size: 20

batch_size: 4

Example 2

batch_size: 20

batch_size: 4

Example 3

batch_size: 20

batch_size: 4

Example 4

batch_size: 20

batch_size: 4

4. Dilation

Example 1

dilation: 3ⁱ

dilation: 2ⁱ

Example 2

dilation: 3ⁱ

dilation: 2ⁱ

Example 3

dilation: 3ⁱ

dilation: 2ⁱ

Example 4

dilation: 3ⁱ

dilation: 2ⁱ