Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation

In plain words

Usual speech translation chains two systems: one turns the spoken words into a text transcript, and a second translates that transcript. This paper asks whether a single neural network can go straight from French audio to English text, skipping the French transcript entirely.

The network reads the audio as a sequence of short slices of sound features, and writes the English sentence one word at a time. At each word it looks back over the whole input to decide which part of the audio matters now. The authors test the idea on a small, clean corpus of travel phrases that was read out by synthetic voices. The result is a promising proof of concept, not a system for real recordings.

TL;DR

A single attention-based encoder-decoder maps French speech features straight to English text, with no source transcript used in training or decoding §1. On a synthetic BTEC corpus, a 5-model ensemble reaches 48.8 BLEU on a held-out speaker's test1 set (7 references), against 52.7 for a Google ASR plus phrase-based SMT baseline tab. 3. The paper calls the results promising but limited by the synthetic data §5. Use it as an early reference for transcript-free speech translation, not as a benchmark on natural speech.

Problem

Speech translation normally needs a source-language transcript, both in training data collection and in the pipeline at inference time. The authors argue that collecting transcripts is costly for under-resourced languages with little standard spelling, and that collecting speech paired with target-language text would be easier §1. They know of only earlier attempts that align speech with text or translate phone sequences; none builds a full end-to-end speech-to-text translation system §1.

Method

The paper compares two attention-based encoder-decoder models that share a decoder design: one for text-to-text translation (the NMT reference) and one for speech-to-text translation §2.

1. Encoder. A multi-layer bidirectional LSTM reads the input sequence and produces one output per position §2. For text, the encoder has 2 layers and the input is word embeddings. For speech, the input is 40 MFCCs plus frame energy per frame, and two fully connected layers of size 256 sit in front of a hierarchical encoder with 3 layers §2 §3. Layers 2 and 3 read every other output of the layer below, so the encoded sequence is one quarter the length of the input frames §2.
2. Decoder initial state. The decoder state starts from a learned linear map of the encoder's last state, passed through tanh §2.
3. Decoder step. At each step an LSTM consumes the embedding of the previous target symbol and emits a new state and an output §2 eq. 1. During training the previous symbol is the reference word; at evaluation it is the model's own prediction §2.
4. Attention. An attention vector over the encoder outputs is computed from the new state and concatenated with the LSTM output §2 eq. 2. The text model uses the additive attention of Vinyals et al. eq. 4 eq. 5. The speech model uses convolutional attention: the previous step's attention weights are filtered with a 1-D convolution, and the filtered signal enters the score as an extra term eq. 6. The authors say this lets the model avoid translating the same part of the signal twice §2.
5. Output. A projection of the concatenated vector gives logits over the target vocabulary, and a softmax gives the next-word distribution §2 eq. 2 eq. 3. The greedy decoder picks the most probable word at each step; the beam-search decoder is used for the beam-search and ensemble settings §2 tab. 2.
6. Training. Adam with an initial learning rate of 0.001 and mini-batch size 64. Dropout of 0.5 is applied on the connections between LSTM layers in the encoder and the decoder §2. Training runs for 20k steps §4.

Shapes and sizes from the paper: W_{init} \in \mathbb{R}^{2m \times 2m}, E \in \mathbb{R}^{n \times \bar{V}}, W_{proj} \in \mathbb{R}^{m \times 2m}, W_{out} \in \mathbb{R}^{\bar{V} \times m}, W_1 \in \mathbb{R}^{m \times m}, W_2 \in \mathbb{R}^{m \times 2m}, \mu \in \mathbb{R}^{m}, F \in \mathbb{R}^{k}, where n is the embedding size, m the LSTM layer size (state size 2m), k the convolution filter size and \bar{V} the target vocabulary size §2.

Equations, in the paper's order:

s_t, o_t = \mathrm{LSTM}(s_{t-1}, E z_{t-1}) eq. 1

y_t = W_{proj}(o_t \oplus d_t) + b_{proj} \qquad d_t = attention(\mathrm{h}, s_t) eq. 2

p(w_t \mid y_{t-1}, s_{t-1}) = \mathrm{softmax}(W_{out} y_t + b_{out}) eq. 3

attention(\mathrm{h}, s_t) = \sum_{i=1}^{A} a_i^t h_i eq. 4

a_i^t = \mathrm{softmax}(u_i^t) \qquad u_i^t = v^T \tanh(W_1 h_i + W_2 s_t + b_2) eq. 5

u_i^t = v^T \tanh(W_1 h_i + W_2 s_t + b_2 + f_{t,i} \mu) \qquad f_t = F \ast a_{t-1} eq. 6

Pseudocode for one decoding step, Python-flavoured and written from the equations above §2 eq. 1-6:

# Attention-based encoder-decoder, one decoding step  [§2] [eq. 1-3]
# Input:  encoder outputs h (A, d_h); previous state; previous symbol z
# Output: new state; distribution over target vocabulary (V,)

def decode_step(h, state_prev, z_prev, z_ref, training):
    # Feed the reference symbol in training, the own prediction otherwise  [§2]
    z = z_ref if training else z_prev
    # New LSTM state and output from the embedded symbol  [§2] [eq. 1]
    state, o = lstm(state_prev, embed(z))
    # Attention vector over encoder outputs from the new state  [§2] [eq. 2]
    d = attention(h, state)
    # Project the concatenation of output and attention  [§2] [eq. 2]
    y = W_proj @ concat(o, d) + b_proj
    # Softmax over the target vocabulary  [§2] [eq. 3]
    probs = softmax(W_out @ y + b_out)
    return state, probs

Text attention (additive, one score per encoder position) §2 eq. 4-5:

# Additive attention for text input  [§2] [eq. 4-5]
# Input:  h (A, d_h); state (2m,)
# Output: context vector d (d_h,)

def text_attention(h, state):
    # Score each encoder position i  [eq. 5]
    scores = [v @ tanh(W1 @ h[i] + W2 @ state + b2) for i in range(A)]
    # Normalise over positions  [eq. 5]
    weights = softmax(scores)                  # a_i^t
    # Weighted sum of encoder outputs  [eq. 4]
    return sum(weights[i] * h[i] for i in range(A))

Convolutional attention for speech §2 eq. 6. The paper says it uses the attention model of Chorowski et al. and gives only the score; the final softmax is assumed from eq. 5:

# Convolutional attention for speech  [§2] [eq. 6]
# Input:  h (A, d_h); state (2m,); previous weights (A,)
# Output: weights (A,)

def speech_attention(h, state, prev_weights):
    # Convolve the previous weights with the filter F (size k = 25)  [§3] [eq. 6]
    filtered = conv1d(prev_weights, F)         # f_t
    # Score each position, with the filter term scaled by mu  [eq. 6]
    scores = [
        v @ tanh(W1 @ h[i] + W2 @ state + b2 + filtered[i] * mu)
        for i in range(A)
    ]
    # Normalise over positions (assumed softmax, see Pitfalls)  [eq. 5]
    return softmax(scores)

Hyperparameters

Results

Author-reported. Setup: BTEC French to English (text) and synthetic French speech (speech); baseline is phrase-based SMT (text) and Google Speech ASR plus SMT (speech) §3 tab. 2 tab. 3.

Corpus sizes tab. 1:

Text translation, BLEU tab. 2. Columns: greedy; + beam search (size 8);

• trigram LM; + ensemble of 5 NMT models; phrase-based SMT.

Speech translation, BLEU tab. 3. Columns: greedy; + beam search; + LM;

• ensemble; pipeline baseline (Google Speech ASR + SMT, ASR WER 23 to 26%).

Agnes is the held-out speaker; Michel is in the training set. The speech baseline's WER is 23 to 26% tab. 3.

Figure 1 fig. 1 shows the attention alignment during training, one panel for the text model and one for the speech model. It is described only, as the figure cannot be reproduced here.

Paper vs code

No verified official code. The paper's footnote §2 links github.com/eske/seq2seq, an Apache-2.0 fork of the TensorFlow seq2seq reference implementation. The GitHub account eske shows no connection to any of the four authors, so the repository is not treated as official. The card therefore uses the paper only. Nothing from that repository is cited here.

Pitfalls

• Synthetic speech. All speech data is concatenative TTS from six training voices and one test voice. The authors say the data has little variability within a speaker, which makes it easier to learn §5. Results on natural speech are not stated.
• Small corpus. BTEC train is about 16 hours of audio and about 20k sentences tab. 1. The authors note overfitting (train.1000 far above test scores) and say it does not appear to hurt evaluation §4.
• Model selection. Non-ensemble systems use the best of 5 runs chosen on dev tab. 2. The paper reports this as model selection, not tuning §4.
• Hidden size wording. The text says the decoder has 256 LSTM units and also that the LSTM state size is 2m §2. If 256 is m, the state is 512. The paper does not say whether the state holds both cell and hidden parts.
• Softmax in speech attention. The speech score is given only as the Chorowski-style term with the filter eq. 6. Normalisation is assumed to be the same softmax as eq. 5. The filter count, the value of \mu and whether it is learned are not stated beyond size m.
• Convolution size and padding. The filter size is 25 §2. Padding, stride and how the first step's weights are initialised are not stated.
• Downsampling details. The paper says the 2nd and 3rd encoder layers read every other output §2. Padding for odd lengths is not stated.
• Dropout scope. Dropout 0.5 is stated only between LSTM layers §2. Use elsewhere (embeddings, attention, output projection) is not stated.
• Learning-rate schedule and clipping. Only the initial rate is given. No decay, warmup or gradient clipping is stated.
• Greedy versus beam. The model section describes a greedy decoder §2. Beam search with size 8 appears only as a decode-time setting in the results tab. 2; the paper does not say which decoder was used for training-time monitoring.
• Ensemble weights. The ensemble is log-linear over five models plus an LM tab. 2, but the weights, and whether the LM weight is tuned, are not stated.
• Baseline comparability. The speech baseline is a pipeline with an external ASR system, and the SMT system was retrained on French without punctuation tab. 3. Gains in the text table are compared to a tuned SMT system; the paper says the SMT system is tuned on dev tab. 2.
• Dataset availability. The BTEC audio is produced with a commercial TTS system §3. Reproduction needs equivalent synthetic audio, which the paper does not provide.
• Cost. Speech training takes about 8 hours on one GTX 1070 for 20k steps, against under 2 hours for text §4.

Lineage

• Builds on: attention-based sequence-to-sequence models for translation (Sutskever et al.; Bahdanau et al.) and for speech recognition (Chorowski et al.; Chan et al.) §1 §2. No cards in this wiki.
• Attention and feature sources: additive text attention from Vinyals et al. §2; hierarchical encoder and convolutional attention from Chorowski et al. §2; MFCC front-end from Chan et al. §3. No cards in this wiki.
• Compared against: Google Speech API with a phrase-based SMT system (Moses), trained on BTEC tab. 3 tab. 2.
• Related prior attempts the paper names: Duong et al. (attention alignment without translation) and Anastasopoulos et al. (joint alignment) §1.
• Superseded by: not stated in this wiki.