Conversational recommendation

Overview

A conversational recommender talks with a seeker, learns what they want, and proposes items during the chat. The question the INSPIRED paper asks is whether the social moves a human recommender makes, such as sharing an opinion or encouraging the seeker, make a suggestion more likely to be accepted, and whether a dialog model can learn to make those moves 2009.14306 §1.

Success is defined from the seeker's behaviour after the suggestion. A recommendation counts as successful when the seeker watched more than half of the trailer and rated it 4 or 5 2009.14306 §5.2. The same paper also treats a reply classified as a recommendation as the trigger for a movie lookup, so the dialog model and the recommender are linked [arxiv:2009.14306 §6.2].

Only one paper with a card covers this concept. Its results are the authors' own, on a dataset they collected, and they are not compared with other conversational recommenders. Treat the numbers below as one data point, not a benchmark.

Variants

Sociable-strategy labels on a GPT-2 dialog model

The data is human-human movie dialogues, with each utterance labelled by the strategy it uses. The scheme has eight sociable strategies (personal opinion, personal experience, similarity, encouragement, offering help, preference confirmation, self-modelling and credibility), two preference-elicitation inquiries, and non-strategy classes such as acknowledgement and recommendation 2009.14306 §4.1. The paper reports 1,001 dialogues and 35,811 utterances, split 801 / 100 / 100 for training, validation and test 2009.14306 §6 2009.14306 tl;dr.

The model has two GPT-2 small language models, one per speaker, and both condition on the shared history 2009.14306 §6.1. The strategy-labelled version writes the strategy as a special token before each recommender turn. The baseline uses the same architecture without the token and is trained on the same dialogues 2009.14306 §6.1 2009.14306 §6.2.

Reported results, author-reported on 100 test dialogues 2009.14306 tab. 7:

In a human comparison with 28 participants, the labelled model was preferred on all five aspects, with the largest gap on engagingness 2009.14306 §6.3. Five-point ratings from 50 participants give the same ordering, with the engagingness scores at 3.56 for the baseline and 3.72 for the labelled model 2009.14306 tab. 10. The text gives no counts or significance test for the pairwise comparison 2009.14306 §6.3.

The link to success is correlational. In a logistic regression over 667 dialogues with a clear outcome, personal opinion (0.12), similarity (0.23), encouragement (0.20) and credibility (0.09) have positive coefficients, and similarity and encouragement are significant at the stated levels 2009.14306 §5.2 2009.14306 tab. 6. The authors do not run a controlled test, so these coefficients show association only 2009.14306 §5.2.

When to use which

There is one variant, so the choice is between using its method or not. The constraints below are the ones the paper's evidence bears on.

Implementation sketch

The core loop is: prepend the strategy label, sample several candidate replies, drop degenerate ones, and pick one. The paper's selection rule differs from the released code, so the sketch shows the paper's steps and the notes below list the differences.

# Fine-tune one GPT-2 model per speaker on labelled turns  [§6.1]
# Each recommender turn starts with its strategy as a special token  [§6.2]
turn_tokens = [STRATEGY_TOKEN[turn.label]] + tokenize(turn.text)

# Sample candidate replies from the same history  [§6.2]
# The paper samples five candidates per turn
candidates = [sample(model_a, history, temperature=0.8,
                     top_k=400, top_p=0.9)
              for _ in range(5)]  # [app. D]

# Drop candidates with repeated bigrams  [code note below]
candidates = [c for c in candidates if not has_repeated_bigram(c)]

# Paper: prefer candidates with encouragement or the longest length
# [§6.2]
chosen = pick_candidate(candidates)

Notes for implementers:

• The paper's selection rule (encouragement or greatest length) is not what its released code runs, which chooses at random among the filtered candidates. The code also samples three candidates, not five. The paper card lists these differences in full.
• The paper gives a temperature of 0.82 for the baseline and 0.8 for the labelled model 2009.14306 app. D. The released code has only 0.8.
• The paper says training runs "until convergence", about two epochs for the baseline and three for the labelled model 2009.14306 app. D. The released code runs ten epochs.
• The labelled model does not enforce its strategy. The label is generated as part of the reply, and nothing checks that the chosen reply carries it.

Contradictions

Only one paper with a card covers this concept, so there are no disagreements between papers. Within the paper, the text reports 79.7% of cases as successful 2009.14306 §3.3, but the case counts in its table do not produce that figure under any simple grouping the paper card checked 2009.14306 tab. 3. Treat that percentage as unreconciled.

Papers

• INSPIRED: Toward Sociable Recommendation Dialog Systems: adds labelled social strategies to a GPT-2 recommender and reports author-run gains on one small dataset.