Comparing and Combining Sentiment Analysis Methods

In plain words

Many tools exist that decide whether a short social media message is positive or negative. This paper puts eight popular ones side by side. Some tools are word lists, some are emoticon lookups, and some are trained classifiers. The authors ask two questions. How many messages does each tool give an answer for at all (coverage)? When two tools give an answer, how often do they agree?

The answers differ a lot. Some tools label almost every message but are often wrong on the labeled data. Others are accurate on the messages they label but skip most of them. Tools also disagree on the mood of real events, so one tool can call the same set of posts positive while another calls them negative.

The authors then build a combined tool from seven of the eight. It covers the most messages while keeping reasonable accuracy. They also publish a free web service, iFeel, where the tools can be run on one text and compared.

TL;DR

• Eight polarity tools were compared on coverage (share of messages they label), agreement (share of joint labels that match), and precision, recall, accuracy and F-measure on six human-labeled datasets §3.1–§4.
• Coverage varies from about 4% to 95% on real events §1. The two tools with the highest coverage were SentiWordNet and SenticNet §4.1.
• A combined method of seven tools reaches 95% coverage with F 0.730 on the labeled data, below Emoticons' average F 0.846 §5.1, tab. 5.
• All numbers are author-reported. The paper has no official code, and its iFeel service is a beta web tool.

Problem

Polarity tools are used on social media in real time, but they were often built for longer text. Without a side-by-side comparison, a user cannot tell how much of a dataset a tool will leave unlabeled, or how much its labels depend on the choice of tool §1. The paper studies only positive versus negative polarity, not other emotions §6.

Method

The paper does not propose a learned model. It runs eight existing tools and compares their outputs. The Combined-method (§5.1) is the only new component, plus the iFeel service (§5.2).

The eight tools §2

• Emoticons §2.1: a fixed list of emoticons grouped into positive, negative and neutral tab. 1. A message takes the polarity of its first listed emoticon. Messages with none are not labeled.
• LIWC §2.2: the LIWC2007 English dictionary (over 4,500 words in about 100 categories). Polarity uses the relative rate of positive and negative feeling words. LIWC is commercial, so it is included in the comparison but excluded from iFeel and the Combined-method §5.1, §5.2.
• SentiStrength 2.0 §2.3: a supervised classifier trained on the labeled datasets of its own paper §3.1.2. Its published results are used instead of running it on those datasets §4.3.
• SentiWordNet 3.0 §2.4: for each word's synsets, the average positive and negative scores are compared. Objective scores are ignored §2.4.
• SenticNet 2.0 §2.5: concept-level polarity scores (for example -0.383 for "boring" and +0.228 for "Monday morning") averaged over the concepts of a text §2.5.
• SASA §2.6: the SASA Python package, version 0.1.3, a machine-learning tool §2.6.
• Happiness Index §2.7: a weighted average of ANEW word valences, on a 1 to 9 scale. The authors map scores in [1, 5) to negative and [5, 9] to positive §2.7.
• PANAS-t §2.8: the authors' Twitter adaptation of the Positive and Negative Affect Schedule. It scores eleven moods against a baseline, as values in [-1, 1]. The authors label joviality, assurance, serenity and surprise as positive, the six negative moods as negative, and attentiveness as neutral §2.8.

Data §3.1

• Near-complete Twitter logs §3.1.1: public Twitter posts from March 2006 to August 2009, with keyword-selected subsets for six events tab. 2. The events are AirFrance, 2008US-Elect, 2008Olympics, Susan Boyle, H1N1 and Harry-Potter.
• Labeled Web 2.0 data §3.1.2, tab. 3: six sets of human-labeled messages from the SentiStrength study: Twitter (4,242), MySpace (1,041), YouTube (3,407), BBC forum (1,000), Runners World (1,046) and Digg (1,077). Ground-truth shares are given in tab. 3.

Measures §3.2

Let a, b, c and d be the counts of true positives, false positives, false negatives and true negatives. The paper uses:

• recall R = a / (a + c);
• precision P = a / (a + b);
• accuracy A = (a + d) / (a + b + c + d);
• F-measure F = 2 · P · R / (P + R).

No equations are numbered in the paper. Coverage is the share of messages in a dataset that a tool labels as positive or negative §3.2.

Agreement is computed per pair of tools. For each pair, it is the share of messages both label on which their labels match §4.2, tab. 4.

Combined-method §5.1

The Combined-method uses PANAS-t, Emoticons, SentiStrength, SentiWordNet, SenticNet, SASA and Happiness Index. LIWC is left out for copyright reasons §5.1. Each included tool gets a weight from 1 to 7, based on the harmonic mean of its precision and recall §5.1. The stated goal is to reach the highest coverage first and then good agreement §5.1.

# Combined-method (as described in prose)  [§5.1]
# Input:  text; labels from the seven included tools
# Output: positive, negative, or not identified
def combined_polarity(text, tools, weights):
    # Each tool's weight is 1..7, from the harmonic mean
    # of its precision and recall  [§5.1]
    # The mapping from F to a weight is not stated  [§5.1]
    labels = [tool(text) for tool in tools]
    # Rule for resolving disagreeing labels: not stated  [§5.1]
    return combine(labels, weights)

iFeel web service §5.2

iFeel runs the seven tools and LIWC is excluded §5.2. It accepts texts of up to 200 characters, is built with the Pyramid Python framework §5.2, and is served as a beta at the URL given in §5.2. The authors plan file input and visualisations §5.2.

Hyperparameters

Results

Author-reported. Setup: eight existing tools, six Twitter events from the 1.7-billion-tweet log §3.1.1, and six labeled datasets tab. 3. The baseline is the best single tool.

Single-tool highlights, all from the paper:

• Emoticons has the best average F, 0.846, but the lowest coverage §4.3, tab. 5.
• LIWC has the most agreement with other tools on average (73.49), and the last column of tab. 4 reports this as its average agreement with the others.
• On the Twitter events, half of the tools put the AirFrance tweets on the positive side, and only two of eight put H1N1 on the negative side §1, §4.4.

Paper vs code

No official code. Search steps:

1. Links in the paper: the paper links the iFeel beta site, the tool sites and their datasets, and no code repository.
2. Project page: the iFeel address (www.ifeel.dcc.ufmg.br) redirects to a non-public host (blackbird.dcc.ufmg.br:1210) that was not fetched, because the redirect is to a host the paper does not give.
3. GitHub search: the search q=ifeel returned repositories by unrelated owners (for example onestraw/iFeel, tiagoetiene/ifeel), none linked to the authors. Two further queries (sentiment coverage agreement iFeel combined-method and pollyanna goncalves sentiment) returned HTTP 403 and could not be read, so the GitHub search is incomplete.

The only code-like artifacts are the third-party tools the paper runs, which are not the authors' code and are not on this card.

Pitfalls

• Text and table disagree on F for the Combined-method. §5.1 says its F of 0.730 is higher than all other tools except Emoticons, but tab. 5 lists SentiStrength at 0.765. The figure in fig. 4a may use a different averaging; the paper does not say.
• The F column of tab. 5 is not the harmonic mean of the table's own precision and recall. For Emoticons, P = 0.867 and R = 0.856 give F of about 0.862, not the listed 0.846. The F values appear to be averaged per dataset §3.2, tab. 5.
• SentiStrength figures are not measured here. The authors used the numbers from the SentiStrength paper because the tool was trained on these datasets §4.3. Its row in tab. 5 is not directly comparable with the others.
• The 1.7 vs 1.8 billion figure. §3.1.1 gives 1.7 billion tweets. §1 says the dataset has 1.8 billion messages §1, §3.1.1.
• Coverage depends on the order of tools. §4.1 combines tools from highest to lowest coverage, and reports above 92.75% after two tools. §5.1 adds tools in the order Emoticons, SentiStrength, Happiness Index, and so on, and reports 70% after two. The two numbers are not comparable.
• Emoticon agreement claim. §1 says Emoticons reach agreement above 85% and also show agreement of 4% to 13%. The source has commented-out positive and negative agreement tables, and no active table gives these numbers §1.
• Pairwise agreement range. §4.2 says agreement ranges from 33% to 80%. tab. 4 has entries from 30.77% to 83.33%. The text and the table do not match.
• Terminology in §3.2. The paper calls precision the "false positive rate" and recall the "true positive rate". The formulas given are the usual precision and recall, so the labels are wrong, not the formulas.
• Event tables. tab. 2 lists dates in an unclear format (for example "11.02--06.2008") and says it shows tweet totals, but no count column is present. Exact date ranges and counts are not stated.
• Matching rule for events. Tweets were selected by keyword search §3.1.1. Case handling, tokenisation and spam filtering are not stated.
• Neutral labels. Neutral emoticons, neutral PANAS-t moods and the Happiness Index mid-band are defined, but how neutral outputs enter coverage and agreement is not stated.
• PANAS-t per-message rule. The method scores mood change over a time period §2.8. How a single message gets a polarity is not stated.
• LIWC and copyright. LIWC appears in the comparison but not in iFeel or the Combined-method §5.1, §5.2. Its software is commercial §2.2.
• Real-time bias. Most tools return more positive than negative labels on the datasets and events studied §4.4. Real-time monitors that count positive and negative rates will inherit that bias §4.4.
• Subset of tools. Only eight tools were compared. Other tools, including POMS, are not covered §6.
• Combined-method mapping is not reproducible from the text. The weight-to-F mapping, and the rule for resolving conflicting labels, are not stated §5.1. An implementer must choose these, and the reported 95% and 0.730 cannot be reproduced from the paper alone.

Lineage

• Builds on: the eight tools themselves, with their original papers as cited in §2 (for example SentiWordNet, LIWC, SentiStrength, SenticNet, the Happiness Index). PANAS-t is the authors' own earlier method §2.8; their bibliography cites it as arXiv abs/1308.1857v1.
• Compared against: the eight tools above. POMS-based tools are not included because the authors could not obtain access §2.8.
• Superseded by: not stated in the content of this wiki.