"I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data

In plain words

Early users of ChatGPT talked about it a lot on Twitter in its first weeks. The authors collected about eleven thousand English tweets that mentioned ChatGPT, grouped them into nine themes with a topic model, and then read a sample of tweets from each theme by hand to label them as positive, negative or neutral.

The result is a snapshot of what early users felt. Most themes were positive: coding help, fun and creative writing, and business ideas. The most worried themes were education, where people feared students would use it to cheat, and answer accuracy, where people noticed confident wrong answers.

The study is descriptive. It does not train or test a model, and the sentiment labels come from people reading the tweets, not from a classifier. It is a useful reference for how such a study was run and for what early reactions looked like.

TL;DR

• Mixed-method study of 10,732 de-duplicated English tweets that mention "ChatGPT", collected 5-7 December 2022 §3.2.
• Topic modelling with LDA in MALLET yields nine topics, chosen by a coherence sweep and a manual check §3.4 tab. I.
• Sentiment is labelled by hand (-1, 0, 1) on 100 random tweets per topic, two annotators per topic §3.5.
• Most topics are mostly positive; education and Q&A carry the largest negative shares §4.3.
• Author-reported and descriptive. No inter-annotator agreement is reported.

Problem

Early adopters' opinions of a new tool can signal how it will be received and where it fails. The authors want a qualitative account of these opinions, with the main topics and per-topic sentiment, drawn from public Twitter data §1 §2.

The paper says it is the first qualitative study of early ChatGPT feedback §2. It names no tool-specific prior work to compare against.

Method

Section numbers are the paper's Roman-numbered sections written in Arabic: §3.2 is section III-B, §4.3 is IV-C. The pipeline has five stages. fig. 1 gives the overview.

1. Collect. Gather tweets from 5 to 7 December 2022 that contain the keyword "ChatGPT" and are in English. Store text, user location, occupation, verification status, date and hashtags. Collection used Python and the Twitter API §3.2.
2. Clean. Remove retweets and duplicate tweets, lowercase, strip punctuation, URLs, emojis and handles, remove NLTK English stop words plus the three most frequent domain words (ChatGPT, OpenAI, AI), and lemmatize with NLTK's WordNet lemmatizer §3.3. Duplicate removal leaves 10,732 tweets from about 18K §3.2 §3.3.
3. Topics. Fit LDA with MALLET over a range of topic counts, select the count with a coherence sweep plus manual reading, then name each topic from its top 20 keywords and 30 sampled tweets §3.4 §4.2.
4. Sentiment. Label 100 random tweets per topic as negative (-1), neutral (0) or positive (1). Neutral covers both-sided and ambiguous tweets. Each topic is labelled by two of the five authors, with disagreements resolved by discussion. Open coding then yields key points and short codes §3.5.
5. Report. Give per-topic percentages, example tweets and implications §4 §5.

The authors tried NLTK's automatic classifier first and dropped it because it mislabelled mixed tweets and ignored images such as screenshots §3.5. This choice means the sentiment numbers are not reproducible with an off-the-shelf classifier.

# Topic and sentiment pipeline  [§3.2] [§3.4] [§3.5]
# Input:  raw tweets (keyword "ChatGPT", English, 5-7 Dec 2022)
# Output: nine topics, per-topic sentiment shares

def run_study(raw_tweets):
    # 1. Remove retweets and duplicate tweets  [§3.3]
    tweets = deduplicate(raw_tweets)           # 18K -> 10,732 reported

    # 2. Normalise text  [§3.3]
    tweets = [lemmatize(remove_noise(lower(t))) for t in tweets]
    tweets = [drop_stopwords(t) for t in tweets]

    # 3. Choose the topic count N  [§3.4]
    best_n = choose_topic_count(tweets, range(5, 61, 2))

    # 4. Fit LDA with MALLET  [§3.4]
    topics = lda_mallet(tweets, n_topics=best_n)

    # 5. Sample and label per topic  [§3.5]
    for topic in topics:
        sample = random_sample(topic.tweets, 100)
        labels = manual_label(sample)           # two annotators per topic
        shares = share_of(labels)               # negative / neutral / positive
    return topics, shares

Values in the pseudocode are the paper's: 18K raw, 10,732 after duplicates, 100 tweets per topic, N = 9 chosen from a search over 5 to 60 §3.2 §3.4. The choose_topic_count step is a compact description; the paper also did a manual check of N in 9 to 12 §3.4.

Hyperparameters

Results

Author-reported. Setup: Twitter, 10,732 tweets, LDA with nine topics, manual labelling of 100 tweets per topic §3 §4.

Table I lists the topics and their top-10 lemmatized keywords. The topic names come from the authors' reading of keywords and tweets §4.2 tab. I.

Other reported facts:

• Location: the top five countries in terms of tweets expressing an opinion were the USA, India, UK, Canada and Germany §4.1 fig. 3.
• Verification: about 2% of the early adopters were verified Twitter accounts §4.1 fig. 4.
• Occupation: software practitioners, academics and students were the three largest groups §4.1 fig. 5.
• Topic sizes: the text ranks Future Careers as largest and Q&A Testing as third largest, but the counts are only in fig. 6, which the text does not state numerically §4.2.

The text gives only a positive share for software development and for entertainment; their other shares are not stated.

Paper vs code

No official code. The paper has no code or data link in the abstract, the TeX source or its footnotes (the only URL is openai.com). A GitHub repository search for "ChatGPT early adopters twitter" returned zero results. The paper does not name a method implementation, so no code is cited.

Pitfalls

• Neutral is not "no opinion". Neutral covers tweets that discuss both sides and tweets that are ambiguous §3.5. Any reuse of the percentages should keep this meaning.
• Manual labels, no agreement score. Two annotators per topic, disagreements resolved by discussion, and no kappa value in the text. A kappa sentence is commented out in the LaTeX source, so the published paper reports no agreement statistic §3.5.
• Dataset size. The number of labelled tweets is nine samples of 100, which is 900 tweets, about 8% of 10,732. This count is derived from the text, not stated as a total §3.5.
• Topic counts. The text gives no count per topic. Ranking claims depend on fig. 6, which has no numbers in the text. Treat the "largest" and "third largest" claims as unverified from the text.
• Tweet-to-topic assignment. LDA yields a probability per topic for each tweet, but the paper does not state how one topic is chosen per tweet. A commented-out sentence describes a highest-probability rule, which is not in the published text §3.4.
• Collection bias. Only tweets with the keyword "ChatGPT", English only, for three days in December 2022. Results describe early discussion in that window, not later users §3.2 §6.
• Occupation and location. The method for deriving occupation from profiles is not stated, and it is not stated how location strings were normalised §3.2 §4.1.
• Stop words. The text says "such as" before a three-word list, so it is not clear whether exactly three domain words were removed §3.3.
• Typeset numbers. Figure and table numbers above follow the LaTeX order. Float placement in the typeset PDF can reorder them; check against the PDF before quoting a number.
• Source typo. Table I contains the token "avlable" in the Search Engines keywords. It is a source typo, not a keyword to reuse.

Lineage

• Builds on: nothing stated beyond general LDA and sentiment practice. The paper cites LDA and MALLET for topic modelling and NLTK for preprocessing, none of which has a card here.
• Compared against: the related-work section names Twitter sentiment studies of IoT and hybrid work §2. This card uses none of their content.
• Superseded by: not stated.