Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities

In plain words

Researchers have long wondered what makes a paper "new". This paper looks at more than 14,000 papers from three big natural language processing (NLP) conferences and asks whether the mix of employers on the author team matters. Each team is put in one of three groups: all university or research-institute authors, all company authors, or a mix of both.

A computer program reads each paper's full text and pulls out four kinds of terms: methods, datasets, tools and metrics. A paper is judged more novel when it combines these terms in unusual pairs, and the most unusual papers form the top 10%. The authors report that papers from university-company teams were more likely to be novel than papers from company-only teams. Their pair analysis also suggests the mixed teams' novelty sits more often in method-metric pairs, while company-only teams' novelty sits more often in method-tool pairs.

TL;DR

• Novelty is measured per paper from unusual pairs of extracted entities (methods, datasets, tools, metrics) in full text §3.3.2.
• Team institutional type comes from author affiliations: academic (A&A), industrial (I&I) or mixed (A&I) §3.2.
• Regression results favour academic and mixed teams over industry-only teams for a binary top-10% novelty label and for a continuous score tab. 8 tab. 9.
• All numbers are author-reported on one corpus, and the paper states that it does not establish causation §6.2.

Problem

Team composition is linked to novelty, but earlier work looked mainly at team size, gender, leadership and international collaboration. Institution type was rarely examined, and most novelty measures return one number per paper with no detail on the kind of novelty §1 §2.2. The authors want a measure that says which kinds of entity combinations make a paper novel §1.

Method

The study runs in four stages fig. 1 §3:

1. Corpus. Full-text PDFs of main-track papers from ACL, EMNLP and NAACL, 1979 to 2021, were converted to HTML with pdf2htmlEX. Rule-based matching pulled out the full text of 15,337 papers. Titles, authors, years and affiliations were also crawled §3.1 tab. 1.
2. Team institutional type. Each affiliation was grouped into one of eight categories following Manjunath et al. (2021), then reduced to academic or industrial. Healthcare institutions were counted as academic. Single-authored papers and other institution types were dropped, leaving 14,812 papers §3.2 tab. 2 tab. 3. Each paper is labelled A&A, I&I or A&I from its authors' types §4.1.
3. Entity extraction. A SciBERT encoder feeds a BiLSTM in a cascade. It tags four entity types: method, dataset, tool and metric §3.3.1 tab. 4. The model was trained on sentences from 50 ACL papers annotated by five researchers in WebAnno §3.3.1. Applied to the 14,812 papers, it yields 151,018 entities §3.3.1 tab. 6.
4. Novelty score. The paper embeds each entity and computes the cosine distance for every pair of entities in a paper eq. 1. Pairs whose distance falls in the top 10% across the NLP corpus count as atypical. The novelty score of a paper is the number of atypical pairs divided by the number of all pairs of entities in that paper eq. 2 §3.3.2. Papers in the top 10% of scores are labelled novel §3.3.2.
5. Novelty bias by pair type. For novel papers, the share of atypical pairs that belong to each of ten entity-type pairs is computed eq. 3 eq. 4 §3.3.3.

Pseudocode for steps 4 and 5, following the paper's formulas:

# Novelty score of one paper  [§3.3.2] [eq. 1] [eq. 2]
# Input:  entities: list of (type, vector) for one paper
# Input:  cutoff: distance threshold at the corpus-wide top 10% of pairs
# Output: score in [0, 1]

def novelty_score(entities, cutoff):
    # All unordered pairs of entities in the paper  [§3.3.2]
    pairs = all_pairs(entities)                    # C(n, 2) pairs
    atypical = 0
    for (first, second) in pairs:
        # Cosine distance between the two entity vectors  [eq. 1]
        dot = dot_product(first.vector, second.vector)
        distance = 1 - dot / (norm(first.vector) * norm(second.vector))
        # Count pairs in the top 10% of distances  [§3.3.2]
        if distance >= cutoff:
            atypical += 1
    return atypical / len(pairs)                   # m / C(n, 2)  [eq. 2]

# Novel-paper label  [§3.3.2]
# Input:  scores: novelty score for each paper in the corpus
# Output: one boolean per paper

def label_novel(scores):
    # Papers in the top 10% of novelty scores are novel  [§3.3.2]
    cutoff = percentile(scores, 90)
    return [score >= cutoff for score in scores]

# Share of atypical pairs per entity-type pair  [§3.3.3] [eq. 3] [eq. 4]
# Input:  atypical_pairs: list of (type_a, type_b) for atypical pairs
# Output: dict mapping each of the ten type pairs to its share

def type_pair_shares(atypical_pairs, all_type_pairs):
    # Denominator m is the count of atypical pairs  [eq. 3] [eq. 4]
    m = len(atypical_pairs)
    shares = {}
    for type_pair in all_type_pairs:
        # n counts atypical pairs of this type  [eq. 3] [eq. 4]
        n = count_of(atypical_pairs, type_pair)
        shares[type_pair] = n / m
    return shares

The paper names ten type pairs: method-method, dataset-dataset, tool-tool, metric-metric (formula 3); method-dataset, method-tool, method-metric, dataset-tool, dataset-metric, tool-metric (formula 4) §3.3.3.

Analysis: group comparisons use the novel-paper share and the novelty score by team type §4.1 §4.2.1 fig. 7. Regressions use the novel label and the score as outcomes, with the team type as the independent variable and industrial teams as the reference group. Controls are task and year (fixed effects), top-level organisation, international collaboration and team size tab. 7 §4.2.2. Pair-type shares by team type are compared with one-way ANOVA §4.2.3 fig. 10.

Hyperparameters

Results

Author-reported. Setup: three NLP conferences, 14,812 papers after filtering, regressions on the same papers §4 tab. 8 tab. 9.

Entity extraction (own annotated test data §3.3.1):

Team type regression, binary novel label (reference group I&I):

Team type regression, continuous novelty score (reference group I&I):

Corpus shares and entity shares:

Other results, described in words:

• Mean novelty score by team type is close across groups, with I&I slightly lower; the text calls the difference significant without a test fig. 7 §4.2.2.
• Pair-type contributions: method-related pairs dominate in all three team types, about 76.61% of contributions to novel papers §4.2 fig. 9.
• One-way ANOVA: method-tool and method-metric shares differ by team type at P<0.1 and P<0.05. The mixed teams have the highest method-metric share, and industry-only teams the highest method-tool share §4.2.3 fig. 10.

Paper vs code

No official code. The paper has no code or data availability statement. The abstract and the raw metadata (raw/arxiv/2606.31058.json, which has no comments field) contain no repository link. A GitHub repository search for "novelty fine-grained knowledge entities" and "team institutional composition novelty" returned no results. The paper's own citations include Zhang et al. (2024) for the entity pipeline and data §3.3.1. That reference is not a repository link, so it is not treated as code.

Pitfalls

• Pair count mismatches. Table 8 and Table 9 report N = 14,812. The first-author binary split sums to 14,809 (12,803 + 2,006) and the three-way split to 14,811 (10,345 + 3,176 + 1,290) fig. 2 §4.1. The paper gives no explanation.
• Text and table disagree on Model 5. §4.2.2 quotes coefficients 0.062 and 0.05 for Model (5), which are the Table 8 Model (1) values. Table 9 Model (5) shows 0.0207 (A&I) and 0.0113 (A&A) §4.2.2 tab. 8 tab. 9.
• Affiliation totals. §3.2 quotes 1,998 unique affiliations and then 1,901. Table 2 sums to 1,998 and Table 3 to 1,901. The academic and industrial shares printed in the text (67.5%, 32.5%) differ slightly from the counts (1,285 and 616 of 1,901).
• Cutoffs not reported. The cosine-distance threshold for "atypical" and the novelty-score threshold for "novel" are not given numerically. A re-implementer has to pick them from the corpus, and the rate of novel papers depends on that choice.
• Entity handling not specified. The paper does not say how multi-word entities are embedded (pooling is not stated), whether duplicate entity strings in one paper are merged before pairing, or whether a pair of identical entities is counted. The worked example uses 190 = C(20, 2) pairs, which suggests duplicates were kept, but the text does not say so fig. 12.
• Formula 2 wording. Eq. 2 is introduced as "n biomedical entities" §3.3.2, which seems to be carried over from a biomedical study. The corpus here is NLP.
• Denominator of eqs. 3–4. Eq. 2 defines m as the number of atypical pairs in the paper. Eqs. 3–4 reuse the symbol without saying whether the denominator is the paper's atypical-pair count. The shares in the §5 case study (fig. 12) sum to 100% over all atypical pairs, which suggests it is. Confirm before implementing.
• Terminology. "Indicator" in the figures and text means metric, and "data" means dataset (tab. 4, fig. 8, fig. 9).
• Significance claims. Fig. 7 shows team-type means of about 0.12, 0.12 and 0.11 without a test, and the text still calls the differences significant §4.2.2 fig. 7. The ANOVA results in §4.2.3 do not report sample sizes or correction for ten pair types.
• Controls. The regression controls do not include the number of entities or paper length, and the novelty score is a fraction over all pairs, so it can depend on the number of entities tab. 7 eq. 2. The score distribution also shifts sharply around 2000 fig. 6, so the year fixed effects matter.
• Causal language. The Discussion describes team-type effects as influences on novelty. The limitations section says causal relations were not examined §6.1 §6.2.
• Annotation quality. Five annotators labelled 50 papers (3,115 sentences). Inter-annotator agreement and guidelines are not reported. The F1 in tab. 5 comes from this set.
• Data coverage. Only three conferences, and the authors note that industry papers are under-represented §6.2. The results may not transfer to other venues or fields.
• Reported outcome. The paper reports no confidence intervals, and the regression standard errors are in parentheses under a note that calls them a "robustness criterion" tab. 8 tab. 9.

Lineage

• Builds on: Uzzi et al. (2013) on atypical combinations and impact §2.1; Lee et al. (2015) on team size and novelty §2.2; Zhang et al. (2024) for the entity extraction method and annotated data §3.3.1. None of these have cards on this site.
• Compared against: the paper compares model baselines (BERT, BERT+CRF, XLM-RoBERTa, XLM-RoBERTa+CRF, SciBERT, SciBERT+CRF) for entity extraction tab. 5. No novelty baseline is compared against.
• Superseded by: not stated.