wikillm

Paper · 2021-02-22 · cs.CV

Hierarchical Graph Representations in Digital Pathology

Builds a hierarchical graph of a breast H&E region: nuclei form a cell-graph, tissue regions a tissue-graph, and each nucleus is linked to its region. HACT-Net, a PNA graph network, classifies it. Author-reported: 61.53% weighted F1 on 7-class BRACS, against 56.62% for the tissue-graph alone and 52.83% for the best multi-scale CNN.

Graph neural networkPrincipal neighbourhood aggregationdigital-pathologyhistologybreast-cancergraph-neural-networkhierarchical-graphprincipal-neighbourhood-aggregation

arXivPDFCodeMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A pathologist reads a tissue sample at two scales at once: single cells, and the larger regions they form, such as gland lining, stroma or necrosis. Most computer models cut the image into fixed squares and look at one square at a time, which loses either the detail of cells or the wider context.

This paper turns one tissue image into a map with two layers. One layer has a node for each cell and links nearby cells. The other has a node for each tissue region and links neighbouring regions. Each cell is also attached to the region it sits in. A neural network reads the cell layer first, passes its summary up to the region layer, and then predicts the tissue type. The authors also release a breast cancer dataset with seven subtypes, labelled by pathologists, and compare the model with pathologists on part of it.

TL;DR

  • Input: one breast H&E tumour region of interest. Output: one of seven subtypes.
  • Method: a hierarchical graph (cell-graph, tissue-graph, and the links between them), processed by PNA graph layers in two stages.
  • Author-reported: 61.53% weighted F1 on the 7-class BRACS task, with the best of the compared baselines near 57%.
  • Code exists under MIT. The paper's stated settings and the released code disagree in several places (see Paper vs code).

Problem

Patch-based CNNs trade field of view against resolution: large patches see context but blur cells, and small patches see cells but not the surrounding tissue §I. Cell-graphs capture cell morphology and arrangement but not tissue regions, and tissue-graphs do the reverse §I. The paper argues that a single entity type cannot describe tissue composition at several scales, so it proposes a graph with two entity types and links between them §I §IV.B.3.

The method assumes that nuclei can be detected reliably, that tissue regions can be segmented into homogeneous superpixels, and that a whole tissue region of interest (ROI) is available as one image §IV.B.

Method

Stages, in order §IV:

  1. Pre-processing. Stain normalisation of the ROI to reduce appearance variation between slides §IV.A.
  2. Cell-graph (CG). Detect nuclei and use their centroids as nodes. Each node carries a CNN feature of a patch around its centroid, plus its normalised centroid. Edges join each node to its nearest neighbours if they are close enough, using the rule in eq. 2 §IV.B.1.
  3. Tissue-graph (TG). Over-segment the tissue into superpixels, merge neighbours with similar colour statistics into tissue regions, then build one node per region with a mean CNN feature over its superpixels and a normalised centroid. Adjacent regions are linked (region adjacency graph) §IV.B.2.
  4. HACT graph. The CG, the TG and a binary assignment matrix that connects each nucleus to the region containing it. A nucleus on a border goes to the region it overlaps most eq. 3 §IV.B.3.
  5. HACT-Net. Two stages of PNA message passing. The first runs on the CG, with an LSTM that combines the outputs of all layers ("jumping knowledge"). Its per-cell outputs are summed into each tissue node's initial state. The second stage runs on the TG in the same way. A readout sums the tissue-node representations, and a two-layer MLP with softmax gives the class §IV.C.1 eq. 4 eq. 5 eq. 6 eq. 7 fig. 4.

The CG and TG layers use the same PNA update eq. 4. PNA aggregates neighbour messages with four statistics (mean, standard deviation, max, min) and scales each aggregate by a degree term, with three scalers: identity, amplification and attenuation eq. 5 fig. 1. Graph normalisation and batch normalisation follow each PNA layer §IV.C.1.

Pseudocode for graph construction (one ROI):

Pseudocode
# HACT graph construction  [§IV.A] [§IV.B]
# Input:  roi (H, W, 3) H&E image
# Output: cell_graph, tissue_graph, assignment (nuclei x regions)

def build_hact(roi):
    # 1. Normalise the stain of the ROI  [§IV.A]
    roi = normalise_stain(roi)

    # 2. Detect nuclei and keep their centroids  [§IV.B.1]
    nuclei_centroids = detect_nuclei(roi)

    # 3. CNN feature from a patch around each centroid  [§VI.B]
    cell_features = patch_cnn_features(roi, nuclei_centroids, patch=72)

    # 4. Connect each nucleus to its nearest neighbours [eq. 2]
    #    and drop edges longer than a threshold d_min  [§IV.B.1]
    cell_graph = knn_graph(nuclei_centroids, cell_features, d_min)

    # 5. Over-segment into superpixels, then merge similar neighbours
    #    into tissue regions  [§IV.B.2]
    regions = merge_similar(oversegment(roi))

    # 6. Mean CNN feature of the superpixels in each region  [§IV.B.2]
    region_features = region_mean_cnn(roi, regions, patch=144)

    # 7. Link adjacent regions (region adjacency graph)  [§IV.B.2]
    tissue_graph = region_adjacency_graph(regions, region_features)

    # 8. Assign each nucleus to one region  [§IV.B.3] [eq. 3]
    assignment = assign_to_region(nuclei_centroids, regions)

    return cell_graph, tissue_graph, assignment

HACT-Net forward pass (training adds softmax cross-entropy on the output):

Pseudocode
# HACT-Net forward pass  [§IV.C.1]
# Input:  cell_graph, tissue_graph, assignment (nuclei x regions)
# Output: class logits (7)

def hact_net(cell_graph, tissue_graph, assignment):
    # 1. Run T_cg PNA layers on the cell-graph  [eq. 4] [eq. 5]
    cell_states = run_pna_layers(cell_graph, cell_layers)
    # 2. LSTM over the outputs of all cell layers  [eq. 6]
    cell_embeddings = lstm_over_layers(cell_states)
    # 3. Each tissue node starts from its own features plus the sum
    #    of the cell embeddings assigned to it  [eq. 7]
    tissue_start = concat(tissue_graph.node_features,
                          assignment_sum(assignment, cell_embeddings))
    # 4. Run T_tg PNA layers on the tissue-graph  [§IV.C.1]
    tissue_states = run_pna_layers(tissue_graph, tissue_start, tissue_layers)
    # 5. LSTM over tissue layers, then sum over tissue nodes  [§IV.C.1]
    tissue_embeddings = lstm_over_layers(tissue_states)
    roi_embedding = sum_over_nodes(tissue_embeddings)
    # 6. Two-layer MLP gives the class logits  [§VI.B]
    return mlp(roi_embedding)

Key equations, as numbered in the paper:

A_{\mathrm{cg} \rightarrow \mathrm{tg}}[i, j] = 1 \text{ if nucleus } i \text{ lies in region } j, \text{ else } 0 eq. 3

\mathcal{S}(D, \alpha) = \frac{\log (D + 1)^{\alpha}}{\delta}, \quad \delta = \frac{1}{|\text{train}|} \sum_{i \in \text{train}} \log (d_i + 1) eq. 5

h^{(T_{\mathrm{cg}}+1)}_{\mathrm{cg}}(v) = \text{LSTM}\big(\{ h^{(t)}_{\mathrm{cg}}(v) : t = 1..T_{\mathrm{cg}} \}\big) eq. 6

h^{(0)}_{\mathrm{tg}}(w) = \text{CONCAT}\Big(H_{\mathrm{tg}}(w),\; \sum_{v \in \mathcal{M}(w)} h^{(T_{\mathrm{cg}}+1)}_{\mathrm{cg}}(v)\Big) eq. 7

Equations 1 (message passing in general) and 4 (PNA update for the cell-graph) are in §III.B and §IV.C.1.

Hyperparameters

Values in the paper come from §VI.B unless marked. "not stated" means the paper does not give it. Code values are from the official repository at code_commit.

NamePaperCodeWhere
Nuclei detectorHoVer-Net, pretrained on MoNuSeg §IV.B.1NucleiExtractor() from the external histocartography librarygenerate_hact_graphs.py
Cell patch size72×72 px §VI.B72 px, resized to 224 for ResNetgenerate_hact_graphs.py
Tissue patch size144×144 px §VI.B144 px, resized to 224generate_hact_graphs.py
CNN backboneResNet-34 or ResNet-50, ImageNet §VI.BResNet-34 onlygenerate_hact_graphs.py generate_hact_graphs.py
Cell node featuresCNN features plus spatial coordinates §IV.B.1512 CNN + 2 coordinates = 514generate_hact_graphs.py train.py
kNN size knot stated5generate_hact_graphs.py
Edge cut-off d_minnot stated50 pxgenerate_hact_graphs.py
SuperpixelsSLIC, low magnification, then merged §IV.B.2size 500, compactness 20, blur 1, threshold 0.05, downsampling 4generate_hact_graphs.py
Stain normalisationMacenko, then a fast pipeline §IV.AVahadane with a target imagegenerate_hact_graphs.py generate_hact_graphs.py
PNA layers per graph3, 4 or 5 (options) §VI.B3 (CG, TG and HACT configs)bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml
PNA aggregatorsmean, std, max, min eq. 5mean max min stdbracs_hact_7_classes_pna.yml
PNA scalersidentity, amplification, attenuation eq. 5identity amplification attenuationbracs_hact_7_classes_pna.yml
Degree normaliser δmean log-degree over training graphs eq. 5fixed avg_d: 4bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml
Jumping knowledgeLSTM over layer outputs eq. 6readout_op: lstmbracs_hact_7_classes_pna.yml
Readout typesum over tissue nodes §IV.C.1readout_type: meanbracs_hact_7_classes_pna.yml
Dropout in GNNnot stated0bracs_hact_7_classes_pna.yml
NormalisationGraphNorm then BatchNorm after each PNA layer §IV.C.1both onbracs_hact_7_classes_pna.yml
PNA MLP2 layers, 64 channels §VI.Bpre-transform 1 + post-transform 1 layers, output dim 64bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml
Graph embedding size128 §VI.Bnot in config (library)not verifiable here
Output classifier2 layers, 128 channels §VI.Bnum_layers: 2, hidden_dim: 128bracs_hact_7_classes_pna.yml
OptimiserAdam, learning rate 1e-3 §VI.BAdam, default lr 1e-2, weight decay 5e-4train.py train.py
Batch size16 §VI.Bdefault 1; README command uses 8train.py README.md
Epochsnot stateddefault 10; README command uses 60train.py README.md
Losscategorical cross-entropy §VI.BCrossEntropyLosstrain.py
Model selectionbest validation weighted F1 §VI.Bthree checkpoints saved, each testedtrain.py train.py
Repeatsthree runs with random initialisation §VI.Bno seed set in the codetrain.py
ROI size filternot statedkeep 50,000 to 50,000,000 pixelsgenerate_hact_graphs.py generate_hact_graphs.py
Class orderseven subtypes §VN, PB, UDH, ADH, FEA, DCIS, IC → 0..6generate_hact_graphs.py

Results

Author-reported. Setup: BRACS, 4,391 ROIs from 325 breast carcinoma WSIs, seven classes, WSI-level split into 3,163 train, 602 validation and 626 test ROIs §V tab. I. Metrics are weighted F1 (mean ± standard deviation over three runs) unless stated. Baselines are described in §VI.A.

7-class, BRACS tab. V:

MethodWeighted F1 (%)Source
CNN, 10×50.85 ± 2.64tab. V
CNN, multi-scale 10×+20×+40×52.83 ± 1.92tab. V
CGC-Net (cell-graph baseline)43.63 ± 0.51tab. V
Patch-GNN, 10×52.10 ± 0.61tab. V
CG-GNN (cell-graph only)55.94 ± 1.01tab. V
TG-GNN (tissue-graph only)56.62 ± 1.35tab. V
CONCAT-GNN (two separate graphs, concatenated)57.01 ± 2.27tab. V
HACT-Net (proposed)61.53 ± 0.87tab. V

4-class, BRACS (Normal; Non-cancerous; Precancerous; Cancerous) tab. VI: HACT-Net 69.04 ± 0.46, best baseline CNN at 10× 64.36 ± 1.37 and TG-GNN 66.24 ± 1.11.

Binary decision tree, aggregated over six tasks tab. VII: HACT-Net 84.15 ± 0.60; CONCAT-GNN 82.61 ± 2.15; CG-GNN 81.50 ± 1.70; single-scale CNN at 10× 78.90 ± 1.38.

Ablations, 7-class weighted F1 tab. II tab. III tab. IV:

  • Node features: none 48.70, hand-crafted 52.46, CNN 61.53 for HACT-Net.
  • Layer type for HACT-Net: GIN 59.73, PNA 61.53.
  • Jumping knowledge for HACT-Net: none 49.16, concatenation 59.78, LSTM 61.53.

Pathologists, 7-class, BRACS test ROIs without the WSI context tab. VIII: three independent pathologists averaged 56.36 ± 0.76 weighted F1 and 57.08 ± 0.64 weighted accuracy. HACT-Net, on the same ROIs, scores 61.53 ± 0.87 weighted F1 and 63.21 ± 0.27 weighted accuracy. Pairwise agreement among the pathologists is between 47.60% and 64.38% tab. IX.

BACH, 4-class accuracy, organiser-evaluated tab. X: HACT-Net 91.00 with one network. The listed ensemble methods score 86.00 to 95.00.

Figures, described only: the 7-class confusion matrix shows the most confusion between Normal and Benign, between UDH and ADH, and between ADH and DCIS fig. 8 §VI.D. The attribution maps highlight tumour epithelium and necrotic regions on the tissue-graph fig. 11 §VI.E.

Paper vs code

Official code: https://github.com/histocartography/hact-net, MIT licence, at commit 4b66b41a2f00b768029b6aaa1152c746faf2b9c6. The paper does not link it. The owner, the histocartography organisation, is the group that the authors Pati and Gabrani work with at IBM Research Zurich, and the repository README cites this paper. The model classes (HACTModel, NucleiExtractor, DeepFeatureExtractor, VahadaneStainNormalizer, AssignmnentMatrixBuilder) come from the external histocartography library, which is not in the repository and is not pinned in environment.yml, so behaviour inside those classes is not verified here.

Differences:

  • Stain normalisation. The paper uses Macenko's method, followed by a faster pipeline §IV.A. The code uses Vahadane normalisation against one target image generate_hact_graphs.py generate_hact_graphs.py.
  • Learning rate and batch. The paper uses learning rate 1e-3 and batch size 16 §VI.B. The README command uses 5e-4 and batch 8 README.md. The train.py defaults are 1e-2 and batch 1 train.py train.py.
  • Weight decay. The paper states no weight decay. The optimiser uses 5e-4 train.py.
  • Epochs. Not stated in the paper. The README uses 60 README.md; the default is 10 train.py.
  • PNA degree normaliser. The paper defines δ as the mean log-degree over the training set eq. 5. The configs fix avg_d: 4 bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml.
  • Readout. The paper sums tissue-node representations §IV.C.1, and the CG-GNN baseline uses a sum readout §VI.A. The configs set readout_type: mean for all three models bracs_cggnn_7_classes_pna.yml bracs_hact_7_classes_pna.yml.
  • PNA depth. The paper lists 3, 4 or 5 layers §VI.B. The configs use 3, and the paper does not say which depth produced the table values.
  • CNN backbone. The paper allows ResNet-34 or ResNet-50 §VI.B. The code uses only ResNet-34 generate_hact_graphs.py.
  • Patch resize. The code resizes each patch to 224 px before the CNN generate_hact_graphs.py. The paper gives no resize step.
  • Assignment rule. The paper assigns border nuclei to the region with the largest overlap §IV.B.3. The code passes nucleus centroids and the tissue map to a library builder generate_hact_graphs.py. The overlap rule is not visible in this repository.
  • Data split. The paper uses a random WSI-level split §V. The code reads the train, val and test folders of the BRACS release generate_hact_graphs.py and does no splitting.
  • Accuracy in the README. The README reports 61.7% accuracy and 61.5% weighted F1 for the pretrained HACT-Net checkpoint README.md. The paper's weighted accuracy of 63.21 tab. VIII is a different summary, so the two numbers should not be compared directly.
  • Graph embedding size. The paper states 128 §VI.B. The HACT config sets output_dim: 64 for the PNA stages bracs_hact_7_classes_pna.yml. Where the 128 comes from is not visible in this repository.

No discrepancy was found for the classifier head (two layers, 128 channels), the label order, or the cross-entropy loss.

Pitfalls

  • Dataset access. BRACS requires a registration and an FTP download of the previous version of the regions-of-interest set README.md. The paper's footnote says the release was "shortly" to follow, so the released split may not match the paper's random WSI-level split §V.
  • Silent image filtering. ROIs outside 50,000 to 50,000,000 pixels are skipped, and the code prints only a message for them generate_hact_graphs.py generate_hact_graphs.py. Check the count of graphs produced against Table I.
  • Hidden parameters. The paper leaves k, d_min, the superpixel sizes, the dropout, the epoch count and the seed unstated. The code values above fill them, but the paper's numbers may come from different settings.
  • Edge threshold is in pixels. The code's 50 px cut-off depends on the resolution at which graphs are built (0.25 µm/pixel for BRACS §V). A different scan resolution changes the graph without any code change.
  • Node-count spread. Graph sizes vary widely: cell-graphs average 1,468 ± 1,642 nodes and tissue-graphs 172 ± 217, with max/min ratios up to 256 for cell-graphs tab. I. Use per-graph normalisation, and expect large memory use for the largest ROIs.
  • Which tensor is summed. The paper says the graph-level embedding is the sum of tissue-node representations §IV.C.1. It does not say whether this is before or after the LSTM over layers. Check the library before implementing.
  • Assignment matrix orientation. The code transposes the stored matrix when it loads it dataloader.py dataloader.py. An implementer must match the orientation the library writes.
  • Graph augmentation. For BACH the paper uses rotation, mirroring and colour changes before graph building, and no node or edge dropping §VI.D. The BACH test set is 100 images, so small accuracy gaps are not significant §VI.D.
  • Expensive preprocessing. Graph building runs a nucleus detector, a CNN on every nucleus patch and every tissue-region patch, and superpixel merging for each ROI. The training script can load all graphs into memory with --in_ram train.py.
  • Code bugs to fix when running. yaml.load(f) is called without a loader in train.py and inference.py train.py inference.py; PyYAML 6 and later require one. The README gives config paths under ../data/config/, but the files are in core/config/ README.md. The model is selected by substring match on the config file name train.py.
  • No seeds. The code sets no random seed train.py, so the three runs in the paper cannot be repeated exactly without adding one.

Lineage

  • Builds on: PNA layers [corso20] (the paper's own citation; no card in this wiki), GraphNorm [dwivedi20] and jumping knowledge with an LSTM [xu18]. The paper also cites its own earlier conference version [pati20] and a cell-graph method with CGC-Net [zhou19].
  • Compared against: CNN baselines [sirinukunwattana18] [mercan19], CGC-Net [zhou19], Patch-GNN [aygunes20], and independent pathologists [elmore15].
  • Superseded by: not known from the content in this wiki.

Source and license

Adapted from “Hierarchical Graph Representations in Digital Pathology” by Pushpak Pati, Guillaume Jaume, Antonio Foncubierta, Florinda Feroce, Anna Maria Anniciello, Giosuè Scognamiglio, Nadia Brancati, Maryse Fiche, Estelle Dubruc, Daniel Riccio, Maurizio Di Bonito, Giuseppe De Pietro, Gerardo Botti, Jean-Philippe Thiran, Maria Frucci, Orcun Goksel, Maria Gabrani (arXiv:2102.11057), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/histocartography/hact-net at commit 4b66b41a2f00 (MIT). Any quoted code is shown under that license.