Paper · 2021-02-22 · cs.CV
Hierarchical Graph Representations in Digital Pathology
Builds a hierarchical graph of a breast H&E region: nuclei form a cell-graph, tissue regions a tissue-graph, and each nucleus is linked to its region. HACT-Net, a PNA graph network, classifies it. Author-reported: 61.53% weighted F1 on 7-class BRACS, against 56.62% for the tissue-graph alone and 52.83% for the best multi-scale CNN.
Graph neural networkPrincipal neighbourhood aggregationdigital-pathologyhistologybreast-cancergraph-neural-networkhierarchical-graphprincipal-neighbourhood-aggregation
arXivPDFCodeMarkdown11 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A pathologist reads a tissue sample at two scales at once: single cells, and the larger regions they form, such as gland lining, stroma or necrosis. Most computer models cut the image into fixed squares and look at one square at a time, which loses either the detail of cells or the wider context.
This paper turns one tissue image into a map with two layers. One layer has a node for each cell and links nearby cells. The other has a node for each tissue region and links neighbouring regions. Each cell is also attached to the region it sits in. A neural network reads the cell layer first, passes its summary up to the region layer, and then predicts the tissue type. The authors also release a breast cancer dataset with seven subtypes, labelled by pathologists, and compare the model with pathologists on part of it.
TL;DR
- Input: one breast H&E tumour region of interest. Output: one of seven subtypes.
- Method: a hierarchical graph (cell-graph, tissue-graph, and the links between them), processed by PNA graph layers in two stages.
- Author-reported: 61.53% weighted F1 on the 7-class BRACS task, with the best of the compared baselines near 57%.
- Code exists under MIT. The paper's stated settings and the released code disagree in several places (see Paper vs code).
Problem
Patch-based CNNs trade field of view against resolution: large patches see context but blur cells, and small patches see cells but not the surrounding tissue §I. Cell-graphs capture cell morphology and arrangement but not tissue regions, and tissue-graphs do the reverse §I. The paper argues that a single entity type cannot describe tissue composition at several scales, so it proposes a graph with two entity types and links between them §I §IV.B.3.
The method assumes that nuclei can be detected reliably, that tissue regions can be segmented into homogeneous superpixels, and that a whole tissue region of interest (ROI) is available as one image §IV.B.
Method
Stages, in order §IV:
- Pre-processing. Stain normalisation of the ROI to reduce appearance variation between slides §IV.A.
- Cell-graph (CG). Detect nuclei and use their centroids as nodes. Each node carries a CNN feature of a patch around its centroid, plus its normalised centroid. Edges join each node to its nearest neighbours if they are close enough, using the rule in eq. 2 §IV.B.1.
- Tissue-graph (TG). Over-segment the tissue into superpixels, merge neighbours with similar colour statistics into tissue regions, then build one node per region with a mean CNN feature over its superpixels and a normalised centroid. Adjacent regions are linked (region adjacency graph) §IV.B.2.
- HACT graph. The CG, the TG and a binary assignment matrix that connects each nucleus to the region containing it. A nucleus on a border goes to the region it overlaps most eq. 3 §IV.B.3.
- HACT-Net. Two stages of PNA message passing. The first runs on the CG, with an LSTM that combines the outputs of all layers ("jumping knowledge"). Its per-cell outputs are summed into each tissue node's initial state. The second stage runs on the TG in the same way. A readout sums the tissue-node representations, and a two-layer MLP with softmax gives the class §IV.C.1 eq. 4 eq. 5 eq. 6 eq. 7 fig. 4.
The CG and TG layers use the same PNA update eq. 4. PNA aggregates neighbour messages with four statistics (mean, standard deviation, max, min) and scales each aggregate by a degree term, with three scalers: identity, amplification and attenuation eq. 5 fig. 1. Graph normalisation and batch normalisation follow each PNA layer §IV.C.1.
Pseudocode for graph construction (one ROI):
# HACT graph construction [§IV.A] [§IV.B]
# Input: roi (H, W, 3) H&E image
# Output: cell_graph, tissue_graph, assignment (nuclei x regions)
def build_hact(roi):
# 1. Normalise the stain of the ROI [§IV.A]
roi = normalise_stain(roi)
# 2. Detect nuclei and keep their centroids [§IV.B.1]
nuclei_centroids = detect_nuclei(roi)
# 3. CNN feature from a patch around each centroid [§VI.B]
cell_features = patch_cnn_features(roi, nuclei_centroids, patch=72)
# 4. Connect each nucleus to its nearest neighbours [eq. 2]
# and drop edges longer than a threshold d_min [§IV.B.1]
cell_graph = knn_graph(nuclei_centroids, cell_features, d_min)
# 5. Over-segment into superpixels, then merge similar neighbours
# into tissue regions [§IV.B.2]
regions = merge_similar(oversegment(roi))
# 6. Mean CNN feature of the superpixels in each region [§IV.B.2]
region_features = region_mean_cnn(roi, regions, patch=144)
# 7. Link adjacent regions (region adjacency graph) [§IV.B.2]
tissue_graph = region_adjacency_graph(regions, region_features)
# 8. Assign each nucleus to one region [§IV.B.3] [eq. 3]
assignment = assign_to_region(nuclei_centroids, regions)
return cell_graph, tissue_graph, assignmentHACT-Net forward pass (training adds softmax cross-entropy on the output):
# HACT-Net forward pass [§IV.C.1]
# Input: cell_graph, tissue_graph, assignment (nuclei x regions)
# Output: class logits (7)
def hact_net(cell_graph, tissue_graph, assignment):
# 1. Run T_cg PNA layers on the cell-graph [eq. 4] [eq. 5]
cell_states = run_pna_layers(cell_graph, cell_layers)
# 2. LSTM over the outputs of all cell layers [eq. 6]
cell_embeddings = lstm_over_layers(cell_states)
# 3. Each tissue node starts from its own features plus the sum
# of the cell embeddings assigned to it [eq. 7]
tissue_start = concat(tissue_graph.node_features,
assignment_sum(assignment, cell_embeddings))
# 4. Run T_tg PNA layers on the tissue-graph [§IV.C.1]
tissue_states = run_pna_layers(tissue_graph, tissue_start, tissue_layers)
# 5. LSTM over tissue layers, then sum over tissue nodes [§IV.C.1]
tissue_embeddings = lstm_over_layers(tissue_states)
roi_embedding = sum_over_nodes(tissue_embeddings)
# 6. Two-layer MLP gives the class logits [§VI.B]
return mlp(roi_embedding)Key equations, as numbered in the paper:
A_{\mathrm{cg} \rightarrow \mathrm{tg}}[i, j] = 1 \text{ if nucleus } i \text{ lies in region } j, \text{ else } 0 eq. 3
\mathcal{S}(D, \alpha) = \frac{\log (D + 1)^{\alpha}}{\delta}, \quad \delta = \frac{1}{|\text{train}|} \sum_{i \in \text{train}} \log (d_i + 1) eq. 5
h^{(T_{\mathrm{cg}}+1)}_{\mathrm{cg}}(v) = \text{LSTM}\big(\{ h^{(t)}_{\mathrm{cg}}(v) : t = 1..T_{\mathrm{cg}} \}\big) eq. 6
h^{(0)}_{\mathrm{tg}}(w) = \text{CONCAT}\Big(H_{\mathrm{tg}}(w),\; \sum_{v \in \mathcal{M}(w)} h^{(T_{\mathrm{cg}}+1)}_{\mathrm{cg}}(v)\Big) eq. 7
Equations 1 (message passing in general) and 4 (PNA update for the cell-graph) are in §III.B and §IV.C.1.
Hyperparameters
Values in the paper come from §VI.B unless marked. "not stated" means the paper
does not give it. Code values are from the official repository at code_commit.
| Name | Paper | Code | Where |
|---|---|---|---|
| Nuclei detector | HoVer-Net, pretrained on MoNuSeg §IV.B.1 | NucleiExtractor() from the external histocartography library | generate_hact_graphs.py |
| Cell patch size | 72×72 px §VI.B | 72 px, resized to 224 for ResNet | generate_hact_graphs.py |
| Tissue patch size | 144×144 px §VI.B | 144 px, resized to 224 | generate_hact_graphs.py |
| CNN backbone | ResNet-34 or ResNet-50, ImageNet §VI.B | ResNet-34 only | generate_hact_graphs.py generate_hact_graphs.py |
| Cell node features | CNN features plus spatial coordinates §IV.B.1 | 512 CNN + 2 coordinates = 514 | generate_hact_graphs.py train.py |
| kNN size k | not stated | 5 | generate_hact_graphs.py |
| Edge cut-off d_min | not stated | 50 px | generate_hact_graphs.py |
| Superpixels | SLIC, low magnification, then merged §IV.B.2 | size 500, compactness 20, blur 1, threshold 0.05, downsampling 4 | generate_hact_graphs.py |
| Stain normalisation | Macenko, then a fast pipeline §IV.A | Vahadane with a target image | generate_hact_graphs.py generate_hact_graphs.py |
| PNA layers per graph | 3, 4 or 5 (options) §VI.B | 3 (CG, TG and HACT configs) | bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml |
| PNA aggregators | mean, std, max, min eq. 5 | mean max min std | bracs_hact_7_classes_pna.yml |
| PNA scalers | identity, amplification, attenuation eq. 5 | identity amplification attenuation | bracs_hact_7_classes_pna.yml |
| Degree normaliser δ | mean log-degree over training graphs eq. 5 | fixed avg_d: 4 | bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml |
| Jumping knowledge | LSTM over layer outputs eq. 6 | readout_op: lstm | bracs_hact_7_classes_pna.yml |
| Readout type | sum over tissue nodes §IV.C.1 | readout_type: mean | bracs_hact_7_classes_pna.yml |
| Dropout in GNN | not stated | 0 | bracs_hact_7_classes_pna.yml |
| Normalisation | GraphNorm then BatchNorm after each PNA layer §IV.C.1 | both on | bracs_hact_7_classes_pna.yml |
| PNA MLP | 2 layers, 64 channels §VI.B | pre-transform 1 + post-transform 1 layers, output dim 64 | bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml |
| Graph embedding size | 128 §VI.B | not in config (library) | not verifiable here |
| Output classifier | 2 layers, 128 channels §VI.B | num_layers: 2, hidden_dim: 128 | bracs_hact_7_classes_pna.yml |
| Optimiser | Adam, learning rate 1e-3 §VI.B | Adam, default lr 1e-2, weight decay 5e-4 | train.py train.py |
| Batch size | 16 §VI.B | default 1; README command uses 8 | train.py README.md |
| Epochs | not stated | default 10; README command uses 60 | train.py README.md |
| Loss | categorical cross-entropy §VI.B | CrossEntropyLoss | train.py |
| Model selection | best validation weighted F1 §VI.B | three checkpoints saved, each tested | train.py train.py |
| Repeats | three runs with random initialisation §VI.B | no seed set in the code | train.py |
| ROI size filter | not stated | keep 50,000 to 50,000,000 pixels | generate_hact_graphs.py generate_hact_graphs.py |
| Class order | seven subtypes §V | N, PB, UDH, ADH, FEA, DCIS, IC → 0..6 | generate_hact_graphs.py |
Results
Author-reported. Setup: BRACS, 4,391 ROIs from 325 breast carcinoma WSIs, seven classes, WSI-level split into 3,163 train, 602 validation and 626 test ROIs §V tab. I. Metrics are weighted F1 (mean ± standard deviation over three runs) unless stated. Baselines are described in §VI.A.
7-class, BRACS tab. V:
| Method | Weighted F1 (%) | Source |
|---|---|---|
| CNN, 10× | 50.85 ± 2.64 | tab. V |
| CNN, multi-scale 10×+20×+40× | 52.83 ± 1.92 | tab. V |
| CGC-Net (cell-graph baseline) | 43.63 ± 0.51 | tab. V |
| Patch-GNN, 10× | 52.10 ± 0.61 | tab. V |
| CG-GNN (cell-graph only) | 55.94 ± 1.01 | tab. V |
| TG-GNN (tissue-graph only) | 56.62 ± 1.35 | tab. V |
| CONCAT-GNN (two separate graphs, concatenated) | 57.01 ± 2.27 | tab. V |
| HACT-Net (proposed) | 61.53 ± 0.87 | tab. V |
4-class, BRACS (Normal; Non-cancerous; Precancerous; Cancerous) tab. VI: HACT-Net 69.04 ± 0.46, best baseline CNN at 10× 64.36 ± 1.37 and TG-GNN 66.24 ± 1.11.
Binary decision tree, aggregated over six tasks tab. VII: HACT-Net 84.15 ± 0.60; CONCAT-GNN 82.61 ± 2.15; CG-GNN 81.50 ± 1.70; single-scale CNN at 10× 78.90 ± 1.38.
Ablations, 7-class weighted F1 tab. II tab. III tab. IV:
- Node features: none 48.70, hand-crafted 52.46, CNN 61.53 for HACT-Net.
- Layer type for HACT-Net: GIN 59.73, PNA 61.53.
- Jumping knowledge for HACT-Net: none 49.16, concatenation 59.78, LSTM 61.53.
Pathologists, 7-class, BRACS test ROIs without the WSI context tab. VIII: three independent pathologists averaged 56.36 ± 0.76 weighted F1 and 57.08 ± 0.64 weighted accuracy. HACT-Net, on the same ROIs, scores 61.53 ± 0.87 weighted F1 and 63.21 ± 0.27 weighted accuracy. Pairwise agreement among the pathologists is between 47.60% and 64.38% tab. IX.
BACH, 4-class accuracy, organiser-evaluated tab. X: HACT-Net 91.00 with one network. The listed ensemble methods score 86.00 to 95.00.
Figures, described only: the 7-class confusion matrix shows the most confusion between Normal and Benign, between UDH and ADH, and between ADH and DCIS fig. 8 §VI.D. The attribution maps highlight tumour epithelium and necrotic regions on the tissue-graph fig. 11 §VI.E.
Paper vs code
Official code: https://github.com/histocartography/hact-net, MIT licence, at
commit 4b66b41a2f00b768029b6aaa1152c746faf2b9c6. The paper does not link it. The
owner, the histocartography organisation, is the group that the authors Pati and
Gabrani work with at IBM Research Zurich, and the repository README cites this
paper. The model classes (HACTModel, NucleiExtractor, DeepFeatureExtractor,
VahadaneStainNormalizer, AssignmnentMatrixBuilder) come from the external
histocartography library, which is not in the repository and is not pinned in
environment.yml, so behaviour inside those classes is not verified here.
Differences:
- Stain normalisation. The paper uses Macenko's method, followed by a faster pipeline §IV.A. The code uses Vahadane normalisation against one target image generate_hact_graphs.py generate_hact_graphs.py.
- Learning rate and batch. The paper uses learning rate 1e-3 and batch size 16 §VI.B. The README command uses 5e-4 and batch 8 README.md. The
train.pydefaults are 1e-2 and batch 1 train.py train.py. - Weight decay. The paper states no weight decay. The optimiser uses 5e-4 train.py.
- Epochs. Not stated in the paper. The README uses 60 README.md; the default is 10 train.py.
- PNA degree normaliser. The paper defines δ as the mean log-degree over the training set eq. 5. The configs fix
avg_d: 4bracs_hact_7_classes_pna.yml bracs_hact_7_classes_pna.yml. - Readout. The paper sums tissue-node representations §IV.C.1, and the CG-GNN baseline uses a sum readout §VI.A. The configs set
readout_type: meanfor all three models bracs_cggnn_7_classes_pna.yml bracs_hact_7_classes_pna.yml. - PNA depth. The paper lists 3, 4 or 5 layers §VI.B. The configs use 3, and the paper does not say which depth produced the table values.
- CNN backbone. The paper allows ResNet-34 or ResNet-50 §VI.B. The code uses only ResNet-34 generate_hact_graphs.py.
- Patch resize. The code resizes each patch to 224 px before the CNN generate_hact_graphs.py. The paper gives no resize step.
- Assignment rule. The paper assigns border nuclei to the region with the largest overlap §IV.B.3. The code passes nucleus centroids and the tissue map to a library builder generate_hact_graphs.py. The overlap rule is not visible in this repository.
- Data split. The paper uses a random WSI-level split §V. The code reads the
train,valandtestfolders of the BRACS release generate_hact_graphs.py and does no splitting. - Accuracy in the README. The README reports 61.7% accuracy and 61.5% weighted F1 for the pretrained HACT-Net checkpoint README.md. The paper's weighted accuracy of 63.21 tab. VIII is a different summary, so the two numbers should not be compared directly.
- Graph embedding size. The paper states 128 §VI.B. The HACT config sets
output_dim: 64for the PNA stages bracs_hact_7_classes_pna.yml. Where the 128 comes from is not visible in this repository.
No discrepancy was found for the classifier head (two layers, 128 channels), the label order, or the cross-entropy loss.
Pitfalls
- Dataset access. BRACS requires a registration and an FTP download of the previous version of the regions-of-interest set README.md. The paper's footnote says the release was "shortly" to follow, so the released split may not match the paper's random WSI-level split §V.
- Silent image filtering. ROIs outside 50,000 to 50,000,000 pixels are skipped, and the code prints only a message for them generate_hact_graphs.py generate_hact_graphs.py. Check the count of graphs produced against Table I.
- Hidden parameters. The paper leaves k, d_min, the superpixel sizes, the dropout, the epoch count and the seed unstated. The code values above fill them, but the paper's numbers may come from different settings.
- Edge threshold is in pixels. The code's 50 px cut-off depends on the resolution at which graphs are built (0.25 µm/pixel for BRACS §V). A different scan resolution changes the graph without any code change.
- Node-count spread. Graph sizes vary widely: cell-graphs average 1,468 ± 1,642 nodes and tissue-graphs 172 ± 217, with max/min ratios up to 256 for cell-graphs tab. I. Use per-graph normalisation, and expect large memory use for the largest ROIs.
- Which tensor is summed. The paper says the graph-level embedding is the sum of tissue-node representations §IV.C.1. It does not say whether this is before or after the LSTM over layers. Check the library before implementing.
- Assignment matrix orientation. The code transposes the stored matrix when it loads it dataloader.py dataloader.py. An implementer must match the orientation the library writes.
- Graph augmentation. For BACH the paper uses rotation, mirroring and colour changes before graph building, and no node or edge dropping §VI.D. The BACH test set is 100 images, so small accuracy gaps are not significant §VI.D.
- Expensive preprocessing. Graph building runs a nucleus detector, a CNN on
every nucleus patch and every tissue-region patch, and superpixel merging for
each ROI. The training script can load all graphs into memory with
--in_ramtrain.py. - Code bugs to fix when running.
yaml.load(f)is called without a loader intrain.pyandinference.pytrain.py inference.py; PyYAML 6 and later require one. The README gives config paths under../data/config/, but the files are incore/config/README.md. The model is selected by substring match on the config file name train.py. - No seeds. The code sets no random seed train.py, so the three runs in the paper cannot be repeated exactly without adding one.
Lineage
- Builds on: PNA layers [corso20] (the paper's own citation; no card in this wiki), GraphNorm [dwivedi20] and jumping knowledge with an LSTM [xu18]. The paper also cites its own earlier conference version [pati20] and a cell-graph method with CGC-Net [zhou19].
- Compared against: CNN baselines [sirinukunwattana18] [mercan19], CGC-Net [zhou19], Patch-GNN [aygunes20], and independent pathologists [elmore15].
- Superseded by: not known from the content in this wiki.
Source and license
Adapted from “Hierarchical Graph Representations in Digital Pathology” by Pushpak Pati, Guillaume Jaume, Antonio Foncubierta, Florinda Feroce, Anna Maria Anniciello, Giosuè Scognamiglio, Nadia Brancati, Maryse Fiche, Estelle Dubruc, Daniel Riccio, Maurizio Di Bonito, Giuseppe De Pietro, Gerardo Botti, Jean-Philippe Thiran, Maria Frucci, Orcun Goksel, Maria Gabrani (arXiv:2102.11057), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/histocartography/hact-net at commit 4b66b41a2f00 (MIT). Any quoted code is shown under that license.