Waste detection in Pomerania: non-profit project for detecting waste in environment

In plain words

Litter comes in many shapes, sizes and settings, and cities sort it by different rules. This project asks whether a computer can find litter in a photo and also say which kind it is, so that recycling can be sorted more efficiently.

It works in two steps. First, a detector draws boxes around anything that looks like litter, without caring what kind it is. Then a second network looks at each boxed piece and decides which of seven sorting categories it belongs to, or whether it is only background. The team built its training data by merging several public waste collections, because no single one is large or consistent enough.

The detector was clearly better than the other object detectors tried. Sorting turned out to be the harder part: the categories are unevenly represented and some labels are poor, so the final classifier is far less accurate than the detector is at finding litter.

TL;DR

• Two-stage pipeline: a one-class litter detector (EfficientDet-D2) followed by an EfficientNet-B2 crop classifier with seven waste categories plus a background class §3.
• Author-reported detection result: 66.4% AP@0.5 on the merged detect-waste+ dataset, the best of the detectors compared tab. 6.
• Author-reported classification result: 73.02% accuracy for the 8-class weighted, per-batch pseudo-label configuration tab. 7; the abstract's "around 75%" comes from other rows.
• Splitting detection from classification was chosen because the seven-class detector scored far lower (16.2% AP@0.5) tab. 6 §3.2.
• The official code is MIT-licensed and differs from the paper in several training settings; read the Paper vs code section before reproducing numbers.

Problem

Litter is found in many environments (indoors, streets, underwater), with varied backgrounds and sorting rules that differ between regions §1 §2.3. Existing public datasets are small, use different annotation types and class sets, and do not provide well-defined sorting categories for industry §2.3. The authors argue that a single model cannot use all of the available annotations at once, so they split localisation from classification §3.

Small litter remains hard to localise, and the authors name stronger detectors and a more balanced dataset as the next steps §4.

Method

Stage 1, localisation. A detector finds litter as one class, "litter", without a category. The authors compared EfficientDet, DETR and Mask R-CNN on the detect-waste dataset and kept EfficientDet-D2 for all later experiments §3.2 tab. 4. The detector is trained on the merged detect-waste+ dataset, which adds cigarette-butt images to the seven merged collections §3.1 fig. 3.

Stage 2, classification. Each detected region is cropped and passed to a classifier with seven waste categories plus a background class, which reduces false positives from the detector §3.3 §4. The seven categories follow the sorting rules of Gdańsk: bio, glass, metals and plastic, non-recyclable, other, paper and unknown §1 §3.3. Two classifier families were compared, EfficientNet-B2 and ResNet-50, and EfficientNet-B2 was kept §3.3.2 tab. 7.

Data for classification. The classification set (classify-waste) is made of cut-outs of litter from TACO, Drinking Waste and Open Litter Map, plus extra bio, glass, other and paper images from TrashNet, Waste Pictures and web searches. Background images come from the Places dataset §3.1 tab. 2. A further set of Open Litter Map images receives pseudo-labels from the detector and the classifier (classify-waste+) §3.1 tab. 3. Metals and plastic dominates and unknown is second, so the set is strongly imbalanced §3.1.

Sampling. Training uses either a random sampler or a weighted sampler that equalises class frequencies §3.3.1 train_effnet.py.

Pseudo-labelling. Unlabelled Open Litter Map crops are labelled with the current classifier and used for further training. The authors tested no pseudo-labelling, updates once per epoch, and updates before every batch §3.3.1 fig. 7. The official code only starts pseudo-labelling at epoch 5 (not stated in the paper) efficientnet.py efficientnet.py.

# Two-stage litter pipeline  [§3] [fig. 2]
# Input:  image (H, W, 3)
# Output: list of (box, category, probability)

def sort_litter(image, detector, classifier):
    # Find litter as one class, with no category  [§3.2]
    boxes, scores = detector(image)          # (N, 4), (N,)
    results = []
    for box, score in zip(boxes, scores):
        # Drop weak detections; the script uses 0.17, the paper gives no value
        # [code: make_predictions.py#L51]
        if score < DETECTOR_THRESHOLD:
            continue
        # Cut the region out and resize it to the classifier input  [§3.3.1]
        crop = resize(image_crop(image, box), 224)    # (224, 224, 3)
        probs = softmax(classifier(crop))              # (8,) incl. background
        category = argmax(probs)
        # The script keeps a box when its top probability is at least 0.5,
        # with no check for background  [code: make_predictions.py#L134]
        if probs[category] >= CLASSIFIER_THRESHOLD:
            results.append((box, category, probs[category]))
    return results

# Classifier training with pseudo-labels  [§3.3.1] [fig. 7]
# Input:  labelled crops, unlabelled crops, number of epochs, mode
# Output: trained classifier

def train_classifier(model, labelled, unlabelled, epochs, mode):
    for epoch in range(epochs):
        # Supervised pass with cross-entropy  [code: classifier/models/efficientnet.py#L45]
        train_one_epoch(model, labelled)
        # The code starts pseudo-labelling at epoch 5  [code: classifier/models/efficientnet.py#L19]
        if epoch < PSEUDO_START_EPOCH:
            continue
        if mode == "per_epoch":
            # Relabel all unlabelled crops, then one optimiser pass over them
            # [code: classifier/models/efficientnet.py#L97-L99]
            # [code: classifier/models/efficientnet.py#L106-L117]
            relabel(model, unlabelled)
            train_one_epoch(model, unlabelled)
        if mode == "per_batch":
            # Relabel each unlabelled batch just before training on it
            # [code: classifier/models/efficientnet.py#L100-L101]
            # [code: classifier/models/efficientnet.py#L155-L166]
            for batch in unlabelled:
                relabel(model, batch)
                train_step(model, batch)
    return model

Input preprocessing (classifier). The crop is enlarged by 60 pixels, randomly cropped to 224 x 224, flipped horizontally and vertically, shifted/scaled/rotated, brightness/contrast jittered, and cut out. Test images are resized to 224 x 224 §3.3.1 train_effnet.py.

Preprocessing (detector). Images are resized to the EfficientDet input size and normalised with the COCO mean and standard deviation §3.2.1. Only blur and brightness augmentation are active in the repository (see Paper vs code).

Hyperparameters

Dataset composition (not a training hyperparameter, but needed to reproduce the data).

Images per category by source collection for classify-waste tab. 2. Rows are categories; "-" means no images from that source.

Open Litter Map images with a pre-assigned category, sorted by count tab. 3:

Results

Author-reported. Setup: detect-waste (one class, litter) for the architecture comparison; EfficientDet-D2 on the listed datasets for the per-dataset study; the classifier on classify-waste with the split described in §3.1. AP is at IoU 0.5 unless stated; the metric is PASCAL-style AP §3.2.1.

Detector architecture comparison, one class, detect-waste tab. 4

EfficientDet-D2 per dataset tab. 5 (mAP@0.50 per dataset; class counts differ, so rows are not directly comparable)

EfficientDet-D2 final results tab. 6

Classifier comparison tab. 7

Per-class results for the EfficientNet-B2, weighted sampler, per-batch configuration tab. 8 fig. 8:

The text states that pseudo-labelling updated per batch "can slightly raise accuracy" and that it gave only a slight gain for classification §3.3.2 §4. The 73% figure is the weighted-sampler result; the 74.6% figure uses a random sampler §3.3.2.

Paper vs code

Searched: the abstract links https://github.com/wimlds-trojmiasto/detect-waste (abstract, footnote). The owner's group is the Women in Machine Learning and Data Science Trójmiasto community, whose project leads are named in the acknowledgements §Acknowledgements. Cloned at commit 0a4367b121e4bfabdfb148b323090573ac2dadc2; LICENSE is MIT. The repository has the detector (EfficientDet, DETR, Mask R-CNN), the classifier, pseudo-labelling scripts and annotation files.

• Detector schedule: the paper gives 20 epochs and LR decay 0.95 §3.2.1; the binary script uses 30 epochs and decay 0.95 train_binary.sh train_binary.sh; the 7-class script uses decay 0.85 and no epoch count (default 300) train.sh train.py.
• Detector augmentation: the paper lists Gaussian blur, brightness, rain, fog, snow, rotation and crop §3.2.1. The active pipeline applies only blur (limit 7, p 0.5) and brightness (limit 0.2, p 0.5); flips, crop, downscale, fog, rain and snow are commented out transforms_albumentation.py. Augmentation is only used when training loader.py.
• Mask R-CNN: the paper gives learning rate 1e-3 and decay 1e-4 §3.2.1; the run script passes only the epoch count, so the default learning rate 0.005 and weight decay 5e-4 apply run.sh train.py.
• DETR: the paper says the loss saturates around 63 epochs §3.2.1; the code default is 300 epochs with an LR drop at 200 main.py, and the fine-tuning notebook note says 300 epochs with a drop at 100 Detect_Waste_finetuning_detr.ipynb.
• Classifier optimiser and decay: the paper names neither; the code uses Adam with a per-epoch multiplicative decay of 0.99 efficientnet.py train_effnet.py.
• Classifier pseudo-labelling: the paper does not give a start epoch; the code starts at epoch 5 and the CLI default is per_epoch efficientnet.py train_effnet.py.
• Classifier shared settings: the paper says all experiments used lr 1e-4, batch 16 and 20 epochs §3.3.1. The ResNet-50 script defaults to 30 epochs and lr 1e-3 train_resnet.py. The CLI default for --num-classes is 7, although the reported best models use 8 train_effnet.py.
• Class order: the classifier training map starts with background = 0 efficientnet.py, but the inference label list starts with bio make_predictions.py. As read, these two orders are off by one.
• Pseudo-label script: pseudolabel.py defines --score-thr (default 0.3) pseudolabel.py but reads args.prob_threshold pseudolabel.py. As read, this raises an error when run as shipped. Not executed.
• Pseudo-label license metadata: the script writes "Attribution-NonCommercial 4.0 International" into the generated annotation file pseudolabel.py.
• Evaluation split: the paper calls the held-out part "validation/testing" §3.1; the training script passes the test folder as the validation set and checkpoints on its loss train_effnet.py train_effnet.py train_effnet.py.

Pitfalls

• Headline numbers differ. The abstract says "up to 70%" average precision for detection, but the best AP@0.5 in any table is 66.4% tab. 6. The abstract's "around 75%" classification accuracy comes from the 74.6% and 74.4% rows tab. 7; the configuration described in the method, 8 classes with weighted sampling and per-batch pseudo-labels, scores 73.02% tab. 7.
• Per-dataset numbers are not comparable. The 91.3%, 97.9% and 99.4% values in tab. 5 are single-dataset scores with different class sets; they should not be read as the pipeline's performance.
• Seven-class detection is not usable. A detector trained directly on the seven categories reached 16.2% AP@0.5, almost four times below the best one-class result of 66.4% tab. 6 §3.2.2. Keep the detector one-class and classify crops separately.
• Accuracy is ambiguous. The paper does not say whether accuracy is plain or class-weighted. The code logs both efficientnet.py. Check which one a comparison uses.
• Split conflict. Section 3.1 says 80% of images go to training §3.1, while the classification results say "almost 25%" of the data are test images §3.3.2. The split code is not in the classifier scripts, so the exact split cannot be reproduced from the repository.
• Checkpoint selection on the test set. The training script reports and selects checkpoints using the test folder train_effnet.py train_effnet.py. Treat the classification numbers as selected on test data, which makes them optimistic (inferred from the code).
• Background handling. The detector-to-classifier script keeps a box whenever the top probability is at least 0.5, including when the top class is background make_predictions.py. The paper says background was added to remove false positives §3.3.1 §4.
• Label order. If the inference label list does not match the training order, every category is shifted by one. Check the order against the dataset folders before deploying.
• Pseudo-label cost. Per-batch and per-epoch pseudo-labelling both run an extra pass over the unlabelled set from epoch 5 onward efficientnet.py. Budget for this, and note the paper found the gain small §4.
• Label noise and manual cleaning. The sources contain annotation errors, and the authors removed some malformed or mislabelled images by hand §3.1. This step is not documented enough to repeat.
• Source licences. The paper's own text is CC BY 4.0. The datasets it draws on (TACO, TrashCan, Open Litter Map and others) have their own terms. This card does not license any dataset, and it does not check the annotation files bundled in the repository. Pseudo-labels produced by the repository script are tagged non-commercial (see Paper vs code).
• Small litter. The authors report weak detection of small objects (AP_S 9.3 for detect-waste+) tab. 6 and name it as the main open problem §4.
• Paper inconsistency. Table 1 gives 205 scene classes for Places, but the text gives 434 tab. 1 §2.3.

Lineage

• Builds on: EfficientDet (detector, scaling), DETR, Mask R-CNN and EfficientNet-B2/B4 and ResNet-50 (classifiers), all used as off-the-shelf components §2.2 §2.1 §3.2 §3.3. Pseudo-labelling follows the classic formulation the authors cite §3.3.
• Compared against: DETR (two backbones), Mask R-CNN, ResNet-50, and per-dataset single-class detectors tab. 4 tab. 5.
• Future work named by the authors: Deformable DETR and EfficientNetV2 for better small-object detection and classification §4.