Road Damage Detection Using Deep Neural Networks with Images Captured Through a Smartphone

In plain words

Cities have to find cracks and potholes before they become dangerous, and today that mostly means an expert driving around and looking, or an expensive survey vehicle. Small towns often have neither the staff nor the budget. Earlier research on spotting road damage in photos used private image sets and usually only answered "damaged or not", which does not tell a road crew what kind of repair to plan.

This paper builds a public dataset for the job. The authors mounted an ordinary smartphone on a car's dashboard, drove through seven Japanese municipalities taking one photo per second, and had every damaged spot boxed and labelled with one of eight types from the Japanese road maintenance guidebook: lengthwise cracks, crosswise cracks, "alligator" cracking that looks like crocodile skin, potholes and ruts, and faded lane or crosswalk paint.

They then trained two off-the-shelf lightweight detectors (SSD with a MobileNet or Inception backbone) on it, so that later work has numbers to beat. Common, visually distinct damage such as faded crosswalks is found fairly reliably, while rare classes such as potholes are mostly missed. The MobileNet model runs on the phone itself in about a second and a half per photo, which is fast enough to keep up with a car taking one photo a second.

TL;DR

A dataset paper with baselines. It releases 9,053 images (600×600, from an LG Nexus 5X on a car dashboard) with 15,435 bounding boxes in eight Japan-specific damage classes (D00, D01, D10, D11, D20, D40, D43, D44), in PASCAL VOC format, plus trained SSD Inception V2 and SSD MobileNet models. The models use 300×300 input and a random 80/20 split (7,240 train / 1,813 eval), and are evaluated at IoU 0.5. Results are per-class recall, precision and an undefined "accuracy", not mAP. SSD MobileNet takes 30.6 ms per image on a GRID K520 and 1.5 s on the phone. Use it as the origin of the RDD series of road-damage benchmarks; for current work, the README points to the larger multi-country RDD2020/RDD2022 releases.

Problem

Road inspection relies on visual checks by experienced staff or on costly mobile mapping and laser-scanning systems, and many municipalities lack the experts or money for either §1 §2.1. Image-based research before this paper had three gaps, according to the authors §1:

1. no shared dataset, so each method was evaluated on its own images;
2. no end-to-end deep object detector applied to road damage (prior deep work classified patches or images, e.g. 99×99 patches or 256×256 crops with a sliding window) §2.1;
3. damage was classified into at most a few types (longitudinal / lateral, sometimes alligator), while Japanese road managers use eight categories that map to different repairs §1 §2.1.

They also argue that top-down road imagery used by many prior studies is hard to reproduce, because mounting a camera outside the car body is illegal in many countries; a dashboard phone avoids this §2.2.

Method

The contribution is a data-collection and annotation pipeline plus a baseline training setup with standard detectors. Nothing new is proposed on the model side.

Data collection §3.1 fig. 1

Algorithm: Road Damage Dataset 2018 collection  [§3.1]-[§3.5]
Input: 7 municipalities (Ichihara, Chiba, Sumida ward, Nagakute, Adachi,
       Muroran, Numazu), cooperating road administrators
1. Mount an LG Nexus 5X on the dashboard of an ordinary passenger car  [§3.1] [fig. 1]
2. A custom app captures a 600x600 image and the GPS position once per second;
   at ~40 km/h (~10 m/s) this is meant to cover the road with neither gaps
   nor overlap. The camera sees the road surface about 10 m ahead      [§3.1] [fig. 1]
3. Drive ~1,500 km in total, over 40 hours (abstract) -> 163,664 images   [§3.1]
4. Keep the images that show damage: 9,053                               [§3.4] [fig. 2]
5. Annotate manually: draw a bounding box, then attach a class label;
   class assignment used a professional road administrator's expertise  [§3.3] [fig. 3] [§1]
6. Blur faces and licence plates that are clearly visible, found by visual
   inspection                                                            [§3.5]
7. Store in PASCAL VOC layout (JPEGImages / Annotations / ImageSets), one
   folder per municipality                                               [§3.3] [code: README.md#L317-L333]
Output: 9,053 images, 15,435 boxes

How step 4 selected images from the 163,664 (who screened them, by what rule) is not stated.

Damage classes tab. 1, from the Road Maintenance and Repair Guidebook 2013 (Japan):

The table note says rutting, bumps, potholes and separation are separate damage types in practice; they are merged into D40 because they could not be told apart in the images tab. 1.

Watch out: the table labels D43 "white line blur" and D44 "cross walk blur", but §5 calls D43 the "blur of the pedestrian crossing" §5. The table is the class definition; the §5 sentence appears to be a slip.

Baseline detectors §4. The authors pick SSD with Inception V2 and SSD with MobileNet, because Huang et al. (speed/accuracy trade-offs, on COCO) found these to have low CPU load and memory use for their accuracy, and the goal is to run on a smartphone §4. SSD itself is described as a single-stage detector predicting classes and anchor offsets from multi-scale feature maps, with no second per-proposal stage §2.3.4.

Algorithm: baseline training and evaluation  [§4] [§5]
1. Randomly split the 9,053 images 8:2 -> 7,240 train, 1,813 eval        [§4.2]
2. Resize 600x600 -> 300x300                                             [§4.1.1] [§4.1.2]
3. Train SSD Inception V2 and SSD MobileNet following the original SSD
   paper's methodology, with the learning rates in Hyperparameters       [§4.1]
4. Evaluate per class with IoU threshold 0.5; report recall, precision,
   "accuracy"                                                           [§5] [tab. 2]
5. Time inference on the training machine (GPU) and on the Nexus 5X      [§5] [tab. 3]

Training used the TensorFlow Object Detection API's tooling, judging by the shipped utilities and the frozen-graph inference in the tutorial (code only; the paper does not name the framework) RoadDamageDatasetTutorial.ipynb.

Smartphone app §5 fig. 5: runs SSD MobileNet on the phone; after "START DETECTION" is pressed, it sends the image of the damaged part and its position to an external server only when damage is detected.

Hyperparameters

Results

Author-reported. Setup: 1,813 evaluation images from the random 80/20 split, 300×300 input, IoU 0.5 §4.2 §5.

Dataset statistics. 9,053 images, 15,435 boxes, 7 municipalities §3.4. The paper shows the per-municipality class counts only as a bar chart fig. 4; the tutorial notebook prints them from the released annotations RoadDamageDatasetTutorial.ipynb RoadDamageDatasetTutorial.ipynb:

The class totals sum to 15,435, matching the paper. The paper credits the many D20 boxes in Muroran (1,192) to snow: it is a snowy area where alligator cracks form as snow thaws fig. 4. D40 is rare because dangerous damage gets repaired quickly §3.4.

Per-class detection results tab. 2. SIx = SSD Inception V2, SMx = SSD MobileNet; R = recall, P = precision, A = "accuracy":

The authors' reading §5: D01 and D44 are found with relatively high recall and precision; D11 and D40 have low recall, which they attribute to few training examples. D43 does well despite few examples because it fills a large part of the image and has a clear striped pattern. They conclude that SSD MobileNet is better overall.

Inference speed, 300×300 input tab. 3:

Inception V2 is about twice as slow as MobileNet, in line with Huang et al. §5. With one photo per second while driving, the authors call 1.5 s per image real-time §5.

Paper vs code

Code checked: sekilab/RoadDamageDetector at code_commit. The repository holds a tutorial notebook, a label map and copies of TensorFlow Object Detection API utilities and protos. It has no training script or pipeline config, so none of the training hyperparameters can be checked against code. The trained models and dataset are downloads, not in the repo README.md.

• Box count. The paper gives 15,435 boxes §3.4. The notebook counts 15,457 object elements in the released XML RoadDamageDatasetTutorial.ipynb, while its per-class counts over the eight classes sum to 15,435 RoadDamageDatasetTutorial.ipynb. So 22 annotations carry a name outside the eight classes. Their names are not printed.
• Score threshold. The paper does not state the detection score cut-off behind the recall and precision values §5. The only threshold in the code is 0.3, used for drawing boxes in the tutorial RoadDamageDatasetTutorial.ipynb.
• Split size matches. The notebook reads ImageSets/Main/val.txt from each municipality and gets 1,813 images, as in §4.2 RoadDamageDatasetTutorial.ipynb RoadDamageDatasetTutorial.ipynb.
• Dataset license differs from the paper's. The paper is CC BY 4.0, but the README releases the images under CC BY-SA 4.0 README.md. Anything derived from the images inherits share-alike terms.

Pitfalls

• "Accuracy" is undefined. Table 2 reports a per-class accuracy next to recall and precision, but the paper never says how it is computed tab. 2. It stays high where recall collapses (D40: recall 0.02–0.03, accuracy 0.95), so it almost certainly counts true negatives and mostly reflects class rarity. Do not compare it with mAP or use it to rank models.
• No mAP and no score threshold. Recall and precision depend on a score threshold that is not stated §5. Rebuilt numbers will differ unless you sweep thresholds; report mAP@0.5 per class for anything comparable.
• Headline claim doesn't match the table. The conclusion says the best class reaches recall and precision above 75% with 1.5 s smartphone inference §6. The only class above 0.75 on both is D43 with SSD Inception V2 (0.81 / 0.77) tab. 2; the phone timing is for SSD MobileNet, whose best classes (D43 0.71 / 0.85, D44 0.85 / 0.66) do not clear 0.75 on both. "MobileNet is better overall" §5 is not uniform either: Inception wins recall on D43 and precision on D01, D20 and D44.
• Ambiguous row labels. The caption expands SMR/SMP/SMA as "SSD Recall", "SSD Precision", "SSD Accuracy" without the word MobileNet tab. 2; from context they are the SSD MobileNet rows.
• Random image-level split. Train and eval images are drawn from the same drives and municipalities §4.2. Frames are one second apart and the paper only claims "no duplication" at ~40 km/h §3.1, so slower driving can put the same damage into both splits. The numbers measure in-distribution performance, not transfer to new cities. For a fair test, split by municipality or route.
• Severe class imbalance. D40 has 409 boxes and D11 636, against 3,789 for D01 RoadDamageDatasetTutorial.ipynb; the paper blames the low recall on D11 and D40 on this §5. Distribution also shifts by city (Muroran's alligator cracks) fig. 4. Use class-balanced sampling or loss weighting, and report per-city numbers if generalisation matters.
• Positive-only images. Only images containing damage were released (9,053 of 163,664 captured) §3.4 fig. 2. False-positive behaviour on undamaged road is never measured. In a real deployment on a phone, most frames show no damage.
• Merged and Japan-specific classes. D40 lumps together rutting, bumps, potholes and separation tab. 1, so it cannot drive a pothole-specific repair. D01/D11 (construction-joint cracks) and D43/D44 (paint wear) are categories from the Japanese guidebook tab. 1; later RDD releases keep only D00, D10, D20, D40 README.md, so labels don't map one to one between versions.
• Unstated training details. Optimizer, batch size, iteration count, anchors, augmentation and pretraining are not given; the paper only says it "followed" the SSD paper §4.1. The MobileNet run states a truncated-normal init with std 0.03 §4.1.2, but not which layers it applies to or whether ImageNet or COCO weights were loaded first. Expect to retune.
• Resolution loss. 600×600 images are halved to 300×300 §4.1.1. Thin linear cracks (D00–D11) lose detail, which plausibly contributes to their low recall; this is not analysed in the paper.
• Text inconsistencies. §2.4 says "seven" base networks, then "six" feature extractors, and evaluates only two §2.4 §4. §5 calls D43 the pedestrian-crossing blur, while Table 1 defines it as white-line blur §5 tab. 1. Trust the class table.
• Phone timing is old hardware. The 1.5 s is a Nexus 5X (MSM8992, 2 GB RAM) with a TF 1.4 frozen graph tab. 3; it says little about modern NPUs, and the GPU timings are on a GRID K520 §5.
• Privacy blurring was manual. Faces and plates were blurred only where clearly visible on visual inspection §3.5; downstream users may still find identifiable content.

Lineage

• Builds on: SSD (Liu et al., arXiv:1512.02325) with MobileNet (Howard et al., arXiv:1704.04861) and Inception V2 (Ioffe & Szegedy, arXiv:1502.03167) backbones; detector choice follows Huang et al., "Speed/accuracy trade-offs for modern convolutional object detectors" (arXiv:1611.10012) §4.
• Compared against: no other method is evaluated on the dataset; earlier road-damage work (CrackNet, Zhang et al. 2016 patch classifier, Maeda et al. 2016, Zalama et al., Akarsu et al.) is discussed only qualitatively §2.1.
• Followed by (from the authors' repository): Road Damage Dataset 2019 (GAN-augmented), RDD2020 (Japan, India, Czech Republic), RDD2022 (six countries), and the IEEE Big Data Cup challenges GRDDC 2020, CRDDC 2022 and ORDDC 2024 README.md.