A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector

In plain words

Reading a car's license plate from a camera sounds like one job, but it is really a chain of smaller ones: find the vehicle, find the plate on it, cut out each character, and decide which letter or digit each one is. If any link fails, the whole read fails. This paper builds every link out of the same kind of fast object detector (YOLO), each one trained for its own small job.

Two tricks do most of the work. First, letters and digits get separate recognizers: Brazilian plates always have three letters followed by four digits, so the position of a character already tells you which kind it is, and each recognizer only has to choose among its own classes. Second, rare letters get extra training examples for free, by colour-inverting plates (so dark text on light looks like light text on dark) and by mirroring characters that look the same when flipped, such as H, O or X.

Because the input is video, the system reads the plate in every frame of a vehicle and then takes a vote per character position. That vote fixes many single-frame mistakes. The authors also release a harder dataset, UFPR-ALPR, recorded from a moving car with cars and motorcycles, where both the commercial systems they tested and their own system score noticeably lower than on the older, static-camera benchmark.

TL;DR

A cascade of four Darknet/YOLO detectors (vehicle → plate → characters → letter/digit classifier) for Brazilian license plates, plus temporal majority voting over all frames of a vehicle §4. Character recognition is split into a 26-class letter network and a 10-class digit network, with letters/digits assigned by position after sorting §4.3. Augmentation by negative plate images and flip-invariant characters is credited for most of the letter-recognition gain §5.1. Author-reported recognition rate is 93.53% on SSIG (47 FPS on a Titan Xp) and 78.33% on the new UFPR-ALPR dataset (35 FPS) tab. 4 tab. 6. Useful if you need a simple, fast, layout-specific ALPR baseline; the layout assumption makes it unsuitable as-is for countries with mixed plate formats §4.3.

Problem

ALPR is usually split into plate detection, character segmentation and character recognition, and an error early in the chain sinks the later stages §1. Existing systems tend to rely on constraints such as fixed cameras, fixed viewing angles, simple backgrounds, good lighting or a single vehicle type, and often miss plates on motorcycles, buses or trucks §1. The largest public Brazilian dataset at the time (SSIG) has fewer than 800 training examples, a static camera, similar backgrounds and no motorcycles §1. Deep detectors need more and more varied annotated data than that §1.

Method

Each stage has its own CNN so it can be tuned separately; the networks are Fast-YOLO, YOLOv2 and CR-NET (a Fast-YOLO-inspired network from Montazzolli and Jung) §4. The pipeline is described in words in fig. 5.

1. Vehicle detection. A YOLO detector (Fast-YOLO on SSIG, YOLOv2 on UFPR-ALPR) runs on the full frame §4.1 §5.1.1 §5.2.1. Classes: one ("car") on SSIG; two (car, motorcycle) on UFPR-ALPR, which worked better than a single "vehicle" class §4.1. The confidence threshold is tuned on validation so that all vehicles are found with the fewest false positives; if no vehicle is found, the result is a negative recognition §4.1.
2. Plate detection. A second detector (Fast-YOLO in both datasets) runs on the vehicle crop, enlarged by a margin §4.1 §5.2.2. The margin is the one needed on validation for every plate to lie inside its predicted vehicle box, then doubled for test and training when that is still reasonable §4.1 §5.1.2 §5.2.2. Threshold 0, keep only the highest-confidence box, since a vehicle has exactly one plate §4.1.
3. Character segmentation. The plate crop (with a margin chosen the same way) is stretched horizontally to an aspect ratio of 2.75:1 and fed to CR-NET with a single "character" class §4.2 tab. 2. The network input is 240×80, chosen for the 3:1 Brazilian car plate; motorcycle plates are close to 1.17:1, hence the stretch §4.2. If more than 7 boxes come back: overlapping boxes are merged into one, otherwise the lowest-confidence boxes are dropped. Overlap means IoU ≥ 0.25 for cars and IoU ≥ 0.75 for motorcycles, whose characters can be strongly tilted §4.2.
4. Character recognition. Each segmented character is cropped with a small padding (1–3 px evaluated) §4.3. Characters are sorted horizontally (cars) or vertically (motorcycles); the first three go to the letter network, the last four to the digit network §4.3. Both are YOLO detectors on the single character crop with threshold 0, keeping the top detection, so every segmented character always gets a class §4.3.
5. Temporal redundancy. For all frames of one vehicle, the final plate is the per-position majority vote over the per-frame predictions §4.4.

Output-layer width of each YOLO head, with A = 5 anchors, four box coordinates, one objectness score and C classes §4.1:

filters = (C + 5) \times A eq. 1

This gives 30 filters for one class (vehicle on SSIG, plate, character segmentation), 35 for two vehicle classes, 75 for digits and 155 for letters §4.1 §4.3.

Architectures. Fast-YOLO for vehicle and plate detection tab. 1:

The same output-layer change is applied to YOLOv2, whose table is not given §4.1.

Character segmentation CNN (CR-NET with 30 output filters) tab. 2:

The digit network is this CNN with its first four layers removed, input 42×26 and output grid 21×13; the letter network keeps the whole CNN, input 270×80 and output grid 33×10 §4.3.

Augmentation. For segmentation and recognition, every plate (or character) also appears as its negative image, simulating the other plate colour schemes §4.2 §4.3. For recognition, characters that are invariant to a flip are flipped to create new samples, and 0/1 also serve as samples of O/I tab. 3:

"6(9)" means a 6 flipped both ways becomes a sample of 9, and vice versa tab. 3.

# Per-vehicle plate reading with temporal majority vote  [§4]
# Input:  frames of one vehicle track; trained detectors
# Output: 7-character plate string, or None

def read_plate_frame(frame):
    # Vehicle detection, threshold tuned on validation (0.125)  [§4.1] [§5]
    vehicles = vehicle_detector(frame, threshold=0.125)
    if vehicles is empty:
        # No vehicle found counts as a negative recognition  [§4.1]
        return None
    vehicle = vehicles[0]  # one annotated vehicle per frame in both datasets
    # Crop vehicle with dataset-specific margin (0% SSIG, 10% UFPR)  [§5]
    vehicle_crop = crop(frame, expand(vehicle.box, vehicle_margin))
    # Plate detection at threshold 0, keep top-confidence box  [§4.1]
    plates = plate_detector(vehicle_crop, threshold=0.0)
    plate = argmax_confidence(plates)
    # Crop plate with margin (10% in both datasets at test time)  [§5]
    plate_crop = crop(vehicle_crop, expand(plate.box, plate_margin))
    # Stretch horizontally to 2.75:1 before segmentation  [§4.2]
    plate_crop = resize_to_aspect(plate_crop, width_over_height=2.75)
    # Segment characters; threshold 0.1 used on SSIG test  [§5.1.3]
    chars = char_segmenter(plate_crop, threshold=char_threshold)
    # Reduce to 7 boxes by merging overlaps or dropping weakest  [§4.2]
    iou_limit = 0.75 if vehicle.cls == "motorcycle" else 0.25
    chars = reduce_to_seven(chars, iou_limit)
    # Sort by x for cars, by y for motorcycles  [§4.3]
    if vehicle.cls == "motorcycle":
        chars = sort_by(chars, key=center_y)
    else:
        chars = sort_by(chars, key=center_x)
    result = []
    for position, box in enumerate(chars):
        # First three are letters, last four digits  [§4.3]
        is_letter = position < 3
        pad = letter_padding if is_letter else digit_padding
        char_crop = crop(plate_crop, expand_pixels(box, pad))
        net = letter_net if is_letter else digit_net
        # Threshold 0, top detection, so a class is always produced  [§4.3]
        result.append(argmax_confidence(net(char_crop, threshold=0.0)).cls)
    return result

def read_plate_track(frames):
    per_frame = [read_plate_frame(f) for f in frames]
    per_frame = [p for p in per_frame if p is not None]
    # Most frequent character at each position across frames  [§4.4]
    return [
        most_common([p[i] for p in per_frame])
        for i in range(7)
    ]

reduce_to_seven follows §4.2: if any pair overlaps above iou_limit, the pair is replaced by the union of the boxes; otherwise the lowest-confidence boxes are discarded until 7 remain. The order in which merges and discards are applied is not stated.

Hyperparameters

Results

Author-reported. Hardware: one NVIDIA Titan Xp (12 GB), Darknet §5. "Recognition rate" counts a frame as correct only if all 7 characters are right; with redundancy, each frame of a vehicle takes the voted plate, so the percentage is frame-weighted §5.1.4. Commercial systems were queried through their cloud APIs in January 2018, untuned for these datasets; Sighthound output was post-processed to the Brazilian format (e.g. swapping 0 and O) §5.

SSIG dataset (2,000 frames, 101 vehicles; 40/20/40 train/val/test) §5.1 tab. 4:

Per-stage results and timing on SSIG tab. 5:

Without redundancy, all three letters were right in 86.32% of frames and all four digits in 98.63%; with redundancy, digits were right for every vehicle and letters in 93.53% §5.1.4. Vehicle detection on SSIG test had 7 false positives (precision above 99%) §5.1.1; segmentation recall was 5,614/5,628 §5.1.3.

UFPR-ALPR dataset (4,500 frames, 150 vehicles; 40/20/40 split; test = 60 vehicles) §3 tab. 6:

Per-stage results and timing on UFPR-ALPR tab. 7:

Fast-YOLO for vehicle detection on UFPR-ALPR reached at best 97.33% recall, which is why YOLOv2 was used there §5.2.1. Plate detection missed one vehicle entirely (all 30 frames) because a false positive outscored the real plate, giving 1,770/1,800 §5.2.2 fig. 9. Segmentation recall is 97.59% on detected plates and 95.97% on the whole test set §5.2.3. Many commercial-system errors were on motorcycles §5.2.4.

Ablations. No ablation table is given. The authors attribute the main gain to splitting letters and digits into two tuned networks, and say augmentation was essential for letters because some classes (e.g. C, V) have fewer than 20 training examples §5.1.4. They credit negative images for handling red plates despite few examples §5.1.4.

Paper vs code

No official code. The paper says the architectures and weights are publicly available for academic purposes but gives no repository link in the source §1. Note that "academic purposes" would not cover commercial reuse of those weights.

Pitfalls

• Layout-specific. The letter/digit split relies on the Brazilian format of three letters then four digits, with sorting by x (cars) or y (motorcycles) §4.3. The authors note it does not fit countries with several plate layouts §4.3. Motorcycle plates have letters on one row and digits on another fig. 2; sorting purely by y within a row is how the paper states it, so a tilted two-row plate could still interleave rows. How ties or tilt are handled beyond "even when tilted" is not stated §4.3.
• Frame-weighted "vehicle" accuracy. The "All correct (vehicles)" column is computed over frames, so 37/40 vehicles is reported as 93.53%, not 92.5% §5.1.4 tab. 4. Compare with care against systems that report per-vehicle accuracy.
• Thresholds and margins were tuned per dataset on validation, and some choices differ between datasets (vehicle margin, padding, flip augmentation for digits, vehicle detector model) §5. The character segmentation threshold for UFPR-ALPR is not stated.
• Threshold 0 everywhere downstream. Plate detection and recognition keep the top box at threshold 0, so the pipeline always emits a plate if a vehicle is found; a confident false positive beats a weak true plate, which caused the only plate miss on UFPR-ALPR §4.1 §5.2.2. There is no rejection option for "no readable plate".
• Plate stretch ratio. Stretching all plates to 2.75:1 before resizing to 240×80 (3:1) means car plates are first squeezed and then stretched slightly; the exact resize order and interpolation are not stated §4.2.
• Reducing to 7 characters is under-specified: the order of merges vs. discards, and what happens with fewer than 7 detections, is not stated §4.2. Fewer than 7 boxes would break the "first three letters, last four digits" assignment.
• Timing excludes some work. Times assume weights are loaded and are averages per stage; recognition is run once per character (×7) rather than batched §5.1 tab. 5. Cropping and resizing overhead is not itemised. The text says the per-frame time is "21.31 seconds", which is a typo for milliseconds given tab. 5 §5.1.
• Learning-rate notation. The paper writes the rates as "1^-3, 1^-4, 1^-5"; read as 1e-3, 1e-4, 1e-5 §5. With 80k iterations and the last step at 35k, more than half of training runs at 1e-5. Batch size and optimizer are not stated (Darknet defaults would apply, but the paper does not say which).
• Letter imbalance. Paraná plates range AAA-0001 to BEZ-9999, so A and B dominate the first letter and many letters are rare §3 fig. 4. Expect letters, not digits, to limit accuracy §5.1.4.
• Dataset privacy. UFPR-ALPR is available subject to privacy restrictions, and SSIG's terms prevent showing qualitative results §5.1 p. 1.
• Commercial comparison date. OpenALPR and Sighthound numbers are from their cloud APIs in January 2018 and may not reflect later versions §5.

Lineage

• Builds on: YOLOv2 (arXiv:1612.08242) and Fast-YOLO (arXiv:1506.02640) for detection §1 §4; CR-NET by Montazzolli and Jung for character segmentation and recognition, here split into a segmenter plus two recognizers instead of one 35-class network §4.2; separate letter/digit classification and temporal voting follow Gonçalves et al. §4.2 §4.4.
• Compared against: Montazzolli and Jung, Gonçalves et al., Sighthound and OpenALPR (commercial) tab. 4 tab. 6.
• Introduces: the UFPR-ALPR dataset (4,500 frames, 150 vehicles, three cameras, moving camera and vehicles, cars and motorcycles) §3.