Paper · 2021-05-30 · cs.RO
Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review
Review of the BRL TacTip, a soft biomimetic optical tactile sensor whose pins move marker tips that a camera tracks. Covers the shear-sensing hypothesis, the 3D-printed TacTip family and robot hands, and deep learning on tactile images for servo control and grasping. Results are summarised from cited works, e.g. 0.1-0.5 mm 3D pose localisation.
Tactile sensingConvolutional neural networkIn-hand manipulationtactile-sensingsoft-roboticsrobot-manipulationreview
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Robot hands can grip objects, but most of them cannot feel very well. This review follows one touch sensor, the TacTip, from its start in 2009 to the point where it is used with modern neural networks. The TacTip copies the inside of a human fingertip: a soft skin sits on top of many small pins, and the tips of the pins carry visible dots. When the skin is pressed, the pins tilt and the dots move sideways. A small camera inside the sensor watches the dots, so the sensor does not need to measure pressure directly.
The author argues that this sideways movement is a good signal for the shape of whatever touches the sensor. Over the years 3D printing made the sensor quick to rebuild in new shapes, and it was fitted to the fingertips of several robot hands. The big change came when neural networks started to read the whole camera image, which allowed robots to do tasks such as sliding along an edge or recognising what a hand is holding.
The review is a summary of work by the author's group and others, so it is a guide to what has been shown and what is still open, not a new experiment.
TL;DR
A review of the BRL TacTip, a soft biomimetic optical tactile sensor that transduces skin indentation into lateral marker shear through stiff pins embedded in a soft gel §IV.A. It argues for shear as the signal that carries shape information §IV.B, tracks a family of 3D-printed sensors and hands §VI.A, and reports a shift from marker-based classifiers and regressors to convolutional neural networks on tactile images from 2019 onward §VII. Treat every numeric result as the cited work's own report, not the review's.
Problem
Robot manipulation lacks a sense of touch that is rich, cheap and easy to combine with learning. Optical tactile sensors give high-resolution contact data, but many designs measure only normal pressure, or they use reflection or marker schemes whose output is hard to interpret §II §IV.A. The review asks how a biomimetic design can make touch useful for dexterous robots §I §VIII. It assumes that soft, camera-based, biomimetic sensors are a route to human-like dexterity, and it does not evaluate alternatives in depth.
Method
The review describes the sensor, its transduction, and the processing that turns camera images into useful signals. Nothing here is a training recipe.
-
Sensor structure. A rubber-like outer epidermis covers an elastomer gel. Stiff nodular pins extend from the skin into the gel, and a marker sits on each pin tip. The markers are imaged through the transparent gel that forms the dermis, and a transparent acrylic cap holds the deeper gel in place §III fig. 1 tab. I. The review maps each part to a feature of glabrous skin: ridges and papillae to the pins, the dermis to the gel, and the Merkel-cell and Meissner-corpuscle receptors to marker displacement and marker velocity §III tab. I.
-
Transduction. Indentation tilts and moves the pins, so the markers shift sideways. The review's central claim is that the sensor responds mainly to shear: normal and shear strain both occur in practice, and they produce different marker patterns, with slip giving a more uniform shear field and indentation giving dipole or multipole patterns §IV.A fig. 2.
-
Shear-sensing hypothesis. Platkiewicz and colleagues propose that shape is better recovered from shear strain than from normal strain, because zero crossings of the shear field mark edges without a gradient step §IV.B. The review gives two simplified models. The first treats the skin as a uniform elastomer with blurred pressure eq. 1. The second treats the TacTip as stiff pins that lever the indentation into shear eq. 2. Both give a shear field that scales with depth and with a gradient:
\gamma(x,z)\approx \frac{z}{E/3}\frac{{\rm d}}{{\rm d}x}p_\epsilon(x,z), \quad p_\epsilon(x,z)=p(x)*\phi_\epsilon(x,z) eq. 1
\gamma(x,z)= z\sin\arctan\left(\frac{{\rm d}}{{\rm d}x}\delta(x)\right) \approx z\frac{{\rm d}}{{\rm d}x}\delta(x) eq. 2
Here \gamma is shear strain at depth z and horizontal position x, E is Young's modulus, p is surface pressure, \phi_\epsilon is a Gaussian blur whose width grows with depth, and \delta is the indentation. Equation 2 holds when the surface slope is small §IV.B.
-
Feature extraction (2015 to about 2019). The markers are tracked in each camera frame. The review uses two signals derived from the tracked markers: displacement from rest, which plays the role of the slowly adapting receptor, and velocity, which plays the role of the rapidly adapting receptor §VI intro §VI.B §III.
Pseudocode# Marker features from the tracked TacTip pins [§VI] [§III] # Input: camera frames of the marker pattern # Output: per-pin displacement and velocity, one row per pin def marker_features(frames, rest_markers): # 1. Track each pin marker through the frames [§VI] markers = track_markers(frames) # (T, N_pins, 2) # 2. Displacement from the resting position [§III] [§VI.B] displacement = markers - rest_markers # (T, N_pins, 2) # 3. Velocity: change of marker position between frames [§III] velocity = diff(markers, axis=0) # (T-1, N_pins, 2) return displacement, velocity -
Learning on marker features (2015 to about 2019). The review lists probabilistic classifiers on pin displacements tab. IV, support vector machines on pin velocities for slip detection tab. IV, Gaussian process and polynomial regressors for sliding and grasp control §VI.B, and online Gaussian-process latent-variable models §VI.B. The review judges that these approaches transfer poorly once the task becomes continuous and moving §VI.B.
-
Deep learning on tactile images (from 2019). Convolutional neural networks take the whole tactile image as input, with cropping, thresholding or concatenation as the only preprocessing. The review says the networks predict quantities such as pose, grasp success or object class directly from the image §VII tab. IV. The reviewed tasks are pose-based servo control, pushing to a goal, braille typing with deep reinforcement learning, object and grasp-success prediction, in-hand grasping and fine contact control §VII fig. 5.
Pseudocode# Whole-image learning path, as described in the review [§VII] [tab. IV] # Input: tactile image, e.g. cropped to (128, 128) or (60, 120) # Output: quantity of interest (pose, grasp success, class) def predict_from_tactile_image(tactile_image, convnet): # 1. Feed the image directly; no marker tracking step [§VII] # Network architecture and layer sizes are not stated. outputs = convnet(tactile_image) # 2. Read out the labelled quantity, e.g. edge pose [§VII] return outputs -
Hardware family. The review tracks the TacTip through 3D-printed versions with different tip sizes, pin counts and cameras tab. II, and through integration into hands: the Tactile Model M2, Model GR2, Model O, Shadow Modular Grasper and Pisa/IIT SoftHand tab. III fig. 4. Multi-material 3D printing enabled rapid prototyping of these probes and hands §VI.A.
Hyperparameters
The review states no training hyperparameters. The table lists the physical design values it does give, plus the input sizes of its learning tasks.
| Name | Value | Where | Notes |
|---|---|---|---|
| tip diameter, 2009 TACTIP | 40 mm | tab. II | 532 pins, webcam at 480p, 30 fps |
| tip diameter, 2016/2018 TacTip | 40 mm | tab. II | 127 pins, rigid body 85 mm |
| tip diameter, 2021 TacTip | 40 mm | tab. II | 330 pins, wide-angle lens, focal length about 10 mm |
| Mini TACTIP tip | 20 mm | tab. II | 276 pins, not integrated with camera |
| magnification of tactile contact lens | about 4x | §V | cited as a design inspiration, not a TacTip value |
| input size, pose and edge tasks | 128 x 128 | tab. IV | ConvNet regressor on tactile image |
| input size, object and grasp tasks | 60 x 120 | tab. IV | ConvNet classifier on tactile image |
| shear model, E and z | not stated | §IV.B | the review gives no values for E, z or the blur width |
| learning rate, optimiser, epochs, training set size | not stated | §VI §VII | the review gives none for any learning method |
Results
Author-reported. The review's Table IV reproduces results from the cited works, so each row is a different study with its own robot, sensor and metric. The review does not give a common baseline. Units in the accuracy column are mixed: millimetres, degrees, percent, microns and newtons.
| Capability | Year | Accuracy | Robot | Method | Source |
|---|---|---|---|---|---|
| single-axis position estimate | 2015 | 0.7 mm | TacTip on ABB arm | probabilistic model over marker displacements | tab. IV |
| 1D localization; curvature estimation | 2017 | 0.1 mm; about 1 mm | TacTip-FP1 on ABB arm | classifier over marker displacements | tab. IV |
| Slip detection | 2018 | more than 95% | TacTip on UR5 arm | SVM on pin velocities | tab. IV |
| 2D contour following (sliding) | 2019 | about 1 mm, about 10 deg | TacTip on ABB arm | convolutional regressor, 128 x 128 input | tab. IV |
| 1D localization; normal force | 2019 | about 0.5 mm, about 0.2 N | TacTip on ABB arm | convolutional autoencoder, 32 x 32 input | tab. IV |
| Object recognition | 2020 | more than 95% | Tactile Model O hand | ConvNet classifier, 60 x 120 image | tab. IV |
| Grasp success prediction | 2020 | more than 90% | Tactile Model O hand | ConvNet classifier, 60 x 120 image | tab. IV |
| Ultrasound detection | 2020 | about 10 micron (upper bound) | TacTip on ABB arm | GP regressor, pin displacements | tab. IV |
| 3D surface localization (depth, roll/pitch) | 2020 | 0.1 mm; 0.3 deg | TacTip on ABB arm | ConvNet regressor, 128 x 128 image | tab. IV |
| 3D edge localization | 2020 | 0.3 mm, 0.2 mm; 1-2 deg, 4 deg | TacTip on ABB arm | ConvNet regressor, 128 x 128 image | tab. IV |
| pinch held steady by a multi-fingered hand | 2021 | 0.1 mm; 0.4 deg | Tactile Modular Grasper | feedback loop plus convolutional network | tab. IV |
Headline numbers in the text, both for the Tactile Model O §VI.B: object classification reaches 93% across a set of 26 objects, and lifting success is predicted at 95% on that same set; slip-based re-grasping on 11 objects, 6 of them novel §VI.B. The review also says accuracy improved from about 1 mm to 0.1-0.5 mm for 3D pose tasks §VI.B, and that the latest results reach sensitivity at the level of single pixels of the tactile image §VI.B.
Paper vs code
No official code. The review has no code of its own, and its text names no
repository or project page (the LaTeX source and arXiv metadata contain no
code link). Searches run: a GitHub repository keyword search for tactip,
which returned 20 repositories and was cut off there; no result was from the
Bristol Robotics Laboratory or the author as a TacTip implementation. The
author's own GitHub account (nlepora) has five repositories:
digitac-design (GPL-3.0, design files for a DIGIT-based skin, not cited in
the review), afferents-tactile-gratings-jrsi2022 and
afferents-tactile-textures-jrsi2022 (notebooks for other papers, no license),
and two forks (DegenerateBoundaries_NatComm22, MIT; video_stream_processor,
GPL-3.0). None contains the TacTip sensor or the methods in the review. A
GitHub organisation search for the lab was not run.
Pitfalls
- Review, not method. Nothing in the card is an experiment by the author of this card. Numbers come from cited works, using different robots, sensors and metrics tab. IV. Do not rank rows of Table IV against each other.
- Text and table disagree. The text gives 93% object classification and 95% grasp success for the Tactile Model O §VI.B. Table IV lists more than 95% and more than 90% for the same two tasks tab. IV. Cite whichever source you use.
- Year of the 2018 TacTip. The table lists the TacTip cited as
ward-cherrier_tactip_2018with year 2016 tab. II, while the text calls it the 2016/2018 TacTip §VI.A. Use the text. - Shear model is approximate. Equation 2 replaces the arctangent by its argument, which is valid only for small surface slopes §IV.B eq. 2. Equation 1 needs a depth-dependent blur width that the review does not specify §IV.B eq. 1. The review says both models are consistent with the hypothesis, and it does not validate them against measurements.
- Claims that are the author's argument. That the TacTip is unique in sensing touch purely from shear, and that a pressure sensor in the gel would be a biomimetic vibration sense, are the author's view and are not measured in the review §IV.B §III.
- Missing learning details. The review names no network architecture, dataset size, loss, optimiser or training schedule for the deep-learning tasks §VII. Treat any reimplementation as your own design, and read the cited works for these values.
- Costs the abstract hides. Marker tracking, image-based learning and reinforcement learning are each costly. The review says deep reinforcement learning on physical robots takes too long, which is why it expects most learning in simulation §VII.
- Figures. Figure 1 contains third-party images credited to Wikipedia fig. 1. This card does not reproduce any figure; the figures are described in words only.
- Source version. This card is based on the arXiv v2 LaTeX source (updated 2021-07-22). Commented-out text in the source is ignored.
Lineage
- Builds on: the TacTip's first design by Chorley and colleagues, 2009 §V tab. II; the tactile contact lens, Kikuuwe and colleagues §V; the shear-sensing hypothesis of Platkiewicz, Lipson and Hayward §IV.B.
- Compared against: GelSight, Johnson and Adelson, 2009 §II §IV.A; BioTac, Wettels and colleagues, 2008 §II §IV.A; GelForce and ChromaTouch §IV.A.
- Related sensor from the same group: Tactile Model O, James and colleagues, 2020 §VI.A tab. III tab. IV.
- Superseded by: not stated in this wiki.
Source and license
Adapted from “Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review” by Nathan F. Lepora (arXiv:2105.14455), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.