---
kind: concept
slug: tactile-sensing
title: Tactile sensing
summary: >-
  Getting contact information at a robot fingertip from a soft, camera-based sensor. Pins under a skin move marker tips that a camera tracks. A review of one such sensor argues that sideways shear carries shape, and reports a shift from marker features with classical learners to CNNs on whole tactile images. Results are author-reported and come from separate studies.
tags: [tactile-sensing, robot-manipulation, soft-robotics]
papers: ["2105.14455"]
related: [in-hand-manipulation, convolutional-neural-network, haptic-feedback-rendering]
status: draft
updated_at: 2026-11-15
---

## Overview

Tactile sensing gives a robot information about contact at its fingertips. The
problem is that many optical tactile sensors measure only normal pressure, or
use reflection or marker schemes whose output is hard to interpret
[arxiv:2105.14455 §II] [arxiv:2105.14455 §IV.A].

The variants share one design. A soft skin covers a transparent gel, and stiff
pins run from the skin into the gel. Each pin tip carries a marker, and a camera
inside the sensor images the markers through the gel [arxiv:2105.14455 §III].
When the skin is indented, the pins tilt and the markers shift sideways. The
camera image of the marker pattern is therefore the signal, and everything
downstream is a function of marker motion or of the whole image. The review's
central claim is that the sensor responds mainly to shear strain, not to normal
strain alone [arxiv:2105.14455 §IV.A].

## Variants

### Marker features with classical learners (about 2015 to 2019)

Each frame is processed by tracking every pin marker. Two signals come out of
the tracking: displacement from the resting position, and velocity between
frames [arxiv:2105.14455 §VI] [arxiv:2105.14455 §VI.B]. Probabilistic
classifiers work on displacements, and support vector machines on velocities
for slip detection [arxiv:2105.14455 tab. IV]. Gaussian process and polynomial
regressors serve sliding and grasp control [arxiv:2105.14455 §VI.B]. The
review reports a slip-detection accuracy of more than 95% on a UR5 arm in 2018
[arxiv:2105.14455 tab. IV], and a single-axis position estimate of 0.7 mm in 2015
[arxiv:2105.14455 tab. IV]. It judges these approaches to transfer poorly once
the task is continuous and moving [arxiv:2105.14455 §VI.B].

Card: [TacTip review](/papers/2105.14455).

### Whole-image CNN on tactile images (2019 onward)

A convolutional network reads the whole tactile image. The only preprocessing
is cropping, thresholding or concatenation, and there is no marker-tracking
step. The network predicts a quantity directly, such as pose, grasp success or
object class [arxiv:2105.14455 §VII] [arxiv:2105.14455 tab. IV]. Reported
results include object recognition above 95% and grasp success above 90% on the
Tactile Model O hand, from 60 x 120 tactile images [arxiv:2105.14455 tab. IV],
and 3D surface localisation of 0.1 mm and 0.3 degrees from 128 x 128 images
[arxiv:2105.14455 tab. IV]. The review does not state the network architecture,
training set size, loss or optimiser for these tasks [arxiv:2105.14455 §VII].

Card: [TacTip review](/papers/2105.14455).

### Hardware variants: printed sensors and hands

The sensor has been rebuilt many times with 3D printing. The review lists
40 mm tips with 127 to 532 pins across versions, and a 20 mm version with 276
pins that is not integrated with a camera [arxiv:2105.14455 tab. II]. The same
design has been fitted to several hands, including the Tactile Model M2, GR2
and O, the Shadow Modular Grasper and the Pisa/IIT SoftHand
[arxiv:2105.14455 tab. III]. Multi-material printing enabled rapid prototyping of
these probes and hands [arxiv:2105.14455 §VI.A].

Card: [TacTip review](/papers/2105.14455).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Recover edges or shape from contact with a soft camera sensor | Shear-based marker sensing | The shear-sensing hypothesis, which the review reports, holds that zero crossings of the shear field mark edges without a gradient step [arxiv:2105.14455 §IV.B]. The review's two shear models are approximations and are not validated against measurements [arxiv:2105.14455 §IV.B]. |
| Detect slip from a short, mostly static event | Marker velocity with an SVM | The review reports more than 95% slip detection on one arm and sensor [arxiv:2105.14455 tab. IV]. |
| Continuous, moving contact with a tactile image stream | Whole-image CNN | The review judges marker-feature methods to transfer poorly to continuous, moving tasks [arxiv:2105.14455 §VI.B], and reports CNN results for servo control and grasping [arxiv:2105.14455 §VII]. |
| Need a sensor of a new shape or size quickly | 3D-printed TacTip family | Printing enabled rapid prototyping of these probes and hands [arxiv:2105.14455 §VI.A]. |
| Need deep reinforcement learning with touch | Train mostly in simulation | The review says deep reinforcement learning on physical robots takes too long, and expects most learning in simulation [arxiv:2105.14455 §VII]. |
| Need to reproduce a learner | Read the cited works | The review gives no architecture, dataset size, loss or optimiser for its learning tasks [arxiv:2105.14455 §VII]. |

The review does not compare its methods on a common benchmark, and each row of
its results table comes from a different robot, sensor and metric
[arxiv:2105.14455 tab. IV]. Compare the rows only within their own setup.

## Implementation sketch

The two learning paths share the same front end: the camera frames go in, and
the question is whether the markers are tracked first or the whole image is
used. Shapes are shown in the comments where they matter.

```pseudocode
# Front end: track each pin marker through the frames  [§VI] [§III]
# Input:  camera frames, resting marker positions
# Output: displacement and velocity per pin

def marker_features(frames, rest_markers):
    # 1. Track markers; shape (T, N_pins, 2)  [§VI]
    markers = track_markers(frames)
    # 2. Displacement from rest, the slow-adapting signal  [§VI.B]
    displacement = markers - rest_markers
    # 3. Frame-to-frame velocity, the fast-adapting signal  [§III]
    velocity = diff(markers, axis=0)
    return displacement, velocity
```

```pseudocode
# Whole-image path: no marker tracking  [§VII]
# Input:  tactile image, cropped or thresholded
# Output: quantity of interest, e.g. pose or grasp success

def predict_from_tactile_image(tactile_image, network):
    # 1. Feed the image directly  [§VII]
    # Architecture and sizes are not stated in the review.
    return network(tactile_image)
```

Where the variants differ is the front end and the learner. The review gives
no training recipe for either path [arxiv:2105.14455 §VI] [arxiv:2105.14455 §VII].

## Contradictions

Within the review, the text and Table IV give different numbers for the Tactile
Model O hand. The text reports 93% object classification and 95% lifting
success over 26 objects [arxiv:2105.14455 §VI.B]. The table reports more than
95% for object recognition and more than 90% for grasp success
[arxiv:2105.14455 tab. IV]. The review does not say which is correct. Cite the
source you use.

## Papers

- [TacTip review](/papers/2105.14455): a review of a soft, camera-based,
  biomimetic tactile sensor, covering shear sensing, the printed sensor family,
  and the move from marker features to CNNs on tactile images.
