Sensing The Frictional State of A Robotic Skin Via Subtractive Color Mixing
Sensing The Frictional State of A Robotic Skin Via Subtractive Color Mixing
Sensing The Frictional State of A Robotic Skin Via Subtractive Color Mixing
org/
2386 IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 4, NO. 3, JULY 2019
Abstract—The perception of surface properties such as shape These chains of events involve fine and dexterous control of the
and slipperiness is crucial to ensure that the hand-held object frictional contact between the object and the skin. In humans,
is stable. Without touch, precise manipulation becomes difficult.
the state of friction is thought to be assessed not by directly
Some robotic tactile sensors use cameras that observe the elastic
deformation of a membrane to detect edges or slippage of the con- determining the normal and tangential components of the force,
tact. The perception of the contact state drove innovative control but rather depending on which part of the fingertip is stuck to
strategies. However, previous methods on these lines do not include the object and which part is starting to slide [3], [4].
quantitative means of measuring the three-dimensional (3-D) de- The contribution of tactile sensing to grasping and manipula-
formation of the skin or suffer from a lack of spatial resolution. tion has been well recognized [5]. Artificial skin based on piezo-
Here, we present a tactile sensor based on a subtractive color mix-
ing process designed to track the 3-D displacement of an array of resistive [6], piezoelectric [7], electrostatic [8], [9], optical [10]
markers, using the information delivered by the color channel of and even ultrasonic transducers [11] have been previously tested.
off-the-shelf cameras. The distributed shear and normal deforma- These tactile sensors convert the localized deformation of the
tion can be assessed from the spectrum of the light reflected and surface into a signal that is stored and interpreted by a computer.
refracted by an array of diffusive and transmissive markers placed These sensors can collect information that is not accessible vi-
on two superimposed layers. The markers show various blends of
colors, depending on the displacement at the surface. The color sually, not only because the contact is often hidden from view,
pattern of each marker can be tracked with little computation but mostly because the information about the contact, such as
and remains robust to external lighting. The ability to sense the the slipperiness of the surface or the compliance of the material,
3-D deformation field can improve robotic perception of frictional requires mechanical interactions to be revealed.
properties that have applications in the fields of robotic control and Many of the latest methods designed for this purpose focus
human-robot interactions.
on measuring the pressure field applied normal to the surface,
Index Terms—Force and tactile sensing, soft material robotics. which suffices to recognize objects [12]. It has been established,
however, that the lateral traction produced by friction is essential
I. INTRODUCTION to control robotic grippers [13], [14]. A dense array of sensors
OR both robots and humans, tactile perception is essential also requires a cumbersome set of electrical interconnections
F to be able to learn and perform appropriate hand gestures
for grasping and manipulating objects [1]. In particular, humans’
and conditioning electronics when a large number of transducers
are used.
tactile perception of the state of contact between a finger and Some sensors use a camera to transduce the deformation of
an object generates information on which the stability of the an elastic body or membrane [15]–[18]. The usual procedure
grasp depends [2]. The object might have to be moved reliably starts by locating the center of black or white markers. The
from one place to another without inducing any perception of lateral motion of each marker can be easily determined with
relative motion with respect to the fingers. In other scenarios, computer-vision algorithms and the distributed measurements
the opposite problem might arise when the object has to be are sufficiently rich to recognize the nature and orientation of
slid to a certain part of the hand in order to be properly lifted. an object. However, these methods do not directly provide the
normal and lateral pressure field at the interface. In particular,
Manuscript received September 10, 2018; accepted December 29, 2018. Date the local friction coefficient, expressed by the ratio between
of publication January 16, 2019; date of current version March 21, 2019. This
letter was recommended for publication by Associate Editor H. Liu and Editor
the lateral shear stress and the normal stress cannot be directly
A. M. Okamura upon evaluation of the reviewers’ comments. This work was observed although this parameter is essential for characterizing
supported by Agence Nationale de la Recherche (ANR) under Grant 16-CE10- the adherence of an object and its stability in the hand.
0003 and in part by the CNRS Auton program. (Corresponding author: Michaël
Wiertlewski.)
To address these challenges, we developed the camera-based
The authors are with the Aix-Marseille University, CNRS, ISM, Marseille sensor, shown in Fig. 1(a) and 1(b), which not only tracks the
13009, France (e-mail:, xi.lin@univ-amu.fr; michael.wiertlewski@univ-amu. lateral motion of an array of markers but also resolves the mo-
fr).
This letter has supplementary downloadable material available at http://
tion normal to the surface. The sensor recruits a double array
ieeexplore.ieee.org, provided by the authors. The Supplemental Materials con- of overlapping semi-transparent colored markers. The deforma-
tain a video presenting a tactile sensor based on a subtractive color mixing tion of the markers, which are attached to the interaction surface,
process designed to track the 3-D displacement of an array of markers, using
the information delivered by the color channel of off-the-shelf cameras. This
affects their shape and their color content, which makes it pos-
material is 6.53 MB in size. sible to reconstruct the 3-dimensional deformation field at the
Digital Object Identifier 10.1109/LRA.2019.2893434 interface.
2377-3766 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.
LIN AND WIERTLEWSKI: SENSING THE FRICTIONAL STATE OF A ROBOTIC SKIN VIA SUBTRACTIVE COLOR MIXING 2387
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.
2388 IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 4, NO. 3, JULY 2019
Fig. 3. Effect of the change in the size of the magenta markers on the histogram
of the hue channel. The histogram is only slightly affected by the number of
pixels in the image.
Fig. 2. (a) Diagram of the image of the marker observed by the camera.
(b) The white light is scattered by either the background or the magenta marker.
Some of the scattered light crosses through the yellow filter, which further
filters the color spectrum. (c) Color spectra of the light after the reflection by the
magenta markers, of the light after being transmitted trough the yellow markers,
and of the light which is both scattered by the magenta markers and filtered by
the transparent yellow filter. (d) Corresponding histograms of the hue channel
in the HSV colorspace.
Fig. 4. (a) The pinhole camera model explaining how sensor motion is cor-
yellow filter, see Fig. 1(c). Shear forces will shift the center of related to changes in apparent size. (b) The large Poisson coefficient of soft
each marker relative to each another. The combination of stretch, material results in significant stretching of the soft marker, which increases
compression, and lateral shift creates a colored pattern, which the signal to noise ratio. (c) The optimum thickness maximizing the apparent
change in the case of a given external force is presented here, depending on the
specifies the direction and the magnitude of the displacement camera’s angle of view.
vector of the surface above the markers Fig. 1(d).
Fig. 2(a) and 2(b) are typical views of the two layers, showing
the three colors magenta (which is white without any green), C. The Opto-Mechanical Model
yellow (which is white without any blue) and red (which is This section describes models for the optical and mechanical
white any green or blue). Marker arrays are flooded with diffuse components of the complete sensor.
white light, which can be either diffused by the magenta markers 1) Effects of the Focal Length on the Resolution: The reli-
or the white layer and possibly filtered by the yellow markers on ability of the measurements depends on the camera and lens
its way back to the camera. All four combinations of the color used to detect the markers. A longer telephoto lens will reduce
spectrum shown in Fig. 2(b) can be seen in the resulting image. the apparent changes in the marker size, and a simple model
Physically, shifting from white to red occurs when the magenta shows that the shorter the focal length, the more pronounced the
markers reflect only the red and the blue parts of the spectrum motion of the moving marker will be.
and the blue is filtered out by the yellow layer, as described in The model illustrated in the Fig. 4(a) is based on the well-
Fig. 2(c). known pinhole camera model, in which light rays reflected by
Humans and cameras alike detect only three bands in the objects and reaching the image plane cross a point located one
optical spectrum, in the blue (≈450 nm), green (≈580 nm) and focal length from the image sensor.
red (≈ 690 nm) ranges. Images detected in the Red-Green-Blue The problem is constrained by the fact that n markers have
color space can be converted into Hue-Saturation-Value (HSV) to fit into the field of view θ. Each of the markers therefore has
color space, where the value and the saturation depend only on to cover a fraction of the field of view, L0 /(d + t) = tan(θ/n).
the illumination of the markers and the hue channel contains the The pinhole model states that the angular size of the markers
color information. An example of the hue intensity of typical is the same on both sides of the focal point, tan (θ/n) = l0 /f .
light rays is presented in Fig. 2(d). The hue channel is presented With these constraints in mind, maximizing the sensitivity of
in the form of a color, where the colors are shown at an angle with the sensor in the normal direction amounts to maximizing the
respect to an arbitrary origin, set at red. In the HSV color space, relative changes in the apparent size (l − l0 )/l0 with a given nor-
the center of mass of the histogram of the hue channel depends mal displacement of the marker δz . Let us take Thales’ intercept
on the normal displacement of the soft marker with respect to theorem:
the transparent marker, see Fig. 3. Since changes in the hue
l L0 l0 L0
of the image involve a large number of pixels carrying 24 bits = and = (1)
of information (versus 1 bit in the case of segmented black and f d + t − δz f d+t
white images), these fluctation will be theoretically more visible Combining these equations leads to the relative change:
in the case of small displacements than the apparent change in −1
l − l0 δz δz
the marker size, which translates into greater sensitivity to the = 1− −1= + O δz2 (2)
motion normal to the surface. l0 d+t d+t
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.
LIN AND WIERTLEWSKI: SENSING THE FRICTIONAL STATE OF A ROBOTIC SKIN VIA SUBTRACTIVE COLOR MIXING 2389
Fig. 5. (a) The soft magenta layer is pushed down towards the yellow filter.
(b) The model predicts that a wide angle lens will give greater magnification at
a given normal displacement of the marker.
Fig. 6. (a) The hue measurements of the marker are almost insensitive to the
apparent size. The simulation showed that the mean hue shifts smoothly towards
The result of this equation is shown in Fig. 5(a). The smaller the hue of the flexible magenta marker. (b) Effects of the apparent size on the
accuracy of the polynomial fit. (c) Effects of the non-uniform luminosity on the
the distance to the object is, the more noticeable the changes displacement measurements.
will be. Smaller focal length optics are therefore preferred.
2) Mechanics and Optimal Thickness Between Layers: The
top layer of the assembly is soft and the magenta markers can leverages a Gaussian filter with standard deviation set to 1/8th
be stretched elastically. Stretching increases the actual size of of the size of the image was added to remove artifacts.
the marker and therefore further enhances the sensitivity, see Fig. 6(a) shows that even with a apparent size of 4 × 4 pixels,
Fig. 4(b). displacements can be satisfactorily approximated by a second
A simple model for the deformation of the marker can be order polynomial fit (R2 > 0.98). We also ran simulations on
drawn up, taking in consideration only the elasticity of the ma- the effects of changing the pixel density on the estimated size of
terial sandwiched between the two layers. In this simplified black and white markers. Fig. 6(b) shows the dramatic effects
model, which is presented in Fig. 4(c), the behavior of the ma- of decreasing pixel density on the black and white markers,
terial boils down to a compression ratio that can be described resulting in a steady decrease in the accuracy of the polynomial
by the materials Young’s modulus E, the thickness t and the fit, which goodness of fit reach as low as R2 = 0.5 when the
marker size L, as well as a lateral extension corresponding to apparent size is 4 × 4 pixels, while the hue-based method is
the Poisson’s ratio ν. The compressive elasticity of the material only slightly affected.
just below the marker can be obtained by deriving Hooke’s and Lastly, in order to gauge the robustness to illumination non-
Poisson’s laws: uniformity, we looked at the effect of adding a 25%-opacity gra-
dient overlay. Fig. 6(c) reports the difference value of the hue or
Fz δz
δz = t and L = L0 1 + ν (3) luminosity for the color-mixing and the black and white method
EL20 t
respectively, between the non-uniform and the uniform illumi-
We can see that a thicker sensor gives greater marker mobil- nation, relative to the overall range of measurement. This metric
ity at a given external force, at the expense of a smaller change compares both methods using a dimensionless number. The re-
in the apparent area. A thickness that maximized the compli- sults show that the accuracy of the color-based method decreased
ance while keeping a large stretch was obtained by combining by less than 1% under non-uniform illumination, whereas the
equations 1 and 3. Assuming that we have a Young’s modulus black and white markers have relative error as high as 50%.
of E = 0.4 MPa, Poisson’s ration ν = 0.5 and a marker size
of L0 = 2 mm projecting an image onto a ls = 35 mm sensor, E. Methods & Manufacturing
the results obtained with three different lenses are presented in
Because it relies only on color and transparency, the sen-
Fig. 5(b). The model argues in favor of a soft material with a
sor can be constructed with inexpensive equipment and materi-
low Young’s modulus, which could be thin and deformable.
als. The procedure used to make the two layers is presented in
Fig. 7. First, a soft white compound (SortaClear 12 with Pig-
D. Robustness to Pixel Density and Lighting Conditions ment Ignite, Smooth-On, Macungie, PA, USA) is poured into a
One of the main advantages of using color channels is that the 3-D printed mould (TPU95A, Ultimaker, Geldermalsen, Nether-
markers configuration can be sorted out using just a few pixels. A lands) to form the outer layer of the sensor. The soft material
simulation was run to verify the robustness of the method when has a Young’s modulus of E = 0.4 Mpa and a Poisson’s ratio of
only a few pixels were used. A single marker was first drawn ν = 0.5. The white color serves to block out the light from the
using a vector graphics editor (Illustrator, Adobe, San Jose, outside, while at the same time diffusing the white illumination.
CA, USA) to depict an opaque magenta marker underlying a Once the outside layer has been cured, a rigid mask is set in place
yellow marker with an opacity of 50%. The size of the magenta and a mixture of the same soft compound and a magenta dye is
marker was changed to provide a range of artificial normal screen printed and heat-cured. A transparent layer (SortaClear
displacements. Images were rasterized in a 512 × 512 image and 12, Smooth-On, Macungie, PA, USA) is cast on top of the ma-
a pyramid gaussian process was used to create smaller versions, genta markers to fill the holes left by the mask. The transparent
with the goal to emulate the effect of having smaller markers. layer also protects the markers and sets the right thickness for
Once the small version was created, an anti-aliasing filter that the sensor, depending on the intended design. All the elastomer
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.
2390 IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 4, NO. 3, JULY 2019
Fig. 8. Direct measurements show the linear relationship between the mean
hue degree and the normal displacement of the surface.
Fig. 7. The production process. (a) A white light-diffusive soft layer is placed
in a cast. (b) Once it has been cured, a rigid mask is applied, and (c) the magenta
markers are screen printed. (d) After the curing process, the mask is removed
and the remaining markers are covered with a transparent compound. (e) The
soft part is mounted on the rigid backing support.
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.
LIN AND WIERTLEWSKI: SENSING THE FRICTIONAL STATE OF A ROBOTIC SKIN VIA SUBTRACTIVE COLOR MIXING 2391
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.
2392 IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 4, NO. 3, JULY 2019
information about the shape of the object and the friction forces [15] N. J. Ferrier and R. W. Brockett, “Reconstructing the shape of a de-
at work. formable membrane from image data,” Int. J. Robot. Res., vol. 19, no. 9,
pp. 795–816, 2000.
[16] B. Ward-Cherrier et al., “The tactip family: Soft optical tactile sensors
ACKNOWLEDGMENT with 3D-printed biomimetic morphologies,” Soft Robot., vol. 5, no. 2,
pp. 216–227, 2018.
The authors would like to thank J. Diperi and S. Viollet for [17] K. Sato, K. Kamiyama, N. Kawakami, and S. Tachi, “Finger-shaped
enlightening discussions. gelforce: sensor for measuring surface traction fields for robotic hand,”
IEEE Trans. Haptics, vol. 3, no. 1, pp. 37–47, Jan.–Mar. 2010.
[18] M. K. Johnson and E. H. Adelson, “Retrographic sensing for the mea-
REFERENCES surement of surface texture and shape,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., 2009, pp. 1070–1077.
[1] L. Natale and E. Torres-Jara, “A sensitive approach to grasping,” in Proc. [19] C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, “Development of a
6th Int. Workshop Epigenetic Robot., 2006, pp. 87–94. tactile sensor based on biologically inspired edge encoding,” in Proc. Int.
[2] R. S. Johansson and J. R. Flanagan, “Coding and use of tactile signals Conf. Adv. Robot., 2009, pp. 1–6.
from the fingertips in object manipulation tasks,” Nature Rev. Neurosci., [20] K. Kamiyama, K. Vlack, T. Mizota, H. Kajimoto, K. Kawakami, and
vol. 10, no. 5, pp. 345–359, 2009. S. Tachi, “Vision-based sensor for real-time measuring of surface trac-
[3] T. André, V. Lévesque, V. Hayward, P. Lefèvre, and J.-L. Thonnard, “Effect tion fields,” IEEE Comput. Graph. Appl., vol. 25, no. 1, pp. 68–75,
of skin hydration on the dynamics of fingertip gripping contact,” J. Roy. Jan./Feb. 2005.
Soc. Interface, vol. 8, no. 64, pp. 1574–1583, 2011. [21] F. Guo, C. Zhang, Y. Yan, P. Li, and Z. Wang, “Measurement of three-
[4] B. Delhaye, P. Lefevre, and J.-L. Thonnard, “Dynamics of fingertip contact dimensional deformation and load using vision-based tactile sensor,” in
during the onset of tangential slip,” J. Roy. Soc. Interface, vol. 11, no. 100, Proc. IEEE 25th Int. Symp. Ind. Electron., 2016, pp. 1252–1257.
2014, Art. no. 20140698. [22] R. Li et al., “Localization and manipulation of small parts using gelsight
[5] R. S. Dahiya, G. Metta, M. Valle, and G. Sandini, “Tactile sensing—from tactile sensing,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2014,
humans to humanoids,” IEEE Trans. Robot., vol. 26, no. 1, pp. 1–20, pp. 3988–3993.
Feb. 2010. [23] W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson, “Measurement of
[6] M. Strohmayr, H. Wörn, and G. Hirzinger, “The DLR artificial skin step shear and slip with a gelsight tactile sensor,” in Proc. IEEE Int. Conf.
I: Uniting sensitivity and collision tolerance,” in Proc. IEEE Int. Conf. Robot. Autom., 2015, pp. 304–311.
Robot. Autom., 2013, pp. 1012–1018. [24] K. O. Johnson, J. R. Phillips, “Tactile spatial resolution. i. two-point
[7] R. D. Howe and M. R. Cutkosky, “Dynamic tactile sensing: Perception of discrim- ination, gap detection, grating resolution, and letter recognition,”
fine surface features with stress rate sensing,” IEEE Trans. Robot. Autom., J. Neurophysiology, vol. 46, no. 6, p. 1177, 1981.
vol. 9, no. 2, pp. 140–151, Apr. 1993. [25] A. Bicchi, “On the closure properties of robotic grasping,” The Int. J.
[8] B. Li, Y. Shi, A. Fontecchio, and Y. Visell, “Mechanical imaging of soft Robot. Res., vol. 14, no. 4, pp. 319–334, 1995.
tissues with a highly compliant tactile sensing array,” IEEE Trans. Biomed. [26] M. Wiertlewski, J. Lozada, and V. Hayward, “The spatial spectrum of
Eng., vol. 65, no. 3, pp. 687–697, Mar. 2018. tangential skin displacement can encode tactual texture,” IEEE Trans.
[9] N. Wettels, V. J. Santos, R. S. Johansson, and G. E. Loeb, “Biomimetic Robot., vol. 27, no. 3, pp. 461–472, Jun. 2011.
tactile sensor array,” Adv. Robot., vol. 22, no. 8, pp. 829–849, 2008. [27] M. Shimojo, “Mechanical filtering effect of elastic cover for tactile sensor,”
[10] A. Kadowaki, T. Yoshikai, M. Hayashi, and M. Inaba, “Development of IEEE Trans. Robot. Autom., vol. 13, no. 1, pp. 128–132, Feb. 1997.
soft sensor exterior embedded with multi-axis deformable tactile sensor [28] K. L. Johnson and K. L. Johnson, Contact Mechanics. Cambridge, U.K.:
system,” in Proc. RO-MAN 18th IEEE Int. Symp. Robot Human Interactive Cambridge Univ. Press, 1987.
Commun., 2009, pp. 1093–1098. [29] T. Maeno, T. Kawamura, and S.-C. Cheng, “Friction estimation by pressing
[11] H. Shinoda, K. Matsumoto, and S. Ando, “Acoustic resonant tensor cell an elastic finger-shaped sensor against a surface,” IEEE Trans. Robot.
for tactile sensing,” in Proc. IEEE Int. Conf. Robot. Autom., 1997, vol. 4, Autom., vol. 20, no. 2, pp. 222–228, Apr. 2004.
pp. 3087–3092. [30] Y. Hu, R. Katragadda, H. Tu, Q. Zheng, Y. Li, and Y. Xu, “Bioinspired
[12] D. Xu, G. E. Loeb, and J. A. Fishel, “Tactile identification of objects using 3-D tactile sensor for minimally invasive surgery,” J. Microelectromech.
Bayesian exploration,” in Proc. IEEE Int. Conf. Robot. Autom., 2013, Syst., vol. 19, no. 6, pp. 1400–1408, 2010.
pp. 3056–3061. [31] A. Yamaguchi and C. G. Atkeson, “Combining finger vision and optical
[13] T. Maeno, S. Hiromitsu, and T. Kawai, “Control of grasping force by tactile sensing: Reducing and handling errors while cutting vegetables,” in
detecting stick/slip distribution at the curved surface of an elastic finger,” Proc. IEEE-RAS 16th Int. Conf. Humanoid Robots, 2016, pp. 1045–1051.
in Proc. IEEE Int. Conf. Robot. Autom., 2000, vol. 4, pp. 3895–3900. [32] L. Osborn, R. R. Kaliki, A. B. Soares, and N. V. Thakor, “Neuromimetic
[14] S. Hirai et al., “A novel model for assessing sliding mechanics and tactile event-based detection for closed-loop tactile feedback control of upper
sensation of human-like fingertips during slip action,” Robot. Auton. Syst., limb prostheses,” IEEE Trans. Haptics, vol. 9, no. 2, pp. 196–206, Apr.–
vol. 63, pp. 253–267, 2015. Jun. 2016.
Authorized licensed use limited to: Japan Advanced Institute of Science and Tech. Downloaded on July 10,2020 at 05:06:52 UTC from IEEE Xplore. Restrictions apply.