SE-402 RESEARCH REPORT 1

Experiments in Single-Sensor Acoustic

Localization
Seth Pollen and Nathan Radtke
pollens@msoe.edu, radtken@msoe.edu

Abstract—This research project explores the area of tangible acoustic the rest of signal processing to take place in software.
interfaces as an alternative to traditional touch screens. Past research Finally, we want to improve usability of the system when
has shown the feasibility of single sensor acoustic localization. The a projector display is overlaid onto the touch surface,
weakness of existing solutions is the high effort required for calibration.
allowing calibration to take place interactively.
In this paper, we present our attempts to create a low cost, single sensor
acoustic localization solution, which employs algorithms intended to
reduce the user effort in calibration. We provide the results of our
implementation and present future research opportunities for the area
2 P REVIOUS R ESEARCH
of tangible acoustic interfaces. Significant research into tangible acoustic interfaces
took place in 2005-6 as a part of the TAI-CHI project
Index Terms—Tangible acoustic interfaces, touch screens, acoustics, (www.taichi.cf.ac.uk). This project identified three major
neural networks, signal processing
algorithmic approaches to the design of TAIs: time de-
lay of arrival (TDOA), time reversal (TR), and acoustic
1 P ROBLEM S TATEMENT holography [2]. Of these three, the time reversal ap-
proach has the special advantage of operating with as
Touch interfaces to computer devices have been the focus
little as one acoustic sensor.
of significant innovation in recent years, but mainstream
A single-sensor solution has many advantages. The
touch screen technologies remain expensive and do not
sensor can be positioned almost anywhere on the physi-
scale easily to large sizes or complex shapes. Thus, while
cal object being touched, eliminating the potential for hu-
touch surfaces could find numerous applications in the
man error in positioning multiple sensors relative to one
context of everyday human life, they are, for the most
another. Furthermore, a single sensor permits applica-
part, currently limited to traditional computing devices
tions to use existing sound input hardware (available on
such as cell phones and PCs.
most personal computers) to receive the incoming signal,
One technology which promises to make touch sur-
allowing all signal processing to be done in software [3].
faces more ubiquitous and economical relies on acoustic
The time reversal technique relies on the assumption that
sensors to detect impacts (such as taps from a fingertip
impacts at different locations on the sensitive object will
or solid object) on a surface by sensing the vibrations
produce unique acoustic patterns in the object (through
caused by those impacts. This technology, known as tan-
reverberations off the object’s boundaries as well as dif-
gible acoustic interfaces (TAI), is much simpler, cheaper,
ferent wave propagation modes), which can be received
and more power-efficient than existing technologies, and
by a single sensor and distinguished from one another
it scales very well to large and complex (non-planar)
in software. Calibration of a time-reversal system, which
surfaces. This opens up the possibility of making nearly
involves tapping various points on the surface to provide
any solid object into a responsive touch interface [1].
an initial data set to the time reversal algorithm, allows
The goal of our project is, first, to explore TAI solu-
the system to store the particular acoustic characteristics
tions using a single, inexpensive sensor and, second, to
of the medium and sensor placement.
reduce the calibration effort required for such systems
However, the research done on the TAI-CHI project
without reducing the system’s touch input resolution.
left some problems unsolved. Most importantly, their
This improvement will require the implementation of
time-reversal solutions are entirely discrete. Extant litera-
some kind of interpolation which allows the system to
ture does not seem to provide methods for interpolating
more precisely localize taps not coincident with given
between calibration points, meaning a large number of
calibration points. To reduce the cost of our solution,
calibration points is required to achieve a resolution that
we also hope to be able to use an ordinary laptop
appears continuous to users. The literature also seems to
audio line-in to digitize the incoming signal, allowing
lack significant inquiry into methods for distinguishing
Seth Pollen and Nathan Radtke are undergraduate software engineering valid impacts on the sensitive area from extraneous noise
students at the Milwaukee School of Engineering in Milwaukee, Wisconsin. or impacts to other areas on the object not intended
2 SE-402 RESEARCH REPORT
to be sensitive. In this paper, we discuss our efforts to
integrate interpolation and noise exclusion into existing
single-sensor acoustic localization techniques.
3 F OUR -P OINT A LGORITHM
The first software technique we investigated deviates
from existing single-sensor algorithms by making sig-
nificant assumptions about the acoustic medium. Based
on those assumptions, it attempts to model the acoustic
transfer function across the whole medium. This algo-
rithm is named “four-point” because of our initial hopes
that it would achieve acceptable localization across the
entire sensitive area based only on calibration data from
the four corners of that area. While we did not find
success with this algorithm, its outcomes informed our
further research and are discussed here.
3.1 Assumptions and Theory
For this algorithm, the sensitive object, or medium,
is assumed to be an ideal acoustic conductor with a
rectangular shape and smooth, straight edges. Under
these assumptions, a burst of sound produced by an
impact propagates through the medium without being
altered and reverberates off the edges of the medium
until being fully attenuated. A sensor attached to the
medium receives, in addition to the original sound pulse,
a series of echoes of that pulse, corresponding to all the
linear reflected paths between the impact point and the
sensor. Part of the four-point algorithm reduces each
received tap to a series of timestamps describing the
delays between successive echoes. Assuming a constant
speed of sound in the medium, these timestamps can
be used to reason about the lengths of the linear paths
followed by the sound waves, and from that reasoning
the location of the impact may be derived.
In addition to assumptions about the acoustic prop-
erties of the medium, this approach also assumes the
impact itself to consist of a brief burst of sound followed
by silence. The duration of this initial burst must be
shorter than the time delay of echoes in the medium
so that echoes do not overlap and obscure one another
in the received signal.
3.2 Tap Identification
A threshold-detection algorithm is used to identify im-
pacts in the signal coming in through the sound card.
Each time the audio signal crosses a certain threshold
h, a snapshot of the signal is taken with a number of
samples L prior to the threshold-crossing sample and T
samples following it; this snapshot is then sent to the
next stage of processing. After one snapshot is created,
new snapshots are not returned until a certain interval
of P samples passes without any of them crossing the
threshold. This ensures that an impact followed by a
particularly long string of echoes is not interpreted as
two separate impacts. Based on our experiments, the
following set of values for these parameters were found
to be the most adaptable and robust:
h = 0.015
L = 25
T = 575
P = 3000
With these settings, the software is able to correctly
identify taps on a variety of surfaces (chalkboards and
tabletops) and with a variety of tap devices (rubberized
bolts, human knuckles, and chalk). Note that these set-
tings are dependent on the capabilities of our sound
card; they are designed for operation with an amplitude
range of ±1.0 and a sample rate of 44.1 kHz. These
settings for T and L mean that each window of data
prepared for further processing is 600 samples (13.6 ms)
in length, and the minimum spacing between impacts is
3,025 samples (68.6 ms).
3.3 Quantization
The next step in the four-point algorithm is the quan-
tizer, which converts incoming sound waves into a
discrete time series of echoes. Several signal-processing
approaches to quantization were considered, includ-
ing conversion to the frequency domain. We rejected
frequency-based techniques, however, because the over-
laying of echoes into a single signal affects the amplitude
of the signal far more than its frequency.
While several variations were tried, all of our ap-
proaches to the quantizer consisted roughly of two
steps, illustrated in Figure 1. First, to smooth out the
oscillations in the incoming sound wave, the signal is
squared and filtered using a low-pass FIR filter with
a cutoff around 640 Hz, and the square root is then
taken to restore the signal to its original scale. Note
that the frequency envelope of a normal chalkboard tap
peaks around 3 kHz. In the figure, the red line shows
the original sound data captured by our microphone,
and the blue line shows the curve produced by this
smoothing technique. This smoothed curve provides a
continuous approximation of the amplitude of the sound
wave. In the second step of quantization, a median filter
is used to detect peaks in the smoothed curve. When it
encounters a peak of the correct width, the median filter
outputs a flat plateau which can easily be identified by
examining the differences between successive samples.
The positions of these detected peaks (illustrated as black
boxes in the figure) yield the timestamp series of echoes.
It is possible, however, for this algorithm to detect
waveform features not actually caused by echoes in the
acoustic medium. The quantizer detects any fluctuation
of the correct duration in the smoothed curve, no matter
how slight the amplitude. Therefore, another algorithm
step was added: it filters out all detected peaks except for
a constant number of peaks having the highest absolute
MAY 27, 2011
Fig. 1. Steps of echo quantization.
amplitudes; it was assumed that higher-amplitude peaks
were more likely to represent actual echoes. A further
refinement to this was also tried, which multiplies each
peak’s amplitude by a constant power (usually less
than 1) of its time relative to the start of the window
and then takes the highest peaks. This compensates for
the fact that peaks naturally decrease in amplitude as
time goes on, due to attenuation of the acoustic signal.
3.4 Geometric Calibration
While we spent significant time testing and tuning the
quantizer, we were never able to move to validation
of the four-point calibration algorithm, which uses the
timestamp series produced by quantization to reason
about the physical extent of the medium and the loca-
tions of impacts. Nevertheless, we did draft an initial
solution to this problem. This proposal collects a set
of calibration data, consisting of coordinate points on
the touch area paired with the quantized time series
produced when each point was tapped. This informa-
tion, combined with the known position of the sen-
sor, is used to derive the physical dimensions of the
acoustic medium. To do this, the medium is assumed
to be rectangular with edges parallel to the edges of
the designated touch sensitive area (which may or may
not occupy the entire physical medium). The calibration
process begins by proposing that the medium does not
extend past the edges of the sensitive area. The available
calibration data set is then searched for a timestamp
series that would not have been possible given its impact
location, the sensor location, and the currently proposed
3
medium boundaries. If one such calibration tap is found,
the appropriate proposed boundary for the medium is
extended by a small amount, and the same process is run
again. Hopefully, this method of successive approxima-
tions eventually converges on the correct medium size
and terminates.
Once calibration has determined the physical extent of
the medium, a simple algorithm can use this information
to localize incoming taps by simulating the propagation
of an impact at various points in the medium and
selecting the point whose simulation results most closely
match the observed timestamp series.
3.5 Problems with Four-Point
Although significant effort was spent tuning the four-
point algorithm, it was eventually deemed unworkable.
The biggest problem was the inconsistency of the quan-
tizer. Running our quantizer on data recorded from
different taps at the same point on the surface produced
significantly different timestamp series. This instability
in quantization was probably a result of our oversimplifi-
cation of the acoustic problem. It is quite possible that the
sound wave received by our sensor is the combination of
several different wave propagation modes which differ
from our assumed model of simple reflections [4]. In any
case, unstable quantization proved disastrous for our
geometric calibration technique. Because our calibration
technique regards each calibration sample (that is, each
series of timestamps) individually when negotiating the
physical dimensions of the medium, any error in the
quantizer output could cause the calibration algorithm
to fail to converge or cause it to produce wildly incorrect
estimates of the medium’s size.
While several workarounds were proposed for these
problems, a lack of time and understanding led us to
abandon the four-point algorithm and pursue localiza-
tion techniques which required fewer assumptions. It
may be possible, however, to improve the four-point
algorithm by requiring the user to provide additional
data during calibration. We could, for example, prompt
the user to measure the physical dimensions of the
medium so that our system does not have to derive
this information from sound samples using such an
unstable algorithm. Our calibration algorithm could also
be improved to consider multiple taps simultaneously
during its reasoning about the size and characteristics
of the medium; this would make it less sensitive to
variances in individual calibration taps.
4 N EURAL N ET A LGORITHM
4.1 Choosing a Neural Network
As we pursued the four point quantizer approach, we
also researched the use of neural networks. Our desire
was to calculate continuous output for localization. We
discovered that neural networks may be a solution for
this problem as they are commonly used to approximate
complicated functions.
4 SE-402 RESEARCH REPORT
Our research efforts led us to further investigate neu-
ral networks which would use a supervised learning
mechanism. This meant that we would require users to
collect data which, in turn, would be used to train the
neural network. The two neural network types which
we thought fit for our problem were the feed-forward
back-propagation network and the radial basis function
network. Our goal in using the neural network was to be
able to train the network and then use it as a black box
for solving localization between calibration points. Our
expected output was a coordinate pair corresponding to
a location on the user’s screen.
4.2 Implementation Approaches
Our implementation efforts started by looking for
a C# API that implemented the feed-forward back-
propagation network. We looked for this specific net-
work type as it has many common applications. Many
open source neural network APIs were found, but
Aforge.NET [5] and FANN [6] were found to have very
good documentation and appeared to be suitable for our
needs.
Getting the neural network to work proved to be a
difficult task. At first it was unable to train and reach our
performance goal, consisting of a maximum allowed er-
ror in the neural network output. Initially, we concluded
our performance goal error was too small and thus
unattainable. We experimented by raising the tolerance
of the performance goal but found that the root of the
issue was that the neural network was unable to make
training progress in the time we were allowing it to train.
The amount of time we allowed training to run varied
from several seconds to several minutes. To allow any
greater time for training, we feel, would have negatively
impacted the user experience. Additionally, given that
each time the system is set up it may be positioned
differently (based on sensor location, screen resolution,
calibration density, etc.) we consider calibration data
non-portable, therefore, calibrating for several minutes
during each setup is unrealistic and could cause users
to not use this system.
Our lack of results with the neural network libraries
led us to reevaluate the feasibility of using a neural net-
work for tap localization. Through additional research,
we determined that this was an appropriate use of a
neural network so we continued to investigate their
capabilities. We discovered MATLAB’s neural network
toolbox and employed it extensively to test and proto-
type neural networks.
4.3 Testing and Prototyping Neural Networks
The testing and prototyping of Neural Networks became
rigorous with the transition to MATLAB. Aided by cus-
tom functions and scripts, we were able to record data
in wave files and process them directly in the MATLAB
environment, which made the testing process more effi-
cient and automated. We developed a standard process
to test neural network features by adjusting parameters
in the network script and running a validation test to
see how well the configuration performed.
4.4 Processing Theories
By the time of testing in MATLAB we had gained a more
formal introduction to neural networks in an artificial
intelligence course. With this knowledge, we began to
discover some of the missteps taken during setup of the
neural networks. This ultimately led to better and more
thorough analysis.
Moving to MATLAB, we were encouraged to expe-
rience quick training times relative to those we were
expecting from the trials in C#. Typical training times
were on the order of 60 seconds. There were some
network configurations which took vastly longer and
attained the timeout threshold of 10 minutes, however,
this was observed in few cases. A recurring problem we
had was overfitting the network to the training data.
When the training goals were reduced, no improvement
was observed.
At this point, we also changed our goal with the
neural networks. Instead of training the network to
produce a coordinate pair when given a waveform, we
pursued a network that could generate a waveform from
a coordinate pair, thereby simulating the acoustics of
the medium. We understood that this was backwards
from normal neural network operation but we decided
to investigate further. The idea was that a neural network
would be able to generate additional calibration data to
assist the time-reversal algorithm (see section 5) reducing
the calibration effort required by the user. The generated
calibration data would fill in the spaces and increase the
resolution of the system.
4.5 Results
Overall, we experienced many mixed results with neu-
ral networks without gaining consistent improvement.
Our best results running the neural network in the
forward direction were with the radial basis function
network. Training was very quick and the calibration
points stayed consistent in testing. Points distant from
the calibrated points generally performed poorly.
The radial basis function network in the reverse direc-
tion yielded interesting results. We found that smoothed
wave generation was possible. However, from the in-
consistent results, we formed several suspicions. These
suspicions were rooted in inconsistent data and evidence
that our tapping device was the cause.
Later we found that inconsistent tap device and data
may not have been the sole cause for poor performance.
We found in an experiment that calibration data with
tighter spacing yielded better results. The calibration
point spacing used for testing the neural networks was
2-3x greater than what we found to be optimal (see
section 5.2). When we discovered this new information,
however, we had already abandoned the neural network
MAY 27, 2011
solution to shift focus on progressing the project. Given
time constraints we were unable to revisit the neural
network solution.
5 T IME R EVERSAL A LGORITHM
In contrast to our four-point and neural net approaches,
the time reversal algorithm has already been the subject
of extensive research [7] and has even been proposed
as part of a consumer product [8]. It makes only one
assumption about the acoustic properties of the medium,
namely, that impacts at different locations will produce
distinct acoustic patterns. During calibration, a set of
known points are tapped and the resulting sound waves
are stored. Then, in order to localize a new tap, its
sound wave is compared against the stored calibration
waveforms using some matching function, and the cali-
bration point whose waveform best matches the new tap
is returned as the result of the localization algorithm.
The name “time reversal” comes from the underlying
theory that the waveform received by the sensor, if time-
reversed and re-emitted into the medium at that point,
would reproduce the original impact waveform at the
original impact location.
5.1 Choice of Matching Function
The first task in developing a time reversal solution is
choosing the function to use for evaluating the match be-
tween two waveforms. Prompted by previous research,
we evaluated several candidate functions, including the
Pearson correlation coefficient [9] and a technique based
on cross-correlation [7]. The first technique uses the
traditional formula for Pearson’s r-value to measure the
match between two vectors A and B of samples taken
from the audio input hardware:
n    5.2
1 X Ai − Ā Bi − B̄
r1 =
n − 1 i=1 sA sB
where Ai is the ith sample from vector A, Ā is the mean
of A, sA is the standard deviation of A, and similar
definitions hold for B.
The second matching technique uses the maximum
value achieved by the cross-correlation of the two input
vectors:
∞
!
n X
r2 = max Ai Bi+t
t=−n
i=−∞
Where Ai and Bi are defined to be 0 for all indices
i ∈/ [1 n]. This technique effectively searches along
the time axis for a shift value (t) that brings the two
signals into the closest agreement. This is an advantage
over the Pearson technique used for r1 , which has no
tolerance for time-shifted signals. However, unlike the
Pearson technique, this cross-correlation is influenced by
the loudness of the signal as well as its shape. Thus,
louder signals will tend to produce higher r2 values,
even if they do not match well together.
5
Our final matching technique, which provides the
best results, is based on cross-correlation but adds the
statistical normalization used by Pearson’s technique to
compensate for varying loudness in the input signals:
1 n
∞ 
X Ai − Ā

Bi+t − B̄
!
r3 = max
n − 1 t=−n i=−∞ sA sB
The value produced by this formula will always lie in the
interval [0 1], with 1 indicating a perfect match between
the two vectors. Note that, because of the statistical nor-
malization performed on the two vectors, this technique
ignores their relative magnitudes.
Matching functions were evaluated by running them
on two waveforms produced by tapping the same point
on a chalkboard and two waveforms produced by tap-
ping different points. The function which best differen-
tiated between taps at the same location and taps at
different locations was considered the best.
Our implementation of the time-reversal algorithm
uses the same threshold-crossing algorithm as described
in section 3.2 for detecting impacts. As noted in section
3.2, a window size of 600 samples (at a 44.1 kHz sample
rate) is used to represent each impact. Larger window
sizes (up to 1200 samples) were tried, but localization
results did not improve significantly. Besides, increasing
window sizes degrades runtime performance by requir-
ing correlation to be computed over more samples. Var-
ious preprocessing functions were applied before com-
puting the cross-correlation match for two waveforms,
including the Fourier transform and a 640-Hz low-pass
filter (to smooth out oscillations in the signal). None
of these preprocessing functions improved correlation
results, so they were eliminated.
Calibration
The time-reversal algorithm requires more calibration
effort than would be necessary with more intelligent
approaches, such as four-point. This is due to the fact
that time reversal does not make any attempt to solve
or model the acoustic properties of the medium; it must
thus have calibration samples from points covering the
entire sensitive area.
All points in the designated sensitive area must be
within a certain distance of a calibration point so that
the time-reversal algorithm can use impact data gath-
ered from calibration points to approximate the acoustic
profile of all other possible impact locations. Thus, cal-
ibration points are usually arranged in a grid covering
the area of interest. The most important aspect of the
calibration grid is the physical spacing of these points
on the sensitive area. Interestingly, it is possible to cal-
culate an upper bound for this spacing from the acoustic
properties of the medium, such as the wavelength of the
waves produced by impacts; see [7].
We experimented with several different tap surfaces,
including blackboards, solid wood tables, and composite
6 SE-402 RESEARCH REPORT

wood tables. We generally found an acceptable cali- Fig. 2. Discrete matching results.
bration point spacing to be between 3 and 6 inches;
this ensures that our discrete localization algorithm (see
section 5.4) always matches a tap on the sensitive area
to one of the nearest calibration points. Wider spacing
of calibration points tends to degrade both localization
schemes (see sections 5.4 and 5.5).

5.2.1 Point Refinement

To improve the integrity of calibration data, a technique
called point refinement was adopted. During calibration,
each calibration point is tapped three times, yielding
three sample signals. These samples are then matched
to one another using our chosen matching function.
The sample with the highest correlations to the other
two samples is retained in the final calibration data
set, while the other two samples are discarded. This
makes calibration more robust by enabling the system to
identify and discard bad data, which could be caused by
extraneous noise or by some other impact on the sensing
surface not intended as a calibration tap. It also improves
runtime performance of localization by reducing the
number of correlations that must be computed against
the calibration data set.
5.3 Tap Exclusion
Once we had chosen an appropriate matching function,
we applied it to two separate problems: tap exclusion
and tap localization. The first of these problems, tap
exclusion, has received less attention from the research
community to date. The goal here is to ignore ambient
noise and impacts on areas not designated as touch-
sensitive.
This was identified as an important feature if this
technology were to be used in normal settings. Errant
or unintended computer interaction could be prevented
with the implementation of such filtering. We developed
two levels of tap exclusion. The first level is to filter
out input by monitoring the amplitude of sound directly
from the input source; see section 3.2. The second level
is correlation exclusion.
The second level of exclusion filters out errant taps
based on the tap’s correlation to the calibration data.
A linear function was found to express the needed
threshold for the incoming tap’s correlation to the stored
library of calibration data. This function depends on the
spacing of the calibration points. The trend found was
that the wider the spacing between calibration points the
lower the threshold must be. Therefore a larger spacing
will permit more taps than a tighter, or smaller, spacing.
5.4 Discrete Localization
The goal of tap localization is to determine where an
impact originated, based on the received signal. Two
approaches to this problem were tried: one discrete and
the other interpolated. The discrete solution is much
simpler; once a correlation function is selected, input
signals can be matched against the grid of calibration
samples paired with coordinate locations on the sensitive
surface. The calibration point which best matches the
received signal is considered to be the location of the
impact. In our experiments with grids of properly spaced
calibration points, taps not coinciding with a calibration
point almost always matched to one of the four nearest
calibration points, with occasional errors.
To simplify program operation when high resolution
is not required, an alternative discrete solution was
implemented, which matches taps to regions on the
screen rather than to exact points. Each region must be
calibrated with a grid of points that properly cover it, as
described above in section 5.2. Incoming taps are then
matched to the center of the localization results produced
by this technique. The grids of calibration points are
not shown; the regions were calibrated on a classroom
chalkboard with a 3-inch spacing between points. The
calibration points did not coincide with the test points.
The grid overlaid on the regions in the figure shows the
test points that were tapped to verify proper operation of
region-based discrete matching. As you can see, points
falling within the regions are matched to the proper
region with almost perfect accuracy.
The grid vertices not colored indicate test points
whose taps were excluded. The tap excluder performed
poorly in the lower-right corner of the area shown,
permitting several taps not in a designated sensitive
region. This may be due to the fact that the sensor
was attached to the medium in this area. Another factor
be an improperly selected threshold for tap exclusion.
Since creating this figure, we have implemented a tap
exclusion scheme that dynamically selects this threshold
based on the calibration point spacing, addressing part
MAY 27, 2011
Fig. 3. Example correlation surface, using a 3-inch mesh
spacing.
of the tap exclusion problem illustrated here.
5.5 Interpolated Localization
With the time reversal algorithm, it may not be necessary
to constrain localization to the discrete set of known
calibration points. Investigating this possibility is the
goal of interpolated localization. The four-point and
neural net algorithms promise interpolated localization
by solving the acoustic transfer function of the medium.
Time reversal does not attempt to solve the transfer
function, and therefore it must perform interpolated
localization based solely on the correlation of the input
tap with its set of calibration taps. If the calibration grid
points are spaced closely enough, the correlation values
for grid points provide sufficient sampling of a smooth
correlation surface with its peak at the actual location
of the impact [7]. Figure 2 shows an example of such
a surface from one of our own experiments. The next
step is to find the maximum point of the (presumably)
smooth surface of which each calibration point provides
a discrete sample. If the spacing of calibration points is
too wide, peaks in the correlation surface will not be
properly sampled by the calibration grid, which could
cause localization results to be grossly inaccurate.
One possible technique for interpolated localization
would be to fit piecewise-smooth regression curves to
the correlation values in each row and column of the
calibrated grid. The maxima of these horizontal and
vertical curves could then be calculated and combined to
produce a location within the grid. We did not examine
this technique, however, due to a lack of time.
The technique we did investigate performs weighted
averaging of calibration point coordinates in the x- and
y-dimensions (using correlation values as the weights)
7
to yield the coordinates of the output point. Performing
this weighted average over the entire set of calibration
points is not a good solution, however, because it will
always bias the output points toward the center of the
mesh; poorly correlated calibration points still have a
non-zero correlation and will thus be factored into the
average. Therefore, some method must be introduced
for choosing a set of representative points from the
grid so that only those points are averaged together.
To further reduce the effect of poorly correlated points,
another step is added: once the representative set of
points is chosen, the lowest correlation value in the set is
subtracted from the correlation value for each point, with
values dropping below zero being clamped to zero. We
tested this weighted-averaging technique using several
algorithms for selecting the representative set of points.
5.5.1 Selection by Cells
In this technique, we find the highest discrete point on
the surface (that is, the best-correlated calibration point).
From the four grid cells which have this point as a
corner, we choose the cell with the highest average cor-
relations at all of its corners. The four corners of this cell
form the representative set which is used for weighted
averaging. Selecting only a single cell, however, biases
interpolation away from calibration points and towards
the centers of cells, since the center of the representative
set is always the center of a cell, and this is where the
average of the corners’ coordinates tends to fall. In order
to avoid this, the eight points surrounding the best-
correlated point (i.e. the corners of all four adjacent cells)
were taken as the representative set. This technique,
however, biases interpolation towards calibration points,
since the set of representative points is always centered
on a calibration point. In summary, neither of the cell-
based techniques provide consistently good interpolated
localization.
5.5.2 Individual Selection
Alternative methods of selecting the representative set
of points include choosing all points with correlations
above a certain threshold and choosing the best n cor-
related points for some constant n. These methods have
the advantage of not depending on a single, maximal
discrete point for their geometric arrangement (as the
cell-based approaches do). This allows the representative
set to vary more freely as the characteristics of the cor-
relation surface change. Also, under the threshold-based
approach, the shift by which all correlations are reduced
before averaging can be a constant (that is, the threshold
itself) instead of being determined by a single point sam-
pled from the representative set. This makes localization
less dependent on any single datum, improving stability.
In our experiments, these approaches (one based on
a correlation threshold and the other on a constant n
number of points) provided better interpolation results
than the cell-based approaches.
8 SE-402 RESEARCH REPORT
Fig. 4. Results of average-threshold interpolation, with a
power of 25.
There are problems with these techniques, however.
The threshold approach has the disadvantage that it
requires manual selection of the threshold value, which
may need to be re-tuned if the mesh spacing, tap device,
or tap surface change. The “constant-n” approach also
has a disadvantage: it relies on a single point (the nth-
best correlated calibration point) to provide the shift by
which all the other correlation values are reduced before
being averaged. Reliance on a single point for anything
introduces unwanted variability into the system.
To address these concerns, we tried one last inter-
polation algorithm. Instead of manually specifying the
threshold for selecting the representative set of calibra-
tion points, we calculate it each time by taking the mean
of the correlation values of all the calibration points.
To bias this mean upwards, however, we raise all the
correlations to a power p > 1, take the mean, and then
raise the mean to the power p−1 . We experimented with
various values for p on a 7-by-7 calibration point mesh,
finding the best interpolation results around p = 25. With
this value for p, each representative set contained an
average of 3.1 points. Figure 4 shows a vector plot of
the results of this interpolation algorithm with p = 25.
Each green circle is the location of an actual impact,
either at a calibration point or at the middle of a grid
cell, and the corresponding blue arrow points to the
location to which the interpolation algorithm mapped
that impact. Ideally, then, the blue vectors should all
have length zero. From the figure it can be seen that
this interpolation technique works well in some areas
but has a consistent bias in other areas. This bias, which
during experimentation seemed to persist with repeated
taps, may be due to inaccuracies in our calibration data
set or to acoustic features of the underlying medium. In
any case, this was the best interpolation we achieved.
5.6 Performance Concerns
Performing correlation of an input tap with all samples
in the calibration set is a time-intensive task. Our goal
was to keep the time taken by localization small enough
so that users perceive their input as registering instan-
taneously. To achieve this goal, we performed several
optimizations.
The cross-correlation of signals (used by us to calculate
the correlation between two taps) is traditionally com-
puted across its entire domain using Fourier transforms.
We, however, do not usually need to compute the cross-
correlation across its entire domain, since we are only
interested in its maximum value, which usually occurs
with a shift value near zero. Our edge detection algo-
rithms ensure that sampled signals are at least somewhat
aligned along the time axis, so the cross-correlation only
needs to be computed over a small interval centered at
zero in order to compensate for slight timing variations
in the sampled signals. Thus, instead of taking the
Fourier transform of both signals and then multiplying
them together to find the cross-correlation, we calculate
the cross-correlation over a window of ±120 samples
(±2.7 ms at our 44.1 kHz sample rate) using a naive
algorithm (that is, a simple sum of products for each
possible offset):
1 n
120 
X Ai − Ā

Bi+t − B̄
!
0
r3 = max
n − 1 t=−n i=−120 sA sB
Finally, since the time reversal algorithm requires sev-
eral independent correlations to be calculated, it adapts
well to parallel computing. Running on a dual-core
machine, we were able to halve the running time by
using parallel tasks to calculate correlation values for
our whole set of calibration data. As an example, we
ran our parallelized discrete localization algorithm on
a calibration set containing 60 taps; localization took an
average of 101 ms each time. This timing is good enough
to provide a seamless user experience.
6 S ENSING H ARDWARE
6.1 Sensor
The sensor we chose was the Knowles accelerometer
which was used by TAICHI researchers [4]. The Knowles
device, part number BU-21771, is a high sensitivity ce-
ramic vibration transducer. This device was an attractive
choice because it requires low voltage (we were able to
power it by stepping down the voltage from a USB port)
and the output voltage did not require amplification to
be read by the sound card.
Early in the project we pursued our goals using a
standard desktop microphone as our sensor. Dr. Sverre
Holm preceded our usage of this sensor type in his
demonstration [3]. We discovered several weaknesses in
the use of the microphone. The first weakness is that
the microphone was highly susceptible to interference of
ambient noise, whereas the same ambient noises did not
MAY 27, 2011
affect the Knowles vibration transducer. Additionally, we
observed that the sound of the impact carried through
the air has a significant influence on the desktop mi-
crophone. This is a particular problem because we want
to monitor only the vibrations which propagate through
the medium; the Knowles device performs far better in
this respect.
Attempts were made to modify the microphone to
improve the performance, such as removing the casing
and soundproofing it from with added padding. How-
ever, these techniques did little to improve the behavior.
Ultimately, we identified the Knowles vibration trans-
ducer as the sensor appropriate for our project. The
transducer’s cost was slightly higher than the desktop
microphone, but this difference was not a deterrent for
the performance gains.
6.2 Signal Capture
All results described in this paper were achieved using
a normal laptop sound card to digitize the sensor signal;
the hardware sample rate was 44.1 kHz. Extant literature
supports the conclusion that this arrangement is suffi-
cient for good localization [7], [10].
7 TAP D EVICE H ARDWARE
In our early experiments and tests, one theory we had
about the observed poor results was that our tap device
was producing an inconsistent tap when struck against
the surface. Originally, we used chalk against a slate
blackboard as demonstrated by Holm [3]. Our analysis
and observations led us to believe that chalk, when hit
against the chalkboard, can produce a different sound
each time causing our localization techniques to produce
incorrect results.
The goal then became to find a device that would pro-
duce a consistent tap. The proposed device was a spring
loaded punch with a repeatable mechanical action. Upon
running the same tests as with the chalk, we were
surprised to find similar results. The consistently tapping
center punch did not improve our results. What these
tests revealed to us was that the tap generating device
was not the sole cause for poor algorithm performance.
After that, our investigations into the tap device were
shelved while we refined our matching algorithms. Dur-
ing our tests with interpolation, we rediscovered the
inconsistencies of tapping with chalk to be an issue.
This time, we tried tapping with a rubber coated bolt,
which yielded much better results. We surmise that the
rubber coating softens the impact, producing a smoother
and more regular input to the surface’s acoustic trans-
fer function. The results we observed as a result of
changing to this tapping device were encouraging. In
an experiment we observed behavior that showed the
tap matching performance with the rubberized bolt was
much improved over the chalk tests.
Our last tap device test was to tap the medium with
our fingers. Due to time constraints we were unable
9
to fully test finger tapping; however, in several trials
we found that using a finger as the tap generator was
possible. We have identified this as a future opportunity
area of research.
8 S URFACE
The first medium, or surface, we tested upon was a slate
chalkboard. This medium was chosen as the standard
testing surface for consistency and also because of the
proven demonstration by Holm [3]. While testing on the
slate surface we tested a number of tapping techniques
as described in section 7. Overall, this medium when
used with a consistent tap produced good results.
The second medium we tested was a thin plastic
sheet which would be used as an inexpensive pane for
framing. With this medium the importance of the tap’s
intensity was revealed to have a significant impact on
the localization performance. However, we experienced
good localization and tap exclusion results. In order to
accommodate tapping on this surface we had to lower
our impact detection threshold to accommodate taps
with less intense peaks. This was also our first test to
not include tapping with a specific device. We observed
consistent matching by knocking our fingernail against
the medium when both of the calibrated location and
intensity were matched.
The last surfaces we tested our system on were table-
tops. The material of the table varied from solid wood to
wood composite. We tested tapping this surface with our
fingertips and our tapping device. We were able to lower
the impact detection threshold low enough to accommo-
date both tapping techniques, and the localization results
observed were acceptable.
We observed that the properties of the tap were
slightly altered in each medium tested. These differences
required minor changes to the exclusion parameters
of the processing algorithms. However, the parameters
were finally set to a point to accommodate all of the
aforementioned surfaces. Additionally, in our GUI ap-
plication we give the user the ability to modify these
parameters to customize the algorithm performance to
any medium.
9 C ONCLUSIONS
In this project, we were not able to achieve our original
goal of a robust interpolation algorithm. We did, how-
ever, make progress toward this goal and built a strong
understanding of the problem domain, which would
allow us to make real progress on interpolation if we had
more time. We were able to implement a polished dis-
crete time-reversal solution designed for operation with
a projector display overlaid onto the touch surface. This
software projects calibration point locations on the screen
and walks the user through the process of calibrating
each one.
Our investigations into tap exclusion (see section 5.3)
take, we believe, a new direction which has not received
10
significant attention from past researchers. The results
of our tap exclusion algorithm are fairly good, and
exclusion has been incorporated into our final software
deliverables.
10 F URTHER R ESEARCH O PPORTUNITIES
10.1 Alternative Surfaces
The great potential of this technology lies in its applica-
tion to a wide variety of everyday surfaces. It would
be interesting to test the sensing hardware and time-
reversal software we have developed against surfaces
like walls and tile floors. Some of our software param-
eters would need to be tuned to the new environment,
but we surmise that our algorithms would still provide
meaningful results in these situations.
10.2 Tap Devices
To reduce the hardware associated with this system, we
began testing our solutions where the method of input
is the user’s fingertip rather than a dedicated tapping
device. This is a challenge that we were unable to devote
much time to; however, it is one improvement that we
feel would make this technology ready for ubiquitous
use. Challenges we observed using fingertips as the tap
generator included variations is the intensity of the tap.
If the intensity was not consistent a tap would not occur.
This problem was extended for situations when multiple
users tried to use the application. We found that it was
difficult for a user who did not perform the system
calibration to find the right intensity and easily use the
application.
10.3 Transparent Calibration
The biggest obstacle to the use of time-reversal acous-
tic localization is the large amount of calibration data
required from the user each time the system is set up
with a different surface or sensor position. One possible
solution is to collect calibration data transparently dur-
ing normal usage of the system, so that the user does
not realize that calibration is taking place. Meaningful
calibration data must consist of a known location paired
with a received acoustic signal, presumably caused by
an impact at that location. To collect such paired data
during normal usage of the system, the system must
somehow guess where the user actually tapped to pro-
duce the signal it has received. If the user is interacting
with a GUI composed of discrete sensitive regions (like
buttons), one way to guess at this location is to assume
that each time the user taps a button, he or she tried to
tap the center of that button. Thus, though the system
will require enough manual calibration to be able to
correctly identify which button the user tapped, after
that point, it can gradually refine itself further by using
data transparently gathered from user operation.
Other techniques for transparent calibration might
include the development of calibration games which
SE-402 RESEARCH REPORT
deliver some entertainment value while still providing
the system with paired locations and acoustic samples.
10.4 Other Interpolation Techniques
None of our interpolation techniques achieved satisfac-
tory localization. This may be due to our calibration
grids still being too widely spaced, but it may also be
that interpolation techniques based on weighted averag-
ing are not a good solution. It may be better to inves-
tigate a regression-based solution which fits piecewise
smooth curves to the rows and columns of samples on
the correlation surface and then solves those smooth
curves for their maxima.
11 R EFERENCES
For a video demonstration of this project’s results, see
[11].
R EFERENCES
[1] D. T. Pham, Z. Wang, Z. Ji, M. Yang, M. Al-Kutubi, and S. Cathe-
line, “Acoustic pattern registration for a new type of human-
computer interface,” in IPROMS 2005 Virtual Conference, May
2005.
[2] W. Rolshofen, D. T. Pham, M. Yang, Z. Wang, Z. Ji, and M. Al-
Kutubi, “New approaches in Computer-Human Interaction with
tangible acoustic interfaces,” in IPROMS 2005 Virtual Conference,
May 2005.
[3] S. Holm, “Touch sensitive blackboard,” Institutt for Informatikk,
Universitetet i Oslo, 2008. [Online]. Available: http://www.
youtube.com/watch?v=V4NwoiPGkVY
[4] “Technical Solutions for the TDOA Method,” Dipartimento di
Elettronica e Informazione, Politecnico di Milano, Tech. Rep.,
2006.
[5] “Aforge.net,” Andrew Kirillov, et. al. [Online]. Available:
http://www.aforgenet.com/framework/
[6] “Fast artificial neural network library,” Steffen Nissen, et. al.
[Online]. Available: http://leenissen.dk/fann/wp/
[7] R. K. Ing, N. Quieffin, S. Catheline, and M. Fink, “In sold local-
ization of finger impacts using acoustic time-reversal process,”
Applied Physics Letters, vol. 87, 2005.
[8] “Reversys technology,” Sensitive Object. [Online]. Available:
http://sensitive-object.com/-ReverSys-R-
[9] D. T. Pham, M. Al-Kutubi, Z. Ji, M. Yang, Z. Wang, and S. Cathe-
line, “Tangible Acoustic Interface Approaches,” in IPROMS 2005
Virtual Conference, May 2005.
[10] “Technical solutions and demonstration for acoustic pattern
recognition using time reversal method,” Laboratoire Ondes et
Acoustique, Tech. Rep., 2006.
[11] S. Pollen and N. Radtke, “Acoustic touch screen demonstration,”
Milwaukee School of Engineering, 2011. [Online]. Available:
http://www.youtube.com/watch?v=ZoAslMiukAQ