IEEE WIRELESS COMMUNICATIONS LETTERS, VOL. 4, NO.

3, JUNE 2015 285

An Iterative Hybrid Transceiver Design Algorithm

for Millimeter Wave MIMO Systems
Chiao-En Chen, Member, IEEE

AbstractIn this letter, a new algorithm for millimeter wave

multiple-input-multiple-output hybrid (mixed RF and baseband)
transceiver design is proposed. The proposed algorithm iteratively
updates the phases of the phase-shifters in the RF precoder
(or RF combiner) to minimize the weighted sum of squared resid-
uals between the optimal full-baseband design and the hybrid
design, and is guaranteed to converge to at least a local opti-
mal solution. Simulation results show that the proposed iterative
design can achieve almost the same performance as the optimal
full-baseband design, in spite of using a much smaller number of
RF chains.
Index TermsMultiple-input-multiple-output (MIMO), mas-
sive MIMO, millimeter wave, hybrid precoding, transceiver.
I. I NTRODUCTION
B ECAUSE of the ever-increasing mobile traffic from smart
phones, tablets, and other mobile devices, it is expected
that the next-generation wireless networks will have to sup-
port 1000 times more capacity when compared with what we
have today [1]. To achieve this goal, a number of promising
technologies have been proposed [2], [3]. Millimeter wave
(mmWave) multiple-input-multiple-output (MIMO) communi-
cation enables additional access to the 30300 GHz bands, and
hence, has drawn great research interest recently [4].
The transceiver design architecture for mmWave MIMO can
be very different from conventional microwave MIMO. This
is mainly because mmWave MIMO requires tens-to-hundreds
of antennas to leverage the acquired array gain to compen-
sate for the high path loss in the mmWave bands. Owing to
the high hardware cost of RF chains, the conventional full-
baseband design in which a dedicated RF chain is allocated
for each antenna element becomes impractical [5]. To address
this issue, hybrid transceiver architecture, which concatenates
an RF precoder (or RF combiner) with a baseband precoder (or
baseband combiner) is proposed [6], [7]. In [7] and later refined
in [8], the authors formulate the design problem into a sparsity
signal reconstruction problem and then solved approximately
by using the orthogonal matching pursuit (OMP) algorithm
[9]. A modification of the OMP-based hybrid transceiver was
later proposed in [10] to enable efficient parallel computation
without sacrificing the performance when compared with the
OMP-based design [7].
Manuscript received November 29, 2014; accepted February 23, 2015. Date
of publication March 5, 2015; date of current version June 18, 2015. This
work was supported by the Ministry of Science and Technology, Taiwan,
under Grant Number MOST103-2622-E-194-008-CC1. The associate editor
coordinating the review of this paper and approving it for publication was
V. Raghavan.
The author is with the Department of Electrical/Communications Engineer-
ing, National Chung Cheng University, Chiayi 621, Taiwan (e-mail: ieecec@
ccu.edu.tw).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/LWC.2015.2409268
Fig. 1. Block diagram of a hybrid transceiver for mmWave MIMO
communications.
In this letter, we propose an iterative hybrid transceiver de-
sign algorithm aimed at reducing the performance gap between
the existing OMP-based hybrid transceiver [8] and the optimal
full-baseband design. The main idea is to sequentially update
the phases in the RF precoder (or RF combiner) in a greedy
manner and hence, monotonically decrease the weighted sum
of squared residuals between the two precoders (or combiners).
Simulation results show that the proposed iterative design can
achieve almost identical rate performance in the mmWave chan-
nel with a much smaller number of RF chains when compared
with the full-baseband design. A complexity reduction scheme
is also presented in this letter.
Notations: Throughout this letter, matrices and vectors are
set in boldface, with uppercase letters for matrices and lower
case letters for vectors. The superscripts T , H , and 1 denote the
transpose, conjugate transpose, and inverse, respectively.  F
is used to represent the Frobenius norm of a matrix, and the
notation []:, is used to denote the th column of a matrix. The
commas and semicolons in a matrix are used as the column and
row separator, respectively. Finally, IN represents the N N
identity matrix.
II. S YSTEM M ODEL
We consider a single-user mmWave hybrid MIMO system
[8] as shown in Fig. 1. The transmitter is assumed to transmit
Ns spatial streams via its Nt antennas and NtRF RF chains,
while the receiver is assumed to use its Nr receive antennas
and NrRF RF chains for reception. To enable multi-stream
transmission with lower implementation complexity, we as-
sume that number of RF chains is smaller than the number
of antennas, and hence Ns NRF < N for all {t, r}.
The Ns 1 symbol vector s is first linear precoded by the
baseband precoder FBB followed by the RF precoder FRF .
Here, represents the average transmitted power per symbol
vector, while the elements in s are assumed to be independent
and have been normalized to have energy N1s . The precoded
signal vector x = FRF FBB s is then transmitted over the
millimeter wave channel. The RF precoder FRF is assumed to
be implemented via low-cost analog phase shifters, and hence
each element in FRF is constrained to have an identical norm.
We assume |[FRF ],m |2 = 1/Nt , for all  = 1, . . . , Nt , and

2162-2337 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
286 IEEE WIRELESS COMMUNICATIONS LETTERS, VOL. 4, NO. 3, JUNE 2015

m = 1, . . . , NtRF . The concatenated precoder F = FRF FBB is where  = 0, . . . , Ny 1 and m = 0, . . . , Nz 1. With the afore-
assumed to follow the total power constraint tr{FFH } = Ns . mentioned system architecture, the achieved rate can be ex-
Without loss of generality, we can assume that the mil- pressed as R = log2 det{INs + Ns R1 H H H
n W HFF H W},
limeter wave channel is narrowband and block-faded.1 Un- 2
where Rn = n W W. H
der this assumption, the equivalent baseband channel can be
represented by the channel matrix H CNr Nt . Without loss
of generality, the channel matrix is assumed to have been III. R EVIEW ON THE S PATIALLY S PARSE
normalized to satisfy E{H2F } = Nt Nr . Using the above no- H YBRID T RANSCEIVER D ESIGN
tations, the received data vector y can then be represented as
In [8], a spatially sparse transceiver design is proposed to
y = HFRF FBB s+n, where n CNr 1 is the noise vector, maximize an approximation of the achieved rate. After some
modelled as CN (0, 2 INr ) representing a circularly symmet- mathematical approximations, the following optimization prob-
ric complex Gaussian random vector with zero-mean and co- lems are formulated [8]:
variance matrix 2 INr . At the receiver side, the data vector
y is processed by the concatenated combiner matrix WH = (Fopt opt
RF , FBB ) = arg min Fopt FRF FBB F
2
H H H H
WBB WRF , where WRF and WBB are the RF combiner and FBB ,FRF
the baseband combiner matrices, respectively. Again, the RF subject to [FRF ]:, {at (p,q , p,q ), for all p, q} ,
combiner is implemented by analog phase shifters, and hence
we can assume |[WRF ],m |2 = 1/Nr , for all  = 1, . . . , Nr , and for all  = 1, . . . , NtRF ,
m = 1, . . . , NrRF . FRF FBB 2F = Ns (3)
To take the channel characteristics of the millimeter wave  
channel into account, we follow the model adopted in [8] which  opt opt   21 2
WRF , WBB = arg min Ryy (WMMSE WRF WBB )
is based on the widely accepted extended Saleh-Valenzuela WBB ,WRF F
geometric channel model [11]. In this model the channel matrix
subject to [WRF ]:, {ar (p,q , p,q ), for all p, q} ,
H is expressed as
for all  = 1, . . . , NrRF , (4)
 
Ncl N ray
   
H= p,q r rp,q , p,q
r
t tp,q , p,q
t 1 1

p=1 q=1
where Ryy 2
= {(/Ns )HFFH HH + n2 INr } 2 , WMMSE
   t H is the optimal MMSE combiner given by WMMSE =
ar rp,q , p,q
r t
at p,q , p,q , (1)
( /Ns )R1 yy HF, and Fopt C
Nt Ns
is the optimal semi-
unitary precoder obtained as the right singular vectors of
where is a normalization factor such that E{H2F } = H associated with the largest Ns singular values. Fopt and
Nt Nr . p,q , rp,q (p,q r
), and tp,q (p,q t
) denote the complex WMMSE are also referred to as the optimal full-baseband
gain, the azimuth (elevation) angle of arrival (AOA), and the precoders and combiners, respectively.
azimuth (elevation) angle of departure (AOD) associated with In [8], the optimization problems (3) and (4) are then for-
the qth propagation path (ray) from the pth cluster, respectively. mulated into optimization problems with sparsity constraints,
(p,q , p,q

) and a (p,q , p,q
) represent the antenna gain and then solved approximately using the OMP algorithm [9].
and array response vector, respectively, for all {t, r}, p = The OMP-based hybrid transceiver enjoys low computational
1, . . . , Ncl and q = 1, . . . , Nray . The complex gain p,q is mod- complexity and has been shown to achieve rate performance
eled as CN (0, ,p 2
INr ), where ,p 2
denotes the average power close to that of the optimal full-baseband design in the simu-
associated with the propagation paths in the pth cluster. The lated mmWave channel [8].
azimuth angles p,q (elevation angles p,q
) within each cluster
are assumed to be Laplacian random variables with angle
spread ( ) centered at a uniformly distributed mean cluster IV. P ROPOSED I TERATIVE H YBRID T RANSCEIVER D ESIGN
angle p (p ), where {t, r}. For simplicity, we assume ideal A. Derivation of the Proposed Algorithm
sectored transmit and receive antenna elements whose antenna
gain is modeled as ( , ) = 1 for all [min , max ], In this subsection, we propose an iterative algorithm for the
and ( , ) = 0, otherwise. hybrid transceiver design problem in mmWave MIMO systems.
We consider the case of a uniform planar array (UPA) as the We start from the optimization problems in (3) and (4), and then
array geometry. It follows that the normalized array response express the objective functions in the following common form:
for an Ny Nz -element UPA in which Ny and Nz elements are  1 2
 
placed on the y-axis and z-axis can be expressed as J(, X) =  2 (Y AX) . (5)
F
1 
a (, ) =  1, . . . , ejkd( sin sin +m cos ) , Here, = INt , y = Fopt , A = FRF , X = FBB in the
Ny Nz precoder design problem, and = Ryy , y = WMMSE ,
T
, ejkd((Ny 1) sin sin +(Nz 1) cos ) , (2) A = WRF , X = WBB in the combiner design problem.
The(, m)th element of A can be denoted as [A],m =
(1/ N )exp (j,m ) due to the norm constraints of the RF
1 In principle, a frequency-selective channel can always be converted into a precoders and combiners, where ,m = [],m . Here equals
parallel collection of frequency flat sub-channels using orthogonal-frequency- t in the precoder design problem, and equals r in the combiner
division-multiplexing (OFDM). design problem, respectively.
CHEN: AN ITERATIVE HYBRID TRANSCEIVER DESIGN ALGORITHM FOR MILLIMETER WAVE MIMO SYSTEMS 287

The minimization of (5) corresponds to the weighted nonlin- expressed as J( ) = tr{yH PT A(AH A)1 AH Py},
ear least squares problem, where and X are the nonlinear and where = PQ, and = PPT . In this new formulation,
linear parameters, respectively. Following the concentration []1,1 = 1,1 is the optimization variable and has to be varied
technique [12], the weighted least squares solution for X given evaluations in the optimization algorithm. For
for differentJ()
A() is obtained as X = (AH A)1 AH y. Substituting notational convenience, we denote A as A = [a, bH ; c, D],

X as X in the objective function (5), we then obtain the where a = (1/ N )exp(j1,1 ) C, while b, c, and D are
concentrated objective function the corresponding vectors and matrix that will be held constant

  1 H  during the optimization. Using the notation in A, AH A
J() = tr YH Y tr YH A AH A A Y can then be expressed as AH A = [e, f H ; f , G], where e =
which only depends on the nonlinear parameter . Since
tr{yH y} is independent of the parameter , the optimal
weighted least squares solution for can be obtained as =
arg maxJ(), where
J() = tr YH A(AH A)1 AH Y . (6) and = e f H . The matrix inverse (AH A)1 can now be
(6) is known to be a non-convex function of and no closed-
form solution is available to date. As a result, we propose to
search for a local optimal solution through alternating mini-
mization. The algorithm is briefly summarized as follows.
Initialization: The proposed algorithm starts with some low-
cost initial solution A, such as the solution obtained from the
OMP-based design.
Iteration: In each iteration, the algorithm performs a local
search in each element of and then updates [A],m . When
,m is being optimized, the rest of the parameters in are
fixed to their latest updated values. Since ,m is merely a real
variable with bounded ranges [, ], a local optimal point of
(6) around the latest updated solution can be found efficiently
by performing 1-D search using the Nelder-Mead simplex-
method [13]. The algorithm performs N NRF element-wise
local searches per iteration as  proceeds from 1 to N , and m
proceeds from 1 to NRF , respectively. The algorithm iterates
the above procedure until the prescribed number of iterations
has been reached.
Construction of X: After A has been obtained, the linear
parameter X is estimated as X. For the precoding
problem, the
obtained parameter X is further scaled by Ns X/AXF to
satisfy the total power constraint described in (3).
Since the objective function J() monotonically increases
in each update and is upper bounded by tr{yH y}, one can
conclude that the proposed algorithm is guaranteed to converge
to at least a local optimal solution.
B. Complexity Reduction Method for Evaluating J()
The Nelder-Mead simplex-method used in the proposed it-
erative algorithm requires multiple computations of J() as
the algorithm searches for optimized variables. Consequently,
multiple matrix inverses (AH A)1 or associated QR factor-
izations need to be recomputed in (6) even though only a single
element in A has been changed. In this subsection, we derive
a complexity reduction method by exploiting the structure in
AH A via the low-rank update technique.
Without loss of generality, we consider the 1-D optimiza-
tion problem where the phase in [A],m is the optimization
variable. We can construct two permutation matrices P
CN N and Q CN N so that A = PAQ and [A]1,1 =
RF RF
[A],m . With the above permutations, (6) can be equivalently
T T
[a , cH ][aT , cT ] , f = [b, DH ][aT , cT ] are both func-
T
tions of a, whereas G = [b, DH ][b , DT ] is independent
of a. By using the matrix inversion Lemma and also the
1
Woodburys identity results [14], (AH A) can be computed
1
as (AH A) = [ 1 , ; 1 1
H H
, G ], where = G f
replaced by matrix-vector and vector-vector multiplications.
The algorithm is performed as follows. 1/2 , the Hermitian
matrix square root of , is first computed. The resulting com-
plexity overhead is negligible when compared with the overall
complexity of the algorithm and hence will not be counted
in the complexity analysis. At the beginning of each 1-D op-
timization, the proposed algorithm computes 1/2 [b, DH ]H ,
1:N ,2:N c, and G1 only once because all these quantities
1/2
are independent of the current optimization variable. For each
function evaluation, the algorithm then computes 1/2 [aT ,
) using these newly
cT ]T , e, f , , and then computes J(
computed quantities. With the proposed reduction scheme, it
can be shown that the average computational complexity per
function evaluation is reduced from the original O(N2 NRF )
to O(max{N2 , N NRF Ns , N3 /Neval }), where Neval is the
average number of function evaluations needed in the 1-D
Nelder-Mead optimization. The complexity reduction method
is therefore most effective when N3 /Neval  NRF .
V. S IMULATION R ESULTS
Here, we present some simulation results of the proposed
iterative hybrid transceiver design simulated under an mmWave
MIMO channel model. For better comparison and cross-
referencing, we follow the channel parameters used in [8] for
our simulation. Throughout the simulation, the sector angles
at the transmitter are assumed to be 60 wide in the azimuth
and 20 wide in the elevation domain, while the sector angle
at the receiver is assumed to be the whole angle space. The
mean azimuth angle and mean elevation angle are set to be
0 and 90 , respectively. The channel is assumed to consist of
Ncl = 8 clusters and Nray = 10 rays per cluster. The angular
spreads are assumed to be t = t = r = r = , while the
average power associated with the propagation paths is assumed
2
to be ,p = 1 for all clusters. The true angles-of-departure and
angles-of-arrival are assumed to be available when performing
the OMP-based design as in [8]. The received SNR is defined as
SNR = /n2 , and each simulation point is averaged over 5000
independent Monte Carlo runs.
In Fig. 2, the achieved rates of various transceiver designs are
simulated under the scenario where the transmitter is assumed
to have Nt = 64 antennas and NtRF = 4 RF chains, while the
receiver is assumed to have Nr = 16 antennas and NrRF = 4
288
Fig. 2. Achieved rate versus SNR achieved by various transceiver designs in
an mmWave MIMO channel ( = 7.5 ).
Fig. 3. Achieved rate versus angle spread achieved by various transceiver
designs in several mmWave MIMO channel configurations.
RF chains. The proposed iterative designs using various initial-
izations such as the OMP initialization, random phase initializa-
tion, and dominant steering vectors [15] initialization have been
simulated. The simulation results suggest that the proposed
iterative design is not very sensitive to different initializations
(at least in the simulated scenario) as all iterative designs
can achieve near-optimal rate performance in two iterations.
Among the three initializations, the OMP-based iterative design
appears to have the best performance. Running the OMP-based
iterative algorithm for just one iteration appears to have ac-
quired most of the performance gain, while further performance
improvement can be achieved by running more iterations at the
expense of higher complexity.
In Fig. 3, the achieved rates of the proposed OMP-based
iterative design is simulated at SNR = 0 dB with NRF = 4 and
IEEE WIRELESS COMMUNICATIONS LETTERS, VOL. 4, NO. 3, JUNE 2015
NRF = 6. The simulation setting is the same as the one used
in Fig. 2, except that the angle spread now varies from 0
to 15 . It is observed from the figure that the proposed OMP-
based iterative design consistently provides improved rate when
compared with the OMP-based design within the whole range
of . It is further observed that the proposed OMP-based
iterative design equipped with 4 RF chains can even outperform
the OMP-based design equipped with 6 RF chains.
VI. C ONCLUSION
In this letter, an iterative hybrid transceiver design algorithm
for mmWave MIMO systems is presented. The proposed al-
gorithm is based on the nonlinear least-squares formulation in
which the residual is minimized iteratively in a greedy manner.
A complexity reduction method is also proposed to speed up
the optimization involved in each iteration. Simulations show
that the performance gap between the optimal full-baseband
and the existing OMP-based hybrid transceiver designs can be
substantially reduced by the proposed algorithm.
R EFERENCES
[1] Q. Li, H. Niu, A. Papathanassiou, and G. Wu, 5G network capacity:
Key elements and technologies, IEEE Veh. Technol. Mag., vol. 9, no. 1,
pp. 7178, Mar. 2014.
[2] F. Boccardi, R. W. Heath, Jr., A. Lozano, T. L. Marzetta, and P. Popovski,
Five disruptive technology directions for 5G, IEEE Commun. Mag.,
vol. 52, no. 2, pp. 7480, Feb. 2014.
[3] F. Rusek et al., Scaling up MIMO: Opportunities and challenges with
very large arrays, IEEE Signal Process. Mag., vol. 30, no. 1, pp. 4060,
Jan. 2013.
[4] S. Rangan, T. S. Rappaport, and E. Erkip, Millimeter-wave cellular wire-
less networks: Potentials and challenges, Proc. IEEE, vol. 102, no. 3,
pp. 366385, Mar. 2014.
[5] Z. Pi and F. Khan, An introduction to millimeter-wave mobile broadband
systems, IEEE Commun. Mag., vol. 49, no. 6, pp. 101107, Jun. 2011.
[6] J. Nsenga, A. Bourdoux, W. V. Thillo, V. Ramon, and F. Horlin, Joint
Tx/Rx analog linear transformation for maximizing the capacity at
60ghz, in Proc. IEEE ICC, Kyoto, Japan, Jun. 2011, pp. 15.
[7] O. E. Ayach, J. R. W. Heath, S. Abu-Surra, S. Rajagopal, and Z. Pi, Low
complexity precoding for large millimeter wave MIMO systems, in Proc.
IEEE ICC, Ottawa, Canada, Dec. 2012, pp. 37243729.
[8] O. E. Ayach, S. Rajagopal, S. Abu-Surra, Z. Pi, and J. R. W. Heath,
Spatially sparse precoding in millimeter wave MIMO systems, IEEE
Trans. Wireless Commun., vol. 13, no. 3, pp. 14991513, Mar. 2014.
[9] J. A. Tropp and S. J. Wright, Computational methods for sparse solution
of linear inverse problems, Proc. IEEE, vol. 98, no. 6, pp. 948958,
Jun. 2010.
[10] Y.-Y. Lee, C.-H. Wang, and Y.-H. Huang, A hybrid RF/baseband precod-
ing processor based on parallel-index-selection matrix-inversion-bypass
simultaneous orthogonal matching pursuit for millimeter wave MIMO
systems, IEEE Trans. Signal Process., vol. 63, no. 2, pp. 305317,
Jan. 2014.
[11] A. A. M. Saleh and R. A. Valenzuela, A statistical model for indoor mul-
tipath propagation, IEEE J. Sel. Areas Commun., vol. 5, no. 2, pp. 128
137, Feb. 1987.
[12] P. Stoica and A. Nehorai, On the concentrated stochastic likelihood func-
tion in array signal processing, Circuits, Syst. Signal Process., vol. 14,
no. 5, pp. 669674, May 1992.
[13] J. A. Nelder and R. Mead, A simplex method for function minimization,
Comput. J., vol. 7, no. 4, pp. 308313, 1965.
[14] R. A. Horn and C. R. Johnson, Matrix Analysis. New York, NY, USA:
Cambridge Univ. Press, 1985.
[15] T. Kim et al., Tens of gbps support with mmWave beamforming systems
for next generation communications, in Proc. IEEE Conf. GLOBECOM,
Atlanda, GA, USA, Dec. 2013, pp. 36853690.