Pixel Recursive Super Resolution

Ryan Dahl Mohammad Norouzi Jonathon Shlens

Google Brain
{rld,mnorouzi,shlens}@google.com
arXiv:1702.00783v1 [cs.CV] 2 Feb 2017

Abstract
88 input 3232 samples ground truth
We present a pixel recursive super resolution model that
synthesizes realistic details into images while enhancing
their resolution. A low resolution image may correspond
to multiple plausible high resolution images, thus modeling
the super resolution process with a pixel independent con-
ditional model often results in averaging different details
hence blurry edges. By contrast, our model is able to repre-
sent a multimodal conditional distribution by properly mod-
eling the statistical dependencies among the high resolution
image pixels, conditioned on a low resolution input. We
employ a PixelCNN architecture to define a strong prior
over natural images and jointly optimize this prior with a
deep conditioning convolutional network. Human evalua-
tions indicate that samples from our proposed model look
more photo realistic than a strong L2 regression baseline.

1. Introduction
The problem of super resolution entails artificially en-
larging a low resolution photograph to recover a plausi-
ble high resolution version of it. When the zoom factor
is large, the input image does not contain all of the infor-
mation necessary to accurately construct a high resolution
image. Thus, the problem is underspecified and many plau-
sible high resolution images exist that match the low resolu-
tion input image. This problem is significant for improving
the state-of-the-art in super resolution, and more generally
for building better conditional generative models of images.
A super resolution model must account for the complex
variations of objects, viewpoints, illumination, and occlu-
sions, especially as the zoom factor increases. When some
details do not exist in the source image, the challenge lies
not only in deblurring an image, but also in generating
Work done as a member of the Google Brain Residency program
(g.co/brainresidency).
Figure 1: Illustration of our probabilistic pixel recursive
super resolution model trained end-to-end on a dataset of
celebrity faces. The left column shows 88 low resolution
inputs from the test set. The middle and last columns show
32 32 images as predicted by our model vs. the ground
truth. Our model incorporates strong face priors to synthe-
size realistic hair and skin details.
new image details that appear plausible to a human ob-
server. Generating realistic high resolution images is not
possible unless the model draws sharp edges and makes
hard decisions about the type of textures, shapes, and pat-
terns present at different parts of an image.
Imagine a low resolution image of a face, e.g., the 88
images depicted in the left column of Figure 1the details

1
of the hair and the skin are missing. Such details cannot be
faithfully recovered using simple interpolation techniques
such as linear or bicubic. However, by incorporating the
prior knowledge of the faces and their typical variations, an
artist is able to paint believable details. In this paper, we
show how a fully probabilistic model that is trained end-to-
end can play the role of such an artist by synthesizing 3232
face images depicted in the middle column of Figure 1. Our
super resolution model comprises two components that are
trained jointly: a conditioning network, and a prior net-
work. The conditioning network effectively maps a low
resolution image to a distribution over corresponding high
resolution images, while the prior models high resolution
details to make the outputs look more realistic. Our con-
ditioning network consists of a deep stack of ResNet [10]
blocks, while our prior network comprises a PixelCNN [28]
architecture.
We find that standard super resolution metrics such as
peak signal-to-noise ratio (pSNR) and structural similar-
ity (SSIM) fail to properly measure the quality of predic-
tions for an underspecified super resolution task. These
metrics prefer conservative blurry averages over more plau-
sible photo realistic details, as new fine details often do
not align exactly with the original details. Our evalua-
tion studies demonstrate that humans easily distinguish real
images from super resolution predictions when regression
techniques are used, but they have a harder time telling our
samples apart from real images.
2. Related work
Super resolution has a long history in computer vi-
sion [22]. Methods relying on interpolation [11] are easy
to implement and widely used, however these methods suf-
fer from a lack of expressivity since linear models cannot
express complex dependencies between the inputs and out-
puts. In practice, such methods often fail to adequately pre-
dict high frequency details leading to blurry high resolution
outputs.
Enhancing linear methods with rich image priors such
as sparsity [2] or Gaussian mixtures [35] have substantially
improved the quality of the methods; likewise, leveraging
low-level image statistics such as edge gradients improves
predictions [31, 26, 6, 12, 25, 17]. Much work has been
done on algorithms that search a database of patches and
combine them to create plausible high frequency details in
zoomed images [7, 13]. Recent patch-based work has fo-
cused on improving basic interpolation methods by building
a dictionary of pre-learned filters on images and selecting
the appropriate patches by an efficient hashing mechanism
[23]. Such dictionary methods have improved the inference
speed while being comparable to state-of-the-art.
Another approach for super resolution is to abandon in-
ference speed requirements and focus on constructing the
high resolution images at increasingly higher magnification
factors. Convolutional neural networks (CNNs) represent
an approach to the problem that avoids explicit dictionary
construction, but rather implicitly extracts multiple layers
of abstractions by learning layers of filter kernels. Dong et
al. [5] employed a three layer CNN with MSE loss. Kim et
al. [16] improved accuracy by increasing the depth to 20
layers and learning only the residuals between the high res-
olution image and an interpolated low resolution image.
Most recently, SRResNet [18] uses many ResNet blocks to
achieve state of the art pSNR and SSIM on standard super
resolution benchmarkswe employ a similar design for our
conditional network and catchall regression baseline.
Instead of using a per-pixel loss, Johnson et al.[14]
use Euclidean distance between activations of a pre-trained
CNN for models predictions vs. ground truth images. Us-
ing this so-called preceptual loss, they train feed-forward
networks for super resolution and style transfer. Bruna et
al. [3] also use perceptual loss to train a super resolution
network, but inference is done via gradient propagation to
the low-res input (e.g., [9]).
Ledig et al. [18] and Yu et al. [33] use GANs to cre-
ate compelling super resolution results showing the ability
of the model to predict plausible high frequency details.
Snderby et al. [15] also investigate GANs for super res-
olution using a learned affine transformation that ensures
the models only generate images that downscale back to the
low resolution inputs. Snderby et al. [15] also explore a
masked autoregressive model like PixelCNN [27] but with-
out the gated layers and using a mixture of gaussians in-
stead of a multinomial distribution. Denton et al. [4] use a
multi-scale adversarial network for image synthesis, but the
architecture also seems beneficial for super resolution.
PixelRNN and PixelCNN by Oord et al. [27, 28] are
probabilistic generative models that impose an order on im-
age pixels representing them as a long sequence. The proba-
bility of each pixel is then conditioned on the previous ones.
The gated PixelCNN obtained state of the art log-likelihood
scores on CIFAR-10 and MNIST, making it one of the most
competetive probabilistic generative models.
Since PixelCNN uses log-likelihood for training, the
model is highly penelized if negligible probability is as-
signed to any of the training examples. By contrast, GANs
only learn enough to fool a non-stationary discriminator.
One of their common failure cases is mode collapsing were
samples are not diverse enough [21]. Furthermore, GANs
require careful tuning of hyperparameters to ensure the dis-
criminator and generator are equally powerful and learn at
equal rates. PixelCNNs are more robust to hyperparame-
ter changes and usually have a nicely decaying loss curve.
Thus, we adopt PixelCNN for super resolution applications.
3. Probabilistic super resolution
We aim to learn a probabilistic super resolution model
that discerns the statistical dependencies between a high
resolution image and a corresponding low resolution im-
age. Let x and y denote a low resolution and a high resolu-
tion image, where y represents a ground-truth high res-
olution image. In order to learn a parametric model of
p (y | x), we exploit a large dataset of pairs of low res-
olution inputs and ground-truth high resolution outputs, de-
noted D {(x(i) , y(i) )}Ni=1 . One can easily collect such
a large dataset by starting from a set of high resolution im-
ages and lowering their resolution as much as needed. To
optimize the parameters of the conditional distribution p,
we maximize a conditional log-likelihood objective defined
as, X PixelCNN
O( | D) = log p(y | x) . (1)
(x,y )D
The key problem discussed in this paper is the exact form
of p(y | x) that enables efficient learning and inference,
while generating realistic non-blurry outputs. We first dis-
cuss pixel-independent models that assume that each out-
put pixel is generated with an independent stochastic pro-
cess given the input. We elaborate why these techniques
result in sub-optimal blurry super resolution results. Finally
we describe our pixel recursive super resolution model that
generates output pixels one at a time to enable modeling
the statistical dependencies between the output pixels using
PixelCNN [27, 28], and synthesizes sharp images from very
blurry input.
3.1. Pixel independent super resolution
The simplest form of a probabilistic super resolution
model assumes that the output pixels are conditionally inde-
pendence given the inputs. As such, the conditional distri-
bution of p(y | x) factorizes into a product of independent
pixel predictions. Suppose an RGB output y has M pixels
each with three color channels, i.e., y R3M . Then,
3M
X represented as,
log p(y | x) = log p(yi | x) . (2)
i=1
Two general forms of pixel prediction models have been ex-
plored in the literature: Gaussian and multinomial distribu-
tions to model continuous and discrete pixel values respec-
tively. In the Gaussian case,
1
log p(yi | x) = 2 kyi Ci (x)k22 log 2 2 , (3)
2
where Ci (x) denotes the ith element of a non-linear trans-
formation of x via a convolutional neural network. Ci (x)
is the estimated mean for the ith output pixel yi , and 2
denotes the variance. Often the variance is not learned, in
How the dataset was created
50%/50%
Model output
Cross-entropy
L2 Regression
Figure 2: Top: A cartoon of how the input and output pairs
for the toy dataset were created. Bottom: Example pre-
dictions for various algorithms trained on this dataset. The
pixel independent L2 regression and cross-entropy models
do not exhibit multimodal predictions. The PixelCNN out-
put is stochastic and multiple samples will place a digit in
either corner 50% of the time.
which case maximizing the conditional log-likelihood of (1)
reduces to minimizing the mean squared error (MSE) be-
tween yi and Ci (x) across the pixels and channels through-
out the dataset. Super resolution models based on MSE re-
gression (e.g., [5, 16, 18]) fall within this family of pixel
independent models, where the outputs of a neural network
parameterize a set of Gaussians with fixed bandwidth.
Alternatively, one could use a flexible multinomial dis-
tribution as the pixel prediction model, in which case the
ouput dimensions are discretized into K possible values
(e.g., K = 256) where yi {1, . . . , K}. The pixel pre-
diction model based on a multinomial softmax operator is
K
X
log p(yi = k | x) = wjkT Ci (x)log exp{wjvT Ci (x)} ,
v=1
(4)
where {wjk }3,K
j=1,k=1 denote the softmax weights for differ-
ent color channels and different discrete values.
3.2. Synthetic multimodal task
To demonstrate how the above pixel independent models
can fail at conditional image modeling, we created a syn-
thetic dataset that is explicitly multimodal. For many gen-
erative tasks like super resolution, colorization, and depth
estimation, models that are able to predict a mode without
averaging effects are desirable. For example, in coloriza-
tion, selecting a strong red or blue for a car is better than
selecting a sepia toned average of all of the colors of cars
that the model has been exposed to. In this synthetic task,
the input is an MNIST digit (1st row of Figure 2), and the
output is the same input digit but scaled and translated ei-
ther into the upper left corner or upper right corner (2nd and
3rd rows of Figure 2). The dataset has an equal ratio of up-
per left and upper right outputs, which we call the MNIST
corners dataset.
A convolutional network using per pixel squared error
loss (Figure 2, L2 Regression) produces two blurry fig-
ures. Replacing the continuous loss with a per-pixel cross-
entropy produces crisper images but also fails to capture the
stochastic bimodality because both digits are shown in both
corners (Figure 2, cross-entropy). In contrast, a model that
explicitly deals with multi-modality, PixelCNN stochasti-
cally predicts a digit in the upper-left or bottom-right cor-
ners but never predicts both digits simultaneously (Figure 2,
PixelCNN).
See Figure 5 for examples of our super resolution model
predicting different modes on a realistic dataset.
Any good generative model should be able to make sharp
single mode predictions and a dataset like this would be a
good starting point for any new models.
4. Pixel recursive super resolution
The main issue with the previous probabilistic models
(Equations (3) and (4)) for super resolution is the lack
of conditional dependency between super resolution pixels.
There are two general methods to model statistical correla-
tions between output pixels. One approach is to define the
conditional distribution of the output pixels jointly by ei-
ther a multivariate Gaussian mixture [36] or an undirected
graphical model such as conditional random fields [8]. With
these approaches one has to commit to a particular form of
statistical dependency between the output pixels, for which
inference can be computationally expensive. The second
approach that we follow in this work, is to factorize the con-
ditional distribution using chain rule as,
M
X pixel can take. Similarly, let Bi (y<i ) : Ri1 RK denote
log p(y | x) = log p(yi | x, y<i ) , (5)
i=1
where the generation of each output dimension is condi-
tioned on the input, previous output pixels, and the previous
channels of the same output pixel. For simplicity of exposi-
tion, we ignore different output channels in our derivations,
and use y<i to represent {y1 , . . . , yi1 }. The benefits of
this approach is that the exact form of the conditional de-
pendencies is flexible and the inference is straightforward.
Inspired by the PixelCNN model, we use a multinomial dis-
tribution to model discrete pixel values in Eq. (5). Alter-
natively, one could use an autoregressive prediction model
prior network!
(PixelCNN)
HR!
image
logits
HR!
+ image
conditioning!
network (CNN)!
Figure 3: The proposed super resolution network com-
prises a conditioning network and a prior network. The
conditioning network is a CNN that receives a low reso-
lution image as input and outputs logits predicting the con-
ditional log-probability of each high resolution (HR) image
pixel. The prior network, a PixelCNN [28], makes predic-
tions based on previous stochastic predictions (indicated by
dashed line). The models probability distribution is com-
puted as a softmax operator on top of the sum of the two
sets of logits from the prior and conditioning networks.
with Gaussian or Logistic (mixture) conditionals as pro-
posed in [24].
Our model, outlined in Figure 3, comprises two ma-
jor components that are fused together at a late stage and
trained jointly: (1) a conditioning network (2) a prior net-
work. The conditioning network is a pixel independent pre-
diction model that maps a low resolution image to a proba-
bilistic skeleton of a high resolution image, while the prior
network is supposed to add natural high resolution details
to make the outputs look more realistic.
Given an input x RL , let Ai (x) : RL RK denote
a conditioning network predicting a vector of logit values
corresponding to the K possible values that the ith output
a prior network predicting a vector of logit values for the ith
output pixel. Our probabilistic model predicts a distribution
over the ith output pixel by simply adding the two sets of
logits and applying a softmax operator on them,
p(yi | x, y<i ) = softmax(Ai (x) + Bi (y<i )) . (6)
To optimize the parameters of A and B jointly, we per-
form stochastic gradient ascent to maximize the conditional
log likelihood in (1). That is, we optimize a cross-entropy
loss between the models predictions in (6) and discrete
ground truth labels yi {1, . . . , K},
M 
X X T layer uses a 1 1 convolution to increase the channels to
O1 = 1[yi ] (Ai (x) + Bi (y<i
))
(x,y )D i=1
 nomial distribution over 256 possible sub-pixel values via a
lse(Ai (x) + Bi (y<i ))
where lse() is the log-sum-exp operator corresponding to
the log of the denominator of a softmax, and 1[k] denotes a
K-dimensional one-hot indicator vector with its k th dimen-
sion set to 1.
Our preliminery experiments indicate that models
trained with (7) tend to ignore the conditioning network
as the statistical correlation between a pixel and previous
high resolution pixels is stronger than its correlation with
low resolution inputs. To mitigate this issue, we include
an additional loss in our objective to enforce the condition-
ing network to be optimized. This additional loss measures
the cross-entropy between the conditioning networks pre-
dictions via softmax(Ai (x)) and ground truth labels. The
total loss that is optimized in our experiments is a sum of
two cross-entropy losses formulated as,
X M 
X T
O2 = 1[yi ] (2 Ai (x) + Bi (y<i
))
(x,y )D i=1

lse(Ai (x) + Bi (y<i )) lse(Ai (x)) .
Once the network is trained, sampling from the model
is straightforward. Using (6), starting at i = 1, first we
sample a high resolution pixel. Then, we proceed pixel by
pixel, feeding in the previously sampled pixel values back
into the network, and draw new high resolution pixels. The
three channels of each pixel are generated sequentially in
turn.
We additionally consider greedy decoding, where one al-
ways selects the pixel value with the largest probability and
sampling from a tempered softmax, where the concentra-
tion of a distribution p is adjusted by using a temperature
parameter > 0,
p bors search (NN). The architecture used for the regres-
p = .
kp k1
To control the concentration of our sampling distribution
p(yi | x, y<i ), it suffices to multiply the logits from A and
B by a parameter . Note that as goes towards , the dis-
tribution converges to the mode1 , and sampling converges to
greedy decoding.
4.1. Implementation details
The conditioning network is a feed-forward convolu-
tional neural network that takes an 88 RGB image through
1 We use a non-standard notion of temperature that represents
standard notation.
a series of ResNet [10] blocks and transpose convolution
layers while maintaining 32 channels throughout. The last
2563 and uses the resulting activations to predict a multi-
(7)
softmax operator.
,
This network provides the ability to absorb the global
structure of the image in the marginal probability distribu-
tion of the pixels. Due to the softmax layer it can capture
the rich intricacies of the high resolution distribution, but
we have no way to coherently sample from it. Sampling
sub-pixels independently will mix the assortment of distri-
butions.
The prior network provides a way to tie together the sub-
pixel distributions and allow us to take samples dependent
on each other. We use 20 gated PixelCNN layers with 32
channels at each layer. We leave conditioning until the late
stages of the network, where we add the pre-softmax ac-
tivations from the conditioning network and prior network
before computing the final joint softmax distribution.
Our model is built by using TensorFlow [1] and trained
across 8 GPUs with synchronous SGD updates. See ap-
pendix A for further details.
5. Experiments
(8)
We assess the effectiveness of the proposed pixel re-
cursive super resolution method on two datasets contain-
ing small faces and bedroom images. The first dataset is
a version of the CelebA dataset [19] composed of a set
of celebrity faces, which are cropped around the face. In
the second dataset LSUN Bedrooms [32], images are center
cropped. In both datasets we resize the images to 32 32
with bicubic interpolation and again to 8 8, constituting
the output and input pairs for training and evaluation. We
present representative super resolution examples on held
out test sets and report human evaluations of our predictions
in Table 1.
We compare our results with two baselines: a pixel inde-
pendent L2 regression (Regression) and a nearest neigh-
sion baseline is identical to the conditioning network used in
our model, consisting of several ResNet blocks and upsam-
pling convolutional layers, except that the baseline regres-
sion model outputs three channels and has a final tanh()
non-linearity instead of ReLU. The regression architecture
is similar in design to to SRResNet [18], which reports state
of the art scores in image similarity metrics. Furthermore,
we train the regression network to predict super resolution
residuals instead of the actual pixel values. The residu-
als are computed based on bicubic interpolation of the in-
put, and are known to work better to provide superior pre-
1
in the dictions [16]. The nearest neighbors baseline searches the
downsampled training set for the nearest example (using eu-
 Input Regression Ours G. Truth NN
Figure 4: Samples from the model trained on LSUN Bed-
rooms at 32 32.
clidean distance) and returns its high resolution counterpart.
5.1. Sampling
Sampling from the model multiple times results in dif-
ferent high resolution images for a given low resolution im-
age (Figure 5). A given model will identify many plausi-
ble high resolution images that correspond to a given lower
resolution image. Each one of these samples may contain
distinct qualitative features and each of these modes is con-
tained within the PixelCNN. Note that the differences be-
tween samples for the faces dataset are far less drastic than
seen in our synthetic dataset, where failure to cleanly pre-
dict modes meant complete failure.
The quality of samples is sensitive to the softmax tem-
perature. When the mode is sampled ( = ) at each sub-
pixel, the samples are of poor quality, they look smooth with
horizontal and vertical line artifacts. Sampling at = 1.0,
the exact probability given by the model, tend to be more
jittery with high frequency content. It seems in this case
there are multiple less certain trajectories and the samples
Figure 5: Left: low-res input. Right: Diversity of super
resolution samples at = 1.2.
jump back and forth between themperhaps this is allevi-
ated with more capacity and training time. Manually tuning
the softmax temperature was necessary to find good looking
samplesusually a value between 1.1 and 1.3 worked.
In Figure 6 are various test predictions with their nega-
tive log probability scores listed below each image. Smaller
scores means the model has assigned that image a larger
probability mass. The greedy, bicubic, and regression faces
are preferred by the model despite being worse quality.
This is probably because their smooth face-like structure
doesnt contradict the learned distributions. Yet sampling
with the proper softmax temperature nevertheless finds re-
alistic looking images.
 Ground Truth NN Bicubic Regression
2.85 2.74 1.76 2.34
2.96 2.71 1.82 2.17
2.76 2.63 1.80 2.35
Figure 6: Our model does not produce calibrated log-probabilities for the samples. Negative log-probabilities are reported
below each image. Note that the best log-probabilities arise from bicubic interpolation and greedy sampling even though the
images are poor quality.
5.2. Image similarity
Many methods exist for quantifying image similarity that
attempt to measure human perception judgements of simi-
larity [29, 30, 20]. We quantified the prediction accuracy
of our model compared to ground truth using pSNR and
MS-SSIM (Table 1). We found that our own visual assess-
ment of the predicted image quality did not correspond to
these image similarities metrics. For instance, bicubic in-
terpolation achieved relatively high metrics even though the
samples appeared quite poor. This result matches recent ob-
servations that suggest that pSNR and SSIM provide poor
judgements of super resolution quality [18, 14] when new
details are synthesized.
To ensure that samples do indeed correspond to the low-
resolution input, we measured how consistent the high res-
olution output image is with the low resolution input image
(Table 1, consistency). Specifically, we measured the L2
distance between the low-resolution input image and a bicu-
bic downsampled version of the high resolution estimate.
Lower L2 distances correspond to high resolutions that are
more similar to the original low resolution image. Note that
the nearest neighbor high resolution images are less consis-
tent even though we used a database of 3 million training
images to search for neighbors in the case of LSUN bed-
rooms. In contrast, the bicubic resampling and the Pixel-
CNN upsampling methods showed consistently better con-
sistency with the low resolution image. This indicates that
Greedy = 1.0 = 1.1 = 1.2
1.82 2.94 2.79 2.69
1.77 3.18 3.09 2.95
1.64 2.99 2.90 2.64
our samples do indeed correspond to the low-resolution in-
put.
5.3. Human study
We presented crowd sourced workers with two images: a
true image and the corresponding prediction from our vari-
ous models. Workers were asked Which image, would you
guess, is from a camera? Following the setup in Zhang et
al [34], we present each image for one second at a time
before allowing them to answer. Workers are started with
10 practice pairs during which they get feedback if they
choose correctly or not. The practice pairs not counted in
the results. After the practice pairs, they are shown 45 ad-
ditional pairs, 5 of which are golden questions designed to
test if the person is paying attention. The golden question
pits a bicubicly upsampled image (very blurry) against the
ground truth. Excluding the golden and practice questions,
we count fourty answers per session. Sessions in which they
missed any golden questions are thrown out. Workers were
only allowed to participate in any of our studies once. We
continued running sessions until fourty different different
workers were tested on each of the four algorithms.
We report in Table 1 the percent of the time users choose
an algorithms output over the ground truth counterpart.
Note that 50% would say that an algorithm perfectly con-
fused the subjects.
 Algorithm pSNR SSIM MS-SSIM Consistency % Fooled
Bicubic 28.92 0.84 0.76 0.006
NN 28.18 0.73 0.66 0.024
Regression 29.16 0.90 0.90 0.004 4.0 0.2
= 1.0 29.09 0.84 0.86 0.008 11.0 0.1
= 1.1 29.08 0.84 0.85 0.008 10.4 0.2
= 1.2 29.08 0.84 0.86 0.008 10.2 0.1
Bicubic 28.94 0.70 0.70 0.002
NN 28.15 0.49 0.45 0.040
Regression 28.87 0.74 0.75 0.003 2.1 0.1
= 1.0 28.92 0.58 0.60 0.016 17.7 0.4
= 1.1 28.92 0.59 0.59 0.017 22.4 0.3
= 1.2 28.93 0.59 0.58 0.018 27.9 0.3

Table 1: Top: Results on the cropped CelebA test dataset at 3232 magnified from 88. Bottom: LSUN bedrooms. pSNR,
SSIM, and MS-SSIM measure image similarity between samples and the ground truth. Consistency lists the MSE between
the input low-res image and downsampled samples on a [0, 1] scale. % Fooled reports how often the algorithms samples
fooled a human in a crowd sourced study; 50% would be perfectly confused.

6. Conclusion [3] J. Bruna, P. Sprechmann, and Y. LeCun. Super-resolution

As in many image transformation tasks, the central prob-
lem of super resolution is in hallucinating sharp details by
choosing a mode of the output distribution. We explored
this underspecified problem using small images, demon-
strating that even the smallest 88 images can be enlarged
to sharp 3232 images. We introduced a toy dataset with
a small number of explicit modes to demonstrate the failure
cases of two common pixel independent likelihood models.
In the presented model, the conditioning network gets us
most of the way towards predicting a high-resolution im-
age, but the outputs are blurry where the model is uncer-
tain. Combining the conditioning network with a PixelCNN
model provides a strong prior over the output pixels, allow-
ing the model to generate crisp predications. Our human
evaluations indicate that samples from our model on aver-
age look more photo realistic than a strong regression based
conditioning network alone.
References
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghe-
mawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,
R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,
J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
V. Vanhoucke, V. Vasudevan, F. Viegas, O. Vinyals, P. War-
den, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor-
Flow: Large-scale machine learning on heterogeneous sys-
tems, 2015. Software available from tensorflow.org. 5
[2] M. Aharon, M. Elad, and A. Bruckstein. Svdd: An algorithm
for designing overcomplete dictionaries for sparse represen-
tation. Trans. Sig. Proc., 54(11):43114322, Nov. 2006. 2
with deep convolutional sufficient statistics. CoRR,
abs/1511.05666, 2015. 2
[4] E. L. Denton, S. Chintala, A. Szlam, and R. Fergus. Deep
generative image models using a laplacian pyramid of adver-
sarial networks. NIPS, 2015. 2
[5] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-
resolution using deep convolutional networks. CoRR,
abs/1501.00092, 2015. 2, 3
[6] R. Fattal. Image upsampling via imposed edge statistics.
ACM Trans. Graph., 26(3), July 2007. 2
[7] W. T. Freeman, T. R. Jones, and E. C. Pasztor. Example-
based super-resolution. IEEE Computer graphics and Appli-
cations, 2002. 2
[8] W. T. Freeman and E. C. Pasztor. Markov networks for super-
resolution. In CISS, 2000. 4
[9] L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm
of artistic style. CoRR, abs/1508.06576, 2015. 2
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. CVPR, 2015. 2, 5
[11] H. Hou and H. Andrews. Cubic splines for image interpola-
tion and digital filtering. Acoustics, Speech and Signal Pro-
cessing, IEEE Transactions on, 26(6):508517, Jan. 2003.
2
[12] J. Huang and D. Mumford. Statistics of natural images and
models. In Computer Vision and Pattern Recognition, 1999.
IEEE Computer Society Conference on., volume 1. IEEE,
1999. 2
[13] J.-B. Huang, A. Singh, and N. Ahuja. Single image super-
resolution from transformed self-exemplars. In IEEE Con-
ference on Computer Vision and Pattern Recognition), 2015.
2
[14] J. Johnson, A. Alahi, and F. Li. Perceptual losses
for real-time style transfer and super-resolution. CoRR,
abs/1603.08155, 2016. 2, 7
 Ours Ground Truth Ours Ground Truth age super-resolution using a generative adversarial network.
arXiv:1609.04802, 2016. 2, 3, 5, 7
[19] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face
attributes in the wild. In Proceedings of International Con-
ference on Computer Vision (ICCV), 2015. 5
[20] K. Ma, Q. Wu, Z. Wang, Z. Duanmu, H. Yong, H. Li, and
23/40 = 57% 34/40 = 85% L. Zhang. Group mad competition - a new methodology
to compare objective image quality models. In The IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), June 2016. 6
[21] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein. Unrolled
generative adversarial networks. CoRR, abs/1611.02163,
17/40 = 42% 30/40 = 75% 2016. 2
[22] K. Nasrollahi and T. B. Moeslund. Super-resolution: A com-
prehensive survey. Mach. Vision Appl., 25(6):14231468,
Aug. 2014. 2
[23] Y. Romano, J. Isidoro, and P. Milanfar. RAISR: rapid and ac-
curate image super resolution. CoRR, abs/1606.01299, 2016.
16/40 = 40% 26/40 = 65%
2
[24] T. Salimans, A. Karpathy, X. Chen, D. P. Kingma, and Y. Bu-
latov. Pixelcnn++: A pixelcnn implementation with dis-
cretized logistic mixture likelihood and other modifications.
under review at ICLR 2017. 4
1/40 = 2% 3/40 = 7% [25] Q. Shan, Z. Li, J. Jia, and C.-K. Tang. Fast image/video
upsampling. ACM Transactions on Graphics (TOG),
27(5):153, 2008. 2
[26] J. Sun, Z. Xu, and H.-Y. Shum. Image super-resolution us-
ing gradient profile prior. In Computer Vision and Pattern
Recognition, 2008. CVPR 2008. IEEE Conference on, pages
1/40 = 2% 3/40 = 7% 18. IEEE, 2008. 2
[27] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu.
Pixel recurrent neural networks. ICML, 2016. 2, 3
[28] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt,
A. Graves, and K. Kavukcuoglu. Conditional image genera-
tion with pixelcnn decoders. NIPS, 2016. 2, 3, 4
1/40 = 2% 4/40 = 1% [29] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simon-
celli. Image quality assessment: from error visibility to
Figure 7: The best and worst rated images in the human structural similarity. IEEE transactions on image process-
study. The fractions below the images denote how many ing, 13(4):600612, 2004. 6
times a person choose that image over the ground truth. [30] Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multiscale
See the supplementary material for more images used in the structural similarity for image quality assessment. In Sig-
study. nals, Systems and Computers, 2004. Conference Record of
the Thirty-Seventh Asilomar Conference on, volume 2, pages
13981402. Ieee, 2004. 6
[15] C. Kaae Snderby, J. Caballero, L. Theis, W. Shi, and [31] C. Y. Yang, S. Liu, and M. H. Yang. Structured face hallu-
F. Huszar. Amortised MAP Inference for Image Super- cination. In 2013 IEEE Conference on Computer Vision and
resolution. ArXiv e-prints, Oct. 2016. 2 Pattern Recognition, pages 10991106, June 2013. 2
[16] J. Kim, J. K. Lee, and K. M. Lee. Accurate image super- [32] F. Yu, Y. Zhang, S. Song, A. Seff, and J. Xiao. Lsun: Con-
resolution using very deep convolutional networks. CoRR, struction of a large-scale image dataset using deep learning
abs/1511.04587, 2015. 2, 3, 5 with humans in the loop. arXiv preprint arXiv:1506.03365,
[17] K. I. Kim and Y. Kwon. Single-image super-resolution using 2015. 5
sparse regression and natural image prior. IEEE Transactions [33] X. Yu and F. Porikli. Ultra-Resolving Face Images by Dis-
on Pattern Analysis and Machine Intelligence, 32(6):1127 criminative Generative Networks, pages 318333. Springer
1133, 2010. 2 International Publishing, Cham, 2016. 2
[18] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Aitken, A. Te- [34] R. Zhang, P. Isola, and A. A. Efros. Colorful image coloriza-
jani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single im- tion. ECCV, 2016. 7
[35] D. Zoran and Y. Weiss. From learning models of natural
image patches to whole image restoration. In Proceedings
of the 2011 International Conference on Computer Vision,
ICCV 11, pages 479486, Washington, DC, USA, 2011.
IEEE Computer Society. 2
[36] D. Zoran and Y. Weiss. From learning models of natural
image patches to whole image restoration. In CVPR, 2011.
4
A.

Operation Kernel Strides Feature maps

Conditional network 8 8 3 input
B ResNet block 33 1 32
Transposed Convolution 33 2 32
B ResNet block 33 1 32
Transposed Convolution 33 2 32
B ResNet block 33 1 32
Convolution 11 1 3 256
PixelCNN network 32 32 3 input
Masked Convolution 77 1 64
20 Gated Convolution Layers 55 1 64
Masked Convolution 11 1 1024
Masked Convolution 11 1 3 256
Optimizer RMSProp (decay=0.95, momentum=0.9, epsilon=1e-8)
Batch size 32
Iterations 2,000,000 for Bedrooms, 200,000 for faces.
Learning Rate 0.0004 and divide by 2 every 500000 steps.
Weight, bias initialization truncated normal (stddev=0.1), Constant(0)

Table 2: Hyperparameters used for both datasets. For LSUN bedrooms B = 10 and for the cropped CelebA faces B = 6.
B. LSUN bedrooms samples

Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN

Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
C. Cropped CelebA faces

Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN

Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN
Input Bicubic Regression = 1.0 = 1.1 = 1.2 Truth NN