Introduction
This project explores ways that morph images of faces into each other, a two-step process including warping the image shapes and cross-dissolving image colors. I explored methods including Delaunay triangulation, affine transformation, and inverse warping that can be used to produce more natural and believable face morphings.
Defining Correspondences
Approach
For this project, I used the face images of myself and my friend Eric. To warp the two faces into
the same shape, such that the facial features and face shapes in the two images
align with each other, I defined pairs of corresponding points on the two images using ginput
in a consistent order, and added the four corners of the image such that each image has 72
corresponding points.
To generate trianges within each set of corresponding points, I first calculated the mid-way shape (the
average value of the two point sets), and then generated a Delaunary triangulation for each image using
scipy.spatial.Delaunay.
Results
Jiayang (me)
Correspondences + Triangulation
Eric
Correspondences + Triangulation
From the results, we can see that Delaunary triangulation does not produce overly skinny triangles, and computing the midway shape helps in avoiding potential triangle deformations.
Computing the "Mid-way Face"
Approach
The mid-way face can be computed by warping both shapes into the mid-way shape calculated in the previous part, and then cross-dissolve the colors.
Warping triangles can be represented as multiplying the triangles with an affine transformation matrix A.
Therefore, I calculated A for each pair of triangles and took the inverse of the matrices, multiplying
the inverses with the target triangles (the mid-way shape) to get the coordinates that I needed to interpolate from the original
images. I first tried scipy.interpolate.griddata with linear interpolation but the efficiency is too
low. Therefore, I switched to nearest neighbor interpolation and reduced the processing time to under 2 minutes.
After warping both image shapes into the mid-way shape, the cross-dissolving step is just simply averaging the pixel intensities between the two warped images.
Results
Jiayang
Jiayang + Eric = Jeric
Eric
The Morph Sequence
Approach
Expanding the morph function in the last part, I modified it so it can take any two images as input, and
added the parameters warp_frac and dissolve_frac, bounded at [0, 1], that control
the extent of warping and cross-dissolving respectively, such that the starting frame has both parameters
set to 0, and the ending frame has both parameters set to 1.
To create an animated morphing sequence with 45 frames and 30 fps, I repeatedly called the morph function,
increasing both warp_frac and dissolve_frac by 1/45 every frame, and then merge
the computed 45 morphed images into a gif.
Results
Jeric morphing
The "Mean face" of a population
Approach
For this part of the project, I used the Danes dataset with annotated correspondences. I computed the average shape of the whole population, which is the mean value of each corresponding point across the whole population, and warped each of the faces in the dataset into the average shape. Then I cross-dissolved all warped images to compute the average face of the whole population.
Results
Example 1 (08-1f.bmp)
Warped
Example 2 (34-1m.bmp)
Warped
Example 3 (35-1f.bmp)
Warped
Example 4 (36-1m.bmp)
Warped
Average Face
Average Female Face
Average Male Face
Using the calculated average shape of the whole population, and my face with re-defined correspondences that matched the keypoints used in the Danes dataset, I warped my face into the average geometry, and also warped the average face into my geometry.
Jiayang to Average Danes
Average Danes to Jiayang
Caricatures: Extrapolating from the mean
Approach
I calculated caricatures of my face by extrapolating from the the average shape of the Danes dataset, setting
warp_frac to values greater than 1. I tried 1.5 and 3, and the result images are quite funny.
Results
warp_frac = 1.5
warp_frac = 3
Bells and Whistles
Gender Change
I used the average Chinese woman image off the web, and morphed it with my face image with various parameter settings. The result images below showed morphing just the shape (only warping), just the appearance (only cross-dissolving), and both.
Jiayang
Reference
Shape only
Appearance only
Both
PCA Basis
With the Danes dataset warped into the average shape, I computed the dataset's PCA basis for the face space. Below are the first 12 "eigenfaces" - the principal components with the most variance, sorted in decreasing order.
Eigenface 1
Eigenface 2
Eigenface 3
Eigenface 4
Eigenface 5
Eigenface 6
Eigenface 7
Eigenface 8
Eigenface 9
Eigenface 10
Eigenface 11
Eigenface 12
Then, I tried to reconstruct the faces in the warped Danes dataset with only the first 12 eigenfaces as the face space. The reconstructed faces are already similar to the original faces, and easily distinguishable, which supports the idea that the first few principal components contains the most variance, thus details that distinguish each face from other faces.
Example 1 (01-1m.bmp)
Reconstructed
Example 2 (07-1m.bmp)
Reconstructed
Example 3 (35-1f.bmp)
Reconstructed
I also tried transforming my own face image, without any modification, into this face space of 37 eigenfaces, and then reconstruct it. The result is kinda disturbing, and also does not look like my face at all, probably because my face is not in the dataset that originally established this face space, and that the background color of my face image is too different from the dark green background of the Danes dataset. I changed the background color of my face image to dark green, and the reconstructed result is significantly better.
Jiayang
Reconstructed
Jiayang with Background Change
Reconstructed with Background Change
A Different Morphing Algorithm Using PCA and Face Space
In this face space computed from the warped Danes dataset, I tried a different morphing algorithm that produce a smooth transition
between faces. Instead of the morphing algorithm that consists of a warping step and a cross-dissoling step that is discussed in the parts above,
my new morphing algorithm in this face space morph faces by changing the weights of each of the 37 eigenface during reconstructing the face image.
For example, I want to morph face A into face B, which are both in the Danes dataset, in 37 frames. The first frame is a reconstruction of face A
using all 37 eigenfaces and corresponding weights, which is represented by the V^T matrix of face A calculated using PCA. For all frames after this, the n-th frame
is a reconstructed face also using all 37 eigenfaces and weights, but have the first n weights of face A replaced with the first n
weights of face B. Therefore, the 37th frame will have all weights replaced with the weights (V^T) of face B, and the reconstructed face is exactly
face B. This process will successfully create a smooth morphing sequence from face A to face B.
Using this new morphing algorithm, I morphed between the first 5 faces of the warped Danes dataset, and merged the result image sequence into a gif loop.
The First 5 Faces in Danes Dataset
Produced Using a Different Morphing Algorithm
While the original morphing algorithm produces a smooth transition of shapes and colors, this new morphing algorithm produces a unique transition animation that consists of "ghosts" that are the weighted combinitions of eigenfaces. This algorithm also requires less computation as each frame is just a weighted linear combination of eigenfaces. An interesting thing to note is that the first few frames of each morphing sequence contains the most changes, while the frames toward the end of each morphing contains less changes and converge to the target face. This is another representation of the property of principal components and eigenfaces, that first few principal components contains the most variance.