Finite Difference Operator

Edges in an image can be coarsely detected where neighboring pixels have high differences in their intensity values, and the gradient magnitude image of an image shows exactly this information. To create the gradient magnitude image, I convolved finite difference operators D_x = np.array([[1], [-1]]) and D_y = np.array([[1, -1]]) onto the original image to get the partial derivatives of the image on x-axis and y-axis, and stack them to get the gradient magnitude image. I then binarized the result with the intensity threshold of 71, over the range of 0~255, to filter out the noise while retaining the real edges.

Derivative of Gaussian (DoG) Filter

While an appropriate threshold can filter out most of the noises, the result edge image is still relatively noisy, and the edges are also not smooth. To smooth the image, I applied a Gaussian filter (size = 9, sigma = 1.5) on the image that blurred it, thus diffusing the noises to neighboring pixels while smoothing the sharp edges. As the edges are smoothed, the threshold must also be lowered. Repeating the procedure in the previous part on this blurred image with the intensity threshold of 16 shows significant differences between the two result edge images. The edges in the new edge image are smoother and thicker, clearly showing the shape of the cameraman and the camera. The noises that presented in the lower half of the old edge image are almost all filtered out, leaving a cleaner and clearer edge image.

Since the convolution operation is associative, convolving the original image with the finite difference operators and the Gaussian filter separately should result in the same image as convolving the Gaussian filter with the finite difference operators first to create the Derivative of Gaussian (DoG) filter, and then convolve the original image with the DoG filter. I applied the DoG filter method on the same original image, and the result is shown below.

As expected, the two results are identical, apart from some neglectable differences caused by floating point precision errors.

Image "Sharpening"

Images with stronger high frequencies often look sharper, and this is the core theory of image "sharpening" - add the high frequencies of the image back to the original image as the fake sharpness. While low-pass filters, such as Gaussian filters, retain only the low frequencies of the image by blurring it and filtering out the details (high frequencies), the high frequencies of the same image can be calculated by subtracting the blurred image from the original image, such that all low frequencies are subtracted from the original image and only the high frequencies are left.

Rewrite the process as a formula, result = image + alpha(image - image * Gaussian) where the Gaussian filter is (size = 9, sigma = 3). Using the properties of convolution, the entire formula can be combined as a single convolution operation called the unsharp mask filter. To combine the formula, replace image with impulse filters since image * Impulse = image. The formula thus becomes result = image * (Impulse + alpha(Impulse - Gaussian)) where Impulse + alpha(Impulse - Gaussian) is the unsharp mask filter.

To confirm that the "sharpness" added using this method is not actually creating details, I picked a sharp image, blurred it using a low-pass Gaussian filter, and then used this method to try sharpening the blurred image. The sharpened image looks sharper than the blurred image, but is still blurry compared to the original image, since this method cannot recover the lost information caused by the low-pass filter filtering out the highest frequencies.

Hybrid Images

Following the SIGGRAPH 2006 paper by Oliva, Torralba, and Schyns, I used the same method to create hybrid images that change in interpretation when the distance between the image and the viewer changes, such that when the viewing distance is close, the image with high frequencies is dominant, and when the viewing distance is far, only the image with low frequencies can be seen. For each pair of images, the im_low is filtered with a Gaussian low-pass filter with parameters varying between pairs such that only the low frequencies are retained, and the im_high is filtered with the Impulse - Gaussian high-pass filter, also with parameters varying between pairs, such that only the high frequencies are retained. After both images are filtered, simply adding them produces the desired hybrid image.

This process of creating hybrid image can be illustrated through frequency analysis. The images below show the log magnitude of the Fourier transform of the two input images, the filtered images, and the hybrid image. For onigiri.png, the image is filtered with a low-pass filter so the FFT image has higher intensity around the center and the axes, and much lower intensity elsewhere. For birb.png, the image is filtered with a high-pass filter so the FFT image has lower intensity around the very center, and higher intensity elsewhere. The FFT image of the hybrid image is indeed a combination of the FFT images of the two filtered images.

However, when the shape of the two images are too similar, the hybrid effect will be less visible, and the hybrid image will stay the same independent of the viewing distance. Also, when the original high-frequency image is vibrant, the filtered image will lose the high-saturation color, leaving only a transparent shape. Below is one example of the said failure.

Gaussian and Laplacian Stacks

Gaussian stacks are Gaussian pyramids without the downsampling, such that images in the stack are all the same dimension as the original image. Instead, each level of the Gaussian stack can be created by convolving the original image with Gaussian filters where both size and sigma of the Gaussian filter increase through each level. In my case, the maximum level of stacks is 6, the initial Gaussian filter is (size = 3, sigma = 1), and both the size and sigma double for each level. The Laplacian stack can be calculated by taking the difference between every two consecutive levels of the Gaussian stack, such that laplacian_stack = gaussian_stack[:-1] - gaussian_stack[1:]. The last level of the Laplacian stack is the same as the last level of the Gaussian stack.

The following images are the levels 0, 2, and 4 of the Laplacian stack for both apple.jpeg and orange.jpeg. The full Gaussian stack and Laplacian stack are also presented below, stacked horizontally.

Multiresolution Blending (a.k.a. the oraple!)

Following the 1983 paper by Burt and Adelson, I used the same method of multi-resolution blending that blends two images seamlessly. This method first calculates the Laplacian stacks of the two input images, and the Gaussian stack of the mask image. Then, for each level (frequency band) i, the blended stack is calculated using im_blended[i] = mask_stack[i] * im1_stack[i] + (1 - mask_stack[i]) * im2_stack[i]. The final blended image is produced by summing all levels of the blending stack. For this project, I decided to limit each stack to 6 levels, and use the Gaussian filter with (size = 6, sigma = 2), with both the size and sigma double for each level.

The images below are the Laplacian stacks of my favorite result!