Activity 14 - Pattern Recognition

This activity aims to recognize objects using their differenct characteristics. Again we will be using our skills in image processing to be able to attain this objective. 

The first thing i thought when I knew this activity, is what object will I classify. So after taking a lot of time thinking of interesting objects to classify, i came up with the following image.

Figure 1. Objects to be recognized.

I bought 4 different candies XO, Maxx, Pochi and Mentos. To further view their differences, the following is an image of the candies themselves.

Figure 2. Different candies, XO (top), Maxx(left),Pochi(right),Mentos(bottom)

After reading the handout it says, I need to take 20 pictures of each type for the training set. I can't afford to buy that much candy, and I don't eat a lot of candies myself. So yeah let me think of another object.

-----------------------------------------------------------------------------------------------------------

OK, i changed my mind with the candies, Now i will classify the contents of the following image. 

Figure 3. Image of Dingdong, mixed snack of chips,nuts and curls.

Activity 13 - Image Compression using Principal Components Analysis

This activity aims to compressed an image using a method called Principal Component Analysis.

Like we previously discussed in Activity 4, image sizes is one of the property of an image that is always looked into. Consequently, a much more detailed image produced a much bigger file sizes. 

In this activity we will try to compressed the image using a particular method without the expense of degrading the quality of the image as well as its dimensions. We will also look into how the image is compressed.

What is PCA?

For a sample image compression, I used the following image taken from [2]

Figure 1. Candies image [2]

This jpeg image has a 700x438 dimensions which consists of a wide variety of colors. This image is converted to the following gray-scaled image using the function rgb2gray()

Figure 2. Gray-scaled image of Figure 1.

With the same dimensions, this gray-scaled image has a file size = 77.6 KB

Using the PCA we obtained reconstructed images with varying PC,

Figure 3. PC =5, 60.4KB

Figure 4. PC = 10, 68.7KB

Figure 5. PC = 25, 75 KB

Figure 6. PC = 50 77.5 KB

Figure 7. Comparison of Gray-scaled(left) image(77.6KB) and PC =100 reconstructed (right) image (76.3KB)

The results showed a slightly lower file size. See that there almost no visible difference unlike the previous samples. I think the objective of this activity is sufficed. Maybe next time i could try a much bigger image, so we can really see a significant change in the size due to compression. I guess a 77.6KB image is maybe a small file for the grayscale of a very colorful object.

References:

[1]AP 186 Handouts. Activity 13 - Image Compression using Principal Components Analysis. Maricor Soriano 2013

[2]http://good-wallpapers.com/misc/12162
------------------------------------------------------------------------------------------------------
Even though I understand the methods, I really dont know how to explain it here, sorry for the poor discussion. I think i can have a grade of  6 or 7 for producing the desired output. I guess I'll go back to this activity next time to improve the discussion.

Activity 12 - Playing Notes by Image Processing

The activity aims to be able to play a musical scoresheet using image processing. 

With the built-in function of scilab to produce sound, the only problem at hand is how we will be reading the scoresheet. We will be using all our skills in image processing to be able to identify notes and and we will be able to listen to it.

First lets crop the staffs so that the clefs are removed and only the staff and the notes are left. Actually we can also remove the clefs by morphological operations, but you can do this for convenience. Also later, I want you to recognize what song did I used to finish this activity. 

If you still dont know a lot about morphological operations, you better read my previous blogs regarding this. I wont go in to detail on the cleaning and enhancement of the image of the scoresheet.

So first lets try this sample scoresheet. The first thing to do is to eliminate all unnecessary symbols so that we can be left by blobs of notes. So using erosion which is implemented in the following code and images. I obtained the following results.

reading the scoresheet

clear

a = imread('C:\Users\Jazz\Desktop\Notes\1st staf.png');

imshow(a)

Figure 1. Sample staff , cropped from a bigger and complete scoresheet.

im = im2bw(~a,0.6);

imshow(im);

SEvline = CreateStructureElement('vertical_line', 2);

SEhline = CreateStructureElement('horizontal_line', 3);

SEcircle = CreateStructureElement('circle',5);

b = ErodeImage(im,SEvline);

imshow(b);

c = ErodeImage(b,SEhline);

imshow(c);

imwrite(c,'C:\Users\Jazz\Desktop\Notes\notes.png')

Figure 2. Cleaned image of the sample staff.

Figure 2 shows the image that already can be subjected to analysis. We can see that with these simplification we can still identify the kind of note it is, and its height related to its pixel coordinates.

Consequently, if we apply the SearchBlob function, which is thoroughly discussed in the previous blog (Activity 11), I can detect 9 blobs instead of 8. As you can see, obviously, this error comes from the half note (6th blob from left), which can be detected to be 2 blobs.

To eliminate this error we again use morphological operations that will enhance the image (especially the half note). So I implemented the following code utilizing the CloseImage function. Again if you are at lost of the morphological operations, please see my previous blogs. (Activity 10, Morphological Operations and Activity 11 Binary Operations)

d = CloseImage(c,SEcircle);

imshow(d);

d = imwrite(d,'C:\Users\Jazz\Desktop\Notes\distinctnotes.png')

Figure 3 Enhanced image of Figure 2

Right now, we already have a clear 8 blobs for our image while preserving the difference of the eight note and the half note.

Since we know how to call each blobs and we can differentiate its area, we can easily identify the corresponding note equivalent of each blobs. So obviously, we have 7 eight notes and 1 half note(6th blob).

 //AnalyzeBlobs
IsCalculated = CreateFeatureStruct(%f);
IsCalculated.Centroid = %t

BlobStatistics = AnalyzeBlobs(Blobs,IsCalculated);

Area = size(find(Blobs==6),2);
disp(Area);

xpixel = zeros(max(blobmax),1);
ypixel = zeros(max(blobmax),1);
arealist = zeros(max(blobmax),1);

for i=1:blobmax
    xpixel(i,1) = BlobStatistics(i).Centroid(1);
    ypixel(i,1) = BlobStatistics(i).Centroid(2);
    arealist(i,1) = size(find(Blobs==i),2);

end

Lastly, we need to be able to know what note it is, I sampled the range of which the notes can be found and assigned it to the its equivalent note. With the previous code, i also identify and obtained  the values of the centroids of each blobs. This centroids will be the identifier of the level or the pitch of the corresponding blob.

Figure 5. Sampled staff with the pixel range of different notes.

Figure 5, shows how we will know the pitch of the note. If its a C, D, or E, etc. So for this image

the red pixel = D which is around 65-68 pixels, E which is around 60-64 pixels, F which is around 54-58 and G = 49-42 pixels.

As you may notice, as we go higher  the staff (which means higher notes), the values of the centroids is decreasing. This is true because the y-pixel coordinate starts from 0 at the top.

That's it!!!. I guess all the problem has been covered. So as a summary, we identify the notes with the blob areas, while we identify the pitch of the note, with the blob centroids. 

All that is left now is to convert it to sound!. With the built-in function of scilab to produce sound, and save it, I implemented this code.

Convertion to Music 

//MusicPart

function n = note(f, t)

n = sin (2*%pi*f*t);

endfunction;

C = 261.63*2;

D = 293.66*2;

E = 329.63*2;

F = 349.23*2;

G = 392.00*2;

A = 440.00*2;

B = 493.88*2;

C1 = 523.25*2;

D1 = 587.33*2;

E1 = 659.26*2;

F1 = 698.46*2;

G1 = 783.99*2;

ypixel(find(ypixel >49 & ypixel< 52)) = G;

ypixel(find(ypixel >43 & ypixel< 47)) = A;

ypixel(find(ypixel >38 & ypixel< 41)) = B;

ypixel(find(ypixel >33 & ypixel< 37)) = C1;

ypixel(find(ypixel >31 & ypixel< 28)) = D1;

ypixel(find(ypixel >26 & ypixel< 23)) = E1;

ypixel(find(ypixel >21 & ypixel< 18)) = F1;

ypixel(find(ypixel >16 & ypixel< 11)) = G1;

arealist(find(arealist<70)) = 1.0;

arealist(find(arealist>70)) = 0.5;

BlobDetails = [xpixel,ypixel,arealist]

notelist = []

for j=1:length(blobmax)

    notelist($+1,:) = note(BlobDetails(j,2),t);

    end

s = matrix(notelist, 1, length(notelist));

With these code, I obtained all the necessary details, with their corresponding conversion in the musical scale.

BlobDetails

xpixel coordinate(order)   note(λ)     t (sec)
14.03370786516854 784.0 0.5
93.04494382022472 784.0 0.5
188.72826086956522 880.0 0.5
298.13793103448273 784.0 0.5
406.95652173913044 1046.5 0.5
532.9666666666667 987.76 1.0
718.7176470588236 784.0 0.5
797.9333333333333 784.0 0.5

The x pixel coordinate just tells you which note will be played first, basically, its already in order. The second column is the Note frequency which is obtained from the reference [2], and the third column is the distinction of the half note and the eight note, so I assigned a value of 0.5 for the eight notes and 1 second for the half note.

So yeah! Were done!! so try to identify the sound I produced. Im sure your familiar with it!. Anyway, i save this using the following code and uploaded it on the web, so you can play it.

savewave('C:\Users\Jazz\Desktop\Notes\happybirthday.mp3',s1)

ENJOY!
Here is the link of the sound, please comment if you have problems accessing the sound.
https://soundcloud.com/jazzlisten-1/ap-186
--------
I forgot to say this, but I have done this for all the staff on the original scoresheet, so its a complete tune, I hope it resembles the original. Hahahaha

References:
[1] AP 186 Handouts. Activity 12 - Playing Notes by Image Processing. Maricor Soriano 2013
[2] http://www.phy.mtu.edu/~suits/notefreqs.html
-----------------------------------------------------------------------------------------------------

I will give myself a 11/10 if you can identify the sound i produced :). I guess that's the main point of this activity.

Activity 11 - Binary Operations

This activity aims to used our knowledge in morphological and binary operations previously discussed in the last activity.

Figure 1. Circles002

Figure 2. Circles with cancer

I. Segment by Thresholding 

First I cut the 823x658 image of Figure 1, to 12 subimages that are overlapping at some parts using the following code

im = imread('C:\Users\Jazz\Desktop\Binary Operations\Circles002.jpg');

//Part 1

Circle11 = im(1:256,1:256);

Circle12 = im(1:256,201:456);

Circle13 = im(1:256,403:658);

Circle21 = im(189:444,1:256);

Circle22 = im(189:444,201:456);

Circle23 = im(189:444,403:658);

Circle31 = im(378:633,1:256);

Circle32 = im(378:633,201:456);

Circle33 = im(378:633,403:658);

Circle41 = im(568:823,1:256);

Circle42 = im(568:823,201:456);

Circle43 = im(568:823,403:658);

imwrite(Circle11,'C:\Users\Jazz\Desktop\Binary Operations\C1_01.jpg');

imwrite(Circle12,'C:\Users\Jazz\Desktop\Binary Operations\C1_02.jpg');

imwrite(Circle13,'C:\Users\Jazz\Desktop\Binary Operations\C1_03.jpg');

imwrite(Circle21,'C:\Users\Jazz\Desktop\Binary Operations\C1_04.jpg');

imwrite(Circle22,'C:\Users\Jazz\Desktop\Binary Operations\C1_05.jpg');

imwrite(Circle23,'C:\Users\Jazz\Desktop\Binary Operations\C1_06.jpg');

imwrite(Circle31,'C:\Users\Jazz\Desktop\Binary Operations\C1_07.jpg');

imwrite(Circle32,'C:\Users\Jazz\Desktop\Binary Operations\C1_08.jpg');

imwrite(Circle33,'C:\Users\Jazz\Desktop\Binary Operations\C1_09.jpg');

imwrite(Circle41,'C:\Users\Jazz\Desktop\Binary Operations\C1_10.jpg')

imwrite(Circle42,'C:\Users\Jazz\Desktop\Binary Operations\C1_11.jpg')

imwrite(Circle43,'C:\Users\Jazz\Desktop\Binary Operations\C1_12.jpg')

The code resulted to this following images. Take note that some parts of each image may be overlapping with the other. The image are labeled as Circle11, Circle12, Circle13, Circle21,Circle22, and so on. The number index represents their position in the image(like a matrix).

Figure 3. 12 subimages of Circles002.png labeled from C1_01 to C1_12.

to segment it by threshold, there is a built-in function in Image Processing Design(IPD) Module of Scilab under AnalyzeBlobs, that can be used. The function is depicted by the following code implementing threshold segmentation.

Threshold = CalculateOtsuThreshold(Circle12);

SegmentedImage = SegmentByThreshold(Circle12, Threshold+10);

imshow(SegmentedImage)

I picked Circle12 as the first sample and the code resulted to the following image. Take note that I segmented it slightly higher to its threshold to produce this better result.

Figure 4. Thresholding using the built-in function SegmentByThreshold()

Alternative to the built-in function of calculating the threshold is by studying first the image's histogram to find the grayscale threshold that will separate the background from the cells. 

Figure 5. Histogram plot of Circle12 with 10 bins

Using this image, the average threshold of the image is around 200, which almost equal to the calculated threshold of the built-in function  CalculateOtsuThreshold, which is 193. It is also advised to us that we would segment the image slightly higher than the threshold value.

Another easier method of thresholding is by using im2bw function of SIVP, that is already used to my previous activities.

II. Morphological Operations

In this part I tried to enhance the binary image so that it could be analyzed easier for the later parts. It consists of three morphological operations such as Closing, Opening and TopHat, where there is a very convenient built-in function again in IPD. 

The enhancement is done by using the following code (varying the morphological function as well as the radius of the SE which is a circle). 

//Part3 Morphological Operations

SE = CreateStructureElement('circle', 15)

Blobs = TopHat(SegmentedImage, SE);

imwrite(Blobs,'C:\Users\Jazz\Desktop\Binary Operations\tophat15.jpg')

imshow(Blobs)

Figure 6. Closing of image Circle12 using a circle SE of radius 3(left) and radius 15(right)

Figure 7.Opening of image Circle12 using a circle SE of radius 3(left) and radius 15(right)

Figure 8. TopHat of image Circle12 using a circle SE of radius 3(left) and radius 15(right)

We could clearly see the distinction of the different morphological operations. For example, the CloseImage function with a larger radius connects all the circles while in contrast it is all eroded in the OpenImage function. The TopHat on the other hand preserved a lot of details in the image. Now to further understand this morphological operations,As another requirement, we are asked to research these functions.

CloseImage -Performs morphological closing (dilation followed by erosion).[2]

OpenImage - Performs morphological opening (erosion followed by dilation).[2]

TopHat - Performs morphological "top hat" operation, returning the image minus the morphological opening of the image (erosion followed by dilation).[2]

You may visit the reference for a lot more different morphological operations. The only problem is they are in MATLAB, as built-in functions, but its really helpful to learn other morphological operations.

III. Area Calculations

In this part I, repicked another subimage to be studied; which is the 4th row and 1st column subimage denoted by Circle41. 

Using the everything we did previously and enhancing the image using the OpenImage function with the correct SE circle radius, I produced the following. 

Figure 9. Cleaned image using morphological operations

Right now what we lack is the analyis of this circles in the image. But surprisingly, these is all covered under one function in IPD, which is the SearchBlobs function, and there are still some more related function to this.

Blobindexed = SearchBlobs(Blobs);

imshow(mat2gray(Blobindexed));

imwrite(mat2gray(Blobindexed),'C:\Users\Jazz\Desktop\Binary Operations\blobs.jpg')

Figure 10. Blob image using built-in function SearchBlobs in IPD

With the previous code, we can now specify which blobs do we want to analyze. Virtually, after doing the previous code, the blobs where numbered accordingly. All we need to do is to call them by their number. And this is implemented by the following code, which also displays their Area.

Area = size(find(Blobs==2),2);

disp(Area)

Given these I averaged the Area and get their standard deviation using the simple loop code.

blobmax = max(Blobindexed);

disp(blobmax);

arealist = zeros(max(blobmax),1);

for i=1:blobmax

    Area = size(find(Blobs == i),2)

    arealist(i,1) = Area;

end

disp(mean(Area));

disp(std2(Area));

The results displayed that the average area = 554.1667, with the standard deviation of 13.19084. 

Take note that this is only true to image Circle41.

IV. Identifying Abnormal Cells.

Lastly to achieve the main objective of this activity, we analyze the image in Figure 2 and we do everything we've done previously.

Figure 11. Cleaned Image (upper) and SearchBlob image(lower) of Figure 2.

Knowing the average size of the a normal cell, we could separate the abnormal cells 

Figure 11. Identified abnormal cells

References
[1][1]AP 186 Handouts. Activity 11 - Binary Operations. Maricor Soriano 2013

[2] http://www.mathworks.com/help/images/ref/bwmorph.html

-----------------------------------------------------------------------------------------------
Self-Grade

I think I deserve a 10/10 after finishing and understanding all the necessary results.

Activity 10 - Morphological Operations

This is one of my favorite activity, the morphological operations

There are a lot of morphological operations and it is really convenient to know how to implement them or how they work. Morphological operations are one of the techniques in image processing to enhance our remove details of an image. This is applied especially to binary image which are easy to understand and implemented.


Table 1. Morphological Operations Supported by Matlab lifted from [2]

Operation	Description
'bothat'	Performs the morphological "bottom hat" operation, returning the image minus the morphological closing of the image (dilation followed by erosion).
'branchpoints'	Find branch points of skeleton. For example: 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 becomes 0 0 0 0 0 1 1 1 1 1 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 Note: To find branch points, the image must be skeletonized. To create a skeletonized image, usebwmorph(BW,'skel').
'bridge'	Bridges unconnected pixels, that is, sets 0-valued pixels to 1 if they have two nonzero neighbors that are not connected. For example: 1 0 0 1 1 0 1 0 1 becomes 1 1 1 0 0 1 0 1 1
'clean'	Removes isolated pixels (individual 1s that are surrounded by 0s), such as the center pixel in this pattern. 0 0 0 0 1 0 0 0 0
'close'	Performs morphological closing (dilation followed by erosion).
'diag'	Uses diagonal fill to eliminate 8-connectivity of the background. For example: 0 1 0 0 1 0 1 0 0 becomes 1 1 0 0 0 0 0 0 0
'dilate'	Performs dilation using the structuring element ones(3).
'endpoints'	Finds end points of skeleton. For example: 1 0 0 0 1 0 0 0 0 1 0 0 becomes 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 Note: To find end points, the image must be skeletonized. To create a skeletonized image, usebwmorph(BW,'skel').
'erode'	Performs erosion using the structuring element ones(3).
'fill'	Fills isolated interior pixels (individual 0s that are surrounded by 1s), such as the center pixel in this pattern. 1 1 1 1 0 1 1 1 1
'hbreak'	Removes H-connected pixels. For example: 1 1 1 1 1 1 0 1 0 becomes 0 0 0 1 1 1 1 1 1
'majority'	Sets a pixel to 1 if five or more pixels in its 3-by-3 neighborhood are 1s; otherwise, it sets the pixel to 0.
'open'	Performs morphological opening (erosion followed by dilation).
'remove'	Removes interior pixels. This option sets a pixel to 0 if all its 4-connected neighbors are 1, thus leaving only the boundary pixels on.
'shrink'	With n = Inf, shrinks objects to points. It removes pixels so that objects without holes shrink to a point, and objects with holes shrink to a connected ring halfway between each hole and the outer boundary. This option preserves the Euler number.
'skel'	With n = Inf, removes pixels on the boundaries of objects but does not allow objects to break apart. The pixels remaining make up the image skeleton. This option preserves the Euler number.
'spur'	Removes spur pixels. For example: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 becomes 0 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 1 1 0 0
'thicken'	With n = Inf, thickens objects by adding pixels to the exterior of objects until doing so would result in previously unconnected objects being 8-connected. This option preserves the Euler number.
'thin'	With n = Inf, thins objects to lines. It removes pixels so that an object without holes shrinks to a minimally connected stroke, and an object with holes shrinks to a connected ring halfway between each hole and the outer boundary. This option preserves the Euler number. See Algorithms for more detail.
'tophat'	Performs morphological "top hat" operation, returning the image minus the morphological opening of the image (erosion followed by dilation).

In this activity, we will only observe erosion and dilation which is the very basic of morphological operations. The previous table, is worthwhile reading for you to know that there are a lot of morphological operation that you can explore more than this activity.

The results and the images in this activity is very small, ranging from 4-50 pixels, I really dont know how I will present it here, systematically and to make it presentable. Maybe its just that there are so many results. Anyway here are the results by implementing morphological operation to a predescribed binary images as well as structure elements.

Figure 1. 5x5 square (left) and a triangle, base = 4 boxes, height = 3 boxes (right)

Figure 2. A hollow 10×10 square, 2 boxes thick(left), and plus sign, one box thick, 5 boxes along each line (right)

STRUCTURE ELEMENTS

Figure 3. (from left to right) 2×2 ones, 2×1 ones, 1×2 ones, cross(3 pixels long,1 pixel thick), diagonal line(two boxes long)

DILATION (top) EROSION (bottom)

Figure 4 Dilation and erosion following the same order of SE, for the 5x5 square.

Figure 5. Dilation and erosion following the same order of SE, for the triangle

Figure 6.  Dilation and erosion following the same order of SE, for the hollow 10x10 square

Figure 7 Dilation and erosion following the same order of SE, for the cross

Figure 4-7 is the expected results, I will upload the hand drawn some time,

Anyway the obvious result is that the dilation expands the image while the erosion shrinks it accordingly to the Structure element.

References:

[1]AP 186 Handouts.Activity 10 - Morphological Operations. Maricor Soriano 2013

[2]http://www.mathworks.com/help/images/ref/bwmorph.html

[3]Erosion.http://www.youtube.com/watch?v=yLdjNi2rncI

[4]Dilation.http://www.youtube.com/watch?v=xO3ED27rMHs

-------------------------------------------------------------------------------------

Self Grade:

8/10