The decomposition of the mixed pixels into individual pure material (endmember) along with its proposition is called spectral unmixing for hyperspectral images. Spectral unmixing is considered a three-stage problem (as shown in Figure 1) for the hyperspectral image. The first is the subspace dimension which finds the number of pure materials in the image. The second one is endmember extraction which extracts the pure material spectra from the image and the third one is abundance estimation which estimates the proportions of each material in mixing. The endmember extraction is a very challenging stage in spectral unmixing as abundance mapping greatly depends on extracted endmembers. In the literature, endmember extraction is addressed using a geometrical, statistical, sparse regression, and deep learning approach. Due to simplicity and easy understanding, many researchers use the geometrical approach. We will discuss our three novel algorithms: Entropy-based convex set optimization (ECSO), Convex Geometry and K-medoids based Noise Robust (CGKNR), Convex polygon maximization (CPM). All the extracted endmembers of the three novel algorithms have been compared with the extracted endmembers of the prevailing algorithms on benchmark real and synthetic datasets using standard evaluation parameters such as Spectral Angle Mapper (SAM), Spectral Information Divergence (SID), and Normalized cross-Correlation (NXC). The Root Mean Square Error (RMSE) is used to test the efficacy of the extracted endmember for abundance mapping. The RMSE error is calculated between FCLS based abundance maps by the endmember of the proposed algorithm and FCLS based abundance maps by the Ground Truth. We have used the Hyperspectral Imagery Synthesis Toolbox (HIST) for generating five types of synthetic images. The synthetic images are added with Gaussian noise to test the noise robustness. The proposed algorithms in this thesis can be used for a variety of hyperspectral applications, including classification, target detection, and many others. Table 1 compares the three novel endmember extraction algorithms.