Multi-Choice Wavelet Thresholding Based Binary Classification Method

Perception-Decision-Cognition Methodology (PDCM)

The proposed Perception-Decision-Cognition Methodology (PDCM) for discriminant analysis is conceptually represented in Figure 1. As indicated in this figure, it consists of three steps:

The algorithm used by the PDCM consists of three steps conceptually described below, after assuming that all data have been classified according to three sets: training set, cognition (validation) set, and test set.

Step 1: Wavelet Thresholding-Based Dimension Reduction Techniques

Dimension reduction is a preferred strategy in the area of machine learning. As anticipated, there are several approaches to perform dimensional reduction. The following methods are among the most popular: principal component analysis (Jolliffe, 2002), rotational linear discriminant analysis technique (Sharma & Paliwal, 2008), independent component analysis (Stone, 2004), semi-definite embedding (Weinberger & Saul, 2006), multifactor dimensionality reduction (Ritchie & Motsinger, 2005), factor analysis (Basilevsky, 1994), and wavelet-based dimension reduction (Chang & Vidakovic, 2002; Cho et al., 2009; Donoho & Johnstone, 1994; Jung et al., 2006).

The dimension reduction strategy has important benefits that can be measured not only in terms of computational time savings, but also in accuracy improvement. In the novel PDCM, the wavelet-based dimension reduction is applied in Step 1. The wavelets approach was selected because of several attractive attributes, among which the following two are most relevant: (a) wavelets adapt effectively to spatial features of a function such as discontinuities and varying frequency behavior; (b) wavelets have efficient O(n) algorithms to do transformations (Mallot, 1999). Based on perceived knowledge, wavelet-based techniques are applied to obtain a well-fitted reduced-dimension representation of the original data.

Discrete Wavelet Transformation (DWT) is often used for dimension reduction (also known as shrinkage or threshold). The data constructed with the scaling and wavelet functions based on orthogonal base in time domain is as follows:

where $Z$ is the set of all the possible integer values, $c_{L,k}$ is the coarse level coefficient and $d_{L,k}$ is the finer level coefficient. Let $y_m={[y_{m1},y_{m2},\cdots ,y_{mN}]}^T$ is an $m^{th}$ observed sample. For a single sample, the DWT procedure uses the orthonormal matrix W of dimension $N\times N$ to find the wavelet coefficient

where J > L, L corresponds to the lowest decomposition level.

through the transformation

For multiple samples, let vector $Y=[y_1,y_2\mathrm{,...,}{y}_M]$ be the data set with M observed samples. The wavelet coefficient vector is obtained from the transformation

where $D=[d_1,d_2\mathrm{,...,}{d}_M]$, and $d_m=(c_{mL},d_{mL},d_{mL+1},\cdots ,d_{mJ})$, $m=\mathrm{1,2,...,}M$.

Small absolute values of wavelet coefficients are undesirable since they may be influenced more by noise than by information. On the other hand, large absolute values are more influenced by information than noise. This observation motivates the development of threshold methods. There are two threshold rules usually referred to as soft and hard thresholds. The soft rule is a continuous function of the data that shrinks each observation, while the hard rule retains unchanged only large observations (Donoho & Johnstone, 1994). The hard and soft threshold methods are defined as following:

here $\lambda$ is the threshold value. The threshold method can be used not only for data reduction but also for de-noising. One of the 5 different thresholding methods will be selected based on the performance of PDCM (Multi-Choice). Also, there are 5 different wavelet-based thresholding methods which will handle high-dimensional data. Therefore, the dimension reduction method is referred as Multi-Choice Wavelet Thresholding (MCWT).

Step 2: Variable Selection Based on Information Complexity and Recursive Feature Elimination

Once the reduced sample space is determined in Step 1, the decision regarding which of the remaining variables should be selected for ranking is made on the basis of minimal information complexity values, following the Information Complexity Performance Testing with Recursive Feature Elimination (ICOMP_PERF-RFE) procedure proposed by Bozdogan and Baek (2018). Since this procedure resulted in better performance than other RFE-based methods. Also, this procedure essentially generates a stabilized and smoothed covariance estimator to calculate the information complexity measure, and, finally performs ranking using recursive elimination on the remaining variables.

The development of information complexity for the discriminant analysis is evaluated using the modified maximal entropic complexity C_1F

where s is the rank of $\widehat{\Sigma }$, ${\lambda }_j$ is the j^th eigenvalue of $\widehat{\Sigma }>\mathrm{0,}j=\mathrm{1,\; 2\; ,}\cdots ,s$ and ${\overline{\lambda }}_a$ is arithmetic means of the eigenvalues

ICOMP_PERF can be evaluated as indicated below:

where lack of fit is assessed by means of the first three terms and complexity by the fourth one. In the above expression, ${\widehat{\sigma }}^2$ is the estimated mean squared error given by

and ${\widehat{\Sigma }}_{STA\_CSE}$ is the stabilized and smoothed convex sum covariance matrix estimator given by

where ${\widehat{\Sigma }}_{STA}$ is the stabilized covariance matrix proposed by Thomaz (2004), h is the number of variables, $I_h$ is h×h identity matrix, and k is chosen such that

and

Specific details on this procedure are provided by Bozdogan and Baek (2018).

Step 3: Cognition Accuracy of Selected Models

When the ranking decision is finished in Step 2, the corresponding accuracies are determined using the corresponding cognition sets and the support vector machines for classification (SVMC) described below. Once the accuracies are calculated for the selected models the most-accurate one is chosen.

The SVMC find an optimal separating hyperplane that maximizes the margin between the classes (Vapnik, 1995). Consider the case of classifying a set of linearly separating data into two groups. Assume a set of training data is given by $[(x_1,y_1),(x_2,y_2)\mathrm{,...,}(x_n,y_n)]$ where $x_i\in {\Re }^p$ is an input vector, $y_i\in \left\{\mathrm{-1,1}\right\}$ is a binary class index, and n is the size of the training data set. Then, a decision boundary that partitions the underlying vector space into two classes can be represented by the hyperplane

where $w$ is the weight vector and $b$ is the bias. The objective of SVMC is to find a maximum margin decision boundary between two parallel hyperplanes, $w^{T}x+b=1$ and $w^{T}x+b=-1$. The dual model of the corresponding primal model can be formulated as

subject to

where $K(x_i^T,x_j)$ is the kernel function and C is a predefined coefficient. Kernel functions used in the numerical experiments are described in Table 1.

The point $x^o$ with coordinates corresponding to new data can be classified as indicated below:

and

where ${\alpha }^{ov}$ and $b^{ov}$ are optimal values found based on the training data. A classification example based on the PDCM is illustrated in Figure 2.

Heart Data (44 Variables)

The data set includes 267 samples and 44 variables on cardiac single proton emission computed tomography (SPECT) images with two categories, i.e., normal and abnormal. There are 55 normal and 212 abnormal classes (Cios & Kurgan, 2001). The data set is divided into 134 samples as a training set, 13 samples as a cognition set, and 120 samples as a test set. The training set has 25 normal and 109 abnormal classes. The cognition set has 1 normal and 12 abnormal classes. The test set has 29 normal and 91 abnormal classes¹. Table 2 and Table 3 show comparison results in terms of the variables selected from ICOMP_PERF-RFE, the cognition accuracy, and the test accuracy. Cauchy and inverse multi-quadratic kernel functions are used in Table 2 and Table 3, respectively.

As observed in Table 2, PDCM achieves better cognition accuracies comparing to SVMC-RFE and two-stage and yields more accurate results in test. Also, as shown in Table 3, PDCM (visu intersect) and two-stage both reach the highest test accuracy, although two-stage requires fewer variables.

Near Infrared Spectroscopy Data (100 Variables)

These data were collected by a Tecator infratec food and feed analyzer to predict the fat content of a meat sample based on near infrared (NIR) spectroscopy. The data set was divided into two classes defined on the basis of fat content; one class (low-fat) corresponded to 20% or less, and another class (high-fat) to more than this level (Rossi & Villa, 2006). There are 77 low-fat and 138 high-fat classes. The entire data set consists of 215 samples with measured values for each of 100 predictive variables (wavelengths). These samples are divided randomly to configure a training set consisting of 108 samples; a cognition set consisting of 11 samples for cognition; and a test set consisting of the remaining 96 samples. The training set has 39 low-fat and 69 high-fat classes. The cognition set has 4 low-fat and 7 high-fat classes. The test set has 34 low-fat and 62 high-fat classes². Table 4 and Table 5 show comparison results in terms of the variables selected from ICOMP_PERF-RFE, the cognition accuracy, and the test accuracy. Cauchy and Gaussian kernel functions are used in Table 4 and Table 5, respectively. Although both PDCM and two-stage reach the 91% accuracy level for the cognition set in Table 4, the test accuracy of PDCM is higher than that of SVMC-RFE and two-stage except PDCM (verti). Additionally, Table 5 shows that the cognition accuracy of PDCM is 91% and the test accuracy is higher than that of other methods. Furthermore, PDCM uses only less than 9 variables to reach the accuracy levels previously mentioned.

Handwritten Data (240 Variables)

This data set has features of handwritten numerals from 0 to 9 extracted from a collection of Dutch utility maps. The entire set consists of 200 samples digitized in binary images per numerals and six different variable sets: 76 fourier coefficients, 216 profile correlations, 64 Karhunen-Love coefficients, 240 pixel averages, 47 Zernike moments and 6 morphological features (Van Breukelen et al., 1998). Two of six variable sets (216 profile correlations and 240 pixel averages) are used for the experiment. Two (0 and 9) out of 10 numerals are selected to verify the proposed method. Each numeral has 100 samples which are included in the experimental data set. For the experiment, 100 samples are used as a training set, 10 samples are used as a cognition set and 90 samples are used as a test set. The training set has 45 zero-numeral and 55 nine-numeral classes. The cognition set has 4 zero-numeral and 6 nine-numeral classes. The test set has 51 zero-numeral and 39 nine-numeral classes³. Table 6 and Table 7 show comparison results in terms of the selected variables from ICOMP_PERF-RFE, the cognition accuracy, and the test accuracy.

Cauchy and inverse multi-quadratic kernel functions are used in Table 6 and Table 7, respectively. As shown in the tables, PDCM reaches 100% cognition accuracy levels as did the SVMC-RFE, except two-stage. Furthermore, PDCM achieves a higher accuracy level than the other methods for the test set, except PDCM (msvet) in Cauchy kernel.