Comparação do desempenho de classificadores multiclasses em dados químicos: abordando o problema de sobreajuste com o teste de permutação

Abstract

This work aimed to apply pattern recognition techniques to three sets of data available in the literature. The dataset called Glass Identification Dataset was evaluated using the classification models: Linear Discriminating Analysis, Regularized Discriminating Analysis, Mixture Discriminating Analysis, Discriminating Analysis of Partial Least Squares, Artificial Neural Networks, Support Vector Machine with the function of the radial core, K-nearest neighbors, Naive Bayes, Learning Vector Quantization, Classification, and Regression Tree, Bagging, Random Forest, C5.0 and Generalized Boosted Machine using accuracy and Kappa index as figures of merit. The Learning Vector Quantization (OLVQ1) classifier was the only model that provided 100% accuracy and the Kappa index equal to 1,000. Because of the result obtained, a new approach for the permutation test was proposed to verify the occurrence of overfitting of the model, which indicated the absence of overfitting. The dataset called Wine Quality Dataset was divided into Red Wine and White Wine. Regression models for both sets of wine were tested: Partial Least Squares, Support Vector Machine with radial base function, Support Vector Machine with a linear function, K-nearest neighbors, Classification and Regression Tree, Bagging, Random Forest and Generalized Boosted Machine using the mean absolute error (MAE), the root of the mean square error (RMSE) and the coefficient of determination (R2) as figures of merit. For all models tested, the results were unsatisfactory. Because of the results of the predictive regression models, the same classification models used in the Glass Identification Dataset data set were used. The obtained results outperformed the classification concerning previously works published in the literature. Once again, the Learning Vector Quantization (OLVQ1) classifier was the model with the best performance indicating the accuracy of 99.54% and 99.33% and the Kappa index equal to 0.9932 and 0.9896 for White and Red Wines, respectively. Therefore, the computational experiments show that the Learning Vector Quantification was the one method able to correctly classify all three data sets, reaching the accuracy of approximately 100% and the perfect Kappa index. The results showed that the overfitting problem was absent, which was confirmed by the pairwise Wilcoxon signed-rank test