I have a non linear data set, and I am using SVM (RBF kernel) to build a classification model, but not sure how to set the best hyperparameters of the SVM, C and gamma in Matlab fitcsvm. And is it acceptable if it is trial and error approach setting arbitrary values, until finding the best performance?
$\begingroup$
$\endgroup$
Add a comment |
2 Answers
$\begingroup$
$\endgroup$
4Well, there is a bunch of articles that tries to tackle this problem but basically, to guarantee a good solution you will need to do Grid Search (sklearn tutorial on it)
You can use various techniques for that, for example:
- Binary Gridsearch: Try the maximum and the minimum value in the interval you want, and the middle point, check which side is best and set the minimum or the maximum as your previous middle point according.
That can help you find an average solution fast, but remember that this comes without warranty.
Exhaustive Grid Search
Randomized Parameter Optimization
Genetic Algorithms
Check the link for sklearn for more information. I research on kernel methods and this is aways a bit annoying thing to tackle.
Note: If you're only going to try RBF I would advise to use $\sigma$ as a value betwen the norm of your training samples $\pm 80\%$. You could also normalize you vectors to norm 1 and limit the search to $0.2$ to $1.8$. Usually too small $\sigma$ gives high sensibility to noise while too big aproximates a straight line (loses non-linear power)
answered Mar 26, 2019 at 11:22
- $\begingroup$what do you think of using 'OptimizeHyperparameters' function to get best parameters mathworks.com/help/stats/bayesian-optimization-case-study.html$\endgroup$– AtheerCommentedApr 6, 2019 at 21:07
- $\begingroup$Never used it before, but you should try$\endgroup$CommentedApr 6, 2019 at 21:15
- $\begingroup$tried it and it works fine. I would like to ask about soft mragin SVM, i believe it needs tunning the C value as well, correct?$\endgroup$– AtheerCommentedApr 9, 2019 at 18:16
- $\begingroup$You should post it as a separated question as this may help others. But remember to search for an previously answered question on soft margin parameter tuning$\endgroup$CommentedApr 9, 2019 at 18:40
$\begingroup$$\endgroup$
Sadly, there is no easy solution for hyperparameter tuning. Basically, you have two options:
- Manual Adjustment: Read the documentation of the SVM and dive into the corresponding literature. Try to understand how the different kernel functions work and which functions are appropriate for your classification problem. Set the hyperparameters to the best of your knowledge and then observe what difference slight deviations make. Of course, there will still be some degree of trial and error in this approach.
- Grid Search: For all categoric parameters (e.g. kernel function), set up a list of all possible categories and for all metric parameters (e.g. epsilon), define a range of reasonable values. This is your hyperparameter space. Subsequently, run a multitude of SVMs with randomly selected parameters from the hyperparameter space. In the end, you choose the hyperparameter combination with the highest accuracy. (Note: there are also other approaches with more sophisticated methods than random selection)
Both methods are widely used, however, if you are not very familiar with SVM's, I highly recommend the manual adjustment.
Of course, it is very tempting to simply set up a grid search, grab something to eat and later pick the best performing hyperparameter combination. But in the end, you will not learn anything and you will not know if these hyperparameters are a reasonable choice for your problem. Broadening your knowledge will be beneficial in the future, both because it reduces the effort for future manual adjustments and because in the future you will be able to narrow down the hyperparameter space for the grid search, which will significantly reduce computing time.
