Overfitting

Overfitting
In the previous example we used a network with two hidden units. Just by looking at the data, it was possible to guess that two tanh functions would do a pretty good job of fitting the data.

Overfitting/Overtraining in supervised learning (e.g. neural network). Training error is shown in blue, validation error in red, both as a function of the number of training cycles.

Overfitting
One of the virtues of backprop is that it will fit any function even if you don't know the form of the function you need to fit.

Overfitting the training data is an important issue because the training examples are only a sample of all possible instances, ...

Overfitting. When attempting to fit a curve to a set of data points, producing a curve with high curvature that fits the data points well but does not model the underlying function well, its shape being distorted by the noise inherent in the data.

Combatting overfitting - getting a model you can use somewhere else ...

Overfitting - This can occur if an AI agent is taught a certain section of a game, and then expected to display intelligent behaviour based on its experience.

This problem is referred to as overfitting. Overfitting is a critical problem in most all standard NNs architecture. Furthermore, NNs and other AI machine learning models are prone to overfitting (Lawrence et al., 1997).

If you've got a learning algorithm in one hand and a dataset in the other hand, to what extent can you decide whether the learning algorithm is in danger of overfitting or underfitting?

Lam and Bacchus address the issue of bias versus variance and propose a solution to the overfitting problem in the context of learning Bayesian networks.

Overparametrization and Overfitting
The true performance of a model is that observed on new data that did not take part to the construction of the model, not the observed performance on the design data.

"Preventing overfitting of Cross-Validation data"
"Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation"
"Efficient Algorithms for Minimizing Cross Validation Error" ...

The training of SVMs is based on quadratic programming, a form of optimization that (usually) has only one global minimum. Therefore, and because SVMs have means to reduce the danger of overfitting, ...

the best one according to a given criterion (for instance the Schwarz Criterion - see Moore's slides), but we need to be careful because increasing k results in smaller error function values by definition, but also an increasing risk of overfitting.

See also: Classification, Validation, Distribution, Machine learning, Regression