Regularization

Regularization has annoying side effects :
A "tuning" parameter has to be introduced, whose optimal value is difficult to identify.
Increased bias of the model (but more than compensated for by its reduced variance).

Regularization Any estimation technique designed to impose a prior assumption of "smoothness" on the fitted function. See "Regularization Theory and Neural Networks Architectures".

[edit] Regularization
Fitting the training set too closely can lead to degradation of the model's generalization ability. Several so-called regularization techniques reduce this overfitting effect by constraining the fitting procedure.

Regularization (in Neural Networks). A modification to training algorithms that attempts to prevent over- or under-fitting of training data by building in a penalty factor for network complexity (typically by penalizing large weights, ...

Chapter 7 Regularization Theory 313
7.1 Introduction 313
7.2 Hadamard's Conditions for Well-Posedness 314
7.3 Tikhonov's Regularization Theory 315
7.4 Regularization Networks 326
7.5 Generalized Radial-Basis-Function Networks 327
7.

Regularization refers to a set of techniques which help to ensure that the function computed by the network is no more curved than necessary. This is achieved by adding a penalty to the error function, giving: ...

See also: Regression, Distribution, Generalization, Neural network, Classification