Unlike model evaluation, sometimes we need to evaluate the machine learning performance from a system view.


In reliability engineering, the reliability of a system is estimated by considering the dependencies between the system’s components. The probability of a system failure is then expressed in terms of the states of its components [1].


A post on stability of ML system can be found here.

An algorithm is stable, intuitively speaking, if its output doesn’t change much if we perturb the input sample in a single point. We will see that this property by itself is necessary and sufficient for generalization.


[1] Ursani, Z., & Corne, D. W. (2017, November). Use of reliability engineering concepts in machine learning for classification. In Soft Computing & Machine Intelligence (ISCMI), 2017 IEEE 4th International Conference on (pp. 30-34). IEEE.