Machine Learning System Evaluation

Unlike model evaluation, sometimes we need to evaluate the machine learning performance from a system view.

Reliablity

In reliability engineering, the reliability of a system is estimated by considering the dependencies between the system’s components. The probability of a system failure is then expressed in terms of the states of its components [1].

Stability

A post on stability of ML system can be found here.

An algorithm is stable, intuitively speaking, if its output doesn’t change much if we perturb the input sample in a single point. We will see that this property by itself is necessary and sufficient for generalization.

Ref.

[1] Ursani, Z., & Corne, D. W. (2017, November). Use of reliability engineering concepts in machine learning for classification. In Soft Computing & Machine Intelligence (ISCMI), 2017 IEEE 4th International Conference on (pp. 30-34). IEEE.