Accuracy is not Enough
To fully understand NIST’s FRVT Ongoing Reports the reader must look past singular statistics for individual evaluations of accuracy. Attaining high marks for matching accuracy on discrete tests is an impressive achievement, but its only part of the story because accuracy among recent submissions has increased to the point where one is practically indistinguishable from the next. Looking at these results, it’s easy to see why the focus must shift from an accuracy leader board to overall system performance.
Evaluating Overall System Performance
To obtain an equivalent assessment of face recognition systems you must find a ‘common language’ because each variable has different units of measure e.g., accuracy in percentage or proportion, template size in bytes, template generation time in milliseconds, and match time in nanoseconds. Arriving at this common language is called dimension scaling and it is the standard method of comparing data with variables which have different units of measure. Using the average of each variable, we produce a 4D vector which provides a fair comparison of each submission. The magnitude (length) of this 4D vector represents system performance. When plotted, smaller magnitudes are closer to the origin (pyramid apex) and represent better system performance.
Ranking FRVT Results
No single variable – especially accuracy – should be looked at by itself when evaluating a system’s operational suitability. For example, if an algorithm gets very high accuracy rates but takes 3 seconds to create a template or the template size is 4kb or it takes 0.5 seconds to compare templates, it would not be an operationally deployable solution for surveillance usage.
Aureus Insight v6.0 Outperforms All Others
When plotting the performance vectors as described in this brief, we can demonstrate the commercially available Aureus Insight v6.0 algorithm significantly outperforms all other NIST tested algorithms in the Visa, Mugshot and Wild image datasets.
