The BioSecure Reference and Evaluation Framework
When addressing identity verification and even if numerous algorithms for different biometric modalities are being developed, it is always difficult to compare methods. Unfortunately, the majority of the experimental results are reported on different data sets, or with different experimental protocols, making the comparison of the biometric systems irrelevant. In the case of results reported on common evaluation campaigns (with common databases and evaluation protocols) the comparison between different systems being tested becomes fair, but the precise reasons why a given system performs better than others remain difficult to figure out. In order to compare methods without prejudice, to understand their real weak and strong parts, to avoid continuous re-development of existing state-of-the-art approaches, and to measure the progress achieved, a common evaluation framework is needed.
In this way, the BioSecure Network of Excellence has taken in charge the development or improvement of existing open-source identity verification systems for the most widely used biometric modalities (2D and 3D face, speech, signature, fingerprint, hand, iris and talking face). These open-source systems are based on principles that have proven their efficiency and their purpose is to serve as comparison point to measure the real progress achieved with new research methods. Consequently, these open-source systems are known as reference systems. These systems accompanied with benchmarking (calibration) publicly available databases and benchmarking (calibration) assessment protocols define the building blocks of the Evaluation Framework, which also reports benchmarking results (i.e. results obtained by applying the reference system on the benchmarking database according to the benchmarking protocol).
The Reference and Evaluation Framework can then be used in two main ways :
- In the case where someone publishes results on a specific database according to a specific protocol, experiments using the reference system have to be added as a way of calibrating the difficulty of this particular task.
- In the case where someone publishes results using the benchmarking database and protocol, the benchmarking results of the reference system can be used as a way of calibrating the performance.
In both cases, the results of the reference systems permit to measure the progress achieved with the newly developed algorithm being tested. Now, it is hoped that the published results of a new biometric algorithm will be systematically accompanied with the results obtained by the reference system in the same experimental conditions. Such a methodology for reporting experimental results will facilitate deeply advances of the state of the art.
Through the Evaluation Framework, an absolute reproducibility of the benchmarking results is also assured. Indeed, for each modality, a documentation explains how to install and use the reference system. The documentation describes also a publicly available database and an associated evaluation protocol. Finally, the documentation provides the results obtained on the reference database according to the reference protocol. In this way, this documentation should allow full reproducibility of the benchmarking framework results.
Relevance of using open-source reference systems
The fact that the proposed reference systems are open-source and made of replaceable C/C++ software modules with well defined inputs/outputs is of great interest. Indeed, researchers often work on a specific part of the system and do not have the time nor the interest in building a complete system. In this way, a researcher could show the improvement of his new features extraction algorithm (for example) simply by replacing the corresponding module in the reference system and without having to bother about the pattern recognition algorithm. This new methodology for conducting research and experiments will enable to assess the contribution of each module to the final performance of a biometric system.
Using an open-source reference system as a basis for researching a specific area is also a good way to save time, human resources and money. For example, young researchers who are just starting in the field of biometrics, could immediately be faced with the reference system and they could put all their efforts to improve our efficient system rather than developing a state-of-the-art algorithm from A to Z again.
At present, the BioSecure Reference and Evaluation Framework concerns eight biometric modalities. It is comprised of five publicly available databases and ten benchmarking open-source reference systems. Among these systems, eight were developed within the framework of the BioSecure Network of Excellence and only three were existing ones (the fingerprint and speech reference systems) but all systems are accompanied with a documentation which mentions a benchmarking database, asssessment protocol and benchmarking results to compose the Evaluation framework.
The evaluation tool is currently available at the following address:
More information about the BioSecure Reference and Evaluation Framework can be found in the BioSecure public deliverable entitled “Description and documentation of the BioSecure software library”.