The BioSecure Multimodal Evaluation Campaign 2007 (BMEC’2007)
offcial website : http://biometrics.it-sudparis.eu/BMEC2007/
One of the key aspirations of the BioSecure Network of Excellence was to investigate in depth the potential and merits of multimodal biometrics. In order to create the conditions which would enable such studies to be undertaken in a meaningful way, the network developed a performance Evaluation Framework including reference systems and collected a substantial multimodal biometric database. Then, and as a final step, the network designed, set up and launched BMEC’2007, the BioSecure Multimodal Evaluation Campaign. The initial objective of this evaluation, not restricted to BioSecure members, was to test the real impact of multimodality on well chosen applicative scenarios, namely mobile and access control. In the meantime, the campaign allowed the participants to assess the performance of their algorithms (either monomodal or multimodal approaches) and to compare and evaluate the benefits of the various solutions investigated.
The main objective of the mobile scenario was to test the robustness of monomodal and multimodal biometric systems against degraded acquisition conditions. Such conditions can be found when the biometric verification of identity is done either indoor or outdoor using a device with limited capabilities such as Webcam, a PDA or a mobile phone. The mobile scenario included two different types of evaluation:
· monomodal evaluation: This evaluation was indeed a continuation of the work initiated during the BioSecure residential workshop. Through BMEC’2007, the consortium supported again comparative evaluations of monomodal biometric algorithms with accent on:
o robustness to degraded conditions;
o robustness to forgeries;
o robustness to elapsed time between sessions.
The modalities considered for this evaluation included fingerprint, signature, talking-face and 2D face on video sequences.
· multimodal evaluation: Multimodality is often presented as a way to improve performance of monomodal systems, specially in the case of degraded conditions, and to improve resistance to forgeries. To validate such assessments, the participants had to fuse scores provided by the fingerprint, signature and 2D face reference systems. In this way, this evaluation allowed to test and evaluate the two following points:
o the enhancement of performance in degraded conditions (in relation to those obtained with monomodal systems) ;
o the robustness to forgeries.
The goal of the access control scenario was to propose two evaluation schemes to benchmark score-level fusion algorithms rigorously. These schemes are quality-dependent and cost-sensitive evaluation schemes. The first one was designed to evaluate quality-dependent fusion algorithms whereas the second was designed to evaluate conventional fusion algorithms. In both cases, the possibility of having two further sub-problems was also considered. The first one involves client-specific or user-dependent fusion where one can train a fusion classifier that is tailored to each identity claim. The second one involves handling the fusion problem with missing information. For instance, when one or more baseline systems are not operational due to failure to acquire or failure to match a biometric sample, nevertheless the fusion system shall be able to output a combined score.
In total, more than 20 organizations, mostly but not only from BioSecure, participated to these evaluations. Only development data were distributed to the participants while the final tests have been performed on sequestered data in order to avoid the disclosure of sensitive biometric data.
The final BioSecure workshop, held by the end of September 2007, gathered all the participants to discuss to a large extent the results of BMEC’2007, which were disclosed there for the first time.
The results of the BioSecure evaluations are publicly available on the BMEC’s website, through the BioSecure public deliverable entitled “Report on the BioSecure multimodal evaluation campaigns” and through several publications in reviews and conferences.