Benchmarking that Matters: Rethinking Benchmarking in Continuous Optimisation for Practical Impact

Kononova, Anna V.; van Stein, Niki; Mersmann, Olaf; Bäck, Thomas; Bartz-Beielstein, Thomas; Glasmachers, Tobias; Hellwig, Michael; Krey, Sebastian; Kůdela, Jakub; Naujoks, Boris; Papenmeier, Leonard; Raponi, Elena; Renau, Quentin; Rook, Jeroen; Schäpermeier, Lennart; Vermetten, Diederick; Zaharie, Daniela

Abstract

Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints and information limitations of continuous and mixed-integer optimisation problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection and industrial decision-making, despite these suites being designed for different purposes. We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features and challenges in multi-objective and noisy settings. We propose a vision centred on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use.

Keywords

Benchmarking, continuous optimization, real-world problems, single-objective, multi-objective

Cite as

Details