Optimisation problems can be thought of as “landscapes” with peaks, valleys, plateaus, and other topographical “characteristics” which influence the performance of an algorithm looking for an optimum. Therefore, measuring these characteristics is essential for finding better solutions faster.
Unfortunately, in a black-box scenario, the landscape is unknown and we are forced to explore it by sampling at different locations. This can be expensive, as each sample may represent a simulation or an experiment that could take hours or days to be completed. In this paper, we examined how accurate the methods are for measuring landscape characteristics on black-box problems, as the data available decreases or the landscape slightly changes.
We found that several measurements can vary significantly, resulting in contradictory conclusions, that could mislead any method dependent on them. Therefore, we identified a subset that provides the most reliable results. Our work also provided a methodology and curated data that can be used by other researchers to identify robust measuring methods.
