Fix evaluation bias to unlock better models on Amazon Bedrock

From the source

This post was co-authored with Khaled Thabit, Meriem Boughrara, and Mohamed Desouky from Property Finder. The engineering team at Property Finder , a digital real estate application, discovered that 96.8 percent of their evaluation labels had been inherited directly from a single model’s predictions rather than verified by human experts.

Their ground truth was systematically skewed toward the incumbent model, making every alternative appear to underperform. The labels also had errors, but the bias was what blocked objective model comparison. They uncovered this when migrating their vision AI workload to a more cost-effective option on Amazon Bedrock .

Property Finder processes tens of thousands of property images every day across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image. These cover watermark detection, room classification, furnishing status, photo quality, and construction indicators.

That’s thousands of new property listings with an average of 15 images each. All need quality assessment before going live. Property Finder needed a more cost-effective option. …

aws.amazon.com