From the source
More and more enterprises are now asking agents to work with their proprietary documents and answer questions about their contents.
However, much of the important information lives inside figures and charts.
Many customers have been finding that agents struggle to answer questions that require reading and counting values in charts.
For agents to work reliably in diverse enterprise settings, we need to make charts more interpretable.
How can we make a chart easier for agents to understand?
We ran a simple, quick test: we asked different agents, “How many local maxima are on this chart?”
Below is a comparison of a frontier agent and Databricks Genie at answering this question.
The frontier agent was passed just the image, spent 50 seconds reasoning, but still got an incorrect answer of 17.
Meanwhile, Databricks Genie used a structured extraction of the chart through ai_parse_document, and got the correct answer 18.
We noticed these shortcomings in our OfficeQA Pro benchmark , where models performed worse on chart-based and multimodal questions than on questions that did not require chart understanding.
…
