From the source
IBM researchers benchmarked three open-source agent harnesses with the same open-weight model and found harness choice moved AppWorld task completion from 30% to 81%.
- The comparison tested CUGA,
Nous Research's Hermes and LangChain's Deep Agents on 90 AppWorld tasks in 30 scenarios, each run with GLM-5.3-flash and one attempt per task.
- IBM says the results are not a verdict on the other two harnesses because neither was built for multi-app API work, and the point is that a harness has to fit the job.
- CUGA is IBM's open-source harness for enterprise work, built to enforce rules the model cannot override and to deploy inside a customer's environment.
- IBM reports consistency figures of 20 scenarios for CUGA, 8 for Hermes and 2 for Deep Agents, where a scenario passes only if all three of its variants pass.
- The company states three limits: the 90-task sample may not generalize, the setups were not identical, and the runs used no policies so they do not measure policy enforcement.
