Cognition· Devin / SWE

Cognition developed an automated system to estimate how many productive engineering hours each Devin session is worth, achieving a model with an r_log of 0.74 and showing no bias. The system is now running with customers.

From the source

After carefully tuning the system, however, our model has an r l o g r_{log} of 0.74 0.74 and appears to be unbiased. Individual predictions aren't perfect, but the model is good enough to be used for estimating aggregated totals. They're also convertible to dollar amounts using engineering salaries, getting us closer to business value. In our system, an agent reviews each completed Devin session — first classifying whether it produced useful output, then estimating how long a human engineer would have taken to produce the same work. We validated it by asking human engineers how long they would have spent on the same tasks. The system is now running with customers.

cognition.com