From the source
Researchers at the AI Accountability Lab in Trinity College, Dublin , have developed a documentation quality assessment framework for evaluating Transparency and Usefulness of the documentation of an AI model.
They have shared their framework and assessments, including that of Apertus, through a website and report.
This work was funded by Mozilla.
Header photo by AAIbarra - CC0 To assess Transparency , we identified 6 dimensions: Clarity, Completeness, Consistency, and Correctness, and for Usefulness we identified Accessibility and Comprehension.
We then developed 242 metrics as specific questions or criteria that evaluate each field in each section of the public summary template.
Since the public summary is structured such that each section has different implications for stakeholders, e.g. some may be interested only in Section 2.2 or Section 3, we evaluate each section using these metrics and then aggregate their scores to get the overall quality.
( Methodology ) Again, note that this evaluation strictly relates to the documentation, and makes no judgement of the performance, accessibility or openness of the model described.
…
