HuggingFace QAEvaluator

from evaluate import evaluator
from datasets import load_dataset
task_evaluator = evaluator("question-answering")
data = load_dataset("squad", split="validation[:2]")
results = task_evaluator.compute(
    model_or_pipeline="sshleifer/tiny-distilbert-base-cased-distilled-squad",
    data=data,
    metric="squad",
)

• SQuAD

• SQuAD v2

Using the `evaluator`

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/docs/evaluate/v0.4.0/en/base_evaluator#question-answering

Evaluator

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/docs/evaluate/v0.4.0/en/package_reference/evaluator_classes#evaluate.QuestionAnsweringEvaluator