HuggingFace Evaluate Perplexity

import evaluate

model_id='seonglae/yokhal-md'

perplexity = evaluate.load("perplexity", module_type="metric")
input_texts = ["lorem ipsum", "Happy Birthday!", "Bienvenue"]
results = perplexity.compute(model_id=model_id,
                             add_start_token=True,
                             predictions=input_texts)
print(round(results["mean_perplexity"], 2))

Perplexity - a Hugging Face Space by evaluate-metric

Perplexity (PPL) is one of the most common metrics for evaluating language models. It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base `e...

https://huggingface.co/spaces/evaluate-metric/perplexity