Benchmarking Text Embedding Models for Vietnamese Retrieval Tasks

UPDATE: This post now includes new Gemma Embedding model.

When building Retrieval-Augmented Generation (RAG) applications, choosing the right embedding model is very important. If your retrieval step gets the wrong results, your language model will like to give the wrong answer.

For Vietnamese, there are many embedding models available, but no official benchmark exists for retrieval tasks. It’s also hard to compare commercial models (like OpenAI) with open-source options. So, in this blog, i would like to share my benchmark results on Vietnamese embedding models for retrieval.

Approach

I used the MTEB library for evaluation. MTEB is a library behind the popular text embedding leaderboard. This library provides both evaluation framework and datasets. The other alternative is to use sentence embeddings evaluator but i found MTEB is simpler, refer to this for the official document. I also found a bug while running the evaluation!

From the MTEB library, I selected three datasets for evaluation:

• VieQuAD: Vietnamese version of the SQuAD dataset, focused on question-answer retrieval.
• WebFAQ: Multilingual FAQ retrieval dataset. I used the vietnamese eval split.
• Zalo Legal Text: Vietnamese legal text retrieval dataset from the Zalo AI contest (2021).