evaluation script for MMTEB

Hi, recently, I found your MMTEB work. The benchmark looks good. I would like to evaluate some models on this benchmark. Can you share a sample script for the evaluation? Thanks!

You can find example scripts in readme

mteb/README.md

Lines 41 to 97 in 4d23c6c

    
           * Using a Python script: 
        
           ```python 
        
           import mteb 
        
           from sentence_transformers import SentenceTransformer 
        
           # Define the sentence-transformers model name 
        
           model_name = "average_word_embeddings_komninos" 
        
           model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name) 
        
           tasks = mteb.get_tasks(tasks=["Banking77Classification"]) 
        
           evaluation = mteb.MTEB(tasks=tasks) 
        
           results = evaluation.run(model, output_folder=f"results/{model_name}") 
        
           ``` 
        
           <details> 
        
             <summary> Running SentenceTransformer model with prompts </summary> 
        
           Prompts can be passed to the SentenceTransformer model using the `prompts` parameter. The following code shows how to use prompts with SentenceTransformer: 
        
           ```python 
        
           from sentence_transformers import SentenceTransformer 
        
           model = SentenceTransformer("average_word_embeddings_komninos", prompts={"query": "Query:", "passage": "Passage:"}) 
        
           evaluation = mteb.MTEB(tasks=tasks) 
        
           ``` 
        
           In prompts the key can be: 
        
           1. Prompt types (`passage`, `query`) - they will be used in reranking and retrieval tasks  
        
           2. Task type - these prompts will be used in all tasks of the given type 
        
              1. `BitextMining` 
        
              2. `Classification` 
        
              3. `MultilabelClassification` 
        
              4. `Clustering` 
        
              5. `PairClassification` 
        
              6. `Reranking` 
        
              7. `Retrieval` 
        
              8. `STS` 
        
              9. `Summarization` 
        
              10. `InstructionRetrieval` 
        
           3. Pair of task type and prompt type like `Retrival-query` - these prompts will be used in all classification tasks 
        
           4. Task name - these prompts will be used in the specific task 
        
           5. Pair of task name and prompt type like `NFCorpus-query` - these prompts will be used in the specific task 
        
           </details> 
        
           * Using CLI 
        
           ```bash 
        
           mteb available_tasks 
        
           mteb run -m sentence-transformers/all-MiniLM-L6-v2 \ 
        
               -t Banking77Classification  \ 
        
               --verbosity 3 
        
           # if nothing is specified default to saving the results in the results/{model_name} folder 
        
           ```

Thanks for the quick response. Actually, I am looking for the evaluation script for the whole MMTEB (the benchmark, not one task from MTEB). It looks the pointed file only includes the evaluation of a task in MTEB.

Similarly you can run

import mteb
benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
evaluation = mteb.MTEB(tasks=benchmark)
evaluation.run(model)

Navigation Menu

Repository navigation

evaluation script for MMTEB #2150

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	* Using a Python script:

	```python
	import mteb
	from sentence_transformers import SentenceTransformer

	# Define the sentence-transformers model name
	model_name = "average_word_embeddings_komninos"

	model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)
	tasks = mteb.get_tasks(tasks=["Banking77Classification"])
	evaluation = mteb.MTEB(tasks=tasks)
	results = evaluation.run(model, output_folder=f"results/{model_name}")
	```

	<details>
	<summary> Running SentenceTransformer model with prompts </summary>

	Prompts can be passed to the SentenceTransformer model using the `prompts` parameter. The following code shows how to use prompts with SentenceTransformer:

	```python
	from sentence_transformers import SentenceTransformer


	model = SentenceTransformer("average_word_embeddings_komninos", prompts={"query": "Query:", "passage": "Passage:"})
	evaluation = mteb.MTEB(tasks=tasks)
	```

	In prompts the key can be:
	1. Prompt types (`passage`, `query`) - they will be used in reranking and retrieval tasks
	2. Task type - these prompts will be used in all tasks of the given type
	1. `BitextMining`
	2. `Classification`
	3. `MultilabelClassification`
	4. `Clustering`
	5. `PairClassification`
	6. `Reranking`
	7. `Retrieval`
	8. `STS`
	9. `Summarization`
	10. `InstructionRetrieval`
	3. Pair of task type and prompt type like `Retrival-query` - these prompts will be used in all classification tasks
	4. Task name - these prompts will be used in the specific task
	5. Pair of task name and prompt type like `NFCorpus-query` - these prompts will be used in the specific task
	</details>

	* Using CLI

	```bash
	mteb available_tasks

	mteb run -m sentence-transformers/all-MiniLM-L6-v2 \
	-t Banking77Classification \
	--verbosity 3

	# if nothing is specified default to saving the results in the results/{model_name} folder
	```

evaluation script for MMTEB #2150

Description

Activity

Samoed commented on Feb 24, 2025

lifu-tu commented on Feb 24, 2025

Samoed commented on Feb 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions