Skip to main content

Fine-tuning

tip

For detailed end-to-end fine-tuning examples and FAQ, check out our fine-tuning guide.

warning

Every fine-tuning job comes with a minimum fee of $4, and there's a monthly storage fee of $2 for each model. For more detailed pricing information, please visit our pricing page.

Fine-tuning basics​

Fine-tuning vs. prompting​

When deciding whether to use prompt engineering or fine-tuning for an AI model, it can be difficult to determine which method is best. It's generally recommended to start with prompt engineering, as it's faster and less resource-intensive. To help you choose the right approach, here are the key benefits of prompting and fine-tuning:

  • Benefits of Prompting

    • A generic model can work out of the box (the task can be described in a zero shot fashion)
    • Does not require any fine-tuning data or training to work
    • Can easily be updated for new workflows and prototyping

    Check out our prompting guide to explore various capabilities of Mistral models.

  • Benefits of Fine-tuning

    • Works significantly better than prompting
    • Typically works better than a larger model (faster and cheaper because it doesn't require a very long prompt)
    • Provides a better alignment with the task of interest because it has been specifically trained on these tasks
    • Can be used to teach new facts and information to the model (such as advanced tools or complicated workflows)

Common use cases​

Fine-tuning has a wide range of use cases, some of which include:

  • Customizing the model to generate responses in a specific format and tone
  • Specializing the model for a specific topic or domain to improve its performance on domain-specific tasks
  • Improving the model through distillation from a stronger and more powerful model by training it to mimic the behavior of the larger model
  • Enhancing the model’s performance by mimicking the behavior of a model with a complex prompt, but without the need for the actual prompt, thereby saving tokens, and reducing associated costs
  • Reducing cost and latency by using a small yet efficient fine-tuned model

Dataset Format​

Data must be stored in JSON Lines (.jsonl) files, which allow storing multiple JSON objects, each on a new line.

Datasets should follow an instruction-following format representing a user-assistant conversation. Each JSON data sample should either consist of only user and assistant messages ("Default Instruct") or include function-calling logic ("Function-calling Instruct").

1. Default Instruct​

Conversational data between user and assistant, which can be one-turn or multi-turn. Example:

{
    "messages": [
        {
            "role": "user",
            "content": "User interaction n°1 contained in document n°2"
        },
        {
            "role": "assistant",
            "content": "Bot interaction n°1 contained in document n°2"
        },
        {
            "role": "user",
            "content": "User interaction n°2 contained in document n°1"
        },
        {
            "role": "assistant",
            "content": "Bot interaction n°2 contained in document n°1"
        }
    ]
}
  • Conversational data must be stored under the "messages" key as a list.
  • Each list item is a dictionary containing the "content" and "role" keys. "role" is a string: "user", "assistant", or "system".
  • Loss computation is performed only on tokens corresponding to assistant messages ("role" == "assistant").

2. Function-calling Instruct​

Conversational data with tool usage. Example:

{
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant with access to the following functions to help the user. You can use the functions if needed."
        },
        {
            "role": "user",
            "content": "Can you help me generate an anagram of the word 'listen'?"
        },
        {
            "role": "assistant",
            "tool_calls": [
                {
                    "id": "TX92Jm8Zi",
                    "type": "function",
                    "function": {
                        "name": "generate_anagram",
                        "arguments": "{\"word\": \"listen\"}"
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "{\"anagram\": \"silent\"}",
            "tool_call_id": "TX92Jm8Zi"
        },
        {
            "role": "assistant",
            "content": "The anagram of the word 'listen' is 'silent'."
        },
        {
            "role": "user",
            "content": "That's amazing! Can you generate an anagram for the word 'race'?"
        },
        {
            "role": "assistant",
            "tool_calls": [
                {
                    "id": "3XhQnxLsT",
                    "type": "function",
                    "function": {
                        "name": "generate_anagram",
                        "arguments": "{\"word\": \"race\"}"
                    }
                }
            ]
        }
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "generate_anagram",
                "description": "Generate an anagram of a given word",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "word": {
                            "type": "string",
                            "description": "The word to generate an anagram of"
                        }
                    },
                    "required": ["word"]
                }
            }
        }
    ]
}
  • Conversational data must be stored under the "messages" key as a list.
  • Each message is a dictionary containing the "role" and "content" or "tool_calls" keys. "role" should be one of "user", "assistant", "system", or "tool".
  • Only messages of type "assistant" can have a "tool_calls" key, representing the assistant performing a call to an available tool.
  • An assistant message with a "tool_calls" key cannot have a "content" key and must be followed by a "tool" message, which in turn must be followed by another assistant message.
  • The "tool_call_id" of tool messages must match the "id" of at least one of the previous assistant messages.
  • Both "id" and "tool_call_id" are randomly generated strings of exactly 9 characters. We recommend generating these automatically in a data preparation script as done here.
  • The "tools" key must include definitions of all tools used in the conversation.
  • Loss computation is performed only on tokens corresponding to assistant messages ("role" == "assistant").

Upload a file​

Once you have the data file with the right format, you can upload the data file to the Mistral Client, making them available for use in fine-tuning jobs.

import os
from mistralai.client import MistralClient

api_key = os.environ.get("MISTRAL_API_KEY")
client = MistralClient(api_key=api_key)

with open("training_file.jsonl", "rb") as f:
    training_data = client.files.create(file=("training_file.jsonl", f))

Create a fine-tuning job​

The next step is to create a fine-tuning job.

  • model: the specific model you would like to fine-tune. The choices are open-mistral-7b (v0.3) and mistral-small-latest (mistral-small-2402).
  • training_files: a collection of training file IDs, which can consist of a single file or multiple files
  • validation_files: a collection of validation file IDs, which can consist of a single file or multiple files
  • hyperparameters: two adjustable hyperparameters, "training_step" and "learning_rate", that users can modify.
from mistralai.models.jobs import TrainingParameters

created_jobs = client.jobs.create(
    model="open-mistral-7b",
    training_files=[training_data.id],
    validation_files=[validation_data.id],
    hyperparameters=TrainingParameters(
        training_steps=10,
        learning_rate=0.0001,
        )
)
created_jobs

List/retrieve/cancel jobs​

You can also list jobs, retrieve a job, or cancel a job.

You can filter and view a list of jobs using various parameters such as page, page_size, model, created_after, created_by_me, status, wandb_project, wandb_name, and suffix. Check out our API specs for details.

# List jobs
jobs = client.jobs.list()
print(jobs)

# Retrieve a jobs
retrieved_jobs = client.jobs.retrieve(created_jobs.id)
print(retrieved_jobs)

# Cancel a jobs
canceled_jobs = client.jobs.cancel(created_jobs.id)
print(canceled_jobs)

Use a fine-tuned model​

When a fine-tuned job is finished, you will be able to see the fine-tuned model name via retrieved_jobs.fine_tuned_model. Then you can use our chat endpoint to chat with the fine-tuned model:

from mistralai.models.chat_completion import ChatMessage

chat_response = client.chat(
    model=retrieved_job.fine_tuned_model,
    messages=[ChatMessage(role='user', content='What is the best French cheese?')]
)

Delete a fine-tuned model​

client.delete_model(retrieved_job.fine_tuned_model)