Made withOllamaClip

Fine Tune a model with MLX for Ollama

M

Matt Williams

Creator on the Ollama channel

Watch on YouTube ↗

167K viewsAug 30, 20248:40 runtime

Learn to fine-tune a large language model on Apple Silicon using MLX for use in Ollama. This guide demystifies the process, breaking it down into three manageable steps: crafting a model-specific dataset in JSONL format, running the fine-tuning command with LoRA, and integrating…

Chapters

#ollama#clip#aillm

Fine-Tuning vs. Prompt Engineering

There are two main ways to customize a model's output:

Prompt Engineering / RAG: Add new information directly into the prompt to give the model context it doesn't already have. This is for teaching the model new facts for a single response.
Fine-Tuning (with LoRA): Tweak the model's internal weights to change how it responds. This is for adjusting the style, syntax, and format of its output, not for teaching it new information.

The Three-Step Fine-Tuning Process

Create a Dataset: Compile a set of high-quality question-and-answer examples.
Run the Fine-Tuning Process: Use a tool like MLX to train the model on your dataset, which generates an adapter.
Use the New Adapter: Combine the adapter with the base model in Ollama to create a new, fine-tuned model.

Important
Step 1, creating the dataset, is the most difficult and critical part of the process. The quality of your data directly determines the quality of your fine-tune.

Step 1: Preparing the Dataset

Finding the Prompt Template

Before creating data, you must find the correct prompt format for your base model.

Navigate to ollama.com and find your base model (e.g., Mistral).
Click the "Template" tab.
Identify the structure the model expects, typically involving special tokens like [INST] and [/INST].

The JSONL Format

Your training data must be in a JSON Lines (.jsonl) file. Each line in the file is a complete, self-contained JSON object.

Each object must contain a single key, "text".
The value is a string containing your formatted prompt and the ideal answer.

Example for Mistral:

{"text": "[INST] Your question or instruction here [/INST] The perfect answer you want the model to learn."}

Data Splitting

For the mlx-lm tool, split your dataset into three files:

train.jsonl: ~60% of your data, used for training.
valid.jsonl: ~20% of your data, used for validation during training.
test.jsonl: ~20% of your data, used for testing the final model.

Step 2: Running the Fine-Tune with MLX

Prerequisites

Install MLX-LM: pip install mlx-lm
Install and Configure Hugging Face access:
- Install the CLI: pip install -U "huggingface_hub[cli]"
- Log in to your account: huggingface-cli login (This is required for models like Mistral).

The Fine-Tuning Command

Use the mlx_lm.lora command to start the process.

mlx_lm.lora \
    --model mistralai/Mistral-7B-v0.1 \
    --train \
    --data path/to/your/dataset/folder/ \
    --batch-size 4

--model: The repository ID of the base model from Hugging Face.
--train: A flag to indicate you are running a training job.
--data: The path to the directory containing your train.jsonl, valid.jsonl, and test.jsonl files.
--batch-size: The number of examples to process in each iteration. A larger size is faster but requires more memory.

The output will be a new directory named adapters containing the LoRA adapter files.

Step 3: Using the Adapter with Ollama

Create a `Modelfile`

Create a new file (e.g., Modelfile) to define your new custom model.

FROM mistral
ADAPTER ./adapters

FROM: Specifies the base model to use (must match the model you fine-tuned).
ADAPTER: Specifies the path to the newly created adapters directory.

Create the Ollama Model

Run the ollama create command to build your new model.

ollama create my-tuned-mistral -f Modelfile

You can now run your fine-tuned model with ollama run my-tuned-mistral.

Note
This feature requires a recent version of Ollama. Ensure you are up-to-date. Using mlx_lm.lora is crucial as it creates adapters in the format Ollama expects natively, which is simpler than other methods that produce .npz files requiring a manual fusion step.

Introduction to AI Model Fine-Tuning

AI models are amazing, what they know and how they can respond to any question you give it is nothing short of miraculous. But we are all still in search of a way to make them better, to know more, to sound like us, to do our work for us.

But the model you download from Hugging Face or the Ollama library is a general model designed to be really good for everyone. How can we tweak the model?

Well, there are two main approaches we can take to modify what a model spits out. We can add new information in the prompt that the model can use to know things it didn't, or we can tweak the model weights to change how the model responds, the style it uses. That tweaking is called fine-tuning.

Fine-tuning isn't about teaching the model new information, although it is possible to do that. Instead, fine-tuning adjusts how the model will phrase what it says, the syntax it uses, the format of the output.

A Simple Three-Step Process with MLX

But when you look online for tutorials on how to fine-tune, you get buried in code. It looks really hard. It's probably going to need an expert to perform all the steps. But it's not. It's actually pretty easy, though you definitely get better with practice.

I think one of the biggest problems is that most folks seem to like to demonstrate fine-tuning using Python notebooks, which has to be the worst format for teaching. So let's take a look at how to fine-tune. I'm specifically using MLX this time around because it supports Apple silicon, but in future videos, I'll take a look at Unsloth and maybe Axolotl as well. You might be shocked at how easy this is to do.

Step one: come up with a data set of questions and answers. Step two: run the fine-tuning process. And step three: use the new adapter with a model. That's it.

But Step One is definitely the hardest part and the one most wrapped up in the dark arts that are hard to explain. But coming into this, I thought two and three were going to be the tough ones, and there's almost nothing to those steps.

Step 1: Finding the Model's Prompt Template

So let's start with step one. The goal here is to get the model to understand how you want it to answer certain questions, how to respond to a prompt. And so you need to provide a question and the answer in a format the model will understand. This is going to be specific to the model.

Let's look at doing a fine-tune of Mistral. So the first step is to go to ollama.com and find the Mistral model. Now click on the template. Now what I'm looking for is how the model expects to see a question and then the answer. This model accepts the new tool calling method with Ollama, but I don't really care about that for now. So at the end, there is an [INST] in square brackets, and then if there's a system prompt, include it, then include the user prompt, then close that [/INST] block, and then you have the answer.

Step 1: Structuring Data with the JSONL Format

So now we need to replicate this a bunch of times and we're going to save them in a JSONL file. That's kind of like a regular JSON file, but not really. In JSONL, each line is a new object. It's not an array of objects separated by commas, and I have no idea why. It certainly would be a whole lot easier for everyone involved if it was a simple array, but oh well.

Now each object has a single key and value. The key is text and the value is the string we just talked about before with that [INST] block.

Step 1: Generating & Splitting the Dataset

So now think about what you want to train it on. You need to come up with at least 50 to 100 of these objects, and more can be better. And again, we aren't teaching the model new facts, but rather the style to respond with. Often folks want a model that will be able to write emails or documents as themselves, or at least a first draft of themselves. So that's what I'm going to try.

If you've been here before, you know that I have a lot of videos, hundreds, and I have the scripts for most of the videos I've made in the last few years here on my Mac. And the way I talk in those videos is the same way I actually talk and the way I write. So I'm going to read in the markdown files with those scripts, and then I'll split up each file by new lines. And then I'll get Ollama to summarize each paragraph. That summarization, I have Ollama convert into an instruction for someone to write about. And there is my question. The original paragraph, that's my answer.

This probably isn't an ideal example, but it seems to work somewhat well for this case. Now, save 60% of the answers to the train.jsonl file and then 20% to each of the other two files: valid.jsonl and test.jsonl. The fine-tuning process will need these.

Step 2: Installing Prerequisites for Fine-Tuning

Now that we have that done, let's move on to step two: run the fine-tuning process. Since I'm using MLX here, we need to install mlx-lm. So pip install mlx-lm.

Next, we need to access the Mistral model on Hugging Face. MLX needs the original safe tensor files. Apparently, you can also use quantized files, but they need to be in MLX's format, and I haven't found a way to convert the standard GGUF files that Ollama uses to that special format.

The Mistral model requires you to log in to Hugging Face to download, so you need to install the Hugging Face CLI and log in. Visit this page on Hugging Face Doco to get that set up. Then run huggingface-cli login to log into your account.

Step 2: Running the Fine-Tuning Command

Now we can get to the command to run to fine-tune. We start with the command mlx_lm.lora. Pass the --train flag to tell it we want to do a fine-tune of the model. The --model flag says which model we want to use. This should be org/model-name. You can find this by going to the repo and clicking that copy button next to the name. The --data flag points to the path where your JSONL files are.

The next flag is --batch-size. This tells mlx-lm how many examples are used in each iteration. You want this to be larger to go through everything faster, but larger requires more memory, so it takes some experimentation. Run that command and now it's time to listen to the fans spin up. I have an M1 Max with 64 gigs of unified RAM and it takes several minutes to run, but it's pretty much hands-off while it does its thing. When it's done, you have a new directory called adapters and in there you'll find the safe tensor files and a config file.

Step 3: Using the Adapter in Ollama

So now we need to define a new model using our adapter. Since we use Mistral, we can create a model file with FROM set to mistral and ADAPTER set to the directory with our new adapters in it.

Now, ollama create -m mistral -f Modelfile, and this will take a second or so. And now you can run ollama run mistral and we can start using our fine-tune of Mistral.

If you're on a much older version of Ollama, this may not work. Support for these adapters was added just in the last couple of days. Also, you need to create the adapter with the mlx_lm command. If you use the Laura script in the mlx-examples repo, it'll create an NPZ file which needs to be fused to the original model, which is just a bit more of a hassle. MLX-LM creates an adapter that's used by Ollama natively.

Conclusion and Next Steps

Is this a perfect fine-tune? No, not really. I have some work to do to make it better, but at least now I know it's not as scary as I thought it was. What do you think? Have you done any fine-tuning of models yourself? Any specific fine-tunes you think are really compelling? Do you have any ideas for a fine-tune you think you might like to do? Let me know in the comments below. Thanks so much for watching. Goodbye.