🌌 A Comprehensive Guide to Unsloth Finetuning
Unsloth has emerged as a powerful and efficient library for fine-tuning large language models (LLMs) and vision language models (VLMs), offering significant speedups and memory reductions compared to traditional methods. This guide provides a comprehensive overview of Unsloth’s features, guidance on selecting the best models, methods for converting models to the GGUF format, and examples of its current applications.
🌟 Key Features of Unsloth for Efficient Finetuning
Unsloth incorporates several innovative features that contribute to its efficiency. At its core is a custom autograd engine with kernels rewritten in OpenAI’s Triton language, including optimized forward and backward passes for RoPE embeddings and a custom Fast Cross Entropy Loss kernel. This foundational optimization results in substantial acceleration during the training process. One of the most notable aspects of Unsloth is its ability to significantly reduce memory usage. It supports 4-bit and 16-bit QLoRA/LoRA finetuning through bitsandbytes. Furthermore, Unsloth introduces dynamic 4-bit quantization, a technique that selectively avoids quantizing certain layers, leading to increased accuracy compared to standard 4-bit methods with only a marginal increase in VRAM. This allows users with limited GPU resources to fine-tune larger models effectively. Unsloth also provides functionalities that streamline the finetuning workflow. Updating Unsloth to the latest version can be done without updating dependencies, ensuring quick access to new features and bug fixes. Continued finetuning from previously saved LoRA adapters is supported, allowing users to resume training without restarting. For conversational models, Unsloth offers the ability to train on completions or responses only, ignoring the input prompts. Finetuning can be resumed from the last saved checkpoint, providing flexibility and preventing loss of progress. Saving models in the 16-bit format required by vLLM for efficient inference is also a straightforward process. Users working in Colab can override the default saving format to use safetensors instead of .bin. Unsloth facilitates the conversion of finetuned models to the GGUF format, which is essential for running models on CPUs using llama.cpp. This conversion can be done directly using the save_pretrained_gguf function. For managing conversational data, Unsloth provides utilities for formatting datasets according to various chat templates like zephyr, chatml, and llama. Furthermore, Unsloth claims to offer a 2x faster inference speed for QLoRA, LoRA, and non-LoRA models without requiring code changes. This is enabled through the FastLanguageModel.for_inference(model) function. For training reasoning models, Unsloth supports Generative Reasoning with Preference Optimization (GRPO), which now works with QLoRA and LoRA. Unsloth also extends its capabilities to vision models with the FastVisionModel class and UnslothVisionDataCollator for efficient finetuning of VLMs. Full finetuning and 8-bit finetuning are also supported by setting the respective flags to True .
🌟 Selecting Optimal Base Models for Unsloth Finetuning
Unsloth boasts compatibility with a wide range of popular LLM architectures, including Llama 3, Gemma, Mistral, Phi, Qwen, and DeepSeek-R1. This broad support allows users to leverage Unsloth’s optimizations with their preferred model structures. For those new to finetuning, starting with a smaller instruct model like Llama 3.1 (8B) is often recommended. When choosing a base model, it’s important to understand the distinction between instruct models and base models. Instruct models are pre-trained with built-in instructions, making them readily usable without extensive finetuning, especially for smaller datasets (less than 300 rows).
Loading a recommended model with Unsloth is straightforward. For example, to load the 4-bit quantized version of Llama 3.1 8B, the following code can be used :
from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( “unsloth/llama-3.1-8b-bnb-4bit”, max_seq_length=2048, load_in_4bit=True, )
Similarly, to load Gemma 3 4B, the code would be:
from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( “unsloth/gemma-3-4b-it-bnb-4bit”, max_seq_length=2048, load_in_4bit=True, )
It’s crucial to consider the VRAM requirements of the chosen model, which depend on the model size and the quantization level. Unsloth’s documentation and community resources often provide guidelines on VRAM usage for different models and configurations. To provide a quick reference, here are some recommended base models for Unsloth finetuning and their potential use cases:
| Model Name | Size | Recommended Use Cases |
|---|---|---|
| unsloth/llama-3.1-8b-bnb-4bit | 8B | General-purpose instruction tuning, beginners |
| unsloth/gemma-3-4b-it-bnb-4bit | 4B | Text generation, smaller-scale tasks |
| unsloth/mistral-7b-bnb-4bit | 7B | Conversational AI, instruction following |
| unsloth/phi-4-bnb-4bit | 14B | Reasoning, code generation |
| unsloth/qwen2-5-7b-bnb-4bit | 7B | Multilingual tasks, long context |
| unsloth/deepseek-r1-8b-bnb-4bit | 8B | Reasoning, complex problem-solving |
This table offers a starting point for selecting a model based on common use cases and model sizes supported by Unsloth.
🌟 Converting Unsloth Finetuned Models to GGUF Format
The GGUF format is essential for running LLMs on CPUs using tools like llama.cpp and Ollama. Unsloth simplifies the process of converting finetuned models to this format. A direct method involves using the save_pretrained_gguf function :
model.save_pretrained_gguf(“gguf_model”, tokenizer, quantization_method=“q4_k_m”)
The quantization_method parameter allows users to specify the desired quantization level for the GGUF file. Common options include q4_k_m, q8_0, and f16. An alternative, manual conversion method involves first saving the finetuned model in 16-bit format :
model.save_pretrained_merged(“merged_model”, tokenizer, save_method=“merged_16bit”)
Then, the user can utilize the conversion scripts provided by the llama.cpp repository. While generally straightforward, GGUF conversion can sometimes lead to a perceived loss of finetuning, which might be related to factors like insufficient training data, an inappropriate batch size, or the incorrect chat template being used. Experimenting with these parameters can often resolve such issues.
🌟 Current Applications and Use Cases of Unsloth-Finetuned Models
The efficiency of Unsloth has enabled a wide range of applications for finetuned LLMs and VLMs. In the financial domain, models finetuned with Unsloth can be used for sentiment analysis of news headlines to gauge their impact on companies. For customer support, Unsloth facilitates the creation of chatbots with customized responses derived from historical customer interactions. Developers are leveraging Unsloth to create specialized coding tools, such as autocomplete models trained on specific development data. In healthcare, Unsloth has been used to finetune models for mental health counseling applications. The library also supports the development of reasoning models through GRPO, which can be applied to domains like law and medicine to enhance the model’s ability to provide well-reasoned outputs. To illustrate basic inference with an Unsloth-finedtuned model, consider the following Python code snippet:
from unsloth import FastLanguageModel
🌌 Assuming ‘finetuned_model’ is the directory where your finetuned model is saved
model, tokenizer = FastLanguageModel.from_pretrained(“finetuned_model”)
prompt = “Write a short story about a lost dog.” inputs = tokenizer(prompt, return_tensors=“pt”).to(model.device) outputs = model.generate(**inputs, max_new_tokens=200) print(tokenizer.decode(outputs, skip_special_tokens=True))
The versatility and efficiency afforded by Unsloth are opening new avenues for deploying specialized LLMs in various resource-constrained environments and for niche applications.
🌟 Conclusion
Unsloth represents a significant advancement in the field of LLM and VLM finetuning. Its emphasis on speed and memory efficiency, coupled with a comprehensive set of features and broad model support, makes it an invaluable tool for researchers and practitioners alike. The streamlined process for GGUF conversion further enhances its utility by enabling easy deployment for local inference.
🔧 Works cited
1. Re-introducing Unsloth - Unsloth AI, https://unsloth.ai/blog/reintroducing 2. Entreprenerdly/unsloth - GitHub, https://github.com/Entreprenerdly/unsloth 3.
unslothai/unsloth: Finetune Llama 3.3, DeepSeek-R1, Gemma 3 & Reasoning LLMs 2x faster with 70% less memory! - GitHub, https://github.com/unslothai/unsloth 4. Fine-Tuning LLMs for Domain-Specific Tasks using Unsloth - ADaSci, https://adasci.org/fine-tuning-llms-for-domain-specific-tasks-using-unsloth/ 5. Unsloth AI - Hugging Face, https://huggingface.co/unsloth 6. Fine-tune Gemma 3 with Unsloth, https://www.unsloth.ai/blog/gemma3 7. Home · unslothai/unsloth Wiki - GitHub, https://github.com/unslothai/unsloth/wiki 8. Saving to GGUF - Unsloth Documentation, https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf 9. How to convert my fine-tuned model to .gguf ? : r/LocalLLaMA - Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1amjx77/how\_to\_convert\_my\_finetuned\_model\_to\_gguf/ 10. A Rapid Tutorial on Unsloth - Stephen Diehl, https://www.stephendiehl.com/posts/unsloth/ 11. Finetuning Llama-3.1 8b model Using Unsloth - YouTube, https://www.youtube.com/watch?v=dFj0xVOHz68 12. Fine-tuning Guide - Unsloth Documentation, https://docs.unsloth.ai/get-started/fine-tuning-guide 13. Train your own R1 reasoning model locally (GRPO) - Unsloth AI, https://unsloth.ai/blog/r1-reasoning 14. Qwen2 Vision Finetuning Unsloth - Kaggle, https://www.kaggle.com/code/danielhanchen/qwen2-vision-finetuning-unsloth-kaggle 15. Efficient Fine-Tuning for Vision-Language Models (VLMs) with Unsloth — Example with Qwen2-VL 7B - Colab, https://colab.research.google.com/drive/1zk\_naZas436nX7fZ8Wq9MB3Gwvy4YIy4?usp=sharing 16. Llama 3.2 Vision fine-tuning - Unsloth AI, https://unsloth.ai/blog/vision 17.
unslothai/unsloth-zoo: Utils for Unsloth - GitHub, https://github.com/unslothai/unsloth-zoo 18. All Our Models | Unsloth Documentation, https://docs.unsloth.ai/get-started/all-our-models 19. What Model Should I Use? - Unsloth Documentation, https://docs.unsloth.ai/get-started/beginner-start-here/what-model-should-i-use 20. Unsloth fine-tuning is lost when model is saved as GGUF : r/LocalLLaMA - Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1fxaxo6/unsloth\_finetuning\_is\_lost\_when\_model\_is\_saved\_as/ 21. unsloth/QwQ-32B-GGUF - Hugging Face, https://huggingface.co/unsloth/QwQ-32B-GGUF 22.
unsloth/r1-1776-GGUF · Discussions - Hugging Face, https://huggingface.co/unsloth/r1-1776-GGUF/discussions 23. unsloth/DeepSeek-R1-GGUF · A Step-by-step deployment guide with ollama, https://huggingface.co/unsloth/DeepSeek-R1-GGUF/discussions/16 24. R1-1776 Dynamic GGUFs by Unsloth : r/LocalLLaMA - Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1it0ocl/r11776\_dynamic\_ggufs\_by\_unsloth/ 25. Gemma 3 - GGUF + 16-bit uploads : r/unsloth - Reddit, https://www.reddit.com/r/unsloth/comments/1j9h9f8/gemma\_3\_gguf\_16bit\_uploads/ 26. FAQ + Is Fine-tuning Right For Me? | Unsloth Documentation, https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me 27. Install Unsloth on your Windows and Finetune LLM models Locally | by Vinod Kumar G R, https://medium.com/@vinodkumargr/install-unsloth-on-your-windows-and-finetune-llm-models-locally-20ebcce34014 28.