🌌 Creating Synthetic Data for Instruction Fine-Tuning with Local Large Language Models
🌟 1. Introduction: The Ascendancy of Synthetic Data in Large Language Model Training
🚀 Welcome to this comprehensive guide! This section will give you the foundational knowledge you need. The training of large language models (LLMs) necessitates vast quantities of high-quality data to achieve optimal performance across a diverse range of tasks. While traditional methods often rely on human-generated or web-scraped datasets, these approaches can be limited by factors such as data scarcity, cost, privacy concerns, and inherent biases. Synthetic data, which is artificially generated information designed to mimic real-world data, has emerged as a compelling alternative and a crucial component in the LLM fine-tuning landscape 1. This approach offers the unique ability to create tailored datasets that address specific needs and overcome the limitations associated with conventional data sources 1. The increasing accessibility and power of local LLMs further enhance the potential of synthetic data generation, allowing for privacy-preserving and cost-effective data creation directly on user machines 4.
🌟 2. Benefits of Synthetic Data for Large Language Model Fine-Tuning
The adoption of synthetic data for fine-tuning LLMs presents a multitude of advantages that contribute to the development of more robust, adaptable, and ethically sound models. One of the primary benefits is scalability, as synthetic data can be generated rapidly and in virtually unlimited quantities, enabling comprehensive model training without the delays associated with real-world data collection 1. This scalability often translates to significant cost-effectiveness, as generating artificial data is typically more affordable than the labor-intensive process of collecting, annotating, and curating real-world datasets 1. Beyond these core advantages, synthetic data provides a powerful means for bias mitigation. By carefully designing the generation process, developers can create balanced datasets that counteract biases present in human-generated data, leading to fairer and more inclusive AI models 1. The ability to generate data for custom scenarios and edge cases is another significant benefit, allowing for the training and testing of models in situations that are rare or difficult to capture in real-world data 1. This ensures that models are better prepared for the complexities and nuances of real-world applications 1.
🌟 3. Methods for Creating Synthetic Instruction Datasets
Several techniques exist for generating synthetic instruction datasets tailored for LLM fine-tuning, each with its own strengths and considerations. One common approach involves leveraging existing datasets as a foundation for generating new examples 7. This can involve using a larger LLM to generate variations or expansions of existing data points, effectively augmenting the original dataset with synthetic instances 10. Another method is rule-based synthetic data generation, where artificial data is created based on predefined rules and constraints 12.
Knowledge distillation is a technique where a stronger, often larger, “teacher” model is used to generate synthetic data that a smaller, more efficient “student” model can then be fine-tuned on 10. This method allows for the transfer of knowledge and reasoning capabilities from the teacher to the student model 11. Self-improvement is another approach where a model learns from its own generated responses through an iterative loop 7. This method avoids reliance on external models but requires careful monitoring to prevent the amplification of biases or errors 6.
🌟 4. Step-by-Step Guide: Generating Alpaca-Formatted Synthetic Data Using Local Large Language Models with Python Code
The Alpaca dataset format, characterized by “instruction,” “input,” and “output” fields, has become a popular structure for instruction fine-tuning 21. Generating data in this format using local LLMs with Python involves several steps:
🌟 4.1. Setting Up the Environment and Choosing a Local LLM. First, ensure that you have a Python environment set up with the necessary libraries. This typically includes libraries for interacting with local LLMs, such as ollama for models served via Ollama 27 or llama-cpp-python for models compatible with llama.cpp 4. You will also need the json library for handling JSON data. Choose a local LLM that suits your hardware capabilities and the desired quality of synthetic data. Models like Llama 2, Mistral, or their fine-tuned variants are commonly used 4.
🌟 4.2. Defining Instructions or Prompts. Prepare a list of instructions or prompts that you want the local LLM to generate responses for. These instructions should be diverse and cover the range of tasks you intend to fine-tune your model for 33.
🌟 4.3. Interacting with the Local LLM to Generate Responses. Use the chosen Python library to interact with your local LLM. For example, if using Ollama, you can use the ollama.generate() function 27. For llama-cpp-python, you would instantiate the Llama model and use its __call__ or create_completion methods 4. Ensure that your prompts are formatted appropriately for the chosen model.
🌟 4.4. Formatting the Generated Data into the Alpaca Template. Process the responses from the local LLM to fit the Alpaca format. This involves creating a dictionary for each instruction-response pair with the keys “instruction,” “input” (which can be an empty string if no specific input is needed), and “output” containing the generated response 21.
🌟 4.5. Saving the Alpaca-Formatted Data to a JSONL File. Save the generated list of Alpaca-formatted dictionaries into a JSON Lines (JSONL) file. Each line in the file should be a valid JSON object representing one instruction-input-output example 21. This format is commonly used for training LLMs with libraries like Hugging Face Transformers 25.
⚡ Example Python Code using Ollama:
Python
import ollama import json
def generate_alpaca_example(instruction_prompt, input_prompt="", model=“llama2”): """Generates an Alpaca-formatted example using a local LLM via Ollama.""" prompt = f”Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction_prompt}\n\n### Input:\n{input_prompt}\n\n### Response:”
response = ollama.generate(model=model, prompt=prompt) return {“instruction”: instruction_prompt, “input”: input_prompt, “output”: response[‘response’]}
def generate_alpaca_data(instructions, model=“llama2”): """Generates a list of Alpaca-formatted examples for a list of instructions.""" data = for instruction in instructions: example = generate_alpaca_example(instruction, model=model) data.append(example) return data
instructions_list =
alpaca_dataset = generate_alpaca_data(instructions_list)
with open(“alpaca_synthetic_data.jsonl”, “w”) as f: for entry in alpaca_dataset: json.dump(entry, f) f.write(“\n”)
print(“Alpaca-formatted synthetic data saved to alpaca_synthetic_data.jsonl”)
🌟 5. Step-by-Step Guide: Generating ChatML-Formatted Synthetic Data Using Local Large Language Models with Python Code
The ChatML format structures conversations with clear role indicators like “system,” “user,” and “assistant” 37. Generating synthetic data in this format with local LLMs and Python involves the following steps:
🌟 5.1. Setting Up the Environment and Choosing a Local LLM. This step is identical to the first step for generating Alpaca-formatted data. Ensure you have the necessary Python libraries installed and have chosen a suitable local LLM 4.
🌟 5.2. Defining Conversation Turns and Roles. Plan the structure of your synthetic conversations, including the number of turns and the role of each turn (system, user, or assistant) 37. The system role typically sets the context, the user provides instructions or queries, and the assistant provides responses.
🌟 5.3. Crafting Prompts for Each Role. Develop prompts that are specific to each role. For the system role, this might involve setting the persona or providing instructions. For the user role, it would be the actual query or instruction. For the assistant role, the prompt might guide the type of response expected 37.
🌟 5.4. Interacting with the Local LLM to Generate Content for Each Turn. Use your chosen Python library to interact with the local LLM for each turn in the conversation. Ensure the prompts align with the specific role you are generating content for 4.
🌟 5.5. Formatting the Generated Content into the ChatML Structure. Organize the generated content into a list of dictionaries, where each dictionary represents a turn in the conversation and has “role” and “content” keys 37.
🌟 5.6. Saving the ChatML-Formatted Data to a JSON File. Save the list of conversations (each being a list of turns) into a JSON file. Using indentation can improve readability 41.
⚡ Example Python Code using Ollama:
Python
import ollama import json
def generate_chatml_turn(role, content_prompt, model=“llama2”): """Generates content for a specific role in a ChatML conversation.""" prompt = f”Generate content for the ‘{role}’ role in a ChatML conversation based on the following: {content_prompt}” response = ollama.generate(model=model, prompt=prompt) return {“role”: role, “content”: response[‘response’]}
def generate_chatml_conversation(topic, model=“llama2”): """Generates a ChatML-formatted conversation about a given topic.""" conversation = system_prompt = f”You are a helpful assistant discussing the topic: {topic}.” conversation.append({“role”: “system”, “content”: ollama.generate(model=model, prompt=system_prompt)[‘response’]})
user_prompt = f”Start a conversation about {topic}.” conversation.append({“role”: “user”, “content”: ollama.generate(model=model, prompt=user_prompt)[‘response’]})
assistant_prompt = f”Continue the conversation about {topic} as the helpful assistant.” conversation.append({“role”: “assistant”, “content”: ollama.generate(model=model, prompt=assistant_prompt)[‘response’]})
return conversation
topic = “the process of fine-tuning LLMs” chatml_dataset = [generate_chatml_conversation(topic) for _ in range(3)]
with open(“chatml_synthetic_data.json”, “w”) as f: json.dump(chatml_dataset, f, indent=2)
print(“ChatML-formatted synthetic data saved to chatml_synthetic_data.json”)
🌟 6. Best Practices for Ensuring Quality, Diversity, and Realism in Synthetic Data
Generating high-quality, diverse, and realistic synthetic data requires careful planning and execution. Several best practices can enhance the effectiveness of this process.
🌟 6.1. Prompt Engineering Strategies for Diverse Data Generation. Employing a variety of prompts and instruction types is crucial for generating diverse synthetic data 33. This includes using open-ended generation prompts, classification tasks, editing instructions, and more. Incorporating different personas or styles by instructing the LLM to adopt specific roles can also significantly increase diversity 55.
Leveraging seed instructions and iterative refinement, starting with a small set of high-quality examples and using the LLM to generate more, can lead to broader coverage 3. Introducing randomness and controlling temperature during generation allows for a balance between creativity and coherence in the output 11.
🌟 6.2. Techniques for Maintaining Data Quality and Avoiding Bias. Implementing data quality checks to identify inconsistencies, inaccuracies, and errors is essential 1. Validating generated synthetic data against real-world data or expert knowledge helps ensure realism and accuracy 1. It is crucial to monitor for and mitigate bias in the generation process to ensure balanced representation and avoid amplifying existing biases 1.
Using model-in-the-loop approaches for continuous feedback and improvement, where the LLM itself helps evaluate and refine the synthetic data, can be beneficial 1.
🌟 6.3. Incorporating Human Oversight and Validation. Human review plays a vital role in ensuring the quality and relevance of synthetic data, as automated methods may miss nuances or subtle errors 2. Establishing methods for human annotation and verification of synthetic data, such as setting up feedback loops and utilizing annotation tools, can significantly improve data quality 8. Human experts can provide valuable corrections and improvements to the generated data, addressing factual errors, coherence issues, or biases 6.
🌟 7. Challenges and Considerations When Using Synthetic Data for Instruction Fine-Tuning
While synthetic data offers numerous benefits, it also presents certain challenges and considerations that need to be addressed.
🌟 7.1. Potential for Bias Transfer and Amplification. Biases present in the teacher model used for distillation or the prompts used for generation can be transferred to the synthetic data, potentially leading to skewed model outcomes 1. There is also a risk of amplifying existing biases present in the source data or the generation algorithm, resulting in unfair or discriminatory models 1.
🌟 7.2. Ensuring Realism and Avoiding Model Autophagy Disorder (MAD).
Ensuring that synthetic data closely mirrors real-world complexity and nuances can be challenging. If the generated data lacks realism, models trained on it may not perform well in practical situations 1. The concept of Model Autophagy Disorder (MAD) describes the degradation of LLM performance when trained solely on self-generated synthetic data without the infusion of new information or human oversight 61.
🌟 7.3. Evaluation and Validation of Synthetic Data. Evaluating the quality and utility of synthetic data is paramount to ensure it serves as a viable substitute for real data 1. Various validation techniques can be employed, including statistical checks, expert reviews, comparison with real-world data, and benchmarking on downstream tasks 1. Metrics for evaluating fidelity, utility, and privacy of synthetic data help ensure it meets the required standards 1.
🌟 8. Conclusion: Empowering Large Language Model Instruction Fine-Tuning with Locally Generated Synthetic Data
Synthetic data has emerged as a powerful tool for instruction fine-tuning of large language models, offering numerous advantages over traditional data sources. The ability to generate scalable, cost-effective, and privacy-preserving datasets, coupled with the potential for bias mitigation and customization for specific scenarios, makes synthetic data an invaluable asset in the development of robust and adaptable LLMs. To effectively leverage local LLMs for synthetic data generation, it is recommended to start with a clear definition of the target task and desired model behavior. Experimentation with different prompting strategies and local LLM frameworks is crucial for optimizing the generation process. Implementing robust data quality checks and validation processes, along with incorporating human oversight and feedback, are essential for refining the synthetic data and ensuring its suitability for fine-tuning. Continuous monitoring of the fine-tuned model’s performance and iterative adjustments to the synthetic data generation process will further enhance the effectiveness of this approach.
🔧 Works cited
1. Synthetic Data: Benefits and Techniques for LLM Fine-Tuning in 2025, accessed on March 19, 2025, https://labelyourdata.com/articles/llm-fine-tuning/synthetic-data 2. LLM synthetic data: Fine-tuning LLMs with AI-generated data | SuperAnnotate, accessed on March 19, 2025, https://www.superannotate.com/blog/llm-synthetic-data 3. Supervised Fine Tuning(SFT) with Synthetic data generation | by Sulbha Jain - Medium, accessed on March 19, 2025, https://medium.com/@sulbha.jindal/supervised-fine-tuning-sft-with-synthetic-data-generation-264d6a325ce5 4. Running Open Source LLMs In Python - A Practical Guide - Christopher Samiullah, accessed on March 19, 2025, https://christophergs.com/blog/running-open-source-llms-in-python 5. The 6 Best LLM Tools To Run Models Locally - GetStream.io, accessed on March 19, 2025, https://getstream.io/blog/best-local-llm-tools/ 6. Synthetic Data in LLMs: Human Supervision Required - Reworked, accessed on March 19, 2025, https://www.reworked.co/information-management/llms-are-hungry-for-data-synthetic-data-can-help/ 7. Using LLMs for Synthetic Data Generation: The Definitive Guide - Confident AI, accessed on March 19, 2025, https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms 8. Creating and Validating Synthetic Datasets for LLM Evaluation & Experimentation - Arize AI, accessed on March 19, 2025, https://arize.com/blog/creating-and-validating-synthetic-datasets-for-llm-evaluation-experimentation/ 9. Generate synthetic data with BigQuery DataFrames and LLMs | Google Cloud Blog, accessed on March 19, 2025, https://cloud.google.com/blog/products/data-analytics/generate-synthetic-data-with-bigquery-dataframes-and-llms 10. Fine-tune LLMs with synthetic data for context-based Q&A using Amazon Bedrock - AWS, accessed on March 19, 2025, https://aws.amazon.com/blogs/machine-learning/fine-tune-llms-with-synthetic-data-for-context-based-qa-using-amazon-bedrock/ 11. How to Generate and Use Synthetic Data for Finetuning - Eugene Yan, accessed on March 19, 2025, https://eugeneyan.com/writing/synthetic/ 12. Rule-Based Synthetic Test Data Generator | Syntho, accessed on March 19, 2025, https://www.syntho.ai/rule-based-synthetic-data/ 13. Synthetic Data 101: What is it, how it works, and what it’s used for - Syntheticus, accessed on March 19, 2025, https://syntheticus.ai/guide-everything-you-need-to-know-about-synthetic-data 14. The Latest Methods and Advancements in Using Synthetic Data for AI - Sapien, accessed on March 19, 2025, https://www.sapien.io/blog/the-latest-methods-and-advancements-in-using-synthetic-data-for-ai 15. Synthetic Data Generation: Addressing Data Scarcity and Bias in ML Models - Dataversity, accessed on March 19, 2025, https://www.dataversity.net/synthetic-data-generation-addressing-data-scarcity-and-bias-in-ml-models/ 16. Powering AI Innovation: Techniques for Synthetic Data Generation - Founding Minds, accessed on March 19, 2025, https://www.foundingminds.com/powering-ai-innovation-techniques-for-synthetic-data-generation/ 17. How To Generate Synthetic Data for Fine-Tuning LLMs with AI Alignment - Medium, accessed on March 19, 2025, https://medium.com/@dongchaochen/how-to-create-synthetic-data-for-fine-tuning-llms-with-ai-alignment-b7e04bb9ebdb 18. www.confident-ai.com, accessed on March 19, 2025, https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms#:~:text=When%20it%20comes%20to%20generating,from%20a%20more%20advanced%20model. 19. Synthetic data for LLM fine-tuning and alignment - Argilla, accessed on March 19, 2025, https://argilla.io/blog/synthetic-data/ 20. How to Create Synthetic Data at High Quality for Fine-Tuning LLMs - Gretel.ai, accessed on March 19, 2025, https://gretel.ai/blog/how-to-create-high-quality-synthetic-data-for-fine-tuning-llms 21. How to create a custom Alpaca instruction dataset for fine-tuning LLMs - Modern Coding, accessed on March 19, 2025, https://zackproser.com/blog/how-to-create-a-custom-alpaca-dataset 22. tatsu-lab/stanford_alpaca: Code and documentation to train Stanford’s Alpaca models, and generate the data. - GitHub, accessed on March 19, 2025, https://github.com/tatsu-lab/stanford_alpaca 23. What are Instruction Datasets for Fine-Tuning LLMs? - Hopsworks, accessed on March 19, 2025, https://www.hopsworks.ai/dictionary/instruction-datasets-for-fine-tuning-llms 24. Custom Fine-Tuning: Alpaca - Levanter - Read the Docs, accessed on March 19, 2025, https://levanter.readthedocs.io/en/latest/Fine-Tuning/ 25. Instruction Tuning GPT2 on Alpaca Dataset - DebuggerCafe, accessed on March 19, 2025, https://debuggercafe.com/instruction-tuning-gpt2-on-alpaca-dataset/ 26. Alpaca Instructions for Large Lanuage Model - Kaggle, accessed on March 19, 2025, https://www.kaggle.com/datasets/mnavaidd/alpaca-instructions-for-large-lanuage-model 27. Ollama Tutorial: Running LLMs Locally Made Super Simple - KDnuggets, accessed on March 19, 2025, https://www.kdnuggets.com/ollama-tutorial-running-llms-locally-made-super-simple 28. Using Ollama with Python: Step-by-Step Guide - Cohorte Projects, accessed on March 19, 2025, https://www.cohorte.co/blog/using-ollama-with-python-step-by-step-guide 29. OLLAMA: How to Run Local Language Models Like a Pro - Cheatsheet.md, accessed on March 19, 2025, https://cheatsheet.md/llm-leaderboard/ollama 30. A Simple, Practical Guide to Running Large-Language Models on Your Laptop - Medium, accessed on March 19, 2025, https://medium.com/predict/a-simple-comprehensive-guide-to-running-large-language-models-locally-on-cpu-and-or-gpu-using-c0c2a8483eee 31. Python with Stanford Alpaca and Vicuna 13B AI models - A llama-cpp-python Tutorial!, accessed on March 19, 2025, https://www.youtube.com/watch?v=-BidzsQYZM4 32. Llama.cpp Python Examples: A Guide to Using Llama Models with Python - Medium, accessed on March 19, 2025, https://medium.com/@aleksej.gudkov/llama-cpp-python-examples-a-guide-to-using-llama-models-with-python-1df9ba7a5fcd 33. Self-Instruct Style Data Generation: The Secret Behind Stanford Alpaca | by Okan Yenigün, accessed on March 19, 2025, https://medium.com/@okanyenigun/self-instruct-style-data-generation-the-secret-behind-stanford-alpaca-e1575ea9ad71 34. Synthetic Data Generation Strategies for Fine-Tuning LLMs - Scale AI, accessed on March 19, 2025, https://scale.com/blog/synthetic-data-fine-tuning-llms 35. ollama/docs/api.md at main · ollama/ollama - GitHub, accessed on March 19, 2025, https://github.com/ollama/ollama/blob/main/docs/api.md 36. Responsible Synthetic Data Creation for Fine-Tuning with RAFT Distillation, accessed on March 19, 2025, https://techcommunity.microsoft.com/blog/educatordeveloperblog/responsible-synthetic-data-creation-for-fine-tuning-with-raft-distillation/4259367 37. smol-course/1_instruction_tuning/chat_templates.md at main - GitHub, accessed on March 19, 2025, https://github.com/huggingface/smol-course/blob/main/1_instruction_tuning/chat_templates.md 38. InternLM/chat/chat_format.md at main - GitHub, accessed on March 19, 2025, https://github.com/InternLM/InternLM/blob/main/chat/chat_format.md 39. Templates - Hugging Face, accessed on March 19, 2025, https://huggingface.co/docs/transformers/main/chat_templating 40. Demystifying Chat Templates of LLM using llama-cpp and ctransformers | by Ahmet Celebi, accessed on March 19, 2025, https://medium.com/@ahmet_celebi/demystifying-chat-templates-of-llm-using-llama-cpp-and-ctransformers-f17871569cd6 41. ChatML + chat templates + Mistral 7b full example.ipynb - Colab - Google, accessed on March 19, 2025, https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing 42. Step 5: Defining LLM Prompt - Prompt Roles | Automated hands-on| CloudxLab, accessed on March 19, 2025, https://cloudxlab.com/assessment/displayslide/8696/step-5-defining-llm-prompt-prompt-roles 43. Enum “AOAI Chat Roles” - Microsoft Learn, accessed on March 19, 2025, https://learn.microsoft.com/en-us/dynamics365/business-central/application/system-application/enum/system.ai.aoai-chat-roles 44. ChatGPT Roles Explained: User, Developer (System), Assistant - YouTube, accessed on March 19, 2025, https://www.youtube.com/watch?v=xbpdMkTz8L4 45. Understanding User, Assistant, and System Roles in ChatGPT - Baeldung, accessed on March 19, 2025, https://www.baeldung.com/cs/chatgpt-api-roles 46. What is the difference between System, User, and Assistant roles in ChatGPT?, accessed on March 19, 2025, https://community.make.com/t/what-is-the-difference-between-system-user-and-assistant-roles-in-chatgpt/36160 47. How to Get Responses From Local LLM Models With Python - HackerNoon, accessed on March 19, 2025, https://hackernoon.com/how-to-get-responses-from-local-llm-models-with-python 48. Using LLMs in Python - YouTube, accessed on March 19, 2025, https://www.youtube.com/watch?v=uT1WmUC_Aj8 49. How to run LLM locally with ollama | Python example - YouTube, accessed on March 19, 2025, https://www.youtube.com/watch?v=IcBnE6J2gpk 50. whats the best local llm for code in python? : r/LocalLLaMA, accessed on March 19, 2025, https://www.reddit.com/r/LocalLLaMA/comments/1jbi8xm/whats_the_best_local_llm_for_code_in_python/ 51. Run models locally | 🦜️ LangChain, accessed on March 19, 2025, https://python.langchain.com/docs/how_to/local_llms/ 52. How to Run LLMs Locally with Python - Picovoice, accessed on March 19, 2025, https://picovoice.ai/blog/how-to-run-llms-locally-with-python/ 53. No more hard prompts: SoftSRV prompting for synthetic data generation - arXiv, accessed on March 19, 2025, https://arxiv.org/html/2410.16534v2 54. Leveraging LLMs for Synthetic Data Generation - Deepchecks, accessed on March 19, 2025, https://www.deepchecks.com/leveraging-llms-synthetic-data-generation/ 55. LLM-Driven Synthetic Data Generation, Curation & Evaluation | by Cobus Greyling | Medium, accessed on March 19, 2025, https://cobusgreyling.medium.com/llm-driven-synthetic-data-generation-curation-evaluation-33731e33b525 56. How To T̶r̶a̶i̶n̶ Synthesize Your D̶r̶a̶g̶o̶n̶ Data - Answer. AI, accessed on March 19, 2025, https://www.answer.ai/posts/2024-10-15-how-to-synthesize-data.html 57. On the Diversity of Synthetic Data and its Impact on Training Large Language Models - arXiv, accessed on March 19, 2025, https://arxiv.org/html/2410.15226v2 58. LLM on Your Laptop: A Synthetic Data Generation Guide — Using Ollama and Small LLMs | by Nikhil Shrimali | Analytics Vidhya | Medium, accessed on March 19, 2025, https://medium.com/analytics-vidhya/generating-synthetic-data-locally-a-guide-using-ollama-and-small-llms-9dafa4bd0f93 59. What’s the best way to create a large synthetic data set with a Local LLM? : r/LocalLLaMA, accessed on March 19, 2025, https://www.reddit.com/r/LocalLLaMA/comments/1feh2xv/whats_the_best_way_to_create_a_large_synthetic/ 60. How Synthetic Data Advances And Protects LLM Development - Forbes, accessed on March 19, 2025, https://www.forbes.com/councils/forbestechcouncil/2024/05/16/how-synthetic-data-advances-and-protects-llm-development/ 61. Three Key Insights on Synthetic Data for LLM Training | by Joyce Birkins | Medium, accessed on March 19, 2025, https://medium.com/@joycebirkins/three-key-insights-on-synthetic-data-for-llm-training-36f00b271679 62. Synthetic Data Generation with LLMs: What You Need to Know - Deepchecks, accessed on March 19, 2025, https://www.deepchecks.com/what-to-know-synthetic-data-generation-llms/ 63. Local Synthetic Data Generation using LLama 3.2 and Ollama - Analytics Vidhya, accessed on March 19, 2025, https://www.analyticsvidhya.com/blog/2025/01/local-synthetic-data-generation/ 64. Understanding Quality in Generative AI Training Datasets - Anolytics, accessed on March 19, 2025, https://www.anolytics.ai/blog/understanding-quality-in-generative-ai-training-datasets/ 65. Best Practices and Lessons Learned on Synthetic Data for Language Models - arXiv, accessed on March 19, 2025, https://arxiv.org/html/2404.07503v1 66. How to create LLM test datasets with synthetic data - Evidently AI, accessed on March 19, 2025, https://www.evidentlyai.com/llm-guide/llm-test-dataset-synthetic-data