⚡ Technical Documentation

Formatting Documents For Rag Ingestion With Llms: An Evaluation Of Microsoft Markitdown And Python Alternatives

Technical guide covering **formatting documents for rag ingestion with llms: an evaluation of microsoft markitdown and python alternatives**

👤
Author
Cosmic Lounge AI Team
📅
Updated
6/1/2025
⏱️
Read Time
10 min
Topics
#llm #ai #model #api #server #configuration #development #code #introduction #design

📖 Reading Mode

📖 Table of Contents

🌌 Formatting Documents for RAG Ingestion with LLMs: An Evaluation of Microsoft MarkItDown and Python Alternatives

🌟 Introduction

🚀 Welcome to this comprehensive guide! This section will give you the foundational knowledge you need. The effectiveness of Retrieval-Augmented Generation (RAG) systems, which enhance the capabilities of Large Language Models (LLMs) by grounding their responses in external knowledge, is significantly influenced by the format of the ingested documents 1. Well-structured and easily parsable content allows LLMs to better understand and utilize the retrieved information, leading to more accurate and contextually relevant outputs 1. This report explores solutions for formatting documents specifically for RAG ingestion, focusing on Microsoft’s MarkItDown utility and a range of alternative Python libraries.

🌟 Microsoft MarkItDown: A Purpose-Built Tool for LLM-Friendly Content

Microsoft MarkItDown is a Python utility designed to convert various file formats into Markdown, a lightweight markup language favored for its simplicity and readability by both humans and machines 1. Its development acknowledges the increasing importance of structured data for generative AI and LLM applications, particularly in the context of RAG 4.

⚡ Implementation and Python Code Examples

MarkItDown can be installed using pip with the command pip install ‘markitdown[all]’ to include dependencies for all supported file formats 3. Alternatively, specific dependencies can be installed as needed, for example, pip install markitdown[pdf, docx, pptx] 5. Basic usage in Python involves instantiating the MarkItDown class and calling the convert() method with the file path 3. The converted content is accessible through the text_content attribute of the result object 3. For instance, to convert a PowerPoint file:

Python

from markitdown import MarkItDown md = MarkItDown() result = md.convert(“/content/Dickinson_Sample_Slides.pptx”) print(result.text_content)

MarkItDown can also leverage Azure Document Intelligence for enhanced conversion, particularly for PDFs. This requires providing the Document Intelligence endpoint during initialization:

Python

from markitdown import MarkItDown md = MarkItDown(docintel_endpoint=“<document_intelligence_endpoint>”) result = md.convert(“test.pdf”) print(result.text_content)

Furthermore, MarkItDown can integrate with Large Language Models, such as OpenAI’s GPT-4o, to generate descriptions for images. This is achieved by providing an LLM client and model during initialization:

Python

from markitdown import MarkItDown from openai import OpenAI client = OpenAI() md = MarkItDown(llm_client=client, llm_model=“gpt-4o”) result = md.convert(“example.jpg”) print(result.text_content)

⚡ Functions and Abilities for RAG

MarkItDown supports a wide array of file types, including PDF, PowerPoint, Word, Excel, Images (with EXIF metadata and OCR), Audio (with EXIF metadata and speech transcription), HTML, text-based formats (CSV, JSON, XML), ZIP files, YouTube URLs, and EPubs 4. Its primary focus is on preserving important document structure as Markdown, including headings, lists, tables, and links, which is crucial for effective RAG 5. The utility can handle multi-tab Excel spreadsheets and recursively parses ZIP archives 6. For image processing, it uses OCR and integrates with LLMs to generate descriptions, enhancing the context for RAG systems 6.

⚡ Limitations

Despite its capabilities, MarkItDown has certain limitations. It cannot process PDF files without prior OCR, and formatting is not preserved during PDF extraction 6. Additionally, the need for an LLM for image descriptions means that image extraction might initially yield no results without proper configuration 6.

⚡ Extensibility

MarkItDown supports third-party plugins, which are disabled by default 5. These plugins can extend the functionality of the tool. Installed plugins can be listed using the command markitdown —list-plugins, and plugins can be enabled during conversion with markitdown —use-plugins path-to-file.pdf 5. Developers interested in creating plugins can refer to the packages/markitdown-sample-plugin directory within the repository 5.

🌟 Python-Based Alternatives for Document Formatting in RAG Systems

While MarkItDown offers a focused solution, several other Python libraries provide document formatting and processing capabilities that are suitable for RAG ingestion.

⚡ Unstructured: Flexible Document Parsing for Diverse Formats

Unstructured is a library designed to preprocess and structure unstructured text documents from various file formats, including PDFs, HTML, Word documents, and more, making them suitable for downstream machine learning tasks like RAG 9. It aims to enable organizations to access all their data for building RAG pipelines 9.

Implementation and Python Code Examples: Unstructured can be installed with pip install unstructured[all-docs] to include dependencies for all supported document types 12. Its primary usage is through data loaders, such as UnstructuredLoader in Langchain, which can partition documents locally or remotely using the serverless Unstructured API 11. Python

from langchain_unstructured import UnstructuredLoader loader = UnstructuredLoader(file_path=“path/to/your/document.pdf”, mode=“elements”) elements = loader.load() for element in elements: print(element.type, element.metadata, element.page_content)

The library offers strategies for parsing PDFs, including “fast” and “hi-res” modes 9. It can extract various elements like titles, narrative text, images, and tables 9. Specific elements, such as tables, can be targeted for extraction 13.

Strengths and Weaknesses for RAG: Unstructured supports a wide range of document types and can extract diverse elements, making it versatile for RAG applications dealing with heterogeneous data 9. However, the open-source version has limitations in performance and features compared to the paid API 14. Using the API might incur costs depending on usage 9.

⚡ Docling: Advanced Document Understanding and Structured Output

Docling is a Python library developed by IBM for parsing various document formats, including PDF, DOCX, HTML, Markdown, and PPTX, and exporting their content into structured formats like JSON and Markdown 15. It utilizes state-of-the-art AI models for layout analysis and table structure recognition 16.

Implementation and Python Code Examples: Docling can be installed using pip install docling 15. Basic conversion involves importing DocumentConverter and specifying the file path 15. Python

from docling.document_converter import DocumentConverter converter = DocumentConverter() source = “path/to/your/document.pdf” document = converter.convert(source) print(document.to_markdown())

Docling integrates with Langchain via DoclingLoader, allowing easy use of its capabilities within Langchain workflows 16. It supports exporting to both JSON and Markdown formats, providing structured representations of the document content and metadata 15. The library can also identify and extract tables, exporting them to CSV or HTML formats 15. It avoids OCR when possible, relying on computer vision models for faster and more accurate processing 18.

Strengths and Weaknesses for RAG: Docling’s strength lies in its multi-format support and advanced document understanding, including layout and table structures, which are crucial for complex RAG scenarios 15. Its efficient performance and open-source nature make it a strong alternative for RAG systems dealing with diverse and intricate document formats.

⚡ PyMuPDF4LLM: Efficient PDF to Markdown Conversion for LLMs

PyMuPDF4LLM is a Python package built on top of PyMuPDF, specifically designed to facilitate the extraction of PDF content in formats suitable for LLM and RAG environments 21. It aims to simplify the process of using PDF data with LLMs by providing efficient conversion to Markdown and direct integration with RAG frameworks.

Implementation and Python Code Examples: PyMuPDF4LLM can be installed using pip install pymupdf4llm 23. Converting a PDF to Markdown is straightforward:

Python

import pymupdf4llm md_text = pymupdf4llm.to_markdown(“input.pdf”) print(md_text)

The library supports multi-column pages and can extract images and vector graphics, including references in the Markdown output 22. It can also output page content as a list of dictionaries, providing metadata along with the Markdown text for each page 22. PyMuPDF4LLM offers direct support for LlamaIndex Documents and integrates with Langchain via PyMuPDFLoader, making it seamless to use with popular RAG frameworks 21.

Strengths and Weaknesses for RAG: PyMuPDF4LLM is particularly well-suited for RAG applications that primarily work with PDF documents. Its efficient and reliable PDF processing, coupled with direct integration with LLM frameworks, makes it an excellent choice for such scenarios. However, its focus is mainly on PDF documents, which might be a limitation for RAG systems dealing with a broader range of file types.

⚡ Marker: Accurate and Fast Conversion to Markdown, JSON, and HTML

Marker is a Python library focused on accurately and quickly converting documents, including PDF, image, PPTX, DOCX, XLSX, HTML, and EPUB files, into Markdown, JSON, and HTML formats 9. It is optimized for handling complex documents like books and scientific papers, preserving layout, formatting, and content.

Implementation and Python Code Examples: Marker can be installed using pip install marker-pdf 27. To convert a single PDF file to Markdown:

Python

import os os.makedirs(“output”, exist_ok=True) os.system(“marker_single /path/file.pdf /path/output”)

Multiple PDF files can be converted simultaneously using parallel processing:

Python

os.system(“marker /path/data /path/output —workers 4”)

Marker can optionally use an LLM (specifically Google’s Gemini via API key) to improve conversion accuracy, especially for inline math 28. It can extract and save images, remove headers and footers, and format tables and code blocks 26.

Strengths and Weaknesses for RAG: Marker’s high accuracy in maintaining the original layout and formatting of complex documents makes it a strong contender for RAG systems where fidelity is crucial, such as those dealing with scientific literature or highly structured content 26. Its efficient processing capabilities and support for multiple output formats add to its appeal. However, it might require pre-processing for non-OCRed PDFs, similar to MarkItDown.

⚡ Python-Markdown: A Versatile Library for Markdown Processing

Python-Markdown is a Python implementation of John Gruber’s Markdown specification, primarily intended for use as a library to convert Markdown syntax into HTML 29. It is highly compliant with the original implementation and offers an extensive API for customization and extension.

Implementation and Python Code Examples: Python-Markdown can be installed using pip install markdown 31. Basic conversion from a text string is simple:

Python

import markdown html = markdown.markdown(“This is some **markdown** text.”) print(html)

Markdown files can also be converted directly:

Python

import markdown with open(“input.md”, “r”) as f: text = f.read() html = markdown.markdown(text) with open(“output.html”, “w”) as f: f.write(html)

Python-Markdown supports various extensions that can add or modify the base syntax, such as table support, fenced code blocks, and more 29.

Strengths and Weaknesses for RAG: While Python-Markdown does not directly convert from other document formats to Markdown, it is essential for RAG pipelines that involve processing or manipulating existing Markdown content. Its flexibility and extensive extension support make it a valuable tool for customizing Markdown for optimal LLM ingestion.

⚡ spaCy (with spaCy Layout): Leveraging NLP for Structured Document Extraction

spaCy is a popular Python library for advanced Natural Language Processing, known for its speed and efficiency 34. The spaCy Layout plugin integrates with Docling to bring structured processing of PDFs, Word documents, and other formats into the spaCy pipeline 38. This combination allows for the application of spaCy’s NLP techniques to structured document content, including linguistic analysis, named entity recognition, and text classification, which can be beneficial for RAG.

Implementation and Python Code Examples: First, install spaCy and spaCy Layout: pip install spacy spacy-layout. Then, download a spaCy language model (e.g., python -m spacy download en_core_web_sm) 39. Processing a document involves loading the spaCy model and the spaCyLayout extension:

Python

import spacy from spacy_layout import spaCyLayout nlp = spacy.load(“en_core_web_sm”) layout = spaCyLayout(nlp) doc = layout(“document.pdf”) print(doc.text)

spaCy Layout outputs clean, structured data in a text-based format and creates spaCy Doc objects with labeled spans for sections, headings, and tables 38. Tables are converted to pandas DataFrames, and layout spans provide information about the content type and layout features 40. The pipe method allows for processing multiple documents efficiently 40.

Strengths and Weaknesses for RAG: The integration of NLP capabilities with document parsing makes spaCy with spaCy Layout a powerful tool for preparing documents for RAG. Its ability to understand document structure and extract labeled spans can facilitate more intelligent chunking and information retrieval. However, it relies on Docling for the underlying document conversion, so its capabilities are tied to Docling’s format support.

⚡ Gensim: Topic Modeling and Document Similarity for Semantic Chunking

Gensim is a Python library focused on topic modeling, document indexing, and similarity retrieval for large text corpora 34. While not a direct document formatting tool, Gensim’s capabilities can be leveraged in RAG pipelines for semantic analysis and chunking.

Implementation and Python Code Examples: Gensim can be installed with pip install gensim 36. A typical workflow involves creating a dictionary of words from the documents, converting documents to a bag-of-words format, and then applying transformations like TF-IDF or topic modeling algorithms like LSI or LDA 43. Python

from gensim import corpora, models

documents = texts = [doc.lower().split() for doc in documents] dictionary = corpora. Dictionary(texts) corpus = [dictionary.doc2bow(text) for text in texts] tfidf = models. TfidfModel(corpus) corpus_tfidf = tfidf[corpus]

Gensim can be used to calculate document similarity, which can inform semantic chunking strategies by identifying related content 43.

Strengths and Weaknesses for RAG: Gensim’s ability to perform topic modeling and document similarity analysis can be valuable for RAG by enabling semantic chunking and improving the relevance of retrieved information. However, it operates on text and does not handle document format conversion directly.

⚡ NLTK (Natural Language Toolkit): Fundamental Tools for Text Processing and Chunking

NLTK is a comprehensive Python library for working with human language data, providing tools for tasks like tokenization, stemming, lemmatization, tagging, parsing, and more 20. It offers fundamental text processing capabilities that can be integrated into RAG pipelines.

Implementation and Python Code Examples: NLTK can be installed using pip install nltk 20. Common tasks include sentence tokenization:

Python

import nltk nltk.download(‘punkt’) text = “This is the first sentence. Here is the second sentence.” sentences = nltk.sent_tokenize(text) print(sentences)

NLTK also provides tools for word tokenization, stemming, and lemmatization, which can be used to normalize text before ingestion into a RAG system 20. Langchain also provides text splitters that utilize NLTK for chunking 46.

Strengths and Weaknesses for RAG: NLTK offers basic but essential tools for text preprocessing and chunking in RAG pipelines. It is widely used and well-documented. However, for large-scale applications or more advanced NLP tasks, other libraries like spaCy might offer better performance. NLTK does not handle document format conversion directly.

🌟 Comparative Analysis: Choosing the Right Tool for Your RAG Pipeline

Selecting the appropriate tool for formatting documents for RAG ingestion depends on various factors, including the types of documents being processed, the desired level of structure in the output, the need for advanced NLP capabilities, and the ease of integration with existing RAG frameworks.

⚡ Feature Comparison Table

FeatureMicrosoft MarkItDownUnstructuredDoclingPyMuPDF4LLMMarkerPython-MarkdownspaCy (with Layout)GensimNLTK
Document FormatsWide rangeWide rangePDF, DOCX, PPTX, HTML, Markdown, ImagePDF, Office (via Pro)PDF, Image, PPTX, DOCX, XLSX, HTML, EPUBMarkdownPDF, Word, Other (via Docling)TextText
Markdown Output QualityGoodVariesGoodExcellent (for PDFs)ExcellentExcellentGoodN/ABasic
Handling of TablesGoodYesExcellentGoodExcellentBasicGood (via Docling)N/AN/A
Handling of Images/AudioYes (with LLM)YesYesYesYesNoYes (via Docling)N/AN/A
Integration with LLMsYes (e.g., OpenAI)Indirect (output to text)Indirect (output to Markdown/JSON)Excellent (designed for LLMs)Yes (via Gemini API)NoYesYes (for semantic analysis)Yes (for preprocessing)
Integration with RAGYesExcellent (via Langchain)Excellent (via Langchain, LlamaIndex)Excellent (via Langchain, LlamaIndex)Indirect (output to Markdown/JSON/HTML)NoYesYes (for semantic chunking)Yes (for chunking)
Ease of UseVery GoodGoodGoodVery GoodGoodVery GoodGoodModerateGood
PerformanceGoodVaries (API faster)GoodExcellent (for PDFs)ExcellentExcellentGoodGoodGood
Extensibility (Plugins)YesNoYesNoYes (via processors)Yes (via extensions)YesYesYes
LicensingMITApache 2.0MITMITMITBSD-3-ClauseApache 2.0LGPL-2.1Apache 2.0

⚡ Use Case Scenarios and Recommendations

  • PDF-heavy RAG systems: PyMuPDF4LLM and Marker are excellent choices due to their strong PDF processing capabilities and focus on high-quality Markdown conversion. PyMuPDF4LLM’s direct integration with Langchain and LlamaIndex makes it particularly attractive.

  • Diverse document formats: Unstructured and Docling offer broad format support and are well-integrated with RAG frameworks. Docling’s advanced document understanding can be beneficial for complex layouts.

  • Processing existing Markdown content: Python-Markdown is the go-to library for parsing and manipulating Markdown files.

  • Leveraging NLP for document understanding: spaCy with spaCy Layout combines document parsing with powerful NLP features, enabling more sophisticated preprocessing for RAG.

  • Semantic chunking and topic analysis: Gensim can be used to understand the semantic relationships within documents, which can inform chunking strategies.

  • Basic text preprocessing and chunking: NLTK provides fundamental tools for these tasks, suitable for simpler RAG pipelines or as part of a broader preprocessing workflow. The choice ultimately depends on the specific requirements of the RAG application, the types and complexity of the documents involved, and the desired balance between ease of use, performance, and advanced features.

🌟 Best Practices for Formatting Documents for Optimal RAG Ingestion

Effective document formatting for RAG involves more than just converting to Markdown. It requires strategies that enhance the LLM’s ability to understand and utilize the ingested knowledge.

⚡ Semantic Chunking Strategies and Markdown Structure

Maintaining semantic context during chunking is crucial for high-quality retrieval in RAG systems 2. Markdown’s structure, with headings and lists, provides natural boundaries for creating meaningful chunks 1. For instance, splitting documents based on headings ensures that each chunk represents a coherent section of information. Tables and code blocks should be formatted correctly in Markdown to be easily interpreted by LLMs 2.

⚡ Handling Metadata and Document Structure

Preserving and incorporating metadata, such as document titles, authors, and creation dates, can significantly enhance the performance of RAG systems by allowing for filtering and providing additional context to the LLM 50. Markdown supports metadata through conventions like YAML front matter, which can be included at the beginning of a Markdown file.

⚡ Considerations for Different Document Types

Formatting different document types presents unique challenges. PDFs might require OCR if they are scanned images, and the formatting can be complex to preserve 2. Office documents often need to be converted to an intermediary format like HTML before being transformed into Markdown 6. Web content might require cleaning and extraction of relevant information before formatting 2. The presence and quality of OCR can significantly impact the accuracy and formatting of extracted text 2.

🌟 Conclusion: Empowering LLMs with Well-Formatted Knowledge

This report has examined Microsoft MarkItDown and a range of Python-based alternatives for formatting documents for RAG ingestion. MarkItDown offers a user-friendly solution for converting various file types to Markdown, with specific features for LLM integration and Azure Document Intelligence. However, its limitations with non-OCRed PDFs and formatting preservation highlight the need to consider alternative tools based on specific requirements. Libraries like Unstructured and Docling provide broader format support and advanced document understanding, making them suitable for diverse and complex document collections. PyMuPDF4LLM and Marker excel in accurately converting PDFs to Markdown, with features tailored for LLM and RAG applications. Python-Markdown remains essential for processing existing Markdown content, while spaCy with spaCy Layout offers a unique approach by integrating NLP capabilities. Choosing the right tool is crucial for optimizing the performance of RAG pipelines. By carefully considering the document types, complexity, and the specific needs of the application, developers can leverage these tools and best practices to ensure that LLMs are empowered with well-formatted and easily understandable knowledge, leading to more accurate and relevant generated responses.

🔧 Works cited

1. Boosting AI Performance: The Power of LLM-Friendly Content in Markdown, accessed on March 24, 2025, https://developer.webex.com/blog/boosting-ai-performance-the-power-of-llm-friendly-content-in-markdown 2. Retrieval-Augmented Generation (RAG) with Azure AI Document Intelligence, accessed on March 24, 2025, https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation?view=doc-intel-4.0.0 3. Microsoft MarkItDown + Ollama and LLaVA: Markdown Conversion with LLM - Medium, accessed on March 24, 2025, https://medium.com/@giacomo__95/markitdown-ollama-and-llava-markdown-conversion-with-microsofts-markitdown-and-ollama-s-llm-2141bba9d183 4. MarkItDown utility and LLMs are great match - Kalle Marjokorpi, accessed on March 24, 2025, https://www.kallemarjokorpi.fi/blog/markitdown-utility-and-llms-are-great-match/ 5. microsoft/markitdown: Python tool for converting files and office documents to Markdown. - GitHub, accessed on March 24, 2025, https://github.com/microsoft/markitdown 6. Deep Dive into Microsoft MarkItDown - Leapcell, accessed on March 24, 2025, https://leapcell.io/blog/deep-dive-into-microsoft-markitdown 7. Deep Dive into Microsoft MarkItDown - DEV Community, accessed on March 24, 2025, https://dev.to/leapcell/deep-dive-into-microsoft-markitdown-4if5 8. MarkItDown is a new Python Library from Microsoft that aims to convert everything to Markdown - Corti.com, accessed on March 24, 2025, https://corti.com/markitdown-is-a-python-library-that-aims-to-convert-everything-to-markdown-2/ 9. RAG — Three Python libraries for Pipeline-based PDF parsing - AI Bites, accessed on March 24, 2025, https://www.ai-bites.net/rag-three-python-libraries-for-pipeline-based-pdf-parsing/ 10. The RAG Engineer’s Guide to Document Parsing : r/LangChain - Reddit, accessed on March 24, 2025, https://www.reddit.com/r/LangChain/comments/1ef12q6/the_rag_engineers_guide_to_document_parsing/ 11. Unstructured - ️ LangChain, accessed on March 24, 2025, https://python.langchain.com/docs/integrations/providers/unstructured/ 12. How to convert PDF DOCX to Structured TXT Formats for RAG! (UNSTRUCTURED Tutorial), accessed on March 24, 2025, https://www.youtube.com/watch?v=iPiYVCl002o 13. Local RAG Explained with Unstructured and LangChain - YouTube, accessed on March 24, 2025, https://www.youtube.com/watch?v=7ru0oj0qIys 14. Unstructured Open Source, accessed on March 24, 2025, https://docs.unstructured.io/open-source/introduction/overview 15. Building Document Parsing Pipelines with Python | by Lasha Dolenjashvili | Medium, accessed on March 24, 2025, https://lasha-dolenjashvili.medium.com/building-document-parsing-pipelines-with-python-3c06f62569ad 16. Docling - ️ LangChain, accessed on March 24, 2025, https://python.langchain.com/docs/integrations/document_loaders/docling/ 17. RAG with Docling and LlamaIndex for Personal Documents - GitHub, accessed on March 24, 2025, https://github.com/homerokzam/rag-docling-llamaindex 18. Enhancing Multimodal RAG Capabilities Using Docling - Analytics Vidhya, accessed on March 24, 2025, https://www.analyticsvidhya.com/blog/2025/03/enhancing-multimodal-rag-capabilities-using-docling/ 19. Docling: AI Powered Document Parsing for LLMs and RAG - YouTube, accessed on March 24, 2025, https://www.youtube.com/watch?v=0vmQ7BmLax8 20. Using NLTK to Improve RAG (Retrieval Augmented Generation) Text Quality - Mindfire Technology, accessed on March 24, 2025, https://www.mindfiretechnology.com/blog/archive/using-nltk-to-improve-rag-retrieval-augmented-generation-text-quality/ 21. PyMuPDF, LLM & RAG - Read the Docs, accessed on March 24, 2025, https://pymupdf.readthedocs.io/en/latest/rag.html 22. PyMuPDF4LLM - PyMuPDF 1.25.4 documentation, accessed on March 24, 2025, https://pymupdf.readthedocs.io/en/latest/pymupdf4llm 23. pymupdf4llm - PyPI, accessed on March 24, 2025, https://pypi.org/project/pymupdf4llm/ 24. Introducing PyMuPDF4LLM - Medium, accessed on March 24, 2025, https://medium.com/@pymupdf/introducing-pymupdf4llm-d2c39442f445 25. PyMuPDF and PyMuPDF4LLM - Prepare PDF for LLM and RAG - Install Locally - YouTube, accessed on March 24, 2025, https://www.youtube.com/watch?v=xR7er853eek 26. Marker: A New Python-based Library that Converts PDF to Markdown Quickly and Accurately - AI Toolhouse Blog, accessed on March 24, 2025, https://blog.aitoolhouse.com/marker-a-new-python-based-library-that-converts-pdf-to-markdown-quickly-and-accurately/ 27. Simple Ways to Parse PDFs for Better RAG Systems | by kirouane Ayoub | GoPenAI, accessed on March 24, 2025, https://blog.gopenai.com/simple-ways-to-parse-pdfs-for-better-rag-systems-82ec68c9d8cd 28. VikParuchuri/marker: Convert PDF to markdown + JSON quickly with high accuracy - GitHub, accessed on March 24, 2025, https://github.com/VikParuchuri/marker 29. Library Reference — Python-Markdown 3.7 documentation, accessed on March 24, 2025, https://python-markdown.github.io/reference/ 30. Python-Markdown — Python-Markdown 3.7 documentation, accessed on March 24, 2025, https://python-markdown.github.io/ 31. A Python implementation of John Gruber’s Markdown with Extension support. - GitHub, accessed on March 24, 2025, https://github.com/Python-Markdown/markdown 32. Awesome Markdown | Curated list of awesome lists, accessed on March 24, 2025, https://project-awesome.org/BubuAnabelas/awesome-markdown 33. Markdown - PyPI, accessed on March 24, 2025, https://pypi.org/project/Markdown/ 34. Top 10 NLP Tools in Python for Text Analysis Applications - The New Stack, accessed on March 24, 2025, https://thenewstack.io/top-10-nlp-tools-in-python-for-text-analysis-applications/ 35. Top 25 NLP Libraries for Python for Effective Text Analysis - upGrad, accessed on March 24, 2025, https://www.upgrad.com/blog/python-nlp-libraries-and-applications/ 36. NLP Libraries in Python - GeeksforGeeks, accessed on March 24, 2025, https://www.geeksforgeeks.org/nlp-libraries-in-python/ 37. 7 Top NLP Libraries For NLP Development [Updated] - Labellerr, accessed on March 24, 2025, https://www.labellerr.com/blog/top-7-nlp-libraries-for-nlp-development/ 38. spacy-layout · spaCy Universe, accessed on March 24, 2025, https://spacy.io/universe/project/spacy-layout 39. Introduction to spaCyLayout and PDF Extraction | by Abonia Sojasingarayar - Medium, accessed on March 24, 2025, https://medium.com/@abonia/introduction-to-spacylayout-and-pdf-extraction-a945e7a627cc 40. spacy-layout/README.md at main - GitHub, accessed on March 24, 2025, https://github.com/explosion/spacy-layout/blob/main/README.md 41. explosion/spacy-layout: Process PDFs, Word documents and more with spaCy - GitHub, accessed on March 24, 2025, https://github.com/explosion/spacy-layout 42. Retrieval-Augmented Generation (RAG): How to Work with Vector Databases | Edlitera, accessed on March 24, 2025, https://www.edlitera.com/blog/posts/rag-vector-databases 43. Core Concepts — gensim - Radim Řehůřek, accessed on March 24, 2025, https://radimrehurek.com/gensim/auto_examples/core/run_core_concepts.html 44. Topic Identification with Gensim library using Python - Analytics Vidhya, accessed on March 24, 2025, https://www.analyticsvidhya.com/blog/2022/02/topic-identification-with-gensim-library-using-python/ 45. Topics and Transformations — gensim - Radim Řehůřek, accessed on March 24, 2025, https://radimrehurek.com/gensim/auto_examples/core/run_topics_and_transformations.html 46. Retrieval-Augmented Generation (RAG) from basics to advanced | by Tejpal Kumawat, accessed on March 24, 2025, https://medium.com/@tejpal.abhyuday/retrieval-augmented-generation-rag-from-basics-to-advanced-a2b068fd576c 47. RAG Pipeline: How It Transforms Natural Language Processing - Bombay Softwares, accessed on March 24, 2025, https://www.bombaysoftwares.com/blog/rag-pipeline-how-it-transforms-natural-language-processing 48. Mastering Document Chunking Strategies for Retrieval-Augmented Generation (RAG), accessed on March 24, 2025, https://medium.com/@sahin.samia/mastering-document-chunking-strategies-for-retrieval-augmented-generation-rag-c9c16785efc7 49. RAG Step-by-Step - DEV Community, accessed on March 24, 2025, https://dev.to/spara_50/rag-step-by-step-3fof 50. Prep your Data for RAG with Azure AI Search: Content Layout, Markdown Parsing & Improved Security | Microsoft Community Hub, accessed on March 24, 2025, https://techcommunity.microsoft.com/blog/azure-ai-services-blog/prep-your-data-for-rag-with-azure-ai-search-content-layout-markdown-parsing—imp/4303538 51. Use Markdown formatting in Microsoft Teams, accessed on March 24, 2025, https://support.microsoft.com/en-us/office/use-markdown-formatting-in-microsoft-teams-4d10bd65-55e2-4b2d-a1f3-2bebdcd2c772 52. Markdown syntax for files, widgets, wikis - Azure DevOps | Microsoft Learn, accessed on March 24, 2025, https://learn.microsoft.com/en-us/azure/devops/project/wiki/markdown-guidance?view=azure-devops 53. Top 5 Beginner-Friendly Open Source Libraries for RAG - DEV Community, accessed on March 24, 2025, https://dev.to/llmware/top-5-beginner-friendly-open-source-libraries-for-rag-1mhb 54. The Benefits of Using Markdown for Efficient Data Extraction | ScrapingAnt, accessed on March 24, 2025, https://scrapingant.com/blog/markdown-efficient-data-extraction 55. Exploring Microsoft Markitdown: Practical Use Cases with LLAMA and LLAVA Integration | by Dinesh Raghupatruni | Medium, accessed on March 24, 2025, https://medium.com/@dineshraghupatruni/exploring-microsoft-markitdown-practical-use-cases-with-llama-and-llava-integration-96ad3ed5576d

Back to Knowledge Base Explore More Knowledge