An LLM-driven RAG system for contract review
Contract review is a critical but time-consuming process in legal practice, with significant financial consequences when errors occur. Although Large Language Models (LLMs) have shown great promise for processing legal documents, they continue to face challenges with long contracts and complex legal relationships. This research presents an advanced approach to automated contract review by integrating knowledge graphs into Retrieval-Augmented Generation (RAG) frameworks, thereby addressing the limitations of existing methodologies.
Literature review and experimental design
Based on an extensive literature review of contract review automation and RAG systems, systematic experiments were conducted to compare RAG approaches with the in-context learning capabilities of LLMs. The empirical analysis confirms that RAG-based methods significantly improve the analysis of long texts and information extraction in legal documents, particularly in terms of accuracy and consistency.
Optimization of the Retrieval Phase
Building on these findings, in-depth research was conducted into optimization techniques for the retrieval phase of RAG, as that phase is critical to the accuracy of contract review. The experimental evaluation included various chunking strategies, query expansion methods, and re-ranking approaches, which helped identify best practices for legal document processing. The main contribution is a new KG-RAG system that enhances contextual understanding in legal document analysis. The approach was evaluated using the Contract Understanding Atticus Dataset (CUAD) and the ContractNLI dataset and demonstrates better performance than traditional RAG implementations and long-context models. In addition, the research investigates optimal chunking strategies and the trade-offs between efficiency and effectiveness of different model architectures.
Key results and benefits
The results show that the KG-enriched RAG framework delivers superior performance in identifying and analyzing complex legal relationships, while maintaining computational efficiency. Knowledge graphs, in particular, effectively capture hierarchical and cross-referenced relationships in legal documents—something that conventional approaches often fail to do.
Contribution to Legal AI
This work contributes to the advancement of Legal AI by offering a more reliable, context-aware approach to contract review, as well as practical insights for the implementation of AI systems in legal practice. The findings point to promising directions for future research on legal document processing, particularly in domains that require in-depth contextual understanding and relational modeling.
