An LLM-driven RAG system for contract review
Contract review is a critical but time-consuming process within legal practice, with significant financial consequences when errors occur. Although Large Language Models (LLMs) have shown promise for processing legal documents, they continue to face challenges with long contracts and complex legal relationships. This research presents an advanced approach to automated contract review by integrating knowledge graphs into Retrieval-Augmented Generation (RAG) frameworks, addressing the limitations of existing methodologies.
Literature review and experimental design
Based on an extensive literature review of contract review automation and RAG systems, systematic experiments were conducted comparing RAG approaches with the in-context learning capabilities of LLMs. The empirical analysis confirms that RAG-based methods significantly improve the analysis of long-context texts and information extraction in legal documents, particularly in terms of accuracy and consistency.
Optimization of the retrieval phase
Building on these findings, in-depth research was conducted into optimization techniques for the retrieval phase of RAG, given its crucial role in the accuracy of contract review. The experimental evaluation included various chunking strategies, query expansion methods, and re-ranking approaches, which identified best practices for legal document processing. The main contribution is a new KG-RAG system that enhances contextual understanding in legal document analysis. The approach was evaluated using the Contract Understanding Atticus Dataset (CUAD) and the ContractNLI dataset and shows better performance than traditional RAG implementations and long-context models. In addition, the research investigates optimal chunking strategies and the trade-offs between efficiency and effectiveness of different model architectures.
Key results and benefits
The results show that the KG-enriched RAG framework delivers superior performance in identifying and analyzing complex legal relationships, while maintaining computational efficiency. In particular, the integration of knowledge graphs excels at capturing hierarchical and cross-referencing relationships within legal documents, an essential aspect that is often lacking in conventional approaches.
Contribution to Legal AI
This work contributes to the advancement of Legal AI by providing a more robust and context-aware approach to contract review, as well as practical insights for the implementation of AI systems in legal practice. The findings point to promising directions for future research in legal document processing, particularly in domains where deep contextual understanding and relational modeling are required.
