RAG Analytics

The RAG (Retrieval-Augmented Generation) analytics dashboard provides insight into how AcaTrove's AI retrieval pipeline is performing. It tracks embedding generation, chunk quality, retrieval accuracy, and cache performance. This data helps administrators optimize AI features and troubleshoot issues with AI-powered search and chat. Admin role is required.

Accessing RAG Analytics

Navigate to /admin/rag-analytics from the admin sidebar.

RAG analytics dashboard with pipeline metrics and performance charts

Pipeline Overview

The dashboard displays the end-to-end RAG pipeline status:

Document Indexing -- Number of documents successfully indexed (chunked and embedded) versus total documents uploaded. A low indexing rate may indicate processing failures or unsupported document formats.

Total Chunks -- The number of document chunks in the vector store. Each document is split into semantic chunks for fine-grained retrieval.

Embedding Coverage -- The percentage of chunks that have valid vector embeddings. This should be 100% for a healthy system. Below-100% coverage indicates embedding generation failures.

Index Health -- The status of the HNSW vector index, including the number of vectors, index size, and last rebuild time.

Retrieval Performance

Track how well the RAG system retrieves relevant information:

• Average retrieval latency -- Time from query to retrieved chunks. Target is under 200ms.
• Relevance scores -- Distribution of similarity scores for retrieved chunks. A healthy system shows most retrievals scoring above 0.7.
• Retrieval volume -- Number of retrieval queries per day, broken down by feature (Paper Chat, Multi-Doc Chat, Search, Q&A).

Cache Performance

AcaTrove caches frequently accessed embeddings and retrieval results to improve performance:

• Cache hit rate -- The percentage of retrieval queries served from cache. Higher is better.
• Cache size -- Current cache memory usage.
• Cache eviction rate -- How frequently cached items are evicted to make room for new ones.

A low cache hit rate with high retrieval volume may indicate that the cache size should be increased.

Chunk Quality Metrics

• Average chunk size -- The average number of tokens per chunk. Chunks that are too small lack context; chunks that are too large reduce retrieval precision.
• Chunk distribution -- A histogram of chunk sizes across the document library.
• Orphaned chunks -- Chunks that are no longer linked to a valid document (can occur after document deletion). These should be cleaned up periodically.

Troubleshooting

Common issues visible through RAG analytics:

• Low relevance scores -- May indicate that the embedding model needs reindexing or that documents are not being chunked optimally. Consider re-running the indexing pipeline.
• High retrieval latency -- Often caused by a large vector store without HNSW index optimization. Check the index health section.
• Low embedding coverage -- Check the upload analytics page for documents that failed processing.

Tips

• Monitor the retrieval relevance scores after adding large batches of documents to ensure the index is performing well.
• If cache hit rates are below 50%, consider increasing the cache allocation in your infrastructure settings.
• Schedule periodic reindexing (monthly or quarterly) to benefit from embedding model improvements.
• Use the chunk quality metrics to fine-tune your semantic chunking parameters.