Key Design Principle: Use LLMs for Language, Not Logic
As described by the Head AI Architect on the project, Amel Spahec: “The issue wasn’t the model. It was how much context we were giving it.” This insight became the foundation of the new design principle: Use LLMs for natural language understanding, and deterministic systems for structured computation.
The Approach
Key elements of the approach included: Intelligent Metric Search (IMS). - Retrieves relevant metric identifiers using keyword search, with semantic fallback, and resolves them into structured data for downstream use Unified retrieval and processing layer. - Combines metric lookup and deterministic data transformation into a single step to ensure consistency and reduce pipeline complexity Structured data grounding. - Uses resolved metrics and attributes as primary inputs to generation, minimizing reliance on unstructured context Data source integration. - Connects internal datasets and external providers through a unified access layer Controlled prompt construction. - Formats structured data into consistent inputs for language models across workflows Human review workflows. - Allows analysts to review and refine outputs, supporting quality control Cost-aware system design. - Reduces token usage and compute cost by limiting unnecessary context and focusing on relevant structured inputs
The Solution
The platform integrates IMS into an end-to-end workflow that supports research, analysis, and content generation. When a user submits a query, the system performs keyword-based retrieval to identify relevant metric IDs, with semantic search used only when necessary. These metrics are then resolved into structured data through deterministic processing within the same pipeline step. The resulting structured inputs are passed to a language model, which generates responses aligned with the query and grounded in the retrieved data. Because outputs are based on structured inputs tied to specific metrics, users can inspect the underlying data used in generation. This improves transparency compared to systems that rely primarily on unstructured text retrieval. The system also standardizes how queries are processed and how outputs are generated, reducing variability across users and teams.
The Impact
The system delivered measurable operational improvements: - Reduced time required to produce research outputs - Lowered compute and infrastructure costs through efficient retrieval and prompt design - Improved consistency of outputs across workflows - Increased visibility into the data used to generate responses - Enabled scaling of research workflows without proportional increases in manual effort
Conclusion
By focusing on structured data retrieval and deterministic processing, Roko Labs helped demonstrate that the client could improve the efficiency and reliability of its investment research workflows. Intelligent Metric Search (IMS) provides a practical alternative to traditional retrieval-based approaches in domains where precision, traceability, and cost control are critical.
