Credit Genie Improves AI Assistant, Uncovers Hidden Needs with LangSmith Insights Agent

TL;DR

• Credit Genie utilized LangSmith Insights Agent to deeply analyze user interactions with their AI financial assistant, AskGenie.
• The Insights Agent V2 configuration (powered by Anthropic Sonnet 4.5 and Haiku 4.5) unexpectedly revealed a high volume of customer support requests.
• These crucial insights led Credit Genie to ship new AskGenie features, directly addressing previously unfulfilled user needs like cash advance status lookups and repayment date modifications.
• Insights Agent is now integrated into a weekly scheduled workflow, ensuring a continuous feedback loop and agile product iteration.

At Credit Genie, a mobile-first financial wellness platform, the goal is to empower users with a deeper understanding of their finances. To achieve this, they developed AskGenie, an AI financial assistant built with LangGraph and seamlessly integrated into their mobile application. However, deploying AskGenie was just the beginning; the real challenge was to understand how users were interacting with it at scale. For this, Credit Genie turned to the powerful analytics capabilities of the LangSmith Platform and its Insights Agent.

Credit Genie configured Insights Agent in two ways, with a key differentiator being a V2 configuration that allowed for autogenerated categories. This unconstrained approach proved instrumental in uncovering genuine user intent. This V2 configuration intelligently processed data using Anthropic's Sonnet 4.5 as its thinking model and Haiku 4.5 for summarization. The results were telling: it revealed an unexpected, yet significant, volume of customer support requests, including inquiries about cash advance status lookups and repayment date modifications—critical gaps their existing support systems weren't adequately covering. This highlighted how granular trace analysis isn't merely observability; it's potent product discovery.

Armed with these insights, Credit Genie swiftly enhanced AskGenie, translating user pain points into actionable improvements. New features now include direct capabilities for cash advance status lookups, repayment date modifications, and even the ability to submit support requests directly within the chat interface, eliminating the need to redirect users to a separate chatbot. This proactive approach ensures AskGenie is not just reactive but continually evolving to meet user needs, as detailed further in the Credit Genie Case Study. Users can explore AskGenie's capabilities firsthand at Credit Genie AskGenie.

Recognizing the immense value of this continuous feedback, Credit Genie has integrated Insights Agent into a weekly scheduled workflow. This ensures a proactive approach to monitoring user behavior shifts and emerging trends, enabling faster cycles of product ideation, experimentation, and refinement. This strategic use of LangSmith Insights Documentation exemplifies a best practice in AI product development, allowing Credit Genie to maintain AskGenie's relevance and optimize the user experience continuously. For more on the latest AI tools and workflows, you can visit the LangChain Blog.

Summary

• Credit Genie effectively used LangSmith Insights Agent to identify unexpected yet crucial user needs within their AskGenie AI financial assistant.
• The Insights Agent V2 configuration provided deep analysis, revealing a high volume of customer support requests related to cash advance status lookups and repayment date modifications.
• These findings directly led to the implementation of new AskGenie features, significantly enhancing user satisfaction and addressing critical functional gaps.

Source: How Credit Genie used Insights Agent to improve their AI financial assistant