I’m developing a system and could use some expert insights on its architecture and implementation.
Here’s a brief overview of the current flow:
User fills out an intake questionnaire with X questions
System references datasets/examples from existing system folders
Multiple parallel flows run with Claude API to generate texts based on guidelines and relevant intake responses
Now, I’m grappling with some key issues:
Overall Architecture
Should I use Agentic AI frameworks? If so, which ones?
Or is there a more efficient approach for a system like this?
Data Storage and Management
How to store intake responses for efficient AI flow access?
Is RAG (Retrieval-Augmented Generation) advisable for storage and retrieval? If not, what are the alternatives?
Existing Dataset Integration
What’s the most efficient way to incorporate examples from existing folders into the flow?
Better to train a small model on the dataset or integrate it directly?
Parallel Process Optimization
How to efficiently manage and synchronize the parallel flows?
Important notes:
Output (generated texts) and number of intake questions remain constant
It’s a complex flow split into several sub-flows, not a single long prompt
