I was wondering whether there’s a similar feature as we have in the UI where we can upload a file that is used with chat-interactions. The use case here would be that we can i.e. upload the documentation of a website like this but our local version of it so we can ensure that llms behave well with our new version of the documentation.
The big issue at the moment is that we would need to include it in the input-token window which makes the input tokens explode to, in this case, 35.000. It would be really great if we had the same possibility.
An alternative is prefix caching which could be semi long-lived, i.e. for a few days.
The last thing I can think of to solve this would be to have multiple steps where we first provide all the titles and the description and make the LLM drill down in multiple steps.
I know this sounds like a RAG however, in one of the steps I want it to make improvements to the docs based on the question, which means that the topic might be missing so it would need to choose a section and write an entry. Which kind of discourages this RAG case in my eyes.
Thank you so much for reading this already, y’all have a great tool!