Can I give additional context to Foundation Models?

Question

Created 1w

Replies 4

Boosts 0

Participants 4

I'm interested in using Foundation Models to act as an AI support agent for our extensive in-app documentation. We have many pages of in-app documents, which the user can currently search, but it would be great to use Foundation Models to let the user get answers to arbitrary questions.

Is this possible with the current version of Foundation Models? It seems like the way to add new context to the model is with the instructions parameter on LanguageModelSession. As I understand it, the combined instructions and prompt need to consume less than 4096 tokens.

That definitely wouldn't be enough for the amount of documentation I want the agent to be able to refer to. Is there another way of doing this, maybe as a series of recursive queries? If there is a solution based on multiple queries, should I expect this to be fast enough for interactive use?

Answered by DTS Engineer in 849318022

Hi @interferon,

You are correct, adding your entire documentation to the prompt will likely exceed the context window size and is probably not the right approach here. One option could be to find just the relevant information in your documentation and then include that information in the prompt.

This approach is sometimes called retrieval augmented generation (RAG). It's up to you as the developer, however, to choose how you set up your data to retrieve the most relevant chunks of information.

Just keep in mind that Foundation Models gives access to powerful but comparatively small on-device models that are optimized for a subset of tasks like summarization. Unexpected results may occur if relied on for tasks which require a large amount of context or advanced reasoning.

Best,

-J

Boost

Answer 1

DTS Engineer OP

Apple

4d

Recommended

Hi @interferon,

You are correct, adding your entire documentation to the prompt will likely exceed the context window size and is probably not the right approach here. One option could be to find just the relevant information in your documentation and then include that information in the prompt.

This approach is sometimes called retrieval augmented generation (RAG). It's up to you as the developer, however, to choose how you set up your data to retrieve the most relevant chunks of information.

Just keep in mind that Foundation Models gives access to powerful but comparatively small on-device models that are optimized for a subset of tasks like summarization. Unexpected results may occur if relied on for tasks which require a large amount of context or advanced reasoning.

Best,

-J

0

Answer 2

manojios99 OP

3d

What we can do for RAG ? Because we need relevant information as per our application's content(data).

0

Answer 3

aryanarayantiwari OP

3d

Suggested, it is the developer’s responsibility to set up Retrieval Augmented Generation (RAG). If you have any examples or suggestions for setting up data, it would be helpful.

1

Answer 4

DTS Engineer OP

Apple

2d

Hi all,

Take a look at our documentation on Foundation Model Tools, which is one option to initiate actions. In this instance it could search your app's existing database, for example.

For embedding documents and setting up a semantic search on-device, you might look into CoreML.

These are some of the frameworks you could use but all that said, there is no one correct way to implement RAG, and the exact implementation will depend on your app's specific use-case.

-J

1