Control response quality with personal score threshold

How to control response quality

Personal AI’s output is a blend between the data used to train your SLM and the general output of LLMs like OpenAI’s GPT-4o. Why is this combination important? Sometimes, your AI doesn’t have enough data on the topic to give a good answer or doesn’t know how to continue the conversation. When this happens, the platform blends in general LLM data to help complete the thought or message, as well as generate questions to keep the conversation flowing.

This doesn’t mean that your AI will deviate from the data you used for training: it will not fall back to LLM hallucinations or factual imprecisions. It’s merely used to improve response fluency and give a better user experience to whoever is chatting to your AI.

In this context, Personal AI has a set of controls that help you decide what happens when the response’s personal score is too low. This is only relevant when using your AI in the AI Autopilot mode.

What is the personal score threshold?

The basic functionality of the personal score threshold is stop your AI from replying if a certain personal score hasn’t been met. For example, if you set the threshold to 55%, any response generated below 55% personal score will not be sent to the user.

For messages filtered out using this setting, you can have your AI send a default message, informing the user it doesn’t have enough information on how to reply. It can also give a fallback suggestion, drafting an AI Copilot message that you can review and approve instead.

Adjust the personal score threshold

When adjusting the personal score threshold, notice the 3 breakpoints in the slider bar.

1. Before the first breakpoint, your AI will still respond with general data. For example, for an answer with 15% personal score, 85% of it will be generated resorting to an LLM.
2. After the first breakpoint, your AI will reply with relevant data. For example, an answer with 45% will have more of your memories into it when compared with the former, but 55% of it will still be generated with general data.
3. Finally, after the second breakpoint, your AI will respond with data heavily reinforced in the memory. The higher you adjust the setting, from 60% to 100%, will determine how high the personal score needs to be for the message to come through.

Higher isn’t always better

In general, setting the personal score threshold to a high percentage yields better responses. But, if your AI doesn’t have enough training data to reply to questions, setting it too high could limit its responses—and, as a result, also limit its usefulness.

Consider training your AI first and monitoring the personal score of each response it generates. Then, as you share your AI with others, progressively increase the personal score threshold to provide higher quality responses, without having it resort to the default message too often.

While very high scores are possible and often seen for well-trained Personas, consider not setting your personal score threshold above 90%, as that could filter out too many messages.