Generated Knowledge

Optimizing AI Prompts

When it comes to optimizing prompts for AI models, there are various techniques and strategies to enhance the performance and accuracy of the models. Two key aspects of optimizing AI responses include enhancing language models and incorporating knowledge into prompts, which we'll discuss today.

By optimizing AI prompts through language model enhancements and knowledge incorporation, you can improve the performance and reliability of your AI models. These techniques allow you to generate more accurate and contextually-grounded responses, enhancing the overall user experience.

‍

Techniques for Prompt Engineering

When it comes to optimizing prompts for AI models, there are various techniques and strategies you can employ to enhance the performance and accuracy of the models. In this section, we will explore two popular techniques: the generated knowledge approach and the dual prompt strategy.

Generated Knowledge Approach

The generated knowledge approach is a technique commonly used to enhance the accuracy of LLMs. It involves asking the LLM to generate potentially useful information about a series of similar questions to show a pattern before requesting a final response. By incorporating generated knowledge, the LLM's output becomes more informative and contextually grounded.

Step 1. Crafting the initial Prompt:

First, let's consider what we're building up to. Let's say we need the following question answered:

Input: Part of golf is trying to 
get a <X> point total than others. 

This is a logical/informational question. Let's start building up a few-shot knowledge generator prompt, to get a Knowledge item for it.

The generator prompt consists of an instruction, a knowledge item for each input, and a new-question placeholder.

Prompt example:

Input: penguins have <X> wings.
Knowledge: Birds have two wings. Penguin is a kind of bird.
... ...
Input: penguins have <X> wings.
Knowledge: Birds have two wings. Penguin is a kind of bird.

Input: <question placeholder>
Knowledge:

The initial prompt is crafted by hands. Do not use knowledge that would directly answer the question yet. You have to provoke LLM to do some analysis. This way the Knowledge below would be a worse building block for your prompt.

Input: penguins have <X> wings.
Knowledge: Penguins are known to have 2 wings.

Accordingly, this Knowledge would be better:

Input: penguins have <X> wings.
Knowledge: Birds have two wings. Penguin is a kind of bird.

Now this can already get you a correct answer with a much higher probability. This is somewhat similar to chain-of-thought prompting, which we already know works. But here we can use AI to build up the prompt. As we're following an actual study by Liu et. al (2022) we'll push this method to its limits and discuss, how to get the best chance at success with knowledge generation.

Step 2 (Optional). Generating knowledge:

In this step, we need to learn generating more knowledge, until we maximize the potential of our prompt to answer primitive logical two-liners correctly.

Now we come up with the next question:

Input: The process by which genes are passed is <X>

To continue the pattern, we need new knowledge. So we modify our base prompt:

Input: penguins have <X> wings.
Knowledge: Birds have two wings. Penguin is a kind of bird.
... ...
Input: The process by which genes are passed is <X>
Knowledge:

Now to generate multiple knowledge points:

Knowledge 1:

Heredity involves the transmission of genetic information

We append this knowledge to the last question and request an answer. Repeating multiple times for accuracy, we see that this knowledge provides confident results.

Knowledge 2:

Inheritance occurs through the transmission of DNA from parents to their offspring.

This knowledge shows worse results, and so we choose the first one.
As a result, we get an improved prompt that looks like this:

Input: penguins have <X> wings.
Knowledge: Birds have two wings. Penguin is a kind of bird.
... ...
Input: The process by which genes are passed is <X>
Knowledge: Heredity involves the transmission of genetic information

Input: {question placeholder}
Knowledge:

We'll use this to get a Knowledge item for our initial question. Appending input-knowledge pairs, we maximize the prompt's effectiveness at 20 items. After that, it starts to go down, probably because too much info starts to accumulate noise. As soon as you are contempt with the depth of the context, you can ask the goal question we've planned for. Or almost any simple logical two-liner task, for that matter.

Step 3. Getting the correct answer

At the last step, we generate multiple knowledge items, and then pick the answer to the original question by majority vote.

Input: penguins have <X> wings.
Knowledge: Birds have two wings. Penguin is a kind of bird.
... ...
Input: The process by which genes are passed is <X>
Knowledge: Heredity involves the transmission of genetic information

Input: Part of golf is trying to 
get a <X> point total than others. 
Knowledge:

Knowledge  1:

Golf is a sport in which players compete
to hit a small ball into a hole using various clubs

Question 1:

Question: Part of golf is trying to 
get a <X> point total than others. 

Knowledge: Golf is a sport in which players
compete to hit a small ball
into a hole using various clubs

Answer:

Answer 1:

The player with the highest score wins.

We'll omit repeating the question with substituted knowledge in the future, and instead give you the new answers we'll get.

Knowledge 2:

Golf is a sport where the competitor 
with the least amount of points gets the victory

Answer 2:

The golfer with the least amount of points wins

Knowledge 3: 

Golf points are scored based on the number of hits

Answer 3:

Lower

By majority vote, we get "lower" amount of points as the correct answer.

‍

Now, state-of-the-art AI Models like ChatGPT 4o would crush these simple questions, yet the technique is still applicable to them in more complex cases. And in case you choose to opt for a more budget-friendly AI, such built-in prompting might make a big difference to the end user, allowing them to converse with AI more freely.

Quick recap:

1. Make a knowledge-generating prompt, suitable for your question. You can also find it online, or take it from the article. Try to make the questions within it similar to the one you want answered.
2. (Optional) Generate up to 20 items of knowledge with majority voting.
3. Generate multiple knowledge items for your original question, and pick an answer by majority vote.

Benchmarks

According to the study where it was proposed, generated knowledge context is on par with the Retrieval-based context. QASC dataset can be excluded, as it used golden data from the model's training data to provide context. IR context on other datasets was gathered from Google snippets or Wikipedia on the topic.

This shows, that generating knowledge is a very strong technique. By the end of the article, authors come to the conclusion, that this can be considered state-of-the-art prompting. Yet we'll discuss simpler techniques and more recent papers in the next articles on the topic.

‍

You can also explore Retrieval Augmented Generation (RAG) to learn how it can be effectively integrated with this technique.

‍

Test out the prompts with AI/ML AI Playground.

We're excited to see what amazing projects you will bring to life. Happy prompting!