A.7. Prompt Optimization

<ol style="margin-left: 1em; line-height: 1.6; padding-left: 0.5em;"> <li style="margin-bottom: 1.2em;"> <strong>Personas</strong><br /> <p style="margin-top: 0.5em; margin-bottom: 0.5em;"> Introducing a persona to the model is an effective way to drive performance. Once you have narrowed down the specialism of the component you are developing the prompt for, you can create a persona in the system prompt that helps to shape the model's behaviour. We used this in our planner model to create a system prompt like this: </p> <pre style="margin-top: 0.5em; margin-bottom: 0.5em; background-color: #f5f5f5; padding: 0.75em; border-radius: 5px;"><code>initial_planner_system_prompt = ( "You work for the leading financial firm, ABC Incorporated, one of the largest financial firms in the world. " "Due to your long and esteemed tenure at the firm, various equity research teams will often come to you " "for guidance on research tasks they are performing. Your expertise is particularly strong in the area of " "ABC Incorporated's proprietary knowledge base of earnings call transcripts. This contains details that have been " "extracted from the earnings call transcripts of various companies with labelling for when these statements are, or " "were, valid. You are an expert at providing instructions to teams on how to use this knowledge graph to answer " "their research queries. \n" )</code></pre> <p style="margin-top: 0.5em; margin-bottom: 0.5em;"> Persona prompts can become much more developed and specific than this, but this should provide an insight into what this looks like in practice. </p> </li>

Few-Shot Prompting and Chain-of-Thought

For extraction-related tasks, such as statement extraction, a concise few-shot prompt (2–5 examples) will typically deliver higher precision than a zero-shot prompt at a marginal increase in cost.

For e.g., temporal reconciliation tasks, chain-of-thought methods where you guide the model through comparison logic are more appropriate. This can look like:

Example 1: [Old fact], [New fact] → Invalidate
Example 2: [Old fact], [New fact] → Coexist
Now: [Old fact], [New fact] →

Dynamic Prompting & Context Management

You can also lean on other LLMs or more structured methods to prune and prepare material that will be dynamically passed to prompts. We saw an example of this when building the tools for our retriever above, where the timeline_generation tool sorts the retrieved material before passing it back to the central orchestrator.

Steps to clean up the context or compress it mid-run can also be highly effective for longer-running queries.

<li style="margin-bottom: 1.2em;"> <strong>Template Library & A/B Testing</strong><br /> <p style="margin-top: 0.5em; margin-bottom: 0.5em;"> Maintain a set of prompt templates in a version-controlled directory (e.g., <code>prompts/statement_extraction.json</code>, <code>prompts/entity_extraction.json</code>) to enable you to audit past changes and revert if necessary. You can utilize OpenAI's reusuable prompts for this. In the OpenAI dashboard, you can develop <a href="https://platform.openai.com/docs/guides/text#reusable-prompts">reusable prompts</a> to use in API requests. This enables you to build and evaluate your prompts, deploying updated and improved versions without ever changing the code. </p> <p style="margin-top: 0.5em; margin-bottom: 0.5em;"> Automate A/B testing by periodically sampling extracted facts from the pipeline, re-running them through alternative prompts, and comparing performance scores (you can track this in a separate evaluation harness). </p> <p style="margin-top: 0.5em; margin-bottom: 0.5em;"> Track key performance indicators (KPIs) such as extraction latency, error rates, and invalidation accuracy. </p> <p style="margin-top: 0.5em; margin-bottom: 0.5em;"> If any metric drifts beyond a threshold (e.g., invalidation accuracy drops below 90%), trigger an alert and roll back to a previous prompt version. </p> </li> </ol>