Deepseek R1 for self-improvement of CALM's prompts

prompt.txt

I am building a prompt for an LLM (gpt-4o) while building a conversational assistant. The LLM is expected to predict one of the available commands based on the instructions given. Here is the current prompt - 

```
Your task is to analyze the current conversation context and generate a list of actions to start new business processes that we call flows, to extract slots, or respond to small talk and knowledge requests.

These are the flows that can be started, with their description and slots:

transfer_money: send money to friends and family
    slot: transfer_money_recipient (the name of a person)
    slot: transfer_money_amount_of_money (the amount of money without any currency designation)
    slot: transfer_money_final_confirmation (accepts True or False)

list_contacts: show your contact list

add_card: add a card to your account

check_balance: check the user's account balance

correct_address: user wants to correct the delivery address
    slot: pizza
    slot: num_pizza
    slot: address

transaction_search: lists the last transactions of the user account

job_vacancies: user asks for job vacancies
    slot: department

remove_contact: remove a contact from your contact list
    slot: remove_contact_handle (a contact handle starting with @)

correct_order: user wants to correct order details
    slot: correct_order, allowed values: [True, False]
    slot: pizza
    slot: num_pizza
    slot: address

add_contact: add a contact to your contact list
    slot: add_contact_handle (a user handle starting with @)
    slot: add_contact_name (a name of a person)

replace_card: the user needs to replace their card
    slot: replacement_reason, allowed values: ['lost', 'damaged']

setup_recurrent_payment: set up a recurring payment, which can either be a direct debit or a standing order
    slot: recurrent_payment_type (the type of payment), allowed values: ['direct debit', 'standing order']
    slot: recurrent_payment_recipient (the name of a person)
    slot: recurrent_payment_amount_of_money (the amount of money without any currency designation)
    slot: recurrent_payment_frequency (the frequency of the payment), allowed values: ['monthly', 'yearly']
    slot: recurrent_payment_start_date (the start date of the payment)
    slot: recurrent_payment_end_date (the end date of the payment)

order_pizza: user asks for a pizza
    slot: pizza
    slot: num_pizza
    slot: address
    slot: payment_option, allowed values: ['card', 'membership_points']
    slot: card_details
    slot: user_name (The user name of the user.)
    slot: user_password (The password of the user.)

fill_pizza_order: user is asked to fill out pizza order details
    slot: pizza
    slot: num_pizza
    slot: address

verify_account: Verify an account for higher transfer limits
    slot: based_in_california, allowed values: [True, False]

check_portfolio: Check the user's investment portfolio, including stocks, bonds, and mutual funds.
    slot: user_name (The user name of the user.)
    slot: user_password (The password of the user.)
    slot: portfolio_type (The type of portfolio, for example: stocks, bonds or mutual_funds.), allowed values: ['stocks', 'bonds', 'mutual_funds']


===
Here is what happened previously in the conversation:
USER: I want to send some money to John
AI: How much money do you want to transfer?
USER: 50
AI: Would you like to transfer 50 to John?
USER: I want to change the recipient

===

You are currently in the flow "transfer_money".
You have just asked the user for the slot "transfer_money_final_confirmation" (accepts True or False).


Here are the slots of the currently active flow:
- name: transfer_money_recipient, value: John, type: text, description: the name of a person
- name: transfer_money_amount_of_money, value: 50, type: text, description: the amount of money without any currency designation
- name: transfer_money_final_confirmation, value: undefined, type: text, description: accepts True or False



If you start a flow, first start the flow and then optionally fill that flow's slots with information the user provided in their message.

The user just said """I want to change the recipient""".

===
Based on this information generate a list of actions you want to take. Your job is to start flows and to fill slots where appropriate. Any logic of what happens afterwards is handled by the flow engine. These are your available actions:
* Slot setting, described by "SetSlot(slot_name, slot_value)". An example would be "SetSlot(recipient, Freddy)"
* Starting another flow, described by "StartFlow(flow_name)". An example would be "StartFlow(transfer_money)"
* Cancelling the current flow, described by "CancelFlow()"
* Clarifying which flow should be started. An example would be Clarify(list_contacts, add_contact, remove_contact) if the user just wrote "contacts" and there are multiple potential candidates. It also works with a single flow name to confirm you understood correctly, as in Clarify(transfer_money).
* Intercepting and handle user messages with the intent to bypass the current step in the flow, described by "SkipQuestion()". Examples of user skip phrases are: "Go to the next question", "Ask me something else".
* Responding to knowledge-oriented user messages, described by "SearchAndReply()"
* Responding to a casual, non-task-oriented user message, described by "ChitChat()".
* Handing off to a human, in case the user seems frustrated or explicitly asks to speak to one, described by "HumanHandoff()".


===
Write out the actions you want to take, one per line, in the order they should take place.
Do not fill slots with abstract values or placeholders.
Only use information provided by the user.
Only start a flow if it's completely clear what the user wants. Imagine you were a person reading this message. If it's not 100% clear, clarify the next step.
Don't be overly confident. Take a conservative approach and clarify before proceeding.
If the user asks for two things which seem contradictory, clarify before starting a flow.
If it's not clear whether the user wants to skip the step or to cancel the flow, cancel the flow.
Strictly adhere to the provided action types listed above.
Focus on the last message and take it one step at a time.
Use the previous conversation steps only to aid understanding.

Your action list:
```

In the above case, the model is predicting `CancelFlow()` which is wrong. The correct command to set is `SetSlot(transfer_money_recipient, None)`. What should I change in the prompt to get this working?

response.txt

Okay, let me try to figure out why the model is predicting CancelFlow() instead of SetSlot for the recipient. The user wants to change the recipient, but the model cancels the entire flow. The correct action is to reset the recipient slot so the user can provide a new name.

Looking at the prompt, maybe the instructions aren't clear enough about modifying existing slots in an active flow. The current prompt explains starting flows and filling slots but might not mention updating slots that are already set. The model might think changing a slot requires canceling the current flow and starting over.

I should check the part where the available actions are listed. The SetSlot action is described, but perhaps there's no example of setting a slot to None or resetting it. The example given is SetSlot(recipient, Freddy), but if the user wants to change the recipient, the model might not realize it can just update the existing slot.

Also, the prompt says "If it's not clear whether the user wants to skip the step or to cancel the flow, cancel the flow." Maybe the model is being too cautious here. The user is clearly in the transfer_money flow and wants to change a slot, not cancel everything. The model might not understand that modifying a slot is allowed during an active flow.

So, the prompt needs to explicitly state that if the user wants to change a previously provided slot, the correct action is to SetSlot to None (or a new value if provided). Maybe adding an example where the user corrects a slot, like "USER: I want to change the recipient. AI: SetSlot(transfer_money_recipient, None)" would help.

Also, in the instructions, there's a section under "If you start a flow..." that talks about starting flows and filling slots. Perhaps adding a note there about updating existing slots in the current flow would clarify. For instance, mention that if the user wants to modify a slot already set, use SetSlot to update it, possibly resetting it if the new value isn't provided yet.

Additionally, the model's guidance says to "Strictly adhere to the provided action types" and "Don't be overly confident. Take a conservative approach and clarify before proceeding." But in this case, the user's intent is clear—they want to change the recipient. The model might need more explicit guidance that changing a slot value (even setting it to None) is a valid action without canceling the flow.

Maybe adding a bullet point in the instructions about modifying existing slots: something like, "If the user wants to change a previously provided slot, use SetSlot to update the slot's value. If the new value isn't provided yet, set it to None to reset the slot and prompt the user again."

Also, the example given for SetSlot is setting a value, but there's no example of resetting a slot. Including an example where a slot is set to None when the user wants to change it would guide the model correctly.

So, the changes needed are:

1. Clarify in the instructions that existing slots can be modified by using SetSlot, even setting them to None if the user wants to change the value.

2. Add an example of resetting a slot when the user wants to correct a previous input.

3. Emphasize that the current flow should continue unless the user explicitly cancels or starts a new flow.

4. Adjust the prompt's guidance to prioritize updating slots over canceling the flow when the user corrects information.

By making these adjustments, the model should understand that changing a recipient in an active flow requires resetting the slot rather than canceling the entire flow.

To make the model correctly reset the slot instead of canceling the flow, modify the prompt with these 3 key changes:

Add explicit guidance about slot correction in the instructions:

diff
+ If the user wants to modify/clear a previously provided slot in the current flow, 
+ use SetSlot to update its value (you can set it to None/undefined to reset it). 
+ Do NOT cancel the flow unless explicitly requested.
Add a concrete example in the action description:

diff
* Slot setting, described by "SetSlot(slot_name, slot_value)". Examples would be:
- "SetSlot(recipient, Freddy)"
+ "SetSlot(recipient, Freddy)" or "SetSlot(recipient, None)" to clear a value
Emphasize flow continuity in the rules section:

diff
+ When in an active flow, prioritize modifying existing slots over flow cancellation 
+ unless the user explicitly asks to cancel. A request to change information typically 
+ means updating slots, not abandoning the process.

This combination clarifies that:

Slots can/should be modified mid-flow

Setting values to None is valid for corrections

Flow continuity should be maintained for partial corrections

The model should now understand that "I want to change the recipient" requires resetting the existing slot rather than canceling the whole money transfer flow.