ReSRer Meta Prompt V4

Meta prompt v1

###Instruction### Suggest improved summarizer's prompt instruction method using history of prompt to improve top-k 10 EM score to 40%. Generated summary is provided to reader which generates final answer.

You can make prompt instruction that treats passages dynamically which means extracting more from important passages and extract less from less important passages Our main goal is optimizing summarizer's instruction prompt by maximizing summarizer's exact match (retaining rate) to finally maximizing reader's exact match score. You can use backtracking, sentence highlighting or structured text or for machine readability. You can use several several metrics for each prompt version to tuning prompt. Below are description for each metric

f1 score is final answer's f1 score corelated to exact match

psgs_tokens are retrieved token length count which are before summarization.

summary_tokens are summary's token length count which you can control flexible but make it shorter than total psgs_tokens

sum_em is exact match span retained rate in summary. This is important for summary performance

You can also suggest new meta instruction instead of this if necessary

###Reader prompt###


Extract a concise noun-based answer from the provided context for the question. Your answer should be under three words and extracted directly from a context of no more than five words. You can analyze the context step by step to derive the answer. Avoid using prefixes that indicate the type of answer; simply present the shortest relevant answer span from the context.

###Baseline###

topk-4 gpt


{'exact_match': 30.81967213114754, 'f1': 40.40037545072675, 'psgs_tokens': 544.1409836065574, 'ret_em': 64.09836065573771}

topk-8 gpt


{'exact_match': 31.606648199445985, 'f1': 42.93274362444592, 'psgs_tokens': 1094.1650969529087, 'ret_em': 72.90858725761773}

topk-16 gpt


{'exact_match': 31.994459833795013, 'f1': 42.9381781336727, 'psgs_tokens': 2189.9739612188364, 'ret_em': 78.53185595567868}

###Prompt v3###


Rewrite the given passages to be easier for the reader answering the given question.
The rewrited text should be half the total length of the original passages. Your response must be at least 200 words long.
The given passages are related about the question topic.
Do not make up information that is not in the document, and do not answer the question.
Reduce the noise unrelated to answer the question.
Remove unrelated phrases and sentences to answer the question.
Find the evidences that support the answer to the question and retain them.
Print only the rewrited texts
The final answer for this question is contained is the passages so maintain the exact span of answer smaller than 5 words.

v3 prompt's metric per scores are like these

topk 4 → summarized


{'exact_match': 33.37950138504155, 'f1': 43.50731017213017, 'psgs_tokens': 546.4903047091412, 'summary_tokens': 161.60775623268697, 'ret_em': 66.64819944598338, 'sum_em': 57.25761772853185}

topk 8→1


{'exact_match': 35.40166204986149, 'f1': 46.4832925615475, 'psgs_tokens': 1094.1232686980609, 'summary_tokens': 243.2825484764542 'ret_em': 73.37950138504155, 'sum_em': 62.548476454293635}

topk 16 → 1


{'exact_match': 36.094182825484765, 'f1': 46.92736784040187, 'psgs_tokens': 2189.9326869806096, 'summary_tokens': 233.7049861495845, 'ret_em': 78.58725761772854, 'sum_em': 62.18836565096952}

topk 10->summarized (in test case count 410/3610


{'exact_match': 32.19512195121951, 'f1': 42.62078977932637, 'psgs_tokens': 1357.424390239025, 'summary_tokens': 239.94878048780487, 'ret_em': 75.60975609756098, 'sum_em': 63.0243902439025}

ReSRer Meta Prompt V4

Meta prompt v1

Recommendations