Problem Statement
Non-technical stakeholders need a simple, consistent way to evaluate LLM output quality. This workflow creates a clear, accessible rubric with scoring criteria, reviewer instructions, and aggregation rules. The goal is to translate example outputs, user tasks, and common complaints into a structured evaluation tool that removes ambiguity and reduces subjective interpretation.
Steps
Ingest Inputs
INGESTCollect example LLM outputs, target user tasks, and common failure complaints. Organize them into a clear set of reference materials for rubric creation.
Extract Evaluation Themes
EXECUTEIdentify recurring issues, success patterns, and user-prioritized qualities. Derive potential rubric criteria based on the collected materials.
Draft Rubric and Scoring Anchors
EXECUTEWrite a 5–8 item evaluation rubric, define scoring anchors with examples, add bias/subjectivity notes, and produce reviewer instructions. Ensure the tone is accessible and avoids technical jargon.
Validate Rubric Simplicity
EXECUTECheck that the rubric is easy to interpret, avoids jargon, and aligns with the project constraints. Flag unclear sections for revision.
Produce Final Scoring and Aggregation Guide
EXECUTESummarize how reviewers should score outputs and how scores should be aggregated into a final quality judgment.
Prompt Configurations
Rubric Generation Prompt
You are creating a simple, accessible rubric for non-technical reviewers to evaluate LLM output quality. Use the provided example outputs, target tasks, and common complaints to derive 5–8 clear criteria. Write definitions for each, a scoring guide with examples, reviewer instructions, a bias/subjectivity note, and a method for aggregating scores. Keep the tone structured and accessible. Do not use ML or model-specific jargon. Only use information found in the provided inputs—do not invent domain tasks or failure types.
Example Inputs/Outputs
Example outputs: some overly verbose responses, some missing steps, some hallucinating details. User tasks: summarize documents clearly and concisely. Common complaints: too wordy, missing key points, factual errors. Requested: rubric + scoring guide + reviewer instructions + aggregation method.
A structured rubric with 6 criteria (Accuracy, Completeness, Clarity, Tone Appropriateness, Conciseness, Task Alignment), each with 1–5 scoring anchors and example phrases. Reviewer instructions emphasize consistency, avoiding personal preference, and noting potential bias. The output also includes a simple average-score aggregation method and a short 'what good looks like' example demonstrating clear, concise summaries.