PR Risk Classifier — Build-It-Yourself Prompt Pack

This document is a sequence of self-contained prompts that you can feed, one at a time, to an AI coding agent (Claude Code, Cursor, etc.) to build a PR risk classification system from scratch in your own Rails repository.

The system being built is a 3-step pipeline that classifies open pull requests as LOW_RISK or HIGH_RISK:

Static Analysis (deterministic) → Semantic Analysis (LLM) → Agentic Analysis (Claude Code CLI)

Low-risk PRs can then be auto-approved by a GitHub Action, with a 90/10 split kept aside for human review to validate the classifier's accuracy.

How to use this document

Read "System Overview" below first so you (the human) understand what's being built.
Work through the phases in order. Each phase has one or more numbered sub-tasks.
For each sub-task, copy the prompt block verbatim into your AI agent. The prompts are written to be self-contained — they tell the agent what to build, where to put it, how to wire it up, and what the public API should look like.
Verify after each prompt. Each sub-task ends with an "Acceptance criteria" list. Before moving on, confirm the agent's output matches.
Don't skip ahead. Later prompts assume earlier files exist with the right interfaces.

The prompts assume:

A Ruby on Rails app with lib/ autoloaded.
Ruby ≥ 3.2, bundler-cache, rake, httparty, yaml, json, open3 available.
gh (GitHub CLI) installed on the host that runs the rake task.
claude (Claude Code CLI: npm install -g @anthropic-ai/claude-code) installed for Step 3.
An Anthropic API key for Step 2.

System Overview

Pipeline

                                 YAML Config
              (lib/pr_risk_classifier/config.backend.yml or
               lib/pr_risk_classifier/config.frontend.yml)
                                      |
                                      v
+------------------------------------------------------------------+
|           Classification::RiskClassificationPipeline             |
|  +-------------+      +----------------+      +----------------+ |
|  |   Step 1    |      |     Step 2     |      |     Step 3     | |
|  |   Static    | ---> |   Semantic     | ---> |   Agentic      | |
|  |  Analysis   |      |  (LLM)         |      |  (Claude Code) | |
|  +-------------+      +----------------+      +----------------+ |
|       |                    |                       |             |
|       | early exit         | early exit            | final       |
|       | LOW or HIGH        | HIGH only             | result      |
+------------------------------------------------------------------+

Escalation rules

Static Result	Semantic Result	Agentic Result	Final
LOW_RISK	(skipped)	(skipped)	LOW_RISK
HIGH_RISK	(skipped)	(skipped)	HIGH_RISK
INCONCLUSIVE	HIGH_RISK	(skipped)	HIGH_RISK
INCONCLUSIVE	LOW_RISK	runs	uses Step 3 result

Rationale: HIGH_RISK signals are trusted early. LOW_RISK from the LLM is verified by deeper agentic exploration before we trust it.

File layout being built

lib/
├── pr_risk_classifier.rb                              # top-level require loader
├── pr_risk_classifier/
│   ├── config.common.yml                              # shared config
│   ├── config.backend.yml                             # Ruby/Rails profile
│   ├── config.frontend.yml                            # React/TS profile
│   ├── configuration.rb                               # YAML loader
│   ├── configuration/
│   │   ├── static_analysis_config.rb
│   │   └── instructions.rb
│   ├── github_client.rb                               # gh CLI wrapper
│   ├── llm_client.rb                                  # Anthropic API wrapper
│   ├── task_runner.rb                                 # rake-task glue
│   └── classification/
│       ├── runner.rb                                  # public entry point
│       ├── risk_classification_pipeline.rb            # the 3-step pipeline
│       ├── pipeline_result.rb
│       ├── pr_context.rb                              # pre-fetched PR data
│       ├── static_analyser.rb                         # Step 1
│       ├── semantic_analyser.rb                       # Step 2
│       ├── backend_semantic_prompt_builder.rb
│       ├── frontend_semantic_prompt_builder.rb
│       ├── agentic_analyser.rb                        # Step 3
│       ├── agentic_prompt_builder.rb                  # base class
│       ├── agentic_prompt_sections.rb                 # shared mixin
│       ├── backend_agentic_prompt_builder.rb
│       ├── frontend_agentic_prompt_builder.rb
│       ├── agentic_result.rb
│       └── streaming_output_handler.rb
└── tasks/
    └── pr_risk_classifier.rake

.github/workflows/
└── pr-risk-classifier.yml

Phase 1 — Foundations

Sub-task 1.1 — Top-level autoloader and module skeleton

Prompt:

Create a new module PrRiskClassifier for a Ruby on Rails project that classifies pull requests as LOW_RISK or HIGH_RISK using a 3-step pipeline (static rules → LLM → Claude Code CLI).

Create the file lib/pr_risk_classifier.rb containing:

# frozen_string_literal: true

module PrRiskClassifier with a class Error < StandardError; end

require_relative lines for every file we will create under lib/pr_risk_classifier/ and lib/pr_risk_classifier/classification/. Order matters — load llm_client, github_client, configuration, configuration/static_analysis_config, configuration/instructions, then everything in the classification/ directory (runner, pipeline_result, risk_classification_pipeline, pr_context, static_analyser, backend/frontend semantic prompt builders, semantic_analyser, agentic_prompt_sections, agentic_prompt_builder, backend/frontend agentic prompt builders, agentic_result, streaming_output_handler, agentic_analyser), then finally task_runner.

Do not create the referenced files yet — they will come in later sub-tasks. Only create lib/pr_risk_classifier.rb.

Acceptance criteria:

File exists at lib/pr_risk_classifier.rb

Defines PrRiskClassifier module + PrRiskClassifier::Error

All require_relative lines listed (Ruby will fail at runtime until later phases create the files — that's expected)

Sub-task 1.2 — Config YAML files

Prompt:

Create three YAML config files for the PR Risk Classifier. They are loaded by PrRiskClassifier::Configuration (built later) and drive both the static rules and the LLM prompts.

File 1: lib/pr_risk_classifier/config.common.yml — settings shared by all profiles. Keys:
static_analysis:
  pr_metadata:
    require_description: true
    min_description_length: 50
  auto_approval_teams:
    - YourOrg/your-team-slug   # replace with the GitHub team allowed to be auto-approved
File 2: lib/pr_risk_classifier/config.backend.yml — for Ruby/Rails PRs. Use this template; tune the values to your codebase:
---
prompt_profile: backend

static_analysis:
  low_risk_file_patterns:
    - "_spec.rb"
    - ".md"
    - "config/locales/"
    - "spec/fixtures/"
    - "spec/factories/"
    - ".github/workflows/"
  high_risk_file_patterns:
    - "db/migrate/"
    - "db/data/"
    - "schema.graphql"
    # add paths in your repo that are business-critical (payments, fulfilment, auth, etc.)
  high_risk_keywords:
    - "\\bpayment\\b"
    - "\\bstripe\\b"
    - "\\brefund\\b"
    - "\\bdestroy_all\\b"
    - "\\bdelete_all\\b"
    - "\\bauthenticate\\b"
    - "\\bcredentials\\b"
    - "\\bapi_key\\b"
    - "\\bsecret\\b"
  thresholds:
    max_diff_lines: 1000
    max_production_diff_lines: 400
  test_file_patterns:
    - "_spec.rb"
    - "_test.rb"
    - "spec/"
    - "test/"

instructions:
  warnings:
    - "Describe domain-specific landmines here, e.g. fulfilment chain, daily triggers, etc."
  context:
    - "One-line facts the LLM should know, e.g. 'Stripe is the payment processor.'"
  sensitive_areas:
    - "app/models/some_critical_model.rb - why it matters"

critical_paths:
  - "app/services/payments"
  - "app/jobs/fulfilment"

external_services:
  - "Stripe"
  - "Sendgrid"
File 3: lib/pr_risk_classifier/config.frontend.yml — for React/TS PRs. Same shape as backend. Use frontend-relevant patterns instead:

low_risk_file_patterns: .test.tsx, .test.ts, .spec.tsx, .stories.tsx, cypress/, .d.ts, **/__generated__/

high_risk_file_patterns: paths under app/javascript/components/pages/CheckoutPage/, app/javascript/context/, package.json, webpack, tsconfig.json etc.

high_risk_keywords: \\bpayment\\b, \\bstripe\\b, \\bcheckout\\b, \\bcookie\\b, \\blocalStorage\\b, \\bsessionStorage\\b, \\btoken\\b, \\blogin\\b, \\bcheckout\\b

same thresholds

test_file_patterns should additionally include .test.tsx, .spec.tsx, cypress/, __tests__/

prompt_profile: frontend

Fill in instructions, critical_paths, external_services with frontend-flavoured content.

Acceptance criteria:

All three YAML files exist and parse with YAML.load_file

Backend config has prompt_profile: backend, frontend has prompt_profile: frontend

Common config has auto_approval_teams and pr_metadata only (no profile-specific keys)

Phase 2 — Configuration classes

Sub-task 2.1 — `StaticAnalysisConfig`

Prompt:

Create lib/pr_risk_classifier/configuration/static_analysis_config.rb. It wraps the static_analysis: block from a YAML config and exposes typed accessors. It is initialised with a Hash (the static_analysis sub-tree of the loaded YAML).

Requirements:

Module path: PrRiskClassifier::StaticAnalysisConfig

Two Structs declared inside the class with keyword_init: true:

PrMetadata with members :require_description, :min_description_length

Thresholds with members :max_diff_lines, :max_production_diff_lines

attr_reader for: pr_metadata, low_risk_file_patterns, high_risk_file_patterns, high_risk_keywords, thresholds, test_file_patterns, auto_approval_teams

initialize(data) builds:

@pr_metadata = PrMetadata.new(require_description: data.dig('pr_metadata','require_description') || true, min_description_length: data.dig('pr_metadata','min_description_length') || 0) — use data.fetch('pr_metadata', {}).fetch(...) style with sensible defaults

@thresholds = Thresholds.new(max_diff_lines: data.dig('thresholds','max_diff_lines') || 1000, max_production_diff_lines: data.dig('thresholds','max_production_diff_lines') || 400)

@low_risk_file_patterns = Array(data['low_risk_file_patterns']) and similar for high_risk_file_patterns, high_risk_keywords, test_file_patterns, auto_approval_teams

Use # frozen_string_literal: true. No external dependencies.

Acceptance criteria:

StaticAnalysisConfig.new({}).pr_metadata.require_description == true

StaticAnalysisConfig.new({}).thresholds.max_diff_lines == 1000

All array accessors default to []

Sub-task 2.2 — `Instructions`

Prompt:

Create lib/pr_risk_classifier/configuration/instructions.rb (PrRiskClassifier::Instructions). It wraps the instructions: block of the YAML config and renders it as a Markdown section that gets injected into LLM prompts.

Requirements:

attr_reader :warnings, :context, :sensitive_areas

initialize(data) assigns @warnings = Array(data['warnings']) etc.

to_prompt_section returns a heredoc starting with ## Custom Instructions\n followed by sub-sections ### Warnings, ### Context, ### Sensitive Areas. Each sub-section is rendered as a bullet list (- item). Empty sub-sections are omitted. If all three are empty, return ''.

Acceptance criteria:

Instructions.new({}).to_prompt_section == ''

Instructions.new('warnings' => ['x']).to_prompt_section includes ## Custom Instructions, ### Warnings, - x, but NO ### Context or ### Sensitive Areas

Sub-task 2.3 — `Configuration` loader

Prompt:

Create lib/pr_risk_classifier/configuration.rb (PrRiskClassifier::Configuration). It loads a YAML config file, deep-merges it on top of config.common.yml, and exposes typed config objects.

Requirements:

require 'yaml', require_relative 'configuration/static_analysis_config', require_relative 'configuration/instructions'

Constant: COMMON_CONFIG_FILENAME = 'config.common.yml'

attr_reader :static_analysis, :instructions, :critical_paths, :external_services, :prompt_profile

Class method Configuration.load(base_path: nil, config_filename:):

base_path ||= File.dirname(__FILE__)

Load File.join(base_path, config_filename) via YAML.load_file

If File.join(base_path, COMMON_CONFIG_FILENAME) exists, load it and deep-merge the profile YAML on top: common.deep_merge(profile). (Rails' Hash#deep_merge is fine; if not running inside Rails, require active_support/core_ext/hash/deep_merge.)

Return new(merged_yaml, base_path: base_path)

initialize(yaml_hash, base_path:):

@prompt_profile = yaml_hash.fetch('prompt_profile', 'backend').to_s.downcase

@static_analysis = StaticAnalysisConfig.new(yaml_hash.fetch('static_analysis', {}))

@instructions = Instructions.new(yaml_hash.fetch('instructions', {}))

@critical_paths = yaml_hash.fetch('critical_paths', [])

@external_services = yaml_hash.fetch('external_services', [])

prompt_instructions → delegate to @instructions.to_prompt_section

Acceptance criteria:

Configuration.load(base_path: 'lib/pr_risk_classifier', config_filename: 'config.backend.yml').prompt_profile == 'backend'

The frontend config returns 'frontend'

static_analysis.thresholds.max_diff_lines reflects the merged value

Phase 3 — External clients

Sub-task 3.1 — `GithubClient`

Prompt:

Create lib/pr_risk_classifier/github_client.rb (PrRiskClassifier::GithubClient). It wraps the gh CLI (so the calling environment must have gh installed and authenticated). All shell calls use Open3.capture3.

Public API:

Custom error: class CliError < StandardError with extra attr_reader :stderr, :exit_status. Constructor: initialize(message, stderr: nil, exit_status: nil).

PrData = Struct.new(:number, :title, :description, :diff, :files_changed, :additions, :deletions, :diff_truncated, :author, keyword_init: true).

DEFAULT_MAX_DIFF_LINES = 2000 (truncate diff to this many lines if no config is supplied).

attr_reader :owner, :repo.

initialize(owner: 'YourOrg', repo: 'YourRepo', config: nil) (defaults must match your repo). Calls a private verify_gh_installed! that runs Open3.capture2('which', 'gh') and raises CliError with a helpful install message if not found.

fetch_pr(pr_number) → returns a PrData. Internally calls:

gh pr view <n> --repo <owner>/<repo> --json title,body,additions,deletions,changedFiles,author

gh pr diff <n> --repo <owner>/<repo> Then truncates the diff to max_diff_lines lines (record truncated: true/false). author is pr_info.dig('author', 'login').

fetch_pr_branch(pr_number) → string. Uses gh pr view <n> --repo ... --json headRefName -q .headRefName.

fetch_pr_base_branch(pr_number) → string. Same with baseRefName.

fetch_pr_files(pr_number) → Array<Hash> of { file:, additions:, deletions: }. Uses gh pr view <n> --repo ... --json files. The path can come from any of path, filePath, name, filename, or file keys (different gh versions disagree). Skip rows with no path.

diff_exceeds_limit?(pr_number) → bool. Compares additions + deletions (from fetch_pr_info) to max_diff_lines.

team_member?(team_slug, username) → bool. team_slug is "org/team". Hits gh api /orgs/<org>/teams/<team>/members/<username> --silent. Returns true iff exit status is success. Use a separate helper run_gh_with_org_token that, when ENV['GH_ORG_TOKEN'] is set, runs Open3.capture3({'GH_TOKEN' => ENV['GH_ORG_TOKEN']}, 'gh', *args). This lets the workflow use a PAT for the org-membership check while the regular workflow token is used for everything else.

Helpers (private):

run_gh(*args) = Open3.capture3('gh', *args)

truncate_diff(diff) returns { diff:, truncated: }

max_diff_lines returns @config.static_analysis.thresholds.max_diff_lines if config given else DEFAULT_MAX_DIFF_LINES

raise_cli_error(operation, stderr, status) raises CliError

Use # frozen_string_literal: true, require 'json', require 'open3'.

Acceptance criteria:

GithubClient.new raises CliError if gh is not on PATH

fetch_pr_branch(N) returns a stripped branch name string

fetch_pr_files(N) returns an array of {file:, additions:, deletions:} (filters out paths-less rows)

Sub-task 3.2 — `LlmClient`

Prompt:

Create lib/pr_risk_classifier/llm_client.rb (PrRiskClassifier::LlmClient) — a thin wrapper around the Anthropic Messages API. Use httparty.

Constants:

BASE_URL = 'https://api.anthropic.com/v1'

DEFAULT_MODEL = 'claude-sonnet-4-20250514' (or your preferred Sonnet model)

API_VERSION = '2023-06-01'

Public API:

class ApiError < StandardError with attr_reader :response. Constructor: initialize(message, response = nil).

initialize(api_key: ENV.fetch('ANTHROPIC_API_KEY', nil), model: DEFAULT_MODEL) — raise ArgumentError if api_key.blank? (or nil? if not in Rails).

message(prompt, system: nil, max_tokens: 4096) — POSTs to /messages with body { model:, max_tokens:, messages: [{role:'user', content: prompt}] } and includes system: only when provided. Returns the parsed JSON body.

generate_text(prompt, system: nil, max_tokens: 4096) — calls message and returns the text from response.dig('content', 0, 'text') (only when content[0]['type'] == 'text', else '').

Private helpers:

post(path, body) — uses HTTParty.post("#{BASE_URL}#{path}", headers:, body: body.to_json, timeout: 120). Calls handle_response.

headers — { 'Content-Type' => 'application/json', 'x-api-key' => @api_key, 'anthropic-version' => API_VERSION }.

handle_response(response) — return parsed_response on success, else raise ApiError with status + body.

Acceptance criteria:

LlmClient.new(api_key: nil) raises ArgumentError

generate_text returns a String

120-second HTTP timeout

Phase 4 — Pipeline scaffolding

Sub-task 4.1 — `PrContext`

Prompt:

Create lib/pr_risk_classifier/classification/pr_context.rb (PrRiskClassifier::Classification::PrContext). It holds all pre-fetched PR data so each analyser doesn't redo network calls.

Requirements:
attr_reader :pr_number, :branch_name, :base_branch, :pr_data, :files, :diff_stats

delegate :title, :description, :diff, :diff_truncated, :author, to: :pr_data (use Forwardable if you don't have ActiveSupport).

initialize(pr_number:, branch_name:, base_branch:, pr_data:, files:) — sets the ivars and computes @diff_stats.

diff_truncated? returns diff_truncated.
compute_diff_stats returns:
{ files: files,
  file_count: ...,
  total_additions: ...,
  total_deletions: ...,
  total_changes: total_additions + total_deletions }
When files is non-empty, sum f[:additions].to_i etc. When files is empty, fall back to pr_data.files_changed.to_i / additions / deletions.
Acceptance criteria:

PrContext.new(...).diff_stats[:total_changes] == additions + deletions

Falls back to PR-level stats when files == []

Sub-task 4.2 — `PipelineResult`

Prompt:

Create lib/pr_risk_classifier/classification/pipeline_result.rb (PrRiskClassifier::Classification::PipelineResult). Holds the per-step results and exposes derived properties.

Requirements:

attr_accessor :pr_number, :step1, :step2, :step3 (initialised to nil).

initialize(pr_number:).

final_step → step3 || step2 || step1.

delegate :classification, to: :final_step.

steps_run → %i[step1 step2 step3].reject { |s| send(s).nil? }.

early_exited_at → :step1 if step1&.early_exit? and step2.nil?; :step2 if step2&.high_risk? and step3.nil?; else nil.

to_h → hash with pr_number, classification, steps_run, early_exited_at, step1: step1&.to_h, step2: step2&.to_h, step3: step3&.to_h.

Acceptance criteria:

With only step1 set and early_exit? true → early_exited_at == :step1, classification comes from step1

to_h[:step3] is nil when step3 wasn't run

Sub-task 4.3 — `RiskClassificationPipeline`

Prompt:

Create lib/pr_risk_classifier/classification/risk_classification_pipeline.rb (PrRiskClassifier::Classification::RiskClassificationPipeline).

This is the orchestrator. It takes a config + clients, fetches all PR data once into a PrContext, runs each step that's requested, and stops early when the result is conclusive.

Requires (relative): pipeline_result, pr_context, static_analyser, semantic_analyser, agentic_analyser (these classes will be built in later phases — that's fine).

Requirements:

initialize(config:, github_client:, llm_client: nil, **options) — store @config, @github_client, @llm_client, @verbose = options.fetch(:verbose, false), @on_progress = options[:on_progress] (a callable).

classify(pr_number, steps: %i[static semantic agentic]) → PipelineResult:

Build a PipelineResult.new(pr_number:).

Build pr_context via build_pr_context(pr_number) — calls @github_client.fetch_pr/pr_branch/pr_base_branch/pr_files and constructs PrContext.

Run steps in order. For each step:

If steps.include?(:static) → run static, then early exit if result.step1.early_exit?.

If steps.include?(:semantic) → run semantic, then early exit if result.step2.high_risk?.

If steps.include?(:agentic) → run agentic.

Report :pipeline_complete to @on_progress and return the result.

run_static_analysis(ctx) → StaticAnalyser.new(config:, github_client:).analyse(ctx).

run_semantic_analysis(ctx, static_context:) — raises ArgumentError unless @llm_client set; calls SemanticAnalyser.new(config:, llm_client:).analyse(ctx, static_context:).

run_agentic_analysis(ctx) → AgenticAnalyser.new(config:, verbose:).analyse(ctx).

report_progress(event, **details) calls @on_progress&.call(event, **details). Emit these events: :fetching_pr, :pr_fetched, :step_started, :step_completed, :early_exit, :pipeline_complete. Each has step name + relevant details.

Don't worry that the analysers don't exist yet — autoloading will resolve them when the user actually runs the pipeline.

Acceptance criteria:

classify(pr, steps: [:static]) runs only step1

When step1 is early-exit it does NOT call the LLM

When step2 returns HIGH_RISK it does NOT call the agentic analyser

All progress events emit through on_progress if provided

Sub-task 4.4 — `Runner` facade

Prompt:

Create lib/pr_risk_classifier/classification/runner.rb (PrRiskClassifier::Classification::Runner). A thin facade so external callers don't talk to the pipeline directly.
require_relative 'risk_classification_pipeline'

module PrRiskClassifier
  module Classification
    class Runner
      def initialize(config:, github_client:, llm_client:, verbose: false, on_progress: nil)
        @pipeline = RiskClassificationPipeline.new(config:, github_client:, llm_client:, verbose:, on_progress:)
      end

      def classify(pr_number, steps: %i[static semantic agentic])
        @pipeline.classify(pr_number, steps:)
      end
    end
  end
end
Acceptance criteria: A unit test instantiating the Runner with stub clients and asserting the call gets forwarded to the pipeline passes.

Phase 5 — Step 1: Static Analyser

Sub-task 5.1 — `StaticAnalyser`

Prompt:

Create lib/pr_risk_classifier/classification/static_analyser.rb (PrRiskClassifier::Classification::StaticAnalyser).

This is the deterministic, no-LLM Step 1. It returns LOW_RISK, HIGH_RISK, or INCONCLUSIVE. Only LOW_RISK and HIGH_RISK are early exits (early_exit: true); INCONCLUSIVE falls through to Step 2.

Result struct
StaticResult = Struct.new(:classification, :confidence, :early_exit, :matched_patterns, :reasoning, keyword_init: true) do
  def early_exit? = early_exit
  def low_risk?  = classification == 'LOW_RISK'
  def high_risk? = classification == 'HIGH_RISK'
  def to_h
    { classification:, confidence:, early_exit:, matched_patterns:, reasoning: }
  end
end
Class skeleton
def initialize(config:, github_client: nil)
  @config = config
  @github_client = github_client
end

def analyse(pr_context)
  # 1. If PR base branch != 'master' → LOW_RISK early exit (auto-approve only applies to main line)
  # 2. If author is not a member of any auto_approval_teams → HIGH_RISK
  # 3. Compute diff stats (split into production_changes vs test_changes by test_file_patterns)
  # 4. If all files match low_risk_file_patterns → LOW_RISK early exit
  # 5. Else check high-risk rules in order:
  #    a. PR description missing or shorter than min_description_length → HIGH_RISK
  #    b. production_changes > max_production_diff_lines OR total_changes > max_diff_lines → HIGH_RISK
  #    c. Any changed file path includes any high_risk_file_patterns → HIGH_RISK
  #    d. Any high_risk_keywords (regex, case-insensitive) match the diff → HIGH_RISK
  # 6. Otherwise INCONCLUSIVE.
end
Detailed rules

Rule constants — declare a frozen STATIC_RULES hash keyed by :low_risk_early_exit, :high_risk, :low_risk so reasoning strings are uniform. Each entry: { key:, description:, confidence: }. Confidences:

non_master_target: 1.0

author_not_in_team: 1.0

missing_description: 1.0

diff_too_large: 1.0

sensitive_file_patterns: 0.95

dangerous_keywords: 0.9

safe_files_only: 1.0

Author team check: only when auto_approval_teams non-empty AND @github_client present. Iterate teams.any? { |t| @github_client.team_member?(t, author) }. If none match → HIGH_RISK with reason "PR author '<author>' is not a member of any auto-approval GitHub team".

Description rule: only when pr_metadata.require_description == true. Fail if description.to_s.strip.empty? or length < min_description_length. Reason: "PR description missing or too short (minimum: <n> chars)".

Diff size rule: split files into production vs test using test_file_patterns (file path includes any of those substrings → test file). Compare production_changes to thresholds.max_production_diff_lines AND total_changes to thresholds.max_diff_lines. Reason example: "Diff too large - production: 1234/400, total: 1500/1000 (test: 0 lines)".

High-risk file patterns: case-sensitive String#include? substring match. Record matched as "<pattern> (<file>)" strings.

High-risk keywords: each pattern is a regex string; build with Regexp.new(pattern, Regexp::IGNORECASE). Rescue RegexpError and skip invalid patterns.

All-files-low-risk rule: pass only if low_risk_file_patterns non-empty AND every changed file path includes at least one of those patterns.

Inconclusive: classification: 'INCONCLUSIVE', confidence: 0.0, early_exit: false, matched_patterns: [], reasoning: 'No definitive patterns matched - requires Semantic Analysis'.

Use # frozen_string_literal: true. Add # rubocop:disable Metrics/ClassLength around the class.

Acceptance criteria:

All-spec PR → LOW_RISK, early_exit? true, no LLM ever called

PR with db/migrate/... file → HIGH_RISK, matched_patterns includes "db/migrate/ (...)"

PR targeting develop → LOW_RISK with reason about non-master target

Empty diff with valid description, valid author, no patterns → INCONCLUSIVE

Phase 6 — Step 2: Semantic Analyser (LLM)

Sub-task 6.1 — `BackendSemanticPromptBuilder`

Prompt:

Create lib/pr_risk_classifier/classification/backend_semantic_prompt_builder.rb (PrRiskClassifier::Classification::BackendSemanticPromptBuilder).

Builds the system + user prompts that the Semantic Analyser sends to Claude. Domain-specific text is for a Rails / Ruby backend.

Public API:
initialize(config:, static_context: nil) — stores both.

system_prompt — joins, with blank lines, the base system prompt below + @config.prompt_instructions (omit when blank).
classification_prompt(pr_data) — builds the user message. Include:
## PR Information
- PR Number: #<n>
- Title: <t>
- Description: <body or 'No description provided'>
- Files Changed: <count>
- Additions: <n>
- Deletions: <n>
- Diff Truncated: <bool>

<static_context_section if @static_context>

## Diff
<pr_data.diff>
Based on the patterns and rules provided, classify this PR as HIGH_RISK or LOW_RISK.
Respond with the JSON object as specified.
Private static_context_section — when @static_context is present, include ## Static Analysis Context with Result:, Matched patterns: (joined comma list or None), Reasoning:. Else ''.
Base system prompt (verbatim — adjust app name)

Tell Claude it's a backend PR risk classifier for a specific Rails app. List High Risk Signals:

Data integrity violations (validations, AR callbacks)

Authorization/auth changes (current_user, sessions, cookies, tokens, Pundit/CanCanCan)

Background-job changes touching critical paths

AR scope/default scope changes

Hot-path SQL changes

Routing changes

Payment/billing logic

External API integrations (idempotency, retries)

Global config changes (initializers, application.rb)

Serializer/JSON API changes

End with the Response Format:
{
  "classification": "HIGH_RISK" or "LOW_RISK",
  "confidence": 0.0 to 1.0,
  "reasoning": "Brief explanation of why this classification was chosen",
  "risk_factors": ["List", "of", "specific", "risk", "factors", "identified"]
}
Acceptance criteria:

system_prompt always includes the High Risk Signals list

When the config has instructions, they appear under a ## Custom Instructions block

When static_context is nil, the ## Static Analysis Context section is absent

Sub-task 6.2 — `FrontendSemanticPromptBuilder`

Prompt:

Create lib/pr_risk_classifier/classification/frontend_semantic_prompt_builder.rb. Same shape and API as BackendSemanticPromptBuilder but the base system prompt targets a React/TypeScript frontend.

List High Risk Signals with frontend-flavoured items: shared UI primitives, global state (Context, Redux, Apollo cache), checkout/payment/login flows, Stripe Elements, cookies / localStorage / session storage, error boundaries, build config / dependency updates.

Add a Low Risk Signals section: tests, docs, Storybook-only changes, cosmetic styling, narrow bug fixes.

Same JSON Response Format as backend.

Acceptance criteria: system_prompt mentions React/TypeScript, Stripe Elements, Apollo Client, and at least one Low Risk Signal.

Sub-task 6.3 — `SemanticAnalyser`

Prompt:

Create lib/pr_risk_classifier/classification/semantic_analyser.rb (PrRiskClassifier::Classification::SemanticAnalyser).

Step 2 of the pipeline: sends a prompt to Claude via LlmClient and parses a JSON response.

Requires (relative): backend_semantic_prompt_builder, frontend_semantic_prompt_builder. require 'json'.

Result struct
SemanticResult = Struct.new(:classification, :confidence, :reasoning, :risk_factors, keyword_init: true) do
  def high_risk? = classification == 'HIGH_RISK'
  def low_risk?  = classification == 'LOW_RISK'
  def to_h = { classification:, confidence:, reasoning:, risk_factors: }
end
Class
def initialize(config:, llm_client:)
def analyse(pr_context, static_context: nil)
  builder = (config.prompt_profile == 'frontend' ? FrontendSemanticPromptBuilder : BackendSemanticPromptBuilder).new(
    config: @config, static_context: static_context
  )
  response = @llm_client.generate_text(
    builder.classification_prompt(pr_context.pr_data),
    system: builder.system_prompt,
    max_tokens: 1000
  )
  parse_response(response)
end
parse_response:

Extract JSON via response.match(/\{[\s\S]*\}/). Raise ParseError if no match.

JSON.parse(match[0]). Wrap parse errors in ParseError.

Build SemanticResult.new(classification:, confidence: parsed['confidence'].to_f, reasoning:, risk_factors: Array(parsed['risk_factors'])).

Define class ParseError < StandardError; end inside the class.

Acceptance criteria:

With a stubbed LlmClient returning '```json\n{"classification":"LOW_RISK","confidence":0.9,"reasoning":"x","risk_factors":[]}\n```', analyse returns a SemanticResult with classification LOW_RISK

Raises ParseError on garbage output

Selects FrontendSemanticPromptBuilder only when config.prompt_profile == 'frontend'

Phase 7 — Step 3: Agentic Analyser (Claude Code CLI)

Sub-task 7.1 — `AgenticPromptSections` mixin

Prompt:

Create lib/pr_risk_classifier/classification/agentic_prompt_sections.rb — module PrRiskClassifier::Classification::AgenticPromptSections. Holds shared prompt fragments used by both backend and frontend agentic prompt builders.

Methods (all def, no module_function; the module is included):

introduction_section — heredoc explaining we're assessing blast radius, not code quality. Goal: how much of the codebase would be affected if these changes have bugs.

response_format_section — ## Response Format heredoc telling the model to respond with the JSON shown in json_schema_example(file_count). file_count comes from pr_context.diff_stats[:file_count].

json_schema_example(file_count) — joins four chunks with ,\n:

Header: opens the JSON object, includes "risk_level": "LOW|MEDIUM|HIGH|CRITICAL", "confidence": 0.0-1.0, and a "blast_radius" block with files_directly_changed set to file_count, estimated_files_affected, critical_paths_touched, entry_points_affected.

Change summary: "change_summary": { methods_modified, methods_added, methods_removed, nature: "feature|bugfix|refactor|config|dependencies" }.

Factors: "risk_factors": [{ factor, severity: "low|medium|high", evidence }] and "mitigating_factors": [{ factor, evidence }].

Footer: closes with "exploration_notes" (narrative) and "recommended_review_focus" (array), then }.

Use <<~JSON.chomp for each chunk so the joins look clean.

Acceptance criteria: introduction_section, response_format_section, and json_schema_example(3) all return non-empty strings; the JSON example, when wrapped in {}, is conceptually a valid JSON shape (it's a prompt template — string fields are placeholders).

Sub-task 7.2 — `AgenticPromptBuilder` base class

Prompt:

Create lib/pr_risk_classifier/classification/agentic_prompt_builder.rb — PrRiskClassifier::Classification::AgenticPromptBuilder. Base class for the per-profile builders. Includes AgenticPromptSections.
require_relative 'agentic_prompt_sections'
HIGH_CALLER_COUNT_THRESHOLD = 5

initialize(config:)

exploration_prompt(pr_context) — stores @pr_context = pr_context, returns build_prompt

build_prompt — joins (with \n) the result of core_sections + analysis_sections, dropping nils.

core_sections returns [introduction_section, context_section, codebase_knowledge_section, config.prompt_instructions, static_checks_note] — context_section will be defined in subclasses.

analysis_sections returns [pre_analysis_section, diff_section, exploration_task_section, response_format_section, risk_criteria_section, closing_section] — exploration_task_section, risk_criteria_section, closing_section are defined in subclasses.

codebase_knowledge_section — heredoc with ## Codebase-Specific Knowledge listing Critical paths that require extra scrutiny: (config.critical_paths as a bullet list) and External service integrations: (config.external_services as a bullet list).

static_checks_note — single line: "Static checks have already handled diff size and obvious file/keyword flags; focus on call-chain impact and indirect risk."

pre_analysis_section — heredoc starting ## Pre-Analysis: What Changed, including the optional PR #<n> - prefix (only when pr_number present), Branch: \` compared to ``, and stats from pr_context.diff_stats. Followed by a bulleted list of changed files: - (+/-)`.

diff_section — heredoc with ## Diff Content, then a fenced diff block with pr_context.diff, plus a "Note: Diff was truncated due to size." note when pr_context.diff_truncated?.

Helpers: format_list(items) and format_file_changes(files) returning bullet-list strings.

Mark all the section methods private except exploration_prompt. attr_reader :config, :pr_context (private).

Acceptance criteria: A subclass that defines context_section, exploration_task_section, risk_criteria_section, closing_section produces a non-empty string from exploration_prompt(ctx).

Sub-task 7.3 — `BackendAgenticPromptBuilder`

Prompt:

Create lib/pr_risk_classifier/classification/backend_agentic_prompt_builder.rb extending AgenticPromptBuilder. Provides Rails-flavoured exploration instructions for Claude Code.

Define these private methods:

context_section — ## Context: <App Name> paragraph explaining it's a Ruby on Rails app + listing business-critical areas (payments, subscriptions, fulfilment, auth — adapt to your domain).

exploration_task_section — ## Your Exploration Task heredoc with five numbered steps:

Review the provided diff.

Trace impact: use Grep with "ClassName", "method_name", check app/controllers/, app/jobs/.

Follow Rails conventions: model → controllers/services; service → callers; concern → including classes; check callbacks.

Assess critical-path involvement (payments, auth, fulfilment, etc.).

Look for risk amplifiers: rescue removals, signature changes, validation removal, scope changes.

risk_criteria_section — ## Risk Level Criteria heredoc covering:

Measurable indicators: caller count, inheritance depth, test ratio.

Change nature: additions / modifications / deletions / renames.

Code Pattern Red Flags: guard removal, signature changes, raw SQL, callback mods, security filters, scope changes.

Hard Rule (use HIGH_CALLER_COUNT_THRESHOLD from the parent class): "If the changed code has #{threshold} or more callers/usages across the codebase, the PR MUST NOT be classified as LOW risk regardless of any mitigating factors."

LOW criteria: purely additive, < 5 usages, simple logging additions, runtime-neutral config, isolated new features with tests.

HIGH criteria: many callers, deletions, shared modules/concerns/base classes, critical paths, red-flag patterns.

closing_section — ## Important heredoc: "Actually run the search commands. Don't guess at usage - verify it. Trace call chains 2-3 levels deep. Consider Rails magic. Lean toward higher risk when uncertain. Cite file names and line numbers. Begin your exploration."

Acceptance criteria: BackendAgenticPromptBuilder.new(config:).exploration_prompt(ctx) includes the substrings "Hard Rules", "5 or more", "Begin your exploration.", and "app/controllers/".

Sub-task 7.4 — `FrontendAgenticPromptBuilder`

Prompt:

Create lib/pr_risk_classifier/classification/frontend_agentic_prompt_builder.rb extending AgenticPromptBuilder. Same four private methods as the backend builder, but target React/TypeScript:

context_section: React/TS frontend, business-critical areas (checkout/Stripe Elements, auth/session, subscription mgmt, account details). Note that app/javascript/components/elements/ (or your equivalent path) contains shared primitives that fan out widely.

exploration_task_section: 1) review diff; 2) trace shared component usage (Grep imports & renders, watch components/elements/, components/shared/, hooks/); 3) state/data flow (Context, Redux, Apollo cache, GraphQL queries/mutations, useEffect deps); 4) critical-path involvement; 5) risk amplifiers (validation/error-handling changes, default-prop changes, hook side effects, global styles).

risk_criteria_section: Measurable indicators (usage count, route reach, state surface area). Change nature. Frontend-specific Red Flags (shared UI primitives, useEffect deps, Apollo cache shape, auth storage, payment flow, routing). Same Hard Rule wording about ≥ HIGH_CALLER_COUNT_THRESHOLD usages → cannot be LOW. Concrete LOW / HIGH bullet lists.

closing_section: "Actually run the search commands. Trace component usage across pages and flows. When a shared component is widely used, lean toward HIGH. Cite files + line numbers. Begin your exploration."

Acceptance criteria: Output mentions Apollo, Stripe Elements, useEffect, and the same Hard Rule about 5 usages.

Sub-task 7.5 — `AgenticResult`

Prompt:

Create lib/pr_risk_classifier/classification/agentic_result.rb (PrRiskClassifier::Classification::AgenticResult). Wraps the JSON Claude Code returned and exposes helper predicates.
FIELD_DEFAULTS = {
  'blast_radius' => {},
  'risk_factors' => [],
  'mitigating_factors' => [],
  'change_summary' => {},
  'recommended_review_focus' => [],
  'auto_classified' => false
}.freeze
Constructor takes a hash with any of these string keys: pr_number, risk_level, confidence, blast_radius, risk_factors, mitigating_factors, change_summary, exploration_notes, recommended_review_focus, raw_response, error, auto_classified. Apply defaults from FIELD_DEFAULTS.

attr_reader for all of them.

Predicates / helpers:

high_risk? — %w[HIGH CRITICAL].include?(@risk_level)

auto_classified? — @auto_classified

classification — 'HIGH_RISK' if high_risk? else 'LOW_RISK' (this is what the pipeline reads back to make the final decision)

reasoning — @exploration_notes or, when blank, a ;-joined risk_factors_summary

auto_approvable? — @risk_level == 'LOW' && @confidence.to_f >= 0.8

needs_senior_review? — 'CRITICAL' OR ('HIGH' AND touches_payments?)

touches_payments? — true if any string in @blast_radius['critical_paths_touched'] (downcased) includes 'payment'

to_h — base attrs + computed { auto_approvable:, needs_senior_review: }

Acceptance criteria:

AgenticResult.new('risk_level' => 'LOW', 'confidence' => 0.9).classification == 'LOW_RISK'

AgenticResult.new('risk_level' => 'CRITICAL').needs_senior_review? == true

to_h[:auto_approvable] returns a bool

Sub-task 7.6 — `StreamingOutputHandler`

Prompt:

Create lib/pr_risk_classifier/classification/streaming_output_handler.rb (PrRiskClassifier::Classification::StreamingOutputHandler). A module_function module that parses one line of Claude Code CLI's --output-format stream-json output and either prints it (for visibility in verbose: true mode) or returns a tagged event that the caller acts on.

Public:

process_line(line) — strips empty lines, parses as JSON, calls handle_event. Rescues JSON::ParserError; if line text contains 'error' it prints a debug line, returns nil either way.

handle_event(event) — switches on event['type']:

'assistant' → call print_assistant_content(event['message'])

'content_block_delta' → print_delta(event['delta'])

'result' → return { type: :result, data: event }

'error' → return { type: :error, data: event.dig('error', 'message') || event.to_s }

else nil

print_assistant_content — iterates message['content'] array, prints each block:

text blocks → puts text

tool_use → puts "\n[Tool: <name>]"

tool_result → puts '[Tool result received]'

print_delta(delta) — print delta['text'] (no newline) when present.

output(text, newline: true) — puts or print, then $stdout.flush.

Use # rubocop:disable Rails/Output around the output method.

Acceptance criteria: Calling process_line('{"type":"result","result":"x"}') returns a hash with type: :result. Calling on a malformed line returns nil.

Sub-task 7.7 — `AgenticAnalyser`

Prompt:

Create lib/pr_risk_classifier/classification/agentic_analyser.rb (PrRiskClassifier::Classification::AgenticAnalyser).

Step 3 of the pipeline. Shells out to the locally installed claude CLI (Claude Code) so it can explore the actual repo on disk. The repo MUST already be checked out at repo_path (or Dir.pwd).

Requires (relative): backend_agentic_prompt_builder, frontend_agentic_prompt_builder, agentic_result, streaming_output_handler. require 'open3', require 'json'.
RISK_LEVELS = %w[LOW MEDIUM HIGH CRITICAL].freeze

def initialize(config:, repo_path: nil, verbose: false)
  @repo_path = File.expand_path(repo_path || Dir.pwd)
  @config = config
  @verbose = verbose
end

def analyse(pr_context)
  builder = (config.prompt_profile == 'frontend' ? FrontendAgenticPromptBuilder : BackendAgenticPromptBuilder).new(config: @config)
  prompt = builder.exploration_prompt(pr_context)
  raw = execute_claude_code(prompt)
  data = parse_result(raw)
  AgenticResult.new(data.merge('pr_number' => pr_context.pr_number, 'raw_response' => raw))
end
execute_claude_code(prompt) — when @verbose, use streaming PTY mode; else quiet capture3.

Quiet mode: Open3.capture3('claude', '-p', prompt, '--output-format', 'json', chdir: @repo_path). Raise on non-zero exit. JSON.parse(stdout).

Streaming mode: PTY.spawn('claude', '-p', prompt, '--output-format', 'stream-json', '--verbose', chdir: @repo_path). For each line, call StreamingOutputHandler.process_line(line) and update local final_result / last_error based on :result / :error events. Rescue Errno::EIO (PTY raises it on EOF), Process.wait(pid) in ensure. After exit, raise on non-zero. Return final_result || { 'result' => '' }.

parse_result(claude_response):

content = claude_response['result'] || claude_response['content'] || claude_response.to_s

Try to extract JSON: first ```json\n...\n``` fence; fallback to (\{.*"risk_level".*\}) regex.

JSON.parse it, then normalise:

risk_level: if not in RISK_LEVELS, set to 'MEDIUM' and append 'Invalid risk level, defaulted to MEDIUM' to analysis['validation_warnings'].

confidence: analysis['confidence']&.to_f&.clamp(0.0, 1.0) || 0.5.

Default these to empty hash/array: blast_radius, risk_factors, mitigating_factors, recommended_review_focus.

On any parse failure, return {'risk_level' => 'HIGH', 'confidence' => 0.3, 'error' => '<msg>', 'exploration_notes' => content} (fail-safe to HIGH).

Acceptance criteria:

Returns an AgenticResult whose pr_number equals the input

On claude exit non-zero, raises with stderr in the message

On JSON-fence response from Claude, returns the parsed risk level

On garbage response, returns risk_level HIGH and error set

Phase 8 — Glue: TaskRunner & Rake tasks

Sub-task 8.1 — `TaskRunner`

Prompt:

Create lib/pr_risk_classifier/task_runner.rb (PrRiskClassifier::TaskRunner). It encapsulates everything the rake task needs: which configs to load for a given PR, how to combine multi-profile results, and how to print pretty CLI output. Keep this fat so the rake task is thin.

Constants:

FRONTEND_CONFIG_FILENAME = 'config.frontend.yml'

BACKEND_CONFIG_FILENAME = 'config.backend.yml'

FRONTEND_EXTENSIONS = %w[.js .jsx .ts .tsx .css .scss .sass].freeze

Public methods:
initialize(root_path:) — store @root_path (the Rails root).

build_classification_runner(config:, steps:, verbose: false, on_progress: nil) — returns a Classification::Runner wired with GithubClient.new(config:) and LlmClient.new(api_key: ENV['ANTHROPIC_API_KEY']) only when :semantic is in steps (otherwise llm_client: nil).

configs_for_pr(pr_number) — calls GithubClient.new.fetch_pr_files(pr_number), then:

If any file has a frontend extension AND any file does not → return [backend_config, frontend_config]

If only frontend → [frontend_config]

Else → [backend_config] Use File.extname(path).downcase for extension match. Loads each config via Configuration.load(base_path: File.join(@root_path, 'lib/pr_risk_classifier'), config_filename:).
combine_profile_results(profile_results) — profile_results is an array of {profile:, result:}. Returns:
{ classification:, final_step:, reasoning:, selected_profile:, selected_result:, profile_results: }
Logic:

If any profile is HIGH_RISK → classification = 'HIGH_RISK' and pick that profile (first one that's HIGH_RISK).

Else LOW_RISK; pick the profile whose result.steps_run.length is largest (most-explored).

final_step: 'agentic' if any result ran step3, else 'semantic' if any ran step2, else 'static'.

reasoning: selected.result.final_step.reasoning || 'No reasoning provided'. When more than one profile, prepend "Mixed PR results (backend=X, frontend=Y). Selected <profile> reasoning: ...".
print_pipeline_result(result) — coloured CLI output for one profile:

Summary header showing Final classification, Steps run, Early exit.

For each step, a formatted block (Step 1 (Static), Step 2 (Semantic), Step 3 (Agentic) deep block with blast radius, risk factors, mitigating factors, recommended review focus, exploration notes). Use ANSI colour codes (red for HIGH, green for LOW, yellow for MEDIUM, purple for CRITICAL).

handle_error(error) — prints red error + first 5 backtrace lines, then re-raises.
Methods can be split into many small private helpers (format_static_result, format_semantic_result, format_deep_result, format_blast_radius, format_factors_section, format_risk_factors_brief, format_risk_factors_detailed, format_mitigating_factors, severity_to_color, risk_level_color, format_list_or_none) to keep each method short.

Acceptance criteria:

Frontend-only PR loads only the frontend config

Mixed PR loads two configs and runs them both

combine_profile_results selects HIGH_RISK if any profile flagged HIGH_RISK

print_pipeline_result writes coloured output to stdout

Sub-task 8.2 — Rake tasks

Prompt:

Create lib/tasks/pr_risk_classifier.rake. Two tasks under the pr_risk: namespace.

Shared helpers

task_runner — memoised PrRiskClassifier::TaskRunner.new(root_path: Rails.root.to_s). Inside, require_relative '../pr_risk_classifier'.

verbose? — ENV.fetch('verbose', 'false').downcase == 'true'

parse_steps — reads ENV['steps'] (default '1,2,3'), maps '1' => :static, '2' => :semantic, '3' => :agentic. Raise on empty.

step_name(step) — friendly label.

classification_progress_callback — lambda that prints progress lines for each pipeline event.

final_step_for_result(result) — :agentic / :semantic / :static.

Task 1: pr_risk:classify

Description: 'Classify a PR (usage: rake pr_risk:classify pr=12345 [steps=1,2,3] [verbose=true])'. Depends on :environment.

Steps:

Parse pr= (must be present; accept either an integer or the full PR URL — String#scan(/\d+/).join.to_i).

steps = parse_steps.

configs = task_runner.configs_for_pr(pr_number) (auto-detect backend / frontend / both).

Print a header. For each config:

Build runner with verbose: verbose? and on_progress: classification_progress_callback.

Call runner.classify(pr_number, steps:).

Collect into profile_results.

combined = task_runner.combine_profile_results(profile_results).

Print each profile's result via task_runner.print_pipeline_result.

Print Final classification: and Final step:.

rescue StandardError => e → task_runner.handle_error(e).

Task 2: pr_risk:classify_json

Description: 'Classify a PR and output JSON (for CI use)'. Same steps but never verbose:. Output must be a single line of JSON to stdout (last line — the GitHub Action greps for it). Shape:
{
  "pr_number": 12345,
  "classification": "LOW_RISK",
  "steps_run": ["step1","step2","step3"],
  "early_exited_at": null,
  "final_step": "agentic",
  "reasoning": "...",
  "profiles": [{"profile":"backend","classification":"LOW_RISK","final_step":"agentic"}]
}
On any error, write {"error":"...","classification":"HIGH_RISK"} and exit 1.

Acceptance criteria:

bundle exec rake pr_risk:classify pr=N prints a multi-section coloured report

bundle exec rake pr_risk:classify_json pr=N outputs valid JSON parseable by jq

On any internal error, classify_json exits non-zero with a HIGH_RISK JSON payload

Phase 9 — GitHub Action

The action wires the rake task into your repository: it runs on PR events, executes the classifier in CI, applies a label, posts a summary comment, and (optionally) auto-approves.

Sub-task 9.1 — Workflow scaffold

Prompt:

Create .github/workflows/pr-risk-classifier.yml. It is a multi-job workflow.

Top-level:
name: PR Risk Classifier

on:
  workflow_dispatch:
    inputs:
      branch:
        description: 'Branch name or owner:branch (optional)'
        required: false
        type: string
  pull_request:
    types: [opened, ready_for_review, synchronize, labeled]
    branches:
      - master   # change to your default branch

concurrency:
  group: 'pr-risk-classifier-${{ github.event.pull_request.number || github.event.inputs.branch || github.ref_name }}'
  cancel-in-progress: true

env:
  ANTHROPIC_API_KEY: ${{ secrets.PR_RISK_CLASSIFIER_ANTHROPIC_API_KEY }}
  RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY_DEVELOPMENT }}
  RAILS_ENV: development
Define five jobs (next sub-tasks fill them in): check-classified, cleanup-on-push, remove-request-label, classify, label, auto-approve, notify-low-risk-agentic. (You can spread these across the next four sub-tasks.)

Acceptance criteria: YAML is valid (yamllint or actionlint clean) even if jobs are stubs at this point.

Sub-task 9.2 — `check-classified`, `cleanup-on-push`, `remove-request-label`

Prompt:

In .github/workflows/pr-risk-classifier.yml, add three pre-flight jobs. They share no resources besides the PR number; each is self-contained.

Job: check-classified

Runs first. Resolves the PR (from event payload, or by gh pulls list --head for workflow_dispatch) and exposes outputs the rest of the workflow uses.
check-classified:
  runs-on: ubuntu-latest
  permissions:
    issues: read
    pull-requests: read
  outputs:
    already_classified: ${{ steps.check.outputs.already_classified }}
    pr_number: ${{ steps.check.outputs.pr_number }}
    pr_title: ${{ steps.check.outputs.pr_title }}
    pr_html_url: ${{ steps.check.outputs.pr_html_url }}
    pr_draft: ${{ steps.check.outputs.pr_draft }}
  steps:
    - uses: actions/github-script@v7
      id: check
      with:
        script: |
          // 1. let pr = context.payload.pull_request
          // 2. If absent (workflow_dispatch), resolve from input branch or context.ref via github.rest.pulls.list({ state:'open', head: 'owner:branch' }).
          //    Fail if no open PR found.
          // 3. Set outputs pr_number, pr_title, pr_html_url, pr_draft.
          // 4. Fetch labels via github.rest.issues.listLabelsOnIssue and set already_classified='true' iff any label name === 'Auto-approved'.
Job: cleanup-on-push

Runs only on synchronize (new commit) when the PR was already classified — so any prior auto-approval becomes stale.

Trigger: if: github.event.action == 'synchronize' && needs.check-classified.outputs.already_classified == 'true'

Steps (single actions/github-script@v7):

Remove the Auto-approved label (try/catch — ignore "label not present" errors).

List comments, delete any whose body includes 'PR Risk Classification' or 'Generated by PR Risk Classifier'.

Post a one-time notice comment instructing the user to add label Auto-approve: Request if they want the classifier to run again.

Job: remove-request-label

Runs when someone manually adds the Auto-approve: Request label — strip the label immediately so this re-run can apply it again next time.
if: "github.event.action == 'labeled' && github.event.label.name == 'Auto-approve: Request'"
Steps: single github-script call to github.rest.issues.removeLabel({ name: 'Auto-approve: Request', ... }). Wrap in try/catch.

All three jobs needs: check-classified (the latter two), and use permissions: pull-requests: write for the write jobs.

Acceptance criteria: Push a commit to a previously auto-approved PR — the workflow removes the label and bot comments.

Sub-task 9.3 — `classify` job

Prompt:

Add the classify job to .github/workflows/pr-risk-classifier.yml. This is the heavy lifter: it boots Ruby + Node, installs Claude Code CLI, runs the rake task, and exposes the JSON outputs.
classify:
  runs-on: ubuntu-latest
  timeout-minutes: 15
  needs: check-classified
  if: "github.event.action != 'synchronize' && needs.check-classified.outputs.pr_draft == 'false' && (github.event_name == 'workflow_dispatch' || github.event.label.name == 'Auto-approve: Request' || (needs.check-classified.outputs.already_classified != 'true' && github.run_attempt == 1))"
  outputs:
    classification: ${{ steps.classify.outputs.classification }}
    label: ${{ steps.classify.outputs.label }}
    final_step: ${{ steps.classify.outputs.final_step }}
    reasoning: ${{ steps.classify.outputs.reasoning }}
  steps:
Steps in order:
Checkout the PR head commit. Use actions/checkout@v4 with fetch-depth: 0 for normal PR runs (Step 3 needs full git history for git diff). For workflow_dispatch and the Auto-approve: Request re-trigger, use fetch-depth: 1 (cheaper).

Setup Node (actions/setup-node@v4 with node-version-file: '.nvmrc').

Install Claude Code CLI: npm install -g @anthropic-ai/claude-code.

Setup Ruby (ruby/setup-ruby@v1, your repo's Ruby version, bundler-cache: true). Pass BUNDLE_GITHUB__COM env if you have private gems behind a PAT.

Create dummy .env: shell heredoc writing fake values for every env var your Rails app requires to boot. The classifier only needs Rails to load enough to run the rake task — no real services are hit. Replace this list to match your codebase (Stripe, S3, Redis, DB URL, SECRET_KEY_BASE, sentry DSN, etc.).
Run classifier with id: classify. Set env GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} and GH_ORG_TOKEN: ${{ secrets.GH_PAT_TOKEN_FOR_GH_OPERATIONS }} (PAT for the team-membership check). Then:
set +e
output=$(bundle exec rake pr_risk:classify_json pr=${{ needs.check-classified.outputs.pr_number }} 2>&1)
exit_code=$?
set -e

if [ $exit_code -ne 0 ]; then
  result='{"classification":"HIGH_RISK","error":"classifier_failed"}'
else
  # Take the LAST line that looks like a JSON object on its own line:
  result=$(echo "$output" | awk 'BEGIN{res=""} {if ($0 ~ /^\{.*\}$/) res=$0} END{print res}')
  [ -z "$result" ] && result='{"classification":"HIGH_RISK","error":"no_json_output"}'
fi
if ! echo "$result" | jq -e . >/dev/null 2>&1; then
  result='{"classification":"HIGH_RISK","error":"invalid_json_output"}'
fi

classification=$(echo "$result" | jq -r '.classification // "HIGH_RISK"')
final_step=$(echo "$result"   | jq -r '.final_step   // "unknown"')
reasoning=$(echo "$result"    | jq -r '.reasoning    // "No reasoning provided"')

# Escape backticks / ${ for downstream JS template literals.
reasoning_escaped=$(printf '%s' "$reasoning" | sed 's/\\/\\\\/g; s/`/\\`/g; s/\${/\\${/g')

label=""
[ "$classification" = "LOW_RISK" ] && label="Auto-approved"

echo "classification=$classification" >> $GITHUB_OUTPUT
echo "label=$label"                   >> $GITHUB_OUTPUT
echo "final_step=$final_step"         >> $GITHUB_OUTPUT
{
  echo "reasoning<<EOF"
  echo "$reasoning_escaped"
  echo "EOF"
} >> $GITHUB_OUTPUT
The fail-safe behaviour is critical: every failure path returns HIGH_RISK so a broken classifier can never auto-approve.

Acceptance criteria:

On a clean low-risk PR the job sets classification=LOW_RISK, label=Auto-approved

When the rake task crashes the job still completes with classification=HIGH_RISK

Multi-line reasoning round-trips through $GITHUB_OUTPUT without breaking later jobs

Sub-task 9.4 — `label` & `auto-approve` jobs

Prompt:

Add two jobs to .github/workflows/pr-risk-classifier.yml that consume classify's outputs.

Job: `label`

Runs only when classification is LOW_RISK (HIGH_RISK PRs are silently left alone — humans review them anyway). Posts (or updates) a sticky bot comment with the classification, the final step, and the reasoning.

label:
  runs-on: ubuntu-latest
  needs: [classify, check-classified]
  if: needs.classify.outputs.classification == 'LOW_RISK'
  permissions:
    pull-requests: write
  steps:
    - name: Post classification summary
      uses: actions/github-script@v7
      with:
        script: |
          const classification = '${{ needs.classify.outputs.classification }}';
          const label = '${{ needs.classify.outputs.label }}';
          const finalStep = '${{ needs.classify.outputs.final_step }}';
          const reasoning = `${{ needs.classify.outputs.reasoning }}`;
          const issueNumber = Number('${{ needs.check-classified.outputs.pr_number }}');

          const emoji = classification === 'LOW_RISK' ? ':white_check_mark:' : ':warning:';
          const body = `## PR Risk Classification ${emoji}

          **Classification:** ${classification}
          **Label applied:** ${label}
          **Final step:** ${finalStep}

          ### Reasoning

          ${reasoning}

          ---
          <sub>Generated by PR Risk Classifier</sub>`;

          // Find existing bot comment and update it instead of spamming new ones.
          const { data: comments } = await github.rest.issues.listComments({
            owner: context.repo.owner, repo: context.repo.repo, issue_number: issueNumber
          });
          const botComment = comments.find(c => c.user.type === 'Bot' && c.body.includes('PR Risk Classification'));
          if (botComment) {
            await github.rest.issues.updateComment({ owner: context.repo.owner, repo: context.repo.repo, comment_id: botComment.id, body });
          } else {
            await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: issueNumber, body });
          }

Job: `auto-approve`

Implements the 90/10 sampling: 90% of low-risk PRs get auto-approved, 10% are deliberately left for a human to validate the classifier.

auto-approve:
  runs-on: ubuntu-latest
  needs: [classify, label, check-classified]
  if: needs.classify.outputs.classification == 'LOW_RISK' && vars.AUTO_APPROVE_ENABLED == 'true'
  outputs:
    approved: ${{ steps.should_approve.outputs.approve }}
  permissions:
    pull-requests: write
  steps:
    - id: should_approve
      run: |
        random=$((RANDOM % 100 + 1))
        if [ $random -le 90 ]; then
          echo "approve=true"  >> $GITHUB_OUTPUT
        else
          echo "approve=false" >> $GITHUB_OUTPUT
        fi

    - name: Auto-approve PR
      if: steps.should_approve.outputs.approve == 'true'
      env:
        GH_TOKEN: ${{ secrets.GH_PAT_TOKEN_FOR_PR_CLASSIFIER }}    # PAT, not GITHUB_TOKEN — branch protection blocks self-approval
      run: |
        gh pr review ${{ needs.check-classified.outputs.pr_number }} \
          --repo ${{ github.repository }} \
          --approve \
          --body "✅ **Auto-approved** by PR Risk Classifier

        This PR was classified as **LOW_RISK** and has been automatically approved.

        _Note: 10% of low-risk PRs are randomly selected for manual review._"

    - name: Add Auto-approved label
      if: steps.should_approve.outputs.approve == 'true'
      env:
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: gh pr edit ${{ needs.check-classified.outputs.pr_number }} --repo ${{ github.repository }} --add-label "Auto-approved"

    - name: Comment on manual review hold
      if: steps.should_approve.outputs.approve == 'false'
      env:
        GH_TOKEN: ${{ secrets.GH_PAT_TOKEN_FOR_PR_CLASSIFIER }}
      run: |
        gh pr comment ${{ needs.check-classified.outputs.pr_number }} \
          --repo ${{ github.repository }} \
          --body "🎲 **Manual review requested**

        This PR was classified as **LOW_RISK** but was randomly selected (10% sample) for manual review.

        Please review and approve manually."

Why a PAT for gh pr review --approve? GitHub Actions cannot satisfy required-review branch protection with the default GITHUB_TOKEN. Generate a fine-grained PAT with pull_requests: write scope, store as GH_PAT_TOKEN_FOR_PR_CLASSIFIER. The PAT-owning user must not be the PR author (you can't approve your own PR).

Acceptance criteria:

vars.AUTO_APPROVE_ENABLED == 'false' skips the entire job
Approval comment posts on ~90% of LOW_RISK PRs, manual-review comment on ~10%
The Auto-approved label is applied so future runs short-circuit via check-classified

Sub-task 9.5 — `notify-low-risk-agentic` job (optional)

Prompt:

Add a final job to .github/workflows/pr-risk-classifier.yml that posts a Slack message whenever the agentic step (Step 3) is the one that classified the PR as LOW_RISK and the PR was auto-approved. This is the highest-value monitoring signal: agentic LOW_RISK is the only path where the LLM-deep dive overrode an inconclusive static result, so it's where false positives would first appear.
notify-low-risk-agentic:
  runs-on: ubuntu-latest
  needs: [classify, check-classified, auto-approve]
  if: |
    needs.classify.outputs.classification == 'LOW_RISK' &&
    needs.classify.outputs.final_step == 'agentic' &&
    needs.auto-approve.outputs.approved == 'true'
  steps:
    - uses: slackapi/slack-github-action@v2.1.1
      with:
        errors: true
        method: chat.postMessage
        token: ${{ secrets.SLACK_TOKEN_PR_RISK_CLASSIFIER }}
        payload: |
          "channel": "<your-channel-id>"
          "text": ":white_check_mark:  *PR Auto-approved*: <${{ needs.check-classified.outputs.pr_html_url }}|#${{ needs.check-classified.outputs.pr_number }} - ${{ needs.check-classified.outputs.pr_title }}>"
Acceptance criteria: When step3 is the deciding step, a Slack message appears in your monitoring channel.

Phase 10 — Required GitHub configuration

These aren't code, but the action won't work without them. Have your AI agent (or you) verify each:

Sub-task 10.1 — Secrets, vars, labels, branch protection

Prompt:

In the GitHub repository where this workflow lives, configure these settings under Settings → Secrets and variables → Actions and Settings → Branches:

Secrets (Settings → Secrets):

Name Used for

PR_RISK_CLASSIFIER_ANTHROPIC_API_KEY Anthropic API key (Step 2)

RAILS_MASTER_KEY_DEVELOPMENT Allows Rails to decrypt credentials in CI

GH_PAT_TOKEN_FOR_GH_OPERATIONS PAT used by the rake task for team_member? lookups

GH_PAT_TOKEN_FOR_PR_CLASSIFIER PAT used to approve PRs (must NOT belong to a user who opens PRs in this repo)

SLACK_TOKEN_PR_RISK_CLASSIFIER Bot token if you wire up Slack notifications

Variables (Settings → Variables):

Name Value

AUTO_APPROVE_ENABLED 'true' to enable auto-approve, 'false' to dry-run

Labels (Settings → Labels): create

Auto-approved — added by the auto-approve job

Auto-approve: Request — manually applied to re-trigger the classifier on an already-classified PR

Branch protection on master:

"Require pull request reviews before merging" — at least 1 review

"Allow specified actors to bypass required pull requests" — leave OFF

The PAT user must have write access; otherwise gh pr review --approve will 403.

Acceptance criteria:

Opening a tiny non-critical PR (touch _spec.rb only) results in: classification = LOW_RISK from Step 1, comment posted, label applied, PR approved (~90% of the time).

Opening a PR that touches db/migrate/ results in: classification = HIGH_RISK from Step 1, no comment, no label, no approval.

Name	Used for
`PR_RISK_CLASSIFIER_ANTHROPIC_API_KEY`	Anthropic API key (Step 2)
`RAILS_MASTER_KEY_DEVELOPMENT`	Allows Rails to decrypt credentials in CI
`GH_PAT_TOKEN_FOR_GH_OPERATIONS`	PAT used by the rake task for `team_member?` lookups
`GH_PAT_TOKEN_FOR_PR_CLASSIFIER`	PAT used to approve PRs (must NOT belong to a user who opens PRs in this repo)
`SLACK_TOKEN_PR_RISK_CLASSIFIER`	Bot token if you wire up Slack notifications

Name	Value
`AUTO_APPROVE_ENABLED`	`'true'` to enable auto-approve, `'false'` to dry-run

Phase 11 — Documentation (optional but recommended)

Sub-task 11.1 — README

Prompt:

Create lib/pr_risk_classifier/docs/README.md that documents the classifier for future maintainers. Cover:

Overview of the 3-step pipeline.

Pipeline diagram (ASCII boxes for Step 1 / 2 / 3).

Step Escalation table explaining when each step early-exits.

Per-step description — what each step checks, what its inputs are, what triggers it, what triggers the next step.

Configuration layout — the YAML structure with a small example.

Config selection — backend vs frontend vs combined runs based on file extensions.

Data flow diagram showing GithubClient → PrContext → analysers.

Key files list with descriptions.

Usage: bundle exec rake pr_risk:classify pr=12345 [steps=1,2,3].

Notes & gotchas: ANTHROPIC_API_KEY for Step 2, claude CLI for Step 3, gh for everything else.

Then create lib/pr_risk_classifier/docs/GITHUB_WORKFLOW.md that documents the workflow:

Triggers (opened, ready_for_review, synchronize, labeled).

Concurrency control (cancel in-progress runs).

Job-by-job walkthrough.

The 90/10 sampling rationale.

Why auto-approve uses a PAT instead of GITHUB_TOKEN.

Failure modes (every failure path → HIGH_RISK).

How to disable auto-approve without removing the workflow.

Acceptance criteria: Both docs render cleanly in GitHub's markdown viewer.

Final checklist

Once every phase is complete, verify the system end-to-end:

bundle exec rake pr_risk:classify pr=<a small PR> runs locally and prints a multi-section coloured report.
bundle exec rake pr_risk:classify_json pr=<small PR> prints a single-line JSON object that jq can parse.
bundle exec rake pr_risk:classify pr=<small PR> steps=1 exits without ever calling Anthropic or Claude Code.
Open a test PR that touches only _spec.rb files → workflow labels Auto-approved and posts a comment.
Open a test PR that adds a file under db/migrate/ → workflow records HIGH_RISK and does not auto-approve.
Push a new commit to a previously auto-approved PR → cleanup-on-push removes the label and bot comments and posts the re-trigger notice.
Manually add the Auto-approve: Request label → remove-request-label strips it and classify runs again.
Confirm the 90/10 sampling — ~10% of LOW_RISK PRs receive a "manual review requested" comment instead of approval.

Appendix — Test PRs to use during development

To stress-test the pipeline cheaply, open these synthetic PRs against a scratch branch:

PR shape	Expected classification	Expected step
Add a single `_spec.rb` file	LOW_RISK	static (early-exit on `safe_files_only`)
Add a 1500-line refactor across `app/services/`	HIGH_RISK	static (`diff_too_large`)
Touch `db/migrate/...`	HIGH_RISK	static (`sensitive_file_patterns`)
Add `payment` keyword usage in a service	HIGH_RISK	static (`dangerous_keywords`)
Small frontend tweak in `app/javascript/components/elements/Button.tsx`	HIGH_RISK	static or agentic (broad fan-out)
Add a new isolated service class with tests, < 5 callers	LOW_RISK	agentic
Tweak a comment in `app/jobs/foo_job.rb` (no description)	HIGH_RISK	static (`missing_description`)

Each row is a useful regression test before rolling out to production PRs.

adrianbooth-eng/PR_RISK_CLASSIFIER_AGENT_INPUT.md

PR Risk Classifier — Build-It-Yourself Prompt Pack

How to use this document

System Overview

Pipeline

Escalation rules

File layout being built

Phase 1 — Foundations

Sub-task 1.1 — Top-level autoloader and module skeleton

Sub-task 1.2 — Config YAML files

Phase 2 — Configuration classes

Sub-task 2.1 — StaticAnalysisConfig

Sub-task 2.2 — Instructions

Sub-task 2.3 — Configuration loader

Phase 3 — External clients

Sub-task 3.1 — GithubClient

Sub-task 3.2 — LlmClient

Phase 4 — Pipeline scaffolding

Sub-task 4.1 — PrContext

Sub-task 4.2 — PipelineResult

Sub-task 4.3 — RiskClassificationPipeline

Sub-task 4.4 — Runner facade

Phase 5 — Step 1: Static Analyser

Sub-task 5.1 — StaticAnalyser

Result struct

Class skeleton

Detailed rules

Phase 6 — Step 2: Semantic Analyser (LLM)

Sub-task 6.1 — BackendSemanticPromptBuilder

Base system prompt (verbatim — adjust app name)

Sub-task 6.2 — FrontendSemanticPromptBuilder

Sub-task 6.3 — SemanticAnalyser

Result struct

Class

Phase 7 — Step 3: Agentic Analyser (Claude Code CLI)

Sub-task 7.1 — AgenticPromptSections mixin

Sub-task 7.2 — AgenticPromptBuilder base class

Sub-task 7.3 — BackendAgenticPromptBuilder

Sub-task 7.4 — FrontendAgenticPromptBuilder

Sub-task 7.5 — AgenticResult

Sub-task 7.6 — StreamingOutputHandler

Sub-task 7.7 — AgenticAnalyser

Phase 8 — Glue: TaskRunner & Rake tasks

Sub-task 8.1 — TaskRunner

Sub-task 8.2 — Rake tasks

Shared helpers

Task 1: pr_risk:classify

Task 2: pr_risk:classify_json

Phase 9 — GitHub Action

Sub-task 9.1 — Workflow scaffold

Sub-task 9.2 — check-classified, cleanup-on-push, remove-request-label

Job: check-classified

Job: cleanup-on-push

Job: remove-request-label

Sub-task 9.3 — classify job

Sub-task 9.4 — label & auto-approve jobs

Job: label

Job: auto-approve

Sub-task 9.5 — notify-low-risk-agentic job (optional)

Phase 10 — Required GitHub configuration

Sub-task 10.1 — Secrets, vars, labels, branch protection

Phase 11 — Documentation (optional but recommended)

Sub-task 11.1 — README

Final checklist

Appendix — Test PRs to use during development

Sub-task 2.1 — `StaticAnalysisConfig`

Sub-task 2.2 — `Instructions`

Sub-task 2.3 — `Configuration` loader

Sub-task 3.1 — `GithubClient`

Sub-task 3.2 — `LlmClient`

Sub-task 4.1 — `PrContext`

Sub-task 4.2 — `PipelineResult`

Sub-task 4.3 — `RiskClassificationPipeline`

Sub-task 4.4 — `Runner` facade

Sub-task 5.1 — `StaticAnalyser`

Sub-task 6.1 — `BackendSemanticPromptBuilder`

Sub-task 6.2 — `FrontendSemanticPromptBuilder`

Sub-task 6.3 — `SemanticAnalyser`

Sub-task 7.1 — `AgenticPromptSections` mixin

Sub-task 7.2 — `AgenticPromptBuilder` base class

Sub-task 7.3 — `BackendAgenticPromptBuilder`

Sub-task 7.4 — `FrontendAgenticPromptBuilder`

Sub-task 7.5 — `AgenticResult`

Sub-task 7.6 — `StreamingOutputHandler`

Sub-task 7.7 — `AgenticAnalyser`

Sub-task 8.1 — `TaskRunner`

Task 1: `pr_risk:classify`

Task 2: `pr_risk:classify_json`

Sub-task 9.2 — `check-classified`, `cleanup-on-push`, `remove-request-label`

Job: `check-classified`

Job: `cleanup-on-push`

Job: `remove-request-label`

Sub-task 9.3 — `classify` job

Sub-task 9.4 — `label` & `auto-approve` jobs

Job: `label`

Job: `auto-approve`

Sub-task 9.5 — `notify-low-risk-agentic` job (optional)