Flaky Test Guidance

Audience: coding agents (and humans) writing or editing tests in this repo.

This document is distilled from every "flaky"/"flake" test-fix PR merged in 2026 (33 PRs). Each rule below maps to a real flake we shipped, found in CI, and had to fix again later. The goal is simple: write the test correctly the first time so it never lands on this list.

Why flakes happen here

Our CI runs the full RSpec suite in parallel workers, in a randomized order seeded per run (the seed is derived from the CI run id). A test is flaky when its result depends on anything that isn't fully controlled by the test itself:

the order other tests ran in (and the state they left behind),
the order the database returns rows in (Postgres gives no order without ORDER BY),
the wall clock or calendar date,
a random value (Faker, RNG, hash-based feature-flag rollout),
an auto-assigned id / sequence position,
which class got autoloaded first under lazy loading.

Every rule below closes one of these holes.

⚡ Pre-flight checklist (run through this on every test you write or touch)

Asserting on a collection? → Use match_array / contain_exactly, never eq([...]), unless the production code has an explicit ORDER BY / .order(...). (§1)
Picking one record out of many? → Don't use .first/.second/[0] on an unordered relation, and don't find_by(flag: true) when fixtures create several matches. (§1, §4)
Touching global/shared state (Current, class-level @vars, feature-flag stores, Warden, caches)? → Reset it in a before/after hook for the whole group; never assume a prior test set it up. (§2)
Using a date or time? → Express it relative to Date.current/Time.current; create time-sensitive records inside the travel_to block; never hardcode Date.new(2021, 2, 8). (§3)
Building fixtures? → Don't create a duplicate of a record a factory/callback already builds; override default "primary" flags on secondary records; pass explicit associations. (§4)
Tempted to write id: 123 or parent_id: 101? → Don't. Let ids auto-assign and reference .id. (§5)
Relying on a random value (Faker city, RNG) to be unique or to produce a specific outcome? → Pin the values that matter; never assert exact statistical distributions. (§6)
Code path gated by a feature flag? → Stub the flag to a deterministic value. (§7)
Inside a nested namespace that shadows a top-level one (e.g. Orders, Concerns)? → Fully-qualify constants with a leading ::. (§8)
API/GraphQL/controller spec? → Assert the happy path completed (errors nil, expected redirect/status) before asserting on the payload or a side effect. (§9)

1. Unordered collections — the #1 cause of flakes

This single category accounts for ~⅓ of all flaky-test fixes. Postgres returns rows in no guaranteed order unless the query has an explicit ORDER BY. ActiveRecord methods like .ids, .pluck, has_one/LIMIT 1, and anything with .unscope(:order) inherit that non-determinism. Locally the DB usually returns insertion order; CI, a different seed, or a different worker returns something else.

Rule 1a — Match collections order-independently

Use match_array (alias contain_exactly) instead of eq on an array literal.

# ❌ flaky — order of Box.ids is undefined
expect(result).to eq([boxes.first.id, boxes.last.id])
expect(pack_file.shipments).to eq(boxes)                 # PackFile#shipments calls unscope(:order)
expect(service).to have_received(:call).with([a.id, b.id])

# ✅ deterministic
expect(result).to match_array([boxes.first.id, boxes.last.id])
expect(pack_file.shipments).to match_array(boxes)
expect(service).to have_received(:call).with(match_array([a.id, b.id]))

PRs: #29501, #30383, #30880, #31083, #31102.

Rule 1b — Match nested collections at every level

contain_exactly only ignores order at the level you apply it. Nest it.

# ❌ inner arrays still order-sensitive
expect(grouped).to contain_exactly([dpd_a.id, dpd_b.id], [yodel.id])

# ✅
expect(grouped).to contain_exactly(contain_exactly(dpd_a.id, dpd_b.id), contain_exactly(yodel.id))

PR: #29501.

Rule 1c — Don't `eq` a string built by joining DB-ordered values

Split it back into a collection and match unordered.

# ❌ controller joins emails from an unordered query
let(:emails) { records.map { |r| r.user.email }.join(', ') }
expect(assigns(:results)).to eq(emails)

# ✅
let(:emails) { records.map { |r| r.user.email } }            # plain array
expect(assigns(:results).split(', ')).to match_array(emails)

PRs: #30908, #30958.

Rule 1d — Never identify a record by position into an unordered set

.first / .second / [0] over an unordered relation grabs a random row. Navigate through a known, stable association instead.

# ❌ flat_map has no stable order; .first/.second are a coin flip
parts = subscription.applicable_discount_parts.flat_map(&:discounts)
expect(parts.first.value).to eq(...)

# ✅ reach the record through the box you know it belongs to
paid_box   = subscription.boxes.first
unpaid_box = subscription.boxes.second
expect(paid_box.reload.discounts.first&.value).to eq(...)

PR: #30777.

Rule 1e — `detect`/`find` must match on a fully discriminating key

If setup creates several records of the same type, matching on type alone returns whichever the queue/DB surfaced first.

# ❌ two centres enqueue this job type — returns a random one
jobs.detect { |dj| dj.payload_object.object.instance_of?(CacheFulfilmentLimitCounts) }

# ✅ pin to the specific trigger too
jobs.detect do |dj|
  dj.payload_object.object.instance_of?(CacheFulfilmentLimitCounts) &&
    dj.payload_object.object.fulfilment_trigger == afternoon_centre_a_trigger
end

PR: #31000.

Rule 1f — If order is part of the contract, enforce it in production code

A test that needs a specific order is telling you the production code should guarantee that order. Fix the code (add ORDER BY, or re-sort in Ruby to match the requested ids) and add a regression test — don't paper over it in the spec.

# production fix: re-emit results in the caller's requested order
records_by_id = QUERY_BY_TYPE[order_type].call(order_ids).index_by(&:id)
order_ids.filter_map { |order_id| records_by_id[order_id] }

PR: #31490.

Heuristic: before writing eq on anything array-shaped, ask "does the code that produced this have an ORDER BY?" If you can't point to one, use match_array.

2. Leaky global / shared state between examples

Anything stored outside a single example — class-level @variables, request-scoped Current, in-memory caches, feature-flag adapter stores, Warden auth state — survives into the next test in the same worker. With randomized ordering, the polluting test and the victim test pair up unpredictably.

Rule 2a — Reset shared stores in a hook, suite-wide if needed

Class-level mutable state must be cleared in a before/after, not ad-hoc inside one example.

# spec/rails_helper.rb — reset the in-memory feature-flag store before every example
config.before do
  ButternutBox::FeatureFlag::Adapters::InMemoryAdapter.instance_variable_set(:@store, nil)
end

# request-scoped cache — reset for any group that exercises a path writing to Current
after { Current.reset }

# auth state — stop Warden leaking a signed-in user into the next example
config.after { Warden.test_reset! }

PRs: #30872 (InMemoryAdapter store), #30918 (Current.reset), #30983 (Warden.test_reset!).

Rule 2b — Reset memoized state for the whole group, not inside one example

If a reset lives in the body of one it, the other example in the group runs with stale state when it executes first.

# ❌ reset buried in one example
it 'does X' do
  described_class.instance_variable_set(:@launchdarkly, nil)
  ...
end

# ✅ reset in the shared before, covering both examples regardless of order
before { described_class.instance_variable_set(:@launchdarkly, nil) }

PR: #30899.

Rule 2c — Never depend on state another test initialized

If your example needs a collaborator, stub it so the example is self-contained — don't reach for a global that only gets populated as a side effect of some other test running first.

# ❌ relies on Rails.configuration.ld_client, set only by a prior boot test
allow(described_class.client).to receive(:close)

# ✅ provide the double the example needs
let(:ld_client) { instance_double(LaunchDarkly::LDClient) }
before { allow(described_class).to receive(:client).and_return(ld_client) }

PR: #30829.

Rule 2d — Beware shared default hashes

Hash.new({}) shares one default object across all keys — writes bleed everywhere. Use the block form.

# ❌ every key points at the same hash
@store = Hash.new({})

# ✅ each key gets its own
@store = Hash.new { |h, k| h[k] = {} }

PR: #30983.

3. Time & date dependence

Hardcoded calendar dates "age" — the relationship between a fixed date and "today" drifts as real time passes, eventually crossing a cutoff/day-of-week boundary the test relied on. Records created outside a travel_to block are evaluated against the real clock.

Rule 3a — Express dates relative to now, never as absolute literals

# ❌ brittle — these relationships rot over time
travel_to(Date.new(2021, 2, 8))
delivery_date = Date.new(2021, 2, 11)

# ✅ encode the *relationship*, anchored to the present
delivery_date = Date.current + 3.days
travel_to(delivery_date - 3.days) do
  last_box.update!(delivery_date: delivery_date - 2.days)
end

PRs: #29755, #29818.

Rule 3b — Create time-sensitive records inside the `travel_to` block

Dates on records are evaluated when the record is built. Build them under frozen time.

# ✅
travel_to(Date.current - 3.months) do
  create(:box, delivery_date: (Date.current + 3.days).iso8601)
end

And keep factory anchor dates relative too — e.g. a price's effective_from { 5.years.ago }, not Date.new(2000, 1, 1), so it always sits on the right side of effective_from <= delivery_date. PRs: #29755, #29818.

4. Factory & data-setup ambiguity

Flakes here come from creating more than one record that a lookup can't distinguish between, or from factory defaults that quietly create competing records.

Rule 4a — Don't create a duplicate of something a factory/callback already builds

# ❌ Box auto-builds an invoice via after_initialize; this makes a 2nd one,
#    and has_one's LIMIT 1 (no ORDER BY) returns either at random
let(:old_invoice) { create(:invoice, box:, due_date: 1.day.ago) }

# ✅ reuse and mutate the one the model already built
let(:old_invoice) { box.invoice.tap { |i| i.update!(due_date: 1.day.ago) } }

Likewise, don't create(:delivery_area, name: 'Poland', ...) when the address factory already made a Poland area and the code looks it up by find_by!(name: 'Poland') — reference the existing one: let(:parent_delivery_area) { user.address.delivery_area }. PRs: #31499 (has_one invoice), #31185 (duplicate delivery area).

Rule 4b — Override default "primary/main" flags on secondary records

If a factory defaults a flag to true (e.g. main_trigger { true }) and the code does find_by(main_trigger: true, ...), every extra record you build becomes an ambiguous match.

# ❌ both are main_trigger: true → find_by returns a random one
let(:second_trigger) { create(:fulfilment_trigger, shipment_type:, fulfilment_centre:) }

# ✅ leave exactly one record with the flag set
let(:second_trigger) { create(:fulfilment_trigger, shipment_type:, fulfilment_centre:, main_trigger: false) }

PRs: #31260, #31403.

Rule 4c — Factories must not blindly create global / uniquely-named records

A factory that always creates a uniquely-named global (a Setting, a config row) spawns a competing duplicate if one already exists. Guard it.

# ✅ idempotent — only create if absent
SETTINGS.each do |name, evaluator_method|
  next if Setting.exists?(name: name)
  create(:setting, name: name, ...)
end
# or: Setting.find_or_create_by(name: name) { ... }

PR: #31228 (and the test-side mitigation #31128: pick a distinct country so a leaked GB setting can't satisfy the lookup).

Rule 4d — Pass explicit associations when factory defaults trigger callbacks

If a factory's default association fires a callback that depends on shared/global records, wire up the associations explicitly to the records your context already created.

# ❌ default address → set_delivery_area callback → NoRootDeliveryAreaError when none exists
let(:user) { create(:client_user) }

# ✅
let(:user_address) { create(:british_address, shipping_country: gb_shipping_country, delivery_area: gb_delivery_area) }
let(:user) { create(:client_user, address: user_address) }

PR: #29589.

5. Hardcoded ids & sequence collisions

Postgres sequences are non-transactional and climb across the whole suite. An explicit-id insert (id: 667) does not advance the sequence, so when nextval eventually reaches that number, the auto-assigned insert collides → PG::UniqueViolation. Hardcoded ids in factories can also produce absurd states (a closure_tree node whose parent_id equals its own future id → infinite recursion → SystemStackError).

Rule 5 — Never hardcode primary keys or foreign keys to literals

# ❌
let(:gb_shipping_country) { create(:shipping_country, :gb, id: 667) }
factory(:reason) { parent_id { 101 } }
let(:delivery_area_id) { 666 }

# ✅ let ids auto-assign; reference the created object
let(:gb_shipping_country) { create(:shipping_country, :gb) }
factory(:reason) { }                          # roots have no parent
let(:delivery_area_id) { gb_delivery_area.id }

PRs: #31540 (id 667 sequence collision), #30975 (parent_id 101 → recursion).

6. Randomness, Faker & probabilistic assertions

Rule 6a — Pin every column of a uniqueness constraint; don't let Faker be the differentiator

If two rows share a uniqueness tuple and a random value is the only thing keeping them apart, a seed that repeats the value collides.

# ❌ same delivery_area + postcode, only random Faker city differs → "City has already been taken"
build(:address, delivery_area:)
build(:address, delivery_area:)

# ✅ set the distinguishing column explicitly
build(:polish_address, delivery_area:, postcode: '00-001')
build(:polish_address, delivery_area:, postcode: '30-001')

PR: #31348.

Rule 6b — Don't assert exact statistical outcomes of randomized logic

Sampling-based assertions (e.g. "high-weight count > medium > low" over 100 draws) will eventually violate the inequality. Assert the mechanism returns a valid result, or stub the RNG.

# ❌ flakes on variance
expect(high_count).to be > medium_count
# ✅ prove correctness without statistics
expect(DiscountCode.default_by(...)).to be_in(configured_codes)

PR: #29588.

7. Feature flags that change control flow

Hash-based percentage rollout buckets a given test user the same way every time for a given flag+user, but the bucket is effectively arbitrary — and if it lands on a before_action redirect, your assertion silently never runs. Always stub flags that gate behaviour.

# ✅ pin the flag so the code path is deterministic
allow(ButternutBox::FeatureFlag::Client.default).to receive(:variant)
  .with(flag: 'should_use_new_ambassador_portal', fallback: 'control', user: anything)
  .and_return('control')

PR: #30919.

8. Namespace & load-order traps (lazy autoloading)

In the test env eager_load is off, so constants resolve when first referenced. Two failure modes:

Rule 8a — Fully-qualify constants inside a namespace that shadows a top-level one

If feature code lives in Admin::OrderIssuesV2::Orders and references Orders::Finder, the lookup walks the ancestor chain and — depending on what's been autoloaded yet — can fall through to the top-level ::Orders, raising NameError only on certain suite orderings.

# ❌ relative — resolves differently depending on load order
Orders::Finder.new(...)

# ✅ absolute — never consults the ancestor chain
::Admin::OrderIssuesV2::Orders::Finder.new(...)

A custom cop Butternut/AmbiguousNamespaceReference enforces this for the configured ambiguous namespaces (Orders, Concerns). PR: #31530.

Rule 8b — Force-load a class before stubbing global state it reads at load time

If a class reads Rails.env / ENV / config at class-load time, and your test stubs that state, the value baked in depends on whether the class loaded before or after your stub.

before do
  ButternutBoxSchema.name              # force the schema to load *before* the stub
  allow(Rails.env).to receive_messages(test?: false, development?: false)
end

Relatedly, keep config.cache_classes = true in config/environments/test.rb so stubs on autoloaded constants aren't discarded when Zeitwerk reloads mid-request. PRs: #30928 (schema load order), #30983 (cache_classes).

9. Make failures loud — assert the happy path before the payload

Several "flakes" were really intermittent errors hidden behind an unhelpful assertion. When a request silently errors, asserting on a missing field gives a confusing message and looks like a flake. Assert success first so the real cause surfaces.

Rule 9a — In GraphQL/API specs, assert `errors` is nil before asserting on `data`

expect(query_result['errors']).to be_nil                 # fails loudly with the real message
expect(result_of_query['redirectUrl']).to include('users/auto_sign_in?...')

PRs: #31136, #31499.

Rule 9b — Assert the action completed before asserting its side effect

A swallowed rescue StandardError can skip the side effect you're testing. Confirm the happy path (redirect/status) first, under :aggregate_failures, so the failure points at the real cause.

it 'deletes the signup link', :aggregate_failures do
  expect(response).to redirect_to(ambassadors_dashboard_summary_path)   # did we even get here?
  expect { signup_link.reload }.to raise_error(ActiveRecord::RecordNotFound)
end

PR: #31096.

Red-flag patterns to grep for in a diff

When reviewing or writing a spec, treat any of these as a prompt to apply the matching rule:

Pattern in the spec	Likely problem	Rule
`.to eq([` / `.to eq(boxes)` on a relation/array	unordered collection	§1a
`.with([` on a mock expecting an id array	unordered argument	§1a
`.join(', ')` then `eq`	order-dependent string	§1c
`.first` / `.second` / `[0]` on a query result	positional pick from unordered set	§1d
`find_by(...: true)` with multiple fixtures of that kind	ambiguous match	§1e / §4b
`Date.new(` / `Time.new(` / `Time.parse('2021-...')`	hardcoded date	§3a
`create(:box, ...)` outside a `travel_to` in a time-sensitive spec	clock drift	§3b
`create(:invoice, box:)` where the model auto-builds one	duplicate `has_one`	§4a
a second `create(:<thing>, name: '…')` for a uniquely-named record	duplicate global	§4a / §4c
`id:` / `parent_id:` set to an integer literal	sequence collision	§5
only Faker distinguishes two rows under a unique index	random collision	§6a
`be >` / count comparisons over sampled random draws	probabilistic	§6b
a `before_action`/redirect gated by an unstubbed flag	rollout decides behaviour	§7
relative constant ref inside a shadowing namespace (`Orders::`, `Concerns::`)	load-order `NameError`	§8a
stubbing `Rails.env`/`ENV`/config a class reads at load	load-order baking	§8b
GraphQL/controller spec asserting only the payload	hidden error masked as flake	§9

Appendix — the PRs this guidance is built from

All merged in 2026; titles contain "flaky"/"flake".

Family	PRs
§1 Unordered collections	#29501, #30383, #30777, #30880, #30908, #30958, #31000, #31083, #31102, #31490
§2 Leaky shared state	#30829, #30872, #30899, #30918, #30983
§3 Time/date	#29755, #29818
§4 Factory/data-setup	#29589, #31128, #31185, #31228, #31260, #31403, #31499
§5 Hardcoded ids	#30975, #31540
§6 Randomness/Faker	#29588, #31348
§7 Feature flags	#30919
§8 Namespace/load-order	#30928, #30983, #31530
§9 Loud failures (diagnostics)	#31096, #31136

Some PRs appear under more than one family (e.g. #30983 fixed both leaky state and a load-order config issue; #30975 combined a hardcoded id with a missing :disable_transactional_fixtures tag).

adrianbooth-eng/flaky_test_guidance.md