URL validation divergences: Claude's findings

FINDINGS.md

Differential testing: isValidURLString vs. the URL parser

Verdict

The alignment work on this branch is solid for the entire ASCII grammar — which is where the #924 changes live (scheme-relative, path-absolute, file URLs, opaque-path bases). Across ~1.6 billion differential evaluations, including an exhaustive enumeration of every ASCII string up to length 6 over a 19-character structurally-significant alphabet against every base, there is zero divergence.

There is exactly one remaining family of divergence, and it is entirely non-ASCII: a host that UTS-46 (IDNA) mapping turns into a complete, canonical IPv4 address. The parser accepts these with no validation error, but isValidURLString correctly rejects them. This is not a bug the branch introduced, and not fixable in the string validator — it is an inherent gap between the lenient host parser and the strict "valid URL string" writing grammar. See the root-cause section.

The invariant under test

The branch aims to make these two notions coincide for every (input, base):

isValidURLString(input, { baseURL })  ⟺  parseURLWithValidationErrors reports
                                          a successful parse with zero validation errors

The fuzzer flags any pair where the left side and the right side disagree, in either direction, plus any exception ("crash") from either function.

Methodology

Three complementary techniques (all under fuzz/):

1. Random differential fuzzer (fuzz.js, multi-process): dictionary-driven token soup, valid-by-construction generators (special/file/opaque/relative URLs, domains, IPv4, IPv6, ports, paths), corpus mutation with coverage-style feedback on novel validation-error signatures, and a broad Unicode generator (unicodeLab) covering every UTS-46 status class plus astral planes, bidi, combining marks, noncharacters and lone surrogates. Findings are auto-minimized and de-duplicated into buckets.

2. Exhaustive ASCII enumeration (exhaustive.js, multi-process): every string up to length N over a chosen alphabet of structurally-significant ASCII characters, evaluated against all base URLs. This upgrades "we didn't find a bug" to "no divergence exists in this space." The alphabet a 1 0 x n - _ : / \ . ? # @ [ ] | % and space (19 chars) captures every parser state transition plus the host-classification code paths (IPv4 decimal/hex/octal via 0/x, the xn-- punycode prefix, hyphen/underscore STD3 domain checks).

3. Targeted IDNA probe (analyze-idna.js): plants representatives of each UTS-46 behavior class (mapped-to-digit, mapped-to-dot, ignored, non-ASCII digits, bidi, combining, case-folded, disallowed, astral) into host / path / query / fragment / opaque positions to enumerate the divergence mechanisms and confirm none exist outside hosts.

Results

Space	Coverage	Divergences
ASCII, length ≤ 5, 15-char alphabet, all bases	20,340,400	0
ASCII, length ≤ 6, 15-char alphabet, all bases	305,106,025	0
ASCII, length ≤ 5, 19-char alphabet, all bases	65,341,500	0
ASCII, length ≤ 6, 19-char alphabet, all bases	1,241,488,525	0
Random ASCII-only (dictionary + generators)	~10,000,000	0
Random full-Unicode (dictionary + generators + unicodeLab)	434,844,000	only the family below (1,494 records, all IDNA→IPv4)

Total across all passes: > 2 billion differential evaluations.

Every divergence the random full-Unicode fuzzer finds is the single family below; a triage classifier (triage.js) re-parses each finding and confirms it is a non-ASCII input whose host parses to a numeric (IPv4) value. No crashes, no valid-but-parser-flags (the other direction), no silent-failure, no ASCII findings.

The one divergence family: IDNA maps a host to a complete IPv4 address

Symptom. For a special-scheme URL whose host, after UTS-46 mapping, becomes a complete valid IPv4 address written with ASCII digits and dots, the parser succeeds with no validation error, yet isValidURLString returns false.

Representative reproducers (all verified; see make-reproducers.js). Each "view" link opens the production live viewer with the exact input preloaded; the fragment carries the input as base64-encoded UTF-8, so the invisible and fullwidth code points survive copy/paste intact:

Input	Parser host	Parser errors	isValidURLString	Live viewer
https://１.2.3.4 (fullwidth digit U+FF11)	1.2.3.4	[]	false	view
https://𝟏.2.3.4 (math digit U+1D7CF)	1.2.3.4	[]	false	view
https://².2.3.4 (superscript U+00B2)	2.2.3.4	[]	false	view
https://1。2。3。4 (ideographic dot U+3002)	1.2.3.4	[]	false	view
https://1．2．3．4 (fullwidth dot U+FF0E)	1.2.3.4	[]	false	view
https://1｡2｡3｡4 (halfwidth dot U+FF61)	1.2.3.4	[]	false	view
https://1.2.3.4­ (soft hyphen U+00AD, ignored)	1.2.3.4	[]	false	view
https://1.2.3.4​ (zero-width space U+200B, ignored)	1.2.3.4	[]	false	view
https://1.2.3.4 (BOM U+FEFF, ignored)	1.2.3.4	[]	false	view
https://①.2.3.4 (circled digit U+2460)	1.2.3.4	[]	false	view

Shortest reproducer: ws://０.0.0.0 (view in live viewer).

Root cause (with spec references).

• The host parser (§host parser) runs the domain parser with beStrict = false, then: "If asciiDomain ends in a number, then return the result of IPv4 parsing asciiDomain."
• The domain parser (§domain-to-ASCII) computes a strictResult via UTS-46 ToASCII with the strict flags. A domain-to-ASCII validation error is reported only if strictResult is a failure. UTS-46 mapping (fullwidth/math/super/subscript/circled digits → ASCII digits; ideographic/fullwidth dots → .; soft hyphen / zero-width characters → removed) is silent — it is not a failure. 1.2.3.4 also passes the strict STD3 / CheckHyphens / VerifyDnsLength checks, so strictResult succeeds and no validation error fires. The result then "ends in a number" and IPv4 parsing of 1.2.3.4 is itself clean.
• The writing grammar (§host writing) is stricter. A valid IPv4-address string must be four ASCII-digit decimal parts — １.2.3.4 is not (１ is not an ASCII digit). A valid domain string must be a string that is a valid domain, i.e. the strict domain parser returns non-failure, the result does not end in a number, and it has no forbidden domain code points — １.2.3.4's strict result 1.2.3.4 does end in a number, so it is not a valid domain string either. Hence the input is not a valid host string, and the URL is not a valid URL string. lib/url-string-validator.js implements exactly this (isValidIPv4AddressString requires ASCII digits; isValidDomainString rejects endsInANumber).

Both sides are individually spec-correct. The spec only guarantees the forward direction — a valid URL string parses with no validation error — and does not guarantee the converse. This family is a witness that the converse is false, so the ⟺ invariant cannot hold in general.

When it does not diverge (mechanism boundaries), all confirmed:

• Mapped host that becomes a non-numeric domain (e.g. ⅰ→i, https://ⅰ.2.3.4): parser IPv4-parses i.2.3.4, hits IPv4-non-numeric-part, so it is not clean → both reject → agree.
• Non-ASCII digits that UTS-46 does not map to ASCII (Arabic-Indic ٤, Devanagari ४): the domain fails ToASCII → domain-to-ASCII error → both reject → agree.
• Mapped host that is an incomplete/too-many/out-of-range IPv4: parser emits an IPv4-* error → both reject → agree.
• The same characters in path / query / fragment / opaque host / opaque path: no IDNA mapping happens there, and they are URL units on both sides → agree.

Is it fixable?

Not in the string validator: the validator already matches the writing grammar, and the grammar genuinely classifies these as invalid. Aligning would require one of:

1. A parser-side validation error for a domain that UTS-46 mapping changes into an IPv4 (or, more broadly, a non-canonical-host / "domain is not already in ASCII IPv4 form" validation error). This is a spec change to the host parser.
2. Broadening the writing grammar so that a host whose UTS-46 mapping yields a valid IPv4 counts as a valid IPv4-address string. Also a spec change, and a semantically awkward one.
3. Documenting this as a known, accepted limitation of the isValidURLString ⟺ parser correspondence.

Because both functions are spec-faithful, the practical recommendation is to raise it upstream (whatwg/url) as "the parser accepts IDNA-mapped IPv4 hosts with no validation error, but such URLs are not valid URL strings" and decide there between (1) and (3). It is out of scope for the #924 alignment, which targeted the ASCII grammar and is clean.

Reproducing

# One case, with code-point breakdown and agree/diverge verdict:
node fuzz/repro.js 'https://１.2.3.4'
node fuzz/repro.js '#frag' 'foo:opaque'      # input + optional base

# Canonical reproducer set (verified):
node fuzz/make-reproducers.js

# Live-viewer links used in this document (verified, base64-encoded):
node fuzz/make-viewer-links.js

# Random differential fuzzer (multi-process; Ctrl-C to stop):
node fuzz/fuzz.js --seconds 120 --workers 14 --out fuzz/out
node fuzz/fuzz.js --seconds 120 --ascii-only  # isolate non-IDNA divergences

# Exhaustive ASCII enumeration against all bases:
node fuzz/exhaustive.js --max-len 5
node fuzz/exhaustive.js --max-len 6 --alphabet 'a10xn-_:/\.?#@[]|% '

# Targeted IDNA mechanism probe:
node fuzz/analyze-idna.js

# Triage a run's findings into "known family" vs "novel":
node fuzz/triage.js fuzz/out-long