Skip to content

Instantly share code, notes, and snippets.

@architectureman

architectureman/CBOE PITCH Protocol Testing Guide for Beginners.md

Created June 19, 2025 11:24
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save architectureman/a825bc9faacc57b4fa468f10df6e6e67 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save architectureman/a825bc9faacc57b4fa468f10df6e6e67 to your computer and use it in GitHub Desktop.
Download ZIP
CBOE PITCH Protocol Testing Guide for Beginners
Raw
CBOE PITCH Protocol Testing Guide for Beginners.md

CBOE PITCH Protocol Testing Guide for Beginners

Introduction

Welcome to the CBOE PITCH Protocol Testing Guide! This document is designed for testers who are new to financial trading systems. We'll use simple language, step-by-step instructions, and visual examples to help you understand how to test systems that use the CBOE PITCH protocol.

What Are We Testing?

Before we dive into testing, let's understand what the CBOE PITCH protocol does in simple terms:

PITCH is a data feed that sends information about stock market activity to trading systems. Think of it like a play-by-play sports commentator who reports:

  • New orders to buy or sell stocks
  • Changes to existing orders
  • Trades that have occurred
  • Market status changes

Your job as a tester is to make sure that systems correctly:

  1. Receive these messages
  2. Interpret them accurately
  3. Update their view of the market properly
  4. Recover from any errors or gaps in the data

The Testing Landscape

Here's a simplified view of what you'll be testing:

                                   YOUR SYSTEM
 ┌───────────────┐              ┌──────────────────────────────────────┐
 │               │   Messages   │                                      │
 │  CBOE PITCH   │─────────────▶│  Message      Order Book    Trading  │
 │   Feed        │              │  Processor ──▶ Builder  ──▶ Logic    │
 │               │              │                                      │
 └───────────────┘              └──────────────────────────────────────┘

Basic Testing Concepts

Types of Testing for PITCH Systems

  1. Functional Testing: Does the system correctly process each message type?
  2. Performance Testing: Can it handle the volume and speed of messages?
  3. Recovery Testing: Can it recover from gaps or disconnections?
  4. Integration Testing: Does it work with other systems in the environment?
  5. Regression Testing: Do changes break existing functionality?

Test Environment Requirements

To test PITCH systems effectively, you'll need:

  • Test Feed: A source of PITCH messages (either recorded or simulated)
  • Test Data: Pre-defined order scenarios to verify specific behaviors
  • Validation Tools: Ways to verify the system's order book state
  • Monitoring Tools: To observe message processing performance

Test Scenarios by Message Type

Let's look at key message types and how to test them. We'll include example messages and expected outcomes.

1. Testing Add Order Messages

An Add Order message tells the system someone wants to buy or sell.

Example Message:
ADD ORDER
Time: 09:15:30.123456789
Order ID: ABC123
Side: Buy
Quantity: 100
Symbol: DEMO
Price: $10.25

Test Scenario: Basic Order Addition

Steps:

  1. Send the Add Order message above
  2. Verify the order book now shows this order
  3. Check that it's in the correct location (buy side, price $10.25)
  4. Verify the quantity is correct (100 shares)

Expected Result:

Before:                          After:
   DEMO ORDER BOOK                 DEMO ORDER BOOK
┌─────────┬───────────┐         ┌─────────┬───────────┐
│  BUY    │   SELL    │         │  BUY    │   SELL    │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.20: │ $10.30:   │         │ $10.25: │ $10.30:   │ ← New order appears here
│   200   │   150     │         │   100   │   150     │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.15: │ $10.35:   │         │ $10.20: │ $10.35:   │
│   500   │   300     │         │   200   │   300     │
└─────────┴───────────┘         ├─────────┼───────────┤
                                │ $10.15: │           │
                                │   500   │           │
                                └─────────┴───────────┘

Additional Test Cases:

  • Add orders with different prices and quantities
  • Add orders for different symbols
  • Add buy and sell orders
  • Add undisclosed orders (where quantity shows as 0)

2. Testing Order Executed Messages

An Order Executed message indicates a trade has happened.

Example Message:
ORDER EXECUTED
Time: 09:18:15.123456789
Order ID: ABC123
Executed Quantity: 100
Execution ID: EX12345

Test Scenario: Full Order Execution

Steps:

  1. Start with a known order (ABC123 for 100 shares)
  2. Send the Order Executed message above
  3. Verify the order is removed from the book (since fully executed)

Expected Result:

Before:                          After:
   DEMO ORDER BOOK                 DEMO ORDER BOOK
┌─────────┬───────────┐         ┌─────────┬───────────┐
│  BUY    │   SELL    │         │  BUY    │   SELL    │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.25: │ $10.30:   │         │ $10.20: │ $10.30:   │ ← Order ABC123 is gone
│   100   │   150     │         │   200   │   150     │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.20: │ $10.35:   │         │ $10.15: │ $10.35:   │
│   200   │   300     │         │   500   │   300     │
├─────────┼───────────┤         └─────────┴───────────┘
│ $10.15: │           │
│   500   │           │
└─────────┴───────────┘

Test Scenario: Partial Order Execution

Example Message:
ORDER EXECUTED
Time: 09:18:15.123456789
Order ID: DEF456
Executed Quantity: 50  (Original order was 200)
Execution ID: EX12346

Expected Result:

Before:                          After:
   DEMO ORDER BOOK                 DEMO ORDER BOOK
┌─────────┬───────────┐         ┌─────────┬───────────┐
│  BUY    │   SELL    │         │  BUY    │   SELL    │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.20: │ $10.30:   │         │ $10.20: │ $10.30:   │ ← Order DEF456 reduced
│   200   │   150     │         │   150   │   150     │   from 200 to 150
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.15: │ $10.35:   │         │ $10.15: │ $10.35:   │
│   500   │   300     │         │   500   │   300     │
└─────────┴───────────┘         └─────────┴───────────┘

Additional Test Cases:

  • Execute orders with various quantities
  • Execute multiple orders in sequence
  • Test execution reports for hidden portions of iceberg orders

3. Testing Delete Order Messages

A Delete Order message removes an order from the book.

Example Message:
DELETE ORDER
Time: 09:17:30.123456789
Order ID: GHI789

Test Scenario: Order Deletion

Steps:

  1. Start with a known order (GHI789)
  2. Send the Delete Order message
  3. Verify the order is removed from the book

Expected Result:

Before:                          After:
   DEMO ORDER BOOK                 DEMO ORDER BOOK
┌─────────┬───────────┐         ┌─────────┬───────────┐
│  BUY    │   SELL    │         │  BUY    │   SELL    │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.20: │ $10.30:   │         │ $10.20: │ $10.30:   │
│   200   │   150     │         │   200   │   150     │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.15: │ $10.35:   │         │ $10.15: │           │ ← Order GHI789 is gone
│   500   │   300     │         │         │           │   from this price level
└─────────┴───────────┘         └─────────┴───────────┘

4. Testing Modify Order Messages

A Modify Order message changes an existing order's price or quantity.

Example Message:
MODIFY ORDER
Time: 09:16:45.123456789
Order ID: JKL012
NEW Quantity: 300  (Original was 200)
NEW Price: $10.25  (Original was $10.20)

Test Scenario: Order Modification (Price Change)

Steps:

  1. Start with a known order (JKL012 at $10.20 for 200 shares)
  2. Send the Modify Order message
  3. Verify the order is moved to the new price level with the new quantity

Expected Result:

Before:                          After:
   DEMO ORDER BOOK                 DEMO ORDER BOOK
┌─────────┬───────────┐         ┌─────────┬───────────┐
│  BUY    │   SELL    │         │  BUY    │   SELL    │
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.20: │ $10.30:   │         │ $10.25: │ $10.30:   │ ← Order JKL012 moved here
│   200   │   150     │         │   300   │   150     │   with new quantity
├─────────┼───────────┤         ├─────────┼───────────┤
│ $10.15: │ $10.35:   │         │ $10.20: │ $10.35:   │ ← No longer at this price
│   500   │   300     │         │    0    │   300     │
└─────────┴───────────┘         ├─────────┼───────────┤
                                │ $10.15: │           │
                                │   500   │           │
                                └─────────┴───────────┘

Additional Test Cases:

  • Modify just the quantity (not the price)
  • Modify to a higher price (lose priority in the book)
  • Modify to a lower price (gain better position in the book)

5. Testing Trading Status Messages

A Trading Status message changes the state of a symbol.

Example Message:
TRADING STATUS
Time: 10:00:00.000000000
Symbol: DEMO
Status: H (Halted)

Test Scenario: Trading Halt

Steps:

  1. Start with DEMO in normal trading mode
  2. Send the Trading Status message with Halted status
  3. Verify the system updates the symbol's status
  4. Verify the system handles new orders appropriately (should reject them)

Expected Result:

  • System shows DEMO as halted
  • New orders for DEMO are rejected
  • Existing orders remain in the book
  • Executions for DEMO are not processed

6. Testing Auction Messages

Auctions are special events where orders are matched at a single price.

Test Scenario: Opening Auction

Steps:

  1. Start with Pre-Open status
  2. Send several Auction Update messages showing indicative price and volume
  3. Send final Auction Summary message
  4. Send Order Executed at Price messages for matched orders
  5. Verify the system updates the book correctly

Auction Update Message:

AUCTION UPDATE
Time: 09:29:45.123456789
Symbol: DEMO
Auction Type: O (Opening)
Buy Shares: 1000
Sell Shares: 800
Indicative Price: $10.25

Auction Summary Message:

AUCTION SUMMARY
Time: 09:30:00.000000000
Symbol: DEMO
Auction Type: O (Opening)
Price: $10.25
Shares: 800

Expected Result:

  • System shows the auction information during pre-open
  • After auction completes, matched orders are executed at the auction price
  • Remaining orders stay in the book
  • Trading status changes to normal trading

Testing Special Order Types

Testing Undisclosed (Hidden) Orders

Undisclosed orders don't show their quantity in the order book.

Test Scenario: Undisclosed Order Execution

Steps:

  1. Send an Add Order message with quantity = 0 (indicates undisclosed)
  2. Send a Trade message for this order
  3. Verify the system handles the hidden execution correctly

Add Order Message (Undisclosed):

ADD ORDER
Time: 09:15:30.123456789
Order ID: MNO345
Side: Buy
Quantity: 0  (indicates undisclosed)
Symbol: DEMO
Price: $10.25

Trade Message (for Undisclosed Order):

TRADE
Time: 09:16:15.123456789
Symbol: DEMO
Quantity: 100
Price: $10.25
Execution ID: EX12347
Order ID: MNO345

Expected Result:

  • The undisclosed order is tracked internally but not shown in the visible book
  • The Trade message updates the internal state of the order
  • Statistics about traded volume are updated

Testing Iceberg Orders

Iceberg orders show only a portion of their total quantity.

Test Scenario: Iceberg Order Replenishment

Steps:

  1. Send an Add Order for the visible portion
  2. Send an Order Executed message that consumes all visible quantity
  3. Send a new Add Order with a new Order ID (replenishment)
  4. Verify the system recognizes this as an iceberg replenishment

Initial Add Order:

ADD ORDER
Time: 09:20:30.123456789
Order ID: PQR678
Side: Buy
Quantity: 100  (visible portion)
Symbol: DEMO
Price: $10.25

Order Executed:

ORDER EXECUTED
Time: 09:21:15.123456789
Order ID: PQR678
Executed Quantity: 100
Execution ID: EX12348

Replenishment Add Order:

ADD ORDER
Time: 09:21:15.123789012
Order ID: STU901  (new ID)
Side: Buy
Quantity: 100  (another portion)
Symbol: DEMO
Price: $10.25

Expected Result:

  • System processes the execution of the first portion
  • System adds the replenishment order at the same price
  • Time priority of the replenishment order is based on the new timestamp

Testing Data Recovery Scenarios

Testing Gap Detection and Recovery

Test Scenario: Message Gap Recovery

Steps:

  1. Send messages with sequence numbers 1001, 1002, 1003
  2. Skip to sequence 1007 (creating a gap from 1004-1006)
  3. Verify the system detects the gap
  4. Send a Gap Request message from the system to the Gap Request Proxy
  5. Send the missing messages 1004-1006
  6. Verify the system processes all messages in order

Expected Result:

  • System detects the sequence gap
  • System requests the missing messages
  • System waits for recovery before processing message 1007
  • Once recovered, system has processed all messages in order

Gap Request:

GAP REQUEST
Unit: 1
Sequence: 1004
Count: 3

Gap Response:

GAP RESPONSE
Unit: 1
Sequence: 1004
Count: 3
Status: A (Accepted)

Testing Spin Recovery

Test Scenario: Full Book Recovery via Spin

Steps:

  1. Start with an empty order book
  2. Request a spin of the current state
  3. Receive spin data (all current orders)
  4. Verify the order book is properly constructed
  5. Continue processing real-time messages

Spin Request:

SPIN REQUEST
Sequence: 5000  (current sequence)

Spin Response:

SPIN RESPONSE
Sequence: 5000
Order Count: 25
Status: A (Accepted)

Expected Result:

  • System sends a proper Spin Request
  • System receives and processes Trading Status and Add Order messages
  • System builds a complete order book from the spin data
  • System continues processing real-time messages from sequence 5001 onward

Performance Testing

Message Rate Testing

Test Scenario: Peak Message Rate Handling

Steps:

  1. Prepare a test feed with messages at gradually increasing rates
  2. Monitor system performance as message rates increase
  3. Identify the point where the system begins to fall behind
  4. Verify recovery when rates decrease

Expected Result:

  • System processes messages without delay up to target rate
  • CPU and memory usage remain within acceptable limits
  • If system falls behind, it recovers when message rate decreases

Latency Testing

Test Scenario: Processing Latency Measurement

Steps:

  1. Send messages with timestamps
  2. Measure time from message receipt to order book update
  3. Calculate min, max, average, and 99th percentile latency

Expected Result:

  • Average processing latency remains under target threshold
  • 99th percentile latency remains under target threshold
  • No processing outliers beyond acceptable limits

Creating Comprehensive Test Plans

Test Matrix Approach

Create a matrix that combines:

  • Message types (Add, Execute, Modify, Delete, etc.)
  • Order types (Regular, Undisclosed, Iceberg)
  • Market conditions (Normal, High Volume, Auction)
  • Recovery scenarios (Gaps, Spins, Disconnects)

Example test matrix (small portion):

                │ Regular Orders │ Undisclosed Orders │ Iceberg Orders │
────────────────┼────────────────┼────────────────────┼────────────────┤
Add Order       │      TC-1      │        TC-2        │      TC-3      │
                │                │                    │                │
Order Executed  │      TC-4      │        TC-5        │      TC-6      │
                │                │                    │                │
Modify Order    │      TC-7      │        TC-8        │      TC-9      │
                │                │                    │                │
Delete Order    │     TC-10      │       TC-11        │     TC-12      │
                │                │                    │                │

Typical Test Case Structure

Each test case should include:

  1. Test ID: Unique identifier
  2. Description: What is being tested
  3. Preconditions: Initial state required
  4. Test Steps: Detailed actions
  5. Expected Results: What should happen
  6. Actual Results: What actually happened
  7. Pass/Fail: Test outcome

Example:

Test ID: TC-4
Description: Test full execution of a regular order
Preconditions: 
  - Order book contains a buy order (ID: XYZ123) for 100 shares of DEMO at $10.25
  - Order is visible in the book

Test Steps:
  1. Send Order Executed message for order XYZ123, quantity 100
  2. Check order book state

Expected Results:
  - Order XYZ123 is removed from the order book
  - Total buy quantity at $10.25 is reduced by 100
  - System records the execution properly

Actual Results: [Filled in during testing]
Pass/Fail: [Filled in during testing]

Automated Testing Approaches

Replay Testing

Record actual market data and replay it through your system:

  1. Capture a full day of PITCH messages
  2. Replay at normal or accelerated speed
  3. Verify system state at key points

Simulation Testing

Create simulated market scenarios:

  1. Generate synthetic orders, executions, etc.
  2. Create specific edge cases and stress conditions
  3. Test recovery from unusual situations

Regression Testing

Create a suite of tests that run automatically:

  1. Basic order book functionality
  2. Special order types
  3. Recovery mechanisms
  4. Performance under load

Common Testing Challenges

Timing Issues

Problem: Messages need to be processed in the correct order.

Testing approach:

  • Deliberately send messages out of sequence
  • Verify system buffers and reorders correctly
  • Test with varying gaps between messages

Data Volume

Problem: Systems must handle large volumes of data.

Testing approach:

  • Test with realistic volumes
  • Gradually increase message rates
  • Test with spikes in activity

Recovery Scenarios

Problem: Systems must recover from gaps and disconnections.

Testing approach:

  • Simulate network disconnections
  • Create message gaps of varying sizes
  • Test spin recovery at different times

Verification Techniques

Order Book Validation

Verify the accuracy of the order book by:

  1. Visual Inspection: For small test cases, manually check the book
  2. Balance Checks: Total quantity at each price level should match sum of individual orders
  3. Reference Comparison: Compare against a reference implementation
  4. Replay Validation: Apply all messages to an empty book and verify final state

State Logging

Implement detailed logging to track state changes:

  1. Log order book state after each message
  2. Create before/after snapshots for key operations
  3. Record all detected anomalies

Checksum Verification

Use checksums to verify data integrity:

  1. Calculate a checksum of the entire order book
  2. Verify checksum matches expected value after known sequences

Testing Best Practices

  1. Start Simple: Begin with basic order operations before complex scenarios
  2. Isolate Tests: Each test should validate one specific behavior
  3. Automate Everything: Manual testing doesn't scale for market data
  4. Use Real Data: Include recorded market data in your test suite
  5. Test Edge Cases: Zero-quantity orders, maximum values, etc.
  6. Test Recovery: Gap detection and recovery are critical
  7. Performance Matters: Test at peak message rates
  8. Document Everything: Record all test cases and results

Glossary for Testers

  • Sequence Number: Identifier that increases by one for each message
  • Gap: Missing messages in a sequence
  • Spin: Complete snapshot of the current order book state
  • Order Book: Collection of all buy and sell orders at their price levels
  • Price Level: Group of orders at the same price
  • Time Priority: Orders at the same price are prioritized by time
  • Execution: When a buy and sell order match, creating a trade
  • Undisclosed Order: Order where the quantity is hidden from the market
  • Iceberg Order: Order where only a portion of the quantity is visible
  • Auction: Process to match multiple buyers and sellers at a single price
  • Trading Status: Current state of a symbol (trading, halted, etc.)

Final Testing Checklist

Use this checklist to ensure you've covered all essential aspects:

  • Basic order operations (add, execute, modify, delete)
  • Special order types (undisclosed, iceberg)
  • Trading status changes
  • Auction processing
  • Gap detection and recovery
  • Spin recovery
  • Performance under load
  • Recovery from errors
  • Integration with other systems
  • Data validation

Remember: A well-tested system ensures accurate market data, which is essential for making proper trading decisions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment