Security design prompt

Sec.txt

Act as a Senior Capital Markets Solution Architect.
Design an enterprise-grade, cloud-native Security Master platform for a financial institution consuming reference and market data from S&P Global Market Intelligence.
The solution must be deployed on Microsoft Azure, use event-driven principles, and expose data to downstream systems via REST APIs.
1️⃣ Ingestion Layer
API-based ingestion implemented using Spring Boot.
File-based ingestion via AutoSys jobs running on Linux VM over SFTP.
Secure credential storage using Azure Key Vault.
Idempotent ingestion with retry, rate-limit handling, and delta processing.
Support both full loads and incremental updates.
2️⃣ Data Processing & Storage
Raw data stored in Azure Data Lake Gen2 (Bronze layer).
Data transformation and normalization using Azure Databricks.
Medallion architecture (Bronze → Silver → Gold).
Golden Security Master stored in MongoDB.
Golden database requirements:
Internal persistent surrogate IDs.
Identifier history (ISIN, CUSIP, Ticker).
Point-in-time modeling (SCD Type 2).
Corporate action support.
Source lineage and audit tracking.
Optimized indexing for low-latency lookups.
3️⃣ Real-Time Event Publishing
Change events published via Confluent Kafka.
Prefer MongoDB Change Streams (CDC) over dual-write pattern.
Topic partitioning by internal_security_id.
Schema governance using Schema Registry (Avro or Protobuf).
Support replay capability and idempotent consumers.
4️⃣ Consumer & Access Layer
Downstream applications are developed using Spring Boot.
Applications are deployed on Azure Kubernetes Service (AKS).
Security Master data exposed via REST APIs.
API layer supports:
Search by ISIN, Ticker, Internal ID.
Asset-class filtering.
Point-in-time queries.
Corporate action history.
API gateway and security using OAuth2/JWT.
Horizontal scaling with AKS auto-scaling.
5️⃣ Governance & Compliance
Data quality validation framework.
Conflict resolution strategy (API vs File feed).
Field-level lineage tracking.
Audit support for Basel, IFRS, SEC, SEBI.
HA/DR architecture for MongoDB, Kafka, and AKS.
Role-based access control (RBAC).
6️⃣ Non-Functional Requirements
Support millions of instruments.
Low-latency REST response (<100ms target for lookup).
High availability (multi-zone AKS + Mongo replica set).
Secure networking (Private Endpoints, VNet isolation).
Observability (distributed tracing, metrics, alerts).
CI/CD pipeline with containerized deployments.
Provide:
Logical end-to-end architecture diagram.
MongoDB collection design.
REST API contract design.
Kafka topic and event schema design.
HA/DR strategy.
Scalability and performance tuning approach.
Enterprise anti-patterns to avoid.