Case Study: Real-Time Fraud Detection for a National Insurance Carrier

Client: National Insurance Carrier (anonymized)

Industry: Insurance / Property & Casualty

Data Scale: 2PB+ claims and policy data lake on Databricks

Engagement Type: Streaming Architecture Build & Real-Time Fraud Detection Platform

The Challenge

A national property and casualty insurance carrier was losing an estimated eight figures annually to fraudulent claims. The fraud was not subtle — organized rings submitting duplicate claims across policy numbers, staged accidents with coordinated medical billing, and identity reuse patterns that were obvious in retrospect. The problem was not detection capability. The problem was timing.

The carrier's existing fraud detection operated on a nightly batch cycle. By the time a fraudulent claim was flagged, it had often already been processed. In some cases, checks had been issued. Recoveries were expensive and unreliable. The fraud team needed to move detection upstream — catching suspicious patterns at submission time, not after payment.

The existing Databricks platform was batch-native. There was no streaming infrastructure, no real-time feature computation, and no mechanism to score a claim in flight. LakeVector was engaged to build it.

Why Structured Streaming + Kafka?

The architecture decision was straightforward: Apache Kafka as the real-time event backbone, Databricks Structured Streaming for stateful stream processing, and Delta Lake as the unified storage layer that bridged streaming and batch workloads.

The key architectural advantage of Structured Streaming for this use case was stateful processing — the ability to maintain sliding windows of claim history per policy, claimant, and provider and continuously evaluate new events against that accumulated state. Batch systems fundamentally cannot do this. A nightly job sees yesterday's claims. A streaming job sees every claim the moment it enters the system and can cross-reference it against the full historical window in milliseconds.

Kafka provided the durable, replayable event log that decoupled claim submission systems from the detection layer — ensuring no events were lost and the streaming system could be restarted or redeployed without data loss.

The Engagement

Phase 1: Event Stream Design and Kafka Integration

Lakevector worked with the carrier's claims engineering team to instrument the claims submission API as a Kafka producer. Every claim submission, update, and status change was published as a structured event to a set of Kafka topics partitioned by claim type, region, and policy line.

Schema governance was implemented from day one using Confluent Schema Registry — ensuring that every event conformed to a versioned Avro schema, preventing the schema drift that had plagued previous integration efforts.

Phase 2: Feature Engineering in Structured Streaming

The fraud detection model required features that could only be computed in real time against a stateful view of historical activity. LakeVector built a Structured Streaming pipeline that maintained continuously updated feature aggregations per key entity:

Claimant-level features — claim frequency, claim velocity (claims per rolling 30/60/90-day window), provider network overlap
Provider-level features — billing pattern anomalies, geographic clustering of claimants, specialty-to-billing-code mismatch rates
Policy-level features — multi-claimant submission patterns, time-to-claim distribution relative to policy inception
Network features — shared addresses, phone numbers, and banking details across claimants and providers, computed via streaming graph lookups

These features were written to a Delta Lake feature store, queryable by both the real-time scoring layer and the offline model training pipeline — ensuring training and serving used identical feature definitions.

Phase 3: Model Serving and Real-Time Scoring

Fraud models trained on the offline feature store were deployed via MLflow Model Registry and served through Databricks Model Serving. The streaming pipeline called the model endpoint for every incoming claim, returning a fraud probability score within the latency budget required for in-flight claim adjudication.

High-confidence fraud scores triggered immediate routing to the fraud investigation queue. Medium-confidence scores flagged claims for enhanced documentation requirements before processing. Low-confidence claims flowed through the standard adjudication path uninterrupted.

Phase 4: Alert Infrastructure and Case Management Integration

Lakevector built a streaming alert layer on top of the scoring pipeline: pattern-based rules evaluated in Structured Streaming alongside the model score, with alerts published back to Kafka and consumed by the carrier's case management system. Fraud investigators received real-time queues populated by the streaming system — replacing the manual, next-day review process with a continuous feed of actionable cases.

The Results

Detection Latency (Batch -> Streaming): 24 hours -> under 2 seconds
Fraud caught before payment: Increased from ~20% to ~78% of flagged cases
Estimated annual fraud loss reduction: 8 figures
False positive rate vs. prior model: Reduced by 31% (richer real-time features)
Claims processed with zero latency impact: 99.4% (fraud scoring added <50ms to non-flagged claims)
Investigator productivity: 3x increase (real-time queue vs. next-day batch review)

The Lasting Impact

The shift from batch to streaming was not just a technical upgrade — it was a fundamental change in when fraud could be stopped. Payment clawback is expensive and uncertain. Real-time prevention is neither. The carrier now has a platform that hardens continuously: as new fraud patterns emerge, new streaming rules and model versions are deployed without pipeline downtime, using Structured Streaming's rolling deployment capability.

The same streaming infrastructure built for fraud detection has since been extended to claims triage, SLA monitoring, and real-time underwriting signals — the platform proved itself and became the foundation for the carrier's next generation of data products.

Why Lakevector

Lakevector builds streaming architectures for organizations where latency is money. We combine deep Spark and Databricks expertise with production-grade data engineering to deliver real-time platforms that actually work at scale.

If your orchestration layer is limiting your platform, we should talk.

Talk to an Expert Today