How a single rogue account broke a system—and the fix that saved it
A quiet heartbeat system, designed to keep facilities and emergency services updated in real time, suddenly found itself drowning in eight times its normal traffic—all because one account was running months-old cached client code. The incident didn’t spike and recover; it lingered, exposing a critical flaw in the underlying architecture: one partition in Azure Event Hubs.
For weeks, the system hummed along at 51,000–58,000 heartbeats per hour—flat, predictable, forgettable. Then, on January 5 at 7:00 AM PST, it stopped being forgettable. By 9:00 PM that night, heartbeats had surged to 466,760 per hour, while SignalR connection counts skyrocketed to eleven times normal levels. The cause wasn’t a cyberattack or a traffic surge, but a single user account responsible for 33% of all token API traffic—traffic generated by stale client code that refused to expire. The system was built to handle heartbeats and sessions efficiently, but it had a hidden ceiling: one partition in its Event Hub.
The Bottleneck That Broke the Loop
Event Hubs partitions act as parallel lanes for event ingestion. With only one partition, all events funnel through a single consumer group, processed sequentially by one instance at a time. Even as the system tried to scale up—more instances, more CPU, optimized code—nothing could bypass this bottleneck. HeartbeatMonitor processed events one by one, falling further behind as the backlog grew. Facility connectivity status, the critical output relied on by hospitals and EMS, started lagging real-world conditions.
The pain didn’t stop there. The Azure Consumption plan added another layer of failure: SNAT port exhaustion. Under sustained load, outbound connections consumed ephemeral ports faster than the system could release them. New connections failed. Clients retried. The cycle fed itself, turning a manageable surge into a self-inflicted denial of service. Worse, the Consumption plan offered no visibility into individual workers or control over scaling—because even if more instances had spun up, they couldn’t help without more lanes to process events.
A Quick Fix—and a Lasting Lesson
Before rebuilding the system, the team needed to survive. They turned to a simple lever: the heartbeat interval, which was already a variable controlled by the API. By increasing the interval from the default 30 seconds, they bought breathing room while planning a more durable fix.
That fix came in three parts: more Event Hub partitions, Azure Container Apps, and KEDA. Multiple partitions allowed parallel processing of events across consumer groups. Container Apps provided better control over scaling and resource visibility. KEDA automated scaling based on actual event backlogs, ensuring capacity matched demand. Together, they removed the single-lane bottleneck and gave the system room to grow.
Why it matters
This wasn’t just a traffic spike—it was a stress test of architectural assumptions. Systems that scale well under steady load can collapse under uneven or misdirected demand. The lesson is clear: parallelism isn’t optional, and visibility into scaling behavior is non-negotiable. For teams running real-time systems on event-driven platforms, this story highlights the need to design for failure modes you can’t predict—and to build in levers that let you respond when they happen.
Source: DEV Community. AI-assisted editorial synthesis — TechnoExpress.