Event-driven architecture: Challenges and how to overcome them

Small startups are often confused about why enterprises invest so much time and money in heavy-duty project management software. “We just Slack each other,” they might think, “Why can’t they do the same?” Inevitably, as startups scale and the number of employees grow, this kind of complex software starts to seem more appealing. Eventually, the complexity of the company warrants similarly complex software and the burden of the tool becomes worth the tradeoff. 

EDA follows a similar pattern. In another article, we covered some of the major benefits of adopting EDA but not all of those benefits are applicable in all contexts. For many use cases, particularly companies with relatively simple applications, EDA can impose tradeoffs that aren’t worth the benefits.

EDA tends to offer the most benefits to enterprises with large codebases and businesses with complex lines of business applications. 

For small development teams, the complexity inherent to EDA can be daunting. Traditional request/response systems are often more familiar and managing the lifecycle of events in an EDA system can create ongoing strain. 

This complexity is especially challenging in troubleshooting and monitoring contexts. Because EDA-based applications are, by definition, distributed and decoupled, development teams can struggle to trace events passing from publishers to subscribers. In a request/response system, it tends to be easier to trace the passage of events because the process is linear. 

In an EDA system, numerous subscribers can subscribe to a single event stream, which offers advantages even as it creates complexity. The harder it is to trace the publishing and passing of events, the harder it can be to test and debug the resulting functions. 

Monitoring can be difficult for similar reasons because the independence and isolation of the decoupled services require development teams to build and operate a specialized monitoring system. 

This complexity breaks down to the API level too. Event-driven APIs maintain event states and track any state changes. The publisher is responsible for orchestrating the message delivery process – itself a complex task – and failures have to be kept statistically independent to ensure a reliable experience for end users.

Businesses are often drawn to EDA systems because they have discrete tasks or processes that would seemingly benefit from asynchronous processing. If there aren’t enough of these tasks or if the value of processing these tasks in realtime is minimal, however, then EDA is likely not the best option. 

If companies are running regular cron jobs, for example, the extent to which EDA would make this work faster is likely outweighed by the burden of adopting and maintaining the overall system. And if the tasks that need to be processed are periodic or can occur in the background, one of the primary benefits of EDA – asynchronous communication – doesn’t apply and the rest of EDA can become more of a drawback than a benefit. 

EDA is only as good as its implementation. Just like realtime updates infrastructure, an effective EDA implementation can be enormously beneficial whereas a poor implementation can be detrimental.

Remember, companies usually adopt EDA when they need greater scalability or resiliency, but if the implementation process goes poorly, then companies can end up with systems that are harder to scale than traditional, off-the-shelf alternatives. 

Implementation is particularly likely to be difficult due to a lack of external support and internal experience. As we’ve written elsewhere, there’s little to no support for event-driven protocols and patterns amongst API getaways. There’s also a wide range of implementation methods, meaning there’s little standardization to rely on. API governance, which was already difficult, becomes even harder.

This problem makes a likely skills gap even harder. Most developers are familiar with request/response systems, so adopting an EDA system can often sit outside of a developer's current skill set or require companies to hire new developers. In both cases, the adoption process can cause stress, and implementation quality – as well as the developer experience – can suffer.

For many companies, the benefits of EDA are real and the challenges appear equally daunting. With the right implementation strategies, however, companies can make EDA work for them. 

EDA systems are notoriously complex but much of this complexity is revealed, not created. For example, many developers read about EDA and imagine a complex solar system of publishers, subscribers, and events orbiting and circulating and colliding. It can appear complex compared to request/response systems, but these traditional systems contain much of the same complexity – it’s just centralized into a monolith. 

The advantage of adopting EDA, as well as other distributed and service-oriented systems, is that revealing the complexity gives you the opportunity to do a better job managing it. The initial work will likely be difficult, but if you can embrace complexity, you can get a deeper, more granular understanding of the components that make up your application. And once you have that understanding, even more flexibility is unlocked as you become better able to control, manage, and maintain each component. 

Companies can also address complexity issues by reframing their approaches and rethinking how they handle different challenges. In a request/response system, monitoring might be as easy as adopting a monitoring tool built to handle the linear passage of events inherent to request/response. That same solution might not be possible for EDA, but that doesn’t mean monitoring is impossible. Instead, companies can work out different approaches and find tools that suit their new circumstances. 

Development teams tend to struggle with event-driven architecture if they don’t think through event-driven design. EDA poses a deep system-level change to how teams build and maintain software. If teams try to graft previous workflows and practices onto a new architecture, they’ll likely encounter friction. At that point, it becomes tempting to attribute the friction to the system and not to the practices applied to the system.

Instead, if companies offer developers time and resources to learn how best to operate an event-driven system, many of the tradeoffs mentioned above can be managed. For example, designing events – the atomic units of EDA – can be difficult at first. It’s tempting to slip to one of two extremes by either designing events to be bespoke to each subscriber, which makes development costly and slow, or designing events to be reusable, which makes each event so generic that the intent of each event becomes hard to understand. 

Despite these two extremes, there’s a balance to be struck and as developers find the right midpoint, they can begin to reliably design events that fit each subscriber and are reusable enough to not require constant from-scratch development.

Some of the complexity resulting from EDA systems can be handled with better, more comprehensive documentation.

Because EDA decouples event publishers and subscribers, the messaging process can become abstract and the information flow can become harder to discern at a glance. Even if you look at the code of a publisher component, for example, there might not be a clear direction on what happens next once that publisher produces an event. 

With comprehensive documentation, outside developers and newly onboarded developers don’t have to rely on the code alone to understand what it does. Creating documentation of this depth and quality might require hiring a technical writer or dedicating sprints to documentation work, but the investment is often worth the payoff because it can make an otherwise complex system easy to understand and use. 

Companies feeling compelled by the benefits of event-driven architecture should be mindful about choosing this approach over another that might provide similar benefits. With software design, one approach might appear to be better, at first glance, than another but the right fine-tuning might mean a different design can be much more suitable than it first appeared. 

We’ve compared EDA against request/response systems throughout this article, and though it’s the more traditional approach, request/response is frequently a good option. In some cases, for example, a benefit of EDA – such as the usage of realtime data – is really only nice to have and a carefully crafted request/response system can be better.

An ecommerce company might want to send out shipment updates in realtime, for example, but the benefit of that speed might be outweighed by an otherwise unwieldy EDA system – especially when companies like Amazon provide a good enough experience sending confirmation emails minutes after processing an order. 

Event streaming, a particular implementation of event-driven architecture, can appear by name to be a good fit for companies wanting to stream video and audio content. Event streaming, however, is a better fit for use cases when events need to be processed in realtime and referenced after the fact, such as in fraud detection.

Media streaming requires entirely different architecture design decisions, so companies that will be prioritizing this kind of streaming will likely want to consider other architecture choices. 

Service-oriented architecture, or SOA, is a broad category of software design that emphasizes breaking application components into services. Microservices, a model that often benefits from EDA, falls into this category. 

Many companies are enticed by EDA because they like the idea of decoupling components. Companies interested in that benefit should be wary of pursuing EDA for that benefit alone. There are numerous SOA approaches that provide similar kinds of decoupling and those might be more suitable to a given company’s needs. 

EDA can be a huge advantage for the right business but it can also be a detriment weighing down a different business. The choice to adopt EDA will ultimately come down to one based on ROI, but companies considering their options will have to ensure the factors feeding into their measurement of ROI are holistic. 

It’s often tempting, for example, to calculate ROI based on cost input and savings output. But if EDA provides resiliency that makes your application more reliable at a global scale, your users might be satisfied in a way that isn’t clear in blunt usage metrics. Similarly, EDA could enable the building of realtime data features but if you don’t have the developers and resources ready to maintain such a complex system, those features can suffer if the EDA infrastructure underlying the system is too unwieldy. 

The challenges of EDA – including excessive complexity, overkill for periodic tasks, and implementation difficulty – then become important considerations. EDA is effective when done well but for many companies, it’s detrimental if implemented poorly or not maintained over time.

Companies looking to adopt EDA should examine which best practices will best support EDA and search for partners that can help them realize the potential of EDA.