The Significance Of Solution Design In Serverless Developments — Part I
What does it bring, and why do we need it?
Many in the industry say that Serverless is easy to get started! It’s always good to have that kind of positive perception. After all, we need technologies that everyone can adopt quickly.
Does this perception of serverless always live up to the expectation? After the start, is it easy to keep going?
The ‘Quick-and-Easy’ Syndrome
Engineers demonstrate how easy it is to develop and run serverless applications in minutes. This early success often makes us get carried away, and we try to replicate this ‘quick-and-easy’ mantra for everything.
Teams initially experiment serverless with something small. Elated with their initial success, they keep piling things on top to maintain their success streak, which can lead to unpleasant situations.
Spaghetti code, big ball of mud, monolith stack, unstructured repository, tangled communications, coupled services, and similar terms soon become synonymous with such a situation.
Developing serverless applications that can evolve according to business demands requires more than a quick-and-easy mindset. For decades, software engineering has employed several practices that have enabled us to develop maintainable applications.
Solution Design is a process that helps us think and visualize the solution before building. It certainly has a place in serverless development.
This is the first of the two-part series on Solution Design in Serverless.
In this article, we will discuss the following topics.
What is Solution Design?
Why do we need Solution Design in serverless?
What is the thought process involved in Solution Design?
The second part will look in detail at the content of a Solution Design document and how it enables teams to develop sustainable serverless solutions.
What Is Solution Design?
Over the years, several definitions of Solution Design have emerged. They vary based on technology, industry, development lifecycle, and other factors.
Besides the differences, Solution Design mainly digests the requirements and proposes a technical solution to be realized by the development team.
Contrary to Enterprise Architecture, Solution Design often focuses on a single product. It could be for a new feature, web application, or microservice. A system’s architecture, however, can be related to Solution Design and be part of it.
Solution Design in Serverless
The purpose and process of solution design in serverless environments are almost the same as those of other technologies.
However, serverless applications differ in architecture and details such as service interactions, security, storage, scaling, resource configurations, etc.
Taking these into account, we can say that in serverless-
Solution Design is a process of translating product requirements into architectural visualization of granular microservices, along with engineering details to build an event-driven serverless application.
Why Do We Need Solution Design?
The only system that wouldn’t benefit from architecture is one that doesn’t change at all — Gregor Hohpe in The Software Architect Elevator.
Serverless is just a technology to solve business problems. In life, anything we build, not just software, requires planning.
Building a house starts with an architectural drawing. The structural design is then completed, and the interior design will likely be completed after that. One cannot construct a house on top of a few bricks that we lay to check the appeal or to test the mortar strength.
It's the same with serverless. Though it’s easy to assemble a few services to build a Proof of Concept (PoC), developing the actual product on top of this shaky (hacky) foundation without a design could end in disappointment.
In my view, three main areas augment the case of why we need Solution Design in Serverless.
Technical agility
Engineering clarity
Business Visibility
Technical agility
There is a difference between experimentation with serverless and adopting serverless. Successful experiments can lead to adoption.
When we adopt serverless, we should consider sustaining the solutions we build and operate for the foreseeable future. I like to call this sustaining rather than maintaining. Sustainability does not come out of the box (OOTB). It requires constructive thoughts and best practices. The Solution Design process enables the design and building of technically agile applications.
When I say technical agility, I mean the ability to change, the flexibility to extend, the possibility to upgrade, and the simplicity to operate. These are essentials for serverless, just picking up its adoption momentum.
Technical agility is the ability to change, the flexibility to extend, the possibility to upgrade, and the simplicity to operate.
Engineering clarity
Many a time, we have experienced the following in our professional lives!
Incompatible interfaces, broken domain boundaries, leaking credentials, sensitive data breaches, open security holes, and several other incidents occur due to ignorance and lack of clarity while building.
Performing Solution Design can address these mistakes by provoking the engineering minds. This process will challenge the judgments, question the choices, and engage in conversations to perfect the solution.
Serverless has best practices, many patterns, and proven architectural constructs. The ingenuity lies in bringing all these together and proposing a solution that results in less ambiguity during implementation.
The Solution Design process will challenge the judgments, question the choices, and engage in conversations to perfect the solution.
Business Visibility
The days businesses set up many monitors in dark dungeons of engine rooms to keep multiple eyes on system status are slowly changing.
These engine rooms exist within every product team, not as dark dungeons but as engaging dashboards alongside engineers’ screens and mobile devices. Teams now own and operate their services.
Observability and traceability are key requirements of every serverless application. The granularity of serverless microservices enables us to get deeper visibility on the metrics.
For business insights, adequate measures must be incorporated into the architecture and the application.
The granularity of serverless microservices enables us to get deeper visibility on metrics and business insights.
Solution Design Thought Process
It’s a balanced act. We should consider all the serverless best practices and comply with any organizational principles and guidelines.
For example, the use case may suit the DynamoDB Single Table Design. However, you may have an internal principle that advises otherwise!
Here are a few guidelines for designing solutions.
Understand the requirements
The process starts with a thorough analysis of the product’s requirements. The functional, non-functional, business, operational, infrastructure, and data security requirements should be given equal importance.
Remember that the requirements change in an agile environment, and so does the design!
Team ideation session
Gather everyone (not just engineers) from the team for an open session to trigger ideas. If it is a larger team, perhaps include those who are closely associated with the product.
Identify and group the ideas such as business logic related, security, data specifics, platform, monitoring, etc. The details collected here will feed into the subsequent stages of the process.
Domain, contexts, and boundaries
If you’ve been introduced to Domain-Driven Design (DDD), it will naturally reflect in your thinking. If not, get some understanding of DDD and the concepts around it. It will become helpful as you identify events, API contracts, datasets, and service boundaries, among other things.
Not necessarily every serverless development will stretch the entirety of a domain. It may be for a feature within a domain that already has microservices in operation.
Think events and event-driven services.
Events are first-class citizens in serverless. They fuel the event-driven architecture. We can compare the role of events in serverless to the power of steam in a steam engine.
Events drive the flow of execution. Events connect the distributed microservices. Events carry crucial information. Events are everywhere in a serverless application. Hence, identifying events is part of the Solution Design process.
Ben Ellerby has written a great guide on how to perform event storming in serverless.
While defining events, think like an event producer. Don’t get influenced by the potential subscribers of the events. Event filters, routing rules, and other event gates control what flows into a subscriber. It is not the responsibility of the event producer.
Follow an event template that suits your domain. Uniformity in the structure will be beneficial.
Events drive a serverless application as steam powers a steam engine. There’s no drive without power. Hence, event-driven!
Serverless patterns
Patterns offer solutions to common problems. Several architectural constructs in serverless are already available as patterns, and there are also infrastructure patterns.
Remember this Super-DRY rule!
Do not Repeat Yourself, but Do Reuse Yourself!
Security
Think and incorporate security measures right from the start of the design process. As you progress with the design, your application’s attack surface should become defined. A good grasp of the Least Privilege Principle and Shared Responsibility Model is essential.
Security is everyone’s job now, not just the security team’s — Dr. Werner Vogels.
Data security is often overlooked. While defining data stores and transport, identify what level of protection is needed. It includes restricting access to tables, encrypting the data payload, redacting data from the logs, etc.
In addition to securing the APIs, set quota limits to prevent the unintended flood of invocations.
Think granular microservices
One thing I like about serverless is the granularity. A serverless microservice doesn’t need to span the entire domain or sub-domain. It is quite possible to design and build several smaller, granular, independent, and event-driven microservices.
Here is an example that depicts the view of granular microservices as part of a Loyalty System. They belong to the same domain and abode within its boundary.
The above are just a handful of ideas. Several others go into a Solution Design. It’s hard to identify and pack everything here.
The basic rule is, if there is something that brings clarity to the implementation, then consider it and add to the Solution Design.
Continuation
I hope this article has given an overview of the need for Solution Design in serverless developments. In the subsequent article, we will see how to capture the design thoughts in a technical document and have a practical understanding of the entire process.
Fabulous article!