Capturing Solution Design Details In Serverless Developments — Part II
What does it contain, and who is it for?
Capturing Solution Design Details In Serverless Developments — Part II
My previous article on the importance of Solution Design in serverless development discussed its need for the following reasons.
Technical agility
Engineering clarity, and
Business Visibility
Nested among these is another one— Service Longevity. I often refer to this as the Sustainability of a serverless application (not to be confused with the new AWS Well-Architected Sustainability Pillar).
In a fast-paced agile environment, teams evolve, and engineers move around. Due to this, the domain and design knowledge fade from them. Storing the traces of what they went through and how they conceptualized the solution is vital. True, the source code repository plays its part, but it doesn’t explain the design unless captured and kept for everyone to understand.
This article introduces a documentation template to collate and keep the design details in one place.
Solution Design is a process of translating product requirements into architectural visualizations of granular microservices and engineering details to build an event-driven serverless application.
Serverless Solution Design Template
What I am describing below is a documentation pattern as a template. I do not want it to be cookie-cutter. Instead, consider it a reference and add your touches.
Though technical details dominate a Solution Design, its target audience is not just engineers. You should be comfortable taking it and traveling the architect elevator onto different floors and sharing it with stakeholders, domain experts, security teams, product owners, leaders, fellow architects, and other engineering teams.
The sub-headings below represent the sections in a Solution Design document.
Overview
This section briefs the purpose of the document. It sets the context, as in the sample text below.
This solution design is for a feature to resend account verification emails to customers to activate their accounts. The service is accessible via online as well as from apps across the stores. Add or refer to the Domain Vision Statement if you maintain one for the development.
Revision history
It is the place to keep a record of the changes. This section is particularly helpful if the design needs to be shared with distant teams or external partners.
Requirements
This section provides the details of the context. It refers to Business Decision Records (BDR), Architectural Decision Records (ADR), Jira tickets, stakeholder documents, etc.
Logical Flow
I like this section because it adds a non-technical perspective to the architecture so that everyone can understand the proposed solution. It’s for the logical design and structure of the application.
Use domain models, flow charts, sequence diagrams, user journeys, box diagrams, and similar tools without tech icons to capture things at a high level. Use narratives if necessary.
I used these diagrams while discussing the solution with the business and found them very effective.
Architecture
This section contains one or more diagrams. Depending on the solution's nature, it can include high-level and detailed architecture diagrams.
Following is an example of a high-level microservices view of a solution.
Detailed architecture diagrams follow the high-level services view. These diagrams are aimed at the engineers and contain cloud service icons and other notations.
The depth and details of these diagrams depend on the team and its serverless maturity level.
Design Details
This is one of the core sections of a Solution Design. The focus here is on the implementation details.
API contract details, quota limits, data validation needs, computation algorithms, workflows and orchestration, integration logic, caching, failure retries, etc., to name a few.
During the early days of serverless adoption, I used to add the microservice names, Lambda function names, resource limits, access policies, and other granular details. As the teams matured, naming conventions and standards emerged; these no longer feature in the solution design.
Make sure the design details incorporate the Well-Architected principles, serverless best practices, and your organizational standards and guidelines.
Data Modelling
I prefer to keep a separate section for data modeling, which will focus on two main areas: event structures and data storage.
As mentioned in the previous article, events drive the execution flow in an event-driven architecture. So, give them the attention they need.
Defining your events: Below is a simple event structure. The event storming exercise would identify the categories, types, and other details. Stating these, perhaps following a common structure, would make it easier to share events across services and domains.
{
"detail": {
"metadata": {
"version": 1.2,
"event_id": "some-unique-id",
"domain": "service-domain",
"service": "microservice-name",
"category": "category", // domain-event, internal, etc.
"type": "type of event", // data, alarm, etc.
"name": "name of the event" // account_created, email_sent, etc.
},
"data": {
"account": "account-number",
"rating": "account-rating"
}
}
}Data storage: There are a myriad of data storage options available, so spend enough time here. Thanks to the granularity of serverless, the design details of a particular instance within a specific type of data storage can be different. For example, the scaling needs of two DynamoDB tables need not be the same; two S3 buckets may have different life-cycle policies.
Below is a sample DynamoDB table design. The solution design can capture data access patterns, the composition of PK and SK values, the need for streams, DAX, etc.
Security and Threat Analysis
Always incorporate security from the start of this process and not as an afterthought. For simple services, add the security and threat analysis details here. Otherwise, perform a separate Threat Modelling exercise and add the reference.
Refer to the following articles on how to conduct Threat Modelling.
“Why Threat Modelling Should Be Part of Your Serverless Development.”
“A Few Steps To Conduct Threat Modelling in Serverless Applications.”
Trade-offs
This section contains the answers to the ‘whys’ and ‘why not’ in the design. Highlight any trade-offs considered, along with the reasons.
For example, any deviation from a defacto pattern would be valuable to the readers to know why. Or, you proposed to use an object store where a NoSQL data store might seem the obvious choice, so people would like to know why not.
Cost Analysis
This is an interesting section to have in a design document, but it can also be informative, especially for discussions with stakeholders on budget approvals, project costing, etc.
Estimating the cost of a serverless application from its design may sound challenging and futile. The idea here is not to count the dollars but to bring awareness and identify spending hot spots requiring a debate or justification.
I recommend that new serverless teams attempt to estimate the cost. This will enhance their understanding of the cost of operation (TCO).
Serverless cost estimation during solution design brings cost awareness and helps identify spending hot spots.
Appendix
Placeholder for all the references, diagram scripts, sample data, etc.
That’s a quick rundown of the common parts of a Solution Design document. A reviewed and accepted Solution Design provides clear guidance to the team that implements, configures, tests, deploys, and owns the solution.
As business demands change, the solution we have built will also change. Future advances in serverless technology can also cause changes to the solution. A good Solution Design enables the team to get through such situations easily. Eric Evans calls this Supple Design in his book Domain-Driven Design.
The Solution Design document plays a vital role as a living reference. Hence, it must be updated to capture and reflect all changes.
FAQ On Solution Design
Who is responsible for doing Solution Design?
To me, engineers are responsible for doing it for their team's work. They will work with engineering leads, architects, and domain experts, but their effort should go into it.
Depending on the complexity and the engineers' experience, one or two engineers can take the lead in implementing the design.
Solution Design is not something drawn up by IT architects in the ivory tower and descended to the team for implementation.
How long should it take to complete a Solution Design?
As a Solution Design mostly focuses on one product, feature, or microservice, it will take days, not weeks or months, to complete a Solution Design.
When handling a large feature or product, it is sensible to break it down and detail the initial parts first so that the implementation work can start while completing the rest.
“Visualize the whole, but Focus in parts.”
As the idiom states, See the forest first before Looking at the trees!
Where does a Solution Design document live?
It depends on the practice followed by the team. Some prefer to keep it in the code repository, while others maintain it as a confluence page or a document.
Irrespective of where we keep it, it should be accessible to everyone, including the non-technical members of the team as well.
Who maintains a Solution Design?
As it is designed and built by the team, it becomes the team's responsibility to maintain it.
Teams can keep a step to update the design docs as part of their development process and their ‘definition of done.’
What happens once a Solution Design is approved?
Once formally agreed upon —or approved, as in some organizations—it becomes an actionable entity. Engineers, along with their Product Owners, can identify the epics, create stories, and plan their development cycle.
Conclusion
“Speed matters in business” — Jeff Bezos
While we acknowledge the need for businesses to move fast to survive in this competitive world, it’s essential to have guardrails in place to prevent mishaps.
Performing Solution Design enables teams to transition from an amateurish, quick-and-easy approach to a planned-and-prepared one.
The process is not about following another ritual in your serverless application development cycle. Instead, consider it a guiding compass that shows you how to build lasting, supple, and sustainable serverless solutions!