Revive Your Well-Architected Thinking With SCORPS!
An architecture review process that suits busy serverless engineers!
Currently, I am reading a few books on writing. Though I write blogs and co-authored a book for O’Reilly, I’ve not received formal coaching in writing.
A common theme in these books is the value of readers’ time in the modern world. The advice, therefore, is to write just enough stuff and cut all the noise!
Writing changes because we change! (from Everybody Writes by Ann Handley)
Understanding The Well-Architected Framework
Eight years ago, while exploring the AWS Certification, I came across the AWS Well-Architected Framework. It had five pillars: Security, Cost Optimization, Operational Excellence, Reliability, and Performance Efficiency, with the Sustainability pillar added later.
To score high in the exams, one must deeply understand each pillar. For reference, I maintained a physical folder with the printed content for each pillar. That was then.
As I write, the following are each pillar’s current PDF page counts.
Security — 200 pages
Cost Optimization — 148 pages
Operational Excellence — 189 pages
Reliability — 276 pages
Performance Efficiency — 118 pages
Sustainability — 86 pages
The limited reach of the framework in the engine rooms
Everyone who architects, builds, and operates applications on the AWS cloud knows the undeniable benefits of the AWS Well-Architected Framework. However, the familiarity of it is not uniform across teams and titles.
In many organizations, architects read, understand, and apply well-architected principles. For an architect, having a good grasp of the AWS Well-Architect Framework is part of their curriculum, and they dedicate the time (and money) to getting acquainted with it.
However, engineers in the engine rooms have several responsibilities in their daily work. They have problems to ideate, solutions to build, ceremonies to attend, operational issues to deal with, and social activities to participate in. Unless engineers prepare for AWS Certification, they rarely allocate time to reading a six-volume and 1000-page framework.
The ignorance is more in serverless teams
Serverless Development on AWS calls serverless engineers multiskilled and diverse. Still, there are more reasons for serverless engineers not to learn the well-architected framework.
Serverless engineers primarily compose their applications using managed cloud services. Due to this, their proximity to AWS’s core computing, storage, and networking capabilities is limited. This is a barrier to understanding many principles and practices in the well-architected pillars.
For a serverless engineer, the amount of (relevant) information in the framework for their specialization of the cloud ecosystem can be overwhelming.
Thus, the closeness of serverless engineers and the Well-Architected Framework is far from cordial.
Serverless engineers rarely allocate time to reading a six-volume and 1000-page well-architected framework.
Thank Goodness For The Serverless Lens
In November 2017, AWS released the Serverless Application Lens to ease the application of the Well-Architected Framework in serverless — well-architected pillars’ goodness served as a single capsule one-tenth in size.
Serverless was still in its infancy, and it took time for the Serverless Lens to get noticed and adopted across teams.
Serverless Lens found its place in the engine rooms
With the increasing popularity of Function as a Service (FaaS) and AWS Lambda, serverless began to make inroads into many organizations.
Serverless and the following elements changed the dynamics of the teams and roles.
The recognition of Domain-Driven Design in tackling organizational complexity and team boundaries — bounded contexts.
The benefits of Microservices as the pattern for developing loosely coupled distributed applications.
The simplicity of Amazon’s two-pizza team concept advocated by Jeff Bezos.
The Spotify squad model of autonomous cross-functional product teams.
As serverless teams owned and operated their services, engineers also became involved in architecture decision-making. The Serverless Lens questionnaire checklist helped engineers audit their applications according to well-architected principles.
Thus, the Serverless Application Lens found its place in serverless teams.
Serverless Lens acting as a service audit measure
The initial version of the Serverless Application Lens included ten well-architected questions, each with multiple checklist items.
Engineers compiled them into a template and shared it with the teams. The following are sample questions and checklists for the security pillar.
With this, teams conducted a simple process of well-architected review before deploying an application to production.
The process worked well and improved quality. It initiated discussions and led engineers to explore the Serverless Lens and the Well-Architected Framework.
Soon, the excitement faded away
After the early enthusiasm, for some reason, the interest slowly faded away. Here are my observations on why:
Auditing a service with a checklist is a one-time activity. Unless there are substantial future changes to the service, it is unnecessary to review it again.
Once an audited service is deployed, any operational issues are handled through incident management, postmortem, and tickets.
As a team gains experience, well-architected principles are incorporated into the development process, making the well-architected review less significant and redundant.
New engineers follow their teams’ development processes, which include well-architected principles. As a result, many become unaware of the framework.
Uniformity in adopting well-architected principles across autonomous teams can be challenging in a growing organization. It requires someone familiar with the principles work across the teams. In the article Grow your serverless teams, I mention a serverless enabler role for this purpose.
So, we condensed the 1000-page, Well-Architected Framework into a 100-page Serverless Application Lens.
The 100-page serverless lens then became a 10-point questionnaire.
Still, many serverless teams are not fully engaged with the AWS Well-Architected Framework or the Serverless Application Lens.
What’s the alternative? Can we improve the situation?
The Serverless Application Lens is the Well-Architected Framework’s goodness, served as a single capsule one-tenth in size!
Working Backward
Once, during a casual team chat:
Me: This month’s cloud bill has gone up by X dollars.
Engineer 1: How do you know?
Me: Oh! I received a billing alert, then looked at the console.
Engineer 2: How much is it?
Me: Y dollars.
Engineer 2: Sigh! I don’t have access to this account to check.
Engineer 1: Same.
If most engineers are unaware of their operational cost, how can we teach them about Cost Optimization?
How can we make engineers think about Operational Excellence, Performance, Security, and so on?
The answer is — we must change our approach to working backward!
Start with something interesting (such as knowing the cost of operation) or something that hurts (such as a production incident) and tread back to map these topics to the respective well-architected principles and practices.
SCORPS — To The Rescue
The SCORPS process is from The Value Flywheel Effect: Power the Future and Accelerate Your Organization to the Modern Cloud (IT Revolution Press, 2022) by David Anderson, Mark McCann, and Michael O’Reilly.
SCORPS stands for Security, Cost Optimization, Operational Excellence, Reliability, Performance Efficiency, and Sustainability.
The Value Flywheel Effect describes SCORPS as a collaborative team activity in which you regularly review your architecture with the well-architected framework — a well-architected review!
SCORPS is a continuous activity you conduct every two weeks (preferred), month, or quarter. It is not a one-time audit.
I recommend reading The Value Flywheel Effect to conduct the SCORPS review process in an organization with multiple teams. For brevity, I will adapt the process in a single serverless team context.
Conducting SCORPS in your serverless team
Most serverless teams follow an incremental and iterative development process.
Your team’s long-term business goals are divided into quarterly goals (OKRs—Objectives and Key Results), which are then divided into two-week goals, referred to as sprint goals.
At the end of this period (sprint), the team reviews its achievements (sprint review) before identifying the goals for the next sprint.
1. Make SCORPS a mandatory ceremony: Append to your sprint review or dedicate separate time for the well-architected review. Make it part of your team’s ways of working.
2. Discuss the highs and lows of your architecture about each pillar: Review each well-architected pillar and discuss the team’s good and bad observations. Use the following as an idea guide.
Security
Reported or observed security incidents
Recognize secure measures implemented by the team
Highlight any threat modeling work completed
Cost Optimization
Share the cloud cost of PROD and other environments
Discuss cost variation since the last meeting
Highlight the main cost influencers
Operational Excellence
Celebrate the number of deployments
Discuss engineer productivity-related matters
Highlight observability improvements
Reliability
Review of API usage versus quota
Discuss any service scalability and outage issues
A quick round-up of the status of all the services in general
Performance
Noticeable or reported performance degradations and improvements
Critical API latency
Sustainability
Recognize sustainability-promoting designs, decisions, etc.
Highlight unused resource cleanup, data expiry mechanisms, etc.
Review data lifecycle policies
3. Record your findings and actions: Once you have a structure for conducting a SCORPS review, you can record the highlights in a document and raise tickets for any identified actions.
Here is a sample template.
4. Encourage discussions and references to the well-architected principles: The main goal of SCORPS as the well-architected review process is to create a space for everyone to share their experiences and learn about the AWS Well-Architected Framework principles and practices. It will promote further conversations among engineers while building new solutions, handling incidents, etc.
How many serverless engineers know the existence of AWS Well-Architected Tool and its purpose?
Be Well-Architected!
Every serverless engineer should aim to understand the AWS Well-Architect Framework well. Every serverless team must provide an ecosystem to foster well-architected thinking among its engineers.
Everyone’s life is indeed busy, and everyone’s time is precious.
As a serverless engineer, you must choose the approach that suits you rather than distancing yourself from the voluminous, well-architected framework.
Your knowledge of the AWS Well-Architected Framework and the Serverless Application Lens is crucial to becoming a successful, accomplished serverless engineer.
And your ambition must be to become Well-Architected!
Further reading:
How to make well-architected reviews work for your organization by The Serverless Edge
SCORP — the well-architected tool for Architecture Reviews by The Serverless Edge
The Value Flywheel Effect: Power the Future and Accelerate Your Organization to the Modern Cloud (IT Revolution Press, 2022)