Data Engineering: An Essential Skill To Weather The AI Storm!
Since AI relies on data, why aren't you boosting your career prospects by building data skills?
Thirty-five years ago, I wrote a small program to manage stock for the procurement team at my first employer, using a language called Clipper, a variant of the popular database management product of the time, dBase!
Those who worked with data in the 80s and early 90s would recall dBase, Clipper, FoxPro, and IBM Db2! The days before Oracle and Sybase ruled the Relational Database Management (RDBMS) market.
Fast-forward to today, and the myriad data products available, along with their purposes, are mind-boggling.
This data (r)evolution, from simple data management systems (DMS) that ran on microcomputers to the mammoth data systems of today distributed across hundreds or thousands of computers around the globe, is the magic of science and the power of technology.
The reason for such growth? Human need!
The demand for bigger and better systems.
Systems capable of capturing, processing, and storing more data than before!
Over the years, especially since the widespread adoption of cloud computing, data has become ubiquitous, like the air we breathe, everywhere.
Big Data, the term coined to describe the data explosion, quickly became the norm.
Yet many software engineers are ignorant of — or even avoid — data and data operations.
But here is an opportunity for many engineers, especially those who fear that AI could automate their jobs or make their roles redundant, to learn about an area essential to LLMs and data science models.
It’s about equipping yourself with skills that will save your career — and make you an agent of growth!
If the algorithm is the brain of an LLM, data is its soul. Even the best algorithm is useless without meaningful data.
Are YOU an Agent? series:
Part 1: Who made Aaron redundant?
Part 2: AI Boom or Career Doom?
Part 3: Are You A Systems Thinker?
Part 4: Fast Code, Fragile Systems!
Part 5: This article
AI Feeds on Data. You Grow on Data Knowledge
Like Aaron in part 1, many engineers are unaware — or in denial — of AI’s impact on their careers until it hits home.
Do you fear AI will replace or automate your job?
If so, what are you doing about it?
Why not plan ahead and move to an area essential to AI?
Life is like riding a bicycle. To keep your balance, you must keep moving. — Albert Einstein.
Given the pace of technological evolution, practitioners cannot stand still. If you do, you will be knocked down and run over by the next tech wave. Therefore, everyone — regardless of whether your role is in danger or not — must keep moving.
Data is as essential as the air we breathe
In this digital age, everyone, not just software professionals, handles data. It’s like the air we breathe; we do it unconsciously.
Here are some examples of engineers being oblivious to data operations.
A software engineer writes a function to connect to a database table and query data.
An infrastructure engineer writes a script to provision a data bucket to store clickstream events.
A QA engineer prepares test data and stores it in files or tables.
A site reliability engineer (SRE) traces through the realms of logs, investigating a production issue.
An AI engineer writes a detailed context-aware prompt and maintains it in GitHub.
The list grows when you consider how architects, analysts, managers, and C-suite executives work with data in different shapes and forms.
Although there are data specialist roles such as Data Engineer or Data Administrator, you don’t need to chase coveted titles. What matters more is the knowledge you possess and your expertise.
It’s about equipping yourself with skills that will save your career.
To equip you with data and data awareness, this article focuses on three essential data concepts and how each presents unique opportunities.
Data Lifecycle
Data Pipelines
Data Stores
1. Data Lifecycle
As with many things in the world, every piece of data has an origin and a journey. The length and duration of this journey, however, depend on the path or pipeline it takes. Similarly, the end of the journey depends on its purpose, as some are deleted sooner, while others may remain longer.
For example,
Your one-time password (OTP) will get deleted in a few minutes.
Your browsing history or clickstream data may stay for weeks or months.
Your land registry data will be kept forever!
The business rules, regulatory obligations, and data usage requirements shape the phases of the data lifecycle.
Harvard Business School divides the data lifecycle into 8 stages, as shown below.
Generation — the birth of data from good and bad sources, producing useful and useless data.
Collection — mode (manual, automated), volume, speed (real-time, batch), etc.
Processing — clean, enrich, transform, encrypt, etc., as required. The Data Pipelines section below covers processing pipelines.
Storage — the location, medium, type, etc., discussed further in the Data Stores section below.
Management — the process of organizing, storing, retrieving, controlling access, and maintaining change logs.
Analysis — gather insights and identify outliers in data using statistical models, algorithms, machine learning (ML), etc.
Visualization — data representation using graphs, charts, dashboards, etc., to improve understanding and communication.
Interpretation — the stage where you assess the impact and implications of the data insights you gained from the analysis and visualization.
Recognize when your data gold turns into data dust. Delete it before it piles up into a dune.
Why do data lifecycle skills matter to you?
My ability to talk about data stems from my early work with it.
Data engineering is a life skill.
A clear understanding of the data lifecycle will empower you to influence and guide your organization toward becoming data-smart and data-efficient.
Your ability to classify data — good vs bad, useful vs useless — at the time of collection can significantly reduce storage and processing costs.
Your data expiration policies can contribute to your organization’s sustainability targets, among other benefits.
Your guidance on privacy and protection can help your organization comply with regulatory requirements and prevent data breaches.
Your expertise in organizing data into features will provide your department’s ML models with meaningful input.
There are several other ways to be impactful. AI may assist with parts of the lifecycle, but it requires your intelligence to steer.
Data is the new uranium: incredibly powerful and amazingly dangerous! — Mark Pesce
2. Data Pipelines
Every software engineer knows the CI/CD Pipeline.
Data Pipeline? Not many!
Just as a CI/CD Pipeline moves code from a developer’s computer to production, a Data Pipeline moves data from sources to destinations.
A Data Pipeline is a series of processes that move data from one or more source systems to one or more destination systems.
Although data pipelines have existed for a long time, their importance has grown as technology has evolved. For example,
Data pipelines enable high-volume big data flows with real-time ingestion, filtering, and transformation.
Cloud computing, in many ways, expanded data pipelines by offering affordable storage and processing options.
ML and AI rely on pipelines to ensure that high-quality data is fed into the models.
In a data pipeline, data is cleaned, validated, transformed, enriched, and aggregated to meet business requirements.
The key components of a data pipeline are,
Data sources — applications, web, log files, devices, etc., where data is created.
Data ingestion — the process of collecting and importing data into the pipeline.
Data transformation — filtering, cleaning, validating, and converting data into the desired format.
Data storage — tables, databases, data warehouses, or data lakes that store raw or processed data.
Data consumption — consumers of processed data for business analytics, dashboards, machine learning systems, etc.
The two popular types of data pipelines are:
ETL (Extract, Transform, Load)
ELT (Extract, Load, Transform)
ETL: Extract, Transform, Load
As shown in the following diagram, the ETL approach first extracts data from source systems, transforms it into the required format, and then loads it into the target system.
ETL is traditionally popular among resource-constrained systems.
ETL ensures that high-quality, structured data is stored while unnecessary data is not retained.
ETL is common in batch data processing.
ELT: Extract, Load, Transform
The main difference in an ELT pipeline is that raw data is stored and then transformed either within the system or at the point of consumption.
ELT is a relatively modern approach made possible by the highly scalable storage and processing power of cloud platforms.
ELT enables faster, high-volume, near-real-time ingestion of raw data.
Due to the availability of cloud resources, ELT is popular for multi-format data transformations.
Why do data pipeline skills matter to you?
Modern organizations rely on data-driven decision-making. To do this, they need a structured mechanism for moving and processing data. You can be the one to help your company automate this process.
Pipelines aren’t only for your code; they’re for data, too!
Here are a few reasons:
AI is only as good as the data it receives! You can be the engineer who ensures your team’s AI applications are built on high-quality data.
Businesses run on data flows, not just applications. Data visibility enhances your systems thinking and increases your value.
Data engineering requires a human in the loop. AI can help, but you can be the one with the business context for data governance and trade-offs.
Data pipeline skills will make you adaptable. Tech trends change, but data remains vital, and data skills offer you greater career resilience.
The data pipeline is not a niche or specialty but mainstream. In an AI-driven world, your ability to design data flows will make you far more valuable than a colleague who focuses only on writing code.
Data engineering isn’t about perfect data; it’s about building the perfect environment for data to thrive. — Benjamin Rogojan
3. Data Stores
For many, the data store feels like a mysterious box that sits in the back — not just back, but even further back, behind those shiny microservices!
Traditionally, we’ve been taught about the three-tier client-server architecture, with the data layer at the far end.
Like me, engineers at the end of the last millennium mostly worked with relational database systems (RDBMS), and the data language was SQL.
All changed, within a short span — thanks to the cloud, then AI!
Today, systems are more distributed than the stars in the sky. Data flows at nearly the speed of light. Data volume is as vast as the universe.
Therefore, how data is classified and stored has become as important as understanding algorithms and design patterns. Still, many software engineers are unaware of its importance and try to solve every data problem with a single tool: a relational database.
Popular data store types
Think of a relational database as one tool in the data store toolbox.
Not every data problem is a nail to be solved with a hammer. There are screws, nuts, bolts, and many other types.
Here are some of the most popular data stores.
Relational Database Systems (RDBMS)
The most popular and widely used for storing structured data.
Used in banking, e-commerce, CRM, and several other cases.
PostgreSQL, MySQL, Oracle, Amazon Aurora, and Microsoft SQL Server are popular relational databases.
Relational databases are commonly known as SQL (Structured Query Language) databases because they enforce a schema and use SQL to perform data operations such as create, read, update, and delete (CRUD).
Document Databases
Data store for self-contained documents in JSON or BSON format.
Used for content management, product catalogs, user profiles, etc.
Prominent ones include Amazon DocumentDB, MongoDB, Couchbase, and Azure CosmosDB.
Key-Value Stores
The simplest type that stores every piece of data as a key-value pair.
Used for caching, session management, config data, shopping carts, etc.
Amazon DynamoDB is a popular and high-performance key-value store.
Amazon DynamoDB — also known as a NoSQL database — stores unstructured and semi-structured data.
Graph Databases
Data is stored as nodes (entities) and edges (relationships) for efficient traversal.
Used to build knowledge graphs, social networks, and more.
Amazon Neptune is a popular vector database.
Vector Databases
Data is stored as vector embeddings that capture semantic meaning.
Designed for AI and machine learning workloads.
Pinecone is a popular vector database.
Object Stores
For large volumes of data in an unstructured format, called objects.
Ideal for cloud-scale data storage and management.
Amazon S3 (Simple Storage Service) is the most popular object store.
In addition, there are wide-column databases, time-series databases, data warehouses, data lakes, search engines, and other types.
For those new to data, I highly recommend Designing Data-Intensive Applications, 2nd Edition, by Martin Kleppmann and Chris Riccomini.
Why does data store knowledge matter to you?
Gone are the days when a company chose the most popular database on the market. Today, a single data store type cannot meet all the demands.
When every architectural decision in a modern software system eventually becomes a data decision, you, as an engineer, should be in a position to influence those decisions for the betterment of your organization.
Rather than asking,
“Which database is popular?”
You ask,
“What is the nature of my data, and how will it be used?”
Many system failures are data modeling failures, not code failures!
Hence, your expertise in data stores gives you an edge over others who focus solely on coding.
The more you understand the nature of your data — whether it’s structured, unstructured, transactional, analytical, connected, textual, time-based, or semantic — the better your recommendation will be.
Your data knowledge enables you to make justifiable trade-offs: data consistency vs. access speed, static vs. dynamic schema, performance vs. cost, short-term vs. long-term data, etc.
Your data integration skills are crucial for working with applications such as CRM, Personalization, Logistics, Analytics, AI, and many others.
Your ability to extract features from raw data and store them in a feature store will enhance your organization’s MLOps.
Although the current technology peak is dominated by AI, the software industry is broader and deeper than you imagine. Being a successful programmer does not mean you cannot equip yourself with data knowledge to be skilled and competitive.
Better prepared than sorry. You can be a data agent, too!
Data outlives code. A database is just a tool: how you use it is up to you. — Martin Kleppmann
Now that you appreciate what feeds algorithms and brings AI models to life, in the next article, I will discuss the importance of Model Thinking and why it’s essential for software engineers to go beyond prompts.