What are the Fundamentals of Data Engineering?

Definition of Data Engineering

Data engineering is a process of designing, building, and maintaining systems that make it possible to collect data, store it, analyze it, and make decisions based on it.

It’s one of the so-called ‘data providers’ jobs, as its purpose is to make data accessible to other data users (e.g., data analysts, data scientists, ML engineers) while ensuring data quality, accuracy, and format suitability.

To go deeper into the job description, read the What Does a Data Engineer Do article.

Understanding the Fundamentals of Data Engineering

Data engineering will be best understood by understanding its fundamentals.

Understanding the Fundamentals of Data Engineering

Fundamental #1: Data Sources and Ingestion

Data engineers typically pull data from many different sources and store it in one place, such as a data warehouse. This process is called data ingestion.

There are several different types of data sources, and data ingestion methods and tools.

Data Source Types

Data sources fall into one of these three categories based on data type.

Data Source and Data Ingestion Fundamentals of Data Engineering

1. Structured Data Sources

Structured here means data follows a predefined schema that organizes data in tables consisting of rows and columns. Each row is a data record, while each column is a data attribute. If that sounds awfully like a definition of relational databases, it’s no surprise, as they are one of the structured data source examples.

Structured data is suitable for representing data that requires high data consistency and the efficiency of complex querying.

Understand Fundamentals of Data Engineering For Structured Data Sources

a) Relational Databases

As we already said, relational databases organize data in tables. Each table stores data for a specific entity type, such as customers, employees, or orders.

For the employees table, each row would represent one employee, and the columns like first_name, last_name, date_of_birth, and address are the employee’s attributes.

Data in relational databases is managed by Relational Database Management Systems (RDBMS), which employ the SQL programming language.

The most popular RDBMSs are:

b) Customer Relationship Management (CRM) Systems

These are systems that store detailed data about customers, such as personal information, contact details, order history, status, interaction history, etc. As you would suspect, they are used to manage customer relationships and sales and create personalized promotions.

The examples of CRMs include:

c) Enterprise Resource Planning (ERP) Systems

ERPs’ purpose is to integrate business processes within a company. It basically collects data from various departments, such as Finance, Manufacturing, HR, Order Management & Inventory, and integrates it into one repository. The purpose of this is to streamline operations, improving collaboration between departments, data accuracy, and decision-making.

Popular ERPs are:

2. Semi-Structured Data Sources

The semi-structured data employs some level of data organization (it uses tags or markers to separate data), but at the same time there’s no fixed schema like in structured data. This makes such data something between structured and unstructured data.

There are four main sources of such data.

Understand Fundamentals of Data Engineering For Semi Structured Data Sources

a) JSON Files

JavaScript Object Notation (JSON) is a data format derived from C-family languages (it includes Java) conventions, but is language-independent. Data in JSON files is organized in name-value pairs or an ordered list of values.

b) Extensible Markup Language (XML) Files

XML files are flexible text data sources typically used for exchanging data over the Internet. Web services and APIs often use this format. XML files store data and metadata, with tags providing context. They use markup language to provide the structure of data.

c) HTML Documents

This is another data source based on the markup language; this time, it’s HTML or HyperText Markup Language. It’s a standard language for creating web pages, and it contains information about the page layout and its content, allowing web browsers to display all that data as a web page you see in a browser. Like XML files, the HTML documents also use tags to structure data. The content to be displayed on the webpage is placed between these tags.

d) Emails

Emails are typical representatives of semistructured data. The structured elements of an email are defined by email protocols, e.g., SMTP (Simple Mail Transfer Protocol). These elements are sender (From:), recipient (To:), subject, date & time, reply-to, message ID, and attachments.

The unstructured elements of an email are the body text, inline images, and media.

3. Unstructured Data Sources

Unstructured data is a data type that lacks a predefined data model and is not systematically organized.

Here are some of the most common examples of unstructured data sources.

Understand Fundamentals of Data Engineering For UnStructured Data Sources

a) Text Documents

These include text processing files (e.g. Word or Google Docs documents), PDFs, and other sources containing text data in a freeform.

b) Social Media Posts

Social media posts from platforms such as Facebook, X, Instagram, or LinkedIn include various data types, such as text, images, videos, hashtags, user mentions, etc.

c) Videos

They contain audio and visual content, which have to be stored and analyzed separately.

d) Images

These include photos, graphics, and other visual data.

Methods of Data Ingestion

There are two main ways of ingesting data.

Data Ingestion Methods To Understand Fundamentals of Data Engineering

1. Batch Processing

Batch processing means data is collected and processed at scheduled intervals, e.g., at the end of the day. This approach is used when the instant access to data is necessary. In the example of daily batch processing, it means that the latest data available to data users will be from the previous day.

This method is simple compared to real-time data processing. It’s also efficient, as it allows optimizing resources by processing data in bulk.

Credit card billing, payroll, system backups, and financial data are examples of data that is usually processed in batches.

2. Real-Time Streaming

This is a more complex approach to data processing which involves continuous data collecting and processing as soon as it becomes available.

Real-time streaming allows immediate data insights and improves an organization’s responsiveness to events.

Stock market data, retail inventory management, IT system monitoring, fraud detection, social media feeds, and location data for GPS systems are typical examples of real-time streaming.

Tools and Technologies for Data Ingestion

Here are some examples of tools that excel in batch data ingestion:

Informatica PowerCenter – a data integration platform that supports batch data ingestion and processing from various sources
Talend – batch data ingestion, data migration, and synchronization tasks
Apache Flume – designed for collecting, aggregating, and moving large amounts of log data
Apache NiFi – automating data flow management and batch data ingestion from various sources to multiple destinations

The tools commonly used in real-time streaming ingestion are:

Apache Kafka – a distributed event streaming platform for building real-time data pipelines and streaming applications
Amazon Kinesis – allows continuous capture, processing, and analysis of streaming data
Apache Nifi – it’s also suitable for real-time data streaming, as it has extensive support for various data formats and protocols
Airbyte – supports real-time data ingestion with a range of connectors for different data sources

Fundamental #2: Data Storage and Management

After data is ingested, it has to be stored somewhere, and for that, we use data storage systems.

Data Storage Systems

There are three primary data storage systems.

Understand Fundamental for Data Storage Systems For Data Engineering

1. Databases

Databases are collections of data organized in a way that allows efficient data storage, management, and querying.

They are ideal for transactional systems (e.g., banking systems, CRM, e-commerce platforms), which require fast processing of data that is consistent and of high integrity.

When talking about databases, this usually means relation databases – data is stored in tables using predefined schemas. Relational databases are used for storing structured data.

We have already learned that the tools used for managing data in relational databases are called Relational Database Management Systems (RDBMS) and listed the most popular ones

However, non-relational (or NoSQL) databases are also commonly used. They are used for storing and managing semi-structured and unstructured data. Here are several main types of NoSQL databases and common tools:

MongoDB, CouchDB – document-oriented databases
Redis, DynamoDB – key-value stores
Apache Cassandra, HBase – wide-column stores
Neo4j, Amazon Neptune – graph databases

2. Data Lakes

These are centralized data repositories that store large amounts of raw structured, semi-structured, and unstructured data in their original format. They are used in big data analytics, machine learning, and AI.

These are several common data lakes:

3. Data Warehouses

Data warehouses are, just like data lakes, centralized data repositories. They, too, integrate data from various resources. However, unlike data lakes, data warehouses are designed for storing structured (and sometimes unstructured) data optimized for querying, analysis, and reporting.

The above definition makes data warehouses typically used for storing historical data and business intelligence (BI).

Choosing Appropriate Storage Solutions

The best storage solution is the one that best fits your needs. In finding it, you should consider several factors, such as the data used, its volume, storage scalability, planned storage use cases, and, of course, cost.

Choosing Appropriate Storage Solutions For Fundamental of Data Engineering

Data Governance and Security in Data Storage

Managing data and ensuring its security has become an increasingly important topic in recent years, recognized through regulations such as the EU’s General Data Protection Regulation (GDPR) or the California Consumer Privacy Act (CCPA).

Data governance refers to a set of policies and procedures within an organization that ensure data integrity, availability, and usability while protecting data privacy in line with the regulatory requirements.

Efficient data governance usually includes these elements.

Data Governance and Security in Data Storage For Fundamentals of Data Engineering

1. Data Governance Framework: Outlines the roles, responsibilities, and processes for managing data.
2. Data Policies and Procedures: They cover data management practices, including data quality, data privacy, data lifecycle management, and data usage.
3. Data quality management: Measures for securing data quality, including data audits, cleansing, and validation.

Data security refers to protecting data from unauthorized access, security breaches, and other similar threats. Its goal is to ensure the data remains confidential and available.

It typically involves these security measures.

Data Security Measures For Fundamentals of Data Engineering

1. Access Control: Only authorized users can access sensitive data.
2. Data Encryption: Protecting data in transit and at rest by encoding it into a format that can only be decrypted by personnel with the appropriate encryption key.
3. Regulation Compliance: Ensuring compliance with the regulations such as GDPR or CCPA.
4. Incident Response Plan: This involves having procedures in place for detecting, responding to, and recovering from security breaches.

Fundamental #3: Data Processing and Transformation

The data you ingest is often incoherent, incomplete, and inconsistent. For it to be usable in analysis and insight-gaining, it has to be processed and transformed.

Data cleaning and preprocessing are crucial steps in doing so.

Data Preprocessing Techniques

Data preprocessing encompasses transforming data into a usable format. It usually involves these techniques.

Data Preprocessing Techniques For Fundamental of Data Engineering

1. Data Cleaning: It means you remove errors, inconsistencies, and inaccuracies from data.
2. Data Normalization: This is a process of scaling numerical data to a standard range (e.g., from 0 to 1) so all features contribute equally to the analysis.
3. Data Transformation: This process refers to mathematically transforming data (e.g., using log transformation) to make data more normally distributed.
4. Encoding: This technique takes categorical data and converts it into numerical formats, e.g., one-hot encoding or label encoding. It makes such data readable by ML algorithms.
5. Data Aggregation: The purpose of this technique is to get some insights from data by aggregating it on several levels, e.g., daily or monthly totals, customer levels, order levels, etc.

While all these data preprocessing techniques are important, data cleaning is usually the most time-consuming and the most important.

Data Cleaning Techniques

There are several techniques commonly used when cleaning data.

Data Cleaning Techniques For Fundamental of Data Engineering

1. Error removal: This involves removing duplicate data, NULLs, and incorrect entries and imputing missing data.
2. Standardization: This technique refers to standardizing data formats, such as applying consistent date and time formats or categorical labels.
3. Outlier Detection: Outliers could skew the data analysis and insights, so it’s important to identify and address them in this stage.

ETL (Extract, Transform, Load) Processes

ETL process refers to extracting, transforming, and loading data. It’s a critical process in data integration, as its purpose is to gather data from various sources, transform it into a usable format, and load it into data storage for further usage.

Data extraction refers to gathering data from multiple sources, such as databases, APIs, and flat files. In this stage, the required data is identified, located in the data sources, and retrieved.

Data transformation in the ETL process means turning data into a usable format through data preprocessing techniques.

Loading data means the data is moved to the data storage systems, be it a database, data lake, or data warehouse, for further use.

Role of Data Pipelines in Automated Data Processing

ETL is often conflated with data pipelines. Generally, ETL is one type of data pipeline; namely, the one that processes data in batches.

Data pipelines are a broader concept that refers to steps that ensure data flow from its source to the final destination. They are designed to be scalable, reliable, and stream real-time data, which makes them essential in establishing automated data processing.

They are used to automate these tasks.

Role Of Data Pipelines For Fundamental of Data Engineering

Fundamental #4: Data Integration and Aggregation

Data integration refers to a process of gathering data from multiple sources into a single data source.

This ensures that data is consolidated so everybody in the organization works with the same data. As this involves data preprocessing and cleaning, it contributes to improving data quality, accessibility, and usability. This, in general, makes business more efficient, especially its decision-making process.

Data Integration Methods

You already learned that ETL is critical in data integration, but it’s not the only data integration technique.

Data Integration Methods of Data Engineering

We covered ETL in a separate section, so we’ll focus on the other two data integration methods.

Data federation means a virtual database is created, which allows users to access data from multiple sources as if it were a single data repository. This is the way of providing a unified data view without the need for actual, physical integration.

Here are the key benefits of data federation.

Data Federation Benefits for Fundamentals of Data Engineering

Another data integration method is API integration. An API, or Integration Programming Interface, is a set of rules, protocols, and tools that allow different software applications to communicate with each other. They act as a bridge between applications, regardless of their underlying technologies.

The API integration works in a way that when one application requests data from another, it does so through an API call. This call is processed, and the data is provided to the application that requested it.

These are the API integration benefits.

API integration benefits for Fundamental of Data Engineering

Strategies for Data Aggregation and Summarization

In data engineering, aggregating data means collecting and combining data from multiple sources into a single dataset. This allows data engineers to make data more manageable for analysis.

There are three fundamental data aggregation strategies.

Data Aggregation Strategies For Fundamentals of Data Engineering

1. Data Grouping: Organizing data into categories (or groups) based on the shared attributes. For example, sales can be grouped by time period, region, or salesperson.
2. Data Summarization: Data is condensed into a more compact form, highlighting main insights. This is by far the most common data aggregation strategy, so we’ll dedicate a separate section to it.
3. Roll-Up Data Aggregation: It summarizes data at increasing granularity. For example, sales might be aggregated on a daily level, then rolled up to weekly, then monthly, quarterly, and yearly basis.

Data Summarization Techniques

Common techniques for summarizing data are shown below.

Data Summarization Techniques for Fundamentals of Data Engineering

1. Averaging Data: Calculating the average (or mean) values, e.g., average sales per month, average salary per department, average order value, etc.
2. Summing Data: Calculating sums, such as total sales per month, total order value per customer, etc.
3. Counting: This means counting the occurrences of data, e.g., the number of transactions per month, the number of new customers per week, or the number of employees.
4. Min/Max: This refers to finding the minimum and maximum values in the dataset, such as the highest and lowest salary or the earliest and latest order.

Data Integration Tools

Here are some popular tools for integrating data.

Data Integration Tools for Fundamentals of Data Engineering

1. Apache NiFi: A data integration tool that automates data flow between different systems and supports a wide range of data sources and destinations.
2. Talend: An ETL tool that offers, among others, a suite of data integration apps. With it, you can connect, transform, and manage data across various systems.
3. Informatica: A comprehensive data integration tool with solutions for data integration, quality, and governance. It supports complex data workflows and integrates with numerous data sources and destinations.
4. Fivetran: A relatively simple data integration tool that automates connecting different data sources and loading the data into a data warehouse.
5. AWS Glue: A serverless ETL tool by Amazon that’s great for integrating data for analytics, machine learning, and application development.
6. Apache Spark: An open-source unified analytics tool for big data processing, known for its speed and seamless integration with other big data tools.

Fundamental #5: Data Quality and Validation

Data quality is a broader concept that refers to overall data accuracy, completeness, consistency, reliability, and validity.

Data validation is a narrower concept and one way of ensuring data quality; it ensures data accuracy and compliance with specific standards before it’s processed.

Importance of Data Quality

Data quality is extremely important for accurate data analysis and decision-making. We can refer to the popular garbage-in-garbage-out (GIGO) principle. No matter how sophisticated your analysis is, if you base it on inaccurate and incomplete data, be sure that your analysis will also be garbage, as well as decisions made based on it.

Techniques for Data Validation and Quality Assurance

Data Validation Techniques

The data validation techniques can be numerous, and here are the most important ones.

Data Validation Techniques for Fundamentals of Data Engineering

If some of those look familiar, it’s true – we already mentioned them in the data cleaning section. The relationship between data validation and data cleaning is that data validation is preventive as it tries to prevent data inconsistencies and accuracies. Data cleaning is a corrective measure since it deals with inconsistencies already present in the data.

So, back to data validation techniques.

1. Schema Validation: Ensuring data complies with predefined data types, structure, and relationships.
2. Format and Data Type Checks: Checking formats and types verifies that data is the right type (e.g., dates are date type, not text type) and format (e.g., dates are in the YYYY-MM-DD, not DD-MM-YYYY format).
3. Null and Missing Value Checks: Ensuring data completeness by checking for null and missing values.
4. Range Checks: A data validation technique that confirms data falls within a specified range, e.g., that e-commerce platform users are all above the age of 18.
5. Duplicate Detection: This technique deals with finding and removing duplicate data.
6. Consistency Checks: Checking that data is consistent across different datasets and systems, e.g, checking that the sales amount in the sales database is the same as in the inventory management database.
7. Source System Loop-Back Verification: It verifies that the data extracted from the system matches the original data; e.g., if the sales data is being migrated, you should check that the sales amount in the new system matches the sales amount in the old system.
8. Ongoing Source-to-Source Verification: A continuous process of comparing data between various systems.

Data Quality Assurance Techniques

These are the common techniques used in data quality assurance.

Data Quality Assurance Techniques For Fundamentals of Data Engineering

1. Automated Checks: Data quality checks can be automated via automation scripts and tools, which reduces the possibility of mistakes.
2. Manual Verification: It can be used in conjunction with automated checks to ensure the errors that might be missed by automated checks are caught.
3. Data Profiling: This means analyzing the data’s structure, content, and quality (e.g., its format, value distribution, missing values, duplicates, outliers, and inconsistencies). It helps identify issues with data before it enters and impacts the system.
4. Third-party Verification: Involves cross-checking data with external sources, e.g., checking that customers’ information matches the national identity databases.

Monitoring and Maintaining Data Quality Over Time

Ensuring data quality is a continuous process. New data constantly flows into the organization and is constantly moved between systems. So, data quality must be continuously monitored and maintained over time using these techniques.

Techniques For Monitoring and Maintaining Data Quality

1. Regular Audits: Conducting audits means systematically reviewing data to maintain consistency, accuracy, and completeness. If you do that regularly, you’ll always be on top of any potential data quality problems with data.
2. Automated Validation Process: If you automate your data validation process, this will decrease the manual effort (and errors) to maintain data quality. When the validation process is automated, it can continuously and independently deal with data quality issues, e.g., missing values, duplicate data, and inconsistencies. When the issue is detected, the system notifies the designated persons within an organization to solve the problem.
3. Monitoring Tools: Employing tools for data quality monitoring tools also allows continuous monitoring of data quality across an organization. These tools provide you with real-time dashboards and automated reports, showing the current status of data quality and any deviations from established standards. This allows for immediate reaction and prompt solving of data quality issues.

Here are some suggestions for the tools you could use:

Fundamental #6: Data Modeling and Analysis

Data Modeling Techniques

Three techniques are employed when modeling data.

Data Modeling Techniques For Fundamentals of Data Engineering

1. Conceptual Data Modeling: Outlines a high-level framework for an organization’s data structure. The approach is based on entity-relationship diagrams (ERDs) – they illustrate entities, attributes, and relationships between data. Since this is a conceptual model, it doesn’t deal with how data will be actually implemented in practice. This approach is typically used in the initial stages, when trying to understand business data requirements.
2. Logical Data Modeling: Adds more details to the conceptual model, like defining primary and foreign keys and constraints. However, it still doesn’t deal with the actual RDBMs this will be implemented in.
3. Physical Data Modeling: It’s database-specific and deals with the actual implementation of the logical data model. Here, you create an actual database schema and define how data will be represented and stored in an actual RDBMS. This model also takes into account performance, storage, and retrieval mechanisms. The visualization of the model adds details about triggers, procedures, and data types to the logical data model.

Here are some tools that are commonly used in data modeling:

Role of Data Modeling in Designing Databases and Data Systems

Data modeling plays three critical roles when designing and developing databases and data systems.

Data Modeling Role #1: Data consistency, integrity, and security are helped by clearly defining data relationships and constraints in the data modeling stage.

Data Modeling Role #2: Data models help ensure database performance, scalability, and ease of maintenance by organizing data logically and efficiently.

Data Modeling Role #3: Data models provide an unambiguous and understandable visual representation of data requirements, bridging the gap between business and technical stakeholders and improving mutual understanding.

Data Analysis and Visualization for Deriving Insights

Data engineers use data analysis to clean, transform, and model data. Data analysis helps them in making data accurate and consistent.

Data visualization helps in discovering data trends, outliers, and insights by showing them on charts, graphs, and dashboards.

Here are some data analysis and visualization tools data engineers often use:

On top of that, data engineers use these Python libraries for analyzing and visualizing data:

Fundamental #7: Scalability and Performance Optimization

An important part of a data engineer’s job is to manage the data system’s scalability and optimize its performance.

Challenges in Scaling Data Engineering Solutions

Scaling involves handling increasing data volumes and their complexity while ensuring the system’s (efficient) performance.

Scalability and Performance Optimization For Fundamental of Data Engineering

1. Increasing Data Volumes: With the growing data volume, data systems may struggle to process and store it.
2. Increasing Data Complexity: As the data volume increases, its complexity does so, too. This can involve the growing number of data sources or the necessity to deal with many data types, such as structured, semi-structured, and unstructured data. As the data complexity increases, it becomes more difficult to integrate it, maintain its quality, and ensure consistency across data systems.
3. System Performance: The challenge here is to ensure that the system performance doesn’t drop or doesn’t do so significantly as the data volume and complexity increase.

Techniques for Optimizing Performance of Data Pipelines and Systems

These are the main principles used for optimizing the performance of data pipelines and systems.

Fundamentals for Optimizing Performance of Data Pipelines and Systems For Data Engineering

1. Distributed Computing Frameworks: Tools like Apache Hadoop and Apache Spark use distributed computing to allow more efficient data processing of large-scale data and its high availability.
2. Cloud-Based Solutions: One of the main advantages of cloud-based solutions is their scalability and flexibility. The popular cloud platforms are AWS, Google Cloud, and Microsoft Azure.
3. Data Indexing: Another technique is creating indexes of frequently queried columns.
4. Data Partitioning: This technique involves splitting data into smaller datasets. Since each partition can now be processed separately, it can reduce processing time.
5. Caching: Cache is a temporary storage for storing copies of frequently used data. Using this technique also improves the system’s performance.
6. Microservices Architecture: Applying this approach means breaking down applications into microservices. This ensures that a surge in data volume in one service doesn’t impact the whole application.

Handling Big Data

You need special tools falling into these categories to handle big data.

Fundamental of Data Engineering For Handling Big Data Tools

1. Scalable Storage Systems: These tools can scale horizontally to accommodate growing data volumes. The common tools used are:

2. Distributed Processing: Reducing processing time by processing large datasets in parallel across multiple nodes is necessary for handling big data. Here are some distributed processing tools:

3. Advanced Analytics Tools: These tools can be used for real-time data streaming (e.g., Apache Kafka) and data querying and analytics, like data warehouses such as Amazon Redshift or Google BigQuery. They are considered advanced because they incorporate techniques such as machine learning, real-time data processing, complex event processing, data wrangling and preparation, and data integration.

Here are some other tool suggestions for real-time data streaming:

Also, several tools that you can use for data warehousing, apart from Amazon Redshift and Google BigQuery:

Emerging Technologies and Trends

Current Trends in Data Engineering

The hottest trend in data engineering is integrating AI and ML algorithms into data engineering workflow. These technologies can automate data workflows and uncover patterns and trends in data faster and more accurately than humans.

In addition, ML algorithms are getting integrated into data pipelines. By doing so, predictive analytics and decision-making become automated and more efficient.

Impact of Cloud Computing and Serverless Architecture

Cloud computing cannot be considered a trend anymore; it’s become commonplace in data engineering. These platforms, such as AWS, Azure, and Google Cloud, are efficient at storing, processing, and analyzing large datasets. This provided organizations with scalable and flexible solutions.

They did so at a fraction of the cost of a traditional on-premises data infrastructure due to their serverless architecture. Such architectures frees data engineers from managing infrastructure and allows them to focus on data solutions development and deployment.

Future Outlook for Data Engineering Professionals

With companies heavily investing in data infrastructure and focusing on real-time data processing, the demand for data engineering skills is expected to increase.

Besides traditional data engineering skills, future data engineers will increasingly have to be in the know regarding AI and machine learning as they become even more integrated into the data engineering process.

Conclusion

Data engineering is a very complex field, as shown by the length of this article, which only covers the fundamentals.

These fundamentals of data engineering include:

Data Sources and Ingestion
Data Storage and Management
Data Processing and Transformation
Data Integration and Aggregation
Data Quality and Validation
Data Modeling and Analysis
Scalability and Performance Optimization

Mastering these fundamental concepts is essential for your data engineering career growth. However, you shouldn’t stick only to theory. If you want to do well at your job interview, you should be familiar with many of the tools we mentioned here. Data engineer interview questions also heavily test your coding skills, so you should practice languages such as SQL and Python.

Take your time, start with these fundamentals, be dedicated, and we’re sure you’ll land a data engineer job.