PDF Summary:Fundamentals of Data Engineering, by Joe Reis and Matt Housley

Mastering the complexities of data engineering is crucial for any organization seeking to harness the power of data analytics, machine learning, and business intelligence. In Fundamentals of Data Engineering, Joe Reis and Matt Housley provide a comprehensive overview of this multifaceted discipline.

The authors guide readers through the data engineering lifecycle, from gathering data from diverse sources to transforming it into valuable insights. They discuss architectural principles for building scalable and secure data systems, evaluate cloud technologies and open-source tools, and explore methods for handling both batch and streaming data. Whether you're a beginner or an experienced practitioner, this book offers a solid foundation for navigating the ever-evolving data engineering landscape.

(continued)...

• Use a decision-making app like "Decide Now!" to help you make swift choices in areas where the stakes are low, thereby training yourself to value quick decision-making. For example, use the app to decide what to have for dinner or which movie to watch, reinforcing the habit of swift value delivery over prolonged deliberation.
• Create a "done list" at the end of each day where you write down all the tasks you've completed. This shifts your focus from what's left to be perfect to what you've already accomplished. Seeing a list of completed tasks can reinforce the value of progress over perfection.
• Create a "quality improvement suggestion box" in your workplace. Encourage employees to submit their ideas for improving quality in any aspect of the business. Review these suggestions regularly with a team and implement feasible ones. This not only taps into the collective intelligence of your workforce but also fosters a sense of ownership and engagement among employees.
• Experiment with a small, non-critical project to get comfortable with cloud migration processes. Choose a set of files or a simple application that you don't rely on daily and use a free or trial cloud service to migrate it. This way, you can learn about the steps involved, such as selecting the right tools, preparing your data, and managing the transfer without the pressure of impacting important data.
• Experiment with no-code or low-code platforms to prototype your ideas before committing to full-scale development. Platforms like Zapier or Airtable allow you to create functional models of the tools you're considering. This hands-on approach can reveal potential challenges or confirm the utility of the tool, helping you make an informed decision with minimal investment.
• Consider starting a small-scale project that would benefit from a managed connector and track the performance metrics before and after implementation. This could be as simple as setting up a new e-commerce platform that requires integration with your inventory management system. Monitor metrics like order processing time and error rates to measure the impact.

In evaluating the financial aspects, it's important to take into account the complete ownership expenses, continuous operational costs, and monetary operations.

The comprehensive financial obligation of a project, often referred to as the sum of all monetary and personnel contributions necessary for procuring and deploying a technology over its active duration, is understood as the Total Cost of Ownership (TCO). The Total opportunity cost of ownership (TOCO) includes the costs associated with the potential opportunities that are forfeited by choosing a particular technology for implementation. And FinOps is a new operational model for managing data technology spending, evolving monitoring to dynamically track costs and apply DevOps-like practices to financial accountability.

When choosing a technological or architectural solution, it is imperative for data engineers to conduct a comprehensive assessment of all associated costs. Choosing the seemingly most cost-effective solution can be unwise if it results in significant potential losses or requires hiring additional personnel. Investing in expensive technologies can sometimes be more cost-effective in the long run than depending on an open-source project that requires ongoing maintenance and support.

Context

• Accurate TCO calculations can aid in budgeting and financial forecasting, helping organizations allocate resources more effectively over time.
• The long-term implications of technology choices are crucial, as they can affect scalability, adaptability, and the ability to pivot in response to market changes or new opportunities.
• Implementing FinOps requires a cultural shift within organizations, promoting accountability and transparency in spending decisions across all departments.
• DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) to shorten the development lifecycle and provide continuous delivery. FinOps applies similar principles to financial management, promoting collaboration and iterative processes to manage costs effectively.
• The cost assessment should include how a solution's performance and efficiency can affect operational costs, such as energy consumption and processing time, which can indirectly influence financial outcomes.
• Lower-cost solutions might offer limited support, requiring companies to hire additional personnel to manage and troubleshoot issues.
• Investing in commercial solutions can free up internal resources, allowing teams to focus on strategic initiatives rather than maintenance and support tasks.

Selecting tools that facilitate the smooth and efficient movement of data.

The construction of data pipelines generally entails employing a range of technological tools. To achieve peak efficiency, it's crucial that the design of your system's components facilitates seamless communication and allows them to operate together with little initial configuration. Data engineers must have a wide range of abilities to acquire data, including proficiency in handling files, APIs, databases, and systems for distributing data, and they should also have an understanding of different data interchange formats like CSV, JSON, and Parquet. Technologies that are incompatible with current systems can create obstacles, slowing down progress and reducing the advantages accrued to the company.

For example, traditional ETL pipelines often exchange data using semicolon-delimited text. Whereas this approach may seem simple and efficient, it has a variety of drawbacks. Certain fields might include semicolons, potentially complicating the parsing process. The target system is responsible for precisely specifying the data's structure or reconciling any inconsistencies, given that CSV files do not inherently contain schema details. Modern data storage formats like Apache Parquet offer enhanced capabilities for managing nested data, evolving schemas, and various data types, and they are also more finely tuned for processing tasks compared to the CSV method.

Practical Tips

• Try using IFTTT (If This Then That) or Zapier to automate simple data tasks you perform regularly. For example, if you track your workouts and meals in separate apps, you can create an automation that logs your workout data into your meal tracking app to see how your exercise affects your dietary needs over time.
• Map out your daily interactions and identify communication bottlenecks. Take a week to note down every instance where you communicate with others, whether it's through email, phone calls, or face-to-face. Look for patterns where misunderstandings or delays frequently occur, and then brainstorm ways to streamline these interactions. For example, if you notice that email threads with your colleagues often lead to confusion, propose a daily 10-minute stand-up meeting to quickly align on tasks and responsibilities.
• Develop a habit of exploring open-source tools and software that facilitate data management. For example, use Python scripts to automate the process of file handling and data cleaning. Familiarize yourself with libraries like Pandas for data analysis, which can help you understand how to manipulate large datasets. Explore GitHub repositories that offer API interaction examples, and try to contribute or modify the code to suit your learning project.
• You can evaluate your current technology stack by creating a simple compatibility chart. Draw a grid and list all the software and hardware you use in two columns. For each pair, mark if they are fully compatible, partially compatible, or incompatible. This visual will help you identify which technologies are not working well together, potentially slowing down your progress.
• You can explore alternative data formats by converting a small dataset from semicolon-delimited text to JSON or XML using an online converter. This will give you a hands-on understanding of the structure and readability differences. For example, take a CSV file of your monthly expenses and convert it to JSON using a free online tool, then observe how the data is nested and whether it feels more intuitive to navigate.
• Create a simple pre-processing script in a spreadsheet program to replace semicolons with another delimiter before importing data. Even if you're not a programmer, many spreadsheet applications like Excel or Google Sheets allow you to record macros or use find-and-replace functions. You could record a macro that searches for semicolons and replaces them with a vertical bar or another unique character. This way, you ensure the data is clean before it enters the ETL pipeline, minimizing the risk of errors.
• Use online CSV validation tools before sharing your data to ensure it meets common standards. Before sending out a CSV file, upload it to a CSV validator that checks for common issues like missing values, inconsistent formats, or incorrect data types. This preemptive step can help you catch errors and make necessary corrections, reducing the burden on the recipient to decipher the structure and content of your data.
• Use Parquet's schema evolution capabilities to collaborate on a shared dataset with friends or colleagues, where each participant is responsible for updating or adding to the data. This could be a shared recipe book, a fantasy sports league with evolving stats, or a joint research project. Monitor how Parquet allows each user to make updates independently and how it maintains data integrity across different versions.
• Enhance your data analysis skills by taking an online course that includes a module on using Parquet. Look for courses that focus on big data technologies and include practical exercises. This will not only teach you the theoretical advantages of Parquet but also give you hands-on experience with the format, which could be beneficial if you're dealing with large datasets in your personal or professional life.

Assessing various offerings based in the cloud.

Cloud computing has transformed the approach to developing and deploying tools and technology for data engineering. The cloud's pricing model, which charges based on actual consumption and its ability to scale flexibly, in addition to its extensive range of functionalities, have made it the favored choice for burgeoning startups. Companies that previously relied on in-house tech capabilities are increasingly turning to services provided through the web.

The benefits and drawbacks of opting for cloud-based services rather than on-premises hosting solutions.

Utilizing cloud-based services provides multiple advantages over the use of on-premises servers. Businesses have the option to rent the required assets on-demand for their current initiatives and technological needs, rather than committing substantial capital to infrastructure and software in preparation for prospective future needs. The expenses related to cloud resources are generally tied to how extensively they are used. Advancements are being made toward greater automation in scaling resources, which allows for the dynamic and temporary increase of computing power, as well as the requisite storage and networking capacities, to support extensive projects and to reduce costs after their conclusion. Cloud domain service providers place a high emphasis on ensuring that their services are reliable and sturdy. Engineers can dynamically establish and decommission resources and infrastructure thanks to cloud architecture. An architecture that relies on long-running “special snowflakes” can be a disaster for system reliability and maintenance.

Certainly, the advantages of using cloud services must be considered alongside the limitations linked to the implementation within a cloud-based setting. Organizations frequently encounter unexpectedly high costs as they navigate the intricacies of setting up applications and services for cost-effectiveness, because understanding and adjusting the intricate cost structures of various cloud services can present difficulties. Continuously operating servers often result in higher costs within a cloud environment compared to the expenses associated with managing and operating personal servers. In addition, cloud providers often offer a closed ecosystem of products and services, and getting data off a platform through data egress can be extremely expensive.

Context

• Utilizing cloud services can be more environmentally friendly, as large cloud providers often invest in energy-efficient data centers and renewable energy sources, reducing the carbon footprint compared to traditional on-premises solutions.
• The complexity of cloud billing can be a challenge, as it involves understanding various pricing tiers, data transfer costs, and potential hidden fees, which can lead to unexpected expenses if not carefully managed.
• Automation in scaling helps in efficiently utilizing resources, reducing waste, and ensuring that the infrastructure is used to its full potential without over-provisioning.
• Comprehensive disaster recovery strategies are in place to quickly restore services and data in the event of a catastrophic failure, minimizing downtime and data loss.
• Many cloud providers offer tools and dashboards that help engineers monitor usage and costs in real-time, making it easier to manage resources efficiently and decommission them when they are no longer cost-effective.
• In computing, "special snowflakes" refer to unique, custom-configured servers or systems that are not easily replicated or replaced. These systems often require manual intervention for setup and maintenance, making them less efficient in environments that benefit from automation and standardization.
• Ensuring that cloud deployments meet security and compliance standards can incur additional costs for tools and services.
• Different service tiers offer varying levels of performance, availability, and support, which can affect costs. Choosing the right tier requires a clear understanding of the organization's needs and budget constraints.
• Closed ecosystems can make it difficult to integrate with third-party tools or services that are not part of the provider's offerings. This can limit an organization's ability to use best-of-breed solutions and may require additional development work to achieve interoperability.

Understanding the subtleties of cost and pricing models in environments based on cloud computing.

The authors recommend that individuals in the data engineering field should have a thorough understanding of the cost structure associated with cloud services to make the most efficient use of cloud resources. The cost structure is marked by a variety of complex and less obvious components, and in many ways, it resembles the methods used by traders to reduce risk in financial markets.

Vendors of cloud storage usually offer various storage options, each with distinct pricing structures. Vendors of cloud services provide a protective option designed for the extended preservation of data, based on the premise that customers rarely request access to their archived data and accept the possibility of incurring higher expenses for swift data recovery when necessary. Professionals in the field of data engineering must evaluate these various classifications, considering the possible compromises, and be ready to adapt as needed.

Other Perspectives

• The premise assumes that all data engineering roles require direct interaction with cloud resource management, which may not be true for all positions or projects within the field.
• The comparison to financial market risk reduction methods might be misleading, as cloud pricing does not inherently involve speculative elements and can be budgeted and controlled with proper planning and monitoring tools.
• The variety of pricing structures might lead to analysis paralysis for some customers, as they may struggle to determine which option is the most cost-effective for their specific needs.
• Vendors may not always provide the most cost-effective protective options for extended data preservation, as their pricing models could be designed to maximize profit rather than truly reflecting the infrequency of data access or recovery needs.
• The rapid pace of change in cloud services can make it challenging to stay informed about the latest storage classifications and cost models, which could hinder the ability to effectively evaluate and adapt.

To effectively meet future technological demands, it's essential to identify the optimal cloud deployment approach, whether that involves a single system or a combination of platforms.

The shift towards solutions centered on cloud technology is currently in its early phases. Some leading technology companies are opting to reintegrate specific cloud capabilities internally, while other enterprises are dedicating themselves to solutions that are completely reliant on cloud-based infrastructure. Organizations that do not have intricate or specialized data needs might want to explore alternative solutions prior to deciding on adopting strategies that involve both hybrid and multicloud due to their complexity.

Businesses in the early stages of developing their data infrastructure are frequently counseled by Reis and Housley to commit to a sole cloud service provider. The authors advise evaluating a mix of various cloud offerings or adopting a strategy that involves multiple clouds if it's evident that the business benefits surpass the costs, intricacies, and possible operational hazards.

Practical Tips

• Consider learning the basics of network security to better understand the implications of reintegrating cloud capabilities. You don't need to become an expert, but familiarizing yourself with the fundamentals through free online courses or tutorials can help you make more informed decisions about your personal data management and the potential risks and benefits of moving away from third-party cloud services.
• Explore cloud-based software to enhance your productivity and collaboration skills. Adopt a cloud-based task management tool or a document collaboration platform for your personal or volunteer projects. This will allow you to experience firsthand the benefits of real-time collaboration, remote access, and the integration capabilities of cloud-based applications.
• Assess your current data management needs by conducting a simple inventory of your data sources, types, and usage patterns. This will help you understand whether your organization's data complexity justifies a hybrid or multicloud approach. For example, if you find that your data is mostly stored in a few key systems and accessed by a limited number of applications, a simpler cloud solution might suffice.
• Develop a decision matrix to compare cloud service providers based on your specific business needs. Include factors such as cost, scalability, security features, and customer support. Score each provider based on these criteria to help you make an informed decision without being overwhelmed by the multitude of options.

Understanding when the Dropbox scenario is applicable to your specific circumstances.

Several widely cited case studies involving well-recognized technology companies, including Dropbox, Netflix, and Cloudflare, describe the substantial benefits these companies realized by moving away from the cloud and running certain key systems directly on their own hardware. Advocates for repatriation have been promoting a significant move away from internet-based centralized data storage systems.

The authors are of the opinion that such evaluations are frequently made too hastily, without considering the unique situations of the respective companies. These companies excel in niche markets through the development of unique software that integrates seamlessly with their specific equipment and foundational systems. They also produce data and deliver services at a scale that vastly exceeds the needs of more conventional businesses.

Context

• Companies like Netflix and Cloudflare have the technical expertise to manage complex infrastructure, which might not be feasible for smaller businesses without similar resources.
• By not relying on a single cloud provider, companies can avoid dependency on specific vendor technologies and pricing models, allowing for more strategic decision-making.
• Developing unique software allows companies to own the intellectual property, which can be a valuable asset and provide additional revenue streams through licensing or partnerships.
• Direct integration with hardware gives companies greater control over their operational environment, allowing for more precise tuning and troubleshooting, which can lead to improved reliability and uptime.
• Large companies often benefit from economies of scale, meaning they can reduce costs per unit by increasing production. This is because fixed costs are spread over a larger number of goods or services.

Deciding whether to construct in-house or purchase externally.

Deciding between building systems in-house or utilizing solutions from external providers is a crucial decision in the process of technological progression. Decisions should be based on financial factors, the skills present, and the ability of the technology to provide a unique advantage in the market.

Choosing between open-source, commercially supported, or proprietary technologies exclusive to certain vendors.

When acquiring technology, one often has to select from a range of familiar options such as open-source platforms, enhanced open-source services, and exclusive proprietary systems, which might be hosted on the cloud or come with specific vendor licenses. Deciding between these options is akin to assessing whether to develop in-house or purchase externally, considering the overall expenses and liabilities, including potential missed chances; opting for community-supported platforms with professional backing can greatly alleviate the challenges of independently operating a system, whereas solutions necessitating a unique license frequently lower the overall financial commitment.

Housley recommends employing well-known tools and systems when possible to prevent undue workload. In most cases, engineers should opt for open source or cloud-based solutions, reserving the selection of proprietary options for particularly distinct circumstances.

Practical Tips

• Reach out to peers or online communities for firsthand testimonials about their experiences with different technology types. Use social media, forums, or professional networks to ask specific questions about the pros and cons they've encountered. This real-world feedback can provide insights that go beyond theoretical knowledge, helping you make more informed decisions about which technology type might suit your personal or professional projects.
• Develop a mini-survey to gather opinions from colleagues or peers who have faced similar decisions. Ask them to share their experiences with in-house development and external purchasing, focusing on outcomes, challenges, and satisfaction levels. Use their insights to inform your decision, as real-world experiences can highlight considerations you might not have thought of on your own.
• You can create a "Missed Opportunity Ledger" to track potential opportunities alongside actual expenses. Start by listing your monthly expenses in a spreadsheet, then add a column where you estimate the value of any opportunities you've passed up. For example, if you declined a freelance job to focus on another project, note the income you missed. This will give you a clearer picture of your financial decisions' impact.
• Create a virtual mastermind group with individuals from various backgrounds but with similar professional aspirations. Use video conferencing tools to meet regularly and discuss challenges, share resources, and provide feedback. This collective approach can help you tackle personal and professional challenges with the support of a diverse community, and the professional backing comes from the collective expertise within the group.
• Look into community education programs that offer licensing as part of their curriculum at a lower cost than traditional avenues. Many community colleges or adult education centers provide courses in areas like real estate, insurance, or cosmetology that include the licensing exam fees in their tuition. This approach can save you money on the path to obtaining a professional license while also providing you with a structured learning environment.
• Implement a weekly "self-audit" to evaluate and adjust your workload management strategies. At the end of each week, take 30 minutes to review what tasks took longer than expected, what could have been done more efficiently, and what tools or systems helped you the most. Use this insight to refine your approach for the following week, such as delegating tasks that are outside your expertise or eliminating unnecessary steps in your workflow.
• Engage in role-playing exercises with friends or colleagues to practice identifying when a proprietary option is the best choice. Take turns presenting each other with various scenarios, ranging from personal to professional, and discuss the pros and cons of standard versus proprietary solutions. This activity will sharpen your ability to discern the distinct circumstances that merit a more tailored approach. For example, in a role-play about planning a vacation, you might explore when it would be better to design a custom itinerary yourself versus choosing a pre-packaged tour.

Understanding the differences between community-led open-source projects and initiatives managed by commercial entities.

Software is often viewed as an economical alternative to high-cost proprietary systems, but the reality is more complex than it seems. Organizations may face challenges when they embrace an open-source approach without a complete understanding of the related costs and trade-offs, and without a thorough knowledge of the available community support.

Teams should evaluate the community's engagement and the breadth of the project's development when choosing a community-driven open-source initiative, as industry professionals recommend. The broad utilization of the tool indicates that numerous groups have effectively overcome its obstacles in real-world settings. Support from experts is easily accessible within a vibrant community. Projects without significant community support can become difficult to execute and manage, or may ultimately be abandoned altogether.

Many products that are derived from community-driven open source initiatives are available in cloud-based formats managed by commercial entities for a periodic subscription cost. Data engineering experts should meticulously evaluate the overall expenses associated with ownership and operational costs while also considering the advantages, support, dependability, and ease of deployment when examining the free community edition.

Other Perspectives

• Some open-source projects are backed by large organizations or foundations, which can provide a level of stability and support that reduces the challenges for organizations looking to adopt these projects.
• Community support can sometimes lead to fragmented or inconsistent advice, which may not align with an organization's specific needs or standards.
• The vibrancy of a community can fluctuate, and today's active project could see reduced engagement in the future, potentially leaving teams without the support they anticipated.
• Widespread use of a tool can sometimes lead to a misconception that it is the best solution available, potentially stifling innovation or consideration of newer, more effective tools that could provide better outcomes.
• Vibrant communities can sometimes be overwhelming or intimidating for newcomers, making it difficult to seek out expert support.
• Some projects may start with minimal community support but can quickly gain momentum if they prove to be innovative or meet an emerging need in the market.
• The availability of cloud-based formats managed by commercial entities might imply a level of support and reliability, but it does not guarantee it; some commercial offerings may have poor support or reliability issues despite being paid services.
• While evaluating overall expenses is important, focusing solely on costs may lead to overlooking the strategic value and innovation potential that community editions can bring to an organization.
• The ease of deployment for a community edition might not translate into ease of scalability or security, which are essential for enterprise environments.

Evaluating the expenses and aspects related to customer service of proprietary products.

While proprietary solutions are attractive for various reasons-high performance, simple setup and installation, and streamlined support-they come with some significant limitations. Organizations, from small startups to major established firms, must adhere to the guidelines established by the service provider without exception. Moreover, costs can rise sharply if users are tied to extended procurement contracts with the suppliers.

The authors highlight a pragmatic approach to making choices between proprietary solutions and the range of other options. Begin by comprehensively grasping the expenses and constraints associated with every alternative. Evaluate the vendor's pricing and support, along with the strength and effectiveness of their business, to ensure that the anticipated advantages and assistance are likely to be obtained from their offerings.

Practical Tips

• Streamline your home office setup with a single-brand ecosystem. Choose a brand that offers a range of devices and services designed to work seamlessly together. This could mean getting a computer, tablet, smartphone, and even smart home devices from the same manufacturer, which can simplify setup and troubleshooting.
• Develop a habit of scheduling regular reviews with your service provider to stay aligned with their guidelines. Set up monthly or quarterly meetings to discuss any updates to the guidelines and how they affect your ongoing and future projects. This proactive approach keeps you informed and compliant.
• You can evaluate your current contracts by creating a 'contract calendar' that tracks expiration dates, terms, and conditions. This calendar will help you identify when contracts are up for renewal, giving you the opportunity to renegotiate or consider alternative suppliers before automatically renewing. For example, if you have a contract for office supplies that is set to renew in six months, set a reminder for four months from now to start researching the market for better deals.
• You can evaluate the true cost of ownership by creating a simple spreadsheet that compares proprietary solutions with open-source alternatives. Start by listing initial costs, ongoing maintenance, support fees, and potential scalability for each option. This will give you a clear financial picture over time, rather than just the upfront cost.
• Use a mobile app that tracks decision outcomes to record the real-world results of your choices. After evaluating expenses and constraints and making a decision, input the expected versus actual outcomes in the app. This will help you refine your evaluation process over time by showing you where your assessments were accurate and where they could be improved.
• Conduct a 'mock project' with a shortlist of vendors before committing to a long-term contract. This trial run can reveal potential issues with vendor support and business stability that aren't apparent in initial meetings or proposals. You could, for example, give each vendor a small, non-critical task and observe their performance, using this as a basis for your final decision.
• Conduct a 'pre-mortem' analysis to anticipate vendor-related challenges. Imagine a future scenario where a partnership with a vendor has failed to meet expectations. Identify all possible reasons for this failure, such as lack of support, hidden costs, or mismatched goals. This exercise can help you ask the right questions and set clear expectations with the vendor beforehand.

Data structures can be classified as either integrated or modular.

A system characterized as monolithic typically operates independently, while a modular system consists of several interlinked components. A system described as monolithic might be characterized by an extensive, singular codebase or a server specifically configured for a unique function. A system structured for modularity might use a web-based interface to facilitate communication between various applications or to separate specific functions into microservices. Segmenting large, unified systems enhances their scalability and boosts dependability.

The benefits and limitations of isolated, individual system designs.

A single-layer system amalgamates various components into a unified whole, making both understanding and setup processes simpler. The process of creating, implementing, and maintaining integrated systems often proceeds with ease and simplicity. Engineers acquainted with the structure will gain a comprehensive understanding of every component they engage with.

Attempting to scale up monolithic systems may result in catastrophic consequences for the data infrastructure. Scaling data systems to accommodate growing volumes and evolving requirements can present significant challenges, and such systems are often inherently fragile. Running even a small, highly optimized BI query could risk overloading and halting the production system of a SaaS platform, which may not be scalable enough to handle complex data analysis tasks. The authors characterize the cumbersome and chaotic monoliths as "large, disordered clusters."

Context

• Deploying updates or new features can be simpler, as there is only one system to update, reducing the complexity of coordinating deployments across multiple services.
• Users benefit from a consistent interface and experience across the system, which can improve usability and reduce the learning curve for new users.
• Engaging with all parts of a system can enhance an engineer's skill set, as they gain experience in various areas such as networking, databases, and application logic.
• Monolithic systems are software architectures where all components are interconnected and interdependent, often resulting in a single, large codebase. This can make updates and scaling more complex compared to modular systems.
• Monolithic systems often rely on a specific technology stack, making it difficult to incorporate new tools or languages without significant refactoring.
• In monolithic architectures, a single performance issue can affect the entire system, as all components share the same resources and infrastructure.

The transition to a modular architectural approach has significant implications for performance and involves complex considerations.

By segmenting a system into individual components, each one can be customized for particular situations, thereby improving its efficiency in various respects. Enhancing the system's scalability can be achieved by architecting its elements to communicate effectively through mechanisms like asynchronous message queues or well-defined application programming interfaces. Upgrades or changes to the various elements of the system can be executed seamlessly, with the rest of the infrastructure remaining unaffected.

While it offers advantages, a system based on modularity also increases the complexity related to its management, supervision, and diagnostic processes. Engineers must now be familiar with a variety of systems to pinpoint data flow and identify potential performance bottlenecks. Worries are mounting regarding the interoperability of different platforms. Incompatible systems may create new obstacles to efficiency, which could negate the benefits that come from using a system designed with interchangeable components.

Context

• By isolating components, failures in one part of the system are less likely to impact others, improving overall system reliability and making it easier to identify and fix issues.
• This refers to a system's ability to handle increased loads without compromising performance. By using asynchronous communication and APIs, systems can add more resources or components to manage higher demand efficiently.
• This design principle involves creating services that communicate over a network, allowing for independent updates and maintenance of each service.
• Managing different versions of modules can be complex, especially when different parts of the system need to be updated independently without causing compatibility issues.
• Understanding network protocols, such as HTTP/HTTPS, TCP/IP, and gRPC, is important for ensuring smooth communication between modules.
• Ensuring interoperability is essential for future-proofing systems, allowing for easier integration of new technologies and platforms as they emerge.
• Incompatible systems often arise from differences in data formats, communication protocols, or software versions, which can prevent seamless interaction between components.

Understanding the growing dominance of architectures based on event-driven approaches and microservices.

The limitations of monolithic software systems have led to the rise of microservice architectures, where each service provides a specific functionality and communicates with other services through a standard, loosely coupled interface. The strategy facilitates rapid scaling and accelerated deployment of software applications.

Reis and Housley believe that data engineers should borrow from microservices to build modular systems, utilizing open source engines as well as cloud-based managed services. For instance, instead of creating and implementing a sophisticated internal ETL dependent on a proprietary API, engineers might utilize Airbyte to capture data, store it in an S3 bucket in its original form, and subsequently employ a timed Spark cluster to process the information.

Practical Tips

• Start using apps and tools that are built on microservice architecture to get a feel for their flexibility and scalability. For instance, if you're into photography, use a photo editing app that offers different modules for editing, organizing, and sharing photos. Notice how each module works independently but also collaborates with others, giving you a practical sense of how microservices function.
• Engage with a community garden project by assigning specific roles to volunteers, akin to services in microservice architecture. Each role, like watering, weeding, or harvesting, should have clear tasks and ways to communicate with others, such as a shared logbook or scheduled meetings. This real-life application can help you grasp the importance of clear interfaces and dedicated functionalities in collaborative environments.
• Experiment with a microservice platform using a free or trial version to get hands-on experience without a financial commitment. Platforms like Heroku, AWS Lambda, or Google Cloud Functions offer free tiers or trial periods where you can deploy microservices and test their scalability and deployment speed. Create a simple application, such as a to-do list, and try to deploy each feature (adding tasks, marking them as complete, etc.) as a separate microservice.
• You can start by mapping out your personal data management tasks as if they were services in a business. Imagine each task, like managing your emails, photos, or financial records, as a separate 'service' that needs to communicate with the others efficiently. This helps you understand the concept of modularity in a practical, everyday context.
• Explore IoT (Internet of Things) devices for your home that leverage open source software and cloud services to enhance modularity. Start with something simple, like smart bulbs or a smart thermostat, which can be controlled via open source apps and often rely on cloud services for remote access and updates. This personal application gives you a tangible understanding of how modularity works in everyday technology, and how cloud services can enhance the functionality of open source systems.
• Enhance your understanding of data processing by creating a visual map of how a Spark-like system works. Use a flowchart tool or even pen and paper to draw out the process from data input to processing to output. This exercise will help you grasp the flow of data and the role of timing in processing, even if you're not ready to set up an actual cluster.

The difference is characterized by the adoption of architectures that do not rely on servers, as opposed to conventional ones that do.

The serverless model represents a considerable transformation in the realm of cloud computing. Engineers are liberated from the task of managing the servers that execute the software by being able to launch applications in a serverless cloud setting. Serverless products frequently rely on containers, a technology that packages a portable filesystem with applications.

Serverless technology enhances the efficiency of the workflows in development and operations.

The field of data engineering is evolving to encompass higher levels of abstraction by incorporating serverless architectures. When appropriate, serverless architecture can greatly reduce the costs and complexities associated with setting up and sustaining data infrastructures. Serverless services typically possess the ability to initiate at no capacity and enhance their scale autonomously in response to increasing demand. The appeal of cloud services stems from their implementation of a pay-per-use billing structure.

A data engineer, for instance, could set up an API endpoint that executes a specialized transformation through Amazon's serverless functions service, known as Lambda. The API provides the capability for users to obtain results in real-time. Lambda skillfully handles the complexities of security, network setups, and server management, while also streamlining the code deployment process. Data engineers possess the skills necessary to focus on writing software that alters data and constructs endpoints for APIs. Serverless frameworks offer advantages, but they also come with unique limitations and characteristics.

Practical Tips

• Collaborate with a local developer group to create a serverless solution for a community problem. Even if you're not a developer, you can contribute by defining the problem, brainstorming solutions, and testing the application. This could be anything from a community event notifier to a shared resource tracker, utilizing serverless technology to handle the backend processes.
• Create a personal project, like a photo-sharing website, using a serverless architecture to understand how scaling works firsthand. Use a platform like AWS Lambda or Google Cloud Functions to handle the backend, and observe how the service automatically adjusts to the number of uploads and views your site receives.
• You can evaluate your current subscriptions and services to see if switching to cloud-based alternatives could save you money. Look at the software and services you pay for on a regular basis, such as data storage, project management tools, or office software. Compare the costs of these services to cloud services that offer similar functionalities but with a pay-per-use model. This could help you only pay for what you actually use, potentially reducing your overall expenses.
• Enhance your blog or website by integrating a Lambda-powered API that provides dynamic content or services. For instance, if you run a cooking blog, you could use Lambda to create an API that generates random weekly meal plans based on a database of recipes. This not only adds interactivity to your site but also encourages repeat visits.
• Explore API consumption by using no-code platforms that allow you to connect to various APIs and retrieve data. Platforms like Zapier or IFTTT provide a user-friendly interface where you can experiment with pulling data from public APIs, such as weather services or social media feeds, and see how data can be extracted and used in different contexts.
• Create a simple cost-benefit analysis tool using a spreadsheet to compare the costs of traditional server-based setups versus serverless setups for hypothetical projects. Input variables like expected traffic, data storage needs, and compute time to see how the costs stack up over time. Share your tool with friends or online communities to get feedback and refine your analysis.

Evaluating various approaches for overseeing deployment management.

Serverless architectures now fundamentally rely on containers as a key component of their deployment tactics. Kubernetes orchestrates the distribution and management of containerized applications across various servers within a cluster, ensuring resource allocation, scalability, networking, and other associated duties are handled efficiently.

Container technology is increasingly being integrated into data engineering, a practice that was previously mainly associated with software applications. Containerized applications can easily incorporate platforms like Spark and Flink, and the rise of orchestrators such as Kubernetes, exemplified by tools like Argo, signals an increasing inclination towards employing container orchestration for managing extensive data and streaming operations.

Practical Tips

• Join online forums or groups focused on containerization and big data to learn from real-world scenarios. Engaging with a community can provide insights into how others are combining containerized applications with platforms like Spark and Flink. You can ask questions, participate in discussions, and even find partners for collaborative learning projects, which can help solidify your understanding of the integration process.
• Apply the principles of container orchestration to manage your personal productivity apps. If you use multiple productivity tools, try using an app like Station or Shift, which aggregates different web applications into a single interface. This mimics the idea of orchestrating containers by managing multiple services from a single point, helping you understand the benefits of centralized management.

Servers that are native to the cloud are characterized by their ephemeral existence and the capacity to scale autonomously.

Data engineering positions frequently depend on server-based solutions, especially for tasks requiring unique software configurations or for operations that are not handled effectively by serverless technologies. Leveraging the scalability that cloud platforms naturally offer is essential for distributing data throughout a server network.

The authors advocate for the view that servers should be seen as assets with a limited lifespan rather than as machines that function continuously without breaks. This approach yields positive outcomes when teams embrace additional tenets of robust data architecture, characterized by a modular design and an inherent tolerance for mistakes. Autoscaling considerably cuts costs by enhancing the infrastructure's capacity to handle increases in data processing and intake, subsequently scaling down as the need diminishes.

Other Perspectives

• Cloud-native servers, while scalable, may still encounter limits imposed by the cloud provider's infrastructure or service quotas, which can prevent them from scaling as needed.
• The use of containers and microservices can provide similar benefits to server-based solutions, such as isolation and specific software configurations, while also offering advantages in terms of efficiency and portability.
• Some regulatory or compliance requirements may limit the ability to distribute data across a server network, especially in a multi-tenant cloud environment.
• Traditional servers, especially in legacy systems, are often expected to have long lifespans and high reliability, as they host critical applications that may not be suitable for ephemeral environments.
• Overemphasis on robustness can lead to overengineering, increasing costs and resource consumption without proportional benefits.
• Scaling down as needed can save costs, but it can also cause performance issues if the scaling actions are too aggressive or not responsive enough to sudden spikes in demand.

The procedure includes the collection of data, its secure preservation, modification, and guaranteeing its availability as required.

Collecting information from the initial sources

Data integration techniques and methodologies must be thoroughly understood by those specializing in data engineering, despite the simplification of this task by sophisticated data tools. These patterns and techniques can be divided into batch versus streaming and essential considerations for ingestion.

Evaluating the distinct uses and trade-offs between batch and real-time data processing methods.

Data is immediately processed as it becomes available, in contrast to batch ingestion where data is collected into separate groups before it is transferred to intermediate storage or transformed. Many applications in the field of data engineering often favor the method of batch processing. Organizations reliant on consistent reporting and analysis find their capabilities limited by the intrinsic restrictions of their data sources and internal organizational boundaries. Batch processing remains a widely used method and suits numerous traditional applications well.

The authors believe that the importance of seamlessly integrating data into data pipelines is poised to increase substantially. The use of specialized applications for continuous monitoring, analyzing security data, and managing operational intelligence is on the rise. Many applications that operate with live data often incorporate analytical capabilities and dashboard functionalities into standard application workflows. The utilization of cloud computing eases the complexity of setting up and administering sophisticated streaming infrastructures.

Practical Tips

• Optimize your social media usage by dedicating specific times to batch process your interactions and using real-time notifications for important updates. Set aside time blocks during your day when you check and respond to social media messages and comments in bulk. Meanwhile, enable notifications for direct messages or mentions from key contacts or topics you follow closely, allowing you to engage with these immediately, similar to real-time processing.
• Organize your grocery shopping by creating a comprehensive list throughout the week and designate one day for shopping. This approach saves time and reduces the frequency of trips to the store. As you run out of items or think of things you need, add them to a list on your phone or a notepad on the fridge, so when shopping day comes, you're ready to go with a complete list in hand.
• You can explore the benefits of batch processing by organizing your digital files during a scheduled time each week. Instead of handling files as they come, set aside an hour every Sunday to sort, file, and back up your documents. This mimics batch processing and can increase your efficiency by reducing the constant context-switching throughout the week.
• Start a cross-functional book club at work to break down organizational silos. By reading and discussing books from various disciplines with colleagues from different departments, you'll gain insights into how other parts of the organization operate and think about data. This could lead to more integrated and holistic reporting practices.
• When cooking meals for the week, use a batch cooking strategy to prepare multiple portions of a few dishes at once. Spend a few hours on a weekend day cooking large quantities of staple items like rice, proteins, and vegetables. Then, mix and match these prepped ingredients to create different meals throughout the week. This not only saves time on daily meal preparation but also helps with portion control and maintaining a balanced diet.
• You can enhance your personal cybersecurity by setting up your own simple network monitoring solution using free tools like Wireshark or OpenVAS. These tools can help you understand the traffic on your home network and identify any unusual patterns that could indicate a security threat. For example, if you notice an unknown device connecting to your network or a spike in outbound traffic, it could be a sign of a compromised device.
• You can enhance your personal finance tracking by using a live data dashboard app that aggregates your bank accounts, investments, and expenses. By setting up custom alerts and analytical graphs, you'll be able to monitor your financial health in real-time, identify spending trends, and make informed decisions on where to cut costs or invest more.
• Use cloud-based tools to organize a virtual movie night with friends. Platforms like AWS or Google Cloud offer services that can host and stream video content. You can upload a movie you've created or have the rights to share, set up a virtual environment, and invite your friends for a movie night, managing the entire experience through the cloud.

Essential considerations for managing both batch and streaming data include

When developing and building pipelines for data ingestion, it is crucial for data engineers to take into account numerous factors such as the data's categorization as either limited or unending, the frequency at which data is received, whether the data intake occurs simultaneously or in a series, along with the techniques for encoding and decoding data, the requirements for processing power and the ability to scale, and the key characteristics of the data itself, which include its size, shape, composition, the types of data involved, and the accompanying metadata.

Data is ingested in separate chunks when volumes are small. Data is utilized right away following its generation. For example, we might pull a daily snapshot of sales data from an application database for batch ingestion, or we could ingest the same sales data as events occur in the database. Data ingestion frequency refers to the regularity with which data enters the system. Data can be gathered daily or updated instantaneously as new transactions are recorded in the original system. The velocity at which data is taken in and the strategies used for handling modifications are critical factors in determining appropriate technologies for data storage and processing.

Other Perspectives

• While considering data categorization and frequency of data reception is important, it can be argued that too much emphasis on these aspects may lead to over-engineering of solutions for simple use cases, where a more straightforward approach could suffice.
• The choice between batch and streaming ingestion is also influenced by the existing technical infrastructure and expertise available within an organization, which might not align with the implied simplicity of choosing one method over the other based on volume and timing alone.
• While it's true that data ingestion can range from daily snapshots to real-time updates, this dichotomy oversimplifies the spectrum of ingestion frequencies, which can include near-real-time, hourly, or other intervals that are not captured by the phrase "daily snapshots to real-time updates."
• The statement assumes a direct correlation between data velocity and technology selection, but budget constraints and existing infrastructure can also heavily influence which technologies are feasible, regardless of data velocity.

Techniques for sourcing data from a range of origins, including databases, flat files, and APIs.

Data engineers compile data from a variety of sources, encompassing both modern automated systems like APIs and traditional methods such as electronic data interchange. Numerous instruments are available to streamline and automate the collection of data from different origins. SQL drivers, such as JDBC and ODBC, offer interfaces that enable interaction with a range of relational databases and some types of NoSQL databases at a fundamental level. Various object storage platforms offer Python libraries and command-line tools, along with interfaces that can be programmed for user interaction. Cloud providers enable the collection of data through comprehensive management services, which encompass dedicated support for a range of message queues and streaming platforms.

The authors recommend opting for high-quality instruments tailored to the particular source system. For example, change data capture is an ideal pattern for ingesting from relational databases that offer a log of all database events, especially if the database is expected to see a high rate of change, while tools that automate API ingestion can dramatically reduce the operational burden of maintaining these connections.

Other Perspectives

• The statement does not consider the potential for data privacy and security concerns that arise when compiling data from various sources, which can be a significant part of the data engineering process in sensitive industries.
• Data sourced from APIs and traditional methods like EDI may require additional processing to ensure consistency and compatibility, as these sources often have different formats and standards.
• Relying on automated instruments for data collection can create a single point of failure in the data pipeline, which can be risky if the tool encounters an error or outage.
• JDBC and ODBC are standards that can sometimes lag behind the latest database features or optimizations, potentially limiting their effectiveness with newer or more advanced NoSQL database functionalities.
• The performance and scalability of Python libraries and command-line tools can vary, and they may not meet the requirements of high-throughput or low-latency applications.
• Integration with existing systems can be challenging when adopting cloud provider services, particularly if those systems are legacy or custom-built.
• CDC solutions might be overkill for databases with low to moderate change rates, where simpler data ingestion methods could suffice.
• Automated API ingestion tools may come with a financial cost, which could be significant for small businesses or projects with limited budgets.

Ensuring the preservation of data's integrity.

Storage serves as the critical foundational framework essential for the entirety of the data engineering lifecycle. Various systems, including data lakes, data warehouses, and other storage solutions, comprise the data repositories that act as sources of data, in addition to serving functions like caching and object storage. Data engineers must grasp the assembly of raw storage media into functional storage systems that cater to application and end-user requirements, as well as selecting storage solutions that fulfill their applications' demands for performance, durability, reliability, and affordability.

Essential elements encompass strategies for encoding data, minimizing its volume, and implementing systems for data storage efficiency.

Data storage necessitates crucial components, both hardware and software. The system includes multiple components like flash-based storage devices, temporary memory modules, central processing units, and network structures, along with approaches for encoding data, strategies to reduce data size, and temporary data storage in memory caches. Grasping the functionality of these basic components and their integration is essential for assessing the compromises inherent in storage systems.

Solid-state drives (SSDs) markedly improve performance compared to conventional magnetic hard drives by providing faster access times, higher input/output operations per second (IOPS), and increased data transfer speeds. When developing databases tasked with managing transactions, designers need to take into account a range of important factors. Caching strategies are often utilized by systems spread across multiple locations to improve the efficiency of their data storage and access. To fully comprehend the impact on performance, engineers need to be well-versed in the various layers of caching, from those incorporated within the CPU to those associated with magnetic disk storage.

Other Perspectives

• Overemphasis on minimizing data volume could potentially lead to loss of information or reduction in data quality, which might be counterproductive for certain applications that rely on high-fidelity data.
• The focus on CPUs might overshadow the increasing importance of dedicated hardware for storage operations, such as storage processors and hardware accelerators, which can offload tasks from the CPU and improve overall system performance.
• In some cases, the evaluation of storage systems may rely more on empirical performance data and benchmarks rather than a deep understanding of the underlying components, as the end-user experience is the ultimate measure of system effectiveness.
• SSDs typically have a higher cost per gigabyte than magnetic hard drives, which can make them less economical for bulk storage needs.
• The idea that designers must consider various factors could be seen as too broad; it might be more useful to specify which factors are most critical or to acknowledge that the importance of different factors can vary depending on the specific use case and requirements of the database.
• Over-reliance on caching can mask underlying performance issues in the system, which might need to be addressed for long-term stability and efficiency.
• Engineers specializing in areas other than performance optimization, such as security or user experience, may not need in-depth knowledge of caching layers.

Data storage systems encompass numerous paradigms such as object storage, filesystems, and virtualized block storage.

The predominant storage architectures encompass systems for managing objects, organizing files, and diverse types of block storage virtualization. Analytical data is more and more often being stored in cloud-based object storage systems that leverage magnetic disk technology at a local level to facilitate substantial scalability. Block storage serves as the bedrock for filesystems, organizing data into a hierarchy of directories and files, which suits applications and operating systems well, yet this setup is less suited for large-scale distributed data analysis. In cloud settings, configuring virtual machines typically includes implementing virtualized block storage, providing each VM with a simulated storage unit that has all the functionalities of a physical hard disk.

Data engineers have the responsibility of integrating diverse storage systems to meet the needs associated with each phase of the data engineering lifecycle. A data lake supported by object storage integrates seamlessly with a lakehouse management system, whereas databases engineered for transaction processing are often chosen for applications that demand rapid access and assured data consistency.

Other Perspectives

• The statement may oversimplify the diversity within each category; for instance, filesystems can range from local file systems to distributed file systems, each with different characteristics and use cases.
• While cloud-based object storage systems do offer scalability, they may not always be the most cost-effective solution for storing analytical data, especially when data transfer and access costs are considered.
• The hierarchical structure of directories and files is a logical organization that can be abstracted away from the underlying storage architecture, which could be block, object, or even something else like a distributed file system that spans across multiple storage nodes.
• The simulation of a physical hard disk in virtualized block storage may not always provide all the functionalities of a physical hard disk, such as certain low-level hardware controls and performance characteristics.
• The integration process can be time-consuming and may delay other critical tasks in the data engineering lifecycle.
• In some cases, the need for rapid access and data consistency can be met with in-memory data stores or caching layers that sit in front of traditional databases, which can provide faster access without the need for a database that is specifically designed for transaction processing.

Matching storage system type with use case, scale, and performance needs

Each phase of the data engineering lifecycle is underpinned by the essential support of storage systems. Each stage and its associated application present unique obstacles in the handling of data. It is crucial to comprehend the data's provenance, magnitude, anticipated efficacy, and associated storage expenses. As the business expands, it will be crucial to establish a storage infrastructure that can scale effectively.

Consider a scenario where you are incorporating event information into a streaming service like Kafka. How do you plan to store your data? Do you need to maintain your data for long-term examination, or would a retention span as short as a week be adequate? What is the appropriate location for long-term data storage? Employing the platform as an intermediary buffer can be advantageous for the subsequent distribution of data into diverse storage solutions such as warehouses or lakes. The choice is determined by the quantities of data and their intended use.

Practical Tips

• Improve your learning retention by applying data versioning to your study notes. As you learn new information, keep a version history of your study materials. When you update your notes with new insights or corrections, save them as a new version rather than overwriting the old ones. This allows you to track your progress over time and revisit previous understandings, similar to how data engineers manage changes in datasets.
• Experiment with different data organization tools to find the best fit for your needs. Try out a few free or trial versions of data management software like Trello for task management, Google Sheets for data analysis, or Evernote for data storage. Use each tool for a different aspect of your life, such as work, personal finance, or health tracking. After a few weeks, evaluate which tools helped you handle data more effectively and consider adopting them for long-term use.
• Consider subscribing to a cloud storage service that offers pay-as-you-go pricing. This allows you to increase your storage capacity as your business grows without a significant upfront investment. Look for services that provide easy scalability options, so you can adjust your plan as needed with minimal technical know-how.
• You can simulate storage needs using a spreadsheet to estimate future data growth. Start by tracking the average size of your current event messages and the frequency at which they're generated. Then, project these figures over time, considering any expected increases in data volume or velocity. This will give you a rough estimate of storage needs over different timeframes, allowing you to plan for expansion before it becomes critical.
• Create a data retention decision tree for your personal documents and media. Sketch out a flowchart that guides you through a series of questions about each piece of data, like "Is this data sentimental?", "Will I need this for tax purposes?", or "Does this contain important personal information?". Depending on your answers, the flowchart will lead to actions such as "keep for 1 year", "archive indefinitely", or "safe to delete". This visual tool can simplify your decision-making process regarding data retention.
• Develop a routine to regularly check the integrity of your stored data. Set a calendar reminder every three months to open and review a random selection of files from different storage locations. This habit ensures that you're not only storing your data but also maintaining its usability over time, catching potential corruption or loss issues early.
• Experiment with IFTTT (If This Then That) or similar automation services to create custom workflows that distribute your data. For example, you could create an applet that saves email attachments to a designated cloud storage folder, which then triggers a secondary action to back up that data to a NAS (Network Attached Storage) at home. This hands-off strategy helps you manage data distribution without manual intervention.
• Organize a "data decluttering" day where you go through your digital files and categorize them based on importance and usage. Similar to tidying up a physical space, this process involves deleting redundant files, archiving old data, and organizing the remaining data into a structured system. This will not only free up space but also give you a clearer idea of what storage solutions are most practical for your current data landscape.

Methods and tactics for modifying data

During the transformation stage, data is processed and refined into a valuable end product. Data undergoes parsing, cleansing, enrichment, and is transformed into a reliable form suitable for subsequent processes such as analytics, machine learning, and automated tasks. The data engineer’s job is to build solid systems that streamline the creation, maintenance, deployment, and monitoring of transformation pipelines, while ensuring high performance, data quality, compliance, and security.

Data modeling techniques for analysis incorporate strategies conceived by Ralph Kimball, as well as approaches associated with Data Vault.

The authors emphasize the continued relevance of traditional data infrastructures, even as data quantities expand and data engineers place greater emphasis on models that process data in real time. In the past, data models were designed to facilitate batch processing of structured data. The foundational frameworks for structuring data are derived from the concepts put forth by Kimball and Inmon, along with the techniques related to Data Vault. In practice, these can be mixed and combined into a hybrid architecture.

The methodologies introduced by Inmon, Kimball, and Data Vault extend beyond simple technical schematics for the arrangement and setup of database tables. Every method provides a unique viewpoint on how the company's operational processes are depicted and executed. Organizations often integrate these strategies to create a data governance structure that clearly defines the different layers of data abstraction, such as the conceptual, logical, and physical levels.

Other Perspectives

• Data modeling techniques for analysis are evolving, and new paradigms such as NoSQL databases and Big Data technologies offer alternative strategies that may not align directly with Kimball's dimensional modeling or Data Vault's ensemble modeling.
• While data models have historically aimed at facilitating batch processing of structured data, they have also been designed with the flexibility to handle ad-hoc queries and reporting, which may not always fit into the batch processing paradigm.
• Critics might also point out that while these methodologies provide a framework for understanding operational processes, they do not always account for the rapidly changing data privacy and security landscape, which is becoming increasingly important in data governance.
• These methods may be too rigid or prescriptive, potentially constraining the ways in which data can be used to reflect the actual workings of a company.
• The assertion that organizations often combine these methodologies does not account for the possibility that some organizations may prefer to adopt a single methodology for the sake of simplicity and focus.

Data lakes and data warehouses frequently utilize large structures and setups that are not normalized.

The rise of cloud-based storage options, including expansive data reservoirs, has made it easier to streamline data structures due to their affordable storage solutions and designs that accommodate a wide range of analytical questions. In the contemporary landscape of cloud computing and vast data, engineers gain advantages from cost-effective storage solutions, as well as the clear separation of data processing from storage, coupled with powerful and flexible systems for query processing.

One consequence of these powerful new capabilities is a trend towards wide tables, which are highly denormalized and can contain many fields. The detailed tables often have intricate designs that incorporate multiple forms of data. Engineers are permitted to construct pipelines with greater flexibility and agility by easing the strictness of conventional modeling practices. However, this lack of structure can also lead to inconsistent data definitions across the organization and other complications for downstream consumers.

Other Perspectives

• Denormalized structures can sometimes lead to increased storage costs and complexity in data processing, as more data is duplicated across the system.
• Designs that accommodate a wide range of analytical questions can sometimes result in overly complex systems that are hard to maintain and can suffer from performance issues if not carefully optimized.
• Cloud-based query processing systems may come with a steep learning curve, requiring specialized knowledge or training, which can be a barrier for some organizations.
• In scenarios where data access patterns are well understood and consistent, a normalized schema might provide better organization and efficiency.
• Such designs might compromise query performance, as more complex joins and transformations might be required to retrieve specific insights from the data.
• The flexibility in pipeline construction could lead to a proliferation of ad-hoc solutions that are difficult to audit and secure against data breaches or leaks.
• While a lack of structure can lead to inconsistent data definitions, it can also foster a more flexible environment that encourages innovation and rapid adaptation to new data sources and types.