Data-driven decision making refers to the use of data and analytics to inform and guide decision-making processes. It involves collecting, organizing, and analyzing data in order to generate insights that can be used to make informed decisions. Data engineers play a crucial role in supporting decision making by building and maintaining the infrastructure and systems […]
How Data Engineers Support Data Driven Decision Making? Read More »
Clustering is a way of organizing data in Snowflake tables to improve the performance of queries. It works by grouping similar values of one or more columns together physically. This makes it easier for Snowflake to find the data that you need, which can lead to faster query results. Clustering is typically beneficial for tables
Micro Partitions and Clustering in Snowflake — Part 2 Read More »
Traditional data warehousing solutions often rely on rigid partitioning techniques to enhance performance and enable scalability. In such systems, partitions act as independent units, managed through specialized DDL and syntax. However, static partitioning comes with its well-known limitations, including maintenance complexities and the potential for data skew, resulting in unevenly sized partitions. In stark contrast
Micro Partitions and Clustering in Snowflake — Part 1 Read More »
Snowflake Time Travel is a fascinating feature that grants you the power to access historical data. Imagine you have an Employee table, and you accidentally delete it. With Time Travel, you can effortlessly journey back in time, let’s say 5 minutes, and rescue your data. The versatility of Snowflake Time Travel goes beyond just data
Time Travel with Snowflake Read More »
Within the Snowflake architecture, virtual warehouses serve as the essential computing engines responsible for query execution. Optimizing the allocation of computing resources within a warehouse can significantly enhance the performance of individual queries or a group of queries. For warehouse owners or administrators aiming to enhance query performance, various warehouse-related strategies are at their disposal.
Strategies for Optimizing Snowflake Warehouse Performance – Part 1 Read More »
In Apache Airflow, an executor is a component responsible for executing tasks within a workflow. When you define a workflow in Airflow, you specify a set of tasks and the dependencies between them. Executors are responsible for executing these tasks according to their specified dependencies. Local and Remote Executors Apache Airflow has two types of executors: local and remote. Local Executors Local executors run tasks on
Apache Airflow Executors 101 Read More »
Apache Airflow is a “data orchestration” platform that data engineers use to schedule and coordinate steps in a data pipeline or data workflows. It’s one of the most popular open-source orchestration platform. Airflow helps define workflows in python code and provides a rich UI to manage and monitor these workflows. Airflow supports easy integration with a
What is Apache Airflow? An Overview Read More »