Discover and read the best of Twitter Threads about #dataengineering
Most recents (24)
Troubleshooting slow Spark jobs is a special type of data engineering torture!
What are the common culprits? 1/5
What are the common culprits? 1/5
- Inadequate initial file parallelism If your upstream data tables are written to too few files, you can't increase the parallelism much unless you work with the upstream pipeline to write more files. This can increase the speed of your job dramatically! 2/5
- Not enough memory/disk spillage
Disk spillage happens when Spark needs more RAM to process the data and uses ROM instead. Disk spillage hurts perf! You can solve bumping up executor memory or increasing parallelism. This can also be caused by skew 3/5
Disk spillage happens when Spark needs more RAM to process the data and uses ROM instead. Disk spillage hurts perf! You can solve bumping up executor memory or increasing parallelism. This can also be caused by skew 3/5
Testing your pipelines before merging is crucial to ensure they do not fail in production. However, testing data pipelines is complex (and expensive) due to the data size, confidentiality, and time it takes to test a data pipeline.
๐งต
#data #dataengineering #testing #dataops
๐งต
#data #dataengineering #testing #dataops
Here are a few ways to get data for your tests:
1. Copying data: An exact copy of the prod data for testing will ensure that our changes are not breaking the pipeline. This is expensive! You can use a part of data for testing, accepting possible edge case misses.
1. Copying data: An exact copy of the prod data for testing will ensure that our changes are not breaking the pipeline. This is expensive! You can use a part of data for testing, accepting possible edge case misses.
2. Data git: Projects like Nessie and LakeFS can help set up different environments without replicating entire data.
Data engineers work with multiple systems & it's crucial to understand DevOps. Shown below are a few DevOps concepts to familiarize oneself with:
1. Docker: docs.docker.com/get-started/
2. Kubernetes: kubernetes.io/docs/concepts/โฆ
3. CI/CD: resources.github.com/ci-cd/
#dataengineering
#data
1. Docker: docs.docker.com/get-started/
2. Kubernetes: kubernetes.io/docs/concepts/โฆ
3. CI/CD: resources.github.com/ci-cd/
#dataengineering
#data
4. IAC: pulumi.com/what-is/what-iโฆ
5. Monitoring: atlassian.com/devops/devops-โฆ
6. Access control: techtarget.com/searchsecurityโฆ
7. Key management: aws.amazon.com/kms/?c=sc&sec=โฆ
8. Encrypted connections: dev.mysql.com/doc/refman/8.0โฆ
5. Monitoring: atlassian.com/devops/devops-โฆ
6. Access control: techtarget.com/searchsecurityโฆ
7. Key management: aws.amazon.com/kms/?c=sc&sec=โฆ
8. Encrypted connections: dev.mysql.com/doc/refman/8.0โฆ
That's a wrap!
If you enjoyed this thread:
1. Follow me @startdataeng for more of these
2. RT the tweet below to share this thread with your audience
If you enjoyed this thread:
1. Follow me @startdataeng for more of these
2. RT the tweet below to share this thread with your audience
Things Iโve learned running my data engineering boot camp for 2 weeks!
- donโt use platforms like Maven which take 10%. Google classroom is free and good enough. 1/5
- donโt use platforms like Maven which take 10%. Google classroom is free and good enough. 1/5
- be more than 1 day ahead of the boot camp. When curricula building you should be working on the next weeks curricula this week so your students can review the materials before.
- use Postgres 14+ on your SQL sections otherwise you donโt have access to BIT_COUNT function 2/5
- use Postgres 14+ on your SQL sections otherwise you donโt have access to BIT_COUNT function 2/5
- use 1 long-standing Zoom call instead of 1 Zoom call per session. This makes it easier for students to know where to go and donโt use the Google Calendar + Zoom integration, itโs really bad!
- give people more time on the homework. Strict deadlines are for college not bootโฆ twitter.com/i/web/status/1โฆ
- give people more time on the homework. Strict deadlines are for college not bootโฆ twitter.com/i/web/status/1โฆ
How to use ChatGPT to speed up data pipeline dev in a few easy steps:
1. Supply ChatGPT with your input schemas. Paste your CREATE TABLE statements directly. ChatGPT knows how to parse these and make sense of the fields inside
1/4
1. Supply ChatGPT with your input schemas. Paste your CREATE TABLE statements directly. ChatGPT knows how to parse these and make sense of the fields inside
1/4
2. Specify what type of analytical pattern you want to do on this data. ChatGPT understands aggregation (for GROUP BY queries), cumulation (for FULL OUTER JOIN queries), enrichment (for JOIN queries) and slowly changing dimension patterns.
2/4
2/4
3. ChatGPT needs more context on WHICH fields it should apply these patterns to. If you donโt supply that, itโll give you flaming garbage.
Example: โapply slowly changing dimension type 2 pattern to fields age, gender, and phone osโ
3/4
Example: โapply slowly changing dimension type 2 pattern to fields age, gender, and phone osโ
3/4
My data eng boot camp curricula has been updated:
- Week 1: Dimension Data Modeling
Day 1: understanding dimensions. Daily dimensions vs SCDs. How to pick SCD type 1/2/3
Day 2: applied dimension data modeling. backfilling SCD tables. Incremental building SCD tables.
1/8
- Week 1: Dimension Data Modeling
Day 1: understanding dimensions. Daily dimensions vs SCDs. How to pick SCD type 1/2/3
Day 2: applied dimension data modeling. backfilling SCD tables. Incremental building SCD tables.
1/8
- Week 2: Fact Data Modeling
Day 1: understand facts. denormalized facts vs normalized facts. How to collaborate and get logging right.
Day 2: applied fact data modeling. Reduced facts for efficient long-term analysis. Cumulative table design for efficient fact analysis. 2/8
Day 1: understand facts. denormalized facts vs normalized facts. How to collaborate and get logging right.
Day 2: applied fact data modeling. Reduced facts for efficient long-term analysis. Cumulative table design for efficient fact analysis. 2/8
- Week 3: Spark
Day 1: understanding when to use Spark. Deep dive into Spark architecture and bottlenecks
Day 2: applied Spark.
understand the parallelism vs network overhead. SparkSQL vs DataFrame vs Dataset.
Iโll lead Scala Spark and @ADutchEngineer will lead PySpark
3/8
Day 1: understanding when to use Spark. Deep dive into Spark architecture and bottlenecks
Day 2: applied Spark.
understand the parallelism vs network overhead. SparkSQL vs DataFrame vs Dataset.
Iโll lead Scala Spark and @ADutchEngineer will lead PySpark
3/8
Live from #GartnerDA | 5 Ways to Enhance Your Data Engineering Practices with Robert Thanaraj, Gartner Director Analyst: gtnr.it/3JOkYPF 
About this session: Analytics relies on a successful data foundation; it must be backed with the right data and processes. Robert explores 5 ways to enhance your #DataEngineering practices: gtnr.it/3JOkYPF #GartnerDA
#DataEngineering is a critical skill in high demand amongst employers with a 7.5% increase in demand. #GartnerDA
" SQL Puzzle Interview Question "
๐งต
๐งต
Table script:
create table input (
id int,
formula varchar(10),
value int
)
insert into input values (1,'1+4',10),(2,'2+1',5),(3,'3-2',40),(4,'4-1',20);
create table input (
id int,
formula varchar(10),
value int
)
insert into input values (1,'1+4',10),(2,'2+1',5),(3,'3-2',40),(4,'4-1',20);
" Exploratory Data Analysis on Terrorism "
๐งต
๐งต
We are performed Exploratory Data Analysis on terrorism #dataset to find out the hot zone of #terrorism. #EDA nothing but #analyzing the given data & finding the #trends, patterns & making some assumptions. #DataVisualization #DataScience #MachineLearning
In this #dataset, there are many features including countries, states, regions, gang names, weapon types, target types, years, months, days, and many more features.
What is a correct Data Engineering Learning Path?
My thoughts in the ๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
My thoughts in the ๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
I believe that the following is a correct order to start in ๐ฌ๐ผ๐๐ฟ ๐๐ฎ๐๐ฎ ๐๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐ถ๐ป๐ด ๐ฃ๐ฎ๐๐ต:
๐
๐
โก๏ธ ๐จ๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ ๐๐ฎ๐๐ถ๐ฐ ๐ฃ๐ฟ๐ผ๐ฐ๐ฒ๐๐๐ฒ๐:
๐ Data Extraction
๐ Data Validation
๐ Data Contracts
๐ Loading Data into a DWH / Data Lake
๐ Transformations in a DWH / Data Lake
๐ Scheduling
๐
๐ Data Extraction
๐ Data Validation
๐ Data Contracts
๐ Loading Data into a DWH / Data Lake
๐ Transformations in a DWH / Data Lake
๐ Scheduling
๐
What are the basics of Writing Data to a Kafka Topic?
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
Kafka is an extremely important ๐๐ถ๐๐๐ฟ๐ถ๐ฏ๐๐๐ฒ๐ฑ ๐ ๐ฒ๐๐๐ฎ๐ด๐ถ๐ป๐ด ๐ฆ๐๐๐๐ฒ๐บ to understand as it was the first of its kind and most of the new products are built on the ideas of Kafka.
๐ฆ๐ผ๐บ๐ฒ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐น ๐ฑ๐ฒ๐ณ๐ถ๐ป๐ถ๐๐ถ๐ผ๐ป๐:
๐
๐ฆ๐ผ๐บ๐ฒ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐น ๐ฑ๐ฒ๐ณ๐ถ๐ป๐ถ๐๐ถ๐ผ๐ป๐:
๐
โก๏ธ Clients writing to Kafka are called ๐ฃ๐ฟ๐ผ๐ฑ๐๐ฐ๐ฒ๐ฟ๐,
โก๏ธ Clients reading the Data are called ๐๐ผ๐ป๐๐๐บ๐ฒ๐ฟ๐.
โก๏ธ Data is written into ๐ง๐ผ๐ฝ๐ถ๐ฐ๐ that can be compared to ๐ง๐ฎ๐ฏ๐น๐ฒ๐ ๐ถ๐ป ๐๐ฎ๐๐ฎ๐ฏ๐ฎ๐๐ฒ๐.
๐
โก๏ธ Clients reading the Data are called ๐๐ผ๐ป๐๐๐บ๐ฒ๐ฟ๐.
โก๏ธ Data is written into ๐ง๐ผ๐ฝ๐ถ๐ฐ๐ that can be compared to ๐ง๐ฎ๐ฏ๐น๐ฒ๐ ๐ถ๐ป ๐๐ฎ๐๐ฎ๐ฏ๐ฎ๐๐ฒ๐.
๐
Quick guide to go from 0 to #dataengineering hero:
- learn SQL
Data Lemur is a great resource here
- learn Python
Do likeโฆ 30-40 leetcode easy and medium questions
- distributed compute
Get a trial of Databricks or Snowflake and find a training to learn about it
1/3
- learn SQL
Data Lemur is a great resource here
- learn Python
Do likeโฆ 30-40 leetcode easy and medium questions
- distributed compute
Get a trial of Databricks or Snowflake and find a training to learn about it
1/3
- data modeling
Find a dimension table like users that you can snapshot daily. Learn about slowly changing dimensions.
Find a fact/event table that you can aggregate and learn about fact modeling
- job orchestration
Learn Mage or Airflow to do your daily automated tasks
2/3
Find a dimension table like users that you can snapshot daily. Learn about slowly changing dimensions.
Find a fact/event table that you can aggregate and learn about fact modeling
- job orchestration
Learn Mage or Airflow to do your daily automated tasks
2/3
- data story telling
Take a training by Tableau on data visualization and how to tell stories with data
- communication
Read crucial conversations and radical candor books. Theyโll help a lot!
If you just do this, youโll be a lot closer to a great data engineering job! 3/3
Take a training by Tableau on data visualization and how to tell stories with data
- communication
Read crucial conversations and radical candor books. Theyโll help a lot!
If you just do this, youโll be a lot closer to a great data engineering job! 3/3
So what is the difference between Row Based and Column Based file formats?
๐งต
#Data #DataEngineering #MLOps #MachineLearning
๐งต
#Data #DataEngineering #MLOps #MachineLearning
๐ฅ๐ผ๐ ๐๐ฎ๐๐ฒ๐ฑ:
โก๏ธ Rows on disk are stored in sequence.
โก๏ธ New rows are written efficiently since you can write the entire row at once.
๐
โก๏ธ Rows on disk are stored in sequence.
โก๏ธ New rows are written efficiently since you can write the entire row at once.
๐
โก๏ธ For select statements that target a subset of columns, reading is slower since you need to scan all sets of rows to retrieve one of the columns.
๐
๐
What are the main use cases for Apache Kafka or any other Distributed Messaging System?
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
We have covered lots of concepts around Kafka already. But what are the most common use cases for The System that you are very likely to run into as a Data Engineer?
๐๐ฒ๐โ๐ ๐๐ฎ๐ธ๐ฒ ๐ฎ ๐ฐ๐น๐ผ๐๐ฒ๐ฟ ๐น๐ผ๐ผ๐ธ:
๐
๐๐ฒ๐โ๐ ๐๐ฎ๐ธ๐ฒ ๐ฎ ๐ฐ๐น๐ผ๐๐ฒ๐ฟ ๐น๐ผ๐ผ๐ธ:
๐
๐ช๐ฒ๐ฏ๐๐ถ๐๐ฒ ๐๐ฐ๐๐ถ๐๐ถ๐๐ ๐ง๐ฟ๐ฎ๐ฐ๐ธ๐ถ๐ป๐ด.
โก๏ธ The Original use case for Kafka by LinkedIn.
โก๏ธ Events happening in the website like page views, conversions etc. are sent via a Gateway and piped to Kafka Topics.
๐
โก๏ธ The Original use case for Kafka by LinkedIn.
โก๏ธ Events happening in the website like page views, conversions etc. are sent via a Gateway and piped to Kafka Topics.
๐
Roadmap to becoming Data Analyst in three months absolutely free. No need to pay a penny for this.
I have mentioned a roadmap with free resources.
A thread๐งต๐
I have mentioned a roadmap with free resources.
A thread๐งต๐
1. First Month Foundations of Data Analysis
A. Corey Schafer - Python Tutorials for Beginners:
B. StatQuest with Josh Starmer - Statistics Fundamentals:
C. Ken Jee - Data Analysis with Python
A. Corey Schafer - Python Tutorials for Beginners:
B. StatQuest with Josh Starmer - Statistics Fundamentals:
C. Ken Jee - Data Analysis with Python
2. Second Month - Advanced Data Analysis Techniques
A. Sentdex - Machine Learning with Python
B. StatQuest with Josh Starmer - Machine Learning Fundamentals
C. Brandon Foltz - Business Analytics
A. Sentdex - Machine Learning with Python
B. StatQuest with Josh Starmer - Machine Learning Fundamentals
C. Brandon Foltz - Business Analytics
Python project ideas for beginners with source code
A thread ๐งต๐
A thread ๐งต๐
Python for data science beginners roadmap
A thread ๐งต๐
A thread ๐งต๐
1. Python Basics
Codecademy's Python Course (codecademy.com/learn/learn-pyโฆ)
Python for Everybody Course (py4e.com)
Codecademy's Python Course (codecademy.com/learn/learn-pyโฆ)
Python for Everybody Course (py4e.com)
2. Data Analysis Libraries
NumPy User Guide (numpy.org/doc/stable/useโฆ)
Pandas User Guide (pandas.pydata.org/docs/user_guidโฆ)
Matplotlib Tutorials (matplotlib.org/stable/tutoriaโฆ)
NumPy User Guide (numpy.org/doc/stable/useโฆ)
Pandas User Guide (pandas.pydata.org/docs/user_guidโฆ)
Matplotlib Tutorials (matplotlib.org/stable/tutoriaโฆ)
How should you ๐ฅ๐๐๐ซ๐ง ๐๐จ๐ฐ๐๐ซ ๐๐? ๐
๐งต
๐งต
It's easy to be overwhelmed by how broad #PowerBI is๐
If you're starting out, here's the path I recommendโฌ
If you're starting out, here's the path I recommendโฌ
๐ ๐๐๐ญ๐ ๐๐จ๐๐๐ฅ๐ข๐ง๐
Begin by learning to organize data into tables, create relationships, and add calculated columns and measures.
This is the most important part of your journey, as understanding the #data is always the first step in Power BI development.
Begin by learning to organize data into tables, create relationships, and add calculated columns and measures.
This is the most important part of your journey, as understanding the #data is always the first step in Power BI development.
โถ๏ธPractice Writing SQL Queries using Real
Dataset ๐ฏ
๐งต
Dataset ๐ฏ
๐งต
โThe very first thing, we must do when writing #SQL queries, is to understand the underlying data. Once we understand the data and how this data is stored across different tables, it becomes much simpler to write SQL #Queries to retrieve any information from that dataโ
โ
List of SQL Queries:
We shall write SQL #Queries using this data. For each of these queries, you would find the problem statement and then the screen shot of the expected output. Under each of these 20 problem statement
We shall write SQL #Queries using this data. For each of these queries, you would find the problem statement and then the screen shot of the expected output. Under each of these 20 problem statement
Considering switching to a ๐ ๐๐ข๐ฝ๐ ๐๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ role?
My thought in the ๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
My thought in the ๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
Usually MLOps Engineers are professionals tasked with building out the ML Platform in the organization.
๐
๐
This means that the skill set required is very broad - naturally very few people start off with the full set of skills you would need to brand yourself as a MLOps Engineer. This is why I would not choose this role if you are just entering the market.
๐
๐
What is the difference between Splittable and Non-Splittable Files?
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
You are very likely to run into a ๐๐ถ๐๐๐ฟ๐ถ๐ฏ๐๐๐ฒ๐ฑ ๐๐ผ๐บ๐ฝ๐๐๐ฒ ๐ฆ๐๐๐๐ฒ๐บ ๐ผ๐ฟ ๐๐ฟ๐ฎ๐บ๐ฒ๐๐ผ๐ฟ๐ธ in your career. It could be ๐ฆ๐ฝ๐ฎ๐ฟ๐ธ, ๐๐ถ๐๐ฒ, ๐ฃ๐ฟ๐ฒ๐๐๐ผ or any other.
๐
๐
Also, it is very likely that these Frameworks would be reading data from a distributed storage. It could be ๐๐๐๐ฆ, ๐ฆ๐ฏ etc.
๐
๐
So how do we implement ๐ฃ๐ฟ๐ผ๐ฑ๐๐ฐ๐๐ถ๐ผ๐ป ๐๐ฟ๐ฎ๐ฑ๐ฒ ๐๐ฎ๐๐ฐ๐ต ๐ ๐ฎ๐ฐ๐ต๐ถ๐ป๐ฒ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐ ๐ผ๐ฑ๐ฒ๐น ๐ฃ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ in ๐ง๐ต๐ฒ ๐ ๐๐ข๐ฝ๐ ๐ช๐ฎ๐?
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
Letโs zoom in:
๐ญ: Everything starts in version control: Machine Learning Training Pipeline is defined in code, once merged to the main branch it is built and triggered.
๐
๐ญ: Everything starts in version control: Machine Learning Training Pipeline is defined in code, once merged to the main branch it is built and triggered.
๐
๐ฎ: Feature preprocessing stage: Features are retrieved from the Feature Store, validated and passed to the next stage. Any feature related metadata is saved to an Experiment Tracking System.
๐
๐
How do we ๐๐ฒ๐ฐ๐ผ๐บ๐ฝ๐ผ๐๐ฒ ๐ฅ๐ฒ๐ฎ๐น ๐ง๐ถ๐บ๐ฒ ๐ ๐ฎ๐ฐ๐ต๐ถ๐ป๐ฒ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐ฆ๐ฒ๐ฟ๐๐ถ๐ฐ๐ฒ ๐๐ฎ๐๐ฒ๐ป๐ฐ๐ and why should you care to understand the pieces as a ML Engineer?
Find out in the ๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
Find out in the ๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
Usually, what is cared about by the users of your Machine Learning Service is the total endpoint latency - the time difference between when a request is performed (1.) against the Service till when the response is received (6.).
๐
๐
Certain SLAs will be established on what the acceptable latency is and you will need to reach that. Being able to decompose the total latency is even more important as you can improve each piece independently. Let's see how.
๐
๐
El ecosistema de ingenierรญa de datos evoluciona a altรญsima velocidad. Ya es tiempo de que subas de level y conozcas mรกs allรก de numpy, pandas y matplotlib.
๐Abro hilo pythรณnico
๐งต[1/x]
#python #dataengineering
๐Abro hilo pythรณnico
๐งต[1/x]
#python #dataengineering
Redpanda ๐ผ : redpanda.com
Redpanda ofrece un performance superior a Apache Kafka y manteniendo la compatibilidad con el API.
ยฟSerรก tan poderoso como Google PubSub?
๐งต[2/x]
#python #dataengineering
Redpanda ofrece un performance superior a Apache Kafka y manteniendo la compatibilidad con el API.
ยฟSerรก tan poderoso como Google PubSub?
๐งต[2/x]
#python #dataengineering
DuckDB ๐ฆ : duckdb.org
DuckDB nos permite hacer OLAP desde nuestro navegador web y tener un motor que funciona bastante bien con Parquet. MotherDuck motherduck.com estรก buscando ofrecer como Saas DuckDB a gran escala.
๐งต[3/x]
#python #dataengineering
DuckDB nos permite hacer OLAP desde nuestro navegador web y tener un motor que funciona bastante bien con Parquet. MotherDuck motherduck.com estรก buscando ofrecer como Saas DuckDB a gran escala.
๐งต[3/x]
#python #dataengineering
