Discover and read the best of Twitter Threads about #systemDesign

Most recents (17)

How Canva efficiently removes duplicates from billions of images

T๐˜„๐—ถ๐—ป ๐—ง๐—ฟ๐—ผ๐˜‚๐—ฏ๐—น๐—ฒ๐˜€: ๐—›๐—ผ๐˜„ ๐—–๐—ฎ๐—ป๐˜ƒ๐—ฎ ๐—–๐—ผ๐—ป๐—พ๐˜‚๐—ฒ๐—ฟ๐˜€ ๐—•๐—ถ๐—น๐—น๐—ถ๐—ผ๐—ป๐˜€ ๐—ผ๐—ณ ๐—œ๐—บ๐—ฎ๐—ด๐—ฒ๐˜€ ๐˜„๐—ถ๐˜๐—ต ๐—˜๐—ณ๐—ณ๐—ถ๐—ฐ๐—ถ๐—ฒ๐—ป๐—ฐ๐˜†
๐ŸŽฏย #Refactor75ย โžกDay41 #systemdesign #content #engineers Image
Canva has more than 20 billion user-uploaded images, with 50 million uploaded daily.

Dealing with a large variety of media creates difficulties when it comes to moderating and minimizing unnecessary duplicate content
To solve this problem, Canva uses perceptual hashing with an internally built reverse image search system
Read 9 tweets
[1/13] As we're witnessing recurring layoffs, it's crucial to stay prepared. This thread is based on my experience of interviewing with Western companies in early 2022. I'll share insights & resources that proved invaluable. Hopefully, it aids your #InterviewPreparation journey.
[2/13] Struggling with HTTPS and digital certificates? This comprehensive guide will clear your doubts - cryptobook.nakov.com. A strong foundation in #CyberSecurity is essential in today's digital landscape.
[3/13] Mobile System Design was a challenging topic due to scarce resources. Until I found this repository - github.com/weeeBox/mobileโ€ฆ. It's a goldmine of information for any aspiring #MobileSystemDesign enthusiast.
Read 13 tweets
After months of interviewing for my next #SoftwareEngineering gig, I wanted to give my two cents on the process 1/N ๐Ÿงต
Most notable for me was how drastically different the interview experience was compared to my last time job hunting 6 years ago. 2/N
Back then, my first coding interview was to implement a roman numeral parser on a whiteboard in a language of the interviewer's choice. It was an experience that felt much closer to hazing than a relevant technical assessment. 3/N
Read 15 tweets
What happens behind the scenes when we ๐ฌ๐ก๐จ๐ฉ ๐จ๐ง๐ฅ๐ข๐ง๐ž?
.
.
Disclaimer: I have limited knowledge of the eCommerce system. The diagram below is based on my research. Please suggest better names for the components or let me know if you spot an error.
The diagram shows the 4 key business areas in a typical e-commerce company: procurement, inventory, eComm platform, and transportation.
1๏ธโƒฃ Procurement
Step 1 - The procurement department selects suppliers and manages contracts with them.

Step 2 - The procurement department places orders with suppliers, manages the return of goods, and settles invoices with suppliers.
Read 9 tweets
How does Netflix scale push messaging for millions of devices?

This post draws from an article published on Netflixโ€™s engineering blog. Hereโ€™s my understanding of how the online streaming giantโ€™s system works. Image
๐‘๐ž๐ช๐ฎ๐ข๐ซ๐ž๐ฆ๐ž๐ง๐ญ๐ฌ & ๐ฌ๐œ๐š๐ฅ๐ž
- 220 million users
- Near real-time
- Backend systems need to send notifications to various clients
- Supported clients: iOS, Android, smart TVs, Roku, Amazon FireStick, web browser
๐“๐ก๐ž ๐ฅ๐ข๐Ÿ๐ž ๐จ๐Ÿ ๐š ๐ฉ๐ฎ๐ฌ๐ก ๐ง๐จ๐ญ๐ข๐Ÿ๐ข๐œ๐š๐ญ๐ข๐จ๐ง
1. Push notification events are triggered by the clock, user actions, or by systems.

2. Events are sent to the event management engine. Image
Read 8 tweets
How does Disney Hotstar capture 5 Billion Emojis during a tournament?

Dedeepya Bonthu [1] wrote an excellent engineering blog that captures this nicely. Here is my understanding of how the system works. Image
1. Clients send emojis through standard HTTP requests. You can think of Golang Service as a typical Web Server. Golang is chosen because it supports concurrency well. Threads in GoLang are lightweight. Image
2. Since the write volume is very high, Kafka (message queue) is used as a buffer. Image
Read 10 tweets
How does Twitter work? Letโ€™s take a look at it from the architectural point of view before Elon takes it.

๐“๐ก๐ž ๐‹๐ข๐Ÿ๐ž ๐จ๐Ÿ ๐š ๐“๐ฐ๐ž๐ž๐ญ:
1๏ธโƒฃ A tweet comes in through the Write API.
2๏ธโƒฃ The Write API routes the request to the Fanout service.

#twitter #systemdesign
3๏ธโƒฃ The Fanout service does a lot of processing and stores them in the Redis cache.
4๏ธโƒฃ The Timeline service is used to find the Redis server that has the home timeline on it.
5๏ธโƒฃ A user pulls their home timeline through the Timeline service.
๐’๐ž๐š๐ซ๐œ๐ก & ๐ƒ๐ข๐ฌ๐œ๐จ๐ฏ๐ž๐ซ๐ฒ
๐Ÿ”น Ingester: annotates and tokenizes Tweets so the data can be indexed.
๐Ÿ”น Earlybird: stores search index.
๐Ÿ”น Blender: creates the search and discovery timelines.

๐๐ฎ๐ฌ๐ก ๐‚๐จ๐ฆ๐ฉ๐ฎ๐ญ๐ž
๐Ÿ”นHTTP push
๐Ÿ”นMobile push
Read 6 tweets
Implementing Distributed Transactions โšก [in a gist]

Distributed Transactions are not theoretical; they are very well used in many systems. An example of it is 10-min food/grocery delivery.

๐Ÿ‘‡
โœจ The UX we want is: Users should see orders placed only when we have one food item and a delivery agent available to deliver.

A key feature we want from our databases is atomicity. Our storage layer can choose to provide it through atomic operations or transactions.

๐Ÿ‘‡
We will have 3 microservices: Order, Store, and Delivery.

Important decision: Store services have food, and every food has packets that can be purchased and assigned.

Hence, instead of just playing with the count, we will play with the granular food packets while ordering.

๐Ÿ‘‡
Read 14 tweets
Why is a solid-state drive (SSD) fast?

โ€œA solid state drive reads up to 10 times faster and writes up to 20 times faster than a hard disk drive.โ€ [1].
โ€œAn SSD is a flash-memory based data storage device. Bits are stored into cells, which are made of floating-gate transistors. SSDs are made entirely of electronic components, there are no moving or mechanical parts like in hard drives (HDD)โ€ [2].
The diagram below illustrates the SSD architecture.

Step 1: โ€œCommands come from the user through the host interfaceโ€ [2]. The interface can be Serial ATA (SATA) or PCI Express (PCIe).
Read 10 tweets
#systemdesign
What is CAP theorem?
CAP stands for Consistency, Availability, and Partition tolerance. The theorem states that you cannot achieve all the properties at the best level in a single database.
But we can only pick two out of three at a time and that totally depends on our prioritize based on requirements.
What is Data Consistency?
Consistency means that any read request will return the most recent write.
Read 5 tweets
#systemdesign #ConsistentHashing
Does load balancer use hashing?
Hashing is one of the common methods used in Load Balancing for distributing requests among the web servers.
How load balancer uses hashing?
We have something known as request_id that is uniquely and randomly generated. Each request by the client is uniquely identified by this request_id. Load balancer hashes the request_id with the proper hash function and distributes the requests.
But here comes the problem, if one server is added or removed from the pool of servers, we need to rehash all requests from the start and distribute them to the servers. So we can use the Consistent Hashing technique to solve this problem.
Read 5 tweets
Here is a simplified design diagram that explains how a stock exchange such as Nasdaq or Newyork exchange works under the hood: #developers #systemdesign

Step 1: A client places an order via the brokerโ€™s web or mobile app.

Step 2: The broker sends the order to the exchange.1/6 Stock exchange design
Step 3: The exchange client gateway performs operations such as validation, rate limiting, authentication, normalization, etc., and sends the order to the order manager.

Step 4-5: The order manager performs risk checks based on rules set by the risk manager.2/6
Step 6: Once risk checks are passed, the order manager checks if there is enough money in the wallet.

Step 7 - 9: The order is sent to the matching engine. The matching engine sends back the execution result if a match is found. 3/6
Read 6 tweets
10 System design concepts that can help you in coding interviews
1. Load Balancing
2. Caching
3. Data Partitioning
4. Indexes
5. Proxies
6. Redundancy and Replication
7. SQL vs. NoSQL
8. Distributed Systems
9. CAP Theorem
10.Consistent Hashing
#systemdesign #programming #design
And, if you need a resource to master system design, you can checkout Grokking the System Design on @EducativeInc
educative.io/courses/grokkiโ€ฆ
And, if you need more choices, I have shared few more courses here
medium.com/javarevisited/โ€ฆ
Read 8 tweets
Ever wonder how replication happens between Master and Replica? How changes on Master propagates to Replica?

This is a short thread of how it happens

#systemdesign #distributedsystems

๐Ÿงต
Any write operation happening on the Master is logged in the Replication log file as an event. The format in which these events are logged in the Log file is called Replication Format.

The two common Replication formats:

- Statement-based format
- Row-based format
โœจ Statement-based Format

The Master records the operation as an event in its log, and when the Replica reads this log, it executes the same operation on its copy of data.

This way, the operation on the Master is executed on the Replica, which keeps it in sync with the Master.
Read 11 tweets
Just wrapped up my 1:1 call with one of my cohort-ian and we ended up building an infinitely scalable Distributed Task Scheduler, AWS CloudWatch Events, DKron, and Quartz Scheduler, in under 30 minutes.

When foundations are clear, no system is harder to design ๐Ÿ’ช

#systemdesign
The features we discussed and designed were:

- Infinite task ingestion
- 30 second SLA of execution
- Execution Framework that supports Binaries, Scripts, Remote Executions
- Fault tolerance of Scheduler Nodes
- Repeatability of tasks
- Exactly-once schedule and execution
The design we discussed did not just have random boxes of high-funda components but rather the actual tools and techs that we would be using, along with their pros and limitations. ๐Ÿ’ช
Read 7 tweets
The progression from coding to designing architectures is the practice of refactoring the structure and behavior of elements, at increasingly higher levels of scale.

#softwareDesign #softwareArchitecture
... where system elements can be

field/ var/ param -> method/ function/ procedure -> class -> aggregate/ composite/ collection -> module/ package -> application/ service -> sub-system -> system -> system of systems

#codenewbie #systemDesign
... where at each level of scale, the goal of refactoring is to win the trade-offs between time, cost, quality

#architects #management #visionary #leadership
Read 3 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!