-: So let's begin specifically talking about scalability.
How do you design systems
that can handle millions of people hitting it all at once?
At Amazon, when I was there,
it was routine for our servers to have to handle
tens of thousands of transactions per second.
I'm sure it's gotten even worse since I've left,
but modern techniques allow you
to horizontally scale these systems
such that they can actually handle that traffic.
And if you get more traffic,
you just throw more servers at it.
So let's see how that works exactly.
Let's start by talking about a single-server design.
So we're gonna start with a design that does not scale,
and we're gonna work our way up to one that does.
So in a sort of legacy single-server design,
you have a bunch of clients
out there in the internet, right?
People running their phones or laptops or PCs,
whatever they might be.
And they are coming through the internet,
which is what this cloud represents.
And they talk to one box on the other side.
So my personal website, for example, is set up this way.
It doesn't get a whole lot of traffic
and I don't really care too much
if it goes down for a while.
So, hey, one server's enough for that,
for my personal thing where I keep my hobbies.
No big deal.
I make sure the data's backed up somewhere
so I can restore it if I need to,
but it just doesn't get enough traffic
for me to care about it that much.
And I don't really care if it goes down
for a length of time either.
That's really the only situation
where you'd want to deal with a single-server design.
But sometimes that is the right solution.
Maybe there's some little internet tool
that just isn't that important
and you don't have a lot of budget for,
and you don't want to spend a lot of time maintaining it.
So it still has its place.
That's just running HTTP of some sort
and maybe there's WordPress or something on top of that.
I don't know.
But the protocol is HTTP, generally speaking,
talking to the web.
And there's probably some database as well.
Again, for a really small website,
maybe that database resides on that server itself.
Again, that's how my personal web server runs.
That's just serving stuff about my hobbies.
Why spend more money on it than that?
It just has a little MySQL instance running on there
in the same server as my Apache instance.
And it's all on WordPress or whatever.
Doesn't matter.
But of course, if you're gonna be doing this
for a real commercial system,
this is a terrible idea, right?
I mean, that host is a single point of failure.
So if that box goes down, I'm gonna have a really bad day
if I care about what that box was doing, right?
I'm gonna have to go and provision a new server.
I'm gonna have to restore everything on it from a backup,
and I'll probably have to go switch the DNS entry somewhere
to point to that new server
so that people can actually hit it.
And this does happen to me pretty often
with my personal site.
It's kind of a pain.
And if this was happening on a real commercial system
where I'm getting paged about it at two in the morning,
I would not want to deal with that.
So generally speaking,
unless you're specifically being asked
to develop something that is unimportant in small scale,
which is pretty unlikely in a system design interview,
this is not the approach you want to take.
So how can we make this better?
Well, one thing we can do
is to separate out the database, at least.
So again, let's start at the top here.
We have all the clients in the world out there
that want to hit your system,
and they're connecting to you through the internet somehow.
They're being routed from all over the globe to someplace.
And again, in this case,
we'll just have a single HTTP server,
but we're gonna at least
split out the database part of this, right?
Well, that's a little bit better.
So at least now I'm not gonna lose my web server
and data and my pages and the database itself
all at the same time.
They're actually scaled independently.
So that's a little bit better from a resiliency standpoint.
Not much though, right?
But the real benefit of this
is that I can scale these two things independently.
So let's say I have an application
that's really hitting the database hard.
That might be eating up a lot of CPU time on my web server,
where I'm spending more time in my database processes
than in my HTTP processes, right?
Maybe MySQL is eating up all my resources
and not leaving enough resources
for the web to actually serve enough connections.
So kind of the next step up
for scaling up a system like that,
where we have one server,
is to at least scale out the database.
And now I have two computers out there instead of one
that I can spread resources around,
and I can scale those independently, right?
So I can choose a beefier box for the database
if I have a really data-intensive application
that's doing complicated joins or whatever it is,
and maybe something less powerful
for the front-end web server, if that's appropriate.
So that gives me a little more control
over how I scale these things, which is good.
But still, these are both
single points of failures now, right?
So if my database goes down, my website goes down.
If my web server goes down, my website goes down.
So from a resiliency standpoint, not really any better.
So let's talk about one way of scaling up further.
Let's say that I do want to take more traffic
than I could have handled on that one server originally.
Again, we start with our clients
connecting through the internet,
and maybe I just throw a bigger server at it.
This is what we call vertical scaling.
So vertical scaling
means that instead of adding more servers,
I just add a bigger server,
and that can get you to a certain point, right?
For a long time with databases,
that's kind of what we did.
You started off with an Oracle instance running somewhere,
and as you threw more and more traffic at it
and more and more data at it,
the way to make it scale was to just get a bigger
and more expensive database host.
And these things got really, really expensive,
really, really quickly.
Same thing applies to your web server here.
So I have a big, fat host here running HTTPD presumably,
and maybe instead of a little virtual machine
running on one CPU,
I'm running a whole machine here now or something,
or at least a bigger virtual machine, right?
So to some extent,
you can just throw more hardware at the problem
and get away with that for a little while.
So if I have a little personal website that, again,
I don't care too much about
and it suddenly gets a surge of traffic,
maybe I deal with that by just throwing
a bigger virtual machine at it.
And in fact, that's what I'm doing right now.
I've had some websites of my own
that kind of buckled under the pressure
of crawlers hitting them, in particular.
And by just throwing a bigger virtual machine at it,
it solved that problem.
But again, not something
you generally want to be doing in production
because vertical scaling has its limits.
You can only get so big of a computer
before you can't get anything bigger.
There's only so many CPUs you can get in a system,
there's only so much memory.
At some point, you're gonna hit a wall.
So vertical scaling only gets you so far,
but you need to know what it is.
Similarly, I can make the database server bigger too.
We talked about that a little bit
in the context of like a giant Oracle database.
Same idea.
Maybe I need more storage.
Maybe I need more CPU power for processing those queries.
To a point, I can just throw bigger and bigger
and more powerful machines at my database server and scale,
to an extent, just that way.
The only good thing about this
is that there's not a lot of things to maintain, right?
So in this system,
I still only have two hosts to take care of.
So that's kind of a plus, right?
The fewer computers that I have,
the fewer servers that I have,
the fewer things that are likely to break
in any given day, okay?
But when it does break, I'm in trouble, right?
Either of those things goes down, my website's down
and it's gonna stay down
until I can replace that host with something else, right?
Maybe that's okay, probably isn't though.
So again, just to summarize there,
servers only come so large.
That is the problem.
So you still have single points of failure here.
The only upside of this
is that you have fewer servers to maintain.
Now, horizontal scaling.
This is kind of where it's at.
So it's a pretty safe bet
that for most modern system design problems,
where you're being asked to develop a system at large scale,
they want you to take a more horizontal scaling approach.
So again, you have your clients out there in the world
hitting you through the internet.
The idea here is that instead of a single server,
you have multiple servers
and you have some sort of a load balancer
between the internet and those servers
that are distributing that load in some fair, even manner.
We could talk about load balancers strategies.
There's round robin, there's partitioning it,
there's load bouncers that can actually
look at the available capacity on a server
and try to route traffic intelligently that way.
So that's a whole other can of worms.
But the idea is that there is some,
either device or software process running somewhere
that is distributing that load coming in from the internet
to a whole fleet of servers.
So the beauty of that is that if one of those goes down,
assuming I have enough excess capacity provisioned,
the user doesn't need to know.
The load bouncer can realize that a host is down
and reroute the traffic around it automatically
so that you have no downtime.
Pretty cool, right?
Maybe they all still point to a single database.
We could scale that out as well, if we wanted to,
but in this case,
maybe our database isn't really a choke point
and we have good redundancy on it somehow.
We'll get more to database scaling later on,
but at least in this architecture,
our front-end web servers are gonna keep on serving, right?
Maybe they have some sort of a cache
in front of the database,
so they can still do something meaningful
if that database were to go down.
But the idea of horizontal scaling
is that as I get more and more traffic,
I just add more and more servers to the fleet
and the load balancer will distribute that traffic
amongst those servers.
So you can see, you can pretty much scale infinitely
with this idea, right?
If I have millions of transactions coming in,
throw in enough servers in there and you'll be good.
There are some finer points about,
where these are located geographically
and what servers, what data centers.
We'll get into that.
But this is the basic idea of horizontal scaling.
It allows an infinite,
well, practically infinite way of scaling things up,
where the more servers you can throw in there,
the more traffic you can handle.
There's really no bound to it in this case.
The downside, of course,
is that there's more stuff to maintain.
So let's talk about stateful and stateless stuff here.
So the catch here is that this really only works well
if your web servers are what we call stateless.
What I mean by that is that subsequent requests
should not depend on something being stored on that server
from a previous request, okay?
Now, you can store stuff in the database.
That's fine.
But the servers themselves,
those web servers need to be stateless
because I don't know
where that previous request got routed to.
So I cannot assume on any given server
that I have any information about the previous hits
from that given user, right?
So a stateless web server means that any web server
can handle any request at any time, okay?
Again, it's okay to use the database
to tie those together and have sort of a history,
but any individual server cannot assume that that server
is the one that actually served
previous requests to a given user.
That's what I mean by stateless.
It just means that we're baking in the assumption
that any request for a user could have gone to any server.
And we cannot assume that our server
knows anything about those previous requests directly.
Okay?
So how do you go about choosing one of those architectures?
Well, in general, a theme in this course
that I'm gonna be preaching is simplicity.
You should always choose the simplest architecture
that meets your projected requirements,
but no simpler than that.
So, yeah, I mean, if you're being asked
to develop something on the internet
where it's just like a little phone tool, maybe,
for one team, there's probably not a whole lot of point
in over-engineering that
and building some huge horizontally scalable system, right?
Maybe a simpler one is okay.
Maybe vertical scaling is okay in that instance.
But the truth is, for most of these system design interviews
at the big companies, that's not the sort of thing
they're gonna ask you to design.
They're gonna ask you to design YouTube
or design Google, design some massive system
where you need to have that scalability.
So usually, it's a safe bet that you want to go
with a more horizontally scaled,
horizontally partitioned architecture there instead.
So let's dive into that in more depth
in our next section here, but big takeaway here,
vertical scaling just means that you're throwing
bigger machines at the problem.
Horizontal scaling means you're throwing
more machines at the problem and distributing the load
throughout those different machines, okay?
That's the main thing to take away from this lecture.