- ran out of interest in taking notes about 60% of the way through
Cloud Computing: on-demand delivery of IT resources, managed over the internet
all AWS services have APIs for you to use to manage everything
options for interacting with AWS (these all use AWS APIs)
- AWS Management Console (browser based)
- AWS Command Line Interface (CLI)
- AWS Software Development Kit (SDKs)
- other tools are available
- AWS Elastic Beanstalk (for EC2 environments)
- AWS Cloudformation (has broader support)
- use JSON or YML config files (cloudformation templates) to declare the setup you want
- update a template and AWS will handle the API calls for all the setup
aws api logs
for computations
compute-as-a-service model
can share host server with other instances (virtual machines)
- multitenancy
- hypervisor manages the server running multiple instances
- instances are secure and separate from each other
vertically scaling an instance
- take more or less resources (memory, cpus)
instance types
- each type in grouped under an instance family, and is optimized for different tasks
- families
- general purpose: balanced resources, ok for web services and small/medium databases
- compute optimized: intensive computing, ok for gaming servers, an intensely used web services, and batch processing
- memory optimized: for processing large datasets in memory
- accelerated computing: hardware accelerators, ok for graphics, data pattern matching, floating point calculations
- storage optimized: for high sequential read/write access to large datasets, ok for distributed file systems, data warehousing, and OLTP (online transaction processing)
auto-scaling of instances
- scale up: add more power/resources to the existing instances
- scale out: add more instances
- set min and max capacity, plus desired capacity (the amount to use by default)
uses Docker containers
- the container includes the code, any dependencies, and configuration
- containers are isolated from each other similar to a virtual machine
cluster: a collection of containers
these can run on EC2, but don't have to
- Amazon Elastic Container Service (ECS)
- Amazon Elastic Kubernetes Service (EKS)
- you can host these on Fargate instead
load balancer: directing/routing traffic among the available instances of a service
Elstastic Load Balancer
- AWS's flavor of load balancer that you can turn on
- a managed service
- a regional service
- cost of one ELB does not change when the number of servers it is balancing between changes
AWS messaging and queuing system
payload: the data contained in a message
queue: holds messages until they are processed
sends out messages
- can send a message to a specific place
- or publish a message and have subscribers
serverless: you cannot see or access the infrastructure (provisioning, scaling, maintenance is all handled by AWS)
- your code runs on servers, but you don't need to provision or manage them
a function with a trigger
- one instance per trigger that occurs
- runtime limited to 15 minutes
a serverless way to host ECS/EKS
Cloudfront is a Content Delivery Networks (CDN)
ex of a CDN: cache a copy of data physically close to a city so that users there can access it quickly
uses edge locations around the world to be close to users
- edge locations are different than regions
block level storage
- for files
- updates just the pieces of storage that have been edited
- ex: your harddrive
instance store volumes
- physical storage attached to the server your EC2 instance is running on
- this data is deleted if you stop your instance (because when it is restated, it will likely be on a different server)
- so it is for ephemeral data
attach EBS volumes to your EC2 instance
- this data persists when your EC2 instance stops and restarts
- allows incremental backups (snapshots)
- sized up to 16 Terabytes
an availability zone resource
data is stored as objects in buckets
- unlimited storage
- max object size is 5 Terabytes
- can version objects (save previous versions)
- create multiple buckets
- security for who can access objects
tiers
- S3 standard: high durability, data stored in at least three facilities at a time
- S3 static web hosting: for static web pages hosted from an S3 bucket
- S3 infrequent access (S3 Standard-IA): for backups and disaster recovery
- S3 glacier: for long term archives, can lock "vaults" to stop edits
lifecycle policies
- can move data between tiers automatically
optimized for write once, read many
every object automatically has a url
object storage: each object is a complete file, each update will cause the entire object to be saved as a whole
a managed Linux file system
multiple EC2 instances can access it at once
automatically scales to give you more storage
a regional resource
supports: mysql, postgresql, oracle, microsoft sql server, and others
"lift-and-shift" migration: copies your local database to the cloud
- Amazon Database Migration Service (DMS)
- homogeneous: from a database type to a matching type (such as between relationship databases)
- heterogeneous: use schema conversion tool to get from one db to a new schema type
build for business analytics
for realtime read/write functionality
a database solution with mysql or postgres
- with data replication
- and continuous backups
for realtime read/write functionality
a serverless database
- you don't need to manage the infrastructure
- create tables of items and attributes
- millisecond response time
- a nosql database, a nonrelational database
- so can only handle simple queries against one table at a time
for realtime read/write functionality
data warehouse
- for historical analytics and querying multiple databases
- historical data refers to data that is set (no longer being edited)
for content management
a graph database, like for social networks
100% immutability, to data can be removed from the audits
TODO find out more about custom metrics and cloudwatch alarms and cloudwatch dasboards
CloudWatch > Logs Insights
query example
fields @timestamp, StatusCode, @message
| sort @timestamp desc
| filter @message ~= "provisioning"
get fields:timestamp and message
sort: by timestamp descending
where: message includes the text "provisioning"
fields can be parsed with regex
each * part will be output as a new named field
# with @message format 'stuff stuff,"@l":"Debug",stuff stuff'
fields @timestamp, @message
| parse @message '*"@l":"*"*' as message_before, loggingType, message_after
| filter loggingType = "Error"
| display loggingType, message_after #or comment out this line to see full messages as normal
| sort @timestamp desc
| limit 20
another way to get this filtering is
fields @timestamp as Timestamp, @@l as LogLevel, ServiceName, @@m
| filter LogLevel == "Error"
| sort @timestamp desc
| limit 20
or more specifically for current job
fields @timestamp as Timestamp, @@l as LogLevel, ServiceName, @@m
| filter LogLevel != "Error" and LogLevel != "Verbose" and LogLevel != "Debug" and not isblank(LogLevel)
| sort @timestamp desc
| limit 20
useful fields
| filter StatusCode ~= '5'
looking for 5xx response codes
trying to find error responses
fields @timestamp, StatusCode, @message
| sort @timestamp desc
| filter not isblank(StatusCode)
aws route 53: a domain name service (DNS): a DNS translates website names into ip addresses
- runs in edge locations
- can also register domain names
routing policies
- latency-based routing
- geolocation dns: routing is based on where the user is located
- geoproximity routing
- weighted round robin
your own private network within AWS
- define your ip range
- can define what traffic can enter the VPC
internet gateway (IGW): an access point for public traffic to enter your VPC
- (this is not automatic)
virtual private gateway: an access point for private traffic (encrypted internet traffic) to enter you VPC
- (this is not automatic)
- make a VPN connection between a private network and your VPC
AWS direct connect:
- a dedicated private connection (not shared with normal internet traffic)
- a dedicated fiber connection (wire) between your data center and an AWS data center
gateways are configured to be "traffic in" or "traffic out"
subnets: used to control access to the gateways
- public subnets have access to the internet gateways
- has a Network ACL
network access control lists (Network ACL)
- define which packets are allowed to cross the boundary? in either direction
- stateless: checks every single packet
- allows all in and out traffic by default
security groups
- ECS instance level access security
- what traffic is allowed to access a particular ECS instance?
- by default, no incoming traffic is accepted and all outgoing traffic is allowed
- stateful: has some memory for what to allow in
- will allow the responses to outgoing packets back in
shared responsibility model: AWS manages security "of" the cloud, customer manages the security "in" the cloud
aws responsible
- physical layer
- network layer
- hypervisor layer
customer
- operating system (when on EC2, when not serverless)
- applications
- data
AWS root account user: the user will complete access to your whole system
- do not use this often, create other users with granular access
multifactor authentication (MFA)
- requires a randomized token to login
create new users
- by default, they have no permissions
- create an IAM policy and associate it to a user to grant them access
- IAM policy: a json document that lists the api calls the user can make
- can also use groups to connect users and policies
- can use roles to give temporary permissions for an amount of time
- an identity will assume a role, and ignore all its normal permissions for a time
least privileged principle: give users minimal access to what they need
manage multiple AWS accounts, separate resources
- centralized management
- consolidated billing
- hierarchical groupings of accounts
- organize into OU (Organizational Units)
- service and api action access control
Amazon has data centers located all over the world
- they are organized into regions
- this is helpful for providing high availability, because a disaster in one region does not block processing in other regions
- regions are built to be geographically close to high-demand population centers
- the regions are interconnected
- users can decide which region to run their services out of
- you data is only stored and handled in the region(s) you specify (for security)
- the AWS features available vary between regions
- pricing varies between regions
availability zone (AZ "ay-zed"): one or more discrete data center
- reach region is made up of one or more availability zones
- a single EC2 instance will only run in one data center
- run multiple instances so they can run in multiple data centers, and even in multiple availability zones
- the geographically distributed availability zones provide high availability
Amazon will install a little data center inside your building
- Amazon will manage this data center
- make use of AWS infrastructure and services in your on-premises data center