Chapter 3: Terraform from First Principles

Chapter 3:
Terraform from First Principles

Context

When you configure infrastructure manually, either by clicking through a web console or running commands one at a time in a terminal, the result exists in two places: the system itself, and your memory. If a colleague needs to recreate the same setup, they have to ask you to walk them through it. If you need to recreate it yourself six months later, you have to reverse-engineer what you did. If someone makes a change without telling you, you might not discover it until something breaks.

This is the configuration drift problem. Drift is the gap between what you intended to build and what currently exists. It accumulates silently. A security group rule added for a quick test that never gets removed. An S3 bucket setting changed manually and never documented. A resource created in the wrong region because no one noticed the console was pointing at the wrong account.

Infrastructure as code exists to eliminate drift. When your infrastructure is defined in files, the files are the source of truth. What is in the files is what should exist. Any deviation is detectable and correctable. The infrastructure can be recreated from scratch by anyone with access to the files and the right credentials.

Terraform is the tool this project uses to manage infrastructure as code. This chapter explains how Terraform works, what its files contain, and why the configuration in this project is structured the way it is.

Concept: From Shell Scripts to Declarative Infrastructure

Before infrastructure as code tools existed, engineers automated provisioning through shell scripts. A script to create an S3 bucket, configure its permissions, set up a CloudFront distribution, and wire the two together. Shell scripts are imperative: they tell the system what to do, step by step, in a specific order. They are better than clicking through a console because they are repeatable and can be version-controlled. But they have a fundamental limitation.

A shell script that creates an S3 bucket will fail on the second run if the bucket already exists. To handle that, you add a check: if the bucket exists, skip creation. Then you add checks for every other resource. The script grows in complexity. It stops being about describing infrastructure and starts being about managing state: what exists, what does not, what needs to be created, what needs to be updated, what needs to be deleted. That logic is tedious to write and easy to get wrong.

This is the problem that declarative infrastructure tools solve. Instead of describing the steps to build infrastructure, you describe the desired end state. Terraform reads the desired state from your files, compares it to the actual state of your AWS account, and makes whatever changes are necessary to bring the two into alignment. You do not tell Terraform how to create a bucket. You tell Terraform what the bucket should look like. Terraform figures out the how.

That distinction, imperative versus declarative, is the core mental model shift required to work with Terraform effectively.

AWS Mapping: How Terraform Works

Terraform is not an AWS tool. It is a general-purpose infrastructure as code platform that supports hundreds of cloud providers and services through a plugin system. Each plugin is called a provider.

Providers

A provider is a plugin that teaches Terraform how to communicate with a specific platform. The AWS provider knows how to make API calls to create, read, update, and delete AWS resources. An API call is a request one program sends to another over a network. AWS exposes an API: a defined set of commands it accepts over HTTPS, such as "create this S3 bucket" or "attach this policy to this role." When you click through the AWS console, the console is making those same API calls on your behalf. Terraform does the same thing directly, without the interface in between. The random provider generates random values, which this project uses to create unique resource names. Both providers are used here.

Providers are downloaded when you run terraform init. They are not stored in the repository. The .terraform.lock.hcl file records the exact provider versions that were downloaded, so that future runs use the same versions rather than automatically picking up whatever happens to be newest.

Resources

A resource block in a Terraform file describes a single piece of infrastructure. It has a resource type, a local name, and a set of arguments that configure it.

resource "aws_s3_bucket" "website" {

bucket = "my-website-bucket"

}

aws_s3_bucket is the resource type, corresponding to an S3 bucket in AWS. website is the local name, used to reference this resource from other parts of the Terraform configuration. The arguments inside the block configure the bucket. This project defines several resource types in main.tf: S3 buckets, a CloudFront distribution, an IAM role, an OIDC identity provider, and supporting resources. Each is a separate block.

State

Terraform keeps a record of every resource it has created. This record is the state file: a JSON document that maps the resources in your .tf files to the actual resources in your AWS account, including their IDs and current configuration.

The state file is how Terraform knows what already exists. When you add a new resource block and run Terraform again, it reads the state file, compares it to your current configuration, and determines that only the new resource needs to be created. It does not attempt to recreate everything that is already in the state.

The state file also holds the IDs AWS assigns to resources at creation time, which Terraform needs for future updates and deletions. If you delete the state file, Terraform has no record of what it created. It will attempt to create everything from scratch on the next run, which will fail because the resources already exist in AWS.

Keeping the state file safe and accessible is critical. This project stores it in S3. Chapter 8 covers the setup.

The init, plan, apply cycle

Terraform operations follow a three-step sequence.

terraform init downloads the providers specified in the configuration and sets up the backend, which is where the state file will be stored. Run this once when you first work with a project, and again whenever you change provider versions or backend configuration.

terraform plan compares your configuration to the current state and produces a list of changes Terraform will make if you proceed. It does not change anything. It shows you exactly what will be created, modified, or destroyed. Reading the plan output before applying is the discipline that prevents accidental changes.

terraform apply executes the changes shown in the plan. Terraform handles resource dependencies automatically: if resource B requires an attribute from resource A, Terraform creates A first and passes its output to B.

You will not run any of these commands in this chapter. The backend configuration in backend.tf contains a placeholder bucket name that does not point to a real bucket. Running terraform init against the current file will fail because no bucket with that name exists in your account. Chapter 8 walks through creating your own state bucket and lock table, updating backend.tf, and then running the commands in the correct order.

Build: Read the Configuration Files

The build in this chapter is reading, not running. You are going to work through three files: backend.tf, variables.tf, and .terraform.lock.hcl. By the end, you will understand what every line does and why it is there.

Open your terminal in the aws-terraform-s3-cloudfront-cicd directory.

Reading backend.tf

cat backend.tf

The file has two top-level blocks: a terraform block and a provider block.

The terraform block configures Terraform itself. It has three sections.

required_version specifies which version of the Terraform CLI this configuration is compatible with. A constraint like ">= 1.0" means any Terraform CLI version 1.0 or higher will work.

required_providers lists the providers this project uses and where to download them from. Each provider entry specifies a source (the registry address) and a version constraint.

Look at the version constraint syntax. You will see the ~> operator, called the pessimistic constraint operator. It means "at least this version, but less than the next major version." For example, ~> 5.0 permits versions 5.0, 5.1, 5.47, and so on, but blocks version 6.0. ~> 3.6 permits 3.6 and 3.7 but blocks 4.0. The operator pins the major version and allows minor updates within it. This prevents a provider's major version upgrade from silently changing the behavior of your configuration.

The backend "s3" block tells Terraform where to store the state file. It specifies the S3 bucket name, the path within that bucket where the state file will live (called the key), the region, the DynamoDB table used for locking concurrent operations, and an encryption flag.

Note: Replace the Bucket Placeholder

The bucket value in this backend block is set to "YOUR-IDENTIFIER-tf-state". It is a placeholder, not a real bucket. Terraform backend blocks cannot reference variables, so the value has to be written into the file by hand. If you run terraform init without replacing it, the command fails because no bucket with that name exists. Chapter 8 walks through creating your own state bucket and updating this file before you run anything. A practical naming convention is [your-identifier]-tf-state: lowercase, unique to you, and immediately recognizable as a Terraform state bucket.

The provider "aws" block configures the AWS provider. It specifies the region. Rather than hardcoding a region string here, it reads from a variable, which means you can change the deployment region without editing the provider block directly.

Reading variables.tf

cat variables.tf

Variable blocks define the inputs to a Terraform configuration. Each block declares a name, a type, and typically a description. Some include a default value; others require you to supply a value at runtime.

Variables let you parameterize your configuration. Instead of hardcoding a region or a GitHub username in main.tf, you declare them as variables and reference them throughout the configuration. When you run Terraform, you supply the values either through a .tfvars file, environment variables prefixed with TF_VAR_, or interactive prompts. The resource blocks in main.tf stay clean and reusable.

The variables you will see here include aws_region, project_name, github_username, and repo_name. Each of these is referenced in main.tf.

Reading .terraform.lock.hcl

cat .terraform.lock.hcl

This file is generated by Terraform, not written by hand. It records the exact provider versions resolved when terraform init was last run. Not just the version range that satisfies the constraint in backend.tf, but the specific version installed and a cryptographic hash of the provider binary.

The purpose of this file is reproducibility. If .terraform.lock.hcl is committed to your repository, anyone who runs terraform init gets exactly the same provider versions, not whatever happens to be newest at the time they run it. The ~> constraint in backend.tf defines what is acceptable. The lock file records what was actually chosen.

Commit this file to version control. Updating it is a deliberate action: run terraform init -upgrade when you want to move to a newer provider version within the constraints you have set. Leave the .terraform/ directory itself out of version control. That directory contains the downloaded provider binaries, which are large and re-downloadable. The lock file is small and critical.

What You Just Did

You read through the three Terraform files that form the foundation of this project. You understand what a provider is, what a resource block represents, what the state file does, and why the init/plan/apply sequence exists.

You noted that the backend configuration contains a placeholder bucket name that you must replace with a bucket in your own account before any commands run. Chapter 8 walks through that.

The next three chapters each cover a specific set of resources from main.tf: S3 in Chapter 4, CloudFront in Chapter 5, IAM in Chapter 6. By the time you reach Chapter 8, you will understand every resource in the project, and setting up the state backend will be a short task rather than a confusing one.

Common Mistakes and How to Catch Them

Running `terraform apply` without reading the plan. The plan step exists so you can see what Terraform is about to do before it does it. Running apply directly, or auto-approving it in a script without reviewing the output, is how infrastructure gets accidentally deleted. Read the plan every time.

Editing or deleting the state file manually. The state file is Terraform's internal record, not a configuration file you manage. Editing it by hand or deleting it to "start fresh" breaks the relationship between Terraform's view of the world and what actually exists in AWS. To remove a resource from Terraform's control without deleting it from AWS, use terraform state rm. To bring an existing resource under Terraform management, use terraform import. Do not touch the file directly.

Not committing `.terraform.lock.hcl`. This file is generated, which makes it feel like it belongs in .gitignore. It does not. It is the record of exactly which provider versions this configuration has been tested with. Leave it out and different machines will use different provider versions, producing inconsistent behavior. Keep .terraform/ out of version control. Keep .terraform.lock.hcl in it.

Skipping version constraints. Omitting required_version and required_providers constraints is common in quick personal projects. It becomes a problem when a provider releases a major version with breaking changes and Terraform silently adopts it on the next run. Set constraints from the start. Update them deliberately.

Self-Documentation

Add the following to your Book 1 — Static to Production knowledge base page:

Define in your own words: provider, resource block, state file, and the purpose of each step in the init/plan/apply cycle

Record the ~> constraint operator with a concrete example of what it permits and what it blocks

Note the backend bucket placeholder flagged in this chapter. Write down what you need to do about it before running Terraform for the first time.

Note that you have not run any Terraform commands yet, and record why.