Introduce S3-native state locking

[bschaatsbergen]

This draft PR prototypes state locking for Terraform’s s3 backend using a .tflock file. It uses Amazon S3’s recently introduced conditional writes feature to implement a locking mechanism. When a lock is acquired, other Terraform clients attempting to lock the same Terraform state file will encounter an error until the lock is released.

Context

The internal/states/statemgr package provides interfaces and functionality for state managers, including the s3 backend. These state managers are responsible for writing and retrieving state from persistent storage.

This PR focuses on the methods implemented by s3: Lock and Unlock.

Currently, the s3 backend implements state locking using Amazon DynamoDB, by writing a LockID and a digest. Terraform uses these values in the Lock and Unlock methods in the s3 backend to manage state locking and unlocking. While DynamoDB has long been used for state locking, leveraging Amazon S3’s newly released conditional writes feature offers an S3-native approach. By using S3 directly, we eliminate the need for an additional AWS component, simplifying the backend architecture.

Implementation

A .tflock file, containing lock information, is used to represent the lock on the state file. This file contains lock information, including a unique lock ID and other metadata.

Acquiring a Lock
To acquire a lock, a .tflock file is uploaded to an S3 bucket to establish a lock on the state file. If the lock file does not already exist, the upload succeeds, thereby acquiring the lock. If the file already exists, the upload fails due to a conditional write, indicating that the lock is already held by another Terraform client.

Releasing a Lock
To release a lock, the corresponding lock file is deleted from the S3 bucket. This action removes the file, thereby releasing the lock and making it available for other Terraform clients.

Conditional Writes

Conditional writes in Amazon S3 allows clients to perform write operations only if certain conditions about the existing object are met. Specifically:

• Successful Write: If no object exists with the same key name, the write operation succeeds, resulting in a 200 response.
• Failed Write: If an object with the same key name already exists, the write operation fails with a 412 Precondition Failed response.

For buckets with versioning enabled, S3 checks for the presence of a current object version. If no current version exists or if the version is a delete marker, the write operation succeeds; otherwise, it fails.

In scenarios where multiple Terraform processes attempt to acquire a lock (i.e., multiple concurrent conditional writes to the same S3 object), only the first process to successfully acquire the lock will succeed. Subsequent processes will receive a 412 Precondition Failed response, indicating that the lock is already held by another Terraform process.

According to Amazon S3 documentation, a 409 Conflict response may occur if a delete request for the object completes before a conditional write operation finishes. In such cases, uploads might need to be retried. However, given that delete operations should generally occur after a lock is successfully acquired, encountering this situation in the context of state locking for Terraform is unlikely (?)

This locking mechanism ensures that only one Terraform process can hold the lock at any given time, preventing concurrent modifications and ensuring consistent state management using the s3 backend and Amazon S3's conditional writes.

An opt-in feature

DynamoDB has long been the standard for Terraform state locking in the s3 backend, and it's fair to assume that many users depend on this mechanism. To provide a practitioner-friendly way to introduce the S3-native state locking via a lock file, this draft PR deprecates DynamoDB-related attributes in the s3 backend. It introduces a new use_lockfile boolean attribute to enable the S3-native locking mechanism.

Switching to the new S3-native locking mechanism

To be worked out.

Draft

I’m still working on this. The foundations of S3-native state locking are already in place. This is still a prototype, and I'm testing it to make sure it fits well with the current state locking experience in the S3 backend. Your feedback, comments, and suggestions are welcome and appreciated.

[gdavison]

Hi @bschaatsbergen, I have a few suggestions in the code.

A few other items:

• The DynamoDB lock stores MD5 hashes to prevent stale content. We should consider an equivalent mechanism for the native S3 lock
• We should add force-unlock tests that set both locks, but delete one or both out-of-band, and then try unlocking
• We should add tests that validate that the correct lock file is created, even with workspace_key_prefix
• We should test that if a backend is locked with only DynamoDB, a second backend locked with both will not acquire a lock and will clean up its lock file
• Consider applying encryption to the lock file in the same way we now handle encryption for the state file. (Added by @bschaatsbergen)

[YakDriver]

Double-triple check my changes but otherwise LGTM 🎉

1. Switched logic in Lock and Unlock to return early, Go style
2. Added log Infos
3. Tightened up error messages
4. Add error handling for deferred closing

Note, seems like the design decision is that double locking is acceptable but that if one fails, they both fail. This seems like a good choice but just wanted to spell it out to make sure we're on the same page.

[jar-b]

LGTM 🎉

% TF_ACC=1 go test -count=1 ./...
ok      github.com/hashicorp/terraform/internal/backend/remote-state/s3 198.502s

[gdavison]

Looks good! 🚀

[jbardin]

The DynamoDB lock stores MD5 hashes to prevent stale content. We should consider an equivalent mechanism for the native S3 lock

I added this a very long time ago because S3 was not consistent after write and you would often read back stale data for a while. This isn't for any consistency problem within Terraform, so if the new consistency rules ensure read-after-write then the dynamodb checksum shouldn't be needed.

[jar-b]

Thanks for all of your effort on this, @bschaatsbergen! 🎉

[github-actions]

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.