The shared staging environment was a recurring bottleneck. Multiple feature branches deployed to the same environment resulted in overwritten configurations, data conflicts, and a sequential, queue-based testing process that stifled parallel development. The feedback loop for a developer to see their changes live in a production-like setting was measured in hours, sometimes days. This friction was unacceptable.
The initial goal was clear: create ephemeral, fully-isolated preview environments for every single pull request. Our first attempt involved a collection of imperative shell scripts orchestrated by our CI server. This was an operational disaster. The scripts were brittle, failed to clean up resources correctly, and had no concept of state, leading to resource drift and escalating cloud costs from orphaned infrastructure. It became obvious that managing infrastructure lifecycle required a declarative, state-aware model, just like we manage our applications. This led us to GitOps, not just for applications, but for the underlying infrastructure itself.
Our technology stack selection was driven by specific, pragmatic needs. For the CI/CD orchestration, GitHub Actions was the native and obvious choice. The primary performance bottleneck in our old process was the frontend asset build; esbuild was selected for its near-instantaneous build times, a critical factor for a rapid feedback loop. For application deployment, Flux CD was our established GitOps tool, reconciling Kubernetes manifests from a Git repository to our clusters. The missing piece was declarative infrastructure. We evaluated Terraform, but its state file management and execution model felt external to our Kubernetes-centric universe. Crossplane presented a compelling alternative: manage external cloud resources as native Kubernetes API extensions. An S3 bucket or a Postgres database could be represented by a YAML manifest and reconciled by a controller, exactly like a Kubernetes Deployment. This unification of application and infrastructure control planes was the architectural lynchpin we needed.
The final architecture coalesces these tools into a single, automated workflow. A pull request triggers a GitHub Action that uses esbuild to build assets, packages a container, and then—crucially—generates and commits Kubernetes manifests to a Git repository. These aren’t just application manifests; they include a Crossplane manifest that declaratively provisions a new, isolated set of cloud resources for that specific branch. Flux CD detects the new manifests and orchestrates the deployment, while Crossplane’s controllers work in parallel to provision the infrastructure.
The core of this system resides in a management Kubernetes cluster where Crossplane and its cloud provider configurations are installed. We use the AWS provider, so the first step is ensuring Crossplane can authenticate with our AWS account. This is handled by creating a Kubernetes secret with AWS credentials and referencing it in a ProviderConfig resource.
# 01-crossplane-aws-provider-config.yaml
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: aws-creds
key: credsIn a real-world project, a more secure method like IRSA (IAM Roles for Service Accounts) is non-negotiable, but for clarity, a secret-based approach illustrates the connection.
With Crossplane configured, we can define the “shape” of our ephemeral environment. We don’t want developers to deal with the low-level details of an S3 bucket or an RDS instance. Instead, we want to offer a high-level abstraction: a PreviewEnvironment. This is achieved through Crossplane’s Composition model. First, we define the abstract API using a CompositeResourceDefinition (XRD).
# 02-xrd-preview-environment.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: previewenvironments.platform.acme.io
spec:
group: platform.acme.io
names:
kind: PreviewEnvironment
plural: previewenvironments
claimNames:
kind: PreviewEnvironmentClaim
plural: previewenvironmentclaims
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
branchName:
type: string
description: "The git branch name, used for naming resources."
deletionPolicy:
type: string
description: "Defines behavior on deletion. 'Delete' for ephemeral, 'Orphan' for retaining data."
enum:
- Delete
- Orphan
default: Delete
required:
- branchNameThis XRD establishes a new Kubernetes resource kind, PreviewEnvironment, within our custom API group platform.acme.io. It exposes a simple interface requiring only a branchName.
Next, we create the Composition that teaches Crossplane how to satisfy a request for a PreviewEnvironment by provisioning actual AWS resources. This is where the real infrastructure logic lives.
# 03-composition-preview-environment.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: previewenvironment.aws.platform.acme.io
labels:
provider: aws
spec:
compositeTypeRef:
apiVersion: platform.acme.io/v1alpha1
kind: PreviewEnvironment
resources:
- name: s3-bucket
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
spec:
forProvider:
region: us-west-2
acl: public-read
deletionPolicy: Delete # Ensure bucket is deleted when the claim is
patches:
- fromFieldPath: "spec.branchName"
toFieldPath: "metadata.name"
transforms:
- type: string
string:
fmt: "acme-preview-%s-assets"
- fromFieldPath: "spec.deletionPolicy"
toFieldPath: "spec.deletionPolicy"
- name: rds-instance
base:
apiVersion: rds.aws.upbound.io/v1beta1
kind: Instance
spec:
forProvider:
region: us-west-2
instanceClass: db.t3.micro
allocatedStorage: 20
engine: postgres
engineVersion: "13"
username: masteruser
skipFinalSnapshot: true
publiclyAccessible: true # Not for production, but necessary for simple preview access
writeConnectionSecretToRef:
namespace: default # The application namespace
deletionPolicy: Delete
patches:
- fromFieldPath: "spec.branchName"
toFieldPath: "metadata.name"
transforms:
- type: string
string:
fmt: "acme-preview-db-%s"
- fromFieldPath: "spec.branchName"
toFieldPath: "spec.writeConnectionSecretToRef.name"
transforms:
- type: string
string:
fmt: "%s-db-conn"
- fromFieldPath: "spec.deletionPolicy"
toFieldPath: "spec.deletionPolicy"
- name: db-subnet-group # RDS instances require a subnet group
base:
apiVersion: rds.aws.upbound.io/v1beta1
kind: SubnetGroup
spec:
forProvider:
region: us-west-2
subnetIds:
- subnet-0123456789abcdef0
- subnet-fedcba9876543210f
patches:
- fromFieldPath: "spec.branchName"
toFieldPath: "metadata.name"
transforms:
- type: string
string:
fmt: "acme-preview-sng-%s"The Composition uses patches to dynamically construct resource names and configurations from the branchName provided in the PreviewEnvironment resource. A critical feature here is writeConnectionSecretToRef. Crossplane will provision the RDS instance and then automatically create a Kubernetes Secret containing the database host, port, username, and password. This is how our application will securely connect to its dedicated database.
The entire process is orchestrated by a GitHub Actions workflow. This workflow is the imperative glue that connects our declarative components.
# .github/workflows/preview-deploy.yml
name: Create Preview Environment
on:
pull_request:
types: [opened, synchronize]
jobs:
build-and-deploy:
runs-on: ubuntu-latest
permissions:
contents: write # To commit manifests back to the repo
pull-requests: write # To comment on the PR
steps:
- name: Checkout Application Code
uses: actions/checkout@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
# The esbuild step is incredibly fast, crucial for a tight feedback loop
- name: Install dependencies and Build Frontend
run: |
npm ci
npm run build
# Example esbuild script in package.json: "build": "node ./esbuild.config.js"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and Push Docker Image
id: docker_build
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: your-docker-repo/preview-app:${{ github.sha }}
- name: Checkout Manifests Repository
uses: actions/checkout@v3
with:
repository: your-org/k8s-manifests
path: k8s-manifests
token: ${{ secrets.MANIFEST_REPO_PAT }}
- name: Generate and Commit Manifests
run: |
BRANCH_NAME=$(echo ${{ github.head_ref }} | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9-]//g')
IMAGE_TAG=${{ github.sha }}
MANIFEST_DIR="k8s-manifests/previews/${BRANCH_NAME}"
mkdir -p ${MANIFEST_DIR}
# Generate the Crossplane infrastructure claim
cat <<EOF > ${MANIFEST_DIR}/01-infra-claim.yaml
apiVersion: platform.acme.io/v1alpha1
kind: PreviewEnvironment
metadata:
name: ${BRANCH_NAME}-env
namespace: default
spec:
branchName: ${BRANCH_NAME}
deletionPolicy: Delete
EOF
# Generate the Flux Kustomization for the application
cat <<EOF > ${MANIFEST_DIR}/02-flux-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: preview-${BRANCH_NAME}
namespace: flux-system
spec:
interval: 1m0s
path: ./apps/base
prune: true
sourceRef:
kind: GitRepository
name: app-source-repo # Assuming a Flux GitRepository source for the app code
targetNamespace: preview-${BRANCH_NAME}
patches:
- patch: |-
- op: replace
path: /spec/template/spec/containers/0/image
value: your-docker-repo/preview-app:${IMAGE_TAG}
target:
kind: Deployment
name: webapp
- patch: |-
- op: add
path: /spec/template/spec/containers/0/envFrom/-
value:
secretRef:
name: ${BRANCH_NAME}-db-conn
target:
kind: Deployment
name: webapp
EOF
# Commit the changes to trigger GitOps
cd k8s-manifests
git config user.name "GitHub Actions"
git config user.email "[email protected]"
git add previews/${BRANCH_NAME}
git commit -m "Deploy preview environment for branch ${BRANCH_NAME} at ${IMAGE_TAG}" || echo "No changes to commit"
git push
- name: Post PR Comment
uses: actions/github-script@v6
with:
script: |
const branchName = '${{ github.head_ref }}'.toLowerCase().replace(/[^a-z0-9-]/g, '');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `🚀 Preview environment provisioning initiated. URL: http://${branchName}.preview.acme.io`
})The esbuild configuration itself is minimal, highlighting its ease of use for achieving drastic build performance improvements.
// esbuild.config.js
const esbuild = require('esbuild');
const process = require('process');
const isProduction = process.env.NODE_ENV === 'production';
esbuild.build({
entryPoints: ['src/index.js'],
bundle: true,
minify: isProduction,
sourcemap: !isProduction,
outfile: 'dist/bundle.js',
logLevel: 'info',
// Error handling for CI/CD pipelines
}).catch(() => process.exit(1));The core logic resides in the “Generate and Commit Manifests” step. It creates a directory in the manifest repository unique to the branch. Inside, it places the PreviewEnvironment manifest, which triggers Crossplane, and a Flux Kustomization. This Kustomization is powerful: it points to a base set of application manifests but uses inline patches to inject the correct Docker image tag and, critically, to mount the database connection secret that Crossplane will create.
The full end-to-end flow can be visualized.
sequenceDiagram
participant Dev as Developer
participant GH as GitHub
participant GHA as GitHub Actions
participant CR as Container Registry
participant Git as Manifests Repo
participant Flux as Flux CD
participant CP as Crossplane
participant Cloud as AWS
Dev->>+GH: git push to feature branch
GH->>GHA: Trigger Pull Request workflow
GHA->>GHA: Run esbuild & build container
GHA->>CR: Push image:sha-12345
GHA->>Git: Generate & commit manifests (PreviewEnvironment, Kustomization)
GHA-->>GH: Post comment to PR with URL
activate Flux
Flux->>Git: Detects new commit
Flux->>CP: Applies PreviewEnvironment manifest
deactivate Flux
activate CP
CP->>Cloud: Provision S3 Bucket
CP->>Cloud: Provision RDS Instance
Cloud-->>CP: Infrastructure is ready
CP->>CP: Create db-conn Secret in K8s
deactivate CP
activate Flux
Flux->>Git: Reads Kustomization for app
Flux->>Flux: Applies Deployment, Service, etc.
Note right of Flux: Pods start, read db-conn Secret,
and connect to the new RDS instance.
deactivate Flux
GH-->>-Dev: Preview ready!
To tear down the environment, a second GitHub Action runs on pull request closed events. Its sole job is to remove the branch-specific directory from the manifest repository and commit the change. This single git rm command triggers a cascade: Flux sees the manifests are gone and deletes the application resources from Kubernetes. Crossplane observes that the PreviewEnvironment resource has been deleted and, respecting the deletionPolicy: Delete, de-provisions the S3 bucket and RDS instance from AWS. This automated cleanup is a fundamental improvement over manual scripting, preventing resource leakage and controlling costs.
This architecture isn’t without its own set of trade-offs and remaining challenges. The provisioning time for stateful infrastructure like an RDS database can be significant—often 5-10 minutes. This extends the feedback loop. For faster previews, one might consider a composition that provisions a new schema on a shared, more powerful database instance, trading complete isolation for speed.
Furthermore, the complexity of this system is non-trivial. It requires deep expertise across CI/CD, containerization, Kubernetes, and cloud provider specifics. Onboarding a new engineer requires understanding how a single commit can ripple through multiple systems to materialize physical infrastructure. The potential for error also increases; a misconfigured Composition or a faulty CI script can lead to failed provisions or dangling resources. Robust error handling, alerting on failed reconciliations in both Flux and Crossplane, and clear documentation are not optional luxuries but core requirements for maintaining such a system. The benefit is a high-fidelity, isolated testing environment, but the operational cost of the platform itself must be acknowledged and managed.
