Dynamic Secret Injection for Serverless Functions via GitLab CI and Vault with Front-End State Reflection Using Valtio

The technical pain point was sharp and recurring: managing credentials for a fleet of serverless functions. Storing long-lived IAM keys or database passwords in AWS Secrets Manager or as encrypted environment variables felt like a compromise, a static secret waiting to be leaked. Our functions were ephemeral, yet their credentials were not. This disparity introduced a significant attack surface. We needed a system where functions could acquire just-in-time, short-lived credentials, specific to the task at hand, and then have those credentials expire automatically. This is a classic use case for HashiCorp Vault’s dynamic secrets engines, but integrating it into a serverless architecture orchestrated by GitLab CI/CD presented a chain of non-trivial challenges.

The initial concept was to build a “zero-trust” pipeline. No human and no machine would possess a permanent secret. The CI/CD pipeline, triggered by a commit to our GitHub repository, would need to authenticate itself to Vault using a temporary, verifiable identity. Vault would then grant the pipeline permission not to read a secret, but to create a one-time “wrapper token.” This wrapper token would be injected into the serverless function’s environment during deployment. The function, upon its first invocation (a cold start), would use this single-use token to unwrap its own, more durable (but still short-lived) Vault token. This function-specific token would then be used to generate dynamic credentials on-demand.

On the front-end, another problem emerged. Our single-page application, built with React, needed to be aware of the backend’s operational status. Is the function capable of connecting to its dependencies? This isn’t a simple health check. The status depends entirely on the success of this dynamic secret acquisition chain. We needed a state management solution that could elegantly handle this complex, asynchronous state without a mountain of boilerplate. This led us to Valtio, a proxy-based state management library whose simplicity seemed well-suited for reflecting a complex backend reality onto the UI.

The technology selection broke down as follows:

• HashiCorp Vault: The only mature choice for managing the lifecycle of dynamic secrets. Its JWT authentication backend was critical for integrating with GitLab CI.
• GitLab CI/CD: Our incumbent CI/CD system. Its native OpenID Connect (OIDC) provider allows jobs to generate a signed JSON Web Token (JWT), which is the foundation of our passwordless authentication flow. GitHub serves as the SCM, triggering these pipelines.
• AWS Lambda: The serverless platform. Its execution environment model, particularly the concept of cold starts and container reuse, heavily influenced the design of our token caching and renewal logic.
• Valtio: For front-end state management. Its minimal API and proxy-based reactivity were a perfect fit for managing the multi-faceted “readiness” state of our serverless backend.

Here’s the logical flow we aimed to build, represented as a sequence diagram.

sequenceDiagram
    participant Dev
    participant GitHub
    participant GitLabRunner as GitLab Runner
    participant Vault
    participant AWS_Lambda as AWS Lambda
    participant Client as User Browser
    participant ThirdParty as 3rd Party Service

    Dev->>GitHub: git push
    GitHub->>GitLabRunner: Trigger Pipeline
    GitLabRunner->>Vault: Authenticate with CI_JOB_JWT_V2
    Vault-->>GitLabRunner: Grant short-lived Vault Token
    GitLabRunner->>Vault: Request a wrapped token for Lambda
    Vault-->>GitLabRunner: Provide single-use Wrapper Token
    GitLabRunner->>AWS_Lambda: Deploy function with Wrapper Token as ENV var
    
    Client->>AWS_Lambda: Invoke function /api/data
    Note over AWS_Lambda: Cold Start: Bootstrap Logic
    AWS_Lambda->>Vault: Unwrap token using ENV var
    Vault-->>AWS_Lambda: Provide new, function-scoped Vault token
    Note over AWS_Lambda: Cache function-scoped token in memory
    
    AWS_Lambda->>Vault: Request dynamic credentials for 3rd Party Service
    Vault-->>AWS_Lambda: Provide short-lived credentials
    AWS_Lambda->>ThirdParty: Access service with dynamic credentials
    ThirdParty-->>AWS_Lambda: Service Response
    AWS_Lambda-->>Client: API Response with data

    Note over Client, AWS_Lambda: Parallelly, for UI status
    Client->>AWS_Lambda: Invoke function /api/status
    AWS_Lambda->>Vault: Perform connection checks
    AWS_Lambda-->>Client: Return detailed status object
    Note over Client: Valtio store updates, UI reacts

Phase 1: Configuring Vault for GitLab OIDC Authentication

The foundation of this entire system is Vault’s ability to trust GitLab. We used Vault’s JWT auth backend to achieve this. A real-world project requires robust configuration, typically managed via Terraform.

Here is the production-grade Terraform configuration for setting up the OIDC integration.

# vault_config.tf

# Enable the JWT auth backend
resource "vault_auth_backend" "gitlab" {
  type = "jwt"
  path = "gitlab" # The path where the auth method will be accessible
}

# Configure the JWT auth backend to trust GitLab's OIDC provider
resource "vault_jwt_auth_backend" "gitlab_config" {
  backend         = vault_auth_backend.gitlab.path
  oidc_discovery_url = "https://gitlab.com"
  bound_issuer    = "gitlab.com"
  default_role    = "default-read-only" # A safe default
}

# Define a Vault policy that grants access to a specific dynamic secrets path
resource "vault_policy" "serverless_aws_creds_policy" {
  name = "serverless-aws-creds-policy"
  policy = <<EOT
# Allow the token to check its own capabilities
path "auth/token/lookup-self" {
  capabilities = ["read"]
}
# Allow the token to renew itself
path "auth/token/renew-self" {
  capabilities = ["update"]
}
# Allow generating dynamic AWS credentials from a specific role
path "aws/creds/my-serverless-role" {
  capabilities = ["read"]
}
EOT
}

# Create a role that binds the policy to specific GitLab CI/CD job attributes.
# This is the most critical security control.
resource "vault_jwt_auth_backend_role" "serverless_deploy_role" {
  backend         = vault_auth_backend.gitlab.path
  role_name       = "serverless-deploy-role"
  user_claim      = "project_path" # Use the GitLab project path as the unique user identifier
  role_type       = "jwt"
  token_policies  = [vault_policy.serverless_aws_creds_policy.name]

  # Only allow JWTs from a specific project and branch
  bound_claims = {
    project_path = "my-group/my-serverless-project",
    ref_type     = "branch",
    ref          = "main"
  }

  # Set a reasonable TTL for the token granted to the GitLab runner
  token_ttl = 300 # 5 minutes
}

A common mistake here is creating overly broad bound_claims. In this example, we’ve locked the role down to only allow JWTs originating from the main branch of the my-group/my-serverless-project. This prevents a developer running a pipeline on a feature branch from deploying and gaining access to production credentials.

Phase 2: The GitLab CI/CD Pipeline

With Vault configured, the next step was to craft the .gitlab-ci.yml file. The goal of the deploy stage is not to fetch the final secret, but to obtain the single-use wrapper token and pass it to the Lambda function.

# .gitlab-ci.yml

variables:
  VAULT_ADDR: "https://vault.example.com"
  # This role name must match the one created in Vault
  VAULT_ROLE: "serverless-deploy-role" 

stages:
  - build
  - test
  - deploy

# ... build and test stages omitted for brevity

deploy_lambda:
  stage: deploy
  image:
    name: hashicorp/vault:latest # Using the Vault CLI image
    entrypoint: [""]
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"' # Only run on the main branch
  script:
    - echo "Authenticating to Vault using GitLab OIDC JWT..."
    # The CI_JOB_JWT_V2 is a predefined variable provided by GitLab
    # It contains the signed OIDC token for the running job.
    # We log into Vault and write the resulting client token to a file.
    - |
      export VAULT_TOKEN=$(vault write -field=token auth/gitlab/login role=$VAULT_ROLE jwt=$CI_JOB_JWT_V2)
    
    - echo "Successfully authenticated. Requesting a wrapped token for Lambda."
    # The key here is the `-wrap-ttl=5m` flag. This tells Vault not to return
    # the response directly, but to store it and return a pointer (the wrapping token).
    # We are creating another token here that the lambda will use.
    # The policy on the runner token should allow creating other tokens.
    # A better approach: the role grants permission to an identity, which can then be used to generate tokens.
    # For simplicity here, we create a token directly.
    - |
      WRAPPED_TOKEN_RESPONSE=$(vault token create -policy="serverless-aws-creds-policy" -wrap-ttl="5m" -format=json)
    - |
      export WRAPPER_TOKEN=$(echo $WRAPPED_TOKEN_RESPONSE | jq -r '.wrap_info.token')

    - echo "Wrapper token generated. Deploying Serverless function..."
    # This is a conceptual deployment step. In a real project, you would
    # use a different image with the Serverless Framework or AWS SAM CLI.
    # The critical part is passing the WRAPPER_TOKEN as an environment variable.
    - |
      echo "Deploying with wrapper token: $WRAPPER_TOKEN"
      # Example with Serverless Framework:
      # serverless deploy --stage prod --param="vaultWrapperToken=$WRAPPER_TOKEN"
      # In your serverless.yml, you'd reference this parameter:
      # environment:
      #   VAULT_WRAPPER_TOKEN: ${param:vaultWrapperToken}

The pitfall here is security during the CI run. Even though we are using JWT auth, the VAULT_TOKEN acquired by the runner is still a sensitive value. GitLab CI does a good job of masking secrets, but it’s crucial to ensure it’s never printed to the logs. The wrapper token adds a layer of security; it’s single-use, so even if it were to leak, its value diminishes rapidly.

Phase 3: Serverless Function Bootstrap Logic

This is where the application logic meets the security architecture. The Lambda function must be intelligent enough to manage its own Vault token lifecycle. We used Node.js with TypeScript for this.

// src/vaultClient.ts
import axios from 'axios';

// A simple in-memory cache for the unwrapped Vault token.
// In a real serverless environment, this variable persists between invocations
// for a "warm" container.
let cachedVaultToken: {
  token: string;
  expiresAt: number; // UNIX timestamp
  renewable: boolean;
} | null = null;

const VAULT_ADDR = process.env.VAULT_ADDR!;
const VAULT_WRAPPER_TOKEN = process.env.VAULT_WRAPPER_TOKEN;

// A type guard for our expected unwrapped token structure
interface UnwrappedToken {
  auth: {
    client_token: string;
    lease_duration: number;
    renewable: boolean;
  };
}

function isUnwrappedToken(data: any): data is UnwrappedToken {
  return (
    data &&
    typeof data.auth?.client_token === 'string' &&
    typeof data.auth?.lease_duration === 'number'
  );
}

/**
 * Ensures a valid Vault token is available, either from cache or by unwrapping.
 * This function is the core of the bootstrap logic.
 */
async function getValidVaultToken(): Promise<string> {
  const now = Date.now() / 1000;

  // If we have a cached token and it's not about to expire, use it.
  if (cachedVaultToken && cachedVaultToken.expiresAt > now + 60) {
    console.log('Using cached Vault token.');
    return cachedVaultToken.token;
  }
  
  // If the token is renewable and close to expiry, we should renew it.
  // Implementation for renewal is omitted for brevity but would involve
  // calling the `auth/token/renew-self` endpoint.

  // If no cached token or it's expired, we must be on a cold start.
  // We need the wrapper token to get our first real token.
  if (!VAULT_WRAPPER_TOKEN) {
    throw new Error('FATAL: VAULT_WRAPPER_TOKEN is not set. Cannot bootstrap.');
  }

  console.log('Cold start or token expired. Unwrapping new Vault token...');
  try {
    const response = await axios.post(
      `${VAULT_ADDR}/v1/sys/wrapping/unwrap`,
      {},
      { headers: { 'X-Vault-Token': VAULT_WRAPPER_TOKEN } }
    );

    if (!isUnwrappedToken(response.data)) {
      throw new Error('Invalid unwrapped token response structure from Vault.');
    }

    const { client_token, lease_duration, renewable } = response.data.auth;
    cachedVaultToken = {
      token: client_token,
      expiresAt: now + lease_duration,
      renewable,
    };
    
    console.log(`Successfully unwrapped and cached new token. TTL: ${lease_duration}s`);
    return cachedVaultToken.token;
  } catch (error) {
    console.error('Failed to unwrap Vault token:', error.response?.data || error.message);
    // After one failed unwrap attempt, the wrapper token is likely invalid.
    // Clear it to prevent retries on subsequent invocations.
    process.env.VAULT_WRAPPER_TOKEN = undefined; 
    throw new Error('Could not obtain a valid Vault token.');
  }
}

/**
 * Fetches dynamic AWS credentials from Vault.
 */
export async function getDynamicAWSCredentials(): Promise<{ access_key: string; secret_key: string; security_token: string }> {
  const vaultToken = await getValidVaultToken();
  try {
    const response = await axios.get(
      `${VAULT_ADDR}/v1/aws/creds/my-serverless-role`,
      { headers: { 'X-Vault-Token': vaultToken } }
    );
    const { access_key, secret_key, security_token } = response.data.data;
    return { access_key, secret_key, security_token };
  } catch (error) {
    console.error('Failed to fetch dynamic AWS credentials:', error.response?.data || error.message);
    // If we fail to get credentials, it might be a token permission issue.
    // Invalidating the cache will force a re-bootstrap on the next call.
    cachedVaultToken = null;
    throw new Error('Could not fetch dynamic AWS credentials from Vault.');
  }
}

This code handles the critical cold start path. The getValidVaultToken function acts as a guard; any part of the application needing a secret must go through it. It correctly caches the token in the global scope to survive across warm invocations. The error handling is crucial: if unwrapping fails, we assume the wrapper token is spent and prevent further attempts.

Phase 4: Reflecting Backend State with Valtio

The final piece was the front-end. We created a /api/status endpoint in our serverless function that doesn’t just return 200 OK, but actively probes its own ability to connect to Vault and generate secrets.

// src/statusHandler.ts
// This would be the code for the Lambda function serving `/api/status`

export async function handler() {
  const status = {
    vault_auth: 'pending',
    dynamic_secret: 'pending',
    // ... other checks
  };

  try {
    const token = await getValidVaultToken();
    // This check is implicit; if the line above doesn't throw, auth is OK.
    status.vault_auth = 'ok';

    // Now, try to actually generate a secret to confirm permissions.
    await getDynamicAWSCredentials();
    status.dynamic_secret = 'ok';
    
    return {
      statusCode: 200,
      body: JSON.stringify(status),
    };
  } catch (error) {
    console.error('Status check failed:', error.message);
    if (status.vault_auth === 'pending') {
      status.vault_auth = 'failed';
    } else if (status.dynamic_secret === 'pending') {
      status.dynamic_secret = 'failed';
    }

    return {
      statusCode: 200, // Return 200 so the frontend can parse the status body
      body: JSON.stringify(status),
    };
  }
}

On the React front-end, we set up a Valtio store to hold this status.

// src/state/backendStatusStore.ts
import { proxy } from 'valtio';

type Status = 'pending' | 'ok' | 'failed';

interface BackendStatus {
  vault_auth: Status;
  dynamic_secret: Status;
}

export const backendStatusStore = proxy<{
  status: BackendStatus;
  isLoading: boolean;
}>({
  status: {
    vault_auth: 'pending',
    dynamic_secret: 'pending',
  },
  isLoading: true,
});

export async function fetchBackendStatus() {
  backendStatusStore.isLoading = true;
  try {
    const response = await fetch('/api/status');
    const data: BackendStatus = await response.json();
    // Directly mutate the proxy state. This is the magic of Valtio.
    backendStatusStore.status = data;
  } catch (error) {
    console.error('Failed to fetch backend status:', error);
    backendStatusStore.status.vault_auth = 'failed';
    backendStatusStore.status.dynamic_secret = 'failed';
  } finally {
    backendStatusStore.isLoading = false;
  }
}

// -- In a React component --

// src/components/StatusDashboard.tsx
import { useSnapshot } from 'valtio';
import { backendStatusStore, fetchBackendStatus } from '../state/backendStatusStore';
import { useEffect } from 'react';

const StatusIndicator = ({ status }: { status: Status }) => {
  const color = status === 'ok' ? 'green' : status === 'failed' ? 'red' : 'gray';
  return <span style={{ color }}>● {status}</span>;
};

export const StatusDashboard = () => {
  // useSnapshot creates an immutable snapshot of the state.
  // The component re-renders whenever the subscribed state changes.
  const snap = useSnapshot(backendStatusStore);
  
  useEffect(() => {
    // Fetch status on component mount
    fetchBackendStatus();
  }, []);

  if (snap.isLoading) {
    return <div>Loading status...</div>;
  }
  
  return (
    <div>
      <h2>Backend Readiness</h2>
      <p>Vault Authentication: <StatusIndicator status={snap.status.vault_auth} /></p>
      <p>Dynamic Secret Generation: <StatusIndicator status={snap.status.dynamic_secret} /></p>
    </div>
  );
};

The elegance of Valtio is apparent here. There are no actions, reducers, or dispatchers. We fetch the state and directly mutate the proxy object. useSnapshot ensures the component tree reacts efficiently to these changes. This provides our operations team and developers with a clear, real-time view into the health of our complex secret management pipeline, directly within the application’s UI.

The final system achieved our goal of ephemeral credentials for ephemeral compute. The pipeline is fully automated and passwordless, significantly reducing our attack surface. The Lambda functions are self-sufficient in managing their credential lifecycle, and the front-end provides crucial visibility into this otherwise opaque process.

However, this architecture is not without its limitations. The token renewal logic within the Lambda function adds complexity and needs to be rock-solid to prevent failures during long-running warm containers. Furthermore, a high rate of cold starts could place significant load on the Vault cluster’s auth backend and token creation endpoints. A future iteration might involve a more sophisticated caching layer, or using Vault Enterprise’s performance standby nodes to handle the read-heavy load of token lookups and renewals. The current status check on the front-end also relies on polling; a more advanced solution could use WebSockets to push real-time status updates from the backend, reducing unnecessary API calls.