Implementing a Monitored Static Site Generation Pipeline with ASP.NET Core, Puppeteer, and Gatsby

The core problem was insidious: our Gatsby build pipeline was a black box. Build durations would fluctuate wildly, and more importantly, front-end performance regressions—a bloated component bundle or a render-blocking script—were only discovered in production. The feedback loop was too long. We needed a system to treat our static site generation not as a simple build script, but as a first-class, observable service. The goal was to shift performance monitoring left, catching regressions at build time by programmatically auditing the generated site and shipping those metrics to our existing observability stack.

Our initial concept was to wrap the entire generation process in an orchestrator. This orchestrator would be responsible for triggering the Gatsby build, serving the output locally, and then running a headless browser against a set of critical pages to capture performance metrics like Largest Contentful Paint (LCP). These metrics would then be exposed for scraping by Prometheus.

The technology selection process led to a somewhat unconventional stack. While a Node.js script would be the obvious choice, our core infrastructure and team expertise are heavily invested in the .NET ecosystem. Opting for an ASP.NET Core Worker Service provided a production-grade foundation with robust dependency injection, configuration, logging, and background service management via IHostedService. For browser automation, Puppeteer-Sharp, a mature .NET port of Puppeteer, was the clear choice, giving us direct access to the Chrome DevTools Protocol. Gatsby remained our static site generator, and Grafana with Prometheus was our non-negotiable for monitoring.

The resulting architecture is a durable service that transforms a volatile build script into a predictable, measurable process.

graph TD
    subgraph ASP.NET Core Worker Service
        A[IHostedService: BuildOrchestrator] -- Triggers --> B{Process: gatsby build};
        B -- On Success --> C[Serve 'public' directory];
        C -- Serves content to --> D[Puppeteer-Sharp Instance];
        D -- Captures metrics --> E[PerformanceAuditor Service];
        E -- Exposes --> F[Prometheus Metrics Endpoint];
    end

    G[Prometheus] -- Scrapes --> F;
    F -- Pushes data to --> G;
    H[Grafana] -- Queries --> G;

    style A fill:#3498db,stroke:#2980b9,stroke-width:2px,color:#fff
    style F fill:#e67e22,stroke:#d35400,stroke-width:2px,color:#fff
    style G fill:#e74c3c,stroke:#c0392b,stroke-width:2px,color:#fff
    style H fill:#9b59b6,stroke:#8e44ad,stroke-width:2px,color:#fff

Part 1: The ASP.NET Core Orchestrator

The heart of the system is a BuildOrchestratorService, an implementation of IHostedService running within an ASP.NET Core Worker. It’s not just a fire-and-forget script; it’s a long-running service responsible for managing the state and lifecycle of the build process.

The project is set up using the dotnet new worker template. The primary dependencies are PuppeteerSharp, prometheus-net, and prometheus-net.AspNetCore.

Here is the core structure of the orchestrator service. In a real-world project, the trigger mechanism would likely be a message queue listener (RabbitMQ, SQS) or a file system watcher, but for this implementation, a periodic timer demonstrates the flow.

// BuildOrchestratorService.cs
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System.Diagnostics;

public class BuildOrchestratorService : BackgroundService
{
    private readonly ILogger<BuildOrchestratorService> _logger;
    private readonly IServiceProvider _serviceProvider;
    private readonly BuildMetrics _metrics;
    private readonly string _gatsbyProjectPath;

    public BuildOrchestratorService(ILogger<BuildOrchestratorService> logger, IServiceProvider serviceProvider, BuildMetrics metrics, IConfiguration configuration)
    {
        _logger = logger;
        _serviceProvider = serviceProvider;
        _metrics = metrics;
        _gatsbyProjectPath = configuration.GetValue<string>("Gatsby:ProjectPath")
            ?? throw new ArgumentNullException("Gatsby:ProjectPath is not configured.");
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Build Orchestrator Service starting.");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await RunFullBuildAndAuditCycle(stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "An unhandled exception occurred in the build and audit cycle.");
                _metrics.RecordBuildFailure();
            }
            
            // In a real system, this would be triggered by a webhook or message.
            // For demonstration, we wait for a configured interval.
            await Task.Delay(TimeSpan.FromMinutes(15), stoppingToken);
        }

        _logger.LogInformation("Build Orchestrator Service stopping.");
    }

    private async Task RunFullBuildAndAuditCycle(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Starting new build and audit cycle.");
        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            // Step 1: Execute `gatsby build`
            var buildSuccess = await ExecuteGatsbyBuild(stoppingToken);
            if (!buildSuccess)
            {
                _logger.LogError("Gatsby build failed. Aborting audit cycle.");
                _metrics.RecordBuildFailure();
                return;
            }

            stopwatch.Stop();
            _metrics.RecordBuildDuration(stopwatch.Elapsed);
            _logger.LogInformation("Gatsby build completed successfully in {Duration} seconds.", stopwatch.Elapsed.TotalSeconds);

            // Step 2: Run performance audit using a scoped service
            // Using a scope ensures that any transient dependencies for the audit are handled correctly.
            using (var scope = _serviceProvider.CreateScope())
            {
                var auditor = scope.ServiceProvider.GetRequiredService<IPerformanceAuditor>();
                await auditor.RunAuditAsync(stoppingToken);
            }
        }
        catch (OperationCanceledException)
        {
            _logger.LogWarning("Build and audit cycle was cancelled.");
        }
        catch (Exception ex)
        {
            stopwatch.Stop();
            _logger.LogError(ex, "Failed during build and audit cycle after {Duration} seconds.", stopwatch.Elapsed.TotalSeconds);
            _metrics.RecordBuildFailure();
        }
    }

    private async Task<bool> ExecuteGatsbyBuild(CancellationToken stoppingToken)
    {
        var processStartInfo = new ProcessStartInfo
        {
            FileName = "npm", // Assumes npm is in the system's PATH
            Arguments = "run build",
            WorkingDirectory = _gatsbyProjectPath,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
            CreateNoWindow = true,
        };

        using var process = new Process { StartInfo = processStartInfo };
        
        // A common pitfall is not handling the output streams asynchronously,
        // which can lead to deadlocks if the buffer fills up.
        var outputTcs = new TaskCompletionSource<bool>();
        var errorTcs = new TaskCompletionSource<bool>();

        process.OutputDataReceived += (sender, args) => {
            if (args.Data != null) _logger.LogInformation("[Gatsby STDOUT] {Data}", args.Data);
        };
        process.ErrorDataReceived += (sender, args) => {
            if (args.Data != null) _logger.LogError("[Gatsby STDERR] {Data}", args.Data);
        };

        process.Start();
        process.BeginOutputReadLine();
        process.BeginErrorReadLine();
        
        await process.WaitForExitAsync(stoppingToken);

        return process.ExitCode == 0;
    }
}

The configuration in appsettings.json is straightforward:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "Gatsby": {
    "ProjectPath": "C:\\path\\to\\your\\gatsby-project"
  },
  "Audit": {
    "BaseUrl": "http://localhost:9000",
    "PathsToAudit": [ "/", "/about", "/products/widget-a" ]
  }
}

A crucial detail here is the asynchronous handling of the process’s standard output and error streams. A naive process.WaitForExit() can lock up if the child process generates enough output to fill the OS pipe buffer. Asynchronously reading the streams prevents this common production issue.

Part 2: Performance Auditing with Puppeteer-Sharp

With a successful build, the public directory contains our static site. The next step is to serve it and audit it. We can use the simple dotnet tool install --global dotnet-serve for local serving, or a more robust embedded Kestrel server. For this implementation, we assume a separate process runs serve -d public -p 9000.

The PerformanceAuditor service encapsulates all Puppeteer-Sharp logic.

// IPerformanceAuditor.cs
public interface IPerformanceAuditor
{
    Task RunAuditAsync(CancellationToken stoppingToken);
}

// PerformanceAuditor.cs
using PuppeteerSharp;
using Microsoft.Extensions.Logging;
using System.Text.Json;

public class PerformanceAuditor : IPerformanceAuditor
{
    private readonly ILogger<PerformanceAuditor> _logger;
    private readonly BuildMetrics _metrics;
    private readonly string _baseUrl;
    private readonly IReadOnlyList<string> _pathsToAudit;

    public PerformanceAuditor(ILogger<PerformanceAuditor> logger, BuildMetrics metrics, IConfiguration configuration)
    {
        _logger = logger;
        _metrics = metrics;
        _baseUrl = configuration.GetValue<string>("Audit:BaseUrl") ?? "http://localhost:9000";
        _pathsToAudit = configuration.GetSection("Audit:PathsToAudit").Get<string[]>() ?? Array.Empty<string>();
    }

    public async Task RunAuditAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Starting performance audit for {PathCount} paths.", _pathsToAudit.Count);

        // Download the browser if it's not already present.
        // In a containerized environment, this should be pre-installed in the Docker image.
        using var browserFetcher = new BrowserFetcher();
        await browserFetcher.DownloadAsync();

        await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions
        {
            Headless = true,
            Args = new[] { "--no-sandbox" } // Often required in container environments
        });

        foreach (var path in _pathsToAudit)
        {
            if (stoppingToken.IsCancellationRequested)
            {
                _logger.LogWarning("Audit was cancelled.");
                return;
            }
            await AuditPageAsync(browser, path);
        }

        _logger.LogInformation("Performance audit finished.");
    }
    
    private async Task AuditPageAsync(IBrowser browser, string path)
    {
        _logger.LogInformation("Auditing path: {Path}", path);
        IPage? page = null;
        try
        {
            page = await browser.NewPageAsync();
            
            // A common mistake is not setting a reasonable timeout. The default can be very long.
            await page.SetViewportAsync(new ViewPortOptions { Width = 1920, Height = 1080 });
            
            var url = $"{_baseUrl}{path}";
            var response = await page.GoToAsync(url, new NavigationOptions
            {
                WaitUntil = new[] { WaitUntilNavigation.NetworkIdle0 }, // Wait for network to be quiet
                Timeout = 30000 // 30 second timeout
            });

            if (response == null || !response.Ok)
            {
                _logger.LogError("Failed to load page {Url}. Status: {Status}", url, response?.Status);
                _metrics.RecordPageLoadError(path);
                return;
            }

            var performanceMetrics = await GetPerformanceMetrics(page, path);
            if (performanceMetrics != null)
            {
                _metrics.RecordPageLCP(path, performanceMetrics.Lcp);
                _metrics.RecordPageFCP(path, performanceMetrics.Fcp);
                _logger.LogInformation("Metrics for {Path}: LCP={Lcp}ms, FCP={Fcp}ms", path, performanceMetrics.Lcp, performanceMetrics.Fcp);
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "An error occurred while auditing path: {Path}", path);
            _metrics.RecordPageLoadError(path);
        }
        finally
        {
            if (page != null && !page.IsClosed)
            {
                await page.CloseAsync();
            }
        }
    }

    private async Task<PagePerformanceMetrics?> GetPerformanceMetrics(IPage page, string path)
    {
        try
        {
            // This JS code runs in the browser context to access the Performance API.
            // It uses PerformanceObserver to reliably get LCP.
            var script = @"
                async () => {
                    return new Promise((resolve) => {
                        let lcp;
                        const observer = new PerformanceObserver((entryList) => {
                            const entries = entryList.getEntries();
                            if (entries.length > 0) {
                                lcp = entries[entries.length - 1].renderTime || entries[entries.length - 1].loadTime;
                            }
                        });
                        observer.observe({ type: 'largest-contentful-paint', buffered: true });

                        // Fallback for LCP after a short delay
                        setTimeout(() => {
                            observer.disconnect();
                            const fcpEntry = performance.getEntriesByName('first-contentful-paint')[0];
                            resolve({
                                lcp: lcp || 0,
                                fcp: fcpEntry ? fcpEntry.startTime : 0
                            });
                        }, 2000);
                    });
                }";

            var result = await page.EvaluateFunctionAsync<JsonElement>(script);
            return JsonSerializer.Deserialize<PagePerformanceMetrics>(result);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to evaluate performance script on path: {Path}", path);
            _metrics.RecordMetricCollectionError(path);
            return null;
        }
    }

    private class PagePerformanceMetrics
    {
        public double Lcp { get; set; }
        public double Fcp { get; set; }
    }
}

The key piece of logic is the JavaScript executed via page.EvaluateFunctionAsync. It uses the PerformanceObserver API within the browser to accurately capture the LCP timing. A simple query might miss it if the LCP element appears late. This is a robust way to get real user-centric metrics, even in a synthetic test. The --no-sandbox argument for Puppeteer.LaunchAsync is a critical detail for running inside Docker containers where user namespaces might not be available.

Part 3: Instrumenting with Prometheus

Now we need to wire these measurements into our observability stack. The prometheus-net library makes this exceptionally clean. We define a BuildMetrics class to encapsulate all our custom metrics. This class is registered as a singleton in the DI container.

// BuildMetrics.cs
using Prometheus;

public class BuildMetrics
{
    private readonly Gauge _buildDurationSeconds = Metrics.CreateGauge(
        "gatsby_build_duration_seconds", 
        "The total duration of the 'gatsby build' command in seconds.");

    private readonly Counter _buildTotal = Metrics.CreateCounter(
        "gatsby_builds_total", 
        "Total number of gatsby builds attempted.",
        new CounterConfiguration { LabelNames = new[] { "status" } });

    private readonly Gauge _pageLcpMilliseconds = Metrics.CreateGauge(
        "puppeteer_page_lcp_ms", 
        "Largest Contentful Paint for a given page in milliseconds.",
        new GaugeConfiguration { LabelNames = new[] { "path" } });
        
    private readonly Gauge _pageFcpMilliseconds = Metrics.CreateGauge(
        "puppeteer_page_fcp_ms", 
        "First Contentful Paint for a given page in milliseconds.",
        new GaugeConfiguration { LabelNames = new[] { "path" } });

    private readonly Counter _pageErrorsTotal = Metrics.CreateCounter(
        "puppeteer_page_errors_total", 
        "Total errors encountered by Puppeteer when auditing a page.",
        new CounterConfiguration { LabelNames = new[] { "path", "error_type" } });

    public void RecordBuildDuration(TimeSpan duration)
    {
        _buildDurationSeconds.Set(duration.TotalSeconds);
        _buildTotal.WithLabels("success").Inc();
    }

    public void RecordBuildFailure()
    {
        _buildTotal.WithLabels("failure").Inc();
    }

    public void RecordPageLCP(string path, double lcp)
    {
        _pageLcpMilliseconds.WithLabels(path).Set(lcp);
    }
    
    public void RecordPageFCP(string path, double fcp)
    {
        _pageFcpMilliseconds.WithLabels(path).Set(fcp);
    }

    public void RecordPageLoadError(string path)
    {
        _pageErrorsTotal.WithLabels(path, "load_failure").Inc();
    }

    public void RecordMetricCollectionError(string path)
    {
        _pageErrorsTotal.WithLabels(path, "metric_collection_failure").Inc();
    }
}

This class is then injected into our services. The final step is to configure the ASP.NET Core application to expose the /metrics endpoint.

// Program.cs
using Prometheus;

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureServices((hostContext, services) =>
            {
                services.AddSingleton<BuildMetrics>();
                services.AddScoped<IPerformanceAuditor, PerformanceAuditor>();
                services.AddHostedService<BuildOrchestratorService>();
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                // This adds the web server capabilities needed to expose the metrics endpoint.
                webBuilder.Configure(app =>
                {
                    app.UseRouting();
                    app.UseEndpoints(endpoints =>
                    {
                        // Expose the metrics endpoint for Prometheus scraping.
                        endpoints.MapMetrics();
                    });
                });
            });
}

A common mistake is using a Worker Service template and forgetting to add the web host configuration (ConfigureWebHostDefaults). Without it, there’s no Kestrel server to respond to Prometheus’s scrape requests on the /metrics endpoint.

Part 4: Visualization and Alerting in Grafana

With the service running and Prometheus scraping the /metrics endpoint, the final piece is visualization. In Grafana, we can now build a dashboard to track the health of our generation pipeline over time.

A useful dashboard would include panels for:

1. Build Duration: A time-series graph showing the build time.

   • PromQL Query: gatsby_build_duration_seconds

2. Build Success/Failure Rate: A stat panel or pie chart showing the ratio of successful to failed builds.

   • PromQL Query: sum(rate(gatsby_builds_total{status="success"}[5m])) / sum(rate(gatsby_builds_total[5m])) * 100

3. LCP per Page: A time-series graph showing the LCP for each audited page. This is the most critical panel for detecting front-end performance regressions.

   • PromQL Query: puppeteer_page_lcp_ms
   • The legend can be configured to show the {{path}} label.

4. Audit Errors: A time-series graph showing the rate of errors during the audit process.

   • PromQL Query: sum by (path) (rate(puppeteer_page_errors_total[5m]))

With these visualizations, a performance regression becomes immediately obvious. If a commit adds a large, unoptimized image to the homepage, the puppeteer_page_lcp_ms{path="/"} metric will spike on the next build. We can then configure a Grafana alert to fire if this metric exceeds a defined threshold, notifying the team in Slack or PagerDuty and effectively blocking a bad deployment before it ever starts. We have successfully closed the feedback loop.

This solution is not without its limitations. The orchestrator is a single point of failure and processes builds serially. For a large-scale operation, this architecture would evolve. The orchestrator would become a dispatcher, pushing build jobs onto a message queue (like RabbitMQ or Kafka). A fleet of stateless “build-and-audit” workers would consume from this queue, allowing for parallel processing and greater resiliency. Furthermore, Puppeteer is resource-intensive; running it requires careful CPU and memory management, ideally within dedicated containers with resource limits. The current implementation also only audits a static list of pages; a more advanced version could dynamically discover all generated pages from a sitemap and perform a more comprehensive audit.