Implementing Full-Stack Observability with OpenTelemetry Across a Go-Fiber gRPC Service and a Pinia-Powered Vue Frontend

The system went live, and for a week, all metrics looked stable. Then came the first intermittent bug report: “The dashboard update occasionally fails during peak hours.” The user-facing component, built with Shadcn UI on Vue, would hang, and a Pinia store action would time out. The Go-Fiber backend logs showed no corresponding errors. The gRPC service, which handles the data fetching, reported healthy status. We were flying blind. The disconnect between a client-side user action and its server-side execution trace was a critical visibility gap. Simple logging on either end was insufficient; we needed a single, continuous narrative for each request.

This situation demanded a move from fragmented logging to holistic observability. The goal was to implement end-to-end distributed tracing, creating a causal chain from the moment a user clicks a button in the browser, through the Pinia state management layer, across the gRPC-Web network boundary, and deep into the Go-Fiber backend handler. The chosen tool for this was OpenTelemetry, due to its vendor-agnostic nature and robust support for both Go and browser-based JavaScript.

The core technical challenge is not just instrumenting each service in isolation, but correctly propagating the trace context across the stack’s boundaries: the browser’s Fetch API, the gRPC-Web protocol translation, and the Go server’s middleware stack. A failure at any point in this chain breaks the trace, rendering the entire effort useless.

The Foundation: Protobuf and Service Definition

Everything starts with the contract. Using gRPC forces a schema-first approach, which is invaluable for type safety between the Go backend and the TypeScript frontend. Our service is simple: fetch user profile data.

proto/user/v1/user.proto:

syntax = "proto3";

package user.v1;

option go_package = "example/gen/user/v1;userv1";

// UserService defines the RPCs for user management.
service UserService {
  // GetUserProfile retrieves a user's profile by their ID.
  rpc GetUserProfile(GetUserProfileRequest) returns (GetUserProfileResponse);
}

message GetUserProfileRequest {
  string user_id = 1;
}

message UserProfile {
  string user_id = 1;
  string name = 2;
  string email = 3;
  int64 created_at_unix = 4;
}

message GetUserProfileResponse {
  UserProfile profile = 1;
}

This contract is the source of truth. We use protoc to generate Go server stubs and TypeScript client code.

buf.gen.yaml:

version: v1
plugins:
  # Go code generation for server
  - plugin: buf.build/protocolbuffers/go
    out: gen
    opt: paths=source_relative
  - plugin: buf.build/grpc/go
    out: gen
    opt: paths=source_relative,require_unimplemented_servers=false

  # TypeScript code generation for client
  - plugin: buf.build/grpc-ecosystem/typescript
    out: ../frontend/src/gen
    opt:
      - target=ts
      - long_type_string
  - plugin: buf.build/grpc-ecosystem/grpc-web
    out: ../frontend/src/gen
    opt:
      - import_style=typescript
      - mode=grpcwebtext

Running buf generate creates the necessary boilerplate in both our backend and frontend projects, ensuring the communication layer is robust and type-checked from the start.

Backend Instrumentation: Go-Fiber, gRPC, and OpenTelemetry

Our backend is built with Go-Fiber for its high performance, combined with the standard google.golang.org/grpc library. The first step is to establish a baseline server without tracing.

cmd/server/main.go:

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"os"
	"os/signal"

	userv1 "example/gen/user/v1"
	"github.com/gofiber/fiber/v2"
	"github.com/gofiber/fiber/v2/middleware/cors"
	"google.golang.org/grpc"
)

// userService is our implementation of the gRPC service.
type userService struct{}

func (s *userService) GetUserProfile(ctx context.Context, req *userv1.GetUserProfileRequest) (*userv1.GetUserProfileResponse, error) {
	// In a real application, this would fetch from a database.
	// We add a small delay to simulate work.
	// time.Sleep(150 * time.Millisecond)

	if req.UserId == "" {
		return nil, fmt.Errorf("user_id cannot be empty")
	}

	log.Printf("Fetching profile for user ID: %s", req.UserId)

	return &userv1.GetUserProfileResponse{
		Profile: &userv1.UserProfile{
			UserId:        req.UserId,
			Name:          "John Doe",
			Email:         "[email protected]",
			CreatedAtUnix: 1672531200,
		},
	}, nil
}

func main() {
	// Standard gRPC server setup
	grpcServer := grpc.NewServer()
	userv1.RegisterUserServiceServer(grpcServer, &userService{})

	// For gRPC-Web to work, it typically needs a proxy that translates
	// HTTP/1.1 requests to gRPC. Fiber can act as this proxy.
	app := fiber.New()

	app.Use(cors.New(cors.Config{
		AllowOrigins: "*",
		AllowHeaders: "Content-Type, X-Grpc-Web, X-User-Agent",
	}))

	// This is a simplified proxy. For production, consider using
	// a dedicated solution like Envoy or the Improbable gRPC-Web proxy.
	// Here, we'll just forward all requests to the gRPC server.
	app.Use(func(c *fiber.Ctx) error {
		// This is a placeholder for a real gRPC-Web wrapper.
		// Libraries like `github.com/gofiber/adaptor` can bridge this,
		// but for tracing, the interceptor is the key part.
		// For simplicity, we'll focus on the gRPC interceptor logic.
		return c.Next()
	})

	// This part is complex. The ideal scenario is that Fiber serves the gRPC-Web
	// requests which are then handled by the grpcServer.
	// Let's assume we have a mechanism to route traffic correctly.
	// The crucial part for tracing happens in the gRPC server's interceptors.
	
	// Start the Fiber server in a goroutine
	go func() {
		if err := app.Listen(":8080"); err != nil {
			log.Fatalf("Fiber server failed to start: %v", err)
		}
	}()

	// Start the gRPC server in another goroutine (on a different port or unix socket)
	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatalf("Failed to listen for gRPC: %v", err)
	}
	go func() {
		log.Println("gRPC server starting on :50051")
		if err := grpcServer.Serve(lis); err != nil {
			log.Fatalf("gRPC server failed to start: %v", err)
		}
	}()


	// Graceful shutdown
	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt)
	<-quit
	log.Println("Shutting down servers...")
	grpcServer.GracefulStop()
}

This initial setup is functional but lacks observability. Now, we integrate OpenTelemetry.

internal/trace/provider.go:

package trace

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

// InitTracerProvider initializes an OpenTelemetry tracer provider.
// In a production environment, you would use an OTLP exporter to send
// traces to a collector (e.g., Jaeger, Zipkin, Datadog).
// For this example, we'll export to standard output.
func InitTracerProvider(serviceName string) (*sdktrace.TracerProvider, error) {
	// Create a new exporter to print traces to the console.
	exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
	if err != nil {
		return nil, err
	}

	// Create a new resource with service name attribute.
	res, err := resource.Merge(
		resource.Default(),
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		),
	)
	if err != nil {
		return nil, err
	}

	// Create a new tracer provider with the batch span processor
	// and the console exporter.
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
		// In production, you would configure sampling.
		// For development, Sample-all is useful.
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
	)

	// Set the global tracer provider.
	otel.SetTracerProvider(tp)
	// Set the global propagator to W3C Trace Context and Baggage.
	// This is crucial for context propagation across services.
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

	log.Println("Tracer provider initialized.")
	return tp, nil
}

The most critical piece is the gRPC interceptor. This middleware intercepts every incoming RPC, extracts the trace context from the metadata, and creates a new server-side span as a child of the client-side span.

internal/middleware/otel_grpc.go:

package middleware

import (
	"context"
	"log"
	
	"google.golang.org/grpc"
	"google.golang.org/grpc/metadata"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/attribute"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
	"go.opentelemetry.io/otel/trace"
)


// metadataCarrier is a custom carrier for extracting trace context from gRPC metadata.
type metadataCarrier struct {
	md *metadata.MD
}

// Get returns the value for a given key from the metadata.
func (c metadataCarrier) Get(key string) string {
	vals := c.md.Get(key)
	if len(vals) > 0 {
		return vals[0]
	}
	return ""
}

// Set sets a value in the metadata.
func (c metadataCarrier) Set(key, value string) {
	c.md.Set(key, value)
}

// Keys returns all keys from the metadata.
func (c metadataCarrier) Keys() []string {
	keys := make([]string, 0, c.md.Len())
	for k := range *c.md {
		keys = append(keys, k)
	}
	return keys
}

// OtelUnaryServerInterceptor returns a gRPC unary server interceptor for OpenTelemetry.
func OtelUnaryServerInterceptor() grpc.UnaryServerInterceptor {
	tracer := otel.Tracer("grpc-server")
	propagator := otel.GetTextMapPropagator()

	return func(
		ctx context.Context,
		req interface{},
		info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler,
	) (interface{}, error) {
		// Extract trace context from incoming gRPC metadata.
		md, ok := metadata.FromIncomingContext(ctx)
		if !ok {
			md = metadata.New(nil)
		}

		// The propagator extracts the context (traceparent, tracestate).
		carrier := metadataCarrier{md: &md}
		ctx = propagator.Extract(ctx, carrier)

		// Create a new span.
		spanOpts := []trace.SpanStartOption{
			trace.WithSpanKind(trace.SpanKindServer),
			trace.WithAttributes(
				semconv.RPCSystemKey.String("grpc"),
				semconv.RPCServiceKey.String(info.FullMethod),
				semconv.RPCMethodKey.String(info.FullMethod),
			),
		}

		ctx, span := tracer.Start(ctx, info.FullMethod, spanOpts...)
		defer span.End()

		log.Printf("GRPC server span started: TraceID=%s, SpanID=%s", span.SpanContext().TraceID(), span.SpanContext().SpanID())

		// Call the actual RPC handler.
		resp, err := handler(ctx, req)
		
		// Set span status based on the error.
		if err != nil {
			span.RecordError(err)
			// A common mistake is not setting the status, which can make
			// traces appear successful in UI even when they failed.
			span.SetStatus(otel.code.Error, err.Error())
		} else {
			span.SetStatus(otel.code.Ok, "OK")
		}

		return resp, err
	}
}

Now, we update main.go to use our tracer provider and interceptor.

// In main()
tp, err := trace.InitTracerProvider("user-service-go")
if err != nil {
	log.Fatalf("failed to initialize tracer provider: %v", err)
}
defer func() {
	if err := tp.Shutdown(context.Background()); err != nil {
		log.Printf("Error shutting down tracer provider: %v", err)
	}
}()

// When creating the gRPC server:
grpcServer := grpc.NewServer(
	grpc.UnaryInterceptor(middleware.OtelUnaryServerInterceptor()),
)
// ... rest of the setup

With this, the backend is fully instrumented. Any gRPC request carrying W3C trace context headers will now correctly initiate a child span.

Frontend Instrumentation: Vue, Pinia, and gRPC-Web

The frontend presents a different set of challenges. We’re in a browser environment, dealing with asynchronous user interactions. The goal is to start a trace when a user performs an action, and ensure that context is injected into the outgoing gRPC-Web request.

First, let’s set up the OpenTelemetry Web SDK.

src/boot/opentelemetry.ts:

import { ZoneContextManager } from '@opentelemetry/context-zone';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { FetchInstrumentation } from '@opentelemetry/instrumentation-fetch';
import { Resource } from '@opentelemetry/resources';
import { BatchSpanProcessor, ConsoleSpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const serviceName = 'user-frontend-vue';

// Use a WebTracerProvider for browser environments.
const provider = new WebTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
  }),
});

// In production, use OTLPTraceExporter to send to a collector.
// const exporter = new OTLPTraceExporter({
//   url: 'http://localhost:4318/v1/traces',
// });
// For this example, we log to the console.
const exporter = new ConsoleSpanExporter();

// BatchSpanProcessor is recommended for production.
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

// ZoneContextManager is crucial for propagating context in async browser operations.
provider.register({
  contextManager: new ZoneContextManager(),
});

// Auto-instrument the Fetch API. This is what gRPC-Web uses under the hood.
registerInstrumentations({
  instrumentations: [
    new FetchInstrumentation({
      // The gRPC-Web client sends a preflight OPTIONS request.
      // We often don't want to trace this.
      ignoreUrls: [/.*\/v1\.UserService\/GetUserProfile/], // Example for ignoring specific patterns
      propagateTraceHeaderCorsUrls: [
         // Match the backend URL to ensure trace headers are sent.
        new RegExp('http://localhost:8080/.*'),
      ],
    }),
  ],
});

console.log('OpenTelemetry Web provider registered.');

export const tracer = provider.getTracer('pinia-tracer');

This file needs to be imported early in the application’s lifecycle, for instance, in main.ts.

Next, we create the gRPC-Web client and a Pinia store to manage the state. The critical part is wrapping the gRPC call in a manually created span. This gives us a root span tied to a specific business action.

src/stores/user.ts:

import { defineStore } from 'pinia';
import { ref } from 'vue';
import { UserServicePromiseClient } from '@/gen/user/v1/user_grpc_web_pb';
import { GetUserProfileRequest, UserProfile } from '@/gen/user/v1/user_pb';
import { tracer } from '@/boot/opentelemetry';
import { context, trace, SpanStatusCode } from '@opentelemetry/api';

export const useUserStore = defineStore('user', () => {
  const profile = ref<UserProfile.AsObject | null>(null);
  const isLoading = ref(false);
  const error = ref<string | null>(null);

  // In a real app, the client would be configured in a separate service file.
  const grpcClient = new UserServicePromiseClient('http://localhost:8080');

  async function fetchUserProfile(userId: string) {
    // This is the starting point of our trace on the frontend.
    // We create a root span for this specific user action.
    const span = tracer.startSpan('fetchUserProfile.PiniaAction');
    
    // Use `context.with` to ensure all async operations within this block
    // are associated with our new span.
    await context.with(trace.setSpan(context.active(), span), async () => {
      try {
        isLoading.value = true;
        error.value = null;
        
        // Add attributes to the span for better context.
        span.setAttribute('user.id', userId);

        const request = new GetUserProfileRequest();
        request.setUserId(userId);

        console.log('Pinia action: Sending gRPC-Web request...');
        // The FetchInstrumentation will automatically pick up this call,
        // create a child span for the HTTP request, and inject the trace
        // context headers (traceparent) because we configured
        // `propagateTraceHeaderCorsUrls`.
        const response = await grpcClient.getUserProfile(request);
        
        const userProfile = response.getProfile();
        if (!userProfile) {
          throw new Error('Profile not found in response');
        }

        profile.value = userProfile.toObject();
        span.setStatus({ code: SpanStatusCode.OK });

      } catch (e: any) {
        console.error('Error fetching user profile:', e);
        error.value = e.message || 'An unknown error occurred';
        
        // A common mistake is forgetting to record errors on the span.
        span.recordException(e);
        span.setStatus({ code: SpanStatusCode.ERROR, message: e.message });
        profile.value = null;
      } finally {
        isLoading.value = false;
        // The span must always be ended.
        span.end();
        console.log(`Span ended: TraceID=${span.spanContext().traceId}`);
      }
    });
  }

  return { profile, isLoading, error, fetchUserProfile };
});

The UI component using Shadcn-Vue simply triggers this action.

src/components/UserProfileCard.vue:

<template>
  <Card class="w-[350px]">
    <CardHeader>
      <CardTitle>User Profile</CardTitle>
      <CardDescription>Fetch user data via gRPC.</CardDescription>
    </CardHeader>
    <CardContent>
      <div v-if="userStore.isLoading">Loading...</div>
      <div v-else-if="userStore.error" class="text-red-500">
        Error: {{ userStore.error }}
      </div>
      <div v-else-if="userStore.profile">
        <p><strong>ID:</strong> {{ userStore.profile.userId }}</p>
        <p><strong>Name:</strong> {{ userStore.profile.name }}</p>
        <p><strong>Email:</strong> {{ userStore.profile.email }}</p>
      </div>
       <div v-else>
        Click the button to fetch data.
      </div>
    </CardContent>
    <CardFooter>
      <Button @click="handleFetchProfile">
        Fetch Profile (ID: 123)
      </Button>
    </CardFooter>
  </Card>
</template>

<script setup lang="ts">
import { Button } from '@/components/ui/button';
import {
  Card,
  CardContent,
  CardDescription,
  CardFooter,
  CardHeader,
  CardTitle,
} from '@/components/ui/card';
import { useUserStore } from '@/stores/user';

const userStore = useUserStore();

const handleFetchProfile = () => {
  // This click is the true origin of our distributed trace.
  userStore.fetchUserProfile('123');
};
</script>

The Complete Trace Flow

When the Fetch Profile button is clicked, the following sequence occurs, with context seamlessly propagated at each step.

sequenceDiagram
    participant User
    participant ShadcnUI as Shadcn UI Component
    participant Pinia as Pinia Store
    participant OTelWeb as OTel Web SDK
    participant gRPCWeb as gRPC-Web Client
    participant OTelGo as OTel Go Interceptor
    participant GoService as Go gRPC Service

    User->>ShadcnUI: Clicks 'Fetch Profile'
    ShadcnUI->>Pinia: Calls fetchUserProfile('123')
    Pinia->>OTelWeb: tracer.startSpan('fetchUserProfile.PiniaAction')
    activate Pinia
    Note over Pinia, OTelWeb: Creates root span (Span A) with new TraceID.
    OTelWeb-->>Pinia: Returns active context with Span A.

    Pinia->>gRPCWeb: client.getUserProfile(request)
    activate gRPCWeb
    
    gRPCWeb->>OTelWeb: (via Fetch API instrumentation)
    OTelWeb->>OTelWeb: Starts child span for HTTP POST (Span B).
    OTelWeb->>gRPCWeb: Injects 'traceparent' header into request.
    
    gRPCWeb->>GoService: POST /user.v1.UserService/GetUserProfile
    deactivate gRPCWeb
    
    GoService->>OTelGo: Interceptor receives request
    activate OTelGo
    OTelGo->>OTelGo: Extracts 'traceparent' header.
    OTelGo->>OTelGo: Creates server span (Span C) as child of Span B.
    
    OTelGo->>GoService: Calls handler logic with new context.
    activate GoService
    Note over GoService: Business logic executes.
    GoService-->>OTelGo: Returns response/error.
    deactivate GoService
    
    OTelGo->>OTelGo: Sets span status (OK/Error) and ends Span C.
    OTelGo-->>GoService: Forwards response.
    deactivate OTelGo
    
    GoService-->>gRPCWeb: HTTP 200 OK Response
    activate gRPCWeb
    
    gRPCWeb-->>OTelWeb: (Fetch instrumentation)
    OTelWeb->>OTelWeb: Ends HTTP span (Span B).
    
    gRPCWeb-->>Pinia: Returns Promise with response.
    deactivate gRPCWeb
    
    Pinia->>Pinia: Processes response, updates state.
    Pinia->>OTelWeb: span.end() for Span A.
    deactivate Pinia
    
    Pinia-->>ShadcnUI: Updates reactive state.
    ShadcnUI-->>User: Renders profile data.

When viewed in a tracing backend like Jaeger, this entire flow appears as a single, unified trace. We can see the total time taken, the breakdown between the frontend action, the network call, and the backend processing. If the Go service had called another instrumented microservice, the trace would continue even further. This is the power of context propagation.

The pitfall here is configuration. A mismatch in the propagateTraceHeaderCorsUrls on the frontend, or CORS headers on the Go-Fiber backend, will cause the browser to block the traceparent header, silently breaking the trace link. Debugging this requires inspecting network request headers in the browser’s developer tools.

This architecture, while adding some initial complexity, fundamentally changes how we debug and understand our system. It moves us from reactive log-sifting to proactive, context-rich analysis of our application’s behavior. The limitations, of course, are that this only covers what is instrumented. A call to a non-instrumented third-party API or a database without an instrumented driver will appear as a gap in the trace. The next logical step would be to add database client instrumentation in the Go service, providing a complete query-level view, and to correlate these traces with structured logs by injecting the trace_id into every log entry.