Implementing a Dynamic Page Composition Engine Using Scala JPA and a Nacos-Driven UI Schema

The core pain point was latency, not in network or database calls, but in development cycles. Our front-end team, building a highly dynamic content platform with a Headless UI, was perpetually blocked. Every new page variant or even a minor tweak to an existing layout—like adding a user recommendations carousel to a product page—necessitated a new bespoke API endpoint from the backend team. The cycle of specification, development, testing, and deployment for these trivial changes was taking weeks, completely crippling our ability to experiment and iterate.

The initial concept was to dismantle this tight coupling. We needed to shift the definition of a page’s structure from backend code to dynamic configuration. This led to the “Page Assembler” service idea: a single, generic endpoint that could compose and deliver the data for any page, driven entirely by a schema that defined the page’s layout and data requirements. The front end would receive a structured JSON object describing the components to render and the data for each, effectively becoming a stateless renderer of backend-driven instructions.

For this to work, the technology choices were critical. We settled on Nacos for configuration, Scala for the service implementation, and JPA/Hibernate for data access, a stack that initially seemed slightly unconventional but proved robust.

• Nacos as a Schema Store: Using a distributed configuration center like Nacos, rather than a database or static files, was a deliberate choice. It provides a versioned, auditable, and accessible UI for managing these page schemas. More importantly, its ability to push configuration changes in near real-time meant we could alter page layouts without a single service restart. A product manager could, in theory, re-order components on the homepage and see the change live in seconds.
• Scala for Composition: The core task of the assembler is to parse a schema, fetch data for multiple independent components, and stitch them together. This is a problem of concurrent composition and robust error handling. Scala, with its powerful Future implementation and functional constructs like for-comprehensions and Either, provided the tools to perform these asynchronous operations concurrently and handle partial failures gracefully. A single component’s data source failing shouldn’t bring down the entire page.
• JPA/Hibernate for Pragmatism: While Scala has excellent functional database libraries, our organization has deep expertise and existing infrastructure around JPA/Hibernate. The challenge was to bridge the gap between Scala’s functional, immutable world and JPA’s imperative, mutable entity management. In a real-world project, leveraging existing skills and tooling often outweighs the pursuit of pure ideological consistency.

The architecture can be visualized as a clear flow of control and data.

sequenceDiagram
    participant FE as Headless UI
    participant PA as Page Assembler (Scala)
    participant NC as Nacos Server
    participant DB as Database

    FE->>+PA: GET /page/product-detail?productId=123
    PA->>+NC: Get schema for 'product-detail'
    NC-->>-PA: Return page schema (JSON)
    PA-->>PA: Parse schema, identify required components (e.g., Header, Specs, Reviews)
    
    par
        PA->>+DB: Fetch data for Header (Product ID: 123)
        DB-->>-PA: Product entity
    and
        PA->>+DB: Fetch data for Specs (Product ID: 123)
        DB-->>-PA: Product specs entities
    and
        PA->>+DB: Fetch data for Reviews (Product ID: 123)
        DB-->>-PA: List of review entities
    end

    PA-->>PA: Assemble JSON payload from component data
    PA->>-FE: Return composed page JSON

Schema Definition in Nacos

The foundation of the entire system is the page schema. We store it in Nacos as a JSON object. The Data ID in Nacos corresponds to our internal pageId, for instance, page.product-detail. The content is a JSON structure that defines the components.

Here is a sample schema for a product detail page:

{
  "pageId": "product-detail",
  "description": "The main product detail page layout.",
  "components": [
    {
      "componentId": "header",
      "componentType": "ProductHeader",
      "dataSource": {
        "provider": "ProductProvider",
        "params": {
          "productId": "${request.path.productId}"
        }
      }
    },
    {
      "componentId": "attributes",
      "componentType": "ProductAttributes",
      "dataSource": {
        "provider": "ProductAttributeProvider",
        "params": {
          "productId": "${request.path.productId}"
        }
      }
    },
    {
      "componentId": "reviews",
      "componentType": "CustomerReviews",
      "dataSource": {
        "provider": "ReviewProvider",
        "params": {
          "productId": "${request.path.productId}",
          "limit": 5
        }
      }
    },
    {
      "componentId": "static-promo",
      "componentType": "PromoBanner",
      "dataSource": {
        "provider": "StaticContentProvider",
        "params": {
          "contentKey": "winter-sale-banner"
        }
      }
    }
  ]
}

A critical design choice here is the ${...} syntax for parameter interpolation. This allows the schema to be generic while deriving specific values from the incoming HTTP request (path variables, query parameters, headers, etc.). The provider key maps directly to a data-fetching implementation on the backend.

The Scala Implementation

Our Scala project is built around Akka HTTP for the web layer, Circe for JSON handling, and the standard Hibernate/JPA libraries.

Project Dependencies (build.sbt)

// build.sbt
val AkkaVersion = "2.6.20"
val AkkaHttpVersion = "10.2.10"
val CirceVersion = "0.14.3"
val HibernateVersion = "6.1.7.Final"
val NacosClientVersion = "2.2.0"

libraryDependencies ++= Seq(
  "com.typesafe.akka" %% "akka-actor-typed" % AkkaVersion,
  "com.typesafe.akka" %% "akka-stream" % AkkaVersion,
  "com.typesafe.akka" %% "akka-http" % AkkaHttpVersion,
  "io.circe" %% "circe-core" % CirceVersion,
  "io.circe" %% "circe-generic" % CirceVersion,
  "io.circe" %% "circe-parser" % CirceVersion,
  "com.alibaba.nacos" % "nacos-client" % NacosClientVersion,

  // JPA / Hibernate
  "org.hibernate.orm" % "hibernate-core" % HibernateVersion,
  "org.postgresql" % "postgresql" % "42.5.1", // Or your specific DB driver

  // Logging
  "ch.qos.logback" % "logback-classic" % "1.4.5",
  "com.typesafe.scala-logging" %% "scala-logging" % "3.9.5",

  // Testing
  "com.typesafe.akka" %% "akka-http-testkit" % AkkaHttpVersion % Test,
  "org.scalatest" %% "scalatest" % "3.2.14" % Test,
  "com.h2database" % "h2" % "2.1.214" % Test
)

Configuration Service: Interfacing with Nacos

This service is responsible for fetching schemas from Nacos and, importantly, caching them. It also sets up a listener to invalidate the cache when a configuration changes, ensuring we don’t have to restart the service to see UI updates.

// src/main/scala/com/example/assembler/config/NacosSchemaProvider.scala
package com.example.assembler.config

import com.alibaba.nacos.api.NacosFactory
import com.alibaba.nacos.api.config.ConfigService
import com.alibaba.nacos.api.config.listener.Listener
import com.typesafe.scalalogging.LazyLogging
import java.util.Properties
import java.util.concurrent.{ConcurrentHashMap, Executor}
import scala.concurrent.{ExecutionContext, Future, Promise}
import scala.util.Try

// Represents the parsed schema from Nacos
// Using Circe for decoding later
import io.circe.Json

trait SchemaProvider {
  def getSchema(pageId: String): Future[Option[String]]
}

class NacosSchemaProvider(implicit ec: ExecutionContext) extends SchemaProvider with LazyLogging {

  private val serverAddr = "localhost:8848" // Should come from application config
  private val group = "DEFAULT_GROUP"
  private val configCache = new ConcurrentHashMap[String, String]()

  private val configService: ConfigService = {
    val properties = new Properties()
    properties.put("serverAddr", serverAddr)
    NacosFactory.createConfigService(properties)
  }

  override def getSchema(pageId: String): Future[Option[String]] = {
    val dataId = s"page.$pageId"
    // First, check local cache
    Option(configCache.get(dataId)) match {
      case Some(cachedSchema) =>
        logger.debug(s"Schema for page '$pageId' found in cache.")
        Future.successful(Some(cachedSchema))
      case None =>
        fetchAndCacheSchema(pageId, dataId)
    }
  }

  private def fetchAndCacheSchema(pageId: String, dataId: String): Future[Option[String]] = {
    logger.info(s"Fetching schema for page '$pageId' (dataId: $dataId) from Nacos.")
    val p = Promise[Option[String]]()
    Future {
      // Nacos client is blocking, so we run it in a Future
      val schemaContent = Try(configService.getConfig(dataId, group, 5000)).toOption
      
      schemaContent.foreach { content =>
        if (content != null && content.nonEmpty) {
          logger.info(s"Successfully fetched schema for '$pageId'. Caching.")
          configCache.put(dataId, content)
          // Setup a listener to invalidate cache on change
          registerListener(dataId)
        } else {
          logger.warn(s"Schema for page '$pageId' (dataId: $dataId) is null or empty.")
        }
      }
      p.success(schemaContent)
    }
    p.future
  }

  private def registerListener(dataId: String): Unit = {
    configService.addListener(dataId, group, new Listener {
      override def getExecutor: Executor = (runnable: Runnable) => ec.execute(runnable)
      override def receiveConfigInfo(configInfo: String): Unit = {
        logger.info(s"Configuration for dataId '$dataId' has changed. Invalidating cache.")
        if (configInfo != null && configInfo.nonEmpty) {
          configCache.put(dataId, configInfo)
        } else {
          // If the config is deleted or cleared, remove it from the cache
          configCache.remove(dataId)
        }
      }
    })
  }
}

Schema Parsing and Model Definition

Using Circe’s automatic derivation, we map the JSON schema to strongly-typed Scala case classes. This prevents a large class of runtime errors.

// src/main/scala/com/example/assembler/model/SchemaModel.scala
package com.example.assembler.model

import io.circe.Json
import io.circe.generic.auto._
import io.circe.parser._

case class DataSource(provider: String, params: Map[String, Json])
case class ComponentSchema(componentId: String, componentType: String, dataSource: DataSource)
case class PageSchema(pageId: String, components: List[ComponentSchema])

object PageSchema {
  def fromString(jsonString: String): Either[io.circe.Error, PageSchema] = {
    decode[PageSchema](jsonString)
  }
}

Data Fetching Layer

This is where JPA/Hibernate comes into play. We define a generic DataProvider trait and concrete implementations for each provider key defined in our schemas.

A common pitfall here is managing the EntityManager. We must ensure it’s handled correctly, especially in an asynchronous context. A typical approach is to have a request-scoped EntityManager or a thin wrapper service that provides it.

// src/main/scala/com/example/assembler/repository/Persistence.scala
package com.example.assembler.repository

import jakarta.persistence.{EntityManager, EntityManagerFactory, Persistence}

object PersistenceModule {
  lazy val entityManagerFactory: EntityManagerFactory =
    Persistence.createEntityManagerFactory("assembler-pu")

  def createEntityManager(): EntityManager =
    entityManagerFactory.createEntityManager()
}

// persistence.xml in src/main/resources/META-INF
/*
<persistence xmlns="https://jakarta.ee/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="https://jakarta.ee/xml/ns/persistence https://jakarta.ee/xml/ns/persistence/persistence_3_0.xsd"
             version="3.0">
    <persistence-unit name="assembler-pu" transaction-type="RESOURCE_LOCAL">
        <provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
        <class>com.example.assembler.domain.Product</class>
        <class>com.example.assembler.domain.Review</class>
        <properties>
            <property name="jakarta.persistence.jdbc.driver" value="org.postgresql.Driver"/>
            <property name="jakarta.persistence.jdbc.url" value="jdbc:postgresql://localhost:5432/mydatabase"/>
            <property name="jakarta.persistence.jdbc.user" value="user"/>
            <property name="jakarta.persistence.jdbc.password" value="password"/>
            <property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect"/>
            <property name="hibernate.show_sql" value="true"/>
            <property name="hibernate.format_sql" value="true"/>
            <property name="hibernate.hbm2ddl.auto" value="update"/>
        </properties>
    </persistence-unit>
</persistence>
*/

Now, the DataProvider trait and implementations.

// src/main/scala/com/example/assembler/service/DataProvider.scala
package com.example.assembler.service

import com.example.assembler.model.RequestContext
import io.circe.Json
import scala.concurrent.{ExecutionContext, Future}

// A simple context object to pass request-specific data
case class RequestContext(pathParams: Map[String, String], queryParams: Map[String, String])

// The result of a data provider can be a success (Json) or an error
sealed trait DataProviderError
case class DataNotFound(message: String) extends DataProviderError
case class InvalidParams(message: String) extends DataProviderError
case class ProviderInternalError(cause: Throwable) extends DataProviderError

trait DataProvider {
  def name: String
  def fetchData(params: Map[String, Json], context: RequestContext)
               (implicit ec: ExecutionContext): Future[Either[DataProviderError, Json]]
}

An example implementation for fetching product data:

// src/main/scala/com/example/assembler/service/ProductProvider.scala
package com.example.assembler.service

import com.example.assembler.model.RequestContext
import com.example.assembler.repository.PersistenceModule
import com.example.assembler.domain.Product // Assuming a JPA entity
import io.circe.Json
import io.circe.generic.auto._
import io.circe.syntax._
import scala.concurrent.{ExecutionContext, Future}
import scala.jdk.OptionConverters._

class ProductProvider extends DataProvider {
  override def name: String = "ProductProvider"

  override def fetchData(params: Map[String, Json], context: RequestContext)
               (implicit ec: ExecutionContext): Future[Either[DataProviderError, Json]] = Future {
    // A production system would use a dedicated DB execution context
    val em = PersistenceModule.createEntityManager()
    try {
      // Resolve productId from request context using our interpolation syntax
      val productIdOpt = params.get("productId")
        .flatMap(_.asString)
        .flatMap(key => Interpolator.resolve(key, context))
        .flatMap(_.toLongOption)

      productIdOpt match {
        case Some(id) =>
          // Using .toScala on Java's Optional
          Option(em.find(classOf[Product], id)).toScala match {
            case Some(product) => Right(product.asJson)
            case None => Left(DataNotFound(s"Product with ID $id not found."))
          }
        case None =>
          Left(InvalidParams("Missing or invalid 'productId' parameter."))
      }
    } catch {
      case t: Throwable => Left(ProviderInternalError(t))
    } finally {
      if (em.isOpen) em.close()
    }
  }
}

// A simple helper for resolving ${...} placeholders
object Interpolator {
  private val regex = """\$\{request\.path\.(\w+)\}""".r
  def resolve(template: String, context: RequestContext): Option[String] = {
    template match {
      case regex(key) => context.pathParams.get(key)
      // Extend this to support query params, headers etc.
      case _ => Some(template) // Not a variable, return as is
    }
  }
}

Note: In a production-grade application, managing the EntityManager lifecycle would be handled more robustly, likely with a connection pool and a dedicated blocking dispatcher for database calls to avoid starving the main async thread pool.

The Core PageAssembler Service

This service orchestrates the entire process. It gets the schema, parses it, and invokes the correct DataProvider for each component concurrently.

// src/main/scala/com/example/assembler/service/PageAssemblerService.scala
package com.example.assembler.service

import com.example.assembler.config.SchemaProvider
import com.example.assembler.model._
import com.typesafe.scalalogging.LazyLogging
import io.circe._
import io.circe.syntax._

import scala.concurrent.{ExecutionContext, Future}

// The final assembled component for the response
case class AssembledComponent(componentId: String, componentType: String, data: Option[Json], error: Option[String])

class PageAssemblerService(
  schemaProvider: SchemaProvider,
  dataProviders: Map[String, DataProvider]
)(implicit ec: ExecutionContext) extends LazyLogging {

  def assemblePage(pageId: String, context: RequestContext): Future[Option[Json]] = {
    schemaProvider.getSchema(pageId).flatMap {
      case None =>
        logger.warn(s"No schema found for pageId: $pageId")
        Future.successful(None)
      case Some(schemaString) =>
        PageSchema.fromString(schemaString) match {
          case Left(err) =>
            logger.error(s"Failed to parse schema for pageId: $pageId. Error: ${err.getMessage}")
            Future.successful(None) // Or a specific error response
          case Right(pageSchema) =>
            // Concurrently fetch data for all components
            val componentFutures: List[Future[AssembledComponent]] =
              pageSchema.components.map(assembleComponent(_, context))
            
            // Wait for all futures to complete
            Future.sequence(componentFutures).map { assembledComponents =>
              Some(Map("pageId" -> pageId.asJson, "components" -> assembledComponents.asJson).asJson)
            }
        }
    }
  }

  private def assembleComponent(
    schema: ComponentSchema,
    context: RequestContext
  ): Future[AssembledComponent] = {
    dataProviders.get(schema.dataSource.provider) match {
      case None =>
        val errorMsg = s"DataProvider '${schema.dataSource.provider}' not found."
        logger.error(errorMsg)
        Future.successful(AssembledComponent(schema.componentId, schema.componentType, None, Some(errorMsg)))
      case Some(provider) =>
        provider.fetchData(schema.dataSource.params, context).map {
          case Right(data) =>
            AssembledComponent(schema.componentId, schema.componentType, Some(data), None)
          case Left(error) =>
            val errorMsg = formatError(error)
            logger.warn(s"Failed to fetch data for component '${schema.componentId}': $errorMsg")
            AssembledComponent(schema.componentId, schema.componentType, None, Some(errorMsg))
        }
    }
  }

  private def formatError(error: DataProviderError): String = error match {
    case DataNotFound(msg) => s"Data not found: $msg"
    case InvalidParams(msg) => s"Invalid parameters: $msg"
    case ProviderInternalError(t) => s"Internal error: ${t.getClass.getSimpleName}"
  }
}

The use of Future.sequence is key here. It transforms a List[Future[A]] into a Future[List[A]], running all data-fetching operations in parallel. This is a massive performance win over a sequential approach. Our error handling strategy ensures that even if one DataProvider fails, the others can succeed, and the page can still be rendered partially.

The Final Result and Lingering Issues

The API endpoint using Akka HTTP becomes trivial. It’s just a thin layer that extracts parameters from the request and calls the PageAssemblerService.

// Akka HTTP Route
path("page" / Segment) { pageId =>
  get {
    // Example of extracting a path param from a more complex route
    // e.g., /page/product-detail/123
    val productId = ... // extract '123'
    val context = RequestContext(pathParams = Map("productId" -> productId), queryParams = request.uri.query().toMap)
    
    onComplete(pageAssemblerService.assemblePage(pageId, context)) {
      case Success(Some(json)) => complete(StatusCodes.OK, json.toString)
      case Success(None) => complete(StatusCodes.NotFound, s"Page or schema '$pageId' not found.")
      case Failure(ex) =>
        log.error(s"Internal server error while assembling page $pageId", ex)
        complete(StatusCodes.InternalServerError)
    }
  }
}

When a Headless UI requests /page/product-detail?productId=123, it receives a payload like this:

{
  "pageId": "product-detail",
  "components": [
    {
      "componentId": "header",
      "componentType": "ProductHeader",
      "data": {
        "id": 123,
        "name": "Super Widget",
        "price": 99.99
      },
      "error": null
    },
    {
      "componentId": "attributes",
      "componentType": "ProductAttributes",
      "data": {
        "weight": "2.5kg",
        "dimensions": "10x20x5 cm"
      },
      "error": null
    },
    {
      "componentId": "reviews",
      "componentType": "CustomerReviews",
      "data": null,
      "error": "Data not found: No reviews found for product 123."
    },
    {
      "componentId": "static-promo",
      "componentType": "PromoBanner",
      "data": {
        "imageUrl": "/banners/winter.png",
        "link": "/sales/winter"
      },
      "error": null
    }
  ]
}

The front end can now iterate through this list, rendering the correct React/Vue component for each componentType, passing it the data object. If error is present, it can render a fallback state. We successfully decoupled the front-end layout from backend deployments.

However, this architecture is not without its own set of challenges and required future iterations. The most immediate concern is the N+1 query problem. Each DataProvider issues its own database call. While they run concurrently, this still results in multiple round trips to the database. A potential solution is to implement a DataLoader-style batching mechanism where all data requirements are collected from the schema first, and a batching service makes a more optimized set of queries (e.g., using WHERE id IN (...)).

Furthermore, the schema itself is limited. It has no concept of conditional rendering or dependencies between components. A more advanced version might use a more expressive DSL or graph-based definition to describe the page. Caching is another obvious next step; component data that doesn’t change often can be aggressively cached in Redis, keyed by the component’s provider and parameters, adding another layer of performance improvement.