Our mobile release train was derailing, slowly. The core issue wasn’t regressions or feature creep, but a fundamental bottleneck between our data science and Android engineering teams. Every new ML model for our recommendation engine, every tweaked image filter, or any data-driven component required a full-cycle app release through the Google Play Store. This meant a two-week lead time at best. A/B testing different model versions was a kludge of hardcoded feature flags and manual rollouts, prone to error and incredibly inefficient. We were building sophisticated models that were stale by the time they reached our users. The problem was clear: we needed a platform to decouple model and configuration delivery from the monolithic application binary.
Our initial concept was an internal delivery platform focused on providing three core capabilities: versioning and storage for large data artifacts (our ML models), a secure, configurable gateway to expose these artifacts to mobile clients, and a robust, automated infrastructure layer to manage the whole system. The goal was to empower data scientists to push a model and, after automated validation and promotion, have it appear in the production app within minutes, not weeks. This required a careful selection of tools that could work in concert, managed with the same rigor as our production services.
The choice of Puppet for infrastructure management was pragmatic. Our SRE team had deep expertise in it, and we had an extensive library of internal modules for managing our base compute instances. While Terraform is excellent for provisioning, Puppet’s strength in continuous configuration enforcement and state management made it a better fit for maintaining the long-lived components of this new platform. For the API Gateway, Kong was a strong contender. Its plugin-centric architecture was the killer feature; we anticipated needing custom authentication and routing logic that went beyond standard JWT or OAuth2 validation. DVC (Data Version Control) was selected by the data science team over alternatives like Git LFS because its framework of pipelines (dvc.yaml) and metric tracking was a more natural fit for their ML experimentation workflow. Finally, the Android team was already migrating to Jetpack Compose, whose declarative and reactive nature was perfectly suited for building UI that could respond dynamically to configuration and models fetched from a remote endpoint.
Phase 1: Declarative Infrastructure with Puppet
The foundation of the platform had to be solid, reproducible, and managed as code. Puppet was tasked with deploying and configuring the Kong gateway and its backing Cassandra database, as well as establishing the secure S3 bucket that would serve as our DVC remote storage.
A common pitfall when starting with Puppet and a complex application like Kong is trying to manage every single configuration detail within Puppet manifests. Our first attempt did exactly this, defining Kong Services, Routes, and Consumers directly as Puppet resources. This became unmanageable. Puppet’s convergence loop is not ideal for the high frequency of changes associated with API configurations.
The superior approach we settled on was to split responsibilities. Puppet’s role is to manage the existence and base state of the infrastructure. It ensures Kong is installed, the service is running, the database connection is correctly configured, and crucial directories have the right permissions. The actual API objects within Kong would be managed via GitOps using Kong’s deck CLI tool.
Here is a simplified Puppet class for managing the Kong installation. It handles repository setup, package installation, and templating the core kong.conf file.
# modules/kong/manifests/install.pp
class kong::install (
String $version = '2.8.1',
String $database_backend = 'cassandra',
Array[String] $cassandra_cp = ['cassandra-node-1', 'cassandra-node-2'],
String $admin_listen = '127.0.0.1:8001',
) {
# Manages the official Kong repository
apt::source { 'kong':
location => 'https://kong.bintray.com/kong-deb',
release => $facts['lsbdistcodename'],
repos => 'main',
key => {
id => 'C9E628864622244F7576B7A88366D63481234567', # Example key
server => 'keyserver.ubuntu.com',
},
include => {
'src' => false,
},
}
package { 'kong':
ensure => $version,
require => Apt::Source['kong'],
}
$cassandra_contact_points = join($cassandra_cp, ',')
# Template out the core configuration file.
# Puppet is responsible for infrastructure-level config, not API objects.
file { '/etc/kong/kong.conf':
ensure => file,
owner => 'kong',
group => 'kong',
mode => '0640',
content => template('kong/kong.conf.erb'),
notify => Service['kong'],
}
service { 'kong':
ensure => running,
enable => true,
require => [Package['kong'], File['/etc/kong/kong.conf']],
subscribe => File['/etc/kong/kong.conf'],
}
}The corresponding template file (kong.conf.erb) would look like this:
# /etc/kong/kong.conf managed by Puppet
database = <%= @database_backend %>
pg_host = /var/run/postgresql
cassandra_contact_points = <%= @cassandra_contact_points %>
cassandra_keyspace = kong
proxy_listen = 0.0.0.0:8000, 0.0.0.0:8443 ssl
admin_listen = <%= @admin_listen %>Similarly, we used Puppet to manage the DVC remote. This involved creating an S3 bucket with a strict bucket policy and generating IAM credentials that would be securely passed to our CI/CD system using Vault. Puppet ensures the bucket exists and its policies are enforced, decoupling the infrastructure from the application logic that uses it.
# modules/dvc_remote/manifests/s3.pp
class dvc_remote::s3 (
String $bucket_name = 'mobile-ml-models-prod',
String $aws_region = 'us-east-1',
) {
# This assumes the aws-sdk is available on the Puppet master
# In a real-world project, this would likely use the puppetlabs/aws module
# Ensure the bucket exists
exec { "create-dvc-s3-bucket-${bucket_name}":
command => "aws s3api create-bucket --bucket ${bucket_name} --region ${aws_region} --create-bucket-configuration LocationConstraint=${aws_region}",
unless => "aws s3api head-bucket --bucket ${bucket_name}",
path => '/usr/bin:/bin/',
# Provider, credentials etc. configured elsewhere
}
$policy_file_path = '/etc/puppetlabs/code/environments/production/modules/dvc_remote/templates/s3_policy.json.epp'
$policy_content = epp($policy_file_path, { 'bucket_name' => $bucket_name })
# Apply a strict bucket policy
file { "/tmp/s3_policy_${bucket_name}.json":
ensure => file,
content => $policy_content,
}
exec { "apply-s3-bucket-policy-${bucket_name}":
command => "aws s3api put-bucket-policy --bucket ${bucket_name} --policy file:///tmp/s3_policy_${bucket_name}.json",
refreshonly => true,
path => '/usr/bin:/bin/',
subscribe => File["/tmp/s3_policy_${bucket_name}.json"],
}
}With this foundation, our platform’s infrastructure was declarative and version-controlled. Any change to Kong’s base configuration or the DVC remote’s policy would go through a standard code review and deployment process via Puppet.
Phase 2: The Custom Logic Layer in Kong
The mobile client couldn’t just be given direct, permanent access to the S3 bucket. That would be a massive security hole. We needed a mechanism where the app could request a specific model version for a given environment (e.g., recommender:v2.1.0 for production), and our gateway would validate this request before granting temporary access. This was the perfect use case for a custom Kong plugin written in Lua.
The plugin’s logic flow is as follows:
- Intercept incoming requests to a specific endpoint (e.g.,
/model-delivery/v1/request-url). - Extract query parameters:
model_name,model_version, andenvironment. - Consult a “promotion manifest.” This manifest, a simple JSON file stored in a git repository, is the source of truth for which model versions are approved for which environments.
- If the requested version is promoted for the requested environment, the plugin uses AWS credentials (securely stored as Kong plugin configuration) to generate a short-lived, pre-signed S3 GET URL for the specific DVC artifact.
- The plugin aborts the original request to an upstream service and instead immediately returns a JSON response containing the pre-signed URL.
- If the version is not promoted, or the parameters are invalid, it returns a
403 Forbiddenor400 Bad Requesterror.
Here’s a core snippet from the access.lua handler file of our plugin.
-- kong/plugins/dvc-presigner/access.lua
local BasePlugin = require "kong.plugins.base_plugin"
local cjson = require "cjson"
local http = require "resty.http"
local aws_signer = require "kong.plugins.dvc-presigner.aws-v4-signer"
local DvcPresignerHandler = BasePlugin:extend()
DvcPresignerHandler.PRIORITY = 1000
function DvcPresignerHandler:new()
DvcPresignerHandler.super.new(self, "dvc-presigner")
end
-- This is the core logic run on every request to the configured route.
function DvcPresignerHandler:access(conf)
DvcPresignerHandler.super.access(self)
local model_name = kong.request.get_query_arg("model_name")
local model_version = kong.request.get_query_arg("model_version")
local environment = kong.request.get_query_arg("environment") or "production"
if not model_name or not model_version then
return kong.response.exit(400, { message = "Bad Request: model_name and model_version are required." })
end
-- In a production system, this manifest would be fetched from a reliable source
-- and cached in-memory (e.g., using lua_shared_dict) to avoid filesystem reads on every request.
-- For this example, we read it directly. A production system might fetch it from S3 or a service.
local manifest_path = conf.manifest_path
local file = io.open(manifest_path, "rb")
if not file then
kong.log.err("Failed to open promotion manifest at: ", manifest_path)
return kong.response.exit(500, { message = "Internal Server Error: Cannot read promotion manifest." })
end
local manifest_content = file:read("*a")
file:close()
local manifest_data = cjson.decode(manifest_content)
-- Check if the requested model version is promoted for the environment.
local is_promoted = manifest_data[environment] and manifest_data[environment][model_name] == model_version
if not is_promoted then
return kong.response.exit(403, { message = "Forbidden: The requested model version is not promoted for this environment." })
end
-- The object key in S3 is derived from the first two characters of the DVC hash (the .dvc file content)
-- and the rest of the hash. This logic needs to be replicated. For simplicity, we'll assume a path.
-- A more robust implementation would fetch the .dvc file to get the true md5 hash.
local object_key = "dvc-store/" .. model_name .. "/" .. model_version
local expiration_seconds = 300 -- 5 minutes
local request_params = {
region = conf.aws_region,
service = "s3",
method = "GET",
host = conf.s3_bucket_name .. ".s3." .. conf.aws_region .. ".amazonaws.com",
path = "/" .. object_key,
headers = {
host = conf.s3_bucket_name .. ".s3." .. conf.aws_region .. ".amazonaws.com",
},
query_args = {},
}
-- The aws_signer module handles the complex AWS Signature v4 process.
local signed_url = aws_signer.get_presigned_url(
request_params,
conf.aws_key,
conf.aws_secret,
expiration_seconds
)
-- Terminate the request processing here and return our generated URL.
return kong.response.exit(200, {
model_name = model_name,
model_version = model_version,
download_url = signed_url,
expires_in = expiration_seconds
}, { ["Content-Type"] = "application/json" })
end
return DvcPresignerHandlerThis plugin became the linchpin of our platform, providing a secure and dynamic bridge between our internal data store and our public-facing clients.
Phase 3: The Data Science Workflow and CI/CD Pipeline
With the infrastructure in place, we needed to define the workflow for data scientists. The process had to be simple and integrated into their existing tools.
- Experimentation: The data scientist works on a feature branch, training models and tracking experiments. The
dvc.yamlfile defines the pipeline. - Versioning: Once a candidate model is ready, they commit the
.dvcpointer file and rundvc pushto upload the actual model artifact to our S3 remote. - Promotion PR: They open a pull request to the
mainbranch. This PR contains the updated.dvcfile and potentially updates to the model training code. - Automated Validation: The PR triggers a CI/CD pipeline (we used GitLab CI). This pipeline runs a suite of tests:
- It pulls the DVC artifact (
dvc pull). - It runs a performance benchmark against a golden dataset.
- It checks for model format correctness.
- It generates a performance report.
- It pulls the DVC artifact (
- Manual Promotion: If all automated checks pass, a senior data scientist or ML engineer can approve the PR. Merging it to
mainis the signal for production promotion. - Deployment Pipeline: The merge to
maintriggers a separate deployment pipeline. This pipeline’s crucial step is to automatically update thepromotion_manifest.jsonfile, check in the change, and then trigger adeck synccommand to ensure Kong is aware of any new API routes or configurations.
A simplified dvc.yaml might look like this:
stages:
featurize:
cmd: python src/featurize.py data/raw/
deps:
- data/raw/
- src/featurize.py
params:
- featurize.max_features
outs:
- data/processed/features.pkl
train:
cmd: python src/train.py data/processed/features.pkl
deps:
- data/processed/features.pkl
- src/train.py
params:
- train.n_estimators
- train.seed
outs:
- models/model.pkl:
cache: true
metrics:
- metrics.json:
cache: falseAnd here is the critical shell script snippet from our deployment pipeline that updates the promotion manifest:
#!/binbin/bash
set -eo pipefail
# This script runs after a merge to the main branch.
# Assumes jq is installed.
MANIFEST_FILE="deploy/promotion_manifest.json"
MODEL_NAME="recommender_v2" # This would be determined dynamically
NEW_MODEL_VERSION=$(git tag --sort=-v:refname | head -n 1) # Assumes we tag releases
if [ -z "$NEW_MODEL_VERSION" ]; then
echo "Error: No git tag found for the new version."
exit 1
fi
echo "Promoting ${MODEL_NAME} to version ${NEW_MODEL_VERSION} for production..."
# Create a temporary file to work with
temp_manifest=$(mktemp)
cp "$MANIFEST_FILE" "$temp_manifest"
# Use jq to update the manifest. This is idempotent.
jq --arg model "$MODEL_NAME" --arg version "$NEW_MODEL_VERSION" \
'.production[$model] = $version' "$temp_manifest" > "$MANIFEST_FILE"
rm "$temp_manifest"
# Check for changes in the manifest
if ! git diff --quiet "$MANIFEST_FILE"; then
echo "Promotion manifest updated. Committing and pushing changes."
git config --global user.name "CI Bot"
git config --global user.email "[email protected]"
git add "$MANIFEST_FILE"
git commit -m "feat(promote): Promote ${MODEL_NAME} to ${NEW_MODEL_VERSION} [skip ci]"
git push origin main
else
echo "Manifest already up-to-date. No changes needed."
fi
# Here we would also trigger a reload or sync for the Kong plugin
# to pick up the new manifest, e.g., by calling a Kong admin API endpoint.Phase 4: The Dynamic Jetpack Compose Client
The final piece was the Android application. The client needed to be architected to handle the dynamic nature of the models and configurations. We established a clear pattern using a Repository, ViewModel, and Composable UI components.
First, the network layer using Retrofit to define the API call to our Kong gateway:
// src/main/java/com/example/myapp/data/network/ModelDeliveryApiService.kt
package com.example.myapp.data.network
import retrofit2.http.GET
import retrofit2.http.Query
data class ModelMetadataResponse(
val model_name: String,
val model_version: String,
val download_url: String,
val expires_in: Int
)
interface ModelDeliveryApiService {
@GET("model-delivery/v1/request-url")
suspend fun getModelDownloadUrl(
@Query("model_name") modelName: String,
@Query("model_version") modelVersion: String, // Or could be 'latest'
@Query("environment") environment: String = "production"
): ModelMetadataResponse
}Next, a ModelRepository to orchestrate fetching the metadata, downloading the actual model file, and caching it on the device to avoid repeated downloads.
// src/main/java/com/example/myapp/data/ModelRepository.kt
package com.example.myapp.data
import android.content.Context
import android.util.Log
import com.example.myapp.data.network.ModelDeliveryApiService
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.File
import java.io.FileOutputStream
import java.net.URL
class ModelRepository(
private val apiService: ModelDeliveryApiService,
private val context: Context
) {
private val TAG = "ModelRepository"
suspend fun getAndCacheModel(modelName: String, modelVersion: String): File {
val modelFile = File(context.cacheDir, "${modelName}_${modelVersion}.tflite")
// In a production app, we'd check a database or SharedPreferences
// for metadata to see if we already have the latest version.
if (modelFile.exists()) {
Log.d(TAG, "Model found in cache: ${modelFile.path}")
return modelFile
}
Log.d(TAG, "Model not in cache. Fetching from remote.")
val metadata = apiService.getModelDownloadUrl(modelName, modelVersion)
return downloadModel(metadata.download_url, modelFile)
}
private suspend fun downloadModel(url: String, outputFile: File): File = withContext(Dispatchers.IO) {
// This is a simplified download implementation.
// A production app should use a robust download manager like Fetch or WorkManager.
val connection = URL(url).openConnection()
connection.connect()
connection.getInputStream().use { input ->
FileOutputStream(outputFile).use { output ->
input.copyTo(output)
}
}
Log.d(TAG, "Model downloaded successfully to ${outputFile.path}")
outputFile
}
}The ViewModel ties this logic to the UI layer, managing the state (Loading, Success, Error) and holding a reference to the loaded TensorFlow Lite interpreter.
// src/main/java/com/example/myapp/ui/recommender/RecommenderViewModel.kt
package com.example.myapp.ui.recommender
import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import com.example.myapp.data.ModelRepository
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch
import org.tensorflow.lite.Interpreter
import java.io.File
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel
sealed class RecommenderUiState {
object Loading : RecommenderUiState()
data class Success(val interpreter: Interpreter) : RecommenderUiState()
data class Error(val message: String) : RecommenderUiState()
}
class RecommenderViewModel(private val repository: ModelRepository) : ViewModel() {
private val _uiState = MutableStateFlow<RecommenderUiState>(RecommenderUiState.Loading)
val uiState: StateFlow<RecommenderUiState> = _uiState.asStateFlow()
fun loadRecommenderModel(version: String) {
viewModelScope.launch {
_uiState.value = RecommenderUiState.Loading
try {
val modelFile = repository.getAndCacheModel("recommender_v2", version)
val interpreter = Interpreter(loadModelFile(modelFile))
_uiState.value = RecommenderUiState.Success(interpreter)
} catch (e: Exception) {
Log.e("RecommenderViewModel", "Error loading model", e)
_uiState.value = RecommenderUiState.Error("Failed to initialize recommender: ${e.message}")
}
}
}
private fun loadModelFile(modelFile: File): MappedByteBuffer {
val fileDescriptor = modelFile.inputStream().channel
return fileDescriptor.map(FileChannel.MapMode.READ_ONLY, 0, modelFile.length())
}
}Finally, the Jetpack Compose UI reactively observes the state from the ViewModel and renders the appropriate content.
// src/main/java/com/example/myapp/ui/recommender/RecommenderScreen.kt
package com.example.myapp.ui.recommender
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.material.CircularProgressIndicator
import androidx.compose.material.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.lifecycle.viewmodel.compose.viewModel
@Composable
fun RecommenderScreen(
// In a real app, this would be passed in or retrieved from a config service.
targetModelVersion: String = "2.1.0",
viewModel: RecommenderViewModel = viewModel()
) {
LaunchedEffect(key1 = targetModelVersion) {
viewModel.loadRecommenderModel(targetModelVersion)
}
val uiState by viewModel.uiState.collectAsState()
Box(
modifier = Modifier.fillMaxSize(),
contentAlignment = Alignment.Center
) {
when (val state = uiState) {
is RecommenderUiState.Loading -> {
CircularProgressIndicator()
}
is RecommenderUiState.Error -> {
Text(text = state.message)
}
is RecommenderUiState.Success -> {
// Here you would use the state.interpreter to run inference
// and display the results in another Composable.
RecommendationsList(interpreter = state.interpreter)
}
}
}
}This architecture gives us the end-to-end flow. A data scientist pushes a tag, and after the pipeline succeeds, the mobile app, on its next launch, can query for that version, securely download it, and use it, all without a single line of client code changing.
sequenceDiagram
participant DS as Data Scientist
participant Git
participant GitLab_CI as CI/CD Pipeline
participant DVC_S3 as DVC Remote (S3)
participant Puppet
participant Kong_GW as Kong Gateway
participant MobileApp as Jetpack Compose App
DS->>Git: git push (code + .dvc file)
DS->>DVC_S3: dvc push (model artifact)
Git->>GitLab_CI: Triggers pipeline on merge to main
GitLab_CI->>Git: Updates promotion_manifest.json
GitLab_CI->>Kong_GW: Triggers config reload if needed
Note right of Puppet: Puppet continuously ensures
Kong instance is healthy.
Puppet-->>Kong_GW: Enforces base configuration
MobileApp->>Kong_GW: GET /model-delivery/v1/request-url?model_name=rec&model_version=2.1.0
Kong_GW->>Kong_GW: Executes custom dvc-presigner plugin
Kong_GW-->>DVC_S3: Generates temporary pre-signed URL
Kong_GW->>MobileApp: 200 OK { "download_url": "s3://..." }
MobileApp->>DVC_S3: GET s3://... (using pre-signed URL)
DVC_S3->>MobileApp: Returns raw model file bytes
MobileApp->>MobileApp: Caches model, loads into TFLite, updates UI
The platform is functional, but it’s far from complete. The single JSON file for the promotion manifest is a clear technical debt; it doesn’t scale to many model types or environments and lacks any audit trail. A future iteration must replace this with a dedicated microservice backed by a proper database. Our Kong plugin’s performance could be improved significantly by caching the manifest in memory (lua_shared_dict) instead of reading from disk on every request. Furthermore, the client-side download logic is simplistic; it needs to be hardened with a background download manager like WorkManager to handle network failures and large files without blocking the user experience. The current system only supports full artifact downloads, making it inefficient for large models with minor updates; exploring binary diffs and patching on the client is a necessary next step for optimization.
