51eshare's Tech Blog
This site is dedicated to sharing IT-related technologies.
  Post Categories
Architecture 35 Observability 23 Backend Development 17 Backend Architecture 15 Distributed Systems 33 Cloud Native 22 Software Architecture 15 Security 3 MLOps 17 Cloud Native & DevOps 1 DevOps 35 Data Engineering 41 Web Development 2 API Gateway 1 Mobile Development 3 Software Engineering & Architecture 1 Security Architecture 1 System Design 9 AI Engineering 2 Frontend Architecture 1 Platform Engineering 3 Build Tools 1 System Architecture 6 Real-Time Systems 1 Resilience Engineering 3 DevSecOps 1 MLOps Engineering 1 Cloud Native Security 1 Backend Engineering 2 Cloud Architecture 4 Full-Stack Development 2 Rust Engineering 1 Build Systems 1 Frontend Engineering 2 WebAssembly 1 Security Engineering 2 Cloud Security 1
Implementing a Go-Based Data Pipeline Using WASM for Dynamic Routing to InfluxDB and Elasticsearch Implementing a Go-Based Data Pipeline Using WASM for Dynamic Routing to InfluxDB and Elasticsearch
Our data ingestion pipeline was becoming a bottleneck, not in terms of throughput, but in operational agility. The core
2023-10-27 Data Engineering
Go WebAssembly InfluxDB System Design Elasticsearch
Implementing a Zero-Trust Data Analytics Pipeline with Trino, Jupyter, and Consul Connect Implementing a Zero-Trust Data Analytics Pipeline with Trino, Jupyter, and Consul Connect
The initial request seemed straightforward: allow our data science team to run ad-hoc queries against a Trino cluster an
2023-10-27 Data Engineering
SSG Zero Trust Consul Connect Service Mesh Presto / Trino Jupyter
Building a Unified Data API Layer from a Spark Pipeline for Both Static Gatsby and Dynamic NuxtJS Frontends Building a Unified Data API Layer from a Spark Pipeline for Both Static Gatsby and Dynamic NuxtJS Frontends
The technical debt was accumulating at an alarming rate. Our data engineering team operated a series of Apache Spark job
2023-10-27 Data Engineering
Micro-frontends Pandas GraphQL Nuxt.js Gatsby Apache Spark
Engineering an Idempotent Data Pipeline for Synchronizing Apache Hudi Commits to a Meilisearch Index via AWS SNS Engineering an Idempotent Data Pipeline for Synchronizing Apache Hudi Commits to a Meilisearch Index via AWS SNS
The initial request was straightforward: provide sub-second search capabilities over our rapidly growing multi-terabyte
2023-10-27 Data Engineering
Meilisearch Idempotency AWS SNS Spark Apache Hudi Code Review
Implementing a Verifiable Delta Lake Commit Hook with Vault's Transit Engine and AWS SNS Implementing a Verifiable Delta Lake Commit Hook with Vault's Transit Engine and AWS SNS
The central problem was decoupling and security. Our core data platform revolves around a set of critical Delta Lake tab
2023-10-27 Data Engineering
Security HashiCorp Vault Delta Lake AWS SNS Spark Scala
Implementing a Queryable Parquet-Based Data Lake on DigitalOcean for Cross-Stack Telemetry Aggregation Implementing a Queryable Parquet-Based Data Lake on DigitalOcean for Cross-Stack Telemetry Aggregation
The cost of third-party observability platforms became unsustainable. With a growing number of Python microservices and
2023-10-27 Data Engineering
Python Observability Data Lake Webpack DigitalOcean Parquet
3 / 7