51eshare's Tech Blog
This site is dedicated to sharing IT-related technologies.
  Post Categories
Architecture 35 Observability 23 Backend Development 17 Backend Architecture 15 Distributed Systems 33 Cloud Native 22 Software Architecture 15 Security 3 MLOps 17 Cloud Native & DevOps 1 DevOps 35 Data Engineering 41 Web Development 2 API Gateway 1 Mobile Development 3 Software Engineering & Architecture 1 Security Architecture 1 System Design 9 AI Engineering 2 Frontend Architecture 1 Platform Engineering 3 Build Tools 1 System Architecture 6 Real-Time Systems 1 Resilience Engineering 3 DevSecOps 1 MLOps Engineering 1 Cloud Native Security 1 Backend Engineering 2 Cloud Architecture 4 Full-Stack Development 2 Rust Engineering 1 Build Systems 1 Frontend Engineering 2 WebAssembly 1 Security Engineering 2 Cloud Security 1
Constructing a Real-Time Security Event Correlation Pipeline with a Data Lake, TimescaleDB, and Meilisearch Constructing a Real-Time Security Event Correlation Pipeline with a Data Lake, TimescaleDB, and Meilisearch
Our Security Operations Center (SOC) was drowning in data. The primary culprit was the firehose of logs from our Web App
2023-10-27 Data Engineering
Security Go Meilisearch Data Lake WAF TimescaleDB
Constructing a Resumable Real-Time Data Pipeline Observability UI with Qwik and Apache Hudi Constructing a Resumable Real-Time Data Pipeline Observability UI with Qwik and Apache Hudi
Our core data ingestion pipelines were becoming black boxes. When a job failed or data quality issues arose, the post-mo
2023-10-27 Data Engineering
Qwik Observability Apache Hudi Google Cloud Pub/Sub Real-Time Systems
Building a Type-Safe Data Hydration Layer in Haskell for SSR Using SQL Server CDC and a NoSQL Cache Building a Type-Safe Data Hydration Layer in Haskell for SSR Using SQL Server CDC and a NoSQL Cache
The project started with a familiar, frustrating constraint: a mission-critical, monolithic SQL Server database that we
2023-10-27 Data Engineering
Redis SSR Architecture SQL Server CDC NoSQL Haskell
Building a Cost-Aware Node.js ELT Orchestrator for Snowflake Through Iterative Refinement Building a Cost-Aware Node.js ELT Orchestrator for Snowflake Through Iterative Refinement
The initial problem wasn’t a technical marvel; it was a bog-standard operational headache. Our data ingestion pipeline w
2023-10-27 Data Engineering
Node.js Observability Snowflake Scrum TypeScript ELT
Implementing an Archival Pipeline for Event Sourcing Streams with Kafka Connect and HDFS Sink Implementing an Archival Pipeline for Event Sourcing Streams with Kafka Connect and HDFS Sink
The operational event store, a PostgreSQL table holding our core business events as JSONB, was growing at over 500GB per
2023-10-27 Data Engineering
Hadoop Event Sourcing Message Queue Kafka HDFS Architecture
Building a Metadata-Driven Query Gateway with a Custom Kong Lua Plugin Interfacing Trino and Firestore on AWS EKS Building a Metadata-Driven Query Gateway with a Custom Kong Lua Plugin Interfacing Trino and Firestore on AWS EKS
The initial problem wasn’t about data access; it was about the operational friction of data access. Our data science and
2023-10-27 Data Engineering
API Gateway Kong Lua Trino AWS EKS Firestore
6 / 7