51eshare's Tech Blog
This site is dedicated to sharing IT-related technologies.
  Post Categories
Architecture 35 Observability 23 Backend Development 17 Backend Architecture 15 Distributed Systems 33 Cloud Native 22 Software Architecture 15 Security 3 MLOps 17 Cloud Native & DevOps 1 DevOps 35 Data Engineering 41 Web Development 2 API Gateway 1 Mobile Development 3 Software Engineering & Architecture 1 Security Architecture 1 System Design 9 AI Engineering 2 Frontend Architecture 1 Platform Engineering 3 Build Tools 1 System Architecture 6 Real-Time Systems 1 Resilience Engineering 3 DevSecOps 1 MLOps Engineering 1 Cloud Native Security 1 Backend Engineering 2 Cloud Architecture 4 Full-Stack Development 2 Rust Engineering 1 Build Systems 1 Frontend Engineering 2 WebAssembly 1 Security Engineering 2 Cloud Security 1
Implementing an End-to-End Exactly-Once WebSocket Delivery System for Apache Spark Streaming Data Implementing an End-to-End Exactly-Once WebSocket Delivery System for Apache Spark Streaming Data
The initial system was straightforward, almost naive. A real-time operations dashboard, designed to give stakeholders a
2023-10-27 Data Engineering
Jest Headless UI WebSockets Apache Spark Exactly-Once Semantics
Building a Transactional Data Lakehouse Ingest Pipeline with TiDB CDC NATS and Apache Hudi Building a Transactional Data Lakehouse Ingest Pipeline with TiDB CDC NATS and Apache Hudi
The 24-hour latency on our core analytics dataset was no longer acceptable. Our batch ETL jobs, pulling snapshots from a
2023-10-27 Data Engineering
NATS GitHub Astro TiDB Apache Hudi
Implementing a Real-Time Feature Pipeline with FastAPI Cassandra and PyTorch for Mobile Personalization Implementing a Real-Time Feature Pipeline with FastAPI Cassandra and PyTorch for Mobile Personalization
The technical pain point was latency. Our initial personalization engine ran server-side. A user action in the Flutter a
2023-10-27 Data Engineering
PyTorch MLOps Database Cassandra Flutter FastAPI
Integrating a Hadoop Batch Feature Pipeline with a Real-Time FastAPI Serving Layer Integrating a Hadoop Batch Feature Pipeline with a Real-Time FastAPI Serving Layer
The core technical pain point was training-serving skew. Our machine learning models, trained on batch features computed
2023-10-27 Data Engineering
Babel Grafana Relay Hadoop MLOps FastAPI
Constructing a High-Throughput Search Pipeline from Hadoop to Solr with a Shadcn UI Frontend and Playwright-Driven Validation Constructing a High-Throughput Search Pipeline from Hadoop to Solr with a Shadcn UI Frontend and Playwright-Driven Validation
The operational challenge was straightforward: terabytes of semi-structured application logs were being dumped into HDFS
2023-10-27 Data Engineering
Solr Hadoop Shadcn UI Playwright Search Big Data E2E Testing
Constructing an Idempotent Batching Consumer for Bridging a REST API and HBase via RabbitMQ Constructing an Idempotent Batching Consumer for Bridging a REST API and HBase via RabbitMQ
The initial system architecture was straightforward, almost deceptively so. A fleet of RESTful API endpoints, built with
2023-10-27 Data Engineering
RabbitMQ RESTful API HBase Java System Design
4 / 7