51eshare's Tech Blog
This site is dedicated to sharing IT-related technologies.
  Post Categories
Architecture 35 Observability 23 Backend Development 17 Backend Architecture 15 Distributed Systems 33 Cloud Native 22 Software Architecture 15 Security 3 MLOps 17 Cloud Native & DevOps 1 DevOps 35 Data Engineering 41 Web Development 2 API Gateway 1 Mobile Development 3 Software Engineering & Architecture 1 Security Architecture 1 System Design 9 AI Engineering 2 Frontend Architecture 1 Platform Engineering 3 Build Tools 1 System Architecture 6 Real-Time Systems 1 Resilience Engineering 3 DevSecOps 1 MLOps Engineering 1 Cloud Native Security 1 Backend Engineering 2 Cloud Architecture 4 Full-Stack Development 2 Rust Engineering 1 Build Systems 1 Frontend Engineering 2 WebAssembly 1 Security Engineering 2 Cloud Security 1
Implementing Record-Level Traceability in an Apache Hudi Pipeline Using OpenTelemetry and Nacos on Alibaba Cloud Implementing Record-Level Traceability in an Apache Hudi Pipeline Using OpenTelemetry and Nacos on Alibaba Cloud
The Spark UI reported success, the Hudi timeline showed a new commit every five minutes, and our downstream dashboards w
2023-10-27 Data Engineering
OpenTelemetry Alibaba Cloud Nacos Spark Apache Hudi Data Observability
Building a Real-Time Data Reconciliation Pipeline with Koa Delta Lake and Redux Building a Real-Time Data Reconciliation Pipeline with Koa Delta Lake and Redux
The core technical challenge was twofold: our operations team required a live, sub-second view of critical inventory tab
2023-10-27 Data Engineering
Node.js Kafka Koa Delta Lake WebSocket CDC Redux
Implementing a Sandboxed WebAssembly UDF Engine within a Java Data Pipeline for Stateful Transformation Implementing a Sandboxed WebAssembly UDF Engine within a Java Data Pipeline for Stateful Transformation
Our real-time analytics pipeline was hitting a wall. The core of the system, a Java application consuming from Kafka, wa
2023-10-27 Data Engineering
Rust DynamoDB ClickHouse WebAssembly Java UDF
Constructing a Resilient MySQL to Hadoop CDC Pipeline with Docker and Debezium Constructing a Resilient MySQL to Hadoop CDC Pipeline with Docker and Debezium
The operational pain point was clear: our primary MySQL database, backing a critical monolithic application, was bucklin
2023-10-27 Data Engineering
Hadoop Kafka Docker MySQL CDC Debezium
Constructing a Stateful Scraping Pipeline with Python Puppeteer and Neo4j for Complex Relational Data Constructing a Stateful Scraping Pipeline with Python Puppeteer and Neo4j for Complex Relational Data
The initial requirement seemed straightforward: map the ownership and directorship structures of a portfolio of private
2023-10-27 Data Engineering
Python Neo4j Puppeteer Web Scraping Graph Database ETL
Constructing a Real-Time NLP Enrichment Pipeline with Batch Processing and Multi-Tier Caching Constructing a Real-Time NLP Enrichment Pipeline with Batch Processing and Multi-Tier Caching
The initial system design was straightforward and, in hindsight, naive. An API endpoint received a block of text, passed
2023-10-27 Data Engineering
Python Rollup Memcached spaCy Snowflake asyncio
2 / 7