51eshare's Tech Blog
This site is dedicated to sharing IT-related technologies.
  Post Categories
Architecture 35 Observability 23 Backend Development 17 Backend Architecture 15 Distributed Systems 33 Cloud Native 22 Software Architecture 15 Security 3 MLOps 17 Cloud Native & DevOps 1 DevOps 35 Data Engineering 41 Web Development 2 API Gateway 1 Mobile Development 3 Software Engineering & Architecture 1 Security Architecture 1 System Design 9 AI Engineering 2 Frontend Architecture 1 Platform Engineering 3 Build Tools 1 System Architecture 6 Real-Time Systems 1 Resilience Engineering 3 DevSecOps 1 MLOps Engineering 1 Cloud Native Security 1 Backend Engineering 2 Cloud Architecture 4 Full-Stack Development 2 Rust Engineering 1 Build Systems 1 Frontend Engineering 2 WebAssembly 1 Security Engineering 2 Cloud Security 1
Building a Reproducible ML Training Pipeline with Apache Iceberg as a Versioned Dataset on DigitalOcean Building a Reproducible ML Training Pipeline with Apache Iceberg as a Versioned Dataset on DigitalOcean
The project started with a familiar, sinking feeling. We were retraining a computer vision model in Keras, built six mon
2023-10-27 Data Engineering
Keras Python MLOps DigitalOcean Apache Iceberg
Managing Concurrent Delta Lake Writes from a Multi-User Web Application Managing Concurrent Delta Lake Writes from a Multi-User Web Application
The core challenge began with a simple, yet deceptive, requirement: allow our data science team to collaboratively clean
2023-10-27 Data Engineering
Flask React Delta Lake Concurrency Data Lakehouse
Engineering a Frontend Visualization Pipeline for Massive Time-Series Datasets from TimescaleDB Engineering a Frontend Visualization Pipeline for Massive Time-Series Datasets from TimescaleDB
The initial requirement seemed straightforward: build a frontend dashboard to visualize high-resolution infrastructure m
2023-10-27 Data Engineering
Observability Frontend React TimescaleDB Data Warehouse Canvas
Implementing a Low-Latency Analytics Pipeline from DynamoDB to ClickHouse Using a Quarkus Stream Consumer and Dask for Parallel Processing Implementing a Low-Latency Analytics Pipeline from DynamoDB to ClickHouse Using a Quarkus Stream Consumer and Dask for Parallel Processing
The core operational dataset, residing in DynamoDB, was generating write traffic exceeding 10,000 transactions per secon
2023-10-27 Data Engineering
Dask Python DynamoDB CDC ClickHouse Java Quarkus
Constructing a Real-Time Hybrid Search Index by Streaming TiDB Changes to OpenSearch Vector Engine via Kotlin Constructing a Real-Time Hybrid Search Index by Streaming TiDB Changes to OpenSearch Vector Engine via Kotlin
The operational data store, a distributed TiDB cluster, handles our transactional workload with ACID guarantees. However
2023-10-27 Data Engineering
Kafka Kotlin CDC TiDB OpenSearch Vector
7 / 7