HARVEST

Highlights

• Engineered an automated data collection system for e-commerce stores utilizing the client’s app, focusing on extracting comprehensive order histories (up to 60 days) from the Shopify platform.
• Spearheaded the technical transition from legacy REST APIs (limited to 7-day lookbacks) to GraphQL Bulk Operations, enabling the system to extract massive datasets efficiently.
• Resolved critical Out-Of-Memory (OOM) worker crashes caused by gigantic data export files (4-6GB JSONL). Redesigned the data ingestion approach to stream data chunks directly from the network instead of loading entire files into RAM, utilizing Java-based workers.
• Architected complex, long-running workflows using Temporal.io to mitigate severe third-party rate-limiting issues. Since Shopify bulk export jobs could take up to 24 hours, Temporal efficiently managed task states without aggressive polling that would exhaust shared API tokens.
• Leveraged Temporal.io’s robust infinite retry mechanisms and activity recovery features, ensuring 100% fault tolerance and zero data loss during network failures or service disruptions.
• The robust architecture and high-quality codebase earned stellar reviews from the client. The system’s long-term operational stability directly led to the acquisition of a subsequent data analytics and visualization project.