Jikan API Scraper: Fault-Tolerant ML Dataset Generator
An autonomous, fault-tolerant data harvesting engine designed to extract MyAnimeList datasets for machine learning models, featuring exponential backoff and low-level disk I/O flushing.
Tech stack
Performance & metrics
- Elimination of memory ballooning by employing a chunked append strategy, bypassing the need to load the entire dataset into RAM at once
- Stable memory footprint maintained by ensuring temporal HTTP response objects are immediately released for garbage collection post-processing
- Dynamic calculation of thread sleep intervals based strictly on the API's current throttling state to optimize execution time
Why I built it
Machine learning pipelines require massive, highly structured datasets, but public APIs like Jikan enforce strict rate limits and suffer from unpredictable downtime. I engineered this robust scraping infrastructure to autonomously harvest raw data for future model training, ensuring absolutely zero data loss during multi-hour execution cycles.
Architecture & decisions
The script operates entirely on Python, utilizing requests for network I/O and pandas for DataFrame manipulation. The core execution loop implements a resilient retry mechanism equipped with exponential backoff and randomized jitter to seamlessly absorb HTTP 429 (Too Many Requests) and 5xx server errors. To guarantee data integrity during potential OS crashes or exhaustive long-running tasks, file descriptors are forcibly flushed directly to disk using the low-level os.fsync() system call after every pagination cycle. Standard output (stdout) is dynamically intercepted and routed into a multi-channel structured logging architecture.
Key features
- 01Aggressive HTTP 429 rate-limit mitigation via an exponential backoff algorithm integrated with randomized jitter
- 02Absolute persistence of incremental DataFrame writes guaranteed by OS-level file syncing (
os.fsync) - 03Dual-depth CSV exporting strategy (core metrics vs. complex entity relationships) pre-optimized for ML feature engineering
- 04Autonomous error-handling and dead-letter logging mechanisms to isolate network disconnects and malformed JSON payloads
Technical review
This project is not just a standard web scraper; it is a resilient data ingestion pipeline. By implementing OS-level file synchronization (fsync) and a custom exception hierarchy, the engine survives API timeouts, connection resets, and rigid throttling limits entirely without manual intervention. The dual-table export strategy—separating core metrics from heavy relational nodes—is a calculated data modeling decision that optimizes the payload for immediate ingestion into Pandas/Polars training environments with minimal preprocessing overhead.