From the source
From the source
How Harvey rebuilt file ingestion for scale, adding folder uploads and one-way sync.
Harvey built a new asynchronous file ingestion system using Temporal to handle hundreds of thousands of files from document management systems.
The system adds folder uploads and one-way sync, so changes in the DMS are automatically detected and synced.
The architecture uses granular Temporal activities for crawling and downloading, with automatic retries and checkpointing to resume from failures.
From the source
To solve these pain points, we decided to build two new features. Folder uploads: Select any folder from a DMS and import the entire hierarchy into Harvey — including nested folders, their associated files, and all relevant metadata. Our goal was to support one click import of hundreds of thousands of files. One-way sync: Connect a folder or client matter for continuous synchronization. When files or folders change in the DMS, Harvey automatically detects and syncs the updates.
harvey.ai
