Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

Amazon's guide shows how to prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI, an open source Python crawler, when the managed Web Crawler connector is not enough.

• The managed connector handles seed URLs or sitemaps, scheduled syncs and JavaScript-loaded content, and Amazon says it is the starting point for most public website content.
• Crawl4AI is for sources needing structured Markdown preparation, extraction limited to a CSS-selector region, or custom metadata attributes on each document.
• The preparation step writes Markdown and a metadata sidecar per page to Amazon S3, and the knowledge base ingests both through its S3 data source, working with managed and customer-managed knowledge bases.
• The sample runs the task on AWS Fargate in a public subnet, triggered on a schedule by Amazon EventBridge, and starts an ingestion job after upload.
• The workflow processes public pages only and does not implement authenticated sessions; Amazon points to the Web Crawler's basic, form-based and SAML authentication options for sites requiring sign-in.