# Amazon — Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

- Company: Amazon (amazon.com)
- Announced: 2026-11-02T17:11:21+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/prepare-web-content-for-amazon-bedrock-knowledge-bases-with-crawl4ai/
- Record: https://forck.live/items/30603-prepare-web-content-for-amazon-bedrock-knowledge-bases-with-crawl4ai
- Subject: Bedrock / Nova

Amazon's guide shows how to prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI, an open source Python crawler, when the managed Web Crawler connector is not enough. The managed connector handles seed URLs or sitemaps, scheduled syncs and JavaScript-loaded content, and Amazon says it is the starting point for most public website content. Crawl4AI is for sources needing structured Markdown preparation, extraction limited to a CSS-selector region, or custom metadata attributes on each document. The preparation step writes Markdown and a metadata sidecar per page to Amazon S3, and the knowledge base ingests both through its S3 data source, working with managed and customer-managed knowledge bases. The sample runs the task on AWS Fargate in a public subnet, triggered on a schedule by Amazon EventBridge, and starts an ingestion job after upload. The workflow processes public pages only and does not implement authenticated sessions; Amazon points to the Web Crawler's basic, form-based and SAML authentication options for sites requiring sign-in.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/prepare-web-content-for-amazon-bedrock-knowledge-bases-with-crawl4ai/:

> When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base , you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default.
> 
> For web content, start with the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. You give it the seed URLs or sitemaps that you’re authorized to crawl, and it crawls within the scope and rate limits that you set. Scheduled syncs keep your knowledge base current as pages change, and the connector can wait for content that loads with JavaScript. There’s no crawler for you to build, host, or operate, so this connector is the starting point for most public website content.
> 
> Some sources need more control over what reaches the knowledge base.

---

Record: https://forck.live/items/30603-prepare-web-content-for-amazon-bedrock-knowledge-bases-with-crawl4ai
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md