agentcore-browser-web-scraping

Name: agentcore-browser-web-scraping
Author: aws-samples

37stars20forks

How to install agentcore-browser-web-scraping?

aws-samples/sample-agent-skills-for-builders/agentcore-browser-web-scraping

$npx -y skills add aws-samples/sample-agent-skills-for-builders --skill agentcore-browser-web-scraping

Installs into the current project.

Summary

Build production web scraping on Bedrock AgentCore Browser — connect Playwright over signed CDP WebSocket, drive extraction with an LLM agent over fixed tool primitives (navigate, scroll, extract-by-selector, screenshot), reuse login state via Browser Profiles with a DCV live-vie

Files · 1

View on GitHub

SKILL.md

1	# AgentCore Browser Web Scraping
2
3	Amazon Bedrock AgentCore Browser gives you a managed cloud Chromium — no
4	crawler fleet, no resident containers, per-session billing, natural
5	isolation. This skill captures a production scraping architecture built on
6	it: an LLM agent decides what to do on the page, but only through **fixed
7	code primitives** it can parameterize, never arbitrary scripts. Login state
8	lives in Browser Profiles that users populate once through a live-view
9	session, so credentials never transit the conversation.
10
11	```
12	Collector (Python, worker thread)
13	├─ browser_session(region, profile_configuration=..., proxy_configuration=...)
14	│ └─ StartBrowserSession → generate_ws_headers() (SigV4)
15	│ └─ playwright chromium.connect_over_cdp(ws_url, headers)
16	└─ Strands Agent (LLM) with 6 fixed tools:
17	navigate / scroll_to_bottom / click_load_more /
18	get_page_text / screenshot / extract_by_selector
19	```
20
21	## When to Apply
22
23	Reference this skill when:
24
25	- Scraping dynamic, JS-rendered, or lazy-loading pages (social feeds,
26	forums, review sites) with AgentCore Browser + Playwright.
27	- Scraping sites that require login (X/Twitter, Reddit, Instagram) and you
28	need reusable login state that never leaves AWS.
29	- Scrapes return 0 items on pages that clearly have content — usually a
30	scroll-container or login-wall misdiagnosis, both covered here.
31	- Google SSO inside the cloud browser silently fails (third-party cookies).
32	- The target site blocks AWS egress IPs and you need an external proxy.
33
34	Not for: general AgentCore Browser service overview or session APIs —
35	see the `aws-agentic-ai` skill's Browser service docs for that.
36
37	## How It Works
38
39	### 1. Session, connection, and the LLM-driven extraction loop
40
41	Connect Playwright to the managed browser over a SigV4-signed CDP WebSocket,
42	then let an LLM agent drive a bounded tool loop. Key decisions:
43
44	- LLM decides, code executes. A hand-written script per site doesn't
45	scale across arbitrary DOMs; raw LLM-generated JS is an injection surface.
46	The middle path: six fixed tools, the model only supplies parameters.
47	- `extract_by_selector`: a constant JS extraction template evaluated with
48	the model's CSS selectors passed as data arguments — zero string
49	concatenation, zero eval. Preserves item boundaries and attributes that a
50	flat `innerText` dump loses.
51	- `screenshot` (downscaled JPEG returned as an image tool-result) lets
52	the model see the page and distinguish "genuinely empty" from login
53	wall / CAPTCHA / cookie banner before declaring failure.
54	- Adaptive step budget: scale the agent's max tool-steps (and the session
55	timeout) with the requested record count — lazy feeds yield 10–20 items
56	per scroll, so a fixed small budget silently under-collects.
57	- Scrolling that actually works: `document.body.scrollHeight` is 0 on
58	flex layouts (YouTube) — scroll `document.scrollingElement` instead, and
59	also iterate inner overflow panels; comments often live in an overlay
60	that window scrolling never touches.
61	- SSRF guard: HTTPS-only, no IP literals, private ranges and IMDS
62	blocked for the hostname and every resolved IP — re-validated inside the
63	`navigate` tool on every call, so the model can't pivot mid-session.
64
65	See [references/session-and-extraction.md](references/session-and-extraction.md)
66	— includes IAM specifics (the `browser/` vs `browser-custom/` ARN pitfall,
67	`InvokeModelWithResponseStream`), the asyncio × sync-Playwright worker-thread
68	requirement, and the error-code taxonomy.
69
70	### 2. Login state: Browser Profiles + live-view login
71
72	Users log in once inside the cloud browser via a DCV live-view page; the
73	session's cookies/tokens are saved to an AgentCore Browser Profile and reused
74	by every later scrape (`profile_configuration={"profileIdentifier": ...}`).
75	Credentials never appear in the conversation or your database.
76
77	Flow: mint a one-time URL token → `start_browser_session` with the profile →
78	presign the `live-view` stream endpoint (SigV4 query auth) → embed the DCV
79	Web Client → on completion `save_browser_session_profile` then
80	`stop_browser_session` (that order — save requires a live session).
81
82	The subtle part is

People who liked this also viewed

6 picks

Type

agentcore-browser-web-scraping

37stars20forks

View on GitHub

How to install agentcore-browser-web-scraping?

aws-samples/sample-agent-skills-for-builders/agentcore-browser-web-scraping

$npx -y skills add aws-samples/sample-agent-skills-for-builders --skill agentcore-browser-web-scraping

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AgentCore Browser Web Scraping
2
3	Amazon Bedrock AgentCore Browser gives you a managed cloud Chromium — no
4	crawler fleet, no resident containers, per-session billing, natural
5	isolation. This skill captures a production scraping architecture built on
6	it: an LLM agent decides what to do on the page, but only through **fixed
7	code primitives** it can parameterize, never arbitrary scripts. Login state
8	lives in Browser Profiles that users populate once through a live-view
9	session, so credentials never transit the conversation.
10
11	```
12	Collector (Python, worker thread)
13	├─ browser_session(region, profile_configuration=..., proxy_configuration=...)
14	│ └─ StartBrowserSession → generate_ws_headers() (SigV4)
15	│ └─ playwright chromium.connect_over_cdp(ws_url, headers)
16	└─ Strands Agent (LLM) with 6 fixed tools:
17	navigate / scroll_to_bottom / click_load_more /
18	get_page_text / screenshot / extract_by_selector
19	```
20
21	## When to Apply
22
23	Reference this skill when:
24
25	- Scraping dynamic, JS-rendered, or lazy-loading pages (social feeds,
26	forums, review sites) with AgentCore Browser + Playwright.
27	- Scraping sites that require login (X/Twitter, Reddit, Instagram) and you
28	need reusable login state that never leaves AWS.
29	- Scrapes return 0 items on pages that clearly have content — usually a
30	scroll-container or login-wall misdiagnosis, both covered here.
31	- Google SSO inside the cloud browser silently fails (third-party cookies).
32	- The target site blocks AWS egress IPs and you need an external proxy.
33
34	Not for: general AgentCore Browser service overview or session APIs —
35	see the `aws-agentic-ai` skill's Browser service docs for that.
36
37	## How It Works
38
39	### 1. Session, connection, and the LLM-driven extraction loop
40
41	Connect Playwright to the managed browser over a SigV4-signed CDP WebSocket,
42	then let an LLM agent drive a bounded tool loop. Key decisions:
43
44	- LLM decides, code executes. A hand-written script per site doesn't
45	scale across arbitrary DOMs; raw LLM-generated JS is an injection surface.
46	The middle path: six fixed tools, the model only supplies parameters.
47	- `extract_by_selector`: a constant JS extraction template evaluated with
48	the model's CSS selectors passed as data arguments — zero string
49	concatenation, zero eval. Preserves item boundaries and attributes that a
50	flat `innerText` dump loses.
51	- `screenshot` (downscaled JPEG returned as an image tool-result) lets
52	the model see the page and distinguish "genuinely empty" from login
53	wall / CAPTCHA / cookie banner before declaring failure.
54	- Adaptive step budget: scale the agent's max tool-steps (and the session
55	timeout) with the requested record count — lazy feeds yield 10–20 items
56	per scroll, so a fixed small budget silently under-collects.
57	- Scrolling that actually works: `document.body.scrollHeight` is 0 on
58	flex layouts (YouTube) — scroll `document.scrollingElement` instead, and
59	also iterate inner overflow panels; comments often live in an overlay
60	that window scrolling never touches.
61	- SSRF guard: HTTPS-only, no IP literals, private ranges and IMDS
62	blocked for the hostname and every resolved IP — re-validated inside the
63	`navigate` tool on every call, so the model can't pivot mid-session.
64
65	See [references/session-and-extraction.md](references/session-and-extraction.md)
66	— includes IAM specifics (the `browser/` vs `browser-custom/` ARN pitfall,
67	`InvokeModelWithResponseStream`), the asyncio × sync-Playwright worker-thread
68	requirement, and the error-code taxonomy.
69
70	### 2. Login state: Browser Profiles + live-view login
71
72	Users log in once inside the cloud browser via a DCV live-view page; the
73	session's cookies/tokens are saved to an AgentCore Browser Profile and reused
74	by every later scrape (`profile_configuration={"profileIdentifier": ...}`).
75	Credentials never appear in the conversation or your database.
76
77	Flow: mint a one-time URL token → `start_browser_session` with the profile →
78	presign the `live-view` stream endpoint (SigV4 query auth) → embed the DCV
79	Web Client → on completion `save_browser_session_profile` then
80	`stop_browser_session` (that order — save requires a live session).
81
82	The subtle part is

People who liked this also viewed

6 picks

Type