extraction-magnus-revert-with-main-with-extension #2464

MagMueller · 2025-07-16T19:47:13Z

Auto-generated PR for branch: extraction-magnus-revert-with-main-with-extension

Summary by cubic

Rebuilt the DOM extraction and serialization system to use Chrome DevTools Protocol (CDP) for more accurate detection of interactive elements, improved accessibility data, and better performance.

New Features
- Added enhanced DOM tree and snapshot extraction using CDP, including visibility, interactivity, and accessibility properties.
- Introduced new serializers for LLM-friendly DOM representations and selector maps.
- Enabled default browser extensions for ad blocking and cookie handling.
Refactors
- Removed legacy DOM processing modules and replaced them with new CDP-based implementations.
- Updated tests and examples to use the new DOM state and element types.

…to new-extraction-layer-magnus

…idation

…ert-with-main

…proved handling of complex pages.

delve-auditor · 2025-07-16T19:49:20Z

✅ No security or compliance issues detected. Reviewed everything up to 99f285e.

Security Overview

🔎 Scanned files: 31 changed file(s)

Detected Code Changes

The diff is too large to display a summary of code changes.

Reply to this PR with @delve-auditor followed by a description of what change you want and we'll auto-submit a change to this PR to implement it.

cubic-dev-ai

cubic found 8 issues across 24 files. Review them in cubic.dev

_{React with 👍 or 👎 to teach cubic. Tag @cubic-dev-ai to give specific feedback.}

cubic-dev-ai · 2025-07-16T19:50:13Z

browser_use/browser/profile.py

+		import urllib.request
+
+		try:
+			with urllib.request.urlopen(url) as response:


urllib.request.urlopen is invoked without a timeout, so a slow or unresponsive server can block the entire process indefinitely. Provide an explicit timeout to make the download operation fail fast.

Suggested change

with urllib.request.urlopen(url) as response:

with urllib.request.urlopen(url, timeout=30) as response:

cubic-dev-ai · 2025-07-16T19:50:13Z

browser_use/dom/views.py


+		# Get attributes hash
+		attributes_string = ''.join(f'{key}={value}' for key, value in self.attributes.items())


Hash computation depends on the iteration order of a dict, but dict order is not considered in equality. Two nodes that are considered equal (same key-value pairs in different insertion order) can generate different hashes, breaking the requirement that equal objects have identical hash values.

Suggested change

attributes_string = ''.join(f'{key}={value}' for key, value in self.attributes.items())

attributes_string = ''.join(f'{key}={self.attributes[key]}' for key in sorted(self.attributes))

cubic-dev-ai · 2025-07-16T19:50:14Z

browser_use/dom/enhanced_snapshot.py

+
+				# Extract stacking contexts if available
+				if layout_idx < len(layout.get('stackingContexts', [])):
+					stacking_contexts = layout.get('stackingContexts', {}).get('index', [])[layout_idx]


stackingContexts is a RareBooleanData object; directly indexing into get('index', [])[layout_idx] risks IndexError and ignores the helper used elsewhere for rare boolean parsing.

cubic-dev-ai · 2025-07-16T19:50:14Z

browser_use/dom/enhanced_snapshot.py

+]
+
+
+def _parse_rare_boolean_data(rare_data: RareBooleanData, index: int) -> bool | None:


The implementation always returns a boolean, never None, so the return type annotation is misleading and could confuse type-checkers or readers.

Suggested change

def _parse_rare_boolean_data(rare_data: RareBooleanData, index: int) -> bool | None:

def _parse_rare_boolean_data(rare_data: RareBooleanData, index: int) -> bool:

cubic-dev-ai · 2025-07-16T19:50:14Z

browser_use/dom/playground/extraction.py

+	# Show startup info
+	print('\n🌐 BROWSER-USE DOM EXTRACTION TESTER')
+	print(f'📊 {len(websites)} websites total: {len(sample_websites)} standard + {len(difficult_websites)} complex')
+	print('🔧 Controls: Type 1-15 to jump | Enter to re-run | "n" next | "q" quit')


The control hint hard-codes the range 1-15 even though the actual website list length is dynamic (currently 17). This can mislead users and cause unnecessary input errors.

Suggested change

print('🔧 Controls: Type 1-15 to jump | Enter to re-run | "n" next | "q" quit')

print(f'🔧 Controls: Type 1-{len(websites)} to jump | Enter to re-run | "n" next | "q" quit')

cubic-dev-ai · 2025-07-16T19:50:14Z

browser_use/dom/service.py


-	def __init__(self, page: 'Page', logger: logging.Logger | None = None):
+	Either browser or page must be provided.


The docstring states that either browser or page must be provided, but the constructor requires both parameters. This mismatch can confuse users and maintainers.

cubic-dev-ai · 2025-07-16T19:50:14Z

browser_use/dom/service.py

+				# cache the session id for this playwright page
+				# self.playwright_page_to_session_id_store[page_guid] = target['targetId']
+
+				session = await cdp_client.send.Target.attachToTarget(params={'targetId': target['targetId'], 'flatten': True})


A new Target.attachToTarget is issued every time _get_current_page_session_id runs, but the returned session is never detached or reused. This can leak DevTools sessions and exhaust browser resources over time.

cubic-dev-ai · 2025-07-16T19:50:14Z

browser_use/dom/service.py

-				interactive_count,
-				total_nodes,
-				# processed_nodes,
+			print(f'⚠️  Viewport size detection failed: {e}')


Direct print statements inside library code bypass the project's logging system, making it harder to control verbosity and aggregate logs. Use the standard logger instead.

github-actions · 2025-07-16T19:50:25Z

Agent Task Evaluation Results: 2/3 (67%)

View detailed results

Task	Result	Reason
captcha_cloudflare	❌ Fail	The agent reported that it attempted to solve the Cloudflare Turnstile captcha as instructed, but the captcha was not solved successfully. Consequently, the 'hostname' value under the success message was not found or extracted. Since the agent did not solve the captcha and did not extract the hostname 'example.com', the task was not completed successfully.
amazon_laptop	✅ Pass	The agent successfully navigated to amazon.com, performed a search for 'laptop', and returned the name and details of the first laptop result. Therefore, it fulfilled all the criteria specified for the task.
browser_use_pip	✅ Pass	The agent successfully provided the pip installation command 'pip install browser-use' as requested, meeting the success criteria.

Check the evaluate-tasks job for detailed task execution logs.

…nsion

cursor

Bug: Click Logic Fails for Larger Elements

The fallback coordinate clicking logic incorrectly uses the top-left corner (element_node.snapshot_node.bounds.x/y) instead of the element's center. This change from the previous element_node.viewport_coordinates.center.x/y causes clicks to miss their intended targets, especially for larger elements. The correct coordinates should be calculated as (bounds.x + bounds.width/2, bounds.y + bounds.height/2).

browser_use/browser/session.py#L2203-L2213

browser-use/browser_use/browser/session.py

Lines 2203 to 2213 in 99f285e

    
           # Final fallback - try clicking by coordinates if available 
        
           if element_node.snapshot_node and element_node.snapshot_node.bounds: 
        
           	try: 
        
           		# TODO: instead of using the cached center, we should use the actual center of the element (easy, just get it by nodeBackendId) 
        
           		self.logger.warning( 
        
           			f'⚠️ Element click failed, falling back to coordinate click at ({element_node.snapshot_node.bounds.x}, {element_node.snapshot_node.bounds.y})' 
        
           		) 
        
           		await page.mouse.click( 
        
           			element_node.snapshot_node.bounds.x, 
        
           			element_node.snapshot_node.bounds.y, 
        
           		)

Fix in Cursor • Fix in Web

Bug: Async Context Manager Scope Issue

The finally block attempts to access the dom_service variable, which is defined within an async with context manager. This causes a NameError because dom_service is out of scope when the finally block executes.

browser_use/browser/session.py#L3323-L3325

browser-use/browser_use/browser/session.py

Lines 3323 to 3325 in 99f285e

    
           finally: 
        
           	await self.remove_highlights(dom_service)

Fix in Cursor • Fix in Web

Was this report helpful? Give feedback by reacting with 👍 or 👎

gregpr07 and others added 30 commits July 9, 2025 12:22

wip, started migrating to pure cdp

8a2eda1

wip, generated types for the dom tree

b6ba350

save huge models to slots -> HUGE speedup

73413ee

created serializers

1e57a8d

added snapshot information

b867019

wip, added serializer

900fdb6

added more snapshot data

ca71470

nicer types

779196d

started migration to pure cdp

7bf1706

fixed action param xpath test

d64913c

simple highlights

5cce5af

fixed out of viewport elements

bb253bf

fixed logic for new elements

fb5eec4

tiny refactor

66a52ab

vibe coded serializer

1a24cf2

added highllights natively

c3f0b8d

refactored to detect ALL elements

dd7586b

extraction, sage entire tree

65f94f7

Merge remote-tracking branch 'origin/eval/testing-new-cdp-ax-tree' in…

17d9f0e

…to new-extraction-layer-magnus

Basic extraction

a345dec

Basic editable properties

4ceb2eb

Improve debug script

1fa8e0d

Remove everything unaccesible for screenreaders

6a9e00b

Include timing

0de8497

Enter continue same state

ac607fa

New websites

01820f9

Highlights higher z index

3bc2175

Include non visable elements

36be62f

Move negative checks further up

f6e3fa1

Remove size 0 for svg to keep them interactive

3407e60

MagMueller added 9 commits July 12, 2025 21:29

Highlights scroll absolute

9d355b6

Cache for clickable

c0ca8bb

Should be visable

bdb6dcb

Enhance visibility check with robust edge case handling and input val…

44b0e38

…idation

Dont remove highlights in multiact

a72e0ac

Merge remote-tracking branch 'origin/main' into extraction-magnus-rev…

061e24b

…ert-with-main

Increase DOM processing timeout from 45 seconds to 120 seconds for im…

e05c027

…proved handling of complex pages.

Load addblocker and cookie banner

3f825fb

Install by default

8b8c08b

cubic-dev-ai bot reviewed Jul 16, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into extraction-magnus-revert-with-main-with-exte…

99f285e

…nsion

MagMueller marked this pull request as draft July 16, 2025 19:51

cursor bot reviewed Jul 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

extraction-magnus-revert-with-main-with-extension #2464

extraction-magnus-revert-with-main-with-extension #2464

MagMueller commented Jul 16, 2025 •

edited by cubic-dev-ai bot

Loading

Uh oh!

delve-auditor bot commented Jul 16, 2025 •

edited

Loading

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Uh oh!

github-actions bot commented Jul 16, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

	with urllib.request.urlopen(url) as response:
	with urllib.request.urlopen(url, timeout=30) as response:


		# Get attributes hash
		attributes_string = ''.join(f'{key}={value}' for key, value in self.attributes.items())

	attributes_string = ''.join(f'{key}={value}' for key, value in self.attributes.items())
	attributes_string = ''.join(f'{key}={self.attributes[key]}' for key in sorted(self.attributes))

		]


		def _parse_rare_boolean_data(rare_data: RareBooleanData, index: int) -> bool \| None:

	print('🔧 Controls: Type 1-15 to jump \| Enter to re-run \| "n" next \| "q" quit')
	print(f'🔧 Controls: Type 1-{len(websites)} to jump \| Enter to re-run \| "n" next \| "q" quit')


		def __init__(self, page: 'Page', logger: logging.Logger \| None = None):
		Either browser or page must be provided.

	# Final fallback - try clicking by coordinates if available
	if element_node.snapshot_node and element_node.snapshot_node.bounds:
	try:
	# TODO: instead of using the cached center, we should use the actual center of the element (easy, just get it by nodeBackendId)
	self.logger.warning(
	f'⚠️ Element click failed, falling back to coordinate click at ({element_node.snapshot_node.bounds.x}, {element_node.snapshot_node.bounds.y})'
	)
	await page.mouse.click(
	element_node.snapshot_node.bounds.x,
	element_node.snapshot_node.bounds.y,
	)

extraction-magnus-revert-with-main-with-extension #2464

Are you sure you want to change the base?

extraction-magnus-revert-with-main-with-extension #2464

Conversation

MagMueller commented Jul 16, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

delve-auditor bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Agent Task Evaluation Results: 2/3 (67%)

Uh oh!

This comment was marked as outdated.

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Click Logic Fails for Larger Elements

Bug: Async Context Manager Scope Issue

Uh oh!

Uh oh!

MagMueller commented Jul 16, 2025 •

edited by cubic-dev-ai bot

Loading

delve-auditor bot commented Jul 16, 2025 •

edited

Loading

github-actions bot commented Jul 16, 2025 •

edited

Loading