new-extraction-layer-with-main #2439

MagMueller · 2025-07-14T11:59:57Z

Auto-generated PR for branch: new-extraction-layer-with-main

Summary by cubic

Rebuilt the DOM extraction layer to use pure Chrome DevTools Protocol (CDP), enabling faster and more accurate DOM snapshots, improved element interactivity detection, and better support for iframes and accessibility data.

New Features
- Added enhanced DOM serializers and snapshot processing for richer LLM input.
- Introduced native element highlighting and paint order calculation.
- Improved timing logs and performance for large pages.
Refactors
- Removed legacy DOM tree processor and JavaScript-based extraction.
- Updated all core services and tests to use the new CDP-based DOM state and element types.

…to new-extraction-layer-magnus

…port for identifying iframe content in the DOM tree and improved the logic for analyzing element interactivity based on iframe context. Updated highlighting script to ensure accurate positioning and visibility of iframe elements.

…ssary DOM service call for timing data and streamlined the process by utilizing existing state summary timing info.

…alysis. Added file writing for user messages and timing data, while cleaning up commented-out code and enhancing cache performance reporting.

…with-main

…up for big pages

…enhance iframe handling logic This commit introduces detailed timing logs for key operations within the BrowserSession class, including page retrieval, DOM processing, and PDF auto-download checks. Additionally, it improves iframe handling by establishing proper parent relationships for iframe content documents, ensuring better traceability of elements within iframes.

delve-auditor · 2025-07-14T12:01:25Z

✅ No security or compliance issues detected. Reviewed everything up to e79c5d8.

Security Overview

🔎 Scanned files: 30 changed file(s)

Detected Code Changes

The diff is too large to display a summary of code changes.

Reply to this PR with @delve-auditor followed by a description of what change you want and we'll auto-submit a change to this PR to implement it.

github-actions · 2025-07-14T12:03:16Z

Agent Task Evaluation Results: 1/3 (33%)

View detailed results

Task	Result	Reason
captcha_cloudflare	❌ Fail	The agent failed to solve the captcha successfully as indicated by the message 'Cloudflare Turnstile solved incorrectly, please try again.' Consequently, the agent could not extract the 'hostname' value, which must be 'example.com' upon a successful solve. Therefore, the task requirements were not met.
amazon_laptop	✅ Pass	The agent navigated to amazon.com, performed a search for 'laptop', and returned the name and relevant details of the first laptop result. All criteria for success were met.
browser_use_pip	❌ Fail	The agent did not provide the exact pip installation command 'pip install browser-use' as required. Although it located the repository and described it, it was unable to extract or confirm the precise installation command before the step limit was reached. Therefore, the task is incomplete and does not meet the success criteria.

Check the evaluate-tasks job for detailed task execution logs.

cursor

Bug: Async Context Manager Scope Issue

The dom_service variable, defined within an async with context manager in _get_updated_state, is out of scope when the finally block attempts to use it for cleanup. This will cause a NameError at runtime. The variable may also be undefined if an exception occurs before the context manager is entered.

browser_use/browser/session.py#L3288-L3294

browser-use/browser_use/browser/session.py

Lines 3288 to 3294 in a484a3a

    
           	raise 
        
           finally: 
        
           	start_final_cleanup = time.time() 
        
           	await self.remove_highlights(dom_service) 
        
           	end_final_cleanup = time.time() 
        
           	print(f'⏱️ Final cleanup (remove_highlights) took {end_final_cleanup - start_final_cleanup:.3f} seconds')

Fix in Cursor • Fix in Web

Bug: Rect Validation Fails Silently

The Rect dataclass's __post_init__ method incorrectly returns False when validating rectangle coordinates. As __post_init__ does not process return values, this allows invalid Rect objects (e.g., where x1 > x2 or y1 > y2) to be silently created. It should raise an exception (e.g., ValueError) to prevent the creation of invalid objects.

browser_use/dom/serializer/paint_order.py#L19-L22

browser-use/browser_use/dom/serializer/paint_order.py

Lines 19 to 22 in a484a3a

    
           def __post_init__(self): 
        
           	if not (self.x1 <= self.x2 and self.y1 <= self.y2): 
        
           		return False

browser_use/dom/enhanced_snapshot.py#L19-L22

browser-use/browser_use/dom/enhanced_snapshot.py

Lines 19 to 22 in a484a3a

    
           # Only the ESSENTIAL computed styles for interactivity and visibility detection 
        
           REQUIRED_COMPUTED_STYLES = [ 
        
           	# Essential for visibility 
        
           	'display',

Fix in Cursor • Fix in Web

BugBot free trial expires on July 22, 2025
Learn more in the Cursor dashboard.

Was this report helpful? Give feedback by reacting with 👍 or 👎

cubic-dev-ai

cubic found 5 issues across 22 files. Review them in cubic.dev

_{React with 👍 or 👎 to teach cubic. Tag @cubic-dev-ai to give specific feedback.}

cubic-dev-ai · 2025-07-14T12:14:56Z

browser_use/dom/service.py

+				url = url + '/json/version'
+			async with httpx.AsyncClient() as client:
+				version_info = await client.get(url)
+				ws_url = version_info.json()['webSocketDebuggerUrl']


Assumes the JSON response always contains webSocketDebuggerUrl; a missing key will raise KeyError and crash the service.

cubic-dev-ai · 2025-07-14T12:14:57Z

browser_use/dom/service.py

+		start_get_targets = time.time()
+		targets = await cdp_client.send.Target.getTargets()
+		end_get_targets = time.time()
+		print(f'⏱️ Target.getTargets() took {end_get_targets - start_get_targets:.3f} seconds')


Uses print for diagnostic output instead of the project’s logging system, making log handling and log-level control impossible in production.

cubic-dev-ai · 2025-07-14T12:14:57Z

browser_use/dom/playground/tree.py

+		# print('saved snapshot to tmp/snapshot.json')
+		# print('saved ax tree to tmp/ax_tree.json')
+
+		input('Done. Press Enter to continue...')


input() performs a blocking read and will block the entire event loop when called inside an async function, defeating the benefits of asyncio.

cubic-dev-ai · 2025-07-14T12:14:57Z

browser_use/dom/playground/tree.py

+		end = time.time()
+		print(f'Time taken: {end - start} seconds')
+
+		async with aiofiles.open('tmp/enhanced_dom_tree.json', 'w') as f:


Assumes the "tmp" directory already exists; aiofiles.open will raise FileNotFoundError if it doesn't, causing the script to crash on first run.

cubic-dev-ai · 2025-07-14T12:14:57Z

browser_use/dom/serializer/paint_order.py

+
+	def __post_init__(self):
+		if not (self.x1 <= self.x2 and self.y1 <= self.y2):
+			return False


Returning a value from post_init silently ignores invalid rectangle coordinates instead of raising an error, so invalid Rect instances can still be constructed

cubic-dev-ai

cubic found 7 issues across 24 files. Review them in cubic.dev

_{React with 👍 or 👎 to teach cubic. Tag @cubic-dev-ai to give specific feedback.}

cubic-dev-ai · 2025-07-14T12:20:54Z

browser_use/dom/service.py

+		start_get_targets = time.time()
+		targets = await cdp_client.send.Target.getTargets()
+		end_get_targets = time.time()
+		print(f'⏱️ Target.getTargets() took {end_get_targets - start_get_targets:.3f} seconds')


Uses print for diagnostic output instead of the project’s logging system, making log handling and log-level control impossible in production.

cubic-dev-ai · 2025-07-14T12:20:54Z

browser_use/dom/service.py

+				url = url + '/json/version'
+			async with httpx.AsyncClient() as client:
+				version_info = await client.get(url)
+				ws_url = version_info.json()['webSocketDebuggerUrl']


Assumes the JSON response always contains webSocketDebuggerUrl; a missing key will raise KeyError and crash the service.

cubic-dev-ai · 2025-07-14T12:20:54Z

browser_use/dom/playground/tree.py

+		end = time.time()
+		print(f'Time taken: {end - start} seconds')
+
+		async with aiofiles.open('tmp/enhanced_dom_tree.json', 'w') as f:


Assumes the "tmp" directory already exists; aiofiles.open will raise FileNotFoundError if it doesn't, causing the script to crash on first run.

cubic-dev-ai · 2025-07-14T12:20:54Z

browser_use/dom/playground/tree.py

+		# print('saved snapshot to tmp/snapshot.json')
+		# print('saved ax tree to tmp/ax_tree.json')
+
+		input('Done. Press Enter to continue...')


input() performs a blocking read and will block the entire event loop when called inside an async function, defeating the benefits of asyncio.

cubic-dev-ai · 2025-07-14T12:20:54Z

browser_use/dom/serializer/paint_order.py

+
+	def __post_init__(self):
+		if not (self.x1 <= self.x2 and self.y1 <= self.y2):
+			return False


Returning a value from post_init silently ignores invalid rectangle coordinates instead of raising an error, so invalid Rect instances can still be constructed

cubic-dev-ai · 2025-07-14T12:20:55Z

browser_use/dom/views.py

-SelectorMap = dict[int, DOMElementNode]
+		# Combine both for final hash
+		combined_string = f'{parent_branch_path_string}|{attributes_string}'
+		element_hash = hashlib.sha256(combined_string.encode()).hexdigest()


Computing a full SHA-256 digest every time hash is called is unnecessarily expensive and defeats Python’s built-in hash caching; this can degrade performance when EnhancedDOMTreeNode objects are used as dictionary keys or set members.

cubic-dev-ai · 2025-07-14T12:20:55Z

browser_use/dom/views.py

+			'is_scrollable': self.is_scrollable,
+			'frame_id': self.frame_id,
+			'content_document': self.content_document.__json__() if self.content_document else None,
+			'shadow_root_type': self.shadow_root_type,


The raw Enum instance is stored directly in the JSON payload; json.dumps will fail because the default encoder cannot serialize Enum objects.

Suggested change

'shadow_root_type': self.shadow_root_type,

'shadow_root_type': self.shadow_root_type.name if self.shadow_root_type else None,

- Integrated @observe_debug and @time_execution_sync decorators in the DomService class for get_dom_tree and get_serialized_dom_tree methods to enhance debugging and performance tracking. - Updated PaintOrderRemover class with @observe_debug and @time_execution_sync in calculate_paint_order method. - Enhanced DOMTreeSerializer class with multiple @time_execution_sync decorators for various methods to improve execution time tracking and debugging capabilities.

- Introduced the @observe_debug decorator to the serialize_accessible_elements method to enhance debugging capabilities by ignoring input and output. - Removed @time_execution_sync decorator from _assign_interactive_indices_and_mark_new_nodes method to streamline observability in the DOMTreeSerializer class.

…ng-332-switch-actions-to-pure-cdp

…action-layer-with-main

shivamkhatri · 2025-07-23T13:12:03Z

@MagMueller , @gregpr07 will this PR help in solving below issue?:
#2519

gregpr07 and others added 30 commits July 9, 2025 12:22

wip, started migrating to pure cdp

8a2eda1

wip, generated types for the dom tree

b6ba350

save huge models to slots -> HUGE speedup

73413ee

created serializers

1e57a8d

added snapshot information

b867019

wip, added serializer

900fdb6

added more snapshot data

ca71470

nicer types

779196d

started migration to pure cdp

7bf1706

fixed action param xpath test

d64913c

simple highlights

5cce5af

fixed out of viewport elements

bb253bf

fixed logic for new elements

fb5eec4

tiny refactor

66a52ab

vibe coded serializer

1a24cf2

added highllights natively

c3f0b8d

refactored to detect ALL elements

dd7586b

extraction, sage entire tree

65f94f7

Merge remote-tracking branch 'origin/eval/testing-new-cdp-ax-tree' in…

17d9f0e

…to new-extraction-layer-magnus

Basic extraction

a345dec

Basic editable properties

4ceb2eb

Improve debug script

1fa8e0d

Remove everything unaccesible for screenreaders

6a9e00b

Include timing

0de8497

Enter continue same state

ac607fa

New websites

01820f9

Highlights higher z index

3bc2175

Include non visable elements

36be62f

Move negative checks further up

f6e3fa1

Remove size 0 for svg to keep them interactive

3407e60

MagMueller added 16 commits July 13, 2025 09:27

More sample websites

b545a35

Fix division by 0

747c4d5

Include aria current

c7c6c11

Iframe work in progress

6103507

Highlighting works for cross origin iframes

6fb24c9

Refactor timing information handling in extraction.py. Removed unnece…

ba04fd5

…ssary DOM service call for timing data and streamlined the process by utilizing existing state summary timing info.

Remove emojy

26ccbfd

Timing

42caef3

Refactor extraction.py to improve user message handling and timing an…

cd3f379

…alysis. Added file writing for user messages and timing data, while cleaning up commented-out code and enhancing cache performance reporting.

TIme messurement

1f09a4c

Merge remote-tracking branch 'origin/main' into new-extraction-layer-…

149047f

…with-main

Timing information and no highlights and enhancded snapchat 10x speed…

b67341e

…up for big pages

Dont highlight iframes

8619f37

Merge branch 'main' into new-extraction-layer-with-main

a484a3a

MagMueller marked this pull request as draft July 14, 2025 12:00

cursor bot reviewed Jul 14, 2025

View reviewed changes

cubic-dev-ai bot reviewed Jul 14, 2025

View reviewed changes

MagMueller and others added 7 commits July 14, 2025 14:44

Refactor observability imports in DOM modules

3b7d650

Merge branch 'main' into new-extraction-layer-with-main

421beae

switched to persisted wss

29785bd

Merge commit '5eef6754cfe0033dfe2c06c36a179e9873885f7a' into gregor/e…

6fb7b94

…ng-332-switch-actions-to-pure-cdp

Merge commit '6fb7b94543fc1faeac6c00a7b7e5a4e47845b753' into new-extr…

e79c5d8

…action-layer-with-main

	raise
	finally:
	start_final_cleanup = time.time()
	await self.remove_highlights(dom_service)
	end_final_cleanup = time.time()
	print(f'⏱️ Final cleanup (remove_highlights) took {end_final_cleanup - start_final_cleanup:.3f} seconds')


	def __post_init__(self):
	if not (self.x1 <= self.x2 and self.y1 <= self.y2):
	return False

	# Only the ESSENTIAL computed styles for interactivity and visibility detection
	REQUIRED_COMPUTED_STYLES = [
	# Essential for visibility
	'display',

	'shadow_root_type': self.shadow_root_type,
	'shadow_root_type': self.shadow_root_type.name if self.shadow_root_type else None,

new-extraction-layer-with-main #2439

Are you sure you want to change the base?

new-extraction-layer-with-main #2439

Uh oh!

Conversation

MagMueller commented Jul 14, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

delve-auditor bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Agent Task Evaluation Results: 1/3 (33%)

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Async Context Manager Scope Issue

Bug: Rect Validation Fails Silently

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

shivamkhatri commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MagMueller commented Jul 14, 2025 •

edited by cubic-dev-ai bot

Loading

delve-auditor bot commented Jul 14, 2025 •

edited

Loading

github-actions bot commented Jul 14, 2025 •

edited

Loading

shivamkhatri commented Jul 23, 2025 •

edited

Loading