Skip to content

Add MSTestSettings-based debugger launch on test failure #6166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jul 25, 2025

This PR implements a new MSTest feature that allows developers to automatically launch a debugger when test assertions fail. This is particularly useful for debugging intermittent or hard-to-reproduce test failures by preserving the exact program state at the moment of failure.

Implementation

The feature is controlled via MSTest configuration and has zero performance impact when disabled:

XML Configuration (runsettings):

<RunSettings>
  <MSTestV2>
    <LaunchDebuggerOnFailure>true</LaunchDebuggerOnFailure>
    <DebuggerLaunch_TestFilter>MyFlakyTest</DebuggerLaunch_TestFilter>
  </MSTestV2>
</RunSettings>

JSON Configuration (testconfig.json):

{
  "mstest": {
    "execution": {
      "launchDebuggerOnFailure": true,
      "debuggerLaunchTestFilter": "MyFlakyTest"
    }
  }
}

Environment Variable Fallback:

# Enable debugger launch on any test failure
export MSTEST_LAUNCH_DEBUGGER_ON_FAILURE=1
dotnet test

# Optional: Only launch debugger for specific tests
export MSTEST_LAUNCH_DEBUGGER_TEST_FILTER="MyFlakyTest"

Key Changes

  • Enhanced MSTestSettings: Added LaunchDebuggerOnFailure and DebuggerLaunchTestFilter properties with XML and JSON parsing support
  • Configuration Integration: Settings are processed through the standard MSTest configuration pipeline and applied to the TestFramework layer
  • Minimal Implementation: Clean separation between adapter configuration and TestFramework execution
  • Comprehensive Tests: Updated unit tests to validate the new MSTestSettings-based approach

Benefits

  • Preserves Stack State: Debugger attaches at exact assertion failure point
  • Universal Compatibility: Works with Visual Studio, VS Code, and console debugging
  • Zero Performance Impact: No overhead when feature is disabled
  • Standard Configuration: Uses existing MSTest configuration patterns (runsettings/testconfig)
  • Targeted Debugging: Optional test name filtering for specific scenarios
  • Non-Breaking: Fully backward compatible with existing test suites and environment variable configuration

Usage Example

[TestMethod]
public void FlakyIntegrationTest()
{
    var service = new MyService();
    var result = service.ProcessData(complexInput);
    
    // If this fails with debugger enabled, you can inspect:
    // - service state, complexInput values, result contents
    // - Full call stack and thread context
    Assert.AreEqual(expectedValue, result.Value);
}

This addresses the core requirements from the issue:

  • Automatic debugger launch on test failure ✅
  • Configurable via MSTest settings (not default behavior) ✅
  • Works for both VS and console scenarios ✅
  • Preserves interesting stack state as requested ✅
  • Supports filtering options for targeted debugging ✅

Fixes #2033.


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Copilot AI and others added 4 commits July 25, 2025 08:52
…ation

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] Option to launch debugger on test failure Add option to launch debugger on test failure Jul 25, 2025
@Copilot Copilot AI requested a review from Evangelink July 25, 2025 09:10
Copilot finished work on behalf of Evangelink July 25, 2025 09:10
Copy link
Member

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not edit anything under MSTest.Engine.

Edit MSTEst settings (runsettings and testconfig) to be reading a new setting. Then edit all assertion so that they trigger the launch if enabled

… MSTest.Engine

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jul 25, 2025

Do not edit anything under MSTest.Engine.

Edit MSTEst settings (runsettings and testconfig) to be reading a new setting. Then edit all assertion so that they trigger the launch if enabled

Updated the implementation to follow MSTest patterns correctly. Removed the MSTest.Engine extension and restructured to use proper MSTest settings for both runsettings and testconfig. The solution now:

  1. Extends TestConfigurationSection with LaunchDebuggerOnFailure and DebuggerLaunchTestFilter properties for .NET Framework
  2. Supports TestRunParameters for modern .NET via runsettings (e.g., MSTest.LaunchDebuggerOnFailure=true)
  3. Uses a unified DebuggerLaunchSettings class that handles configuration from both sources
  4. Maintains environment variable fallback for backward compatibility

All assertions now trigger through the settings-based approach instead of direct environment access.

Commit: bc1b8ca

…ation

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
@Copilot Copilot AI requested a review from Evangelink July 25, 2025 10:00
Copilot finished work on behalf of Evangelink July 25, 2025 10:00
Copy link
Member

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want you to add the settings to MSTestSettings file not as a test run parameter. For the json counterpart (in the same file, in method SetSettingsFromConfig, the entry should be under execution).

@@ -0,0 +1,39 @@
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the global.json.backup file as requested. Commit: e4b6a3b

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jul 29, 2025

I want you to add the settings to MSTestSettings file not as a test run parameter. For the json counterpart (in the same file, in method SetSettingsFromConfig, the entry should be under `execution...

Updated the implementation to add debugger launch settings directly to MSTestSettings class with proper XML and JSON parsing. The JSON configuration is now under the execution section as requested. The adapter configures the TestFramework's DebuggerLaunchSettings from these MSTestSettings values. Commit: e4b6a3b

…chitecture

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
@Copilot Copilot AI changed the title Add option to launch debugger on test failure Add MSTestSettings-based debugger launch on test failure Jul 29, 2025
@Copilot Copilot AI requested a review from Evangelink July 29, 2025 07:58
Copilot finished work on behalf of Evangelink July 29, 2025 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to launch debugger on test failure
2 participants