MTM

Table of Contents
Test
Overview
About exploratory and manual testing
About load testing
Quickstarts
Manual testing
Create a test plan
Create test cases
Run manual tests
Test & Feedback extension
Install the extension
Test in Connected mode
Test in Standalone mode
Load testing
Load test with Visual Studio
Load test with VSTS
Load test with Azure portal
View and compare results
Tutorials
Test from the Kanban board
Load test before release
How-to Guides
Manual testing
Track test status
Delete test artifacts
Perform user acceptance testing
Test different configurations
Repeat a test with different data
Collect screenshots and video
Collect diagnostic data
Manage test results
Associate tests with test cases
Run tests from the Test hub
Associate results with requirements
Explore work items
Add to existing bugs
Get insights from tests
Request stakeholder feedback
Provide stakeholder feedback
Get voluntary stakeholder feedback
Track stakeholder feedback requests
Load testing
Add app performance data
Run Apache JMeter load tests
Record and replay tests
Test private and intranet apps
Test using your own servers
Install certificates and software
Microsoft Test Manager
Guidance for MTM usage
Connect to a team project
Plan tests with MTM
Run manual tests with MTM
Run automated tests with MTM
Record and play back tests
Explore your app
Share steps between test cases
Copy test suites and cases
Collect more diagnostic data
Test Microsoft Store apps
Reference
Manual testing FAQs
Load testing FAQs
Unable to connect
Default permissions (Security)
Resources
Get Stakeholder feedback
Using Microsoft Test Manager
Create custom code & plug-ins
Edit load tests
Analyze load test results
REST API for test management
Web Performance Test API
ALM Blog posts
Manual test posts
Test management posts
Test & Feedback posts
Load test posts
Plug-ins for Visual Studio
Continuous testing
Unit testing
Exploratory & Manual Testing
6/1/2018 • 1 min to read • Edit Online
VS 2017 | VS 2015 | VSTS | TFS 2018 | TFS 2017 | TFS 2015

Visual Studio Team Services (VSTS ) and Team Foundation Server (TFS ) provide rich and powerful tools
everyone in the team can use to drive quality and collaboration throughout the development process.
See also: Load testing
5-Minute Quickstarts
Learn how to create tests plans and test cases, and run them using the VSTS or TFS web portal. Use the Test &
Feedback extension to explore and find bugs in your apps.
Create a test plan Create test cases Run manual tests
Install the extension Test in Connected mode Test in Standalone mode
Videos
https://channel9.msdn.com/Series/Visual-Studio-ALM- https://channel9.msdn.com/Series/Test-Tools-in-Visual-
Rangers-Demos/VS-Team-Services-Test-Case-Explorer- Studio/IntroducingTestFeedbackextension/player
v2/player
Step-by-Step Tutorials
Test from the Kanban board
How-to Guides
Track test status
Manage test results
Explore work items
Get insights from tests
Reference
FAQs for manual testing
Unable to connect
Resources
Get Stakeholder feedback
REST API for test management
Blog posts for testing
Test & Feedback posts
Using Microsoft Test Manager
Blog posts for test management
Load and performance testing
Continuous testing
Unit testing
Exploratory and manual testing scenarios and
capabilities

Quality is a vital aspect of software systems, and manual testing and exploratory testing continue to be an
important techniques for maximizing this. In today's software development processes, everybody in the team owns
quality - including developers, managers, product owners, user experience advocates, and more.
VSTS and TFS provide rich and powerful tools everyone in the team can use to drive quality and collaboration
throughout the development process. The easy-to-use, browser-based test management solution provides all the
capabilities required for planned manual testing, user acceptance testing, exploratory testing, and gathering
feedback from stakeholders.
Planned manual testing. Manual testing by organizing tests into test plans and test suites by designated
testers and test leads.
User acceptance testing. Testing carried out by designated user acceptance testers to verify the value
delivered meets customer requirements, while reusing the test artifacts created by engineering teams.
Exploratory testing. Testing carried out by development teams, including developers, testers, UX teams,
product owners and more, by exploring the software systems without using test plans or test suites.
Stakeholder feedback. Testing carried out by stakeholders outside the development team, such as users
from marketing and sales divisions.
Holistic approach to manual testing, types of manual testing, and personas involved
You must install the Test Manager extension to use the advanced features of the Test hub.
Planned manual testing

Manual testing has evolved with the software development process into a more agile-based approach. VSTS and
TFS integrate manual testing into your agile processes; the team can begin manual testing right from their Kanban
boards in the Work hub. Teams that need more advanced capabilities can use the Test hub for all their test
management needs.
Manual testing from the Kanban board
Get started with manual testing easily using the Kanban board in the Work hub. Add, view, and interact with test
cases directly from the cards on the Kanban board, and then progressively monitor status directly from the card.
Developers and testers can use these rich capabilities to simplify maximizing quality within their teams. In VSTS,
you need just Basic access to use these features. See more at Add, run, and update inline tests.
Manual testing in the Test hub

The Test hub in VSTS and TFS provides a rich test management solution for teams that need advanced manual
testing capabilities. The Test hub includes all the capabilities required for the testing lifecycle - including test
planning, authoring, execution, and tracking. Get started using the advanced manual testing features with the Test
Manager extension.
Test planning
Create and manage test plans and test suites for your teams with ease. Create static suites, requirement-based
suites, or query-based suites. Export and share the test plans and test suites with your team. See more at Create
test plans
Test authoring
Create multiple test cases in one operation, or easily add existing test cases to a test suite. Assign single or multiple
testers to execute the tests. View test results and references to a test case across test suites. See more at Create test
cases.
Testing web applications
The Test hub provides a browser-based test runner to run tests for your web apps. Mark test steps and test
outcomes as pass or fail, and collect diagnostic data such as system information, image action logs, screen
recordings, and screen captures as you test. Bugs filed during the tests automatically include all the captured
diagnostic data to help your developers reproduce the issues. See more at Run tests for web apps.
Testing desktop apps

Test your desktop apps with Microsoft Test Runner client, which is part of Microsoft Test Manager. Use the Test
Runner client to collect all the basic diagnostic data such as system information, image action logs, screen
recordings, screen captures, and event logs as you test. In addition, use Microsoft Test Manager to collect advanced
diagnostic data such as code coverage, IntelliTrace traces, and test impact data. See more at Run tests for desktop
apps.
Test tracking
Quickly configure lightweight charts to track your manual test results using the chart types of your choice, and pin
the charts to your dashboard to easily analyze these results. Choose a retention policy to control how long your
manual testing results are retained. See more at Track test status.
User acceptance testing

User acceptance testing (UAT) is a key factor in software development that ensures the value requested by
customers is being delivered by the engineering team. VSTS and TFS include capabilities and tools to manage user
acceptance testing. Quickly create UAT plans and suites, and invite multiple testers to execute these tests using test
artifacts provided by the engineering team. Easily monitor UAT progress and results using lightweight charts. See
more at User acceptance testing.
Exploratory testing for everyone
Maximizing quality in modern software development processes is a shared responsibility between developers,
managers, product owners, user experience teams, and more. Collaborative testing processes and tools are the key
factors in driving quality in these scenarios.
The Test & Feedback extension is a simple browser-based extension you can use to test web apps anytime and
anywhere, and is simple enough for everyone in the team to use. It helps to improve productivity by allowing you
to spend more time finding issues, and less time filing them.
Using the extension is a simple, three step process:

Capture your findings quickly and easily using the tools in the extension. Capture notes, screenshots with
annotations, and screen recordings to describe your findings and highlight issues. Additionally, in the
background the extension automatically captures rich data such as user actions as an image action log, page
load data, and system information about the browser, operating system, memory, and more that can serve
as a starting point for debugging.
Create work items such as bugs, tasks, and test cases directly from the extension. The captured findings
automatically become a part of the work item. Users can file a bug to report an issue with the product, or
create a task that indicates a new work requirement. The extension can also be used to create test cases for
scenarios discovered during exploration.
Collaborate with your team by sharing your findings. Export your session report in Standalone mode, or
connect to VSTS or Team Foundation Server (2015 or later) for a fully integrated experience including
exploring user stories and backlog items, simplified tracking and triaging of bugs and tasks, and managing
feedback requests in one place.
As users perform exploratory testing, you can get insights from the sessions in the Test hub of VSTS or TFS. View
completed exploratory sessions and derive meaningful insights across all the sessions. Get end-to-end traceability
such as a breakdown of the work items created, the work items explored and not explored, session owners, and
more.
Stakeholder feedback
Seeking feedback from stakeholders outside the development team, such as marketing and sales teams, is vital to
develop good quality software. Using VSTS and TFS, developers can request feedback on their user stories and
features. Stakeholders can respond to feedback requests using the browser-based Test & Feedback extension - not
just to rate and send comments, but also by capturing rich diagnostic data and filing bugs and tasks directly. See
more at Request stakeholder feedback and Provide stakeholder feedback.
Key benefits
Test on any platform. With the Test hub in VSTS and TFS, you can use your browser to access all the
manual testing capabilities. The Test hub enables you to create and run manual tests through an easy-to-use,
web-based interface that can be accessed from all major browsers on any platform.
Rich Diagnostic data collection. Using the web-based Test Runner and Test Runner client you can collect
rich diagnostic data during your tests. This includes screenshots, an image action log, screen recordings,
code coverage, IntelliTrace traces, and test impact data for your apps under test. This data is automatically
included in all the bugs you create during test, making it easy for developers to reproduce the issues.
End to End Traceability. VSTS and TFS provide end-to-end traceability of your requirements, builds, tests
and bugs. Users can track their requirement quality from cards on the Kanban board. Bugs created while
testing are automatically linked to the requirements and builds being tested, which helps you track the
quality of the requirements or builds.
Extensible platform. You can combine the tools and technologies you already know with the development
tools that work best for you to integrate with and extend VSTS and TFS. Use the REST APIs and
contribution model available for the Test platform to create extensions that provide the experience you need
for your test management lifecycle.
Additional resources
Get started with manual testing
Advanced manual testing techniques
Get started with exploratory testing
Advanced exploratory testing techniques
Manual testing with Microsoft Test Manager
Guidance for MTM usage
Get stakeholder feedback with exploratory testing
See also Load and performance testing, Continuous testing, Unit testing.
Help and support

Report any problems on Developer Community, make suggestions on UserVoice, get advice on Stack Overflow,
and get support via our Support page.
Load testing scenarios and capabilities
Visual Studio 2017 | Visual Studio 2015 | VSTS | Previous version
Load test your app with hundreds of thousands of users using Visual Studio Team Services (VSTS ).
Comprehensive testing capabilities

Load test web sites, apps, and APIs.
Author tests using Visual Studio, Azure, and VSTS.
Quickly create load tests by specifying a website, referencing a JMeter test file, or recording and replaying
your actions.
Run tests or customize them using powerful tools in Visual Studio.
You can even use existing unit or functional tests to generate load.
Cloud scalable
Scale to hundreds of thousands of concurrent users, and generate hundreds of thousands of connections in
minutes.
Cloud-based load testing leveraging the power of Azure is like having a whole performance lab at your
fingertips.
Of course you can run your load test from on-premises agents too!
Generate load from multiple regions worldwide

Run tests from one of many global Azure datacenter locations to minimize latency and simulate users' real-
world conditions.
Deep analysis with rich diagnostics

Includes trace and exception logging.
View app performance with real-time charts and graphs.
Go even further with Application Insights, and correlate test results with server diagnostics.
Free allocation and flexible low-cost pricing

Pricing is per virtual user minute (VUM ) - a measure of how long your test is and how many users the test
simulates.
You get a generous allocation of virtual user minutes free each month. See the VSTS Pricing page.
More information
Get started with load testing
Run URL -based load tests with VSTS
Run Apache JMeter load tests with VSTS
Performance test your Azure web app under load
Pricing for VSTS features
If you prefer to run your tests in a local environment rather than in the cloud, see Load test with Visual Studio.
See also Manual and exploratory testing, Continuous testing, Unit testing.
Help and support

Create a test plan and test suite
VSTS | TFS 2018 | TFS 2017 | TFS 2015

Create test plans to track manual testing for sprints or milestones. That way, you can see when the testing for a
specific sprint or milestone is complete.
Test plans are used to group together test suites and individual test cases. This includes static test suites,
requirement-based suites, and query-based suites. You can add individual test cases to a test plan without
creating a test suite if you wish, but using a test suite provides a way to group test cases for separate testing
scenarios within a single test plan.
To use the features described in this topic you must have either a Visual Studio Enterprise subscription, or have
installed the Test Manager extension available from Visual Studio Marketplace.
Note: Stakeholders cannot create or manage test plans. You must have at least Basic access. See Default manual
testing permissions and access.
Create a test plan

1. If you haven't already, sign up for VSTS, create your project, and create your backlog.
2. In VSTS, open your project.
3. Go to the Test Plans tab of the Test hub. Create a test plan for your current sprint.
4. Name the test plan. Check the area path and iteration. Then choose *Create.
Add a test suite and select backlog items to test
1. Now add test suites for the backlog items that need manual tests. (These backlog items could be user
stories, requirements, or other work items based on the setup of your project.)
You use requirement-based suites to group your test cases together. That way, you can track the testing
status of a backlog item. Each test case that you add to a requirement-based test suite is automatically
linked to the backlog item.
2. Add a clause to filter by the iteration path for the sprint. Run the query to view the matching backlog items.
3. Select the backlog items that you want to test this sprint. Add them as requirements to your test plan by
creating test suites from them.
Now you've created a requirement-based test suite for each backlog item.
Find a test plan

The Test Plans page shows a single test plan. Use the icon or the drop-down list at the top of the left
column select the test plan you want to work with.
Test plans, suites, and test cases are stored in VSTS and TFS as special types of work items.
See also
Link test cases to work items
Next step
Create manual test cases
Create manual test cases
VSTS | TFS 2018 | TFS 2017 | TFS 2015

Create manual test cases to check that each of the deliverables meet your users' needs. Organize your test cases
by adding test cases to test suites. Then choose which testers you want to run the tests.
Note: Stakeholders cannot create or manage test cases. You must have at least Basic access. See Default manual
testing permissions and access.
Create test cases

1. If you haven't already, create a test plan and requirement-based test suites.
2. Select a requirement-based test suite, and then create a test case for that suite.
The test suite that you selected was created from a backlog item. When you add a test case to this kind of
suite, the test case is linked automatically to the backlog item.
3. Add test steps with actions and expected results so that any team member can run the test. You can add
attachments to a step if you want.
Now you've created a test case that you can run.
Test iterations are design to support data-driven scenarios, not workflow -driven scenarios. From a best
practice perspective, if you have two test scenarios where the the workflows are different, consider creating
separate test cases.
Assign testers
1. You can reassign test cases so that another tester can run them. Select the tests that you want to reassign.
Then open the shortcut menu (choose the "..." ellipses or right-click) and select the tester you want to run
the tests.
Or, you can assign all the test cases in a test suite to multiple testers. This is useful for acceptance testing.
2. After you select the testers, email them so they know the tests are ready for them to run. (You just need
Basic access to run tests from VSTS.)
See also
Link test cases to work items
Next step
Run manual tests
Run manual tests

Run your manual tests and record the test results for each test step using Microsoft Test Runner. If you find an
issue when testing, use Test Runner to create a bug. Test steps, screenshots, and comments are automatically
included in the bug.
You just need Basic access to run tests that have been assigned to you with Visual Studio Team Services
(VSTS ). Learn more about the access that you need for more advanced testing features.
Run tests for web apps

1. If you haven't already, create your manual tests.
2. Select a test from a test suite and run it.
Microsoft Test Runner opens and runs in a new browser.

3. Start the app that you want to test. Your app doesn't have to run on the same computer as Test Runner. You
just use Test Runner to record which test steps pass or fail while you manually run a test. For example, you
might run Test Runner on a desktop computer and run your Windows 8 store app that you are testing on a
Windows 8 tablet.
4. Mark each test step as either passed or failed based on the expected results. If a test step fails, you can
enter a comment on why it failed.
5. Create a bug to describe what failed.

The steps and your comments are automatically added to the bug. Also, the test case is linked to the bug.
If Test Runner is running in a web browser window, you can copy a screenshot from the clipboard directly
into the bug.
6. You can see any bugs that you have reported during your test session.
7. When you've run all your tests, save the results and close Test Runner. All the test results are stored in
VSTS. How do I resume testing, or run one or more tests again?
8. View the testing status for your test suite. You see the most recent results for each test.
9. Open a test and choose the test case in the Related Work section. Then use the Child links in the Related
Work section of that work item to view the bugs filed by the tester.
Can I run tests offline and then import the results?
Run tests for desktop apps

If the only data you want to collect from your desktop app is screen recordings, use the web-based Microsoft Test
Runner in the same way as described above for web apps.
However, if you want to collect more types of data, run your tests using Microsoft Test Manager client.
1. Launch the test runner client from the Test hub by choosing Run with options from the Run menu.
2. In the Run with options dialog, select Microsoft Test Runner 2017 or later, choose the data collectors
you want to enable, and optionally select a build to associate with your test run.
3. Choose OK to start testing. For more information, see Collect diagnostic data.
Can I run tests offline and then import the results?
See also
Next step
View your test progress
Install the Test & Feedback extension
VSTS | TFS 2018 | TFS 2017 | TFS 2015

The Test & Feedback extension helps teams perform exploratory testing and provide feedback. Everyone in
the team, such as developers, product owners, managers, UX or UI engineers, marketing teams, early adopters,
and other stakeholders can use the extension to submit bugs or provide feedback and contribute to the quality
of your product.
Install the extension

1. Check the list of supported browsers and decide which you want to use.
2. Download and install your chosen browser, if you haven't already, then open it.
3. Go to Visual Studio Marketplace > Test & Feedback and choose Install.
4. Follow the instructions shown to install the Test & Feedback extension in your browser:
If you are using Google Chrome, choose the Install link to open the Google Chrome web store
and follow the instructions to install the extension.
If you are using Mozilla Firefox 50.0 and higher, choose the Download link and save the file to a
local folder on your computer.
Select and drag the downloaded file and drop it on any tab in Firefox.
Choose Install.
You need to install the extension or add-on only once. Afterwards your browser will update it automatically.
Select an exploratory testing mode

1. Open the extension you installed in your browser by choosing the icon.
2. Decide if you want to use the extension in Connected or Standalone mode.
Connected mode
Available to all users of VSTS and TFS 2015 or later:
Users with Basic access: Full capture and create capabilities to submit bugs, tasks, and test cases.
Includes collaboration capabilities such as end-to-end traceability, rich insights across completed
exploratory sessions, simplified tracking and triaging for bugs and tasks, and more.
Users with Stakeholder access: Full capture and create capabilities, except for test cases, to submit
feedback and respond to feedback requests from the team.
Feedback experience is available only in VSTS and TFS 2017 or later.
Standalone mode
Available to everyone. No connection to VSTS or TFS is required. Take notes and screenshots with inline
annotations to capture issues. Create bugs and export a session report to share findings.
If you have problems connecting to VSTS or TFS, you may find the topic TF31002: Unable to connect
useful.
See also
Next step
Use the Test & Feedback extension in Connected mode
Exploratory testing with the Test & Feedback
extension in Connected mode
VSTS | TFS 2018 | TFS 2017 | TFS 2015

To use the Test & Feedback extension in Connected mode you must connect to your VSTS account or TFS 2015
and later. This automatically configures the extension based on your access level:
Users with Basic access can use the extension to perform exploratory testing, as described below.
Users with Stakeholder access can use the extension to respond to feedback requests or to provide
feedback voluntarily. More details.
Users with Basic or Stakeholder access can use extension to respond to feedback requests sent by the
team by choosing the Provide feedback link in the email. More details.
Connect to VSTS or TFS

1. If you want to use VSTS, and you haven't already done so, sign up for a VSTS account now. Make sure you
create a project when you create your account.
2. If you haven't already, install the Test & Feedback extension.
3. Open the extension in your web browser and select Connected mode.
4. Enter the URL of the VSTS account or TFS you want to connect to and choose Next.
If you are connecting for the first time, you may be prompted to sign in.
5. After connecting to the server, the extension shows all the collections, projects and teams in that server.
Select the project or team you want to connect to and choose Save.
If there are many projects or teams, use the search textbox to find the one you need.
The extension is now ready to be used in Connected mode. Depending on your access level (Basic or
Stakeholder) you will see the appropriate UI for either exploratory testing or providing feedback. The extension
remembers your selection and remains connected until the session cookies expire or you explicitly disconnect
from the server.
Create bugs or tasks

After you have connected, you are ready to begin testing your app.
1. Start your exploratory testing session.
2. Open the web application you want to test, and start exploring it.
3. When you find an area that has a bug, take a screenshot of any part of the screen, make notes, or record
your actions as a video.
Some browsers may not provide all of the capture capabilities. See Which web browsers does the
extension support?
4. When you are done exploring and capturing information, create a bug or a task.
5. The bug or task form contains all your captured information. It also contains an image action log
describing your interactions with the page (such as mouse clicks, keyboard typing events, touch gestures,
and more) and page load data. Uncheck these options if you do not want to include this data in the bug or
task.
The image action log is the sequence of steps you took that led to the issue. It can be used to
reproduce the issue and understand the context. Page load data provides preliminary information
about the time it takes to load the pages, such as the resource timings and navigation timelines.
6. Enter a title for the bug or task and add any additional notes you require to the description. Then save the
bug or task.
You can also add your findings to an existing similar bug.
7. View a list of all your activities in reverse chronological order in the Session timeline page. It shows all
the screenshots, videos, and notes you've captured, the work items such as bugs, tasks, and test cases
you've already filed, and the work items you've explored.
You can use the extension to explore work items in VSTS or TFS.
8. To view a bug or task in VSTS or TFS, choose the link in the session timeline.
This opens the work item form in VSTS or TFS.
How do I play the video recordings I created with the extension?
Create test cases

The extension lets you create test cases as you explore your application.
1. When you find a scenario where you want to create a test case, choose Create test case.
2. The test case form contains a list of all your actions up to this point while exploring the app (it reads them
from the image action log).
3. Enter a title for the test case and then edit it as required. For example, uncheck the action steps you don't
want to include in the test case, edit the captured text, and add the expected result. Then save the test case.
4. Continue exploring the application. Create more bugs, tasks, or test cases as required.
End your testing session

1. When you're done, stop your session.
2. If you are using VSTS, or TFS 2017 and higher, open the Session timeline page and choose the "view"
icon to see your completed exploratory sessions in VSTS or TFS.
Alternatively, open the Recent exploratory sessions list directly in the Runs tab of the Test hub.
3. Now see how you can view your sessions and get insights.
Next step
Get insights across your sessions
Exploratory testing with the Test & Feedback
extension in Standalone mode
VSTS | TFS 2018 | TFS 2017 | TFS 2015

All teams can use the Test & Feedback extension in Standalone mode. Users don't need a Visual Studio Team
Services (VSTS ) account or Team Foundation Server connection to use this mode.
Start testing in Standalone mode

1. If you haven't already, install the Test & Feedback extension.
2. Open the extension in your web browser and select Standalone mode.
3. Open the web application you want to explore and start the testing session.
4. When you find an area that has a bug, take a screenshot of the entire screen or any part of it.
5. You can annotate the screenshot using the tools available in the inline annotation toolbar.
6. Make notes about the issue to share with your team, and then save the note.
Create a bug
1. When you have finished capturing information for an issue, choose Create bug.
2. The bug form contains all your captured information. Enter a title for the bug and add any additional notes
you require to the description. Then save the bug.
3. View a list of all your activities in reverse chronological order in the Session timeline page. It shows all the
screenshots and notes you've captured and the bugs you've already created.
End your testing session
1. Continue exploring the application. Create more bugs as you encounter issues with the app.
2. When you're done, stop your session.
The extension automatically creates a session report that contains details of all the bugs created during the
session, and any attachments.
3. The report is saved in the default Downloads folder of your web browser. Share it with the rest of your
team as an email attachment, or copy it to OneNote, Word, or in any other format you prefer.
Next step
Use the extension in Connected mode
Load test your app in the cloud using Visual Studio and
VSTS
Visual Studio 2017 | Visual Studio 2015 | VSTS
Check your app or web site's performance before you launch it or deploy updates to production. Find problems
before your customers do. Start running cloud-based load tests in almost no time with Visual Studio and Visual
Studio Team Services (VSTS ).
This example shows how to execute a cloud load test using Visual Studio. You can also run cloud-based load
tests directly using the VSTS portal, or run load tests locally with Visual Studio.
Prepare your environment

Download and install Visual Studio Enterprise, if you don't already have it.
Create a VSTS account, if you don't have one already. You can have any access level assigned to you in
VSTS when you use Visual Studio Enterprise to run load tests.
If you don't have a load test project, use our sample load test project with your web site or app. Just
provide the address for the web site that you want to test. Or, if you have a load test project, jump ahead to
connecting to VSTS to run the load tests.
Get the sample load test project

1. Download the sample load test project, unblock the zip file in its Properties dialog, and unzip the files into
a local folder on your computer.
2. Open the GettingStartedWithLoadTesting.sln solution in Visual Studio Enterprise.
3. Open the SampleWebTest.webtest file. Replace the URL with the URL of your app's web page.
Connect to your VSTS account
Before you can run load tests in the cloud, connect Visual Studio to your VSTS account.
1. In Team Explorer, connect to your VSTS account.
2. Connect to one of your projects.
If you haven't connected to your VSTS account before, add your account to the server list.
Enter your VSTS account name (your-account-name.visualstudio.com ).
If you're prompted to sign in to VSTS, do that now.

3. Select your VSTS account from the list, then choose your project. Now you can connect.
Run and analyze your load test

1. In Solution Explorer, open the load test that you want to run.
2. To run your test closer to where your users are, select a location closer to your users.
3. Now run your load test. This will run in the cloud using VSTS.
Your test appears in the queue and waits for its turn to run. When VSTS is ready to run your test, the test
status changes to "Acquiring resources".
A large test run might take up to 10 minutes while VSTS sets up virtual machines and agents for you.
4. You can watch your app's performance while the test runs. Look at the details to review errors, warnings, or
other information about your test.
5. When the test is done, download the report to view the results.
The results include performance counter data, threshold violations, and error information.
6. Review your test's details. Find the number of users where your app's performance fails to meet your
requirements by examining the step load pattern for virtual users.
7. Fix any performance issues that you find in your app's code, then rerun the test.
8. To simulate real-world loads more closely, you can refine your test by specifying web performance test
properties, load test scenario properties, and run settings properties.
Next step
Run URL-based load tests with VSTS
VSTS
You can run a load test on your web app or site directly using Visual Studio Team Services (VSTS ).

You will need a Visual Studio Enterprise subscription (monthly, annual, or MSDN ) to run URL -based load
tests.
Create your VSTS account, if you don't have one already.
Run a URL-based load test

1. Sign in to your VSTS account (https://your-account-name.visualstudio.com ).
2. Go to the Test hub, open the Load test tab, and choose URL based test from the + New menu.
3. Type a name for the load test, then enter the URL you want to test in the center column and in the details
pane on the right. For a simple load test, leave the HTTP method set to GET.
You can add multiple URLs and select the method for each one, such as POST or PUT. You can also add
headers and querystring values if you need to send these as part of the request. The URL Load Test
accesses each of these URLs multiple times using the parameters you specify, and records the results.
4. Open the Settings tab. Here you can change the parameters of the test such as the duration, load pattern,
number of users, and more. To run the test near to your users, select a Load location. Then choose Save.
5. When you have set up all the URLs and parameters for your test, start it by choosing Run test.
6. As the test runs, you see live information about the progress of the test. You can stop the test by using the
Abort link on the toolbar.
View the results of the load test
1. When your test is done, look at the results to see how well your app performed. For example, you can see
an overview of your app's performance in the Summary tab. This tab shows all of the main metrics such
as average response time, user load, requests per second, failed requests, any errors that might have
occurred, and test usage.
The lower section of the Summary tab shows the settings used for the test, and details of the five slowest
requests during the test. If there are any transaction tests, the tab will also show the five slowest of these.
Use the icon above a column to sort the list based on the contents of that column.
2. Open the Charts tab to see a graphical representation of the test results over time. The charts show the
average performance, throughput, errors, and the results of each test request. Hover your mouse pointer
over a chart to see more details.
3. Open the Diagnostics tab to see detailed information such as a list of errors and status messages.
You can also use the icon in the Errors section of the Summary tab to go directly to the Diagnostics
tab.
4. Open the Logs tab to see a list of test runs. Choose the link in the Attachment column to download the
detailed log as a text file.
5. To run the same test again, choose Rerun.
Key metrics
Response Time. The time it takes for your app to respond to requests is one of the key metrics for
measuring app performance. Most apps perform well when a single user is accessing them, but the
response time can increase dramatically when the app is under load. This can happen if resources such as
CPU, database or other services are at peak capacity, resulting in longer response times.
User Load. The peak user load encountered during the load test run.
Failed Requests. The number of requests that failed, either because the app did not respond or due to
other issues such as connectivity errors. Your app might start throttling requests when under load by
discarding new requests in order to allow existing requests to be processed.
Next step
View and compare load test runs
Load test with the Azure portal
VSTS
Check your web app's performance before you launch it or deploy updates to production. That way, you can
better assess whether your app is ready for release. Feel more confident that your app can handle the traffic
during peak use or at your next marketing push.

You'll need an Azure subscription. You can get one free through Visual Studio Dev Essentials.
You'll need a Visual Studio Team Services (VSTS ) account to keep your performance test history. A suitable
account will be created automatically when you set up your performance test. Or you can create a new
account or use an existing account if you're the account owner. [What else can I do with a VSTS account?]
(reference-qa.md#Team ServicesAccount)
Deploy your app for testing in a non-production environment. Have your app use an App Service plan
other than the plan used in production. That way, you don't affect any existing customers or slow down
your app in production.
Set up and run your performance test

1. Sign in to the Azure Portal. To use a VSTS account that you own, sign in as the account owner.
2. Go to your web app.
3. In the DEVELOPMENT TOOLS section choose Performance test.

4. Now you'll link a VSTS account to keep your performance test history. Choose Set Account.
5. If you have a VSTS account to use, select that account. If you don't, create a new account.
6. Choose + New to create a new performance test.

7. Set the details and run the test. Your web app's default URL is added automatically. You can change the
URL to test other pages (HTTP GET requests only). To simulate local conditions and reduce latency, select a
location closest to your users for generating load.
You simulate load on your app by generating virtual users (customers) who visit your web site at the same
time. This will show how many requests are failing or responding slowly.
As an example, suppose you have an app that gave out coupons at last year's holiday sale. This event lasted
15 minutes with a peak load of 100 concurrent customers. You want to double the number of customers
this year. You also want to improve customer satisfaction by reducing the page load time from 5 seconds to
2 seconds. So, you can test your updated app's performance with 250 users for 15 minutes.
What is the maximum test duration and number of concurrent users?
8. Watch the progress in real time while the test runs. During the first minute, the page loads slower than is
required.
9. After the test is done, view the final results. You can see that the page loads much faster after the first
minute. This helps identify where you might start troubleshooting the problem.
Test multiple URLs

You can also run performance tests incorporating multiple URLs that represent an end-to-end user scenario by
uploading a Visual Studio Web Test file. Some of the ways you can create a Visual Studio Web Test file are:
Capture traffic using Fiddler and export as a Visual Studio Web Test file
Create a load test file in Visual Studio
To upload and run a Visual Studio Web Test file:
1. Follow the steps above to open the New performance test blade. In this blade, choose the CONFIGFURE
TEST USING option to open the Configure test using blade.
2. Check that the TEST TYPE is set to Visual Studio Web Test and select your HTTP Archive file. Use the
icon to open the file selector dialog.
After the file has been uploaded, you see the list of URLs to be tested in the URL DETAILS section.
3. Specify the user load and test duration, then choose Run test.
4. After the test has finished, view the results in the two panes. The left pane shows the performance
information as a series of charts.
The right pane shows a list of failed requests, with the type of error and the number of times it occurred.
5. Rerun the test by choosing the Rerun icon at the top of the right pane.
Next step
View and compare your load test runs

You can review past load test runs or current runs started by anyone on your team, at any time. You can also
compare two test runs to see the gain or loss in performance, and other information.
Open a load test in Visual Studio

1. Open your load test project, open the Load test menu, and choose Load Test Manager.
2. The Load Test Manager page shows all of the load test runs started by you and all of your team members.
Also see Analyze Load Test Results Using the Load Test Analyzer.
Open a load test in VSTS

If you are running URL -based or Apache JMeter load tests, you can see the list of all the test runs in Visual
Studio Team Services Load test list.
Filter and select a load test
1. Filter the list of load tests by state, date, or user who created the test run.
2. Select a test run and open the shortcut menu (in VSTS you can use the icon) to see details of the test run,
or stop a running test.
You can also open a test run by double-clicking it with your mouse.
How do I delete load tests?
Compare two test runs

1. To compare two test runs, select them in the list by holding CTRL while clicking with the mouse. Then
choose the Compare two runs icon on the toolbar, or open the shortcut menu for one of the test runs and
choose Compare.
2. In the comparison page you see the names of the two tests and, at the top of the page, a Summary section
that lists the prime performance factors for each test, then the difference from the baseline as a percentage
(the color of this text indicates a gain or loss in performance).
Use the links in the first row, the names and IDs of the test runs, to open the detailed view of that test run.
3. The Charts section of the page shows a graphical comparison of performance for the two test runs. The
default is a chart for the response time and user load. Choose a different pair of factors from the dropdown
menu to see more performance comparisons.
4. The Test settings section lists the primary settings specified for the two test runs. Again, the names and
IDs of the tests are hyperlinks that open the details of that test.
Next step
Record and replay tests
Add, run, and update inline tests
VSTS | TFS 2018 | TFS 2017

Similar to task checklists, you can quickly define inline tests, or a set of manual tests, for a backlog item from your
Kanban board. Not only can you add tests, you can run them and update their status. If you're new to working with
the Kanban board, see Kanban basics.
In this topic, you'll learn:
How to add inline tests to a backlog item from your Kanban board
How to run tests and update the status of tests
How to expand or collapse inline tests
How to reorder or reparent inline tests
Tests you create from the Kanban board are automatically linked to the user story or backlog item.
Add tests
1. To start adding tests, open the menu for the work item.
Adding inline tests is the same as adding test cases to a test suite. A default test plan and test suite are
automatically created under which the manual test cases are grouped.
For example, a test suite is created for each user story, and all inline tests are added to that suite. Below,
user story 152 is highlighted which has three manual tests defined with IDs of 153, 155, and 161.
To learn more about test plans and test suites, see Plan your tests.
2. If you have a number of tests to add, simply keep typing each title and click Enter.
To add details to the test case, open it. You can click the title, double-click the inline item, or open the context
menu and choose Open.
See Create manual tests to learn more about defining tests.

Prior to running the test, you must add details.
Run test
Run the test by selecting Run test from the actions menu for the inline test.
Microsoft Test Runner starts in a new browser instance. For details on running a test, see Run manual tests.
Update the status of a test

You can update the status of the test from the actions menu .
Updating the status of tests enable you to track test results.

Why doesn't the Kanban board show the status for test suites and plans already created in the Test hub?
Expand or collapse inline tests

Upon first opening the Kanban board, you'll see an unexpanded view of checklists.
Simply click the inline test summary to expand a collapsed set of tests. Click the same summary to collapse an
expanded list.
Copy or reparent a test

To reparent a test, drag and drop the test onto a different user story.
This action automatically changes the linked relationship of the test to point to the new user story.
To create a copy of a test to add to a different user story, select the test, press the CTRL key and then drag and
drop the test onto the card of the user story.
Related articles
Use inline tests for lightweight traceability and to manage manual tests for user stories or other backlog items that
they support. To learn more about test case management, see Create manual tests.
If you find that you don't use this feature, you can disable it from the common configurations dialog.
Additional ways you can quickly add linked items and objects to user stories from the Kanban board:
Add inline tasks
Create a new branch, drive Git development
To initate web-based exploratory testing for a user story, you need to install the Exploratory testing , see
Exploratory test your web app directly in your browser.
Test status in the Kanban board
Test integration with the Kanban board makes it easy for teams to get started with manual testing and then take
advantage of the full testing capabilities in Test Manager later, when required. When test cases are created from
the Kanban board and updated afterwards in Test Manager, the Kanban board shows the correct status. However,
integration is not optimized to work in the other direction; for example, when users create requirement-based
suites with Test Manager instead of in the Kanban board. We intend to make some major performance
improvements to this integration in future releases.
Tutorial: Load test your app before release

Find performance issues before you release your app by running load tests with Visual Studio Enterprise using
Cloud-based Load Testing to provide virtual machines in the cloud that generate the load of many users
accessing your web site at the same time. All you need is a Visual Studio Team Services (VSTS ) account.
In this tutorial, you'll learn how to:
Create a web performance and load test project
Record a web performance test
Create a load test
Improve your load tests
Create a web performance and load test project

You first create web performance tests. These tests are used in your load tests to simulate multiple users
performing actions in your app at the same time.
1. If you don't have Visual Studio Enterprise, get it here.
2. Create a web performance and load test project.
If you don't see the template for the web performance and load test project type, ensure you have installed
the required packages during Visual Studio setup. You can re-run the installer by choosing Get tools and
features on the Visual Studio Tools menu.
Record a web performance test
1. Create a web performance test.
2. Your web browser opens. Enter the URL for the website that you want to test.
3. Use your application like you expect your customers to use it. For example, search for items and add them
to the shopping cart. The recorder will capture the HTTP requests and responses.
4. When you're done, stop recording.
Now, Visual Studio looks for dynamic parameters for the HTTP responses to each of your HTTP requests.
A progress bar appears while this happens. If dynamic parameters are found, a table appears. You can
assign constant values to each dynamic parameter.
5. Rename your test. For example, ShoppingCart.webtest.
6. Edit test properties to specify performance goals. For example, you can set a page response time goal of
one second.
7. Save the test.
Create a load test

1. Create a new load test in the web performance and load test project.
2. When the load test wizard appears, choose the kind of load test that you'd like to run.
3. Change the load pattern to step load. This gradually adds users over time. How many virtual users can I
configure in my load test?
4. Choose the test mix step.
5. Add the web performance test you created.
6. Move the web performance test to the list of tests to run.

7. When you run cloud-based load tests using your VSTS account, you can run those tests and generate load
in an Azure datacenter that's closer to your users. That way, you reduce latency and simulate local
conditions. Select the location where you want to run your load test.
8. After you finish the wizard, the web performance test is added to the load test and appears in the load test
editor.

You can run your load test locally, or you can run it in the cloud using VSTS. All you need is a VSTS account. If
you run the load test in the cloud, you can generate more load without setting up test controllers and test agents.
To learn how easy it is to use Cloud-based Load Testing to run your load tests, go here.
Follow these steps to run your load test on your local machine.
1. Run the load test.
2. While the test runs, you discover that the shopping cart page response time exceeds the value you set.
3. Add an analysis note to track the issue.

4. After the load test is finished, the summary is displayed.
The results for the completed test include performance counter data, threshold violations, and error
information.
5. Choose the detail view. By analyzing the step load pattern for users, you can identify the user count where
your performance fails to meet your requirements.
6. Fix any performance issues in your application's code and rerun the test.
Improve your load tests

You can improve your test to better simulate real-world loads by specifying various load test scenario properties
and run settings properties. For example, you can specify the number of new users that will use web cache data in
your load test.
Next step
Run URL -based load tests
Track test status

Quickly view the status of your testing using lightweight charts. For example, find out how many test cases are
ready to run, or how many tests are passing and failing in each test suite. You can pin these charts to your home
page, then all the team can see the progress at a glance.
Track testing progress

Use test results charts to track how your testing is going. Choose from a fixed set of pre-populated fields related
to results. By default, a pie chart is created for each test plan. This chart is grouped by the outcome field to show
the latest results for all the tests in the test plan.
View this default chart from the Charts tab.
Add your own charts for test results to visualize what's important for your team. If you already know how to add a
chart, jump to the examples below of charts that you can create.
1. Select the test plan or test suite for your chart in the Test plan tab. Then create a new chart.
2. Select the chart type. Based on the chart, configure the fields that you want to use to group by, or for rows
and columns.
All charts roll up the information for any child test suites of the test plan or test suite that you selected.
3. Save the chart. Now it will be displayed in the charts tab for the test plan or test suite that you selected.
Test results examples
What's the test status for a specific test suite?
Select the test suite from the Test plan tab and add a test results pie chart. Group by outcome.
What's the test status for user stories that my team's testing this sprint?
If you have created requirement-based test suites in your test plan for your user stories, you can create a chart for
this.
1. Group these requirement-based test suites together in a static test suite.
2. Select this static test suite in the Test plan tab.
3. Add a test results stacked bar chart. Choose Suite as the rows pivot and Outcome as the columns pivot.
How many tests has each tester left to run?

Select your test plan from the Test plan tab and add a test results pivot table chart. Choose Tester as the rows
pivot and Outcome as the columns pivot.
How can I check quality based on the configuration?

Use either a stacked bar chart or a pivot table chart. Choose Configuration as the rows pivot and Outcome as the
columns pivot.
How can I track why tests are failing for my team?
For failure analysis, use either a stacked bar chart or a pivot table chart. Choose Tester for the rows and Failure
type for the columns. (Failure type for test results can only be set using Microsoft Test Manager.)
How can I track the resolution for failing tests for my team?
For resolution analysis, use either a stacked bar chart or a pivot table chart. Choose Tester for the rows and
Resolution for the columns. (Resolution type for test results can only be set using Microsoft Test Manager.)
Track test case status

Use test case charts to find out the progress of your test case authoring. The charts for test cases give you the
flexibility to report on columns that you add to the Tests tab. By default, test case fields are not added to the view
in the Tests tab.
If you already know how to add a chart, jump to the examples below of charts that you can create for test cases.
1. Add any fields you want to use for your test case chart from the Tests tab with Column options. Then the
fields will appear as choices in the drop-down lists for grouping for your test case charts.
2. Select the test plan or test suite for your chart in the Test plan tab. Then add a test case chart.
All charts roll up the information for any child test suites of the test plan or test suite that you selected.
3. Select the chart type. Based on the chart, configure the fields that you want to use to group by, for rows
and columns, or the range (trend charts only).
You can't group by test suite for the test case charts.
4. Save the chart. Now it will be displayed in the charts tab for the test plan or test suite that you selected.
Test case examples
How can I track burndown for test case creation?
Use a stacked area trend chart to view the burndown for how many test cases are ready to be run. Choose State
for the stack by field and Ascending for the sort field.
How can I track burndown for automation status?

Use a stacked area trend chart to view the burndown for how many test cases have been automated. Choose
Automation status for the stack by field and Ascending for the sort field.
If multiple teams own test cases in my test plan, can I see how many each team owns and the priorities
of the tests?
If your teams are organized by area path, then your can use a test case pie chart. Choose Area path for the group
by field.
If you want to know the priorities of these tests, then create a stacked bar chart. Choose Area path for rows and
priority for the columns.
How can I track test creation status by team members?
Test case owners are tracked by the Assigned to field. Use a stacked bar chart or a pivot table chart. Choose
Assigned to for rows and status for the columns.
Share charts on your team's dashboard

Pin a chart to your team's dashboard for all the team to view. Use the chart's context menu.
You can configure the dashboard widget to show a range of chart types. You must be a team administrator to do
this, but team members with Stakeholder access can view the charts on the dashboard.
Learn more about dashboards. Or learn more about team administration.
See also
Next step
Control how long to keep test results
Move, change, or delete work items

VSTS
Often times you find that someone created a work item of the wrong work item type (WIT) or within an incorrect
team project. You can correct these issues for individual work items or bulk modify several work items. You can also
remove work items added to your backlog or task board that aren't relevant anymore.
Delete or restore work items

TFS 2018 | TFS 2017 | TFS 2015 | TFS 2013
You can remove work items added to your backlog or task board that aren't relevant anymore. Simply change the
State to Remove, or delete the work item. You can perform operations on individual work items or bulk modify
several work items.
TIP
You can't change the work item type for an existing work item, but you can copy the work item and specify a new type. Also,
if you have several work items with type changes you want to make, you can export them using Excel, and then re-add them
as a new type.
In this topic you'll learn:

How to change the work item type of one or more work items
How to move one or more work items to another team project
How to remove work items from the backlog by changing the State to Removed
How to delete work items and test artifacts
How to restore or permanently delete work items (web portal)
What permissions are required to delete work items and test artifacts
How to delete work items and test artifacts
How to restore or permanently delete work items (web portal)
How to permanently delete work items (command-line tool)
What permissions are required to delete work items and test artifacts
How to permanently delete work items (command-line tool)
What permissions are required to delete work items
You only have access to those actions that are supported on your platform and for which you have permissions. If
you are a member of the Contributors group (anyone who has been added as a team member) or Project
Administrators groups, you have access to the following features. For a simplified view of permissions assigned to
built-in groups, see Permissions and access.
You can access the following actions for which you have permissions. If you are a member of the Contributors
group (anyone who has been added as a team member) or Project Administrators groups, you have access to the
following features. For a simplified view of permissions assigned to built-in groups, see Permissions and access.
CONTRIBUTORS PROJECT ADMINISTRATORS
- Change work item type - Move a work item to another team project
- Remove work items (change State) - Permanently delete work items (web portal)
- Delete work items (web portal) - Permanently delete test artifacts
- Restore work items (web portal)
You can't change type, move work items, or delete/restore work items whose WITs support test management or
that belong to the Hidden Types Category. This includes all work items that track tests—such as test cases, shared
steps, and shared parameters—code review requests and responses, and feedback requests and responses.
- Delete work items (web portal) - Permanently delete work items (command-line tool)
- Restore work items (web portal) - Permanently delete test artifacts
- Delete work items - Permanently delete work items (command-line tool)
- Restore work items
- Remove work items (change State) - Permanently delete work items (command-line tool)
TIP
From the web portal, you can multi-select several work items from a backlog or query results page and perform a bulk
update using the associated feature. To change, move, delete, or restore several work items at the same time, see Bulk modify
work items.
Prerequisites
To change the work item type, delete, or remove work items, you must be a member of the Contributors group
or be granted Stakeholder access Or, you must have your View work items in this node, and your Edit work
items in this node permissions set to Allow
To move work items to another team project, you must be a member of the Project Administrators group or
have the Move work items out of this project permission set to Allow. The Contributors group does not have
this permission set at the project-level by default.
To delete work items, you must be a member of the Project Administrators group or have the Delete work
items in this project permission set to Allow. The Contributors group has Delete and restore work items at
the project-level set to Allow by default.
Prerequisites
To remove or delete work items, you must be a member of the Contributors group or be granted Stakeholder
access Or, you must have your View work items in this node, and your Edit work items in this node
permissions set to Allow
items in this project permission set to Allow. The Contributors group has Delete and restore work items at
the project-level set to Allow by default.
Prerequisites
To remove work items, you must be a member of the Contributors group or be granted Stakeholder access
Or, you must have your View work items in this node, and your Edit work items in this node permissions
set to Allow
items in this project permission set to Allow. For TFS 2015.1 and earlier versions, the Contributors group has
Delete work items in this project at the project-level set to Not set by default. This setting causes the
Contributors group to inherit the value from the closest parent that has it explicitly set.
To learn more, see Set permissions and access for work tracking.
Change the work item type

NOTE
You can't change the work item type of work items associated with test management. Both Contributors and users assigned
Stakeholder access can change the work item type.
Changing the work item type refreshes the work item form with the fields defined for the type selected. For
example, you can change a bug to a task and the form will refresh with the fields defined for a task.
You can change a single work item or several multi-selected work items to a new type.
1. Select the Change type... option from the work item form's Actions menu.
Or, from the backlog or query results page, multi-select several work items whose type you want to change.
You can select several work items of the same type or different type so long as you want to change them all
to the same work item type.
Click to open the context menu of one of the selected work items, and choose the Change type…
option.
IMPORTANT
From the Query results page, the Change type… option becomes unavailable if you have checked the Query Editor's
Query across projects checkbox.
2. Select the type and optionally enter a comment.
Comments are automatically added to the Discussion control.

3. Save the work item to complete the change.
NOTE
The system automatically resets the State and Reason fields to the default initial values of the specified type.
4. From the Query results page, you must save all work items that you bulk-modified. When you bulk modify
items from the backlog, they are automatically saved. Work items shown in bold text indicate that local
changes have not yet been saved to the data store. The system automatically saves each work item. Refresh
the page to reflect your changes.
Move a work item to another team project

When you discover that a work item belongs to a different team project within your account or collection, you can
move it where it belongs. You can move a single work item or several multi-selected work items.
You can only move work items from one team project to another team project within the account or collection. You
can't move work items associated with test management. To move work items to another team project, you must
be a member of the Project Administrators group or be granted explicit permissions to move work items.
1. Open the work item and choose the Move... option from the work item form's Actions menu.
If you don't see the option, then you haven't been granted permissions to move work items out of the
project.
Or, from the backlog or query results page, multi-select several work items whose type you want to change.
You can select several work items of the same type or different type so long as you want to change them all
to the same work item type.
Click to open the context menu of one of the selected work items, and choose the Move… option.
2. Select the destination team project and optionally enter a comment.
Comments are automatically added to the Discussion control and an entry is made to the History control.
Also, the system automatically resets the State and Reason fields to the default initial values for the work
item type that you move.
Remove work items by changing the State

By changing the State of a work item to Removed, you effectively remove it from a backlog or board view (product,
portfolio, and sprint backlogs, Kanban board, and task boards).
To cause removed items to not show up in queries, you must add a clause that indicates which states you want the
query to filter for.
Delete work items

NOTE
Feature availability: The Delete and Recycle bin features are available from TFS 2015.2 and later versions.
Deleted work items won't appear in your backlogs, boards, or queries. Deleted items are moved to a recycle bin
from which you can recover them if needed. To delete a test case, test plan, or test suite, or other test-related WITS,
see Delete test artifacts.
1. You can delete a work item from within the work item form, or by multi-selecting work items from a backlog
or query results page.
2. Confirm you want to actually delete the item(s).
NOTE
The Delete work items confirmation dialog indicates there are auto-delete settings (disabled). There are no settings
you can enable or disable. There is only a background process which permanently deletes work items that have been
set to delete.
3. Using multi-select from a backlog or query results list, you can delete several work items at once.
4. You can also delete work items from your Kanban or task board.
Or, you can drag them to the (Recycle bin). You can only access the (Recycle bin) from the
Work hub.
Restore or permanently delete work items

To permanently delete work items from the web portal, you must be a member of the Project Administrators group
or be granted explicit permissions to delete or restore work items.
Browser
Command Line
Restoring or deleting work items from the web portal isn't a supported feature for TFS 2013.
NOTE
Feature availability: The Delete and Recycle bin features require TFS 2015.2 or later version.
1. To restore deleted items, open the Recycle bin from the web portal.
2. Select the items you want to restore and then choose Restore.
Optionally, you can choose to permanently delete the items.
NOTE
You'll only see the Permanently delete option if your Permanently delete work items permission is set to Allow.
3. Confirm your selection.

You must be a member of the Project Administrators group or have the Delete test artifacts permission set to
Allow. You must also have your access level set to Advanced, which provides access to the full Test feature set.
Users with Basic access and with permissions to permanently delete work items and manage test artifacts can only
delete orphaned test cases. That is, they can delete test cases created from the work hub that aren't linked to any
test plans or test suites.
To delete test artifacts, the following restrictions and operations apply:
Users with Basic access and with permissions to permanently delete work items and manage test artifacts can
only delete orphaned test cases. That is, they can delete test cases created from the work hub that aren't linked
to any test plans or test suites.
When you delete a test plan, test suite, test case, shared steps, or shared parameters, you not only permanently
delete them, you also delete all associated test artifacts such as test results.
You can't bulk delete test artifacts. If test artifacts are part of a bulk selection to be deleted, all other work items
except the test artifact(s) will get deleted.
IMPORTANT
Feature availability: The permanently delete feature of test artifacts is available from the Test and Work hubs for TFS 2017.1
and later versions.
We only support permanent deletion of test artifacts such as test plans, test suites, test cases, shared steps and shared
parameters. Deleted test artifacts won't appear in the recycle bin and cannot be restored. Deletion of test artifacts not only
deletes the selected test artifact but also all its associated child items such as child test suites, test points across all
configurations, testers (the underlying test case work item doesn't get deleted), test results history, and other associated
history.
1. To delete a test case, open it from the web portal and choose the Permanently delete option from the actions
menu. (Bulk deletion is not supported from a query results page.)
NOTE
You'll only see the Permanently delete option if you have the necessary permissions and access.
2. Confirm you want to actually delete the item.

3. You can also delete test plans and test suites directly from the Test hub.
4. To delete shared steps and shared parameters you need to first manually remove all references to them
before you can delete them.
Related articles
To learn more about managing test artifacts, see:
Create a test plan
Control how long to keep test results
Delete and restore actions performed under the hood
Delete work items
When you delete a work item, the following actions occur:
Generates a new revision of the work item
Updates the Changed By/Changed Date fields to support traceability
Preserves the work item completely, including all field assignments, attachments, tags, and links
Causes work item to become non-queryable and therefore can't appear in any work tracking experience, query
result, or report
Updates charts accordingly, CFD, velocity, burndown and lightweight charts are updated to remove deleted
work items
Removes WIT extensions
Preserves trend data except for the latest value
Removes the work item from the data warehouse/cube similar to as if it was permanently removed.
Restore work items
When you restore a work item, the following actions occur:
Causes a new revision of the work item to be made
Updates the Changed By/Changed Date fields to support traceability
Becomes queryable
All fields remain unchanged
History contains 2 new revisions, one for deletion, and one for restore
Reattaches WIT extensions
Updates charts accordingly, CFD, velocity, burndown and lightweight charts are updated to include the restored
work items
Restores trend data
Adds the work item back to the data warehouse/cube similar
Sets the area or iteration path fields to the root node if the previous area path or iteration paths were deleted.
1. Removes the deleted test artifact from the test case management (TCM ) data store and deletes the underlying
work item
2. Runs a job to delete all the child items both from the TCM side and the underlying work items. This action may
take time (up to a few minutes) depending on the number of artifacts to be deleted.
3. Causes all information in the WIT data store and TCM data store to be deleted and cannot be reactivated nor
restored.
VSTS | TFS 2018 | TFS 2017 | TFS 2015

Today's faster development pace requires tools that enable test teams to more easily verify value based on
business requirements, and the high quality software demanded by customers. This type of testing is often
referred to as user acceptance testing and is available as a feature in Visual Studio Team Services (VSTS ) and
Team Foundation Server (TFS ).
Typically you create a Test Suite using a formal requirement work item type. However, today's agile teams often
prefer to work from User Stories or Product Backlog items as their requirements.
Before you start

You must have already created work items and a test plan. If not, follow the steps in:
Create your backlog
Create a test plan
Assign and invite testers

VSTS makes it easy to assign testers to individual test cases. A typical scenario for user acceptance testing is the
ability to not just assign one tester to a test case (see Search for and assign testers) but assign multiple testers an
entire set of tests.
This can also be accomplished by selecting the suite and choosing Assign testers to run all tests. This option
also enables the sending of emails to the testers assigned to the tests.
An important feature of user acceptance testing is that the testers may in fact be the business owners who created
the original business requirements.
Search for and assign testers
In scenarios where you have large development teams the ability search for an individual is also important.
Choose Assign tester from the drop-down menu. In the shortcut menu, choose Assign testers to run all tests
and select the testers you want to include.
Set the Send email option to send all of them a notification email.
Easily track results

A key principle of good user acceptance testing practice is to minimize the effort required to determine whether a
requirement has been achieved. There are two ways this can be achieved, you can focus on individual test runs
and tests in the Test hub to see which failed or use the charts views make it much easy and accessible to all
members of VSTS makes this much easier.
Note: The dashboard display shown here is also used for other types of testing such as continuous testing.
If you don't see the data or information you expect in the dashboard charts, verify that the columns in your data
have been added to the Tests view. For details see this blog post.
Help and support

VSTS
Your users will probably install or run your app on a wide variety of configurations, such as different operating
systems, web browsers, and other variations. You will want to run at least some of your tests in environments that
have those different configurations.
Use your test plan to decide which tests you want to run on which configurations. You have to make sure that
when you run your tests that you have set up your environments for the configurations that you need.
You might draw up a schematic matrix of the combinations that you want to test:
Then you can:

Create the configurations and variables
Assign the configurations to test plans and test suites
Run tests with each of the configurations
Track your test results for each configuration
Note: This feature is available only in VSTS. In addition, Stakeholders and Basic users cannot create or
manage configurations.
Create configurations and variables

A test configuration is a combination of configuration variable values. Your configuration variables could be, for
example, operating system, browser, CPU type, database. A configuration might be "Windows 8 + 32-bit CPU" or
"Windows 10 + 64-bit CPU."
You must create the configuration variables first. Then combine multiple variable values to create a configuration.
1. Open the Configurations tab in the Test hub, choose the icon, and select New configuration
variable.
2. Type a name for the variable, such as Browser, and type a value. Add as many values as you wish to the
configuration variable, and then save it.
3. Repeat the steps to create any other configuration variables you need. For example, create a configuration
variable named Operating system with the names of each operating system on which you want to test.
4. Choose the icon and select New test configuration.
5. Type a name for the test configuration and add the configuration variables you created. Choose a value for
each variable for this configuration.
Ensure Assign to new test plans is checked to make this the default configuration for all the new test
plans you create.
6. Save your new test configuration.
Assign configurations to test plans and suites

You can assign configurations to a test plan, a test suite, or an individual test case. Configurations assigned to a
test plan or test suite apply to all tests or suites within it.
1. To assign a configuration to a test plan, open the shortcut menu for the plan and choose Assign
configuration to test plan.
2. To assign a configuration to a test suite, open the shortcut menu for the suite and choose Assign
configuration to test suite.
If you add multiple configurations to a test plan or suite, the tests cases are repeated in the plan or suite
with the each of the configurations you have assigned.
3. If necessary, override the default configuration assigned to a test case and assign the configuration you
need. Select one or more test cases, open the shortcut menu, and choose Assign configurations.
4. Search for and select the configurations to assign to these test case(s).
Run tests with each configuration
1. Set up a testing platform for a particular configuration, such as testing the app using Google Chrome on
Windows 10.
2. Select and run a test that has this configuration assigned.
As you run the test, a reminder of the required configuration in shown in the status bar of the Test Runner
window.
Track test results for each configuration
1. Open the Charts tab for your test plan or test suite, choose New, and select New test result chart.
2. Choose the type of chart you require, select Configuration in the Group by list, and choose OK.
A chart is created that can help you track your tests based on configurations. You can pin this chart to your
dashboard.
If you have a test case that appears in several test plans and test suites, you can set the different configurations
for each of these. The same test case can have different configuration settings in different test suites and test
plans.
See also
Overview of manual and exploratory testing
Exploratory test and submit feedback directly from your browser
Help and support

VSTS | TFS 2018 | TFS 2017 | TFS 2015

When you write a manual test, you often want to specify that the test should be repeated several times with
different test data. For example, if your users can add different quantities of a product to a shopping cart, then you
want to check that a quantity of 200 works just as well as a quantity of 1.
To do this, you insert parameters in your test steps. Along with the test steps, you provide a table of parameter
values. You can also share parameters and their data between test cases when you use the web portal with TFS
2015 and later or VSTS. That way you can run multiple test cases with the same data.
Add parameters to a test case

1. Create a parameter by typing a name preceded by "@" in the actions and expected results of your test
steps.
2. Underneath the list of steps, add combinations of parameter values. You might need to scroll down to see
them.
Share parameters between test cases

1. Convert existing parameters to shared parameters so that you can use them and the associated data in
other test cases.
2. After you've created a shared parameter set, open another test case and add the shared parameter set to
that test case. You can search for the shared parameter set by name.
The shared parameter set is displayed in the Parameter values section after you add it. You can now use
these parameters in your test case steps.
3. If the test case has different parameter names for these shared parameters, map the shared parameters to
the local parameters to use the shared parameter data.
When they are correctly mapped, the data associated with the shared parameter is displayed.
4. Add, edit and rename your shared parameter sets in the Parameters tab. View the test cases that reference
them in the Test cases pane.
5. Each shared parameter set is a work item. Open the Properties tab to view or make changes to this work
item. For example, you can assign owners and track changes.
You can't add more than one shared parameter set to a single test case. If two test cases share similar data, for
example, one test case needs customer ID, name, email, and phone, and the second needs customer ID, name
and address, you might consider creating a single shared parameter set containing all of the parameters -
even though a few of the columns in the set will remain unused in each test case.
Run a test case with parameters

1. Select a test case with parameters and start it running. The Test Runner shows the first row of parameter
values.
2. When you've completed the steps, mark the test passed or failed. Then go on to the next iteration of the
test, which uses the next row of parameter values.
3. Use the drop down to navigate to other iterations.

4. If any of the parameter values are incorrect, fix them without canceling the test by choosing Edit from
step's shortcut menu.
Review the test results

If you marked any test iteration as failed, the outcome of the entire test is shown as failed.
1. Check the test result by opening the details pane.
2. Double-click a test result to view the test run details, and the test results for each iteration.
Speed up test iterations by using record and playback
It can be error-prone and tedious to work through a long table of parameter combinations. To speed things up,
create an action recording when you run the test with the first set of parameter values, and then play it back for
the other sets.
1. Use Microsoft Test Manager to run the test.
2. Select Create action recording before you choose Start.
3. Complete the first test iteration and then move on to the next one.
4. Mark each step as passed or failed as you work. Enter parameter values in the application exactly as they
are displayed in the test script.
5. Choose Play to run the test with the next set of parameter values. Your actions will be played back
automatically, but you must still verify the results.
Record and playback doesn't work with all applications. For details, see Supported Configurations and Platforms
for Coded UI Tests and Action Recordings.
See also
Testing different configurations
Manage test results
Help and support

Collect screenshots, video, logs, and attachments in
continuous tests
VSTS | TFS 2018 | TFS 2017

When running tests in a CI/CD pipeline, collecting diagnostic data such as screenshots, video, logs, and
attachments is often useful to help troubleshoot failures.
Collect screenshots, logs, and attachments

1. Add the file you want to include in the test results as a TestResult file.
Screenshot ss = ((ITakesScreenshot)driver).GetScreenshot();
string path = Directory.GetCurrentDirectory() + "SearchTestScreenshot.png";
ss.SaveAsFile(path);
this.TestContext.AddResultFile( path);
2. Define the TestContext in your test class.
private TestContext testContextInstance;
public TestContext TestContext

{
get
{
return testContextInstance;
}
set
{
testContextInstance = value;
}
}
3. View the test results and the collected files as attachments to your test results in VSTS or TFS.
Collect video using the Visual Studio Test task
1. To collect video, specify the data collector you want to use in a runsettings file. The video data collector
captures a screen recording when tests are run.
2. In the Visual Studio Test task, specify the location of the runsettings file.
Publish attachments when not using the Visual Studio Test task
If you are not using the Visual Studio Test task, you must publish attachments in your test results in a different
way:
If you are running tests in the build (CI) pipeline, you can use the Copy and Publish Build Artifacts task to
publish any additional files created in your tests. These will appear in the Artifacts tab in your build
summary.
Use the REST APIs to publish the necessary attachments. Code samples can be found in this GitHub
repository.
The NUnit and JUnit test result formats do not have a formal definition for attachments in the results schema, so
the Publish Test Results task cannot publish attachments when these formats are used.
See also
Continuous testing scenarios and capabilities
Set up continuous testing for your builds
Test with unified agents and phases
Pass parameters to tests from a build or release pipeline
Help and support

Collect diagnostic data while testing
VSTS | TFS 2018 | TFS 2017

Collect diagnostic data while testing your apps. This data will be included in the bugs you file during the test. You
can collect diagnostic data from web apps and from desktop apps, and view it in Team Services or Team
Foundation Server.
Collect diagnostic data from web apps
Collect diagnostic data from desktop apps
Collect diagnostic data from web apps

For web apps under test, you can use web-based Microsoft Test Runner to collect the following data on demand:
Screen captures
Image action log
Screen recordings
The following diagnostic data collection features currently work only with the web-based Microsoft Test
runner. See Exploratory test and submit feedback directly from your browser
Screen capture
Capture annotated screenshots from your web app.
1. Ensure that the tab of the app from which you want to capture data is the active tab.
2. Open Test Runner and choose the Capture screenshot icon.
3. Choose the title of the tab containing the app you are testing.
If the tab title you want is not shown in the list, switch to the app and activate it by tapping on the title bar,
or on a child window if the app has opened one.
4. Drag to select the area of the screen you want to capture, or just capture the full screen.
5. If required, edit the title of the screenshot and add annotations and text to it using the icons in the toolbar.
6. Save your screenshot.
Image action log

Capture your interactions with the web app as an image action log that provides context.
2. Open or switch to the Test Runner and choose the Capture user actions... icon.
3. Choose the title of the tab containing the app you are testing.
If the tab title you want is not shown in the list, switch to the app and activate it by tapping on the title bar,
or on a child window if the app has opened one.
4. The Test Runner will now record all the actions you take on the app's browser tab.
If you create a bug while recording your actions, all the data collected up to that point will be included in the
bug.
5. Finish capturing your actions by choosing the Stop button. The action log is added to the test results as an
attachment.
6. Choose the ActionLog... link at the bottom of the window to view the data captured in the action log.
The log opens in your web browser.
Screen recording
Capture screen recordings from your web apps.
2. Open or switch to the Test Runner and choose the Record screen icon.
3. Choose the entire screen, or choose an app to start recording.
If you create a bug while recording your screen, the recording automatically stops and is added to the bug.
4. Finish recording your actions by choosing the Stop button. The recording is added to the test results as an
attachment.
If you do not stop the recording after ten minutes, it stops automatically and is saved as an attachment to
your test results. Restart the recording the Record screen icon if required.
5. Choose the ScreenRecording... link at the bottom of the window to view the captured recording.
View the diagnostic data

When you create a bug while capturing diagnostic data, all the data captured up to that point is included in the
bug that is created. You can view it before you save the bug.
Collect diagnostic data from desktop apps

At present you can collect only screen recordings and system information when testing desktop apps using the
web-based Microsoft Test Runner. Instead, use Microsoft Test Manager client to collect additional diagnostic from
desktop apps.
1. In the Test hub in VSTS or TFS, select a test case, test suite, or test plan to execute.
2. Open the Run menu and choose Run with options.
3. In the Run with options dialog, select Microsoft Test Runner 2017 or later in the first drop-down list.
4. Choose the data collectors you want to enable. Options include Event log, Action log, Screen and voice
recorder, and System information.
By default, the test runner client lets you to capture screenshots of your desktop app during testing.
5. If you wish, select a build to associate with your test run. A link to this build will be included automatically in
all the bugs you create during the test run.
6. Choose OK to start testing.
If you want to collect advanced diagnostic data such as code coverage, IntelliTrace, and Test Impact data in
addition to the data items listed above, you must configure the data collectors and other run settings in Microsoft
Test Manager (MTM ) and run your tests using MTM. For more details, see Run manual tests with Microsoft Test
Manager.
NOTE
If you have an older version of Microsoft Test Manager, we recommend you upgrade to the latest version. However, if you
have Microsoft Test Manager 2015 or an earlier version installed, you can choose Microsoft Test Runner 2015 and earlier
when you launch the test runner from Test hub using Run with options. You must configure the data collectors and other
run settings in Microsoft Test Manager (MTM) and specify these as the default settings for the test plan. For more details,
see Run manual tests with Microsoft Test Manager.
See also
Help and support

Control how long to keep test results in VSTS

Running tests, especially automated ones, generates lots of data. To keep your test system responsive and
performing well, have a policy to clear test results that you don't need anymore. Delete automated test results
when you delete your builds. You can keep manual test results while you're still reviewing them, for example, up to
a year.
Manual test results

To delete manual test results after a specific number of days, set the retention limit at the project level. Visual
Studio Team Services (VSTS ) keeps manual test results related to builds, even after you delete those builds. That
way, build policies don't delete your test results before you can analyze the data.
1. Sign to your VSTS account (https://your-account-name.visualstudio.com ). You'll need at least project
administrator permissions.
2. Go to your project.
3. Go to the project control panel.
4. Select a limit for how long you want to keep manual test data.
Automated test results

Automated test results associated with builds
By default, VSTS keeps automated test results related to builds only as long as you keep those builds. To keep test
results after you delete your builds, edit the build retention policy. If you use Git for version control, you can
specify how long to keep automated test results based on the branch.
1. Sign to your VSTS account (https://your-account-name.visualstudio.com ). You'll need at least build level
permissions to edit build definitions.
2. Go to your project. Find and edit your build definition.
By default, test results are deleted when the build is deleted.
3. If you use Git, and you have more than one branch, set the branch filter to delete test results and builds in
one branch. Meanwhile, you can keep test results in another branch, even though you delete the builds in
that other branch.
Automated test results not associated with builds or orphaned from deleted builds
To clean up automated test results that are left over from deleted builds or test results that aren't related to builds,
for example, results published from external test systems, set the retention limits at the project level. Learn more
See also
Help and support

Associate automated tests with test cases
Visual Studio 2017 | Previous version

Consider using Visual Studio to associate automated tests with a test case when:
You created a manual test case that you later decide is a good test to automate, but you still want to be able
to run that test as part of a test plan. Tests can be run in the CI/CD pipeline by choosing the test plan or test
suite in the settings of the Visual Studio Test task. Automated tests can also be run from the Test hub in
VSTS and TFS. If you are using XAML builds you can also run these automated tests by using Microsoft
Test Manager.
You want to enable end-to-end traceability of requirements. If your test cases are linked to requirements or
user stories, the results of the test execution can be used to establish the quality of those requirements.
NOTE: At present you cannot use this procedure to associate MSTest V2 tests or tests written in NUnit and
XUnit. Adding this capability is planned for a future release.
The process to associate an automated test with a test case is:

1. Create a test project containing your automated test. What types of tests are supported?
2. Check your test project into a VSTS or Team Foundation Server (TFS ) repository.
3. Create a build definition for your project, ensuring that it contains the automated test. What are the
differences if I am still using a XAML build?
4. Use Visual Studio Enterprise, Visual Studio Professional, or Visual Studio Test Professional to associate the
automated test with a test case as shown below. The test case must have been added to a test plan that uses
the build you just defined.
If you are using Team Foundation Build and Release Management in VSTS or TFS (not a XAML build), you can
run associated tests in the Build and Release pipeline by using the Visual Studio Test task. You cannot run tests on-
demand using Microsoft Test Manager (MTM ) unless you are using a XAML build.
The parameters in a test case are not used by any automated test that you associate with a test case. Iterations of a
test case that use these parameters are for manual tests only.
For more information about checking in your test project and team build, see Add files to the server and
Continuous integration on any platform. For more information about action recordings and coded UI tests, see
Recording and Playing Back Manual Tests and Use UI Automation To Test Your Code.
Associate your test

1. Open your solution in Visual Studio.
2. If you don't know the identifier of the work item for the test case, locate the test case in the Test hub or
query for the work item in the Work hub.
3. When you know the identifier of the work item for the test case:
If you are using Visual Studio 2017 or later, follow these steps to associate your tests.
If the Test Explorer window is not displayed, open it from the Test | Windows menu.
If your tests are not displayed in Test Explorer, build the solution.
In Test Explorer, select the test method you want to associate and choose Associate to Test Case.
In the dialog that opens, type the test case identifier and choose Add Association, then choose
Save.
The dialog shows a list of test cases currently associated with the selected test method. You cannot
associate more than one test method with a test case, but you can associate a test method with more
than one test case.
If you are using Visual Studio 2015 or earlier, follow these steps to associate your tests.
In Team Explorer open the Work Items tab. If the Team Explorer window is not displayed, open it
from the View menu.
Expand the list of Queries in the Work Items tab to find one that displays your test cases, for
example the default My Test Cases query.
Execute the query by choosing View Results on the shortcut menu (or double-click the query
name).
Open the test case you want to associate by choosing Open on the shortcut menu (or double-click
the test case name).
In the work item, open the ASSOCIATED AUTOMATION tab. All the tests in the solution are
shown in the list together with their associated test projects.
Choose the ellipsis (...) and, in the Choose Test dialog, select the test and then choose OK. The value
in Automation Status is automatically changed to Automated.
Choose Save Work Item to save the changes to the test case.
If a test case already has an automated test associated with it, you must first remove this association
before you can add a different automated test. Choose Remove association to remove the existing
automation.
See Also
Associate automated test results with requirements
Run automated tests from test plans in the Test hub
Run automated tests with Microsoft Test Manager
Help and support

VSTS | TFS 2018 | TFS 2017.2

Automate test cases in your test plans and run them directly from the Test hub:
Provides a user-friendly process for testers who may not be well versed with running tests in Build or
Release workflows.
Gives you the flexibility to run selected tests on demand, rather than scheduled testing in Build or Release
workflows where all tests meeting the filter criteria are run.
Useful when you want to rerun a few tests that failed due to test infrastructure issues, or you have a new
build that includes fixes for failed tests.
You will need:
A test plan containing your automated tests, which you have associated with automated test methods using
Visual Studio 2017, or Visual Studio 2015 or earlier.
A Team Build definition that generates builds containing the test binaries.
The app to test. You can deploy the app as part of the build and release workflow and also use it for on-
demand testing.
Set up your environment

1. In the Test plans tab of the Test hub, choose your test plan, open the shortcut menu, and choose Test plan
settings.
2. In the Test plan settings dialog, select the build definition that generates builds which contain the test
binaries. You can then select a specific build number to test, or let the system automatically use the latest
build when tests are run.
3. You will need a release definition that was created from the Run automated tests from Test Manager
template to run tests from test plans in the Test hub. If you have an existing release definition that was
created using this template, select it and then select the existing environment in the release definition where
the tests will be executed. Otherwise, choose the Create new link in the dialog to create a new release
definition containing a single environment with the Visual Studio Test task already added.
How do I pass parameters to my test code from a build or release pipeline?
4. To configure the Visual Studio Test task and the release definition, start by assigning meaningful names to
the release definition and environment. Then select the Visual Studio Test task and configure it as follows:
Verify that version 2 of the Visual Studio Test task is selected. The version number is shown in the
drop-down list at the top left of the task settings panel.
Verify that Select tests using is set to Test run. What does this setting mean?
If you have UI tests that run on real browsers or thick clients, set (tick) the Test mix contains UI
tests checkbox. This is not required if you are running UI tests on a headless browser.
Select how is the test platform is provisioned, and the version of Visual Studio or the location of the
test platform that is installed on the test machines
If your tests need input parameters such as app URLs or database connection strings, select the
relevant settings file from the build artifacts. You can use the Publish build artifacts tasks in you
build definition to publish the settings file in a drop location if this file is not included in the artifacts.
In the example shown below, the application URL is exposed in the run settings file, and is
overridden to set it to a staging URL using the Override test run parameters setting.
For information about the option settings of the Visual Studio Test task, see Visual Studio Test task.
5. Choose the Agent phase item and verify that the deployment queue is set to the one containing the
machines where you want to run the tests. If your tests require special machines from the agent pool, you
can add demands that will select these at runtime.
You may be able to minimize test times by distributing tests across multiple agents by setting Parallelism
to Multiple executions and specifying the number of agents. More details.
Note: If you are running UI tests such as CodeUI or Selenium on physical browsers such as IE, Firefox,
or Chrome, the agent on the machines must be running in interactive mode and not as a service. More
details.
6. In the Pipeline tab of the release definition, verify that the build definition containing the test binaries is
linked to this release definition as an artifact source.
7. Save the release definition.

8. If you chose Create new in the Test plan settings dialog in step 2 of this example, return to the browser tab
containing your test plan settings. In the Test plan settings dialog, select the release definition and
environment you just saved.
Run the automated tests

1. In the Test hub, open the test plan and select a test suite that contains the automated tests.
2. Select the test(s) you want to run, open the Run menu, and choose Run test.
The test binaries for these tests must be available in the build artifacts generated by your build definition.
3. Choose OK to start the testing process. The system checks that only automated tests are selected (any
manual tests are ignored), validates the environment to ensure the Visual Studio Test task is present and
has valid settings, checks the user's permission to create a release for the selected release definition, creates
a test run, and then triggers the creation of a release to the selected environment.
4. Choose View test run to view the test progress and analyze the failed tests. Test results have the relevant
information for debugging failed tests such as the error message, stack trace, console logs, and attachments.
5. After test execution is complete, the Runs tab of the Test hub shows the test results. The Run summary
page shows an overview of the run.
There is a link to the Release used to run the tests, which makes it easy to find the release that ran the tests
if you need to come back later and analyze the results. Also use this link if you want to open the release to
view the release logs.
What are the typical error scenarios or issues I should look out for if my tests don't run?
6. The Test results page lists the results for each test in the test run. Select a test to see debugging
information for failed tests such as the error message, stack trace, console logs, and attachments.
7. Open the Test Plans page and select the test plan to see the status of your tests if tests are updated after
test execution is complete. Select a test to see the recent test results.
See Also
Help and support


If your test suites include requirements, link these to your test results and view the results on your team's
dashboard. This enables end-to-end traceability of requirements for agile teams. For example, when teams do not
use planned testing (by creating test plans or test case work items), and instead choose to simply write automated
tests that run in the CD/CI pipeline, associating test results with requirements provides an easy way to monitor
test results and ensure requirements are met.
To associate automated test results with requirements:
1. On the test results page, select the tests you want to link to requirements and choose the Associate tests
to work item (link) icon.
2. Select the requirements from the list of suggested work items and choose Associate.
3. To see the related test results, select Requirements in the Group by list.
4. On your team's dashboard, add the Requirements quality widget and configure it for the appropriate
build definition and work item query.
5. This shows the pass rate for each of your requirements. Use the links to view the results in more detail, and
the Expand link to see more.
See Also
Help and support

Explore work items with the Test & Feedback
extension
VSTS | TFS 2018 | TFS 2017 | TFS 2015

Use the Test & Feedback extension to explore existing work items and associate them with a new or an in-
progress exploratory session. After a work item is associated with a session, all new bugs, tasks and test cases
created in the current session are automatically linked to that work item. This enables end-to-end traceability, and
simplifies tracking and management of issues.
You can explore:
Work items belonging to a requirement category, a feature category, or an epic category
Requirements-based test suites and test cases.
You can explore a work item from the Kanban board or from the extension. You can also explore multiple work
items in the same session.
Explore work items from the Kanban board

1. In the Kanban board, open the shortcut menu of the work item you want to explore, and choose Do
exploratory testing.
2. A banner in the Work hub shows which work item is associated with your session.
3. Launch the Test & Feedback extension. If there are acceptance criteria for the work item, these are shown.
If you have not already started a session, start one now. The work item is automatically associated with the
current or new session.
4. All bugs, tasks, and test cases you create will automatically be linked to the current work item.
Explore work items from the Test & Feedback extension
1. Open the Explore work item page in the extension and search for the work item you want to explore.
You can search using the work item identifier or keywords in the work item title.
2. Select the work item in the search results and choose Explore selected work item.
3. The work item is now associated with the in-progress session. If there are acceptance criteria, these are
shown.
4. All bugs, tasks, and test cases you create will automatically be linked to the current work item.
Explore multiple work items in the same session
To explore another work item, you must first dissociate the current work item from the in-progress session.
1. Open the Explore work item page and choose Change.
2. Associate the new work item with the in-progress session as described above.
See Also
Add findings to existing bugs with exploratory testing
Get insights across your exploratory testing sessions
Use the Test & Feedback extension in Standalone mode
Exploratory testing with Microsoft Test Manager
Help and support

VSTS | TFS 2018 | TFS 2017 | TFS 2015

To help avoid duplication, the Test & Feedback extension automatically searches for and displays existing bugs,
based on the keywords in the title, as you file a new bug. You can choose to continue creating a new bug or add
your findings to an existing bug.
1. As you type the title for a new bug, in the background the extension searches for similar bugs that might be
related to the issue you've found and displays a link to the results. Choose this link to see the results that
have similar title keywords.
The form displays 0 Similar if it does not find any matching bugs. In this case, or if you don't see a "similar"
link, you can create a new bug containing your screenshots, notes, and videos as described in this topic.
2. If you see a bug you want to update, instead of creating a new one:
Select it in the list and choose Edit.
The extension appends all your screenshots, notes, and videos to the existing bug.
Save the updated bug.
3. If, instead, you decide not to update an existing bug, ignore the "similar" link and:
Choose the New bug link to return to the bug details form.
Enter the details for the new bug and save it as described in this topic.
4. Continue exploring your app, filing bugs and tasks, and creating test cases.
See Also
Explore work items with exploratory testing
Help and support

VSTS | TFS 2018 | TFS 2017

View completed exploratory testing sessions and derive meaningful insights at team or individual level, and for a
specific period.
1. Open the Recent exploratory sessions page. You can do this:

From the Test & Feedback extension by choosing the "view" icon on the Session timeline page.
From the VSTS or TFS Test hub by opening the Runs tab and choosing Recent exploratory
sessions.
2. Explore the Recent exploratory sessions page. It contains three main sections:
Summary view - shows a graphical breakdown of the work items explored, work items created, session
owners, and the total time for these sessions.
Pivot view - shows a collapsible nested and sortable list of items grouped in different ways.
Details view - shows the work item selected in the Pivot view or a summary of information about a
selection of items.
Get insights from your exploratory testing sessions:

Use the Recent exploratory sessions page to get insights about your app from the information collected during
your exploratory testing sessions.
1. Set the scope for the data. Summary view provides the highest level view of your test results. Use it to
get insights into the overall effort and results of the exploratory testing sessions.
Use the View filter to scope the summary view to all sessions or just your own sessions.
Use the Period filter to scope the summary view to sessions in the period from the last 7 to the last 90
days.
2. Pivot the data on the type of work item. Pivot view lets you to focus on all the work items you created
in your exploratory testing sessions, or just on bugs, tasks, or test cases; and group the results in different
ways.
Use the Pivot filter to group the work items as a nested list based on those that have been explored,
those that have not been explored (requires a query), by the session in which they were created, or by
session owner.
Use the Show filter to show all items; or just bugs, tasks, or test cases.
3. Get deep insights from Details view. Details view gives insights into the items selected in Pivot view.
Depending on the type of item you select, you see the work item as an editable form, or a series of charts.
Select a row in Pivot view to see a summary of all the related information in Details view. For example,
if you have pivoted the list based on sessions, select a session to see a summary of all the information
from the work items in just that session.
Select a child row in Pivot view to display the work item form for that individual item. For example, if
you have pivoted the list based on explored work items, expand a work item and select a child bug, task,
or test case to see the work item form for just that item.
Discover work items not yet explored
Use a query to explore the work items that users have not yet explored.
1. Create a shared query in VSTS or TFS that selects work items that can be explored using the Test &
Feedback extension, such as work items in the epic category, feature category, requirement category,
requirement-based suites, or test cases.
You must use a shared query. If this query returns a mix of supported and unsupported work items,
only those in supported categories will be displayed.
2. Use the View and Period filters to scope the view to the type of session (all sessions or just your own
sessions) and the time span (from the last 7 to the last 90 days). Then open the Query list and choose
Select query.
3. In the Query selector dialog, choose the shared query you created earlier.
4. View all the work items returned by the query in Summary view. You see a breakdown of explored and
unexplored work items, work items filed, sessions, and total session duration.
5. Open the Pivot list and choose Unexplored Work Item.
The view now shows only the unexplored work items.

See Also
Explore work items with exploratory testing
Help and support

Request stakeholder feedback using the Test &
Feedback extension
VSTS | TFS 2018 | TFS 2017

Stakeholders can respond to feedback requests for user stories and features generated in VSTS or TFS using a
lightweight end-to-end flow based on the Test & Feedback extension. Only users with Basic access can request
feedback. Basic users can provide feedback using the flow described in this topic.
Request feedback
Provide feedback
Voluntary feedback
Track requests
NOTE: This lightweight end-to-end flow is applicable only for web apps and by using VSTS or TFS 2017. To
get feedback for desktop apps, or for earlier versions of TFS, use the feedback flow described in this topic
about the Microsoft Feedback Client.
Request feedback from stakeholders

Request feedback from stakeholders directly from a VSTS or TFS work item.
1. Open the work item form for the user story or feature for which you want to request feedback.
2. Open the shortcut menu from the ellipses (...) and choose Request feedback.
3. Type or select the names of the stakeholder(s) you want to send the request to, and optionally add any
instructions or notes that will help them provide meaningful feedback.
4. Choose Send to generate emails to all the selected stakeholders.
NOTE
Teams can request feedback from other team members, such as users having Basic access. Just add their names in the
feedback request form so that a Request feedback email is sent to them. Also see Can users with Basic access respond to
feedback requests?.
See also
Provide stakeholder feedback using the Test & Feedback extension
Voluntary stakeholder feedback using the Test & Feedback extension
Track stakeholder feedback using the Test & Feedback extension
Help and support

Provide feedback using the Test & Feedback
extension
VSTS | TFS 2018 | TFS 2017

Request feedback
Provide feedback
Voluntary feedback
Track requests
get feedback for desktop apps, or for earlier versions of TFS, use the feedback flow described in this topic
about the Microsoft Feedback Client.
Stakeholders and other users can respond to feedback requests using the Test & Feedback extension in two ways:
From the link in a feedback request email
Directly from the Test & Feedback extension
Before you start, ensure you have installed the Test & Feedback extension. This is required in order to respond to
feedback requests.
Any user with a Stakeholder access can use the Test & Feedback extension in Stakeholder mode. This mode is
designed to allow the widest possible range of users to assist test teams by providing feedback.
Provide feedback directly from a feedback request email

1. Open the feedback request email and choose the Provide feedback link.
2. The VSTS or TFS landing page opens to confirm that the extension has been automatically configured with
the feedback request. Choose the icon in the toolbar to launch the extension.
If you are a Stakeholder, you see the Feedback requests page. Read the instructions (if any) in the
feedback form to understand how to give the feedback and what the requestor requires.
If you are a Basic user, you see the Explore work item traceability page showing details of the user story
on which feedback was requested, and the user acceptance criteria (if any).
3. Read any instructions in the email and this page to understand how to give the feedback, and on what
feature.
4. Open the application you need to provide feedback on and begin your feedback. For example, choose
Capture screenshot to take a screenshot.
You can use all the capabilities of the extension such as capturing screenshots, notes, and screen recordings.
For more details, see this topic.
Some browsers may not provide all the capture capabilities. See Which web browsers does the
extension support?
5. When you are done capturing feedback:

If you are a Stakeholder, choose Provide feedback. You can optionally choose to create bugs and
tasks when you submit your feedback. The process is the same as described in this topic.
If you are a Basic user, create a bug or a task.
6. All your feedback captured is shown in the response form, bug, or task. Type a suitable title and, optionally,
select a star rating for the feature you've been testing.
7. Save your feedback. This create a work item in VSTS or TFS containing all your feedback.
8. Continue to capture more feedback if required. You can submit multiple feedback responses, bugs, and tasks
for the same feedback request.
9. If you are a Stakeholder:
When you are done providing feedback, go to the Feedback requests page and choose Feedback
requests.
In the pending feedback requests page, mark the feedback request as Completed.
10. Choose the Stop icon to end your feedback session.
Provide feedback directly from the Test & Feedback extension

1. Open the Test & Feedback extension in your browser using the icon in the toolbar.
2. In the Connection settings page, choose Connected mode.
3. Connect to the server and the project or team that is requesting feedback.
4. Open the Feedback requests page to see all your feedback requests from the project or team you
connected to.
5. Select the feedback request you want to respond to and choose View feedback.
6. Read the instructions in the feedback request details page, then choose Provide feedback.
7. Capture and submit your feedback as shown above.
See also
Request stakeholder feedback using the Test & Feedback extension
Help and support

Voluntarily provide stakeholder feedback using the
VSTS | TFS 2018 | TFS 2017

Request feedback
Provide feedback
Voluntary feedback
Track requests
get feedback for desktop apps, or for earlier versions of TFS, use the feedback flow described in this topic about
the Microsoft Feedback Client.
Provide voluntary feedback

You can use the Test & Feedback extension to provide feedback voluntarily, even if you haven't received a specific
feedback request.
1. Open the Test & Feedback extension in your browser using the icon in the toolbar.
2. In the Connection settings page, choose Connected mode.
3. Connect to the server and the project or team that is requesting feedback.
4. Start the exploratory testing session.
5. Open the application you want to provide feedback on and begin your feedback. For example, choose
Capture screenshot to take a screenshot.
You can use all the capabilities of the extension such as capturing screenshots, notes, and screen recordings.
Some browsers may not provide all of the capture capabilities. See Which web browsers does the
extension support?
6. When you are done capturing feedback, Choose Provide feedback.
You can optionally choose to create bugs and tasks when you submit your feedback. The process is the same
as described here.
7. All your feedback captured is shown in the response form. Type a suitable title and, optionally, select a star
rating for the feature you've been testing.
8. Save your feedback. This create a work item in VSTS or TFS containing all your feedback.
9. Continue to capture more feedback if required. You can submit multiple feedback responses, bugs, and tasks
for the same feedback request.
10. Choose the Stop icon to end your feedback session.
See also
Help and support

Track stakeholder feedback using the Test & Feedback
extension
VSTS | TFS 2018 | TFS 2017

Request feedback
Provide feedback
Voluntary feedback
Track requests
get feedback for desktop apps, or for earlier versions of TFS, use the feedback flow described in this topic about
the Microsoft Feedback Client.
Track feedback requests

1. In VSTS or TFS, select your project and open the Queries tab of the Work hub.
2. In the list of shared queries, select Feedback. This query displays a list of all the feedback responses
received.
3. Open the response work item to see the details of the feedback.
See also
Help and support

Get app performance data with your load tests

When you load test your app in the cloud using Visual Studio Team Services (VSTS ), you can compare app
performance with virtual user load using Application Insights. Then, by doing a quick root cause analysis, you can
figure out which code is causing performance problems.
1. Download and install Visual Studio Enterprise, if you haven't already done so.
2. Enable Azure Active Directory for your VSTS account, if you haven't already done so.
3. Link your VSTS account with your Azure subscription, if you haven't already done so.
4. Sign in to your VSTS account from your web browser to refresh the Azure Resources Manager access
token. The token is valid for 12 hours in the context of VSTS.
If you have already signed, you must sign out and then sign in again.
5. Set up your load test project to run in the cloud, if you haven't already done so.
6. With your load test project open in Visual Studio Enterprise, open the Run Settings section and select your
active run settings. Open the shortcut menu and choose Get Performance Data from Application
Insights.
7. Select the apps you want to monitor and the performance counters you want to view while your load test
runs.
The counters you selected are shown in the load test project.
8. Queue a load test run and view the performance data from Application Insights while your load test runs.
The data might take a few minutes to appear.
Application counters are correlated with user load so that you can understand which issues might cause
performance problems that you find.
The counter samples have a sampling rate of one minute irrespective of the sampling rate configured in
your load test project.
9. To do a more detailed analysis for any performance issue, or to do a quick root cause analysis, go to
Application Insights.
See also
FAQs for load testing
Load test with VSTS
Tutorial: Run load tests before release
Analyze load test results using the Load Test Analyzer
Help and support

VSTS
Before you start:
Create your Visual Studio Team Services (VSTS ) account, if you don't have one already.
To run a JMeter load test:
2. Go to the Load Test hub, open the + New menu and choose Apache JMeter test.
3. Enter your load test parameters. To run your test near to where your users are located, select a closer
location for your load test. Then start your test when you're ready.
For information about the scripts and supporting files used for JMeter web tests, see Build a Web Test
Plan on the Apache JMeter website.
4. As the test runs, you see live information about the progress of the test. You can stop the test by using the
Abort link on the toolbar.
5. When your test is done, look at the results to see how well your app performed. For example, you can see
an overview of your app's performance in the Summary tab. This tab shows all of the main metrics such as
average response time, user load, requests per second, failed requests, any errors that might have occurred,
and test usage.
The lower section of the Summary tab shows the settings used for the test, and details of the five slowest
requests during the test. If there are any transaction tests, the tab will also show the five slowest of these.
Use the icon above a column to sort the list based on the contents of that column.
6. Open the Charts tab to see a graphical representation of the test results over time. The charts show the
average performance, throughput, errors, and the results of each test request. Hover your mouse pointer
over a chart to see more details.
7. Open the Diagnostics tab to see detailed information such as a list of errors and status messages.
You can also use the icon in the Errors section of the Summary tab to go directly to the Diagnostics
tab.
8. Open the Logs tab to see a list of test runs. Choose the link in the Attachment column to download the
detailed log as a text file.
9. If you have a favorite listener that you use to analyze results in the JMeter IDE, download the test results in
.CSV format and the logs as a zip file from the Download Results link.
10. To run the same test again, choose Rerun.
11. Now see how you can view and compare your load test runs.
See also
Load test with VSTS
Help and support

Record and replay cloud-based load tests
VSTS
You can record and then replay cloud-based load tests on your web app or website directly using an HTTP Archive
file and Visual Studio Team Services (VSTS ).
Before you start:
Create a VSTS account, if you don't have one already.
You can use your monthly free 20,000 virtual user minutes (VUM ) allowance to try it. If you want to use
load testing beyond this, you can set up billing for your VSTS account.
About HTTP Archive testing

If you have used cloud-based load testing before, you may be familiar with the ability to get performance
information on your app when under load by using the VSTS web portal. It's a great way to:
Quickly run a URL -based load test for your app; you just need to enter the app URL.
Easily learn about the basic load test features such as running tests from different locations around the world,
viewing and analyzing results in the browser, analyzing the test errors, and more.
At the other end of the spectrum is using Visual Studio Enterprise to create and run load tests. While you can
certainly create a URL -based load test using Visual Studio Enterprise, the real advantage is the ability to write tests
that mimic several end-to-end user scenarios. These may include a distributed mix of tests, the ability to use data
to drive your test, and even use of the rich extensible framework to create plugins necessary to suit complex
testing requirements.
HTTP Archive testing also allows you to create load tests that mimic end-to-end user scenarios, but with fewer
capabilties than the rich Visual Studio Enterprise load tests. However, by using HTTP Archive files you can
represent user scenarios that:
Support multiple URLs
Send requests other than just a simple GET
Simulate a complete user scenario
If you develop or test web apps, you may already be familiar with the F12 developer tools in your browser. These
enable you carry out a range of tasks from inspecting the HTML, CSS, and JavaScript, to viewing the traffic
between the browser and the server. You can use this ability to view traffic between the browser and the server to
create an HTTP Archive load test for your scenario.
The example shown here uses the Google Chrome browser, but you can use the browser of your choice such
as Edge, Internet Explorer, or Firefox. For example, this video shows how to create an HTTP Archive with
different browsers and with Fiddler.
Create an HTTP Archive file

Follow these steps to create an HTTP Archive (.har) file using your browser.
1. Launch the browser and press F12 to open the developer tools.
2. Ensure that the browser traffic is being recorded. For example, if you are using Chrome, check that:
The recording button is "on" (the round button is red).
The Preserve log checkbox is set to ensure that the complete sequence of URLs for your scenario is
captured. If you do not set this option, network activity recordings are discarded whenever you
reload the current page or load a different page - which means that an end-to-end scenario may not
be captured.
3. Exercise your user scenario. Enter the URL in the address bar and go through the sequence of actions a
typical user would follow. For example, in a retail store site, the user would browse for a product, enter a
product name in the search box, choose links of interest, view the product information, and (hopefully) place
an order.
As you go through these steps, you will see that all the traffic between the browser and the server is
shown in the Network tab of the developer tools.
4. After you have finished recording your user scenario, save the URLs as HTTP Archive (.har) file. In Chrome,
do this by opening the shortcut menu for the URL list and choosing Save as HAR with content. Enter a
name for the file and save it on your computer.
You can also create an HTTP Archive using tools other than a browser. For example, Fiddler is a popular
tool for viewing and troubleshooting traffic. Use Fiddler to record traffic in the same way and export
sessions as an HTTP Archive.
Create a load test using the HTTP Archive file
Follow these steps to create a load test in the VSTS web portal using an HTTP Archive (.har) file.
2. Go to the Load Test hub, open the + New menu and choose HTTP Archive based test.
3. In the Import HTTP Archive file dialog, select the .har file that you recorded in the earlier steps and
choose OK.
The HTTP Archive file is uploaded and read, the URLs are extracted, and all the details (the HTTP method,
headers, querystring parameters, form post parameters, and more) are displayed. You can add URLs and
edit the existing URLs to provide a friendly name for each request in the center Add URL pane. This will
help you easily identify each request.
The right-hand section of this page shows details of the selected request URL.
Some dynamic information such as ASP.NET viewstate, event validation, and more that varies from
session to session, and must be retained in order for a sequence of requests to succeed. These values
are automatically identified and extracted from a request and correlated in any subsequent requests
that use them.
4. Open the Settings tab to view and change any load test settings.
5. Enter a name for your load test, then choose Save.
6. Choose the Run test link to execute your load test. A progress bar shows the current status of the test run.
7. As the load test runs, you see a rich set of metrics that indicate how your app is performing under load.
When the test is complete, you can explore the results.
For more information about the results and reports, see URL -based load testing with VSTS.
Troubleshooting
As you run a load test, requests are sent to your application in the sequence you specified. Ideally, you want a clean
run with all requests succeeding. However, requests may fail because of issues in your app or in your test. The load
test results view offers two sections to help you identify these errors: Diagnostics and Logs.
The Diagnostics section provides insights into any test errors and important status messages from the load test
service that occurred during the load test run. For failing requests, you can see the error type, the specific error
subtype, the number of times the failure occurred, and more.
This page also lists the requests that were tested, and the test you executed.
The Logs section gives you access to logs from all of the load-generating agents. If requests in your test are
failing, these logs will help you figure out what went wrong. The test logs are available in HTTP Archive (.har)
format, the same format as you used to record the user scenario and create the test. You can download these logs
and view them in a HAR viewer such as Fiddler, HAR Analyzer, or any other viewer that you prefer. You can then
inspect the details of each request and its response in order to troubleshoot any failures in your test.
This screenshot shows how you can import the log into Fiddler.
Export the test to Visual Studio
Sometimes, one or more requests will fail every time they are executed. If you have eliminated the obvious causes
such as correctness of the URL, the target URL not being reachable, or any application issues, requests may be
failing because the application uses dynamic parameters (such as session ID, cookies, values set in hidden fields
such as ASP.NET VIEWSTATE, and others) in these requests. When you create a test using an HTTP Archive, the
following cases are handled automatically when the test is executed:
Dynamic parameter values set in cookies.
Dynamic parameter values set in hidden fields, such as ASP.NET VIEWSTATE and EVENTVALIDATION.
However, dynamic parameters may also appear elsewhere in requests such as query strings or form post
collections. At present, the load test mechanism does not support these types of dynamic parameters. If you find
that this is the cause of your test failures, you can export and run the test in Visual Studio.
This export mechanism downloads a Visual Studio load test project containing the required web performance test
and load test for your application. See how to fix dynamic parameters using Visual Studio. Sean Lumley's blog
post has a detailed example of how dynamic parameters can be identified by inspecting the test and test results.
See also
Help and support
Test private and intranet apps using cloud-based load
testing
VSTS
The Cloud-based Load Testing (CLT) service can be used for performance and scale testing of an app by
generating load from Azure. This type of load generation can only access and generate load on an Internet-located
or publicly accessible app. However, you may want to load test an app which is not publicly accessible, perhaps to:
Test an app that runs only on an internal network. In many large-scale organizations there are apps and
websites that cater for the needs of the whole organization and it is crucial to test it with peak load to discover
any performance or stress-related issues.
Test an app internally before releasing it on the Internet Before going public, organizations typically want
to ensure there are no performance issues that may affect the app under high user load.
Consider the following decision tree that shows six possible scenarios. This topic discusses only scenarios 3 and 4.
1. The default case; CLT auto-provisions agents. This is the default scenario for load testing using CLT when
the app has a publicly-available endpoint. The load testing service automatically provisions load agents (in
Azure) to simulate the user load. For more details, see Run URL -based load tests with VSTS.
2. Use an ARM template to deploy your own load agents. Solution 1 above provisions the agents within the
CLT service boundaries, so you don't have full control. If you want to access and control the load generation
agents, you can deploy them in your own Azure subscription using an ARM template. These machines will be
registered with the CLT service and can generate load. For more details, see Structure and syntax of ARM
templates.
3. Use an ARM template to deploy load agents in a VNet. If the app under test is within an Azure VNet, or if
there is ExpressRoute connectivity between the app's private network and Azure, you can use a pre-defined
ARM template deploy IaaS VMs in Azure in a specific VNet to act and have these VMs act as load agents. The
machines will be provisioned in your Azure subscription and registered against your VSTS account. The VNet
where you create these machines must have line-of-sight to the app so that the load generators can reach it.
4. Use an ARM template to deploy load agents with static IPs. If you don't have ExpressRoute connectivity
and want to test apps hosted on-premises, you can use an ARM template to deploy IaaS VMs in Azure that act
as load agents. Create these VMs with static IP addresses for which you can configure your firewall to allow
inbound traffic from the CLT service. The machines will be provisioned in your Azure subscription and
registered against your VSTS account.
5. Use cloud load agents on your own infrastructure. A simple PowerShell script can help you configure
physical or virtual machines as load agents. These machines are registered against your own VSTS account and
used for load generation. For more details, see Run cloud-based load tests using your own machines.
6. Use the Test Controller and Test Agents for on-premises testing on your own infrastructure. If you
want to test apps on-premises and have constraints such as not being able to store results in the cloud (perhaps
for regulatory compliance) you can use the Test Controller and Test Agents combination for load testing. This
requires you to use your own infrastructure for load generation and the results are stored in SQL Server. See
Configure test agents and test controllers for running load tests for details.
The following sections describe how you can provision load agents using Azure IaaS VMs (you will need an Azure
subscription). This approach is primarily useful when:
You want to test a private app that is not accessible through the CLT service or over the Internet.
You have your own Azure subscription and you want to use it for load testing. You can also use any Azure free
credits you may have.
Azure also allows you to spread the load testing across different geographical locations to measure response times
from different locations.
Use an ARM template to deploy load agents in a VNet

The following schematic shows the simple topology where load agents reside in a user's VNet and therefore have
line-of-sight to the app.
Use this ARM template on GitHub to provision machines easily and quickly.
Alternatively, providing you have an existing VNet, you can automatically provision load agents into it. This link
loads the template in the Azure portal and shows the following page.
You must then fill in the parameters and choose the subscription, resource group, and location to suit your
scenario. VNet identification requires the resource group name.
Use an ARM template to deploy load agents with static IP addresses

You can use this ARM template if you don't want to use a VNet. This may be because:
You don't have ExpressRoute in Azure but want to perform load testing using your own subscription. This ARM
template deploys a rig with its own VNet. If you need to test a private app, you can deploy the rig with static IP
addresses (provided as an option), then open the firewall for these IPs to enable routing for load agents.
You want to be able to control the load generation agents (CLT auto-provisioned agents can't be accessed by
the user). You can choose to have static or dynamic IP addresses for these VMs.
Alternatively, you can automatically provision load agents in a new VNet. This link loads the template in the Azure
portal and shows the following page.
Notes on deploying agents to VMs

After you deploy the VMs, it may take 10-15 minutes before the machines are configured with CLT and ready for a
load test. The load test runs on these agents and you will not be charged VUMs by the CLT service, but you will
incur the cost of Azure resources consumed under your subscription.
All the VMs in a resource group are registered under an agent group whose name is same as the resource group
name. In order to queue the run on a particular set of agents, you must specify the agent group name when
queuing the run. If you deployed your agents and resource groups before December 14th 2016, all the old agents
are available under an agent group named default. We recommend you reconfigure any such agents using the
latest script.
If you want to have your agents separated based on some configuration (for example, location, VNet, or capacity)
you should locate these agents in separate resource groups. This allows isolated runs and easier management of
machines and agents.
In the ARM templates, the machine size to Standard_D4_V2. This size provides 8 CPU cores and 28 GB of
memory. You can change this by editing the template. See Sizes for Windows virtual machines in Azure.
Queue a run using load agents

You can queue a load test run using the VSTS portal or Visual Studio Enterprise
Using the VSTS portal
Open the Load test page Settings tab and select Use self-provisioned agents. Then select the test rig you have
configured with the ARM template, and optionally the number of agents to use.
Using Visual Studio Enterprise IDE

Add the following context parameters to your Visual Studio Load Test file:
Context parameter name: UseStaticLoadAgents
Context parameter value: true
Context parameter name: StaticAgentsGroupName
Context parameter value:
You can specify the number of machines to be used for a load test run by setting the Agent core count property
present in the Run Settings. Every core is treated as a single machine. For example, shown below, five machines
will be used for the run.
Load test runs performed on your own load agent machines are not charged. You can confirm this by looking at
the status messages for the run.
Check the FAQs at the end of this topic for more details.
Manage self-provisioned agents

You can use this PowerShell script to manage self-provisioned agents. Download the script and unblock the file
before use.
Script parameters
TeamServicesAccountName. The name of your VSTS account you want to manage. Specify just the account
name. For example, if your VSTS account is xyz.visualstudio.com, enter just xyz.
PATToken. Required for authentication. Obtain a PAT token for your VSTS account as described here. Ensure
the selected scope is Load Test (read and write).
The available operations and the switches for the script are:
Get agent groups. Lists all the registered agent groups within the VSTS account. Example:
.\ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://xyz.visualstudio.com

-PATToken olxpldk2...wi3rbaq -GetAgentGroups
Get agents. Lists all the agents and their current status for a specified agent group. Example:

-PATToken olxpldk2...wi3rbaq -AgentGroupName test -GetAgents
Delete an agent. Deletes the agent reference from the service. The agent must be in the offline state. The
agent group name is mandatory. Example:

-PATToken olxpldk2...wi3rbaq -DeleteAgent -AgentGroupName test -AgentName dpk-param
Get more help using get-help on the script ManageVSTSCloudLoadAgent.ps1.
FAQ
Q: How do the load agents communicate with CLT?
A: The load agents communicate with CLT using the HTTPS protocol. As these machines or VMs are inside the
user's private network (Azure or on-premises), they can reach the app under test directly. The results are pushed
back to the CLT service so that analysis occurs in similar way to other types of load test runs.
Q: How I am charged for this?
A: You will not incur load testing VUM charges for the runs where you deploy load agents on your premises or in
your own Azure subscription. However, you will be charged the applicable Azure VM costs.
Q: Can I use these machines for other purposes?
A: These machines can be used for other tasks as well, but we recommended not having anything running while a
load test run is in progress.
Q: Can I shut down the machines where I have configured the load test agent?
A: Yes, the machines can be shut down when not in use. The load test agent service will automatically start to
receive commands from CLT after the machine is restarted. If you are using the Azure ARM template to deploy
these agents, you can start or stop the VMs as required. You can also do this using a PowerShell script. See Stop
All VMs in Specified Azure Resource Group. You should delete the Azure resource group after you are done with
load testing, and re-create it later if required. See Manage Azure resources through portal
Q: I have proxy settings on my machines, will this work?
A: We support only default proxy scenario, when the proxy settings are controlled through your web browser and
it uses the current user's credentials to connect to the proxy server. In other cases, please see our help and support
forum.
Q: Where can I find the log files to debug issues?
A: The PowerShell script logs are stored in the logs subfolder where the PowerShell script resides. These logs are
also displayed in the PowerShell window. Run execution logs are in the %windir%\temp\CloudLoadTest\logs
folder.
Q: How can I discover the outgoing URLs so that I can allow them in my firewall settings?
A: There is a REST API for this. You first need to get the target AgentGroup Id using the REST API:
https://<VSTS account name>.vsclt.visualstudio.com/_apis/clt/agentgroups
Then use this to get the list of outgoing URLs:

https://<VSTS account name>.vsclt.visualstudio.com/_apis/clt/agentgroups?agentGroupId=<Agent Group
Id>&outgoingRequestUrls=true
The output is a list of strings where the first two inputs are the Azure Blob and Table service URLs. Another
outgoing URL is your Visual Studio account URL. Other than these three URLs, you also need to allow the URL
https://<VSTS account name>.vsclt.visualstudio.com .
See also
Load test with VSTS
Help and support

Run cloud-based load tests using your own machines
VSTS
When you run a cloud-based load test, the Cloud Load Test (CLT) service automatically provisions the necessary
machines (load agents) to generate the load on your application. After the load test run has completed, these
resources are removed. However, you can run load tests using your own machines, such as virtual machines in
Azure provisioned in your own subscription, or other virtual or physical machines located on-premises. This topic
shows two primary scenarios where such a configuration may be useful.
Terminology
Auto-provisioned machines. These are load-generating machines that are automatically provisioned by the
CLT service when a load test run request is received, and are also automatically removed when the load test run
has completed execution. When these machines are used, you are charged VUMs as applicable for your load
test run.
Self-provisioned machines. These are load-generating machines that you provision yourself (in your Azure
subscription or on-premises). These machines can be configured to register themselves against your VSTS
account and then they can run a cloud-based load test. This scenario is the focus of this topic.
Cloud load agent. This is the agent that works with the CLT service. This agent will be installed when you use
self-provisioned machines, and must be configured for your VSTS account. Once configured, it can then be
used for running a load test. The cloud-load agent is NOT the same as the Test Controller and Test Agent
combination that you may have used previously for running load tests or automated tests.
Scenarios
More control over agent machines. Sometimes you may need more control over the agent machines; for
example, to install custom software used during a load test run. Simple configuration is easily achieved using
deployment items and setup scripts on auto-provisioned agents. However, if you are installing some bulky
software or doing some time-consuming operation as part of the setup, you may want to do that only once and
reuse the machines over and over again. In addition, using your own machines means that you can set up and
remove the agents only when required, as opposed to the auto-provisioned agents that are removed
automatically after a test run,
Testing private apps and apps behind the firewall. The basic requirement of the cloud load testing service
is that the application endpoint is public or reachable from the cloud. Often this is not the case. The app that
you want to load test may be on-premises behind the firewall, or in a private VNet in Azure. Or you may be
developing new features that will only become publicly accessible when it's released. In this case, you need to
provision agents in the same network as your app (so that they can reach the app) and have them work with the
CLT service so that you can easily load test your app.
Solutions
Use machines located on-premises. You can provision as many machines as you need on-premises and run
a PowerShell script that installs the cloud load agents and configures these machines against your VSTS
account. For more details and to obtain the PowerShell script, see Testing private and intranet apps using cloud-
based load testing. The agents communicate with the CLT service only using HTTP (S ), so you don't need to
open any other ports.
Use virtual machines in Azure. While you can certainly adopt the same approach as above by provisioning
the machines and then running the PowerShell script to install and configure the load agents, a simpler
approach is to use Azure Resource Manager (ARM ) templates. You specify just a few inputs such as your VSTS
account, a PAT token for authentication, and the number of agent machines you need. The machines are
provisioned in your Azure subscription and you have complete control over them. Two ARM templates are
available in the Azure quick-start templates repository on GitHub:
A simple template with a dynamic IP option. This template provisions the number of machines you
specify and assigns them dynamic IP addresses. In this configuration, the application will still need to be
publicly reachable. You can install any software you may need after the machines are provisioned and
are ready, or you can customize the ARM template to install the necessary software as part of the
provisioning process.
A simple template with a static IP option. This template provisions the number of machines you specify
and assigns them static IP addresses. The machines receive the same addresses even after you shut
down and restart them later. In this configuration, you can allow traffic from these known IP addresses
through the firewall to reach an application behind it. The agents communicate with the CLT service
using only HTTP (S ), so you don't need to open any other ports.
A VNet-based ARM template. This template provisions the number of machines you specify in a specific
VNet that you have already set up in Azure. This VNet will be where your private app is hosted, and so
the load agents have a line-of-sight to reach your app.
For more information about using the ARM templates, see Test private and intranet applications using cloud-
based load testing.
Comparison with Test Controllers and Test Agents

If you have previously used Visual Studio load testing (rather than cloud load testing), you will have used Test
Controllers and Test Agents (TC/TA) to run load tests. The differences between these and cloud load test agents
are:
The cloud load agent does not need a controller. The CLT service acts as a lightweight controller instead.
The cloud load agent when running on self-provisioned machines uses the CLT service to store results and
benefits from ongoing enhancements to the CLT service (such as viewing the load test report in a browser). The
Test Controller and Test Agent combination uses SQL Server to store results.
The cloud load agent uses HTTP (S ) to communicate with the CLT service, and is quite resilient to network
issues whereas the communication between Test Controller and Test Agent uses .NET remoting - which can be
susceptible to network issues. The Test Agent also generates much more network traffic than the cloud-load
agent.
FAQ
Q: How do I run JMeter tests using the self-provisioned option?
A: The cloud load agent is designed to also run JMeter tests. Use of self-provisioned agents for JMeter load tests
instead of the default auto-provisioned agent is under development.
Q: What is the cost of using self-provisioned agents? Will I be charged VUMs?
A: No. If you use self-provisioned agents on-premises, you bear the hardware cost. If you use self-provisioned
agents in Azure, the machines will be provisioned in your Azure subscription and you will be charged based on the
number of machines and the duration for which these VMs are running. CLT will not levy any VUM charges.
Q: Can I use the self-provisioned option when running load tests in a CI pipeline?
A: Yes. You can use the existing Azure Resource Group tasks from the build and release catalog with either of the
ARM templates to provision, start, and stop machines. The CLT tasks let you specify whether to use the auto-
provisioned agents or self-provisioned agents. A build template to help with this configuration is under
development.
Q: How do I automate the process of provisioning machines and shutting them down?
A: You can do this with using a build definition. In this context, think of the build (CI pipeline) as an automation
orchestrator rather than a system that does build. A build template to help with this configuration is under
development.
See also
Load test with VSTS
Help and support

Install certificates and custom software on agent
machines
VSTS
In some test scenarios you might need to set up the environment for the test, such as installing certificates or
custom software, and then clean up the environment afterwards (such as removing temporary files or folders
created during test execution). To do this you can include artifact deployments, setup, and cleanup scripts in the Test
Settings of your test solution.
Some examples are:
Install client certificates on agent machines; for example, when you are using HTTP client authentication the
web server authenticates the client using the client's public key certificate. You can use a setup script to
install the relevant certificate on the agent machines that run your load tests.
Install an Azure Management Certificate; for example, if your unit tests must perform operations such as
creating a new storage account or deploying an Azure website by using Windows Azure Management API.
You can use a setup script to install the certificate.
Install software on the machines running the load test agents in the cloud to collect data or metrics; for
example, installing Network Monitor to capture network traffic statistics. You can use a setup script to install
it on the agent machine and use a cleanup script to save the data to remote storage such as Azure Storage
(by using a SAS key).
Change settings on the agent machine before and after running the test; for example, you can use a setup
script and a cleanup script to modify and reset values such as registry entries or other settings as required.
Add certificates and scripts to deploy

1. Double-click the active test settings (such as Local.testsettings) to open Test Settings dialog.
2. Select the Deployment tab and set the Enable deployment checkbox.
3. Choose Add File, browse to the location of your certificate, and add it to the deployment items list.
4. Select the Setup and Cleanup Scripts tab in left-hand navigation bar.
5. Choose the ellipsis (...), browse to the location of the file or other artifact you want to deploy, and add it to
the deployment items list.
6. Choose Apply and then Close.

Deployment Items are in a folder named DeploymentDirectory, which can be accessed through the environment
variable %DeploymentDirectory%
Examples of setup scripts

Script to install a certificate into the Trusted Root Certification Authorities certificate store on the test computer.
This assumes you have added the Certificate Manager Tool CertMgr.exe to the deployment list:
%DeploymentDirectory%/CertMgr.exe -add -c %DeploymentDirectory%\mycertificate.cer -s -r localMachine root
Script to import a certificate into the Trusted Root Certification Authorities certificate store on the test computer:
certutil.exe -f -user -p password -importpfx %DeploymentDirectory%\mycertitficate.pfx NoRoot
See CertMgr and Certutil for more information about using these utilities.
You can use the deployment options and a setup script to add .exe files or other files you want to deploy to the
machines running the agent, and use a setup script to install software on these machines. For example, a script to
install Web Deploy on an agent machine that runs load tests in the cloud, assuming you have added
WebDeploy_x64_en-US.msi to the deployment list:
%DeploymentDirectory%\WebDeploy_x64_en-US.msi /passive.
See also
Load test with VSTS
Help and support

Guidance on Microsoft Test Manager usage
VSTS | TFS 2018 | TFS 2017

Visual Studio Team Services (VSTS ) and Team Foundation Server (TFS ) offer both web-based and client-based
solutions for manual testing:
The Test Center in Microsoft Test Manager (MTM ) client is a desktop-based manual testing solution, which has
traditionally been used by testers for their Manual testing needs (see Run manual tests with Microsoft Test
Manager).
The Test hub in VSTS and TFS is a web-based manual testing solution, which works across all platforms and
with all browsers. We have invested in Test hub over past two years to provide you better experiences across
Plan, Author, Execute and Track phases of Manual testing.
Because the Test hub is a fully featured Test management solution which works across all platforms and with
all browsers, we recommend you use the Test hub over Microsoft Test Manager for all your test management
requirements. You can use Microsoft Test Manager to test your desktop applications by launching the
Microsoft Test Runner (client) from the Test hub.
This topic will help you understand why the Test hub is a more comprehensive solution for manual testing
compared to Microsoft Test Manager.
Manual Testing with the Test hub

The Test hub in VSTS and TFS is a fully-featured test management solution spanning all phases of the testing
lifecycle. The Test hub works on all platforms (such as Linux, macOS, Windows, and others) and all browsers (such
as Edge, Chrome, Firefox, and others). You can easily get started with using manual testing features right from
your Kanban board, and use the Test hub for more advanced manual testing capabilities. This topic shows new
capabilities introduced in the Test hub.
Test planning
Create and manage test plans and test suites for your teams with ease with Test hub. Export and share the test
plans and test suites with your team or assign multiple testers to execute tests. See below the comparison matrix to
know more about the features introduced in Test hub.
Comparison of test planning with the Test hub and test planning with MTM:
TEST PLANNING CAPABILITY WEB-BASED TEST HUB CLIENT-BASED MTM
Create test plan
Create/Manage suites
Add/remove tests from test suite

TEST PLANNING CAPABILITY WEB-BASED TEST HUB CLIENT-BASED MTM
Assign individual testers to test

plan/test suite
Create/edit/assign configurations
Clone test plan/test suite*
Add tests from other test suites*
Order manual tests within suites (RBS,

QBS, Static)
Export test plans and test suites
View test case references across test

suites
Assign multiple testers to test plans and

test suites for user acceptance testing
* These capabilities are included as part of the upcoming version of the Test Case Explorer extension available from
Visual Studio Marketplace.
Test authoring
You can easily get started creating test cases right from your Kanban board in the Work hub. Easily add, view,
interact with, and execute tests from your Kanban cards, and create multiple test cases using a grid in the Test hub.
Create shared parameters and use them in multiple test cases for data driven testing.
Comparison of test authoring with the Test hub and test authoring with MTM:
TEST AUTHORING CAPABILITY WEB-BASED TEST HUB CLIENT-BASED MTM
Create and edit test cases using WIT

form
Create and edit shared steps
Bulk author and edit test cases
Inline tests on Kanban board
Create and edit shared parameters
Test execution
Test your web apps and your desktop apps.
Web apps
The Test hub provides a browser based Test Runner you can use to test your web apps; for example, by marking
test steps and test outcomes as pass or fail, and collecting diagnostic information such as system information,
image action logs, screen recordings and screen captures during your tests.
Desktop apps
You can use the browser based Test Runner for running tests on desktop apps if you only want to mark test steps
and test outcomes as pass or fail, or collect screen recordings during your tests. If you need other data collection
capabilities when testing desktop apps, you can use the Microsoft Test Runner client that is part of Microsoft Test
Manager. You can launch Microsoft Test Runner client directly from the Test hub.
Comparison of test execution with web based Test Runner and test execution with Microsoft Test
Runner (client):
WEB-BASED TEST RUNNER FOR TESTING CLIENT-BASED TEST RUNNER FOR TESTING
TEST EXECUTION CAPABILITY WEB APPS DESKTOP APPS
Bulk mark tests without opening in Test

Runner
Pass or fail tests or test steps using Test

Runner
Inline changes to tests during execution
Pause and resume tests
File bugs during test execution
Capture screenshots, image action log,

and screen recording during test
execution
Update existing bugs during test

execution
Verify bugs
Assign a Build for the test run
Assign test settings
Fast-forward steps using action

recording*
Collect advanced diagnostic data during

manual testing*
Connect to a machine in an
environment
* The web-based Test Runner currently does not support Action Recording (fast-forward test steps) or Advanced
Data collection (code coverage, IntelliTrace, and test impact) during your tests. You can use the Microsoft Test
Runner client, launched from the Test hub, for these requirements.
Test tracking
You can easily track your manual testing results using your chosen light-weight chart types, and pin them to your
dashboard to quickly analyze the test results. View test case result history across test suites and test plans easily by
using the right-hand pane in the Test plans tab of the Test hub. You can also select a retention policy to specify
how long you want to keep your manual testing results.
Comparison of test result tracking with the Test hub and test result tracking with MTM:
TEST TRACKING CAPABILITY WEB-BASED TEST HUB CLIENT-BASED MTM
Test run and results analysis
Create, configure, and pin light-weight

test result charts
Test run and results retention policy
View test results history across test

suites and test plans
Exploratory Testing
Use the light weight Test & Feedback browser extension to perform exploratory testing on your web applications.
You can collect rich diagnostic data such as screen captures, screen recording, and image action logs using this
extension. The extension also has the capability to capture page load data for your web applications. In the Test
hub you can view completed exploratory testing sessions and derive meaningful insights at team or individual
level, and for a specific period.
To explore your desktop applications, you can use the Exploratory Test Runner client in Microsoft Test Manager by
launching it from the Test hub.
Comparison of exploratory testing with the Test & Feedback extension and exploratory testing with
Exploratory runner (client):
CLIENT-BASED EXPLORATORY RUNNER

EXPLORATORY TESTING CAPABILITY WEB-BASED EX TENSION FOR WEB APPS FOR DESKTOP APPS
Explore user stories
File bugs using screen capture and

recording, and image action log
Create test cases and tasks
Exploratory testing session insights
Capture page load performance data
Why the Test hub over Microsoft Test Manager?

As clearly shown above, the Test hub is a much richer, faster, and easier-to-use solution for manual testing
compared to the Test Center in MTM. The Test hub works on all platforms and all browsers, and has a rich and
modern web UI that improves your testing experience across all phases of manual testing.
All the test plans, test suites, test cases, and other test management data you create using MTM are stored in your
VSTS account or TFS, so existing MTM users can easily get started using the Test hub.
Help and support

Connect Microsoft Test Manager to your project and
test plan

Use Microsoft Test Manager (MTM ) to help you test the application you built. MTM stores your test plans and
results on Team Foundation Server (TFS ).
Connect MTM to your project and test plan

1. If you don't have MTM, download and install Visual Studio Enterprise or Visual Studio Test Professional .
Don't have a project? Set up a project
2. Connect to TFS and choose your project.
If you don't see your project, choose Add server and enter the URL of your VSTS or TFS server.
3. Create a new test plan, unless there's already a plan you want to use. Typically, you create a separate test
plan for each sprint.
4. Select a plan.
If you want to connect to a different project or test plan later, choose Home.
Signed in with the wrong user name? Choose Home , Change project, Sign out.
Try this next

Exploratory testing
Plan manual tests with Microsoft Test Manager
Help and support


At the start of a sprint, find out what you need to test. Discussing test cases is a great way to help the team
understand the detail of what your users need. Tests planned in this way provide a clear target for the
development team.
TIP You can also use the web portal to plan manual tests. It is generally more convenient for creating test
cases.
1. Connect to a test plan if you haven't already.

The test plan links together the test cases you'll use in this sprint.
2. Add a manual test case to your test plan.
3. Name the test case.
4. Add the steps to perform during the test. Don't forget to include the expected results.
To add multiple lines in a step, press ALT + Enter.
Now that you've defined a manual test case, you can run it from MTM and keep the results in TFS.
Organize your test cases with test suites

Test suites are folders inside a test plan that help you organize tests. When you run tests, you can choose to run all
the tests in a suite, one after another.
Create a new test suite.
Select a suite and then create new tests in the suite.
Drag test cases from one suite to another, or cut and paste.
CTRL + drag or copy and paste to make the same test case appear in more than one suite.
These operations don't affect the test case itself. Suites contain links to test cases, and it's the links that
you're moving or copying. For the same reason, removing a test case from a suite doesn't delete the test
case from TFS.
Try this next

Run manual tests with Microsoft Test Manager
Or, dig deeper:
Help and support

Run manual tests with Microsoft Test Manager

Microsoft Test Runner sits at the side of the screen while you test your application. It displays the steps you
planned and the results you expected, and you check them off as you work. It can record your actions along with
comments, screenshots, and other data, so that if you find a bug, it's easy to reproduce.
The web portal or Microsoft Test Runner? Use the web-based test runner in the Test hub when you want
to test web applications, and Microsoft Test Runner for desktop applications. You can launch Microsoft Test
Runner from the Test hub instead of using Microsoft Test Manager.
Running test cases with Microsoft Test Runner

1. Get ready to test. Here are a few things you might need to do before running your tests:
Install the latest version of your app.
Create some test cases. Typically you create them at the start of a sprint, and aim to have them all
pass by the end of the sprint. You can create them either with the web portal or Microsoft Test
Manager.
Install Microsoft Test Manager (MTM ) on the machine where you want to run your tests. To get
MTM, install Visual Studio Enterprise or Visual Studio Test Professional .
Connect MTM to your test plan
2. Run a test case.
TIP If you are already looking at a test case in the web portal, you can start Test Runner directly from
there by choosing Run in Client.
Test Runner appears at the side of the screen. It will stay there while you work with your application.
3. Create an action recording so that you can quickly repeat the test later.
4. Follow the steps in the test. Mark each step as either Passed or Failed. When a step fails, add a comment
to describe what was wrong. You can attach screenshots, too.
If you have to attend to something else, Pause the test. You don't want your emails or password included
in the recording.
5. Create a bug if you find a problem.
6. Name the bug and describe the failure.

You can assign the bug if you know who'll fix it.
7. End the test and save the results.
Now the results are stored in TFS.
Replay previous tests

If you ran a test before, you can repeat it quickly by replaying the same actions.
(This works with most applications, though not all).
1. Start the test. Don't overwrite the recording.
2. Play your recorded actions. You have to verify the results of each step.
Track the progress of your tests

Monitor the progress of your project by seeing how many tests have passed.
Tests begin in the Active state, meaning that they are ready to run. When a bug has been fixed, you can set the
state of a failed test back to Active.
See Also
Test configurations: specifying test platforms
Record and play back manual tests
Testing Microsoft Store apps
Help and support

Run automated tests with Microsoft Test Manager
Set up your test plan to use your build

To set up your test plan to run the automated test that you have created, you must select the correct build
definition used to build your automated test or a build definition that has the correct build drop location for your
existing automated test assemblies. You must do this so that the automated test can be found in the share location
for your build definition and then it can be run from Microsoft Test Manager.
If you have multiple build configurations, the test assemblies to run the automated tests are searched for
recursively from the root directory of the build drop folder. If it is important which assemblies are selected
when you run your automated tests, you should use Run with options to specify the build configuration.
To set up your test plan to use your team build:

1. Open Microsoft Test Manager.
2. To select a test plan, choose the down-arrow on the center group switcher and then choose Testing Center.
3. On the center group menu bar, choose Plan.
4. To set up your test plan to run the automated tests, choose Properties and then choose the drop-down
arrow to the right of Filter for builds. The dialog box that shows build definition and quality is displayed.
5. To select the build definition that is used to build your automated tests, choose Build definition.
6. Each build can be given a specific value to reflect the quality of the build. To select the quality of the builds
you want to be able to view, choose Build quality.
For more information about build definitions and build quality, see Continuous integration on any
platform.
7. To save your changes, choose Set build filter.

8. To select the most recent build to use with this test plan that includes the latest changes to the automated
test, you must first choose Save to save the plan and then choose Modify. The Assign Build activity is
displayed. You can compare your current build with a build you plan to take. The associated items list shows
the changes to work items between the builds. You can then assign the latest build to take and use for
testing with this plan.
9. To close the Assign Build activity and return to the test plan properties, choose the Close icon.
10. To save these changes for this test plan, choose Save in the toolbar.
Create your test settings and environment to run your tests

To run your automated tests, you must use a standard or an SCVMM environment. You cannot run automated
tests using Microsoft Test Manager without a lab environment.
You must create an environment that contains the roles in your test settings and then use this environment in your
test plan. For more information about how to create your environment and roles and test settings, see Use a lab
environment for your devops.
Run the automated test using Microsoft Test Manager

To run the automated test using Microsoft Test Manager:
2. To run the automated test, choose the down-arrow on the center group switcher and then choose Testing
Center.
3. On the center group menu bar, choose Test.
4. (Optional) To override the build, the test settings or the environment to use for running the automated tests
that you select in this test plan, right-click the test and then choose Run with options. For example, if you
want to run on a staging environment instead of your standard testing environment then you might select a
different environment. From the Run options dialog box, you can change these settings, and then choose
Run to run the selected test.
If you select a different environment, it must contain the same roles that you selected in the test settings
that you use.
5. To run the automated test without changing any options, right-click the test and then choose Run. The
Analyze Test Runs activity is displayed. It shows the progress of the test run that contains this test.
You can run multiple automated tests by selecting multiple tests, or you can select to run a whole suite
of tests. To run a suite, right-click the test suite and then choose Run.
View and update the test results

To view and update the test results:
2. To view the test results, choose the down-arrow on the center group switcher and then choose Testing
Center.
3. On the center group menu bar, choose Test and then choose Analyze Test Runs. The Analyze Test Runs
activity is displayed. It shows any test runs for this test plan.
4. Double-click a test run to open it and view the details. The test run details are displayed.
5. (Optional) To update the title of your test run to be more meaningful, type the new name in Title.
6. (Optional) If your test failed, you can update the reason for the failure. Choose Resolution and select the
reason for the failure from the list.
7. (Optional) To add comments to the test result, choose the Comments icon. Type your comments and then
choose Save comments.
8. (Optional) To view the details of an individual test, double-click the test. The test result is displayed. It shows
the details from the test run, the attachments for data collected for this test result, and the test results
history for that test. You can close this view to return to the test run.
If, from your analysis, you determine that there is a bug, you can create a bug from this view.
9. To save these changes for this test run, choose Save in the toolbar.
See Also
Help and support

Record and play back manual tests

Record keystrokes and mouse actions with Microsoft Test Manager while you are testing an app. You can then
play back your actions quickly and accurately the next time you run the test.
Playback is very useful for reproducing bugs. You can retrace the exact actions that the tester performed to the
point where the problem was found. Playback also helps you run a test with different data, on multiple
configurations, or where you have shared steps that are the same in many test cases. Playback also speeds up
regression testing - that is, tests that you run from one sprint to the next to make sure that everything still works
correctly.
You can record and play back tests in a wide range of desktop apps, and also web apps that you access through a
supported browser. For a detailed list, see Supported configurations and platforms for coded UI tests and action
recordings.
Run Microsoft Test Manager on your client machine

To record and play back actions, you have to install Microsoft Test Manager on the machine where you'll run your
tests. If you're testing a desktop app, install the latest version of the app and Microsoft Test Manager on the same
machine. If you're testing a web-based app, install the app on a test server, and run Microsoft Test Manager on the
machine where you'll run your web browser.
To get Microsoft Test Manager, install Visual Studio Test Professional or Visual Studio Enterprise
Run a test case
1. Connect Microsoft Test Manager to your project, and select your current test plan.
2. Select a test case and run it.
Record your actions during a test run

1. In the Start Test window, select Create action recording.
2. After each step, make sure to mark that step Pass or Fail.
3. If you want to pause your test, choose Pause.
4. After you finish your test run, choose End Test. This makes sure the recording assigns your actions to the
correct steps.
Caution: All your keystrokes and gestures might be recorded, including passwords, emails, chat messages, and
other sensitive data.
Delete mistakes
Open the editing panel at the bottom of the test runner. You can delete actions there:
Alternatively, you can run the test again and choose Overwrite action recording.
Keep or re -record shared steps
If you come to a sequence of shared steps, you might have already recorded them in an earlier test case. You can
either keep the earlier recording or record them now, like this:
You have to indicate when you finish recording the shared steps:
Capture parameter values

If a test step requires a parameter value that you must type as text, that text is recognized and bound into the
recording. When you play the recording back with other parameter values, the new values are used instead.
Parameter values aren't bound if they're not typed as text, for example, when values are selected instead. When
you play the recording back, you'll have to manually perform that step.
Play back an action recording

1. Select your test and run it. Don't select the overwrite option in the Start Test dialog box.
You can play the whole test, or play individual steps. The test runner will replay the keystrokes and mouse
actions that you recorded.
2. You have to check the result of each step. The recording doesn't check the outputs.
See Also
FAQs for recording and playing back manual tests
Help and support

Exploratory testing using Microsoft Test Manager

While you work with your application, Microsoft Test Manager (MTM ) can record your actions, comments,
screenshots and other data. The recording makes it easy to reproduce bugs. And you can quickly play back your
tests whenever the application is updated.
Explore your app

1. Install the latest build of your application. If it's a desktop application, install it on a machine that has MTM.
If it's a server, you can install it on any other machine, but you'll want to run the browser or client on the
same machine where you have MTM.
2. Connect MTM to your test plan if you haven't already. The test plan stores default settings and test results.
3. Start an exploratory test session in Microsoft Test Manager.
Test Runner appears at the side of the screen. It will stay there while you work with your application.
4. Get your app ready to start testing. For example, if it's a website, start the server.
5. When you're ready, start recording.
6. While you work with the application, add comments and screenshots in the Test Runner window.
Double-click a screenshot if you want to edit it.
7. Create a bug if you find a problem. The bug will automatically include a list of your actions and comments.
8. Create a test case.

Why? When the application is updated, you'll want to run the test again to make sure everything is still
working. With the list of your actions, it's easy for you or someone else to repeat the same test.
You'll probably want to use Change Steps to set how many recent actions are included.
9. Pause the recording if you need to attend to something else. You don't want your emails or passwords
included in the recording.
10. Complete your test.
Give your test run a title that expresses the result, such as "Failed to open account" or "Successfully created
an order."
11. Review the exploratory tests in this plan to see how well the sprint is progressing.
Enable screen recording during testing

1. In your test settings, open Plan, Properties, and - under Manual runs - choose Test settings = <New>.
2. In the test settings wizard, give the new settings a name.
3. On the Data and Diagnostics page, scroll down and select Screen and Voice recorder.
4. Choose Configure if you want to record your voice along with the screen.
5. Close the test settings and Save and Close the test plan properties.
When you start a test, a real-time screen recording will be added.
If you want to run tests without real-time recording, go back to the test plan properties and set Test Settings =
<Default>.
Try this next

Help and support


When you plan manual tests there are some sequences of steps, such as logging in, that occur in many test cases.
To avoid having to enter these sequences again and again, create shared steps.
Create shared steps

While you're editing a test case, select a sequence of steps that you want to share:
The steps you selected are replaced with a link to the new shared steps work item:
Use shared steps

Now you can use the shared steps in another test case:
A TFS query opens. Run it to find the steps you want to insert:
When you run a test with shared steps

When you run a test, you can either mark the whole shared sequence as passed or failed, or mark each step
separately:
See also
Help and support
Copying and cloning test suites and test cases

There are several ways to duplicate test suites and test cases. It's important to understand that a test suite or plan
contains a set of references to test cases. If you delete the suite, or if you delete a test case from every suite, the test
case still exists as a work item in VSTS or TFS, and you can find it there with a query.
For details about copying tests in VSTS or TFS, see this FAQ.
Copying and cloning in Microsoft Test Manager

All these procedures are performed in Microsoft Test Manager. Choose Testing Center, Plan, Contents. (You can
also clone from the command line.)
Reference the same tests in different suites
Copy and paste test cases in order to use the same tests in different suites and plans. For example, you could have
a quick suite that uses a subset of the tests in a more exhaustive suite.
1. Copy a test case with CTRL+C.
2. Select a different suite or plan and paste with CTRL+V.
(If you don't select a different suite, nothing happens when you paste, because each suite can only have one
reference to any test case.)
If you edit the test case in one suite, you'll see the changes when you look at the test case in the other suite.
If you delete a test case from a suite, you're only deleting it from that suite. If you delete it from every suite,
the test case is still in Team Foundation, and you can find it with a work item query.
Clone and edit a test case
Use this to author a new test case that is similar to an existing one.
1. Right click a test case and choose Create copy.
The new test case opens.
2. Edit the new test. You must at least change its title. Under Links, you might want to delete the link to the old
test case.
The two tests can subsequently be edited independently of each other.

Copy suites from another plan or suite
When you're creating the test plan for a new sprint, you often want to repeat some of the tests from the previous
sprints, to make sure that the functionality you already implemented is still working.
1. Create the test plan for the new sprint.

2. Select the destination suite or plan and then get the suites you want to copy.
The test suite and any suites it contains are copied, but they contain references to the same test cases. The
source and destination test plans share the same test cases.
After the copy, you can add or remove test cases from either plan without affecting the other; however, if you edit a
shared test case, the changes will impact both test plans.
Clone a test plan and its test cases
Cloning is useful if you want to branch your application into two versions: after copying, the tests for the two
versions can be changed without affecting each other.
1. On the context menu for the old test plan, choose Clone plan.
2. In the dialog, select the suites you want to copy and set the new area and iteration paths.
Check Clone Requirements if you want to make new user stories or requirements that you will maintain
separately. For example:
If you plan to merge the two branches eventually, you'll want to keep the same requirements for
functionality that has already been implemented and tested. Don't check Clone Requirements.
If you plan to diverge into two similar but separate applications, you might want to change the user
stories of one without changing the stories of the other. Check Clone Requirements to create an
independent set of requirements for the new test cases.
3. Update any query-based suites that you copied to use the new area and iteration paths.
4. Specify a build in the destination test plan if you have cloned automated test cases.
What Gets Cloned?
When you clone a test suite, the following objects are copied from the source test plan to the destination test plan:
TEST PLAN OBJECT COPIED NOTES
Test case Each new test case retains its shared

steps.
A link is made between the source and

new test cases.
The new test cases do not have test

runs, bugs, test results, and build
information.
Shared steps referenced by cloned test

cases
Test suite The following data is retained:
- Names and hierarchical structure of

the test suites
- Order of the test cases
- Assigned testers
- Configurations
Action Recordings linked from a cloned

test case
Links and Attachments
Test configuration The test configuration is reapplied in the

destination test plan.
Test settings The test setting for the destination test

plan is applied.
Test results
Test runs and exploratory test sessions Because test runs are applicable only to
the source test plan, they are not
copied.
Requirements-based suites Requirements-based test suites are

converted to static test suites in the
Without /clonerequirements destination test plan. Cloned test cases
will be referenced under this static test
suite.
Cloned test cases do not include links to

their original requirements work items.
TEST PLAN OBJECT COPIED NOTES
Requirements-based suites Copied and linked to a new copy of the

requirement work item.
with /clonerequirements
Requirements work items (product with /clonerequirements Requirements work items that are
backlog items or user stories) associated with a cloned requirements-
based suite are cloned.
Bug work items with /clonerequirements Cloned in a project that uses the Scrum
process template, or any project in
with /clonerequirements which the Bug work item type is in the
Requirements work item category.
In other projects, bugs are not cloned.
Example test suite cloned by using tcm.exe
Source Test Plan
Destination Test Plan

Clone test suites from the command line
Tcm.exe can be used to copy test suites. Open a command prompt and change directory to
%VS110COMNTOOLS%..\IDE.
Open the Developer Command Prompt. Alternatively, use a standard command prompt and change directory to
%VS110COMNTOOLS%..\IDE. Use tcm.exe:
cd %VS110COMNTOOLS%..\IDE
tcm suites /clone

/collection:http://Server:8080/tfs/Collection
/teamproject:"Project"
[/destinationteamproject: "DestinationProject"]
/suiteid:sourceId
/destinationsuiteid:targetId
[/clonerequirements]
[/overridefield:"field name"="new value"] [/overridefield:"field 2"="value 2" ...]]
Parameters:
Server, Collection, Project: The names of your team foundation server, project collection, and project.
destinationProject : Specify this if the destination test suite is in a different project. It must be in the same
project collection. You must specify override field values for "Iteration Path" and "Area Path" .
suiteIdand destinationSuiteId : The ID of the suite to be copied, and the ID of the suite into which the
new copy will be added. If you want to copy a whole test plan, use the ID of the suite at the root of the test
plan.
The ID of a suite is displayed in the details pane when you select it in the test plan.
You can also get a list of suites by using tcm suites /list .
/clonerequirements: Clone requirements work items that are attached to requirements-based test suites. If
you omit this parameter, requirements-based test suites are converted to static suites.
/overridefield:"field name"="new value" : Change the value of a field in each cloned work item. You can use
multiple occurrences of this parameter to change as many fields as you want.
Examples:
tcm suites /clone /collection:http://tfs.fabrikam.com:8080/tfs/DefaultCollection

/teamproject:IceCream /destinationteamproject:ToyStore
/clonerequirements
/suiteid:234 /destinationsuiteid:567
/overridefield:"Iteration Path"="ToyStore\sprint3"
/overridefield:"Area Path"="ToyStore\catalog"
Help and support

Test configurations: specifying test platforms

Your users will probably install or run your app on a wide variety of configurations, such as different operating
systems, web browsers, and other variations. You will want to run at least some of your tests in environments that
have those different configurations. Use your test plan to decide which tests you want to run on which
configurations. You have to make sure that when you run your tests that you have set up your environments for
the configurations that you need.
You might draw up a schematic matrix of the combinations that you want to test:
Use Microsoft Test Manager to specify test configurations. But you can still run the tests either with the web portal
or with Microsoft Test Manager.
Requirements
Visual Studio Enterprise or Visual Studio Test Professional
Planning tests with configurations

Connect Microsoft Test Manager to your test project and open your test plan. Open your test plan by opening
Testing Center, Plan, Contents.
Select one or more tests, then choose Configurations.
Set the configurations you want to run the tests on.
Don't see the configurations you want? Choose All configurations. If you still don't see what you need, learn how
to define your own configurations.
I have a test case that appears in several test plans and test suites. Do I have to set the configurations for
each of these test points?
Yes. The same test case can have different configuration settings in different test suites and test plans.
Running tests with configurations

When you want to run a test that has multiple configurations, you'll see that it appears more than once in the run
list.
Set up a testing platform for a particular configuration, and then sort the list to show the tests to run on that
configuration.
When you run a test, a reminder of the required configuration appears on the Test Runner window.
WEB PORTAL MICROSOFT TEST MANAGER
Test Runner doesn't verify that you're actually running on the specified configuration. However, if you use Microsoft
Test Manager, system information is stored in the test log.
Create new configurations for your project

A few configurations are already defined for you, but you'll probably want to add your own.
A test configuration is a combination of configuration variable values. Your configuration variables could be, for
example, operating system, browser, CPU type, database. A configuration might be "Windows 8 + 32-bit CPU" or
"Windows 10 + 64-bit CPU."
Choose Testing Center, Organize, Test Configuration Manager.
To add your own configuration variables and values, choose Manage configuration variables:
Create new configurations that your tests can use:

Are different test data a good use of a test configuration variable?
It's better to use parameters when you want a test to be run with different test data, because it's easy to set
different parameters for different test cases. Test configurations are better for variations in the hardware or
software platform on which the application under test is installed.
Improving performance when repeating tests

Repeating tests on different configurations can be slow and error-prone. To speed things up, record your actions on
one configuration, and then play them back on another.
If you play back on a different browser, choose the Change browser for playback option under the play menu in
test runner. However, be aware that record/playback doesn't work for all browsers and applications. In some cases
you might have to play back some steps manually.
Help and support

Collect more diagnostic data in manual tests

While you are testing your application, Microsoft Test Manager can collect data that will help diagnose any fault
that you might find. If you create a bug report while you're testing, the data is automatically attached to the bug
work item.
You can decide what kinds of data you want to collect.
What diagnostic data can I collect in a test?

The diagnostic data is collected in the test results. It will be added to a bug if you create one while performing the
test.
DATA YOU CAN COLLECT HOW
- Link to the test case. Run tests in the web portal.

- The steps you marked as passed or failed.
- Any comments or attachments that you added.
+ Run tests with Microsoft Test Manager.
- Operating system version and other system information. (Use default test settings.)
- Your keystrokes and gestures.
- Screenshots, recorded automatically while you work. Microsoft Test Manager has to be installed on the machine
where you run the tests, or on a machine connected to a
device where the test runs.
+ Use test settings when you perform the tests with Microsoft
Test Manager.
Data collected from your client or desktop application:
Test settings files configure diagnostic data adapters. You can
- Event logs choose a test settings file when you run tests, or you can set
- IntelliTrace a default test settings file for your test plan.
- Video recording of the desktop
- Test impact analysis. This allows you to choose tests based
on changes since a previous build.
DATA YOU CAN COLLECT HOW
+ Use test settings to define the data you want to collect.
Data collected from your server software:
1. Event logs.
2. IntelliTrace
3. Test impact
4. Virtual machine snapshots of the servers, if you are using
an SCVMM lab environment
How do I create test settings?

You need test settings only if you want to collect more data than the default. The default setting collects basic
system information from each lab machine and your keystrokes and gestures from the local machine.
1. If the application you are testing is a website or has a server component, and you want to collect data from
the servers:
Create a lab environment. It can be a standard environment or an SCVMM environment.
In the Properties of your test plan, set the test environment you want to use for manual tests.
2. Choose an existing test settings file, or create a new one.
This sets the default selection for performing tests in this test plan. You can override the selection when you
perform individual test runs.
3. Give the test settings file a name.
4. Choose the lab environment you want to use for your tests. If you aren't using a lab environment, choose
Local.
Each test settings file matches one set of machine roles.

5. For each machine role, choose the data you want to collect from that machine.
The Local role is the client machine on which you'll perform the tests.
What are the diagnostic data options?

On the Data and Diagnostics page you can add and configure diagnostic adapters to collect data for each machine
role in your lab environment. In most cases the diagnostic data is included with the test results.
DIAGNOSTIC DATA ADAPTER CONFIGURATION
Action Log: Allows you to record the actions you perform Record and play back manual tests.
during your test, so that you can play them back rapidly on a
subsequent occasion. The actions are also recorded as text Not all gestures and applications are recorded.
descriptions in any bug report that you create.
ASP.NET Client Proxy for IntelliTrace and Test Impact Select this adapter in a web client role. It is required if you are
testing an ASP.NET application, and you want to collect Test
Impact or Intellisense data on the web server role.
Event log Choose Configure to select the types of events you want.
Collects events that your application wrote to the event logs. Your application has to write events using WriteEntry.
IntelliTrace: Generates an .itrace file that is linked to any bug Using IntelliTrace.
that you create. From this IntelliTrace file, the local session can
be simulated on another computer.
System information: Records information about the No additional configuration.

machine.
Test impact: Enables the Recommended Tests feature in If you are testing an ASP.NET application:
Testing Center, Track. This determines which tests are affected
by the changes since a previous build, based on code 1. On the role where the IIS server will run, enable Test
coverage. Impact and then choose Configure, Advanced, ASP.NET.
2. In the web client role, enable ASP.NET Client Proxy for
IntelliTrace and Test Impact
Restart your server application after enabling this option.

DIAGNOSTIC DATA ADAPTER CONFIGURATION
Video Recorder records the desktop in real time while you To record audio, choose Configure.
work.
Help and support

Testing Microsoft Store apps

You have two options if you want to test Microsoft Store apps on a phone, tablet, or other device:
Plan and perform your tests using the web portal, using a browser on the device or on another computer.
When you use the web portal for testing, the test runner doesn't interact with the software that you're
testing; it simply acts as a checklist of the test steps. Therefore, you don't have to run it on the device that
you're testing.
This option requires no special preparation on the device, other than installing the software.
Couple the device to your computer, and run the tests using Microsoft Test Manager. This option allows you
to capture screenshots and collect diagnostic data from the device.
Prepare the Windows device for testing

1. If possible, use the same user credentials or the same Microsoft Live ID on the Windows device and on the
computer that is running Microsoft Test Manager. If the user is different, the machine that you are running
Microsoft Test Manager from will display a credentials dialog box when you try to connect.
2. Install the Remote Debugger on the device that you want to test. See Installing the Remote Debugger. (This
is only supported for Windows client operating systems. Windows Server 2012 is not supported.)
The Microsoft Test Tools Adapter Configuration Tool will appear on the device as a new tile.
3. Choose the Microsoft Test Tools Adapter Configuration Tool tile in Windows.
4. Choose Start Service in the configuration dialog box for Microsoft Test Tools Adapter to configure the
Microsoft Test Tools Adapter.
Connect to the remote device
1. On the machine that you are testing from, open Microsoft Test Manager.
Create some test cases if you haven't already done so.
2. On the Run Tests page, choose the Modify link next to Perform tests using: to specify the remote
Windows device.
3. Choose the Remote device option and enter the name of the device that you want to test.
By default, port 6905 is used by Microsoft Test Manager to communicate with remote devices. If you want
to use a different port, enter the remote device as deviceName:port. For example, mySlateDevice1:8001 . You
must also change the port on the remote device by opening the service configuration file
mttaservice.exe.config in the Visual Studio installation folder.
4. Choose the Test link to verify that Microsoft Test Manager can communicate with the remote device.
Install your Microsoft Store app
1. Choose Install Microsoft Store App, and then enter the path and name of the .appx file for the Microsoft
Store app that you want to install.
2. Follow the steps in the installation wizard.
Test your Microsoft Store app

1. Choose Start Test.
Test Runner opens.
2. Perform the steps in the test on the remote device.
As you complete each step of the test, mark it passed or failed on the host computer.
The following features are supported while you test on a Windows Remote device:
FEATURE SUPPORT
System info Yes
Capture screenshot Yes
Event logs Yes.
Action record/playback Windows Web apps - Yes.
Windows desktop and store apps - No.
Create a bug Yes
Create environment snapshot of the servers in an Yes

SCVMM lab environment.
Security
Verify that the share location where the .appx file and certificates are stored is properly secured.
Help and support

Creating manual test plans

Go to related topic >
Q: Can I rename my test plan?
A: Yes, open the test plan from the shortcut menu and rename it.
Q: Can I permanently delete my test plan?

A: Yes, do this from the shortcut menu for the test plan.
See also Delete test artifacts
Q: Can I group and reorder my requirement-based test suites together?
A: Yes, you can create a static test suite that can contain any type of test suites - just like folders. Drag test suites to
group them in a static test plan. Drag and drop tests to reorder them.
Q: What are query-based test suites ?

A: Use a query to group together test cases that have a particular characteristic, for example, all tests that have
Priority=1. The suite will automatically include every test case that is returned by the query that you define.
Q: Can I edit other properties of a test plan from the test hub?
A: You can only do this from Microsoft Test Manager (MTM ). If you're using Visual Studio 2017, Visual Studio
2015, Visual Studio 2013, or Visual Studio 2012 Update 3, you can open a test plan in MTM directly from the Test
hub. (The most recently installed version of MTM is launched.)
Q: Can I copy, clone, and move test plans and test suites?
A: Yes, install the Test Case Explorer extension from Marketplace.
Q: Can I export the test plan to share or review offline?
A: Yes, you can export test plans, test suites, and test cases. Select the details that you want in the report. Then
email or print this report for review.
Change the test case fields in the report by adding or removing columns from the list view of the test suite.
Q: When I export a test plan, can I just view the data or copy it to a Word document?
A: Yes, choose Print in the Export dialog box, then choose Cancel in the Print dialog box. This displays the data in
the report. Select all the text, then copy and paste it into a Word document, if you want. All the formatting in the
report is retained.
Q: When I export a test plan, can I customize the report?
A: You can only do this if you are using an on-premises Team Foundation Server. You can edit the XSLT file.
Q: Can I track changes to test plans and test suites that I create with VSTS?
A: Yes, you can track changes to test plans and test suites. Open the work item for the test plan or test suite, then
view the work item history.
For test suites, other actions are tracked in the Test Suite Audit field. For example, adding and removing test cases
from a test suite are tracked in this field.
Creating manual test cases

Q: Can I rename or permanently delete test cases?
A: Yes. Open the test case from its shortcut menu.
Then rename it.
Or permanently delete it.
See also Delete test artifacts

Q: Can I add an extra line to a test step?
A: Yes, press Shift+Enter in the action or expected results field to add an extra line.
Q: How do I insert a test step into a test case?
A: Select a test step. Press Alt+P to insert a new test step above the selected step.
Q: Is there a way to quickly add multiple test cases at the same time?
A: Yes, use the grid view when you add test cases to the test suite.
On the grid shortcut menu, you can add, delete, or clear rows.
Switch between Grid and List views using the View menu at the right of the window.
Note: Do not use the Team plugin for Excel to add or update test case work items. Excel cannot parse the
format used to store test steps, and in some cases this may affect the formatting of the test case work items.
Q: Can I bulk edit multiple test cases?

A: Yes, switch the view from List to Grid. The grid shows all the test cases for the current test suite and all the test
steps for those cases. This is a helpful view if you want to review your test cases with other team members. When
you review, you can update and add new test cases.
Or, you can filter and sort the test cases in list view. Then select just the ones that you want to bulk edit using the
grid.
To return to the test suite view, switch the view from Grid back to List.
Q: Can I copy, clone, and move test plans and test suites?
A: Yes, install the Test Case Explorer extension from Marketplace.
Q: Can I copy test cases and test steps from an existing Excel worksheet?
A: Yes, copy the columns from Excel that you want to use for the title, action, and expected results fields. No
column formatting, other than multiline, is copied from the worksheet. Paste these columns into the grid view, edit
if necessary, and save them. (This is supported only with Internet Explorer and Chrome browsers.)
Q: Can I copy test cases from the grid to an Excel worksheet?

A: Yes, copy the data from the grid and paste it into your Excel worksheet. No test step formatting, other than
multiline, is copied into the worksheet. (This is supported only with Internet Explorer and Chrome browsers.)
Q: Can I edit other fields in the grid view?
A: Yes, in List view use the column options to select the fields in the test case work item.
You can then view and edit these fields when you switch to the grid view.
Q: Can I reorder test cases in a test suite?
A: Yes, you can reorder manual test cases in static suites, requirement-based suites, and query-based suites.
Choose Order tests on the tool bar, then drag and drop one or more tests. Or open the shortcut menu for a test to
move it to the top or to another position. After reordering the tests, you can sort them by the Order field and then
run them in that order with the web runner.
Q: Can I tag test cases so that I can see only tests with specific tags?
A: Yes, you can tag test cases in a suite with any tag that you want. For example, tag all the tests related to login so
that you can rerun these tests if a bug is fixed for the login page. Then you can filter on that tag from the Test hub.
You can add and edit tags when you edit a test case, or bulk edit tags in the grid view. You can also create suites
based on queries when you use tags.
Q: Can I share test steps between test cases?
A: Yes, choose the steps that you want to share. Find out more about sharing test steps.
Q: Can I add parameters to a test case so it can run multiple times with different data?
A: Yes, choose a test step, and then add the parameter. Find out more about repeating test steps with different data.
Q: Can I share parameter data between test cases?

A: Yes. That way, test cases with the same parameters can run with same data, so you get consistent results. To
share parameter data, convert your existing parameters to shared parameters.
After you create a shared parameter set, open another test case, and add the shared parameter set to that test case.
Find out more about sharing parameters.
Add, edit, and rename your shared parameter sets on the Parameters tab. In the test cases pane, view the test cases
that use those parameters.
Each shared parameter set is a work item. On the Properties tab, you can view or make changes to this work item.
For example, you can assign owners and track changes.
Q: Can I import parameter values from an Excel spreadsheet to my shared parameter sets?
A: Yes, copy the data from your Excel spreadsheet and paste it into your shared parameters grid. You can also copy
the data from your grid back into Excel, if necessary.
Q: How can I find out if a test case was added to other test suites?
A: Select a test case, then view the test suites details. The Associated test suites pane shows you any test suite for
any test plan that contains this test case. This includes all projects.
Click the associated test suite to view it. To view the project and the test plan for that test suite, move your pointer
over the test suite.
Q: What happens when I delete a test case from a requirement-based test suite?
A: The test case still exists in your project, but the test case is removed from the test suite. Also, it's no longer linked
to the backlog item for that test suite.
Q: Why do I see the wrong test suite and tests when I click 'View Tests' from the notification email about tests
that are assigned to me?
A: This might happen if you were prompted to enter sign-in credentials for VSTS when you clicked this link.
Without signing out of VSTS, click 'View Tests' again to see the correct test suite and tests.
Running manual tests

Q: How do I rerun a test?
A: Just select any test and choose Run.
Q: Can I run all the tests in a test suite together?
A: Yes, select a test suite and choose Run. This runs all the active tests in the test suite. If you haven't run a test yet,
its state is active. You can reset the state of a test to active if you want to rerun it.
Q: Can I choose a build to run tests against?

A: Yes, Choose Run and then select Run with options.
Select the build you want from the drop-down list.
Any bug filed during the run will automatically be associated with the selected build, and the test outcome will be
published against that build.
Q: Can I fix my test steps while I'm running a test?
A: Yes, if you have the Test Manager for VSTS. You can insert, move, or delete steps. Or you can edit the text itself.
Use the edit icon next to the test step number to do this.
The tool to edit the test steps is shown.
Q: Can I add a screenshot to the test results when I am running a test?

A: If you are using Google Chrome, you can use the web runner to take screenshots of the web app while testing.
For more information, see Collect diagnostic data.

Q: Can I capture my actions on the app as a log?
A: If you are using Google Chrome, you can use the web runner capture your actions on the web app as image
logs while testing.

Q: Can I capture screen recordings of my app?
A: If you are using Google Chrome, you can use the web runner to capture screen recordings of your web and
desktop apps while testing.

Q: How do I control how long I keep my test data?
A: Learn more here.
Q: Can I run tests offline and then import the results?
A: Yes, see the Offline Test Execution extension.
Tracking test status

Q: Can I view the recent test results for an individual test case?
A: Yes, select the test case within a test suite and then choose to view the test details pane.
View the recent test results for this test case.
Q: How is data shown in the charts for test cases that are in multiple test suites?
A: For test case charts, if a test case has been added to multiple test suites in a plan then it's only counted once. For
test result charts, each instance of a test that is run is counted for each of the test suites separately.
Q: Who can create charts?
A: You need at least a Basic access to create charts.
Q: How can I edit or delete a chart?
A: Select the option you want from the chart's context menu.
Q: How do I control how long I keep my test data?
A: Learn more here.
Repeating a test with different data

Q: Are parameters the best way to specify that the test should be run on different operating system platforms?
And with different browsers, databases, and so on?
A: It's better to use test configurations for that. With test case parameters, you run the different parameter values
one after another, which makes it difficult to switch from one platform to another.
Q: Can I use parameters in shared steps?
A: Yes. You set the parameter values in the test cases where you use the shared steps.
Q: Can I import parameter values from an Excel spreadsheet to my shared parameter sets?
A: Yes. Copy the data from your Excel spreadsheet and paste it into your shared parameters grid. You can also copy
the data from your grid back into Excel if you need to.
Managing test results

Q: What are the default retention limits?
A: For projects created before October 2015, VSTS doesn't delete results from automated tests and manual tests
unless you change the retention limit.
For new projects created after October 2015, VSTS deletes all test results after one year (365 days), unless you
chose to indefinitely retain a build associated with those results.
Q: What is the default test retention policy for XAML builds?
A: By default, a XAML build definition is set up to delete builds older than the 10 most recent builds. But related
test results aren't automatically deleted when those builds are deleted. Learn more.
Q: Why isn't test data deleted for XAML builds by default?
A: By default, this is turned off because 10 builds can happen very quickly, especially with continuous integration
builds. Test results are often deleted before you can analyze them.
Q: How do I keep a build indefinitely?
A: Like this:
Sharing steps between test cases

Q: How do I use shared steps in Microsoft Test Manager?**
A: It's almost exactly the same in Microsoft Test Manager as in the web portal. The buttons look slightly different.
Q: Can I find all my shared steps, and all the test cases where they are used?**
A: Yes. Open Microsoft Test Manager and look under Organize, Shared Steps Manager.
Shared steps and test cases are stored as work items in Team Foundation Server.
Q: Can I share steps between test plans and projects?**
A: Yes. But don't forget that if you edit shared steps, the changes appear in every place you use them.
Q: Can I use parameters in shared steps?**
A: Yes. You provide values for the parameters in the test cases where the shared steps are used.
You don't have to provide values in the shared steps definition. However, you can provide one default row of
values, which is used when you create an action recording of a standalone shared step.

Q: Which web browsers does the extension support?
A: The Test & Feedback extension is currently available for Google Chrome and Mozilla Firefox version 50.0 and
higher. Edge support is planned.
Some browser versions do not currently support all the features of the Test & Feedback extension.
FEATURE CHROME FIREFOX
Capture screenshots with inline Yes Yes

annotations
Capture notes Yes Yes
Capture screen recordings Yes No
Capture page load data Yes No
Capture user actions log Yes No
Capture system information Yes No
Create bugs Yes Yes
Create tasks and test cases Yes Yes
Create feedback requests Yes Yes
Export session report for sharing Yes Yes
End-to-end tracability for workitems Yes Yes
Simplified bug and task tracking and Yes Yes

triaging
View and get insights from sessions Yes Yes
View similar existing bugs Yes Yes
Test app on devices using cloud Yes No

providers such as Perfecto
Manage feedback requests Yes Yes
For more details, see Visual Studio Marketplace.

Q: How do I play the video recordings I created with the extension?
A: The video recordings created by the Test & Feedback extension can be viewed in Google Chrome browser and
in the VLC Video Player.
Q: Does the extension support Team Foundation Server?
A: The Test & Feedback extension supports Team Foundation Server 2015 and later. All users, including
stakeholders, can use the extension in Connected mode with all the functionality except session insights and the
request and provide feedback flow, which are supported only for TFS 2017.
Q: Can I edit an existing bug instead of creating a new bug when using the Test & Feedback extension?
A: Yes, the extension automatically shows bugs that may be related to the one you are creating and allows you to
add your screenshots, notes, and videos to this existing bug. For more details, see Add findings to existing bugs
with exploratory testing.
Q: On Google Chrome, the mouse offset towards the left makes it difficult to use. Do you have a workaround?
A: The workaround is:
1. Navigate to chrome://flags/#enable-use-zoom -for-dsf
2. Search for 'Use Blink's zoom for device scale factor'
3. Change it to Disabled
Microsoft Test Manager

Q: Can I record a test in one test plan and play it back in another?
A: Yes, this is a great way to do regression tests quickly and accurately. If you want to repeat some tests you did in
a previous sprint, just add those test cases to the test plan for the current sprint. The recording is linked to the test
case, not to its appearance in any particular test plan or suite.
Q: Can I record a test in one test configuration and play it back in a different configuration? The tests for
different configurations show up as separate tests in the Run page.
A: Yes, the recording is linked to the test case, so you can play it back from any instance of that test case, even in
different test configurations, test suites, or test plans.
Q: Some or all of my actions aren't recorded, or the playback doesn't work properly. Why?
A: Action recording works best for apps in which each user interface field has a unique ID, and for basic actions
such as keystrokes, clicks, and menu selections. It doesn't work for some apps and web browsers. See Supported
configurations and platforms for coded UI tests and action recordings. To learn how to develop your app so that it's
easier to record tests, see Enable coded UI testing of your controls.
Q: Record and playback is great. But can I completely automate a test, including verifying the results?
A: Yes, see Automate system tests.
Help and support

General
Q: How do I learn more about Cloud-based Load Testing?
A: Watch this video, or check out the Cloud-based Load Testing blog.
https://channel9.msdn.com/Events/Visual-Studio/Launch-2013/QE103/player
Q: Do I need anything to load test in the cloud with Visual Studio Ultimate 2013?
A: Yes, you'll need Update 4 or Update 5 installed. Download this version.
Q: My Visual Studio trial period ended, but I still want to run load tests.
A: To continue load testing after the trial, you'll need an active and valid Visual Studio Enterprise 2017 or 2015
license, or a Visual Studio Ultimate 2013 license. Learn more about licensing.
Q: Can I run cloud-based load tests on any app, even behind a firewall?
A: Yes, you can load test apps or sites that are only available to your company, such as internal or pre-release apps,
staging or preproduction deployments. To learn more, see Testing private and intranet applications using cloud-
based load testing.
Or, you can run a load test locally using Visual Studio.
Q: Can I run cloud-based load tests on my on-premises servers or on an Azure Virtual Machine?
A: Yes, several scenarios are supported. To learn more, see Run cloud-based load tests using your own machines.
Q: What are virtual users?
A: Virtual users create load by accessing your app or web site all at the same time during your test run. That way,
you can test performance under more realistic or projected conditions. Virtual users are simulated by test agents.
Q: What are test agents? How do they relate to my test run?
A: Test agents are computing resources, like CPU, memory, and network, that generate load by simulating virtual
users. Test agents use agent cores to create virtual users. Each core creates at least 1 virtual user.
For load test runs in Visual Studio Team Services (VSTS ) with the Visual Studio IDE, you can specify the number of
cores to use. For example, if you get errors when you run your test, you might have to increase the number of
cores.
Otherwise, your tests and the number of virtual users that you specify determine how many cores and agents are
used.
Q: Where do I specify the number of cores for runs in VSTS with the Visual Studio IDE?
A: You can do that here:
What do the values mean?
CORES AGENTS
0 (Default) The number of cores is based on the number of

virtual users that you specify for your test.
1 Your test run will use 1 agent.
2 - 10 Each agent will always use 2 cores.
The maximum number of cores for each test run is 200. If your test run needs more cores, you can run up to 10
load tests at the same time.
The minimum number of virtual users per agent core is 1. If your load test requires more cores, contact
vsoloadtest@microsoft.com.
The number of agents also depends on your text mix (web performance test or unit test). If you have only web
performance tests, we suggest using between 600 and 2,500 virtual users for every two cores. If you have unit
tests, the agent count depends on what your unit tests do. This means you will have to test if you have enough
agents by running a shorter duration load test run or use goal-based load testing.
Q: What are virtual user minutes (VUMs)? How many minutes will my load test use?
A: If your test run uses 25 or more virtual users per core, then VUMs = (max virtual user load for your test run) *
(test run duration in minutes).
If your test run uses fewer than 25 users per core, then VUMs = (number of cores) * (25 virtual users per core) *
(test run duration in minutes).
The minimum values used to calculate VUMs are 25 virtual users and 1 minute. If your test run values are smaller
than the minimum values, then those values are rounded up to meet the minimums. For example, if your test run
specifies 20 virtual users for 30 seconds, then your test run will actually run with 25 virtual users for 1 minute = 25
VUMs, not 15 VUMs.
Also, test run duration is in minutes, not seconds. For example, if your test run duration is 5 minutes and 15
seconds, then that duration is rounded up to 6 minutes.
A minimum of 250 virtual user minutes, including the warm-up period, is deducted from your account for:
Completed runs, based on the full duration of the run
Aborted runs, based on the elapsed run duration
For runs that end in an error state, no virtual user minutes will be deducted from your account.
Note that there is an additional charge for resource retention.
Q: I'm running load tests using test iterations. How is load test duration determined?
A: When load tests are configured to run using test iterations, the test duration cannot be determined when the test
run starts. In such cases, the load test run assumes the maximum duration of 48 hours. If the test iterations
complete in this time, the duration used for calculating the virtual user minutes consumed by your load test is the
actual duration for which the test ran. If the test iterations do not complete in 48 hours, the load test will be stopped
after 48 hours and virtual user minutes will be charged accordingly.
Q: Are there any limits when running the cloud-based load tests?
A: Yes. Based on where you're running the test, each test run duration limit is:
Visual Studio IDE: 48 hours
VSTS Load test hub:
URL -based load tests: 48 hours
JMeter load tests: 4 hours
Azure portal: 1 hour
Q: Do other load tests run on the same virtual machines that host my agents?
A: No, the virtual machines that host your agents host only one load test run.
Q: Are there load test features that aren't supported when you run load tests in the cloud?
A: These features aren't currently supported:
Network mix property
Agent to Use in test settings - use the core count property instead
SQL Trace properties in run settings
IP switching
Q: Can I speed up my load testing cycle by retaining the resources the tests use?
A: Yes, you can reduce the delay while your load test starts each time during a typical test > fix > retest scenario by
retaining the resources for an appropriate period after each test completes, instead of having to wait for them to be
acquired and provisioned for each test run. In Visual Studio Update 3 and later, specify the retention time in your
run settings properties.
In earlier versions of Visual Studio, add a context parameter named ResourcesRetentionTimeInMinutes to your
run settings.
Note:
The maximum resource retention period is 4 hours (240 minutes).
The resources will be released after the specified period if you do not start another test during that period.
When you start another test, the resource retention period you specify in that test will be applied -
effectively allowing you to extend the retention period. You can specify a different retention period each time
you start a test.
There a small additional charge applied when you use this feature. In addition to the usual VUMs that your
load test consumes, a surcharge of 5 VUMs per core is applied for the duration of the run. So if you retain
20 cores for 10 minutes, an additional 1000 VUMs (5 x 20 x 10) will be charged. The status messages
display this information.
Resource retention is not available for Apache JMeter tests at the present time.
For more details, see this blog post.
Q: How do I delete a load test?
A: Currently, only test runs can be deleted, and only by the user that created that test run. The load test itself cannot
be deleted because the data resides at the account level.
Visual Studio load testing

Q: How can I increase the capacity of my load tests?
A: You can use the Cloud-based Load Testing service, so you can run your tests across multiple virtual machines in
the cloud.
Q: How many virtual users can I configure in my load test?
A: In the full version of Visual Studio Enterprise, the number of virtual users is unlimited. In Visual Studio
Enterprise trial version, the virtual user count is limited to 250.
Q: Can I analyze load tests that ran previously?
A: Yes, to open and manage those results, click in the load test editor. You can have multiple tests open at the
same time to compare runs, and create trend analysis reports to compare them.
Q: Is there a difference between what I can analyze during a running test versus a completed test?
A: Yes, these are the differences:
Performance counters A smaller subset of the performance counter data is available while a test is
running.
Views When the load test run has completed, the Summary View and Details View are available.
Q: Can load tests use other test types in their test mix besides web performance tests?
A: Yes, you can include unit tests and coded UI tests.
Q: Can virtual users simulate pausing between test steps?
A: Yes, you can specify think times to simulate the time spent by a user on a web page.
Q: Why should I use Cloud-based Load Testing?
A: If you don't want to set up machines for load testing, or you don't have available resources, you can use the
Cloud-based Load Testing service. It sets up virtual machines in the cloud that will run your load test. Note that
your web site must be publicly available on the internet for load testing using VSTS to access it.
Azure load testing

Q: Why can't I see my existing VSTS account to run load tests?
A: To use a VSTS account for running load tests from the Azure portal, one of the following criteria must be
satisfied:
The account is backed by Azure Active Directory, Has an Azure subscription is linked to it, and the user is a
member of the linked Azure subscription.
The account is backed by Azure Active Directory and the user is an owner of the account.
Q: What is the maximum test duration and number of concurrent users?
A: The limitations for load testing in the Azure Portal depend on the web application service tier license type, as
follows:
LICENSE TYPE MAX DURATION (MINS) MAX USER LOAD (VUSER)
Free 1 40
Shared 30 1,000
Basic/Standard/Premium 60 20,000
Q: Where can I check how much test time I've used so far?
A: You can check this in the Azure Portal. For details, see Manage pricing and data volume in Application Insights.
Q: What is the default option and are my existing tests impacted?
A: The default option for performance load tests is a manual test - the same as before the multiple URL test option
was added to the portal. Your existing tests continue to use the configured URL and will work as before.
Q: What features not supported in the Visual Studio Web Test file?
A: At present this feature does not support Web Test plug-ins, data sources, and extraction rules. You must edit
your Web Test file to remove these. We hope to add support for these features in future updates.
Q: Does it support any other Web Test file formats?
A: At present only Visual Studio Web Test format files are supported. We'd be pleased to hear from you if you need
support for other file formats. Email us at vsoloadtest@microsoft.com.
Q: What else can I do with a VSTS account?
A: To find your new account, go to https://{accountname}.visualstudio.com . Share your code, build, test, track work,
and ship software - all in the cloud using any tool or language. Learn more about how VSTS features and services
help your team collaborate more easily and deploy continuously.
Q: Can I get more detailed profiler information?
A: Yes, see Profiling live Azure web apps with Application Insights.
Setting up tests
Q: Can I have other test types, besides web performance tests, in a load test mix?
A: Yes, you can include unit tests and coded web tests, but not coded UI tests.
Q: How long do I have to wait until I can run my load test after creating a VSTS account?
A: It can take between 5 seconds to 3 hours until you get permissions to run the load test in the cloud. If you
previously created your VSTS account, you might be able to run the load test right away.
Q: How do I provide different values to the same test?
A: Use a .csv file or an Excel spreadsheet. Using SQL Server is currently not supported. Learn how to supply values
to your test.
Q: Help, I'm having problems with my agents!
A: You must have at least 1 virtual user per core. If you're getting status messages that an agent stopped working
due to load, or if the downloaded report shows high CPU use for an agent, try increasing the number of agents
that you're using.
If you need more help, contact vsoloadtest@microsoft.com
Q: Where are the test agents used for my load test runs located?
A: When you set up your load test run, you can select the test agent location from any supported Azure datacenter,
starting with Visual Studio Ultimate 2013 Update 5 and Visual Studio Enterprise 2015. After your run finishes,
your results are stored in the same location as your VSTS account.
If you're using an earlier version of Visual Studio, the agent location is based on the location that you chose when
you created your VSTS account.
VSTS ACCOUNT REGION TEST AGENT AZURE DATACENTER
South Central US East US 2
West Europe West Europe
Q: Can virtual users simulate pausing between test steps?

A: Yes, you can specify think times. Select a scenario in your load test and edit the think time in the Properties view.
Q: Where can I get more information about simulating real-world loads?
A: Learn more about how to specify web performance test properties, load test scenario properties, and run
settings properties.
Q: Can I run load tests locally and in the cloud from the same project?
A: Yes, your project can have multiple test settings files. Add another test settings file to your Solution Items folder.
Now you can use one settings file to run your tests locally and the other settings file to run your load tests in the
cloud. To switch between them, open the file's shortcut menu, then select the test settings file that you want to use.
Q: How do I install certificates or software on agents that run my load tests in the cloud?
A: In the test settings, you can use deployment options and a setup script. In the Deployment window, add the .exe
or other files that you want to deploy on the agents. To install those files on the agents, use the setup script.
All the items deployed on the agents are copied to a directory on the agent. You can access the directory location
by using %DeploymentDirectory% in the setup and cleanup script. For example, if you want to install WebDeploy
on the agent machine, add WebDeploy_x64_en-US.msi to Deployment window. The setup.cmd will look like this:
%DeploymentDirectory%\WebDeploy_x64_en-US.msi /passive
Apache JMeter tests

Q: What are the supported JMeter versions?
A: The latest supported version of Apache JMeter is version 3.2.
Q: Which samplers are supported?
A: Currently, only HTTP samplers are supported.
Q: I want to supply properties to JMeter. How do I do that?
A: Add the properties you need to a user.properties file. Upload it using the Supporting files parameter when
you set up the test, and it will be applied when the load test runs.
Q: Are custom listeners supported?
A: Custom listeners are not currently supported.
Running and monitoring load tests

Q: Can I use mstest to run load tests with VSTS?
A: Yes, you can in VSTS, and in TFS 2015 and later. For more information, see this blog post.
Q: Can I debug a load test while it's running in the cloud?
A: Yes, you can do this when you use Visual Studio Enterprise 2015 or later. Learn more.
Q: How can I check the status of the Cloud-based Load Testing service?
A: You can view the service status at the top of the VSTS support page and on our service blog. You can also
subscribe to alerts for service status by following this post in our support forum.
Q: What are the possible load test run states?
A: When you run load tests with VSTS, the test run states are:
In-Progress: The test run is currently running in the cloud.
Completed: The test run was completed successfully.
Aborted: The test run was stopped because the user clicked the stop button. This state can also result from
issues related to your load test, such as issues with your test scripts.
Error: The test run was stopped due to an error with the service itself. For example, there might be an
infrastructure issue in the service, and it can't continue to run your test. This is not an issue caused by your load
test or test scripts.
Q: Where is my load test report stored after I download it?
A: Your downloaded reports are stored in a local SQL Server Express database. You can change the default
location, if you want. You can also store all the reports together for everyone by changing the location for each user
to the same database.
SQL Server Express works best for storing test results from a trial run. For better performance as you download
more reports, use SQL Server.
Q: How should I view test logs after downloading the test results locally?
A: Due to a known issue, you must currently use this workaround:
1. Start Notepad with administrator privileges.
2. Open devenv.exe.config file. You can usually find this file at: C:\Program Files (x86)\Microsoft Visual Studio
12.0\Common7\IDE
3. Change the value of bindingRedirect to "8.0.0.0-14.0.0.0"
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.QualityTools.LoadTest"
publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="8.0.0.0-14.0.0.0" newVersion="12.0.0.0"/>
</dependentAssembly>
Recording and replaying tests

Q: Can I simulate actions from different users and data-drive my test?
A: No, this capability is planned but is not available at present.
Q: Does the feature automatically identify and correlate all dynamic information from all requests?
A: No, this capability is currently supported only for the ASP.NET information such as VIEWSTATE,
EVENTVALIDATION, and similar values. Your request may fail if it contains other dynamic information. In these
cases, you should run the tests using Visual Studio.
Q: Can I test REST APIs using the functionality provided by the VSTS portal?
A: Yes, you can use the URL -based test to test REST APIs. Enter the request URL of your API and the details
required to create your test.
Q: I want to see what's possible in Visual Studio after I export the test. Do I need to have Visual Studio
Enterprise edition?
A: No, you can use the 90-day trial version of Visual Studio Enterprise edition. This allows you to run cloud-based
load tests. See this blog post.
Application Insights
Q: Can I get more detailed profiler information?
A: Yes, see Profiling live Azure web apps with Application Insights.
Q: Can I view data from other app monitoring tools when load testing in the cloud?
A: No.
Q: Can I increase how often data is collected?
A: No, this is currently a fixed frequency of one minute.
Q: I don't see any counters even after waiting a few minutes. What's wrong?
A: Go to Application Insights and check that you can view performance data for your app there. If you see data
collected there, report your issue to vsoloadtest@microsoft.com.
Q: Why do I get an "Unable to connect to VSTS due to network failure" error when trying to add apps using the
Get Performance Data from Application Insights menu command?
A: This can happen because:
No apps are configured to push analytics data to Application Insights. See Get started with Visual Studio
Application Insights. Also check that you can see the apps in Application Insights in the Azure portal, as
shown here:
The Azure Resource Manager access token has expired. The token is valid for 12 hours in the context of
VSTS. Sign out of your VSTS account and then sign in again to refresh the token.
Azure Active Directory is not enabled for your VSTS account. See Access VSTS with Azure Active Directory.
If none of the above works, contact us at vsoloadtest@microsoft.com.
Troubleshooting
Q: What do I do if Visual Studio stops responding when I try run a load test in the cloud?
A: To resolve this issue, see Known issues with load testing.
Q: How do I record a web performance test with Internet Explorer 11?
A: If the web test recorder is not active when you try and record your web test with Internet Explorer 11, see Using
Internet Explorer 11 and not able to record a web performance test to resolve the issue.
Q: How do I view errors and warnings that happen when my load test is running in the cloud?
A: Status messages and test errors are reported while your load test runs. Status messages give you details about
the load test run itself, such as when a connection to the results database is lost. Test errors relate to the test. View
both these messages from the Details tab on the progress graphs.
Q: I get an error when I try to import downloaded test results. What do I do?
A: If the error states that the connection's current state is closed, you can set the amount of time that a connection
waits before timing out.
Set the ConnectTimeout or Connection Timeout keywords in the connection string. Do not set a value of 0 as a
timeout in a ConnectionString because the connection will keep trying to connect indefinitely.
Q: Why can't I use more than 250 virtual users or plug-ins when I have Visual Studio Ultimate or Visual Studio
Enterprise?
A: If this happens, you must take the Visual Studio product key from your MSDN subscription and use the
"Change my Product License" option on the Product Information page. You must do this on every machine where
you want to run load tests using VSTS. To get the product key, visit this site.
Q: Why did the REST API calls that I use stop working?
A: Starting on 26th November 2014, you must add the version information to your REST API calls. If your call fails
with a VssVersionNotSpecifiedException exception, you must include ?api-version=1.0-preview.1 in your
REST API calls. To do this, see Get started with the REST APIs.
Q: I noticed that user code fails to execute if it depends on the test names. Are test names changed when run
against the service?
A: When the test runs using VSTS, test names in load tests are converted to lower case. Any string match done on
a test name by user code should ignore the case or convert test names to lower case.
Q: How do I enable client-side logs to help troubleshoot issues with load tests run in the cloud?
A: Edit devenv.exe.config with a text editor. You can usually find file at:
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE"
1. Add this line inside the <appSettings< section:
<add key="ElsClientLogLevel" value="XXX"/>
Where XXX can be any of the following:

all - logs all messages
off - stops logging any messages
critical - only logs critical messages
error - only logs error and critical messages
warning - logs error, critical and warning messages (default)
information - logs error, critical, warning and info messages
verbose - logs error, critical, warning, info and verbose messages
2. Add this section to the bottom of the devenv.exe.config file, just above the closing tag. You can specify the
path for the log file by changing the initializeData value.
<system.diagnostics>
<trace autoflush="true" indentsize="4">
<listeners>
<add name="myListener" type="System.Diagnostics.TextWriterTraceListener"
initializeData="d:\VSTestHost.log"/>
</listeners>
</trace>
<switches>
<!-- You must use integral values for "value": 0 = off, 1 = error, 2 = warn, 3 = info, 4 = verbose.--
>
<add name="EqtTraceLevel" value="4" />
</switches>
</system.diagnostics>
3. Restart Visual Studio and reproduce the issue. You can then review the log file or share it with Support. You
can find the log file at %Temp%\ELSClient\ .
Q: Why don't I see the individual timing values in the Load Tests Results Store?
A: For Visual Studio 2013 Update 4, Visual Studio Enterprise 2015, and later versions, the default value for the
TimingDetailsStorage property was changed from AllIndividualDetails to None. If you want to collect the
individual timings, you must specifically set TimingDetailsStorage property to be AllIndividualDetails. See Load
Test Run Settings Properties.
Errors
Q: My test run failed with these errors. What do I do?
A: If you get one of these errors:
VS1550064
VS1550072
VS1550078
VS1550081
VS1550082
VS1550083
Contact VSTS Support. You will have to give them your test run id.
Q: My run was aborted because the .loadtest xml file could not be parsed. What do I do?
A: You might get these errors if you manually edit the .loadtest xml file:
VS1550084
Open the file and revert any changes that you added. Rerun the load test. The run should complete successfully.
Q: Too many applications or counters were selected to run for my load test. What do I do?
A: You might get these errors if you manually edit the .loadtest xml file:
VS1550026
VS1550027
Open the file and revert any changes that you added. Rerun the load test. The run should complete successfully.
Q: No active load test settings were found in my load test. What do I do?
A: You might get this error if you close the load test wizard without completing it:
VS1550030
To fix this problem, create another load test. Delete the failed test run.
Q: My load test got an error when it started or was aborted during the run. What do I do?
A: Generally, these problems happen due to issues with the cloud-based load testing service. Just try and run your
load test again. If these problems still happen, contact VSTS support. You will have to give them your test run id.
Q: Where can I find information about other errors?
A: See Visual Studio Cloud Load Testing error codes to find more details about other errors and their resolutions,
where applicable.
Links to useful resources

Tutorials
All about Load Test planning
Simulating expected load - how to model real world load in CLT
Analyzing Load Test results
Data driven Load Tests with VSO and SQL Azure
Driving Unit Tests from Cloud-based Load test
Parameterizing tests to run in different environments
A Web Performance Test primer
Managing Load Test results
Getting 90th and 95th percentile results in a Load Test
Understanding Load Test results schema
Load Test plug-ins
Generating Excel reports for your Load Test runs
Generating Run Comparison report in Excel
Understanding Virtual User Activity visualization
Using fiddler to create web tests
Creating web tests transactions from fiddler
Creating custom Load profiles via plug-ins
Case Studies
Load Testing Visual Studio Online itself
NORAD Tracks Santa
Blogs and other references
Quick Reference Guide for VS Performance Testing
Geoff Gray's blog
Ed Glas' blog
Sean Lumley's blog
Samples
Sample plugins
Sample code for REST APIs (Test Execution)
Sample code for REST APIs (Importing Results)
Help and support

TF31002: Unable to connect to this Team Foundation
Server {0}. Team Foundation Server URL: {1}
VSTS | TFS 2018 | TFS 2017 | TFS 2015 | TFS 2013

You might receive this error when you try to connect to VSTS or an on-premises Team Foundation Server from
Visual Studio.
You receive this error when you try to connect to VSTS

PROBLEM RESOLUTION
You don't have an active account or license. Check with your account administrator that you're a member
of the account and have an active, valid license. See Assign
licenses to users for details.
Your VSTS account is connected to the Azure Active Directory. When your VSTS account is connected to a directory that is
associated with an Office 365 or Microsoft Azure subscription,
only members in the directory can access the account.
Check with your directory administrator to have them create

an organizational account for you or add your account to the
directory as external member.
You can't switch between different organizational accounts. If you work with several VSTS accounts that connect to
different directories, such as accounts that are created from
the Microsoft Azure Preview Portal, the sign-out function
might not work as expected. For example, you can't switch
between different organizational accounts to connect to
multiple accounts that are linked to directory tenants.
When this problem occurs, you see a flashing blank sign in

dialog box several times. Then, you receive either TF31002 or
TF31003 error after you connect to or add a new connection
in "Connect to Team Foundation Server" dialog box.
To resolve this problem, apply the most recent Visual Studio

update .
To learn more, see KB Article ID 2958966, You can't switch

between different organizational accounts in Visual Studio
Online.
You want to log into VSTS from Visual Studio using a different See Connect to team projects, Change accounts when you
account. connect to VSTS.
You receive this error when you try to connect to an on-premises TFS from your client computer
If you determine that you're receiving this error from one computer but not others, or others aren't receiving this
error, then check the problem resolutions outlined below.
PROBLEM RESOLUTION
Your password has expired. Verify that you typed your user account and password
correctly, and that your password hasn't expired.
You've entered an incorrect server URL. Verify that you have typed the server URL correctly including
the server name, port number, and protocol (http/https). See
Connect to team projects to learn more.
The TFS configuration has changed. If the configuration for the on-premises TFS has changed, you
must create a new connection. You might also need to clear
the client cache.
You work remotely and need to connect to a TFS Proxy server You need to configure Visual Studio to connect to TFS Proxy.
to check in files to Team Foundation version control.
You're connecting to a later version of TFS than your Visual Your version of Visual Studio or Team Explorer might be
Studio client version. incompatible with Team Foundation Server. You might need to
install one or more GDR packs. See Requirements and
compatibility for details.
Your firewall is blocking TFS services. See Allow a program to communicate through Windows
Firewall.
Visual Studio stops responding when you run a query in Your computer might be configured to bypass the proxy
Visual Studio. server. You should verify the configuration of the
BypassProxyOnLocal setting on your computer. For more
information, see BypassProxyOnLocal Configuration.
Several users receive this error when they try to connect to an on-premises TFS
If the problem occurs on more than one computer, you should contact your TFS administrator to confirm whether
the server is operational and available on the network.
As an administrator, you should check the event logs for the application-tier server to try to pinpoint the problem.
Also, you can use the following table to determine whether the server is misconfigured. In the table, problems that
are more likely to occur appear first. Therefore, you should try the resolutions in the order in which they appear so
that you increase the chance that you can solve the problem quickly.
PROBLEM RESOLUTION
The TFSService account password has expired or is incorrect. Many services for Team Foundation Server will stop running
when the service account for Team Foundation has expired.
For more information, see Change the service account or
password for Team Foundation Server.
The application-tier server for Team Foundation is unavailable. You should verify whether each required service is running. If
a required service is not running, you must restart it. If
necessary, set it to start automatically. For more information,
see Stop and start services, application pools, and websites.
The network is unavailable. You should verify whether your network is operational.
A website identity for Team Foundation is configured You should verify or correct the server binding assignments
incorrectly. that are made to websites for Team Foundation.
PROBLEM RESOLUTION
Access to a website for Team Foundation has been restricted. You should verify or correct restrictions that are made to
those websites that are based on IP addresses and domain
names.
The firewall or ports are configured incorrectly. You should verify or correct port binding assignments for
websites and port assignments for the firewall. First, you
should open the administration console for Team Foundation,
display the Application Tier page, and review the URL
assignments. If necessary, you can click Change URL to
modify the URL of a website. Next, you should verify the port
assignments for Internet Information Services (IIS) and the
ports that are allowed through the firewall. For more
information, see Review Server Status and Settings and Verify
or Correct Port Assignments.
Trust relationships between domains are not configured If a group of users cannot access Team Foundation Server, you
correctly. might have trust issues between domains.
When users connect to different versions of TFS from Visual This can occur because the GUIDs for the TFS 2012 collection
Studio, for example, they connect to TFS 2012 and then TFS are the same as that for TFS 2008. This confuses the local
2008, they can get the TF31002 error. client cache because it tries to maintain the same GUID based
local cache for both the 2008 server and the new Project
Collection in 2012.
To fix this, you need to run the TFSConfig ChangeServerID

command. See TFSConfig ChangeServerID command.
If the previous resolutions do not solve the problem, go to the MSDN Forums - Visual Studio Team System —
Team Foundation Server - Administration.
Default manual testing permissions and access

You can access most manual testing features when you are added as a team member or a member of the
Contributors group for a project. The most common built-in groups include Readers, Contributors, and Project
Administrators. For a simplified view of all default permissions assigned to built-in groups, see Default
permissions and access.
Permissions can be given at Project level and at Area path level.
Stakeholders have limited access to manual testing features. To learn more, see About access levels.
ACCOUNT OWNER &

TASK STAKEHOLDERS READERS CONTRIBUTORS PROJECT ADMINS
Exploratory testing,
view test runs
Exploratory testing,
create and delete test
runs
Provide feedback
using the Test &
Feedback extension
Request feedback
using the Test &
Feedback extension
Manage test
configurations and
test environments
Manage test plans

and test suites
Test Manager
(purchased
separately)
Help and support

Feedback

You can request feedback using one of two tools, through the Test & Feedback extension or through the Request
feedback link you access from a dashboard.
NOTE
Feature availability: The Test & Feedback extension is available for VSTS and TFS 2017 and later versions.
How-to Guides
Request feedback (Test & Feedback extension)
Get feedback (Work tracking)
Provide feedback with the Test & Feedback extension
Provide feedback with the Feedback client
Set feedback permissions
Reference
Default permissions and access set for collaboration tools
Give us feedback, get support
Resources
Uservoice Integration (Marketplace extension)
UserVoice UI (Marketplace extension)
Integrating with VSTS and Team Foundation Server
You can build custom applications or services that integrate with your Visual Studio Team Services (VSTS ) and
Team Foundation Server (TFS ) accounts by using the REST APIs to make direct HTTP calls, or utilize our .NET
Client Libraries.
Along with interacting with VSTS or TFS in your application, you can also integrate with popular third-party
services such as Slack or Jenkins.
5-Minute Quickstarts
Check out the quick starts to get you started:
Create a bug
Get work items using queries
Concepts
.NET client libraries
Authentication
Service hooks
REST API Versioning
Cross-origin resource sharing
Samples
Custom application samples
How-to Guides
Authenticate with PATs
Authenticate with OAuth 2.0
Create service hooks subscriptions programmatically
Reference
Service hooks consumers and action reference
Service hooks events reference
Resources
REST API reference
1 min to read •
Edit Online

MTM

Uploaded by

Copyright:

Available Formats

MTM

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MTM

Uploaded by

Copyright:

Available Formats

Table of Contents

VS 2017 | VS 2015 | VSTS | TFS 2018 | TFS 2017 | TFS 2015

Create a test plan Create test cases Run manual tests

Install the extension Test in Connected mode Test in Standalone mode

VS 2017 | VS 2015 | VSTS | TFS 2018 | TFS 2017 | TFS 2015

Planned manual testing

Manual testing in the Test hub

Testing desktop apps

User acceptance testing

Using the extension is a simple, three step process:

Help and support

Visual Studio 2017 | Visual Studio 2015 | VSTS | Previous version

Comprehensive testing capabilities

Generate load from multiple regions worldwide

Deep analysis with rich diagnostics

Free allocation and flexible low-cost pricing

Help and support

VSTS | TFS 2018 | TFS 2017 | TFS 2015

Create a test plan

Find a test plan

VSTS | TFS 2018 | TFS 2017 | TFS 2015

Create test cases

VS 2017 | VS 2015 | VSTS | TFS 2018 | TFS 2017 | TFS 2015

Run tests for web apps

Microsoft Test Runner opens and runs in a new browser.

5. Create a bug to describe what failed.

Can I run tests offline and then import the results?

Run tests for desktop apps

VSTS | TFS 2018 | TFS 2017 | TFS 2015

Install the extension

Select an exploratory testing mode

2. Decide if you want to use the extension in Connected or Standalone mode.

VSTS | TFS 2018 | TFS 2017 | TFS 2015

Connect to VSTS or TFS

Create bugs or tasks

You can also add your findings to an existing similar bug.

This opens the work item form in VSTS or TFS.

How do I play the video recordings I created with the extension?

Create test cases

End your testing session

VSTS | TFS 2018 | TFS 2017 | TFS 2015

Start testing in Standalone mode

Prepare your environment

Get the sample load test project

2. Connect to one of your projects.

If you're prompted to sign in to VSTS, do that now.

Run and analyze your load test

Prepare your environment

Run a URL-based load test

5. To run the same test again, choose Rerun.

Prepare your environment

Set up and run your performance test

3. In the DEVELOPMENT TOOLS section choose Performance test.

6. Choose + New to create a new performance test.

Test multiple URLs

Visual Studio 2017 | Visual Studio 2015 | VSTS

Open a load test in Visual Studio

Open a load test in VSTS