🌐 US-Proxy
class="logged-out env-production page-responsive" style="word-wrap: break-word;" >
Skip to content

Build tests prior to executing them - #4881

Merged
MacGyver Codilla (mcodilla) merged 25 commits into
mainfrom
building-tests2
May 13, 2026
Merged

MacGyver Codilla (mcodilla) merged 25 commits into
mainfrom
building-tests2

Conversation

@mcodilla

@mcodilla MacGyver Codilla (mcodilla) commented Apr 13, 2026 •

Copy link
Copy Markdown
Contributor

Original PR: #4523

Addresses the following issue: #4515

This PR

  • Solves merge conflicts
  • Modifies buildTestTargets() to correctly compute target name
  • Validates changes

Brian Cody and others added 9 commits July 22, 2025 16:10
…ting

- BLOCKER: Replace path.relative() with executableTargets lookup to map
  test program paths to actual CMake target names. The previous approach
  produced file paths (e.g. 'tests/my_test') instead of target names
  (e.g. 'my_test'), causing cmake --build --target to fail.
- BLOCKER: Fix accmulatedTestList.concat() no-op by using push(...) so
  tests are properly marked as errored on build failure.
- MAJOR: Check return value of recursive getTestTargets() call so child
  failures propagate correctly to the caller.
- MINOR: Add guard for empty testProgram() return to prevent nonsensical
  build targets from empty strings.
- MINOR: Fix typo accmulatedTestList -> accumulatedTestList.
- MINOR: Remove extra whitespace in for statement and redundant
  .toString() call.
- NIT: Remove trailing semicolon after for...of block.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mcodilla
MacGyver Codilla (mcodilla) marked this pull request as ready for review April 13, 2026 21:15
@mcodilla MacGyver Codilla (mcodilla) changed the title draft Build tests prior to executing them Apr 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates CTest Explorer “build before run” behavior so running/debugging selected tests builds their required executables instead of defaulting to the ALL target, improving usability for partial builds and excluded-from-all tests.

Changes:

  • Add logic in CTestDriver to collect test executables per CMakeProject and invoke targeted builds.
  • Map test executable paths back to CMake target names using project.executableTargets (with a filename-based fallback).
  • Add a CHANGELOG entry describing the new build-before-test behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 13 comments.

File Description
src/ctest.ts Adds target discovery and per-project targeted build step before running/debugging tests.
CHANGELOG.md Documents the user-visible behavior change for build-before-run in the test workflow.

Comment thread src/ctest.ts
Comment thread src/ctest.ts
Comment thread src/ctest.ts
Comment thread src/ctest.ts
Comment thread src/ctest.ts
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts
Comment thread CHANGELOG.md Outdated
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts Outdated
@hanniavalera

Copy link
Copy Markdown
Contributor

I have some feedback :) Here is what I'm thinking needs to be addressed!

Full-project build still happens

In buildTests(), the original per-folder project.build(undefined, …) loop is still there, buildTestTargets() runs after it, not instead of it:

extensionManager.cleanOutputChannel();
}
const buildResult = await project.build(undefined, false, false);
if (buildResult.exitCode !== 0) {
status = 2;
}
} catch (e) {
status = 2;
}
}
}
builtFolder.set(folder, status);
if (status === 1) {
this.ctestErrored(test, run, { message: localize('no.project.found', 'No project found for folder {0}', folder) });
} else if (status === 2) {
this.ctestErrored(test, run, { message: localize('build.failed', 'Build failed') });
}
}
return this.buildTestTargets(foundTarget, run);

So "Run Test" ends up doing:

  1. cmake --build <dir> — builds everything (exactly what [Feature] Customize or change the CMake command run when using "Run Test" from CTest Explorer #4515 asks us to stop doing)
  2. cmake --build <dir> --target <TestExe> — no-op rebuild

Lines ~1880–1905 should be removed so buildTestTargets() is the only build call. The old builtFolder / status aggregation also becomes dead state and can go.

executableTargets includes install variants

cmakeFileApiDriver.executableTargets adds entries like "MyExe (Install)". If one collides on path, we'll try cmake --build --target "MyExe (Install)" and fail. Filter t.isInstallTarget when building execPathToName:

const execTargets = await project.executableTargets;
// Precompute a lookup map from normalized executable path to target name
const execPathToName = new Map<string, string>();
for (const t of execTargets) {
execPathToName.set(util.platformNormalizePath(t.path), t.name);
}

Fallback to path.parse(targetPath).name breaks non-CMake tests

Not every test's command[0] is a CMake target - python script.py, mpiexec, cmake -E …, custom launchers, etc. When the lookup misses, we end up with --target python / mpiexec and the build fails. Today's full-build path handles these implicitly. Either fall back to a full project build when any test doesn't resolve, or skip unresolved tests from the target list, but the behavior should be intentional, not guessed from a basename.

@mcodilla
MacGyver Codilla (mcodilla) enabled auto-merge (squash) May 13, 2026 20:13
Comment thread src/ctest.ts
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts Outdated
Comment thread src/ctest.ts
@mcodilla
MacGyver Codilla (mcodilla) merged commit 3c10814 into main May 13, 2026
4 checks passed
@mcodilla
MacGyver Codilla (mcodilla) deleted the building-tests2 branch May 13, 2026 20:35
Hannia Valera (hanniavalera) added a commit that referenced this pull request Jul 2, 2026
Running a single test from the inline Test CodeLens or the project
outline "Run Test" went through preTest, which builds the default (or
ALL) target. That is slower than necessary and, when a specific default
build target is configured, could leave the test's executable unbuilt so
the test could not run. This extends the targeted build-before-run added
in #4881 (issue #4515) -- previously wired only into the Test Explorer
run/debug handlers -- to the single-test entry points.

- Add CTestDriver.getTestBuildTargets(), reusing the same test-program ->
  executable-target mapping as buildTestTargets (extracted into a shared
  executableTargetNamesByPath helper). It returns no targets for test
  commands that are not CMake executable targets (e.g. python, cmake -E,
  or a nested ctest build-and-test).
- Thread an optional buildTargets list through CMakeProject.ctest() and
  preTest(); runTest() resolves the test's target and passes it, falling
  back to the default build when nothing resolves. All other ctest/preTest
  callers pass nothing and keep the existing default-build behavior.
Hannia Valera (hanniavalera) added a commit that referenced this pull request Jul 2, 2026
* Fix Test Explorer and CodeLens test runs spinning forever

A test run launched from the Test Explorer or the inline Test CodeLens
created a vscode TestRun but only called run.end() on the happy path.
Any throw between createTestRun and run.end() (e.g. a build or ctest
failure) orphaned the run, so the progress spinner kept spinning
indefinitely even though the work had finished.

- Guarantee run.end() via try/finally at all three createTestRun sites
  (runCTest, runTestHandler, debugTestHandler). runCTest keeps
  propagating errors so task/workflow callers are unaffected; the two
  run-profile handlers log and finalize.
- Remove the now-redundant run.end() in coverageCTestHelper (the caller
  owns finalization) to avoid a double end().
- Await the run in the CodeLens handlers so failures are logged and
  surfaced instead of escaping silently, and route to the project that
  owns the test.
- Never collapse a single-test run to a match-everything regex when the
  full set of tests is unknown (empty superset); use exact anchored
  matches so one test can no longer trigger the whole suite.
- Log a warning when a requested test matches no discovered test.

* Report a clear reason when a CTest run produces no output

When a test was started but CTest returned no matching result -- for
example the test executable was not built (so CTest reports it as
"Not Run") or the run matched nothing -- the test was finalized with no
output attached, so VS Code showed a bare "The test case did not report
any output." This was especially confusing for projects like fmt whose
test executables are not built by the default target.

- Add reportMissingTestResult() to attach an actionable message ("the
  test executable may not have been built -- build the test target,
  then run again") to both the test output and its error state, in the
  non-parallel path and for explicitly targeted tests in the parallel
  path. Tests that genuinely ran and produced no output are unchanged.
- Harden CTest results parsing: decodeOutputMeasurement tolerates
  missing/empty values and failed decompression, and cleanupResultsXml
  no longer throws when a single test lacks Measurement/NamedMeasurement
  (which previously discarded the results of the entire run). Compressed
  (base64+gzip) test output continues to be decoded.
- Add a unit test and fixture covering compressed, missing and empty
  measurements.

* Build only the test's own target for single-test runs

Running a single test from the inline Test CodeLens or the project
outline "Run Test" went through preTest, which builds the default (or
ALL) target. That is slower than necessary and, when a specific default
build target is configured, could leave the test's executable unbuilt so
the test could not run. This extends the targeted build-before-run added
in #4881 (issue #4515) -- previously wired only into the Test Explorer
run/debug handlers -- to the single-test entry points.

- Add CTestDriver.getTestBuildTargets(), reusing the same test-program ->
  executable-target mapping as buildTestTargets (extracted into a shared
  executableTargetNamesByPath helper). It returns no targets for test
  commands that are not CMake executable targets (e.g. python, cmake -E,
  or a nested ctest build-and-test).
- Thread an optional buildTargets list through CMakeProject.ctest() and
  preTest(); runTest() resolves the test's target and passes it, falling
  back to the default build when nothing resolves. All other ctest/preTest
  callers pass nothing and keep the existing default-build behavior.
Hannia Valera (hanniavalera) added a commit that referenced this pull request Oct 9, 2026
…rget (#5094)

Building tests before running them (#4881, for #4515) replaced the previous
per-folder default build with a targeted build of just the executable targets
behind the selected tests. When a test's CTest command is a wrapper program
rather than a CMake executable target -- "/usr/bin/bash" "-c" "mpiexec ...",
mpiexec, a Python launcher, or "${CMAKE_COMMAND} -E env ..." -- the program
cannot be mapped to a target, so it was dropped from the build list. If that
left no targets at all, buildTestTargets skipped the build entirely and CTest
ran against a stale or missing executable, which made cmake.buildBeforeRun
ineffective for those tests.

Such a test still normally runs something the project builds, so a selection
that cannot be fully resolved now falls back to the project's default build
instead of building nothing. Selections whose test programs all map to
executable targets keep the targeted build introduced by #4881.
Hannia Valera (hanniavalera) added a commit that referenced this pull request Oct 12, 2026
…rget (#5094)

Building tests before running them (#4881, for #4515) replaced the previous
per-folder default build with a targeted build of just the executable targets
behind the selected tests. When a test's CTest command is a wrapper program
rather than a CMake executable target -- "/usr/bin/bash" "-c" "mpiexec ...",
mpiexec, a Python launcher, or "${CMAKE_COMMAND} -E env ..." -- the program
cannot be mapped to a target, so it was dropped from the build list. If that
left no targets at all, buildTestTargets skipped the build entirely and CTest
ran against a stale or missing executable, which made cmake.buildBeforeRun
ineffective for those tests.

Such a test still normally runs something the project builds, so a selection
that cannot be fully resolved now falls back to the project's default build
instead of building nothing. Selections whose test programs all map to
executable targets keep the targeted build introduced by #4881.
Hannia Valera (hanniavalera) added a commit that referenced this pull request Oct 16, 2026
…rget (#5094)

Building tests before running them (#4881, for #4515) replaced the previous
per-folder default build with a targeted build of just the executable targets
behind the selected tests. When a test's CTest command is a wrapper program
rather than a CMake executable target -- "/usr/bin/bash" "-c" "mpiexec ...",
mpiexec, a Python launcher, or "${CMAKE_COMMAND} -E env ..." -- the program
cannot be mapped to a target, so it was dropped from the build list. If that
left no targets at all, buildTestTargets skipped the build entirely and CTest
ran against a stale or missing executable, which made cmake.buildBeforeRun
ineffective for those tests.

Such a test still normally runs something the project builds, so a selection
that cannot be fully resolved now falls back to the project's default build
instead of building nothing. Selections whose test programs all map to
executable targets keep the targeted build introduced by #4881.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Customize or change the CMake command run when using "Run Test" from CTest Explorer

5 participants