CI for performance tests #992
Conversation
b357460 to
d15d25d
Compare
Codecov Report
@@ Coverage Diff @@
## master #992 +/- ##
=======================================
Coverage 86.75% 86.75%
=======================================
Files 1 1
Lines 302 302
=======================================
Hits 262 262
Misses 40 40
Continue to review full report at Codecov.
|
|
@filmor if you are to merge this, I recommend squashing the commit, as there are many intermediate changes, which are insignificant on their own. |
af473de to
0eec424
Compare
|
OK, test is passing, and I can see in AppVeyor logs for 3.5 x64 xplat build, that performance is checked. BTW, 3.5 x64 xplat is the only configuration where perf test runs, as it is the only configuration supported by the published NuGet package. |
|
I guess this boils down to how much flakiness we can tolerate in perf tests in CI. Letting a bit more wiggle room in the performance goal should make it very unlikely to fail randomly. |
This reverts commit c5b4e88.
… NuGet being Windows-only
430ea6e to
a6c55d4
Compare
* attempted to add performance tests to CI * attempt to fix PerformanceTests xplat CI build * enabling building PerformanceTests for Mono * fixed AppVeyor path to Python.PerformanceTests.dll * fixed Mono deb sources to bionic * slightly relaxed perf target for WriteInt64Property * PerformanceTests: explicitly specify platform * use framework-specific build of perf tests in xplat and generic otherwise * added perf tests run to Travis CI * better error message for a failure to run benchmarks * appveyor: don't run perf tests in unsupported configurations * fixed performance test Python version condition in AppVeyor * explicitly notify when performance tests are skipped in AppVeyor * relax performance targets to ~10%, improve perf failure message * switch to the release of Microsoft.NETFramework.ReferenceAssemblies package
What does this implement/fix? Explain your changes.
Runs performance tests in CI, comparing build with new changes against the latest (2.3 as of now) NuGet release.
Does this close any currently open issues?
#886
Checklist
Nothing applicable.