Improve benchmark runs, UI helpers and heap profiles of tests

Author: vkryl

app/src/main/java/org/thunderdog/challegram/ui/MessagesController.java

@@ -828,6 +828,7 @@ public boolean onTouchEvent (MotionEvent event) {
     params.addRule(RelativeLayout.ALIGN_TOP, R.id.msg_bottom);
 
     topBar = new CollapseListView(context);
+    topBar.setId(R.id.msg_topBar);
     topBar.setLayoutParams(params);
     topBar.setTotalHeightChangeListener(listView -> onMessagesFrameChanged());
 

app/src/main/res/values/ids.xml

@@ -269,6 +269,7 @@
   <item type="id" name="msg_list" />
   <item type="id" name="msg_bottomShadow" />
   <item type="id" name="msg_bottomReply" />
+  <item type="id" name="msg_topBar" />
 
   <item type="id" name="emoji" />
   <item type="id" name="emoji_recent" />

tests/README.md

@@ -62,6 +62,7 @@ Tests are kept in the repository and guard behavior that must keep working after
 ./gradlew describeCheckResults
 ```
 
+* `-Ptgx.checks=false` builds tests without `checks` folders, e.g. to measure a commit before the change, where checks of the change don't compile.
 * The table lists checks and tests from the repository that don't pass before or after, other tests are counted. `—` means the check didn't run, e.g. before the change for checks of new code.
 * Checks are pushed to `checks/<branch>`, a branch with one commit on top of the pull request branch, deleted after the pull request is merged or closed. Screenshots and recordings shown in the description are added next to the checks and linked from this branch. The command keeps checks in the working tree and doesn't change the pull request branch:
 
@@ -266,10 +267,19 @@ class ChatTest : AuthorizedTest() {
 * `screenshot("name")` saves screenshot to test results, e.g. to check how new UI looks. `recordScreen("name") { }` saves video of the block, e.g. for pull requests with new UI, see [Screenshots and recordings](#screenshots-and-recordings).
 * Animations are on, because some bugs are visible only with them. Tests that don't need them can be annotated with `@DisableAnimations`, as well as their classes.
 * Elements are found by view id: `R.id.chats_list` is `"chats_list"`. To make a new view reachable, declare id in `app/src/main/res/values/ids.xml` and assign it with `setId`.
-* Helpers are in [`Extensions.kt`](core/src/main/kotlin/tgx/test/Extensions.kt): `findElement`, `waitForElement`, `failIfPresent`, `openUrl`, `runShellCommand` and others. Any `UiDevice` and `UiObject2` methods can be used as well, except `executeShellCommand`: use `runShellCommand` or `androidx.test.shell.Shell` instead.
+* Helpers are in [`Extensions.kt`](core/src/main/kotlin/tgx/test/Extensions.kt): `findElement`, `waitForElement`, `failIfPresent`, `openUrl`, `runShellCommand` and others. Helpers of chats are in [`Chats.kt`](ui/src/main/kotlin/tgx/test/Chats.kt): `openSavedMessages`, the only chat where tests send messages, `sendMessage`, `lastMessages`, `visibleTopOfChat` below bars of the chat, e.g. of a pinned message, `openMessageMenu` by a tap at the time of a message, `setAppearance(AppTheme.NIGHT_BLUE, bubbles = false)`, which sets the theme and the chat style without screens of settings, and `textCopiedBy { }`, which returns what the block copied. Any `UiDevice` and `UiObject2` methods can be used as well, except `executeShellCommand`: use `runShellCommand` or `androidx.test.shell.Shell` instead.
 * To see ids of elements currently displayed on device: `adb exec-out uiautomator dump /dev/tty`.
 * On Windows MSYS2 and Git Bash convert arguments that look like paths, e.g. `/dev/tty`, to Windows paths. Prepend `MSYS2_ARG_CONV_EXCL='*'` to `adb` commands with paths on device.
-* Notifications of new messages come without a second account and without sending anything from [`NotificationFlood`](ui/app/kotlin/tgx/test/NotificationFlood.kt), which builds of tests have inside app process: it loads incoming messages of chats of the account from their history and gives the app updates of TDLib about notifications with new ids and dates, as if they had just come. TDLib of these builds sends no notifications of its own. A test starts it by a broadcast, e.g. `runShellCommand("am broadcast -n ${getApplicationId()}/tgx.test.NotificationFloodReceiver --ei count 3000 --ei perMinute 3000 --ei chats 5")`, and waits for `done` in logcat of tag `NotificationFlood`, where each second it logs the notifications it sent with the number of mappings and the native heap of the app.
+* Notifications of new messages come without a second account and without sending anything from [`NotificationFlood`](ui/app/kotlin/tgx/test/NotificationFlood.kt), which builds of tests have inside app process: it loads incoming messages of chats of the account from their history and gives the app updates of TDLib about notifications with new ids and dates, as if they had just come. Builds of tests get notifications from TDLib like release builds, and the flood starts without them. A test starts it by a broadcast, e.g. `runShellCommand("am broadcast -n ${getApplicationId()}/tgx.test.NotificationFloodReceiver --ei count 3000 --ei perMinute 3000 --ei chats 5")`, and waits for `done` in logcat of tag `NotificationFlood`, where each second it logs the notifications it sent with the number of mappings and the native heap of the app. When the system caches the app in background, which ends a long flood by "excessive binder traffic during cached", it warns `the app is cached`.
+
+#### Profiles of native memory
+
+`recordNativeHeap("name") { }` records native allocations of the app while the block runs by heapprofd of Perfetto, sampled every 4 KB and dumped every 30 s, and saves the trace to test results as `<Class>.<test>.name.heapprofd.perfetto-trace`. Builds of tests are profileable by shell, and Perfetto reads the config only from `/data/misc/perfetto-configs`, where the test puts it. `describeHeapProfiles` prints memory allocated and freed during the profile, and the call sites that allocated the most, with the memory of each that wasn't freed at the end, from the code that allocates to its callers. It needs [trace processor](#analyzing-traces) in `PATH` or `-PtraceProcessor=<path>`:
+
+```shell
+./gradlew :tests:ui:connectedLatestNonMinifiedReleaseAndroidTest :tests:ui:describeHeapProfiles -Pabi=arm64 -Pandroid.testInstrumentationRunnerArguments.class=tgx.test.<Check>
+```
+
 
 ### Benchmarks
 
@@ -306,7 +316,7 @@ Results are saved in `build/outputs/connected_android_test_additional_output` of
 ./gradlew :tests:benchmark:compareBenchmarkResults
 ```
 
-`saveBenchmarkResults` keeps results in `build/benchmark-results` of the module until `clean`, microbenchmarks also join results of runs of the same build, see [Optimizing code](#optimizing-code). `compareBenchmarkResults` shows the change of medians of each metric with 95% confidence interval of the shift between iterations, e.g. `-1.2% (-2.0%..-0.4%) lower`, and ends with a summary of clear changes. When the interval includes zero, the change is smaller than measurements can tell: more iterations narrow the interval, 4 times more make it 2 times narrower, and microbenchmarks measure code more precisely. Sampled metrics, e.g. frame durations, are compared by their percentiles. The output is included in the pull request. Tasks that read results run after tests requested in the same command, e.g. `./gradlew :tests:benchmark:connectedLatestBenchmarkReleaseAndroidTest :tests:benchmark:compareBenchmarkResults`.
+`saveBenchmarkResults` keeps results in `build/benchmark-results` of the module until `clean`. Microbenchmarks keep every run there, also of commands that neither save nor compare results, join runs of the same build, and don't save a run without measurements, e.g. a dry run or a run of checks only, as results to compare with, see [Optimizing code](#optimizing-code). `compareBenchmarkResults` shows the change of medians of each metric with 95% confidence interval of the shift between iterations, e.g. `-1.2% (-2.0%..-0.4%) lower`, and ends with a summary of clear changes. When the interval includes zero, the change is smaller than measurements can tell: more iterations narrow the interval, 4 times more make it 2 times narrower, and microbenchmarks measure code more precisely. Sampled metrics, e.g. frame durations, are compared by their percentiles. The output is included in the pull request. Tasks that read results run after tests requested in the same command, e.g. `./gradlew :tests:benchmark:connectedLatestBenchmarkReleaseAndroidTest :tests:benchmark:compareBenchmarkResults`.
 
 * Among many metrics some change clearly by chance, so intervals are widened for the number of metrics. A change that targets one metric names it, e.g. `-Pmetric=frameDurationCpuMsP90`: its intervals are widened only for the number of benchmarks that have it, so it's measured precisely, while other metrics show only clear regressions.
 * Device heats up while benchmarks run, so the next run may be slower without any change. Run benchmarks before and after the change with the same pause, e.g. let the device cool down for a few minutes.
@@ -473,8 +483,8 @@ class TextVariants(sample: TextSample) : TextSampleBenchmark(sample) {
 ```
 
 * Each run is saved with the build it ran, a checksum of the app, tests and arguments of the benchmark library, and runs of the same build, up to 5, are joined: repeating the command narrows intervals, 4 runs before and 4 after make them 2 times narrower than one and one. The first line shows the number of runs, e.g. `Runs: 2 before, 1 after`. The run of the build of saved results, e.g. without changes, is compared with its earlier runs.
-* `Screen:` shows screen of runs, which sizes of texts depend on. When it changes, e.g. on foldable device, results and outputs change too.
-* `Thermal status:` shows the highest thermal status of device in runs, e.g. `none` or `light`. Throttled device runs code slower, differently for each test, so when the status changes between runs, results differ more than the control shows: let device cool down and measure again.
+* Runs are joined only when they ran in the same state of the device as the last run: the same screen, which sizes of texts depend on, e.g. on a foldable device, and the same thermal status, the highest during the run, e.g. `none` or `light`. Runs in another state are listed after `Left out runs in another state`. `Screen:` and `Thermal status:` show the state.
+* Throttled device runs code slower, differently for each test. When saved results have no runs in the thermal status of the last run, the comparison fails with `No saved runs in the state of the last run`, and results are saved again in the current status. `-PacrossThermalStatus` with `-Pcontrol` compares times relative to the control with runs in another status instead, e.g. `Thermal status: light before, none after, times are relative to BoringOrStaticLayout, only to look at: decide by runs in one status`. Such comparisons are only to look at: of the same build, 36 metrics of `TextSuite`, 3 of 5 comparisons across statuses showed a false clear change, up to 9 %, and 1 of 4 in one status. Decisions, e.g. by the decision rule of a task, are made by comparisons in one status.
 * `-Pimplementation` compares only the target implementation.
 * `-Pcontrol` measures changes of times relative to the control: when device becomes 5% slower while it heats up, the control is 5% slower too, and the target is compared as if device didn't change. Times of the same code also differ between runs by a few percent, differently for each test, so intervals include the spread of changes of the control between its tests, shown in the first lines: one pair of runs clearly shows only changes several times larger than it.
 * `-Pmetric=timeNs` measures the target metric more precisely, see [Comparing results](#comparing-results).
@@ -497,9 +507,27 @@ Outputs: 1 same
 Faster checks:
 
 * `-Pandroid.testInstrumentationRunnerArguments.tests_regex=TextBenchmarks#draw_` runs tests whose `class#method` matches the regular expression, e.g. drawing of all inputs, `'-Pandroid.testInstrumentationRunnerArguments.class=tgx.test.TextBenchmarks#draw_Text[text=short]'` runs one test.
-* `-Pandroid.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable=true` runs each measured block once, in seconds: it checks that measurements work and saves outputs, which `compareBenchmarkResults` compares without measurements.
+* `-Pandroid.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable=true` runs each measured block once: it checks that measurements work and saves outputs, which `compareBenchmarkResults` compares without measurements. `TextSuite` runs in 10 s and the command takes about 30 s with the app built, 2 min with a build of changed code. Dry runs don't compile the app ahead of time, which takes a minute, and outputs are packed into one file at the end of a run, because Gradle copies outputs during the run by a command of `adb` for each file, which took 10 minutes for 500 outputs.
 * `Text` is optimized with `-Pandroid.testInstrumentationRunnerArguments.class=tgx.test.TextSuite -Pimplementation=Text -Pcontrol=BoringOrStaticLayout`, the suite also checks its drawing and sizes on more texts and options.
 
+An earlier commit, e.g. the one a pull request is based on, is measured again by the current tests when tests changed since then: its saved runs measured other samples, and their drawings had no background. Both builds run in the same clone, so that their runs are saved together in `tests/microbenchmark/build/benchmark-results`:
+
+```shell
+# The earlier app with the current tests
+git switch --detach <commit> && git submodule update --recursive
+git checkout <branch> -- tests
+# A dry run, 2 minutes with the build: the tests run on the earlier code and draw its outputs
+./gradlew -q :tests:microbenchmark:connectedLatestNonMinifiedReleaseAndroidTest -Pabi=arm64 -Ptgx.checks=false -Pandroid.testInstrumentationRunnerArguments.class=tgx.test.TextSuite -Pandroid.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable=true
+./gradlew -q :tests:microbenchmark:connectedLatestNonMinifiedReleaseAndroidTest :tests:microbenchmark:saveBenchmarkResults -Pabi=arm64 -Ptgx.checks=false -Pandroid.testInstrumentationRunnerArguments.class=tgx.test.TextSuite
+# Back to the branch, the runs stay
+git checkout <branch> -- tests && git switch <branch> && git submodule update --recursive
+./gradlew -q :tests:microbenchmark:connectedLatestNonMinifiedReleaseAndroidTest :tests:microbenchmark:compareBenchmarkResults -Pabi=arm64 -Pandroid.testInstrumentationRunnerArguments.class=tgx.test.TextSuite -Pimplementation=Text -Pcontrol=BoringOrStaticLayout
+```
+
+* `-Ptgx.checks=false` builds tests without `checks`, which use code of the change.
+* Tests that use other code the earlier commit doesn't have, e.g. a benchmark of a new class or a helper that waits for its work, don't compile there. They are deleted or reduced before the runs of the earlier commit, and `git checkout <branch> -- tests` restores them: only metrics and outputs that both runs have are compared.
+* The dry run checks in about 2 minutes with the build of the earlier app that the tests work, before measured runs of 5 minutes each. Outputs before come from the measured run, drawn by the same tests as after, with the same background.
+
 ## Running tests
 
 | Kind | Command |

tests/buildSrc/kotlin/tgx/test/AppTests.kt

@@ -13,6 +13,7 @@ import org.gradle.api.Project
 import org.gradle.kotlin.dsl.configure
 import org.gradle.kotlin.dsl.get
 import org.gradle.kotlin.dsl.of
+import org.gradle.kotlin.dsl.register
 import tgx.gradle.source.KeystoreSource
 import java.io.File
 import java.util.Properties
@@ -26,6 +27,21 @@ const val TEST_DEVICE = "emulator"
 
 internal const val SNAPSHOT_PATH = "tests/core/src/main/assets/snapshot.zip"
 
+// -Ptgx.checks=false builds tests without checks of a change, e.g. to measure a commit before it,
+// where they don't compile
+internal val Project.hasChecks: Boolean
+  get() = providers.gradleProperty("tgx.checks")
+    .map {
+      it != "false"
+    }
+    .getOrElse(true)
+
+// Dry runs of microbenchmarks run each test once, without measuring
+private val Project.isDryRun: Boolean
+  get() = providers
+    .gradleProperty("android.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable")
+    .orNull == "true"
+
 // Libraries of the app that tests inside app process depend on too
 private val APP_LIBRARIES = listOf(
   mapOf("group" to "androidx.annotation"),
@@ -60,11 +76,15 @@ fun Project.configureTests() {
     // Built-in Kotlin compiles only Kotlin source directories
     sourceSets.getByName("test") {
       kotlin.directories += "../tests/unit/kotlin"
-      kotlin.directories += "../tests/unit/checks"
+      if (hasChecks) {
+        kotlin.directories += "../tests/unit/checks"
+      }
     }
     sourceSets.getByName("androidTest") {
       kotlin.directories += "../tests/integration/kotlin"
-      kotlin.directories += "../tests/integration/checks"
+      if (hasChecks) {
+        kotlin.directories += "../tests/integration/checks"
+      }
       if (hasTestTools) {
         kotlin.directories += "../tests/integration/tools"
       }
@@ -117,7 +137,9 @@ fun Project.configureAppTests(
 
     // Checks of a single change, which are ignored by git
     sourceSets.getByName("main") {
-      kotlin.directories += "checks"
+      if (hasChecks) {
+        kotlin.directories += "checks"
+      }
       if (hasTestTools) {
         kotlin.directories += "tools"
       }
@@ -128,8 +150,9 @@ fun Project.configureAppTests(
     experimentalProperties["android.experimental.self-instrumenting"] = !inAppProcess
     if (inAppProcess) {
       // Microbenchmarks fail when the app isn't fully compiled ahead of time,
-      // benchmark plugin of AndroidX turns on the same
-      experimentalProperties["android.experimental.force-aot-compilation"] = true
+      // benchmark plugin of AndroidX turns on the same. Dry runs measure nothing,
+      // and the compilation takes about a minute before each run
+      experimentalProperties["android.experimental.force-aot-compilation"] = !isDryRun
     }
 
     // Tests inside app process need the key of the app, which is debug key when keystore isn't set
@@ -182,6 +205,12 @@ fun Project.configureAppTests(
     }
   }
 
+  tasks.register<DescribeHeapProfilesTask>("describeHeapProfiles") {
+    mustRunAfter(tasks.named { it.startsWith("connected") && it.endsWith("AndroidTest") })
+    results.set(layout.buildDirectory.dir("outputs/connected_android_test_additional_output"))
+    traceProcessor.set(providers.gradleProperty("traceProcessor"))
+  }
+
   if (inAppProcess) {
     // App process loads classes of the test APK before classes of the app,
     // so their other versions would replace ones the app is built with