Skip to content

perf(cicd): shorten the Initial Artifact Build that gates all CI tests - #38329

Open
swicken wants to merge 29 commits into
mainfrom
issue-38326-artifact-build-speed
Open

swicken wants to merge 29 commits into
mainfrom
issue-38326-artifact-build-speed

Conversation

@swicken

@swicken swicken commented Oct 28, 2026

Copy link
Copy Markdown
Member

Fixes #38326

The "Initial Artifact Build" job gates every test job. A cold PR build (run 40742517893) took about 29 minutes in it; a warm one about 13.5. This PR attacks both halves: it keeps the Maven cache warm, and it shortens the warm build.

Changes

Maven repository cache (maven-job/action.yml)

  • PR and merge-queue runs save the ~/.m2 caches only when they restored nothing at all. Their entries are scoped to their own ref, so no other run could ever read them. Each one was about 784 MB, and about a dozen filled the 10 GB repository cache and evicted main's entry, the only one every build can use. On 10-27, PR builds were cold from 14:20 to 22:28 UTC because of this.
  • Branch builds (nightly, trunk, LTS, release) save as before.

Checkout (cicd_comp_build-phase.yml, cicd_pr_workflow-lint.yml)

  • The PR build fetches 2 commits instead of the full history: 364 s in the baseline, about 30 s expected.
  • The .mvn/maven.config guard now diffs the PR merge commit against its first parent (HEAD^1..HEAD), which is exactly the PR's change set. It fails closed when HEAD is not a merge commit. The workflow-lint job uses the same approach.

Reactor

  • PR builds exclude dotcms-integration, dotcms-postman, dotcms-reports, dotcms-ui-e2e, dotcms-test-karate, dotcms-api-data-model and dotcms-cli. Each test job builds its own module with -pl. Merge-queue, trunk, nightly and release builds still build the full reactor.
  • check-previous-build now reuses only merge-queue builds, falling back to trunk's own builds. A PR build is incomplete now, so it must never be reused for trunk.

Uploads and the test image

  • The test image is built and saved in the background while the other artifacts upload. Its status is always recorded, and the wait step fails fast if the build fails or the process vanishes.
  • maven-repo and docker-build-context upload with compression-level: 0, because their contents are already compressed. docker-image uses level 1: measured at 755 MB, against 726 MB at level 6, and 2.3x faster to zip.
  • build-classes is no longer uploaded by default. Its only reader is the uncalled deploy-javadoc action.
  • PR builds skip docker-build-context, which only deployments download.

Frontend (core-web)

  • The Nx build cache is carried between runs. Restoring .nx/cache alone doesn't work in Nx 23 (it reports "Unrecognized Cache Artifacts"), so the job serves the restored directory through a small local server, nx-cache-server.mjs. The server implements Nx's documented self-hosted cache API.
    • Only a push to main (trunk) and the scheduled nightly save the cache. PR and merge-queue runs only restore it.
    • An empty directory is never saved. A port conflict or a server that fails to start means building without the cache.
  • The Nx inputs of every app that ends up in the WAR now cover the files their builds actually read. That includes libs/dotcms-scss, Tailwind v4's workspace-wide class scan, npm files read by path (through pnpm-lock.yaml and package.json), and the Node version. These gaps could already serve stale CSS from a local Nx cache.
  • 13 projects that never reach the WAR are tagged skip:build, including the SDK framework libraries, dotcdn, mcp-server and ai-evals. They still build, with --prod, in the Frontend Unit Tests job through the validate profile. The WAR file list is unchanged: the same 2,356 entries with the same sizes.

Expected savings on a PR build

Change Saving
Main's Maven cache stays warm about 10 min on builds that would otherwise start cold
Shallow checkout about 5.5 min
Uploads and image overlap about 2.5 to 3 min
Skipped test-only modules about 1.5 min warm, 3.5 min cold
Nx cache and pruning about 1 min on average, about 2.2 min on a full hit

These are estimates from logs and local measurements. This PR's own CI run is the first real measurement. Its Nx cache will be empty until a trunk push or nightly saves one on main.

Verification

  • bash .github/workflows/tests/release-build-reuse.test.sh: 179 passed. It covers:
    • the cache save conditions
    • the PR-only module exclusion
    • the reuse search
    • the background image build: success, a failing build, and a vanished process
  • bash .github/workflows/tests/nx-cache.test.sh: 27 passed. It covers:
    • the server's 200, 404 and 409 answers
    • path rejection
    • partial PUTs and read errors
    • a port conflict
    • empty-cache detection and save gating
  • Local Nx builds through the cache server: 87 s cold, 5 s warm, with byte-identical output. An SCSS edit rebuilt only the four apps that use it, and a Tailwind class edit rebuilt only the two Tailwind apps.

What to check on this PR's CI run

  • The build job restores Linux-mavencore-v2-… from main and has no "Save Maven Repository Cache" step.
  • The Checkout step takes well under a minute, and the reactor lists 18 modules.
  • The image build overlaps the uploads, and every test job still passes.
… workflow lint

The PR build checked out with fetch-depth: 0 (every branch, full history,
364 s in run 40742517893) only for the .mvn/maven.config guard. On
pull_request the checkout is the merge commit, whose first parent is the base
tip, so depth 2 is enough to diff HEAD^1..HEAD. The guard now fails closed if
HEAD is not a merge commit. The workflow-lint actionlint job gets the same
treatment.
…ip unused uploads

Build and save the test image in the background while maven-repo and the
docker build context upload, store already-compressed artifacts without
re-deflating them, stop uploading build-classes (no workflow downloads it)
and skip the docker-build-context upload on pull request builds, which are
never deployed.
Pull request and merge-queue runs now save the Maven repository and plugin
caches only when they restored nothing. Their entries are scoped to their own
ref, so no other run can read them, and at ~690 MB each they kept the 10 GB
repository cache full and evicted the main-branch entry that every run reads.

On 2026-10-28, builds that restored main's entry scanned projects in 1-2 s
and ran Maven in 5-8 min; builds that started cold took 76-127 s to scan
and ~18 min in Maven, most of it downloading dependencies.
…fact build

The Initial Artifact Build ran nx run-many -t build over 21 projects, but the
dotcms-core-web war only takes dotcms-ui, dotcms-block-editor,
dotcms-binary-field-builder, dotcms-webcomponents, edit-content-bridge and the
standalone sdk-analytics bundle. The SDK libraries, mcp-server, ai-evals,
dotcdn and http are now tagged skip:build, so build-test leaves them out; the
SDK libraries that dotcms-ui and sdk-analytics depend on still build as their
dependencies.

They are still compiled before merge: the validate profile, which only the
Frontend Unit Tests job uses, gains a build-unshipped execution that builds
exactly the skip:build projects. That job runs on every change under core-web.
Every test job rebuilds its own module with -pl (dotcms-integration,
dotcms-postman, dotcms-ui-e2e, dotcms-test-karate, the CLI pair), and no
step of a PR run reads them from the maven-repo artifact. dotcms-reports is
an aggregator nothing in CI invokes. In run 40742517893 these modules took
3.5 min of the reactor; with a warm ~/.m2 they take about 1.5 min.

Merge-queue, trunk, nightly and release builds are unchanged: trunk reuses
the queue's maven-repo for the CLI native build and the dotcms-cli deploy.
The per-module class bundle was consumed by the SonarQube job, which was
removed in f79b08e (#32772). The only remaining restore-classes caller is
the deploy-javadoc action, which no workflow uses, and the release workflows
already turn the upload off. It cost ~25 s per build (run 40742517893).
Every build job ran the whole core-web Nx build from scratch. A restored
.nx/cache cannot fix that: Nx 21+ indexes it in .nx/workspace-data/<machine-id>-v3.db,
so on a new runner the restored artifacts are unrecognized and never used.
@nx/shared-fs-cache, Nx's own answer for CI-provider caches, is commercially
licensed and deprecated over CVE-2025-36852 (CREEP).

Instead, maven-job starts a small local server (nx-cache-server.mjs) that
speaks Nx's documented self-hosted remote cache protocol from a directory,
and points Nx at it with NX_SELF_HOSTED_REMOTE_CACHE_SERVER. actions/cache
carries the directory. Only builds of main save it, keyed on the core-web
tree so main stores a new entry only when core-web changed. Pull requests
and the merge queue restore the newest one. The saved directory holds only
the artifacts that run used or built (about 37 MB), so it does not grow.
Release builds do not use it.

.github/workflows/tests/nx-cache.test.sh runs the step's real run block and
checks the server's protocol behaviour.
# Conflicts:
#	.github/workflows/cicd_comp_build-phase.yml
check-previous-build searched every workflow for a successful maven-repo
artifact at github.sha. A pull_request run's head_sha is the PR head
commit, so if main received that exact commit (an admin push bypassing the
queue), trunk would reuse a PR build. PR builds now leave the CLI and test
modules out of maven-repo, so trunk's CLI native build and dotcms-cli deploy
would fail.

Restrict the search to cicd_2-merge-queue.yml. That is what trunk reuses
in practice (run 40740068900 reused merge-queue run 40730136845). Only
trunk and a nightly dispatched with reuse-previous-build set the search;
every other caller leaves it off.
The Nx cache reuses a build when its task hash is unchanged, but the targets
behind the war read files their hash did not cover, so a change to only those
files got the previous output back:

- dotcms-ui, dotcms-block-editor, dotcms-binary-field-builder and
  dotcms-webcomponents import libs/dotcms-scss by path (styles, includePaths,
  relative @import), and nothing in the project graph depends on it.
- dotcms-ui and dotcms-block-editor run Tailwind v4, which builds utility CSS
  from every text file under core-web, not only their dependency graph. A
  class written in libs/sdk/vue showed up in dotcms-ui's styles.css.
- npm files read by path (primeicons, prismjs, gridstack, tinymce,
  monaco-editor, flatpickr, regenerator-runtime, daisyui) and the build tool
  versions are invisible to Nx's external dependency hashing.

nx.json gains three named inputs: warBuild (pnpm-lock.yaml, package.json and
the Node version), dotcmsScss and tailwindSources (the whole workspace, which
Nx and Tailwind both read minus git-ignored files). The six shipped build
targets use them as they need. This also fixes local nx build runs, which
could serve stale CSS from the local cache for the same reasons.

Checked with real builds through the CI cache server, each run starting
empty and restoring a cold run's cache: a libs/dotcms-scss edit rebuilt the
four apps that read it and restored the rest; an edit to an unrelated
libs/sdk/vue file rebuilt only the two Tailwind apps.
Restricting previous-build reuse to the merge queue dropped one case that
used to work: a commit that reached main without the queue is built by its
trunk push run, and a later trunk dispatch or reuse-enabled nightly on that
commit (build-on-missing-artifacts false) now failed instead of reusing it.

When the merge-queue search finds nothing, search cicd_3-trunk.yml for the
same commit. Trunk checks out github.sha, so that maven-repo is complete
and matches the commit. The nightly is not a source: it builds the midnight
commit (build-ref) rather than its own head_sha. The PR workflow is never
searched.
… server

Save gating. The save was gated on github.ref == refs/heads/main, the ref a
workflow was dispatched from, not the code it built: a manual deploy of a
feature branch or a CLI release of an auxiliary branch, dispatched from main,
would have saved its outputs as main's newest entry for every PR to restore.
maven-job now saves only when the caller passes save-nx-cache, which the
build phase forwards and only trunk (push to main) and the scheduled nightly
(which always builds a commit of main) set.

Empty saves. A Count Nx Cache Entries step skips the save when the run used
and built nothing, for example when Maven restored core-web from its own
build cache and Nx never ran.

Readiness. The step waited for any listener on the port. The server now
writes a ready file once it is listening and exits if it cannot listen, so a
taken port means a build without the cache, not Nx talking to whatever holds
the port.

Server. GET streams through pipeline and cuts the connection on a read error
instead of leaving the response hanging; a failed PUT removes its partial
file so it is never saved.

The Restore Nx Cache comment now says plainly that only release-build: true
skips the cache, so images from trunk, the nightly and manual deploys can
contain restored outputs.

nx-cache.test.sh covers all of it: 27 checks pass, and 12 of them fail
against the previous version.
…figuration

Frontend Unit Tests passes no -Dprod, so build-unshipped compiled dotcdn
and the SDK libraries with their default configuration, not the production
one the artifact build used before they were moved out of it. Pass --prod
explicitly; locally that took 26 s against 25 s.
…ast when it vanishes

The background subshell inherits the step's errexit, so a failing docker
build exited it before the status file was written and the wait step polled
for 20 minutes. Capture the exit code with || rc=$?, record the pid, and
have the wait step fail at once when the process is gone without a status.
@claude

claude Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @swicken's task in 4m 43s —— View job


Claude Code Review

I reviewed the reviewable portions of this PR — the maven-job composite action, nx-cache-server.mjs, the Nx config/project changes, core-web/pom.xml, and the build-phase/workflow-lint logic. Focused on bugs this PR introduces (cache-save correctness, the background image build, the Nx remote-cache server, and whether cached Nx output can silently ship a stale WAR).

New Issues

  • 🟡 Medium: core-web/apps/*/project.json (shipped-project inputs) — Correctness now depends on each shipped project's declared Nx inputs being complete. Because CI reuses Nx build outputs across runs via the remote-cache server, Nx keys reuse on task hashes, so any file a shipped build reads that is not in its declared inputs can silently serve a stale artifact from main's cache into the WAR (a green build with wrong bytes), where before the WAR was always rebuilt. I confirmed the six WAR projects each carry warBuild plus style coverage (dotcms-ui/dotcms-block-editor: tailwindSources; dotcms-binary-field-builder/dotcms-webcomponents: dotcmsScss; edit-content-bridge: warBuild only; sdk-analytics:build:standalone: warBuild). Assumption: edit-content-bridge reads no SCSS/Tailwind sources and binary-field-builder/webcomponents use no Tailwind class scan. What to verify: that those three genuinely read nothing from libs/dotcms-scss / the Tailwind workspace scan beyond what's declared, and that the pom.xml:354-360 warning about adding new WAR webResources directories is enough to keep this invariant from silently regressing later. This is the single highest-risk behavior change in the PR; the test suites don't (and can't easily) cover "the cached output matches a clean rebuild."

  • 🟡 Medium: core-web/nx.json:176 — tailwindSources: ["{workspaceRoot}/**/*"] makes the entire workspace an input for dotcms-ui and dotcms-block-editor, so any source change anywhere invalidates their Nx cache entries. That's correct-over-fast (Tailwind does scan broadly), but it means the "~2.2 min on a full hit" saving effectively never applies to the main app on a typical PR that touches any core-web file — worth setting expectations against the stated savings. Not a bug.

Looked at and OK

  • action.yml:840-866 — the delete-built-artifacts-from-cache and save-cache-maven gates are byte-identical and paired, so delete never runs without a matching save; PR/merge-queue runs correctly save only when cache-matched-key == ''. Upload (persist-maven-repo:752) runs before the delete, so the artifact keeps the built modules. ✅
  • action.yml:721-801 — the background image build writes its status file as its last act and the wait step checks the status file before concluding the process "vanished", closing the exit/status race; "$(cat STATUS)" != "0" is newline-safe. A hard-kill after docker save but before the status write correctly fails the job (fail-fast, as intended). ✅
  • action.yml:807-814 — upload-docker-image has default success() gating, so a failed wait-docker-image-build skips the upload rather than publishing a missing/partial /tmp/image.tar. ✅
  • nx-cache-server.mjs:41 — /^\/v1\/cache\/(\w+)$/ rules out path traversal; the GET hard-link EEXIST race is handled, PUT is 409-on-exists + write-tmp-then-rename with cleanup, and leftover .tmp files are excluded from both the entry count (action.yml:898) and cross-run propagation (restored dir is only hard-linked by hash). ✅
  • generate-build-classes default is true in the action but false in the build phase; the only restore-classes: true consumer is deploy-javadoc/action.yml, which no workflow calls — so turning off the upload breaks no active job. ✅
  • cicd_comp_build-phase.yml:140-163 / cicd_pr_workflow-lint.yml:98-104 — fetch-depth: 2 plus the HEAD^2 fail-closed guard is correct for diffing HEAD^1..HEAD on the PR merge commit (depth 2 is needed for the parent trees, not just the SHAs). ✅

No 🔴/🟠 findings — nothing blocking. The two mediums are non-blocking; the first is the one worth a second look before merge.
· issue-38326-artifact-build-speed

@github-actions github-actions Bot added Area : CI/CD PR changes GitHub Actions/workflows Area : Frontend PR changes Angular/TypeScript frontend code Area : SDK PR changes SDK libraries labels Oct 28, 2026
actionlint reported 45 shellcheck findings in cicd_comp_initialize-phase.yml
on PR #38329. 44 were already on main; one SC2006 came from the trunk
fallback added to Set Outputs.

Fixed mechanically: the two backtick substitutions become $(...) (SC2006),
and every >> $GITHUB_OUTPUT / >> $GITHUB_STEP_SUMMARY is quoted (SC2086,
40 findings). Both paths are runner-provided with no spaces, so behaviour
is unchanged.

The three SC2129 findings (group the redirects) are in the pre-existing
Change Detection Summary step, where the fix is a restructure of that
script. They get a scoped ignore in .github/actionlint.yaml, as the lint
workflow's policy prescribes for pre-existing debt.
@swicken swicken added the PR : dotbot review Trigger dotbot AI code review and the post-merge QA test plan label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

dotbot code review:

  • Reviewer: meta/muse-spark-1.3 (medium)
  • Overall: patch is correct
  • New findings this run: 0
  • Prior unresolved dotbot findings still relevant: 0
  • Active findings total: 0

Traced reactor exclusions, cache-save gates, background image sync, shallow-checkout guards and Nx inputs; no provable P0/P1 regression introduced by this patch.

Tip: comment with "/dotbot address comments" to attempt automated fixes for unresolved review threads.

reviewed by dotbot · meta/muse-spark-1.3 · medium

@swicken

swicken commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review. On the two mediums:

  1. Nx input completeness for the shipped projects: checked. Every input of the six targets that end up in the war was audited (the styles/includePaths/@import reads, assets copied by path, the Stencil and Vite configs, env reads). edit-content-bridge, dotcms-binary-field-builder and dotcms-webcomponents don't use Tailwind, and none of the six reads anything outside core-web. It was also tested with real builds through the cache server: an edit in libs/dotcms-scss rebuilt exactly the four apps that read it, and a Tailwind class added to an unrelated library rebuilt only dotcms-ui and dotcms-block-editor, with everything else served from the cache. Agreed that nothing in CI checks "cached output equals a clean rebuild". A scheduled job that builds once with the cache and once without and diffs dist/ would cover a future war project with missing inputs; I'll treat that as a follow-up rather than part of this PR.

  2. tailwindSources covering the whole workspace: intended, and the estimate in the description already reflects it (about 1 min on average, 2.2 on a full hit). Narrowing Tailwind's scan would win back hits on dotcms-ui but can change the shipped CSS, so that's a call for the frontend owners.

No code changes from this round.

@github-actions

Copy link
Copy Markdown
Contributor

dotbot code review:

  • Reviewer: ~z-ai/glm-latest (medium)
  • Overall: patch is correct
  • New findings this run: 0
  • Prior unresolved dotbot findings still relevant: 0
  • Active findings total: 0

Traced the Nx cache server (port-conflict fail-closed, tmp-write-then-rename, 409-on-exists, empty-cache never saved, save gated on push-to-main/schedule and github.ref), the background docker image build/wait status-file handshake and its upload gating, the Maven cache-save gating on cache-matched-key plus event name, the shallow-checkout HEAD^2 fail-closed diff guards, and the PR reactor exclusion against the restricted previous-build reuse (merge-queue and trunk only, both of which build the full reactor). No provable P0/P1 regression introduced by this patch.

Tip: comment with "/dotbot address comments" to attempt automated fixes for unresolved review threads.

reviewed by dotbot · ~z-ai/glm-latest · medium

@dotCMS-Machine-User dotCMS-Machine-User left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ dotbot review: all reviewer models (meta/muse-spark-1.3, ~z-ai/glm-latest) agree — patch is correct.

approved automatically by dotbot

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area : CI/CD PR changes GitHub Actions/workflows Area : Frontend PR changes Angular/TypeScript frontend code Area : SDK PR changes SDK libraries PR : dotbot review Trigger dotbot AI code review and the post-merge QA test plan

2 participants