Repository navigation
build: size lerna prepare and gradle workers to the CPUs and memory the build can use - #9446
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9446 +/- ##
============================================
+ Coverage 71.37% 75.10% +3.74%
- Complexity 0 2298 +2298
============================================
Files 347 621 +274
Lines 23077 41253 +18176
Branches 2808 5242 +2434
============================================
+ Hits 16470 30981 +14511
- Misses 5989 8694 +2705
- Partials 618 1578 +960
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…URRENCY A Jest test runs gradlew-with-worker-cap.sh against stub uname, lscpu, nproc and getconf. It checks that physical cores win on an unrestricted host, that a cpuset's lower online count wins, that OMP_NUM_THREADS is ignored, and that GRADLE_WORKER_CAP caps the result. prepare-and-cache.md now says where lerna:prepare takes its concurrency from.
Lerna runs one prepare task per CPU by default. Each task runs bob build (tsc and babel) and can hold well over 1 GiB, and during a cold yarn install the Yarn process itself holds several GiB. On a 16-CPU / 8 GiB Linux machine or container the kernel OOM killer then kills Yarn mid-prepare. Yarn's yarnPath launcher reports a signal-killed child as a bare exit 1, so yarn install --immutable failed with no error text. scripts/lerna-prepare-concurrency.js replaces the inline node -p in lerna:prepare. It keeps RNFB_LERNA_CONCURRENCY and os.availableParallelism(), and adds a cap of one task per 2.5 GiB of memory (the cgroup limit when one is set), with a floor of 1. Machines with plenty of memory per CPU keep one task per CPU.
…roup memory choice The CPU, memory and floor cases now pass override: '' so an RNFB_LERNA_CONCURRENCY exported in the developer's shell no longer changes their result. memoryBytes() is exported with injectable total and constrained values, and four cases cover no limit, an unlimited cgroup value, a limit above total RAM, and a limit below it. prepare-and-cache.md notes where the 2.5 GiB per task comes from.
mikehardy
changed the title
mikehardy
marked this pull request as ready for review
November 8, 2026 18:58
Collaborator
Author
|
Quite a few changes stacked up - all infra + test - need to get them through, merging this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack position: bottom (1 of 4). Base:
main. Merge this first. Stack order: this PR → #9447 (tooling and test fixes) → #9444 (recorded-only e2e release) → #9448 (shared-host e2e hardening).Description
Two build entry points sized themselves from every core on the machine, even when the build was limited to fewer CPUs (Docker
--cpuset-cpus, cgroup cpusets, CPU affinity) or had little memory per CPU:yarn lerna:prepareused lerna's default of onepreparetask per CPU. It now passes--concurrencyfromscripts/lerna-prepare-concurrency.js, which only prints a number:RNFB_LERNA_CONCURRENCYwhen it is a positive integer.os.availableParallelism()(honours CPU affinity) and one task per 2.5 GiB of memory (the cgroup limit when it is lower than total RAM, else total RAM), and at least 1.tests/android/gradlew-with-worker-cap.shderived--max-workersfromlscpu//proc/cpuinfo, which list every host core even under a cpuset. It now also reads the online CPU count (nprocwithOMP_*limits unset, thengetconf _NPROCESSORS_ONLN, thensysctl hw.logicalcpuon macOS) and uses the smaller of physical and online. The log line now prints both values.Why the memory cap. On a plain Linux machine or container with 16 CPUs and 8 GB, a cold
yarn install --immutableexited 1 with no error text. Eachpreparetask runsbob build(tsc and babel) and can hold well over 1 GB, and during a cold install the Yarn process itself holds about 4 GB. With 16 tasks the kernel OOM killer killed Yarn partway throughlerna:prepare. Yarn'syarnPathlauncher reports a child killed by a signal as a bare exit 1, so nothing was printed. GitHub runners (4 CPUs, 16 GB, Nx cache restored) never hit this.Hosts with at least 2.5 GiB per CPU (GitHub Linux runners, Macs with 64 GiB or more) keep one task per CPU. Hosts with less run fewer
preparetasks when the Nx cache misses. Examples: a 16-CPU / 32 GiB Mac (12 tasks), a 10-CPU / 16 GiB Mac (6 tasks), a 3-CPU / 7 GiB macOS runner (2 tasks). The gradle worker cap is unchanged on an unrestricted host.okf-bundle/monorepo-tooling/prepare-and-cache.mddocuments howlerna:preparepicks its concurrency. New Jest tests:gradlewWorkerCap.test.jsandlernaPrepareConcurrency.test.js.Related issues
None. Related PRs in this stack:
Release Summary
None (build tooling only).
Checklist
AndroidiOSOther(macOS, web)e2etests added or updated inpackages/\*\*/e2ejesttests added or updated inpackages/\*\*/__tests__Test Plan
Build scripts only; no package source changes.
First commit, full ts-js-validation sequence from the repo root, each exit 0:
yarn lerna:prepare--concurrencyvalueyarn tsc:compileyarn tsc:compile:consumeryarn reference:apiyarn lintyarn tests:jestyarn format:jsyarn compare:typesSecond commit (gradle cap test and docs):
yarn lintexit 0.gradlewWorkerCap.test.jspasses 4/4 on macOS and in a Linuxnode:24container. It stubsuname,lscpu,nprocandgetconfand checks that physical cores win on an unrestricted host, that a cpuset's lower online count wins, thatOMP_NUM_THREADSis ignored, and thatGRADLE_WORKER_CAP(default 6) caps the result.Third commit (memory cap):
yarn lintexit 0.lernaPrepareConcurrency.test.js(6 tests) andgradlewWorkerCap.test.js(4 tests) pass.yarn install --immutablein anubuntu:26.04+ Node 24 container with 16 CPUs and 7.9 GB, from a cleangit archive:oom_kill1.oom_kill0,Done with warnings in 1m 16s.Fourth commit (review follow-ups):
override: '', so an exportedRNFB_LERNA_CONCURRENCYno longer affects them.memoryBytes()is exported, with four new cases: no limit, an unlimited cgroup value, a limit above total RAM, and a limit below it.yarn lintexit 0. Both Jest files pass 14/14, and the lerna file also passes 10/10 withRNFB_LERNA_CONCURRENCY=5set.memoryBytes()always return total RAM fails the below-total case (1 failed, 9 passed).Think
react-native-firebaseis great? Please consider supporting the project with any of the below:React Native FirebaseandInvertaseon Twitter