Allow excluding node_modules folder from artifacts via variable
Problem/Motivation
We often get pinged in the #gitlab channel or the issue queue about ERROR: Uploading artifacts as "archive" to coordinator... 413 Request Entity Too Large.
We've had some issues where we have already ignored ".git" files, which helps when bringing "dev" packages.
Howerver, looking at the size of one module artifact for the composer job, we can see (first numeric value is MB):
$ du -m ~/Downloads/artifacts-538085-composer | sort -nr | head -n 10
645 ~/Downloads/artifacts-538085-composer
546 ~/Downloads/artifacts-538085-composer/web
489 ~/Downloads/artifacts-538085-composer/web/core
364 ~/Downloads/artifacts-538085-composer/web/core/node_modules
118 ~/Downloads/artifacts-538085-composer/web/core/node_modules/@ckeditor
98 ~/Downloads/artifacts-538085-composer/vendor
66 ~/Downloads/artifacts-538085-composer/web/core/modules
57 ~/Downloads/artifacts-538085-composer/web/modules/contrib
57 ~/Downloads/artifacts-538085-composer/web/modules
We can see that "node_modules" is responsible for more than 50% of the size of the artifact. It appears that the size of this folder for D11 is even bigger than for D10, so pipelines might be failing purely because of this.
This issue is related #3464400: Consider merging the build and validate stages, but probably broader in scope. This one is only about "node_modules" folder and we'd get a +50% reduction in artifact size.
Steps to reproduce
Example: https://git.drupalcode.org/project/contribution_records/-/jobs/5765518
Normal module with very few dependencies.
Proposed resolution
Exclude the "node_modules" folder in the "composer" artifact and run yarn install in the subsequent jobs where needed.
- stylelint
- eslint
- cspell
- .nightwatch-base
If we don't want to risk BC (not sure how), we could do this behind a variable.
Remaining tasks
MR