Execute CSPELL in project root folder and make fixing words easier

Issue information

From user: jonathan1055

Related to !155 (merged)

Contribution record

Fork management

Problem/Motivation

Due to the CSPELL work-around for copying the projects files into $CI_PROJECT_DIR/gitlabci_cspell/$CI_PROJECT_NAME the paths in the downloadable _cspell_json.txt are not valid any more.

Steps to reproduce

Download _cspell_json.txt and save it as .cspell.json to form the basis for the project's own file. The cspell job then fails with

Dictionary Error with (drupal) Error: ENOENT: no such file or directory, open '/builds/project/core/misc/cspell/drupal-dictionary.txt'
Dictionary Error with (dictionary) Error: ENOENT: no such file or directory, open '/builds/project/core/misc/cspell/dictionary.txt'
CSpell: Files checked: 0, Issues found: 0 in 0 files

Proposed resolution

This MR also contains several other improvements to make the cspell job easier for maintainers to fix and run green. Here is a summary of the changes:

  1. cspell is executed directly in the project root $CI_PROJECT_DIR instead of the copied folder $CI_PROJECT_DIR/gitlabci_cspell/$CI_PROJECT_NAME
  2. This eliminates the need to copy everything, so that folder is no longer created
  3. /webroot and /vendor are automatically ignored
  4. Manipulation of .cspell.json (either the projects own, or the default from /assets) is done in a new script prepare-cspell.php not the inline bash in .gitlab-ci.yml job step. This gives more flexibility and imporves readability
  5. The two drupal core dictionaries are always added, and that the paths to them are always corrected, regardless of what the maintainer might need for other running of cspell, locally or pre-existing on DrupalCI
  6. The input variables _cspell_words and _cspell_ignore_paths no longer require the tedious quoting of each item. This is all done in the php script. This reduces maintauiner effort in support
  7. A new variable _cspell_ignore_standard_files allows flexibility for ignoring the common list (composer.json, maintainers.text, etc). The default value is 1, so that these are ignored unless the maintainer specifically sets the value to 0
  8. Additional internal dictionaries (already part of cspell) are used to save maintainers adding some extra words

Here is the latest updated doc page for these changes

To test this MR use the following in your .gitlab-ci.yml

include:
  # MR 155 Run cspell in root folder
  - project: 'issue/gitlab_templates-3426136'
    ref: 3426136-cspell-in-root-folder

variables:
  _CURL_TEMPLATES_REPO: issue/gitlab_templates-3426136
  _CURL_TEMPLATES_REF: 3426136-cspell-in-root-folder

Testing

Tetsing needs to cover the following scenarios:

  1. Project without any cspell related files or variables. Project does have composer.json
    Search API
    Totally un-customised. No .cspell-project-words.txt, no variables, no .cspell.json. The project does have a composer.json file
    Issue #3427068: Fix cspell errors in Search API
    MR https://git.drupalcode.org/project/search_api/-/merge_requests/116
    Test https://git.drupalcode.org/issue/search_api-3427068/-/pipelines/120292
    Decoupled Pages
    Does not have .cspell.json, no .cpell-project-words.txt, no _CSPELL variables. There is currently one unrecognised word in composer.json, but with this MR the cspell job is green.
    MR https://git.drupalcode.org/project/decoupled_pages/-/merge_requests/6
    Existing https://git.drupalcode.org/project/decoupled_pages/-/pipelines/121257
    Test https://git.drupalcode.org/issue/decoupled_pages-3422594/-/pipelines/121458
  2. Project without any cspell variables, no .cspell.json. Project does not have composer.json
    CDN
    Issue #3347168: Match core's code quality checks: cspell
    https://git.drupalcode.org/project/cdn/-/merge_requests/18
    https://git.drupalcode.org/issue/cdn-3347168/-/jobs/1076281
  3. Project with cspell variables
    Module Filter
    #3431234: Fix spelling and unrecognized words
    https://git.drupalcode.org/project/module_filter/-/merge_requests/45
    The project does not have any .cspell.json or .cspell-project-words.txt
    Fixed entirely with additions to _CSPELL_WORDS. The sequence of commits and pipelines demonstrates that the new MR caters for unquoted words in the list, which makes it easier for maintainers.
    Able Player
    #3427424: Resolve cspell errors found by gitlab pipeline
    https://git.drupalcode.org/project/ableplayer/-/merge_requests/37
    Testing shows that existing _CSPELL_WORDS and _CSPELL_IGNORE_PATHS work with the new changes. It also shows simplification and reduction in work. See the linked issue for details.
  4. Project with existing .cspell-project-words.txt file
    Open Telemetry
    Project has existing .cspell-project-words.txt and _CSPELL_IGNORE_PATHS and the cspell job currently passes green.
    Issue https://www.drupal.org/project/opentelemetry/issues/3425641
    switch to MR155 - cspell job still green
    remove variable _CSPELL_IGNORE_PATHS: "\"composer.json\"" - job remains passing green
  5. Project with existing .cspell.json file
    Automatic Updates
    This project contains a dictionary.txt file and was running CSPELL on DrupalCI before Gitlab Templates had it.
    Issue #3426716: gitlab CI phpunit tests in 3.0.x are now failing
    MR 1030
    The current CSPELL job fails
    With MR155 it passes with no other changes
  6. Project with .cspell.json file as provided in artifacts when no file is present
    Scheduler
    First run on main branch (not new MR), with no project .cspell.json, to get a cspell_json.txt artifact that projects may have now
    https://git.drupalcode.org/project/scheduler/-/jobs/1120118
    The unrecognised words are in copyright.txt and License.md.
    Downloaded the cspelll_json.txt artifact, saved as .cspell.json with no changes, uploaded and run using MR155
    https://git.drupalcode.org/project/scheduler/-/pipelines/125105
    The copyright.txt and License.md files are now ignored as expected. Only two unrecognised words - intentionally added to test _CSPELL_WORDS.
    Add two words to _CSPELL_WORDS - job ends green
    https://git.drupalcode.org/project/scheduler/-/pipelines/125164
  7. Project using the new _CSPELL_IGNORE_STANDARD_FILES variable set to 0 to not those files.
    Scheduler
    As above, but with _CSPELL_IGNORE_STANDARD_FILES=0
    https://git.drupalcode.org/project/scheduler/-/pipelines/125204 the jobs ends amber.
    You can see in the log here that cspell_json['ignorePaths'] only contains 'vendor' 'web' 'node_modules' and '.git'. The unrecognised words are now reported in 4 files: composer.json, COPYRIGHT.txt, License.md and src/copyright.txt
Edited by drupalbot
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information