Skip to content

Development

How to build, check and test Paperless Sync, where the pictures and the look of the website come from, and how a change becomes a release. CONTRIBUTING.md has the rules for a change: its tests, the sign-off of every commit and the title of its pull request.

flowchart TB
    accTitle: The way of a change into a release
    accDescr: A change on a branch passes bash scripts/check.sh and the end-to-end tests in Docker on the machine of its author. Its pull request, titled as a Conventional Commit, passes the required checks: the CI, the end-to-end tests against every supported Nextcloud version when the change can affect the app, CodeQL, the secret scan, the licenses of every file and the sign-off. A failed check sends it back to the branch. Once every check passes, it is squashed into main, and the release bot adds it to the pull request of the next release; merging that one publishes the signed package on GitHub and in the Nextcloud App Store. Your steps are orange, the checks and the bots blue, and the release green.

    change["A change<br>on a branch"]
    local["bash scripts/check.sh<br>bash tests/e2e/run.sh"]
    request["A pull request with a<br>Conventional Commit title"]
    checks{"The required checks:<br>CI, end-to-end tests,<br>CodeQL, secret scan,<br>licenses, sign-off"}
    main["Squashed<br>into main"]
    bot["chore: release x.y.z<br>by the release bot"]
    release(["The signed package<br>on GitHub and in the<br>Nextcloud App Store"])

    change --> local --> request --> checks
    checks -- "a check fails" --> change
    checks -- "every check passes" --> main --> bot
    bot -- "merged" --> release

    classDef yours fill:#f59e0b2e,stroke:#f59e0b,stroke-width:2px
    classDef bots fill:#526cfe2e,stroke:#526cfe,stroke-width:2px
    classDef result fill:#16a34a2e,stroke:#16a34a,stroke-width:2px
    class change,local,request yours
    class checks,main,bot bots
    class release result

Set-up

You need PHP 8.2+, Composer, Node.js and Docker; the dev container in .devcontainer/ has them ready, for VS Code and for GitHub Codespaces.

composer install
bash scripts/check.sh
bash tests/e2e/run.sh

scripts/check.sh runs the checks of the CI: Composer, appinfo/info.xml against the schema of the App Store, PHP syntax, the coding standard, Psalm, PHPUnit, the package that krankerl builds, which scripts/check-package.sh compares file by file with the runtime files of the commit (what the package holds), and the checks of this app alone in scripts/check-project.sh: the version of Playwright in the end-to-end tests, the rules of tests/e2e/affected.sh, the icon of the website, the tests of its hook, the versions in the README, the same description of the app on GitHub, on the website and in composer.json, an area label for every file of the app and of its unit tests, the tests of scripts/check-package.sh, and the JavaScript and the translations.

Tests

Unit tests cover path generation, state transitions, export, metadata moves, exclusions, trash handling, guarded deletion, with permanent deletion off and without direct deletion too, incomplete scans of Paperless, the batch size with failing writes, inbox success and failure, dry-run behavior, and release version management. scripts/check.sh fails when a line of lib/ runs in no test. The doubles of the tests are strict and Nextcloud's clock stands still in them, and mutation tests check every week that the tests notice a change of the code; Architecture shows how, with the score and the way to run them on your computer.

tests/package/test_check_package.sh tests the check of the release package in a throwaway repository, in a few seconds and without krankerl: the package of a small app passes, packed as krankerl packs it and again as the release workflow packs it after signing, and every package that lacks a runtime file, holds a stray, changed or linked one, or the wrong signature fails with the difference. scripts/check-project.sh runs it.

The Docker end-to-end suite mounts this checkout into real Nextcloud containers and uses a deterministic Paperless API mock. It configures the app, runs dry-runs, exports, moves, the trash and its restore, conflicts, documents that vanish, guarded deletion and the inbox, and starts the background job as system cron would. run-all.sh runs it against the current release of every Nextcloud version from min-version to max-version in appinfo/info.xml, side by side, and the newest of them also on PostgreSQL, and on MariaDB with S3 object storage; run.sh runs it against one version, on SQLite by default:

bash tests/e2e/run-all.sh
bash tests/e2e/run.sh
E2E_DATABASE=mysql E2E_OBJECTSTORE=s3 bash tests/e2e/run.sh

CI runs run-all.sh on every push to main, every night, and in every pull request that can affect the app; a pull request that changes only the documentation, other workflows or the version of a release passes the check at once. PostgreSQL, MariaDB and S3 run in a job of their own, a required check as well, on every push to main, every night, and in a pull request that changes the migrations, the state, the files in Nextcloud, the uploads to Paperless or the tests. The contract tests start a real Paperless-ngx instead of the mock, the last release of 2.x or the current release of 3.x, check every answer that the app reads, run the app against it, and record the answers as the fixtures of the unit tests in tests/fixtures/; CI runs them every Monday and in every pull request that changes how the app talks to Paperless:

bash tests/e2e/run-contract.sh 3
CONTRACT_RECORD=1 bash tests/e2e/run-contract.sh 2

See tests/e2e/README.md for details and tests/e2e/MANUAL_ACCEPTANCE_TESTS.md for the release matrix.

Pictures

The pictures of the documentation and of the App Store come from the same Docker setup, with a synthetic household archive; take them anew after a change of the interface:

bash tests/e2e/screenshots.sh

The README and the website

The first screen of the README shows the version in the App Store and the versions of Nextcloud in badges that read appinfo/info.xml on main, so that they follow every release and every raise of max-version by the upstream bot without an edit. scripts/check-project.sh fails when one of them reads anything else, and when a version in the text of the README, of the home page or of the comparison, such as <span data-app="nextcloud">33–35</span>, names another lowest version or more than appinfo/info.xml; text that still names a lower max-version after the upstream bot raised it only gets a note, so that the pull request of the bot passes on its own.

The website takes its look from the app. The icon of the app is its logo, its favicon and the logo of its preview cards, from a copy in docs/images/app.svg that scripts/check-project.sh keeps equal to img/app.svg. docs/stylesheets/brand.css holds the colors of the icon, in shades that pass WCAG AA in the light and the dark palette. A link to the home page shows the social preview of the repository, .github/social-preview.png, which bash scripts/social-preview.sh renders from .github/social-preview.html. The versions of Nextcloud and PHP on the website come from appinfo/info.xml through the hook .github/mkdocs_project.py, which writes them into the fields such as <span data-app="php">8.2+</span> of every page, those of the requirements that the guide takes from the README included; GitHub shows the text of the fields as written:

flowchart TB
    accTitle: Where the website of Paperless Sync takes its look from
    accDescr: The icon of the app, img/app.svg, is copied to docs/images/app.svg, which scripts/check-project.sh compares with it; the website shows the copy as its logo, its favicon and on the preview cards of its pages. The social preview of the repository, rendered from .github/social-preview.html, is the preview card of the home page. The colors come from docs/stylesheets/brand.css, and the hook .github/mkdocs_project.py writes the versions of appinfo/info.xml into the pages. The files of the repository are green, the parts of the website blue.

    icon["img/app.svg<br>the icon of the app"]
    copy["docs/images/app.svg"]
    preview["social-preview.html<br>→ social-preview.png"]
    colors["docs/stylesheets/<br>brand.css"]
    info["appinfo/info.xml"]
    logo(["Logo, favicon and<br>preview cards"])
    home(["Preview card of<br>the home page"])
    palette(["Colors, light<br>and dark"])
    chips(["Versions of Nextcloud<br>and PHP on the pages"])

    icon -- "copied, compared by<br>check-project.sh" --> copy
    copy --> logo
    preview --> home
    colors --> palette
    info -- "mkdocs_project.py" --> chips

    classDef source fill:#17803d2e,stroke:#17803d,stroke-width:2px
    classDef website fill:#0082c92e,stroke:#0082c9,stroke-width:2px
    class icon,copy,preview,colors,info source
    class logo,home,palette,chips website

The same hook shows an alert of GitHub, a quote that begins with [!NOTE], [!TIP], [!IMPORTANT], [!WARNING] or [!CAUTION], as an admonition of the website. So a page, or a part of the README that the website takes by a snippet, writes an alert once for both, such as the warning about edits of the archive in the conflict policy.

Code

The app ID is paperless_sync and the PHP namespace is OCA\PaperlessSync. Psalm analyzes the PHP code, CodeQL scans the JavaScript, and the SBOM workflow continuously inventories dependencies. The architecture describes the parts of the app, and the decisions why it works as it does.

Translations

Paperless Sync speaks English and German. The settings page, its messages and the connection checklist follow the language that Nextcloud has for the administrator, the Activity stream that of the target user, and the times on the page the locale of the personal settings. The German texts are in l10n/de.json, which PHP reads, and in its copy l10n/de.js for the browser.

node scripts/check-l10n.mjs, which scripts/check-project.sh runs, compares them with the code, so that no message stays English on a German page:

---
config:
  flowchart:
    nodeSpacing: 24
---
flowchart TB
    accTitle: What the check of the translations finds
    accDescr: scripts/check-l10n.mjs reads the texts that the code translates, translate() and translatePlural() in js, ->t() and ->n() in templates and lib, and the texts that lib translates later, of a ConnectionCheck and an InvalidSettingException, with the constants that they name. It compares them with the German catalog l10n/de.json. A text that the catalog lacks, or a German text with other placeholders than its English one, fails the check with the place in the code; it needs German words. A text that no code translates, a text twice, or an l10n/de.js that differs from l10n/de.json fails as well, and node scripts/check-l10n.mjs --fix drops the text and writes l10n/de.js anew. Otherwise the check passes.

    code["The code<br>js · templates · lib"]
    texts["`The texts it translates<br>*and those that lib*<br>*translates later*`"]
    catalog["The catalog<br>l10n/de.json"]
    compare{"Each text once in the catalog,<br>with its placeholders,<br>and nothing more?"}
    missing["❌ A missing text or<br>other placeholders:<br>add the German words"]
    unused["🔧 An unused text, one twice<br>or another l10n/de.js:<br>--fix drops it and writes it"]
    passed["✅ Passes: every text<br>is in German"]

    code --> texts --> compare
    catalog --> compare
    compare -- "missing" --> missing
    compare -- "more" --> unused
    compare -- "yes" --> passed

    classDef source fill:#f59e0b2e,stroke:#f59e0b,stroke-width:2px
    classDef check fill:#526cfe2e,stroke:#526cfe,stroke-width:2px
    classDef done fill:#16a34a2e,stroke:#16a34a,stroke-width:2px
    classDef refused fill:#c9302c1f,stroke:#c9302c,stroke-width:2px
    class code,catalog source
    class texts,compare,unused check
    class passed done
    class missing refused
  • A text that the code translates is the first argument of translate() and translatePlural() in js/, of $l->t() and $l->n() in templates/ and lib/, and a text that lib/ translates later, when it knows the language of the reader: the message and the hint of a ConnectionCheck and the text of an InvalidSettingException. Write it as a string at that place, or as a class constant that the call names, such as self::STATUS_HINT, never as a variable: the check can't read it, so it fails with the file and the line, and --fix then changes nothing, instead of dropping the German words of a text that looks unused.
  • A missing text fails with its file and line, and a German text whose placeholders differ from those of its English one, {name} in JavaScript, %s or %1$s in PHP, %n in a plural, fails with both texts: PHP stops at a placeholder without its value, and one that is missing loses it.
  • A text that no code translates any more, a text twice and an l10n/de.js that differs fail as well; node scripts/check-l10n.mjs --fix removes them and writes l10n/de.js from l10n/de.json.

A new text is English in the code; add it with its German words to l10n/de.json, in the terms of the settings page, such as Basisordner for Base folder, and run node scripts/check-l10n.mjs --fix. A message about a setting names it by its label in quotes, and an InvalidSettingException names the setting as well, so that the page moves the focus there. The unit tests read the German of l10n/de.json through tests/Doubles/GermanTranslations.php, and fail on a text that it lacks.

Release process

The protected main branch requires the checks of the CI, the Docker end-to-end tests against every supported Nextcloud version, the dependency review, CodeQL, the secret scan, the licenses of every file (REUSE), the sign-off of every commit and a Conventional Commit title. Renovate keeps the dependencies, the actions and the images current, and Dependabot the Features of the dev container; routine updates merge on their own once every check passes. Automation shows every workflow and bot, and when it runs.

The release bot keeps a pull request for the next release. Its version follows from the titles of the merged pull requests, and what they wrote under Unreleased in CHANGELOG.md becomes its notes. Merging it publishes the release: the package, checked before and after signing with the app's certificate, its detached signature, an SPDX SBOM and signed build provenance, then the same package in the Nextcloud App Store, verified afterwards as users can verify it. See the release guide.

Project decisions and support expectations are documented in GOVERNANCE.md, SUPPORT.md, and CODE_OF_CONDUCT.md.