Repository navigation
Sync current Funnel docs and automate updates from develop #107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
8e49e30
docs: sync Funnel guides from develop
matthewpeterkort a4f8dbf
docs: clarify sync organization permissions
matthewpeterkort 91c44f1
fix: protect pending docs during sync
matthewpeterkort af6a562
ci: check Funnel docs updates daily
matthewpeterkort 1729ee0
chore: leave Funnel content to scheduled sync
matthewpeterkort 49d239a
docs: sync current Funnel reference pages and repair automation
matthewpeterkort File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| name: Check Funnel documentation sync | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| - main | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| check: | ||
| name: Test sync and build site | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Check out CALYPR docs | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| - name: Set up Python | ||
| uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0 | ||
| with: | ||
| python-version: '3.12' | ||
| - name: Test sync behavior | ||
| run: python -m unittest discover -s tests -v | ||
| - name: Install website dependencies | ||
| run: python -m pip install -r requirements.txt | ||
| - name: Build website | ||
| run: zensical build --clean |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| name: Update Funnel documentation | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '17 6 * * *' | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: write | ||
| pull-requests: write | ||
|
|
||
| concurrency: | ||
| group: sync-funnel-docs | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| sync: | ||
| name: Prepare Funnel docs update | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Check out CALYPR docs | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Check out Funnel develop | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| repository: calypr/funnel | ||
| ref: develop | ||
| path: .funnel-upstream | ||
| persist-credentials: false | ||
|
|
||
| - name: Check Funnel revision | ||
| id: revision | ||
| run: | | ||
| revision=$(git -C .funnel-upstream rev-parse HEAD) | ||
| previous=$(python3 -c 'import json; print(json.load(open("docs/tools/funnel/.funnel-docs.json"))["revision"])') | ||
| echo "revision=$revision" >> "$GITHUB_OUTPUT" | ||
| if [ "$revision" != "$previous" ]; then | ||
| echo "changed=true" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Set up Python | ||
| if: steps.revision.outputs.changed == 'true' | ||
| uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Test sync behavior | ||
| if: steps.revision.outputs.changed == 'true' | ||
| run: python -m unittest discover -s tests -v | ||
|
|
||
| - name: Sync Funnel documentation | ||
| if: steps.revision.outputs.changed == 'true' | ||
| run: python scripts/sync_funnel_docs.py .funnel-upstream --revision "${{ steps.revision.outputs.revision }}" | ||
|
|
||
| - name: Install website dependencies | ||
| if: steps.revision.outputs.changed == 'true' | ||
| run: python -m pip install -r requirements.txt | ||
|
|
||
| - name: Build the website | ||
| if: steps.revision.outputs.changed == 'true' | ||
| run: zensical build --clean | ||
|
|
||
| - name: Create or update pull request | ||
| if: steps.revision.outputs.changed == 'true' | ||
| uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7 | ||
| with: | ||
| branch: automation/sync-funnel-docs | ||
| commit-message: 'docs: sync Funnel develop documentation' | ||
| title: 'docs: sync Funnel develop documentation' | ||
| body: | | ||
| Update the CALYPR site from [Funnel develop](https://github.com/calypr/funnel/tree/${{ steps.revision.outputs.revision }}) at `${{ steps.revision.outputs.revision }}`. | ||
|
|
||
| Review the generated content and navigation before merging. This PR was prepared and built by the scheduled documentation sync. | ||
| delete-branch: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ venv | |
| site/ | ||
| .idea/ | ||
| attic/ | ||
|
|
||
| /.funnel-upstream/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,176 +1,57 @@ | ||
| # Developing CALYPR Docs | ||
|
|
||
| This repo now uses a Zensical-only build pipeline. MkDocs, `.nav.yml`, and the old multirepo plugin are gone. | ||
| The site is built with Zensical from the content in `docs/`. Navigation is | ||
| configured in `zensical.toml`. | ||
|
|
||
| The main thing to understand is that the published site is built from a generated docs tree, not directly from `docs/`. | ||
| ## Build locally | ||
|
|
||
| ## Mental Model | ||
|
|
||
| - `docs/` contains repo-authored source content. | ||
| - `.generated/docs/` is the staged build input that Zensical actually reads. | ||
| - `site/` is the rendered output from `zensical build`. | ||
| - `zensical.toml` is the only navigation source of truth. | ||
|
|
||
| The build pipeline is: | ||
|
|
||
| 1. Copy repo-authored docs from `docs/` into `.generated/docs/` | ||
| 2. Import selected upstream docs from `git-drs` and `syfon` | ||
| 3. Apply a few link rewrites needed for imported pages | ||
| 4. Add generated assets like `termynal.css` and `termynal.js` | ||
| 5. Run Zensical against `.generated/docs` | ||
|
|
||
| ## Core Commands | ||
|
|
||
| From the repo root: | ||
|
|
||
| ```bash | ||
| ```sh | ||
| python3 -m venv venv | ||
| source venv/bin/activate | ||
| pip install -r requirements.txt | ||
| make serve | ||
| make build | ||
| ``` | ||
|
|
||
| Useful commands: | ||
|
|
||
| - `make prepare`: rebuild `.generated/docs` | ||
| - `make build`: run a clean Zensical build | ||
| - `make serve`: prepare docs and start the local server | ||
| - `make update-branches`: update GitHub source links in authored docs to the preferred upstream branches | ||
|
|
||
| If you want the raw commands: | ||
|
|
||
| ```bash | ||
| python scripts/prepare_docs.py | ||
| zensical build --clean | ||
| zensical serve | ||
| ``` | ||
|
|
||
| ## Where Navigation Lives | ||
|
|
||
| Navigation is fully defined in [zensical.toml](./zensical.toml). | ||
|
|
||
| If you want to: | ||
|
|
||
| - rename a sidebar label | ||
| - move a section | ||
| - add a new page to the nav | ||
| - change the Tools grouping | ||
|
|
||
| edit the `nav = [...]` structure in `zensical.toml`. | ||
|
|
||
| There are no `.nav.yml` files anymore. If you add a page under `docs/` and do not add it to `zensical.toml`, it will not appear in the intended nav structure. | ||
|
|
||
| ## Source of Truth Rules | ||
|
|
||
| ### Repo-authored pages | ||
|
|
||
| These live in `docs/` and should be edited directly there. | ||
|
|
||
| Examples: | ||
|
|
||
| - `docs/index.md` | ||
| - `docs/calypr/...` | ||
| - `docs/tools/index.md` | ||
| - `docs/tools/funnel/...` | ||
| - `docs/tools/grip/...` | ||
|
|
||
| ### Imported tool pages | ||
|
|
||
| These are not authored in this repo anymore. | ||
|
|
||
| Current imported repos: | ||
|
|
||
| - `git-drs` | ||
| - `syfon` | ||
|
|
||
| Their imported file mappings live in [scripts/prepare_docs.py](./scripts/prepare_docs.py) under `REPO_IMPORTS`. | ||
| Use `make serve` for local development. The rendered site is written to `site/`. | ||
| The [GitHub Pages workflow](.github/workflows/publish.yml) builds and publishes | ||
| when changes reach `main`. | ||
|
|
||
| If you need to: | ||
| ## Funnel documentation | ||
|
|
||
| - add a new imported page | ||
| - stop importing a page | ||
| - change where an imported page lands under `tools/...` | ||
| The canonical Funnel reference pages live in the `docs/` | ||
| directory on [Funnel's `develop` branch](https://github.com/calypr/funnel/tree/develop/docs). | ||
| A scheduled workflow checks that branch daily and opens or updates a pull request | ||
| with changed pages and generated navigation. Review and merge the pull request | ||
| to publish the update. The workflow also runs the sync tests and builds the site. | ||
|
|
||
| update `REPO_IMPORTS` and usually also update `zensical.toml`. | ||
| The Funnel landing and download pages remain CALYPR site content. Reference | ||
| pages, section overviews, navigation, and documentation assets come from Funnel. | ||
| The public installer is copied from Funnel's `install.sh`. | ||
|
|
||
| Do not hand-edit imported pages under `.generated/docs`. They are build artifacts and will be replaced on the next `make prepare`. | ||
| The sync reads page titles from Markdown headings and keeps the existing | ||
| Overview URL by importing `docs/README.md` as `docs/docs.md`. Repository-source | ||
| links are rewritten to the imported commit on GitHub. Retired pages are removed, | ||
| including Elasticsearch. Private `docs/internal/` content and hidden files are | ||
| excluded. Unknown sections and Hugo shortcodes stop the sync before publication. | ||
|
|
||
| ## How Upstream Imports Work | ||
| The earlier exceptions for the GCP Batch, Kubernetes, and S3 guides have been | ||
| removed: their reconciled content is now in Funnel `develop`. | ||
|
|
||
| The importer prefers local sibling repos by default: | ||
| To run the sync locally against a Funnel checkout: | ||
|
|
||
| - `../git-drs` | ||
| - `../syfon` | ||
|
|
||
| So on a typical local machine, the imported content comes from whatever is currently checked out in those repos. | ||
|
|
||
| If local sibling repos are unavailable, the importer falls back to GitHub and picks the first available branch from [scripts/branch_config.json](./scripts/branch_config.json). | ||
|
|
||
| Current branch preference order: | ||
|
|
||
| - `development` | ||
| - `develop` | ||
| - `main` | ||
| - `master` | ||
|
|
||
| The last import decision is written to `.generated/imports.json`. | ||
|
|
||
| Environment overrides: | ||
|
|
||
| - `CALYPR_IMPORT_PREFER_LOCAL=0`: force remote import mode | ||
| - `CALYPR_IMPORT_GIT_DRS_DIR=/path/to/git-drs`: override local `git-drs` source | ||
| - `CALYPR_IMPORT_SYFON_DIR=/path/to/syfon`: override local `syfon` source | ||
|
|
||
| ## When You Add or Change Pages | ||
|
|
||
| ### Adding a repo-authored page | ||
|
|
||
| 1. Create the file under `docs/` | ||
| 2. Add it to `zensical.toml` | ||
| 3. Run `make serve` or `make build` | ||
|
|
||
| ### Adding an imported tool page | ||
|
|
||
| 1. Add the file mapping in `scripts/prepare_docs.py` | ||
| 2. Add the page to `zensical.toml` | ||
| 3. If the imported markdown links break after relocation, add a targeted rewrite in `MARKDOWN_REWRITES` | ||
| 4. Run `make prepare` | ||
| 5. Run `make build` | ||
|
|
||
| ## Styling and Overrides | ||
|
|
||
| Site-level assets still come from the repo: | ||
|
|
||
| - custom CSS: `docs/stylesheets/extra.css` | ||
| - custom JS: `docs/javascripts/nav-open-level1.js` | ||
| - theme overrides: `overrides/` | ||
|
|
||
| Those are copied into `.generated/docs` during preparation and then used by Zensical. | ||
|
|
||
| ## Deployment Behavior | ||
|
|
||
| Deployment uses the same pipeline as local builds: | ||
|
|
||
| - GitHub Pages runs `python scripts/prepare_docs.py` and then `zensical build --clean` | ||
| - Netlify does the same | ||
|
|
||
| That means deployment now depends on the import step succeeding. In CI, the importer will usually use remote GitHub repos rather than local sibling repos. | ||
|
|
||
| ## Files You Should Usually Ignore | ||
|
|
||
| - `.generated/` | ||
| - `site/` | ||
| - `.cache/` | ||
|
|
||
| These are build/runtime artifacts. | ||
|
|
||
| ## Known Caveat | ||
|
|
||
| Zensical currently reports some warnings from imported and legacy docs content, especially around bracketed literals like: | ||
| ```sh | ||
| python scripts/sync_funnel_docs.py /path/to/funnel --revision "$(git -C /path/to/funnel rev-parse HEAD)" | ||
| python -m unittest discover -s tests -v | ||
| make build | ||
| ``` | ||
|
|
||
| - `[SIGNED]` | ||
| - `[0]` | ||
| - `[]string` | ||
| - `map[string]string` | ||
| Generated Funnel pages and the marked Funnel navigation block should be changed | ||
| in the Funnel repository, not edited manually here. The sync branch is reused | ||
| while an update PR is open. If the Funnel revision is unchanged, no PR is created. | ||
|
|
||
| Those are content/rendering warnings, not a sign that the Zensical migration is broken. | ||
| The workflow needs both the organization policy and the repository setting that | ||
| allow GitHub Actions to create pull requests, plus `contents: write` and | ||
| `pull-requests: write` job permissions. | ||
| GitHub places PR workflows created with `GITHUB_TOKEN` in an approval-required | ||
| state. The sync workflow runs its own tests and site build before creating the | ||
| PR; a repository writer can approve the PR workflows separately. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whenever an upstream revision changes, the preceding unittest step creates
scripts/__pycache__/andtests/__pycache__/, neither of which is ignored by.gitignore(confirmed by running the tests locally). This step does not restrictadd-paths, whose documented default is that “all new and modified files are added,” so each generated documentation PR also commits those binary build artifacts. Their checkout-dependent timestamps can cause further binary churn on subsequent updates to the open PR. Ignore Python bytecode or restrict the action to the intended documentation and navigation paths. See the action's input documentation.Useful? React with 👍 / 👎.