DevOps/truecharts
2
0
Fork 0
Code Issues Pull Requests Actions 33 Packages Projects Releases 1 Wiki Activity

chore(ci): Improved lint speed (#8809)

Browse Source 
**Description**
<!--
Please include a summary of the change and which issue is fixed. Please
also include relevant motivation and context. List any dependencies that
are required for this change.
-->
⚒️ Fixes  # <!--(issue)-->

TODO: Switch to scripted checks (To replace ct-list's tasks):
- [x] Helm Lint
- [x] Version Checking
- [x] Run the lint inside the devcontainer (so it have the below tools
available)
- [x] YAML Schema check on Chart.yaml
- [ ] Maintener check on Chart.yaml (Moved to another PR in the future)
- [x] YAML Lint on Chart.yaml
- [x] YAML Lint on values.yaml
- [x] Sort output of lint results (failed on the bottom)
- [x] Combine verify deps in the lint job, since the verification runs
inside the fetch dep (which runs on lint job too)
- [x] Remove one of the 2 "get changed" functions (one based on tags
other on commit)
After that, move linting to single step 


https://github.com/helm/chart-testing/blob/main/pkg/tool/linter.go
https://github.com/helm/chart-testing/blob/main/pkg/tool/account.go

**⚙️ Type of change**

- [ ] ⚙️ Feature/App addition
- [ ] 🪛 Bugfix
- [ ] ⚠️ Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] 🔃 Refactor of current code

**🧪 How Has This Been Tested?**
<!--
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration
-->

**📃 Notes:**
<!-- Please enter any other relevant information here -->

**✔️ Checklist:**

- [x] ⚖️ My code follows the style guidelines of this project
- [x] 👀 I have performed a self-review of my own code
- [x] #️⃣ I have commented my code, particularly in hard-to-understand
areas
- [x] 📄 I have made corresponding changes to the documentation
- [x] ⚠️ My changes generate no new warnings
- [x] 🧪 I have added tests to this description that prove my fix is
effective or that my feature works
- [x] ⬆️ I increased versions for any altered app according to semantic
versioning

**➕ App addition**

If this PR is an app addition please make sure you have done the
following.

- [ ] 🪞 I have opened a PR on
[truecharts/containers](https://github.com/truecharts/containers) adding
the container to TrueCharts mirror repo.
- [ ] 🖼️ I have added an icon in the Chart's root directory called
`icon.png`

---

_Please don't blindly check all the boxes. Read them and only check
those that apply.
Those checkboxes are there for the reviewer to see what is this all
about and
the status of this PR with a quick glance._
This commit is contained in:
Stavros Kois
2023-05-23 10:26:41 +02:00
committed by GitHub
parent 66a1b19196
commit 702a0c3aba
39 changed files with 522 additions and 486 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/charts-lint.yaml
+50 -370
View File
@@ -11,400 +11,80 @@ on:
type: string
jobs:
verify-dep-sig:
lint-and-verify:
name: Lint Charts and Verify Dependencies
runs-on: ubuntu-latest
name: Verify Dependency Signatures
container:
image: ghcr.io/truecharts/devcontainer:3.1.10@sha256:c239addf725eb5cedf79517f8089fdafdc32b5270d1893ee87ae6e511b9bcae3
image: tccr.io/truecharts/devcontainer:v3.1.38@sha256:2fd9355bc6791461430db87a04f5fe7b49c46bc5957bc348f1efbff4c6c7913d
steps:
- name: Install Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- name: Prep Helm
run: |
helm repo add truecharts https://charts.truecharts.org
helm repo add truecharts-library https://library-charts.truecharts.org
helm repo add truecharts-deps https://deps.truecharts.org
helm repo update
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: Setting repo parent dir as safe safe.directory
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
- name: Collect changes
id: collect-changes
uses: ./.github/actions/collect-changes
- name: Fetch and Verify dependencies
shell: bash
if: |
steps.collect-changes.outputs.changesDetectedAfterTag == 'true'
- name: Prep Helm
run: |
CHARTS=(${{ steps.collect-changes.outputs.modifiedChartsAfterTag }})
charts_path="charts"
for changed in ${CHARTS[@]}; do
.github/scripts/fetch_helm_deps.sh "${changed}"
done
helm repo add truecharts https://charts.truecharts.org
helm repo add truecharts-deps https://deps.truecharts.org
helm repo add truecharts-library https://library-charts.truecharts.org
helm repo add truecharts-deps https://deps.truecharts.org
helm repo update
generate-lint-matrix:
name: Generate matrix for lint
runs-on: ubuntu-latest
outputs:
matrix1: |
{
"chart": ${{ steps.list-changed.outputs.charts1 }}
}
matrix2: |
{
"chart": ${{ steps.list-changed.outputs.charts2 }}
}
matrix3: |
{
"chart": ${{ steps.list-changed.outputs.charts3 }}
}
matrix4: |
{
"chart": ${{ steps.list-changed.outputs.charts4 }}
}
matrix5: |
{
"chart": ${{ steps.list-changed.outputs.charts5 }}
}
matrix6: |
{
"chart": ${{ steps.list-changed.outputs.charts6 }}
}
detected1: ${{ steps.list-changed.outputs.detected1 }}
detected2: ${{ steps.list-changed.outputs.detected2 }}
detected3: ${{ steps.list-changed.outputs.detected3 }}
detected4: ${{ steps.list-changed.outputs.detected4 }}
detected5: ${{ steps.list-changed.outputs.detected5 }}
detected6: ${{ steps.list-changed.outputs.detected6 }}
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Collect changes
- name: Collect changes (branch-based)
id: list-changed
if: inputs.chartChangesDetected == 'true'
shell: bash
run: |
EXCLUDED=$(yq eval -o=json '.excluded-charts // []' .github/ct-lint.yaml)
CHARTS=$(ct list-changed --config .github/ct-lint.yaml)
CHARTS_JSON=$(echo "${CHARTS}" | jq -R -s -c 'split("\n")[:-1]')
OUTPUT_JSON=$(echo "{\"excluded\": ${EXCLUDED}, \"all\": ${CHARTS_JSON}}" | jq -c '.all-.excluded')
OUTPUT_JSON1=$((echo $OUTPUT_JSON | jq -r -c '.[:50]'; echo $OUTPUT_JSON | jq -r -c '.[301:350]'; echo $OUTPUT_JSON | jq -r -c '.[601:650]'; echo $OUTPUT_JSON | jq -r -c '.[901:950]'; echo $OUTPUT_JSON | jq -r -c '.[1201:1250]') | jq -s add)
OUTPUT_JSON2=$((echo $OUTPUT_JSON | jq -r -c '.[51:100]'; echo $OUTPUT_JSON | jq -r -c '.[351:400]'; echo $OUTPUT_JSON | jq -r -c '.[651:700]'; echo $OUTPUT_JSON | jq -r -c '.[951:1000]'; echo $OUTPUT_JSON | jq -r -c '.[1251:1300]') | jq -s add)
OUTPUT_JSON3=$((echo $OUTPUT_JSON | jq -r -c '.[101:150]'; echo $OUTPUT_JSON | jq -r -c '.[401:450]'; echo $OUTPUT_JSON | jq -r -c '.[701:750]'; echo $OUTPUT_JSON | jq -r -c '.[1001:1050]'; echo $OUTPUT_JSON | jq -r -c '.[1301:1350]') | jq -s add)
OUTPUT_JSON4=$((echo $OUTPUT_JSON | jq -r -c '.[151:200]'; echo $OUTPUT_JSON | jq -r -c '.[451:500]'; echo $OUTPUT_JSON | jq -r -c '.[751:800]'; echo $OUTPUT_JSON | jq -r -c '.[1051:1100]'; echo $OUTPUT_JSON | jq -r -c '.[1351:1400]') | jq -s add)
OUTPUT_JSON5=$((echo $OUTPUT_JSON | jq -r -c '.[201:250]'; echo $OUTPUT_JSON | jq -r -c '.[501:550]'; echo $OUTPUT_JSON | jq -r -c '.[801:850]'; echo $OUTPUT_JSON | jq -r -c '.[1101:1150]'; echo $OUTPUT_JSON | jq -r -c '.[1401:1450]') | jq -s add)
OUTPUT_JSON6=$((echo $OUTPUT_JSON | jq -r -c '.[251:300]'; echo $OUTPUT_JSON | jq -r -c '.[551:600]'; echo $OUTPUT_JSON | jq -r -c '.[851:900]'; echo $OUTPUT_JSON | jq -r -c '.[1151:1200]'; echo $OUTPUT_JSON | jq -r -c '.[1451:1500]') | jq -s add)
echo ::set-output name=charts1::${OUTPUT_JSON1}
echo ::set-output name=charts2::${OUTPUT_JSON2}
echo ::set-output name=charts3::${OUTPUT_JSON3}
echo ::set-output name=charts4::${OUTPUT_JSON4}
echo ::set-output name=charts5::${OUTPUT_JSON5}
echo ::set-output name=charts6::${OUTPUT_JSON6}
if [[ $(echo ${OUTPUT_JSON1} | jq -c '. | length') -gt 0 ]]; then
echo "::set-output name=detected1::true"
fi
if [[ $(echo ${OUTPUT_JSON2} | jq -c '. | length') -gt 0 ]]; then
echo "::set-output name=detected2::true"
fi
if [[ $(echo ${OUTPUT_JSON3} | jq -c '. | length') -gt 0 ]]; then
echo "::set-output name=detected3::true"
fi
if [[ $(echo ${OUTPUT_JSON4} | jq -c '. | length') -gt 0 ]]; then
echo "::set-output name=detected4::true"
fi
if [[ $(echo ${OUTPUT_JSON5} | jq -c '. | length') -gt 0 ]]; then
echo "::set-output name=detected5::true"
fi
if [[ $(echo ${OUTPUT_JSON6} | jq -c '. | length') -gt 0 ]]; then
echo "::set-output name=detected6::true"
EXCLUDED_JSON=$(go-yq eval -o=json '.excluded-charts // []' .github/ct-lint.yaml)
CHARTS_JSON=$(echo "${CHARTS}" | jq --raw-input '.' | jq --compact-output --slurp '.')
OUTPUT_JSON=$(echo "{\"excluded\": ${EXCLUDED_JSON}, \"all\": ${CHARTS_JSON}}" | jq --compact-output '.all-.excluded')
echo CHANGED_CHARTS=${OUTPUT_JSON} >> "$GITHUB_OUTPUT"
if [[ $(echo ${OUTPUT_JSON} | jq --compact-output '. | length') -gt 0 ]]; then
echo "detected=true" >> "$GITHUB_OUTPUT"
fi
lint-charts1:
needs:
- generate-lint-matrix
if: needs.generate-lint-matrix.outputs.detected1 == 'true'
name: Lint charts Part 1
strategy:
matrix: ${{ fromJson(needs.generate-lint-matrix.outputs.matrix1) }}
fail-fast: false
max-parallel: 10
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: lint Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- uses: actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b # v4
with:
python-version: "3.10"
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Run chart-testing (lint)
run: ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}" || ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}"
lint-charts2:
needs:
- generate-lint-matrix
if: needs.generate-lint-matrix.outputs.detected2 == 'true'
name: Lint charts Part 2
strategy:
matrix: ${{ fromJson(needs.generate-lint-matrix.outputs.matrix2) }}
fail-fast: false
max-parallel: 10
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: lint Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- uses: actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b # v4
with:
python-version: "3.10"
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Run chart-testing (lint)
run: ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}" || ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}"
lint-charts3:
needs:
- generate-lint-matrix
if: needs.generate-lint-matrix.outputs.detected3 == 'true'
name: Lint charts Part 3
strategy:
matrix: ${{ fromJson(needs.generate-lint-matrix.outputs.matrix3) }}
fail-fast: false
max-parallel: 10
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: lint Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- uses: actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b # v4
with:
python-version: "3.10"
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Run chart-testing (lint)
run: ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}" || ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}"
lint-charts4:
needs:
- generate-lint-matrix
if: needs.generate-lint-matrix.outputs.detected4 == 'true'
name: Lint charts Part 4
strategy:
matrix: ${{ fromJson(needs.generate-lint-matrix.outputs.matrix4) }}
fail-fast: false
max-parallel: 10
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: lint Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- uses: actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b # v4
with:
python-version: "3.10"
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Run chart-testing (lint)
run: ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}" || ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}"
lint-charts5:
needs:
- generate-lint-matrix
if: needs.generate-lint-matrix.outputs.detected5 == 'true'
name: Lint charts Part 5
strategy:
matrix: ${{ fromJson(needs.generate-lint-matrix.outputs.matrix5) }}
fail-fast: false
max-parallel: 10
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: lint Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- uses: actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b # v4
with:
python-version: "3.10"
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Run chart-testing (lint)
run: ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}" || ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}"
lint-charts6:
needs:
- generate-lint-matrix
if: needs.generate-lint-matrix.outputs.detected6 == 'true'
name: Lint charts Part 6
strategy:
matrix: ${{ fromJson(needs.generate-lint-matrix.outputs.matrix6) }}
fail-fast: false
max-parallel: 10
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
with:
fetch-depth: 0
ref: ${{ inputs.checkoutCommit }}
- name: lint Kubernetes tools
uses: yokawasa/action-setup-kube-tools@af4ebb1af1efd30c5bd84a2e9773355ad6362a33 # v0.9.3
with:
setup-tools: |
helmv3
helm: "3.8.0"
- uses: actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b # v4
with:
python-version: "3.10"
- name: Set up chart-testing
uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0
- name: Run chart-testing (lint)
run: ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}" || ct lint --config ".github/ct-lint.yaml" --charts "${{ matrix.chart }}"
# Summarize matrix https://github.community/t/status-check-for-a-matrix-jobs/127354/7
lint_success:
needs:
- generate-lint-matrix
- lint-charts1
- lint-charts2
- lint-charts3
- lint-charts4
- lint-charts5
- lint-charts6
if: |
always()
name: Lint successful
runs-on: ubuntu-latest
steps:
- name: Check lint matrix status
- name: Fetch and Verify dependencies
shell: bash
if: steps.list-changed.outputs.detected == 'true'
env:
charts_path: "./"
run: |
res1="${{needs.lint-charts1.result}}"
res2="${{needs.lint-charts2.result}}"
res3="${{needs.lint-charts3.result}}"
res4="${{needs.lint-charts4.result}}"
res5="${{needs.lint-charts5.result}}"
res6="${{needs.lint-charts6.result}}"
CHANGED=$(echo '${{ steps.list-changed.outputs.CHANGED_CHARTS }}' | jq --raw-output '.[]')
for changed in ${CHANGED}; do
.github/scripts/fetch_helm_deps.sh "${changed}"
done
if [[ $res1 == "success" || $res1 == "skipped" ]]; then
echo "lint tests 1 successfull..."
else
echo "lint tests 1 error, stopping due to failure..."
exit 1
fi
- name: Run Chart Linting
continue-on-error: true
id: lint
if: steps.list-changed.outputs.detected == 'true'
env:
result_file: /tmp/lint_result.txt
run: |
CHANGED=$(echo '${{ steps.list-changed.outputs.CHANGED_CHARTS }}' | jq --raw-output '.[]')
# If the github.base_ref is empty (eg it runs outside of a PR) it fails back to origin/master
.github/scripts/tc-lint.sh '${{ steps.list-changed.outputs.CHANGED_CHARTS }}' "origin/${{ github.base_ref }}"
if [[ $res2 == "success" || $res2 == "skipped" ]]; then
echo "lint tests 2 successfull..."
else
echo "lint tests 2 error, stopping due to failure..."
exit 1
fi
- name: Create/Update comment
if: steps.list-changed.outputs.detected == 'true'
uses: thollander/actions-comment-pull-request@632cf9ce90574d125be56b5f3405cda41a84e2fd # v2
with:
filePath: /tmp/lint_result.txt
comment_tag: lint_results
mode: recreate
if [[ $res3 == "success" || $res3 == "skipped" ]]; then
echo "lint tests 3 successfull..."
else
echo "lint tests 3 error, stopping due to failure..."
exit 1
fi
if [[ $res4 == "success" || $res4 == "skipped" ]]; then
echo "lint tests 4 successfull..."
else
echo "lint tests 4 error, stopping due to failure..."
exit 1
fi
if [[ $res5 == "success" || $res5 == "skipped" ]]; then
echo "lint tests 5 successfull..."
else
echo "lint tests 5 error, stopping due to failure..."
exit 1
fi
if [[ $res6 == "success" || $res6 == "skipped" ]]; then
echo "lint tests 6 successfull..."
else
echo "lint tests 6 error, stopping due to failure..."
- name: Lint Result
if: steps.list-changed.outputs.detected == 'true'
shell: bash
run: |
if [ "${{ steps.lint.outcome }}" != "success" ]; then
echo "Linting failed"
exit 1
fi
.github/workflows/charts-release.yaml
-2
View File
@@ -11,7 +11,6 @@ on:
- "charts/**"
jobs:
release-helm:
runs-on: ubuntu-latest
container:
@@ -238,7 +237,6 @@ jobs:
git commit -sm "Commit released docs for TrueCharts" || exit 0
git push
- name: Checkout Catalog
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3
if: |