mirror of
https://github.com/hashicorp/vault.git
synced 2026-03-28 08:57:31 +00:00
61b6ae14e5
The `pipeline` utility started as collection of small CLI utilities that we found useful for the Vault CI/CD pipeline. Rather than engineering complex bash scripts in YAML blocks, instead, we could build small, reusable, testable actions and integrate the into a single binary. No more copying and pasting loads of bash from YAML, instead we can copy a single command and run the same thing locally that we can in CI.
As we've continued to invest in the utilities capability, it's become clear that other CI pipelines would benefit from the same functionality that we've been building. This change represents the first significant work to make the utility truly generic in a HashiCorp repo that utilizes CRT sense. Once all the Vault specifics have been extracted we hope to move the utility out of the repo and make it available everywhere.
The primary change here is to move our changed file grouping configuration out of the `changed` package entirely. Instead of checkers that are written as Go code, we have created a new configuration file for the `pipeline` utility called `pipeline.hcl` While there are certainly other things that will eventually be configurable here, the only thing we've added support for is `changed_files`, which allows configuring how to match a given changed files path to a group name.
The DSL is fairly simple:
```hcl
changed_files {
// One or more groups can be defined
group "group_name_label" {
// Zero or more ignore blocks can be defined
ignore {
base_dir = []
base_name = []
base_name_prefix = []
contains = []
extension = []
file = []
}
// One or more match blocks can be defined
match {
base_dir = []
base_name = []
base_name_prefix = []
contains = []
extension = []
file = []
}
}
}
```
For example,
```hcl
// Create a changed_files block where we can define our changed files groups
changed_files {
// Group blocks take one label which is the name of the group
group "app" {
// Groups can ignore based on some criteria.
ignore {
// In this instance, we'll ignore any file that begins with
// tools/pipeline. All paths will be relative to the git repository
// root directory. The joinpath() function is here to support paths
// that are agnostic to the operating systems path separator. While
// it's unlikely that you'll need them, several cty stdlib functions
// are available.
base_dir = [joinpath("tools", "pipeline")]
}
// Groups must define at least one match block.
match {
// This will match any file with the .go extension (except for
// those that will be excluded with our ignore directive aboe
extension = [".go"]
}
// Groups can contain more than one match block. If any of the match
// blocks meet their criteria the group will be associated with the
// changed file
match {
base_name = ["go.mod", "go.sum"]
}
// If groups have more than one attribute set, each attribute group
// must match in order for the match.
match {
// Here we only match files that contain "raft_autopilot" in the
// path with the .go extension
extension = [".go"]
contains = ["raft_autopilot"]
}
}
group "autopilot" {
// Ignore blocks have the same attributes as match blocks
match {
// The base directory.
base_dir = [
"changelog",
joinpath("tools", "codechecker"),
]
// The base of the file
base_name = ["README.md"]
// A prefix string match on a files name.
base_name_prefix = ["buf."]
// Any string match in the files full path
contains = [
"-ce",
"_ce",
"-oss",
"_oss",
]
// The file's extension
extension = [
".hcl",
".md",
".sh",
".yaml",
".yml",
]
// An exact file match
file = [
# These exist on CE branches to please Github Actions.
joinpath(".github", "workflows", "build-artifacts-ent.yml"),
joinpath(".github", "workflows", "backport-automation-ent.yml"),
]
}
}
}
```
The default location of the config is `.release/pipeline.hcl`. All of our prior checks have been migrated to the DSL file present in this change.
- We had several commands that used the changed files groups that were built into the library. This change requires us to instead load the configuration from the file and use the user defined groupings.
- Several commands now take some part of that configuration in the request type. When possible we use the version parsed by the root command and verify in the request body rather than attempt to load the configuration.
- We also refactor the loading and parsing of `.release/versions.hcl` in the same manner. Now we automatically parse the file in the default locations relative to the git repo root.
- Our root command now has two new flags `--pipeline-config` and `--versions-config` which allow specifying a default location for each file. Commands which previously accepted flags or args to configure the versions file have been updated to use the global root flags instead. We've also removed the previous implementation that would recursively search backwards from the working directory to find the `versions.hcl` file. Instead we only support loading the file from the default location relative to the Git repo root.
- All instances of changed `pipeline` command invocations have been update to support the new auto-loading of configuration.
- A new configuration sub-command with validation exists to quickly validate a configuration file. `pipeline config validate`
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
153 lines
2.0 KiB
Plaintext
153 lines
2.0 KiB
Plaintext
# Compiled Object files, Static and Dynamic libs (Shared Objects)
|
|
*.o
|
|
*.a
|
|
*.so
|
|
|
|
# Folders
|
|
_obj
|
|
_test
|
|
.cover
|
|
|
|
# Architecture specific extensions/prefixes
|
|
*.[568vq]
|
|
[568vq].out
|
|
|
|
*.cgo1.go
|
|
*.cgo2.c
|
|
_cgo_defun.c
|
|
_cgo_gotypes.go
|
|
_cgo_export.*
|
|
|
|
_testmain.go
|
|
|
|
*.exe
|
|
*.test
|
|
*.prof
|
|
|
|
# Other dirs
|
|
/bin/
|
|
/pkg/
|
|
|
|
# Generated Web UI goes here
|
|
/http/web_ui/*.*
|
|
/http/web_ui/**/*.*
|
|
|
|
# Vault-specific
|
|
example.hcl
|
|
example.vault.d
|
|
|
|
# Without this, the *.[568vq] above ignores this folder.
|
|
!**/graphrbac/1.6
|
|
|
|
# Ruby
|
|
website/vendor
|
|
website/.bundle
|
|
website/build
|
|
website/tmp
|
|
|
|
# Vagrant
|
|
.vagrant/
|
|
Vagrantfile
|
|
|
|
# Configs
|
|
*.hcl
|
|
!.copywrite.hcl
|
|
!.release/ci.hcl
|
|
!.release/pipeline.hcl
|
|
!.release/security-scan.hcl
|
|
!.release/linux/package/etc/vault.d/vault.hcl
|
|
!enos/**/*.hcl
|
|
!**/fixtures/*.hcl
|
|
!**/test-fixtures/**/*.hcl
|
|
!**/testdata/*.hcl
|
|
|
|
# Enos
|
|
.enos
|
|
enos-local.vars.hcl
|
|
enos/**/support
|
|
enos/**/kubeconfig
|
|
.terraform
|
|
.terraform.lock.hcl
|
|
.tfstate.*
|
|
|
|
.DS_Store
|
|
.idea
|
|
.vscode
|
|
|
|
# VSCode debugger executable
|
|
__debug_bin*
|
|
|
|
dist/*
|
|
|
|
# ignore ctags
|
|
./tags
|
|
|
|
# Editor backups
|
|
*~
|
|
*.sw[a-z]
|
|
|
|
# IntelliJ IDEA project files
|
|
.idea
|
|
*.ipr
|
|
*.iml
|
|
|
|
# compiled output
|
|
ui/dist
|
|
ui/tmp
|
|
ui/root
|
|
|
|
# dependencies
|
|
ui/node_modules
|
|
ui/bower_components
|
|
|
|
# misc
|
|
ui/.DS_Store
|
|
ui/.sass-cache
|
|
ui/connect.lock
|
|
ui/coverage/*
|
|
ui/libpeerconnection.log
|
|
ui/npm-debug.log
|
|
ui/test-reports/*
|
|
ui/testem.log
|
|
|
|
# used for JS acceptance tests
|
|
ui/tests/helpers/vault-keys.js
|
|
ui/vault-ui-integration-server.pid
|
|
|
|
# for building static assets
|
|
node_modules
|
|
|
|
# Website
|
|
website/.bundle
|
|
website/build/
|
|
website/npm-debug.log
|
|
website/vendor
|
|
website/.bundle
|
|
website/.cache
|
|
website/assets/node_modules
|
|
website/assets/public
|
|
website/components/node_modules
|
|
|
|
.buildcache/
|
|
.releaser/
|
|
*.log
|
|
|
|
tools/godoctests/.bin
|
|
tools/gonilnilfunctions/.bin
|
|
tools/codechecker/.bin
|
|
tools/pipeline/.bin
|
|
tools/pipeline/pipeline
|
|
.ci-bootstrap
|
|
|
|
# Ignore stubmaker outputs on ent so they don't get checked in
|
|
*_stubs_ent.go
|
|
|
|
# scratch directory holding random stuff
|
|
scratch
|
|
|
|
# ignore Idea/Goland .run directory
|
|
.run
|
|
|
|
# bob
|
|
.bob/notes
|