So I created this post because with Helm v3, if you use --wait option and the timeout is reached and the deployment isn't successful, Helm marks the deployment as failed. The problem is that subsequent deployment (upgrade) that may contain the fix won't work as expected and it will end up in error like this:
[some-name] has no deployed release
The only way to fix this is manual intervene and deleting the previous failed release. This is not only scary but also against the automation process.
If we remove --wait option, Helm will mark the deployment as successful regardless.
My solution to this that works nicely is as per below:
- Remove --wait option from helm deploy
- Use this command to retrieve the list of deployment for that namespace that you are deploying against: kubectl get deployments -n ${namespace} -o jsonpath='{range .items[*].metadata}{.name}{","}{end}'
- You can use split to turn the comma separated list above into an array
- Then you can run multiple commands in parallel (we use Jenkins so it's easy to do so) as kubectl rollout status deployment ${deploymentName} --watch=true --timeout=${timeout} -n ${namespace}
- If after the timeout, for example 7m means 7 minutes, the deployment still not successful, the command exits with error
Problem solved.
A Jenkins groovy function can look like this:
Note: devopsToolsContainerName is used to switch to a container in which kubectl command is available.
def call(map) {
def namespace = map?.namespace
assert namespace : '[namespace] argument must be procided'
def devopsToolsContainerName = map?.devopsToolsContainerName ?: 'devops-tools'
def timeout = map?.timeout ?: '7m'
def runInStage = map?.runInStage ?: true
def func = {
container(devopsToolsContainerName) {
String result = sh(returnStdout: true, script: "kubectl get deployments -n ${namespace} -o jsonpath='{range .items[*].metadata}{.name}{\",\"}{end}'")
log "=> ${result}"
String[] deployments = result.split(',')
log "=> ${deployments}"
def stages = [failFast: true]
def report = [:]
deployments.each {
stages["${it}"] = {
report["${it}"] = sh(returnStdout: true, script: "kubectl rollout status deployment ${it} --watch=true --timeout=${timeout} -n ${namespace}")
log "${it} report => ${report["${it}"]}"
}
}
parallel stages
}
}
if(runInStage) {
stage("Waiting for deployment") {
func()
}
} else {
func()
}
}
Comments