My requirement is to automate the creation of secrets in Keyvault without exposing the values of these secrets during the pipeline execution(AzureDevops Server).
tried, different options such as having input paramaters (it will display the values when its perform initialization) , runtime variables etc (list of variables, or dynamic creation as per input is not possible), but couldn't fulfil our requirements.
Finally planned to use the below script to read values from library group and create the values in key vault. But when we are masking the secret values in Library group (by locking), the secrets are getting created with null value only. So is there any way to retrieve the values secret variables and create them in key vault only if the secret value is different(if existing)
-bash: |
az login --service-principal --username $(spid) --password $(spsecret) --tenant $(tenantid)
az account set --subscription ${{ variables.subscription }}
az config set extension.use_dynamic_install=yes_without_prompt
groupID=`az pipelines variable-group list -p myproject --group-name ${{ environment }}-${{ parameters.myapp}}-kv-secret --query '[].id' -o tsv`
echo "grouup id is $groupID"
variables=$(az pipelines variable-group variable list -p myproject --group-id $groupID --output json)
echo "$variables" | jq -r 'keys[] as $k | "\($k)=\(.[$k].value)"' | while read variable; do
name=$(echo "$variable" | cut -d= -f1)
value=$(echo "$variable" | cut -d= -f2)
if ! az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" &>/dev/null; then
expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')
az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate
secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)
if [[ "$secretValue" != "$value" ]]; then
echo "Failed to create secret $secretName in Key Vault $keyVaultName"
exit 1
fi
if [[ "$secretValue" == "$value" ]]; then
echo "Created secret $secretName in Key Vault $keyVaultName, hence cleaning the variable from the keyvault"
az pipelines variable-group variable delete --group-id $groupID --name "$name --yes"
fi
fi
done
Updated script,
where the secret is getting updated each time even though the same value is in the file.
while IFS= read -r line
do
name=$(echo "$line" | cut -d '=' -f 1)
value=$(echo "$line" | cut -d '=' -f 2)
if ! az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" &>/dev/null; then
expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')
az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate --output none
secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)
if [[ "$secretValue" != "$value" ]]; then
echo "Failed to create secret $name in Key Vault ${{ variables.podkv }}"
exit 1
else
echo "Created secret $name in Key Vault ${{ variables.podkv }}, hence cleaning the variable from the keyvault"
fi
else
echo "The secretName $name already exists, so checking if the secret value is the same"
existing_secretvalue=$(az keyvault secret show --name "$name" --vault-name "$(${{ variables.podkv }})" --query "value")
if [[ "$existing_secretValue" != "$value" ]]; then
echo "there is secret value change for the secretname $name , so Creating a new secret value for the existing secret $name"
expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')
az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate --output none
new_secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)
if [[ "$new_secretValue" != "$value" ]]; then
echo "Failed to create secret $name in Key Vault ${{ variables.podkv }}"
exit 1
else
echo "Created secret $name in Key Vault ${{ variables.podkv }}, hence cleaning the variable from the keyvault"
fi
fi
fi
done < "$(keyvault.secureFilePath)"
Best Answer
This is an expected behavior. When we set the secret variable in Variable Group, we are not able to use Rest API or Azure DevOps CLI to get the variable. It will show as NULL Value.
In this case, I am afraid that the variable group method can not meet your requirement.
Based on your description, I suggest that you can save the variables in a file and add the file to Azure DevOps Secure file(Pipelines -> Library -> Secure file).
The secure file can only be downloaded with the Download secure file v1 task in Azure Pipelines.
Here are the steps:
Step1: We can create a file and save the variables.
For example:
test.txt
Step2: Upload the file to secure file.
For example:
Step3: We can use the Download secure file v1 task to download the secure file and then use the bash script to loop the values in the file. Finally we can set the Key vault value.
Here is an example:
In this case, the variable value will not be exposed during the pipeline run. And the secure file will be automatically deleted(On agent machine) after Pipeline run completing.
For more detailed info, you can refer to this doc: Use secure files
You can add the Azure CLI: Retrieve a secret from Key Vault to bash loop to compare secret value and current variable value.
Update:
I can reproduce the same issue when using the same bash script.
The cause of the issue is that when we use the Azure CLI to get the existing secret value. it will automatically add
\rat the end of the value.This line:
For example:
To solve this issue, you can add
--output tsvto format the output of the secret value.For example: