Problem
I recently encountered an issue where a customer wasn’t able to deploy AKS clusters on Azure Local via Arc. I was able to reproduce the same issue in my lab and it turned out to be a widespread issue. In this blog I will detail the error and provide a work-a-round.
The failure message I saw in my lab cluster was “Kubernetes version 1.29.4 is not ready for use on Linux” (see below screenshot).
This was strange as I was able to select kubenetes version 1.29.4 when using the deployment wizard in the Portal, and I had previously deployed an Aks cluster at that version.
Troubleshooting
I attempted to recreate/reset the versions using the steps in the below article, but that did not work:
Troubleshoot K8sVersionValidation error code – AKS enabled by Azure Arc | Microsoft Learn
I then confirmed the latest supported kubenetes versions, including 1.29, are listed for my instance.
The commands in the same article above with a different output view:
#Complete az device login
$Subscription = "<<subsid>>"
$ResourceGroup = "<<rgname>>"
$CustomLocationName = "<<customlocname>>"
az login --use-device-code
az account set --subscription $Subscription
#add aksarc extension f needed
#az extension add --name aksarc
#Set CL Resource Id
$cl_id = "/subscriptions/$Subscription/resourceGroups/$ResourceGroup/providers/Microsoft.ExtendedLocation/customLocations/$CustomLocationName"
#Get image versions
(az aksarc get-versions --custom-location $cl_id | ConvertFrom-Json).properties.values
Cause
It appears that since updating to version 10.2411.3.2, the Aks Images in the MOC Gallery are in a failed state and not downloaded to the cluster and thus not ready to use. This can occur after reboots.
Check the status of images:
#complete az device login
$Subscription = "<<subsid>>"
$ResourceGroup = "<<rgname>>"
$CustomLocationName = "<<customlocname>>"
az login --use-device-code
az account set --subscription $Subscription
#add aksarc extension f needed
#az extension add --name aksarc
#Set CL Resource Id
$cl_id = "/subscriptions/$Subscription/resourceGroups/$ResourceGroup/providers/Microsoft.ExtendedLocation/customLocations/$CustomLocationName"
# Get the status of images
az aksarc get-versions --custom-location $cl_id
Resolution
*Please note that this is a work-a-round and not a permanent fix. It also involves changing the version of MOC, which could even mean it’s downgraded, which is a cluster wide change and so please be careful in your production systems and test this first!
To fix the issue, you can delete the images in a ‘false’ ready state and that will trigger them to be downloaded again.
#Login to Azure
$Subscription = "<<subsid>>"
$ResourceGroup = "<<rgname>>"
$Tenant ="<<tenantid>>"
Connect-AzAccount -SubscriptionId $Subscription -TenantId $Tenant -DeviceCode
#Get moc version
Get-MocVersion
#Update moc if needed
#Update-Moc -version 1.11.3.10307
#Get failed images
get-mocgalleryimage -location MocLocation | ForEach-Object { [PSCustomObject]$_ } | Where-Object { $_.name -match "linux-cblmariner|windows-windows" -and $_.properties.statuses.ProvisionState -match "delete_failed" } | ConvertTo-Json -Depth 10
#Remove failed images
remove-mocgalleryimage -location MocLocation -name "<image in delete_failed state>"
#E.g Image names
"windows-windows2019-0.4.1.11203"
"linux-cblmariner-0.4.1.11203"
"windows-windows2022-0.4.1.11203"After this process, the images should start downloading again and an example of how they are storage on the CSV’s is below:
You can then check the status again using he above commands, and when the issue is resolved you should see the Ready state is ‘true’.
Once the images are ‘ready’ and the images are downloaded, you should then be able to deploy AKS clusters again.
Conclusion
I hope the has helped you if facing this issue and we expect Microsoft will fix this in the next update.