Create a Secure Service Fabric cluster on Azure in a fully automated manner

Scenario

In this post we will cover how DevOps teams can fully automate the process of creating Secure Service Fabric clusters on Azure.
By fully automating the process, DevOps teams will achieve better productivity and a more predictable release and deployment pipeline. We will use Azure Resource Manager json templates and some powershell scripts.
By using the Azure portal you can now create Azure Clusters, more info here Setting up a Service Fabric Cluster from the Azure Portal
Also, you can manually create Secured Service Fabric clusters from the Azure Portal, How to secure Service Fabric cluster using certificates , but this requires multiple steps.
But in this blog post we will cover how DevOps teams can fully automate the process of creating secure Service Fabric Clusters.

Service Fabric is a distributed systems platform used to build scalable, reliable, and easily-managed applications for the cloud, and finally Service Fabric is available on Azure.
For more info on Service Fabric : Overview of Service Fabric

Get The Code !

Use this ARM template and PS Script to create Service Fabric clusters on Azure in a fully automated manner.

Steps :

In order to create a secured Service Fabric cluster on Azure DevOps teams will have to:
Step 0 : Clone the Repo and Specify your own settings in the parameters file and the PS Script.
-Step 1 : Generate (or use an existing) Certificate
-Step 2 : Create an Azure Key Vault
-Step 3 : Upload the Certificate to Azure KeyVault
-Step 4 : Create a VNET, Load balancer and Public IP and all the related objects.
-Step 5 : Create the VMs ad add the ServiceFabric extension to all the VMs.

Step 0 : Specify your own settings in the parameters file and the PS Script.

Clone the github repo : https://github.com/GONZALORUIZ/AzureServiceFabric.git
Before you execute the script, make sure that you specify the following settings:
-Location (in the PS Script) : The Azure location in which you want to create all the services
-dnsName (In the PS Script) : The dns name that you will to assign to the Public IP that you will use to access the service fabric cluster.

$dnsName = "YOURNAME$instanceNumber"
$resourceGroupName = "YOURRESOURCEGROUP$instanceNumber"
"vmSize": {
        "value": "Standard_D1"
      },
"numberOfNodes": {
        "value": 5
      },
"adminUserName": {
        "value": "YOURADMIN"
      }

Step 1 : Generate (or use an existing) Certificate

For Dev and Test purposes, we will generate Self-signed certificates. The following function will create the certificate , export it to your current folder and add it to the Cert:CurrentUserTrustedPeople the Cert:CurrentUserMy stores.

function SetupCertificates()
{
 
    If (-not (Test-Path $certificateFilePath)){
        $newCer = New-SelfSignedCertificate -CertStoreLocation Cert:CurrentUserMy -DnsName $dnsName
        $newCer    | Export-PfxCertificate -FilePath $certificateFilePath -Password  $certificatePassword 

        
        $newCer |Export-Certificate -FilePath $cerCertificateFilePath -Type CERT
        ######## Set up the Certs
        #If this is a self signed cert, then add it to the Trusted People Store.Else skip.
        $importedCer = Import-PfxCertificate -Exportable -CertStoreLocation Cert:CurrentUserTrustedPeople -FilePath $certificateFilePath -Password $certificatePassword

        #####import the cert into your local store. this is so that you can use the cert to view the secure cluster 
        $importedCer = Import-PfxCertificate -Exportable -CertStoreLocation Cert:CurrentUserMy -FilePath $certificateFilePath -Password $certificatePassword

    }    
    $clusterCertificates = new-object System.Security.Cryptography.X509Certificates.X509Certificate2 $certificateFilePath, $certificatePassword
    $clusterCertificates
} 

Step 2 : Create an Azure KeyVault

As explained in the documentation to secure Service Fabric Clusters, you will need to create an Azure KeyVault and upload the Certificate to the KV. After that, when you create the Service Fabric cluster you will have to reference both the KeyVault and the Certificate. You can specify these settings manually from the Azure Portal, but this script will help you to fully automate this process. The following function will create a KeyVault.

function GetOrCreateKeyVault
{
    
        if (-not (Get-AzureRmResourceGroup | ? ResourceGroupName -eq $resourceGroupName))
        {
            $newResourceGroup = New-AzureRmResourceGroup  -Name $ResourceGroupName -Location $Location -Verbose 
        }

        

    
        if( -not (Get-AzureRmKeyVault -ResourceGroupName $ResourceGroupName | ? VaultName -eq $VaultName ))
        {
            Write-Host "Creating vault $VaultName in $location (resource group $ResourceGroupName)"    
            $keyVault = New-AzureRmKeyVault -VaultName $VaultName -ResourceGroupName $ResourceGroupName -Location $Location `
                                            -EnabledForDeployment -Verbose  -Sku premium   
            
            
        }
        else 
        {
            $keyVault = Get-AzureRmKeyVault -VaultName $VaultName -ResourceGroupName $ResourceGroupName  
        }        
    
    $keyVault

} 

Step 3 : Upload the Certificate to Azure KeyVault

After creating the KeyVault, you will need to upload the Certificate KeyVault. ServiceFabric will use this certificate to secure the cluster.
The following function shows how to upload the Certificate to KeyVault.

After creating the KeyVault, you will need to upload the Certificate KeyVault. ServiceFabric will use this certificate to secure the cluster.
The following function shows how to upload the Certificate to KeyVault.

function AddCertificateToKeyVault
{
  Param(
      [string] $SecretName,  
      [string] $PfxFilePath,
      [System.Security.SecureString] $Password,
      [string] $ResourceGroupName,  
      [string] $Location,  
      [string] $VaultName
     )
     $ErrorActionPreference = 'Stop'   
       
    
    $keyVaultSecret = Get-AzureKeyVaultSecret -Name $SecretName  -VaultName $VaultName -ErrorAction Ignore
    if(($keyVaultSecret ) -eq $null)
      {
                 

            $bytes = [System.IO.File]::ReadAllBytes($PfxFilePath)
            $base64 = [System.Convert]::ToBase64String($bytes)
            

            $jsonBlob = @{
                data = $base64
                dataType = 'pfx'
                password = $clearPassword
            } | ConvertTo-Json

            $contentbytes = [System.Text.Encoding]::UTF8.GetBytes($jsonBlob)
            $content = [System.Convert]::ToBase64String($contentbytes)

            $secretValue = ConvertTo-SecureString -String $content -AsPlainText -Force
     
            Write-Host "Writing secret $SecretName to vault $VaultName"
            $keyVaultSecret = Set-AzureKeyVaultSecret -VaultName $VaultName -Name $SecretName -SecretValue $secretValue -Verbose         
     }
    $keyVaultSecret

 }

Step 4 : Create a VNET, Load balancer, Public IP and all the Infrastructure objects

We will use an ARM template to define all the objects that a Service Fabric needs. By doing this we will have a very repeatable process that will allow DevOps teams to very quickly create new clusters.
The azuredeploy.json template defines all the Azure objects that we need in order to create a Service Fabric cluster.
Also in the azuredeploy.parameters.json you can specify your own parameters like the size of the VMs and the number of Nodes you want to add to the cluster.

The other parameters will be dynamically generated by the PS Script in the following function :

function SetClusterTemplateParameters()
{
    
    # Read the Json Parameters file and Convert to HashTable
    $parameters = New-Object -TypeName hashtable 
    $jsonContent = Get-Content $parametersFileLocation  -Raw | ConvertFrom-Json 
    $jsonContent.parameters.psobject.Properties.Name `
            |ForEach-Object {$parameters.Add($_ ,$jsonContent.parameters.$_.Value)}
    
    # Complete Parameters Values 
    $parameters["adminPassword"] = $clearPassword
    $parameters["certificateThumbprint"] = $clusterCertificate.Thumbprint   
    $parameters["sourceVaultValue"] = $keyVault.ResourceId
    $parameters["certificateUrlValue"] = $keyVaultSecret.Id
    $parameters["dnsName"] = $dnsName
    $parameters["vmStorageAccountName"] = $dnsName+"stg"
    $parameters["clusterName"] = $dnsName+"FabcClu"


    
    $parameters
} 

The script will also prompt the user to specify the password for the admin user, in a fully automated build process you would typically add the password in the parameters section, but this script assumes that the cluster administrator will execute the script and specify the password where prompted.

if($certificatePassword -eq $null) {
    $certificatePassword = Read-Host -Prompt "Enter password" -AsSecureString 
    $clearPassword = (New-Object System.Management.Automation.PSCredential 'N/A', $certificatePassword).GetNetworkCredential().Password    
} 

Step 5 : Create the VMS and add the ServiceFabric extension to all the VMs using an ARM Template

The azuredeploy.json templates defines the Microsoft.ServiceFabric/clusters and it uses the CopyIndex function to create multiple VMs. Also note, that we are adding the ServiceFabric and IaaSDiagnpstics extensions to the VMS.
I find this model based on extensions extremely powerful and we will love you hear your feedback about this model.

{
      "apiVersion": "2015-05-01-preview",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[concat(parameters('vmName'),copyIndex())]",
      "location": "[parameters('clusterLocation')]",
      "dependsOn": [
        "[concat('Microsoft.Compute/availabilitySets/', parameters('availSetName'))]",
        "[concat('Microsoft.Storage/storageAccounts/', parameters('vmStorageAccountName'))]",
        "[concat('Microsoft.Network/networkInterfaces/', concat(parameters('nicName'),'-',copyIndex()))]"
      ],
      "properties": {
        "availabilitySet": {
          "id": "[resourceId('Microsoft.Compute/availabilitySets', parameters('availSetName'))]"
        },
        "hardwareProfile": {
          "vmSize": "[parameters('vmSize')]"
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces',concat(parameters('nicName'),'-',copyIndex()))]"
            }
          ]
        },
        "osProfile": {
          "adminPassword": "[parameters('adminPassword')]",
          "adminUsername": "[parameters('adminUsername')]",
          "computername": "[concat(parameters('vmName'),copyIndex())]",
          "secrets": [
            {
              "sourceVault": {
                "id": "[parameters('sourceVaultValue')]"
              },
              "vaultCertificates": [
                {
                  "certificateStore": "[parameters('certificateStoreValue')]",
                  "certificateUrl": "[parameters('certificateUrlValue')]"
                }
              ]
            }
          ],
          "windowsConfiguration": {
            "enableAutomaticUpdates": false,
            "provisionVMAgent": true
          }
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "[parameters('vmImagePublisher')]",
            "offer": "[parameters('vmImageOffer')]",
            "sku": "[parameters('vmImageSku')]",
            "version": "[parameters('vmImageVersion')]"
          },
          "osDisk": {
            "name": "osdisk",
            "vhd": {
              "uri": "[concat('http://',parameters('vmStorageAccountName'),'.blob.core.windows.net/',parameters('vmStorageAccountContainerName'),'/',resourcegroup().name,'-',parameters('vmName'),copyIndex(),'.vhd')]"
            },
            "caching": "ReadWrite",
            "createOption": "FromImage"
          }
        }
      },
      "copy": {
        "name": "vmLoop",
        "count": "[parameters('numberOfNodes')]"
      }
    },
    {
      "apiVersion": "2015-05-01-preview",
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "[concat(parameters('vmName'),copyIndex(0),'/ServiceFabricNode')]",
      "location": "[parameters('clusterLocation')]",
      "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'), copyIndex(0))]"
      ],
      "properties": {
        "type": "ServiceFabricNode",
        "publisher": "Microsoft.Azure.ServiceFabric",
        "protectedSettings": {
          "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('diagnosticStorageAccountName')),'2015-05-01-preview').key1]",
          "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('diagnosticStorageAccountName')),'2015-05-01-preview').key2]"
        },
        "settings": {
          "clusterEndpoint": "[reference(parameters('clusterName')).clusterEndpoint]",
          "nodeTypeRef": "NodeType1",
          "dataPath": "D:SvcFab",
          "certificate": {
            "thumbprint": "[parameters('certificateThumbprint')]",
            "x509StoreName": "[parameters('certificateStoreValue')]"
          }
        },
        "typeHandlerVersion": "1.0"
      },
      "copy": {
        "name": "vmServiceFabExtensionLoop",
        "count": "[parameters('numberOfNodes')]"
      }
    },
    {
      "apiVersion": "2015-05-01-preview",
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "[concat(parameters('vmName'),copyIndex(),'/Microsoft.Insights.VMDiagnosticsSettings')]",
      "location": "[parameters('clusterLocation')]",
      "dependsOn": [
        "[concat('Microsoft.Storage/storageAccounts/', variables('diagnosticStorageAccountName'))]",
        "[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'), copyIndex(0))]",
        "[concat('Microsoft.Compute/virtualMachines/',parameters('vmName'),copyIndex(0),'/extensions/ServiceFabricNode')]"        
      ],
      "properties": {
        "type": "IaaSDiagnostics",
        "autoUpgradeMinorVersion": true,
        "protectedSettings": {
          "storageAccountName": "[variables('diagnosticStorageAccountName')]",
          "storageAccountKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('diagnosticStorageAccountName')),'2015-05-01-preview').key1]",
          "storageAccountEndPoint": "https://core.windows.net/"
        },
        "publisher": "Microsoft.Azure.Diagnostics",
        "settings": {
          "WadCfg": {
            "DiagnosticMonitorConfiguration": {
              "overallQuotaInMB": "50000",
              "EtwProviders": {
                "EtwEventSourceProviderConfiguration": [
                  {
                    "provider": "Microsoft-ServiceFabric-Actors",
                    "scheduledTransferKeywordFilter": "1",
                    "scheduledTransferPeriod": "PT5M",
                    "DefaultEvents": {
                      "eventDestination": "ServiceFabricReliableActorEventTable"
                    }
                  },
                  {
                    "provider": "Microsoft-ServiceFabric-Services",
                    "scheduledTransferPeriod": "PT5M",
                    "DefaultEvents": {
                      "eventDestination": "ServiceFabricReliableServiceEventTable"
                    }
                  }
                ],
                "EtwManifestProviderConfiguration": [
                  {
                    "provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
                    "scheduledTransferLogLevelFilter": "Information",
                    "scheduledTransferKeywordFilter": "4611686018427387904",
                    "scheduledTransferPeriod": "PT5M",
                    "DefaultEvents": {
                      "eventDestination": "ServiceFabricSystemEventTable"
                    }
                  }
                ]
              }
            }
          },
          "StorageAccount": "[variables('diagnosticStorageAccountName')]"
        },
        "typeHandlerVersion": "1.5"
      },
      "copy": {
        "name": "vmExtensionLoop",
        "count": "[parameters('numberOfNodes')]"
      },
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[parameters('clusterName')]"
      }

Enjoy the Results!

After executing the template, you will see a new Resource Group that will contain all the objects :

Also, note that AzureKeyVault is not exposed yet on the Resource Group information, but you can use the Resouce Explorer to check the KeyVault information

Step 6 – Connect to the Cluster

After you script executes the JSON template, it waits two minutes and then it will try to connect to the Cluster. In some occasions more than two minutes will be needed, so you might need to retry :

After you script executes the JSON template, it waits two minutes and then it will try to connect to the Cluster. In some occasions more than two minutes will be needed, so you might need to retry :

After you script executes the JSON template, it waits two minutes and then it will try to connect to the Cluster. In some occasions more than two minutes will be needed, so you might need to retry :