AzurePowerShell

Upload Multiple Files to Azure Storage with PowerShell

By 14 March 2020June 24th, 2020No Comments

A simple code example that takes a pipeline input of file names, appends an optional date stamp and uploads the file(s) to Azure Storage. It needs the new Azure Az Module. You can install that here.

Install-Module Az -AllowClobber -Scope CurrentUser
Install-Module Az -AllowClobber -Scope CurrentUser

Here is the PowerShell Script. Copy-ItemToAzStorage.ps1

param (
    [Parameter(Mandatory = $True, ValueFromPipeline = $True, Position = 1)]
    [string]$SourcePath,
    [Parameter(Mandatory = $True, Position = 2)]
    [string]$TempPath,
    [Parameter(Mandatory = $True, Position = 3)]
    [string] $AzureContainerPath,
    [Parameter(Mandatory = $False, Position = 4)]
    [switch]$AppendTimeStamp
)

Begin {

    $module = Get-module -name "Az.Storage" -Verbose

    If (($module.Name) -match ("Az.Storage")) {
        Write-Host "Module Az.Storage is loaded"
    }
    else {
		Import-Module Az.Storage
    }

    # create  temp folder
    if (-not ($TempPath | Test-Path)) {
        New-Item -ItemType Directory -Path $TempPath
    }
    else {
      Get-ChildItem -Path $TempPath -Filter *.* | Remove-Item -Force
    }

    #Set up variables.

    $storageAccountName = "YOUR STORAGE ACCOUNT NAME"
    $storageAccountKey = "YOUR STORAGE KEY"
    $ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
    $containerName = "incoming"
    $containerPath = $AzureContainerPath       
    
}

Process {

    try {

        ForEach ($sourcePathInput in $SourcePath) {

            if (-not ($sourcePathInput | Test-Path)) {
                throw  "Source File does not exist."
            }
            if (-not ($TempPath | Test-Path)) {
                throw  "Temp Path does not exist."
            }
        
            $destinationFile = Copy-Item -Path $sourcePathInput -Destination $TempPath -PassThru -Force
        
            $dirName = [io.path]::GetDirectoryName($destinationFile)
            $filename = [io.path]::GetFileNameWithoutExtension($destinationFile)
            $ext = [io.path]::GetExtension($destinationFile)
            if ($AppendTimeStamp){
                $newPath = "$($dirName)\$($filename)_$(get-date -f yyyy-MM-dd)$ext"
            }
            else {
                $newPath = "$($dirName)\$($filename)$ext"
            }
        
            $blobName = "$($containerPath)/$($filename)_$(get-date -f yyyy-MM-dd)$ext"

            Write-Host "Renaming $destinationFile to $newPath"
        
            Rename-Item -Path $destinationFile -NewName $newPath -Force
        
            ## upload to azure.
            Write-Host "Uploading $newPath to Azure DataLake"

            Set-AzStorageBlobContent -File $newPath -Container $containerName -Blob $blobName -Context $ctx -Force
        }

    }
    catch {
        
        $e = $_.Exception
        $msg = $e.Message
        while ($e.InnerException) {
            $e = $e.InnerException
            $msg += "`n" + $e.Message
        }
        
        $msg        
    }        
}

End {

    # Cleanup tasks
    Remove-Module Az.Storage -ErrorAction Ignore

}

You can execute it like this

Get-ChildItem -Path "C:\Raw Data\*.txt" | .\Copy-ItemToAzStorage.ps1 -TempPath "C:\Raw Data\Temp\" -AzureContainerPath "DestinationFolder" -AppendTimeStamp
Follow me