In this article, we are going to see how we can download multiple files from the Azure Blob Storage using ASP.NET Core. We can say that this article is an appendix to our multiple articles that cover Azure Blob Storage, and we encourage you to read them as well.
Let’s start.
Interacting With Blobs in ASP.NET Core
ASP.NET Core has three clients to interact with different types of resources: BlobServiceClient, BlobContainerClient, and BlobClient. A client that has methods to work with blobs is the BlobClient. Unfortunately, a method to download multiple blobs or download an entire folder doesn’t exist. Instead, we are going to perform simultaneous downloads for each blob, and we are going to do it in parallel to be somewhat performant.
How to Setup Azure Blob Storage Locally?
Instead of working with the actual Azure storage, we are going to use Azurite, an emulator used for testing and development. We are going to create a small console application with Visual Studio 2022 since Azurite comes with it by default.
Note that you can easily switch to the actual Azure storage later by changing the connection string. You can see how to set up everything on Azure as part of this article.
Let’s create a console application from Visual Studio. Since Azurite starts automatically only for ASP.NET and Azure Functions projects, we are going to start Azurite from the command line:
C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator\azurite.exe
After we run the executable file, Azurite will start listening for connections:
Azurite Blob service is starting at http://127.0.0.1:10000 Azurite Blob service is successfully listening at http://127.0.0.1:10000 Azurite Queue service is starting at http://127.0.0.1:10001 Azurite Queue service is successfully listening at http://127.0.0.1:10001 Azurite Table service is starting at http://127.0.0.1:10002 Azurite Table service is successfully listening at http://127.0.0.1:10002
We are going to use Microsoft Azure Storage Explorer to connect to Azurite blob storage and to prepare some files. After the download and launch, we can find our Azurite storage under the local-1 account (since we already run it):
Then let’s right-click on the Blob Containers and choose Create Blob Container in the dialog to create one container called multiple-files.
Finally, we can upload 50 random files to that container.
Great, we are now ready for the coding.
Download Multiple Files From Azure Blob Storage
Let’s install the Azure Blob Storage client library for the .NET by running the command in the Package Manager Console:
Install-Package Azure.Storage.Blobs -Version 12.9.1
Why this specific version? At the moment of writing this article, there is a bug with the latest package version and the Azurite version (3.14.1) that comes with Visual Studio. It turns out that they are not backward compatible and that we should either update Azurite to version 3.16.0 or downgrade the SDK a bit, which we decided to do.
To continue, let’s connect to our local storage in the Program class:
var connectionString = "UseDevelopmentStorage=true"; var blobServiceClient = new BlobServiceClient(connectionString);
And let’s create MultipleDownloadExamples class:
public class MultipleDownloadExamples
{
private readonly BlobServiceClient _blobServiceClient;
public MultipleDownloadExamples(BlobServiceClient blobServiceClient)
{
_blobServiceClient = blobServiceClient;
}
}
In our examples, we are going to fetch all the files from our container. Although we have only 50 blobs, there can be thousands of them, so we will fetch them in pages.
That said, we are going to add a method in the MultipleDownloadExample class that downloads blobs in parallel:
public async Task DownloadMultipleFilesUsingParallelForEachAsync(string containerName)
{
var blobContainerClient = _blobServiceClient.GetBlobContainerClient(containerName);
var blobPages = blobContainerClient.GetBlobsAsync().AsPages();
await Parallel.ForEachAsync(blobPages, new ParallelOptions { MaxDegreeOfParallelism = 2 }, async (blobPage, token) =>
{
await Parallel.ForEachAsync(blobPage.Values, new ParallelOptions { MaxDegreeOfParallelism = 2 }, async (blob, token) =>
{
var blobClient = blobContainerClient.GetBlobClient(blob.Name);
using var file = File.Create(blob.Name);
await blobClient.DownloadToAsync(file);
});
});
}
We utilize the Parallel.ForEachAsync method that we got in the .NET 6 version. First, the Parallel.ForEachAsync method is going through pages of blobs in parallel, and then it is downloading blobs from each page.
Lastly, we are going to call that method in our Program class:
var multipleDownloadExamples = new MultipleDownloadExamples(blobServiceClient);
// Multiple download with parallel foreach async
await multipleDownloadExamples.DownloadMultipleFilesUsingParallelForEachAsync("multiple-files");
Using the SemaphoreSlim Class as an Alternative
Sometimes, we don’t want to use the Parallel.ForEach method. For example, they don’t work as expected when used in Azure Functions.
So, let’s add another method where we achieve parallelism in a different way:
public async Task DownloadMultipleFilesUsingSemaphoreSlim(string containerName)
{
var blobContainerClient = _blobServiceClient.GetBlobContainerClient(containerName);
var blobPages = blobContainerClient.GetBlobsAsync().AsPages();
var semaphore = new SemaphoreSlim(4);
var tasks = new List<Task>();
await foreach (var blobPage in blobPages)
{
foreach (var blob in blobPage.Values)
{
await semaphore.WaitAsync();
tasks.Add(Task.Run(async () =>
{
try
{
var blobClient = blobContainerClient.GetBlobClient(blob.Name);
using var file = File.Create(blob.Name);
await blobClient.DownloadToAsync(file);
}
finally
{
semaphore.Release();
}
}));
}
await Task.WhenAll(tasks);
tasks.Clear();
}
}
Although we don’t iterate through blob pages in parallel, we still manage to download blobs from each page in parallel, which is still great. We control the degree of parallelism by passing an initial value of the granted concurrent requests to the constructor of the SemaphoreSlim class.
Now, we can call this method as well in the Program class to download our files:
await multipleDownloadExamples.DownloadMultipleFilesUsingSemaphoreSlim("multiple-files");
Conclusion
In this article, we’ve learned how to download multiple files from Azure Blob storage. Also, we have seen how to set up Azurite, a new storage emulator used for testing and development.
