Monday, 8 January 2024

Docker repo

With our Azurite docker image running, it's time to configure our docker repo. Let's start by adding a docker profile to our 'launchSettings.json' file


"Docker": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": true,
"launchUrl": "swagger",
"applicationUrl": "http://localhost:5050",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
}
},

As I've mention in the past, your naming conventions are very important, they are what let future you figure out what the hell you did; in this case all you're going to have to remember is that you have to check your 'launchSettings.json' file and you'll be able to follow the bread crumb trail of what this profile is meant for.

before we continue we're going to have to add a nuget to our project to work with azureBlobStorage, go to your .csproj file and enter in the following command 

dotnet add package azure.storage.blobs

you should see the following in your terminal

We need to add one more nuget package aht that is the 'azure.Identity' package

Your .csproj should now have the"azure.storage.blobs" package reference added

<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="azure.storage.blobs" Version="12.19.1" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
</ItemGroup>

</Project>

these will let us leverage Azures prebuilt classes for interacting with our blob storage.

Before we dive into our DockerRepo class, let's open our app settings file and add our azurite development connection string

{
"flatFileLocation": "DefaultEndpointsProtocol=https;
AccountName=devstoreaccount1;
AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;
BlobEndpoint=https://127.0.0.1:10000/devstoreaccount1;"
}

I broke the string up onto multiple lines for ease of reading, however you'll have to concatinate it onto one line.

Now we can finally start coding Next open up your 'DockerRepo.cs' file, it should look like the following


using pav.mapi.example.models;

namespace pav.mapi.example.repos
{
public class DockerRepo : IRepo
{
public Task<IPerson[]> GetPeopleAsync()
{
throw new NotImplementedException();
}

public Task<IPerson> GetPersonAsync(string id)
{
throw new NotImplementedException();
}
}
}

We configured the class but never implemented the methods, Let's take the opportunity to do so now; we're going to create a constructor that takes in a connection string and a the logic for our two get functions.

using System.Text.Json;
using Azure.Storage.Blobs;
using pav.mapi.example.models;

namespace pav.mapi.example.repos
{
public class DockerRepo : IRepo
{
BlobContainerClient _client;
public DockerRepo(string storageUrl)
{
this._client = new BlobContainerClient(storageUrl, "flatfiles");
}

public async Task<IPerson[]> GetPeopleAsync()
{
var openingsBlob = _client.GetBlobClient("people.json");
var openingJSON = (await openingsBlob.DownloadContentAsync()).Value.Content.ToString();

if (openingJSON != null)
{
var opt = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true
};

return JsonSerializer.Deserialize<Person[]>(openingJSON, opt);
}

throw new Exception();
}

public async Task<IPerson> GetPersonAsync(string id)
{
var ppl = await this.GetPeopleAsync();
if(ppl != null)
return ppl.First(p=> p.Id == id);
throw new KeyNotFoundException();
}
}
}

Now if we take a look at our main

using pav.mapi.example.models;
using pav.mapi.example.repos;

namespace pav.mapi.example
{
public class Program
{
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
var flatFileLocation = builder.Configuration.GetValue<string>("flatFileLocation");

if (String.IsNullOrEmpty(flatFileLocation))
throw new Exception("Flat file location not specified");

if(builder.Environment.EnvironmentName != "Production")
switch (builder.Environment.EnvironmentName)
{
case "Local":
builder.Services.AddScoped<IRepo, LocalRepo>(x => new LocalRepo(flatFileLocation));
goto default;
case "Development":
builder.Services.AddScoped<IRepo, DockerRepo>(x => new DockerRepo(flatFileLocation));
goto default;
default:
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
break;
}

var app = builder.Build();

switch (app.Environment.EnvironmentName)
{
case "Local":
case "Development":
app.UseSwagger();
app.UseSwaggerUI();
break;
}

app.MapGet("/v1/dataSource", () => flatFileLocation);
app.MapGet("/v1/people", async (IRepo repo) => await GetPeopleAsync(repo));
app.MapGet("/v1/person/{personId}", async (IRepo repo, string personId) => await GetPersonAsync(repo, personId));

app.Run();
}

public static async Task<IPerson[]> GetPeopleAsync(IRepo repo)
{
return await repo.GetPeopleAsync();
}

public static async Task<IPerson> GetPersonAsync(IRepo repo, string personId)
{
var people = await GetPeopleAsync(repo);
return people.First(p => p.Id == personId);
}
}
}

we pass our connection string to our Docker repo service in the form our flatfilelocation variable which is populated based on the profile loaded.

Now if we run our application with the 'local' profile our static GetPersonAsync and GetPeopleAsync functions will receive the LocalRepo implementation of our IRepo interface and if we run our application with the 'docker' profile, it will use the DockerRepo implementation of our IRepo interface.

Tuesday, 2 January 2024

Azurite Powershell

In order to upload files to our azurite storage, we're going to leverage PowerShell... on a mac, so please read the MSDN documentation on that since, it may change.


Though it may seem as if we are adding unnecessary complexity to our project, with powershell we can automate the process of uploading data to our blob storage, which may not seem very important now, however in the future when you we'll be deploying to the cloud, it will make life much simpler.

In your application create a 'powershell' folder, if you haven't already, let's start with a simple powershell script to copy our people.json file from our local hard drive to our docker container.



if(0 -eq ((Get-Module -ListAvailable -Name az).count) ){
Write-host "installing AZ module." -foregroundcolor Yellow
Install-Module -Name Az -Repository PSGallery -Force
Write-host "AZ module installed." -foregroundcolor Yellow
}
else{
Write-host "AZ module already installed." -foregroundcolor Green
}

$cs = "DefaultEndpointsProtocol=https;"
$cs += "AccountName=devstoreaccount1;"
$cs += "AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
$cs += "BlobEndpoint=https://127.0.0.1:10000/devstoreaccount1;"
$cs += "QueueEndpoint=https://127.0.0.1:10001/devstoreaccount1;"
$cs += "TableEndpoint=https://127.0.0.1:10002/devstoreaccount1;"

$context = New-AzStorageContext -ConnectionString $cs

function UploadFiles {
Param(
[Parameter(Mandatory=$true)][String]$containerName,
[Parameter(Mandatory=$true)][String]$localPath)
# upload all flat files to local azurite
Get-ChildItem ` -Path $localPath ` -Recurse | `
Set-AzStorageBlobContent `
-Container $containerName `
-Context $context `
}

function createContainer { Param(
[Parameter(Mandatory=$true)][String]$containerName,
[Parameter(Mandatory=$true)][Int32]$permission)

$container = $context | Get-AzStorageContainer -Name $containerName -ErrorAction SilentlyContinue
if($null -ne $container){
Remove-AzStorageContainer -Name $containerName -Context $context
}

$container = $context | Get-AzStorageContainer -Name $containerName -ErrorAction SilentlyContinue
if($null -eq $container){
Write-Host "Createing local $containerName contaier" -ForegroundColor Magenta
$context | New-AzStorageContainer -Name $containerName -Permission $permission
}
else {
Write-Host "local $containerName container already exists" -ForegroundColor Yellow
}
}


createContainer -containerName "flatfiles" -permission 0;

UploadFiles -containerName "flatfiles" -localPath "/Volumes/dev/data/"

$sasToken = New-AzStorageContainerSASToken -name "flatfiles" -Permission "rwdalucp" -Context $context

Write-Host "your SAS token is:$sasToken it has been copied to your clipboard" -ForegroundColor Green
Write-Host "The full connection URL has been copied to your clipboard" -ForegroundColor Green
Write-Output "https://127.0.0.1:10000/devstoreaccount1/flatfiles/people.json?" $sasToken | Set-Clipboard

You should have the URL to the people.json file copied to your clipboard with a SAS token to let you access it, simply past the URL in your browser or postman or insomnia and you should be able to get your json file.

Next let's create a script that will list all of the blobs in our Azurite blob storage


if(0 -eq ((Get-Module -ListAvailable -Name az).count) ){
Write-host "installing AZ module." -foregroundcolor Yellow
Install-Module -Name Az -Repository PSGallery -Force
Write-host "AZ module installed." -foregroundcolor Yellow
}
else{
Write-host "AZ module already installed." -foregroundcolor Green
}

$cs = "DefaultEndpointsProtocol=https;"
$cs += "AccountName=devstoreaccount1;"
$cs += "AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
$cs += "BlobEndpoint=https://127.0.0.1:10000/devstoreaccount1;"
$cs += "QueueEndpoint=https://127.0.0.1:10001/devstoreaccount1;"
$cs += "TableEndpoint=https://127.0.0.1:10002/devstoreaccount1;"

$context = New-AzStorageContext -ConnectionString $cs
$containerName = "flatfiles"

Get-AzStorageBlob -Container $containerName -Context $context | Select-Object -Property Name

And finally a powershell script to delete files.


Param (
[Parameter()][String]$blobToDelete,
[Parameter()][Boolean]$deleteAllFiles)

if(0 -eq ((Get-Module -ListAvailable -Name az).count) ){
Write-host "installing AZ module." -foregroundcolor Yellow
Install-Module -Name Az -Repository PSGallery -Force
Write-host "AZ module installed." -foregroundcolor Yellow
}
else{
Write-host "AZ module already installed." -foregroundcolor Green
}

$cs = "DefaultEndpointsProtocol=https;"
$cs += "AccountName=devstoreaccount1;"
$cs += "AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
$cs += "BlobEndpoint=https://127.0.0.1:10000/devstoreaccount1;"
$cs += "QueueEndpoint=https://127.0.0.1:10001/devstoreaccount1;"
$cs += "TableEndpoint=https://127.0.0.1:10002/devstoreaccount1;"

$context = New-AzStorageContext -ConnectionString $cs
$containerName = "flatfiles"

if($true -ne ([string]::IsNullOrWhiteSpace($blobToDelete))){
Remove-AzStorageBlob -Blob $blobToDelete -Container $containerName -Context $context
}

if($true -eq $deleteAllFiles) {
Get-AzStorageBlob -Container $containerName -Context $context | Remove-AzStorageBlob
}


the above unlike the other two scripts takes in parameters letting us specify if we want to delete a specific file or all files. it can be called several ways, but the following two are probably the easiest

./deleteBlobs.ps1 people.json         
./deleteBlobs.ps1 -deleteAllFiles $true     

and that's it for now, keep in mind that theses scripts are specific to our azurite environment, when it comes time to deploy to production we'll cross that bridge when we get to it.

Saturday, 30 December 2023

Azurite - docker


In our previous post we configured a local repository which can read flat files from our hard drive, next we're going to set up a docker image with azurite.

The Azurite open-source emulator provides a free local environment for testing your Azure Blob, Queue Storage, and Table Storage applications. When you're satisfied with how your application is working locally, switch to using an Azure Storage account in the cloud. The emulator provides cross-platform support on Windows, Linux, and macOS.


Though one can connect to Azurite using standard http, We're going to opt for https, it adds a bit of complexity, however it's worth the investment, since later on it will be simpler to create a production version of an AzureRepo.

We're going to have to install the mkcert
mkcert is a simple tool for making locally-trusted development certificates. It requires no configuration.
GitHub - FiloSottile/mkcert: A simple zero-config tool to make locally trusted development certificates with any names you'd like.

I always suggest you go to the official documentation, because it may very well change and to be honest generally whenever i read my own blog posts, I usually go to back to the source material. That said at the time of this article you can install mkcert on a mac using homebrew 

brew install mkcert
brew install nss # if you use Firefox

next with mkcert installed you'll have to install a local CA 

mkcert -install


You'll have to provide your password to install the mkcert utility, but once it's setup you'll be able to easily make as many locally trusted development certificates as you please.

Now that you have the mkcert utility installed create a folder in your project called _azuriteData,  realistically this folder can have any name you like, however it's nice when folders and variables describe what they are used for; makes future you not want to kill present you. Once you have your folder created navigate into it and enter in the following command
 
mkcert 127.0.0.1


This will create two files inside your _azuriteData folder or whatever directory you're currently in, a key.pem file and a .pem file; I'm not going to get into the deep details of what these two files are, partially because that's not the aim of this post and partially because I have a passing awareness when it comes to security, high level, and with little practical experience, you know like most consultants.

However I can tell you that PEM stand for Privacy Enhanced Mail, but the format is widely used for various cryptographic purposes beyond email security. You can think of the two files as the public key (.pem) and the corresponding private key (key.pem) for symmetric encryption.

In essence what happens is that the message is encrypted with the public key (127.0.0.1.pem), and then decrypted with the private one (127.0.0.1-key.pem); however you really don't need to worry too much about the details.

Your folder structure should like something like this:


next we are going to create a docker compose file, for this to work you need to have docker installed on your system, you can download the install file from https://www.docker.com/.

At the root of your project create a docker-compose-yaml file, this file will contain all the instructions you need to install and run an azurite storage emulator locally for testing purposes.


version: '3.9'
services:
azurite:
image: mcr.microsoft.com/azure-storage/azurite
container_name: "azurite-data"
hostname: azurite
restart: always
command: "azurite \
--oauth basic \
--cert /workspace/127.0.0.1.pem \
--key /workspace/127.0.0.1-key.pem \
--location /workspace \
--debug /workspace/debug.log \
--blobHost 0.0.0.0 \
--queueHost 0.0.0.0 \
--tableHost 0.0.0.0 \
--loose"
ports:
- "10000:10000"
- "10001:10001"
- "10002:10002"
volumes:
- ./_azuriteData:/workspace


One of the trickiest things about this compose file is that whatever local folder contains our public/private pem keys must be mapped to the 'workspace' of our docker container. This command basically tells docker to use the azurite-data folder as the volume for this container. 

now if we go to our terminal and we enter the command 

docker-compose up

The first thing that docker will do is download the azurite image, once the download is complete, the image should be running in our docker container, that is of course if you have docker installed and running on your local machine.


You should see something like the above. Now if you navigate to your docker dashboard you should see your environment up and running

some caveats you may get some sort of 

open /Users/[user]/.docker/buildx/current: permission denied

in that case run the following command, or commands

sudo chown -R $(whoami) ~/.docker
sudo chown -R $(whoami) ./_azurite-data

The above sets the current user to the owner of the docker application folder as well as the folder which you will map to your azurite container running in docker, this should fix your docker permission issues.

Thursday, 28 December 2023

Dependency injection

I've spoken about the inversion of control pattern before, in essence it is a fundamental design pattern in object oriented languages. It separates the definition of capability from the logic that executes that capability. That may sound a bit strange but what it means is that the class which you interact with defines functions and methods, however the logic which those functions and methods use are defined in a separate class. At run time the class you interact with, either receives or initiates a class which implements the methods and functions needed.

The idea is that you can have multiple implementations for the same methods and then swap out those implementations without modifying your main codebase. There are a number of advantages to writing code this way, one of the primary ones is that this patter facilitates unit testing, however in this post we'll focus on the benefits of having multiple implementations of the same interface.


in the above UML diagram, we define an interface for a repository as well as a person class, we then define two repo classes which would both implement the IRepo interface, finally the Program class would either receive a concrete implementation of the IRepo class or it would receive instructions on which concrete implementation to use. (I've minimised the number of arrows to try and keep the diagram somewhat readable)

Let's try and implement the above in our Minimal API, start by creating the following folder structure with classes and interfaces


It's common to have a separate folder for all interfaces, personally I prefer to have my interfaces closer to their concrete classes, to be honest it really doesn't matter how you do it, what matters is that you are consistent, personally this is the way I like to set up my projects, but there's nothing wrong with doing it differently.

So let's start with the IPerson interface


namespace pav.mapi.example.models
{
public interface IPerson
{
string Id { get; }
string FirstName { get; set; }
string LastName { get; set; }
DateOnly? BirthDate { get; set; }
}
}

as you can see from the namespace, I don't actually separate it from the Person class, the folder is just there as an easy way to group our model interfaces. Next let's implement our Person class


using pav.mapi.example.models;

namespace pav.mapi.example.models
{
public class Person : IPerson
{
public string Id { get; set; } = "";
public string FirstName { get; set; } = "";
public string LastName { get; set; } = "";
public DateOnly? BirthDate { get; set; } = null;

public Person(){}

public Person(string FirstName, string LastName, DateOnly BirthDate) : this()
{
this.Id = Guid.NewGuid().ToString();
this.FirstName = FirstName;
this.LastName = LastName;
this.BirthDate = BirthDate;
}

public int getAge()
{
var today = DateOnly.FromDateTime(DateTime.Now);
var age = today.Year - this.BirthDate.Value.Year;
return this.BirthDate > today.AddYears(-age) ? --age : age;
}

public string getFullName()
{
return $"{this.FirstName} {this.LastName}";
}
}
}

An important thing to call out is, that this is a contrived example, setting the ID property with both a public setter as well as a public getter is not ideal, however I want to focus on the Inversion of control pattern and not get bogged down in the ideal ways of defining properties; I may tackle this in a future post.

Now on to defining our repo interface, simply define two functions, one to get people and one to get a specific person.


using pav.mapi.example.models;

namespace pav.mapi.example.repos
{
public interface IRepo
{
public Task<IPerson> GetPersonAsync(string id);
public Task<IPerson[]> GetPeopleAsync();
}
}

with that done, let's implement our Local repo, it's going to be very simple we are going to use use our path from our local appsettings file to load a JSON file full of people.


using System.Text.Json;
using pav.mapi.example.models;

namespace pav.mapi.example.repos
{
public class LocalRepo : IRepo
{
private string _filePath;
public LocalRepo(string FilePath)
{
this._filePath = FilePath;
}

public async Task<IPerson[]> GetPeopleAsync()
{
var opt = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true
};

try
{
using (FileStream stream = File.OpenRead(this._filePath))
{
var ppl = await JsonSerializer.DeserializeAsync<Person[]>(stream, opt);
if (ppl != null)
return ppl;

throw new Exception();
}

}
catch (Exception)
{
throw;
}
}

public async Task<IPerson> GetPersonAsync(string id)
{
var people = await this.GetPeopleAsync();
var person = people?.First(p => p.Id == id);
if (person != null)
return person;

throw new KeyNotFoundException($"no person with id {id} exitsts");
}
}
}

Next let's implement our DockerRepo


using pav.mapi.example.models;

namespace pav.mapi.example.repos
{
public class DockerRepo : IRepo
{
public Task<IPerson[]> GetPeopleAsync()
{
throw new NotImplementedException();
}

public Task<IPerson> GetPersonAsync(string id)
{
throw new NotImplementedException();
}
}
}

For now we'll just leave it unimplemented, later on when we set up a docker container, we'll implement our functions.

Finally let's specify which implementation to use based on our loaded profile in our Main


using pav.mapi.example.models;
using pav.mapi.example.repos;

namespace pav.mapi.example
{
public class Program
{
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
var flatFileLocation = builder.Configuration.GetValue<string>("flatFileLocation");

if (String.IsNullOrEmpty(flatFileLocation))
throw new Exception("Flat file location not specified");

if(builder.Environment.EnvironmentName != "Production")
switch (builder.Environment.EnvironmentName)
{
case "Local":
builder.Services.AddScoped<IRepo, LocalRepo>(x => new LocalRepo(flatFileLocation));
goto default;
case "Development":
builder.Services.AddScoped<IRepo, DockerRepo>(x => new DockerRepo());
goto default;
default:
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
break;
}

var app = builder.Build();

switch (app.Environment.EnvironmentName)
{
case "Local":
case "Development":
app.UseSwagger();
app.UseSwaggerUI();
break;
}

app.MapGet("/v1/dataSource", () => flatFileLocation);
app.MapGet("/v1/people", async (IRepo repo) => await GetPeopleAsync(repo));
app.MapGet("/v1/person/{personId}", async (IRepo repo, string personId) => await GetPersonAsync(repo, personId));

app.Run();
}

public static async Task<IPerson[]> GetPeopleAsync(IRepo repo)
{
return await repo.GetPeopleAsync();
}

public static async Task<IPerson> GetPersonAsync(IRepo repo, string personId)
{
var people = await GetPeopleAsync(repo);
return people.First(p => p.Id == personId);
}
}
}

Before we test our code, we first need to setup a flat file somewhere on our hard drive, 


[
{
"id": "1",
"firstName": "Robert",
"lastName": "Smith",
"birthDate": "1984-01-26"
},
{
"id": "2",
"firstName": "Eric",
"lastName": "Johnson",
"birthDate": "1988-08-28"
}
]

I created the above file in the directory "/volumes/dev/people.json",  it's important to know the path because in the next step we are going to add it to our appsettings.Local.json file

{
"flatFileLocation": "/Volumes/dev/people.json"
}

With all that complete let's go into our main terminal and run our api with our local profile

dotnet watch run -lp local --trust

in our Swagger we should see three get endpoints we can call


Each one of these should work

/v1/dataSource should return where our flatfile is located on our local computer

/Volumes/dev/people.json

/v1/people should return the contents of our people.json file

[
  {
    "id": "1",
    "firstName": "Robert",
    "lastName": "Smith",
    "birthDate": "1984-01-26"
  },
  {
    "id": "2",
    "firstName": "Eric",
    "lastName": "Johnson",
    "birthDate": "1988-08-28"
  }
]

/v1/person/{personId} should return the specified person.

 {
    "id": "2",
    "firstName": "Eric",
    "lastName": "Johnson",
    "birthDate": "1988-08-28"
  }

Next if we stop our api (cntrl+c) and restart it with the following command

dotnet watch run -lp http --trust

we again run our application however this time with the development profile, if we navigate to our swagger and call our functions the only one which will work is 

/v1/dataSource which should return the placeholder we specified in our appsettings.Development.json file.

Dev

the other two endpoints should both fail, since we never implemented them