Persistence v2

docs/eShop-Db-Design.md

@@ -0,0 +1,25 @@
+# Nethereum eShop Db Design
+
+The eShop takes a hybrid approach to decentralisation.  It attempts to balance Blockchain benefits with some requirements for off-chain processing and storage.  For instance, data which is subject to privacy rules may not belong on chain (e.g. personal postal addresses) whilst some documents (e.g. proof of purchase) may suit on chain storage perfectly.
+
+## Data Layer
+
+The core business services in the Nethereum.eShop are agnostic of the actual data persistence provider.  A mixture of the repository and CQRS patterns are present.  The repositories are typically involved in business unit transactions whilst queries are for highly optimised, performance intensive requirements which may require a provider specific implementation or additional resources such as data warehouses, external API's, search services etc.   As a basic example, Entity Framework Core can provide a repository layer implementation whilst Dapper can help provide optimised queries which in some organisations remain under DBA control.  
+
+One of the primary reasons for the design was that some entities would be likely to move from one storage solution to another in the near future.   Some entities which can't be stored on chain at present may be able to go on chain in the future once specific privacy concerns have been addressed in core Blockchain technology.  Abstracting the storage implementation via a repository or query interface will make it easier to move entity storage in the future.
+
+## Currently Supported Db's
+
+* Sql Server
+* Sqlite
+* MySql
+* InMemory
+
+All of these are currently supported with a mixture of Entity Framework core for Repository implementations and Dapper backed Query implementations.
+
+Non Entity Framework Core providers could be written as neither the repository or query interfaces are dependent on EF and Dapper.
+
+Entity Framework Core was chosen because: 
+* it's already widely used by .net developers
+* minimal code is required to implement other EF core based providers
+* it provides a migration solution (whatever you may think of it!)

docs/eShop-Db-Setup.md

@@ -2,7 +2,7 @@
 
 ## Configuration
 
-Connection strings are loaded from configuration in the following order.  Each source will potentially overwrite settings in previous sources.
+Connection strings and settings are loaded from configuration in the following order.  Each source will potentially overwrite settings in previous sources.
 
 * appsettings.json
 * appsettings.{environment}.json
@@ -12,61 +12,98 @@ Connection strings are loaded from configuration in the following order.  Each s
 
 For access keys and connection strings, user secrets is preferred for the development environment.  It ensures they are excluded from source control and are able to differ from one developer to another.
 
+## Supported DB's
+
+* SqlServer (tested against 15.0.2000.5 (2019), SQL Server Developer Edition running locally)
+* Sqlite
+* InMemory
+
 ## Initial Db Setup
 
-### Prerequisites
-* Access to a SQL Server (tested against 15.0.2000.5 (2019), SQL Server Developer Edition running locally)
-* A Connection string which allows a Database to be created on server
-    * In a windows environment the current windows user would typically be setup as as sysadmin on the SQL Server and the connection strings would use Integrated Security to map the windows user to SQL Server.  This avoids setting a password in the connection string.  The initial setup involves the migration creating a database so the connection must be for a user with create db access.
-* Install the most recent dotnet-ef tool (necessary to create and run migrations)
-```
-dotnet tool install --global dotnet-ef --version 3.1.1
-```
+### Options
+
+* Automatic migration at Startup.
+    * During startup of the main "Web" app migrations will be applied to create and update the database.
+    * This can be enabled with the following settings in appsettings.json
+        * "CatalogApplyMigrationsOnStartup": true,
+        * "IdentityApplyMigrationsOnStartup": true,
+    * *Warning* For SQL Server this will may require a user with necessary permissions to create a DB.
+* Manually applying SQL Scripts
+    * You need to create the DB First (e.g. ``` create database EShop ```) then apply the appropriate scripts below to create the schema.
+        * src\Nethereum.eShop.SqlServer\Catalog\Migrations\Scripts
+        * src\Nethereum.eShop.SqlServer\Identity\Migrations\Scripts
+        * src\Nethereum.eShop.Sqlite\Catalog\Migrations\Scripts
+        * src\Nethereum.eShop.Sqlite\Identity\Migrations\Scripts
 
 ### DB Connections
-* CatalogConnection - the main DB containing products, quotes, orders etc
+
+Database providers are dictated by the ``` CatalogDbProvider ``` and `` AppIdentityDbProvider` ``` settings.  
+
+Supported values are:
+* InMemory
+* SqlServer
+* Sqlite
+* MySql
+
+``` json
+  "ConnectionStrings": {
+    "CatalogConnection_SqlServer": "Server=localhost;Integrated Security=true;Initial Catalog=eShop;",
+    "IdentityConnection_SqlServer": "Server=localhost;Integrated Security=true;Initial Catalog=eShop;",
+    "CatalogConnection_Sqlite": "Data Source=C:/temp/eshop_catalog.db",
+    "IdentityConnection_Sqlite": "Data Source=C:/temp/eshop_app_identity.db",
+    "CatalogConnection_MySql": "server=localhost;database=eShop;user=eShop;password=oDEcHyOmr1ujVIWLBtQp;Allow User Variables=True",
+    "IdentityConnection_MySql": "server=localhost;database=eShop;user=eShop;password=oDEcHyOmr1ujVIWLBtQp;Allow User Variables=True",    
+    "BlockchainProcessingProgressDb": "Server=localhost\\sqldev;Integrated Security=true;Initial Catalog=eShopWebJobs;"
+  },
+  "CatalogDbProvider": "SqlServer",
+  "AppIdentityDbProvider": "SqlServer",
+  "CatalogApplyMigrationsOnStartup": true,
+  "IdentityApplyMigrationsOnStartup": true
+```
+
+* CatalogConnection_{DbProvider} - the main DB containing products, quotes, orders etc
     * This connection is used by both the Web and WebJob projects
-* IdentityConnection - authentication DB
+* IdentityConnection_{DbProvider} - authentication DB (web login)
     * Only used by the Web project at present
 * BlockchainProcessingProgressDb - stores event log processing progress (last processed block number etc)
     * Only used by the WebJobs project
 
-### Steps
-
-* Set up user-secrets
-
-    * Main Web Project: 
-        * Right Click the "Web" project and select "Manage User-Secrets", add the settings below and amend as necessary.
-        * By default the Web project runs with an in-memory database.  To use a real database, the "use-in-memory-db" setting must be set to false.
-        ``` json
-        {
-        "use-in-memory-db": false,
-        "ConnectionStrings": {
-            "CatalogConnection": "Server=<YourServer>;Integrated Security=true;Initial Catalog=eShop;",
-            "IdentityConnection": "Server=<YourServer>;Integrated Security=true;Initial Catalog=eShop;"
-        }
-        }
-        ```
-    * Web Jobs Project: 
-        * Right Click the "Web" project and select "Manage User-Secrets", add the settings below and amend as necessary.
-        ``` json
-        {
-        "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=<AccountName>;AccountKey=<AccountKey>;EndpointSuffix=core.windows.net",
-        "ConnectionStrings": {
-            "CatalogConnection": "Server=<YourServer>;Integrated Security=true;Initial Catalog=eShop;",
-            "IdentityConnection": "Server=<YourServer>;Integrated Security=true;Initial Catalog=eShop;",
-            "BlockchainProcessingProgressDb": "Server=<YourServer>;Integrated Security=true;Initial Catalog=eShopWebJobs;"
-        }
-        }
-        ```
-* Ensure the Web Project builds! (essential for running the initial migration)
-* Ensure the SQL Server is running
-* Ensure the database you require HAS NOT already been created
-* Run a script to create the DB
-    * In a command prompt - navigate to the root of the "Web" Project
-    * Run ApplyDbMigrations.bat
-        * This will create the DB and the necessary tables for the Catalog and Identity
-        * The "BlockchainProcessingProgressDb" referenced by the WebJobs project is setup differently, it is setup at run-time and created if it does not exist
-    * Run the Web project to ensure it works as expected
+Connection strings names are suffixed with the provider name (e.g. CatalogConnection_SqlServer) to allow easy switching between providers.  The developer can multiple  connection strings in settings and swap over using only the "CatalogDbProvider" and "AppIdentityDbProvider" settings.
+
+### User Secrets
+
+The "Web" and "WebJobs" project share the same User Secrets Id.  This is because they are expected to share the same persistence stores.  To set or change user secrets, right click the project and select "manage user-secrets".
+
+### Adding Migrations and DB Creation Scripts
+
+When schema changes have been made, migrations should be created so that DB's can be created or updated to the most recent schema.
+
+The solution is setup so that migrations for multiple DB providers can be managed from one place which ensures they are in sync.  Note: creating or adding a migration does not update the database, it simply creates code or scripts which can be run later.
 
+Migrations are created using the "dotnet-ef" tool with a specific start up project ('Nethereum.eShop.Migrations').  The migration process requires a start up project and it is not practical to use the main "Web" or "WebApp" projects as these are only configured for one Db provider at any one time.  The goal here is to create migrations for each DB provider at once.
+
+* Installing dotnet-ef tool (you may need to update to .Net Core 3.1.2 first)
+```
+dotnet tool install --global dotnet-ef --version 3.1.2
+```
+
+#### Adding a migration
+
+Batch files which create a named migration for Catalog and Identity for all supported Db Providers.
+
+```
+AddCatalogMigration.bat <NameOfMigration>
+AddIdentityMigration.bat <NameOfMigration>
+```
+
+Creates a new migration in each DB provider project.  
+
+#### Creating Db Scripts
+
+Generates a complete DB creation script for each supported Db Provider for both the Catalog connection and Identity connection.  This script can be run manually against the chosen database. 
+
+```
+ScriptCatalogDb.bat
+ScriptIdentityDb.bat
+```
 

src/Nethereum.eShop.DbFactory/EShopDbBootstrapper.cs

@@ -0,0 +1,43 @@
+using Microsoft.Extensions.Configuration;
+using Nethereum.eShop.ApplicationCore.Interfaces;
+using Nethereum.eShop.InMemory.Catalog;
+using Nethereum.eShop.InMemory.Infrastructure.Identity;
+using Nethereum.eShop.MySql.Catalog;
+using Nethereum.eShop.MySql.Identity;
+using Nethereum.eShop.Sqlite.Catalog;
+using Nethereum.eShop.Sqlite.Identity;
+using Nethereum.eShop.SqlServer.Catalog;
+using Nethereum.eShop.SqlServer.Identity;
+
+namespace Nethereum.eShop.DbFactory
+{
+    public static class EShopDbBootstrapper
+    {
+        public static IEShopDbBootstrapper CreateInMemoryDbBootstrapper() => new InMemoryEShopDbBootrapper();
+        public static IEShopIdentityDbBootstrapper CreateInMemoryAppIdentityDbBootstrapper() => new InMemoryEShopAppIdentityDbBootrapper();
+
+        public static IEShopDbBootstrapper CreateDbBootstrapper(IConfiguration configuration)
+        {
+            var name = configuration["CatalogDbProvider"]?.ToLower();
+            return name switch
+            {
+                "sqlserver" => new SqlServerEShopDbBootstrapper(),
+                "sqlite" => new SqliteEShopDbBootstrapper(),
+                "mysql" => new MySqlEShopDbBootstrapper(),
+                _ => new InMemoryEShopDbBootrapper()
+            };
+        }
+
+        public static IEShopIdentityDbBootstrapper CreateAppIdentityDbBootstrapper(IConfiguration configuration)
+        {
+            var name = configuration["AppIdentityDbProvider"]?.ToLower();
+            return name switch
+            {
+                "sqlserver" => new SqlServerEShopAppIdentityDbBootstrapper(),
+                "sqlite" => new SqliteEShopAppIdentityDbBootstrapper(),
+                "mysql" => new MySqlEShopAppIdentityDbBootstrapper(),
+                _ => new InMemoryEShopAppIdentityDbBootrapper()
+            };
+        }
+    }
+}

src/Nethereum.eShop.DbFactory/Nethereum.eShop.DbFactory.csproj

@@ -0,0 +1,14 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>netstandard2.1</TargetFramework>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\Nethereum.eShop.InMemory\Nethereum.eShop.InMemory.csproj" />
+    <ProjectReference Include="..\Nethereum.eShop.MySql\Nethereum.eShop.MySql.csproj" />
+    <ProjectReference Include="..\Nethereum.eShop.Sqlite\Nethereum.eShop.Sqlite.csproj" />
+    <ProjectReference Include="..\Nethereum.eShop.SqlServer\Nethereum.eShop.SqlServer.csproj" />
+  </ItemGroup>
+
+</Project>

src/Nethereum.eShop.EntityFramework/Catalog/Cache/RuleTreeCache.cs

@@ -0,0 +1,23 @@
+using Nethereum.eShop.ApplicationCore.Entities.RulesEngine;
+using Nethereum.eShop.ApplicationCore.Interfaces;
+using Nethereum.eShop.Infrastructure.Cache;
+using System.Threading.Tasks;
+
+namespace Nethereum.eShop.EntityFramework.Catalog.Cache
+{
+    public class RuleTreeCache : GeneralCache<RuleTree>, IRuleTreeCache
+    {
+        public RuleTreeCache()
+        {}
+
+        public Task<RuleTree> GetByIdAsync(string id)
+        {
+            return Task.FromResult(new RuleTree(new RuleTreeSeed()));
+        }
+
+        public Task<RuleTree> GetLastRuleTreeCreatedAsync()
+        {
+            return Task.FromResult(new RuleTree(new RuleTreeSeed()));
+        }
+    }
+}

src/Nethereum.eShop.EntityFramework/Catalog/CatalogContext.cs

@@ -0,0 +1,115 @@
+using MediatR;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.EntityFrameworkCore.Storage;
+using Nethereum.eShop.ApplicationCore.Entities;
+using Nethereum.eShop.ApplicationCore.Entities.BasketAggregate;
+using Nethereum.eShop.ApplicationCore.Entities.BuyerAggregate;
+using Nethereum.eShop.ApplicationCore.Entities.ConfigurationAggregate;
+using Nethereum.eShop.ApplicationCore.Entities.OrderAggregate;
+using Nethereum.eShop.ApplicationCore.Entities.QuoteAggregate;
+using Nethereum.eShop.ApplicationCore.Interfaces;
+using System;
+using System.Data;
+using System.Threading;
+using System.Threading.Tasks;
+
+namespace Nethereum.eShop.EntityFramework.Catalog
+{
+    public class CatalogContext : DbContext, IUnitOfWork
+    {
+        private readonly IMediator _mediator;
+        private IDbContextTransaction _currentTransaction;
+
+        public CatalogContext(
+            DbContextOptions options, IMediator mediator) : base(options)
+        {
+            _mediator = mediator;
+        }
+
+        public DbSet<Buyer> Buyers { get; set; }
+        public DbSet<Basket> Baskets { get; set; }
+        public DbSet<BasketItem> BasketItems { get; set; }
+        public DbSet<CatalogItem> CatalogItems { get; set; }
+        public DbSet<CatalogBrand> CatalogBrands { get; set; }
+        public DbSet<CatalogType> CatalogTypes { get; set; }
+        public DbSet<StockItem> StockItems { get; set; }
+        public DbSet<Quote> Quotes { get; set; }
+        public DbSet<QuoteItem> QuoteItems { get; set; }
+        public DbSet<Order> Orders { get; set; }
+        public DbSet<OrderItem> OrderItems { get; set; }
+
+        public DbSet<Setting> Settings { get; set; }
+
+        protected override void OnModelCreating(ModelBuilder builder)
+        {
+            base.OnModelCreating(builder);
+        }
+
+        public async Task<bool> SaveEntitiesAsync(CancellationToken cancellationToken = default(CancellationToken))
+        {
+            // Dispatch Domain Events collection. 
+            // Choices:
+            // A) Right BEFORE committing data (EF SaveChanges) into the DB will make a single transaction including  
+            // side effects from the domain event handlers which are using the same DbContext with "InstancePerLifetimeScope" or "scoped" lifetime
+            // B) Right AFTER committing data (EF SaveChanges) into the DB will make multiple transactions. 
+            // You will need to handle eventual consistency and compensatory actions in case of failures in any of the Handlers. 
+            await _mediator.DispatchDomainEventsAsync(this).ConfigureAwait(false);
+
+            // After executing this line all the changes (from the Command Handler and Domain Event Handlers) 
+            // performed through the DbContext will be committed
+            var result = await base.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
+
+            return true;
+        }
+
+        public async Task<IDbContextTransaction> BeginTransactionAsync()
+        {
+            if (_currentTransaction != null) return null;
+
+            _currentTransaction = await Database.BeginTransactionAsync(IsolationLevel.ReadCommitted).ConfigureAwait(false);
+
+            return _currentTransaction;
+        }
+
+        public async Task CommitTransactionAsync(IDbContextTransaction transaction)
+        {
+            if (transaction == null) throw new ArgumentNullException(nameof(transaction));
+            if (transaction != _currentTransaction) throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not current");
+
+            try
+            {
+                await SaveChangesAsync().ConfigureAwait(false);
+                transaction.Commit();
+            }
+            catch
+            {
+                RollbackTransaction();
+                throw;
+            }
+            finally
+            {
+                if (_currentTransaction != null)
+                {
+                    _currentTransaction.Dispose();
+                    _currentTransaction = null;
+                }
+            }
+        }
+
+        public void RollbackTransaction()
+        {
+            try
+            {
+                _currentTransaction?.Rollback();
+            }
+            finally
+            {
+                if (_currentTransaction != null)
+                {
+                    _currentTransaction.Dispose();
+                    _currentTransaction = null;
+                }
+            }
+        }
+    }
+}