.NET configuration evolution

Every programmer imagined - well, or might want to imagine - himself as an airplane pilot, when you have a huge project, a huge panel of sensors, metrics and switches for it, with which you can easily configure everything as it should. Well, at least not running to manually lift the chassis yourself. Both metrics and graphs are all good, but today I want to tell you about those same tumblers and buttons that can change the parameters of the aircraft's behavior, configure it.



The importance of configurations is difficult to underestimate. Everyone uses one or another approach in configuring their applications, and, in principle, there is nothing complicated about it, but is it that simple? I propose to look at the "before" and "after" in the configuration and understand the details: how what works, what new features we have and how to use them to the fullest. Those who are not familiar with configuring in .NET Core will get the basics, and those who are familiar will get food to think about and use new approaches in their daily work.

Pre-.NET Core configuration

In 2002, the .NET Framework was introduced, and since it was the time of the XML hype, the developers from Microsoft decided “let's have it everywhere”, and as a result, we got XML configurations that are still alive. At the head of the table we have a static ConfigurationManager class through which we get string representations of parameter values. The configuration itself looked something like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="Title" value=".NET Configuration evo" />
    <add key="MaxPage" value="10" />
  </appSettings>
</configuration>

The problem was solved, the developers got a customization option, which is better than INI files, but with its own peculiarities. So, for example, support for different settings values ​​for different types of application environments is implemented using XSLT transformations of the configuration file. We can define our own XML schemas for elements and attributes if we want something more complex in terms of grouping data. Key-value pairs have a strictly string type, and if we need a number or a date, then "let's do it yourself somehow":

string title = ConfigurationManager.AppSettings["Title"];
int maxPage = int.Parse(ConfigurationManager.AppSettings["MaxPage"]);

In 2005 we added configuration sections , they allowed grouping parameters, building their own schemes, avoiding naming conflicts. We also presented * .settings files and a special designer for them.







Now you can get a generated, strongly typed class that represents configuration data. The designer allows you to conveniently edit the values, sorting by editor columns is available. The data is retrieved using the Default property of the generated class, which provides the Singleton configuration object.

DateTime date = Properties.Settings.Default.CustomDate;
int displayItems = Properties.Settings.Default.MaxDisplayItems;
string name = Properties.Settings.Default.ApplicationName;

We also added scopes of configuration parameter values. The User area is responsible for user data, which can be changed by him and saved during program execution. Saving takes place in a separate file along the path% AppData% \ * Application name *. The Application scope allows you to retrieve parameter values ​​without the possibility of user override.



Despite the good intentions, the whole thing became more complicated.

• In fact, these are the same XML files that began to grow in size faster and, as a result, became inconvenient to read.
• The configuration is read from the XML file once, and we need to reload the application to apply the changes to the configuration data.
• Classes generated from * .settings files were marked with the sealed modifier, so this class could not be inherited. In addition, this file could be changed, but if a regeneration occurs, we lose everything we wrote ourselves.
• Working with data only according to the key-value scheme. To get a structured approach to working with configurations, we need to additionally implement this ourselves.
• The data source can only be a file, external providers are not supported.
• Plus, we have a human factor - private parameters get into the version control system and become exposed.

All of these problems remain in the .NET Framework to this day.

.NET Core configuration

In .NET Core, they reimagined configuration and created everything from scratch, removed the static ConfigurationManager class and solved many of the problems that were "before". What have we got new? As before - the stage of forming the configuration data and the stage of consuming this data, but with a more flexible and extended life cycle.

Setting up and filling with configuration data

So, for the stage of data generation, we can use many sources, not limiting ourselves only to files. The configuration is done through the IConfgurationBuilder - the basis into which we can add data sources. NuGet packages are available for various types of sources:



Each source is added as a new layer and overrides the parameters with matching keys. Here is the Program.cs example that comes by default in the ASP.NET Core app template (version 3.1).

public static IHostBuilder CreateHostBuilder(string[] args) => 
    Host.CreateDefaultBuilder(args).ConfigureWebHostDefaults(webBuilder => 
        { webBuilder.UseStartup<Startup>(); });

I want to focus on CreateDefaultBuilder . Inside the method, we will see how the initial configuration of sources occurs.

public static IWebHostBuilder CreateDefaultBuilder(string[] args)
{
    var builder = new WebHostBuilder();

    ...

    builder.ConfigureAppConfiguration((hostingContext, config) =>
    {
        IHostingEnvironment env = hostingContext.HostingEnvironment;

        config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
              .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true);

        if (env.IsDevelopment())
        {
            Assembly appAssembly = Assembly.Load(new AssemblyName(env.ApplicationName));
            if (appAssembly != null)
            {
                config.AddUserSecrets(appAssembly, optional: true);
            }
        }

        config.AddEnvironmentVariables();

        if (args != null)
        {
            config.AddCommandLine(args);
        }
    })
            
    ...

    return builder;
}

So we get that the base for the entire configuration will be the appsettings.json file; further, if there is a file for a specific environment, then it will have a higher priority, and thereby override the matching values ​​of the base. And so with each subsequent source. The order of addition affects the final value. Visually, everything looks like this:







If you want to use your order, then you can simply clear it and define how you need it.

Host.CreateDefaultBuilder(args)
    .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); })
    .ConfigureAppConfiguration((context,
                                builder) =>
     {
         builder.Sources.Clear();
         
         //  
     });

Each configuration source has two parts:

• Implementation of IConfigurationSource. Provides a source of configuration values.
• Implementation of IConfigurationProvider. Converts the original data to the resulting key-value.

By implementing these components, we can get our own data source for configuration. Here is an example of how you can implement getting parameters from a database through the Entity Framework.

How to use and retrieve data

Now that everything is clear with the setting and filling with configuration data, I propose to take a look at how we can use this data and how to get it more conveniently. The new approach to configuring projects makes a big bias towards the popular JSON format, and this is not surprising, because with its help we can build any data structures, group data and have a readable file at the same time. Take the following configuration file for example:

{
  "Features" : {
    "Dashboard" : {
      "Title" : "Default dashboard",
      "EnableCurrencyRates" : true
    },
    "Monitoring" : {
      "EnableRPSLog" : false,
      "EnableStorageStatistic" : true,
      "StartTime": "09:00"
    }
  }
}

All data forms a flat key-value dictionary, the configuration key is formed from the entire file key hierarchy for each value. A similar structure would have the following data set:





We can get the value using the IConfiguration object . For example, here's how we can get the parameters:

string title = Configuration["Features:Dashboard:Title"];
string title1 = Configuration.GetValue<string>("Features:Dashboard:Title");
bool currencyRates = Configuration.GetValue<bool>("Features:Dashboard:EnableCurrencyRates");
bool enableRPSLog = Configuration.GetValue<bool>("Features:Monitoring:EnableRPSLog");
bool enableStorageStatistic = Configuration.GetValue<bool>("Features:Monitoring:EnableStorageStatistic");
TimeSpan startTime = Configuration.GetValue<TimeSpan>("Features:Monitoring:StartTime");

And this is already quite good, we have a good way to get data that is cast to the required data type, but somehow not as cool as we would like. If we receive data as given above, then we will end up with a duplicate code and make mistakes in the names of the keys. Instead of individual values, you can assemble a complete configuration object. Binding data to an object through the Bind method will help us with this. Example of class and data retrieval:

public class MonitoringConfig
{
    public bool EnableRPSLog { get; set; }
    public bool EnableStorageStatistic { get; set; }
    public TimeSpan StartTime { get; set; }
}

var monitorConfiguration = new MonitoringConfig();
Configuration.Bind("Features:Monitoring", monitorConfiguration);

var monitorConfiguration1 = new MonitoringConfig();
IConfigurationSection configurationSection = Configuration.GetSection("Features:Monitoring");
configurationSection.Bind(monitorConfiguration1);

In the first case, we bind by the section name, and in the second, we get a section and bind from it. The section allows you to work with a partial view of the configuration - this way you can control the data set with which we are working. Sections are also used in standard extension methods - for example, getting a connection string uses the "ConnectionStrings" section.

string connectionString = Configuration.GetConnectionString("Default");

public static string GetConnectionString(this IConfiguration configuration, string name)
{
    return configuration?.GetSection("ConnectionStrings")?[name];
}

Options - typed configuration view

Creating a configuration object manually and binding to data is not practical, but there is a solution in the form of using Options . Options are used to get a strongly typed view of a configuration. The view class must be public with a constructor without parameters and public properties for assigning a value, the object is filled through reflection. More details can be found in the source .



To start using Options, we need to register the configuration type using the Configure extension method for IServiceCollection indicating the section that we will project onto our class.

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.Configure<MonitoringConfig>(Configuration.GetSection("Features:Monitoring"));
}

After that, we can receive configurations by injecting a dependency on the IOptions, IOptionsMonitor, IOptionsSnapshot interfaces. We can get the MonitoringConfig object from the IOptions interface through the Value property.

public class ExampleService
{
    private IOptions<MonitoringConfig> _configuration;
    public ExampleService(IOptions<MonitoringConfig> configuration)
    {
        _configuration = configuration;
    }
    public void Run()
    {
        TimeSpan timeSpan = _configuration.Value.StartTime; // 09:00
    }
}

A feature of the IOptions interface is that in the dependency injection container, the configuration is registered as an object with the Singleton lifecycle. The first time a value is requested by the Value property, an object is initialized with data that exists as long as this object exists. IOptions does not support data refresh. There are IOptionsSnapshot and IOptionsMonitor interfaces to support updates.



The IOptionsSnapshot in the DI container is registered with the Scoped lifecycle, which makes it possible to get a new configuration object on request with a new container scope. For example, during one web request we will receive the same object, but for a new request we will receive a new object with updated data.



IOptionsMonitor is registered as a Singleton, with the only difference that each configuration is received with the actual data at the time of the request. In addition, IOptionsMonitor allows you to register a configuration change event handler if you need to respond to the data change event itself.

public class ExampleService
{
    private IOptionsMonitor<MonitoringConfig> _configuration;
    public ExampleService(IOptionsMonitor<MonitoringConfig> configuration)
    {
        _configuration = configuration;
        configuration.OnChange(config =>
        {
            Console.WriteLine(" ");
        });
    }
    
    public void Run()
    {
        TimeSpan timeSpan = _configuration.CurrentValue.StartTime; // 09:00
    }
}

It is also possible to get IOptionsSnapshot and IOptionsMontitor by name - this is necessary if you have several configuration sections corresponding to one class, and you want to get a specific one. For example, we have the following data:

{
  "Cache": {
    "Main": {
      "Type": "global",
      "Interval": "10:00"
    },
    "Partial": {
      "Type": "personal",
      "Interval": "01:30"
    }
  }
}

The type to be used for the projection:

public class CachePolicy
{
    public string Type { get; set; }
    public TimeSpan Interval { get; set; }
}

We register configurations with a specific name:

services.Configure<CachePolicy>("Main", Configuration.GetSection("Cache:Main"));
services.Configure<CachePolicy>("Partial", Configuration.GetSection("Cache:Partial"));

We can receive values ​​as follows:

public class ExampleService
{
    public ExampleService(IOptionsSnapshot<CachePolicy> configuration)
    {
        CachePolicy main = configuration.Get("Main");
        TimeSpan mainInterval = main.Interval; // 10:00
            
        CachePolicy partial = configuration.Get("Partial");
        TimeSpan partialInterval = partial.Interval; // 01:30
    }
}

If you look at the source code of the extension method with which we register the configuration type, you can see that the default name is Options.Default, which is an empty string. So we implicitly always pass in a name for the configurations.

public static IServiceCollection Configure<TOptions>(this IServiceCollection services, IConfiguration config) where TOptions : class
            => services.Configure<TOptions>(Options.Options.DefaultName, config);

Since configuration can be represented by a class, we can also add parameter value validation by marking up the properties using validation attributes from the System.ComponentModel.DataAnnotations namespace. For example, we specify that the value for the Type property must be required. But we also need to indicate when registering the configuration that validation should occur in principle. There is an extension method ValidateDataAnnotations for this.

public class CachePolicy
{
    [Required]
    public string Type { get; set; }
    public TimeSpan Interval { get; set; }
}

services.AddOptions<CachePolicy>()
        .Bind(Configuration.GetSection("Cache:Main"))
        .ValidateDataAnnotations();

The peculiarity of such validation is that it will happen only at the moment of receiving the configuration object. This makes it difficult to understand that the configuration is not valid when the application starts. There is an issue on GitHub for this problem . One solution to this problem can be the approach presented in the article Adding validation to strongly typed configuration objects in ASP.NET Core.

Disadvantages of Options and how to get around them

Configuring via Options also has its drawbacks. For use, we need to add a dependency, and each time we need to access the Value / CurrentValue property to get a value object. You can achieve cleaner code by getting a clean configuration object without the Options wrapper. The simplest solution to the problem may be additional registration in the container of a pure configuration type dependency.

services.Configure<MonitoringConfig>(Configuration.GetSection("Features:Monitoring"));
services.AddScoped<MonitoringConfig>(provider => provider.GetRequiredService<IOptionsSnapshot<MonitoringConfig>>().Value);

The solution is straightforward, we do not force the final code to know about IOptions, but we lose the flexibility for additional configuration actions if we need them. To solve this problem, we can use the "Bridge" pattern, which will allow us to get an additional layer in which we can perform additional actions before receiving the object.



To achieve this goal, we need to refactor the current example code. Since the configuration class has a restriction in the form of a constructor without parameters, we cannot pass the IOptions / IOptionsSnapshot / IOptionsMontitor object to the constructor; for this we will separate the configuration reading from the final view.



For example, let's say we want to specify the StartTime property of the MonitoringConfig class with a string representation of minutes with a value of "09", which doesn't fit the standard format.

public class MonitoringConfigReader
{
    public bool EnableRPSLog { get; set; }
    public bool EnableStorageStatistic { get; set; }
    public string StartTime { get; set; }
}

public interface IMonitoringConfig
{
    bool EnableRPSLog { get; }
    bool EnableStorageStatistic { get; }
    TimeSpan StartTime { get; }
}

public class MonitoringConfig : IMonitoringConfig
{
    public MonitoringConfig(IOptionsMonitor<MonitoringConfigReader> option)
    {
        MonitoringConfigReader reader = option.Value;
        
        EnableRPSLog = reader.EnableRPSLog;
        EnableStorageStatistic = reader.EnableStorageStatistic;
        StartTime = GetTimeSpanValue(reader.StartTime);
    }
    
    public bool EnableRPSLog { get; }
    public bool EnableStorageStatistic { get; }
    public TimeSpan StartTime { get; }
    
    private static TimeSpan GetTimeSpanValue(string value) => TimeSpan.ParseExact(value, "mm", CultureInfo.InvariantCulture);
}

To be able to get a clean configuration, we need to register it in the dependency injection container.

services.Configure<MonitoringConfigReader>(Configuration.GetSection("Features:Monitoring"));
services.AddTransient<IMonitoringConfig, MonitoringConfig>();

This approach allows you to create a completely separate life cycle for the formation of a configuration object. It is possible to add your own data validation, or additionally implement a data decryption stage if you receive it in encrypted form.

Ensuring data security

An important configuration task is data security. File configurations are insecure because the data is stored in clear text, which is easy to read; often the files are in the same directory as the application. By mistake, you can commit the values ​​to the version control system, which can declassify the data, but imagine if this is public code! The situation is so common that there is even a ready-made tool for finding such leaks - Gitleaks . There is a separate article that gives statistics and the variety of disclosed data.



Often a project must have separate parameters for different environments (Release / Debug, etc.). For example, as one of the solutions, you can use substitution of final values ​​using continuous integration and delivery tools, but this option does not protect the data at design time. The User Secrets tool is designed to protect the developer . It is included in the .NET Core SDK (3.0.100 and higher). What is the main principle of this tool? First, we need to initialize our project to work with the init command.

dotnet user-secrets init

The command adds a UserSecretsId element to the .csproj project file. With this parameter, we get a private storage that will store a regular JSON file. The difference is that it is not located in your project directory, so it will only be available on the current computer. The path for Windows is% APPDATA% \ Microsoft \ UserSecrets \ <user_secrets_id> \ secrets.json, and for Linux and MacOS ~ / .microsoft / usersecrets / <user_secrets_id> /secrets.json. We can add the value from the example above with the set command:

dotnet user-secrets set "Features:Monitoring:StartTime" "09:00"

A complete list of available commands can be found in the documentation.



Data security in production is best ensured using specialized storage, such as: AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, Consul, ZooKeeper. To connect some, there are already ready-made NuGet packages, and for some it is easy to implement them yourself, since there is access to the REST API.

Conclusion

Modern problems require modern solutions. Along with the move away from monoliths to dynamic infrastructures, configuration approaches have also undergone changes. There was a need, regardless of the location and type of sources of configuration data, the need for a prompt response to data changes. Together with .NET Core, we got a good tool for implementing all kinds of application configuration scenarios.