Clever Geek Handbook

Elite: Dangerous and CosmosDB

image



o7 cmdr!



On a warm quarantine evening, in one of the telegram chats on Elite: Dangerous, a discussion flared up on the topic: what type of stars have Earth-like planets most often?



The fact is that the exploration of planets is one of the main mechanics of the game. And in the hierarchy of the usefulness of the planets, the Earthlike ones are at the very top. But their rarity is also quite high. So the commanders wanted to know: which stars to pay attention to when moving along the Milky Way?



From this discussion, a whole project was born, which I eventually buried. No, we found the answer to the question posed with it. But I didn't like the project for various reasons, and after a few months of procrastination, I launched the second iteration. What came of this, as well as the answer to the question posed - in this article.



Introduction



In the elite, a huge portion of the gameplay happens, whatever you think, in a browser with a dozen open tabs. We will not argue whether it is right or not, but, definitely, this has its own charm and many people like it.



, , . , Frontier Developments . , .



? , Log- . .



() ( Windows):



C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .



, , , . ? - , ( , , , , POI, etc).



PC:



Tool Commander
E:D Market Connector Otis B.
ED-Intelligent Boardcomputer Extension Duke Jones
edce-client Andargor
EDDI VerticalBlank, Hoodathunk, T'Kael
EDDiscovery Robby
Elite Log Agent John Kozak
Elite G19s Companion app MagicMau
Elite Virtual Assistant
Trade Dangerous + EDAPI orphu


. - . , . , , . , 400 (, ! ). , -.



, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.



, , — . .





EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .



image



% . 6 . .



, , . , , !



, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .



, EDDN ? ...





EDDN



EDDN — :



The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).



HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).



. ZeroMQ tcp://eddn.edcd.io:9500. .



( ):



image



, . ? , . , ( ) ! , - .





, :



image



— Microsoft Azure. - . — Azure, dotnet core/standard



--- , .



, :



1. Message Distributor



EDDN Channel . ( schemaRef , ) — . EDDN 5:





. ? , ( ), , . journal, .. ( ). , , .



, MessageDistributor EDDN . .



2. Azure Storage Queue



( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).



Azure Storage Emulator Azure Storage Explorer


3. Message Processor



, , . Azure Function App.



, - ? .



, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .



, , (Scale Out), , , - , CPU/Memory consumption, etc.



, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .



QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .



. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).



IMHO: , . . 1. , — . (, ) .



Cosmos DB Optimistic Concurrency, . . .



4. CosmosDB



. , ( ) , . , .. .



4: Signals, Systems, Stations, Bodies. , journal ( , ? -).



, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .



CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .





, React Native — dotnet, . .



: EDDNConsumer. , , .



:



  • EDDNConsumerCore — MessageDistributor. EDDN
  • EDDNModels — EDDN
  • JournalContributor — MessageProcessor journal
  • SharedLibrary — . .


EDDNConsumerCore



dotnet core 3.1 . Main DI. HostedService ConsumerService



services.AddHostedService<ConsumerService>();


nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ



StartAsync, — NetMQRuntime ClientAsync ( , ):



private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}


, ( , 1 ), _messageDistributor , , . .



MessageDistributor DistributeAsync:



public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}


, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .



, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .



, , json- NewtonsoftJson ( json, ). Missing Member



_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;


, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .



, header , . , ( ):



public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}


, appsettings json :



{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}


( ):



"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }


Dictionary<string, string> schemaRef. .



1 _queues — QueueClient, . , . , . round trip , . , , . , .



, — , — . . MessageDistributor, .



. 3 . .



EDDNModels



POCO . , . , JournalMessage:



[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}


Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .



. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .



.



JournalContributor



"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).



, :



public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }


QueueTrigger ( ). , Event … .



, , . — , dotnet . Startup Configure, FunctionStartup (assembly):



using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}


DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .



. , , , switch :



public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };


, , . ScanProcessor, .. 2- :



public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }


, , ( !) , , , , . . . , . , : , Bodies Id . , .



UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .



. , 3 , MessageProcessor journal .





, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .



image



continuous, .. .



![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)



, , schemaRef . , .



image



Stream-log : , , … .



image



. , , .



image



RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.



image



. 145 . EDSM, .



image



. Check.



. . .





. . - .



45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .



— . - — , - — "" . .



image



. ~10 . appinsjournalweprivate . Function App. — , . \. :



image



( ), host.json JournalContributor:



{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}


:



image



, ( 21 ):



image



( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .



cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .



— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .



, 175,8 . , .



, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).



-. CosmosDB RU/s ( , ) RU/s (Pricing), :



image



EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .



, Function App Storage Account, , .



, . — .



. .





.



" " " ", . - , . =) , -, ...



— . , - , 145 , EDSM 57 . - - , .



, , , . . " " — .



image



, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .



CosmosDB — Jupyter Notebooks. C#, Python, - . , .



, , . ( , Sol [0,0,0]:



image



? :



image



5 :



image



EDSM, - , . , , — - . , , .



CosmosDB, Azure Function Change Feed ( ), .



— CosmosDB .



For those who have endured to the end and are still interested in where to look for earth-like planets in the elite, here is your answer (the results are based on data obtained from a full EDSM dump about six months ago):



image



F, K, G, A types and, for some reason, neutrons make up the top 5 types of stars from which pilots have found Earth-like planets. Well, these are the results ¯ \ _ (ツ) / ¯



Well. That's all.



Fly safe, cmdr!



More articles:


All Articles