👂🏾 🌠 🤶🏻 Understanding Threat Intelligence Sources 🍸 🚶 ✅

According to the 2021 SANS Cyber Threat Intelligence (CTI) Survey , 66.3% of companies use open sources to collect indicators of compromise and try to work with multiple sources at the same time. It would seem that collecting indicators from open sources is a fairly simple task: you just need to download a txt or csv file from some site, and that's it. In reality, a lot of problems can be encountered along the way. In this article, we will tell you what these difficulties can be, what the structure and format of the feed depend on, what metrics help to assess the usefulness of feeds, and also show with a real example what you can learn from the feed. For greater objectivity, we prepared this article together with Kolya Arefiev from RST Cloud ,engaged in the aggregation, enrichment, cleaning and ranking of indicators published by independent information security researchers.

What are the pitfalls of open sources TI

Let's make a reservation right away: it will not be possible to get around the problems when working with indicators from open sources. The very first time you load such indicators into protection tools, you will receive thousands of responses per day. Without disassembling the first 100 hits, you will most likely spit on this business and just turn off the feed.

Let's try to describe the main problems that need to be resolved when collecting indicators from open sources:

Selection of sources

You should not chase the number of indicators in the source and choose those with the most indicators. Make a spot check of some of the indicators from the feed, evaluate them in terms of the number of false positives.

Understanding the structure of your feed

, . , , : , (, , , , ?). , , IP, , url hash . 2C, , url, .

, ? ? , ? ?

, , . , , . ? , , ?

, , . 4 , , . « » , .

127.0.0.1, microsoft.com SHA256 calc.exe — . , .

- . , , ASN, Whois, Geo-, , .. , , , .

, . , IT , , . RST Cloud IoC, 365 .

: « »

, . , , , .

, , . , « ».

« », 2013 David J Bianco.

. , , txt- csv-. - .

, :

812          |  ROGERS-COMMUNICATIONS           |  99.250.237.110   |  2021-04-05 04:46:22 

852          |  ASN852                          |  66.183.170.4     |  2021-04-05 02:50:15 

1101         |  IP-EEND-AS Surfnet B.V.         |  192.42.116.13    |  2021-03-31 19:52:48 

1101         |  IP-EEND-AS Surfnet B.V.         |  192.42.116.14    |  2021-04-03 14:50:55 

1101         |  IP-EEND-AS Surfnet B.V.         |  192.42.116.16    |  2021-04-05 09:45:03 

1101         |  IP-EEND-AS Surfnet B.V.         |  192.42.116.17    |  2021-04-04 18:13:59

Family,URL,IP,FirstSeen

Pony,http[:]//officeman[.]tk/admin.php,207[.]180.230.128,01-06-2020

Pony,http[:]//learn[.]cloudience[.]com/admin.php,192[.]145.234.108,01-06-2020

Pony,http://vman23[.]com/admin.php,95.213.204.53,01-06-2020

. , . , , (TTP) APT.

, , json, yaml, xml - , . , STIX 2, .

, . , json — 1 3 . , , , . .

:

, , , RST Cloud.

json :

{

    "id": "572f891c-dd92-31d3-a2e7-c448a4b72b22",

    "title": "RST Threat feed. IOC: defender5.coachwithak.com",

    "description": "IOC with tags: c2. Related threats: silverfish_group",

    "threat": ["silverfish_group"],

    "domain": "defender5.coachwithak.com",

    "fseen": 1616630400,

    "lseen": 1617408000,

    "collect": 1617494400,

    "tags": ["c2"],

    "resolved": {

        "ip": {

        "a": ["37.48.84.156"],

            "cname": []

        },

      "whois": {

            "created": "2019-07-31 20:36:52",

            "updated": "2020-08-01 10:58:42",

            "expires": "2021-07-31 20:36:52",

            "age": 608,

            "registrar": "GoDaddycom LLC",

            "registrant": "unknown",

            "havedata": "true"

        }

    },

    "score": {

        "total": 55,

        "src": 73.75,

        "tags": 0.89,

        "frequency": 0.85

    },

    "fp": {

        "alarm": "false",

        "descr": ""

 }

}