Clever Geek Handbook

How to quickly and easily create snapshots and remove them automatically

The story is about how I had a problem that took 2 days to solve. A discrepancy was found in the documentation and the real world of the Yandex cloud was written in Yandex, but there was no answer.



Yandex.Cloud



Task:



According to the schedule, create SNAPSHOT disks in the used instance. At the same time, it should be possible to mark the disks that need to be backed up and which not.



Subtasks:



  • Snapshots older than n days should be automatically deleted from storage. In this case, it should be possible to change n.
  • Snapshots must have a human-readable title. And match the following pattern: 



    %  %- %-%  %


So that, if necessary, it was obvious to a person from what to deploy new cars. (in the final version, it is implemented by a separate bash script executed from a third-party machine).



Progress:



Y.cloud does not have a standard function that performs this task.



The solution is to implement it at the expense of the function inside Y.cloud itself, to save resources. To create snapshots, a function was written in NodeJS12 



const ycsdk = require("yandex-cloud/api/compute/v1");
const FOLDER_ID = process.env.FOLDER_ID;
async function handler(event, context) {
    const snapshotService = new ycsdk.SnapshotService();
    const diskService = new ycsdk.DiskService();
    const diskList = await diskService.list({
        folderId: FOLDER_ID,
    });
    for (const disk of diskList.disks) {
        if ('snapshot' in disk.labels) {
            snapshotService.create({
                folderId: FOLDER_ID,
                diskId: disk.id
            });
        }
    }
}
exports.handler = handler;


* When calling this function, you need to pass FODLER_ID through the environment and specify the service account.



Next, a scheduled call to this function is added. And the task is solved.



Subtask 1.



Initially, it was decided to do it through the same NodeJS12 function.

Logic of work: analyze the snapshot creation date, compare it with the difference between the current date and n, and if the limit is exceeded, delete it.



To do this, refer to the official documentation and see that the CreatedAt parameter must be of the string type.



OK. We are writing a function that will remove snapshots that are less than 1 hour from birth (for the test). We launch. We get nothing. Nothing at all. Not an error in the error output field, but the action we need.



Iteration 1. 



const ycsdk = require("yandex-cloud/api/compute/v1");

const FOLDER_ID = process.env.FOLDER_ID;

const date = new Date();
date.setHours( date.getHours() - 1 );

async function handler(event, context) {
  const snapshotService = new ycsdk.SnapshotService();

  const {snapshots} = await snapshotService.list({folderId: FOLDER_ID});
  for ( let snapshot in snapshots ) {
    const dateSnapshot = new Date( snapshot.createdAt );
    if ( dateSnapshot.getTime() > date.getTime() ) snapshotService.delete({snapshotId: snapshot.id});
  }
}

exports.handler = handler;


Iteration 2. Change the



function so that it displays a response message in the error body. 



const ycsdk = require("yandex-cloud/api/compute/v1");

const FOLDER_ID = process.env.FOLDER_ID;

const date = new Date();
date.setHours( date.getHours() - 1 );

async function handler(event, context) {
  const snapshotService = new ycsdk.SnapshotService();

  const {snapshots} = await snapshotService.list({folderId: FOLDER_ID});
  throw Error( JSON.stringify( snapshots ) );
}

exports.handler = handler;

   :
«"errorMessage": "[{\"labels\":{},\"productIds\":[],\"id\":\"fd813o0n3p753lhqphie\",\"folderId\":\"b1gfub3omefcfvchsd0f\",\"createdAt\":{\"seconds\":{\"low\":1594137358,\"high\":0,\"unsigned\":false}},\"diskSize\":{\"low\":1073741824,\"high\":0,\"unsigned\":false},\"status\":2,\"sourceDiskId\":\"ef3ivjn6340h9e8incbq\"},…..»


Having combed which, we see the following: 



{
    "labels": {},
    "productIds": [],
    "id": "fd813o0n3p753lhqphie",
    "folderId": "b1gfub3omefcfvchsd0f",
    "createdAt": {
      "seconds": {
        "low": 1594137358,
        "high": 0,
        "unsigned": false
      }


From here we draw a conclusion. CreatedAt is not a string, but an object.



Iteration 3.



We are trying to work with CreatedAt. We change the function. 



const ycsdk = require("yandex-cloud/api/compute/v1");

const FOLDER_ID = process.env.FOLDER_ID;

const date = new Date();
date.setHours( date.getHours() - 1 );

async function handler(event, context) {
  const snapshotService = new ycsdk.SnapshotService();

  const {snapshots} = await snapshotService.list({folderId: FOLDER_ID});
  for ( let snapshot in snapshots ) {

    if ( snapshot.createdAt.seconds.low > date.getTime() / 1000 ) {
      snapshotService.delete({snapshotId: snapshot.id});
    }
  }
}

exports.handler = handler;


We get the error: 



{
    "errorMessage": "Cannot read property 'seconds' of undefined",
    "errorType": "TypeError",
    "stackTrace": [
        {
            "function": "Runtime.handler",
            "file": "/function/code/index.js",
            "line": 14,
            "column": 29
        }
    ]


The error tells us that we are trying to grab the seconds property from a non-existent object, although we previously observed the output of the properties of the response object "createdAt":{"seconds":{"low":1594137358,"high":0,"unsigned":false}}

Iteration 4. 



const ycsdk = require("yandex-cloud/api/compute/v1");

const FOLDER_ID = process.env.FOLDER_ID;

const date = new Date();
date.setHours( date.getHours() - 1 );

async function handler(event, context) {
  const snapshotService = new ycsdk.SnapshotService();

  const {snapshots} = await snapshotService.list({folderId: FOLDER_ID});
  for ( let i = 0; i < 5; i++ ) {
    throw Error( JSON.stringify( snapshots[i].createdAt ) );
  }

}

exports.handler = handler;


We get the error: 



{
    "errorMessage": "{\"seconds\":{\"low\":1594137358,\"high\":0,\"unsigned\":false}}",
    "errorType": "Error",
    "stackTrace": [
        {
            "function": "Runtime.handler",
            "file": "/function/code/index.js",
            "line": 13,
            "column": 11
        }
    ]
}


Reduced the loop to 5 iterations for convenience.



Iteration 5. 



const ycsdk = require("yandex-cloud/api/compute/v1");

const FOLDER_ID = process.env.FOLDER_ID;

const date = new Date();
date.setHours( date.getHours() - 1 );

async function handler(event, context) {
  const snapshotService = new ycsdk.SnapshotService();

  const {snapshots} = await snapshotService.list({folderId: FOLDER_ID});
  const list = [];
  list.push( date.getTime() / 1000 );
  for ( let i in snapshots ) {
    const d = new Date( snapshots[i].createdAt );
    list.push( d.getTime() / 1000 );
  }
  throw Error( JSON.stringify( list ) );

}

exports.handler = handler;
;


We get the error: 



{
    "errorMessage": "[1594135869.705,null,null,null,null,null,null,null]",
    "errorType": "Error",
    "stackTrace": [
        {
            "function": "Runtime.handler",
            "file": "/function/code/index.js",
            "line": 18,
            "column": 9
        }
    ]
}


This answer tells us that the Date function was unable to parse the supposedly createdAt property string, which should contain the date and time as a string, according to the documentation.



Total - in the Yandex Cloud platform, another discrepancy between the documentation and the real state of affairs was found. If you have the same task as mine, you can now not spend a whole working day on it.


More articles:


All Articles