In the last article, Google Earth Engine (GEE) as a public supercomputer, it was about working in the cloud editor GEE, where you only need to have Google mail for access. If the needs are limited to one-time tasks and gigabytes of recoverable data, then this is quite enough. But for the automation of many even small tasks, a cloud editor is not the best way to work, and even more so when you need to repeatedly receive rasters with a total size of terabytes. In such cases, other tools will be required and today we will look at the possibilities of access from the console shell and Python scripts and Python Jupyter notebook.
In the screenshot of a Python Jupyter laptop, where a raster with population density data for 2020 from the Earth Engine data Catalog: WorldPop Global Project Population Data is displayed on an OpenStreetMap
Introduction
, . , Google Earth Engine (GEE), , . , . , GEE , . , , , (, ). , (ML) , ! , — GEE, Compute Engine . , , .
Google Cloud SDK google-cloud-sdk. ( ) . :
$ gcloud auth list
Credentialed accounts:
- youremail@gmail.com (active)
To set the active account, run
$ gcloud config set account <account>
:
$ gcloud config set account <account>
:
$ gcloud auth login
buckets Google Drive, GEE GEE. , API .
C GEE buckets Export.table.toCloudStorage Export.image.toCloudStorage , Google Compute Engine. gsutil, :
$ gsutil du -h gs://gcp-pdp-osm-dev-earth-engine
(. -h). gsutil , (cp, rm,...), .
GEE Google Drive Export.table.toDrive Export.image.toDrive, - . Google Drive .
GEE API
Google Earth Engine (GEE) my-service-account@...iam.gserviceaccount.com: Create and register a service account to use Earth Engine. GEE KEYS JSON , Register a new service account. Python :
import ee
service_account = 'my-service-account@...iam.gserviceaccount.com'
credentials = ee.ServiceAccountCredentials(service_account, 'privatekey.json')
ee.Initialize(credentials)
Python API ee.Authenticate() .
$ earthengine earthengine --ee_config
, Python GDAL:
import os
from osgeo import gdal
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my-service-account.json"
GDAL:
export GOOGLE_APPLICATION_CREDENTIALS=my-service-account.json
GEE
API Method: projects.assets.getPixels , 32MB. , GDAL API, .
GDAL Python. WorldPop/GP/100m/pop 2020 . , :
export GOOGLE_APPLICATION_CREDENTIALS=my-service-account.json # fetch collection ogrinfo -ro -al "EEDA:" -oo COLLECTION=projects/earthengine-public/assets/WorldPop/GP/100m/pop -where "year=2020" # show one raster info gdalinfo "EEDAI:projects/earthengine-public/assets/WorldPop/GP/100m/pop/ZWE_2020" # fetch one raster to local drive gdal_translate "EEDAI:projects/earthengine-public/assets/WorldPop/GP/100m/pop/ZWE_2020" ZWE_2020.tif
Python:
import os
from osgeo import ogr, gdal
# define service account key
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my-service-account.json"
# fetch collection
driver = ogr.GetDriverByName('EEDA')
ds = driver.Open('EEDA:projects/earthengine-public/assets/WorldPop/GP/100m/pop')
layer = ds.GetLayer()
# filter collection by attribute
layer.SetAttributeFilter('year=2020')
# select 1st raster
for feature in layer:
name = feature.GetField("name")
crs = feature.GetField("band_crs")
print ('raster name and crs:',name, crs)
break
# fetch 1st raster from the collection to array
ds = gdal.Open(f'EEDAI:{name}')
band = ds.GetRasterBand(1)
array = band.ReadAsArray()
print ('raster shape:', array.shape)
«» Google Earth Engine. GEE , Python Jupyter , . , «» — GEE. , GDAL .
I would be interested to get feedback from readers: is it worth addressing more complex topics, or is this already beyond the scope of what interests the Russian-speaking audience? I know that a lot of readers here use Google Transtale and similar translators, perhaps you should immediately write in English on LinkedIn, as I already do with publications on geophysics.
Links
EEDAI - Google Earth Engine Data API Image
Using GDAL / OGR for Data Processing and Analysis
Raster (gridded) dataset handling
How to load GeoJSON files into BigQuery GIS