Google Earth Engine (GEE) as a public domain for large geodata

In the last article, Google Earth Engine (GEE) as a public supercomputer, it was about working in the cloud editor GEE, where you only need to have Google mail for access. If the needs are limited to one-time tasks and gigabytes of recoverable data, then this is quite enough. But for the automation of many even small tasks, a cloud editor is not the best way to work, and even more so when you need to repeatedly receive rasters with a total size of terabytes. In such cases, other tools will be required and today we will look at the possibilities of access from the console shell and Python scripts and Python Jupyter notebook.









In the screenshot of a Python Jupyter laptop, where a raster with population density data for 2020 from the Earth Engine data Catalog: WorldPop Global Project Population Data is displayed on an OpenStreetMap







Introduction



, . , Google Earth Engine (GEE), , . , . , GEE , . , , , (, ). , (ML) , ! , โ€” GEE, Compute Engine . , , .







Google



Google Cloud SDK google-cloud-sdk. ( ) . :







$ gcloud auth list
Credentialed accounts:
 - youremail@gmail.com (active)
To set the active account, run
 $ gcloud config set account <account>
      
      





:







$ gcloud config set account <account>
      
      





:







$ gcloud auth login
      
      







buckets Google Drive, GEE GEE. , API .







C GEE buckets Export.table.toCloudStorage Export.image.toCloudStorage , Google Compute Engine. gsutil, :







$ gsutil du -h gs://gcp-pdp-osm-dev-earth-engine
      
      





(. -h). gsutil , (cp, rm,...), .







GEE Google Drive Export.table.toDrive Export.image.toDrive, - . Google Drive .







GEE API



Google Earth Engine (GEE) my-service-account@...iam.gserviceaccount.com: Create and register a service account to use Earth Engine. GEE KEYS JSON , Register a new service account. Python :







import ee
service_account = 'my-service-account@...iam.gserviceaccount.com'
credentials = ee.ServiceAccountCredentials(service_account, 'privatekey.json')
ee.Initialize(credentials)
      
      





Python API ee.Authenticate() .







$ earthengine earthengine --ee_config
      
      





, Python GDAL:







import os
from osgeo import gdal

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my-service-account.json"
      
      





GDAL:







export GOOGLE_APPLICATION_CREDENTIALS=my-service-account.json
      
      





GEE



API Method: projects.assets.getPixels , 32MB. , GDAL API, .







GDAL Python. WorldPop/GP/100m/pop 2020 . , :







export GOOGLE_APPLICATION_CREDENTIALS=my-service-account.json

# fetch collection
ogrinfo -ro -al "EEDA:" -oo COLLECTION=projects/earthengine-public/assets/WorldPop/GP/100m/pop -where "year=2020" 
# show one raster info
gdalinfo "EEDAI:projects/earthengine-public/assets/WorldPop/GP/100m/pop/ZWE_2020"
# fetch one raster to local drive
gdal_translate "EEDAI:projects/earthengine-public/assets/WorldPop/GP/100m/pop/ZWE_2020" ZWE_2020.tif
      
      





Python:







import os
from osgeo import ogr, gdal

# define service account key
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my-service-account.json"
# fetch collection
driver = ogr.GetDriverByName('EEDA')
ds = driver.Open('EEDA:projects/earthengine-public/assets/WorldPop/GP/100m/pop')
layer = ds.GetLayer()
# filter collection by attribute
layer.SetAttributeFilter('year=2020')
# select 1st raster
for feature in layer:
    name = feature.GetField("name")
    crs = feature.GetField("band_crs")
    print ('raster name and crs:',name, crs)
    break
# fetch 1st raster from the collection to array
ds = gdal.Open(f'EEDAI:{name}')
band = ds.GetRasterBand(1)
array = band.ReadAsArray()
print ('raster shape:', array.shape)
      
      







ยซยป Google Earth Engine. GEE , Python Jupyter , . , ยซยป โ€” GEE. , GDAL .







I would be interested to get feedback from readers: is it worth addressing more complex topics, or is this already beyond the scope of what interests the Russian-speaking audience? I know that a lot of readers here use Google Transtale and similar translators, perhaps you should immediately write in English on LinkedIn, as I already do with publications on geophysics.







Links



EEDAI - Google Earth Engine Data API Image







Raster API tutorial







Vector layers







Using GDAL / OGR for Data Processing and Analysis







Raster Layers







Raster (gridded) dataset handling







FROM GEE TO NUMPY TO GEOTIFF







How to load GeoJSON files into BigQuery GIS








All Articles