import earthaccess
from pprint import pprint
import xarray as xr
import geopandas as gpd
Data discovery with earthaccess
Summary
In this example we will use the earthaccess
library to search for data collections from NASA Earthdata. earthaccess
is a Python library that simplifies data discovery and access to NASA Earth science data by providing an abstraction layer for NASA’s Common Metadata Repository (CMR) API Search API. The library makes searching for data more approachable by using a simpler notation instead of low level HTTP queries. earthaccess
takes the trouble out of Earthdata Login authentication, makes search easier, and provides a stream-line way to download or stream search results into an xarray
object.
For more on earthaccess
visit the earthaccess
GitHub page and/or the earthaccess
documentation site. Be aware that earthaccess
is under active development.
Prerequisites
An Earthdata Login account is required to access data from NASA Earthdata. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.
Learning Objectives
- How to authenticate with
earthaccess
- How to use
earthaccess
to search for data using spatial and temporal filters - How to explore and work with search results
Get Started
Import Required Packages
import os
"HOME"] = "/home/jovyan" os.environ[
= earthaccess.login()
auth # are we authenticated?
if not auth.authenticated:
# ask for credentials and persist them in a .netrc file
="interactive", persist=True) auth.login(strategy
We are already authenticated with NASA EDL
Search for data
There are multiple keywords we can use to discovery data from collections. The table below contains the short_name
, concept_id
, and doi
for some collections we are interested in for other exercises. Each of these can be used to search for data or information related to the collection we are interested in.
Shortname | Collection Concept ID | DOI |
---|---|---|
GPM_3IMERGDF | C2723754864-GES_DISC | 10.5067/GPM/IMERGDF/DAY/07 |
MOD10C1 | C1646609808-NSIDC_ECS | 10.5067/MODIS/MOD10C1.061 |
SPL4SMGP | C2531308461-NSIDC_ECS | 10.5067/EVKPQZ4AFC4D |
SPL4SMAU | C2537927247-NSIDC_ECS | 10.5067/LWJ6TF5SZRG3 |
But wait…You may be asking “how can we find the shortname
, concept_id
, and doi
for collections not in the table above?”. Let’s take a quick detour.
https://search.earthdata.nasa.gov/search?q=GPM_3IMERGDF
Search by collection
#collection_id = 'C2723754864-GES_DISC'
= 'C1598621096-GES_DISC' collection_id
= earthaccess.search_data(
results = collection_id,
concept_id = True,
cloud_hosted = 10 # Restricting to 10 records returned
count )
Granules found: 7792
In this example we used the concept_id
parameter to search from our desired collection. However, there are multiple ways to specify the collection(s) we are interested in. Alternative parameters include:
doi
- request collection by digital object identifier (e.g.,doi
= ‘10.5067/GPM/IMERGDF/DAY/07’)
short_name
- request collection by CMR shortname (e.g.,short_name
= ‘GPM_3IMERGDF’)
NOTE: Each Earthdata collect has a unique concept_id
and doi
. This is not the case with short_name
. A shortname can be associated with multiple versions of a collection. If multiple versions of a collection are publicaly available, using the short_name
parameter with return all versions available. It is advised to use the version
parameter in conjunction with the short_name
parameter with searching.
We can refine our search by passing more parameters that describe the spatiotemporal domain of our use case. Here, we use the temporal
parameter to request a date range and the bounding_box
parameter to request granules that intersect with a bounding box.
For our bounding box, we are going to read in a GeoJSON file containing a single feature and extract the coordinate pairs for the southeast corner and the northwest corner (or lowerleft and upperright corners) of the bounding box around the feature.
= gpd.read_file('../../NOAAHackDay-Dec-2023/data/sf_to_sierranvmt.geojson') inGeojson
= inGeojson.total_bounds xmin, ymin, xmax, ymax
We will assign our start date and end date to a variable named date_range
and we’ll assign the southeast and the northwest corner coordinates to a variable named bbox
to be passed to our earthaccess
search request.
#date_range = ("2022-11-19", "2023-04-06")
= ("2019-11-19", "2019-12-06")
date_range = (xmin, ymin, xmax, ymax) bbox
= earthaccess.search_data(
results = collection_id,
concept_id = True,
cloud_hosted = date_range,
temporal = bbox,
bounding_box )
Granules found: 18
- The
short_name
andconcept_id
search parameters can be used to request one or multiple collections per request, but thedoi
parameter can only request a single collection.
>concept_ids
= [‘C2723754864-GES_DISC’, ‘C1646609808-NSIDC_ECS’]
- Use the
cloud_hosted
search parameter only to search for data assets available from NASA’s Earthdata Cloud. - There are even more search parameters that can be passed to help refine our search, however those parameters do have to be populated in the CMR record to be leveraged. A non exhaustive list of examples are below:
day_night_flag = 'day'
cloud_cover = (0, 10)
# col_ids = ['C2723754864-GES_DISC', 'C1646609808-NSIDC_ECS', 'C2531308461-NSIDC_ECS', 'C2537927247-NSIDC_ECS'] # Specify a list of collections to pass to the search
# results = earthaccess.search_data(
# concept_id = col_ids,
# #cloud_hosted = True,
# temporal = date_range,
# bounding_box = bbox,
# )
Working with earthaccess
returns
earthaccess
provides several convenience methods to help streamline processes that historically have be painful when done using traditional methods. Following the search for data, you’ll likely take one of two pathways with those results. You may choose to download the assets that have been returned to you or you may choose to continue working with the search results within the Python environment.
Download earthaccess
results
In some cases you may want to download your assets. earthaccess
makes downloading the data from the search results very easy using the earthaccess.download()
function.
= earthaccess.download(
downloaded_files 0:9],
results[='../../NOAAHackDay-Dec-2023/data',
local_path )
Getting 9 granules, approx download size: 0.27 GB
earthaccess
did a lot of heavy lifting for us. It identified the downloadable links, passed our Earthdata Login credentials, and save off the file with the proper name. Amazing right!?
We’re going to remove those files to keep our space clean.
!rm ../../NOAAHackDay-Dec-2023/data/*.nc4
Explore earthaccess
search response
print(f'The results variable is a {type(results)} of {type(results[0])}')
The results variable is a <class 'list'> of <class 'earthaccess.results.DataGranule'>
len(results)
18
We can explore the first item (earthaccess.results.DataGranule
) in our list.
= results[0]
item type(item)
earthaccess.results.DataGranule
Each item contains three keys that can be used to explore the item
item.keys()
dict_keys(['meta', 'umm', 'size'])
'umm'] item[
{'RelatedUrls': [{'URL': 'https://data.gesdisc.earthdata.nasa.gov/data/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4', 'Type': 'GET DATA', 'Description': 'Download 3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4'}, {'URL': 's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4', 'Type': 'GET DATA VIA DIRECT ACCESS', 'Description': 'This link provides direct download access via S3 to the granule'}, {'URL': 'https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4', 'Type': 'USE SERVICE API', 'Subtype': 'OPENDAP DATA', 'Description': 'The OPENDAP location for the granule.', 'MimeType': 'application/x-netcdf-4'}, {'URL': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials', 'Type': 'VIEW RELATED INFORMATION', 'Description': 'api endpoint to retrieve temporary credentials valid for same-region direct s3 access'}], 'SpatialExtent': {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180.0, 'EastBoundingCoordinate': 180.0, 'NorthBoundingCoordinate': 90.0, 'SouthBoundingCoordinate': -90.0}]}}}, 'ProviderDates': [{'Date': '2020-02-27T16:10:05.000Z', 'Type': 'Insert'}, {'Date': '2020-02-27T16:10:05.000Z', 'Type': 'Update'}], 'CollectionReference': {'ShortName': 'GPM_3IMERGDF', 'Version': '06'}, 'DataGranule': {'DayNightFlag': 'Unspecified', 'Identifiers': [{'Identifier': '3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4', 'IdentifierType': 'ProducerGranuleId'}], 'ProductionDateTime': '2020-02-27T16:10:05.000Z', 'ArchiveAndDistributionInformation': [{'Name': 'Not provided', 'Size': 29.92357635498047, 'SizeUnit': 'MB'}]}, 'TemporalExtent': {'RangeDateTime': {'BeginningDateTime': '2019-11-19T00:00:00.000Z', 'EndingDateTime': '2019-11-19T23:59:59.999Z'}}, 'GranuleUR': 'GPM_3IMERGDF.06:3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4', 'MetadataSpecification': {'URL': 'https://cdn.earthdata.nasa.gov/umm/granule/v1.6.5', 'Name': 'UMM-G', 'Version': '1.6.5'}}
Get data URLs / S3 URIs
Get links to data. The data_links()
method is used to return the URL(s)/data link(s) for the item. By default the method returns the HTTPS URL to download or access the item.
item.data_links()
['https://data.gesdisc.earthdata.nasa.gov/data/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4']
The data_links()
method can also be used to get the s3 URI when we want to perform direct s3 access of the data in the cloud. To get the s3 URI, pass access = 'direct'
to the method.
='direct') item.data_links(access
['s3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4']
If we want to extract all of the data links from our search results and add or save them to a list, we can.
= []
data_link_list
for granule in results:
for asset in granule.data_links(access='direct'):
data_link_list.append(asset)
0:9] data_link_list[
['s3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191119-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191120-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191121-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191122-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191123-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191124-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191125-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191126-S000000-E235959.V06.nc4',
's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGDF.06/2019/11/3B-DAY.MS.MRG.3IMERG.20191127-S000000-E235959.V06.nc4']
We can pass or read these lists of data links into libraries like xarray
, rioxarray
, or gdal
, but earthaccess
has a built-in module for easily reading these data links in.
Open results in xarray
We use earthaccess
’s open()
method to make a connection to and open the files from our search result.
= earthaccess.open(results) fileset
Opening 18 granules, approx size: 0.53 GB
Then we pass the fileset
object to xarray.
= xr.open_mfdataset(fileset, chunks = {}) ds
Some really cool things just happened here! Not only were we able to seamlessly stream our earthaccess
search results into a xarray
dataset
using the open_mfdataset()
(multi-file) method, but earthaccess
whether we were working from within AWS us-west-2 and could use direct S3 access or if not, would use https. We didn’t have to create a session or a filesystem to authenticate and connect to the data. earthaccess
did this for us using the auth
object we created at the beginning of this tutorial.
Let’s take a quick lock at our xarray
dataset
ds
<xarray.Dataset> Dimensions: (time: 18, lon: 3600, lat: 1800, nv: 2) Coordinates: * lon (lon) float32 -179.9 -179.8 ... 179.9 179.9 * lat (lat) float32 -89.95 -89.85 ... 89.85 89.95 * time (time) object 2019-11-19 00:00:00 ... 2019-12-... Dimensions without coordinates: nv Data variables: precipitationCal (time, lon, lat) float32 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> precipitationCal_cnt (time, lon, lat) int8 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> precipitationCal_cnt_cond (time, lon, lat) int8 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> HQprecipitation (time, lon, lat) float32 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> HQprecipitation_cnt (time, lon, lat) int8 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> HQprecipitation_cnt_cond (time, lon, lat) int8 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> randomError (time, lon, lat) float32 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> randomError_cnt (time, lon, lat) int8 dask.array<chunksize=(1, 3600, 1800), meta=np.ndarray> time_bnds (time, nv) object dask.array<chunksize=(1, 2), meta=np.ndarray> Attributes: BeginDate: 2019-11-19 BeginTime: 00:00:00.000Z EndDate: 2019-11-19 EndTime: 23:59:59.999Z FileHeader: StartGranuleDateTime=2019-11-19T00:00:00.000Z;\nStopGran... InputPointer: 3B-HHR.MS.MRG.3IMERG.20191119-S000000-E002959.0000.V06B.... title: GPM IMERG Final Precipitation L3 1 day 0.1 degree x 0.1 ... DOI: 10.5067/GPM/IMERGDF/DAY/06 ProductionTime: 2020-02-27T16:09:48.308Z
- lonPandasIndex
PandasIndex(Float64Index([ -179.9499969482422, -179.84999084472656, -179.75, -179.64999389648438, -179.5500030517578, -179.4499969482422, -179.34999084472656, -179.25, -179.14999389648438, -179.0500030517578, ... 179.0500030517578, 179.15000915527344, 179.25001525878906, 179.3500213623047, 179.4499969482422, 179.5500030517578, 179.65000915527344, 179.75001525878906, 179.8500213623047, 179.9499969482422], dtype='float64', name='lon', length=3600))
- latPandasIndex
PandasIndex(Float64Index([-89.94999694824219, -89.8499984741211, -89.75, -89.64999389648438, -89.54999542236328, -89.44999694824219, -89.3499984741211, -89.25, -89.14999389648438, -89.04999542236328, ... 89.05000305175781, 89.15000915527344, 89.25, 89.35000610351562, 89.45001220703125, 89.55000305175781, 89.65000915527344, 89.75, 89.85000610351562, 89.95001220703125], dtype='float64', name='lat', length=1800))
- timePandasIndex
PandasIndex(CFTimeIndex([2019-11-19 00:00:00, 2019-11-20 00:00:00, 2019-11-21 00:00:00, 2019-11-22 00:00:00, 2019-11-23 00:00:00, 2019-11-24 00:00:00, 2019-11-25 00:00:00, 2019-11-26 00:00:00, 2019-11-27 00:00:00, 2019-11-28 00:00:00, 2019-11-29 00:00:00, 2019-11-30 00:00:00, 2019-12-01 00:00:00, 2019-12-02 00:00:00, 2019-12-03 00:00:00, 2019-12-04 00:00:00, 2019-12-05 00:00:00, 2019-12-06 00:00:00], dtype='object', length=18, calendar='julian', freq='D'))
- BeginDate :
- 2019-11-19
- BeginTime :
- 00:00:00.000Z
- EndDate :
- 2019-11-19
- EndTime :
- 23:59:59.999Z
- FileHeader :
- StartGranuleDateTime=2019-11-19T00:00:00.000Z; StopGranuleDateTime=2019-11-19T23:59:59.999Z
- InputPointer :
- 3B-HHR.MS.MRG.3IMERG.20191119-S000000-E002959.0000.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S003000-E005959.0030.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S010000-E012959.0060.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S013000-E015959.0090.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S020000-E022959.0120.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S023000-E025959.0150.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S030000-E032959.0180.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S033000-E035959.0210.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S040000-E042959.0240.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S043000-E045959.0270.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S050000-E052959.0300.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S053000-E055959.0330.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S060000-E062959.0360.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S063000-E065959.0390.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S070000-E072959.0420.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S073000-E075959.0450.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S080000-E082959.0480.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S083000-E085959.0510.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S090000-E092959.0540.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S093000-E095959.0570.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S100000-E102959.0600.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S103000-E105959.0630.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S110000-E112959.0660.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S113000-E115959.0690.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S120000-E122959.0720.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S123000-E125959.0750.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S130000-E132959.0780.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S133000-E135959.0810.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S140000-E142959.0840.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S143000-E145959.0870.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S150000-E152959.0900.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S153000-E155959.0930.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S160000-E162959.0960.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S163000-E165959.0990.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S170000-E172959.1020.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S173000-E175959.1050.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S180000-E182959.1080.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S183000-E185959.1110.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S190000-E192959.1140.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S193000-E195959.1170.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S200000-E202959.1200.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S203000-E205959.1230.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S210000-E212959.1260.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S213000-E215959.1290.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S220000-E222959.1320.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S223000-E225959.1350.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S230000-E232959.1380.V06B.HDF5;3B-HHR.MS.MRG.3IMERG.20191119-S233000-E235959.1410.V06B.HDF5
- title :
- GPM IMERG Final Precipitation L3 1 day 0.1 degree x 0.1 degree (GPM_3IMERGDF)
- DOI :
- 10.5067/GPM/IMERGDF/DAY/06
- ProductionTime :
- 2020-02-27T16:09:48.308Z