5 NASA EarthData Download

Profile-CDM Profile-EDU Profile-GEO

Using NASA EarthData Account to Download Data in R

Date Modified: June 28, 2024

Author: Mitchell Manware author-mm

Key Terms: Geospatial Data, Remote Sensing

Programming Languages: R, Bash

5.1 Introduction

5.1.1 Motivation

NASA’s Earth Observing System Data and Information System (EOSDIS) and its twelve Distributed Active Archive Centers (DAAC) are home to a wide range of open access Earth science data.

  • Alaska Satellite Facility (ASF) DAAC
  • Atmospheric Science Data Center (ASDC)
  • Crustal Dynamics Data Information System (CDDIS)
  • Global Hydrometeorology Resource Center (GHRC)
  • Goddard Earth Sciences Data and Information Services Center (GES DISC)
  • Land Processes DAAC (LP DAAC)
  • Level 1 and Atmosphere Archive and Distribution System (LAADS) DAAC
  • National Snow and Ice Data Center (NSIDC) DAAC
  • Oak Ridge National Laboratory (ORNL) DAAC
  • Ocean Biology DAAC (OB.DAAC)
  • Physical Oceanography DAAC (PO.DAAC)
  • Socioeconomic Data and Applications Center (SEDAC)

See https://www.earthdata.nasa.gov/eosdis/daacs for more information (14).

Many of the NASA EOSDIS data sets are relevant to environmental health research, and accessing the data from within R is important for reproducability and repeatable analyses. Although the data is openly accessible, users are required to register for an EarthData account and/or be logged in if you have an account.

The NASA EarthData Account prerequisite files can be generated directly from the command line or using a text editor (notepad in Windows). See How to Generate Earthdata Prerequisite Files for detailed instructions on creating prerequisite files with both the command line and Python (15).

5.1.2 Objectives

Users will:

  • Register for or log into a NASA EarthData Account
  • Generate NASA EarthData Account prerequisite files using R
  • Practice downloading data from a URL

5.2 NASA EarthData Account

5.2.1 Register or Log In

Visit https://urs.earthdata.nasa.gov/ to register for or log into a NASA EarthData account.

NASA EarthData Account Landing Page

5.2.2 Approved Applications

After creating an account, navigate to “My Profile”(https://urs.earthdata.nasa.gov/profile), and then to “Applications > Authorized Apps”. This “Authorized Apps” page specifies which NASA EarthData applications can use your login credentials. Authorize the applications from which you will be downloading data.

NASA EarthData Approved Applications

5.3 Generate Prerequisite Files

Downloading password-protected data from a URL requires user credentials. Without prerequisite files containing user credentials, the data will not be downloaded correctly.

Without the prerequisite files the download step run without error, but trying to open the zip file will return an error.

To demonstrate, try to download population density data (16) from NASA’s Socioeconomic Data and Applications Center (SEDAC) archive center.

NASA SEDAC Population Density Data Characteristics
Metric Population Density
Year 2020
Resolution ~5 km
Format GeoTiff
URL https://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-density-rev11

Define the data URL and destination file.

url <- paste0(
  "https://sedac.ciesin.columbia.edu/downloads/data/gpw-v4/gpw-v4-population-",
  "density-rev11/gpw-v4-population-density-rev11_2020_2pt5_min_tif.zip"
)
destfile <- paste0(
  "/   YOUR FILE PATH   /sedac_population_2020_5km.zip"
)

Run the download command using system() and unzip the file with unzip().

system(
  command = paste0(
    "curl -n -c -b -LJ -o ",
    destfile,
    " --url ",
    url
  )
)
unzip(destfile)

As expected, the data was not downloaded.

To download the password protected data with command line commands, we must generate the .netrc, .urs_cookies, and .dodsrc prerequisite files.

The following steps return errors for Windows system users. File generation on Windows is currently in development.

5.3.1 .netrc

The following commands create the .netrc file, which contains your NASA EarthData Account credentials.

First, set your working directory to the home directory. Setting a working directory differs between Mac/Linux and Windows machines.

if (.Platform$OS.type == "unix") {
  setwd("~/")
} else if (.Platform$OS.type == "windows") {
  setwd("C:/")
}

Create a file named .netrc with file.create().

file.create(".netrc")

Open a connection to .netrc with sink(). Write the line machine urs... replacing YOUR_USERNAME and YOUR_PASSWORD with your NASA EarthData username and password, respectively. After writing the line, close the connection with sink() again.

sink(".netrc")
writeLines(
  "machine urs.earthdata.nasa.gov login YOUR_USERNAME password YOUR_PASSWORD"
)
sink()

Edit the settings so only you, the owner of the file, can read and write .netrc.

system("chmod 0600 .netrc")

After, check to ensure the file was created properly.

file.exists(".netrc")
TRUE
readLines(".netrc")
paste0(
  "machine urs.earthdata.nasa.gov login YOUR_USERNAME password YOUR_PASSWORD"
)

5.3.2 .urs_cookies

The following commands create the .urs_cookies file.

First, set your working directory to the home directory.

if (.Platform$OS.type == "unix") {
  setwd("~/")
} else if (.Platform$OS.type == "windows") {
  setwd("C:/")
}

Create a file named .netrc with file.create().

file.create(".urs_cookies")

After, check to ensure the file was created properly.

file.exists(".urs_cookies")
TRUE

5.3.3 .dodsrc

The following commands create the .dodsrc file.

First, set your working directory to the home directory.

if (.Platform$OS.type == "unix") {
  setwd("~/")
} else if (.Platform$OS.type == "windows") {
  setwd("C:/")
}

Create a file named “.dodsrc” with file.create()

file.create(".dodsrc")

Open a connection to .dodsrc with sink(). Write the lines beginning with HTTP., replacing YOUR_USERNAME and YOUR_PASSWORD with your NASA EarthData username and password, respectively. After writing the line, close the connection with sink() again.

sink(".dodsrc")
writeLines(
  paste0(
    "HTTP.NETRC=YOUR_HOME_DIRECTORY/.netrc\n",
    "HTTP.COOKIE.JAR=YOUR_HOME_DIRECTORY/.urs_cookies"
  )
)
sink()

After, check to ensure the file was created properly.

file.exists(".dodsrc")
TRUE
readLines(".dodsrc")
paste0(
  c(
    "HTTP.NETRC=YOUR_HOME_DIRECTORY/.netrc",
    "HTTP.COOKIE.JAR=YOUR_HOME_DIRECTORY/.urs_cookies"
  )
)

If working on a Windows machine, copy the .dodsrc file to the project working directory. Replace YOUR_WORKING_DIRECTORY with the absolute path to the project working directory.

if (.Platform$OS.type == "windows") {
  file.copy(
    "C:/.dodsrc",
    "YOUR_WORKING_DIRECTORY/.dodsrc"
  )
}

Enter these commands, as well as your username, password, and home directory, without error. Even a single misplaced character can disrupt the verification of your EarthData credentials.

5.4 Download Data

With the prerequisite files generated, try to download the SEDAC population density data again.

Be sure to authorize the “SEDAC” applications at “My Profile”(https://urs.earthdata.nasa.gov/profile) under “Applications > Authorized Apps” before running the following command.

system(
  command = paste0(
    "curl -n -c ~/.urs_cookies -b .urs_cookies -LJ -o ",
    destfile,
    " --url ",
    url
  )
)
unzip(destfile)

The data is downloaded successfully after the prerequisite files have been generated and the “SEDAC” applications have been authorized.

References

14.
National Aeronautics and Space Administration (NASA). n.d. EOSDIS Distributed Active Archive Centers (DAAC). Website. https://www.earthdata.nasa.gov/eosdis/daacs
15.
National Aeronautics and Space Administration (NASA). n.d. How to Generate Earthdata Prerequisite Files. Website. https://disc.gsfc.nasa.gov/information/howto?title=How%20to%20Generate%20Earthdata%20Prerequisite%20Files
16.
Center for International Earth Science Information Network (CIESIN) - Columbia University. 2018. Gridded Population of the World, Version 4 (GPWv4): Population Density, Revision 11. Palisades, New York: NASA Socioeconomic Data and Applications Center (SEDAC). Dataset. https://doi.org/10.7927/H49C6VHW