How to Extract Chlorophyll-a data in Level-3 Binned AQUA-MODIS (Netcdf4) Files that are Packed by SCAN Lines of the MODIS instrument #10178

Kaikash777 · 2024-10-19T20:31:34Z

Kaikash777
Oct 19, 2024

Issue

I downloaded AQUA-MODIS netcdf files (total = 3422) for chlorophyll-a from the Earthdata website from 2012 to 2024 for The Northern Red Sea, Egypt.

I need to:

Stack all the 3422 AQUA MODIS netcdf4 files to create a SpatRaster using the function terra::rast() to calculate the mean chlorophyll-a levels of all files.

I tried using packages such as ncdf4, terra, raster, and RNetCDF without any success (see my efforts below). I've also tried looking for solutions on Stackoverflow and did some troubleshooting with suggestions from ChatGPT and nothing has worked. I really don't understand how find a solution.

Here's the manifest.txt link to get the data (expires Oct 22nd 2024)

https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_2.tar?h=ocdist410&p=/data1/d0703031e30f7f6f/requested_files

Read the files into R:

 library(terra)
 library(ncdf4)
 library(raster)
 library(RNetCDF)

#Start the file path to all the netcdf files as a variable 'folder'
folder<-"~/Documents/GIS_Data/CHL-A/"

#Make a list of all the files within the folder 
files <-list.files(folder, pattern='*.nc', full.names ="TRUE")

Loop through all the file paths and the AQUA MODIS netcdf4 files to gain access to them:

for (file in files) {
     nc=open.nc(file)
     print.nc(nc)
}

Contents of one of the AQUA MODIS files

netcdf netcdf4 {

// global attributes:
		NC_CHAR :product_name = "AQUA_MODIS.20221217.L3b.DAY.CHL.x.nc" ;
		NC_CHAR :title = "MODIS Level-3 Binned Data" ;
		NC_CHAR :instrument = "MODIS" ;
		NC_CHAR :platform = "Aqua" ;
		NC_CHAR :temporal_range = "day" ;
		NC_INT :start_orbit_number = 109682 ;
		NC_INT :end_orbit_number = 109698 ;
		NC_CHAR :date_created = "2024289095750000" ;
		NC_CHAR :processing_version = "unspecified" ;
		NC_CHAR :source = "satellite observations from MODIS-Aqua" ;
		NC_CHAR :history = "/sdps/sdpsoper/Science/OCSSW/DEVEL/bin/l3bin ifile=AQUA_MODIS.20221217.L3b.DAY.CHL.nc ofile=AQUA_MODIS.20221217.L3b.DAY.CHL.x.nc latnorth=37.0000 latsouth=5.1000 lonwest=26.7200 loneast=54.1400" ;
		NC_CHAR :time_coverage_start = "2022-12-17T00:25:01.000Z" ;
		NC_CHAR :time_coverage_end = "2022-12-18T02:59:59.000Z" ;
		NC_FLOAT :northernmost_latitude = 36.8958320617676 ;
		NC_FLOAT :southernmost_latitude = 5.10416650772095 ;
		NC_FLOAT :easternmost_longitude = 50.6941337585449 ;
		NC_FLOAT :westernmost_longitude = 26.7268047332764 ;
		NC_DOUBLE :geospatial_lat_max = 36.8958320617676 ;
		NC_DOUBLE :geospatial_lat_min = 5.10416650772095 ;
		NC_DOUBLE :geospatial_lon_max = 50.6941337585449 ;
		NC_DOUBLE :geospatial_lon_min = 26.7268047332764 ;
		NC_CHAR :geospatial_lat_units = "degrees_north" ;
		NC_CHAR :geospatial_lon_units = "degrees_east" ;
		NC_CHAR :geospatial_lat_resolution = "4.638312 km" ;
		NC_CHAR :geospatial_lon_resolution = "4.638312 km" ;
		NC_CHAR :spatialResolution = "4.638312 km" ;
		NC_INT :data_bins = 31421 ;
		NC_FLOAT :percent_data_bins = 0.132233932614326 ;
		NC_CHAR :units = "chlor_a:mg m^-3" ;
		NC_CHAR :binning_scheme = "Integerized Sinusoidal Grid" ;
		NC_CHAR :project = "Ocean Biology Processing Group (NASA/GSFC/OBPG)" ;
		NC_CHAR :institution = "NASA Goddard Space Flight Center, Ocean Ecology Laboratory, Ocean Biology Processing Group" ;
		NC_CHAR :standard_name_vocabulary = "CF Standard Name Table v36" ;
		NC_CHAR :Conventions = "CF-1.6 ACDD-1.3" ;
		NC_CHAR :naming_authority = "gov.nasa.gsfc.sci.oceandata" ;
		NC_CHAR :id = "AQUA_MODIS.20221217.L3b.DAY.CHL.x.nc/L3/AQUA_MODIS.20221217.L3b.DAY.CHL.x.nc" ;
		NC_CHAR :license = "https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/" ;
		NC_CHAR :creator_name = "NASA/GSFC/OBPG" ;
		NC_CHAR :publisher_name = "NASA/GSFC/OBPG" ;
		NC_CHAR :creator_email = "data@oceancolor.gsfc.nasa.gov" ;
		NC_CHAR :publisher_email = "data@oceancolor.gsfc.nasa.gov" ;
		NC_CHAR :creator_url = "https://oceandata.sci.gsfc.nasa.gov" ;
		NC_CHAR :publisher_url = "https://oceandata.sci.gsfc.nasa.gov" ;
		NC_CHAR :processing_level = "L3 Binned" ;
		NC_CHAR :cdm_data_type = "point" ;
		NC_CHAR :keywords_vocabulary = "NASA Global Change Master Directory (GCMD) Science Keywords" ;
		NC_CHAR :keywords = "Earth Science > Oceans > Ocean Chemistry > Pigments > Chlorophyll; Earth Science > Oceans > Ocean Chemistry > Chlorophyll" ;

group: level-3_binned_data {
  dimensions:
  	binListDim = UNLIMITED ; // (31421 currently)
  	binDataDim = UNLIMITED ; // (31421 currently)
  	binIndexDim = UNLIMITED ; // (4320 currently)
  types:
    compound binListType {
      NC_UINT bin_num ;
      NC_SHORT nobs ;
      NC_SHORT nscenes ;
      NC_FLOAT weights ;
      NC_FLOAT time_rec ;
    }; // binListType
    compound binDataType {
      NC_FLOAT sum ;
      NC_FLOAT sum_squared ;
    }; // binDataType
    compound binIndexType {
      NC_UINT start_num ;
      NC_UINT begin ;
      NC_UINT extent ;
      NC_UINT max ;
    }; // binIndexType
  variables:
  	binListType BinList(binListDim) ;
  	binDataType chlor_a(binDataDim) ;
  	binIndexType BinIndex(binIndexDim) ;
  } // group level-3_binned_data

group: processing_control {

  // group attributes:
  		NC_CHAR :software_name = "L3BIN" ;
  		NC_CHAR :software_version = "5.14" ;
  		NC_CHAR :input_sources = "AQUA_MODIS.20221217.L3b.DAY.CHL.nc" ;
  		NC_CHAR :l2_flag_names = "ATMFAIL,LAND,HILT,HISATZEN,STRAYLIGHT,CLDICE,COCCOLITH,LOWLW,CHLWARN,CHLFAIL,NAVWARN,MAXAERITER,ATMWARN,HISOLZEN,NAVFAIL,FILTER,HIGLINT" ;

  group: input_parameters {

    // group attributes:
    		NC_CHAR :infile = "AQUA_MODIS.20221217.L3b.DAY.CHL.nc" ;
    		NC_CHAR :ofile = "AQUA_MODIS.20221217.L3b.DAY.CHL.x.nc" ;
    		NC_CHAR :pfile = "AQUA_MODIS.20221217.L3b.DAY.CHL.x.nc" ;
    		NC_CHAR :oformat = "netCDF4" ;
    		NC_CHAR :syear = "9999" ;
    		NC_CHAR :eyear = "9999" ;
    		NC_CHAR :sday = "1970001" ;
    		NC_CHAR :eday = "2038018" ;
    		NC_CHAR :sorbit = "-1" ;
    		NC_CHAR :eorbit = "-1" ;
    		NC_CHAR :out_parm = ":DEFAULT:" ;
    		NC_CHAR :processing_version = "unspecified" ;
    		NC_CHAR :reduce_fac = "1" ;
    		NC_CHAR :resolve = "" ;
    		NC_CHAR :merged = "" ;
    		NC_CHAR :loneast = "54.139999" ;
    		NC_CHAR :lonwest = "26.719999" ;
    		NC_CHAR :latnorth = "37.000000" ;
    		NC_CHAR :latsouth = "5.100000" ;
    		NC_CHAR :verbose = "0" ;
    		NC_CHAR :unit_wgt = "0" ;
    		NC_CHAR :median = "0" ;
    		NC_CHAR :deflate = "5" ;
    		NC_CHAR :composite_prod = "" ;
    		NC_CHAR :composite_scheme = "" ;
    		NC_CHAR :doi = "" ;
    } // group input_parameters
  } // group processing_control
}

Attempts to Stack the netcdf files:

Attempt 1:

chla1<-terra::rast(files)

Error Message

Error: [rast] cannot open this file as a SpatRaster: /Documents/GIS_Data/CHL-A/AQUA_MODIS.20120101.L3b.DAY.CHL.x.nc
In addition: Warning message:
`Documents/GIS_Data/CHL-A/AQUA_MODIS.20120101.L3b.DAY.CHL.x.nc' not recognized as a supported file format. (GDAL error 4)

Attempt 2:

chla1<-terra::rast(nc)

Error Message:

Error in methods::as(x, "SpatRaster") : 
  no method or default for coercing “NetCDF” to “SpatRaster”

Attempt 3:

chl4<-raster::stack(files)

Error Message

Error in ncvar_type_to_string(rv$precint) : 
Error, unrecognized type code of variable supplied: -1

IMPORTANT INFORMATION

I got an answer to my Stackover flow question:

https://stackoverflow.com/questions/79104406/creating-a-spatraster-with-multiple-aqua-modis-netcdf-files-with-the-function

These are "level-3 binned" data. That means that the data are packed by scan lines of the MODIS instrument. Neither terra nor raster will read that, and even ncdf4 reports an error. The only package that I know of that will open the files without error is ncdfCF but even that won't recognise the data set contained in each file. You can, however, read out the raw data and then process it yourself.

library(ncdfCF)

(chl <- open_ncdf(files))

AQUA_MODIS.20030101.L3b.DAY.CHL

Resource : /my_data_dir/AQUA_MODIS.20030101.L3b.DAY.CHL.nc

Format : netcdf4

Conventions: CF-1.6 ACDD-1.3

Keep open : FALSE

Has groups : TRUE

(vars <- chl$root$unused())

I KEEP ON GETTING AN ERROR WHEN I RUN THIS LINE OF CODE AND THIS IS WHERE I'M STUCK

Error: attempt to apply non-function

Many thanks in advance if anyone can help

Desired Output from the answer in my Stackoverflow question:

$`level-3_binned_data.BinList`

[0] BinList

Group : /level-3_binned_data

Data type : binListType

Dimension ids: 0

$`level-3_binned_data.chlor_a`

[1] chlor_a

Group : /level-3_binned_data

Data type : binDataType

Dimension ids: 1

$`level-3_binned_data.BinIndex`

[2] BinIndex

Group : /level-3_binned_data

Data type : binIndexType

Dimension ids: 2

Reading the data

library(RNetCDF)
d <- var.get.nc(vars[[1]]$group$handle, vars[[1]]$name)
str(d)

List of 5

$ bin_num : num [1:2402743(1d)] 260315 262125 262126 263933 263934 ...

$ nobs : num [1:2402743(1d)] 1 2 2 7 6 9 3 1 1 3 ...

$ nscenes : num [1:2402743(1d)] 1 1 1 2 3 3 3 1 1 1 ...

$ weights : num [1:2402743(1d)] 1 1.41 1.41 3.73 4 ...

$ time_rec: num [1:2402743(1d)] 3.16e+08 6.31e+08 6.31e+08 2.21e+09 1.89e+09 ...

JoshuaKGoldberg · 2024-10-19T21:10:34Z

JoshuaKGoldberg
Oct 19, 2024
Maintainer

This gave me a chuckle. You're nowhere near the right place to be asking about science things. This is the Discussions area for typescript-eslint, a project in the JavaScript/TypeScript tooling ecosystem. Good luck with your science. 🔬

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to Extract Chlorophyll-a data in Level-3 Binned AQUA-MODIS (Netcdf4) Files that are Packed by SCAN Lines of the MODIS instrument #10178

{{title}}

Replies: 1 comment

{{title}}

Select a reply

How to Extract Chlorophyll-a data in Level-3 Binned AQUA-MODIS (Netcdf4) Files that are Packed by SCAN Lines of the MODIS instrument #10178

Kaikash777 Oct 19, 2024

AQUA_MODIS.20030101.L3b.DAY.CHL

Resource : /my_data_dir/AQUA_MODIS.20030101.L3b.DAY.CHL.nc

Format : netcdf4

Conventions: CF-1.6 ACDD-1.3

Keep open : FALSE

Has groups : TRUE

$level-3_binned_data.BinList

[0] BinList

Group : /level-3_binned_data

Data type : binListType

Dimension ids: 0

$level-3_binned_data.chlor_a

[1] chlor_a

Group : /level-3_binned_data

Data type : binDataType

Dimension ids: 1

$level-3_binned_data.BinIndex

[2] BinIndex

Group : /level-3_binned_data

Data type : binIndexType

Dimension ids: 2

Reading the data

List of 5

$ bin_num : num [1:2402743(1d)] 260315 262125 262126 263933 263934 ...

$ nobs : num [1:2402743(1d)] 1 2 2 7 6 9 3 1 1 3 ...

$ nscenes : num [1:2402743(1d)] 1 1 1 2 3 3 3 1 1 1 ...

$ weights : num [1:2402743(1d)] 1 1.41 1.41 3.73 4 ...

$ time_rec: num [1:2402743(1d)] 3.16e+08 6.31e+08 6.31e+08 2.21e+09 1.89e+09 ...

Replies: 1 comment

JoshuaKGoldberg Oct 19, 2024 Maintainer

Kaikash777
Oct 19, 2024

$`level-3_binned_data.BinList`

$`level-3_binned_data.chlor_a`

$`level-3_binned_data.BinIndex`

JoshuaKGoldberg
Oct 19, 2024
Maintainer