Data Ingest Plugins

To be discoverable plugins the environmental variable LE_LCCS_PLUGINS_PATH must be set to point at the directory they are in and the name of the file must start with le_ingest

The class must have a function read_to_xarray which takes at least **kwargs.

Examples

The example below inherits from the ‘LEIngestGridded’ class, generates a raster from a vector file using the gdalrasterize command (called via subprocess) and then reads using the function read_data_from_gdal.

import os
import subprocess
import tempfile

# Set up a class to inherit from 'LEIngestGridded'
class LEIngestVector2Raster(LEIngestGridded):
    def read_to_xarray(self, input_vector, variable_name,
                       **kwargs):
        """
        Function to take a vector, rasterise it and then read into an xarray
        """
        # Create temporary output file
        temp_out_raster = tempfile.mkstemp(prefix="lccs_ingest", suffix=".tif")[0]

        # Run gdalrasterize command
        rasterise_cmd = ["gdalrasterize", "-te"]
        rasterise_cmd.extend([self.target_min_x, self.target_min_y,
                                   self.target_max_x, self.target_max_y])
        rasterise_cmd.extend(["-tr", self.target_pixel_size_x, self.target_pixel_size_y])
        rasterise_cmd.extend(["-burn", "1"])
        rasterise_cmd.extend([input_vector, temp_out_raster])

        cmd_out = subprocess.check_output(rasterise_cmd)
        logging.info(cmd_out)

        # Read data using read_data_from_gdal function inherited from LEIngestGridded
        out_xarray = self.read_data_from_gdal(temp_out_raster, variable_name):

        # Remove temporary raster
        os.remove(temp_out_raster)

        return out_xarray

The example below inherits from the ‘LEIngestGridded’ class, reading in data from a static dataset using GDAL using the function read_data_from_gdal, and applying an skimage filter (https://scikit-image.org/docs/dev/api/skimage.filters.html). It was initially used in LCCS system for smoothling static layers on the fly.

import numpy
import xarray
import scipy.ndimage

from le_lccs.le_ingest.gridded_ingest import LEIngestGridded

# Set up a class to inherit from 'LEIngestGridded'
class LEIngest_GDALWithFilter(LEIngestGridded):

    def read_to_xarray(self, input_file, variable_name, expstr=None, band=1,
                       resampling="nearest", **kwargs):
        """
        Function takes a raster and applies skimage filter

        Relevant ingest needs to be added to config file (example)
        layer_name:
            ingest_class: le_ingest_filter_data.LEIngest_GDALWithFilter
            input_file: /g/data/r78/LCCS_Aberwystwyth/ls8_all.vrt
            expstr: "(band > 0.05)"
        """
        # Read from GDAL to xarray
        input_xarray = self.read_data_from_gdal(input_file, variable_name, expstr, band,
                                        resampling, **kwargs)

        # Apply filter and replace values in xarray with filtered
        # values
        input_xarray[variable_name].values = \
            scipy.ndimage.gaussian_filter(input_xarray[variable_name].values, sigma=10)

        # Return
        return input_xarray