Pytroll/Satpy is a collection of open-source Python modules for reading, processing, and writing Earth observation satellite data. Satpy provides high-level processing capabilities for both geostationary and low Earth orbit satellites. It supports many input and output formats and resampling data to different projections. To handle large datasets from new satellites, Satpy uses optimized tools like Dask for parallel and out-of-memory computations, improving performance. The Pytroll community develops these tools in an agile and collaborative manner.
Pytroll/Satpy is a collection of open-source Python modules for reading, processing, and writing Earth observation satellite data. Satpy provides high-level processing capabilities for both geostationary and low Earth orbit satellites. It supports many input and output formats and resampling data to different projections. To handle large datasets from new satellites, Satpy uses optimized tools like Dask for parallel and out-of-memory computations, improving performance. The Pytroll community develops these tools in an agile and collaborative manner.
Pytroll/Satpy is a collection of open-source Python modules for reading, processing, and writing Earth observation satellite data. Satpy provides high-level processing capabilities for both geostationary and low Earth orbit satellites. It supports many input and output formats and resampling data to different projections. To handle large datasets from new satellites, Satpy uses optimized tools like Dask for parallel and out-of-memory computations, improving performance. The Pytroll community develops these tools in an agile and collaborative manner.
Pytroll/Satpy is a collection of open-source Python modules for reading, processing, and writing Earth observation satellite data. Satpy provides high-level processing capabilities for both geostationary and low Earth orbit satellites. It supports many input and output formats and resampling data to different projections. To handle large datasets from new satellites, Satpy uses optimized tools like Dask for parallel and out-of-memory computations, improving performance. The Pytroll community develops these tools in an agile and collaborative manner.
with Pytroll/Satpy Salomon Eliasson, Martin Raspaud, Adam Dybbroe SMHI
martin.raspaud@smhi.se What is Pytroll/Satpy? - Pytroll is a collection of free and open source python modules - For reading, processing and writing EO satellite data - Satpy is an easy to use front-end module, eg to generate imagery
martin.raspaud@smhi.se Sentinel 2A, MSI
Pytroll/Satpy in action from glob import glob from satpy.scene import Scene
martin.raspaud@smhi.se SatPy - High level processing for satellite data - Both GEO and LEO - Indexing of channels by name or wavelength - Many built-in composites - Read many input formats - Write many output formats - Resample data to any PROJ.4 projection
Himawari-8, AHI, fire temperature product
martin.raspaud@smhi.se The challenge
martin.raspaud@smhi.se New Missions, Much more data - GOES 16 and 17 ABI (3x, 4x, 5x) - Himawari 8 and 9 AHI - Sentinel 1, 2, 3 in the order of 10 000 x 10 000 pixels per segment - EPS-SG - MTG FCI
GOES-16, ABI martin.raspaud@smhi.se Data-size problem - Too much data to fit in memory of regular computers - Too long processing times due to single-threading
High Res!
Himawari 8, AHI Credits: Simon R. Proud martin.raspaud@smhi.se Mitigation: Optimized data processing
martin.raspaud@smhi.se Efficient tools increase performance - Python Scientific Stack: Numpy, Scipy - Resampling: Pyresample & Pykdtree - Tiepoint interpolation: Python-geotiepoints - Spectral-domain computations: Pyspectral - Orbital and space computations: Pyorbital Pyresample resampling performance vs Scipy and libANN martin.raspaud@smhi.se Mitigation: Out of memory computations
martin.raspaud@smhi.se Using Dask for parallel computations - Lazy processing - Out-of-memory/Chunked processing - Implements the numpy array interface
Example of chunked processing over time
martin.raspaud@smhi.se Some performance results ● SatPy 0.8.4 - single core numpy ○ First execution crashed at 30m just before saving to geotiff ○ Total Time: ~23m (disk cache) ○ Peak Memory Usage: ~103GB ○ Time spent on I/O: 6m14s
● SatPy 0.9.0a1 - 8 Worker Threads
(Dask): ○ Total Time: 5m38s ○ Peak Memory Usage: ~12GB ○ Time spent on I/O: 3m1s GOES-16, ABI Credits: David J. Hoese martin.raspaud@smhi.se Mitigation: Distributed processing
martin.raspaud@smhi.se Dask distributed
- Client/Server architecture - Works automatically on regular dask code - Works on clusters
martin.raspaud@smhi.se Sentinel 2A, MSI The Pytroll Philosophy
martin.raspaud@smhi.se Pytroll - FOSS ((L)GPL, Github) - Agile development (CI, Code reviews) - Active community (> 100 contributors, Hackathons)
