<strong>This release requires an update to the config file in PEcAn web interface. Without ... more <strong>This release requires an update to the config file in PEcAn web interface. Without this change the web interface will not work. See config.php changes</strong> Following are the major new features of this release. Besides these new features many bugs have been fixed, see milestone 1.4.5 list of issues or complete list of commits. New Models This release now adds support for the PRELES model. PRELES Add some text here about PRELES models, where it comes from etc. New Met Data This release adds support for CRUNCEP CRUNCEP Add some text here about CRUNCEP, where it comes from etc. Web Interface The web interface now allows the user to login using their credentials, event if the flag authorization required is not set. Any run started when logged in will be recorded as belonging to the logged in user. Only users logged in can now delete workflows. Users can now filter the history page, by their own runs, those that are available locally, succeeded and by textual search (in notes, model, site). Notes are no longer an advanced feature, but can now be added during the input section. config.php changes A major change has been done to the web interface configuration files. Please make sure to update your config.php, without these changes your web interface will not allow you to do any runs! The variables $hostlist, $qsuboptions and $qsublist have been merged into a single variable called $hostlist. This list has a list of all hosts that are shown in the host pull down list, as well as any options for qsub. This now also allows you to specify a folder on the remote system to use. All runs will use this remote folder! Please see the example.config.php The following variables are no longer used : $qsuboptions, $qsublist, $ed_soil, $ed_inputs and can be safely removed from you config.php (after fixing the above). Additional changes PEcAn now leverages of Travis CI for continuous integration. To use this in your own branches and forks of PEcAn make sure you have merged in the master branch from [...]
Overview This version supports conversion of met files. The code can now download and convert the... more Overview This version supports conversion of met files. The code can now download and convert the met data to CF standard and to model specific format. The met data will along the way be gap filled. This allows any researchers to pick any Ameriflux site and run the SIPNET model on this site. As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version pecan.xml Changes The met tag can now support a special notation that lets it download and convert the met data. The met tag can have an input="Ameriflux" and output="sipnet" attribute. Currently these are the only supported conversion but more will be added in the next release of PEcAn. web changes When running in the web interface you now have the ability to edit the pecan.xml before the software runs. This allows you to change the number of ensemble runs done, as well as any other changes to the pecan.xml. After the run finishes, you can now download the sipnet clim file used for a particular run. You can now also plot different variables against each other instead of time vs one of hte variables.
Overview This version supports new model types better. BETY now allows to add new model types to ... more Overview This version supports new model types better. BETY now allows to add new model types to the database. You can associate these model types with PFT's. Each model type will have a list of required and optional inputs associated. These inputs will be listed in <run><inputs> tag. As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version pecan.xml Changes Using the new modeltypes we can now check the inputs specified in the pecan.xml file. All inputs should now be listed under the <run><inputs>. The code will try and update your pecan.xml file to this new format. Any missing inputs that are required will quit the run, allowing you to quickly fix your pecan.xml file. For updated information see https://github.com/PecanProject/pecan/wiki/PEcAn-Configuration. Database Changes The database requires updates for the model types table. You will need to update BETY to be able to see this. This requires version 3.2 of BETYdb (https://github.com/PecanProject/bety/releases/tag/betydb_3.2). Code Changes One major change is to the function called get.trait.data(). This function now requires a model type as the second parameter. The model type is used to make sure we get the right PFT. This allows multiple PFTs to have the same name, but be distinguishable by the combination of name and model type.
PEcAn This version of PEcAn now can use BrownDog. BrownDog will help with the conversion of data ... more PEcAn This version of PEcAn now can use BrownDog. BrownDog will help with the conversion of data from input files to model specific files. This version also allows for conversion of the NARR data. PEcAn WEB allow the user to type the name of site to filter the sites ability to use BrownDog if enabled allow to download input files (even if run has failed) ability to run ED without PSS/CSS/Cite As always there are also many bugfixes done
Proceedings of the Practice and Experience on Advanced Research Computing, Jul 22, 2018
Clowder is an open source data management system to support data curation of long tail data and m... more Clowder is an open source data management system to support data curation of long tail data and metadata across multiple research domains and diverse data types. Institutions and labs can install and customize their own instance of the framework on local hardware or on remote cloud computing resources to provide a shared service to distributed communities of researchers. Data can be ingested directly from instruments or manually uploaded by users and then shared with remote collaborators using a web front end. We discuss some of the challenges encountered in designing and developing a system that can be easily adapted to different scientific areas including digital preservation, geoscience, material science, medicine, social science, cultural heritage and the arts. Some of these challenges include support for large amounts of data, horizontal scaling of domain specific preprocessing algorithms, ability to provide new data visualizations in the web browser, a comprehensive Web service API for automatic data ingestion and curation, a suite of social
Permafrost thaw has been observed at several locations across the pan-Arctic in recent decades, y... more Permafrost thaw has been observed at several locations across the pan-Arctic in recent decades, yet the pan-Arctic extent and potential spatial-temporal variations in thaw are poorly constrained. Thawing of ice-rich permafrost can be inferred and quantified with satellite imagery due to the subsequent differential ground subsidence and erosion that in turn affects land surface cover. Information contained within existing and rapidly growing collections of high-resolution satellite imagery (Big Imagery) is here extracted across the Arctic region through a collaboration between software engineers, computer- and earth scientists. More specifically, we are a) developing geospatial data down to sub-meter resolution, and also b) enabling discovery and knowledge-generation through visualization tools. This cyberinfrastructure platform, the Permafrost Discovery Gateway (PDG), is being designed with input from users of the PDG, e.g. primarily the Arctic earth science community but also the general public. The PDG builds upon other NSF supported data management resources (Arctic Data Center and Clowder) and the Fluid Earth Viewer. The Fluid Earth Viewer, which is the first visualization tool implemented into the PDG, was initially created for the public to explore atmospheric and oceanographic visualizations and is here modified to support permafrost geospatial products, and a number of community built analytic tools to identify permafrost artifacts within satellite imagery. The effort also includes workflow optimization of remote sensing code for pan-Arctic sub-meter scale mapping of ice-wedge polygons from optical imagery. We are additionally actively engaging with the user-community to ensure that the PDG becomes useful, both in terms of the type of data contained within the PDG and the design of the visualization tools. The PDG has the potential to fill key Arctic science gaps, such as bridging plot to pan-Arctic scale findings, while also serving as a resource informing decisions regarding the economy, security, and resilience of the Arctic region
In collaboration with the U.S. National Archives and Records Administration Division of Applied R... more In collaboration with the U.S. National Archives and Records Administration Division of Applied Research the Image, Spatial, and Data Analysis group at NCSA has developed a number of tools to aid in the preservation of digital records. In this paper we present these tools in the context of preserving 3D digital files. Tools such as the Conversion Software Registry, Software Servers, Polyglot, 3D Utilities, and Versus provide users with a scalable means of discovering and carrying out large file migration tasks in a manner that can take into account and quantify the unavoidable information loss that occurs when moving from one format to another. We present each of these tools and describe how they can be used.
ABSTRACT Recent advances in sensor technology are facilitating the deployment of real-time sensor... more ABSTRACT Recent advances in sensor technology are facilitating the deployment of real-time sensor networks into the environment that can produce measurements at high spatial and/or temporal resolutions. At the same time, advances in cyberinfrastructure technologies are allowing researchers to use large quantities of heterogeneous sensor data for improving understanding of large-scale environmental processes, as well as in monitoring and modeling the environments in which the sensors are deployed. In their endeavors, researchers often seek the &quot;raw&quot; sensor data in order to ensure that their analyses will be free from bias, but usually such data have already been processed in ways that may not meet the needs of the researchers. Furthermore, acquiring sensor data suitable for a particular application (e.g. at an appropriate spatiotemporal resolution) may require the re-purposing of sensor data that might be beyond the scope of the original sensor design and deployment. Similar to the recent revolution of computer server virtualization, it is now necessary to consider virtualization of sensors and sensor networks so that existing sensor networks and their measurements can be easily re-purposed and used in new ways. This study explores the creation of virtual sensors and their potential impacts on national environmental observatories such as the WATERS Network including how virtual sensors can facilitate broad community collaborations. Additionally, the process of validating virtual sensors through community participation and review will be discussed. To demonstrate how virtual sensors will benefit the real-time modeling of environmental systems, a case study will be presented that addresses the real-time prediction of combined sewer overflows in the Chicago Metropolitan area. Combined sewer overflow (CSO) forecasts rely heavily on measurements regarding the quantity and spatial distribution of rainfall. To improve the accuracy of CSO forecasts, data from virtual rainfall sensors are employed. The virtual sensors are created by performing three transformations on real-time reflectivity data measured by the Next Generation Weather Radar (NEXRAD) system. The first transformation registers the NEXRAD data grid with geographic coordinate system used by the CSO model. Second, the reflectivity data is converted to rainfall rates using a radar-rainfall relationship, and finally, the data are aggregated in time to provide rainfall accumulation estimates that correspond with the temporal discretization of the CSO model.
Automated search of handwritten content is a highly interesting and applicative subject, especial... more Automated search of handwritten content is a highly interesting and applicative subject, especially important today due to the public availability of large digitized document collections. We describe our efforts with the National Archives (NARA) to provide searchable access to the 1940 Census data and discuss the HPC resources needed to implement the suggested framework. Instead of trying to recognize the handwritten text, a still very difficult task, we use a content based image retrieval technique known as Word Spotting. Through this paradigm, the system is queried by the use of handwritten text images instead of ASCII text and ranked groups of similar looking images are presented to the user. A significant amount of computing power is needed to accomplish the pre-processing of the data so to make this search capability available on an archive. The required preprocessing steps and the open source framework developed are discussed focusing specifically on HPC considerations that are relevant when preparing to provide searchable access to sizeable collections, such as the US Census. Having processed the state of North Carolina from the 1930 Census using 98,000 SUs we estimate the processing of the entire country for 1940 could require up to 2.5 million SUs. The proposed framework can be used to provide an alternative to costly manual transcriptions for a variety of digitized paper archives.
This version contains all the model developments for ED-2.2 and described in Longo et al. (2019) ... more This version contains all the model developments for ED-2.2 and described in Longo et al. (2019) (https://dx.doi.org/10.5194/gmd-2019-45).
PEcAn Besides bug fixes, this new version of the VM supports fetching the MET data from NARR and ... more PEcAn Besides bug fixes, this new version of the VM supports fetching the MET data from NARR and converting it to model specific inputs. As well as a new model LINKAGES (https://github.com/araiho/Linkages). BETY Bety is now installed in /home/carya/bety and does not require root to upgrade. Users can simply call the following commands as non root to update the server. git pull bundle install rake db:migrate
<strong>This release requires an update to the config file in PEcAn web interface. Without ... more <strong>This release requires an update to the config file in PEcAn web interface. Without this change the web interface will not work. See config.php changes</strong> Following are the major new features of this release. Besides these new features many bugs have been fixed, see milestone 1.4.5 list of issues or complete list of commits. New Models This release now adds support for the PRELES model. PRELES Add some text here about PRELES models, where it comes from etc. New Met Data This release adds support for CRUNCEP CRUNCEP Add some text here about CRUNCEP, where it comes from etc. Web Interface The web interface now allows the user to login using their credentials, event if the flag authorization required is not set. Any run started when logged in will be recorded as belonging to the logged in user. Only users logged in can now delete workflows. Users can now filter the history page, by their own runs, those that are available locally, succeeded and by textual search (in notes, model, site). Notes are no longer an advanced feature, but can now be added during the input section. config.php changes A major change has been done to the web interface configuration files. Please make sure to update your config.php, without these changes your web interface will not allow you to do any runs! The variables $hostlist, $qsuboptions and $qsublist have been merged into a single variable called $hostlist. This list has a list of all hosts that are shown in the host pull down list, as well as any options for qsub. This now also allows you to specify a folder on the remote system to use. All runs will use this remote folder! Please see the example.config.php The following variables are no longer used : $qsuboptions, $qsublist, $ed_soil, $ed_inputs and can be safely removed from you config.php (after fixing the above). Additional changes PEcAn now leverages of Travis CI for continuous integration. To use this in your own branches and forks of PEcAn make sure you have merged in the master branch from [...]
Overview This version supports conversion of met files. The code can now download and convert the... more Overview This version supports conversion of met files. The code can now download and convert the met data to CF standard and to model specific format. The met data will along the way be gap filled. This allows any researchers to pick any Ameriflux site and run the SIPNET model on this site. As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version pecan.xml Changes The met tag can now support a special notation that lets it download and convert the met data. The met tag can have an input="Ameriflux" and output="sipnet" attribute. Currently these are the only supported conversion but more will be added in the next release of PEcAn. web changes When running in the web interface you now have the ability to edit the pecan.xml before the software runs. This allows you to change the number of ensemble runs done, as well as any other changes to the pecan.xml. After the run finishes, you can now download the sipnet clim file used for a particular run. You can now also plot different variables against each other instead of time vs one of hte variables.
Overview This version supports new model types better. BETY now allows to add new model types to ... more Overview This version supports new model types better. BETY now allows to add new model types to the database. You can associate these model types with PFT's. Each model type will have a list of required and optional inputs associated. These inputs will be listed in <run><inputs> tag. As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version pecan.xml Changes Using the new modeltypes we can now check the inputs specified in the pecan.xml file. All inputs should now be listed under the <run><inputs>. The code will try and update your pecan.xml file to this new format. Any missing inputs that are required will quit the run, allowing you to quickly fix your pecan.xml file. For updated information see https://github.com/PecanProject/pecan/wiki/PEcAn-Configuration. Database Changes The database requires updates for the model types table. You will need to update BETY to be able to see this. This requires version 3.2 of BETYdb (https://github.com/PecanProject/bety/releases/tag/betydb_3.2). Code Changes One major change is to the function called get.trait.data(). This function now requires a model type as the second parameter. The model type is used to make sure we get the right PFT. This allows multiple PFTs to have the same name, but be distinguishable by the combination of name and model type.
PEcAn This version of PEcAn now can use BrownDog. BrownDog will help with the conversion of data ... more PEcAn This version of PEcAn now can use BrownDog. BrownDog will help with the conversion of data from input files to model specific files. This version also allows for conversion of the NARR data. PEcAn WEB allow the user to type the name of site to filter the sites ability to use BrownDog if enabled allow to download input files (even if run has failed) ability to run ED without PSS/CSS/Cite As always there are also many bugfixes done
Proceedings of the Practice and Experience on Advanced Research Computing, Jul 22, 2018
Clowder is an open source data management system to support data curation of long tail data and m... more Clowder is an open source data management system to support data curation of long tail data and metadata across multiple research domains and diverse data types. Institutions and labs can install and customize their own instance of the framework on local hardware or on remote cloud computing resources to provide a shared service to distributed communities of researchers. Data can be ingested directly from instruments or manually uploaded by users and then shared with remote collaborators using a web front end. We discuss some of the challenges encountered in designing and developing a system that can be easily adapted to different scientific areas including digital preservation, geoscience, material science, medicine, social science, cultural heritage and the arts. Some of these challenges include support for large amounts of data, horizontal scaling of domain specific preprocessing algorithms, ability to provide new data visualizations in the web browser, a comprehensive Web service API for automatic data ingestion and curation, a suite of social
Permafrost thaw has been observed at several locations across the pan-Arctic in recent decades, y... more Permafrost thaw has been observed at several locations across the pan-Arctic in recent decades, yet the pan-Arctic extent and potential spatial-temporal variations in thaw are poorly constrained. Thawing of ice-rich permafrost can be inferred and quantified with satellite imagery due to the subsequent differential ground subsidence and erosion that in turn affects land surface cover. Information contained within existing and rapidly growing collections of high-resolution satellite imagery (Big Imagery) is here extracted across the Arctic region through a collaboration between software engineers, computer- and earth scientists. More specifically, we are a) developing geospatial data down to sub-meter resolution, and also b) enabling discovery and knowledge-generation through visualization tools. This cyberinfrastructure platform, the Permafrost Discovery Gateway (PDG), is being designed with input from users of the PDG, e.g. primarily the Arctic earth science community but also the general public. The PDG builds upon other NSF supported data management resources (Arctic Data Center and Clowder) and the Fluid Earth Viewer. The Fluid Earth Viewer, which is the first visualization tool implemented into the PDG, was initially created for the public to explore atmospheric and oceanographic visualizations and is here modified to support permafrost geospatial products, and a number of community built analytic tools to identify permafrost artifacts within satellite imagery. The effort also includes workflow optimization of remote sensing code for pan-Arctic sub-meter scale mapping of ice-wedge polygons from optical imagery. We are additionally actively engaging with the user-community to ensure that the PDG becomes useful, both in terms of the type of data contained within the PDG and the design of the visualization tools. The PDG has the potential to fill key Arctic science gaps, such as bridging plot to pan-Arctic scale findings, while also serving as a resource informing decisions regarding the economy, security, and resilience of the Arctic region
In collaboration with the U.S. National Archives and Records Administration Division of Applied R... more In collaboration with the U.S. National Archives and Records Administration Division of Applied Research the Image, Spatial, and Data Analysis group at NCSA has developed a number of tools to aid in the preservation of digital records. In this paper we present these tools in the context of preserving 3D digital files. Tools such as the Conversion Software Registry, Software Servers, Polyglot, 3D Utilities, and Versus provide users with a scalable means of discovering and carrying out large file migration tasks in a manner that can take into account and quantify the unavoidable information loss that occurs when moving from one format to another. We present each of these tools and describe how they can be used.
ABSTRACT Recent advances in sensor technology are facilitating the deployment of real-time sensor... more ABSTRACT Recent advances in sensor technology are facilitating the deployment of real-time sensor networks into the environment that can produce measurements at high spatial and/or temporal resolutions. At the same time, advances in cyberinfrastructure technologies are allowing researchers to use large quantities of heterogeneous sensor data for improving understanding of large-scale environmental processes, as well as in monitoring and modeling the environments in which the sensors are deployed. In their endeavors, researchers often seek the &quot;raw&quot; sensor data in order to ensure that their analyses will be free from bias, but usually such data have already been processed in ways that may not meet the needs of the researchers. Furthermore, acquiring sensor data suitable for a particular application (e.g. at an appropriate spatiotemporal resolution) may require the re-purposing of sensor data that might be beyond the scope of the original sensor design and deployment. Similar to the recent revolution of computer server virtualization, it is now necessary to consider virtualization of sensors and sensor networks so that existing sensor networks and their measurements can be easily re-purposed and used in new ways. This study explores the creation of virtual sensors and their potential impacts on national environmental observatories such as the WATERS Network including how virtual sensors can facilitate broad community collaborations. Additionally, the process of validating virtual sensors through community participation and review will be discussed. To demonstrate how virtual sensors will benefit the real-time modeling of environmental systems, a case study will be presented that addresses the real-time prediction of combined sewer overflows in the Chicago Metropolitan area. Combined sewer overflow (CSO) forecasts rely heavily on measurements regarding the quantity and spatial distribution of rainfall. To improve the accuracy of CSO forecasts, data from virtual rainfall sensors are employed. The virtual sensors are created by performing three transformations on real-time reflectivity data measured by the Next Generation Weather Radar (NEXRAD) system. The first transformation registers the NEXRAD data grid with geographic coordinate system used by the CSO model. Second, the reflectivity data is converted to rainfall rates using a radar-rainfall relationship, and finally, the data are aggregated in time to provide rainfall accumulation estimates that correspond with the temporal discretization of the CSO model.
Automated search of handwritten content is a highly interesting and applicative subject, especial... more Automated search of handwritten content is a highly interesting and applicative subject, especially important today due to the public availability of large digitized document collections. We describe our efforts with the National Archives (NARA) to provide searchable access to the 1940 Census data and discuss the HPC resources needed to implement the suggested framework. Instead of trying to recognize the handwritten text, a still very difficult task, we use a content based image retrieval technique known as Word Spotting. Through this paradigm, the system is queried by the use of handwritten text images instead of ASCII text and ranked groups of similar looking images are presented to the user. A significant amount of computing power is needed to accomplish the pre-processing of the data so to make this search capability available on an archive. The required preprocessing steps and the open source framework developed are discussed focusing specifically on HPC considerations that are relevant when preparing to provide searchable access to sizeable collections, such as the US Census. Having processed the state of North Carolina from the 1930 Census using 98,000 SUs we estimate the processing of the entire country for 1940 could require up to 2.5 million SUs. The proposed framework can be used to provide an alternative to costly manual transcriptions for a variety of digitized paper archives.
This version contains all the model developments for ED-2.2 and described in Longo et al. (2019) ... more This version contains all the model developments for ED-2.2 and described in Longo et al. (2019) (https://dx.doi.org/10.5194/gmd-2019-45).
PEcAn Besides bug fixes, this new version of the VM supports fetching the MET data from NARR and ... more PEcAn Besides bug fixes, this new version of the VM supports fetching the MET data from NARR and converting it to model specific inputs. As well as a new model LINKAGES (https://github.com/araiho/Linkages). BETY Bety is now installed in /home/carya/bety and does not require root to upgrade. Users can simply call the following commands as non root to update the server. git pull bundle install rake db:migrate
Uploads
Papers by Rob Kooper