Horseweed Online Tool Documentation

The online tool for the Horseweed pollen/seed dispersion is currently hosted from a server located at Room 644 in Dr. Junming Wang's office, Illinois State Water Survey, University of Illinois at Urbana and Champaign. It is an online tool for calculating the dispersion (deposition and concentration), and outcrossing of horseweed pollen grains and seeds from a user defined location and area. HYSPLIT software, version 4.9, from NOAA is used for calculation based on the meteorological data stored in the server. For pollen, the HYSPLIT program is run multiple times (2 hours each time, pollen alive<2 hours) for a user specified duration, and the average concentration and total deposition of the pollen grains is calculated. For seed, the HYSPLIT just run once for the duration.The output data and the Google earth compatible graphical output web links are sent to the users’ email.

The overall implementation of the tool can be described as follows:

  • User submits the pollen/seed calculation request through the online tool website.
  • Each request is stored in a queue, and the online tool is ready to receive other requests. All the requests are also stored in another table for future reference.
  • The queued requests are processed on a First-In-First-Out (FIFO) basis.
  • After the successful processing of a request, it is deleted from the queue, and an email is sent to the requester with the detailed results. The results include the links of output data file (txt), graphics file (png, pdf, kmz).
  • After sending the email, another request in the queue is processed. This process repeats until the queue is empty.

The online tool can be divided into 2 major functional components:

  • Web interface
  • Background php process

The following flowcharts explain the functional flow of the components.

A. Description of Files

The description of the different files used in the system is as follows:

  1. Index.html

This is the homepage of the online tool. A user can either sign in using an existing user id and password or register for a new account. This page directs to loginuser.php after the user clicks on Login button. If sign up is clicked, this page directs to register.html.

  1. Mysql.php

It contains the information about the database in which the existing user information is stored (“$db_name”). It also contains the names of the tablesused to store the parameters submitted by the clients(“$tbl_name1”).The setup is automatically by the daemon. If the daemon finds the table doesn’t exist, it will automatically create one. If you want to change the table structure, you can modify the “daemon.php” file.

$sql="CREATE TABLE IF NOT EXISTS $tbl_name1 (

email varchar(255) DEFAULT NULL,

latitude float DEFAULT NULL,

longitude float DEFAULT NULL,

month tinyint(4) DEFAULT NULL,

day tinyint(4) DEFAULT NULL,

hour tinyint(4) DEFAULT NULL,

myusername varchar(255) DEFAULT NULL,

year_avg tinyint(4) DEFAULT NULL,

weeks tinyint(4) DEFAULT NULL,

datatype varchar(50) DEFAULT NULL,

id int(10) unsigned NOT NULL AUTO_INCREMENT,

PRIMARY KEY (id)

) ENGINE=InnoDB DEFAULT CHARSET=latin1";

  1. Loginuser.php

It uses mysql.php to get user information from the database. If username and password are matched, then the page directs to choose.html, else to index.html.

  1. Register.html

This page provides a registration form for the new users. The information provided in this form is stored in the database for storing the login information. After successful registration, this page directs the user to choose.html.

  1. Choose.html

Using this page, a user can choose between pollen and seed dispersion. This page directs the user to pollen.php/seed.php based on the selection.

  1. Pollen.php

Using this page, user can submit the pollen dispersion calculation request. User provides the simulation parameters including the lat-long coordinates, start and end time, number of years for averaging the results, density of plants, total area and the type of meteorological data (NAM/NARR). It must be ensured that the simulation start time is before the end time. If inserted otherwise, no calculations will be done and the users won’t get the output. Users should also note that the meteorological data is only available from 1/1/1997.

The source location can be selected either from the Google-maps API shown in the form or it can be inserted manually in the text fields. Note thatthis tool works only for the US region because the server has meteorological data for the CONUS region only. After the user specifies the simulation parameters and submits the request, the form is handled by pollendriver.php which puts the user’s request in the queue. User will be redirected to choose.html, and the request will be processed on a First-In-First-Out basis.

  1. Seed.php

This page is similar to the pollen.php; the only difference is that this is used for submitting calculation requests for seed dispersion. After the user specifies the simulation parameters and submits the request, the form is handled by seeddriver.php which puts the user’s request in the queue. User will be redirected to choose.html, and the request will be processed on a First-In-First-Out basis.

  1. Pollendriver.php

The main task of this page is to receive the user requests from pollen.php and queue them into the request queue in database. It also stores the request details in a database table for future reference. Since the same database is being used for storing pollen and seed calculation requests, a flag is used todistinguish between the 2 request types: seed and pollen. For pollen calculation requests, the flag field in the database is set to 1. The database details are provided in file mysql.php.

  1. seeddriver.php

seeddriver.php receives the user requests from seed.php and queues them into the request queue in database. It also stores the request details in a database table for future reference. Since the same database is being used for storing pollen and seed calculation requests, a flag is used to distinguish between the 2 request types: seed and pollen. For seed calculation requests, the flag field in the database is set to 2. More details of the database are provided in mysql.php.

  1. Database fields

Following are the fields in database to store the query request parameters:

Email  for storing email of request submitter. Needed to send the output files.

Latitude  Source latitude for pollen/seed

Longitude  Source longitude for pollen/seed

Month  Calculation start month

Day  Calculation start day

Hour  Calculation start hour

emonth  End month for calculations

eday  End day for calculations

ehour  End hour for calculations

myusername  Username

year_avg  Number of years for which calculations are to be averaged (starts from 2013 and goes backwards until 1997)

datatype  Meteorological data to use: NARR or NAM

flag  Used to distinguish the type of calculation request (1 for pollen, 2 for seed)

weeks  Not used in current version

  1. daemon.php

This is the heart of the system. It runs as a background process, and can be customized and configured in the task scheduler. It retrieves the user request parameters from the database and begins the execution of different programs using those parameters. The following programs are executed from this page for pollen/seed requests:

Pollen requests:

  • sendemail.jar
  • outcrossinglegend.jar
  • generateoutcrossingkml.jar
  • depositionlegend.jar
  • generatedepositionkml.jar
  • concentrationlegend.jar
  • generateconcentrationkml.jar,
  • zipthefiles.jar

Seed requests:

  • seeddispersion.jar
  • depositionlegend.jar
  • generatedepositionkml.jar
  • concentrationlegend.jar
  • generateconcentrationkml.jar,
  • zipthefiles.jar

Finally, daemon.php sends email to the user with links to all the output files.

  1. Logout.php

It is responsible for ending the user’s active session.

  1. Tools.php

It is responsible for finding the current working directory.

  1. Viewresults.php

This page is used to show all the previous output files corresponding to user’s requests. The results can be viewed only if the user is logged in.

  1. Hycs_std.exe

This is an exe file for performing the calculations related to the deposition and concentration of pollens and seeds. It uses the meteorological data from the server and the other parameters as specified in the CONTROL file. If there is no CONTROL file in the same folder, it will ask the user to enter all those parameters one by one. For our system, the CONTROL file has to be updated before each run otherwise the output will always be the same. This update is done by the java program sendemail.jar. hycs_std.exe generates output file “cdump” which contains the concentration and deposition data in binary format.

  1. Con2asc.exe

This program is responsible for generating the readable output data in ASCII format, corresponding to the data in the binary file“cdump”. After running this program, an ASCII file with filename in the format day_hour.txt is created. The exact name of the output file is saved in a new file named CON2ASC.OUT. So, one should read the file “CON2ASC.OUT” in order to get the name of the output file generated by con2asc.exe.

  1. Ref_control

This file is used to generate the CONTROL file that is required for the execution of hycs_std.exe. The location of the meteorological data should be provided in the 8thline of this file.

  1. Pollenstrengthdata

This file contains the source strength data for pollen at different times (each line has month/day/hour information and corresponding pollen source strengths). These values are based on the experimental results and will be used for pollen dispersion calculations. Sendemail.jar uses this file through a program generatepollensetupfile.jar and stores the required source strengths for calculation in a file named EMITIMES.

The emission rate required by the Hysplit is in mass/hour. In our case, it would be pollen #/hour or seed#/hour.

Lets consider the following:

- source strength in "pollenstrengthdata" file = p (lets say unit is x)

- area (input from user) = A (unit is acre)

- density (input from user) = d (unit is plants/acre)

The java program uses p*A*d as the emission rate which has units x*acre*plants/acre = x*plants

To make the unit of emission rate = x*plants equal to pollen(or seed)/hour, we need to have x = pollen(or seed)/plant/hour i.e., the unit of values to be stored in files should be average pollen (or seed) released by a single plant per hour.

The values should be updated in only 2 files. "pollenstrengthdata" for pollen source strength and "strengthdata" for seed source strength. No need to change any other files; other required files (such as EMITIMES) will be automatically updated by the java program.

/////////////

Hi Dr. Wang,

As I was debugging, I found that the source strength values were not entered for 2/29 (leap years have this date) which resulted in calculations with incorrect source strengths. Initially, when I wrote the program, I assumed that source strength file contains data for 2/29 as well, and skips that data for non-leap years.

The program is set to skip certain number of lines from the source strength file, and read only those lines corresponding to the input dates. But since there was no data for 2/29, wrong lines were read for obtaining the source strength data for the input date for 2012 (leap year). I entered it now, and the calculation yield correct results.

In future, if you need to change the source strength files, please keep data for 2/29 as well. The program uses 2/29 data for leap years, and automatically skips those lines for non-leap years.

Now, I am working on fixing the issue with large kmz output file by thresholding and removing Placemarks for smaller values. After I am done with this, I will investigate the issue with long duration pollen requests.

Regards,

Sudeep

////////////////

Dear Dr. Wang,

After investigating the problem, I found the following two major issues:

1. The name of file for storing seed source strength is wrong. It should be "strengthdata" instead of "Strengthdata"

2. The data format in both files is wrong. The java program requires data to be in the same format as that in the attached files. Note that the three columns must be separated by a single space character (I found some columns were separated by a tab which is one of the causes of the problem). Also note that the date in the first column should be formatted as MM/DD but I found it otherwise in the updated file. Similarly, the hour in the second column should be in HH format.

E.g.,

Correct Incorect

01/01 00 0.1230 1/1 0 0.1230

01/02 01 0.3456 1/2 1 0.3456

Right hand side is incorrectly formatted because first column is not formatted as MM/DD, second column isnot formatted as HH, and second and third columns are not separated by exactly one space character.

Fixing these issues should resolve the problem. Please let me know if you have any other questions.

Regards,

Sudeep

  1. Strengthdata

This file contains the source strength data for seed at different times (each line has month/day/hour information and corresponding seed source strengths, seeds/plant/hour). These values are based on the experimental results and will be used for seed dispersion calculations. seeddispersion.jar uses this file through a program generatesetupfile.jar and stores the required source strengths for calculation in a file named EMITIMES.

WHEN

On 06/11/2015, use the new source strength data generated based on 2013 IL seed data to replace the old strength data. Rename the file “strengthdata” into “strengthdata_backup_20150611”, and add the new file “strengthdata”.

WHAT

The new source strength data was generated based on 2013 IL seed data, the original file is

According to the original data, the seeds are released between 8:00 and 18:00 from 08/29/2013 to 10/11/2013. So that will be totally 44*11= 484 source strength values. Because the data in the original file is incomplete, so we need to generate the missing data.

HOW

Then how to generate the missing data? For missing dates, use the closest previous and later day’s data to do linear interpolation. For missing hours, use the current day’s data to do interpolation. The following figure(from the paper “Horseweed (Conyza canadensis) gene flow through atmospheric seed emission, dispersion, and deposition”) shows the diurnal variation of normalized seed source strength (normalized by daily maximum, the diamonds are means and bars are standard deviations). The missing hours’ values are generated by scaling the other hours’ values according to the curve in this figure.

  1. EMITIMES

When one needs to run Hysplit with multiple source strengths, the source strength values cannot be specified in the CONTROL file. Rather, a different file containing the source strength should be used. In our online tool, file “EMITIMES” contains such varying source strength for each Hysplit run. Since multiple Hysplit runs may be necessary for each calculation request, this file is updated with corresponding source strengths for each simulation. Java programsgeneratepollensetupfile.jar and generatesetupfile.jar usethe required source strength information from “pollenstrengthdata” and “strengthdata” files to generate EMITIMES.

  1. Setupfile.cfg

Sendemail.jar uses this file as a reference for generating the SETUP.CFG file. SETUP.CFG is needed for performing Hysplit run using multiple source strengths for pollen dispersion calculations. When Hysplit needs to be run with multiple source strengths, the pollen source strength values are required to be stored in a separate file, but not in the CONTROL file. This SETUP.CFG contains all the information about the name and attributes of such file.

  1. Seedsetupfile.cfg

seeddispersion.jar uses this file as a reference for generating the SETUP.CFG file. SETUP.CFG is required for performing Hysplit run using different seed source strengths. When Hysplit needs to be run with multiple source strengths, the seed source strength values are required to be stored in a separate file, but not in the CONTROL file. This SETUP.CFG contains all the information about the name and attributes of such file.

  1. Dco.exe

This program is used to generate png and pdf contours based on the final results for seed/pollen dispersion.

The source file is called “depconoutcontour.m” under the online tool folder “horseweed”

The documentation of the source file is “Horseweed_Generate_PDF_and_PNG_contours(Revised by Xiufen Cui on 05_01_2014).docx” under the the RSETserver “Phpapp/docs” folder

  1. Reference.kml

This is the file which contains the initial codes for generating Google earth compatible kml file. The kml files are later merged with the legend file (png) to generate a kmz file. Program generatedepositionkml.jar and generateconcentrationkml.jar use this file as a reference to generate the corresponding kml files.

  1. Depositionlegend.png

This image is generated by the java program “depositionlegend.jar”, and serves as a legend in google earth compatible “kmz” file showing the deposition results.

  1. Concentrationlegend.png

This image is generated by the java program “concentrationlegend.jar”, and serves as a legend in google earth compatible “kmz” file showing the concentration results.

  1. outcrossinglegend.png

This image is generated by the java program “outcrossinglegend.jar”, and serves as a legend in google earth compatible “kmz” file showing the outcrossing results.