PECAS Model Development: MapIt - Installation and Configuration
System Documentation Technical Note 10.18.2012/ARM/GFM/JEA/JDPM
File: MapIt documentation_PostGIS2.0.docx
PECAS Model Development
MapIt Installation and Configuration
Tech Memo 2 – WORKING DRAFT
Calgary, Alberta
October 2012
1. What is MapIt?
MapIt is a tool developed by HBA Specto that provides support to a group of tasks involved in every day PECAS modelling work. MapIt allows users to visualize PECAS model results in the form of maps at the LUZ (land use zones) and TAZ (transportation analysis zones). MapIt runs on a webserver and interacts with a database, so that multiple users can simultaneously use it to investigate model run results.
Users define their own query in terms of: scenarios (one or two), which file (ActivityLocations, ExchangeResults, others), which attributes (prices, quantities, utilities, etc.), start year and end year. MapIt then shows a list of options depending on the file selected. For example, MapIt shows activity groups or commodity groups based on the file previously chosen.
Two features make MapIt essential in PECAS modelling work:
· It compares the results from 2 different scenarios (for instance W00 and C01).
· It combines values from different years.
Although MapIt generates rudimentary maps, its main purpose is to combine PECAS outputs into layers in a geodatabase and in shape files. Users can then use the GIS and visualization tools they are comfortable with to view PECAS output.
Figure 1. MapIt interface
MapIt is used by HBA staff to build numeric comparisons between scenarios and years and store the instructions for the comparison in the database. Users can download these comparisons as GIS shape files, or view them from the GIS database. These comparisons are used in calibration and development (plan an implement future changes or improvements in the models) and in policy evaluation and analysis.
The purpose of this brief document is to describe the MapIt installation and configuration on our client’s computer. The description includes how to get the source code, what are the prerequisite software packagess and their corresponding installation, how to load shape files into the database, how to configure MapIt setting files, how to create the map files for MapServer, how to configure Apache and how to run MapIt. Figure 1 shows MapIt interface and some user options.
2. Getting the source code:
1. Install the Tortoise SVN software
2. Create a folder where you want to install the program i.e. “C:\djangoProjects”
3. With the right bottom click on the folder “djangoProjects” and check out the code from SVN using
Mac Users: in a terminal windows, change to your development directory and type
svn co http://svn.hbaspecto.com/svn/pecas/PECASSoftware/trunk/web_output
You will need to write for the URL of repository: http://svn.hbaspecto.com/svn/pecas/PECASSoftware/trunk/web_output
And for the checkout directory:
C:\djangoProjects\web_output
Figure 2. SVN Code Checkout
You will see the Tortoise copying the code to the specified folder like in this window:
Figure 3. SVN windows copying files into specified folder
And you will end up with the files and programs that you need in the following path: C:/djangoProjects/web_output which from now will be called the “PROJECT PATH”
Figure 4. final content of the specified folder
3. Prerequisites Software:
- PostgreSQL 8.4 (pgAdmin III) or higher with PostGIS enabled.
- Python-2.66 32bit (Python 2.7 and later versions work, but Python 3.x is not compatible)
- psycopg2 (e.g. for Windows: Psycopg2-2.0.14.win32-py2.6-pg8.4.3 )
- Django 1.2.1
- MapServer 2.3 (e.g. for Windows ms4w_2.3.1)
- Google Visualization Python (gviz_api)
Note: You can get the all the software you need (except PostgreSQL) from: C:\djangoProject\Web_output\SoftwareToInstallessentials.rar
Mac Users: You can use MacPorts to install everything. Install the prerequisites for mac ports as instructed on http://www.macports.org/ (XCode and its command line utilities.) With macports installed you can install postgresql, except the port for postgresql does not include the required tablefunc module. To fix this, you should first type:
sudo port edit --editor "open -e" postgresql91
And find the line that begins with “set contribs”. In that line there is a list of modules, add the tablefunc module to the list. Save the file, then you can type
sudo port install postgresql91
to install postgres, and
sudo port install postgresql91-server
sudo port load postgresql91-server
to start it running.
In this guide, we will assume that PostgreSQL is already installed, then going down the list the next program to install is python.
3.1 Python & Psycopg
Installing python and psycopg is straightforward. You only need to double click on the installer (e.g python-2.7.3.msi and psycopg2-2.4.5.win32-py2.7-pg9.1.3-release.exe) in the SoftwareToInstall-essentials folder.
Mac Users: with macports installed, type:
sudo port install python26
and
sudo port install py26-psycopg2
Note there are python 27 versions of these, but as of this writing there is no python 27 integration with apache and mod_wsgi, so we are installing the python 26 versions.
3.2 Django 1.2
From this point many programs are going to be installed using the CMD (command prompt) and it is recommended to open CMD as an Administrator. One way to do this in windows vista or in windows 7 is going to the search programs and files bar, type cmd and instead of pressing Enter, press CTR+SHIFT+ENTER. That will launch the user account control option. Click yes and that will take you to the CMD windows as an Administrator.
Once in the CMD, to install Django, change to C:\djangoProjects\web_output\SoftwareToInstall-essentials\Django-1.2.1
Run the following command:
C:\djangoProjects\web_output\SoftwareToInstall-essentials\Django-1.2.1> C:\Python27\python setup.py install
Mac Users: sudo port install py26-django
As you can see it is needed to specify the path of the program we want to install, which is django. But, also the path of python and the command “python setup.py install”.
WARNING: Make sure that the version of python used is the same as you installed earlier in 2.1. For more help on Django installation, please see: http://docs.djangoproject.com/en/1.2/intro/tutorial01/#intro-tutorial01
Mac Users: to check your version of python, type
python --version
3.3 MapServer
To install MapServer, run the ms4w setup program in C:\djangoProject\web_output\SoftwareToInstall-essentials\
(e.g. ms4w-3.0.4-setup.exe) and install ms4w to the C:\ drive. Then, run the following command:
C:\ms4w> apache-install.bat
Mac Users: sudo port install mapserver +proj +postgis
For more help, please see: http://maptools.org/ms4w/index.phtml?page=README_INSTALL.html#b-extracting-ms4w-for-the-first-time
WARNING: if there is any trouble please check that the comments (signed as #) inside of the file httpd_weboutput.conf in the path C:\ms4w\httpd.d start at the beginning of the line
3.4 Gvis_api
This program is used to generate the bar charts for an individual zone. To install it, run the following command:
C:\djangoProject\web_output\google-visualization-python> C:\Python32\ python setup.py install
Mac Users: cd google-visualization-python; sudo python setup.py install
3.5 OWSlib
This program is used to generate the legend of the maps in Mapit. To install it, run the following command:
C:\djangoProjects\web_output\SoftwareToInstall-essentials\OWSLib> C:\Python27\ python setup.py install
This requires installing setuptools for the python version that you have installed in your computer. The file to install is called “ez_setup.py” and you can take it from the SoftwareToInstall-essentials folder located in C:\djangoProject\web_output. Open the program in python and run it. For more complete instructions go to : https://pypi.python.org/pypi/setuptools/1.1.5#windows
4. Setting up the database
You may already have a database setup, which you used for the PECAS Space Development (SD) module. Mapit uses two schemas in the database, the “output” schema and the “analysis” schema, which can coexist in the same database as the SD schemas. However the SD database may not have some of the functionality (PostGIS, crosstab functions, etc.) so it is better to create a new database.
To set up a new database for MapIt, you need to follow the steps below:
1- Open the program pgAdmin III. It will probably ask for the user password, which usually is: usrPostgres
2- In the Object browser pane on the left, right click on the Databases icon (under the PECAS_Demo server), and choose “New Database …” option.
3- Fill in the New Database form as in the figures below. Click OK.
Properties Tab:
Definition Tab:
Figure 5. New Database forms
4- Now that you have created (or chosen) your database for MapIt, you need to right click on the newly created Database (in this case MapIt), and choose “Restore…”
Above to restore only structure
5- On the ”File Options” tab in the Restore Database form, choose C:\djangoProjects\web_output\SoftwareToInstall-essentials\MapIt_DB_template.backup
6- On the “Restore Options #1” tab, make sure no options are checked.
7- On the “Restore Options #2” tab, make sure the “Verbose messages” box is checked.
Figure 6. Restore in mapit database
Note needed on restore errors. There may be many duplicate functions, keys, etc. if restoring over an existing database.
When the Restore finishes, and after Refreshing (View>Refresh), you should be able to see three (3) schemas, namely:
· output: used to store the models’ run
· analysis: used to store MapIt views and crosstabbed data
· public: used to store the zone shapefiles (TAZ and LUZ) and some system tables
Note: if you previously installed the “PECASDemo” database, there will be 2 additional schemas shown: topology and w00.
NOTE: addgeometrycolumn function is changed in PostGIS 2.0, so addgeometrycolumn functions without “boolean” arguments must be dropped (by dropping the postgis2.0 extension if needed using the “Drop cascaded...” command on the postgis extension in the restored database) so shapefiles can be imported (in the next step) if upgrading from a postgreSQL version <9.0 and PostGIS version <2.0.
Refer to http://www.bostongis.com/blog/index.php?/archives/187-How-to-upgrade-your-database-to-PostGIS-2.0-let-me-count-the-ways.html
Running on PostGIS 2.0.0 PostgreSQL 9.0 or lower and moving to 9.1 with extensions
Yes, This also applys to folks who were running 32-bit PostgreSQL on Windows and decidedwhat the hey lets see what this PostGIS on 64-bit PostgreSQL windows is all about.
1. Backup your old database
2. Make sure the PostGIS 2.0.0 binaries are installed on your new PostgreSQL service
3. Restore your old db on new server service
4. run:
postgis_upgrade_20_minor.sql
-- this one only if you had installed with raster support before
rtpostgis_upgrade_20_minor.sql
-- otherwise you'll need to install
rtpostgis.sql
-- run this 00
topology_upgrade_20_minor.sql (if you had topology support this may be in a topology sub folder)
5. Run sql statement
CREATE EXTENSION postgis FROM unpackaged VERSION '2.0.0';
-- only if you have topology support
CREATE EXTENSION postgis_topology FROM unpackaged VERSION '2.0.0';
-- (if you don't have topology and want it)
CREATE EXTENSION postgis_topology;
The query files (*.sql) are located in:
C:\Program Files\PostgreSQL\9.2\share\contrib\postgis-2.0
You will end up with a view as shown in Figure 7:
Figure 7. Schemas in mapit database
5. Load shapefiles to database
1- Make sure you have the shapefiles for TAZs and LUZs. They must be in WGS84 spatial reference which is in PostGIS SRID=4326. (If not, you can convert them after, see below.)
2- Use the PostGIS Shapefile Import/Export Manager plugin (located under “Plugins” menu in pgAdmin III) to load the shapefiles to the public schema.
Figure 8. Importing shape files to PostGIS
WARNING: Depending on the version, you need to specify which file to add, specify the path and the name of the geometry column as “the_geom” as indicated on the window, and the SRID number of your shapefile, then press Import. You need to find the projection of your shapefile, and the corresponding SRID. http://www.bostongis.com/PrinterFriendly.aspx?content_name=postgis_tut01 has some information on finding an SRID. If your shapefile is not in SRID 4326, do not call the column the_geom, rather call it the_geom_mysrid, then execute commands similar to the following (TEST THIS) to update the projection:
select AddGeometryColumn(‘tazs’, ‘the_geom’, 4326, ‘MULTIPOLYGON, 2);
update tazs set the_geom = ST_Transform(the_geom_mysrid, 4326);
WARNING: The name of the columns with the geometry in your luzs and tazs tables should be the same as indicated in Figure 8, “the_geom”. If they are not the same, changes should be done clicking with the right button on the column name and going to properties to change it to “the_geom”. This is shown in Figure 9. And the final column names are shown in Figure 10.
Figure 9. Changing the name of the column with the geometry in the luzs and tazs tables
Figure 10. Column names “the geom” for luzs and tazs
3- Apply the Land Area script: From pgAdmin 3: open and run the following SQL script: C:\djangoProjects\web_output\SQL Changes\addLand_area to SpatialLayers.sql If land area already exists in the table, then this script will return an error.
4- Make note of the extent: similar to the previous step, run the query in C:\djangoProjects\web_output\SQL Changes\calculateLayerExtent.sql
Right the resulted numbers down as you will need to put them in settings.py file
5- Each polygon in the TAZ and LUZ has to have a point associated with it for drawing charts on the polygon. Open the query C:\djangoProjects\web_output\SQL Changes\addGraphPointGeom.sql and run it. (If there is an error about topology, you can change the text “ST_PointOnSurface” to “ST_Centroid”, and the graphs will appear at the centroid of the zone, which in some cases may not be on the zone.)
[NEED TO SET COMMODITY NUMBERS, COMMODITY TYPES, ACTIVITY NUMBERS, ACTIVITY TYPES. COMMODITYNUMBERS.CSV AND ACTIVITYNUMBERS.CSV CAN BE IMPORTED FROM MODEL OUTPUTS, TYPES NEED TO BE DEFINED BY THE USER.]
6. Configure MapIt settings file
1- In Web_output folder, make a copy of “default_settings.py” file, and call it: mapit_settings.py
2- Edit the mapit_settings.py, and make changes as appropriate (follow the CHANGE ME comments)
3- Save and close
WARNING: The name of the database is mapit, and by default postgres is configured to not accept connections by IP address, so if the database is on the same machine as mapit use 'localhost' as a IP address
WARNING: in the “default_settings.py” there is a "fieldNameforJoin" that should be specified, and it is the luz or taz number column name. This field should be integer instead of numeric. If the only pre-existing one is numeric should be changed to integer inside of Postgres.
7. Create the Map files for MapServer
1- Edit the “mapFilesCreator.py” file in Web_output folder
2- Make sure that the script is importing mapit_setting file that was created in Section 6. In other words, the mapFilesCreator.py must have the following line in it:
import mapit_settings as settings
3- In command prompt, run the script:
C:\djangoProject\web_output> C:\Python32\python mapFilesCreator.py