geograph_web HOWTO

This document describes the installation steps I needed to get the code from www.geograph.org.uk working for our tiny geograph project for a small part of Germany.

1. Prerequisites

What do you need to get the software working?

• apache web server
• php5
• mod_rewrite
• permissions for running cron jobs on the machine
• mysql database
• Imagemagick is not neccessary, but highly recommended
• detailed maps of your area which use your preferred grid: you should try to get these maps as software
• formulae which allow you to convert your grid coordinates to the usual geographic latitude/longitude coordinates (and back).
• a lot of disk space on the server (about number of square kilometres times number of pictures per grid square times 0.200 MB = some Gigabytes)
• basic knowledge about php, javascript, sql
• a lot of time

2. Setting up the code

Get the code

The first step was setting up the code: Download the latest nightly from the geograph.org.uk project (you need the latest geograph_web-* file) and unpack it.

Patches

I had to apply some code changes to get the project working. The main changes can be found on the project's web site. If you think that applying some of these changes is neccessary for you, just download the patches you need, test if they are still applying cleanly to the code

cd local
patch --dry-run -p6 < /PATH/TO/THE/PATCH.diff

handle error messages if neccessary, and apply the patches:

cd local
patch -p6 < /PATH/TO/THE/PATCH.diff

You can do this later on, of course, but keep in mind that you might need to replicate the changes of the default template directory ("basic") to your own template directory (see below) and that you have to upload the modified files to your web server.

Important changes are:

• All patches named 0?_bug*.diff and 1?_bug*.diff which are bug fixes.

• 50_mailfrom.diff changes hard coded email addresses and adds an "envelope from" which is needed here. This needs a better solution using configurable mail addresses, though. Please change to your address before applying the patch.

• 51_url.diff changes hard coded URLs. Please change to your address before applying the patch.

Template directory

Find a name for the directory that will contain the templates from which your xhtml code will be created. I've chosen "germany32" which means "Germany, UTM zone 32", maybe "germany" would have been the better choice. To avoid confusion, I'll use tmplname in the following text.

Copy the default templates to your template directory:

cd local/public_html/templates
cp -ai basic tmplname

Copying is actually not needed: If smarty does not find a template, it tries to find it in the "basic" directory.

Directory structure

You need to add some directories:

cd local
mkdir adodbcache rastermaps rastermaps/OS-50k rastermaps/OS-50k/tiffs public_html/geophotos

Permissions

Some directories need special permissions: The web server and cron jobs need write permissions. I.e. you need to execute a command like this one:

cd local
chmod a+w adodbcache public_html/memorymap public_html/rss
chmod a+w public_html/kml public_html/sitemap
chmod a+w public_html/maps public_html/photos public_html/geophotos
chmod a+w public_html/templates/basic/cache public_html/templates/basic/compiled
chmod a+w public_html/templates/tmplname/cache public_html/templates/tmplname/compiled
chmod a+w rastermaps rastermaps/OS-50k rastermaps/OS-50k/tiffs

Furthermore, some scripts need to be executable:

cd local
chmod a+x scripts/*.pl

You should also assure that public_html/geophotos/00 is a symbolic link to ../photos:

cd local/public_html/geophotos
ln -sn ../photos 00

Grid and coordinates

You need to think about which coordinate grid(s) you should use for your area. Is there a national grid? Or do you want to use UTM coordinates and the MGRS? Is one grid enough, or do you need several of them (different Islands, different UTM zones)?

I decided to use different MGRS grids for the UTM zones in Germany.

Reference id

Choose numerical ids, the reference ids, for every grid you use. Currently, there are the following reference ids in the code I use:

Grid prefix

Choose the grid prefixes you use. The grid prefix is a combination of up to three characters which identify each square of 100 km × 100 km. Changing the code to allow four letters might lead to problems because they need too much space on the maps. Using one ore two letters might still lead to problems because the length of the grid prefix was used to determine the region in many parts of the code.

I decided to use the MGRS coordinates without leading UTM Zone digits, which results in grid prefixes of three letters (e.g. coordinates of Füssen: 32T PT 27 69: Grid square TPT2769 with grid prefix TPT).

Coordinate conversion

The hardest task is coordinate conversion. You have to provide php routines that convert from your local coordinates (area given by reference index) to longitude/latitude (WGS84) coordinates and routines that convert from longitude/latitude (WGS84) coordinates (area only given by these coordinates!) back to your local coordinates.

If you want to use google for displaying the maps, you also have to provide JavaScript routines which do basically the same, with one difference: You don't have a reference index here, but must use the grid prefix to identify your area (one of many reasons, why you should use unique grid indexes, even for different grids). See the UTM conversion routines I use for inspiration.

Remark: There is also a so called "internal coordinate system" in the code; these are the "real" coordinates, which are used to display the maps, etc. Internal coordinates are calclulated from the local coordinates by adding a constant offset (configurable for each grid index).

Initial grid

This step is easy but time consuming: For building the grid later on, you need a grayscale bitmap (one png file for every reference index) where each pixel represents the "land percentage" of the gridsquares (1 km × 1 km) of your area:

It is possible to do this in several steps, so you can start with a small area. You don't have to spend too much time with the gray pixels, because there is a much easier way to estimate these percentages later on. That's why I only used black (100% land) and white (< 100% land or land to be added later).

See my files, stage 1, stage 2, stage 3 or the file for Great Britain.

After creating these files, you should copy them to the administration directory of the code:

cd local/public_html/admin
cp -i /PATH/TO/YOUR/FILE.png .

3. Configuration

Web server configuration

You can use local/apache/geograph.conf as template config file for apache:

• Set the document root to the public_html directory of the source tree you will copy to the server.

• Adopt the php_value lines. You might want to add a line like

  php_value allow_call_time_pass_reference 1

  to get rid of warnings in the log.

• Adopt the rewrite rules. If you are using a .htaccess file, you might need to substitute leading slashes (/) with caret symbols (^).

• If you have problems when changing a lot of categories on the category consolidation page, the following lines might help.

  php_value suhosin.post.max_array_depth 0
php_value suhosin.post.max_array_index_length 0
php_value suhosin.post.max_vars 0
php_value suhosin.request.max_array_depth 0
php_value suhosin.request.max_array_index_length 0
php_value suhosin.request.max_vars 0

geograph_web configuration

To configure the geograph_web code, you should copy local/libs/conf/www.exampledomain.com.conf.php to a file with the name of your domain in the same directory. If the URL for your project is http://your.doma.in, you'd do something like

cd local/libs/conf
cp -i www.exampledomain.com.conf.php your.doma.in.conf.php

Now, you can start editing this copy, changing the configuration variables to values which match your area, server setup, etc. The comments in this file are helpful, but I'll still give some hints:

• You should add the following lines which are missing (e.g. below the KML_HOST configuration near the end of the file):

  $CONF['TILE_HOST'] = $_SERVER['HTTP_HOST'];
$CONF['STATIC_HOST'] = $_SERVER['HTTP_HOST'];
$CONF['CONTENT_HOST'] = $_SERVER['HTTP_HOST'];

• If you want to use google for displaying the raster maps, you must get an API key and configure the software in the following way:

  $CONF['raster_service']='Google,Grid';
$CONF['google_maps_api_key'] = 'YOUR-API-KEY';

• For my (one server) setup, the following line is working:

  $CONF['enable_cluster'] = 0;

• If you don't want to use the sphinx search engine, you need the following line:

  $CONF['sphinx_host'] = "";

• Set up your templates with

  $CONF['template']='tmplname';

• On a public server you should use

  $CONF['smarty_debugging']=0;

  because your database password is visible otherwise.

• Use some line like

  $CONF['references_all'] = array(0=>'name of your project')+$CONF['references'];

  in order to get properly named labels for data regarding the whole project.

• You need a line like

  $CONF['origins'] = array(reference_index_1 => array(x_offset_1,y_offset_1),
                         reference_index_2 => array(x_offset_2,y_offset_2));

  in order to define the offsets of the local coordinates of your grid and the internal coordinates (see section about coordinates above). E.g. I use a line like

  $CONF['origins'] = array(3 => array(-200,-5200));

  because I want to map UTM coordinate (200,5200) (in km) to the origin of the internal coordinate system.

• While changing code or templates, you should use

  $CONF['smarty_caching']=0;

  because your changes will be invisible for a long time otherwise. The disadvantage is a slowdown and a higher server load.

4. Installation

Installing the code

Just upload the source tree to the server (scp, ftp, ...). Keep in mind that public_html must be your document root and assure that the permissions and the symbolic link mentioned above are preserved. During the following installation process, it's a good idea to watch the error log of your web server.

Initialization of the database

Setting up the database needs the following steps:

• Import schema/schema.mysql
• Import schema/changes.mysql which contains current changes to the database schema. Because these changes can be somewhat imprecise I suggest importing these lines step by step. If there are major problems waiting for the next nightly might help; if that does not help: I got quick, helpful and friendly answers on my messages to the www.geograph.org.uk developers.

Setting up the grid prefixes

The next step is importing your grid prefixes into the database. schema/gridprefix_creation.txt gives some hints here but is somewhat outdatet. You might want to look at my mysql file which is generated from a tab separated list (grid prefix, reference index, easting (kilometers), northing (kilometers)) using an awk script:

./gridprefix.awk < gridprefix.dat > gridprefix.txt

After putting the "replace ..." line in front of gridprefix.txt and a semicolon at the end, you can import the file into your database.

Adding the administrator account

Now it's time to register the administrator user with /register.php. Just enter your data into the form, confirm, and follow the confirmation link in the mail you'll receive afterwards.

After logging out your new user, you can grant this user admin privileges using some statement like

update user set rights='basic,admin' where email='yourmailadress';

Now you can log in again, edit your profile and grant your user moderator and ticket privileges using the "Moderator Admin" tool (/admin/moderator_admin.php) you find on the "Admin Index" page.

Setting up the grid

Now, it's time to tell the system where your grid squares are situated on the map. This is done by filling in the form you find on /admin/gridbuilder.php: Just enter the path (relative to the document root) to the grayscale png file(s) you created before, the amount of pixels your png file should be shifted (x=0 and y=0 if you used internal coordinates when creating the file(s)) and your reference index. You should check "Redraw maps" and also "Clear existing land squares if marked as sea in this image" if you want to destroy land squares you created during an earlier gridbuilder session.

Testing the installation

For testing your installation, you can use /test.php. Unfortunately, this script seems to be somewhat hypersensitive. This can be patched away using 29_test.diff if that makes you sleep better.

Init scripts

There are some scripts which allow you to initialize the graphs on the statistics page:

cd scripts
./update_rate_graph.pl --init=STARTTIME --base=/PATH/TO/SRC/TREE --db=DBNAME --user=DBUSER --pass=DBPASSWORD
./update_submission_graph.pl --init=STARTTIME --base=/PATH/TO/SRC/TREE --db=DBNAME --user=DBUSER --pass=DBPASSWORD

/PATH/TO/SRC/TREE is the parent directory of the document root; STARTTIME is the unix time stamp of the first data point. You can convert a date string to a unix time stamp using

date -d 'date string' +%s

These commands print a list of commands you should execute afterwards.

Setting up cron

For setting up cron, I basically adopted the settings from local/cron/crontab.example in the source tree: I did substitute GET with wget -q -O /dev/null (curl -s -o /dev/null should work, too) and I omitted the first recreate_maps.php line (which seems to be outdated) and the update_cpu_graph.pl line (because I can't access /proc/loadavg on my server).

This is also a good point for checking the error logs, access logs and mails (from cron).

You can avoid file permission problems resulting from different uids of cron daemon and web server by GETting http://DOMAIN/_scripts/runscript.php?job=maps (or job=kml) instead of running the scripts directly from cron.

Submitting the first picture

Before submitting your first picture, you should register another user and grant him/her moderator and ticket privileges using the "Moderator Admin" tool (/admin/moderator_admin.php) you find on the "Admin Index" page. This is needed, because you can't moderate your own pictures.

Submitting the photograph is straight forward, just follow the instructions on submit.php. After entering the grid reference you'll be asked for the file name, exact position of photographer and photo subject. If you use google for displaying the maps, this is the point where you can test your JavaScript coordinate conversion routines: When you drag the markers over the map, the coordinates in the form should change.

After submitting the picture, the second moderator can log in and should find the image on the admin index (admin/index.php) and moderation index (admin/moderation.php). If this is not the case, you might have forgotten to apply my patch for this problem.

This is the point, where you can test moderating, ticket stuff, ... You can also watch the event mechanism on admin/events.php or speed things up by injecting the event "every_day" on this page.

Setting up forums

If you want to use the forums, you have to activate them in your configutation file (libs/conf/your.doma.in.conf.php):

$CONF['forums']=true;

After logging in as administrator, you can use the link "Admin panel" on the discussion page (/discuss/) and add forums, there.

Unfortunately, there are quite a lot hard coded forum ids in the code. So, you should either create your forums in the same order they were created on geograph.org.uk or use my patch for this (NOTE: this patch is already included in the nightlies) and adopt the $CONF['forum_XXXX'] settings from the example configuration file.

The forum ids on geograph.org.uk are

If you want to offer galleries, you have to add the galleries forum (which will be hidden on the forum list /discuss/).

Fixing the border of the area

If you want to fix land percentages of grid squares you can do this with the map fixer tool admin/mapfixer.php. It allows you to enter the grid reference of a square and change the land percentage.

If the squares you want to change are close to land squares, you can also add them to the map fixer queue (i.e. give them a land percentage of -1) by simply clicking on them on the map (/mapbrowse.php). The square will be black then (if you did not apply my patch to make those squares blue) and appear on the list on the map fixer page.

Adding towns

For adding town to the maps you have to import a mysql file like this. I generated it from a tab separated list (name, grid prefix, easting (10 meters), northing (10 meters), size, reference index, quadrant) using an awk script:

./towns.awk < towns.dat > towns.txt

After putting the "replace ..." line in front of towns.txt and a semicolon at the end, you can import the file into your database.

If you want to use this script you need to enter your grid prefixes in the BEGIN section.

The meaning of the columns:

Unfortunately you have to change the code of libs/geograph/map.class.php in order to tell the software to use loc_towns for your reference id. See my change for reference id 3. You might want to recreate the maps (admin/recreatemaps.php) afterwards.

Creating KML files

For creating KML/KMZ files for GoogleEarth just visit these URLs with your favourite browser: /_scripts/kml-_make.php, /_scripts/kml-style.php, /_scripts/kml-geograph.php

Make "picture of the day" work

To get rid of the error picture on /index.php you should enter your favourite pictures into the form on /admin/pictureoftheday.php.

The code should maybe changed to use a default picture if the list is empty...

5. What ist still to do for our project

Logo and Icon

We still need to create the logo (public_html/templates/tmplname/img/logo.gif) and a favicon (public_html/favicon.ico).

Templates

The templates must be translated and contents that are irrelevant for our area should be removed.

Code changes

• There are still a lot of places in the code, where hard coded user ids, reference indexes, forum ids, etc. need to be removed.
• Hard coded mail addresses should be replaced by one configuration variable. Also the envelope from address should be set.
• Some towns (Ludwigsburg) are not shown on the map; some towns are shown, but there's only a dot and no label.

6. What is still to add to the howto?

Article categories

Sitemap: Updates and Google

kml update

Logo and icon

Moving the area on the map

Gridbuilder and alpha channel

Permissions

• cd public_html/stuff
chmod a+w captcha_tmp
• cd public_html/rss
mkdir germanyde
mkdir germany32
chmod a+w germany32 germanyde
• cd public_html/sitemap
chmod a+w root

Create search for potd