Data Updating Procedures

The data sets in WaterMarque containing pollution and hydrological monitoring stations and the water quality attribute information for these stations are constantly being updated by the Department of Water and Sanitation (DWAF). To supply WaterMarque users with current data these data sets will be regularly updated.

The original data sets reside on databases at DWAF head-office and regional offices in different file formats. The data sets had are reformatted to ARC/INFO format, namely coverages and INFO tables to be incorporated and displayed in WaterMarque. In order to simplify the updating of the data within WaterMarque, procedures were developed to automate the reformatting process where possible. These procedures are documented below.

N.B. Because of changes to the structure of the Department of Water and Sanitation & Forestry, the names of files may not be exactly as specified here.
The data sets that are updated are as follows:

hydrological monitoring stations coverages for the whole country and each province. The coverages are:
- HWQ
- HWQ_MPUM
- HWQ_FSTATE
- HWQ_GAUT
- HWQ_NATAL
- HWQ_NCAPE
- HWQ_NPROV
- HWQ_NWEST
an INFO table containing water quality attribute data called INORGANIC.DAT
pollution monitoring station coverages for South Africa and each DWAF region called:
- POLMON
- FREESTATE
- WCAPE
- ECAPE
- SWCAPE
- TRANSVAAL
- NATAL
- HIGHVELD
an INFO table containing water quality data for pollution monitoring stations called POLMON.DAT
INFO tables containing permit attribute data for pollution monitoring stations for South Africa and each DWAF region called:
- PERMIT.POLMON
- PERMIT.FREESTATE
- PERMIT.ECAPE
- PERMIT.SWCAPE
- PERMIT.TRANSVAAL
- PERMIT.NATAL
- PERMIT.HIGHVELD

A general diagram of the updating procedures. The new data will be collated by IWQS from DWAF head-office and regional offices. The data formatting procedures will be done by GIMS and the formatted data will be send to each installation site.

WaterMarque updating process

Data formatting procedure for Hydrological Monitoring Stations

The hydrological monitoring stations are exported in ASCII file format from the Stations Catalog Database(STATKAT) at DWAF Head-office. Follow these steps to convert the ASCII data into point coverages:

1. Load ASCII file:
Create a temporary workspace for loading the data and running the procedures to ensure that existing data is not overwritten immediately: Arc: cw tempwork Arc: w tempwork
2. Run AML to generate point coverage: Arc: &amlpath $WMSYS/ptool/aml Arc: &r his_pntdef

Usage: &r his_pntdef <directory> <asciiname> <infoname> <coverage name>

Arc: &r his_pntdef $WMSYS/tempwork monpnt.asc stations.dat stations

If you have problems with the ADD FROM line, it could be that:

You need to remove spurious blanks from the input file.
In vi Editor, try
:1,$s/ *,/,/g :1,$s/, */,/g

which removes blanks before and after commas.
You need to remove quote (") marks from the input file. These occur when exporting fields containing commas from packages such as Excel and Quattro. Remember to remove the offending commas as well (usually a fiendish decimal).

3. Create point coverages for DWAF regions: an AML is run that does an INTERSECT with
the monitoring stations and the DWAF regions polygon coverage, and the monitoring
stations coverage and the quaternary catchment polygon coverage. The monitoring stations
are then split up into the different regions.
Arc: &r his_splitstat Usage: &r his_splitstat.aml <directory> <stations coverage> {routine} Arc: &r his_splitstat $WMSYS/tempwork stations
4. Kill existing coverages in WaterMarque data directory: Arc: w $WMSYS/wmdata/general Arc: kill hwq all Killed hwq with the ALL option Arc: kill hwq_wcape all Killed hwq_wcape with the ALL option Arc: kill hwq_ecape all Killed hwq_ecape with the ALL option Arc: kill hwq_mpum all Killed hwq_mpum with the ALL option Arc: kill hwq_fstate all Killed hwq_fstate with the ALL option Arc: kill hwq_gaut all Killed hwq_gaut with the ALL option Arc: kill hwq_natal all Killed hwq_natal with the ALL option Arc: kill hwq_nprov all Killed hwq_nprov with the ALL option Arc: kill hwq_nwest all Killed hwq_nwest with the ALL option Arc: kill hwq_ncape all Killed hwq_ncape with the ALL option
5. Copy new coverages to WaterMarque data directory: Arc: copy hwq $WMSYS/wmdata/general/hwq Copied hwq to $WMSYS/wmdata/general/hwq Arc: copy hwq_ecape $WMSYS/wmdata/general/hwq_ecape Copied hwq_ecape to $WMSYS/wmdata/general/hwq_ecape Arc: copy hwq_wcape $WMSYS/wmdata/general/hwq_ecape Copied hwq_wcape to $WMSYS/wmdata/general/hwq_ecape Arc: copy hwq_mpum $WMSYS/wmdata/general/hwq_mpum Copied hwq_mpum to $WMSYS/wmdata/general/hwq_mpum Arc: copy hwq_fstate $WMSYS/wmdata/general/hwq_fstate Copied hwq_fstate to $WMSYS/wmdata/general/hwq_fstate Arc: copy hwq_gaut $WMSYS/wmdata/general/hwq_gaut Copied hwq_gaut to $WMSYS/wmdata/general/hwq_gaut Arc: copy hwq_natal $WMSYS/wmdata/general/hwq_natal Copied hwq_natal to $WMSYS/wmdata/general/hwq_natal Arc: copy hwq_ncape $WMSYS/wmdata/general/hwq_ncape Copied hwq_ncape to $WMSYS/wmdata/general/hwq_ncape Arc: copy hwq_nprov $WMSYS/wmdata/general/hwq_nprov Copied hwq_nprov to $WMSYS/wmdata/general/hwq_nprov Arc: copy hwq_nwest $WMSYS/wmdata/general/hwq_nwest Copied hwq_nwest to $WMSYS/wmdata/general/hwq_nwest

Data formatting procedure for Water Quality attribute data for Hydrological Monitoring Stations

The water quality data for hydrological monitoring stations are exported as an ASCII file from the (HIS) database at DWAF Head-office. The ASCII file must be in comma delimited format. Follow these steps to convert the ASCII data into point coverages:

1. Load ASCII file:
Create a temporary workspace for loading the data and running the procedures to ensure that existing data is not overwritten immediately: Arc: cw tempwork Arc: w tempwork
2. Run AML to generate INFO table: Arc: &amlpath $WMSYS/ptool/aml Arc: &r his2info.aml

Usage: &r his2info.aml <path Name> <Ascii Name> <Info Name> {routine}
Arc: &r his2info $WMSYS/tempwork hisdata.txt inorganic.dat

3. Delete old inorganic.dat from WaterMarque data directory: Arc: w $WMSYS/wmdata/wq Arc: &sv d [delete inorganic.dat -info]
4. Copy new INFO table to WaterMarque data directory: Arc:copyinfo inorganic.dat $WMSYS/wmdata/wq/inorganic.dat

Data formatting procedure for Pollution Monitoring (POLMON) Stations

The POLMON stations and related water quality information are exported in ASCII file format from the POLMON system at Head-office. The files needed for each DWAF region include: mon_pt.txt, permit.txt, qual_val.txt.

Formatting the POLMON stations:

The exported POLMON stations data for a DWAF region is contained in mon_pt.txt. Follow these steps to convert the ASCII data into a point coverage:
1. Load ASCII file:
Create a temporary workspace for loading the data and running the procedures to ensure that existing data is not overwritten immediately: Arc: cw tempwork Arc: w tempwork
2. Run AML for each DWAF region, to generate point coverage: the AML generates a point
coverage and add additional items such as class, type and z-code. Arc: &amlpath $WMSYS/ptool/aml Arc: &r pol_makepts Usage: &r pol_makepts.aml <new polmon cover> <province> {routine}

Arc: &r pol_makepts transvaal transvaal

3. Combine all DWAF regional coverages into a single POLMON coverage: Once the coverages for all the regions exist, APPEND the coverages to create a coverage containing pollution monitoring stations for the whole country: Arc: append polmon point Enter coverages to be APPENDed. (Type END or a blank line when done): ===================================================== Enter the 1st coverage: natal enter the 2nd coverage: highveld Enter the 3rd coverage: freestate Enter the 4th coverage: transvaal Enter the 5th coverage: ecape Enter the 6th coverage: swcape Enter the 7th coverage: END Appending coverages...

4. Delete old coverages from WaterMarque data directory:
Arc: kill $WMSYS/wmdata/general/polmon all Killed $WMSYS/wmdata/general/polmon with the ALL option Arc: kill $WMSYS/wmdata/general/transvaal all Killed $WMSYS/wmdata/general/transvaal with the ALL option Arc: kill $WMSYS/wmdata/general/highveld all Killed $WMSYS/wmdata/general/highveld with the ALL option Arc: kill $WMSYS/wmdata/general/swcape all Killed $WMSYS/wmdata/general/swcape with the ALL option Arc: kill $WMSYS/wmdata/general/ecape all Killed $WMSYS/wmdata/general/ecape with the ALL option Arc: kill $WMSYS/wmdata/general/natal all Killed $WMSYS/wmdata/general/natal with the ALL option Arc: kill $WMSYS/wmdata/general/freestate all Killed $WMSYS/wmdata/general/freestate with the ALL option

5. Copy new coverages to WaterMarque data directory: Arc: copy polmon $WMSYS/wmdata/general/polmon Copied polmon to $WMSYS/wmdata/general/polmon Arc: copy natal $WMSYS/wmdata/general/natal Copied natal to $WMSYS/wmdata/general/natal Arc: copy freestate $WMSYS/wmdata/general/freestate Copied freestate to $WMSYS/wmdata/general/freestate Arc: copy transvaal $WMSYS/wmdata/general/transvaal Copied transvaal to $WMSYS/wmdata/general/transvaal Arc: copy ecape $WMSYS/wmdata/general/ecape Copied ecape to $WMSYS/wmdata/general/ecape Arc: copy swcape $WMSYS/wmdata/general/swcape Copied swcape to $WMSYS/wmdata/general/swcape Arc: copy highveld $WMSYS/wmdata/general/highveld Copied highveld to $WMSYS/wmdata/general/highveld

Data formatting procedure for Water Quality attribute data for POLMON stations

The exported water quality data for POLMON stations for a DWAF region is contained in qual_val.txt. Follow these steps to convert the ASCII data into a INFO file:

1. Load ASCII file:
Create a temporary workspace for loading the data and running the procedures to ensure that existing data is not overwritten immediately: Arc: cw tempwork Arc: w tempwork
2. Run C program for each DWAF region to reformat data into 4 ASCII files: Arc: &amlpath $WMSYS/ptool/aml Arc: polcon.2 Usage: polcon <POLMON FILE> <OUTPUT FILE> where <POLMON FILE> must be sorted according to station, date eg. sort -o qual_val.srt +3 -4 +7.6 -8 +7.3 -7.6 +7.0 -7.3 qual_val.txt ********************************************************************** * * * P O L M O N C O N V E R T * * * * A program by W. F. Geldenhuys. August 1995. * * * **********************************************************************
The <POLMON FILE> is usually extracted from the POLMON system in the correct sort order, but if not the above usage for the sort can be used to sort the file.
Arc: polcon.2 qual_val.txt gauteng

The output will be gauteng.1, gauteng.2, gauteng.3, gauteng.4

3.AML for each DWAF region to generate INFO table. This AML must be run for each DWAF region:

Arc: &r pol_qual2info

Usage: &r pol_qual2info.aml <Ascii prefix> <INFO prefix> {routine}

Arc: &r pol_qual2info gauteng gauteng
The output will be an INFO table called gauteng

4. Merge the INFO tables of the DWAF regions into one table and rename the table to POLMON.DAT: Arc: copyinfo transvaal polmon.dat Arc: info INFO EXCHANGE CALL 31/03/1998 11:47:22 INFO 9.42 11/11/86 52.74.63* Copyright (C) 1994 Doric Computer Systems International Ltd. All rights reserved. Proprietary to Doric Computer Systems International Ltd. US Govt Agencies see usage restrictions in Help files (Help Restrictions) ENTER USER NAME>ARC ENTER COMMAND >SELECT GAUTENG 839 RECORDS SELECTED ENTER COMMAND >MERGE INTO POLMON.DAT ON STATION ENTER COMMAND >SELECT SWCAPE 176 RECORDS SELECTED ENTER COMMAND >MERGE INTO POLMON.DAT ON STATION ENTER COMMAND >SELECT WCAPE n RECORDS SELECTED ENTER COMMAND >MERGE INTO POLMON.DAT ON STATION ENTER COMMAND >SELECT ECAPE 456 RECORDS SELECTED ENTER COMMAND >MERGE INTO POLMON.DAT ON STATION ENTER COMMAND >SELECT NATAL 332 RECORDS SELECTED ENTER COMMAND >MERGE INTO POLMON.DAT ON STATION ENTER COMMAND >SELECT HIGHVELD 1002 RECORDS SELECTED ENTER COMMAND >MERGE INTO POLMON.DAT ON STATION
5. Delete old POLMON.DAT in WaterMarque data directory and copy new INFO table: Arc: &ty [delete $WMSYS/wmdata/wq/polmon.dat -info] Arc: copyinfo polmon.dat $WMSYS/wmdata/wq/polmon.dat

Data formatting procedure for Permit data

The exported permit data for POLMON stations for a DWAF region is contained in permit.txt. Follow these steps to convert the ASCII data into a INFO table and to create a frequency table:

1. Load ASCII file:
Create a temporary workspace for loading the data and running the procedures to ensure that existing data is not overwritten immediately: Arc: cw tempwork Arc: w tempwork
2. Run AML for each DWAF region to reformat data into 4 ASCII files: Arc: &amlpath $WMSYS/ptool/aml Arc: &r pol_permitup.aml Usage: &r pol_permitup.aml <INPUT ASCII file> <province name> <In Path> <Out Path> {routine}
Arc: &r pol_permitup permit.txt gauteng $WMSYS/tempwork $WMSYS/tempwork

The output will be two INFO tables named permit.gauteng and freq.gauteng. The province
names are changes to the abbreviations and names as used in WaterMarque.

3. Merge the permit INFO tables of the DWAF regions into one table and rename the table to
PERMIT.ALL: Arc: copyinfo permit.gauteng permit.all Arc: info INFO EXCHANGE CALL 31/03/1998 11:47:22 INFO 9.42 11/11/86 52.74.63* Copyright (C) 1994 Doric Computer Systems International Ltd. All rights reserved. Proprietary to Doric Computer Systems International Ltd. US Govt Agencies see usage restrictions in Help files (Help Restrictions) ENTER USER NAME>ARC

ENTER COMMAND >SELECT PERMIT.ECAPE 839 RECORDS SELECTED ENTER COMMAND >MERGE INTO PERMIT.ALL ON STATION ENTER COMMAND >SELECT PERMIT.SWCAPE 176 RECORDS SELECTED ENTER COMMAND >MERGE INTO PERMIT.ALL ON STATION ENTER COMMAND >SELECT PERMIT.TRANSVAAL 456 RECORDS SELECTED ENTER COMMAND >MERGE INTO PERMIT.ALL ON STATION ENTER COMMAND >SELECT PERMIT.NATAL 332 RECORDS SELECTED ENTER COMMAND >MERGE INTO PERMIT.ALL ON STATION ENTER COMMAND >SELECT PERMIT.HIGHVELD 1002 RECORDS SELECTED ENTER COMMAND >MERGE INTO PERMIT.ALL ON STATION

4. Run the frequency command on the merged file:
Arc: frequency permit.all freq.all Enter Frequency item names (type END or a blank line when done): ================================================================ Enter the 1st item: station Enter the 2nd item: end Enter Summary item names (type END or a blank line when done): ============================================================== Enter the 1st item: end
5. Delete existing files in WaterMarque data directory: Arc: &sv d [delete $WMSYS/wmdata/wq/permit.ecape -info] Arc: &sv d [delete $WMSYS/wmdata/wq/permit.swcape -info] Arc: &sv d [delete $WMSYS/wmdata/wq/permit.highveld -info] Arc: &sv d [delete $WMSYS/wmdata/wq/permit.transvaal -info] Arc: &sv d [delete $WMSYS/wmdata/wq/permit.natal -info] Arc: &sv d [delete $WMSYS/wmdata/wq/permit.gauteng -info] Arc: &sv d [delete $WMSYS/wmdata/wq/permit.all -info]

6. Copy new files to data directory:
Permit files:
Arc: copyinfo permit.ecape $WMSYS/wmdata/wq/permit.ecape Arc: copyinfo permit.transvaal $WMSYS/wmdata/wq/permit.transvaal Arc: copyinfo permit.highveld $WMSYS/wmdata/wq/permit.highveld Arc: copyinfo permit.natal $WMSYS/wmdata/wq/permit.natal Arc: copyinfo permit.swcape $WMSYS/wmdata/wq/permit.swcape Arc: copyinfo permit.gauteng $WMSYS/wmdata/wq/permit.gauteng Arc: copyinfo permit.all $WMSYS/wmdata/wq/permit.all
Frequency files: Arc: copyinfo freq.ecape $WMSYS/wmdata/wq/freq.ecape Arc: copyinfo freq.transvaal $WMSYS/wmdata/wq/freq.transvaal Arc: copyinfo freq.highveld $WMSYS/wmdata/wq/freq.highveld Arc: copyinfo freq.natal $WMSYS/wmdata/wq/freq.natal Arc: copyinfo freq.swcape $WMSYS/wmdata/wq/freq.swcape Arc: copyinfo freq.gauteng $WMSYS/wmdata/wq/freq.gauteng Arc: copyinfo freq.all $WMSYS/wmdata/wq/freq.all

Return to index page