RWA dxf to arc.doc
Regional Water Authority Detail Conversion
AML and General Information used in translation.
Prepared by: Brett Melton, July 14, 1997.
General Information
The purpose of the following AML is for AutoCAD DXF conversion into ArcINFO coverages for the Regional Water Authority. It translates 50-scale Detail information based on layer name, elevation, block name, etc. to specific coverages and verifies the integrity of the data.
Much of the data that will come across will be incorrect. That is where this AML really shows it’s worth. The AML was simple and efficient when first written. As time has gone on, the AML has evolved from a translator to a translator and extreme data verifier. Examples of problem areas include values of 2 ½ inside a pipe diameter when the field should read as 2.5. Or worse yet, when the pipe diameter reads something like 5.1.2 which is anybody’s guess. There are cases where lines have been drawn on the incorrect layer. There are dates in fields that haven’t arrived yet or dates that exist before the discovery of America. There are topology errors, block placement errors, town inconsistency errors, and many others not listed here. Many of the problems would have been avoided if the data input methods were more stringent. Date validation, layer setting, and end-point snapping macros should be used when this magnitude of data is being input. When these input methods are complete, a trimmed down version of this AML should be used to make the translation more efficient.
The very first thing the maintenance programmer should notice about this AML (as well as most all my AMLs) is that there is a DEBUG variable near the top. ALWAYS SET THIS TO TRUE IF YOU ARE DEBUGGING. Just trust me on that one - it will make your life easier. The second thing you should notice are the amlname and workspc lines. These are used for the bailout function. If the program ever crashes during execution. Check the line in the aml where it crashed first. If nothing becomes apparent, set the DEBUG to true and run it again. You should be able to see exactly how, why and where the crash is taking place. There is a variable called routname that is set for every routine. At crash / bailout time, you will be shown what routine caused the problem. This bit of knowledge may facilitate your debugging.
General Points:
dtype_list deals with the ascii Data Type files produced from AutoCAD when it’s assigning all the elevation values to the graphical data. Each number represents the data type number and contains all information about that data type. Most of the information from one ascii file will be loaded into one info file, but not always. For instance the data type file for hydrants will have most (if not all) of its information loaded into HYDRANT.DAT info file.
Each town has a town code. This town code is plugged into many of the info files. It is also used in the AML itself when doing different processes/procedures for different towns.
Routine List:
load_dxf_file Bring the AutoCAD dxf file into 3 major coverages for division later.
load_relate_files Load the Ascii Files processed in AutoCAD into ArcINFO.
divide_and_conquer Break the composite coverages into the final specific coverages.
populate_data_files Relate former ascii files into final coverages using elevation.
look_for_service_points Points not belonging in distrib, move to service.
alter_decimal_items Changing 2 ½ to 2.5, correcting dates, and other numeric corrections.
read_wnm50sym_table Assign the wnm50 symbols provided from Baysys.
define_cover_projection Project define all coverages.
add_missing_features Ensure proper PAT and AAT item structures, create if missing.
alter_info_definitions Because ESRI is full of idiots, corrupt the PAT and AAT items to conform.
clean_up_covers Kill all temporary coverages and dropitem extemporaneous fields.
These are the main executing routines. These routines have further sub-routines that assist the main routine is performing it’s task. The AML is setup so that the user/programmer can start anywhere and end anywhere in the routine list if DEBUG is set to TRUE. It is wise to use this ability instead of wasting a great deal of time waiting for something that has processed previously.
A Closer Look at the Routines
load_dxf_file
The first step in the process is to load everything into the “all” coverage. The “anno1” coverage accepts all standard text and it’s insertion point. The “anno2” coverage accepts all attribute annotation and the attribute insertion point as well as block insertion points of specified layers. The “cont” coverage loads contour lines and elevation text; but does not exist for all towns. This routine will also replace all aircock and blowoff blocks with the text string of “AC” and “BO” respectively.
load_relate_files
This routine starts and ends at a do loop. Every value in the dtype_list goes through this routine one at a time. This routine loads the ascii data file (after a little unix manipulation) into info. It then adds elevation and sorts the table for ordered relates. Okay, okay, I’ll talk more about the unix process. The first unix command changes all occurrences of |,| into just | and then deletes the single | at the end and beginning of each line. This is done with the three sed statements. Look up listitem in the AML book. The select statement in the do until length . . . loop is to get field type. Don’t try to understand it, it just works. The next select statement looks to see if the field is character, integer, date, or binary and will perform unix statements to conform the value of the rows.
divide_and_conquer
The routine is long but simple. It merely divides the composite coverages into the various land base and detail coverages. This is where the bulk of the work is done. Be sure to note areas where the [show coverage -exist] line is. A large portion of this routine will not execute unless certain coverages are created from the composite. A sub-routine worth noting here is the calculate_feature_ids. This routine will place unique numbers in the graphic and database files. Review this routine to see what the Feature ID is made up of.
populate_data_files
Load_Relate_Files loaded the ascii files into info. This routine relates those files to the final coverages by the dxf-elevation. It then reads the files and adds the appropriate information into the RWA template files and adds the feature-id from the coverage. The template files are stored at the base directory and are empty info files. This routine uses cursor processing and relates heavily. Caution should be used when [quote]ing a value versus simply assigning a value to a field. A sub-routine used by this routine is “verify coverage info structure” which currently only checks the distrib pat file.
look_for_service_points
Here we do a search for SERVICE valves and hydrant. Standard valves and hydrant get plugged into the distrib coverage. The same type of processing as populate data files is used here just for these service points. There are Four Major steps involved here. The first just creates two temporary coverages called service70 and service72 for further analysis. Step two assigns wnm_type and defines graphic point settings. Step three populates the rwa info file like was done in “populate_data_files”. Finally, step four extracts necessary items and plugs the points into distrib with the proper settings.
alter_decimal_items
For each info table and specific item name, the sub-routine “correct_decimal_item” is called. The sub-routine makes changes like: “UK” in numeric fields to “0.00” and changes numbers with a slash to a true decimal value. The sub-routine is hard to follow, but no editing should be done to the lines. Additional checks can be added in any format.
read_wnm50sym_table
This possibly should be done with relate tables, but got so complex that it just seemed easier to implement dynamically within the AML. It calculates the symcode and symcode2 fields in the coverages based on values provided by BAYSYS. It is very simple code to read but remains a long routine and requires a lot of time. For the person who attempts to make if more efficient --More Power to you !
define_cover_projection
The simplest of all the routines. It merely assigns the projection into all coverages.
add_missing_features
This is a list oriented routine. At the beginning is a list of coverage names followed by the number of items and the item names themselves to add if missing. This is required for proper librarian loading.
alter_info_definitions
Anybody with half a brain will get a chuckle out of this one. ESRI developed the structure of the librarian for RWA. They defined the coverage predefined items (such as coverage#) incorrectly. Again, I say that ESRI, the makers of ArcINFO, assigned incorrect data definitions for the coverages. After the librarian was delivered to RWA, ESRI was too lazy to simply (and I emphasize simply) redefine the items correctly. RWA requests that all the coverages we deliver to him conform to the demented structure. I should have named this routine “conform_to_the_idots”.
clean_up_covers
This is just house cleaning. Not much to comment on other than this is where to clean up after yourself if you create any temporary coverages or temporary items in info files.
/* Program: rwa_dxfarc.aml */
/* Author: Brett D. Melton */
/* Workspace: /b3/rwa/translate/detail */
/* Date: December 13, 1995 */
/* Upd: January 1, 1997 - Correct Service Valves & Hydrants. */
/* Upd: March 27, 1997 - Alter items to 4 10 B - who knew ? */
/* Purpose: To translate dxf into arc coverages for rwa map grids. */
/* Company: The BRODIE Group, Inc. */
/* _/_/_/_/_/_/_/ _/_/_/_/ _/_/_/_/ */
/* _/_/_/_/_/_/_/ _/_/_/_/_/ _/_/_/_/_/_/ */
/* _/_/ _/_/ _/_/ _/_/ */
/* _/_/ _/_/_/_/_/ _/_/ _/_/_/ */
/* _/_/ _/_/ _/_/ _/_/ _/_/_/ */
/* _/_/ _/_/_/_/_/_/ _/_/ _/_/ */
/* _/_/ _/_/_/_/_/ _/_/_/_/_/ */
/* \ / */
/* \\ // */
/* \\\' , / / */
/* \\\//, _/ //, */
/* \_-//' / //< */
/* \ /// <//` */
/* / >\\`__/ */
/* /,)-^>_\`\` */
/* (/ \\ //\\\ */
/* // _// \\ */
/* ((` ((` ` */
/* /> */
/* ( //------, */
/* (*)OXOXOXOXO(*>:::<---=-=-=-=-=-=-=-=-=-=-=-=-=-=-=------> */
/* ( \\------' */
/* \> */
/* Author's Note:
/* This program was developed over a long period of time because of constant */
/* changes by RWA and BAYSYS. This program is not structured well and has */
/* many inefficiencies. Do not be stressed if you notice the program doing */
/* strange things. */
/* allow for up to 10 input grids */
&arg map1 map2 map3 map4 map5 map6 map7 map8 map9 map10
&s amlname := rwa_dxfarc.aml
&s workspc := /b3/rwa/translate/detail
&s dtype_list := 027 041 060 062 066 070 072 074 075 078 080 082 083 103 107
&s DEBUG := .FALSE.
&if %DEBUG% &then; &s MessageEchoStatus := &on
&else ; &s MessageEchoStatus := &off &all
display 0
&system clear
&type \\\\
&type _/_/_/_/ _/_/ _/_/ _/_/
&type _/_/_/_/_/ _/_/ _/_/ _/_/ _/_/
&type _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
&type _/_/_/_/_/ _/_/ _/_/ _/_/ _/_/_/_/_/_/
&type _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
&type _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
&type _/_/ _/_/ _/_/_/ _/_/ _/_/ _/_/
&type \\\\
&severity &error &routine bailout
&if [substr [extract 1 [show version]] 1 3] ne 6.1 &then
&return &inform \\Sorry, but you must run this in version 6.1.x\\
&if NOT [null %map1%] &then
&call mapsheetlist
&else
&s mapsheetlist :=
&s routname := beginning
&if [show program] ne ARC &then
&return &inform \\I can only operate %amlname% in ARC module only\\
&type \\\\\\
&type 4 Milford
&type 5 Branford
&type 9 New Haven
&type 11 North Haven
&type 12 Cheshire
&type 18 Guilford\\
&s ct_town := [response 'Enter Number or Enter Town Name ']
/* &s ct_town := 11; &type \\Working on North Haven\\
&if [length %ct_town%] eq 1 &then
&s ct_towncode := 0%ct_town%
&else
&s ct_towncode := %ct_town%
&if [type %ct_town%] lt 0 &then &do
&select %ct_town%
&when 4
&s ct_town := milford
&when 5
&s ct_town := branford
&when 9
&s ct_town := new_haven
&when 11
&s ct_town := nor_haven
&when 12
&s ct_town := cheshire
&when 18
&s ct_town := guilford
&otherwise
&s ct_town := *Invalid_Numeric_Entry*
&end /* end of select
&end /* end of ct_town numeric
&if NOT [exist %workspc%/%ct_town% -workspace] &then
&return &inform \\%workspc%/%ct_town% workspace does not exist
&workspace %workspc%/%ct_town%
&if [null %mapsheetlist%] &then
&do &until %mapsheetlist% ne '?'
&type Enter List of Map Grids (without dxf extension) separated by a space
&s mapsheetlist := [locase [response 'or enter ( A -> ALL , X -> EXIT , ? -> LIST ) ']]
&if [quote %mapsheetlist%] eq '?' &then &do
&type \
&sys ls -d ???.dxf
&type \
&end
&if [translate [quote %mapsheetlist%]] eq 'X' &then &do
&return;&return &inform \\Terminated by User.\\
&end
&end
&do checkfile &list %dtype_list%
&if NOT [exist %workspc%/dtype%checkfile%.template -info] &then
&return &inform \\Templates Missing\\
&end /* end of do checkfile
/***********************************************************************************/
/********************************** ROUTINE LIST ***********************************/
/***********************************************************************************/
&s Rcount := 1
&s routname%Rcount% := load_dxf_file ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := load_relate_files ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := divide_and_conquer ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := populate_data_files ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := look_for_service_points ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := alter_decimal_items ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := read_wnm50sym_table ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := define_cover_projection ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := add_missing_features ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := alter_info_definitions ; &s Rcount := [calc %Rcount% + 1]
&s routname%Rcount% := clean_up_covers ; &s Rcount := [calc %Rcount% + 1]
&s RoutineMax := [calc %Rcount% - 1]
&s RoutineMin := 1
&type \\************************************************************************
&do RoutineNum = 1 &to %RoutineMax% &by 1
&type Task Number %RoutineNum% ==> [value routname%RoutineNum%]
&end
&type ************************************************************************\\
&if %DEBUG% &then &do
&s RoutineMin := [response [quote Enter Routine Number to start at: <return for %RoutineMin%> ] %RoutineMin%]
&if [type %RoutineMin%] ge 0 &then &do
&workspace %workspc%
&return &inform \\What part of "Number" did you not understand ?? \\
&end
&s RoutineMax := [response [quote Enter Routine Number to stop after: <return for %RoutineMax%> ] %RoutineMax%]
&if [type %RoutineMax%] ge 0 &then &do
&workspace %workspc%
&return &inform \\What part of "Number" did you not understand ?? \\
&end
&end /* end of DEBUG TRUE */
&if %RoutineMin% gt %RoutineMax% &then &do
&workspace %workspc%
&return &inform \\Start Number Set Higher Than End Number.\\
&end
&if %RoutineMax% lt 1 &then &do
&workspace %workspc%
&return &inform \\Enter a Number greater than zero.\\
&end
&s routlist :=
&do RoutineNum = %RoutineMin% &to %RoutineMax% &by 1
&if [variable routname%RoutineNum%] &then
&s routlist := %routlist% [value routname%RoutineNum%]
&end
&do RoutineNum = %RoutineMin% &to %RoutineMax% &by 1
&if [variable routname%RoutineNum%] &then &do
&if %RoutineNum% le %RoutineMin% &then
&type \\---> Executing: [value routname%RoutineMin%]
&else
&type ---> [value routname%RoutineNum%]
&end
&end
&type \\
/***********************************************************************************/
/************************* IF USER SELECTED ALL MAP SHEETS *************************/
/***********************************************************************************/
&if [substr [locase [extract 1 %mapsheetlist%]] 1 1] = a and ~
[type [substr [locase [extract 1 %mapsheetlist%]] 2 1]] ge 0 &then
&do
&messages &on; &type \Translating ALL !!\; &messages %MessageEchoStatus%
&s delfile := [delete /tmp/dxf.list -file]
&system ls ???.dxf | cut -d. -f1 > /tmp/dxf.list /* CREATE LIST OF ALL DXF FILES
&s dxf_file_list := [open /tmp/dxf.list openstat -read] /* READ LIST OF ALL DXF FILES
&if %openstat% eq 0 &then &do
&s mapsheet := [read %dxf_file_list% readstat]
&if %readstat% eq 0 &then
&do &until %readstat% ne 0
&do routname &list %routlist% /* FOR EACH ROUTINE NAME ...
&call %routname% /* CALL THAT ROUTINE **** HEART AND SOUL OF PROGRAM
&end /* end of do routname
&type \I'm Done with %mapsheet%.\
&system echo '0 0'
&system echo ' J '
&system echo '\_/ --> You probably should review my work.'
&type \
&s mapsheet := [read %dxf_file_list% readstat]
&end /* end of do until readstat ne 0
&end /* end of openstat eq 0
&else
&return &inform \\Could not open /tmp/dxf.list
&end
/***********************************************************************************/
/********************** IF USER DID NOT SELECT ALL MAP SHEETS **********************/
/***********************************************************************************/
&else
&do mapsheet &list [unquote %mapsheetlist%]
&workspace %workspc%/%ct_town%
&if NOT [exist %mapsheet%.dxf -file] &then
&type \Cannot Find File: %mapsheet%.dxf\
&else &do
&do routname &list %routlist% /* FOR EACH ROUTINE NAME ...
&call %routname% /* CALL THAT ROUTINE **** HEART AND SOUL OF PROGRAM
&end /* end of do routname
&type \I'm Done with %mapsheet%.\
&system echo '0 0'
&system echo ' J '
&system echo '\_/ --> You probably should review my work.'
&type \
&end /* end of else do
workspace %workspc%
&end /* end of do mapsheet list
&workspace %workspc%
&messages &on
&s closestat := [close -all]
&s delfile := [delete /tmp/dxf.list -file]
&return /* end of program
/***********************************************************************/
/******************************* ROUTINES ******************************/
/***********************************************************************/
&routine mapsheetlist /* USED FOR COMPILING LIST OF MAPS USER TYPED IN
&s j := 1
&s mapsheetlist := [unquote [value map%j%]]
&s j := [calc %j% + 1]
&do &until [null [value map%j%]]
&s mapsheetlist := %mapsheetlist% [unquote [value map%j%]]
&s j := [calc %j% + 1]
&end /* end of do until loop
&if %j% gt 10 &then &do
&type \\****************************************
&type Only processing first 10 grids typed in:
&type %mapsheetlist%
&type ****************************************\\
&end
&return /* end of mapsheetlist
/***********************************************************************/
&routine load_dxf_file
&if [exist %workspc%/%ct_town%/%mapsheet% -workspace] &then
&s deletework := [delete %workspc%/%ct_town%/%mapsheet% -workspace]
createworkspace %workspc%/%ct_town%/%mapsheet%
&workspace %workspc%/%ct_town%/%mapsheet%
&type \\Entering Workspace: %workspc%/%ct_town%/%mapsheet%
&type \\Loading %mapsheet%.dxf ROUND 1 - for lines and points
&messages %MessageEchoStatus%
&if [exist %mapsheet%_all -cover] &then
kill %mapsheet%_all all
dxfarc ../%mapsheet%.dxf %mapsheet%_all /* LOAD ALL ELEMENTS FROM DXF FILE
$REST
y
build %mapsheet%_all lines
build %mapsheet%_all points
build %mapsheet%_all anno.dxf
joinitem %mapsheet%_all.aat %mapsheet%_all.acode %mapsheet%_all.aat ~
%mapsheet%_all-id %mapsheet%_all-id linear
joinitem %mapsheet%_all.pat %mapsheet%_all.xcode %mapsheet%_all.pat ~
%mapsheet%_all-id %mapsheet%_all-id linear
dropitem %mapsheet%_all.aat %mapsheet%_all.aat
dxf-color
dxf-thickness
dxf-type
dxf-curve
end
dropitem %mapsheet%_all.pat %mapsheet%_all.pat
dxf-color
dxf-thickness
dxf-type
dxf-size
end
&messages &on; &type Loading %mapsheet%.dxf ROUND 2 - for text
&messages %MessageEchoStatus%
&if [exist %mapsheet%anno1 -cover] &then
kill %mapsheet%anno1 all
dxfarc ../%mapsheet%.dxf %mapsheet%anno1 /* LOAD TEXT ELEMENTS FROM DXF FILE
$REST textpoint textanno
y
build %mapsheet%anno1 point
build %mapsheet%anno1 anno.dxf
joinitem %mapsheet%anno1.pat %mapsheet%anno1.xcode %mapsheet%anno1.pat ~
%mapsheet%anno1-id %mapsheet%anno1-id linear
&messages &on; &type Loading %mapsheet%.dxf ROUND 3 - for attributes
&messages %MessageEchoStatus%
&if [exist %mapsheet%anno2 -cover] &then
kill %mapsheet%anno2 all
dxfarc ../%mapsheet%.dxf %mapsheet%anno2 /* LOAD ATTRIBUTE ELEMENTS FROM DXF FILE
060ANNO blocks atanno atpoint
027L016 blocks atanno atpoint
060L044 blocks atanno atpoint
066L054 blocks atanno atpoint
0 atpoint atanno
end
y
&if [exist %mapsheet%anno2 -cover] &then &do
build %mapsheet%anno2 point
build %mapsheet%anno2 anno.dxf
joinitem %mapsheet%anno2.pat %mapsheet%anno2.xcode %mapsheet%anno2.pat ~
%mapsheet%anno2-id %mapsheet%anno2-id linear
&end
&messages &on; &type Loading %mapsheet%.dxf ROUND 4 - for Contour \
&messages %MessageEchoStatus%
&if [exist %mapsheet%_cont -cover] &then