Keeping Users' Data Current--Using Geoprocessing Models & Python ...
Keeping p g Users’ Data Current Using Geoprocessing Models & Python Scripts
Michael Hickey Tulare County RMA/GIS [email protected] 3/3/2009
1
The Problem … z
Automate Keeping USERS’ Data Current – Network is too slow to support ‘heavy GIS users’ direct from server users – ‘Tiny stack’ of GIS data copied to local computers (as ‘Drive V:’) – Cannot rely on users to copy data from server
z
Frequently Updated GIS Data – – – –
3/3/2009
PARCELS (revised weekly) Zoning (typically monthly) Address Points (under ‘active development’) LAFCO cases ((typically yp y monthly) y) 2
Overview @GIS z
GIS staff keeps ‘core data’ current on server *
z
Microsoft s ‘Task Microsoft’s Task Scheduler Scheduler’ executes Python scripts at night
z
Python script compares date of FC on server with date of FC on GISmph – If GISmph current, do nothing – If GISmph NOT current current, • Run geoprocessing model on data • Copy revised data to server
3/3/2009
3
Overview @GIS User z z z
When user turns computer ON in morning, a Python script is executed P th script Python i t compares data d t off FC on server with ith data of FC on GISlocal For EACH FC in process: – If GISlocal current, do nothing – If GISlocal NOT current, • Copy revised data from server
3/3/2009
4
Address Points … After each editing session: z Assign point to proper spatial districts (TAZ, census block, block PcnqPt, PcnqPt APN) z Apply “MapMarker” to force addresses to US Postal standards z Evaluate how well each address meets UP Postal standards
3/3/2009
5
Address Points … GP model
3/3/2009
6
Parcels & Parc_Lbls z
Each week the PARCELS theme is revised to match revisions to PIMS and Assessor’s Map … (this is an exact copy of PIMS database)
z
Each time a new PARCELS theme is released, a new Parc_Lbls theme is generated: – [APN_dash] and other fields are reformatted – Standard land use codes are added
z
Future Enhancements: – Zoning, AG G preserve info f – GIS estimate of use/APN for parcels ‘under review’ by assessor (if a parcel is ‘under review’, assessor tends to g [use] [ ] = ‘9000’ and BLANK ALL other data)) assign
3/3/2009
7
Parc_Lbls … GP model
3/3/2009
8
Zoning z z z
Tulare County has 109+/- zoning categories … un-map-able! ‘G ‘Generalized li d Zoning’ Z i ’ (24 categories t i … colors l similar to Land Cover and Planned Uses) Requires adding [gen [gen_zone] zone] field
3/3/2009
9
Zoning … GP model
3/3/2009
10
Old Tricks … retained z
DOS subst command and Network ‘Map p Drive’ z DOS .bat file – Execute program with parameters – Copy files
3/3/2009
11
Python Tricks applied to GP_models and/or scripts Exec BAT file (with wait) z eMAIL (signal success or failure) z Copy C FC tto ZIP file fil z Push ZIP file to FTP site z LOG files to record processing time/status ==================================== z Read CSV file to p pGdB table z Write Excel to pGdB table z Write pGdB table to Excel z
3/3/2009
12
Failure Modes (part one) Just because it’s worked for six months does not mean it will work tomorrow!
z z
Scratch file has 2 gig limit (and deleting does NOT compact!) Schema Lock (Novell) – If someone has file ‘open’ it cannot be copied/updated – DOS copy does d NOT give i error message
z
Schema Lock (ESRI) – If y you ‘touch’ table in catalog, g, GP model called by script may fail…
z
Table Joins (if file name changes)
3/3/2009
13
Failure Modes (part two) Just because it’s worked for six months does not mean it will work tomorrow!
z
Update from ArcGIS 9.2 to 9.3 – If it worked in 9.2, it will continue to work in 9.3 (until it is opened/viewed which causes a ‘recompile’)
z
ArcGIS 9.3:Table Joins – MUST have index on join field – MUST unJOIN after using JOIN
z
ArcGIS 9.3:Copy Files
3/3/2009
14
Python Resources
3/3/2009
15
Looking for Collaborators z
Programming / GP models are ‘crystallized’ intelligence (or lack thereof)
z
How can I share my stuff with you? y
z
How can I get your stuff?
FOR MORE INFO...
[email protected] 3/3/2009
16
Michael Hickey Tulare County RMA/GIS [email protected] 3/3/2009
1
The Problem … z
Automate Keeping USERS’ Data Current – Network is too slow to support ‘heavy GIS users’ direct from server users – ‘Tiny stack’ of GIS data copied to local computers (as ‘Drive V:’) – Cannot rely on users to copy data from server
z
Frequently Updated GIS Data – – – –
3/3/2009
PARCELS (revised weekly) Zoning (typically monthly) Address Points (under ‘active development’) LAFCO cases ((typically yp y monthly) y) 2
Overview @GIS z
GIS staff keeps ‘core data’ current on server *
z
Microsoft s ‘Task Microsoft’s Task Scheduler Scheduler’ executes Python scripts at night
z
Python script compares date of FC on server with date of FC on GISmph – If GISmph current, do nothing – If GISmph NOT current current, • Run geoprocessing model on data • Copy revised data to server
3/3/2009
3
Overview @GIS User z z z
When user turns computer ON in morning, a Python script is executed P th script Python i t compares data d t off FC on server with ith data of FC on GISlocal For EACH FC in process: – If GISlocal current, do nothing – If GISlocal NOT current, • Copy revised data from server
3/3/2009
4
Address Points … After each editing session: z Assign point to proper spatial districts (TAZ, census block, block PcnqPt, PcnqPt APN) z Apply “MapMarker” to force addresses to US Postal standards z Evaluate how well each address meets UP Postal standards
3/3/2009
5
Address Points … GP model
3/3/2009
6
Parcels & Parc_Lbls z
Each week the PARCELS theme is revised to match revisions to PIMS and Assessor’s Map … (this is an exact copy of PIMS database)
z
Each time a new PARCELS theme is released, a new Parc_Lbls theme is generated: – [APN_dash] and other fields are reformatted – Standard land use codes are added
z
Future Enhancements: – Zoning, AG G preserve info f – GIS estimate of use/APN for parcels ‘under review’ by assessor (if a parcel is ‘under review’, assessor tends to g [use] [ ] = ‘9000’ and BLANK ALL other data)) assign
3/3/2009
7
Parc_Lbls … GP model
3/3/2009
8
Zoning z z z
Tulare County has 109+/- zoning categories … un-map-able! ‘G ‘Generalized li d Zoning’ Z i ’ (24 categories t i … colors l similar to Land Cover and Planned Uses) Requires adding [gen [gen_zone] zone] field
3/3/2009
9
Zoning … GP model
3/3/2009
10
Old Tricks … retained z
DOS subst command and Network ‘Map p Drive’ z DOS .bat file – Execute program with parameters – Copy files
3/3/2009
11
Python Tricks applied to GP_models and/or scripts Exec BAT file (with wait) z eMAIL (signal success or failure) z Copy C FC tto ZIP file fil z Push ZIP file to FTP site z LOG files to record processing time/status ==================================== z Read CSV file to p pGdB table z Write Excel to pGdB table z Write pGdB table to Excel z
3/3/2009
12
Failure Modes (part one) Just because it’s worked for six months does not mean it will work tomorrow!
z z
Scratch file has 2 gig limit (and deleting does NOT compact!) Schema Lock (Novell) – If someone has file ‘open’ it cannot be copied/updated – DOS copy does d NOT give i error message
z
Schema Lock (ESRI) – If y you ‘touch’ table in catalog, g, GP model called by script may fail…
z
Table Joins (if file name changes)
3/3/2009
13
Failure Modes (part two) Just because it’s worked for six months does not mean it will work tomorrow!
z
Update from ArcGIS 9.2 to 9.3 – If it worked in 9.2, it will continue to work in 9.3 (until it is opened/viewed which causes a ‘recompile’)
z
ArcGIS 9.3:Table Joins – MUST have index on join field – MUST unJOIN after using JOIN
z
ArcGIS 9.3:Copy Files
3/3/2009
14
Python Resources
3/3/2009
15
Looking for Collaborators z
Programming / GP models are ‘crystallized’ intelligence (or lack thereof)
z
How can I share my stuff with you? y
z
How can I get your stuff?
FOR MORE INFO...
[email protected] 3/3/2009
16
