Downloading, Managing, and Utilizing Large Shapefile Folders

Uses: CygWin, ArcCatalog/ArcMap v.10

The Download:

Government data files from agencies such as NOAA are often placed online for download. Although, accessing a website to manually download the files is sometimes preferred, if there are too many files, then the process can be automated to alleviated any struggles.

Writing a wget script for Cygwin will allow you to download files directly from the webpage. This will place the downloaded files into a folder once the process is completed.


                                                                                                    Sample wget:

wget -c -r -l1 -nH -np --timestamping --cut-dirs=1

--accept="shapefile_$(date +%Y%m%d)*"

url of file-url of file-url of file-url of file-url of file 

                                                                                                                                    Sample 2:

wget -c -r -l1 -nH -np --timestamping --cut-dirs=1

--accept="shapefile_$(date --date yesterday +%Y%m%d)*"

url of file-url of file-url of file-url of file-url of file-url of file-url of file

Managing Tar Ball Data:

Windows Explorer is capable of managing large amount of files in a downloaded bundle, however the sorting process overloads the capability of the computer, which might cause a delay or overheating.

Instead of using Windows Explorer, use Cygwin to move the files. Here are some Cygwin steps in order to move these.

1.       First, Create a folder directory with individual folders for organization (years, months, and days)

2.       Next, using Cygwin, locate the downloaded file’s location:


3.       Once you’ve located the location of the files, (enable advanced editing in the settings) copy and paste an identifying feature of the file’s name (shapfile_YYYYMMDD) and paste it within the command below

Mv (paste)* [folder destination name] 

Mv shapefile_20110121* 01

4.       There should be a short pause (depending on how many files meet your search criteria) after pressing enter and the files have been relocated, the prompt will be responsive for the next command.


ESRI Management of ShapeFiles:


1.       In ArcCatalog create a folder connection for these files.

2.       View the folder in the Catalog, and use the merge tool for each day of shapefiles in the folder. This will make the data easier to project onto a geographic coordinate system, as well as easier to load as a layer in ArcMap. (Arc Toolbox> Data Management Tools> General> Merge.) You can either select all of the files in the Contents menu, and drag them into the box, or you can follow the folder directory to the files location (because of your great [yet tedious] organization of the files by day)

3.       Use the define projection tool to define the projection as WGS 1984, UTM of the desired zone. (Arc Toolbox> Data Management Tools> Projections and Transformations> Define Projection)

4.       Open ArcMap, and add the data from the merged shapefile set, as well as a 1984 WGS topographic map, if it’s in the projection.

5.       Open the attribute table of the shapefile merge and choose ‘select by attribute’

6.       In order to narrow down the scope of the data, geographically, input a query such as latitude and longitude.

Gulf of Mexico Data Query:

“LON”>-96 and “LON”<-82 and “LAT”>18 and “LAT”<30

7.       Export the data as a new layer, saved in the file where the ASCAT data files are, and named as something that is easily found, such as DDMMYYYYGoM

8.       This data can further be processed by changing the symbology associated with the shapefiles. Double Click the layer in the table of contents, and go to the symbology tab. Click the symbol button>Style references> check weather. Use a simple weather arrow symbol, since we are working with relative wind data. Then go back to the symbology tab, and click advanced>rotation>set the parameters to DIR>OK>OK. Now the data actually makes sense.

9.       This data can be matched to the correct time frame, in order to derive information to get an idea of what oceanic wind conditions (surface roughness) were present at the time.