Open Satellite Data
Satellite data is an extremely expensive data source, if you need world-wide coverage at hi-resolution.
Fortunately, there is free hi-resolution data available with world-wide coverage from the
But the latter data is not readily available without a time-consuming pre-processing step called pan-sharpening, which requires expensive geo-spatial processing software and lots of patience.
This project is about establishing a free and ready-to-use version of the above data sources
by providing a free implementation of the pan-sharpening technique.
Here is a glimpse on the result:
And a glimpse on the full data set (please be patient, the full earth coverage takes a while to load):
Acquiring the SRTM Data
First we take care of the SRTM data. That’s no big deal. SRTM data with a resolution of 90m is only about 18GB at world-wide coverage. The latest version 4.1 has established a consistent data quality and is easily downloaded in a couple of days from http://srtm.csi.cgiar.org. The data is mirrored here:
http://schorsch.efi.fh-nuernberg.de/data/terrain/SRTM/v4/ as of 2021 the server is no longer existent unfortunately
The above data is available in GeoTIFF format and is ready to be imported into GDAL, VTP Builder, QTViewer or other common GIS software for the purpose of geo-spatial visualization.
If you are using that data, please acknowledge the data source srtm.csi.cgiar.org and the NASA as original provider.
Landsat Data
Now we take care of the Landsat data, which is a different kind than SRTM. The best freely avaialble data source for Landsat7 data is the EarthSat version available through GLCF. It is already ortho-rectified, but still provided in bands so that there is no true-color version available out-of-the-box.
True-color versions exist but the data providers make a fortune out of selling those although they are just a processed version of otherwise freely available data that has been remotely sensed by the NASA with US government funding. So it does not appear sound to pay for something that has been funded by the public. Access to the public Landsat data should be completely free to the public!
This is why we are going to prepare and publish a free true-color version of the original data.
So first we are going to download all necessary bands of the EarthSat Landsat 7 data from the GLCF server. Then we are going to process the data with libGrid to yield naturally looking images with world-wide coverage at 15m resolution. This processing step is known as pan-sharpening and color-matching. More about those processing techniques in the libGrid tutorial.
Acquiring the Landsat Data
The Landsat data cannot be downloaded manually, since it is more than 1.5 terabyte of compressed data. Even through a good internet connection with a consistent bandwidth of say 1MB/s the download takes more than one month. In practice, I get a download speed of about 400kb/s from the GLCF server.
Therefore, we clearly need to automate the Landsat data download. We are am using a shell script for that purpose. It loops over all bands of the Landsat images that are available on the GLCF server and downloads them one by one with the GNU wget tool.
The following script does the job:
#!/bin/tcsh -f
# pull GLCF Landsat7 EarthSat imagery
# by Stefan Roettger
# ftp path = ftp://ftp.glcf.umd.edu/glcf/Landsat/WRS2
# l7 paths = p001..p233
# l7 rows = r001..r122
# l7 dir = <path><row>_7x<date>.ETM-EarthSat-Orthorectified
set ftppath = "ftp://ftp.glcf.umd.edu/glcf/Landsat/WRS2"
set ftprate = 1024000
set wget = /opt/local/bin/wget
# World coverage
set l7path1 = 1
set l7path2 = 233
set l7row1 = 10
set l7row2 = 100
set p=$l7path1
while ($p <= $l7path2)
set r=$l7row1
while ($r <= $l7row2)
set l7path=p$p
set l7row=r$r
if ($p < 100) set l7path=p0$p
if ($r < 100) set l7row=r0$r
if ($p < 10) set l7path=p00$p
if ($r < 10) set l7row=r00$r
echo "checking path=$l7path row=$l7row"
if (! -e $l7path$l7row'_'nn80.tif.gz) then
$wget -timeout 10 "$ftppath/$l7path/$l7row/$l7path$l7row*.ETM-EarthSat-Orthorectified" >&/dev/null
ls $l7path$l7row*.ETM-EarthSat-Orthorectified >&/dev/null
if ($status == 0) then
set l7subpaths=`ls $l7path$l7row*.ETM-EarthSat-Orthorectified`
rm -f $l7subpaths
foreach l7subpath ($l7subpaths)
end
foreach ext (nn10 nn20 nn30 nn40 nn80)
$wget --limit-rate=$ftprate --timeout=60 "$ftppath/$l7path/$l7row/$l7subpath/$l7path$l7row*_$ext.tif.gz"
mv -f $l7path$l7row*_$ext.tif.gz $l7path$l7row'_'$ext.tif.gz
chmod -x $l7path$l7row'_'$ext.tif.gz
end
endif
endif
@ r++
end
@ p++
end
At the moment of writing (May/24 2013) the script has already downloaded 1.2 terabyte of EarthSat Landsat 7 data in about 5 weeks. So there is still a couple of weeks to go to complete the download.
The compressed bands 1,2,3 and 4 and the panchromatic band 8 are available on my mirror server at:
http://schorsch.efi.fh-nuernberg.de/data/terrain/Landsat/EarthSat
All GeoTIFF images are compressed with the GNU zip tool. The images are named
p<path>r<row>_nn<band>.tif.gz
For example the panchromatic band 8 of path 63 row 46 has the file name:
p063r046_nn80.tif.gz
Compressing the Landsat Data
The Landsat data is available in a lossless compressed format. One patch with compressed bands 1,2,3,4 and 8 is 234MB on the average. There are roughly 8000 patches covering the entire world. This amounts to about 1.5 terabyte in total.
While there is the general rule to store original data with lossless compression, the data is so huge that considering the option of a slightly lossy compression makes sense.
One patch of band 8 is about 230MB of uncompressed and about 80MB compressed data. When considering lossy compression, we want to gain just a slight additional compression ratio of about 1:2, so that the impact on image quality is minimal. For those low compression levels, the JPEG format is a viable option. The GEOTIFF format supports the so called JPEG-in-TIF option where GEOTIFF is used as a container for embedding a JPEG image while keeping the geo-referencing information, which is otherwise lost in pure JPEG.
The following table shows image quality vs. compression ratio with the JPEG-in-TIF option of the GeoTiff library of GDAL for path 63 row 46 band 8 showing Big Island of Hawai’i (crop area 1000×1000+8500+8500):
| Compression | Size | Compression ratio | Relative ratio | Image quality |
|---|---|---|---|---|
| none | 233.0MB | original | ||
| LZW | 83.0MB | 1:2.77 | original | |
| JPEG 95 | 27.4MB | 1:8.50 | 1:3.07 | almost as crisp as original |
| JPEG 90 | 16.0MB | 1:14.6 | 1:5.25 | fine details are blurred, less visible contrast |
| JPEG 80 | 8.8MB | 1:26.5 | 1:9.55 | too much detail loss |
It appears that JPEG compression level 95 is the best compromise to gain a compression ratio of 1:3 while loosing just a bit of image crispness compared to the original data. The JPEG compression even smoothes out some graininess of the original data.
Supposed we want to save a single geotiff image file in the JPEG-in-TIF format, we compress it on the Unix command line by running the gridcopy tool of the libGrid library:
gridcopy image.tif image.jpgintif 95
Saving Compression Time
The compression of a single patch with bands 1,2,3,4 and 8 to the JPEG-in-TIF format takes about 7 minutes.
There are roughly 8000 Landsat patches covering the entire world. Extrapolating the time to compress that amount of data yields 40 days of computing time. On a dual-core processor this goes down to three weeks. Ok, that will not finish over night but it seems reasonable for the huge amount of world-wide data.
To parallelize the conversion we use the -j option of GNUMake, which is able to run multiple make jobs concurrently.
The following Makefile converts all images with the ending .tif.gz in the SRCDIR directory to JPG compressed images with the ending .jpgintif in the DSTDIR directory. It assumes that the libGrid gridcopy tool is available and located in the TOOLDIR directory:
SRCDIR = EarthSat DSTDIR = compressed TOOLDIR = ~/Projects/libgrid/grid/tools TOOL = gridcopy TOPTS = 95 TIFS = $(wildcard $(SRCDIR)/*.tif.gz) JPGS = $(patsubst $(SRCDIR)/%.tif.gz,$(DSTDIR)/%.jpgintif,$(TIFS)) all: output $(JPGS) %.jpgintif: @echo processing $*.jpgintif @-gunzip -c $(patsubst $(DSTDIR)/%,$(SRCDIR)/%,$*).tif.gz >$*.tif @-$(TOOLDIR)/$(TOOL) $*.tif $*.jpgintif $(TOOLOPT) @-rm -f $*.tif output: @-if [ ! -e $(DSTDIR) ]; then mkdir -p $(DSTDIR); fi
Running that Makefile with 4 parallel jobs on a quad core processor with
make -j 4
is estimated to bring down computing time to less than 2 minutes per patch and less than 2 weeks in total.
Landsat Data Summary
After about 9 weeks of downloading the last Landsat patch arrived on 24/16/2013 at my server.
| Total patches | 7866 |
| Total compressed | 1.5 TB |
| Total uncompressed | 4.9 TB (estimated) |
Landsat 7 ETM+ coverage:
+-+-+-+-+---------------------------++-+++--------------+-+--+--+-+++--+-+-+-+-+++-+-+-+-+-++++++-+-+++++++-++-+-+-++++-+-++-++-----------+-+---------+-+-++--++-+----------++--++++-+-+-++++-+++++-++-+-+-+-+-+-+-+------+-+-++++-+--++- +++++-+-+------------------------+-++-+-+--+--++--------++-+-++-+-+-++-+-+-+-+-+++-+-+-+-++-+-+++-+-++-+-++-++-+-++-+++-+-+++++-++-++-++--+-+--------+--+-+++-+-++-++-+-+-+-++-++-++-+++-+-++-++-++-++-+-+-++-++--+-+-------+-+-++++--++- -+-++++--------------------------+-++-+-+--+-+++-+-+--+-+-++--+-+++-+-++-+++-+-+++-+-++++-+-+-+-+-+-++-+-+-+++-++++-+-+-+-+-+-+-++-++-+++-+-+-++------+-+-+++-+-++-+-+-++-+-+-+-++-++-++-+-+-+-++-++-+-+-+-+-++++-+-+-+-+-----+-+-++-+-++ ++-+------+-+-++------------------+++++-++-+++-++-++-++-++-+-++++++-+-++-+++++-+++-+-++++-+-+-+++-+-++++-+-+++-+-++-+-+-+-+-+-++++-++++++++++-++-++-+-+-+++++++-+-+++-+-+++-+-+-++-+++-+-+-+-+-++-++-+-+-+-+-++-+-+++-+-+-+-----+-++-+-++ ++-+------+-+++-++----------------++-++-++-+-+-+++++-++-++++-+-+-+-++-+++++-++-++-++-++++-+-+-+-+++-++++-+-+++-+-++-+-+-+-++++++-+++-+++++++-+-+-++-+++-+-+-+++++-+++-+-+-++++++++++++-+-++-+-+++-++++++++-++-+-+-+-+-+-+-+-----++++-+-+- ++----------+++-++--------------+-++-++-++++-+-++-++++++++-+-+-+-++++-+-+++++-+++-++-++++++-+-+-++-+++++-+++++-+-++++++-+-++-+-+++++-+++-+++++-++--+++--+-+-++-++-+++++-+++++-+-+-++-+-++++-+-+-+++++-+++-+-+-+-+-+-+-+-+++--------+-+-+- +-------------+++---------------+-++-++-++++-++++-+-+-++-+-+++-++++-+-+-++-+-+++-+++-++++-+-+++-++-+-+++-+-+-+++++++++-+++++-++++-++-++++++++++++----+++--+-++++-++++-+-+-+-+-+-+-++-+-+-++-+-+-+++-+-+-+++-+-+-+-+-+-+++++--------+-+-+- +-------------------------------+-+++++-++-+-++-+-+-+-++-++++-++++-++++-++++-+++-+-+-++-+++++++-++-+-+++-++++++-++-+++-++-++-++++-++-+++-++++++-----------+-++-+-+-++-+-++-++-+-++++++-+-++-+-+-+++-+-++-+------+++-+-+-++++---------+-+- ++--------------------------+---++++++++++-+-++++-+-+-+-++-++-+-++-++++-+++-++++-+-+-++-+++-+-++++++++++++++++++++++-+-++++++++++++--+++-+++----------+---+-++++-+-++-+-+++++++-+-++-+-++++-+++-+++-++++-+---------++++-++-+++----------+ -------------------------++++---+++++++++++++++++++++++++++++++-++++++++++-+++++-++++++++++++++++++++++++++++++-++++-+++++-+++-+++++-++++--------------------+-+-+-++-------+++++++-++++++-++++-+++-++-+-++---------+++++++-+++---------- -------------------------+++++-----++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------++++++----------------+-----+-++++----------++++++++++++++++++++++++++++++++----++++++++++++++-------- -------------------------++++++-----++++++-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------+++++-+-------------------+-++-+------------++++++++++++++++++++++++++++++++++++++++++++++++++------- -------------------------+++++++----++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------+++++--++----------------+++++----------------+++++++++++++++++++++++++++++++++++++++++++++++++++----- -------------------------+++++++++-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------++++-------++------+++++++--------------------+++++++++++++++++++++++++++++++++++++++++++++++++++---- -------------------------+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------+++-----------++++++++--------------------------+-++++++++++++++++++++++++++++++++++++++++++++++++--- -----------------------------++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----++------------------------------------------------+++++++++++++++++++++++++++++++++++++++++++++++++-- -----------------------------++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----+---------------------------------------------------++++++++++++++++++++++++++++++++++++++++++++++++- ------------------------------+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----++----------------------------------------------------++++++++++++++++++++++++++++++++++++++++++++++++ --------------------------------+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---++------------------------------------------------------+++++++++++++++++++++++++++++++++++++++++++---- ---------------------------------+++++++++++++++++++++-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--++++++-------------------------------------------------------+++++++++++++++++++++++++++++++++++++++++++---- ----------------------------+++++++++++++++++++++++++------+++++++-+++++++++++++++++++++++++++++++++++++++++++++++++++++++---+++++---------------------------------------------------------++++++++++++++++++++++++++++++++++++-++------- ----------------------------+++++++++---+++++++++++++++++++++++++++-++++++++++++++++++++++++++++++++++++++++++++++++++++-----+++-----------------------------------------------------------++++++++++++++++++++++++++++++++++++---------- -----------------------------+++++++++--++-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------++------------------------------------------------------------+++++++++++++++++++++++++++++++++++----------- ---------------+++-----------+++++++++--++-++++-+++++++++++++++++++--++++++++++++++++++++++++++++++++++++++++++++++++++------++-------------------------------------------------------------++++++++++++++++++++++++++++++++------------- ------------------++---------+++++++-+++++++++--+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---+++--------------------------------------------------------------+++++++++++++++++++++++++++++++------------- -------------------------------++++++++++++-++----+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--+++-++++++--------------------------------------------------------------+++++++++++++++++++++++++++++++------------- -------------------------------+++++++++++++-------++----++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--++++++++++---------------------------------------------------------------++++++++++++++++++++++++++++++------------- -------------------------+----++++++++++++++++---+++-------++++++++++++++++++++++++++++++++++++++++++++++++++++++++--++++++++-+-----------------------------------------------------------------+++++++++++++++++++++++++++-------------- ------------------------------++++++++++++++++++-++++++-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----++----++-----------------------------------------------------------------+++++++++++++++++++++++++---------+----- --------------------------+---+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---++-----+------------------------------------------------------------------++++++++++++++++++++++++--------------- -------------------------++-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---++--------------------------------++-------------------------------------+-++++++++++++++-+++--+++--------------- -------------------------+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----+-------++--------------------------+--------------------------------------+++++++++++++--------+++++------------ ----------------------------++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-++-+------+----------------------------++-------------------------------------+++++++++++---------+++++----------- ---------------------------++++++++++++++++++++++++++++++++++++++++++++++++--++++++++++++++++++++++++++++++++++++++++++---+------+-------+---------------------------------------------------------+--++++++++++---------++++++---------- ---------------------------++++++++++++++++++++++++++++++++++++++++++++++++++-----+++++++++++++++++++++++++++++++++++----------------------------------------------+--+--------------------------------+++++++++--------++++++++--------- ---------------------------++++++++++++++++++++++++++++++++++++++++++++++++++-----+++++++++++++++++++++++++++++++---++-------------------------------------------------+++--------------------------------+++++++---++++++++++++++------- ---------------------------++++++++++++++++++++++++++++++++++++++++++++++++++-------+++++++++++---+++++++++++++--+--++--------+-----+------------------------------------+++-------------------------------+++++++-+++++--+++++++++------ ---------------------------++++++++++++++++++++++++++++++++++++++++++++++++++--------+++++++++-----++++++++++++-----++---------------+------------+-----------------------++-------------------------+-+---+++++++++++++--++-+++++++++++- ----------------------+-+---++++++++++++++++++++++++++++++++++++++++++++++++----------++++++++-----+++++++++++++----+++--------------+----------------------------+------------------------------------------+++++++++++-+---++-++--++++- -----------------------++---++++++++++++++++++++++++++++++++++++++++++++++------------+++++++-------++++++++++++---++++--------------+--------------------------------------------------------------------------+++++++++++-++--------+-+ -----------------------++--++++++++++++++++++++++++++++++++++++++++++++++--------------+++++-------+--+++++++++----+++++-------------+--------------+---------------------------------------------------------------+++++++-+-----------+ +---------------------------+++++++++++++++++++++++++++++++++++++++++++--++-----------++++++-------++-+++++++++------++++------------+--------------------------------------------------------------------------------+++++++----+++----+ +----------------------------+++++++++++++++++++++++++++++++++++++++--+++++-----------+-++++-------+--+++++++++--++-++++++----------------------+-++++------------------------------------------------------------------+++----++++++++++ +-----------------------------++++++++++++++++++++++++++++++++++++++++++++------------++-++++------+---++-+++++--++-++++++-------++-------------+--+-++---------------------------------------------------+--------------+++-+-++++++++++ ++-----------------------------++++++++++++++++++++++++++++++++++++++++++--------------+-++++-------+--+++-+++--+-+++++++++----+-+-+-+-+-++-+------++-+-------------------------------------------------------------------+++++++++++++++ +++----------------------------++++++++++++++++++++++++++++++++++++++++++--------------+---+++------++-++++--------++++++++----+----+++++-+++-++-----+-+--------------------------------------------------------------------+++++++++++++ +++++++--------------------------++++++++++++++++++++++++++++++++++++++++--------------++---++--------++++++-------++++++++---+------------++-+-+-+--+++----------------+------------------------------------------------+-----++++++++++ ++++++++--------------------------+++--+----++++++++++++++++++++++++++++---------------++-------------++++++--+---+++++--++---+--------------+-------+-------------------++-------------------------------------------------+--++++++++++ +++++++++-------------------------------------++++++++++++++++++++++++++----------------+-------------++++++++++++++++---++++++-------------------------+----------------------------------------------------------------------++++++++++ +++++++++------------------------------------+-++++++++++++++++++++++++-----------------+--------------++++++++++++++++++++++-+-+------------+----------++-----+------------+-----------------------------------------+-------+++++++++++ +++++++++++----------------------------------+-+++++++++++++++++++++++------------------+---------------++++++++++++++++++++++++++-------------------+---++-----+---------+--------------------------------------------+------+++++++++++ +++++++++++++--------------------------------+-++++++++++++++++++++++------------------------------------+++++++++++++++++++++++++++-+++++++-----------+--++-----------------------------------------------------------++-----+++++++++++ +++++++++++++++++------------------------------++++++++++++++++++++++-------------------------------------++++++++++++++++-+++++++++++++++++++-------------++------+----------------------------------------------------------+++++++++++ ++++++++++++++++++--+----------------------------+++++++++++++++++++--------+++----------------------------++++-+-++++++++-++++++++++++++-++++++-+---------------++++---------+-----------------------------------------------+++++++++++ ++++++++++++++++++++-----------------------------+++++++++++++++++++--------++----------+-------------------+++++++++++++++-++++++-++++++++++-++-+----------+-----------------+-----------------------------------------------+++++++++++ ++++++++++++++++++++------------------------------+++++++++++++++++++-------+-+---------++--------------------++++++++++++++++++-+-++++++++-+-++++----------+++-------------------------+--------------------------------------++++++++++ ++++++++++++++++++++-------------+----------------+++++++++++++++++++------+-------------------------------------+++++++++++++++---+++++++++++++++++---------++----++--------+----------++--------------------------------------+++++++++ ++++++++++++++++++++-------------------------------+++++++++++++++++++--++-----+-------------------------------+-------+++++---+------++-++++++---++-++-------++-----------+------+------++-------------------------------------+++++++++ +++++++++++++++++++--------------------------------+++++++++++++++++++++-++------------------------------+----------------+----+++++--++----+++---+---+-+------------+--+--------+-----------------------------------------------++++++++ +++++++++++++++++++--------------------------------+++++++++++++++++++-++-++----------------------------------------------++---+++++--+++-------------++-----+----+-+-----+-------------------------------------------------------+++++++ +++++++++++++++++++--------------------------------+++++++++++++++++++---++++---------------------------------------------++++++++++--++++-------------++-------++--++++------------+++-+-+---------------------------------------+++++++ +++++++++++++++++++--------------------+-----------+++++++++++++++++++--+++++-+---+----------------------------------------+++++++++++++++-++----------++------++-+-+-----------+-+-++++++-----------------------------------------++++++ +++++++++++++++++++--------------------------------+++++++++++++++++++-+++++------+-------------------------------------+-+++++++++++++++++++--------+--+-----++++--------------+++++-++++-+-----------------------------------------++++ +++++++++++++++++++--------------------------------+++++++++++++++++----++++----------------------------------------------++++++++++++++++++------+--+---+----++++-++--+---+-++---------+++++------------------------------------------++ +++++++++++++++++++------+--------------------------+++++++++++++++-----++++-----+---+--------------------------------+++++++++++++++++++++++-----+---+++++----+-+-++---------++-------++-++-------------------------------------------++ ++++++++++++++++++-----------------------------------+++++++++++++++--+-++++----+-------------------------------------++++++++++++++++++++++++-+++-+---+++-+-----+-++--------+-+-+--------++++-----------------------------------------++ ++++++++++++++++++------------------------------------++++++++++++++----++++------------------------------------------+++++++++++++++++++++++++---------++-+-------+--------------+++-------+-+----------------------------------------++ ++++++++++++++++--------------------------------------++++++++++++++----++++------------------------------------------+++++++++++++++++++++++++++-------------------------------------+----------++-------------------------------------+ ++++++++++++++----------------------------------------+++++++++++++------++-------------------------------------------+++++++++++++++++++++++++++----------------------------------------------------------------+----------------+-----+ ++++++++++++++-----------------------------------------++++++++++++---------------------------------------------------+++++++++++++++++++++++++++---------------------------------------++---------------------+------------------------+ ++++++++++++++-----------------------------------------++++++++++++----------------------------------------------------++++++++++++++++++++++++++---------+--------+--------------------------------------------------------------------+ ++++++++++++++------------------------------------------++++++++++------------------------------------------------------+++++++++++++++++++++++++------------------+--------------------------------------------------------------------+ ++++++++++++++-------------------------------------------+++++++++------------------------------------------------------+++++++++++++++++++++++++----+----------------------------------------------------------------------------------+ +++++++++++++--------------------------------------------++++++++-------------------------------------------------------++++++++----+++++++++++++----------------------------------------------------------------------------------++---+ ++++++++++++----------------------------------------------++++++--------------------------------------------------------+++++++------++++++++++++-------------+-------------------------------------------------------------------------+ +++++++++++----------------------------------------------------------------------------------------------------------------------------++++++++++-------------+++-----------------------------------------------------------------------+ +++++++++++----------------------------+---------------------------------------------------------+---------------------------------------++++++++--------------++++---------------------------------------------------------------------+ ++++++++++---------------------------------------------------------------------------------------+----------------------------------------+++++----------------++++---------------------------------------------------------------------+ ++++++++---------------------------------+--------------------------------------------------------------------------------------------------++++---------------++++---------------------------------------------------------------------- ++++++++-------------------------------------------------------------------------------------------------------------------------------------+++--------------++++----------------------------------------------------------------------- +++++++---------------------------------------------------------------------------------------------------------------------------------------++-------------++++------+----------------------------------------------------------------- +++++++------------------------------------------------------------------------------------------------------------------------------------------------------++++------+----------------------------------------------------------------- ++++++-------------------------------------------------------------------+-------++-------------------------------------------------------------------------++++------------------------------------------------------------------------- +++++++------------------------------------------------------------------+-----------------------------------------------------------------------------------++-------------------------------------------------------------------------- +++++++---------------------------------------------------------------------------------------++----------------------------------------------------------------------------------------------------------------------------------------- +++++++---------------------------------------------------------------------------------------++-------------------------------------------------------------+-------+------------------------------------------------------------------- -+++++----++++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --+++++-----+-------------------------------------------------------------------------------------+-------------------------------------------------------------+------------------------------------------------------------------------ ---+++++++----------------++-------------------------+---------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------- -------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
A couple of the orthorectified EarthSat patches had missing or broken data:
- missing meta data:
- p037xr037, p195xr050
- corrupted band archive:
- p059xr073, p131xr020, p158xr027
Those broken EarthSat patches have been replaced with the corresponding GLS2000 (non-orthorectified) version.
- For example p037xr037 from
- has been replaced with
Compressing the Landsat Data (Continued)
We are giving it a try now. I am running the above conversion to 95% JPEG on a Ubuntu 13.10 box with 200GB SSD and an extra 1TB data drive. It has an Intel Xeon E3–1275 V2 quad-core cpu clocked at 3.5GHz and 32GB of ram.
Since compressing the entire data is taking too long for testing, we use a subset for Europe. That’s the Landsat paths 180–210 and the rows 10–50 with a total of 906 patches and 188GB of data to process.
We start compressing with two parallel jobs on 11/04/2013 11:22 …
While processing the data, we monitor the cpu temperature with the “sensors” tool of the lmsensors package. We get a consistent temperature of 75${}^\circ$ with almost no fan noise noticable. Good, we can let it run…
Processing finished on 11/05/2013 13:19 after approx. 26 hours.
That’s 207 seconds per patch for a single processor (for bands nn10/nn20/nn30/nn40/nn80).
Total compressed data is 75G vs. 188GB of uncompressed data. Compression ratio is 1:2.5 with an average of 85MB vs. 212MB per patch.
Color-Mapping the Landsat Data
Now we take care of the conversion of the separate satellite bands to true-color rgb imagery.
As outlined in the previous sections, we use the pan-sharpening tool of libgrid to produce the true-color imagery. Since the data contains a black border, which needs to be masked as no-data, we also use the gridcopy tool to set the no-data value of the imagery to zero.
In order to speed up the true-color imagery processing, we run the following Makefile in parallel:
SRCDIR = EarthSat DSTDIR = true-color TOOLDIR = ~/Projects/libgrid/grid TOOL = tools/gridcopy TOPTS = 0 MERGER = merger MOPTS = --quiet --tilesize=0 SRCS = $(wildcard $(SRCDIR)/*_nn80.tif.gz) DSTS = $(patsubst $(SRCDIR)/%_nn80.tif.gz,$(DSTDIR)/%.tif,$(SRCS)) all: $(DSTS) %.tif: output @echo processing $* @-gunzip -c $(patsubst $(DSTDIR)/%,$(SRCDIR)/%,$*)_nn80.tif.gz >$*_nn80.tif @-gunzip -c $(patsubst $(DSTDIR)/%,$(SRCDIR)/%,$*)_nn40.tif.gz >$*_nn40.tif @-gunzip -c $(patsubst $(DSTDIR)/%,$(SRCDIR)/%,$*)_nn30.tif.gz >$*_nn30.tif @-gunzip -c $(patsubst $(DSTDIR)/%,$(SRCDIR)/%,$*)_nn20.tif.gz >$*_nn20.tif @-gunzip -c $(patsubst $(DSTDIR)/%,$(SRCDIR)/%,$*)_nn10.tif.gz >$*_nn10.tif @-$(TOOLDIR)/$(TOOL) $*_nn80.tif $*_nn80.tif $(TOPTS) @-$(TOOLDIR)/$(TOOL) $*_nn40.tif $*_nn40.tif $(TOPTS) @-$(TOOLDIR)/$(TOOL) $*_nn30.tif $*_nn30.tif $(TOPTS) @-$(TOOLDIR)/$(TOOL) $*_nn20.tif $*_nn20.tif $(TOPTS) @-$(TOOLDIR)/$(TOOL) $*_nn10.tif $*_nn10.tif $(TOPTS) @-$(TOOLDIR)/$(MERGER) $*_nn80.tif\ $*_nn40.tif\ $*_nn30.tif\ $*_nn20.tif\ $*_nn10.tif\ $*.tif $(MOPTS) @-rm -f $*_nn*.tif @echo finished processing $* output: @-if [ ! -e $(DSTDIR) ]; then mkdir -p $(DSTDIR); fi
We let the conversion to true-color imagery run with 4 jobs, one for each processor of a quad-core cpu:
make -j 4
We start the conversion on 11.Nov.2013 19:40 … Uups, not enough space left on the drive, we restart the conversion on 21.Nov.2013 19:18 …
Conversion finished on 26.Nov.2013 21:03 after about 6 days. That’s about 30 minutes per patch.
Indexing the Landsat Data
Now that we have nicely processed true-color imagery of entire Europe, we realize that its too much data to navigate through by filename. Therefore, we generate a thumbnail gallery for the entire Landsat matrix.
First, we use ImageMagick to generate the thumbnails:
#!/bin/tcsh -f
set src = true-color
set dst = thumbs
# World coverage
set l7path1 = 1
set l7path2 = 233
set l7row1 = 10
set l7row2 = 100
# which images to thumbnail
set ext = "tif"
set sfx = ""
echo "creating thumbnails for Landsat 7 ETM+ imagery:"
if (! -e $dst) then
mkdir $dst
endif
set r=$l7row1
while ($r <= $l7row2)
set p=$l7path2
while ($p >= $l7path1)
set l7path=p$p
set l7row=r$r
if ($p < 100) set l7path=p0$p
if ($r < 100) set l7row=r0$r
if ($p < 10) set l7path=p00$p
if ($r < 10) set l7row=r00$r
set l7file=$src/$l7path$l7row$sfx.$ext
if (-e $l7file.gz) set l7file=$l7file.gz
if (-e $l7file) then
echo -n "+"
set l7thumb1=$dst/$l7file:t:r.small.jpg
set l7thumb2=$dst/$l7file:t:r.large.jpg
if (! -e $l7thumb1 || ! -e $l7thumb2) then
if ($l7file:e == "gz") then
gunzip -c $l7file | convert - -quiet -quality 95 -geometry 32x32 $l7thumb1
gunzip -c $l7file | convert - -quiet -quality 95 -geometry 2048x2048 $l7thumb2
else
convert $l7file -quiet -quality 95 -geometry 32x32 $l7thumb1
convert $l7file -quiet -quality 95 -geometry 1024x1024 $l7thumb2
endif
endif
else
echo -n "-"
endif
@ p--
end
echo ""
@ r++
end
Second, we produce a HTML table that contains the Landsat thumbnails as a matrix indexed by path and row:
#!/bin/tcsh -f
set src = thumbs
# World coverage
set l7path1 = 1
set l7path2 = 233
set l7row1 = 10
set l7row2 = 100
# which images to index
set ext = "jpg"
set sfx = ""
echo "<html>"
echo "<head>"
echo "<title>Index of Landsat ETM+ imagery:</title>"
echo "</head>"
echo "<body>"
echo "<table border=1>"
set r=$l7row1
while ($r <= $l7row2)
set p=$l7path2
echo -n "<tr>"
while ($p >= $l7path1)
set l7path=p$p
set l7row=r$r
if ($p < 100) set l7path=p0$p
if ($r < 100) set l7row=r0$r
if ($p < 10) set l7path=p00$p
if ($r < 10) set l7row=r00$r
set l7thumb1=$src/$l7path$l7row$sfx.small.$ext
set l7thumb2=$src/$l7path$l7row$sfx.large.$ext
echo -n "<td>"
if (-e $l7thumb1 && -e $l7thumb2) then
echo -n "<a href="$l7thumb2"><image width=32 src="$l7thumb1"></a>"
endif
echo -n "</td>"
@ p--
end
echo "</tr>"
@ r++
end
echo "</table>"
echo "</body>"
echo "</html>"
Here is how the resulting HTML thumbnail matrix of entire Europe looks like:
Please note that this work and the above true-color satellite imagery is licensed under a Creative Commons-Attribution-NonCommercial 3.0 License.
Please also note that the attribution of the original Landsat data is “Landsat imagery courtesy of NASA Goddard Space Flight Center and U.S. Geological Survey” or “USGS/NASA Landsat” for short.
Topo-Mapping the Landsat Data
As outlined in the Croatia Showcase we can also use the Landsat data to produce topographic maps instead of true-color maps. Here is how the resulting thumbnail matrix of the topography of entire Europe looks like:
Please note that this work and the above topo-graphic satellite imagery is licensed under a Creative Commons-Attribution-NonCommercial 3.0 License.
Please also note that the attribution of the original Landsat data is “Landsat imagery courtesy of NASA Goddard Space Flight Center and U.S. Geological Survey” or “USGS/NASA Landsat” for short.
Color-Mapping the Landsat Data (Continued)
The test run for processing Europe Landsat data worked well. It took about one week of processing time. For the entire world, however, processing time is estimated to take about 2 months. But before tackling the entire world, we need to fix a flaw of the data.
Fixing the Landsat Data
After processing the Europe subset, it appeared that some patches were darker than others.
An example for that flaw are the two patches p193r030_nn10 and p193r031_nn10 (covering part of Korsika). One is bright, the other one is much darker.
The .met files, which accompany each set of Landsat bands, state different sun elevation angles for the two Korsika patches with 57 and 63 degrees (p193r030.met and p193r031.met). But that alone cannot account for the large variation in brightness.
When looking closer at the .met files, it came out that the patches were normalized from their original radiance values to so called calibrated digital number images (QCAL DN in the notation of the USGS). Since different patches have different radiance range, the calibrated range will stretch differently after normalization looking darker or brighter.
As a consequence, we need to reverse the normalization. Fortunately, the original radiance values are given in the .met files, so we just need to apply those values as a scaling factor to the calibrated original data.
I added support for the Landsat band radiance correction to the libgrid library so that it tries to read the appropriate scaling factors from the .met files in order to re-adjust the intensities. Those adjusted intensities are called at-sensor reflectances.
The scale and bias factors applied are defined as follows (for normalized reflectances in the range [0..1]):
For more information about the calculation of the at-sensor reflectances see the GRASS GIS manual.
Note: The .met file for path 037 and row 037 is broken on the GLCF server (it contains zero bytes). For that patch we can’t compute at-sensor reflectances.
Color-Mapping the Landsat Data (Continued)
Here it goes again. We recompute the entire subset of Europe with the corrected at-sensor reflectances, to see if that finally works out well.
Started recomputation on 20.2.2014 3pm … Finished recomputation on 24.2.2014 11pm. Significantly faster than the first run, probably because I forgot to compile in release mode the first time.
And was it worth all the tweaking? Yes, the brightness of all Landsat true-color composites is now perfectly balanced! The same holds for the topographic Landsat composites:
http://schorsch.efi.fh-nuernberg.de/data/terrain/Landsat.html http://schorsch.efi.fh-nuernberg.de/data/terrain/Topomap.html
Producing World-Wide Data
To produce world-wide data an estimated disk space of 6TB is required. Since there is not that much space left on the external drives of my server, I am currently waiting for a 16TB Synology DS414 NAS with 4 Western Digital WD40EFRX “Red” 4TB drives to be shipped.
Producing World-Wide Data (Continued)
Got my 16TB Synology DS414 NAS shipped today (18.6.2014).
Producing World-Wide Data (Continued)
The Synology DS414 is still sitting on the table waiting to be unpacked after the exams (16.7.2014).
Producing World-Wide Data (Continued)
The Synology NAS is now up and running (28.7.2014).
Configuring the NAS (30.7.2014)
To configure the Synology NAS, we go to the web interface at http://<ip-address>:5000 and log in as “Admin” with an empty password. Then we follow these steps:
- Enter the EZ Internet Assistant.
- Enter and confirm the network settings.
- In the firewall settings allow NFS (Mac/Linux file server).
- Enter the system settings.
- In the “Services” section, enable the NFS file service.
- In the “Shared Folders” section, create a new shared folder, e.g. “nas”.
- Edit the folder settings
- Edit the NFS priviledges of the folder
- Enable NFS priviledges for the local sub net, e.g. “141.75.33.*”.
- Enable read/write priviledges
- Edit the NFS priviledges of the folder
- The mount point of the shared folder is “/volume1/nas”.
- Edit the folder settings
- If everything worked well, we override the default admin password.
Mounting the NAS on MacOS X 10.8 (31.7.2014)
Here is a nice description how to establish access to the NAS on MacOS X 10.8:
A short summary of the above tutorial:
/etc/autofs.conf:
AUTOMOUNTD_MNTOPTS=nosuid,nodev,resvport
/etc/auto_master:
/Synology /etc/auto_nfs
/etc/auto_nfs:
nas -fstype=nfs,rw,bg,hard,intr,tcp <ip-address>:/volume1/nas
sudo automount -vc
As far as I can tell, the above approach does not work on Mac OS X 10.6 Leopard.
To work around that I enabled AFP on the Synology NAS. Then we add a user with access to the shared folder and mount the shared folder with the following command line:
sudo mount -t afp afp://user:passwd@<ip-address>:548/volume1/nas /Volumes/Synology
Mounting the NAS on Linux (6.8.2014)
sudo mkdir /nas sudo mount -t nfs -o vers=3 <ip-address>:/volume1/nas /nas sudo chmod a+x /nas
If everything went fine, the output of “df /nas/” should look like the following:
Filesystem 1K-blocks Used Available Use% Mounted on 141.75.33.250:/volume1/nas 11614605056 204160 11614298496 1% /nas
Mounting the NAS on Linux permanently (6.8.2014)
Modify /etc/fstab by adding the following line:
<ip-address>:/volume1/nas /nas nfs defaults,vers=3 0 0
Producing World-Wide Data (Finally)
The production of true-color imagery from world-wide Landsat data is now running on a 4 core Xeon.
Summing up: Each Landsat patch has been downloaded by a wget script to my GLCF mirror server at http://schorsch.efi.fh-nuernberg.de/data/terrain/Landsat/EarthSat/ (7866 patches with 1.5TB). For each Landsat path the required bands are transferred via ssh from my GLCF mirror server to a 4 core Xeon Linux box. Then the bands are merged with the libgrid merger tool. Merging is scheduled in parallel using a parallelized GNU Makefile. And finally the merged true-color imagery is uploaded to the NAS, which is mounted per NFS.
Estimated run time for all 7866 Landsat patches is 5 weeks starting on 8/11/2014 for a total a 5TB of true-color imagery.
Producing World-Wide Data (Finally Continued)
Roughly half-way through on 8/28/2014.
Producing World-Wide Data (Finally Continued Again)
Finished on 9/14/2014 after 33 days of number crunching with 5.6 TB of produced data.
Producing World-Wide Data (Finally Continued Yet Again)
Now we mount the NAS data on the web server. On a MacOS X 10.6.8 server we do so by adding the following line to /etc/fstab:
<nas-ip-address>:<nas-nfs-export-path> /<local-mount-point> nfs defaults,vers=3,resvport 0 0
On my server “schorsch.efi.fh-nuernberg.de” this corresponds to:
141.75.33.250:/volume1/nas /nas nfs defaults,vers=3,resvport 0 0
Finally, we create the mount point
sudo mkdir /nas
and update the automounts:
sudo automount -v
Here’s a thumbnail overview of all the Landsat data (please be patient while the page loads, it’s 7866 thumbnails on a single HTML page ;-) :
Fin!