Saving space on your HD – null file compression in GRASS GIS 7.2

The GRASS GIS development team recently released a new stable major release, GRASS GIS 7.2. The release brings more than 1900 fixes and improvements since the previous stable release 7.0.5. You’ll find a detailed overview of all the changes and improvements on this GRASS wiki page.

One important library change in a GRASS library is support for NULL file compression using the r.null function. This may not sound terribly exciting to all of you, but for those that have GRASS databases with large (number of) raster layers, this may save considerable space on the hard disk.

For now, compression of the NULL file is not enabled by default. Instead it must be explicitly turned on. You can do this by running on the command line:

export GRASS_COMPRESS_NULLS=1

This will only set this environmental variable for the current session. If you want to turn it on permanently, you can add the line above to the bashrc file in linux or the env.bat in Windows:

$HOME/.grass7/bashrc # on Linux
$HOME/.grass7/env.bat # on Windows

For more information about setting GRASS GIS and environmental variables, see this GRASS wiki page.

NULL file compression can be managed with r.null and the -z flag. Note that after the NULL file of a raster layer is compressed, it can only be opened with GRASS GIS 7.2.0 or later!

r.null -z map=your_raster_layer

To avoid having to do this for each layer individually, you can run r.null over all raster layers in a mapset or location using a loop. The example below will compress the null files of all raster layers in the location LOC of the grass database GDB. You can alternatively add a loop to include all locations in database.

LOC=latlon
GDB=/home/paulo/Data/GRASSdb
g.mapset mapset=PERMANENT location=$LOC dbase=$GDB
MPS=`g.mapset -l`
for j in $MPS; do
    g.mapset mapset=$j location=$LOC dbase=$GDB
    MAPS=`g.list type=raster pattern=* mapset=$j`
    for i in $MAPS; do
        r.null -z map=$i
    done
done

To give you an example of how much this could save you, after running this on a database of 160 GB, the database was reduced to 116 GB (-27.5%). Another database of 118 GB was reduced to 90 GB (-24%). Of course, your millage may vary considerably depending on the type of data contained in your database.

Let me warn again though, after compressing your null files, you will only be able to open them in GRASS GIS 7.2 or higher! And although I assume this is safe, backing up your data before running is of course never a bad idea.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s