Skip to: Site menu | Main content

GeoTools

The Open Source Java GIS Toolkit

Image Mosaicing Pyramidal JDBC Plugin Print

Funded by Google as part of the GSOC 08 program.

This plugin is intended to handle large images stored as tiles in a JDBC database. Tiles created by pyramids are also stored in the database. The utility uses the indexing of databases to speed up access to the requested tiles, the plugin for itself has a multithreaded architecture to make use of dual/quad core CPUs and multiprocessor systems.

This plugin should give you the possibility to handle large images with their pyramids in a JDBC database.

Tested Database Systems

  • DB2 (with or without Spatial Extender)
  • Oracle (with or without Location Based Services, Oracle Spatial not needed)
  • MySql
  • Postgres (with or without Postgis)
  • H2

How to build image pyramids and tiles

Creating the pyramids and tiles is not part of this module. As a recommendation you can use http://www.gdal.org/gdal_retile.html which has been developed exactly for this requirement.

How to import the tiles and the georeferencing information

Importing the data is database dependent. If you have a spatial extension, the gdal_retile.py utility produces the proper shape or csv files which you can import with a database utility.

Outline of the multithreaded architecture

The multithreaded architecture is based on synchronized Queue available since SDK 5.0. The following threads will be used:

  • One ImageComposer thread which reads GridCoverages from the Queue, mosaicing the image and finally rescales the image for the requested dimension.
  • For each tile, a TileDecoder thread which decodes the tile, crops the tile if necessary und puts the result as GridCoverage in the queue.
  • The Main thread for controlling this threads

The Main threads performs the following steps in chronological order

If the requested CRS ist not equal to the CRS of the tiles, transform the requested CRS

  1. Calculate the best pyramid and query the tiles needed from the database
  2. Create the queue and start the ImageComposer Thread
  3. For each tile fetch the image data and immediately start an ImageDecoder Thread
  4. Wait for all ImageDecoder Threads
  5. Put a predefined End Message in the queue which causes the ImageComposer Thread to finish its work
  6. Wait for the ImageComposer Thread
  7. If the requested CRS ist not equal to the CRS of the tiles, reproject the image
  8. Return the GridCoverage

Database Design

First, we need a Meta Table for holding Information for each pyramid of the maps. This table has 9 columns as described below

  • Name (String): The name of the map
  • SpatialTable (String): The name of the table where to find the georeferencing information
  • TileTable (String): The name of the table where to find the tile data stored as BLOB.
  • ResX (Double): The resolution of the x-axis
  • ResY (Double): The resolution of the y-axis
  • MinX (Double): The minimum X of the extent
  • MinY (Double): The minimum Y of the extent
  • MaxX (Double): The maximum X of the extent
  • MaxY(Double): The maximum Y of the extent

Important:

The only fields you have to fill are Name,SpatialTable and TileTable. These three fields together are the primary key. All other fields will be calculated by the plugin. It is not necessary to do calculations. The plugin implements the following mechanism during the first request:

  1. Check if ResX or ResY is null, if true, calculate the values from a random chosen tile from the corresponding tile table.
  2. Check if one of MinX,MinY,MaxX,MaxY is null, if true, calculate the values from the corresponding spatial table.
  3. If something has been calculated, store these values into to db to avoid recalculation at the next restart

This approach assures that the probably costly calculation is only executed once, but updating,adding or removing tiles requires you to nullify one of the MinX,MinY,MaxX,MaxY Attributes to force a new extent calculation.

First example:

Assume we have a Map called Europe with 2 Pyramids. Since we have 3 levels, we need 3 tile tables and 3 spatial tables. Lets start filling the META table.

Name SpatialTable TileTable

ResX

ResY

MinX

MinY

MaxX

MaxY

Europe EUSPAT0 EUTILE0            
Europe EUSPAT1 EUTILE1            
Europe EUSPAT2 EUTILE2            

Remember: You do not have to fill the other attributes.

A tile table needs the following attributes:

  • Location (String) : A unique identifier for the tile
  • Data (Blob) : The image data (Encoded in a format readable by JAI, e. g. png,jpeg,...)

The table EUTILE0 for example ( 0x.... indicates binary data)

Location Data
Tile_1_1

0x10011...
Tile_1_2 0x00011..

A spatial table contains the georeferencing information. Depending on your database installation, there are 2 possibilites.

  1. You have no spatial db extension implies storing the extent in 4 Double fields
  2. Having a spatial extension offers the possiblity to use a geometry column

A spatial Table without using a spatial extension needs the following attributes:

  • Location (String) : A unique identifier used for joining into the tiles table
  • MinX (Double): min x of the tile extent
  • MinY (Double): min y of the tile extent
  • MaxX (Double): max x of the tile extent
  • MaxY (Double): max y of the tile extent

The table EUSPAT0 for example

Location MinX MinY MaxX MaxY
Tile_1_1 10 10

20

20

Tile_1_2 20 10 30 20

A spatial Table using a spatial extension needs the following attributes:

  • Location (String) : A unique identifier used for joining into the tiles table
  • Geometry (Geometry): A Multipoligon type provided by your db

The table EUTILE0 for example

Location Geom
Tile_1_1 0x00111...
Tile_1_2 0x10011...

Reducing the number of tables

Since the number of rows in a spatial table has to be equal to the number of rows in a tile table, there is a one to one relationship. It is no problem to join these tables. Applying this redesign to our samples results in the following table structure:

Name SpatialTable TileTable

ResX

ResY

MinX

MinY

MaxX

MaxY

Europe EU0 EU0            
Europe EU1 EU1            
Europe EU2 EU2            

The EU0 Table

Location MinX MinY MaxX MaxY Data
Tile_1_1 10 10

20

20

0x10011..
Tile_1_2 20 10 30 20 0x00011..

or using a geometry column

Location Geom Data
Tile_1_1 0x00111... 0x10011..
Tile_1_2 0x10011... 0x00011..

Further DB Design Rules

  • It is possible to add custom columns to your tables
  • It is possible to create one meta table for all your maps and pyramid levels , or create one meta table for each map containing records for the map and its pyramids, or do any mixture of these two approaches, the only rule is to store a map and its pyramids level together in the same meta table.
  • All primary key attributes will be handled as JDBC String, the exact DB Type and length depends on your needs.
  • All numerical fields will be handled as JDBC Double.
  • Tile image data has to be a BLOB

Mapping of table and attribute names

Since there a lot of naming conventions in different enterprises, I is not idea to force the use of predefined table and attribute names. In our example, the names of the spatial and tile tables is selectable, the name of the meta table and all the attribute names were assumed. The mapping of these names is part of the XML configuration. The corresponding XML fragment shows the mapping of the assumed names.

A sample XML fragment file

mapping.xml.inc

<!-- possible values: universal,postgis,db2,mysql,oracle -->
<spatialExtension name="universal"/>
<mapping>
        <masterTable name="META" >
            <coverageNameAttribute name="Name"/>
            <maxXAttribute name="MaxX"/>
            <maxYAttribute name="MaxY"/>
            <minXAttribute name="MinX"/>
            <minYAttribute name="MinY"/>
            <resXAttribute name="ResX"/>
            <resYAttribute name="RresY"/>
            <tileTableNameAtribute    name="TileTable" />
            <spatialTableNameAtribute name="SpatialTable" />
        </masterTable>
        <tileTable>
            <blobAttributeName name="Data" />
            <keyAttributeName name="Location" />
        </tileTable>
        <spatialTable>
            <keyAttributeName name="Location" />
            <geomAttributeName name="Geom" />
            <tileMaxXAttribute name="MaxX"/>
            <tileMaxYAttribute name="MaxY"/>
            <tileMinXAttribute name="MinX"/>
            <tileMinYAttribute name="MinY"/>
        </spatialTable>
</mapping>

The structure of this XML Fragment is kept very simple, use it as a pattern.

The name attribute of the <spatialExtension> has to be one of the following values

  • universal (a vendor neutral JDBC approach which should work with any jdbc database with BLOB support)
  • db2 (use spatial extender)
  • mysql (use the spatial features of mysql)
  • postgis (use the spatial features of postgis)
  • oracle (use location based services included in every oracle edition, no oracle spatial needed !!!)

If your spatial extension is universal you need to specify <tileMinXAttribute>,<tileMinYAttribute>,<tileMaxXAttribute>,<tileMaxYAttribute>, but you can omit <geomAttributeName>.

In all other cases, you need the <geomAttributeName> Element and can omit <tileMinXAttribute>,<tileMinYAttribute>,<tileMaxXAttribute>,<tileMaxYAttribute>.

This approach assures that there are no predefined table and attribute names, allowing seamless integration into existing naming conventions.

Configuration

Each Map with its pyramids is configured by an XML File. The Configuration can be split into 3 parts

  1. The database mapping as explained above
  2. The JDBC connect configuration
  3. The Map configuration

JDBC Connect configuration

Dependent on the deployment environment, there are different possibilities for connection to the database. Using this plugin in a client java application will require a JDBC Driver Connect, operation within a J2EE Container should prefer a JNDI Datasource connect (your admin will be happy) .

A sample XML fragment file for a JDBC Driver Connection to the H2 database

connect.xml.inc

<connect>
        <dstype value="DBCP"/>
        <username value="user" />
        <password value="password" />
        <jdbcUrl value="jdbc:h2:target/h2/testdata" />
        <driverClassName value="org.h2.Driver"/>
        <maxActive value="10"/>
        <maxIdle value="0"/>
</connect>

XML Fragment for connection to a JNDI Datasource

connect.xml.inc

<connect>
        <dstype value="JNDI"/>
         <jndiReferenceName value="jdbc/myDataSourceName"/>
</connect>








Map Configuration;

No we can put the things together and create the config file for a map.

Example file map.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE ImageMosaicJDBCConfig [
	<!ENTITY mapping PUBLIC "mapping" "mapping.xml.inc">
	<!ENTITY connect PUBLIC "connect" "connect.xml.inc">
]>


<config version="1.0">
	<coverageName name="oek"/>
	<coordsys name="EPSG:31287"/>
	<!-- interpolation 1 = nearest neighbour, 2 = bipolar, 3 = bicubic -->
	<scaleop interpolation="1"/>
	<verify cardinality="false"/>
	&mapping;
	&connect;
</config>

The DOCTYPE includes two XML entity references to the connct.xml.inc and mapping.xml.inc. The XML parser includes these fragments at parsing time.

Elements:

config

The root element. The version attribute is reserved for future versions of this plugin. Must be "1.0."

coverageName

The name of your map. This name is used for searching the meta data in the meta table.

coordsys

The name of a coordinate reference system, will be referenced by CRS.decode() of the geotools library.

scaleop

The interpolation method to use (1 = nearest neighbour, 2 = bipolar, 3 = bicubic)

verify

if you have image data and georeferencing information in different tables and the the attribute cardinality is true , the plugin will check the number of records in each table. If the numbers are not equal, the image/pyramid will be removed and you see a warning in the log. This check is intended for testing environments, set the value to false in production environments to avoid bad performance.

Configuration Summary

A map configuration consists of 3 parts, connect configuration, mapping configuration and map configuration. As a result, you have great flexibilty and can reuse parts of the configuration for many maps.

Setup Example

At the beginning we have a large image or a set of tiles which compose a large image. We will use http://www.gdal.org/gdal_retile.html to prepare our image and image pyramid tiles.

Prepare with gdal_retile.py

As a result, we have our tiles, csv or shape files ready to import into a database. The next step is to create the table for the meta info.

Attention: All the table and attribute names have to correspond to the xml mapping fragment, otherwise the plugin will not work. If there is a valid xml config, you can use the following utility to create ddl script templates.

Using the java ddl generation utility

Create the meta table

Next you have to choose on of the following options. The first one will work with any jdbc database and does not require a spatial extension. The other options are dedicated to selected databases using their spatial extension. The spatialExtension element in the xml mapping fragment is used to specify the option to use.

Prepare a jdbc database

Prepare db2 with spatial extender

Prepare for postgis

Prepare for mysql

Prepare oracle location based services

The next step is importing the data. Each database has its own utilities for loading bulk data, discussion is beyond the scope of this documentation. As an alternative, there is a java utility doing the job.

Using the java import utility

Last not least, we have to speed up access times.

Creating indexes for performance

Code Example

This example shows how to use the module.

// First, get a reader
// the configUrl references the config xml&nbsp; and is object of one of the following types
// 1) java.net.URL
// 2) java.io.File
// 3) java.lang.String (A filename string or an url string)


AbstractGridFormat format =(AbstractGridFormat)GridFormatFinder.findFormat(configUrl);
ImageMosaicJDBCReader reader= (ImageMosaicJDBCReader)format.getReader(configUrl,null);

// get a parameter object for a grid geometry
ParameterValue<GridGeometry2D>gg=AbstractGridFormat.READ_GRIDGEOMETRY2D.createValue();

// create an envelope, 2 Points, lower left and upper right, x,y order
GeneralEnvelope envelope = new GeneralEnvelope(new double[] {10,20},new double[] {30,40});

// set a CRS for the envelope
envelope.setCoordinateReferenceSystem(CRS.decode"EPSG:4326");

// Set the envelope into the parameter object
gg.setValue(new GridGeometry2D(new GeneralGridRange(new Rectangle(0, 0, width, heigth)),envelope));

// create a parameter Object for the background color (NODATA), this param is optional
final ParameterValue outTransp=ImageMosaicJDBCFormat.OUTPUT_TRANSPARENT_COLOR.createValue();
outTransp.setValue(Color.WHITE);

// call the plugin passing an array with the two parameter objects
try {
    GridCoverage2D coverage = (GridCoverage2D) reader.read(new GeneralParameterValue[] {
       gg, outTransp});
    } catch (IOException e) {
       e.printStackTrace();
    }