Geotools in gvSIG CE

From gvSIG CE Wiki

Jump to: navigation, search

Contents

Historical context

gvSIG is hard to develop. Association-gvSIG is making a "total refactoring" of the API by implementing their own new API. We're thinking about the opposite: a partial refactoring reusing some existing library. That existing library would be GeoTools. Technically, the key part for making this feasible is to keep a small scope, to keep the refactoring "partial". We're talking about changing the data access. But that may involve changing also geometry model, which may involve changing rendering, symbology, etc. That's the risk. We should try to reduce the refactoring to the data access as much as we can.

During the code sprint we will analyse its feasibility and specify a list of tasks, a roadmap, so this work could be done in a collaborative way.

This is a very important task for the future of gvSIG CE and also it can be a hard task from a technical point of view. Several technical discussions [1], [2], [3] have been started to study the possibility of this important change.

[1] gvSIG CE Developers List

[2] GeoTools User List

[3] gvSIG developers List

Following developers joined the discussions:

  • Fernando Gonzalez (Geomati.co)
  • Benjamin Ducke (gvSIG CE Developer)
  • Victor Gonzalez (Geomati.co)
  • Nacho Varela
  • Francisco Puga (Cartolab)
  • Gabriel Roldan (OpenGeo)
  • Andrea Aime (GeoSolutions S.A.S.)
  • Jody Garnett (LISAsoft)
  • Flavio Pompermaier
  • Fran PeƱarrubia (Software Colaborativo-Asociacion gvSIG)
  • Miguel Montesinos (Prodevelop-Asociacion gvSIG)
  • Cesar OrdiƱana (DiSiD Technologies-Asociacion gvSIG)
  • Cesar Martinez

Advantages

  • We no longer fully maintain a data access API, but collaborate with geotools community.
  • gvSIG developers may need to contribute to geotools to get their job done are seen in more projects
  • Also, developing for a GIS application that has geotools as their API may reduce the learning curve. There are also a lot of documentation in geotools.
  • Geotools also has many other API features, such as topology and map rendering. It would be a good idea for a small project like gvSIG CE to make use of the synergies that this can provide.

Disadvantages

  • we diverge from gvSIG 1.x branch which will make collaboration harder. Patches will no longer be compatible whenever they involve code using data access API.

Evaluation

  • the costs of migrating the extensions compatible with the 1.x branch. An adapter design pattern from the existing API to geotools could be implemented. This seems easiers than the migration of all the extensions of the branch 1.x
  • how to integrate the new GDAL/OGR Java API in this effort. imageio-ext wraps GDAL in a JAI structure, creates JAI ImageIO image readers around it and then builds coverage readers on top of that.
  • GeoTools (GT) relies heavily on JAI and there is a task about getting rid of JAI.

Some results of our research on integrating Geotools (GT)

After some work, we have managed to change the custom projection support in gvSIG for Geotools. Here we are going to discuss what have we done, the cost of the task in hours, what we have achieved and some issues that have arisen.

What we have already done

  • So far we have managed to substitute the custom gvSIG projection support with Geotools. This can be better viewed in the following diagrams that show the before and after status:
Before Geotools integration
After Geotools integration

Basically, we replace the following interfaces:

    • IProjection (gvSIG) -> CoordinateReferenceSystem (Geotools)
    • ICoordTrans (gvSIG) -> MathTransform (Geotools)

and we get rid of all the custom CRS, CRS factories and everything related with implementations. Also, in order to make the change as smooth as possible, a ProjectionUtils class has been created to delegate the tasks on Geotools and perform some tasks that were responsibility of gvSIG CRS related interfaces but don't exist in Geotools replacements.

All these changes can be checked out in the SVN repository [1][2][3][4].

What needs to be reviewed

After the changes, everything that needs to be reviewed is marked with:

// TODO geotools refactoring: <message>

Here is the list:

  • We have lost the ReferencingUtil.getTinAsFMapLyr method (it throws an UnsupportedOperationException) [5].
  • We have lost the IncrementalRectBoundsVoronoiStrategy.createTin method (it throws an UnsupportedOperationException) [6].
  • Renderer has slightly changed its behavior due to minor Geotools API changes [7].
  • On FLyrVect, the QuadtreeGt2 spatial index has been substituted with QuadtreeJts [8].
  • On VectorErrorTable we have lost an error message [9].
  • ViewPort.distanceWorld does not calculate the distance correctly [10].
  • The new ProjectionUtils class should be fully reviewed and maybe add some tests to check that it is correct [11].

What still may to be done

  • Correct and review the bugs/issues explained in the previous section.
  • Remove the ProjectionUtils class completely.
  • An adapter from gvSIG 1.x API to new GT API will be necessary. It includes:
    • IGeometry implementation using JTS geometries.
    • ReadableVectorial implementations that use GT DataStores.
  • Some GT DataStores must be implemented in order to support all the formats that gvSIG currently has but GT does not. A list should be done.
  • Replace the rendering and simbology implementation with Geotools.
  • Replace the several gvSIG query syntaxes with the Geotools one.
  • Replace the gvSIG geoprocessing framework with the Geotools implementation.
  • Integrate the OGR data stores provided by Geotools.

Integrate Geotools in gvSIG CE

The Munich codesprint was some time ago and we did not have the opportunity to devote more time to the Geotools integration since then. Finally we have been able to work on it for a week.

As stated from the start, the aim of the 2012 codesprint in Munich was not to finish the Geotools integration but to get an idea of how to tackle the task and estimate the cost of it (see our Release Planning in Development_and_releases for details on how we want to tie this into the CE release cycle).

First of all, for all that have not dealt with the gvSIG code base before: We have found that this task is not easy at all. The gvSIG 1.x branch (the one CE is based on) has hundreds of thousands of lines of code, organized in ~50 different projects that not only do not completely separate user interface from model but that are also tightly coupled with each other. Little or no documentation, duplicated code, meaningless variable names and 17.000 warnings makes handling the gvSIG source code a really discouraging experience. If you're a developer, I'm sure you get the idea.

After trying different approaches and restarting from scratch each time, we have finally found a method that has the following advantages:

  • It provides value immediately. We'll be able to produce gvSIG versions working on Geotools very soon. Of course, the initial versions will lack a lot of functionality the current gvSIG CE version has, but:
    1. everybody will be able to see the progress of the integration
    2. at a certain point in time, some people may be content with the status of the integration and start using it
  • It defines several more or less precise steps to follow. This is helpful when dealing with a beast of so many lines of code. The method is explained further down.
  • Can be done in parallel by several people. This is very important since this initiative has received some attention and I would say it is likely that someone will join and share the efforts. The development will take place in a GIT repository in github, so that forks and merges can happen easily.

This method has been the first one we have considered good enough to make the integration succeed. Of course, suggestions are welcome.

Let's explain the method. In gvSIG, there is a plugin system, called "andami" that hosts about 25 extensions that provide the user interface. These extensions use several libraries to implement their functionalities. Instead of modifying the existing code base, we have created one more extension project called "main" and one more library project called "core".

The main extension will receive one by one every functionality from the old codebase. As the libraries used by the old code are not present in main, there will be some compilation errors. To solve these compilation errors, the code that uses the old libraries will be changed to use the core library, adding the necessary functionality to core if necessary. Of course, core exposes Geotools API, as it was the main aim of the integration.

So schematically:

  • Move all extensions to main one by one
  • For each extension:
    • Include all classes in the same extension that are necessary to compile. Code can be removed if:
      • Supports old 0.3 version (method name ends with 03)
      • It is not necessary right now and we'll be forced to add it later. For example:
        • Public methods that are not called in main -> Either they will be called with code that will be moved to main and we'll recover the erased method, or they will never be called so it was well removed.
        • Whole classes -> At the end we can compare old and new codebase and we can detect the classes that are missing.
    • Fix the compilation errors of code using old libraries (not accessible anymore from main). This might require adapting the code to the new core library and even adding more functionality to it. In case of new functionality:
      • Discuss in the list the best way to implement
      • Implement
      • Add javadoc
      • Add unit tests

Some considerations when following the method.

  • Our first priority is integrating Geotools. Please don't try to fix the extension code in main. While following the previous method you will see some things that can easily be fixed. Please don't. You'll realize that there are many of these and fixing them will keep you from the real aim, which is the integration. Let's try to do only minimal changes to the code that goes to main.
  • Do not add TODOs. They are ignored. assert false:"todo message"; is much more effective. If you don't want to implement a method, try to remove it (following the rules stated before).
  • If you add some comment related to the integration, include the text "gtintegration" in it, so that we can process it later.
  • Please, try to understand core before making changes. Just use the mailing list to agree on some changes before starting to code.

Methodology

Please read this post in our blog to know more about the methodology we are proposing. This method has been the first one we have considered good enough to make the integration succeed. Of course, suggestions are welcome:

Cost estimation of the remaining tasks

Replacing gvSIG projection library with Geotools

  • Cost estimation: around 30 hours.
  • Some tasks are still to be done, as explained in the previous sections.
  • We may have introduced bugs and this cost does not take the technical debt into account. It may even double the cost of the already done work.

FAQ