Ticket #138 (closed Task: fixed)

Opened 10 years ago

Last modified 8 years ago

Develop a Concept for Sustainable Keeping of Data

Reported by: JuergeN Owned by:
Priority: Major Milestone:
Component: DeepaMehta Standard Distribution Version: 4.0.5
Keywords: Cc: jri, dgf, Malte
Complexity: 3 Area:
Module:

Description (last modified by JuergeN) (diff)

As everyone knows we do not manage data import from one version to another, yet. This was a decision we all shared to make the development of major features faster and easier.

But the closer we get to an useful application, this will become a major caveat for people to really use DeepaMehta for anything else but testing and playing arround with it. Even worse, people might exidently have missed our notice and then get very sad and/or angry about it.

So we should look for ways to somehow enable people to keep their data from one version of DeepaMehta to another.

This could be done by an external script, exporting and importing data from one version to another or by auto-detection when starting deepamehta or what ever.

We will have to fix this - the sooner, the better!

If you have a ggod idea how to achieve this with lowest effort, then please write it down here.

Attachments

dm-core-interfaces.png (65.1 KB) - added by jri 10 years ago.
DeepaMehta Core Interfaces

Change History

comment:1 Changed 10 years ago by JuergeN

  • Description modified (diff)

comment:2 Changed 10 years ago by jri

DM already has a sophisticated migration facility. The core itself and every plugin keep track of the installed software version and perform migrations to bring the DB model and content up-to-date. This happens every time DM is started and when modules are updated.

However, writing the actual migration logic is up to the core/plugin developers and is a time consuming task. For the moment I have no resources to do this. Perhaps other developers can help.

Anyway, DM is possibly at a too early state to strive for content migration now. The DM core model is not even complete. (Completing the core model is planned for the 4.1 release, early 2012). The effort needed for migration logic is possibly better invested at the 4.1 release.

comment:3 Changed 10 years ago by JuergeN

I agree that the implementation of the aspired solution for a convenient content migration might be too time consuming for now. But never the less it also means that no one can use DeepaMehta for the moment. I think it is worthwhile exploring alternative scenarios like dumping and reimporting the data, for example, or any other useful simple and generic approach. I agree that you in person should not spend time for this now, but maybe other people come up with useful ideas, concepts or even solutions. Let's give it a try. I think the earlier we can achive the goal of sustainalble keeping of data, the better.


comment:4 Changed 10 years ago by Malte

One simple approach i could think of would consists of writing a plugin that imports just all topics from a previous dm instance based on it's REST interface because the REST interface is not so likely to change radically in between the upcoming releases.

Two implement this plugin for let's say all current Note-Topics and all current Webpage-Topics, plus all the relations between it wouldn't be that time consuming, if you don't need much of an interface for controlling such an import.

The plugin would operate on server-side (maybe just after activation of that bundle) and fetch at first all composite-topics of type "dm4.core.note", then iterate over the resultset and create a new Note-Topic on the new server for each item in the resultset, associating the respective "dm4.core.note.title" and the "dm4.core.note.description". The same for all Webpage-Topics, just with storing the values to other type-uris of related topics. One could even think of a kind of generic approach for storing more fields than those who are known for the import-plugin-developer if the topictype-uri, assuming the topictype-uris did not change between the releases on the table.

For keeping all the drawn associations it get's a bit more complicated, more and more per types for which you want to keep the associations drawn on instance level.

Keeping all topicmaps would add another dimension to the whole procedure.

comment:5 Changed 10 years ago by Malte

But as jri's says, the "concept for sustainable keeping of data", as the name of the ticket implies, is not only already present, it's implemented as a facility in the core platform. It just needs to be made use of per plugin, e.g. when a new version of the contacts plugin comes out, just write the migration class for the contacts plugin per release, that's the way it should be.

comment:6 Changed 10 years ago by jri

Yes, the DM migration facility is already there, and that's the way to go. A separate plugin and/or exporting all data at first would not save any work but would create more work and problems. Going the canonical path -- using the DM migration facility -- on the other hand let the migration author concentrate on the actual migration logic.

OK, lets assume there is a migration author who cares about the model and content migration. Writing a migration means coding a Java class according to a certain format. No deep Java knowledge is required here but knowledge about the DM concepts and API. That would be the API of the core service and the central domain objects.

For an example of a migration class look here:
https://trac.deepamehta.de/browser/deepamehta/deepamehta-webclient/src/main/java/de/deepamehta/plugins/webclient/migrations/Migration2.java
Within a migration class the author has access to the DM core service through the dms object.

Core service API:
https://github.com/jri/deepamehta/blob/master/deepamehta-core/src/main/java/de/deepamehta/core/service/DeepaMehtaService.java

Central domain objects (see attached class diagram):
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/DeepaMehtaObject.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/Topic.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/Association.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/Type.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/TopicType.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/AssociationType.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/AssociationDefinition.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/Role.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/TopicRole.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/AssociationRole.java
https://trac.deepamehta.de/browser/deepamehta/deepamehta-core/src/main/java/de/deepamehta/core/ViewConfiguration.java

Organizational the migration author must work hand in hand with the core author. Once a change to the data model is introduced the migration author would write a corresponding migration class. The migration classes are an integral part of a DM release.

Thank you, Jürgen, for pushing this issue!
Your persistence is appreciated :)

Last edited 10 years ago by jri (previous) (diff)

Changed 10 years ago by jri

DeepaMehta Core Interfaces

comment:7 Changed 10 years ago by jri

For an in-depth description of DeepaMehta's migration facility look here:
http://groups.google.com/group/deepamehta3/msg/a2a29bf176e464bd

Last edited 10 years ago by jri (previous) (diff)

comment:8 Changed 9 years ago by jri

Starting with DM 4.1 we will ensure content is migrated automatically to every future DM4 release. This applies to the DM core and all the plugins of the DM Standard Distribution.

Providing the migrations for 3rd-party plugins is the responsibility of the respective developers.

I think we can close this ticket now.

comment:9 Changed 8 years ago by jri

  • Status changed from new to closed
  • Resolution set to fixed

Since DM 4.1 (March 2013) we ensure data sustainability.

comment:10 Changed 8 years ago by jri

  • Milestone Sustainable Keeping of Data deleted

Milestone Sustainable Keeping of Data deleted

Note: See TracTickets for help on using tickets.