Storage layer performance: associations should be indexed

[jri]

With "real" application content, e.g. 1000 Person topics, retrieval has proven too slow.
This is mainly when traversing supernodes. This happens e.g. when loading the definition of a type which have e.g. 1000 instances. You have to traverse all 1005 associations then to find the 5 association definitions.

The supernode problem is described in this article (thanks to dgf for the link):
​http://java.dzone.com/articles/solution-supernode-problem

The solution is to index the associations.

[Jörg Richter]

MehtaGraph? Neo4j: add test cases (#389).

See ticket 389.

• Changeset: ef09edc8f64a453e591d4313b439e364fd808cf8
• URL: ​https://github.com/jri/deepamehta/commit/ef09edc8f64a453e591d4313b439e364fd808cf8

[Jörg Richter]

Core: refactor storage layer, pt.6 (#389, #391).

Introduce association index.
Does not yet compile.

See ticket 389.
See ticket 391.

• Changeset: a276f355b8db82cb3443a4e31412678450a2f883
• URL: ​https://github.com/jri/deepamehta/commit/a276f355b8db82cb3443a4e31412678450a2f883

[Jörg Richter]

Core: association metadata index (#389, #338).

The association metadata index basically works. Both, indexing and query. Update is pending.

Core is not yet functional.

Furthermore, in order to determine an object's type the new storage layer doesn't rely on "Instantiation" associations anymore. Instead a "type_uri" property is stored for every object (= Neo4j Node). (The "Instantiation" associations remain to exist).
This provides the basis for solving #338 (An Association's "Instantiation" Association has no type assignment).

Furthermore in Parent POM: we explicitely use an older Surefire version (2.12.4) in order to avoid an exception logging problem introduced in Surefire 2.13

See ticket 389.
See ticket 338.

• Changeset: 7610f63f22fe66a9b1c0c66c5513e5414abd568a
• URL: ​https://github.com/jri/deepamehta/commit/7610f63f22fe66a9b1c0c66c5513e5414abd568a

[Jörg Richter]

Core: bidirectional assoc indexing (#389, #391).

The new association metadata index can be queried "from both sides".
See ticket 389.

First tests of the Neo4j storage implementation are adpapted to the new storage layer API.
See ticket 391.

• Changeset: cff74ad681a040a4646af0e3c04d2b16dd20270c
• URL: ​https://github.com/jri/deepamehta/commit/cff74ad681a040a4646af0e3c04d2b16dd20270c

[Jörg Richter]

Core: bidirectional assoc indexing (#389, #391).

The new association metadata index can be queried "from both sides".
See ticket 389.

First tests of the Neo4j storage implementation are adpapted to the new storage layer API.
See ticket 391.

• Changeset: cff74ad681a040a4646af0e3c04d2b16dd20270c
• URL: ​https://github.com/jri/deepamehta/commit/cff74ad681a040a4646af0e3c04d2b16dd20270c

[Jörg Richter]

Core: fix bidirectional assoc index (#389, #393).

For the first time the server can start with the new storage layer.
Use the "assoc-index" branch.
(Try multiple if "bundle not found" is thrown, see 2) below.)

Pending:

1) Adapt the Webclient to the changed assod def format (#393).
2) Provide storage implementation as OSGi service.
3) Update assoc metadata index on delete and retype.

BREAKING CHANGES

The semantics of the Association Definition URI has changed. Now this is straight the URI as derived from the underling Association. So, it does *not* reflect the assoc def's part type URI (= "child type") anymore, but is usually empty (see #393).

When operating on an assoc def you must replace

    assocDef.getUri()

by

    assocDef.getPartTypeUri()

This is in particular important for genric operations on composite values, e.g.:

    composite.put(assocDef.getPartTypeUri(), childTopic.getModel());

See ticket 389.
See ticket 393.

• Changeset: bda1119bcb65202d7f446b2d35473bfd1b43f77f
• URL: ​https://github.com/jri/deepamehta/commit/bda1119bcb65202d7f446b2d35473bfd1b43f77f

[Jörg Richter]

Core: fix bidirectional assoc index (#389, #393).

For the first time the server can start with the new storage layer.
Use the "assoc-index" branch.
(Try multiple if "bundle not found" is thrown, see 2) below.)

Pending:

1) Adapt the Webclient to the changed assod def format (#393).
2) Provide storage implementation as OSGi service.
3) Update assoc metadata index on delete and retype.

BREAKING CHANGES

The semantics of the Association Definition URI has changed. Now this is straight the URI as derived from the underling Association. So, it does *not* reflect the assoc def's part type URI (= "child type") anymore, but is usually empty (see #393).

When operating on an assoc def you must replace

    assocDef.getUri()

by

    assocDef.getPartTypeUri()

This is in particular important for genric operations on composite values, e.g.:

    composite.put(assocDef.getPartTypeUri(), childTopic.getModel());

See ticket 389.
See ticket 393.

• Changeset: bda1119bcb65202d7f446b2d35473bfd1b43f77f
• URL: ​https://github.com/jri/deepamehta/commit/bda1119bcb65202d7f446b2d35473bfd1b43f77f

[Jörg Richter]

Storage fix: update assoc index on delete (#389).

BREAKING CHANGES

Core API: rename methods

    RelatedTopic.getAssociation()                         ->  getRelatingAssociation()
    RelatedTopicModel.getAssociationModel()               ->  getRelatingAssociation()
    RelatedAssociationModel.getRelatingAssociationModel() ->  getRelatingAssociation()

See ticket 389.

• Changeset: 7c6fe75131ed185cc9a3552c54101d2b3f1762ea
• URL: ​https://github.com/jri/deepamehta/commit/7c6fe75131ed185cc9a3552c54101d2b3f1762ea

[Jörg Richter]

Core: refactor some storage code (#389).

In preparation of "update association metadata index on retype".

New in DeepaMehtaStorage? interface:

• void storeTopicTypeUri(long topicId, String topicTypeUri)
• void storeAssociationTypeUri(long assocId, String assocTypeUri)

Not yet functional.

See ticket 389.

• Changeset: 84f92f75244e960b2a191d3dee55289d20c2dcfe
• URL: ​https://github.com/jri/deepamehta/commit/84f92f75244e960b2a191d3dee55289d20c2dcfe

[Jörg Richter]

Core: update assoc index on assoc retype (#389).

Done:

• update the association metadata index when an *association* is retyped.

Pending:

• ... when an *association role* is retyped.
• ... when a *topic* is retyped.

See ticket 389.

• Changeset: ab8200aa62a506a56c0a1a7064151d9b330a4cdd
• URL: ​https://github.com/jri/deepamehta/commit/ab8200aa62a506a56c0a1a7064151d9b330a4cdd

[Jörg Richter]

Neo4j storage: update index on role retype (#389).

The association metadata index is updated when an *association role* is retyped.

Pending:

• ... when a *topic* is retyped.

=> Building composite types works for the first with the new storage layer.

BREAKING CHANGES

The association metadata index format has slightly changed.
Existing DBs *can* be used without modification. However, roles of existing associations can *not* be retyped.
Only roles of new associations can.

Resetting the DB is recommended.

See ticket 389.

• Changeset: 2c443c571903a2633355097be76727de6fba1663
• URL: ​https://github.com/jri/deepamehta/commit/2c443c571903a2633355097be76727de6fba1663

[Jörg Richter]

Storage: store type URI of retyped topic (#389).

The association metadata index is not yet updated when a topic is retyped.

=> retyping a Topic does not work for the moment.

Further refactoring is required: The retype code must be moved from storage decorator to application layer.

See ticket 389.

• Changeset: 64581b8379240ef74bd3feddf9002f3a450bcdf1
• URL: ​https://github.com/jri/deepamehta/commit/64581b8379240ef74bd3feddf9002f3a450bcdf1

[Jörg Richter]

Neo4j storage: update index on topic retype (#389)

The association metadata index is updated when a topic is retyped.

Still pending:

• when an association is retyped all associations this association is a player in, must be re-indexed.

See ticket 389.

• Changeset: 44db47ff84d54957753106e1af82329a07934201
• URL: ​https://github.com/jri/deepamehta/commit/44db47ff84d54957753106e1af82329a07934201

[Jörg Richter]

Neo4j storage: update assoc metadata index (#389).

When an association is retyped all associations the association is a player in, are re-indexed.

=> For the first time the new storage layer should be complete now.
Minor issues may remain and will be fixed very soon.

See ticket 389.

• Changeset: c4206f43b406c62b68671c4f2b2af4c5b18a0cea
• URL: ​https://github.com/jri/deepamehta/commit/c4206f43b406c62b68671c4f2b2af4c5b18a0cea

[Jörg Richter]

Storage fix: content index update on delete (#389)

When a topic or association is deleted the corresponding entries are removed from the *content indexes*.

=> ready to be merged into "master".
=> when you have an installation build from "master" its DB must be deleted.
=> to build the latest stable release (4.0.13) use branch "4.0.13-release".

See ticket 389.

• Changeset: 97f250fa4549863626ed894b8f8b51de962c1e04
• URL: ​https://github.com/jri/deepamehta/commit/97f250fa4549863626ed894b8f8b51de962c1e04

[Jörg Richter]

MehtaGraph? Neo4j: add test cases (#389).

See ticket 389.

• Changeset: ef09edc8f64a453e591d4313b439e364fd808cf8
• URL: ​https://github.com/jri/deepamehta/commit/ef09edc8f64a453e591d4313b439e364fd808cf8

[Jörg Richter]

Core: refactor storage layer, pt.6 (#389, #391).

Introduce association index.
Does not yet compile.

See ticket 389.
See ticket 391.

• Changeset: a276f355b8db82cb3443a4e31412678450a2f883
• URL: ​https://github.com/jri/deepamehta/commit/a276f355b8db82cb3443a4e31412678450a2f883

[Jörg Richter]

Core: association metadata index (#389, #338).

The association metadata index basically works. Both, indexing and query. Update is pending.

Core is not yet functional.

Furthermore, in order to determine an object's type the new storage layer doesn't rely on "Instantiation" associations anymore. Instead a "type_uri" property is stored for every object (= Neo4j Node). (The "Instantiation" associations remain to exist).
This provides the basis for solving #338 (An Association's "Instantiation" Association has no type assignment).

Furthermore in Parent POM: we explicitely use an older Surefire version (2.12.4) in order to avoid an exception logging problem introduced in Surefire 2.13

See ticket 389.
See ticket 338.

• Changeset: 7610f63f22fe66a9b1c0c66c5513e5414abd568a
• URL: ​https://github.com/jri/deepamehta/commit/7610f63f22fe66a9b1c0c66c5513e5414abd568a

[Jörg Richter]

Core: bidirectional assoc indexing (#389, #391).

The new association metadata index can be queried "from both sides".
See ticket 389.

First tests of the Neo4j storage implementation are adpapted to the new storage layer API.
See ticket 391.

• Changeset: cff74ad681a040a4646af0e3c04d2b16dd20270c
• URL: ​https://github.com/jri/deepamehta/commit/cff74ad681a040a4646af0e3c04d2b16dd20270c

[Jörg Richter]

Core: fix bidirectional assoc index (#389, #393).

For the first time the server can start with the new storage layer.
Use the "assoc-index" branch.
(Try multiple if "bundle not found" is thrown, see 2) below.)

Pending:

1) Adapt the Webclient to the changed assod def format (#393).
2) Provide storage implementation as OSGi service.
3) Update assoc metadata index on delete and retype.

BREAKING CHANGES

The semantics of the Association Definition URI has changed. Now this is straight the URI as derived from the underling Association. So, it does *not* reflect the assoc def's part type URI (= "child type") anymore, but is usually empty (see #393).

When operating on an assoc def you must replace

    assocDef.getUri()

by

    assocDef.getPartTypeUri()

This is in particular important for genric operations on composite values, e.g.:

    composite.put(assocDef.getPartTypeUri(), childTopic.getModel());

See ticket 389.
See ticket 393.

• Changeset: bda1119bcb65202d7f446b2d35473bfd1b43f77f
• URL: ​https://github.com/jri/deepamehta/commit/bda1119bcb65202d7f446b2d35473bfd1b43f77f

[Jörg Richter]

Storage fix: update assoc index on delete (#389).

BREAKING CHANGES

Core API: rename methods

    RelatedTopic.getAssociation()                         ->  getRelatingAssociation()
    RelatedTopicModel.getAssociationModel()               ->  getRelatingAssociation()
    RelatedAssociationModel.getRelatingAssociationModel() ->  getRelatingAssociation()

See ticket 389.

• Changeset: 7c6fe75131ed185cc9a3552c54101d2b3f1762ea
• URL: ​https://github.com/jri/deepamehta/commit/7c6fe75131ed185cc9a3552c54101d2b3f1762ea

[Jörg Richter]

Core: refactor some storage code (#389).

In preparation of "update association metadata index on retype".

New in DeepaMehtaStorage? interface:

• void storeTopicTypeUri(long topicId, String topicTypeUri)
• void storeAssociationTypeUri(long assocId, String assocTypeUri)

Not yet functional.

See ticket 389.

• Changeset: 84f92f75244e960b2a191d3dee55289d20c2dcfe
• URL: ​https://github.com/jri/deepamehta/commit/84f92f75244e960b2a191d3dee55289d20c2dcfe

[Jörg Richter]

Core: update assoc index on assoc retype (#389).

Done:

• update the association metadata index when an *association* is retyped.

Pending:

• ... when an *association role* is retyped.
• ... when a *topic* is retyped.

See ticket 389.

• Changeset: ab8200aa62a506a56c0a1a7064151d9b330a4cdd
• URL: ​https://github.com/jri/deepamehta/commit/ab8200aa62a506a56c0a1a7064151d9b330a4cdd

[Jörg Richter]

Neo4j storage: update index on role retype (#389).

The association metadata index is updated when an *association role* is retyped.

Pending:

• ... when a *topic* is retyped.

=> Building composite types works for the first with the new storage layer.

BREAKING CHANGES

The association metadata index format has slightly changed.
Existing DBs *can* be used without modification. However, roles of existing associations can *not* be retyped.
Only roles of new associations can.

Resetting the DB is recommended.

See ticket 389.

• Changeset: 2c443c571903a2633355097be76727de6fba1663
• URL: ​https://github.com/jri/deepamehta/commit/2c443c571903a2633355097be76727de6fba1663

[Jörg Richter]

Storage: store type URI of retyped topic (#389).

The association metadata index is not yet updated when a topic is retyped.

=> retyping a Topic does not work for the moment.

Further refactoring is required: The retype code must be moved from storage decorator to application layer.

See ticket 389.

• Changeset: 64581b8379240ef74bd3feddf9002f3a450bcdf1
• URL: ​https://github.com/jri/deepamehta/commit/64581b8379240ef74bd3feddf9002f3a450bcdf1

[Jörg Richter]

Neo4j storage: update index on topic retype (#389)

The association metadata index is updated when a topic is retyped.

Still pending:

• when an association is retyped all associations this association is a player in, must be re-indexed.

See ticket 389.

• Changeset: 44db47ff84d54957753106e1af82329a07934201
• URL: ​https://github.com/jri/deepamehta/commit/44db47ff84d54957753106e1af82329a07934201

[Jörg Richter]

Neo4j storage: update assoc metadata index (#389).

When an association is retyped all associations the association is a player in, are re-indexed.

=> For the first time the new storage layer should be complete now.
Minor issues may remain and will be fixed very soon.

See ticket 389.

• Changeset: c4206f43b406c62b68671c4f2b2af4c5b18a0cea
• URL: ​https://github.com/jri/deepamehta/commit/c4206f43b406c62b68671c4f2b2af4c5b18a0cea

[Jörg Richter]

Storage fix: content index update on delete (#389)

When a topic or association is deleted the corresponding entries are removed from the *content indexes*.

=> ready to be merged into "master".
=> when you have an installation build from "master" its DB must be deleted.
=> to build the latest stable release (4.0.13) use branch "4.0.13-release".

See ticket 389.

• Changeset: 97f250fa4549863626ed894b8f8b51de962c1e04
• URL: ​https://github.com/jri/deepamehta/commit/97f250fa4549863626ed894b8f8b51de962c1e04