Replace Neo4j by Datomic

[Jörg Richter <jri@…>]

In 516ba472af80cd130ca1402dfef03573fc09bb51/deepamehta:

Setup module dm4-storage-datomic.

See #1065.

[Jörg Richter <jri@…>]

In 554a0f3dce77341d703edb32bb573f2390744007/deepamehta:

Adapt core service test environment.

See #1065.

[Jörg Richter <jri@…>]

In 17e0153551952cf3e8777f88a99faac78a33685e/deepamehta:

OSGi bundling.

See #1065.

[Jörg Richter <jri@…>]

In 664697b7f224920ff07d02bfc8ca592129210411/deepamehta:

Add Datomic API tests.

See #1065.

[Jörg Richter <jri@…>]

In 153e042cef794d99165629cea38fed001d85078a/deepamehta:

Store topic.

See #1065.

[Jörg Richter <jri@…>]

In d1bf674d816ff66cf8df07d9ef8713741cb4086c/deepamehta:

Fix queries with parameters.

See #1065.

[Jörg Richter <jri@…>]

In 1309fe00fa35e67e022def9d9b4aa2face670521/deepamehta:

Fetch/store properties.

See #1065.

[Jörg Richter <jri@…>]

In 83774e5b5940c8a41633c659acfe6554a11d4535/deepamehta:

Create root topic.

See #1065.

[Jörg Richter <jri@…>]

In a7c69daa6428a5dca2727cf5c62307580a30c809/deepamehta:

Store topic value.

See #1065.

[Jörg Richter <jri@…>]

In dbcc46e424a11fad77ab18ad419b7a049663e4ed/deepamehta:

Store association.

See #1065.

[Jörg Richter <jri@…>]

In 47aa19db773477c5387de2d30baa6a632bd3f350/deepamehta:

Store association value.

See #1065.

[Jörg Richter <jri@…>]

In 504b7efb7dcab4944418454f7c2850701c3e84f1/deepamehta:

Fetch topic by key/value.

See #1065.

[Jörg Richter <jri@…>]

In 96740462a8ebf87af8f96c5b917ee53331e6864c/deepamehta:

Storage API: add fetchTopicByUri().

See #1065.

[Jörg Richter <jri@…>]

In 75e9f8be5fab6be427c087a3b1fd803e79478781/deepamehta:

Fix core model version tracking.

See #1065.

[Jörg Richter <jri@…>]

In b5070d8b4293c901e503ea55b92a5e90dcfa43cc/deepamehta:

Fix value type of dynamic schema attributes.

See #1065.

[Jörg Richter <jri@…>]

In 5510b6458693a09c778e2223e13cd0067dc2cc78/deepamehta:

Core: fix storage specific get-topic-by-uri calls.

See #1065.

[Jörg Richter <jri@…>]

In e0809b86673450cdbdf408803924244b468bc846/deepamehta:

Read schema in edn format.

See #1065.

[Jörg Richter <jri@…>]

In 6d706b410b5cce75ed4f07087dab11e319943882/deepamehta:

Add query builder.

See #1065.

[Jörg Richter <jri@…>]

In 1f02b97a0bb62651052b3b9695532d264aec1bf9/deepamehta:

Fix Clojure class loading.

See #1065.

[Jörg Richter <jri@…>]

In 1793d0c54e7a580d7ad141a1074930ad7d6c46a8/deepamehta:

Traversal.

See #1065.

[Jörg Richter <jri@…>]

In 1868b156d6e68e323d8fbdb343d653015e2c8d02/deepamehta:

Fetch topic/assoc associations (#1065).

See #1065.

[Jörg Richter <jri@…>]

In a72b1c3e98c0ac7e9270872e75ab81ab0bc2b8b6/deepamehta:

Delete topic/association (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 30ea697f8d5b67a00e2e73b136115c5c2668fdf1/deepamehta:

Fetch topics by key/value (#1065).

See #1065.

[Jörg Richter <jri@…>]

In d6a8d05fbcbe5d6385d4c58c4d46572635ab9403/deepamehta:

Extend Storage API: fetch topics/assocs by type.

See #1065.

[Jörg Richter <jri@…>]

In 784ef38e2180d00eb992e1ce0f232ae5b89b8446/deepamehta:

Fetch association between 2 topics (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 1dd2e23b6e555016bc67f925068353e864db89fd/deepamehta:

Etappensieg! All Plugins Active! (#1065).

See #1065.

[Malte]

Hey, congrats!! I think all the third party plugin code will still work after this without any changes but lots of lines third party lines of code (thinking "getOrCreate") could, in future, *simply be removed*... Looking forward to talk about this new milestone in detail soon!

[Jörg Richter <jri@…>]

In 5bd7babd05c67725bb9db21862ae0ec68bb46b5b/deepamehta:

Core Service API: add getTypeUri() (#1065).

1 new method in CoreService:

String getTypeUri(long id)

It gives you the type URI of a topic/assoc.

BREAKING CHANGE

Starting with Datomic (probably DM 4.9) getting the type URI via Property API will not work anymore, like this:

dm4.getProperty(id, "type_uri");

Use the new getTypeUri() call instead.

Note: both properties, uri and type_uri are considered implementation details.
Actually they never were part of the public Core Service API but Neo4j specific.

See #1065.

[Jörg Richter <jri@…>]

In 6ec95044321800285320f9d69259f9c84729e367/deepamehta:

Storage API: store methods return ID (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 341ad2a796e28e2f7b1e71e33bcf0777df44c09c/deepamehta:

hasProperty() (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 18c0328a68f81dc042c6cece5ae8accc28f38e18/deepamehta:

Build assoc models (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 5034d675d92f185db77f34ad7d19d92272cd9cf6/deepamehta:

More on assocs (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 7e4c7700ab15f93361980dc10477fbc56ecd642d/deepamehta:

Add AVET indexes to all attributes (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 23d03d487588618d3633c4f92a653fe2d01b7e95/deepamehta:

Switch from memory DB to H2 (#1065).

See #1065.

[Jörg Richter <jri@…>]

In 4d3218fb1ce3a61cb1bfbc7b58bf9251f9fa49bd/deepamehta:

Migration manager: model version is long (#1065).

Datomic does not support int attributes, only long.

See #1065.

[Jörg Richter <jri@…>]

In fac20199314cc9938734296deef60a7ebeadc696/deepamehta:

Topicmaps: topic coordinates are long (#1065).

Datomic does not support int attributes, only long.

See #1065.

[Jörg Richter <jri@…>]

In 609b693fce5c4bffc48f96ece0c1469c189a304d/deepamehta:

Core service test env: fix DB shutdown (#1065).

See #1065.

[jri]

Most DM platform features are now backed by Datomic, so I could do first performance comparisons now.

The sad result is: DM with Datomic is about 5-10 times slower than with our current Neo4j/Lucene solution.

Some rough Webclient end user experiences:

	Neo4j/Lucene	Datomic
Login	0.5s	5s
Page reload	1.5s	7s

I did also some formal measurement for the sole "get-all-topic-types" DB call. It loads all the 101 topic type definitions of the DM standard distro. These are the average timings:

Neo4j/Lucene	Datomic
1433ms	7667ms

Despite these results I would not say that Datomic is slow in any regard. When regarding the fundamental differences between Datomic and our current Neo4j/Lucene solution the results came not as a surprise:

• The Datomic peer library (= DM app) and the Datomic transactor (= the "change" coordinator) are 2 separate processes, running in different JVM instances, and communicate via TCP sockets. In contrast Neo4j/Lucene runs in the same JVM instance as DM, no TCP communication involved.

• Datomic is an extra layer that runs on top of a 3rd-party DB product (in our case the Java embedded H2 RDBMS) which adds extra complexity (e.g. Datomic's distinctive "immutable" nature and its management of time/change).

• Datomic interprets Datalog queries. In contrast we don't use Neo4j's "Cypher" interpreter but make direct call to the Neo4j Java API in conjunction with Lucene index lookups. It is obvious that the power of a universal DB query language (like Datalog) comes at a (performance) cost.

• Datomic is written in both, Clojure and Java. For most API calls several language "interop" calls are performed. In contrast Neo4j/Lucene is pure Java. However in the face of todays highly optimized JVMs at this point I'm not sure if Datomic's "mixed language" nature implies any performance disadvantage.

What does this mean for DM?
To me these results imply that we can't use Datomic for DM. As sad as this is.

I would say that I understand Datomic and its performance influencing factors enough to say that no further measures would lead to substantially better results. I would even say that no DB engine which offers a universal query language could ever yield the performance of our current storage solution. The Neo4j/Lucene performance is a result of the very fact that it does NOT offer an universal query language.

The perspective I see is to NOT continue the search for a DB alternative for DM at the moment, but to go with what we have. To the DM application developers this means that more post-processing of DB results must be performed at application level.

The need for arbitrarily complex DB queries was the very origin for exploring other DB systems for DM such as Datomic or Neo4j 3.0. However with the new experience described here I rather think we must either 1) lower our performance demand, or 2) rethink DM's data/storage model.

[Jörg Richter <jri@…>]

In c1c5b309baf88d4a733c4d501db982a67791e1d3/deepamehta:

Fix: initial plugin model version is long (#1065).

See #1065.

[jri]

Work on Datomic is discontinued