Ticket #1065 (closed Task: wontfix)

Opened 4 years ago

Last modified 4 years ago

Replace Neo4j by Datomic

Reported by: jri Owned by: jri
Priority: Major Milestone: Release 4.9
Component: DeepaMehta Standard Distribution Version: 4.8.5
Keywords: Cc: dgf, Malte, JuergeN
Complexity: 13 Area:
Module: deepamehta-storage-neo4j

Description


Change History

comment:1 Changed 4 years ago by jri

  • Status changed from new to accepted

comment:2 Changed 4 years ago by Jörg Richter <jri@…>

In 516ba472af80cd130ca1402dfef03573fc09bb51/deepamehta:

Setup module dm4-storage-datomic.

See #1065.

comment:3 Changed 4 years ago by Jörg Richter <jri@…>

In 554a0f3dce77341d703edb32bb573f2390744007/deepamehta:

Adapt core service test environment.

See #1065.

comment:4 Changed 4 years ago by Jörg Richter <jri@…>

comment:5 Changed 4 years ago by Jörg Richter <jri@…>

comment:6 Changed 4 years ago by Jörg Richter <jri@…>

comment:7 Changed 4 years ago by Jörg Richter <jri@…>

In d1bf674d816ff66cf8df07d9ef8713741cb4086c/deepamehta:

Fix queries with parameters.

See #1065.

comment:8 Changed 4 years ago by Jörg Richter <jri@…>

comment:9 Changed 4 years ago by Jörg Richter <jri@…>

comment:10 Changed 4 years ago by Jörg Richter <jri@…>

comment:11 Changed 4 years ago by Jörg Richter <jri@…>

comment:12 Changed 4 years ago by Jörg Richter <jri@…>

comment:13 Changed 4 years ago by Jörg Richter <jri@…>

In 504b7efb7dcab4944418454f7c2850701c3e84f1/deepamehta:

Fetch topic by key/value.

See #1065.

comment:14 Changed 4 years ago by Jörg Richter <jri@…>

In 96740462a8ebf87af8f96c5b917ee53331e6864c/deepamehta:

Storage API: add fetchTopicByUri().

See #1065.

comment:15 Changed 4 years ago by Jörg Richter <jri@…>

In 75e9f8be5fab6be427c087a3b1fd803e79478781/deepamehta:

Fix core model version tracking.

See #1065.

comment:16 Changed 4 years ago by Jörg Richter <jri@…>

In b5070d8b4293c901e503ea55b92a5e90dcfa43cc/deepamehta:

Fix value type of dynamic schema attributes.

See #1065.

comment:17 Changed 4 years ago by Jörg Richter <jri@…>

In 5510b6458693a09c778e2223e13cd0067dc2cc78/deepamehta:

Core: fix storage specific get-topic-by-uri calls.

See #1065.

comment:18 Changed 4 years ago by Jörg Richter <jri@…>

In e0809b86673450cdbdf408803924244b468bc846/deepamehta:

Read schema in edn format.

See #1065.

comment:19 Changed 4 years ago by Jörg Richter <jri@…>

comment:20 Changed 4 years ago by Jörg Richter <jri@…>

In 1f02b97a0bb62651052b3b9695532d264aec1bf9/deepamehta:

Fix Clojure class loading.

See #1065.

comment:21 Changed 4 years ago by Jörg Richter <jri@…>

comment:22 Changed 4 years ago by Jörg Richter <jri@…>

In 1868b156d6e68e323d8fbdb343d653015e2c8d02/deepamehta:

Fetch topic/assoc associations (#1065).

See #1065.

comment:23 Changed 4 years ago by Jörg Richter <jri@…>

In a72b1c3e98c0ac7e9270872e75ab81ab0bc2b8b6/deepamehta:

Delete topic/association (#1065).

See #1065.

comment:24 Changed 4 years ago by Jörg Richter <jri@…>

In 30ea697f8d5b67a00e2e73b136115c5c2668fdf1/deepamehta:

Fetch topics by key/value (#1065).

See #1065.

comment:25 Changed 4 years ago by Jörg Richter <jri@…>

In d6a8d05fbcbe5d6385d4c58c4d46572635ab9403/deepamehta:

Extend Storage API: fetch topics/assocs by type.

See #1065.

comment:26 Changed 4 years ago by Jörg Richter <jri@…>

In 784ef38e2180d00eb992e1ce0f232ae5b89b8446/deepamehta:

Fetch association between 2 topics (#1065).

See #1065.

comment:27 Changed 4 years ago by Jörg Richter <jri@…>

In 1dd2e23b6e555016bc67f925068353e864db89fd/deepamehta:

Etappensieg! All Plugins Active! (#1065).

See #1065.

comment:28 Changed 4 years ago by Malte

Hey, congrats!! I think all the third party plugin code will still work after this without any changes but lots of lines third party lines of code (thinking "getOrCreate") could, in future, *simply be removed*... Looking forward to talk about this new milestone in detail soon!

comment:29 Changed 4 years ago by Jörg Richter <jri@…>

In 5bd7babd05c67725bb9db21862ae0ec68bb46b5b/deepamehta:

Core Service API: add getTypeUri() (#1065).

1 new method in CoreService:

String getTypeUri(long id)

It gives you the type URI of a topic/assoc.

BREAKING CHANGE

Starting with Datomic (probably DM 4.9) getting the type URI via Property API will not work anymore, like this:

dm4.getProperty(id, "type_uri");

Use the new getTypeUri() call instead.

Note: both properties, uri and type_uri are considered implementation details.
Actually they never were part of the public Core Service API but Neo4j specific.

See #1065.

comment:30 Changed 4 years ago by Jörg Richter <jri@…>

In 6ec95044321800285320f9d69259f9c84729e367/deepamehta:

Storage API: store methods return ID (#1065).

See #1065.

comment:31 Changed 4 years ago by Jörg Richter <jri@…>

comment:32 Changed 4 years ago by Jörg Richter <jri@…>

comment:33 Changed 4 years ago by Jörg Richter <jri@…>

comment:34 Changed 4 years ago by Jörg Richter <jri@…>

In 7e4c7700ab15f93361980dc10477fbc56ecd642d/deepamehta:

Add AVET indexes to all attributes (#1065).

See #1065.

comment:35 Changed 4 years ago by Jörg Richter <jri@…>

In 23d03d487588618d3633c4f92a653fe2d01b7e95/deepamehta:

Switch from memory DB to H2 (#1065).

See #1065.

comment:36 Changed 4 years ago by Jörg Richter <jri@…>

In 4d3218fb1ce3a61cb1bfbc7b58bf9251f9fa49bd/deepamehta:

Migration manager: model version is long (#1065).

Datomic does not support int attributes, only long.

See #1065.

comment:37 Changed 4 years ago by Jörg Richter <jri@…>

In fac20199314cc9938734296deef60a7ebeadc696/deepamehta:

Topicmaps: topic coordinates are long (#1065).

Datomic does not support int attributes, only long.

See #1065.

comment:38 Changed 4 years ago by Jörg Richter <jri@…>

In 609b693fce5c4bffc48f96ece0c1469c189a304d/deepamehta:

Core service test env: fix DB shutdown (#1065).

See #1065.

comment:39 Changed 4 years ago by jri

Most DM platform features are now backed by Datomic, so I could do first performance comparisons now.

The sad result is: DM with Datomic is about 5-10 times slower than with our current Neo4j/Lucene solution.

Some rough Webclient end user experiences:

Neo4j/Lucene Datomic
Login 0.5s 5s
Page reload 1.5s 7s

I did also some formal measurement for the sole "get-all-topic-types" DB call. It loads all the 101 topic type definitions of the DM standard distro. These are the average timings:

Neo4j/Lucene Datomic
1433ms 7667ms

Despite these results I would not say that Datomic is slow in any regard. When regarding the fundamental differences between Datomic and our current Neo4j/Lucene solution the results came not as a surprise:

  • The Datomic peer library (= DM app) and the Datomic transactor (= the "change" coordinator) are 2 separate processes, running in different JVM instances, and communicate via TCP sockets. In contrast Neo4j/Lucene runs in the same JVM instance as DM, no TCP communication involved.
  • Datomic is an extra layer that runs on top of a 3rd-party DB product (in our case the Java embedded H2 RDBMS) which adds extra complexity (e.g. Datomic's distinctive "immutable" nature and its management of time/change).
  • Datomic interprets Datalog queries. In contrast we don't use Neo4j's "Cypher" interpreter but make direct call to the Neo4j Java API in conjunction with Lucene index lookups. It is obvious that the power of a universal DB query language (like Datalog) comes at a (performance) cost.
  • Datomic is written in both, Clojure and Java. For most API calls several language "interop" calls are performed. In contrast Neo4j/Lucene is pure Java. However in the face of todays highly optimized JVMs at this point I'm not sure if Datomic's "mixed language" nature implies any performance disadvantage.

What does this mean for DM?
To me these results imply that we can't use Datomic for DM. As sad as this is.

I would say that I understand Datomic and its performance influencing factors enough to say that no further measures would lead to substantially better results. I would even say that no DB engine which offers a universal query language could ever yield the performance of our current storage solution. The Neo4j/Lucene performance is a result of the very fact that it does NOT offer an universal query language.

The perspective I see is to NOT continue the search for a DB alternative for DM at the moment, but to go with what we have. To the DM application developers this means that more post-processing of DB results must be performed at application level.

The need for arbitrarily complex DB queries was the very origin for exploring other DB systems for DM such as Datomic or Neo4j 3.0. However with the new experience described here I rather think we must either 1) lower our performance demand, or 2) rethink DM's data/storage model.

comment:40 Changed 4 years ago by Jörg Richter <jri@…>

In c1c5b309baf88d4a733c4d501db982a67791e1d3/deepamehta:

Fix: initial plugin model version is long (#1065).

See #1065.

comment:41 Changed 4 years ago by jri

  • Status changed from accepted to closed
  • Resolution set to wontfix

Work on Datomic is discontinued

Note: See TracTickets for help on using tickets.