dm4-wikidata: removing all associations for an entity update fails

[Malte]

This ticket is to fix the recently introduced wikidata search "update" mechanism:

Background:
In wikidata a self-referential association is possible (as this example ​https://www.wikidata.org/wiki/Q948 shows) and i did not check for that during my import process. Now the database seems to be corrupt since i cannot simply iterate over a topics associations and delete all of them, as the following error shows:

Nov 09, 2015 12:12:55 PM org.deepamehta.plugins.wikidata.WikidataSearchPlugin removeAllClaimsFromThisItem
INFO: > Associaton "Staat" is deleted (Q948$2923F45A-68F6-4C1C-974A-869D1880E759) from 1: "Tunesien" ==>  to 2: "Tunesien"
Nov 09, 2015 12:12:55 PM de.deepamehta.core.impl.EmbeddedService getAssociation
INFO: assocId=154171
Nov 09, 2015 12:12:55 PM de.deepamehta.plugins.webservice.provider.CatchAllExceptionMapper toResponse
SEVERE: A DeepaMehta resource method or event listener threw an exception resp. an error occurred. Mapping exception/error to response: 500 (Internal Server Error).
java.lang.RuntimeException: java.lang.RuntimeException: Deleting association 154171 failed
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.removeAllClaimsFromThisItem(WikidataSearchPlugin.java:678)
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.processWikidataClaims(WikidataSearchPlugin.java:547)
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.loadClaimsAndRelatedWikidataItems(WikidataSearchPlugin.java:314)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Deleting association 154171 failed
	at de.deepamehta.core.impl.EmbeddedService.deleteAssociation(EmbeddedService.java:278)
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.removeAllClaimsFromThisItem(WikidataSearchPlugin.java:674)
	... 48 more
Caused by: java.lang.RuntimeException: Fetching association 154171 failed
	at de.deepamehta.core.impl.EmbeddedService.getAssociation(EmbeddedService.java:180)
	at de.deepamehta.core.impl.EmbeddedService.deleteAssociation(EmbeddedService.java:276)
	... 49 more
Caused by: java.lang.IllegalStateException: Node[154171] has been deleted in this tx
	at org.neo4j.kernel.impl.core.LockReleaser$CowEntityElement.assertNotDeleted(LockReleaser.java:133)
	at org.neo4j.kernel.impl.core.LockReleaser$CowEntityElement.getPropertyAddMap(LockReleaser.java:126)
	at org.neo4j.kernel.impl.core.LockReleaser$CowNodeElement.getPropertyAddMap(LockReleaser.java:143)
	at org.neo4j.kernel.impl.core.LockReleaser.getCowPropertyAddMap(LockReleaser.java:545)
	at org.neo4j.kernel.impl.core.NodeManager.getCowPropertyAddMap(NodeManager.java:1089)
	at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:150)
	at org.neo4j.kernel.impl.core.NodeImpl.getProperty(NodeImpl.java:52)
	at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:155)
	at de.deepamehta.storage.neo4j.NodeType.of(NodeType.java:45)
	at de.deepamehta.storage.neo4j.Neo4jStorage.checkType(Neo4jStorage.java:1014)
	at de.deepamehta.storage.neo4j.Neo4jStorage.fetchAssociationNode(Neo4jStorage.java:992)
	at de.deepamehta.storage.neo4j.Neo4jStorage.fetchAssociation(Neo4jStorage.java:220)
	at de.deepamehta.core.impl.StorageDecorator.fetchAssociation(StorageDecorator.java:150)
	at de.deepamehta.core.impl.EmbeddedService.getAssociation(EmbeddedService.java:178)
	... 50 more

Nov 09, 2015 12:12:55 PM de.deepamehta.core.impl.TransactionFactory$TransactionResourceFilter$2 filter
WARNING: ### Rollback transaction of AbstractSubResourceMethod(WikidataSearchPlugin#loadClaimsAndRelatedWikidataItems)
Nov 09, 2015 12:12:58 PM de.deepamehta.core.impl.AttachedTopic _update

Question:
Using the dm4-core, is it even possible to create a self-referential association (that is from TopicID: 1 to TopicID:1)? Looking at the source i think that an error should be thrown normally and the association should not even exist.

Sadly the logging i do here is not really revealing.

Next task:
Improving logging to be able to see better what the association is really made of i am trying to delete here.

[Malte]

Please note: It is yet to be confirmed that self-referential associations play a role here, that is just my guess since when updating other items, removing all their wikidata associations runs fine.

Looking into de/deepamehta/core/impl/AttachedAssociation, there could be really just two case this exception would not be caught by dm4 internally, if i am not mistaken. That is when Aa) the getId() does not meet the node which has already been deleted (sounds impossible, no?) and (b) if the neo4j exception message changes.

BTW: The dm4 code running there is version 4.4.3.

[jri]

In your removeAllClaimsFromThisItem() method (master branch):

for (Association edge : claims_to_be_deleted) {
    dms.deleteAssociation(edge.getId());
}

Try this instead:

for (Association edge : claims_to_be_deleted) {
    edge.delete();
}

The exception would not occur then (I suspect).

I would appreciate when you report weather this particular change does work.

DM Core possibly must improve the exception handling when deleting associations.

Also you don't need to create single transactions within the loop as there is already an outer transaction (on a per-request basis).

[Malte]

I will try all that, at latest during the upcoming week and then i will report back here. Thanks!

[jri]

Did comment:2 work for you?

[Malte]

Yes, this is odd, your suggestion did the trick!
Thanks for your help.

It would be great to hear something about the question i posted earlier:

Question:
Using the dm4-core, is it even possible to create a self-referential association (that is > from TopicID: 1 to TopicID:1)? Looking at the source i think that an error should be
thrown normally and the association should not even exist.

Is this right, are self referential associations (assocs involving the same topic as Player1 and Player2) even possible to create when one uses a dms.createAssociation()-call or would that throw an exception?

Notice: I let my code run on 4.7 but i guess that did not make any difference in this case.

[jri]

In 4.8 the Node[..] has been deleted in this tx exception should never occur anymore, regardless which delete call you use. The delete logic has been completely revised in 4.8 (in the course of the #935 refactoring).