Ticket #886 (closed Defect: fixed)

Opened 5 years ago

Last modified 5 years ago

dm4-wikidata: removing all associations for an entity update fails

Reported by: Malte Owned by: Malte
Priority: Major Milestone: Release 4.8
Component: 3rd Party Plugins Version: 4.7
Keywords: Cc: jri, JuergeN
Complexity: 3 Area:
Module:

Description

This ticket is to fix the recently introduced wikidata search "update" mechanism:

Background:
In wikidata a self-referential association is possible (as this example https://www.wikidata.org/wiki/Q948 shows) and i did not check for that during my import process. Now the database seems to be corrupt since i cannot simply iterate over a topics associations and delete all of them, as the following error shows:

Nov 09, 2015 12:12:55 PM org.deepamehta.plugins.wikidata.WikidataSearchPlugin removeAllClaimsFromThisItem
INFO: > Associaton "Staat" is deleted (Q948$2923F45A-68F6-4C1C-974A-869D1880E759) from 1: "Tunesien" ==>  to 2: "Tunesien"
Nov 09, 2015 12:12:55 PM de.deepamehta.core.impl.EmbeddedService getAssociation
INFO: assocId=154171
Nov 09, 2015 12:12:55 PM de.deepamehta.plugins.webservice.provider.CatchAllExceptionMapper toResponse
SEVERE: A DeepaMehta resource method or event listener threw an exception resp. an error occurred. Mapping exception/error to response: 500 (Internal Server Error).
java.lang.RuntimeException: java.lang.RuntimeException: Deleting association 154171 failed
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.removeAllClaimsFromThisItem(WikidataSearchPlugin.java:678)
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.processWikidataClaims(WikidataSearchPlugin.java:547)
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.loadClaimsAndRelatedWikidataItems(WikidataSearchPlugin.java:314)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Deleting association 154171 failed
	at de.deepamehta.core.impl.EmbeddedService.deleteAssociation(EmbeddedService.java:278)
	at org.deepamehta.plugins.wikidata.WikidataSearchPlugin.removeAllClaimsFromThisItem(WikidataSearchPlugin.java:674)
	... 48 more
Caused by: java.lang.RuntimeException: Fetching association 154171 failed
	at de.deepamehta.core.impl.EmbeddedService.getAssociation(EmbeddedService.java:180)
	at de.deepamehta.core.impl.EmbeddedService.deleteAssociation(EmbeddedService.java:276)
	... 49 more
Caused by: java.lang.IllegalStateException: Node[154171] has been deleted in this tx
	at org.neo4j.kernel.impl.core.LockReleaser$CowEntityElement.assertNotDeleted(LockReleaser.java:133)
	at org.neo4j.kernel.impl.core.LockReleaser$CowEntityElement.getPropertyAddMap(LockReleaser.java:126)
	at org.neo4j.kernel.impl.core.LockReleaser$CowNodeElement.getPropertyAddMap(LockReleaser.java:143)
	at org.neo4j.kernel.impl.core.LockReleaser.getCowPropertyAddMap(LockReleaser.java:545)
	at org.neo4j.kernel.impl.core.NodeManager.getCowPropertyAddMap(NodeManager.java:1089)
	at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:150)
	at org.neo4j.kernel.impl.core.NodeImpl.getProperty(NodeImpl.java:52)
	at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:155)
	at de.deepamehta.storage.neo4j.NodeType.of(NodeType.java:45)
	at de.deepamehta.storage.neo4j.Neo4jStorage.checkType(Neo4jStorage.java:1014)
	at de.deepamehta.storage.neo4j.Neo4jStorage.fetchAssociationNode(Neo4jStorage.java:992)
	at de.deepamehta.storage.neo4j.Neo4jStorage.fetchAssociation(Neo4jStorage.java:220)
	at de.deepamehta.core.impl.StorageDecorator.fetchAssociation(StorageDecorator.java:150)
	at de.deepamehta.core.impl.EmbeddedService.getAssociation(EmbeddedService.java:178)
	... 50 more

Nov 09, 2015 12:12:55 PM de.deepamehta.core.impl.TransactionFactory$TransactionResourceFilter$2 filter
WARNING: ### Rollback transaction of AbstractSubResourceMethod(WikidataSearchPlugin#loadClaimsAndRelatedWikidataItems)
Nov 09, 2015 12:12:58 PM de.deepamehta.core.impl.AttachedTopic _update

Question:
Using the dm4-core, is it even possible to create a self-referential association (that is from TopicID: 1 to TopicID:1)? Looking at the source i think that an error should be thrown normally and the association should not even exist.

Sadly the logging i do here is not really revealing.

Next task:
Improving logging to be able to see better what the association is really made of i am trying to delete here.

Change History

comment:1 Changed 5 years ago by Malte

Please note: It is yet to be confirmed that self-referential associations play a role here, that is just my guess since when updating other items, removing all their wikidata associations runs fine.

Looking into de/deepamehta/core/impl/AttachedAssociation, there could be really just two case this exception would not be caught by dm4 internally, if i am not mistaken. That is when Aa) the getId() does not meet the node which has already been deleted (sounds impossible, no?) and (b) if the neo4j exception message changes.

BTW: The dm4 code running there is version 4.4.3.

comment:2 Changed 5 years ago by jri

In your removeAllClaimsFromThisItem() method (master branch):

for (Association edge : claims_to_be_deleted) {
    dms.deleteAssociation(edge.getId());
}

Try this instead:

for (Association edge : claims_to_be_deleted) {
    edge.delete();
}

The exception would not occur then (I suspect).

I would appreciate when you report weather this particular change does work.

DM Core possibly must improve the exception handling when deleting associations.

Also you don't need to create single transactions within the loop as there is already an outer transaction (on a per-request basis).

Last edited 5 years ago by jri (previous) (diff)

comment:3 Changed 5 years ago by Malte

I will try all that, at latest during the upcoming week and then i will report back here. Thanks!

comment:4 Changed 5 years ago by jri

Did comment:2 work for you?

comment:5 Changed 5 years ago by Malte

Yes, this is odd, your suggestion did the trick!
Thanks for your help.

It would be great to hear something about the question i posted earlier:

Question:
Using the dm4-core, is it even possible to create a self-referential association (that is > from TopicID: 1 to TopicID:1)? Looking at the source i think that an error should be
thrown normally and the association should not even exist.

Is this right, are self referential associations (assocs involving the same topic as Player1 and Player2) even possible to create when one uses a dms.createAssociation()-call or would that throw an exception?

Notice: I let my code run on 4.7 but i guess that did not make any difference in this case.

comment:6 Changed 5 years ago by jri

In 4.8 the Node[..] has been deleted in this tx exception should never occur anymore, regardless which delete call you use. The delete logic has been completely revised in 4.8 (in the course of the #935 refactoring).

comment:7 Changed 5 years ago by jri

  • Status changed from new to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.