dm4-files: createFolder during migration fails if per-workspace=true

[Malte]

Because of #324 i had to migrate the creation of my HTML template folders (which are stored within the dm4-filerepo) from my plugins postInstall Hook to a migration.

Now the following exception is thrown during the installation of my plugin if the "per-workspace" option is set to true.

WARNUNG: ROLLBACK! (plugin "DvEW Web Experiments")
09.05.2016 17:53:43 de.deepamehta.core.impl.PluginImpl handleEvent
SCHWERWIEGEND: An error occurred while handling PLUGIN_ACTIVATED event from "de.deepamehta.workspaces" for plugin "DvEW Web Experiments":
java.lang.RuntimeException: Activation of plugin "DvEW Web Experiments" failed
	at de.deepamehta.core.impl.PluginImpl.activate(PluginImpl.java:500)
	at de.deepamehta.core.impl.PluginManager.activatePlugin(PluginManager.java:64)
	at de.deepamehta.core.impl.PluginImpl.checkRequirementsForActivation(PluginImpl.java:464)
	at de.deepamehta.core.impl.PluginImpl.handleEvent(PluginImpl.java:924)
	at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:415)
	at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks$1.run(SyncDeliverTasks.java:145)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.RuntimeException: Installing plugin "DvEW Web Experiments" in the database failed
	at de.deepamehta.core.impl.PluginImpl.installPluginInDB(PluginImpl.java:535)
	at de.deepamehta.core.impl.PluginImpl.activate(PluginImpl.java:487)
	... 11 more
Caused by: java.lang.RuntimeException: Running migration 4 of plugin "DvEW Web Experiments" failed
	at de.deepamehta.core.impl.MigrationManager.runMigration(MigrationManager.java:147)
	at de.deepamehta.core.impl.MigrationManager.runPluginMigration(MigrationManager.java:103)
	at de.deepamehta.core.impl.MigrationManager.runPluginMigrations(MigrationManager.java:70)
	at de.deepamehta.core.impl.PluginImpl.installPluginInDB(PluginImpl.java:525)
	... 12 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Creating folder "web-experiments" at repository path "/" failed
	at de.akmiraketen.plugins.experiments.migrations.Migration4.createFolderWithName(Migration4.java:96)
	at de.akmiraketen.plugins.experiments.migrations.Migration4.run(Migration4.java:48)
	at de.deepamehta.core.impl.MigrationManager.runMigration(MigrationManager.java:140)
	... 15 more
Caused by: java.lang.RuntimeException: Creating folder "web-experiments" at repository path "/" failed
	at de.deepamehta.files.FilesPlugin.createFolder(FilesPlugin.java:259)
	at de.akmiraketen.plugins.experiments.migrations.Migration4.createFolderWithName(Migration4.java:90)
	... 17 more
Caused by: java.lang.RuntimeException: Mapping repository path "/" to an absolute path failed
	at de.deepamehta.files.FilesPlugin.absolutePath(FilesPlugin.java:704)
	at de.deepamehta.files.FilesPlugin.createFolder(FilesPlugin.java:242)
	... 18 more
Caused by: java.lang.RuntimeException: If "dm4.filerepo.per_workspace" is set the request requires a "dm4_workspace_id" cookie
	at de.deepamehta.files.FilesPlugin.getWorkspaceId(FilesPlugin.java:868)
	at de.deepamehta.files.FilesPlugin.absolutePath(FilesPlugin.java:694)
	... 19 more

As the stacktrace message suggest, the dm4-files service expects to be called from a "request" scope but calls from within a migration come from the so called "system" scope.

See also #884, as how i read the new pathPrefix() seems also to depend on a "Cookie" and thus must be affected by this issue (=plugin developer can not specify a workspace/file-repo when calling fileService from "system" scope), too.

[jri]

When calling createFolder() in a migration (while per-workspace=true) you can't state "/" as repoPath. DM can't know for which workspace the folder should be created.

You must explicitly state the workspace, preferably by calling the pathPrefix() method that takes a workspace ID argument:

files.createFolder("web-experiments", files.pathPrefix(workspaceId) + "/")

In a migration (while per-workspace=true) you can't call pathPrefix() without argument.

I'll make this more clear in the 4.8.1 API docs.

Thank you for reporting!

[jri]

Replying to jri:

files.createFolder("web-experiments", files.pathPrefix(workspaceId) + "/")

Actually you don't need add the trailing slash.

[Malte]

Thanks for the info, i did not know about the API allowing me to drop a workspaceId into the pathPrefix-method.

[Malte]

Sorry, but no matter how i try it, i get exceptions thrown at met.

The deciding question (to me) seems to be: When are the workspace folders created?

I thought i needed to create it manually (if workspace filerepo folder is not yet existent, as i did for #821) before being able to create subfolders (corresponding to a specific workspace-filerepo) but that seems to be not possible either (despite i am succesfully doing so in the newest images plugin from request scope):

INFO: Standard workspace assignment for association 61709 (typeUri="dm4.core.aggregation") SUPPRESSED
01.06.2016 14:38:50 de.akmiraketen.plugins.experiments.migrations.Migration4 run
INFO: ### Creating the "/web-experiments/templates" folder in for screen templates in the "Web Experiments"-WORKSPACE, web-experiments 0.5-SNAPSHOT
01.06.2016 14:38:50 de.deepamehta.files.FilesPlugin fileExists
INFO: Checking existence of file/directory at repository path "/workspace-70296/web-experiments"
01.06.2016 14:38:50 de.deepamehta.files.FilesPlugin fileExists
INFO: Checking existence of file/directory at repository path "/workspace-70296"
01.06.2016 14:38:50 de.akmiraketen.plugins.experiments.migrations.Migration4 createFolderWithName
INFO: Creating a new Workspace folder in filerepo/workspace-70296!
01.06.2016 14:38:50 de.deepamehta.files.FilesPlugin createFolder
INFO: Creating folder "/workspace-70296" at repository path "/"
01.06.2016 14:38:50 de.deepamehta.core.impl.PluginImpl installPluginInDB
WARNUNG: ROLLBACK! (plugin "DvEW Web Experiments")
01.06.2016 14:38:51 de.deepamehta.core.impl.PluginImpl handleEvent
SCHWERWIEGEND: An error occurred while handling PLUGIN_ACTIVATED event from "de.deepamehta.workspaces" for plugin "DvEW Web Experiments":
java.lang.RuntimeException: Activation of plugin "DvEW Web Experiments" failed
	at de.deepamehta.core.impl.PluginImpl.activate(PluginImpl.java:500)
	at de.deepamehta.core.impl.PluginManager.activatePlugin(PluginManager.java:64)
	at de.deepamehta.core.impl.PluginImpl.checkRequirementsForActivation(PluginImpl.java:464)
	at de.deepamehta.core.impl.PluginImpl.handleEvent(PluginImpl.java:924)
	at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:415)
	at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks$1.run(SyncDeliverTasks.java:145)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.RuntimeException: Installing plugin "DvEW Web Experiments" in the database failed
	at de.deepamehta.core.impl.PluginImpl.installPluginInDB(PluginImpl.java:535)
	at de.deepamehta.core.impl.PluginImpl.activate(PluginImpl.java:487)
	... 11 more
Caused by: java.lang.RuntimeException: Running migration 4 of plugin "DvEW Web Experiments" failed
	at de.deepamehta.core.impl.MigrationManager.runMigration(MigrationManager.java:147)
	at de.deepamehta.core.impl.MigrationManager.runPluginMigration(MigrationManager.java:103)
	at de.deepamehta.core.impl.MigrationManager.runPluginMigrations(MigrationManager.java:70)
	at de.deepamehta.core.impl.PluginImpl.installPluginInDB(PluginImpl.java:525)
	... 12 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Creating folder "/workspace-70296" at repository path "/" failed
	at de.akmiraketen.plugins.experiments.migrations.Migration4.createFolderWithName(Migration4.java:96)
	at de.akmiraketen.plugins.experiments.migrations.Migration4.run(Migration4.java:52)
	at de.deepamehta.core.impl.MigrationManager.runMigration(MigrationManager.java:140)
	... 15 more
Caused by: java.lang.RuntimeException: Creating folder "/workspace-70296" at repository path "/" failed
	at de.deepamehta.files.FilesPlugin.createFolder(FilesPlugin.java:259)
	at de.akmiraketen.plugins.experiments.migrations.Migration4.createFolderWithName(Migration4.java:90)
	... 17 more
Caused by: java.lang.RuntimeException: Mapping repository path "/" to an absolute path failed
	at de.deepamehta.files.FilesPlugin.absolutePath(FilesPlugin.java:704)
	at de.deepamehta.files.FilesPlugin.createFolder(FilesPlugin.java:242)
	... 18 more
Caused by: java.lang.RuntimeException: If "dm4.filerepo.per_workspace" is set the request requires a "dm4_workspace_id" cookie
	at de.deepamehta.files.FilesPlugin.getWorkspaceId(FilesPlugin.java:868)
	at de.deepamehta.files.FilesPlugin.absolutePath(FilesPlugin.java:694)
	... 19 more
01.06.2016 14:38:51 de.deepamehta.core.impl.PluginImpl activate

Not creating the workspace-filerepo (folder) is also not an option as the creation of folders in a workspace specific filerepo will fail (despite passing the correct workspace specific path-prefix) as parent folder path).

INFO: ### Creating the "/web-experiments/templates" folder in for screen templates in the "Web Experiments"-WORKSPACE, web-experiments 0.5-SNAPSHOT
01.06.2016 14:53:15 de.deepamehta.files.FilesPlugin fileExists
INFO: Checking existence of file/directory at repository path "/workspace-75549/web-experiments"
01.06.2016 14:53:15 de.deepamehta.files.FilesPlugin fileExists
INFO: Checking existence of file/directory at repository path "/workspace-75549"
01.06.2016 14:53:15 de.akmiraketen.plugins.experiments.migrations.Migration4 createFolderWithName
WARNUNG: Parent folder in Filerepo/workspace-75549 - DOES NOT EXIST
01.06.2016 14:53:15 de.akmiraketen.plugins.experiments.migrations.Migration4 createFolderWithName
INFO: Creating the "web-experiments" subfolder for filerepo at /workspace-75549!
01.06.2016 14:53:15 de.deepamehta.files.FilesPlugin createFolder
INFO: Creating folder "web-experiments" at repository path "/workspace-75549"
01.06.2016 14:53:15 de.deepamehta.core.impl.PluginImpl installPluginInDB
WARNUNG: ROLLBACK! (plugin "DvEW Web Experiments")
01.06.2016 14:53:16 de.deepamehta.core.impl.PluginImpl$1 addingService
SCHWERWIEGEND: An error occurred while adding service de.deepamehta.files.FilesService to plugin "DvEW Web Experiments":
java.lang.RuntimeException: Activation of plugin "DvEW Web Experiments" failed
	at de.deepamehta.core.impl.PluginImpl.activate(PluginImpl.java:500)
	at de.deepamehta.core.impl.PluginManager.activatePlugin(PluginManager.java:64)
	at de.deepamehta.core.impl.PluginImpl.checkRequirementsForActivation(PluginImpl.java:464)
	at de.deepamehta.core.impl.PluginImpl.addService(PluginImpl.java:422)
	at de.deepamehta.core.impl.PluginImpl.access$000(PluginImpl.java:45)
	at de.deepamehta.core.impl.PluginImpl$1.addingService(PluginImpl.java:366)
	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)
	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)
	at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
	at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)
	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:317)
	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)
	at de.deepamehta.core.impl.PluginImpl.openServiceTrackers(PluginImpl.java:395)
	at de.deepamehta.core.impl.PluginImpl.start(PluginImpl.java:120)
	at de.deepamehta.core.osgi.PluginActivator.start(PluginActivator.java:56)
	at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:645)
	at org.apache.felix.framework.Felix.activateBundle(Felix.java:2154)
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2072)
	at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:976)
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:1175)
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:1153)
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:458)
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:263)
Caused by: java.lang.RuntimeException: Installing plugin "DvEW Web Experiments" in the database failed
	at de.deepamehta.core.impl.PluginImpl.installPluginInDB(PluginImpl.java:535)
	at de.deepamehta.core.impl.PluginImpl.activate(PluginImpl.java:487)
	... 22 more
Caused by: java.lang.RuntimeException: Running migration 4 of plugin "DvEW Web Experiments" failed
	at de.deepamehta.core.impl.MigrationManager.runMigration(MigrationManager.java:147)
	at de.deepamehta.core.impl.MigrationManager.runPluginMigration(MigrationManager.java:103)
	at de.deepamehta.core.impl.MigrationManager.runPluginMigrations(MigrationManager.java:70)
	at de.deepamehta.core.impl.PluginImpl.installPluginInDB(PluginImpl.java:525)
	... 23 more
Caused by: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: java.lang.RuntimeException: Creating folder "web-experiments" at repository path "/workspace-75549" failed
	at de.akmiraketen.plugins.experiments.migrations.Migration4.createFolderWithName(Migration4.java:96)
	at de.akmiraketen.plugins.experiments.migrations.Migration4.run(Migration4.java:52)
	at de.deepamehta.core.impl.MigrationManager.runMigration(MigrationManager.java:140)
	... 26 more
Caused by: javax.ws.rs.WebApplicationException: java.lang.RuntimeException: Creating folder "web-experiments" at repository path "/workspace-75549" failed
	at de.deepamehta.files.FilesPlugin.createFolder(FilesPlugin.java:257)
	at de.akmiraketen.plugins.experiments.migrations.Migration4.createFolderWithName(Migration4.java:93)
	... 28 more
Caused by: java.lang.RuntimeException: Creating folder "web-experiments" at repository path "/workspace-75549" failed
	... 30 more
Caused by: de.deepamehta.files.FileRepositoryException: File or directory "/home/malted/source/deepamehta/databases/web-exp-test/upgrade/filerepo/workspace-75549" does not exist
	at de.deepamehta.files.FilesPlugin.checkExistence(FilesPlugin.java:740)
	at de.deepamehta.files.FilesPlugin.createFolder(FilesPlugin.java:243)
	... 29 more

I hope the case behind this question is clear, so far:

• If per.workspace-filerepos=true, create the "templates" folder etc. in a specific filerepo corresponding to my custom workspace (i do so through pre-prending the result of a "pathPrefix(myCustomWorkspaceId)"-call
• If per.workspace-filerepos=false, create the "templates" folder etc. in the global filerepo

I guess, if the workspace folder is not yet available but the plugin developer knows about it (knows WorkspaceID, knows filerepo config is set to PER-WORKSPACE) and wants to create folders in it programmatically, the current fileservice fails to support her.

[jri]

Replying to Malte:

I guess, if the workspace folder is not yet available but the plugin developer knows about it (knows WorkspaceID, knows filerepo config is set to PER-WORKSPACE) and wants to create folders in it programmatically, the current fileservice fails to support her.

You're absolutely right!
This case is actually not supported at the moment.
I'll fix that very soon.

The concept is that the developer must not care about creating the workspace folders. This is supposed to happen automatically, at the time of the first access. So, this call is correct, but will only work once I've done the fix:

files.createFolder("web-experiments", files.pathPrefix(workspaceId));

[Jörg Richter <jri@…>]

In b806fdabc5ba8c47d727e6c9c22e2e1c8d5daa17/deepamehta:

Files: fix workspace folder creation (#965).

In case of enabled per-workspace file repos: the workspace folder is created also if the repo-path passed to a Files service call is not "/".

de.deepamehta.files.FileRepositoryException: File or directory ... does not exist should not happen anymore.

See #965.

[Malte]

Great, thanks for the two improvements.