File Upload Facility

[jri]

The user must be able to upload files to the server.
The server must maintain a File Repository.

The File Repository is a dedicated directory, e.g. "deepamehta-files" (next to "deepamehta-db").
As a start the File Repository is flat. Sub-directories is a future subject.

For developers the File Upload Factility provides:

• a means to open an Upload Dialog and specifying a (client-side) callback. The Upload Dialog allows the user to choose a file and to initiate the upload. The uploaded file is stored in the File Repository. Once the upload is complete the callback is invoked.

Upload Dialog usage:
1) Invoke the upload dialog (via a button or menu item)
2) Press "Choose File". The OS's File Open dialog appears.
3) Choose a file and press "Choose" button. The File Open dialog disappears.
4) Press the Upload Dialog's "Upload" button. The file is uploaded.

Technically, the Upload Dialog is a jQuery dialog containing a form with an input type=file element. Once the "Upload" button is pressed the form is submitted via POST.

A lot of questions remain to be discussed:

• Can we suppress the Upload Dialog? Its usage is too cumbersome. A more comfortable usage would consist only of 2 (instead of 4) steps: 1) Invoke the OS's File Open dialog directly, 2) Pressing "Choose" initiaites the upload immediately.
• Can the upload perform in the background?
• Can we have a progress indicator?
• How are the server-side files represented as topics? How they are distinguished from client-side file topics?

In any case WE DONT WANT FLASH.

Broader file handling questions arise in conjunction with single-user installation (client and server run on the same machine) vs. multi-user installation (clients and server run on different machines). 2 requirements can be stated already:

• With a single-user installation no files must be copied. Files remain where they are stored by the user. So, no File Repository is involved here. (That's the mode currently implemented by DM.)
• With a multi-user installation files must be sharable between users. A user must be able to open a file which is uploaded by another user from inside DM (via double-click). So, the server must maintain a File Repository and each client must maintain a mirror File Repository. The per-client individual file organization structure is not respected. Only the respective repositories are involved.

[jri]

1) Invoke the OS's File Open dialog directly
2) Pressing "Choose" initiaites the upload immediately.

Regarding 1) see:
​http://stackoverflow.com/questions/210643/in-javascript-can-i-make-a-click-event-fire-programmatically-for-a-file-input
Especially the answer which bases on hiding the input element via opacity 0.

Regarding 2): this is possible by calling form's submit() *onchange* of the input element.

[Jörg Richter]

Begin of new feature: File Upload (#289).

The "Create" menu provides a new "Upload File" item. It opens a file upload dialog. The choosen file is uploaded to the server. It is not yet processed.

Plugin developers can invoke the file upload dialog programmatically:

dm4c.upload_dialog.show(command, callback)

(Both parameters are not yet used.)

The REST API of the "Files" plugin provides a new method:

void uploadFile(UploadedFile? file, String command, ClientState? clientState)

A multipart/form-data request with the file content as its body can be POSTed to /files/{command}
(The command is not yet used.)

1 class is moved from core "util" to "service" package:

UploadedFile?

The entire "Commands" concept is finally dropped from core.
2 classes are removed from the "service" package:

CommandParams?
ComandResult?

2 provider classes are removed from the "Webservice" plugin:

CommandParamsProvider?
CommandResultProvider?

See ticket 289

• Changeset: 47511b3213d113b9292fcf40fc2815cb27b16849
• URL: ​https://github.com/jri/deepamehta/commit/47511b3213d113b9292fcf40fc2815cb27b16849

[jri]

Files plugin: store uploaded files (#289).

Try it via "Upload File" from the Create menu.
The uploaded file is stored in "deepamehta-files" (next to "deepamehta-db").
The server response is shown in an alert.

There is a new property in global POM:

    dm4.files.path

This is the path of the server-side file repository.
By default it is named "deepamehta-files" and stored next to "deepamehta-db".

The response of the upload request is an JSON object.
For the moment it contains just the name of the uploaded file.

    {
        filename: ...
    }

This JSON object is passed to the callback function that is passed to the show() method:

    dm4c.upload_dialog.show(callback)

The 2nd show() parameter ("command") is dropped.
The URI for POSTing a file is just /files (the "command" path parameter is dropped).

For the upcoming 4.0.12 release the file repository is not realized in a separate module (#291) but
an integral feature of the "Files" plugin. However, it is encapsulated in a new class: FileRepository?

The class UploadedFile? has moved from Core module to Files plugin.

Changeset: 13472711788096ec90771b50010dfceab353e7f4

[Jörg Richter]

Files plugin: avoid file repo name clashes (#289).

Every file stored in the file repository is represented by a File topic.
Its ID is prepended to the file's name. Thus file name clashes are avoided.
(Remember the file repository is flat.)

The File topic ID is included in the response of the upload request:

    {
        file_name: "Model.png",
        topic_id: 2857
    }

See ticket 289.

• Changeset: 41ae887f1b00fa9e1a3907b0b15ae8321ddddeed
• URL: ​https://github.com/jri/deepamehta/commit/41ae887f1b00fa9e1a3907b0b15ae8321ddddeed

[Jörg Richter]

Files plugin: upload to arbitrary directory (#289)

The existing file repository (that is managed by the Proxy plugin) is used as the repository for uploaded files as well.
There are not two different repositories anymore.

So, the recently introduced dm4.files.path POM property is dropped.
Any location inside the Proxy plugin's file repository can be the upload target.
(The file repository's root directory is still configured by the dm4.proxy.files.path POM property.)

The "Upload File" menu item is dropped from the Create menu.
Instead every Folder topic (as revealed through the File Browser) provides an "Uplaod File" button. This button invokes the Upload Dialog. The uploaded file is stored in the directory represented by the respective Folder topic.

Hints for plugin developers:
The method for invoking the Upload Dialog is renamed and takes another mandatory parameter:

    dm4c.upload_dialog.open(storage_path, callback)

The new "storage_path" parameter is the storage location for the uploaded file.
The storage path is relative to dm4.proxy.files.path and must begin with a slash.

At REST API level the storage path is appended to the /files request path

    POST /files//store/file/here

Note the double slash. The first slash separates the storage_path parameter, the second slash is part of the storage_path parameter (which always begins with slash as aforesaid).

Example: assume this POM setting:

    <dm4.proxy.files.path>/home/terry/deepamehta-files</dm4.proxy.files.path>

When posting a multipart/form-data request with a single file part (let's say image.png) to the URI above the file will be stored at

    /home/terry/deepamehta-files/store/file/here/image.png

See ticket 289.

• Changeset: 3e6c020cd4b7b61ca677f716b1d24cb91c3c4b02
• URL: ​https://github.com/jri/deepamehta/commit/3e6c020cd4b7b61ca677f716b1d24cb91c3c4b02

[Jörg Richter]

Files plugin: consolidate security (#292, #289).

Both, the Files API (/files URI) and the file respository (/filerepo URI) are consistently secured.
Each of the respective requests is secured by 2 measures:

• Subnet restriction: the request must originate from an allowed subnet (according to the dm4.filerepo.netfilter setting). Otherwise "403 Forbidden" is send back.
• Fighting directory traversal attacks: once canonicalized (resolving ../ references) the request's "path" parameter must point inside the file repository (according to the dm4.filerepo.path setting). Otherwise "403 Forbidden" is send back.

Furthermore, if the request's "path" parameter refers to a non-existing resource (file or directory)
"404 Not Found" is send back.

Additionally the upload mechanism is revised: the ID of the respective File topic is not included in the name of the stored file anymore. The original (client-side) file name is only changed if such a file already exists in the file repository. In this case a -2, -3, -4, ... suffix is added to the file's basename.

See ticket 292.
See ticket 289.

• Changeset: eba62cfb8f5b728bc4875a6cd6d4c190ac2e36e2
• URL: ​https://github.com/jri/deepamehta/commit/eba62cfb8f5b728bc4875a6cd6d4c190ac2e36e2

[Jörg Richter]

Webclient: polish upload dialog code (#289).

The upload dialog code is moved from Webclient to Files module.
The file upload_dislog.js is deleted.

IMPORTANT: if your plugin makes use of the upload dialog it must be adapted.
To open the dialog:

    dm4c.get_plugin("de.deepamehta.files").open_upload_dialog(...)

See ticket 289.

• Changeset: a18231a716b4937ce89b27fca87192393a4c8620
• URL: ​https://github.com/jri/deepamehta/commit/a18231a716b4937ce89b27fca87192393a4c8620

[Jörg Richter]

Begin of new feature: File Upload (#289).

The "Create" menu provides a new "Upload File" item. It opens a file upload dialog. The choosen file is uploaded to the server. It is not yet processed.

Plugin developers can invoke the file upload dialog programmatically:

dm4c.upload_dialog.show(command, callback)

(Both parameters are not yet used.)

The REST API of the "Files" plugin provides a new method:

void uploadFile(UploadedFile? file, String command, ClientState? clientState)

A multipart/form-data request with the file content as its body can be POSTed to /files/{command}
(The command is not yet used.)

1 class is moved from core "util" to "service" package:

UploadedFile?

The entire "Commands" concept is finally dropped from core.
2 classes are removed from the "service" package:

CommandParams?
ComandResult?

2 provider classes are removed from the "Webservice" plugin:

CommandParamsProvider?
CommandResultProvider?

See ticket 289

• Changeset: 47511b3213d113b9292fcf40fc2815cb27b16849
• URL: ​https://github.com/jri/deepamehta/commit/47511b3213d113b9292fcf40fc2815cb27b16849

[Jörg Richter]

Files plugin: avoid file repo name clashes (#289).

Every file stored in the file repository is represented by a File topic.
Its ID is prepended to the file's name. Thus file name clashes are avoided.
(Remember the file repository is flat.)

The File topic ID is included in the response of the upload request:

    {
        file_name: "Model.png",
        topic_id: 2857
    }

See ticket 289.

• Changeset: 41ae887f1b00fa9e1a3907b0b15ae8321ddddeed
• URL: ​https://github.com/jri/deepamehta/commit/41ae887f1b00fa9e1a3907b0b15ae8321ddddeed

[Jörg Richter]

Files plugin: upload to arbitrary directory (#289)

The existing file repository (that is managed by the Proxy plugin) is used as the repository for uploaded files as well.
There are not two different repositories anymore.

So, the recently introduced dm4.files.path POM property is dropped.
Any location inside the Proxy plugin's file repository can be the upload target.
(The file repository's root directory is still configured by the dm4.proxy.files.path POM property.)

The "Upload File" menu item is dropped from the Create menu.
Instead every Folder topic (as revealed through the File Browser) provides an "Uplaod File" button. This button invokes the Upload Dialog. The uploaded file is stored in the directory represented by the respective Folder topic.

Hints for plugin developers:
The method for invoking the Upload Dialog is renamed and takes another mandatory parameter:

    dm4c.upload_dialog.open(storage_path, callback)

The new "storage_path" parameter is the storage location for the uploaded file.
The storage path is relative to dm4.proxy.files.path and must begin with a slash.

At REST API level the storage path is appended to the /files request path

    POST /files//store/file/here

Note the double slash. The first slash separates the storage_path parameter, the second slash is part of the storage_path parameter (which always begins with slash as aforesaid).

Example: assume this POM setting:

    <dm4.proxy.files.path>/home/terry/deepamehta-files</dm4.proxy.files.path>

When posting a multipart/form-data request with a single file part (let's say image.png) to the URI above the file will be stored at

    /home/terry/deepamehta-files/store/file/here/image.png

See ticket 289.

• Changeset: 3e6c020cd4b7b61ca677f716b1d24cb91c3c4b02
• URL: ​https://github.com/jri/deepamehta/commit/3e6c020cd4b7b61ca677f716b1d24cb91c3c4b02

[Jörg Richter]

Files plugin: consolidate security (#292, #289).

Both, the Files API (/files URI) and the file respository (/filerepo URI) are consistently secured.
Each of the respective requests is secured by 2 measures:

• Subnet restriction: the request must originate from an allowed subnet (according to the dm4.filerepo.netfilter setting). Otherwise "403 Forbidden" is send back.
• Fighting directory traversal attacks: once canonicalized (resolving ../ references) the request's "path" parameter must point inside the file repository (according to the dm4.filerepo.path setting). Otherwise "403 Forbidden" is send back.

Furthermore, if the request's "path" parameter refers to a non-existing resource (file or directory)
"404 Not Found" is send back.

Additionally the upload mechanism is revised: the ID of the respective File topic is not included in the name of the stored file anymore. The original (client-side) file name is only changed if such a file already exists in the file repository. In this case a -2, -3, -4, ... suffix is added to the file's basename.

See ticket 292.
See ticket 289.

• Changeset: eba62cfb8f5b728bc4875a6cd6d4c190ac2e36e2
• URL: ​https://github.com/jri/deepamehta/commit/eba62cfb8f5b728bc4875a6cd6d4c190ac2e36e2

[Jörg Richter]

Webclient: polish upload dialog code (#289).

The upload dialog code is moved from Webclient to Files module.
The file upload_dislog.js is deleted.

IMPORTANT: if your plugin makes use of the upload dialog it must be adapted.
To open the dialog:

    dm4c.get_plugin("de.deepamehta.files").open_upload_dialog(...)

See ticket 289.

• Changeset: a18231a716b4937ce89b27fca87192393a4c8620
• URL: ​https://github.com/jri/deepamehta/commit/a18231a716b4937ce89b27fca87192393a4c8620

[Jörg Richter]

Begin of new feature: File Upload (#289).

The "Create" menu provides a new "Upload File" item. It opens a file upload dialog. The choosen file is uploaded to the server. It is not yet processed.

Plugin developers can invoke the file upload dialog programmatically:

dm4c.upload_dialog.show(command, callback)

(Both parameters are not yet used.)

The REST API of the "Files" plugin provides a new method:

void uploadFile(UploadedFile? file, String command, ClientState? clientState)

A multipart/form-data request with the file content as its body can be POSTed to /files/{command}
(The command is not yet used.)

1 class is moved from core "util" to "service" package:

UploadedFile?

The entire "Commands" concept is finally dropped from core.
2 classes are removed from the "service" package:

CommandParams?
ComandResult?

2 provider classes are removed from the "Webservice" plugin:

CommandParamsProvider?
CommandResultProvider?

See ticket 289

• Changeset: 47511b3213d113b9292fcf40fc2815cb27b16849
• URL: ​https://github.com/jri/deepamehta/commit/47511b3213d113b9292fcf40fc2815cb27b16849

[Jörg Richter]

Files plugin: avoid file repo name clashes (#289).

Every file stored in the file repository is represented by a File topic.
Its ID is prepended to the file's name. Thus file name clashes are avoided.
(Remember the file repository is flat.)

The File topic ID is included in the response of the upload request:

    {
        file_name: "Model.png",
        topic_id: 2857
    }

See ticket 289.

• Changeset: 41ae887f1b00fa9e1a3907b0b15ae8321ddddeed
• URL: ​https://github.com/jri/deepamehta/commit/41ae887f1b00fa9e1a3907b0b15ae8321ddddeed

[Jörg Richter]

Files plugin: upload to arbitrary directory (#289)

The existing file repository (that is managed by the Proxy plugin) is used as the repository for uploaded files as well.
There are not two different repositories anymore.

So, the recently introduced dm4.files.path POM property is dropped.
Any location inside the Proxy plugin's file repository can be the upload target.
(The file repository's root directory is still configured by the dm4.proxy.files.path POM property.)

The "Upload File" menu item is dropped from the Create menu.
Instead every Folder topic (as revealed through the File Browser) provides an "Uplaod File" button. This button invokes the Upload Dialog. The uploaded file is stored in the directory represented by the respective Folder topic.

Hints for plugin developers:
The method for invoking the Upload Dialog is renamed and takes another mandatory parameter:

    dm4c.upload_dialog.open(storage_path, callback)

The new "storage_path" parameter is the storage location for the uploaded file.
The storage path is relative to dm4.proxy.files.path and must begin with a slash.

At REST API level the storage path is appended to the /files request path

    POST /files//store/file/here

Note the double slash. The first slash separates the storage_path parameter, the second slash is part of the storage_path parameter (which always begins with slash as aforesaid).

Example: assume this POM setting:

    <dm4.proxy.files.path>/home/terry/deepamehta-files</dm4.proxy.files.path>

When posting a multipart/form-data request with a single file part (let's say image.png) to the URI above the file will be stored at

    /home/terry/deepamehta-files/store/file/here/image.png

See ticket 289.

• Changeset: 3e6c020cd4b7b61ca677f716b1d24cb91c3c4b02
• URL: ​https://github.com/jri/deepamehta/commit/3e6c020cd4b7b61ca677f716b1d24cb91c3c4b02

[Jörg Richter]

Files plugin: consolidate security (#292, #289).

Both, the Files API (/files URI) and the file respository (/filerepo URI) are consistently secured.
Each of the respective requests is secured by 2 measures:

• Subnet restriction: the request must originate from an allowed subnet (according to the dm4.filerepo.netfilter setting). Otherwise "403 Forbidden" is send back.
• Fighting directory traversal attacks: once canonicalized (resolving ../ references) the request's "path" parameter must point inside the file repository (according to the dm4.filerepo.path setting). Otherwise "403 Forbidden" is send back.

Furthermore, if the request's "path" parameter refers to a non-existing resource (file or directory)
"404 Not Found" is send back.

Additionally the upload mechanism is revised: the ID of the respective File topic is not included in the name of the stored file anymore. The original (client-side) file name is only changed if such a file already exists in the file repository. In this case a -2, -3, -4, ... suffix is added to the file's basename.

See ticket 292.
See ticket 289.

• Changeset: eba62cfb8f5b728bc4875a6cd6d4c190ac2e36e2
• URL: ​https://github.com/jri/deepamehta/commit/eba62cfb8f5b728bc4875a6cd6d4c190ac2e36e2

[Jörg Richter]

Webclient: polish upload dialog code (#289).

The upload dialog code is moved from Webclient to Files module.
The file upload_dislog.js is deleted.

IMPORTANT: if your plugin makes use of the upload dialog it must be adapted.
To open the dialog:

    dm4c.get_plugin("de.deepamehta.files").open_upload_dialog(...)

See ticket 289.

• Changeset: a18231a716b4937ce89b27fca87192393a4c8620
• URL: ​https://github.com/jri/deepamehta/commit/a18231a716b4937ce89b27fca87192393a4c8620