Add timestamps to all items

[JuergeN]

For many reasons it would be important to store timestamps with all items as a db based property:

• creation date | user
• modification date | user (last modified by ...)
• access date | user (last accessed by ...)

This could go into logging, if enabled so.

[JuergeN]

Joerg, can you estimate the effort that is needed to accomplish this task?

In my opinion a timeline based representation of the date is another very important way to show the content (just like the geomap). I really think this would add another "killer feature" to DeepaMehta and proof how flexible data can be displayed from within one system.

[jri]

Fully agreed. Time is an essential aspect (in the EduZEN project we need it as well).

Estimating the effort involves several levels:
1) The pure storage aspect is easy and takes 1 or 2 days.
2) The retrieval aspect requires more attention. What kinds of queries do we need? How to extend the DM Core Service API? Or is ”Time" handled via a separate API? Ad-hoc extension of each Core Service call with some time parameters is probably not the way to go.
So, its not easy to say how long it takes. Further conceptualization is required. Possibly the Core Service API requires a redeign first. I would like to discuss this further with dgf. All in all in could take between 4 days and 4 weeks.
3) The most challenging part will be the GUI. What types of widgets do we provide? Is Time a new Topicmap type? Or rendered in the detail panel? How does it look like? How interact? How could these be generalized? What parts are user-configurable? What facilities can we offer developers at the framework-level? I would like to discuss this further in order to identify a 1st milestone.

Thanks for the impulse!
Lets take the next step on this.

[JuergeN]

To me the first step should be a simple timeline (new MapType?). I think about it just like the Geomap. I can create new maps of the type timeline). All items I search appear with their icons in the map, the details will appear in the details panel if I click that icon.

The map has two dimensions x and y. x is the horizontal dimension and shows days, y is the vertical dimenstion and shows 24 hours. I can scroll through x as an endless timeline and through y from 0 to 23 hours. Everything I search appears in the timeline at the day and hour it was created or last modified. I think, this would be enough for a good starting point.

As a second step I would want to filter the search results in the three categories "created" and/or "last modified" and/or "last accessed". The width of a day row depends on the number of icons it needs to display. If it is empty, it can be quite narrow.

A third step could be to have an extra TopicType? with a date in it, that can be displayed in the timeline (like every type instance that has an address/geodata can be displayed in the geomap). E.g. an event would have a date or a person would have a birthday. I can filter my timesearch for birthdays, or events. Another example are digital fotos that often have a timestamp from when they were taken (meta info). So the timeline could show me all fotos in the order they were taken. (Once again it would be great if each instance of a topic would have a thumbnail of a picture instead of the default icon if a picture is part of the composite.)

[jri]

Thank you, JuergeN, for the use cases and roadmap (steps 1-3)! This is very helpful and I think we are ready to start now.

Also in the EduZEN project the time issue is important NOW. Malte has offered development support also on the DM side.

Planning:

1. I enrich all DM objects with the 3 timestamps (as you describe in the original ticket).
2. Malte develops the GUI components.
3. Together we design the time API, driven by the respective use cases.

I will start now with step 1.

[Malte]

Sorry for the misunderstanding, I communicated to you my interest in developing step 1, not the GUI. But great that you start.

[Malte]

Questions from our first developer talk on this issue:

• What means "last accessed", from DB-perspective this always evaluates to "just now"?
• "Name Lastname" accessed as part of a collection (just the composite was accessed)
• "Full Contact" accessed with all child topics (all topics were accesssed)
• Internationalization: Configurable Timezone vs generic UNIX Timestamp
• Possibly we need a "guest" account (with or without session) here or just a dummy username for logging access.

API Request:

• A range query (with an optional "From" parameter) and an optional "Type" parameter would be great.

6 parameters may be realized as Neo4J Node "Properties". 3 time values each with a username.

No discussion on GUI, yet. New topics require further thoughts.

[Malte]

Some possible answers for the bullet points above:

• "Last accessed" means "now in millisec" AFTER any read/create orupdate operation (POST_READ,_CREATE,_UPDATE)
• if fetchComposite=true, then "all" child-topics were accessed
• when just the composite was accessed (as part of the collection) just the composite was accessed not more (for now)
• Internationalization: no configurable timestamp on server-side is the easiest (and i've heard the sanest way), so we could always deal with UTC in msecs

[jri]

I will start now with "created", "created_by", "modified", and "modified_by" timestamps.

For readability I consider IETF-compliant RFC 2822 as storage format with fixed GMT/UTC timezone [1]. Example:

Mon, 25 Dec 1995 13:30:00 GMT

That is also the exact required format for HTTP date/time headers [2].
Furthermore there is native JavaScript? date support for it [3].

[1] ​http://tools.ietf.org/html/rfc2822#section-3.3
[2] ​http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3
[3] ​https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse

[JuergeN]

Nice to see some progress here! :)

Concerning your questions about last_accessed, this is just purely the last date stamp when content was accessed (now-1) on the level of every single obejct - be it through the machine or a human beeing. Here yu can see what i get under linux:

root@linux:stat example.file
  File: `example.file'
  Size: 2637      	Blocks: 8          IO Block: 4096   regular file
Device: 1bh/27d	Inode: 406700474   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-07-15 13:40:31.000000000 +0200
Modify: 2013-06-04 16:54:23.000000000 +0200
Change: 2013-06-04 16:54:23.000000000 +0200

[jri]

Thanks, JuergeN, for feedback!

[jri]

New Core Event PRE_SEND_RESPONSE (#477, #386).

Fired before the response is send to the client. That is exactly once per request.
This allows plugins to set certain response headers like Last-Modified.

New listener interface:

de.deepamehta.core.service.event.PreSendResponseListener?

Listener method:

void preSendResponse(ContainerResponse? response)

Note: ContainerResponse? is a Jersey-specific class (com.sun.jersey.spi.container.ContainerResponse?).
We should remove the Jersey dependency and move to JAX-RS 2.0 instead.

See ticket 477.
See ticket 386.

Changeset: 1676101075ebb2789642fe7770d0595cb0d1a97e

[jri]

Time Plugin: send Last-Modified headers (#386).

If the response contains a single entity and if a modified timestamp for that entity is stored in the DB the Last-Modified response header is set accordingly.

The Time plugin provides an OSGi service (no REST service yet):

• String getTimeCreated(long objectId)
• String getTimeModified(long objectId)

See ticket 386.

Changeset: d3e987725578630ecab34b390fae448771c5e654

[jri]

Time plugin: change storage format (#386).

The timestamps are stored in Unix time. The Time API has changed accordingly.
This is to better support range queries in the future.

If you've already installed the Time plugin you should reset your DB.

See ticket 386.

Changeset: 356d6c318c639ff16e0c84471c44bfcfca5ca022

[jri]

Replying to jri:

Time Plugin: send Last-Modified headers (#386).

If the response contains a single entity and if a modified timestamp for that entity is stored in the DB the Last-Modified response header is set accordingly.

Just one observation regarding caching (see #478): the sole presence of the Last-Modified response header leverages the browser's own caching mechanism. That is the browser automatically adds a If-Modified-Since header to subsequent GET requests to the same resource. So, in conjunction with the new Caching plugin (which operates only at the server-side so far) caching is already effective. No DM webclient modifications were required. Tested with Safari and Chrome.

[jri]

Add 2 new plugins to reactor build (#386, #478).

DeepaMehta 4 Time
DeepaMehta 4 Caching

See ticket 386.
See ticket 478.

Changeset: b9c5fa2584ff000a6560df1e92ce97b0312d2e27

[jri]

Time plugin: enrich with timestamp (#386, #479).

If the response contains a single DeepaMehtaObject? its composite model is enriched with a dm4.time.modified entry. The value is a UNIX timestamp. If the modification date is unknown the value is -1.

This way an object's timestamp is accessible at the Webclient application layer (client-side). It could be used for display or for setting the If-Unmodified-Since header in a conditional PUT request.

In preparation of #479 (Edit Conflict Detection).

See ticket 386.
See ticket 479.

Changeset: dfaa8f0ba4ee468cb589c267e41df7265bb2e016

[jri]

There is a bug: if a topic is just edited and the Edit button is pressed again the next PUT request fails with 412 (Precondition Failed), see #479. This is because the updated topic contained in the directives has no modification date set.

[jri]

Caching: fix conditional PUT request (#386, #478).

Topics/associations contained in a Directives object are enriched with their modification timestamps.
As a consequence the Caching module uses the up-to-date modification timestamp when constructing a subsequent PUT request.

See ticket 386.
See ticket 478.

Changeset: 4528ffa5017160f17e675cfea16545e563027081

[jri]

Time plugin: time range queries (#386).

The Time API provides methods for retrieving topics and associations based on a created/modified time range.

Time service:

• Collection<Topic> getTopicsByCreationTime(long from, long to)
• Collection<Topic> getTopicsByModificationTime(long from, long to)
• Collection<Association> getAssociationsByCreationTime(long from, long to)
• Collection<Association> getAssociationsByModificationTime(long from, long to)

REST API:

• GET /time/from/{from}/to/{to}/topics/created
• GET /time/from/{from}/to/{to}/topics/modified
• GET /time/from/{from}/to/{to}/assocs/created
• GET /time/from/{from}/to/{to}/assocs/modified

Note: these methods rely on the timestamps and property index recently introduced (4.1.1-SNAPSHOT).
Topics/associations existing in a DM 4.1 installation have no timestamps.

See #386.

Changeset: 60785f0f32083c825bb48f5ea2bd527fd3afa8df

[jri]

Time and Caching fixes (#386, #478, #479).

FIXES

• Browser caching is enabled also for topics/associations which have no timestamp. 0 assumed as default (Jan 1, 1970).
• Edit conflict detection works also for topics/associations which have no timestamp.

CHANGES

Time plugin:

• for topics/associations send to the client their creation timestamp is available in the composite value as well (besides the modification timestamp).
• the time API's timestamp getter methods return 0 as default in case no timestamp is stored (before -1 has been returned).
• the Last-Modified response header is also set for a default timestamp value (0).

Caching plugin:

• a request's precondiction is also checked for a default timestamp value (0).

See #386.
See #478.
See #479.

Changeset: 1b5bbdb7c6b963915168f5a7c69e6ccb8a13d427

[jri]

Time API is polymorph (#472, #386).

BREAKING CHANGE

The timestamp getters and setters do not distinguih between topics and associations but take a generic DeepaMehtaObject?.

long getCreationTime(DeepaMehtaObject? object)
long getModificationTime(DeepaMehtaObject? object)

See #472.
See #386.

Changeset: 854a83b2a42941d0b0befeab3655d479358be922

[jri]

To summarize, now we have:

• creation and modification timestamps
• a Time API and REST interface
• performant range queries (trie-indexing)
• integration with the caching plugin to leverage the browser cache

This ticket is regarded as complete now.
Pending aspects like the access timestamp and the GUI components are supposed to be addressed in separate tickets.