Ticket #386 (closed Enhancement: fixed)

Opened 8 years ago

Last modified 7 years ago

Add timestamps to all items

Reported by: JuergeN Owned by: jri
Priority: Major Milestone: Release 4.2
Component: DeepaMehta Standard Distribution Version: 4.0.12
Keywords: Cc: jri, dgf, Malte
Complexity: 3 Area:
Module:

Description

For many reasons it would be important to store timestamps with all items as a db based property:

  • creation date | user
  • modification date | user (last modified by ...)
  • access date | user (last accessed by ...)

This could go into logging, if enabled so.

Change History

comment:1 Changed 8 years ago by JuergeN

  • Milestone set to Release 4.1

comment:2 Changed 8 years ago by JuergeN

  • Cc jri added

comment:3 Changed 8 years ago by JuergeN

Joerg, can you estimate the effort that is needed to accomplish this task?

In my opinion a timeline based representation of the date is another very important way to show the content (just like the geomap). I really think this would add another "killer feature" to DeepaMehta and proof how flexible data can be displayed from within one system.

comment:4 Changed 8 years ago by jri

  • Cc dgf, Malte added

comment:5 Changed 8 years ago by jri

Fully agreed. Time is an essential aspect (in the EduZEN project we need it as well).

Estimating the effort involves several levels:
1) The pure storage aspect is easy and takes 1 or 2 days.
2) The retrieval aspect requires more attention. What kinds of queries do we need? How to extend the DM Core Service API? Or is ”Time" handled via a separate API? Ad-hoc extension of each Core Service call with some time parameters is probably not the way to go.
So, its not easy to say how long it takes. Further conceptualization is required. Possibly the Core Service API requires a redeign first. I would like to discuss this further with dgf. All in all in could take between 4 days and 4 weeks.
3) The most challenging part will be the GUI. What types of widgets do we provide? Is Time a new Topicmap type? Or rendered in the detail panel? How does it look like? How interact? How could these be generalized? What parts are user-configurable? What facilities can we offer developers at the framework-level? I would like to discuss this further in order to identify a 1st milestone.

Thanks for the impulse!
Lets take the next step on this.

comment:6 Changed 8 years ago by JuergeN

To me the first step should be a simple timeline (new MapType?). I think about it just like the Geomap. I can create new maps of the type timeline). All items I search appear with their icons in the map, the details will appear in the details panel if I click that icon.

The map has two dimensions x and y. x is the horizontal dimension and shows days, y is the vertical dimenstion and shows 24 hours. I can scroll through x as an endless timeline and through y from 0 to 23 hours. Everything I search appears in the timeline at the day and hour it was created or last modified. I think, this would be enough for a good starting point.

As a second step I would want to filter the search results in the three categories "created" and/or "last modified" and/or "last accessed". The width of a day row depends on the number of icons it needs to display. If it is empty, it can be quite narrow.

A third step could be able to show every instance with a date in it (like every type instance that has an address/geodata can be displayed in the geomap). E.g. an event would have a date or a person would have a birthday. I can filter my timesearch by searching by type for birthdays, or events. Another example are digital fotos that often have a timestamp from when they were taken (meta info). So if I search by type for fotos, the timeline could show me all fotos in the order they were taken. (Once again it would be great if each instance of a topic would have a thumbnail of a picture instead of the default icon if a picture is part of the composite.)

Last edited 8 years ago by JuergeN (previous) (diff)

comment:7 Changed 8 years ago by jri

  • Milestone changed from Release 4.1 to Release 4.2

comment:8 Changed 8 years ago by jri

Thank you, JuergeN, for the use cases and roadmap (steps 1-3)! This is very helpful and I think we are ready to start now.

Also in the EduZEN project the time issue is important NOW. Malte has offered development support also on the DM side.

Planning:

  1. I enrich all DM objects with the 3 timestamps (as you describe in the original ticket).
  2. Malte develops the GUI components.
  3. Together we design the time API, driven by the respective use cases.

I will start now with step 1.

comment:9 Changed 8 years ago by Malte

Sorry for the misunderstanding, I communicated to you my interest in developing step 1, not the GUI. But great that you start.

comment:10 Changed 8 years ago by Malte

Questions from our first developer talk on this issue:

  • What means "last accessed", from DB-perspective this always evaluates to "just now"?
  • "Name Lastname" accessed as part of a collection (just the composite was accessed)
  • "Full Contact" accessed with all child topics (all topics were accesssed)
  • Internationalization: Configurable Timezone vs generic UNIX Timestamp
  • Possibly we need a "guest" account (with or without session) here or just a dummy username for logging access.

API Request:

  • A range query (with an optional "From" parameter) and an optional "Type" parameter would be great.

6 parameters may be realized as Neo4J Node "Properties". 3 time values each with a username.

No discussion on GUI, yet. New topics require further thoughts.

comment:11 Changed 8 years ago by Malte

Some possible answers for the bullet points above:

  • "Last accessed" means "now in millisec" AFTER any read/create orupdate operation (POST_READ,_CREATE,_UPDATE)
  • if fetchComposite=true, then "all" child-topics were accessed
  • when just the composite was accessed (as part of the collection) just the composite was accessed not more (for now)
  • Internationalization: no configurable timestamp on server-side is the easiest (and i've heard the sanest way), so we could always deal with UTC in msecs
Last edited 8 years ago by Malte (previous) (diff)

comment:12 Changed 8 years ago by jri

  • Owner set to jri
  • Status changed from new to accepted
  • Type changed from Defect to Enhancement

I will start now with "created", "created_by", "modified", and "modified_by" timestamps.

For readability I consider IETF-compliant RFC 2822 as storage format with fixed GMT/UTC timezone [1]. Example:

Mon, 25 Dec 1995 13:30:00 GMT

That is also the exact required format for HTTP date/time headers [2].
Furthermore there is native JavaScript? date support for it [3].

[1] http://tools.ietf.org/html/rfc2822#section-3.3
[2] http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3
[3] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse

comment:13 Changed 8 years ago by JuergeN

Nice to see some progress here! :)

Concerning your questions about last_accessed, this is just purely the last date stamp when content was accessed (now-1) on the level of every single obejct - be it through the machine or a human beeing. Here yu can see what i get under linux:

root@linux:stat example.file
  File: `example.file'
  Size: 2637      	Blocks: 8          IO Block: 4096   regular file
Device: 1bh/27d	Inode: 406700474   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-07-15 13:40:31.000000000 +0200
Modify: 2013-06-04 16:54:23.000000000 +0200
Change: 2013-06-04 16:54:23.000000000 +0200

comment:14 Changed 7 years ago by jri

Thanks, JuergeN, for feedback!

comment:15 Changed 7 years ago by jri

New Core Event PRE_SEND_RESPONSE (#477, #386).

Fired before the response is send to the client. That is exactly once per request.
This allows plugins to set certain response headers like Last-Modified.

New listener interface:

de.deepamehta.core.service.event.PreSendResponseListener?

Listener method:

void preSendResponse(ContainerResponse? response)

Note: ContainerResponse? is a Jersey-specific class (com.sun.jersey.spi.container.ContainerResponse?).
We should remove the Jersey dependency and move to JAX-RS 2.0 instead.

See ticket 477.
See ticket 386.

Changeset: 1676101075ebb2789642fe7770d0595cb0d1a97e

comment:16 follow-up: ↓ 18 Changed 7 years ago by jri

Time Plugin: send Last-Modified headers (#386).

If the response contains a single entity and if a modified timestamp for that entity is stored in the DB the Last-Modified response header is set accordingly.

The Time plugin provides an OSGi service (no REST service yet):

  • String getTimeCreated(long objectId)
  • String getTimeModified(long objectId)

See ticket 386.

Changeset: d3e987725578630ecab34b390fae448771c5e654

comment:17 Changed 7 years ago by jri

Time plugin: change storage format (#386).

The timestamps are stored in Unix time. The Time API has changed accordingly.
This is to better support range queries in the future.

If you've already installed the Time plugin you should reset your DB.

See ticket 386.

Changeset: 356d6c318c639ff16e0c84471c44bfcfca5ca022

comment:18 in reply to: ↑ 16 Changed 7 years ago by jri

Replying to jri:

Time Plugin: send Last-Modified headers (#386).

If the response contains a single entity and if a modified timestamp for that entity is stored in the DB the Last-Modified response header is set accordingly.

Just one observation regarding caching (see #478): the sole presence of the Last-Modified response header leverages the browser's own caching mechanism. That is the browser automatically adds a If-Modified-Since header to subsequent GET requests to the same resource. So, in conjunction with the new Caching plugin (which operates only at the server-side so far) caching is already effective. No DM webclient modifications were required. Tested with Safari and Chrome.

comment:19 Changed 7 years ago by jri

Add 2 new plugins to reactor build (#386, #478).

DeepaMehta 4 Time
DeepaMehta 4 Caching

See ticket 386.
See ticket 478.

Changeset: b9c5fa2584ff000a6560df1e92ce97b0312d2e27

comment:20 Changed 7 years ago by jri

Time plugin: enrich with timestamp (#386, #479).

If the response contains a single DeepaMehtaObject? its composite model is enriched with a dm4.time.modified entry. The value is a UNIX timestamp. If the modification date is unknown the value is -1.

This way an object's timestamp is accessible at the Webclient application layer (client-side). It could be used for display or for setting the If-Unmodified-Since header in a conditional PUT request.

In preparation of #479 (Edit Conflict Detection).

See ticket 386.
See ticket 479.

Changeset: dfaa8f0ba4ee468cb589c267e41df7265bb2e016

comment:21 Changed 7 years ago by jri

There is a bug: if a topic is just edited and the Edit button is pressed again the next PUT request fails with 412 (Precondition Failed), see #479. This is because the updated topic contained in the directives has no modification date set.

comment:22 Changed 7 years ago by jri

Caching: fix conditional PUT request (#386, #478).

Topics/associations contained in a Directives object are enriched with their modification timestamps.
As a consequence the Caching module uses the up-to-date modification timestamp when constructing a subsequent PUT request.

See ticket 386.
See ticket 478.

Changeset: 4528ffa5017160f17e675cfea16545e563027081

comment:23 Changed 7 years ago by jri

Time plugin: time range queries (#386).

The Time API provides methods for retrieving topics and associations based on a created/modified time range.

Time service:

  • Collection<Topic> getTopicsByCreationTime(long from, long to)
  • Collection<Topic> getTopicsByModificationTime(long from, long to)
  • Collection<Association> getAssociationsByCreationTime(long from, long to)
  • Collection<Association> getAssociationsByModificationTime(long from, long to)

REST API:

  • GET /time/from/{from}/to/{to}/topics/created
  • GET /time/from/{from}/to/{to}/topics/modified
  • GET /time/from/{from}/to/{to}/assocs/created
  • GET /time/from/{from}/to/{to}/assocs/modified

Note: these methods rely on the timestamps and property index recently introduced (4.1.1-SNAPSHOT).
Topics/associations existing in a DM 4.1 installation have no timestamps.

See #386.

Changeset: 60785f0f32083c825bb48f5ea2bd527fd3afa8df

comment:24 Changed 7 years ago by jri

Time and Caching fixes (#386, #478, #479).

FIXES

  • Browser caching is enabled also for topics/associations which have no timestamp. 0 assumed as default (Jan 1, 1970).
  • Edit conflict detection works also for topics/associations which have no timestamp.

CHANGES

Time plugin:

  • for topics/associations send to the client their creation timestamp is available in the composite value as well (besides the modification timestamp).
  • the time API's timestamp getter methods return 0 as default in case no timestamp is stored (before -1 has been returned).
  • the Last-Modified response header is also set for a default timestamp value (0).

Caching plugin:

  • a request's precondiction is also checked for a default timestamp value (0).

See #386.
See #478.
See #479.

Changeset: 1b5bbdb7c6b963915168f5a7c69e6ccb8a13d427

comment:25 Changed 7 years ago by jri

Time API is polymorph (#472, #386).

BREAKING CHANGE

The timestamp getters and setters do not distinguih between topics and associations but take a generic DeepaMehtaObject?.

long getCreationTime(DeepaMehtaObject? object)
long getModificationTime(DeepaMehtaObject? object)

See #472.
See #386.

Changeset: 854a83b2a42941d0b0befeab3655d479358be922

comment:26 Changed 7 years ago by jri

To summarize, now we have:

  • creation and modification timestamps
  • a Time API and REST interface
  • performant range queries (trie-indexing)
  • integration with the caching plugin to leverage the browser cache

This ticket is regarded as complete now.
Pending aspects like the access timestamp and the GUI components are supposed to be addressed in separate tickets.

comment:27 Changed 7 years ago by jri

  • Status changed from accepted to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.