Memento: Updated technical details (May 2011)

Memento: Time Travel for the Web

Herbert Van de Sompel
Robert Sanderson
Michael L. Nelson

http://mementoweb.org/

Memento is funded by
The Library of Congress

Updated Technical Details (May 2011)
Updated Technical Details (05/2011)

Memento wants to make it Easy

to navigate the Web of the Past

Technical Specification
https://datatracker.ietf.org/doc/draft-vandesompel-memento/
2

Tate Online Select Date Tate Online
Today March 16 2008 March 16 2008

From
National Archives

3

Memento achieves this by introducing

a uniform version access capability to

integrate the past and current Web

4

Problem Statement …

5

Resources

6

Resources have Representations

7

Resources have Representations that Change over Time

8

Only the Current Representation is Available from a Resource

9

Old Representations are Lost Forever

10

Archived/Version Resources Exist

11

Resource Versions on the Web

•  Content Management Systems

•  Web Archives

•  Transactional archives

•  Search engine caches

•  …

12

Sep 11 2001, 20:36:10 UTC Dec 20 2001, 4:51:00 UTC

Archived Resources

http://en.wikipedia.org/w/index.php?
http://web.archive.org/web/20010911203610/http:// title=September_11_attacks&oldid=282333 archived
www.cnn.com/ archived resource for http://cnn.com resource for http://en.wikipedia.org/wiki/
September_11_attacks

Updated Technical Details (05/2011) 13

Versions are Not Integrated with the Web

•  Cannot talk about a resource as it
used to exist

•  Cannot access a prior version
knowing the current one

•  Cannot access the current version
knowing a prior one

14

Memento Wants to Integrate the Past and Current Web

15

The Memento Framework:

Protocol to Integrate Past and Current Web

Overview

16

Memento Framework

•  Regards the Web as a big Content
Management System

•  Introduces a uniform capability to
access versions on the Web

•  Does not build new archives but
leverages existing systems that
host versions

17

Memento Framework

•  Is distributed: versions may exist on
several servers

•  Uses time as a global version
indicator

•  Is based on the primitives of the
Web: resource, resource state,
representation, content negotiation,
link

18

Original Resources and Mementos

19

Bridge from Present to Past

20

Bridge from Past to Present

21

Memento Framework

22

Memento Client Server Interaction

23

Memento HTTP Flow

HEAD R, Accept-Datetime

Link  G

GET G, Accept-Datetime

302  M, Vary, Link  M,R,T

GET M, Accept-Datetime

200, Memento-Datetime, Link  M,R,T,G
24



Interesting Cases

25

Multiple Archives

26

Original Resource Gone

27

Original Resource’s Server Gone

28

Original Resource Provides no Link

29



HTTP Headers

30

HTTP Headers used in Memento

•  Defines two new headers:
–  request: Accept-Datetime!
–  response: Memento-Datetime!

•  Introduce new content for two existing headers:
–  response: Vary ; Link!

•  Use one existing headers without modification:
–  response: Location, TCN:!

31

HTTP Request Headers: Accept-Datetime

•  Accept-Datetime 

o  Issued against TimeGate, (Original Resource), (Memento)

o  Header value:
-  Desired datetime of Memento (MANDATORY)
Must be in RFC 1123 format and in GMT

-  Interval indicator to express the client is only interested in
Mementos within the interval (OPTIONAL)
–  Expressed as two ISO8601 durations:
"-P3DT5H;+P2DT6H"!

Accept-Datetime: Mon, 12 Oct 2009 14:20:33 GMT!

32

HTTP Response Headers: Memento-Datetime
•  Memento-Datetime!
o  Returned by Mementos

-  Always. Even when not via a TimeGate
o  Header value: Archival datetime of the Memento

-  Resource has not and will not change beyond that date
o  This header is sticky:

-  Once returned, must always return it with same value
-  Must also be preserved when Mementos are mirrored at
different URIs
•  This header is crucial to allow a client to understand it has arrived
at a Memento
See: http://www.mementoweb.org/guide/resourcetype/

Memento-Datetime: Mon, 12 Oct 2009 14:20:33 GMT!

33

HTTP Response Headers: Vary

•  Vary!
o  Returned by TimeGate

o  Similar to regular content negotiation

o  Header value:

-  negotiate, accept-datetime!

•  TimeGate must first meet the datetime preference, and then – if
possible – other content negotiation preferences

•  Note: accept-datetime value in Vary header is crucial to
allow a client to understand it has arrived at a TimeGate.
See: http://www.mementoweb.org/guide/resourcetype/

Vary: negotiate, accept-datetime!

34

HTTP Response Headers: Location

•  Location 

o  Returned by TimeGate
o  Similar to regular content negotiation
o  Header value: URI of selected Memento

Location: http://web.archive.org/web/20010911223004/
http://cnn.com!

35

HTTP Response Headers: Link

•  Link!
o  Returned by Original Resource, TimeGate and Mementos

o  Various new Relation Types are introduced:

-  “original”!
-  “timegate”!
-  “memento”!
-  “timemap”!
o  HTTP Link Header: RFC 5988

See: https://datatracker.ietf.org/doc/rfc5988/

Link: <http://web.archive.org/web/20010911223004/http://!
cnn.com>;rel="memento";datetime="Mon, 11 Sep 2001 22:30:04 GMT"!

36

Memento HTTP Flow


Link  G
timegate


memento,original,timemap

memento,original,timemap,timegate


•  Link!
o  Returned by Original Resource, TimeGate and Mementos

o  Various new Relation Types are introduced

38



HTTP Interactions

39



HTTP Link Header Details

46


47

Memento HTTP Flow


Link  G
timegate






•  RECOMMENDED "timegate” Link from Original Resource to
TimeGate

•  If this Link is not available, the client must try and find a TimeGate
itself, via:
-  Memento Discovery approaches (see later)
-  User interaction (e.g. Preferences in an application)

49


50

Memento HTTP Flow


Link  G


302  M, Vary, Link  M
memento


200, Memento-Datetime, Link  M
memento


•  MANDATORY ”memento” Links from TimeGate and Memento to
Mementos

•  ”memento” Links point at the following Mementos know to the
responding server:
o  Selected Memento (MANDATORY if a Memento is selected)

o  First Memento, Last Memento (MANDATORY)

o  Memento prev to selected one, Memento next to selected one

(RECOMMENDED)
o  Other Mementos (OPTIONAL, and only if prev and next are

provided)
o  Temporal order of Mementos is expressed using existing Relation

Types (RFC 5829, RFC 5988): first, last, next, prev,
successor-version, predecessor-version

52


•  MANDATORY ”memento” Links from TimeGate and Memento to
Mementos

•  Attributes for a ”memento” Link:
o  datetime (MANDATORY)

datetime of the Memento pointed at by the link
o  license (OPTIONAL)

license associated with the Memento
o  embargo (OPTIONAL)

datetime until which the Memento will remain
inaccessible
o  type (RECOMMENDED)

mime type of the Memento.

53


54

Memento HTTP Flow


Link  M=R
memento,original


•  Both ”memento” and ”original” Link on a resource.

•  The resource is its own Memento, i.e. it is a stable resource.
o  Resource that was born stable or became stable; it will not change

anymore.
o  For example resources with PermaLink on news sites

o  Note the difference with Last-Modified header

•  Can still also provide a ”timegate” Link!
o  For example pointing at TimeGates for Mementos of the resource

before it became stable

56


57

Memento HTTP Flow

GET M, (Accept-Datetime)

200, Memento-Datetime, Link  M,R,T
memento,original,timemap


•  Mementos without a TimeGate, for example:
o  Resources in snapshot archives

o  Version resources in systems that have not yet implemented

TimeGates

•  Should still use Memento-Datetime header
•  Should still use “original” Link
•  Can have “timemap” Link

59


60

Memento HTTP Flow


Link  G


302  M, Vary, Link  T
timemap


200, Memento-Datetime, Link  T
timemap

•  RECOMMENDED ”timemap” Links from TimeGate and
Mementos.

•  A TimeMap is introduced to allow retrieving an inventory of Mementos
for an Original Resource that the responding server is aware of. It lists:
o  URI of Original Resource (MANDATORY)

o  URI and datetime of all known Mementos (MANDATORY)

o  URI of TimeGate for Original Resource (RECOMMENDED)

o  URI of TimeMap itself (RECOMMENDED)

•  Multiple TimeMap serializations possible; link-value format
MANDATORY
o  application/link-format:

see https://datatracker.ietf.org/doc/draft-ietf-core-link-format/

62

Memento HTTP Flow: GET TimeMap

63

Memento HTTP Flow: TimeMap Response



Discovery to Support Integration of Past and Current Web

TimeGate Discovery

65

Batch Discovery of TimeGates: robots.txt

•  robots.txt file is used by Web servers to convey crawling
policies

•  Web crawlers (such as for archives) retrieve and parse it
•  De-facto standard, no official endorsement
•  Extended with new directives, including by Google

User-agent: * # Reject all crawlers!
Disallow: /!
# Google Sitemap extension!
Sitemap: http://some.example.com/me/sitemap.xml!

User-agent: NiceBot # Select only NiceBot!
Crawl-delay: 10 # 10 seconds between requests!

66

Batch Discovery of TimeGates: robots.txt

•  Add TimeGate and Archived directives to support discovery
of TimeGates known to the server
•  User agent should concatenate desired URL with TimeGate link
•  Archived value is truncated host/path or * to describe a general
web archive

http://mementoweb.org/guide/robotstxt/
67


Discovery to Support Integration of Present and Past Web

Discovery via TimeMaps: All Mementos for a given
Original Resource known by an archive

68

TimeMap Overview

•  A TimeMap is an inventory of Mementos for an Original Resource
that the responding server is aware of. It lists at least:
•  URI of Original Resource
•  URI and datetime of all known Mementos
•  URI of TimeGate for Original Resource
•  URI of TimeMap itself

•  Multiple TimeMap serializations possible:
•  application/link-format mandatory

see https://datatracker.ietf.org/doc/draft-ietf-core-link-format/

•  RDF TimeMaps proposed

69

TimeMaps: Link Format Syntax

•  Document in the format of the value of the Link HTTP Header

•  Format:
<URI>;rel="type";attr="val", !
!<URI2>… 

•  rel is the relationship between context URI and the URI in <>s
•  The Context URI for TimeMaps is the URI with rel="original"
•  Other rel types link to Mementos, TimeGates etc.!

<http://cnn.com/>;rel="original", 
<http://web.archive.org/web/20010911223004/http://!

70

TimeMaps: Link Attributes

•  Existing Attributes for Links
•  "rel" The type of relationship
•  "type" The (mime) formatt of the linked resource
•  "title" Title of the linked resource
•  "hreflang" !Language of linked resource
•  "media" Intended media (eg screen)
•  "anchor" URI to override context URI for link

•  New rel types introduced:
•  "original" The Original Resource
•  "memento" A Memento of the Original
•  "timegate" A TimeGate for the Original
•  "timemap" A TimeMap of Mementos of the Original!

71

TimeMaps: Link Attributes

•  New Attributes for Mementos:
•  datetime The Memento-Datetime!
•  license!License associated with Memento
•  embargo!Time after which Memento is available
•  Are others necessary?!

<http://cnn.com/>;rel="original",!
<http://web.archive.org/timegate/cnn.com/>;rel="timegate",!
<http://web.archive.org/timemap/link/cnn.com/>;rel="timemap",!
<http://web.archive.org/web/20010911223004/http:// 
<http://web.archive.org/web/…/cnn.com/>;rel="memento"; 
license="http://archive.org/license/1"; 
datetime="…";embargo="Mon, 20 Jul 2011 00:00:00 GMT",!
… !

72


Discovery to Support Integration of Past and Current Web

Memento Discovery: All Mementos known by an
archive

73

Batch discovery of Mementos: robots.txt

•  robots.txt file is used by Web servers to convey crawling
policies

•  Support discovery of Mementos through robots.txt via existing
User-agent and Allow directives
•  Use value memento for User-agent to convey that the value
for the Memento-Datetime header must remain sticky when
crawling/mirroring Mementos

74

Batch discovery of Mementos: Memento Feeds

•  Concept:
•  Archives publish feeds in which each entry provides details
about a specific Memento, e.g. Memento-Datetime, Original
Resource, etc.
•  As new Mementos become available, new feeds with new
entries are published
•  Once published, feeds remain static

•  Technology:
•  To be decided in collaboration with IIPC
•  Inspired by the approach and functionality of CDX files (see
http://www.archive.org/web/researcher/cdx_legend.php) but:
•  With Memento-specific extensions;
•  Possibly using different serialization;
•  Including mechanisms to discover these feeds.

75


Tools

76

Memento Client Support

•  Several client tools developed by
us and others

•  Add-ons for FireFox (operational)
and Internet Explorer
(experimental)

•  Applications for Android
(operational) and iPhone/iPad (in
development)

•  Paper in Code4Lib Journal
http://journal.code4lib.org/articles/4979

77

Memento Server Support

•  Plug-in for MediaWiki (operational)

•  Used on W3C’s main wiki

•  Please install it for your MediaWiki!

http://www.mediawiki.org/wiki/Extension:Memento
78

Memento Server Support

•  Memento-compliant Wayback
software:

•  In production at the Internet
Archive

•  Available to Web archives,
worldwide

•  Please have your favorite Web
Archive experiment with the
new 1.6 version!

http://mementoweb.org/tools/wayback/
79

Memento Server Validator

•  Server side client:

•  Attempts to perform all
Memento actions against a
given URI

•  Reports success/failure of the
interactions and warnings for
optional aspects

•  Kept up to date with IETF
Internet Draft

http://mementoweb.org/tools/validator/
80

Memento Proxy Support

•  Several systems that host
Mementos made Memento-
compliant “by proxy”

•  Many major Web Archives that
do not yet run Memento-
compliant software
•  3,000+ MediaWiki systems,
including Wikipedia, Wikia

•  We would love all of these to
become natively Memento
compliant!

81

Memento Aggregator TimeGate

•  Aggregates all known TimeGates
•  Proxies
•  Native Implementations

•  Redirects to authoritative
TimeGates (Wikipedia,
Transactional Archives)

•  Currently implemented with
BerkeleyDB
•  Future version to use
FaceBook's Cassandra platform

82

Aggregators Find More Mementos!

•  1000 URIs sampled from delicious.com
•  1 dot = 1 Memento (x=Memento-Datetime, y=URI of Original Resource)
•  Sorted by URI longevity
83

But Still Too Few Mementos To Be Found…

•  1000 URIs sampled from search engine result pages;
•  See: “How Much of the Web is Archived?” JCDL 2011
84

Crawl-Based Web Archives

Observations
For example: Heritrix crawler for Internet Archive

85

Crawl-Based Web Archives

•  Collect discreet observations of resources, not their entire
evolution.

•  Can be rejected (robots.txt, by user-agent, by host IP)

•  Can be deceived (cloaking, by geo-location, by user-agent).

•  Coverage of particular Web server dependent on crawl-strategy.

86

Server-Side Transactional Web Archives

Change History
For example: TTApache, PageVault, Vignette Web Capture

87

Server-Side Transactional Web Archives

•  Collect all representations served by to-be-archived server.

•  To-be-archived server needs to cooperate.
•  Incentives e.g. institutional memory, official record of Web
presence.

•  Archival coverage restricted by to-be-archived server, does not
include external servers (e.g. embedded resources).

•  To be archived server can submit falsified information.

•  Archival collection management: what to keep, what not (e.g.
significant changes, deduplication, …).

88

Development of Transactional Web Archive Software
Capture:
•  Apache connection filter module captures URI, headers, body
•  POSTs in real-time to transactional archive

Access:
•  Online, real time access via Memento TimeGates
•  Batch Export via WARC files for long term preservation

89

Capture:
•  Apache connection filter module captures URI, headers, body
•  POSTs in real-time to transactional archive

Submit:
•  Java-Grizzly-Jersey submission interface application
•  Berkeley DB metadata store
•  FS store for body and headers

90

Access:
•  Transactional archive natively supports Memento
•  Immediate availability of archived content
•  Export of WARC, e.g. for long-term archiving in other environment

Development Timeline:
•  Ongoing development (LANL) and testing (ODU)
•  Submit/Access finalized; coding focus on collection management
•  Expected release as open source, 3rd Quarter 2011

91


Resource Versioning

92

93

94

95

96

97

98

99

Memento Framework

Original Resource: http://lanlsource.lanl.gov/pics/picoftheday.png

100

Time Travel across Versions of a Picture of the Day

Movie at: http://www.mementoweb.org/demo/picoftheday.mov


Memento Framework

Original Resource: http://dbpedia.org/resource/France

107

Time-Series Analysis across DBpedia Versions

Data collected through HTTP Navigation

paper at http://arxiv.org/abs/1003.3661
108


About Memento-Datetime:

Archive Navigation Coherence

109

Resource History Recorded by CMS and Transactional Archives

110

Navigate foo.html @ t4

111

Navigation Coherence for foo.html @ t4

112

Resource Observations Recorded by Crawler-Based Archives

113

Missed Observations

114

Navigate foo.html @ t4

115

Navigation Incoherence foo.html @ t4

116

Increase Coherence with Observations from Multiple Archives?

117


About Memento-Datetime:

Relation to Creation Datetime and Last-Modified

118

Three Notions of Time

•  Creation: Datetime when the resource first came into being
•  Last-Modified: Datetime when the resource was last changed
•  Memento-Datetime: Datetime that the resource was “frozen”, e.g.
as a result of:
o  Archiving it at a different URI (e.g. in a CMS, Web Archive,
Transactional Archive, Snapshot Archive);
o  Deciding never to change it anymore and keeping it at its
original URI.

119

Creation = Memento-Datetime = Last-Modified

Cr
MD
LM

At a particular point in time,
the Original Resource is
observed, and the associated
Memento is created. All time
values for the Memento are
the same.

120

Creation = Memento-Datetime < Last-Modified

Cr
MD
LM

The HTML archive banner
added to a Memento
necessitates a change in
Last-Modified of the
Memento, but Creation date
and Memento-Datetime
remain unchanged.

121

Creation < Memento-Datetime <= Last-Modified

Cr
MD
LM

It is possible that the Original
Resource was a placeholder
resource and returned a 200
response before it started to
identify a Memento (URI-
R=URI-M).

122

Memento-Datetime < Creation <= Last-Modified

Cr
MD
LM

If a Memento is copied to a
new archive, the copied
Memento has a Creation and
Last-Modified equal to the
time of copying. The
Memento-Datetime is “sticky”
and is the same for the
Memento and its copy.

123


Persistent Web Annotations

124

Web-Centric Annotation: No Persistence

Google Sidewiki Annotation on http://news.bbc.co.uk/ as of 2010-06-14
125

Web-Centric Annotation: No Persistence

Archived page from:
http://www.dracos.co.uk/work/bbc-news-archive/2010/03/08/07.05.html
126

Web-Centric Annotation: Desired Persistence

127

Open Annotation: Dealing with Web Time

•  As regular Web resources, Body and Target of an Annotation have
representations that can change over time.

•  Body and Target can change independently of each other.

•  If an Annotation involves resources as they existed at a particular point
in time, this needs to be recorded.

•  The OAC model provides hooks for doing so:
•  Timeless Annotations;
•  Uniform Time Annotations;
•  Varied Time Annotations.


Open Annotation: Uniform Time Annotations
•  The Annotation is not always applicable, but pertains to the state of the
Body and Target at a specific moment in time.

•  Add oac:when property to the Annotation.


Memento + Open Annotation: Persistent Annotations
•  In order to reconstruct the Annotation as intended: Use Memento to
obtain an archived representation of B and T as they existed at the
oac:when datetime.


Create an Annotation

131

Reconstruct the Annotation without Memento

132

Reconstruct the Annotation with Memento

133


The Increasing Value of a URI

134

URI as Access Point to a Page


URI as Access Point to Page and Data


URI as Access Point to Current and Past Pages and Data


References

•  Van de Sompel, H., Nelson, M.L., Sanderson, R.,
Balakireva, L., Ainsworth, S., Shankar, H. (2009)
Memento: Time Travel for the Web.
http://arxiv.org/abs/0911.1112
•  Van de Sompel, H., Sanderson, R., Nelson, M.L.,
Balakireva, L., Ainsworth, S., Shankar, H. (2010) An
HTTP-Based Versioning Mechanism for Linked Data.
Proceedings of the 3rd Workshop on Linked Data on the
Web. http://arxiv.org/abs/1003.3661
•  Sanderson, R., and Van de Sompel, H. (2010) Making Web
Annotations Persistent over Time. Proceedings of the
10th ACM/IEEE-CS Joint Conference on Digital libraries.
http://arxiv.org/abs/1003.2643
•  Sanderson, R., Van de Sompel, H. (2011) Open Annotation
Alpha3 Data Model Guide.
http://www.openannotation.org/spec/alpha3/

138

Memento wants to make navigating the Web’s Past Easy

http://mementoweb.org/
http://groups.google.com/group/memento-dev
139

Memento: Updated technical details (May 2011)

Convert to study guideBETA

More Related Content

Memento: Updated technical details (May 2011)