cif applications

Journal of
Applied
Crystallography Open Database – an open-access
Crystallography collection of crystal structures
ISSN 0021-8898

Saulius Gražulis,a* Daniel Chateigner,b Robert T. Downs,c A. F. T. Yokochi,d

Received 24 October 2008 Miguel Quirós,e Luca Lutterotti,f Elena Manakova,a Justas Butkus,a Peter Moeckg
Accepted 4 May 2009
and Armel Le Bailh

a
Institute of Biotechnology, Graiciuno 8, LT-02241 Vilnius, Lithuania, bCRISMAT-ENSICAEN, Université de Caen
This article is dedicated to Michael Berndt.
Basse-Normandie, Bd. M. Juin, 14050 Caen, France, cDepartment of Geosciences, University of Arizona,
Tucson, Arizona 85721-0077, USA, dSchool of Chemical, Biological and Environmental Engineering, 315
Gleeson Hall, Oregon State University, Corvallis, OR 97331-2702, USA, eDepartamento de Quı́mica Inorgánica,
Facultad de Ciencias, Universidad de Granada, 18071 Granada, Spain, fDepartment of Materials Engineering,
University of Trento, via Mesiano, 77-38050 Trento, Italy, gPortland State University, Department of Physics, PO
Box 751, Portland, OR 97207-0751, USA, and hUniversité du Maine, Laboratoire des Oxydes et Fluorures, CNRS
UMR 6010, Avenue O. Messiaen, 72085 Le Mans Cedex 9, France. Correspondence e-mail: grazulis@ibt.lt

The Crystallography Open Database (COD), which is a project that aims to

gather all available inorganic, metal–organic and small organic molecule
structural data in one database, is described. The database adopts an open-
access model. The COD currently contains 80 000 entries in crystallographic
information file format, with nearly full coverage of the International Union of
Crystallography publications, and is growing in size and quality.

1. Introduction This announcement introduced some new members to the effort

(Brian Toby and Alexandre Yokochi) and, by the end of March 2003,
The Crystallography Open Database (COD) is a recent tool offered the number of entries in the COD had increased to more than 5000.
to the scientific community on the Web at http://www. In order to ensure quality and standardization of uploaded files, the
crystallography.net/. It was founded in February 2003 as a response to CIF2COD computer program was built by modifying CIF2SX with
Michael Berndt’s letter published in the Structure Determination by permission from Louis Farrugia. The first COD search page was
Powder Diffractometry (SDPD) mailing list (http://tech.groups. coded in the PHP language. Uploads of crystallographic information
yahoo.com/group/sdpd/message/1016). files (CIFs) continued in April 2003 (1200 files from the Institut de
The historical fragmentation of structural data into three databases Physique de la Matière Condensée, Grenoble) and after four months
covering inorganic compounds (Inorganic Crystal Structure Data- the number of entries in the COD surpassed 12 000 (Le Bail, 2003) as
base; Belsky et al., 2002), metals, alloys and intermetallics (Crystal a result of uploads by individuals, laboratories and data shared by the
Data for Metals Database, CRYSTMET; White et al., 2002), and AMCSD.
organic and organometallic small molecules (Cambridge Structural In December 2003, a subset of the COD was created and named
Database; Allen, 2002) reflects the fact that in the past most public or PCOD (Predicted Crystallography Open Database) with the goal of
private research laboratories concentrated their activity on one or the gathering computationally predicted structures and with the expec-
other of these very specialized topics. However, nowadays tation that the number of predicted entries could easily exceed the
researchers at a given laboratory frequently extend their activities to number of experimentally determined ones. In January 2004, the
all classes of compounds, focusing instead on materials with specific PCOD offered 200 entries.
properties within more general classes of compounds, such as nano- By October 2005, the COD contained 20 000 entries. The database
materials, hybrids or gas storage. Consequently, there is now a need to also had 30 new volunteers along with three new COD Advisory
have access to all these databases simultaneously [together with Board members (Saulius Gražulis, Miguel Quirós Olozábal and Peter
powder data from the International Center for Diffraction Data Moeck). With the help of these volunteers, the number of entries in
(Kabekkodu et al., 2002) that are generally added for identification the COD increased to 48 000 by December 2006, now including
purposes]. 10 000 structures from the AMCSD (Downs & Hall-Wallace, 2003;
Thus, a group of scientists (Armel Le Bail, Luca Lutterotti and Downs, 2008). In February 2007, a massive PCOD update boosted the
Lachlan Cranswick) responded quickly to Michael Berndt’s letter number of entries to more than 60 000, with the help of the GRINSP
and teamed up to create an open crystallography database. The group software (Le Bail, 2005) for crystal structure prediction. At the same
contacted Professor Robert T. Downs who generously offered strong time, using the PCOD data, the Predicted Powder Diffraction
support for the concept, including the data set from the American Database (P2D2; Le Bail, 2008) was created, which provides identi-
Mineralogist Crystal Structure Database (AMCSD; Downs & Hall- fication by a search–match procedure similar to that of the Powder
Wallace, 2003; Downs, 2008), along with MySQL/PHP scripts written Diffraction File (Kabekkodu et al., 2002).
by Hareesh Rajan. At the same time, Daniel Chateigner joined, and In September 2007, the IUCr Executive Committee decided that
less than three weeks after the letter from Michael Berndt, the COD the CIFs associated with structural papers published in IUCr journals
project was announced through various Internet media (newsgroups, should be made freely available to all databases, including the COD,
various mailing lists and what’s new pages). giving the COD permission to routinely download new files from the

726 doi:10.1107/S0021889809016690 J. Appl. Cryst. (2009). 42, 726–729

 cif applications
IUCr site. This very welcome decision brought about a reorganization REF format is intended to be used in those cases where old data,
of the COD with the center of operations being transferred from Le predating the CIF era, need to be keyed in by hand or converted from
Mans (France) to Vilnius (Lithuania) in December 2007. some other format.
Five years after its foundation, in 2008, the COD passed a major
milestone by archiving the 50 000th entry, while PCOD climbed over 2.2. COD deposition procedure and validation
the 100 000 structure limit in the same year. Our actions to date are Data in REF format, or occasionally in some other formats, are
but the start of this database, and the COD hopes that more crys- converted into the CIF format and then enter the same validation and
tallographers will upload their results in order to accelerate its deposition procedure as CIFs (Fig. 1). Each CIF is checked for syntax
completion. errors, using both the publicly available ‘vcif’ tool from the IUCr
New developments at the COD including automation of data (McMahon, 1998) and our own CIF parser written in Perl (http://
deposition, data validation and correction, a novel search interface, www.perl.com/; Wall et al., 2000). Syntactic errors, if any, must be
and mirror sites and their synchronization, as well as the calculation corrected manually; this task is currently performed by a COD
of powder diffraction patterns, are briefly described in this paper. maintainer responsible for deposition. When the syntax is correct,
structures are assigned a new range of sequential COD numbers.
Bibliographic information is taken either from the data sections of
2. Methodology the CIFs, from the data_global sections, or from auxiliary files in
2.1. COD and PCOD contents BibTeX (Patashnik, 2003) or PubMed XML format (http://
www.ncbi.nlm.nih.gov/entrez/query/static/overview.html). As a last
The COD and PCOD each consist of two major parts: an SQL resort, the bibliographic information may be taken from the names of
database and a collection of structure data files. The structure files
the directories containing CIFs, which are then chosen to reflect
record crystallographic data that were published in peer-reviewed journal, year and journal issue, or the name of the donating person
scientific journals, or that were determined or predicted and donated and laboratory. Each separate CIF is given a full copy of available
by established crystallographic laboratories. The master copy of the bibliographic information, so that it can be further processed and
data is recorded in CIF format (Hall et al., 1991). stored independently.
From the master copy of the (P)COD data collection, data tables The CIFs now can be validated to check whether all necessary data
for the (P)COD SQL databases are generated. These tables abstract items such as cell parameters, symmetry or bibliography are present.
the most important crystallographic, chemical and bibliographic When all quality checks are passed, the existing COD database is
information and are used for online searches. Currently, the data scanned for duplicates. Duplicate structures are as a rule not
tables contain cell constants (a, b, c, , , ), cell volume, Hermann– deposited into the COD. A structure is considered a potential
Mauguin space-group symbol, a summary chemical formula, the duplicate if its cell constants are within 0.5 Å and cell angles within
number of distinct chemical elements, and a descriptive text that 1.2 of any existing entry, the summary chemical formulas match, and
includes the chemical names of the substance and bibliographic both structures have been published in the same paper. If pressure
references. A special field, coeditor code, is also included in the and temperature are specified, these are also checked, and structures
database, in order to generate URL links to the original papers for are considered duplicates only if they were measured under identical
those journals that accept data sharing (currently, the IUCr journals). conditions. All potential duplicates are flagged and reviewed manu-
We check and, when possible, restore systematic and trivial names ally.
of the reported chemical compounds and their formulas (IUPAC, The final step involves insertion of the CIFs into the Subversion
structural and summary), since this information is vital for identifi- repository and insertion of the data dump into the COD SQL table.
cation of the material. Information about chemical and hydrogen The checked CIFs are presented to the CIF2COD program, which
bonds is preserved in the COD CIFs if present in the original data file, computes some derived data and creates a data dump that can be
but is not otherwise inferred from the structure. loaded into the COD MySQL table. The new structures become
Each structure deposited in the COD and the PCOD gets a unique available on the Web immediately after deposition.
seven-digit number, a (P)COD identifier. If a structure of a
Automated procedures have been developed to simplify the
compound is redetermined, with higher precision or under different submission of data for users. For several years now, the MAUD
conditions, it will be deposited in the (P)COD under a new (P)COD software (Lutterotti et al., 1999) has included algorithms for
number. submission. Such functionality makes it simple to submit data to the
Since the COD identifier of a structure, once assigned, remains COD; submission does not even require a visit to the COD Internet
unchanged, a problem might arise when a deposited COD file needs page. The ‘Submit Structure to COD’ submenu lists the CIF of one of
to be changed for some reason, say, a syntax or data error must be the actual phases of a given analysis. The corresponding window
corrected after the deposition. Currently, we have adopted a version allows manual modification of the file, if necessary, before the
control system called Subversion (Collins-Sussman et al., 2008). Each submission is completed by simply clicking the ‘Submit to COD’
change of any COD file is recorded in a central COD repository, and button. After submission, the uploaded CIF is treated identically to
the new version of the file automatically gets a new revision number. other ‘regular’ submissions from the Internet.
These numbers, along with the COD repository address, are inserted
by the software into the COD file header. There is a publicly acces-
sible interface to the COD that allows older revisions of any file to be 3. Discussion
extracted and the COD change logs recorded by COD maintainers to
be read. Having a COD number and the revision number of a file, it is 3.1. Current status of COD and PCOD
always possible to restore a previous version of that file. Currently, the COD stores over 80 000 structures of small organic
Structures are accepted in two formats – standard CIF format and metal–organic molecules, inorganics, and minerals. The PCOD
(http://www.iucr.org/resources/cif/) and a very simple REF (http:// contains over 100 000 predicted inorganic crystal structures in CIF
www.crystallography.net/ref.html) format, devised by A. Le Bail. The format, generated by the GRINSP programs (Le Bail, 2005). The

J. Appl. Cryst. (2009). 42, 726–729 Saulius Gražulis et al.  Crystallography Open Database 727
cif applications
number of different structure types is close
to 30 000, the total number being attained by
adding series of isostructural virtual
compounds. For instance, there are 6400
different (Al/P)O4 compounds, and three
other series of isostructural compounds with
formulations SiO2, (Al/Si)O4 and (Al/S)O4.
Besides the possibility of searching through
the PCOD web interface, in a similar way to
the COD, 48 complete series of compounds
(characterized by the presence of the same
chemical elements) are downloadable for
prospective research.
For anybody who wishes to use the COD
and the PCOD databases, the collected files
are presented using standard open protocols
and formats. The database can be searched
online on the COD server using a simple
web-based search form, and the structural
results can be downloaded either one by one
or in a compressed .zip file. Alternatively, the
whole collection of the COD files and data-
base tables can be downloaded from the
COD web site (using the http protocol) as a
compressed .zip, .tar.gz or .tar.bz2 file, or
updated via an rsync protocol (http://samba.
anu.edu.au/rsync/) from rsync://www.
crystallography.net/cod-cif and rsync://
www.crystallography.net/pcod-cif so that the
files can be used and examined on a user’s
local machine. Finally, the COD and PCOD
CIFs, database dumps and web scripts are
available for anonymous checkouts from
the COD Subversion server (svn://
www.crystallography.net/cod and svn://www.
Figure 1
crystallography.net/pcod). From this server The COD deposition procedure. In this data flow diagram, circles indicate automatic processes and arrows show
an interested user can reconstitute locally the data paths. As in control flow diagrams, a trapezoid indicates manual processes and a rhomb indicates a
the whole COD database and the web site process where a decision to divert data via different paths is taken. Names after the colons in each node are the
for local searches, and also browse COD names of the Unix tools or COD-specific programs that were used for that operation. Rectangles are abstract
(web) data sources – data sources depicted in pink provide crystallographic and chemical information
deposition logs and retrieve older revisions, (coordinates, symmetry data, formulae), while those depicted on a blue background provide bibliographic data.
should they be necessary. Cylinders denote internal COD disk storage facilities (databases). File extensions indicate file formats used. The
To facilitate the use of the COD as a .mrk file format is an intermediate format similar to XML designed for ease of parsing and editing, and used only
internally by the COD deposition scripts.
reference database, it is planned that all data
published in the COD will be assigned
persistent URLs. Thus, any structure deposited in the COD should be human depositors for possible errors. Both steps can be automated
available as http://www.crystallography.net/cif/hCOD numberi, e.g. and parallelized. Finally, the structures still requiring human atten-
http://www.crystallography.net/cif/1000000.cif. tion can be checked and edited in parallel by numerous COD
The open-access nature of the COD and the PCOD permits the reviewers all over the world, provided there is adequate software and
creation of numerous mirrors of the COD and the PCOD. At present, enough volunteers participate in COD maintenance. Currently, the
three mirrors are available at http://cod.ibt.lt/, http://cod.ensicaen.fr/ number of people contributing or willing to contribute to the
and http://nanocrystallography.org/. Currently, one centralized repo- development of the COD amounts to several dozen, apparently
sitory is kept as an authoritative source of data, but with the growth enough to provide qualified peer-review for the incoming structures.
of the databases a decentralized implementation is possible. The development of the automatic data submission, annotation and
CIF correction software is under way. Calculation of powder patterns
is implemented for the PCOD data in the Match! software (http://
3.2. Future directions of COD and PCOD development www.crystalimpact.com/match/match18.htm).
A current challenge for all crystallographic databases, including For researchers who wish to publish their structure-related work,
the COD, is an exponential increase in the number of determined most journals require the deposition of structures with a crystal-
structural data entries. Fortunately, there is plenty of room to lographic database and ask for the database accession number as
improve the efficiency of the COD deposition procedure. The current proof of deposition. For such structures, a special deposition status,
procedure involves a step in which a COD number is assigned by ‘on hold until publication’, will be introduced. The structures
COD coordinator, and a step where the structures are checked by submitted to the COD with the ‘on hold’ flag will be included in the

728 Saulius Gražulis et al.  Crystallography Open Database J. Appl. Cryst. (2009). 42, 726–729
 cif applications
COD SQL database where their cell constants, composition, Laboratoire des Oxydes et Fluorures, Institut de Physique de la
symmetry and authorship will be indicated. A COD number will be Matière Condensée and Portland State University for donating
assigned to the structure and returned to the author, and will be original coordinate files, collected or determined in their premises.
visible through the search interface of the COD. The atomic coor- We acknowledge the numerous volunteers who helped to establish
dinates themselves, however, will not be released to the public until the COD data collection by donating data from their private
either the publication describing them appears, the authors inform collections of structural data. Many thanks to Patrick Ducrot from
the COD team that the coordinates should be released, or one year the Ecole Nationale d’Ingénieurs de Caen (ENSICAEN), France, for
elapses from the original deposition of the CIFs. If the structure is not establishing and maintaining the mirror COD web site, and to Amber
published within one year, an e-mail will be sent to the depositing Lauer of Portland State University and Boris Dušek of the Charles
author asking whether the structure should be released or withdrawn. University in Prague for setting up and maintaining the Portland
At present, one of the main limitations of the functionality of the State University COD mirror. The authors thank Virginijus Šikšnys
COD is the absence of a substructure search engine. In organic and from the Institute of Biotechnology (Vilnius) for comments and
metal–organic chemistry, the best way of defining such similarity is discussion of the manuscript. The work of JB and SG during the
generally the presence of a common group of atoms chemically linked summer of 2007 was partly supported by the Lithuanian Science
in the same way: this is what we call a ‘substructure’. For performing Council Student Research Fellowship Award. The COD Advisory
this task with COD data, we need to represent the chemical Board thanks Crystal Impact GbR for financial support of the open
connectivity of the structures included in the COD in a suitable access publication of this paper.
format, provide a tool for the user to input into the COD the defi-
nition of the substructure and finally employ a search–match engine References
that compares the user input against the COD data. A specialized
Allen, F. H. (2002). Acta Cryst. B58, 380–388.
chemical format such as CML (http://en.wikipedia.org/wiki/ Belsky, A., Hellenbrandt, M., Karen, V. L. & Luksch, P. (2002). Acta Cryst.
Chemical_Markup_Language) or SMILES (http://www.opensmiles. B58, 364–369.
org/) with molecules already ‘grown’ across any possible crystal- Collins-Sussman, B., Fitzpatrick, B. W. & Pilato, C. M. (2008). Version Control
lographic symmetry elements and simplifying the possible presence with Subversion. Sebastopol: O’Reilly Media.
Downs, R. T. (2008). American Mineralogist Crystal Structure Database, http://
of chemically identical but crystallographically different moieties rruff.geo.arizona.edu/AMS/amcsd.php.
could be used for encoding the necessary information. The tools for Downs, R. T. & Hall-Wallace, M. (2003). Am. Mineral. 88, 247–250.
user-friendly structure input and search are available under both free Hall, S. R., Allen, F. H. & Brown, I. D. (1991). Acta Cryst. A47, 655–685.
and commercial licenses (http://xdrawchem.sourceforge.net/, http:// Kabekkodu, S. N., Faber, J. & Fawcett, T. (2002). Acta Cryst. B58, 333–337.
www.cambridgesoft.com/software/ChemOffice/, http://sourceforge. Le Bail, A. (2003). IUCr Commission on Crystallographic Computing
Newsletter, No. 2, pp. 39–41.
net/projects/joelib/, http://sourceforge.net/projects/cdk/). The Le Bail, A. (2005). J. Appl. Cryst. 38, 389–395.
remaining task is the integration of these tools with the COD. Le Bail, A. (2008). Powder Diffr. 23, S5–S12.
Lutterotti, L., Matthies, S. & Wenk, H.-R. (1999). Proceedings of the Twelfth
The authors and the COD Advisory Board are grateful to the International Conference on Textures of Materials (ICOTOM-12), Vol. 1,
International Union of Crystallography for permission to automate pp. 1599–1604. Ottawa: NRC Research Press.
the download of published supplementary data and bibliographic McMahon, B. (1998). vcif, http://www.iucr.org/resources/cif/software/archived/
records of the IUCr online journals. We thank the Mineralogical vcif-0.58.
Patashnik, O. (2003). TUGboat, 24, 25–30.
Society of America, the Mineralogical Association of Canada, Wall, L., Christiansen, T. & Orwant, J. (2000). Programming Perl, 3rd ed.
Laboratoire de Cristallochimie et Physicochimie du Solide, Labor- Sebastopol: O’Reilly Media.
atoire de Cristallographie et Sciences des Matériaux (CRISMAT), White, P. S., Rodgers, J. R. & Le Page, Y. (2002). Acta Cryst. B58, 343–348.

J. Appl. Cryst. (2009). 42, 726–729 Saulius Gražulis et al.  Crystallography Open Database 729