Guides: Research Commons: Data / Metadata Documentation Standards

Overview: what to include in this section

What metadata will be provided to help others identify and discover the data?

Describe the content, formats, and internal relationships of the data in detail

Project Documentation	Dataset Documentation
Rationale and context for data collection	Variable names and descriptions
Data collection methods	Explanation of codes and classification schemes used
Structure and organization of data files	Algorithms used to transform data (may include computer code)
Data sources used (data citation)	File format and software (including version(s)) used.
Transformations of data from the sanitize data through analysis
Information on confidentiality, access and use conditions

Components for Standards: You'll need info from each of the tabs

Where data is stored and how it is kept secure throughout the research process is critical. Funding agencies may require that you retain data for a given period and will likely ask you to explain in your data management plan how you will store and back it up, and how you will manage the security of and access to your data. If you will be working with large data sets (with larger storage and backup needs) you should contact your departmental IT staff.

Be sure to include who will be responsible for ensuring that files are stored and backed-up properly. Funding agencies are increasingly looking for details related to “roles and responsibilities.”

Storage is the act of keeping your data in a secure location that you can access readily. Files in storage should be the working copies of your files that you can access and change regularly.

Backup is the practice of keeping additional copies of your data in separate physical or cloud locations from your files in storage. Backup copies are copies you would access in the case of data loss and needing to access previous versions of your work.

Storage systems often provide mirroring, in which data is written simultaneously to two drives. This is not the same thing as backup since alterations in the primary files will be mirrored in the second copy.

A good rule to go by with storing and backing up copies of your work is LOCKSS (Lots Of Copies Keep Stuff Safe), and to keep each copy as physically far apart from the other copies as possible to prevent damage by natural disaster, such as a fire or flood occurring in the lab where the research is being performed. It’s a good idea to follow the rule of three when thinking about this: you should keep three copies of your data, two backup copies should be kept on different devices or storage media, and one backup copy should be kept off-site. This might look like:

One copy in active storage. This is a copy you are regularly accessing and working on during your research. It will likely be on your computer or a lab’s shared network drive.
A second copy on a different device on- or off-site, such as an external hard drive in your office or a backup server provided by your IT department.
A third copy, preferably off-site. This might be on a cloud application like Box, Google Drive, or another appropriate cloud solution.

Courtesy UW-Madison

Other Considerations:

If any of the following policies affect the management of your data, you will need to address them in a DMP, as they will affect how you can store and share your data.

HIPAA (Health Information Portability and Accountability Act)
- HIPAA’s privacy rule set protections for the privacy of protected health information (PHI) and set limits on sharing it.
FISMA (Federal Information Security and Modernization Act of 2014)
- FISMA works to ensure strong protection over federal information and information systems against cybersecurity threats.
FERPA (Federal Educational Rights and Privacy Act)
- FERPA protects the privacy of data from student records, and applies to all schools that receive funding from the U.S. Department of Education.

DMP Tool: Storage and Security
From DMP.org information about storage and security

Data becomes useful when it has meaning and context associated with it. The most common way to bring context to data is by applying metadata (description and documentation of your data) and through supplementary files, such as a data dictionary. Documenting your data is important for sharing your data, in order for other researchers to understand how to access, view, and possibly re-use your data.

Data dictionaries should provide the key information about the data that you will be collecting, and is used to explain what the variable names and values in a dataset really mean. Data dictionaries are most commonly used when working with tabular data or creating a database.
README files are documents in plain text (.txt) or markdown (.md) format that are often used to describe software packages, programming scripts, and datasets, and can also be used for research projects. It should include information about the creators of the files that it is describing, a list of the files included in the set, relevant funder information, and any associated research outputs, such as articles or presentations. The README should include a citation for the dataset, as well as for any of the byproducts of the research data that was collected and used. For more information about creating a README, see Cornell’s “Guide to Writing ‘README’ Style Metadata.”
- Example: README for a dataset.
A data paper differs from a research paper in that it is “used to present large or expansive data sets, accompanied by metadata which describes the content, context, quality, and structure of the data” (Ecological Society of America). The Ecological Society of America provides a guide on writing a data paper.
- Example: Scientific Data from Nature is a publisher of data papers
A codebook provides descriptions and definitions about the variables and values included in a dataset to assist users in interpreting the data for potential replication or reuse. Codebooks provide variable names and a description for what each variable represents, each variable’s type, the format that the values for each variable should be in, and the range of values, if applicable.
- Example: ICPSR
Metadata is the describing and documenting of your data. There are different ways to do this, depending on the discipline that you are working in and the types and formats of data that you are collecting. The method that you use to describe your data will depend on the project, your team, and the complexity of your data. The documentation for your data should contain the minimum information required to be able to reuse the data that is being described.
- Following disciplinary metadata schemas gives you the opportunity to describe your project, its data, and other outputs, such as publications. When possible, you should follow a disciplinary standard that is common in your field. There are many metadata schemas that are specific to disciplines as well as the format of data being collected.
- General: Dublin Core (Widely used in disciplinary and institutional repositories); Disciplinary Metadata from the DCC (Searchable list of disciplinary metadata standards and related information. Includes biology, Earth science, physical science, social science & humanities and general research data); Altova Schema library (A reference library to common (and uncommon) industry and cross-industry schemas).
- Natural Science: Darwin Core, Ecological Metadata Language (EML), Biodiversity Information Standards (TDWG)
- Social Science: Data Documentation Initiative (DDI)
- Geospatial: Content Standard for Digital Geospatial Metadata (CSDGM), ISO 19115: FGDC Information About Geospatial Metadata, EPA Geospatial Metadata Technical Specification
- Arts and Humanities: Categories for the Descriptions of Works of Art, Dublin Core, Public Broadcasting Core (PBCore), Text Encoding Initiative, Visual Resources Association (VRA)

Courtesy UW-Madison

DMP Tool: Data Documentation
From DMP Tool, guidance for data documentation
All About Metadata
Detailed help from Univ of Central Florida

“Data scientists spend 80% of their time cleaning and manipulating data and only 20% of their time actually analyzing it.” IBM Analytics

Data cleansing (also known as data cleaning) is a process of detecting and rectifying (or deleting) of untrustworthy, inaccurate or outdated information from a data set, archives, table, or database. It helps you to identify incomplete, incorrect, inaccurate or irrelevant parts of the data. By doing this you can then replace, modify, or delete the bad data. Data cleaning can be performed interactively with data wrangling tools, or as batch processing through scripting.

There are 5 data cleansing steps:

Standardize your data
Validate your data
De-duplicate your data
Analyze data quality
Find out if you have a data quality problem

OpenRefine Data Cleansing
OpenRefine, previously known as Google Refine, is an open source, web based data cleaning and transformation tool. OpenRefine allows you to import and clean large datasets in formats such as Excel, XML, RDF as XML, RDF N3 triples, JSON, TSV, and CSV.
8 Ways to Clean Data
A very basic introduction to data cleansing techniques in Excel.
Data Cleaning with R and the Tidyverse: Detecting Missing Values
Learn how to detect missing values using the tidyr and dplyr packages from the Tidyverse.
Data Cleaning with Python and Pandas: Detecting Missing Values
Learn a number of different data cleaning tasks using Python’s Pandas library. Specifically, we’ll focus on probably the biggest data cleaning task, missing values.

A data catalog is a structured collection of data used by an organization. It is a kind of data library where data is indexed, well-organized, and securely stored. Most data catalog tools contain information about the source, data usage, relationships between entities as well as data lineage. This provides a description of the origin of the data and tracks changes in the data to its final form.

The catalog informs users about the available data sets and the metadata around a topic and assists users in locating it quickly.

You can contact your departmental IT staff for help finding a suitable catalog.

Below is a list of data catalogs.

Data Catalogs
From DBMS Tools

“Data curation is the active and ongoing management of data through its lifecycle of interest and usefulness to scholarship, science, and education. Data curation enables data discovery and retrieval, maintains data quality, adds value, and provides for re-use over time through activities including authentication, archiving, management, preservation, and representation.”

-Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign

Data Citation and Linking
Helpful overview from DCC: Digital Curation Centre

Data Curation
Detailed help from Univ of Central Florida
DCC: Curation Lifecycle Model
From the UK Digital Curation Centre

A Data Dictionary is a set of important information about data used within an organization (metadata). This information includes names, definitions, and attributes about data, owners, and creators of assets. Data Dictionary tools provide insights into meaning and purposes of data elements. They add useful aliases about the scope and characteristics of data elements, as well as the rules for their usage and application. (UW Madison)

The OSF (Open Science Framework) provides a tutorial on how to make a data dictionary for tabular data.

Example: USDA’s National Agricultural Library

Data Dictionary Tools
From DBMS Tools