Guides: Research Commons: Data & Text Analysis Software

Before you begin . . .

Text and data mining is unique and thus, can take different timeframes to accomplish. A successful and time-efficient project begins with preplanning.

Step One starts with completing the outline below:

What are the goals of my project?
What data sources are available that meet my needs? How do I want my results to look--the form of my results?
- The text analysis method you choose will depend on your research question.
What funding needs may this project incur?
- How much money and potential sources?
What skills are needed to carry out this project?

TDM (Text and Data Mining) is frequently a fair use under US copyright law, but some resources maybe restricted by license agreement.

Be aware that some vendors restrict automatic downloading; some may require purchase of the data.
Each user is responsible for ensuring that he or she uses products solely for noncommercial, educational, scholarly or research use. Systematic downloading, distribution of content to non-authorized users or indefinite retention of substantial portions of information is strictly prohibited.
The use of software such as scripts, agents, or robots, is generally prohibited and may result in loss of access to these resources for the entire Illinois State University community.
Please refer to the Illinois State University Information Technology Policy

"TDM (Text and Data Mining) is a broad term for developing research practices that involves building and processing a corpus: a collection of text that may contain millions or even billions of words. Remember, less is (becomes) more as you build your corpus --see below -- as it will help avoid hidden biases and unpredictable gaps in coverage.

A source or database describes data collected without a strong organizing principle.

A corpus describes data collected and organized with specific questions in mind about certain geographic regions, time periods, or social phenomena.

Univ of Pennsylvania Guide TDM

Directory of Free API's
An Application Programming Interface, or API, is basically an interface that allows applications to talk to one another. They can be used in a variety of ways, including downloading large amounts of data from a website without requiring user input. In this way, a researcher can even download the entire contents of a digital library hands-free. Using an API does require some technical or programming knowledge. Some, but not all, resources in this guide require use of an API to access data.
From UIUC Guide
MIT Library: API's for Scholarly Resources
Explains API and provides some resources and links.

“Data scientists spend 80% of their time cleaning and manipulating data and only 20% of their time actually analyzing it.” IBM Analytics

Data cleansing (also known as data cleaning) is a process of detecting and rectifying (or deleting) of untrustworthy, inaccurate or outdated information from a data set, archives, table, or database. It helps you to identify incomplete, incorrect, inaccurate or irrelevant parts of the data. By doing this you can then replace, modify, or delete the bad data. Data cleaning can be performed interactively with data wrangling tools, or as batch processing through scripting.

There are 5 data cleansing steps:

Standardize your data
Validate your data
De-duplicate your data
Analyze data quality
Find out if you have a data quality problem

OpenRefine Data Cleansing
OpenRefine, previously known as Google Refine, is an open source, web based data cleaning and transformation tool. OpenRefine allows you to import and clean large datasets in formats such as Excel, XML, RDF as XML, RDF N3 triples, JSON, TSV, and CSV.
8 Ways to Clean Data
A very basic introduction to data cleansing techniques in Excel.
Data Cleaning with R and the Tidyverse: Detecting Missing Values
Learn how to detect missing values using the tidyr and dplyr packages from the Tidyverse.
Data Cleaning with Python and Pandas: Detecting Missing Values
Learn a number of different data cleaning tasks using Python’s Pandas library. Specifically, we’ll focus on probably the biggest data cleaning task, missing values.

Software Tools

D3js
Data-Driven Documents. A JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS.
DataONE Software Tools
List of many software tools
Dirt Directory
A registry of digital research tools for scholarly use.
GitHub
Find (and share) Code and datasets

ICPSR (Inter-University Consortium for Political and Social Research)
Searchable data archive covering demography, crime, economics, education, elections, health care, aging and substance abuse. To download data files, create a free account.

LaTeX
A high-quality typesetting system; it includes features designed for the production of technical and scientific documentation. LaTeX is the de facto standard for the communication and publication of scientific documents.
Milner LibraryGuide General Statistical Sources
Open Data Kit
A community that produces free and open-source software for collecting, managing, and using data in resource-constrained environments.
Piktochart
Create Infographics
SurveyCTO
Design forms, collect data, monitor for quality, and learn.
UIUC Library: Additional Downloadable Tools
List of ready-to-use digital tools.

Atlas.ti
Qualitative analysis of large bodies of textual, graphical, audio and video data.
Compatible with English, Spanish & German.

Dedoose
A cross-platform app for analyzing qualitative and mixed methods research with text, photos, audio, videos, spreadsheet data and more.
Compatible with most languages.
Dedoose Videos
See firsthand how Dedoose can benefit you and your work including: the steps of establishing your account, importing your resources and descriptors, building and perfecting your coding system, excerpting your texts, applying the relevant codes, exploring and interpreting your work, and, exporting your final results in a form ready for use in your article or report.

Ensembl Variant Effect Predictor (VEP)
VEP determines the effect of your variants (SNPs, insertions, deletions, CNVs or structural variants) on genes, transcripts, and protein sequence, as well as regulatory regions.

Ingenuity Pathway Analysis
QIAGEN Ingenuity Pathway Analysis (IPA) is for the analysis, integration and interpretation of data derived from ‘omics experiments. Such experiments include RNA-seq, small RNA-seq, metabolomics, proteomics, microarrays including miRNA and SNP, and small-scale experiments. With QIAGEN IPA you can predict downstream effects and identify new targets or candidate biomarkers. QIAGEN Ingenuity Pathway Analysis helps you perform insightful data analysis and interpretation to understand your experimental results within the context of various biological systems.

Jekyll
Turn plain-text into static text and blogs; uses Markdown language.
Markdown Guide
Guide that explains how to use Markdown.
Markdown is a markup language used to format documents.

LaTeX
Using LaTeX through Overleaf allows you online access-no remembering which device, do I have my jump drive? etc.
Be sure to check out the many great templates.
There is great Learn LaTex in 30 minutes under the Help tab, then documentation

MAXQDA
Software package for qualitative and mixed methods research. It helps you collect, organize, analyze, visualize and publish your data. MAXQDA supports various methodological frameworks, including grounded theory, literature reviews, exploratory market research and qualitative content analyses. You can import documents, PDF files, tables, images, media files, Twitter tweets, YouTube comments, geo references, bibliographical data, and more.

NVivo
For qualitative analysis. Allows storage, organization, query, and visualization of data.
Compatible with: English, Simplified Chinese, French, German, Japanese, Portuguese & Spanish
NVivo Resources
From Case Western Reserve University Guide

Qlucore Omics Explorer
Qlucore Omics Explorer is bioinformatics software for research in the life science, biotech, food and plant industries, as well as academia. The powerful and flexible visualization based analysis tool with built-in powerful statistics delivers results instantly.

Google Style Guide for Python
Matplotlib
Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. Matplotlib can be used in Python scripts, the Python and IPython shells, the Jupyter notebook, web application servers, and four graphical user interface toolkits.
Python
Programming Language
Python Anacondat Distribution
If new to Python, consider this version instead. Anaconda includes Python, Spyder (an integrated development environment, or IDE, for Python), Jupyter notebooks capability, pre-installed Python packages, and more, making it easy to get started quickly.
UC San Diego Guide for Python
Help guide with links to resources, instructions, etc.
W3Schools Python Tutorial

Gentle Guide to R
R for Statistical Computing
To download, you'll need to select the CRAN-Mirror (Comprehensive R Archive Network), that is the closest location to you. Open the CRAN link, scroll down to USA. Close locations to Illinois State are either Iowa, Indiana, or St. Louis, Mo. Once you select the CRAN you want, the downloading automatically starts.
RStudio
To use or work in R, you ALSO need RStudio. Select the RStudio Desktop Open Source License for free software.
UC San Diego Guide for R
Learn R | Codeacademy
"In this course, you’ll be exposed to fundamental programming concepts in R. After the basics, you’ll learn how to organize, modify and clean data frames, a useful data structure in R. Then you’ll learn how to create data visualizations to showcase insights in data! Finish up with statistics and hypothesis testing to become a data analysis expert."
Text Mining with R
This book serves as an introduction of text mining using the tidytext package and other tidy tools in R. The functions provided by the tidytext package are relatively simple; what is important are the possible applications. Thus, this book provides compelling examples of real text mining problems.
Building Rtools
This document is a collection of resources for building packages for R under Microsoft Windows, or for building R itself (version 1.9.0 or later). The original collection was put together by Prof. Brian Ripley and Duncan Murdoch; it is currently maintained by Jeroen Ooms.

Rspberry Pi
A small, affordable computer to learn programming.
YouTube: Getting Started with Raspberry Pi
YouTube overview

SAS Text Miner: UIUC Library
SAS or Statistical Analysis System.
SAS Text miner is a text mining plug-in for the SAS Enterprise Guide that provides different capabilities for analyzing text.

Stanford CoreNLP
"A set of human language technology tools. It can give the base forms of words, their parts of speech, whether they are names of companies, people, etc., normalize dates, times, and numeric quantities, mark up the structure of sentences in terms of phrases and syntactic dependencies, indicate which noun phrases refer to the same entities, indicate sentiment, extract particular or open-class relations between entity mentions, get the quotes people said, etc."

Tableau
Create and share interactive charts and graphs, stunning maps, live dashboards and fun applications in minutes, then publish anywhere on the web.
Tableau 10.2
Blog with info and download link