Antes de iniciar a análise para o projeto Matrix-Methods-Cluster-Analysis, alguns importantes passos foram realizados para preparação da fonte de dados. Estes passos foram descritos nos tópicos a seguir.
O processo preparação de dados, em geral, inclui etapas de extração (retirar os dados de sua localização atual), tranformação (limpar e normalizar os dados) e carga (colocar os dados em um banco de dados onde eles podem ser analisados).
Passo que executa configurações iniciais para execução dos passos seguintes.
# clearing everything before starting -----------------------------------------
ClearRStudioEnvironment()
# setting the environment -----------------------------------------------------
options(encoding = "UTF-8")
info.username <- Sys.info()[["user"]]
info.sysname <- Sys.info()[["sysname"]]
info.machine <- Sys.info()[["machine"]]
info.encoding <- getOption("encoding")
directoryPath <- dirname(rstudioapi::getSourceEditorContext()$path)
directoryPath <- stringr::str_replace(directoryPath, "/src/datapreparation", "")
setwd(directoryPath)
getwd()
# check required libraries
list.of.packages <- c("rawr", 'reticulate', 'ggwordcloud')
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if (length(new.packages)) {
print('one or more required package for this project is not installed in your environment.')
print(new.packages)
print('would you like to install it?')
user.reply <- readline(prompt="y/n: ")
if (user.reply == 'y') {
install.packages(new.packages)
}
}To-do: citar que a base veio do Kaggle.
# performing data download from Kaggle ----------------------------------------
# for this project, we are using the following Kaggle dataset:
# -> https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics
#
# to make it work you will need to install kaggle public API:
# -> https://github.com/Kaggle/kaggle-api#Installation
#
# and set up your API credentials.
# -> https://github.com/Kaggle/kaggle-api#api-credentials
#
# make sure to save your API credentials at: C:\Users\<username>\.kaggle\kaggle.json
#
# following, the API command to download the dataset from Kaggle:
# kaggle datasets download -d gyani95/380000-lyrics-from-metrolyrics
# to-doA etapa de ingestão de dados - data ingestion - é responsável por carregar os dados da fonte de dados de origem.
# performing data loading -----------------------------------------------------
# remarks:
# read_csv is from the readr package;
# read_csv is faster than read.csv for big files.
# loading lyrics ----
start_time = Sys.time()
dataDirectory <- paste(directoryPath, "/data/raw/", sep = "")
lyrics <- read_csv(paste(dataDirectory, "lyrics.csv", sep = ""),
col_types = cols(
index = col_integer(),
song = col_character(),
year = col_integer(),
artist = col_factor(),
genre = col_factor(),
lyrics = col_character()),
locale = locale(encoding = 'UTF-8'))
total_lyrics_loaded <- paste('Total records: ', length(lyrics$index), sep = "")
end_time = Sys.time()
# call logging function
WriteLog(TaskName = "step_03_data_ingestion.R",
StartTime = start_time,
EndTime = end_time,
AdditionalInfo = total_lyrics_loaded)
# applying tokenization in the lirycs ----
start_time = Sys.time()
lyrics_token <- unnest_tokens(lyrics,
input = lyrics,
output = word,
token = 'words',
drop = TRUE,
to_lower = TRUE)
total_words <- paste('Total words after tokenization: ', length(lyrics_token$index), sep = "")
end_time = Sys.time()
# call logging function
WriteLog(TaskName = "step_03_data_ingestion.R",
StartTime = start_time,
EndTime = end_time,
AdditionalInfo = total_words)O objetivo desta etapa é analisar valores ausentes (missing values) e outras condições estranhas.
# analysing missing values and other strange conditions -----------------------
start_time = Sys.time()
lyrics <- filter(lyrics,
!is.na(lyrics),
!(genre %in% c('Not Available', 'Other')),
as.integer(year) >= 1970)
# uncomment for fast prototyping
# lyrics <- sample_n(lyrics, size = 150000)
# invisible(gc())
total_lyrics_loaded <- paste('Total records: ', length(lyrics$index), sep = "")
end_time = Sys.time()
# call logging function
WriteLog(TaskName = "step_04_data_clearing.R",
StartTime = start_time,
EndTime = end_time,
AdditionalInfo = total_lyrics_loaded)O objetivo desta etapa é aprimorar os dados adicionando novas informações que agreguem nas explorações dos mesmos.
# enhancement of data ---------------------------------------------------------
start_time = Sys.time()
lyrics$decade <- paste(str_sub(lyrics$year, 1, 3), '0', sep = '')
lyrics$genre <- trimws(lyrics$genre)
total_lyrics_loaded <- paste('Total records: ', length(lyrics$index), sep = "")
end_time = Sys.time()
# call logging function
WriteLog(TaskName = "step_05_data_enhancement.R",
StartTime = start_time,
EndTime = end_time,
AdditionalInfo = total_lyrics_loaded)Como último passo de preparação de dados, o dataset final é salvo em formato .rds para uso nos demais relatórios e scripts.
# dataset preparation ---------------------------------------------------------
# The objective of this step is to return a DataFrame to be used in the
# predictive modeling.
start_time = Sys.time()
# saving the final dataset to evaluate
rdsFileName <- paste(directoryPath, "/data/processed/lyrics.rds", sep = "")
saveRDS(lyrics, rdsFileName)
end_time = Sys.time()
# call logging function
WriteLog(TaskName = "step_06_data_preparation.R",
StartTime = start_time,
EndTime = end_time,
AdditionalInfo = rdsFileName)Para realização deste trabalho, algumas funções foram criadas para otimizar atividades e procedimentos repetitivos. Estas funções também podem, futuramente, serem reusadas em outros projetos ou mesmo serem adicionadas a uma library R.
# functions -------------------------------------------------------------------
ClearRStudioEnvironment <- function() {
# clear environment and memory
rm(list=ls())
invisible(gc())
# clear console screen
cat("\014")
# clear plots
while (!is.null(dev.list()))
dev.off()
}
WriteLog <- function(TaskName, StartTime, EndTime, AdditionalInfo) {
TotalTime <- difftime(EndTime, StartTime, tz,
units = c("auto", "secs", "mins", "hours",
"days", "weeks"))
log <- paste('[', Sys.time(), '] ',
'Task: ', TaskName, ' | ',
'Time elapsed: ', TotalTime, ' | ',
'Memory used: ', pryr::mem_used(), ' bytes | ',
AdditionalInfo,
sep = "")
write(log, file = "log.txt", append = TRUE)
}