Before starting the analysis for the Spatial-Statistics, a few important steps were taken in order to prepare the source data files. These steps are listed below:
This step do an initial setup configuration for the project.
# clearing everything before starting -----------------------------------------
# clear environment and memory
rm(list=ls())
invisible(gc())
# clear console screen
cat("\014")
# clear plots
while (!is.null(dev.list()))
dev.off()
# setting the environment -----------------------------------------------------
options(encoding = "UTF-8")
info.username <- Sys.info()[["user"]]
info.sysname <- Sys.info()[["sysname"]]
info.machine <- Sys.info()[["machine"]]
info.encoding <- getOption("encoding")
directoryPath <- dirname(rstudioapi::getSourceEditorContext()$path)
directoryPath <- stringr::str_replace(directoryPath, "/src/datapreparation", "")
setwd(directoryPath)
getwd()The process of data ingestion — preparing data for analysis — usually includes steps called extract (taking the data from its current location), transform (cleansing and normalizing the data), and load (placing the data in a database where it can be analyzed).
# performing data loading -----------------------------------------------------
dataRawDirectory <- "./data/raw/"
shapefile_to_read <- paste(dataRawDirectory, "crime_mg.shp", sep = "")
target <- readOGR(shapefile_to_read, encoding="UTF-8")The objective of this step is analysing missing values and other strange conditions. In order to accomplish this task, a few R functions were used to quickly discover missing values, like NA and empty fields.
The command below was used to find out any NA values in each table.
There was no tables with NA values:
| x | |
|---|---|
| CODMUNI | 0 |
| ID | 0 |
| MUNIC | 0 |
| AREA | 0 |
| INDICE94 | 0 |
| INDICE95 | 0 |
| GINI_91 | 0 |
| POP_94 | 0 |
| POP_RUR | 0 |
| POP_URB | 0 |
| POP_FEM | 0 |
| POP_MAS | 0 |
| POP_TOT | 0 |
| URBLEVEL | 0 |
| PIB_PC | 0 |
| X_COORD | 0 |
| Y_COORD | 0 |
Finally, the following command was used in each table to find out where empty values was hidden.
There was no tables with empty values:
| x | |
|---|---|
| CODMUNI.NA | NA |
| ID.NA | NA |
| MUNIC.NA | NA |
| AREA.NA | NA |
| INDICE94.NA | NA |
| INDICE95.NA | NA |
| GINI_91.NA | NA |
| POP_94.NA | NA |
| POP_RUR.NA | NA |
| POP_URB.NA | NA |
| POP_FEM.NA | NA |
| POP_MAS.NA | NA |
| POP_TOT.NA | NA |
| URBLEVEL.NA | NA |
| PIB_PC.NA | NA |
| X_COORD.NA | NA |
| Y_COORD.NA | NA |
For the exploration analysis report, we did not take any additional action, since the missing values was not relevant.
There are 33 cities in which the variables POP_RUR, POP_URB, POP_FEM, POP_MAS, POP_TOT have zero in their values, however the column POP_94 has value greater than zero. so, the strategy is to produce the value for the missing columns based on the average values for the rest of cities in Minas Gerais.
The following adjustments were taken:
# converting columns from text to numeric
target$POP_RUR <- as.numeric(as.character(target$POP_RUR))
target$POP_URB <- as.numeric(as.character(target$POP_URB))
target$POP_FEM <- as.numeric(as.character(target$POP_FEM))
target$POP_MAS <- as.numeric(as.character(target$POP_MAS))
# getting rural population and female's mean ratio
target.dataframe <- as(target, "data.frame")
target.dataframe <- filter(target.dataframe, POP_RUR != 0)
poprur.ratio <- mean(target.dataframe$POP_RUR) / mean(target.dataframe$POP_URB)
pophm.ratio <- mean(target.dataframe$POP_MAS) / mean(target.dataframe$POP_FEM)
popfem.ratio <- mean(target.dataframe$POP_FEM) / mean(target.dataframe$POP_TOT)
# adjusting rows with zero in POP_RUR, POP_URB, POP_FEM, POP_MAS and POP_TOT
target$POP_RUR <- ifelse(target$POP_RUR == 0,
round(target$POP_94 * poprur.ratio, digits = 0),
round(target$POP_RUR, digits = 0))
target$POP_URB <- ifelse(target$POP_URB == 0,
round(target$POP_94 - target$POP_RUR, digits = 0),
round(target$POP_URB, digits = 0))
target$POP_FEM <- ifelse(target$POP_FEM == 0,
round(target$POP_94 * popfem.ratio, digits = 0),
round(target$POP_FEM, digits = 0))
target$POP_MAS <- ifelse(target$POP_MAS == 0,
round(target$POP_94 - target$POP_FEM, digits = 0),
round(target$POP_MAS, digits = 0))
target$POP_TOT <- ifelse(target$POP_TOT == 0,
round(target$POP_94, digits = 0),
round(target$POP_TOT, digits = 0))In order to make the data information more understandable, it was translated some relevant labels and domains from Czech to English.
For this project this step is not required.
This step aims to improve the analysis by adding auxiliary information. Data enhancement is all about making sure any data that is coming into the business is being looked at with a critical eye and is being filtered down to maximize its value.
For this project this step is not required.
The below function was created to be used in the modeling exercises to be performed, the idea is to have a standard way to get the prepared data set already prepared with correct data types and dummies.