paranormal – Weird Data Science

Mapping Paranormal Manifestations in the British Isles

moth — Tue, 10 Apr 2018 20:08:23 +0000

In our last entry we analysed the relative frequency of paranormal manifestations in the British Isles according to the Paranormal Database. The results of that analysis showed that hauntings of various forms are by far the most commonly-reported paranormal encounter in the British Isles, followed by cryptozoological sightings.

This is, however, relatively unsatisfactory. It is much more interesting to know where such sightings and events occur. Are there particular haunts of restless spirits? Do mysterious beasts roam in particular regions more than others? To answer these questions, we need to delve into the specific locations of different reports.

The Paranormal Database does contain location information, but it is given very informally. To map this we can make use of Google’s Geolocation API to convert free text strings, such as “Felbrigg Hall, Norfolk” into usable latitude and longitude coordinates. (In this case: 52.907479, 1.259443.)

The geolocation is not perfect, but with sufficient manipulation of the service it was possible to produce geolocated coordinates for most of the entries in the database. In order to represent these meaningfully, we have also subdivided the entries into a different types. The original Paranormal Database data is subdivided into twenty categories, which we have reduced to six for easier presentation. This includes collapsing the various kinds of haunting, from poltergeists to ‘post-mortem manifestions’, simply to hauntings. Similarly, we combine alien big cats and shuck into the broader family of cryptozoology.

With this in place, we can see the overall distribution of paranormal events in the British Isles.

Paranormal manifestations in the British Isles.

Paranormal manifestations in the British Isles. (PDF Version.)

As might be expected, London is a dark and sinister nexus of paranormal activity. Hauntings, as might be expected from their overall frequency, dominate the majority of the British Isles. Moving north, particularly as we reach the Scottish Highlands, cryptozoological sightings begin to challenge hauntings as the most common supernatural event. We can also see significant cryptozoology in the Hebrides, Orkney, and Shetland — the archipelagos that surround the Scottish mainland.

Both Wales, Ireland, and Cornwall are significantly less densely haunted in the Paranormal Database, with the majority of sightings falling in England.

This overall view, however, combines a number of very different phenomena. Where, for example, are we most likely to receive a visitation from a restless spectre as opposed to being pursued by a savage and unnatural beast?

By breaking down the sightings into different types, and plotting a heatmap of event density over each, we can identify the regions in which different manifestations cluster.

Density plot of paranormal manifestations in the British Isles

Density plot of paranormal manifestations in the British Isles. (PDF Version.)

This view highlights several points of interest.

Firstly, London’s preeminent position is not for all forms of paranormal activity. Hauntings are extremely dense in London, however the rest of England is also well-populated. As might be expected from the first diagram, the less population-dense regions further north have produced fewer sad echoes of mortality. Despite its general reputation Edinburgh, while noticeably haunted, cannot compare with many regions of England.

Cryptozoologically, however, London is far from dominant. Whilst unknown beings may lurk in the foetid sewers of the capital, they clearly prefer the wide open spaces — both the Norfolk and Suffolk Broads are rife with cryptids, as are the Hebrides, Orkney, and Shetland that were noticeable earlier. Finally, visitors to Cornwall will pass through areas of increasing monstrous activity.

UFO’s also appear to be attracted to East Anglia most strongly, and are otherwise most common in the large population centres of England. Less obviously, there is a noticeable density of UFO activity on the Pembrokeshire Coast, in the south-west of Wales.

Monsters, which in this classification includes werewolves, vampires, and dragons, produce a surprising cluster in North Wales, around Snowdonia. The most significant monstrous sightings, however, appear to be in Exmoor; again on the south-western tip of the British Isles.

The final categories of manifestation include legends, fairies, and a catch-all category of ‘other manifestations’ that include mysterious orbs, talking trees, bleeding stones, and the supernatural impressions left by the work of John Dee. As might be expected, this last category is more uniformly distributed across the country, matching high-density population areas. There is, however, another notable cluster in Cornwall for this category.

In conclusion, then, the British Isles are teeming with paranormal activity. Entities from beyond the grave lie close at all times, with twisted monstrosities roaming the wild spaces. UFOs descend from the night sky to terrorise the coastal regions.

From this analysis, ghost hunters should concentrate on London for the best chance of a sighting, although almost any of the large centres of population provide a reasonable chance of spectral apparitions. Cryptid researchers should concentrate in East Anglia or head north to the islands beyond Scotland. Those seeking contact with extraterrestrials should focus particularly on the east coast of Suffolk, or travel to the south-west of Wales. Paranormal investigators whose interests lie in legends or monsters, or less specific strange entities, would be well-advised to visit Cornwall.

Code for the plotting elements of this analyis are given below, following on from the scraping and parsing code in our previous post. The geolocation step required a more significant effort, and will be the focus of a future code-based post.

You can keep up to date with our latest paranormal data mining on Twitter at @WeirdDataSci.

Show analysis code

Data:

The Paranormal Database: http://www.paranormaldatabase.com

Other:

JSL Ancient font: http://www.1001fonts.com/jsl-ancient-font.html
Cowplot: https://cran.r-project.org/web/packages/cowplot/vignettes/introduction.html

Combined Plot Code:

[code language=”r”] library(spatstat)

library(rgdal)
library(maptools)

library(tidyverse)
library(magrittr)
library(ggplot2)
library(ggthemes)
library(raster)
library(viridis)
library(scales)

library(sf)

library(showtext)

library(grid)

library(cowplot)
library(magick)

# Function to combine certain paranormal report types into broader categories
combine_paranormal_types <- function( type ) {

# levels( paranormal_tbl$type )
# [1] "Alien Big Cat" "Crisis Manifestation"
# [3] "Cryptozoology" "Curse"
# [5] "Dragon" "Environmental Manifestation"
# [7] "Experimental Manifestation" "Fairy"
# [9] "Haunting Manifestation" "Legend"
# [11] "Manifestation of the Living" "Other"
# [13] "Poltergeist" "Post-Mortem Manifestation"
# [15] "Shuck" "Spontaneous Human Combustion"
# [17] "UFO" "Unknown Ghost Type"
# [19] "Vampire" "Werewolf"

# Simple lookup
case_when(
type %in% c("Alien Big Cat", "Cryptozoology", "Shuck") ~ "Cryptozoology",
type %in% c("Dragon", "Vampire", "Werewolf" ) ~ "Monster",
type %in% c("Crisis Manifestation", "Environmental Manifestation", "Experimental Manifestation", "Haunting Manifestation", "Manifestation of the Living", "Poltergeist", "Post-Mortem Manifestation", "Unknown Ghost Type") ~ "Haunting",
type %in% c("Legend", "Fairy", "Curse") ~ "Legend",
type %in% c("Spontaneous Human Combustion", "Other") ~ "Other",
type %in% c("UFO") ~ "UFO"
)

}

# Load font
font_add( "mapfont", "/usr/share/fonts/TTF/weird/JANCIENT.TTF" )
showtext_auto()

# Read world shapefile data and tranform to an appropriate projection.
# Limit to the UK, Ireland, and the Isle of Man
world <- readOGR( dsn=’data/ne/10m_cultural’, layer=’ne_10m_admin_0_countries’ )
world_subset <- world[ world$iso_a2 %in% c("GB","IE","IM"), ] world_subset <- spTransform(world_subset,CRS("+init=epsg:4326"))
world_df <- fortify( world_subset )

# Read paranormal database
paranormal_tbl <- as.tibble( read.csv( file="data/paranormal_database.csv" ) )

# Convert the paranormal dataframe to a spatial dataframe that contains
# explicit longitude and latitude projected appropriately for plotting.
coordinates( paranormal_tbl ) <- ~lng+lat
proj4string( paranormal_tbl )<-CRS("+init=epsg:4326")
paranormal_tbl <- spTransform(paranormal_tbl,CRS(proj4string(paranormal_tbl)))

# Restrict paranormal_tbl to those points in the polygons defined by world_subset
paranormal_tbl_rows <- paranormal_tbl %>%
over( world_subset ) %>%
is.na() %>%
not() %>%
rowSums() %>%
`!=`(0) %>%
which

paranormal_tbl <- as.tibble( paranormal_tbl[ paranormal_tbl_rows, ] )

paranormal_tbl$combined_type <- paranormal_tbl$type %>%
map( combine_paranormal_types ) %>%
unlist

# Show the map
gp <- ggplot() +

geom_map( data = world_df, aes( map_id=id ), colour = "#3c3f4a", fill = "transparent", size = 0.5, map = world_df )

# Display each sighting as geom_point. Use a level of transparency to highlight
# more common areas. (On the 20180401 dataset, this reports that 16035 out of
# the original 19387 points lie in the appropriate area. Several are geolocated
# outside of the UK — the geolocation should be run again with better bounds
# checking and region preference.)
gp <- gp + geom_point(data=paranormal_tbl, aes(x=lng, y=lat, colour=combined_type ), size=0.5, shape=17, alpha=0.9) +
expand_limits(x = world_df$long, y = world_df$lat)

gp <- gp +

# Theming
theme_map() +
theme(
plot.background = element_rect(fill = "transparent", colour = "transparent"),
panel.border = element_blank(),
plot.title = element_text( size=24, colour="#3c3f4a", family="mapfont" ),
text = element_text( size=14, color="#3c3f4a", family="mapfont" ),
) +

theme(
legend.background = element_rect( colour ="transparent", fill = "transparent" ),
legend.key = element_rect(fill = "transparent", colour="transparent"),
legend.position = "left",
legend.justification = c(0,0)
) +

guides( fill = guide_colourbar( title.position="top", direction="vertical", barwidth=32, nrow=1 ) ) +
guides(colour = guide_legend(override.aes = list(size=2)) ) +
scale_colour_manual(
values = c("#00EA38","#417CCC","#B79F00","#F564E3","#00BFC4","#F8766D"),
breaks = c("Haunting", "Cryptozoology", "Monster", "Legend", "UFO", "Other" ),
name = "Manifestation" ) +
coord_fixed( ratio=1.2 )

# Cowplot trick for ggtitle
title <- ggdraw() +
draw_label("Paranormal Manifestations in the British Isles", fontfamily="mapfont", colour = "#3c3f4a", size=24, hjust=0, vjust=1, x=0.02, y=0.88) +
draw_label("http://www.weirddatascience.net | @WeirdDataSci", fontfamily="mapfont", colour = "#3c3f4a", size=14, hjust=0, vjust=1, x=0.02, y=0.40)

data_label <- ggdraw() +
draw_label("Data from: http://www.paranormaldatabase.com", fontfamily="mapfont", colour = "#3c3f4a", size=12, hjust=1, x=0.98 )

paranormal_legend <- get_legend(gp)

# Remove legend from internal plot
gp <- gp + theme(legend.position=’none’)

tgp <- plot_grid(title, gp, data_label, ncol=1, rel_heights=c(0.1, 1, 0.1))

vellum_plot <- ggdraw() +
draw_image("img/vellum.jpg", scale=1.4 ) +
draw_plot(tgp) +
draw_plot( paranormal_legend, 0.02, 0.03 )

save_plot("output/paranormal.pdf",
vellum_plot,
base_width = 16,
base_height = 9,
base_aspect_ratio = 1.78 )
[/code]

Per-Manifestation Density Plot Code:

[code language=”r”] library(spatstat)

library(rgdal)
library(maptools)

library(tidyverse)
library(magrittr)
library(ggplot2)
library(ggthemes)
library(raster)
library(viridis)
library(scales)

library(sf)

library(showtext)

library(grid)

library(cowplot)
library(magick)

# Function to combine certain paranormal report types into broader categories
combine_paranormal_types <- function( type ) {

}

# Load font
font_add( "mapfont", "/usr/share/fonts/TTF/weird/JANCIENT.TTF" )
showtext_auto()

# Read world shapefile data and tranform to an appropriate projection.
# Limit to the UK, Ireland, and the Isle of Man
world <- readOGR( dsn=’data/ne/10m_cultural’, layer=’ne_10m_admin_0_countries’ )
world_subset <- world[ world$iso_a2 %in% c("GB","IE", "IM"), ] world_subset <- spTransform(world_subset,CRS("+init=epsg:4326"))
world_df <- fortify( world_subset )

# Read UK Major Cities datafile
#cities <- readOGR( dsn=’data/ons_uk_cities’, layer=’Major_Towns_and_Cities_December_2015_Boundaries’ )
#cities <- spTransform(cities,CRS("+init=epsg:4326"))
#cities_fortified <- fortify( cities )
#cities_centroids_tbl <- as.tibble( coordinates( cities ) )
#cities_tbl <- as.tibble( cbind( cities@data$tcity15nm, cities_centroids_tbl ))
#colnames(cities_tbl) <- c( "city", "lng", "lat" )

# As the polygons are more confusing than useful, instead label centroids

# Read paranormal database
paranormal_tbl <- as.tibble( read.csv( file="data/paranormal_database.csv" ) )

paranormal_tbl$combined_type <- paranormal_tbl$type %>%
map( combine_paranormal_types ) %>%
unlist

# Convert the paranormal dataframe to a spatial dataframe that contains
# explicit longitude and latitude projected appropriately for plotting.
coordinates( paranormal_tbl ) <- ~lng+lat
proj4string( paranormal_tbl )<-CRS("+init=epsg:4326")
paranormal_tbl_spatial <- spTransform(paranormal_tbl,CRS(proj4string(paranormal_tbl)))

# Restrict paranormal_tbl to those points in the polygons defined by world_subset
paranormal_tbl_rows <- paranormal_tbl_spatial %>%
over( world_subset ) %>%
is.na() %>%
not() %>%
rowSums() %>%
`!=`(0) %>%
which

#paranormal_tbl <- as.tibble( paranormal_tbl_spatial[ paranormal_tbl_rows, ] )

# Create window for spatial analysis
paranormal_owin <- as.owin.SpatialPolygons(world_subset)

# Function to plot density of a specific manifestation type.
# plot_resolution is for the density raster, and is mainly used for quick prototyping of the output.
density_plot <- function( paranormal_type, plot_resolution = 1024 ) {

cat( paste0( "Plotting density: ", paranormal_type, "… " ) )

paranormal_tbl_spatial <- paranormal_tbl_spatial[ which( paranormal_tbl_spatial$combined_type == paranormal_type ), ]

paranormal_ppp <-
ppp( x=coordinates(paranormal_tbl_spatial)[,1],
y=coordinates(paranormal_tbl_spatial)[,2],
window = paranormal_owin )

# This discards ‘illegal’ points outside of the window
paranormal_ppp <- as.ppp(paranormal_ppp)

paranormal_density <- density( paranormal_ppp, diggle=T, sigma=0.4, dimyx=c(plot_resolution,plot_resolution) )

# Make density image object usable by ggplot as a raster
paranormal_density_raster <- raster( paranormal_density )
raster_tbl <- as.tibble( rasterToPoints( paranormal_density_raster ) )

# Show the map
gp <- ggplot() +

geom_map( data = world_df, aes( map_id=id ), colour = "#3c3f4a", fill = "transparent", size = 0.8, map = world_df ) +

# Add density of sightings as raster.
geom_raster( data = raster_tbl, alpha=0.8, aes( x=x, y=y, fill=layer), show.legend=TRUE ) +
scale_fill_viridis( option="magma", direction = -1, name="Density" )

gp <- gp +

# Theming
theme_map() +

theme(
plot.background = element_rect(fill = "transparent", colour = "transparent"),
panel.border = element_blank(),
plot.title = element_text( size=12, colour="#3c3f4a", family="mapfont" ),
text = element_text( size=12, color="#3c3f4a", family="mapfont" ),
) +

guides( fill = guide_colourbar( title.position="top", direction="horizontal", barwidth=6, barheight=0.4 ) ) +

# Fix the ratio of the plot to avoid distorting the UK
coord_fixed( ratio=1.2 )

cat("done.\n" )
return(gp)
}

# Calculate densities for each phenomenon
phenomena <- unique( paranormal_tbl$combined_type )

gp_list <- phenomena %>%
map( density_plot, 1024 )

# Plot as a grid with cowplot
theme_set(theme_cowplot(font_size=4, font_family = "mapfont" ) )
gp <- plot_grid( plotlist=gp_list,
labels = phenomena,
label_colour = "#3c3f4a" )

# Cowplot trick for ggtitle
title <- ggdraw() +
draw_label("Density of Paranormal Manifestations in the British Isles", fontfamily="mapfont", colour = "#3c3f4a", size=24, hjust=0, vjust=1, x=0.02, y=0.88) +
draw_label("http://www.weirddatascience.net | @WeirdDataSci", fontfamily="mapfont", colour = "#3c3f4a", size=14, hjust=0, vjust=1, x=0.02, y=0.40)

data_label <- ggdraw() +
draw_label("Data from: http://www.paranormaldatabase.com", fontfamily="mapfont", colour = "#3c3f4a", size=12, hjust=1, x=0.98 )

tgp <- plot_grid(title, gp, data_label, ncol=1, rel_heights=c(0.1, 1, 0.1))

vellum_plot <- ggdraw() +
draw_image("img/vellum.jpg", scale=1.4 ) +
draw_plot(tgp)

save_plot("output/paranormal-density.pdf",
vellum_plot,
base_width = 16,
base_height = 9,
base_aspect_ratio = 1.78 )
[/code]

Paranormal Manifestations in the British Isles

moth — Sat, 17 Mar 2018 09:20:09 +0000

The British Isles are ancient, haunted places. Pre-Roman legends and folk songs are filled with dragons, magic, and strange, wild creatures. Spirits, fairies, and all kinds of imps and goblins roam the countryside with intent ranging from the mischievous to the malevolent. Will-o’-the-wisps lead unwary travellers deep into marshes in the night, before vanishing. The sad ghosts of past tragedies reside in ancient castles and stately homes.

In more recent history, strange beasts have been rumoured to live wild in the open spaces, whether large predators escaped from zoos, the last survivals of prehistory, or spirits. Every village, town, and county has its own stories and traditions.

The Paranormal Database is a collection of both traditional and recent paranormal events in Britain and Ireland. It contains details of almost 20,000 hauntings, cryptozoological sightings, legends, monsters, UFO’s, and other strange phenomena, with details of the date and time of sightings, the location, and brief descriptions.

The data is not easily accessible beyond directly reading pages, and required some effort and time to scrape and make usable. Paranormal Database entries contain names, dates, locations, and comments as unstructured text and so will require further effort to perform a more thorough analysis. The R code used to scrape the website is included at the end of this post.

To understand the range and breadth of the paranormal life of the British Isles, we will focus on the data stored in the Paranormal Database. For this initial entry, we will take a first look at the data and get an overview of what mind-numbing horrors are most commonly encountered by the unsuspecting traveller in the United Kingdom and beyond.

Frequency of paranormal sightings in the British Isles.

Frequency of paranormal sightings in the British Isles. (PDF version)

Show frequency table of manifestations

Paranormal Manifestations in the British Isles

Manifestation Type	Occurrences
Haunting Manifestation	12376
Legend	1662
Cryptozoology	835
Shuck	688
Poltergeist	625
Unknown Ghost Type	614
Fairy	550
Other	427
Alien Big Cat	382
UFO	336
Crisis Manifestation	277
Dragon	190
Curse	183
Post-Mortem Manifestation	179
Environmental Manifestation	152
Manifestation of the Living	44
Werewolf	36
Vampire	34
Spontaneous Human Combustion	28
Experimental Manifestation	2

As we can see from the diagram and the frequency table, hauntings are by far the most common manifestation in paranormal Britain, being an order of magnitude greater than the number of legend recorded. Examining the list, beyond “Haunting Manifestation” we see that several of the most common types are variants: both poltergeist activity and unknown types of ghost represent a significant amount of the total events recorded.

Cryptozoology, in its various forms, is also well-represented. The phenomenon of the Black Shuck, a ghostly black dog, is one of the highest categories next to the main cryptozoology category, and alien big cats are close behind. Dragons, werewolves, and vampires, perhaps, deserve to be classed more as monstrous entities than cryptozoological oddities and are, in any case, far less common.

In brief conclusion, then, the unquiet dead are by far the most numerous beings to trouble the unhappy folk of the British Isles; twisted mockeries of natural fauna are far from rare.

Do particular phenomena cluster in regions and, if so, where? Are werewolves truly more commonly seen when the moon is full? Have certain manifestations become more common as time passes? Are certain sightings clustered temporally as well as geographically? Are the most haunted areas also the most cryptozoologically active? With access to the full horror of the data we can begin to answer these question about the darkest corners of the United Kingdom.

Full code for scraping the data and producing the plot are given below.

You can keep up to date with our latest visions of the statistical unknown on Twitter at @WeirdDataSci.

Show analysis code

Data:

The Paranormal Database: http://www.paranormaldatabase.com

Other:

JSL Ancient font: http://www.1001fonts.com/jsl-ancient-font.html
Rvest web scraping library: http://blog.rstudio.com/2014/11/24/rvest-easy-web-scraping-with-r/ | https://cran.r-project.org/web/packages/rvest/index.html
xkcd styling for ggplot2: http://xkcd.r-forge.r-project.org/

Paranormal Database Scraping Code:

[code language=”r”] library(rvest)
library(magrittr)
library(tidyr)
library(stringr)
library(dplyr)

# Base URL for scraping
paranormal.base <- "http://www.paranormaldatabase.com"

# Extract links from a given page. Return a list with full URLs.
extract.links <- function( in.html, base.url ) {

# Read the page and extract any links from the ‘hero-unit’ div.
# Extract all links, making relatives absolute.
in.html %>%
html_nodes( ‘.hero-unit’ ) %>%
html_nodes( ‘a’ ) %>%
html_attr( ‘href’ ) %>%
url_absolute( base.url )

}

# Function to extract a paranormal database entries page into a dataframe.
# Returns a dataframe of the entries from the given HTML.
extract.entries <- function( in.html ) {

# Select all table cells with a width of 100% — ugly, but identifies the
# entries themselves.
page.entries <- in.html %>%
html_nodes( xpath=’//td[ contains( @width, "100" ) ]’ ) %>%
html_text() %>%
str_replace_all( "[\r\n]" , "" )

# If any entries were found, process them.
if( ( length( page.entries ) > 0 ) &&
(str_detect(page.entries, "(.*[[:graph:]])\\s*Location: (.*[[:graph:]])\\s*Type: (.*[[:graph:]])\\s*Date / Time: (.*[[:graph:]])\\s*Further Comments: (.*[[:graph:]])\\s*")) ) {
# Each entry is a string containing:
# – Location:
# – Type:
# – Date / Time:
# – Further Comments:
# So split the string on these and put the entries into a dataframe.
page.entries.df <- data.frame( entry=page.entries) %>%
tidyr::extract(entry, into=c(‘title’, ‘location’, ‘type’, ‘date’, ‘comments’), "(.*[[:graph:]])\\s*Location: (.*[[:graph:]])\\s*Type: (.*[[:graph:]])\\s*Date / Time: (.*[[:graph:]])\\s*Further Comments: (.*[[:graph:]])\\s*" )

# Further split these into subtypes where appropriate. Both ‘location’ and ‘type’ are often split into a main and a subtype by a space-surrounded hyphen.
# As we’ll do this twice, make it a function. This takes a base column name, a new name for the sub-column, and the separator string.
coalesce.join <- function( base.data, base.column, sub.column, separator ) {

base.data <- base.data %>%
tidyr::extract( base.column, into=c("tmp.main", sub.column), separator, remove=FALSE)

# Combine the old base column with the new tmp.main column, using base.column to fill in the NAs in tmp.main
# where the separator was not matched.
base.data[ base.column ] <- coalesce( base.data[[ "tmp.main" ]], base.data[[ base.column ]] )
base.data <- base.data %>%
select(-one_of("tmp.main"))

}

# Split ‘type’
page.entries.df <- coalesce.join( page.entries.df, "type", "subtype", "(.*) – (.*)" )
page.entries.df <- coalesce.join( page.entries.df, "location", "sublocation", "(.*) – (.*)" )

page.entries.df

}

# Read each page’s links to populate a list to process.
# Each page contains a "Return to Regional Listing" link, and a link to enter the listings.
# That listing can be either a straight entries page, or a link to subpages.

# Starting with the base page, pull every link in the main ‘hero-unit’ div and
# add it to a ‘to.process’ vector, checking that that URL is not already
# listed. Also pass along a ‘processed’ vector in the recursive calls so that
# we don’t overly duplicate results.

# WARNING: As written, this will cause a lot of duplication due to the lack of
# global list of processed links. As this was a one-off script, I haven’t fixed this.

# Base URL to being processing
paranormal.regions <- "http://www.paranormaldatabase.com/regions/regions.htm"

# Recursively process the top-level URL
scrape.recurse <- function( in.url, processed ) {

message( paste("Processing", in.url ) )

# Try to fetch this URL and any subpages.
# Warn on failure
page <- tryCatch(
{
# Fetch and parse HTML of current page.
read_html( in.url )
},
error=function( cond ) {
message( paste( "URL caused an error:", in.url ))
message( "Error message:")
message( cond )
return( NULL )
},
warning=function( cond ) {
message( paste( "URL caused a warning:", in.url ))
message( "Warning message:")
message( cond )
return( NULL )
},
finally={
}
)

if( is.null( page ) ) {
return( NULL )
}
else {
# Get the current page’s entries
links <- extract.links( page, in.url )
entries <- extract.entries( page )

lapply( entries$title, function(x) cat( x, "… ", sep="" ) )
cat( "\n")

# Recurse, but don’t follow any known links
new.entries <-
lapply( setdiff( links, processed ), function(x) scrape.recurse( x, union( links, processed ) ) ) %>%
bind_rows()

# Bind entries from sub-pages to this page’s entries.
all.entries <- bind_rows( entries, new.entries )

# Report success and return results
message( paste( "Successfully processed:", in.url ) )
return( all.entries )
}

}

# Run the scraping and parsing, then remove duplicates.
all.entries <- scrape.recurse( paranormal.regions, paranormal.regions )
paranormal.df <- unique( all.entries )

# Manually fix some type entries.
paranormal.df$type[ which( paranormal.df$type == "legend" ) ] <- "Legend"
paranormal.df$type[ which( paranormal.df$type == "Haunting Manifestation?" ) ] <- "Haunting Manifestation"
paranormal.df$type[ which( paranormal.df$type == "SHC" ) ] <- "Spontaneous Human Combustion"
paranormal.df$type[ which( paranormal.df$type == "ABC" ) ] <- "Alien Big Cat"

# With all values scraped, save data.
save( paranormal.df, file="paranormal.Rdata")

[/code]

Paranormal Manifestations in the British Isles Plotting Code:
[code language=”r”] library(ggplot2)
library(jpeg)
library(xkcd)
library(showtext)
library(grid)
library(gridExtra)

# Create a summary barplot for paranormal activity in the UK.

# Load the data from scraping http://www.paranormaldatabase.com/
load( "../data/paranormal.Rdata" )

# Process the data to frequency counts of each type of event.
counts <- table( paranormal.df$type )
df <- as.data.frame.table( sort(counts, decreasing=FALSE) )
colnames( df ) <- c("type", "frequency" )

# Create the plot.
# Use the xkcd package to create irregular bars for a hand-drawn look.
df$xmin <- 1
df$xmax <- df$frequency
df$ymin <- as.numeric(df$type) – 0.1
df$ymax <- as.numeric(df$type) + 0.1

xrange <- range( min( df$xmin ), max( df$xmax ) )
yrange <- range( min( df$ymin ), max( df$ymax ) + 1 )
mapping <- aes( xmin=xmin, ymin=ymin, xmax=xmax, ymax=ymax )

# Load font
font_add( "handwriting", "/usr/share/fonts/TTF/weird/JANCIENT.TTF" )
showtext_auto()

# Load background image.
bg.image <- jpeg::readJPEG("img/vellum.jpg")
bg.grob <- rasterGrob( bg.image, interpolate=TRUE,
width = 1.8,
height = 1.4 )

gp <- ggplot( data=paranormal.df, aes( x=frequency, y=type ) ) +
annotation_custom( bg.grob, xmin=-Inf, xmax=Inf, ymin=-Inf, ymax=Inf ) +
xkcdrect( mapping, df, alpha=0.6, fill="#8a0707", size=1 ) +
xkcdaxis( xrange, yrange ) +
theme(
text = element_text( size=12, color="#232732", family="handwriting" ),
plot.background = element_rect(fill = "transparent", color="transparent"),
panel.background = element_rect(fill = "transparent", color="transparent"),
plot.title = element_text( size=24, colour="#232732", family="handwriting" ),
axis.title = element_text( size=18, colour="#232732", family="handwriting" ),
axis.text.x=element_text( size=12, color="#232732", family="handwriting" ),
axis.text.y=element_text( size=12, color="#232732", family="handwriting", hjust=1) ) +
scale_y_discrete( labels=c( as.character( df$type ) ) ) +
scale_x_continuous( expand = c( 0, 0 ) ) +
ylab( "Phenomenon" ) +
xlab( "Occurrence" ) +
labs( caption = "Data from: http://www.paranormaldatabase.com" ) +
ggtitle("Paranormal Manifestations in the British Isles", subtitle="http://www.weirddatascience.net | @WeirdDataSci")

# To allow the background image to extend to the edge of the panel, we have to generate
# a gtable directly from gp, then turn off clipping.
p2 <- ggplot_gtable(ggplot_build(gp))
p2$layout$clip[p2$layout$name=="panel"] <- "off"

# To create the output, we need to call grid.draw on the gtable.
# Use the pdf device to draw directly to preserve clipping and fonts.
# showtext has to be called explicitly in here, for some reason.
pdf( file="output/haunting.pdf", width=16, height=9 )
showtext_begin()
grid.draw(p2)
showtext_end()
dev.off()

[/code]