linguistics – Weird Data Science

Readings from the Book

moth — Sun, 28 Jan 2024 16:27:02 +0000

Once again, the Oxford Internet Institute at the University of Oxford — through madness, or through omission brought on by horrified incredulity — saw fit to expose its students to the nightmarish patterns that descend, fractal-like, endlessly below the surface of mundane reality.

This second OII Halloween Lecture drew on the twisted meanderings we travellers have taken through the cryptic verbiage of the Voynich Manuscript. We aim to establish the dread authenticity of the text, by rousing its very statistical bones from the inscrutable fasciae of its pages. Walking a tightrope between careful statistical exploration and ever-burgeoning insanity, we further explore the structures that arise from the text, separating the untranslated knowledge in the book into coherent bodies for future study.

In yet another, almost criminially negligent, oversight, the OII’s 2024 Halloween Lecture was captured, frozen in space and time, for the detriment and despair of the unexpectant world.

For those not driven to blissful negation by the tortured ramblings of the above, the underlying materials for the talk are presented, with neither hope nor tremor, here.

illuminating_the_illuminated

Illuminating the Illuminated – Part Three: Topics of Invention | Topic Modelling the Voynich Manuscript

moth — Tue, 24 Dec 2019 16:51:08 +0000

Our earlier experiments derived some of the darker statistics of the Voynich Manuscript supporting the conjecture, but not erasing all doubt, that the manuscript’s cryptic graphemes are drawn from some natural, or shudderingly unnatural, language.

Despite our beliefs regarding its authenticity, however, the statistical tools we have employed so far can tell us little about the structure, and almost nothing of the meaning, of the Voynich Manuscript. In this post, whilst shying away from the madness and confusion of attempting to translate MS 408, or of definitively identifying its language, we will delve into the extent to which modern natural language processing techniques can reveal its lesser secrets.

The mechanisms we will apply in this post are drawn from the world of topic modelling, an approach widely used in the processing of human language to identify eerily related documents within a corpus of text.

Topic modelling, in its most widely used form, lies in considering each given document as a nebulous admixture of unseen and unknowable topics. These topics, in effect, are themselves probability distributions of words that are likely to occur together. Each document, therefore, is characterised as a set of probability distributions that generate the observed words. This approach, known as Latent Dirichlet Allocation, dispassionately extracts the hidden structure of documents by deriving these underlying distributions.

For known languages, latent Dirichlet allocation extrudes a set of topics characterised by the high-probability words that they generate. These, in turn, can be subjected to human interpretation to identify the semantic underpinnings behind the topics.

To illustrate, we present a topic model of Margaret A. Murray’s seminal 1921 work “The Witch Cult in Western Europe”. There are many uneasy subtleties in producing such a model, into which we will not plunge at this early stage; at a quick glance, however, we can see that from Murray’s detailed research and interweaved arguments for a modern-day survival of an ancient witch cult in Europe, the algorithm can identify certain prevalent themes. The third topic, for example, appears to conjure terms related to the conflict between the accepted state religion and the ‘heathen’ witch cult. The ninth topic concerns itself with the witches’ marks, supposedly identified on the body of practitioners; while the tenth dwells on the clandestine meetings and sabbaths of the cult.

Topic plot for Murray’s “The Witch Cult in Western Europe”

" data-image-caption="

Topic plot for Murray’s “The Witch Cult in Western Europe”

Topic plot for Murray’s “The Witch Cult in Western Europe” | (PDF Version)

Witch Cult Topic Model Code

wcwe_topics.r
[code language=”r”]

library( tidyverse )
library( magrittr )

library( ggthemes )
library( showtext )

library( tidytext )
library( widyr )

library( stm )
library( quanteda )

library(cowplot)

# For reorder_within() for facets:
library( drlib )

# Fonts
font_add( "main_font", "/usr/share/fonts/TTF/weird/alchemy/1651 Alchemy/1651AlchemyNormal.otf")
font_add( "bold_font", "/usr/share/fonts/TTF/weird/alchemy/1651 Alchemy/1651AlchemyNormal.otf")

showtext_auto()

# Read (processed) text of Murray’s "The Witch Cult in Western Europe".
wcwe_raw <-
read_csv( "data/wcwe/wcwe_raw.csv", col_names=FALSE ) %>%
rename( text = X1 ) %>%
rowid_to_column( var = "chapter" )

# Tokenize
# (Remove words of 3 letters or less)
# Stemming and stopword removal apparently not so effective anyway,
# according to Schofield et al.:
wcwe_words <-
wcwe_raw %>%
unnest_tokens( word, text ) %>%
filter( !word %in% stop_words$word ) %>%
filter( str_length( word ) > 3 )

wcwe_word_counts <-
wcwe_words %>%
count( word, chapter, sort = TRUE )

# Generate the corpus
wcwe_dfm <-
wcwe_words %>%
count( chapter, word, sort=TRUE ) %>%
cast_dfm( chapter, word, n )

# Search for a number of topics and output goodness-of-fit measures.

# N=2 is the number of documents ‘held out’ for the goodness-of-fit measure.
# (The model is trained on the main body, then used to calculated the
# likelihood of the held-out documents.) N=2 is used here to produce
# approximately 10% of the corpus.

if( not( file.exists( "work/wcwe_topic_search_k.rds" ) ) ) {

message( "Seaching low-n topic models…" )

wcwe_k <-
searchK( wcwe_dfm, K=c(3:30), N=2 )

saveRDS( wcwe_k, "work/wcwe_topic_search_k.rds" )

} else {

wcwe_k <-
readRDS( "work/wcwe_topic_search_k.rds" )

}

# Plot semantic coherence against exclusivity for model selection
wcwe_k_plot <-
wcwe_k$results %>%
gather( key="variable", value="value", exclus, semcoh )

wcwe_k_semcoh_exclusive <-
ggplot( wcwe_k_plot, aes( x=K, y=value, group=variable) ) +
geom_line() +
facet_wrap( ~variable, ncol=1, scales="free_y" )

# Based on metrics of the above, calculate a 13-topic model
if( not( file.exists( "work/wcwe_topic_stm-13.rds" ) ) ) {

message( "Calculating 13-topic model…" )

wcwe_topic_model_13 <-
stm( wcwe_dfm, K=13, init.type="Spectral" )

# This takes a long time, so save output
saveRDS( wcwe_topic_model_13, "work/wcwe_topic_stm-13.rds" )

} else {

wcwe_topic_model_13 <- readRDS( "work/wcwe_topic_stm-13.rds" )

}

# Work with the 13-topic model for now
wcwe_topic_model <- wcwe_topic_model_13

### Convert output to a tidy tibble
wcwe_topic_model_tbl <-
tidy(wcwe_topic_model, matrix = "beta" )

wcwe_topics_top <-
wcwe_topic_model_tbl %>%
group_by(topic) %>%
top_n(10, beta) %>%
ungroup() %>%
arrange(topic, -beta)

gp <-
wcwe_topics_top %>%
mutate(term = reorder_within(term, beta, topic)) %>%
ggplot(aes(term, beta, fill = factor(topic))) +
geom_col(show.legend = FALSE, alpha=0.8 ) +
facet_wrap(~ topic, scales = "free") +
scale_x_reordered() +
coord_flip()

# Palette of ink colours obtained from screenshots of Diamine inks.
ink_colours <- c( "#753733", "#b6091d", "#e45025", "#232d1d",
"#224255", "#533f50", "#453437", "#7f2430",
"#254673", "#52120e", "#3d2535", "#25464b",
"#2f2a1c" )

gp <-
gp + scale_fill_manual( values=ink_colours )

topic_plot_13 <-
gp + labs( x="Term", y="Probability in Topic" ) +
theme (
plot.title = element_text( family="bold_font", size=16 ),
plot.subtitle = element_text( family="bold_font", size=12 ),
axis.text.y = element_text( family="bold_font", size=10 )
)

theme_set(theme_cowplot(font_size=4, font_family = "main_font" ) )

wcwe_topic_plot <-
topic_plot_13 +
theme (
axis.title.y = element_text( angle = 90, family="main_font", size=12 ),
axis.text.y = element_text( colour="#3c3f4a", family="main_font", size=10 ),
axis.title.x = element_text( colour="#3c3f4a", family="main_font", size=12 ),
axis.text.x = element_text( colour="#3c3f4a", family="main_font", size=10 ),
axis.line.x = element_line( color = "#3c3f4a" ),
axis.line.y = element_line( color = "#3c3f4a" ),
plot.title = element_blank(),
plot.subtitle = element_blank(),
plot.background = element_rect( fill = "transparent" ),
panel.background = element_rect( fill = "transparent" ), # bg of the panel
panel.grid.major.x = element_blank(),
panel.grid.major.y = element_blank(),
panel.grid.minor.x = element_blank(),
panel.grid.minor.y = element_blank(),
legend.text = element_text( family="bold_font", colour="#3c3f4a", size=10 ),
legend.title = element_blank(),
legend.key.height = unit(1.2, "lines"),
legend.position=c(.85,.5),
strip.background = element_blank(),
strip.text.x = element_text(size = 10, family="main_font")
)

# Cowplot trick for ggtitle
title <-
ggdraw() +
draw_label("\"The Witch Cult in Western Europe\" (Murray, 1921) Topic Model", fontfamily="bold_font", colour = "#3c3f4a", size=20, hjust=0, vjust=1, x=0.02, y=0.88) +
draw_label("http://www.weirddatascience.net | @WeirdDataSci", fontfamily="bold_font", colour = "#3c3f4a", size=12, hjust=0, vjust=1, x=0.02, y=0.40)

data_label <-
ggdraw() +
draw_label("Data: Murray, M. \"The Witch Cult in Western Europe\" (1921) | http://www.gutenberg.org/ebooks/20411", fontfamily="bold_font", colour = "#3c3f4a", size=8, hjust=1, x=0.98 )

tgp <-
plot_grid(title, wcwe_topic_plot, data_label, ncol=1, rel_heights=c(0.1, 1, 0.1))

wcwe_topic_plot <-
ggdraw() +
draw_image("img/parchment.jpg", scale=1.4 ) +
draw_plot(tgp)

ggsave( "output/wcwe_topic_plot_13.pdf", width=16, height=9 )

[/code]

As the plot above suggests, topic modelling is a tool to support our limited human understanding rather than a cold, mechanical source of objectivity and, as with much unsupervised machine learning, there are various subjective choices that must be made guided by the intended purpose of the analysis. Drawing together impercetible threads of relation in bodies of text, the approach suggests major themes and, crucially, can associate disparate areas of text that focus on similar concerns.

Topical Remedies

What, then, can we learn by bringing the oppressive weight of latent Dirichlet allocation to bear against a cryptic tome whose words, and indeed letters, resist our best efforts at interpretation?

Without understanding of individual words, we wil be unable to glean the semantic understanding of topics that was possible with Murray’s Witch Cult…. There is a chance, however, that the topic model can derive relations between separated sections of the manuscript — do certain early pages demonstrate a particular textual relationship to later pages? Do sections of the overall manuscript retain an apparent coherence of topics, with contiguous pages being drawn from a small range of similar topics? Which Voynich words fall under similar topics?

Preparations

Topic modelling typically requires text to undergo a certain level of formulaic preparation. The most common of such rituals are stemming, lemmatization, and stopword removal. Briefly, stemming and lemmatization aim to reduce confusion by rendering words to their purest essence. Stemming is a more crude heuristic, unsympathetically incising endings, and so truncating “dark”, “darker”, “darkest” simply to the atomic root word “dark”. Lemmatization requires more understanding, untangling parts of speech and context: that to curse is a verb while a curse is a noun; the two identical words should therefore be treated separately.

Stopword removal aims to remove the overwhelming proportion of shorter, structural words that are ubiquitous throughout any text, but are largely irrelevant to the overall topic: the, and, were, it, they, but, if…. Whilst key to our understanding of texts, these terms have no significance to the theme or argument of a text.

Undermining our scheme to perform topic modelling, therefore, is the lamentable fact that, without understanding of either the text or its structure, we are largely unable to perform any of these tasks satisfactorily. We have neither an understanding of the grammatical form of Voynich words allowing stemming or lemmatization, or a list of stopwords to excise.

Whilst stemming and lemmatization are unapproachable, at least within the confines of this post, we can effect a crude form of stopword removal through use of a common frequency analysis of the text. Stopwords are, in general, those words that are both most-frequently occuring in some corpus of documents and those that are found across the majority of documents in that language. The second criterion ensures that words occurring frequently in obscure and specialised texts are not considered of undue importance.

This overall statistic is known as term frequency-inverse document frequency, or tf-idf, and is widely used in information retrieval to identify terms of specific interest within certain documents that are not shared by the wider corpus. For our purposes, we wish to identify and elide those ubiquitous, frequent terms that occur across the entire corpus. To do so, given our lack of knowledge of the structure of the Voynich Manuscript, we will consider each folio as a separate document, and consider only the inverse document frequency as we are uninterested in how common a word within each document. To avoid words that most commonly appear across the manuscript, with a basis in the distribution of stop words in a range of known languages, we therefore remove the 200 words with lowest inverse document frequency scores¹.

Having contorted the text into an appropriate form for analysis, we can begin the process of discerning its inner secrets. Our code relies on the tidytext and stm packages, allowing for easy manipulation of document structure and topic models²

Numerous Interpretations

Topic models are a cautionary example of recklessly unsupervised machine learning. As with most such approaches, there are a number of subjective choices to be made that affect the outcome. Perhaps the most influential is the selection of the number of topics that the model should generate. Whilst some approaches have been suggested to derive this number purely by analysis, in most cases it remains in the domain of the human supplicant. Typically, the number of topics is guided both by the structure of the text along with whatever arcane purpose the analysis might have. With our imposed lack of understanding, however, we must rely solely on crude metrics to make this most crucial of choices.

Several methods of assessment exist to quantify the fit of a topic model to the text. The two that we will employ, guided by the stm package are semantic coherence, which roughly expresses that words from a given topic should co-occur within a document; and exclusivity, which values models more highly when given words occur within topics with high frequency, but are also relatively exclusive to those topics.

We select an optimal number of topics by the simple process of calculating models with varying numbers of topics, and assessing when these two scores are maximised. For the Voynich Manuscript we observe that 34 topics appears to be initially optimal³.

Selection metrics for Voynich topic model topic numbers.

" data-image-caption="

Selection metrics for Voynich topic model topic numbers.

Selection metrics for Voynich topic model topic numbers. | (PDF Version)

The initial preparation of the code, the search through topic models of varying numbers, and the selection of the final 34 topic model is given in the code below alongside plotting code for the metrics diagram.

Voynich Manuscript Topic Modelling Code

voynich_topics-model.r
[code language=”r”]

library( tidyverse )
library( magrittr )

library( tidytext )
library( widyr )

library( stm )

# install_github("dgrtwo/drlib")
library( drlib )

# References:
#
#
#
#
#

voynich_raw <-
read_csv( "data/voynich_raw.txt", col_names=FALSE ) %>%
rename( folio = X1, text = X2 )

# Read in manually-identiied sections per folio, according to
#
voynich_sections <-
read_csv( "data/voynich_sections.txt", col_names=FALSE ) %>%
rename( folio = X1, section = X2 )

# Merge the above to note section for each folio alongside the text
voynich_tbl <-
left_join( voynich_sections, voynich_raw )

# Tokenize
# Remove words of 3 letters or less.
voynich_words <-
voynich_tbl %>%
unnest_tokens( word, text ) %>%
filter( str_length( word ) > 3 )

# Most common words
voynich_common <-
voynich_words %>%
count( word, sort=TRUE ) %>%
mutate( word = reorder( word, n ) )

# Counts of words per folio
voynich_word_counts <-
voynich_words %>%
count( word, folio, sort = TRUE )

# TF-IDF
voynich_tf_idf <-
voynich_word_counts %>%
bind_tf_idf( word, folio, n ) %>%
arrange( desc( tf_idf ) )

# Based on median stopword count of languages in NLTK
# (),
# remove the 200 lowest-scoring words.
voynich_stopwords <-
voynich_tf_idf %>%
arrange( idf ) %>%
select( word ) %>%
unique() %>%
head( 200 ) %>%
extract2( "word" )

voynich_words <-
voynich_words %>%
filter( !word %in% voynich_stopwords )

# Generate the corpus
voynich_dfm <-
voynich_words %>%
count( folio, word, sort=TRUE ) %>%
cast_dfm( folio, word, n )

# Search for a number of topics and output goodness-of-fit measures.

# N=20 is the number of documents ‘held out’ for the goodness-of-fit measure.
# (The model is trained on the main body, then used to calculated the
# likelihood of the held-out documents.) N=20 is used here to produce
# approximately 10% of the corpus.
if( not( file.exists( "work/voynich_topic_search_k.rds" ) ) ) {

message( "Seaching low-n topic models…" )

voynich_k <-
searchK( voynich_dfm, K=c(3:40), N=20 )

saveRDS( voynich_k, "work/voynich_topic_search_k.rds" )

} else {

voynich_k <-
readRDS( "work/voynich_topic_search_k.rds" )

}

# Based on the metrics above, use 34-topic model
if( not( file.exists( "work/voynich_topic_stm-34.rds" ) ) ) {

message( "Calculating 34-topic model…" )

# Setting K=0 uses (Lee and Minno, 2014) to select a number of topics
voynich_topic_model_34 <-
stm( voynich_dfm, K=34, init.type="Spectral" )

# This takes a long time, so save output
saveRDS( voynich_topic_model_34, "work/voynich_topic_stm-34.rds" )

} else {

voynich_topic_model_34 <- readRDS( "work/voynich_topic_stm-34.rds" )

}

# Based on the metrics above, also calculated a secondary 12-topic model
if( not( file.exists( "work/voynich_topic_stm-12.rds" ) ) ) {

message( "Calculating 12-topic model…" )

# Setting K=0 uses (Lee and Minno, 2014) to select a number of topics
voynich_topic_model_12 <-
stm( voynich_dfm, K=12, init.type="Spectral" )

# This takes a long time, so save output
saveRDS( voynich_topic_model_12, "work/voynich_topic_stm-12.rds" )

} else {

voynich_topic_model_12 <- readRDS( "work/voynich_topic_stm-12.rds" )

}

# Work initially with the 34-topic model
voynich_topic_model <- voynich_topic_model_34

## Convert output to a tidy tibble
voynich_topic_model_tbl <-
tidy(voynich_topic_model, matrix = "beta" )

voynich_terms <-
tidy(voynich_topic_model, matrix = "gamma" )

# Select the top six terms in each topic for display
voynich_topics_top <-
voynich_topic_model_tbl %>%
group_by(topic) %>%
top_n(6, beta) %>%
ungroup() %>%
arrange(topic, -beta)

# Produce a per-folio topic identification.
# Document – topic – score
topic_identity <-
voynich_terms %>%
group_by( document ) %>%
top_n( 1, gamma ) %>%
arrange( document ) %>%
ungroup

# Reinsert manually-identified section information (derived from
# illustrations).
topic_identity$section <- voynich_tbl$section

saveRDS( topic_identity, "work/topic_identity.rds" )

[/code]

Topic Metric Selection Plotting Code

voynich_topics-plot_metric.r
[code language=”r”]

library( tidyverse )
library( magrittr )

library( ggthemes )
library( showtext )

library( ggplot2 )
library( cowplot )

# References:
#
#
#
#
#

font_add( "voynich_font", "/usr/share/fonts/TTF/weird/voynich/eva1.ttf")
font_add( "main_font", "/usr/share/fonts/TTF/weird/alchemy/1651 Alchemy/1651AlchemyNormal.otf")
font_add( "bold_font", "/usr/share/fonts/TTF/weird/alchemy/1651 Alchemy/1651AlchemyNormal.otf")

showtext_auto()

# Read topic model search
voynich_k <-
readRDS( "work/voynich_topic_search_k.rds" )

# Plot semantic coherence against exclusivity for model selection
voynich_k_plot <-
voynich_k$results %>%
as_tibble %>%
rename( "Semantic Coherence"=semcoh, "Exclusivity"=exclus ) %>%
gather( key="variable", value="value", "Semantic Coherence", "Exclusivity" ) %>%
rename( "Topic Count"=K, "Value"=value )

# We will use cowplot, so set the theme here.
theme_set(theme_cowplot(font_size=4, font_family = "main_font" ) )

# Plot semantic coherence against exclusivity.
# (Highlight the selected 34-topic point.)
voynich_k_semcoh_exclusive <-
ggplot( voynich_k_plot, aes( x=`Topic Count`, y=Value, group=variable) ) +
geom_line( colour="#8a0707" ) +
facet_wrap( ~variable, ncol=1, scales="free_y" ) +
geom_vline( xintercept=34, colour="#228b22", linetype="longdash" ) +
scale_x_continuous(breaks=c( seq(0, 40, 10), 34 ) ) +
theme (
axis.title.y = element_text( angle = 90, family="main_font", size=12 ),
axis.title.x = element_text( colour="#3c3f4a", family="main_font", size=12 ),
axis.text.x = element_text( colour="#3c3f4a", family="main_font", size=10 ),
axis.text.y = element_text( colour="#3c3f4a", family="main_font", size=10 ),
axis.line.x = element_line( color = "#3c3f4a" ),
axis.line.y = element_line( color = "#3c3f4a" ),
plot.title = element_blank(),
plot.subtitle = element_blank(),
plot.background = element_rect( fill = "transparent" ),
panel.background = element_rect( fill = "transparent" ), # bg of the panel
panel.grid.major.x = element_blank(),
panel.grid.major.y = element_blank(),
panel.grid.minor.x = element_blank(),
panel.grid.minor.y = element_blank(),
legend.text = element_text( family="bold_font", colour="#3c3f4a", size=10 ),
legend.title = element_blank(),
legend.key.height = unit(1.2, "lines"),
legend.position=c(.85,.5),
strip.background = element_blank(),
strip.text.x = element_text(size = 10, family="main_font")
)

# Cowplot trick for ggtitle
title <-
ggdraw() +
draw_label("Vonich Manuscript Topic Selection Metrics", fontfamily="bold_font", colour = "#3c3f4a", size=20, hjust=0, vjust=1, x=0.02, y=0.88) +
draw_label("http://www.weirddatascience.net | @WeirdDataSci", fontfamily="bold_font", colour = "#3c3f4a", size=12, hjust=0, vjust=1, x=0.02, y=0.40)

data_label <-
ggdraw() +
draw_label("Data: http://www.voynich.nu", fontfamily="bold_font", colour = "#3c3f4a", size=8, hjust=1, x=0.98 )

tgp <-
plot_grid(title, voynich_k_semcoh_exclusive, data_label, ncol=1, rel_heights=c(0.1, 1, 0.1))

voynich_topic_selection_plot <-
ggdraw() +
draw_image("img/parchment.jpg", scale=1.4 ) +
draw_plot(tgp)

ggsave( "output/voynich_topic_selection_metrics.pdf", width=16, height=9 )

[/code]

With these torturous steps on our path finally trodden, our path leads at last to a model deriving the underlying word generating probabilities of the Voynich Manuscript. In each facet, the highest-probability words in each topic are shown in order.

Voynich Manuscript Topic Model (34 topics)

" data-image-caption="

Voynich Manuscript Topic Model (34 topics)

Voynich Manuscript Topic Model (34 topics) | (PDF Version)

Topic Model Plotting Code

voynich_topics-plot_topics.r
[code language=”r”]

library( tidyverse )
library( magrittr )

library( ggthemes )
library( showtext )

library( tidytext )

# install_github("dgrtwo/drlib")
library( drlib )

library( ggplot2 )
library( cowplot )

showtext_auto()

# ####################
# ## 34 Topic Model ##
# ####################

# Work with the 34-topic model
voynich_topic_model <-
readRDS( "work/voynich_topic_stm-34.rds" )

## Convert output to a tidy tibble
voynich_topic_model_tbl <-
tidy(voynich_topic_model, matrix = "beta" )

voynich_terms <-
tidy(voynich_topic_model, matrix = "gamma" )

# Select the top six terms in each topic for display
voynich_topics_top <-
voynich_topic_model_tbl %>%
group_by(topic) %>%
top_n(6, beta) %>%
ungroup() %>%
arrange(topic, -beta)

# We will use cowplot, so set the theme here.
theme_set(theme_cowplot(font_size=4, font_family = "main_font" ) )

# Plot each topic as a geom_col(). Use drlib’s ‘reorder_within’ to order bars
# within each facet. (Note that the scale_x_reordered() is needed to fix
# (flipped!) x-axis labels in the output.)
#
gp <-
voynich_topics_top %>%
mutate(term = reorder_within(term, beta, topic)) %>%
ggplot(aes(term, beta, fill = factor(topic))) +
geom_col( alpha=0.8, show.legend = FALSE) +
theme( axis.text.y = element_text( family="voynich_font", size=10 ) ) +
facet_wrap(~ topic, scales = "free") +
scale_x_reordered() +
coord_flip() +
labs( x="Term", y="Probability in Topic" )

# Theming
gp <-
gp +
theme (
axis.title.y = element_text( margin = margin(t = 0, r = 12, b = 0, l = 0), angle = 90, family="main_font", size=12 ),
axis.title.x = element_text( margin = margin(t = 12, r = 0, b = 0, l = 0), colour="#3c3f4a", family="main_font", size=12 ),
axis.text.x = element_text( colour="#3c3f4a", family="main_font", size=6 ),
axis.line.x = element_line( color = "#3c3f4a" ),
axis.line.y = element_line( color = "#3c3f4a" ),
plot.title = element_blank(),
plot.subtitle = element_blank(),
plot.background = element_rect( fill = "transparent" ),
panel.background = element_rect( fill = "transparent" ), # bg of the panel
panel.grid.major.x = element_blank(),
panel.grid.major.y = element_blank(),
panel.grid.minor.x = element_blank(),
panel.grid.minor.y = element_blank(),
legend.text = element_text( family="bold_font", colour="#3c3f4a", size=10 ),
legend.title = element_blank(),
legend.key.height = unit(1.2, "lines"),
legend.position=c(.85,.5),
strip.background = element_blank(),
strip.text.x = element_text(size = 10, family="main_font")
)

gp <-
gp +
theme(
panel.background = element_rect(fill = "transparent", colour = "transparent"),
plot.background = element_rect(fill = "transparent", colour = "transparent"),
legend.background = element_rect(fill = "transparent", colour = "transparent")
)

# Palette of ink colours (based on screenshots of Diamine inks).
ink_colours <- c( "#753733", "#b6091d", "#e45025", "#232d1d",
"#224255", "#533f50", "#453437", "#7f2430",
"#254673", "#52120e", "#3d2535", "#25464b",
"#2f2a1c" )

# Create a vector of selections from the palette, one for each topic.
ink_palette <-
sample( ink_colours, size=34, replace=TRUE )

# Add fill colours to plot.
gp <-
gp + scale_fill_manual( values=ink_palette )

# Cowplot trick for ggtitle
title <-
ggdraw() +
draw_label("Vonich Manuscript Topic Model (34 Topics)", fontfamily="bold_font", colour = "#3c3f4a", size=20, hjust=0, vjust=1, x=0.02, y=0.88) +
draw_label("http://www.weirddatascience.net | @WeirdDataSci", fontfamily="bold_font", colour = "#3c3f4a", size=12, hjust=0, vjust=1, x=0.02, y=0.40)

data_label <-
ggdraw() +
draw_label("Data: http://www.voynich.nu", fontfamily="bold_font", colour = "#3c3f4a", size=8, hjust=1, x=0.98 )

tgp <-
plot_grid(title, gp, data_label, ncol=1, rel_heights=c(0.1, 1, 0.1))

voynich_topic_plot <-
ggdraw() +
draw_image("img/parchment.jpg", scale=1.4 ) +
draw_plot(tgp)

ggsave( "output/voynich_topic_plot_34.pdf", width=16, height=9 )

[/code]

Of Man and Machine

The topic model produces a set of topics in the form of probability distributions generating words. The association of each topic to a folio in the Voynich Manuscript represents these probabilistic assignments based solely on the distribution of words in the text. There is a secondary topic identification, however, tentatively proposed by scholars of the manuscript. The obscure diagrams decorating almost every folio provide their own startling implications as to the themes detailed in the undeciphered prose.

We might wish to ask, then: do the topic assignments generated by the machine reflect the human intepretation? To what extent do pages decorated with herbal illuminations follow certain machine-identified topics compared with those assigned to astronomical charts?

The illustration-based thematic sections of the Voynich Manuscript fall into eight broad categories, according to Zandbergen. These sections are, briefly:

Herbal, detailing a range of unidentified plants, comprising most of the first half of the manuscript;
astronomical, focusing on stars, planets, and astronomical symbols;
cosmological, displaying obscure circular diagrams of a similar form to the astronomical;
astrological, in which small humans are displayed mostly in circular diagrams alongside zodiac signs;
biological, characterised by small drawings of human figures, often connected by tubes;
pharmaceutical, detailing parts of plants and vessels for their preparation;
starred text, divided into short paragraphs marked with a star, with no other illustrations; and
text only pages.

With these contextual descriptions, we can examine the relationship between the speculative assignments of the topic model against the suggestions of the diagrams.

Voynich Manuscript Folio Topic Heatmap

" data-image-caption="

Voynich Manuscript Folio Topic Heatmap

Voynich Manuscript Folio Topic Heatmap | (PDF Version)

Voynich Manuscript Topic Model Folio Heatmap Plotting Code

voynich_topics-plot_heatmap.r
[code language=”r”]

library( tidyverse )
library( magrittr )

library( ggthemes )
library( showtext )

# install_github("dgrtwo/drlib")
library( drlib )

library( ggplot2 )
library( cowplot )

font_add( "main_font", "/usr/share/fonts/TTF/weird/alchemy/1651 Alchemy/1651AlchemyNormal.otf")
font_add( "bold_font", "/usr/share/fonts/TTF/weird/alchemy/1651 Alchemy/1651AlchemyNormal.otf")

showtext_auto()

# Set the number of topics
n_topics <- 34

# Load the appropriate topic model
voynich_topic_model <-
readRDS( paste0( "work/voynich_topic_stm-", n_topics, ".rds" ))

theme_set(theme_cowplot(font_size=4, font_family = "main_font" ) )

# Load folio topic identity assignments
topic_identity <-
readRDS( "work/topic_identity.rds" )

# Cowplot
theme_set(theme_cowplot(font_size=4, font_family = "main_font" ) )

# Plot topic heatmap
topic_heatmap <-
topic_identity %>%
ggplot( aes( x=document, y=topic, fill=section ) ) +
geom_tile( colour="#3c3f4a", alpha=0.8, size=0.4 ) +
scale_fill_brewer( palette="Dark2", direction=1, name="Section", labels=c("Astrological", "Astronomical", "Biological", "Cosmological", "Herbal", "Pharmaceutical", "Starred Text", "Text Only" ) ) +
ggtitle( "Voynich Folio Topic Assignments", paste( n_topics, "Topic Model" )) +
labs( x="Folio", y="Topic" ) +
theme (
plot.title = element_text( family="bold_font", size=22 ),
plot.subtitle = element_text( family="bold_font", size=12 ),
panel.grid.major.x = element_blank(),
panel.grid.major.y = element_blank(),
panel.grid.minor.x = element_blank(),
panel.grid.minor.y = element_blank(),
) +
scale_y_continuous(labels = seq( 1, n_topics, 1 ), breaks = seq( 1, n_topics, 1 ), minor_breaks = seq(0.5 , n_topics+.5, 1) ) +
scale_x_continuous(minor_breaks = seq(0.5 , 226.5, 5) )

gp <-
topic_heatmap +
theme (
axis.title.y = element_text( margin = margin(t = 0, r = 12, b = 0, l = 0), angle = 90, family="main_font", size=12 ),
axis.title.x = element_text( colour="#3c3f4a", family="main_font", size=12 ),
axis.text.x = element_text( colour="#3c3f4a", family="main_font", size=10 ),
axis.text.y = element_text( colour="#3c3f4a", family="main_font", size=10 ),
axis.line.x = element_line( color = "#3c3f4a" ),
axis.line.y = element_line( color = "#3c3f4a" ),
plot.title = element_blank(),
plot.subtitle = element_blank(),
plot.background = element_rect( fill = "transparent" ),
panel.background = element_rect( fill = "transparent" ), # bg of the panel
legend.text = element_text( family="bold_font", colour="#3c3f4a", size=10 ),
legend.title =element_text( family="bold_font", colour="#3c3f4a", size=12 ),
legend.key.height = unit(1.2, "lines"),
strip.background = element_blank(),
strip.text.x = element_text(size = 10, family="main_font")
)

# Cowplot trick for ggtitle
title <-
ggdraw() +
draw_label("Voynich Manuscript Topic Heatmap", fontfamily="bold_font", colour = "#3c3f4a", size=20, hjust=0, vjust=1, x=0.02, y=0.88) +
draw_label("http://www.weirddatascience.net | @WeirdDataSci", fontfamily="bold_font", colour = "#3c3f4a", size=12, hjust=0, vjust=1, x=0.02, y=0.40)

data_label <-
ggdraw() +
draw_label("Data: http://www.voynich.nu", fontfamily="bold_font", colour = "#3c3f4a", size=8, hjust=1, x=0.98 )

tgp <-
plot_grid(title, gp, data_label, ncol=1, rel_heights=c(0.1, 1, 0.1))

voynich_topic_heatmap <-
ggdraw() +
draw_image("img/parchment.jpg", scale=1.4 ) +
draw_plot(tgp)

ggsave( "output/voynich_folio_topic_heatmap.pdf", width=16, height=9 )

[/code]

The colours in the above plot represent the manual human interpretation, whilst the location on the y-axis shows the latent Dirichlet allocation topic assignment.

We might have harboured the fragile hope that such a diagram would have demonstrated a clear confirmatory delineation between the sectional diagrammatic breakdown of the Voynich Manuscript. At a first inspection, however, the topics identified by the analysis appear almost uniformly distributed across the pages of the manuscript.

The topic model admits to a number of assumptions, not least the selection of stopwords through to the number of topics in the model. We must also be cautious: the apparent distribution of topics over the various sections may be deceptive. For the moment, we can present this initial topic model as a faltering first step in our descent into the hidden structures of the Voynich Manuscript. The next, and final, post in this series will develop both the statistical features and the topic model towards a firmer understanding of whether the apparent shift in theme suggest by the illustrations is statistically supported by the text.

Until then, read deeply but do not trust what you read.

Footnotes

Illuminating the Illuminated Part One: A First Look at the Voynich Manuscript

moth — Thu, 26 Sep 2019 20:15:16 +0000

The Voynich Manuscript

While the world abounds with strange phenomena ripe for analysis in their raw state, there is a peculiar pleasure in scrutinising arcane information curated and obscured by the human mind.

The Voynich Manuscript is one of the most well-known and studied volumes of occult knowledge. The book’s most recent history involves its purchase in 1912 by Wilfrid Voynich, a rare book dealer, from a sale of manuscripts by the Society of Jesus at the Villa Mondragone, Frascati. Following several fruitless years of attempts to decipher the manusript and discover its origin, or to interest others in it, Wilfrid Voynich died. The book passed through a number of other hands before being donated to Yale University by the noted rare book dealer Hans P. Kraus in 1969. It now resides in Yale’s Beinecke Rare Book and Manuscript Library with the designation MS 408.

Written almost entirely in an unknown script, barring a small number of words apparently in Latin and High German, the manuscript is compellingly illustrated with depictions of plants, herbs, human figures, astronomical and astrological symbols. The manuscript has resisted all attempts at interpretation by cryptographers, historians, and linguists.

Voynich Manuscript – Folio 178

" data-image-caption="

Voynich Manuscript – Folio 178

" data-large-file="https://www.weirddatascience.net/wp-content/uploads/2019/09/voynich_folio_178.jpg" src="http://www.weirddatascience.net/wp-content/uploads/2019/09/voynich_folio_178.jpg" alt="Voynich Manuscript Folio 178" width="7486" height="3715" class="size-full wp-image-910" />

Voynich Manuscript – Folio 178

From a linguistic and cryptographic perspective, this lack of success in interpretation is not surprising. The two-hundred or so folios of the manuscript, while beautifully illuminated, present a sadly limited corpus of text for the purposes of traditional analysis.

In this short series of posts we will subject the Voynich Manuscript to a range of text analysis techniques, delving into its structure, gain horrific insight into its composition, and skeptically assessing its credibility. The manuscript has been subjected to almost fifty years of furtive attempts by cryptographers, including the US National Security Agency and a menagerie of others from the distinguished to the deranged. We will crudely mimic some earlier results, and hopefully add our own confusion to the roiling mass of current research into the Voynich Manuscript.

Authenticity

Since its discovery, and throughout the ongoing unsuccessful attempts to decipher its contents, many have questioned the authenticity of the Voynich Manuscript. The theory that the entire book is a hoax, either by contemporary scribes or by more modern players, has been raised repeatedly over the years.

Radiocarbon dating in 2010 asserted that the manuscript’s parchment likely dates from the early 15th century; the volume of parchment in the manuscript, and its consistency across the document, make it unlikely, although not impossible, that the book is a modern-day hoax.

Other supporting evidence has drawn from early mentions of the manuscript in correspondence. According to http://www.voynich.nu, which presents a far more detailed and thorough description of the research around the manuscript and its history than we could hope to offer here, the first extant mention of the manuscript can be found in a 1639 letter from Athanasius Kircher in Rome, replying to a letter forwarded from Georgius Barschius of Prague by the mathematician Theodor Moretus.

The letter refers to a “book of mysterious steganography” (“libellum… …steganographici mysterisi”) illustrated with pictures of plants, stars and chemical secrets that Kirscher had not yet had time to decipher. Barschius had sought out Kirscher’s expertise due to his fame at the time for claiming to have, erroneously as it later transpired, deciphered the hieroglyphic writing system of the Ancient Egyptian language. Later correspondence between Barschius and Kirscher appears, according to Zandbergen⁴, to suggest strongly that the mysterious book in question is the Voynich Manuscript based on its description.

A Statistical Argument: Zipf’s Law

We now turn from historical sources to darker, more statistical realms. There is compelling support for the notion that, regardless of the true meaning of the book, its contents are drawn from a human language and are neither random symbols nor any form of sophisticated cipher.

One of the pillars of this argument is that certain statistical properties of the Voynich Manuscripts text strongly resemble those of natural, human languages, and which are unlikely, although not impossible, to arise from random text, artificially generated text, or most forms of encipherment.

The most well-known of these statistical properties is the apparent adherence of the manuscript to Zipf’s Law. This law, made famous by the US linguist George Zipf, observes that in corpora of natural languages, the frequency of a word is inversely proportional to its rank when words from a corpus are ordered by frequency. More plainly: the most common word in a language is likely to be n times more common than the second most common word; the second word will be roughly n times more common than the third word, and so on. Whilst merely an approximation, this law can be seen to hold for most human languages, and for a range of other natural phenomena.

Random gibberish, on the other hand, would most likely not follow Zipf’s Law, although carefully crafted gibberish certainly could. Rugg has demonstrated that a simple mechanical procedure can produce randomised text that adhered to Zipf’s Law, although the example he provides is both somewhat contrived and also presupposes a knowledge of this statistical quirk of human languages in the first place. Given that the physical makeup of the Voynich Manuscript dates to the early 15th Century, some four centuries before Zipf popularised this mathematical assessment of human languages, the argument that it is a contemporary act of calligraphic glossolalia seems strained.

Similarly, most forms of cryptography beyond the simplest substitution ciphers would also skew the text away from Zipf’s Law. It is notable that the Voynich Manuscript predates even works such as Trithemius‘s Steganographia, or the Book of Soyga and its magic tables of letters that so obsessed John Dee.

In contrast, however, it has been claimed that other features of the text raise doubts. One of the most commonly stated counter-arguments to the natural hypothesis of the Voynich text is that some words are repeated an unnatural number of times. Depending on the transcription, individual words have been reported to be repeated up to five times. Whilst this is not an impossible occurrence in human lanugage, it is highly irregular.

The next post in this short series will focus on the Voynich Manuscript’s adherence, or lack thereof, to Zipf’s Law in full. Following that, we will see the extent to which other forms of modern textual analysis can be applied to dissect the arcane and unrelenting secrets of MS 408.

This post, however, will describe the contortions required to render the Voynich text suitable for our particular form of scrutiny.

Assumptions

Given the format and presentation of the text, we make several assumptions about the writing system contained in the Voynich Manuscript:

It is written in an alphabet, or potentially an abjad or even an abugida⁵, and not a logographic system. That the text is not logographic is justified by the small number of individual symbols. The distinction between the other systems is sufficiently subtle that it will not affect our analyses⁶.
The manuscript is written from left to right, and not the reverse, vertically, boustrophedon. This is uncontroversial and apparent from even a cursory inspection of the text itself; the horizontal flow of the writing is clear, with lines clearly starting at the left margin and ending before the right. The text is separated into paragraphs, of which the final line is justified to the left.

Data

Due to the diligent activity of several generations of Voynich researchers, the text of the manuscript has been transcribed into a machine-readable format. As the alphabet is unknown, there are minor uncertainties in rendering the text, leading to a number of similar but competing transcriptions. The subtle details of the various transcription efforts, and their history, are available at: http://www.voynich.nu/transcr.html, with the raw data available at http://www.voynich.nu/data/. We have settled on the v101 transliteration by Glen Claston, rendered in the Intermediate Voynich Transliteration File Format (IVTFF) of Zandbergen. This is one of the more recent and widely-used transcriptions, and has the added advantage of being supported by the availability of a TrueType font. The underlying file is available here: http://www.voynich.nu/data/GC_ivtff_s.txt.

Crude Manipulations

We perform the following steps to make the data usable for our analyses. For many scenarios, we would develop a generalisable set of steps to allow conversion of many documents to an appropriate form. Until and unless, however, a new cache of documents in the same language are found, it is simpler and easier to perform these one-time steps manually.

Firstly, we delete from the text all incomplete words, as marked in the IVTFF format. This includes:

all text in angle brackets
all words containing ?’s
all words containing []

Secondly, we tokenize the text and remove punctuation. The transcription of the Voynich manuscript that we have chosen uses the following punctuation:

“.” is a space
“,” is a potential space. For simplicity, we do not treat these as a space.

Finally, we organize the document in an appropriate form to be imported into an R data frame, or tidyverse tibble.

The above steps were performed in the Vim text editor, and the commands used are reproduced in the code below:

Show Vim text manipulation commands.

[code]

# Delete all commented lines
:\%g/\^\#.\*/d

# Remove blank lines
:\%g/\^\$/d

# Remove "," — assume that potential spaces are /not/ spaces.
:\%s/,//g

# Replace each folio’s page marker (initial for each page)
# with its contents, followed by a comma. ( -\> f1r,)
:\%s/\^\s*\<.{-}\>\$/\rfRos,/
:\%s/\^\<(f\[0-9\]*\[r\|v\]\[0-9\]*)\>\s*\<.{-}\>\$/\r\1,/

# Remove all \<\> entries (non-greedy)
:\%s/\<.{-}\>\s*//g

# Join all paragraphs (all newlines followed by a character
# other than a newline are removed).
:\%s/\n(\[\^\\n\])/.\\1/
:\%s/\^.f/f/
:\%s/,./,/

# Replace "high ascii" rare characters from the IVTFF with their
# ASCII representation. ()
:\%s/@(\[0-9\]{-});/=nr2char(submatch(1))/g

# Replace full stops with spaces
:\%s/./ /g

[/code]

The resulting raw data file is available here. This file can be read into R simply by use of the read.csv function:

voynich_tbl <- 
	read_csv( "data/voynich_raw.txt", col_names=FALSE ) %>%
	rename( folio = X1, text = X2 )

As a first, horrifying glance into the forms of analysis that this allows, we can now use our raw data to identify the most repeated words in the manuscript, according to our transcription. The following R code extracts the entirety of the text and encodes it as a run length encoding. This conveniently results in a sequential list of words and the number of times that each is repeated in sequence. We can then simply extract the largest number of repetitions for each word in the corpus:

Count longest word repetition sequences in the Voynich Manuscript.

[code language=”r”]

library( tidyverse )
library( magrittr )

# Count the number of repeated words in the Voynich Manuscript text.

# Load the raw data
voynich_tbl <-
read_csv( "data/voynich_raw.txt", col_names=FALSE ) %>%
rename( folio = X1, text = X2 )

# Extract the text as a vector of words
voynich_vector <-
voynich_tbl %>%
extract2( "text" ) %>%
paste( sep=" ", collapse=" " ) %>%
str_split( " " ) %>%
unlist

# Create a run length encoding object from the vector
voynich_rle <-
voynich_vector %>%
rle

# Convert rle object to a data frame and report the maximum number of repeated
# cases for each word
voynich_repetitions <-
voynich_rle %>%
unclass %>%
as.data.frame %>%
group_by( values ) %>%
summarise( max_repetitions = max( lengths ) ) %>%
ungroup %>%
arrange( desc( max_repetitions ) )

[/code]

This simple analysis shows that, in the transcription we have chosen, the longest sequences of repeated words are only three words in length, occuring a total of five times in the text. While there are many other arguments against the potential validity of the Voynich Manuscript, word repetition does in itself present a compelling reason to doubt that the text is a human language.

We have now reduced the strange and beautiful elegance of the Voynich Manuscript’s centuries-old illuminations to a crude, utilitarian abstraction. With this particular act of artistic and literary desecration complete, in the next post we will examine Zipf’s Law in more detail, and interrogate the extent to which this law supports or undermines the text’s authenticity.