Introduction

I’m into light novels so I’ve decided to combine my love for data science and light novel.

Information Retrieval

I started out by retrieving some data from online via scraping using scrapy. I love scrapy and highly recommend it for any serious web scrapping task.

The Data

So what does the data look like?

2739 novels.

It’s in a json format.

The columns are novel title, description, novel type, genre, tags, rating, language, authors, year, and license.

Exploratory Data Analysis

For this portion I will be using R to do some EDA (❤ John Tukey).

There are eight languages in the data including 7 novels with no language category.

The languages are: Japanese, Chinese, Korean, Malaysian, Filipino, Indonesian, Thai, and Vietnamese.

The average ratings only include novels that have at least 30 user ratings.

I did not take any other rating from other countries because there are less than 30 novels. As a statistician I feel 30 or more is a sufficient number to represent the population for each category anything smaller than is insufficient.

I plot a histogram plot for all novels from all countries with 30 or more user ratings. It is left skewed. While I can only speculate that people tend to rate for their favorite novels. It’s interesting because there is quite a lot of quality novels out there and many hover above 4.0 rating out of 5.0.

There is a small number with perfect five ratings. 19 novels with 5 ratings all of them have single digit number of users reviews except for one (“Mum, I Used to Hate You”). It have 18 users that reviewed it with a 5 rating.

Sweet let’s find novels that have high rating and at least 30 user reviews! I tried 4.5 at first but it’s a huge list at least 300+ so I decided to look for 4.7 ratings. There is exactly 75 novels with that criteria.

Here’s the list and hope you guys find it useful.

A Game To Make Him Fall
A Will Eternal
A Slight Smile is Very Charming
Ame no Hi no Iris
At the Northern Fort
By A Slight Mistake
Cultivation Chat Group
Clockwork Planet
Demon Girl ~Tale of a Lax Demon~
Dungeon Defense
Eight Treasures Trousseau
Gekkou
Golden Age Legitimate Fei
Hokuou Kizoku to Moukinzuma no Yukiguni Karigurashi
Hiraheishi wa Kako o Yumemiru
Hikaru ga Chikyuu ni Itakoro……
I Reincarnated into an Otome Game as a Villainess With Only Destruction Flags…
I Don’t Like The World, I Only Like You
It’s Because You Said There Would Be Candy!!
Joy of Life
Kenkyo, Kenjitsu o Motto ni Ikite Orimasu
Kaze no Stigma
Kono Kamen no Akuma ni Sodan wo!
Kuzu to Kinka no Qualidea
Manuscript Screening Boy and Manuscript Submitting Girl
Marietta-hime no Konrei
Marginal Operation
Maoyuu Maou Yuusha
Mimizuku to Yoru no Ou
Mondaiji-tachi ga Isekai kara Kuru Sou Desu yo?
Mulberry Song
My Death Flags Show No Sign of Ending
No Game No Life
Nobunaga’s Imouto is My Wife
Ookami to Koushinryou
Overlord (LN)
Ouroboros Record ~Circus of Oubeniel~
Our Second Master
Quickly Wear the Face of the Devil
Rakuin no Monshou
Rain
Release that Witch
Running Away From The Hero!
Sansheng, Wangchuan Wu Shang
Semi Datte Tensei Sureba Ryuu Ni Naru
Sweet Heart in Honeyed Desire
Tabi ni Deyou, Horobiyuku Sekai no Hate Made
The Bathroom Goddess
The Destruction of a Triad Boss Trilogy
The Girl Who Bore the Flame Ring
The Girl Who Ate a Death God
The Founder of Diabolism
The Legend of Sun Knight
The Grandmaster Strategist
The Magnificent Battle Records of A Former Noble Lady
The Princess Wei Yang
The Probability I Can Kill My Wife Without Being Found Out
The Other World Dining Hall (LN)
The Rebirth of the Malicious Empress of Military Lineage
The Scum Villain’s Self-Saving System
The Tang Dynasty’s Female Forensic Doctor
The Witch and the Gourd of Stories
To Be A Virtuous Wife
Three Days of Happiness
Uchi no Musume no Tame naraba, Ore wa Moshikashitara Maou mo Taoseru kamo Shirenai (LN)
Uchi no Musume no Tame naraba, Ore wa Moshikashitara Maou mo Taoseru kamo Shirenai (WN)
Tsuyokute New Saga (LN)
Vermillion
Utsuro no Hako to Zero no Maria
Why Is the Prettiest Girl in School Trying to Talk to a Loner Like Me during Lunch Break?
When He Comes, Close Your Eyes
Virtual World: Peerless White Emperor
Woof Woof Story ~ I Told You I am a Rich Person’s Dog, Not Fenrir ~
Yahari Ore no Seishun Love Come wa Machigatte Iru
Youjo Senki

Conclusion

I think I’ll be doing more light novel analysis in the future.

I think I’ll do NLP too something like sentiment analysis or something for those tags and categories. Of course this is when I have time.

What Did I Get to Practice? (for me)

• My data science & statistic skills using R.
• My web scraping skill using scrapy.
• My devops skill, I made a custom vagrant for jekyll and used ansible to self provision. I created a new blog just for this post.
• I made a yummy pie chart.

Etc..

The novel picture is taken from google searching for a CC license image.