More harpsicord?!

Seattle is lucky to have KINGFM, a local radio station dedicated to 100% classical music. As one of the few existent classical music fans in his twenties, I listen often enough. Over the past few years, I've noticed that when I tune to the station, I always seem to hear the plinky sound of a harpsicord.

Before I sent KINGFM an email, admonishing them for playing so much of an instrument I dislike, I wanted to investigate whether my ears were deceiving me. Perhaps my own distaste for the harpsicord increased its impact in my memory.

This article outlines the details of this investigation and especially the process of collecting the data.

If it ain't baroque...

A harpsicord is in many ways similar to the piano. Pressing down and releasing one of its keys triggers an internal mechanism that plucks a string inside. Resultant vibration of the string produces the corresponding pitch. Because its strings are plucked, the instrument has no dynamic range. Each note sounds at roughly the same volume; however firmly or softly the player strikes the keys. Some harpsicords have several choirs of strings that allow the player limited control of the volume and timbre.

The harpsicord can sound tinny to modern ears. Thomas Beecham famously said, "The sound of a harpsichord - two skeletons copulating on a tin roof in a thunderstorm."

At the start of the 16th century, the newly invented fortepiano began to push both the harpsicord and its close relative, the clavicord out of favor. The new instrument worked more like the modern piano in that its strings were struck with padded hammers. Compared to the other keyboard instruments of its day, it had a more resonant sound and allowed the player to control the dynamics of each note simply by altering the force with which he struck the keys.

The period before the fortepiano, during which the harpsicord had its heyday is known as the Baroque Era. The history of classical music is often divided into several historical "eras" or "periods". The dates that separate them are somewhat arbitrary with substantial overlap, I'll follow Wikipedia in defining these boundaries because they have the most comprehensive composers data with the most permissive licence.

Wikipedia's dates differ little from those outlined by Naxos, a well respected music label with an extremely comprehensive catalog. Unfortunately, the Naxos ToS are extremely restrictive with respect to their composer data.

These eras are:

• Medieval (476–1400)
• Renaissance (1400–1600)
• Baroque (1600–1760)
• Classical era (1730–1820)
• Romantic era (1815–1910)
• 20th century (1900–2000)
• 21st century (since 2000)

Since King seems to play very little music from before 1600, I ignored the Medieval and Renaissance era in my analysis.

Because of the dominance of the piano and its predecessors starting in the Classical Era, one is less likely to find the sound of the harpsicord in modern recordings of anything but Baroque music. Even then, music originally written for harpsicord is often transcribed to the piano and recorded that way. Glenn Gould, perhaps the most famous modern Bach interpreter, is well-known for recordings of such transcriptions.

One exception is opera. Harpsicord was used for accompanying recitative all the way into the late 18th century. For simplicity, we will blissfully ignore this fact.

Collecting the data

KINGFM's posts their playlist daily to their website. Scraping this data, I was able to build the dataset.

request = require('request')
cheerio = require('cheerio')

parse_page = (error, response, body) ->
  if(error or response.statusCode != 200)
    console.log(error)
  else
    # Load the page into cheerio
    $ = cheerio.load(body)

    # Iterate over the the links on the front page
    $('#siteTable a.title').each (k,v) =>

      console.log($(v).attr('href'))

request.get("http://www.reddit.com", parse_page)

The joys of data normalization

Then came the hard part: associating composer names in the playlist data with historical eras. This is more difficult than it seems because subtle differences in the datasets could result in mismatches. King is mostly consistent in defining its composers but it does so in a different format from Wikipedia. It lists "SCHUBERT" instead of "Franz Schubert". These cases are easy enough: simply convert to lowercase and lop off the first name.

There are several types of more difficult cases where the database contains multiple composers that share a last name. e.g. J.S. Bach and all of his sons. In these cases, we need first names or initials to differentiate. Unfortunately, the formats differ between the data sets. Wikipedia has "Carl Philipp Emanuel Bach" and KINGFM, "BACH, C.P.E". Other tough cases are those where composer has multiple last names e.g. "Vaughan Williams". Other annoying cases occur where diacritics did not match e.g."Dvořák" and "DVORÁK" (no accented r). Since my data set is fairly small (3210 playlist items), I was able to handle these unfortunate cases with regular expressions and frustration.

Results

Play count

2691 of the 3208 playlist items had matches in the database, leaving 472 unidentified tracks. The results were distributed like this

Era distribution

Top composers

Composer	Play count
Mozart	191
Bach	188
Haydn	114
Beethoven	92
Schubert	83
Chopin	78
Debussy	66
Mendelssohn	63
Brahms	51
Tchaikovsky	46

Air time

Analyzing the total play count for each era is useful. But the more interesting question for a listener is not how often tracks from each era are played. Rather, it's what proportion of the airtime each era consumes. This is an important distinction because some movements last less than a minute while others can last 30 minutes or longer.

Top Composers by airtime (including only the top 16 composers or 48% of total):

This data highlights the importance of using airtime over play count. While King plays J.S. Bach almost as often as Mozart, Mozart gets 42% more airtime, more than 20 hours more per month compared to Bach.

Async and ordering

When analyzing airtime, we have to make sure all of our data is properly sorted. Since we are scraping asynchronously and writing the results to disk as they are returned, it's likely that our data will come back in an order different from the order of the HTTP requests.

This can occur because some pages are larger and so take longer to transfer than others. A glance at the data shows that this did indeed happen:

Time	Composer
07/19/2012 11:51pm	PURCELL
07/19/2012 11:54pm	SCHUBERT
07/16/2012 12:02am	CHOPIN

Since we made non-blocking HTTP requests, the data from 7/19 arrived more quickly and so was written to disk before the data from the 16th.

Since we want to access this data as a JavaScript object anyway, we ought not rely on the default ordering of an object's properties. Field ordering is not not part of the ECMAScript spec. To remedy this, we will use moment.js to parse each date string and convert it to a UNIX timestamp.

These timestamps will be cast to two separate data structures, a list and hash. The list will be sorted and used for ordering. The hash maps timestamps to composer names. Iterating through the list, we look up the associated composer and use subtraction to work out the total seconds of airtime for each track.

require('fs')
require('moment')

composers_by_air_time = ->

  dates = []
  map = {}
  playlist = JSON.parse(fs.readFileSync('db/playlist.json'))

  for date, composer of playlist
    # Build an array of timestamps together with a mapping from
    # timestamp to composer
    unix_date = moment(date, 'MM/DD/YYYY hh:mma').unix()

    # Push it onto our list of timestamps
    dates.push(unix_date)

    # Map the composer whose work STARTED to this timestamp
    map[unix_date] = composer

  # Sort the dates (as ints) so we can subtract adjacent members
  dates.sort

  results = {}

  for idx, date of dates when idx > 0
    # Subtract each item from its predecessor, ignoring the first one
    prev_date = dates[idx - 1]
    difference = date - prev_date
    composer = map[prev_date]

    # Group by composer name
    if results[composer] > 0
      results[composer] += difference
    else
      results[composer] = difference

  return results

Airtime by era

We can combine the airtime data with the composer era data to get the total airtime by era

Conclusions