Partager via

The fluid language of the Web

  • Article

We prepared, as we had for the earlier dataset, the top-100K words list for the body stream for Apr10.  You can download it here.

We decided to take a closer look at the dataset to how the top 100K lists changed between Jun09 and Apr10.  Our findings are interesting:

  • The union of the word set is just shy of 110K.  This means that 10% of the words either fell in or out of the top 100K.  This is a turnover rate higher than I expected.
  • Some words that are newly in the top list are what you'd expect (unigram log10 probability difference shown parenthetically):
    • espnlosangeles (21.88993), an ESPN satellite established during 2009
    • debate2010 (21.53613)
  • Some words took a predictable jump:
    • ipad (2.560667), a product introduced mid-year
  • Quite a few words newly in the mix are not conversational words:
    • childreplyhtml (22.09848)
    • focaladvid (21.76564)

Curious indeed.

Comments

  • Anonymous
    September 22, 2011
    > childreplyhtml This suggests the data is dirty in some respect, doesn't it? Humans don't use that sort of word, that's obvious some sort of HTML source fragment creeping into the n-grams.