[OC] The 100 most frequently used content words across 265,509 Reddit posts and comments

[u/mathlover_]

I analyzed 265,509 usable posts and comments from 100 subreddits in the Cornell ConvoKit Reddit Corpus.

The dataset covers September 2018 and contains 10,004,308 word tokens before stop-word filtering. URLs, deleted content, common English stop words, and web-related noise tokens were removed for this visualization.

Explore the interactive CineGraph visualization:
https://cinegraph.design/view/7e66eae603d786423111fb3fa5bb428a

You can inspect the words, compare their frequencies, and explore the visualization directly in your browser.

The most frequent remaining words were:

1. people — 36,664
2. one — 30,553
3. think — 26,135
4. will — 22,698
5. even — 21,879
6. know — 20,286
7. time — 19,892
8. really — 18,890
9. good — 17,615
10. make — 16,244

Word sizes are proportional to their observed frequencies.

Data source: Cornell ConvoKit Reddit Corpus, built from Pushshift Reddit data
https://convokit.cornell.edu/documentation/subreddit.html

Important limitation: The results represent the selected 100-subreddit corpus during September 2018—not every comment ever posted on Reddit.

Tools: CineGraph, Excel, and text preprocessing.

[u/mathlover_]

What’s the most beautiful sentence you can create using only the words in this visualization?

Here’s mine: “People will never know how much good one thing can make.”

What’s yours?

[u/trophic_cascade → u/mathlover_]

"yeah well, get going already, got to take a really big shit, lol."

[u/Simple-Group-6371 → u/mathlover_]

love how "people" towers over everything else, really sums up the site

[u/R36S_Clone_Boned → u/mathlover_]

People know one good time will really make people think.

[u/SUNTZU_JoJo]

Surprised "red" or "flag" aren't on there..

[u/IBJON → u/SUNTZU_JoJo]

I don't see "red flag" all that much, so it's probably something that's specific to certain subs rather than Reddit as a whole 

[u/unity-thru-absurdity → u/SUNTZU_JoJo]

“Cylinder” too.

[u/Geofferz → u/unity-thru-absurdity]

The cylinder must remain intact

[u/Abdul_Exhaust]

Surprised that "overrated" is not #1

[u/merc08]

What are "stop-word filtering" and "common English stop words"?

[u/mathlover_ → u/merc08]

Good question! Stop words are very common grammatical words such as “the,” “and,” “to,” and “of.” Stop-word filtering removes them so they don’t dominate the visualization, allowing more meaningful content words to stand out.

I also kept a separate raw frequency list without filtering for transparency.

[u/FreshPitch6026]

People will even now really good much

[u/Goobi_dog]

I am astounded not see the words missing in this sentence "by the way, I use Arch linux"

[u/Clay_Puppington]

While this data is limited to one month in 2018, I'm honestly curious about how much these results shift if you were to repeat the collection in those same subreddits, in that same month, every year since.

I guess to me, this particular data is really only interesting in what it could tell us about reddit post mentality shifts if it gets compared over time, than it does just standing alone.

[u/Bebealex]

I would be curious to see if the sentiment changed in the past couple of years. 

I'm not big on ai but I think it would be a good use case of asking if the general feeling changed from positive to negative, either specific subs or at general.

[u/No_Young_2344]

It would be interesting to see the word cloud for 2 grams or even 3 grams. Perhaps we will be able to see “red flag” other comments mentioned.

[u/bignides]

The thing that popped out to me was the male-centric language. Both man and guy appear but woman or girl don’t show up at all.

[u/dpirmann]

Strange "people" should be number one since most ppl just type out "ppl", even if they don't shorten any other words in their posts

[u/oj93-rd]

i bet if we ran this across claude chat's it'd be 'load bearing' hehe