Hacker News Generator

How to generate new Hacker News titles on Google Cloud

The time was 1:30PM, I was just coming back from a lunch break at the office with some spare time before my next meeting.

«Let’s just take a quick look at Hacker News before getting back to work»

I remember one of the top post being “My business card runs Linux” (link), I’ve seen this exact same one a few times and couldn’t help thinking this particular title was very «HackerNews-y».
What if we had a script generate new titles from a corpus of all previous HN titles, how far would the apple land from the tree?

^{_{(note please that my goal here is not to point the finger at this particular post, I was actually interested in the challenges of running linux on a system as small as a business card)}}

Let’s do it!

The proof of concept

Obtaining the data

First things first, I need to make sure I can get ahold of previous Hacker News data I could build onto 🧐
Fortunately I had the intuition these would be accessible in a public BigQuery dataset.
After a bit of searching, I found bigquery-public-data.hacker_news.stories!

BigQuery is a cloud-based data warehouse with an easily accessible query engine API.
In a typical project, you would use BigQuery to store any long-term data from your applications, such as logs, that you would then use for analytics, or any application data that you archive.
To showcase its impressive request engine on large datasets, BigQuery also allows you to query public datasets from any user or company.

As you can see on this screenshot, the whole dataset holds 1,959,808 records of stories posted between October 2005 and October 2015!

After a bit of data exploration, I decided to extract only the posts with a score above 100 upvotes to restrict the titles I would be working with.
With a very simple SQL query I got myself around 43,000 lines of data out of the nearly 2 millions, which BigQuery output in under a single second, not too bad!

SELECT
  title
FROM
  `bigquery-public-data.hacker_news.stories`
WHERE
  score > 100

As you can see we can use standard SQL with BigQuery, nothing fancy. Using the bq command line, I ran this query and extracted all titles separated by newlines into a local CSV file.

bq query \
  --format csv \
  --use_legacy_sql=false \
  'SELECT title FROM `bigquery-public-data.hacker_news.stories` where score > 100' \
  > hn_title.csv`

The logic

Now my plan was to generate new sentences from this corpus using Markov chains.

Markov chains are relatively simple systems to generate sequences of random tokens based on a simple rule:

• The first S words in the sentence are always randomly picked from a row in the corpus.
• To pick the next word in a sequence, you look at the S previous words.

The number of previous words you keep in memory is the state.
Using 2 previous words to pick a 3rd one makes your state size of 2.

By increasing the size of your state you can avoid jumping from totally unrelated sequences of words. However, you will be decreasing the number of possible outcomes.
To keep the sentences less random, since the titles were already using some very precise technical jargon, I settled with a size of 3.

import markovify

with open("hn_title.csv") as f:
    text = f.read()

text_model = markovify.NewlineText(text, state_size=3)

def get_title():
    return text_model.make_short_sentence(280)

if __name__ == "__main__":
    get_title()

$ gcloud functions deploy hn_title \
  --runtime python37 \
  --entry-point get_title \
  --trigger-http \
  --region europe-west1

> [Output] https://location-project.cloudfunctions.net/hn_title

pip install functions-framework
functions-framework --target get_title --debug

FROM python:3.7-slim

# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME

# Install production dependencies.
COPY requirements.txt .
RUN pip install gunicorn functions-framework
RUN pip install -r requirements.txt

COPY . .
# Run the web service on container startup. Here we use the gunicorn
# webserver, with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 -e FUNCTION_TARGET=get_title functions_framework:app

$ tree
.
├── Dockerfile
├── hn_title.csv
├── main.py
└── requirements.txt

$ gcloud run deploy hntitlegen \
  --allow-unauthenticated \
  --source=.

> [Output] https://hntitlegen-uanwpkxudq-ew.a.run.app/