How to get Medium stats data and make your own visualizations

I started to write programming & data related articles in Medium recently. Just as a new curious user to Medium, I frequently visit https://medium.com/me/stats to check how many readers has read my posts. Although those numbers rarely change, but these numbers are really what I care, at least for current several days. Since it means something for me, I want to keep trace of those numbers (especially the change trending)

But soon, I learned although Medium has API:

Medium/medium-api-docs

But its API is quite limited, mainly used to publish articles to Medium. But there is no official Medium Stats API. After more research, I found an already Archived project:

CharlesStover/medium-stats-api

The most useful thing I learned is: when you use HTTP Clients to access https://medium.com/me/stats, if you set “SID” & “UID” inside cookie, then you can get data successfully (Although the return data is HTML, but it is very easy to use regex to extract the real stats data in JSON format), Great!

First, just visit https://medium.com/me/stats by web browser, like Chrome. Then, open Chrome’s developer tool, inside “Network” tab, find any request, and look up the “sid” & “uid” inside cookie HTTP Header. (keep those in secret), then I wrote the following Python Script to fetch my article data, and save them into Apache Parquet format.

import re
import json
import pandas as pd
import requests
from datetime import datetime
uid='__your_uid_from_cookie__'
sid='__your_sid_from_cookie__'
r = requests.get('https://medium.com/me/stats?limit=9999',
                  cookies = dict(uid=uid, sid=sid)
                  )
post_list = json.loads(re.compile('window\["obvInit"\]\((.*)\)').findall(r.text)[0])['value']
df = pd.DataFrame(post_list)
df['fetchTime'] = datetime.now()
del df['previewImage']
df.to_parquet('~/data/collections/medium_stats.parquet')

After python script run successfully, I have save those data into my local file. I can add some additional logic to merge those files together to keep all the data change histories.

Since data is ready, Let’s make some plots. I’d like to use open source https://streamlit.io/ tools to do this job. Streamlit is an amazing tool, by using it, I can turn small number of python lines into a fully workable web application! The Python script for my web app is as following:

import pandas as pd
import streamlit as st
import altair as alt
full_df = pd.read_parquet('~/data/collections/medium_stats.parquet')
st.header('My Medium Posts\'s stats')
df = full_df[['title', 'views', 'reads','readingTime', 'claps', 'upvotes', 'fetchTime']]
bars = alt.Chart(df).mark_bar().encode(
    x=alt.X('title', axis=alt.Axis(labels=False)),
    y='views',
    size=alt.Size('readingTime', legend=alt.Legend(orient='bottom-right')),
    color=alt.Color('title', legend=alt.Legend(
        orient='top-right', direction='vertical'
    )),
    tooltip=['reads', 'claps', 'upvotes']
)
text = bars.mark_text(dy=-9).encode(text='views:Q')
st.altair_chart(bars + text, use_container_width=True)
st.table(df)

I personally like to use altair (which wraps Vega-Lite) for visualization, but streamlit also support several alternatives. Finally, I can have my own dashboard!