Most of my experience with ML and Deep learning has been with tabular data and NLP. Here is a basic vision model with the fastai library in the scope of Part 1 (2022) fastai Deep learning course.

I am using the Breast Cancer CT dataset from Kaggle to train a model https://www.kaggle.com/datasets/sabermalek/bcfpp The dataset has 3 categorical labels:

0 : Cancer
1 : Benign
2 : Normal

#
# import libraries
import joblib as jlb
from fastai.vision.all import *

#
# Read data
images, labels, masks = jlb.load('../input/bcfpp/BCFPP.jlb')

# labels contains 3 classes (0 for Cancer, 1 for Benign and 2 for normal)

images = np.uint8(images)

# lets see what we got
print(f"images: {images.shape}")
print(f"labels: {labels.shape}")
print(f"masks: {masks.shape}")

images: (3600, 384, 256)
labels: (3600,)
masks: (3600, 384, 256)

We have 3600 Breast CT scans size 384 x 256 with labels and a mask

Lets look at an image

im = Image.fromarray(images[0])
im.to_thumb(384)

And the corresponding mask

im = Image.fromarray(np.uint8(masks[0]))
im.to_thumb(384)

Not sure what to do with the mask data at this point so we will ignore it for now when training our model.

Lets look at the distribution of our data labels to see if we are dealing with an unbalanced dataset.

(pd.array(labels)
 .value_counts()
 .to_frame(name='labels')
 .set_axis(['count'], axis=1)
 .rename(index={0:'cancer',1:'benign',2:'normal'})
)

Perfectly balanced so no issue there.

DataBlock

We have not covered DataBlock in depth yet and all the examples involve reading image files from disk. The image data from the Breast CT scan dataset is in the form of a Numpy array. We can use the DataBlock by defining functions for get_items, get_x and get_y.

Not the most elegant solution but at this stage it will get us to a model quickly.

I tried using lambda functions but I got and error with learn.export when I did that.

#
# just return the item index for items
def get_items(i):
    return i

# return image
def get_x(i):
    return images[i]

# return mapped label
def get_y(i):
    if labels[i] == 0: return "cancer"
    if labels[i] == 1: return "benign"
    if labels[i] == 2: return "normal"

#  Build DataBlock and keep 20% of our data for validation set
dls = DataBlock(
    blocks=(ImageBlock(cls=PILImageBW), CategoryBlock), 
    splitter=RandomSplitter(valid_pct=0.2, seed=42),
    get_items=get_items,
    get_x=get_x,
    get_y=get_y,
    item_tfms=Resize(384)
).dataloaders(list(range(images.shape[0])))

# look at a few images in our DataBlock
dls.show_batch(max_n=6)

learn = vision_learner(dls, resnet18, metrics=error_rate)
learn.fine_tune(6)

Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth

learn.export('model.pkl')

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(cmap='Purples')

In this classification we are looking to optimize cancer recall. Misclassifying cancer for benign or benign for cancer is not an issue as a biopsy would most likely be performed to see if cancer is actually present. So the bad outcome would be our model predicting a normal scan when it was actually cancer.

In our confusion matrix this was only the case for 2 scans !

print(f'percent of instances where model predicted normal when cancer was present: {round((2/len(dls.valid_ds))*100,2)}%')

percent of instances where model predicted normal when cancer was present: 0.28%

The model missed a cancer diagnosis in only 0.28% of the CT scans! This is considering that the 68 times our model mislabelled cancer for benign it would have been caught with a biopsy.

S3 partners short interest

Short interest is usually measured by shares on loan. Some brokers don't rely on borrowing shares as they have shares in inventory.

eg russell 3000

In their promotional presentation, S3 states:

stock borrowed data is 45% within 10% of actual reported
S3 data is 85% within 10% of actual reported

Common Short Float = shares shorted / shares available to trade

Despite its widespread use, this calculation is flawed in two main ways:

US investors are only required to report short shares twice per month, leading to a short interest number being roughly ten days stale by the time it gets to investors Float does not accurately represent shares available to trade on a daily basis To combat these flaws, S3 provides a true daily shares shorted measure and calculates more accurate “tradeable shares” than the general definition for Float provides.

S3 points out that “what is missing [from the general definition for float] are the ‘synthetic longs’ that are created as a result of a short sale which, in some stocks, can be a very significant number.” The synthetic long is a result of a long shareholder lending out their shares, a short seller borrowing those shares, and a long buyer on the other side of the short sale now owning the shares. In this case, the long buyer on the other side of the short sale has increased the market’s potential tradable quantity of shares. The interesting feature in the S3 data is the Squeeze Risk which we will look in depth.

S3 Data Exploratory Data Analysis

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
import liberator
from datetime import datetime, timedelta

pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', 40)

# S3 data for 2021 as of December 23rd US stock only
df_s3_21_us = pd.read_csv('df_s3_23dec21_us.csv', dtype={"Cusip":"string"});

print(f"S3 short float mean: {df_s3_21_us['S3SIPctFloat'].mean():.3f}")
print(f"S3 short max: {df_s3_21_us['S3SIPctFloat'].max()}")
print(f"S3 short median: {df_s3_21_us['S3SIPctFloat'].median()}")
print(f"S3 short skew: {df_s3_21_us['S3SIPctFloat'].skew():.3f}")

S3 short float mean: 0.035
S3 short max: 1.0
S3 short median: 0.0013
S3 short skew: 6.248

df_s3_21_us.iloc[:,7:].head()

Crowding: S3’s proprietary index score measuring the magnitude of shorting/covering activity relative to the security’s float, borrow capacity and financing rate.

Short Interest: Real-time short interest expressed in shares.

ShortInterestNotional: ShortInterest * Price (USD)

ShortInterestPct: Real-time short interest as a percentage of equity float.

S3Float: The number of tradable shares including synthetic longs created by short selling.

S3SIPctFloat: Real-time short interest projection divided by the S3 float.

IndicativeAvailability: S3's projected available lendable quantity

Utilization: S3's Utilization is defined as S3 Short Interest divided by Total Lendable supply

DaysToCover10Day: Liquidity measure = Short Interest / 10 day average ADTV

DaysToCover30Day: Liquidity measure = Short Interest / 30 day average ADTV

DaysToCover90Day: Liquidity measure = Short Interest / 90 day average ADTV

Finviz scan for tradable stock universe with volume > 500k, price > 10$ and ATR > 0.5

# average volume > 500k, price > 10$, ATR > 0.5
df_finviz = pd.read_csv('finviz_dec_23.csv')

df_finviz.head()

# 1294 stocks
df_finviz.shape

(1294, 19)

Lets extract S3 data for december 23rd to compare with the Finviz data for that day

# extract s3 data for dec 23, the day of our finviz screen
df_s3_dec_23 = df_s3_21_us[df_s3_21_us['timestamp'].str.startswith('2021-12-23')]

# filter s3 data for our target stocks from finviz
df_s3_dec23_filtered = df_s3_dec_23[df_s3_dec_23.symbol.isin(df_finviz.Ticker)]
df_s3_dec23_filtered.head()

Top 10 stocks by S3 Squeeze Risk for December 23

# top 10 s3 for squeeze risk on december 23
s3_top10 = df_s3_dec23_filtered.nlargest(10, 'Squeeze Risk')[['symbol','ShortInterestPct','S3SIPctFloat','Crowded Score','Squeeze Risk']]
s3_top10

# add new column 'fshortn' with Float short as a float
df_finviz['fshortn'] = df_finviz['Float Short'].str.replace('%','').astype(float)

# top 20 finviz Float Short
finviz_top20 = df_finviz.nlargest(20,'fshortn')[['Ticker','Shares Float','Float Short']]
finviz_top20

# function to get common elements in 2 lists
def Intersection(lst1, lst2):
    return set(lst1).intersection(lst2)

How many of our top 10 Squeeze Risk stocks are caught with a Finviz top 20 short float for December 23?

# see what symbols from top 10 float short in S3 are in finviz top 20
Intersection(s3_top10.symbol, finviz_top20.Ticker)

{'ARCH', 'BFRI', 'CRTX', 'HRTX', 'ICPT', 'RLAY', 'TTCF'}

The top 20 float short stocks from finviz catch 7/10 of the top Squeeze Risk S3 stocks.

# top 10 s3 for Short Interest Pct on december 23
s3_SIP_top10 = df_s3_dec23_filtered.nlargest(10, 'ShortInterestPct')[['symbol','ShortInterestPct','S3SIPctFloat','Crowded Score','Squeeze Risk']]
s3_SIP_top10

Looking at the short interest values, we can see that some stocks with similar short float value have very different Squeeze Risk. So the proprietary Squeeze Risk data has different information in it.

How many top 10 S3 short float value do we catch with the finviz top 20 ?

# see what symbols from top 10 float short in S3 are in finviz top 20
Intersection(s3_SIP_top10.symbol, finviz_top20.Ticker)

{'BFRI', 'BLNK', 'CRTX', 'HRTX', 'ICPT', 'LGVN', 'LMND', 'SAVA'}

The top 20 float short stocks from finviz for dec 23 catch 8/10 of the top Short Interest Pct S3 stocks.

# lets merge finviz data our S3 data
merge_df = pd.merge(df_finviz, df_s3_dec23_filtered[['symbol','ShortInterestPct','S3SIPctFloat','Squeeze Risk']],left_on='Ticker',right_on='symbol')
merge_df.head()

Lets see how correlated S3 short float data is the Finviz equivalent

# Scatter plot of finviz Float Short versus S3 Squeeze Risk
sns.set(rc = {'figure.figsize':(10,8)})
sns.scatterplot(x=merge_df['fshortn'], y=merge_df['S3SIPctFloat']*100);
plt.xlabel("Finviz Short Float");

The S3 SIP percentage float values look to be highly correlated with the Finviz Short Float values and would not add anything to a model. This is for one day but this was done for about 20 days with the same results.

print(f"correlation between S3 and Finviz short float: {merge_df['S3SIPctFloat'].corr(merge_df['fshortn']):.3f}")

correlation between S3 and Finviz short float: 0.933

This is highly correlated. Anything above 0.8 is highly correlated. A correlation that high would seem to indicate that based solely on the S3 percent Float data the gain in accuracy S3 advertises is not significant compared to Finviz Elite data. Granted this was done for about 20 days with the same results.

The S3 metric we are really interested in is their proprietary Squeeze Risk.

sns.scatterplot(x=merge_df['S3SIPctFloat']*100, y=merge_df['Squeeze Risk']);

print(f"correlation between S3 short float and S3 Squeeze Risk:\n{merge_df['S3SIPctFloat'].corr(merge_df['Squeeze Risk']):.3f}")

correlation between S3 short float and S3 Squeeze Risk:
0.615

S3 proprietary Squeeze Risk seem to have more information than just their percent float value, this looks more promising.

Lets look at S3 data correlation between data features.

target = ['Short Momentum',
 'Short Interest',
 'ShortInterestNotional',
 'ShortInterestPct',
 'S3Float',
 'S3SIPctFloat',
 'IndicativeAvailability',
 'DaysToCover10Day',
 'DaysToCover30Day',
 'DaysToCover90Day',
 'Crowded Score',
 'Squeeze Risk',
 'symbol']

plt.figure(figsize=(12,7));
sns.heatmap(df_s3_21_us[target].corr(),annot=True,cmap='Blues');

This heatmap indicates that none of the S3 features are highly correlated together. So in a model, every feature could be included and contribute to performance.

# lets filter our s3 data to our Finviz tradable stock universe
df_s3_21_us_filtered = df_s3_21_us.copy()
df_s3_21_us_filtered = df_s3_21_us_filtered[df_s3_21_us_filtered.symbol.isin(list(df_finviz.Ticker))]

df_s3_21_us_filtered.shape

(297388, 26)

# add timestamp date column
df_s3_21_us_filtered['Date'] = pd.to_datetime(df_s3_21_us_filtered.timestamp)
df_s3_21_us_filtered.iloc[:,7:].head()

# set index to date
df_s3_21_us_filtered.index = df_s3_21_us_filtered['Date']
del df_s3_21_us_filtered['timestamp']
del df_s3_21_us_filtered['Date']
df_s3_21_us_filtered.head()

Lets build a dataframe with the daily top 10 S3 Squeeze Risk with quote data including previous day close and previous day vwap. On a given trading day open, one would have all of the S3 data, the Open price and the previous day's close and vwap.

# new dataframe with S3 and quote data
df_top10 = pd.DataFrame(columns=['date','symbol','Open','High','Low','Close','Volume','vwap','prev_close','prev_vwap','Offer Rate','Bid Rate','Last Rate','Short Momentum','Short Interest','ShortInterestNotional','ShortInterestPct','S3Float','S3SIPctFloat','IndicativeAvailability','DaysToCover10Day','DaysToCover30Day','DaysToCover90Day','Crowded Score','Squeeze Risk','squeeze_return','profitable'])

# build date array for 2021
# dates = list(set(list(df_s3_21_us_filtered.index)))
dates = np.unique(np.array(df_s3_21_us_filtered.index))
dates[:5]

array(['2021-01-04T00:00:00.000000000', '2021-01-05T00:00:00.000000000',
       '2021-01-06T00:00:00.000000000', '2021-01-07T00:00:00.000000000',
       '2021-01-08T00:00:00.000000000'], dtype='datetime64[ns]')

# for some reason S3 has data on Thanksgiving, November 25th which is a non trading day
index_to_remove = np.where(np.array([str(d) for d in dates]) == '2021-11-25T00:00:00.000000000')

# remove Thanksgiving
dates = np.delete(dates, index_to_remove)

# get date strings for liberator api
date_strings = [str(d)[:10] for d in dates]
date_strings = np.array(date_strings)

Loop to populate our dataframe. As a rough POC, we add a profitable feature to indicate if a stock reach 10% or above its open price for that day.

So we are assuming a Squeeze occurs if the high for the day reaches 10% above the stock open. This value should probably be higher but this is just a first exploration of the data.

index = 0
for d in dates[1:]:
    # print(d)
    # top 10 s3 for squeeze risk
    # top_10 = df_s3_21_us_filtered.loc[d].nlargest(10, 'Squeeze Risk')[['symbol','ShortInterestPct','S3SIPctFloat','Crowded Score','Squeeze Risk']]
    top_10 = df_s3_21_us_filtered.loc[d].nlargest(10, 'Squeeze Risk')[['symbol','Offer Rate','Bid Rate','Last Rate','Short Momentum','Short Interest','ShortInterestNotional','ShortInterestPct','S3Float','S3SIPctFloat','IndicativeAvailability','DaysToCover10Day','DaysToCover30Day','DaysToCover90Day','Crowded Score','Squeeze Risk']]
    top_10_symbols = list(top_10.symbol)
    date = str(d)[:10]
    # find index for current date
    date_index = np.where(date_strings == date)[0][0]
    # get index for previous trading day
    date_index -= 1
    end = str((pd.Timestamp(d) + timedelta(days=1)).date())
    # get quotes for top 10 Squeeze Risk symbols for current and previous date
    quotes = liberator.get_dataframe(liberator.query(symbols = top_10_symbols, as_of = end, back_to = date_strings[date_index], name = 'daily_bars'))
    # iterate through each top 10 stock for current day
    for dd, row in top_10.iterrows():
        # quote for symbol
        quote = quotes[quotes.symbol == row['symbol']]
        # no quote for symbol at this date, continue
        if quote.empty:
            print(f'{row["symbol"]} has no quotes for {date}')
            continue
        # current quote
        q = quote[quote.timestamp.str.startswith(date)]
        # previous trading day quote
        qprev = quote[~quote.timestamp.str.startswith(date)]
        
        high = q['high'].values[0]
        open = q['open'].values[0]
        # potential_return is max possible return from open
        potential_return = round((high - open)/open, 3)
        # print(q['symbol'] + ': ' + str(potential_return))
        row['date'] = date
        row['Open'] = open
        row['High'] = high
        row['Low'] = q['low'].values[0]
        row['Close'] = q['close'].values[0]
        row['Volume'] = q['volume'].values[0]
        row['vwap'] = q['vwap'].values[0]
        # prev trading day values
        row['prev_vwap'] = qprev['vwap'].values[0]
        row['prev_close'] = qprev['close'].values[0]
        # max potential return
        row['squeeze_return'] = potential_return
        # label profitable > 10%
        row['profitable'] = 1 if potential_return >= 0.1 else 0
        df_top10.loc[index] = row

        index+=1

SOFI has no quotes for 2021-02-05
PCT has no quotes for 2021-02-12
PCT has no quotes for 2021-02-16
SOFI has no quotes for 2021-04-09
SOFI has no quotes for 2021-04-12
SOFI has no quotes for 2021-04-29
SOFI has no quotes for 2021-04-30
SOFI has no quotes for 2021-05-03
SOFI has no quotes for 2021-05-04
SOFI has no quotes for 2021-05-05
SOFI has no quotes for 2021-05-10
SOFI has no quotes for 2021-05-11
SOFI has no quotes for 2021-05-19
SOFI has no quotes for 2021-05-20
SOFI has no quotes for 2021-05-21
SOFI has no quotes for 2021-05-24
SOFI has no quotes for 2021-05-25
SOFI has no quotes for 2021-05-26

# df_top10.to_csv('df_top10_full.csv', index=False)
df_top10 = pd.read_csv('df_top10_full.csv')

# Extract our data with a high - open greater than 10%
profitable = df_top10.copy()[df_top10.profitable == 1]

# top symbol count of squeeze > 10% for the year
pd.DataFrame(profitable.groupby('symbol').count()['squeeze_return'].sort_values(ascending=False)[:20])

This looks good... Top symbols in our rough arbitrary 10% threshold for a squeeze are AMC, the infamous GME and BBBY.

# squeeze count by date
pd.DataFrame(profitable.groupby('date').count()['squeeze_return'].sort_values(ascending=False)[:20])

With the 10% criteria we got a high of 6 hits on January 27th.

# Average of all max squeeze returns > 10% is 18%
print(f"{profitable['squeeze_return'].mean():.3f}")

0.177

# average of 1.5 top 10 stocks a day with max return > 10%
print(f"{profitable.groupby('date').count()['squeeze_return'].mean():.2f}")

1.55

plt.figure(figsize=(10,8))
plt.hist(profitable['Squeeze Risk']);

Most gains > 10% occur at 100 percent Squeeze Risk in our top10 Squeeze Risk dataset

pd.DataFrame(profitable[['Squeeze Risk']].value_counts(), columns=['count'])

pd.DataFrame(profitable.groupby('Squeeze Risk')['squeeze_return'].mean())

Seems to be an outlier at 75% Squeeze Risk

plt.figure(figsize=(10,8))
plt.plot(profitable.groupby('Squeeze Risk')['squeeze_return'].mean());

# check outlier at 75%
profitable[profitable['Squeeze Risk'] == 75]

ISIG had a high of $24.85 on December 8th for an open of $12.80

profitable.shape

(178, 27)

# 248 trading days
len(set(df_s3_21_us_filtered.index))

248

# range of Squeeze Risks for potential trade returns > 10%
profitable['Squeeze Risk'].min(), profitable['Squeeze Risk'].max()

(75.0, 100.0)

# number of data points with Squeeze Risk >= 75%
df_s3_21_us_filtered[df_s3_21_us_filtered['Squeeze Risk'] >= 75].shape[0]

11673

# from the daily top 10 S3 Squeeze risk percentage that reach at least 10% above the open
df_top10['profitable'].value_counts()[1]/df_top10.shape[0]

0.0728910728910729

7.3% in our top 10 dataset will reach 10% above the open. Our dataset is unbalanced.

This data would be useful as part of a model but it would be unlikely to yield a good model on its own.

xgboost model

# Lets do a quick xgboost model with our data
# convert our target profitabe to int
df_top10['profitable'] = df_top10['profitable'].astype(int)

df_top10.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2442 entries, 0 to 2441
Data columns (total 27 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   date                    2442 non-null   object 
 1   symbol                  2442 non-null   object 
 2   Open                    2442 non-null   float64
 3   High                    2442 non-null   float64
 4   Low                     2442 non-null   float64
 5   Close                   2442 non-null   float64
 6   Volume                  2442 non-null   int64  
 7   vwap                    2442 non-null   float64
 8   prev_close              2442 non-null   float64
 9   prev_vwap               2442 non-null   float64
 10  Offer Rate              2442 non-null   float64
 11  Bid Rate                2442 non-null   float64
 12  Last Rate               2442 non-null   float64
 13  Short Momentum          2435 non-null   float64
 14  Short Interest          2442 non-null   float64
 15  ShortInterestNotional   2442 non-null   float64
 16  ShortInterestPct        2435 non-null   float64
 17  S3Float                 2442 non-null   float64
 18  S3SIPctFloat            2442 non-null   float64
 19  IndicativeAvailability  2442 non-null   float64
 20  DaysToCover10Day        2442 non-null   float64
 21  DaysToCover30Day        2442 non-null   float64
 22  DaysToCover90Day        2442 non-null   float64
 23  Crowded Score           2442 non-null   float64
 24  Squeeze Risk            2442 non-null   float64
 25  squeeze_return          2442 non-null   float64
 26  profitable              2442 non-null   int64  
dtypes: float64(23), int64(2), object(2)
memory usage: 515.2+ KB

from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split

X = df_top10.copy()[['Open','prev_close','prev_vwap','Offer Rate',
 'Bid Rate',
 'Last Rate',
 'Short Momentum',
 'Short Interest',
 'ShortInterestNotional',
 'ShortInterestPct',
 'S3Float',
 'S3SIPctFloat',
 'IndicativeAvailability',
 'DaysToCover10Day',
 'DaysToCover30Day',
 'DaysToCover90Day',
 'Crowded Score',
 'Squeeze Risk']]
X.head()

# lets normalize price values from Open and change percent values to the same scale
X['prev_close'] = X['prev_close']/X['Open']
X['prev_vwap'] = X['prev_vwap']/X['Open']
X['ShortInterestPct'] = X['ShortInterestPct']*100
X['S3SIPctFloat'] = X['S3SIPctFloat'] * 100
del X['Open']
X.head()

X['Squeeze Risk'].value_counts()

100.0    1366
95.0      172
97.5      167
92.5      162
90.0      151
87.5      131
85.0      107
82.5       65
75.0       42
80.0       31
77.5       31
72.5       11
70.0        3
67.5        3
Name: Squeeze Risk, dtype: int64

Target is our >10% open to high label

y = df_top10['profitable']

# Split training with 20% test set and maintain positive ratio.
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=2, stratify=y)

X_train.shape, X_test.shape

((1953, 17), (489, 17))

xgb = XGBClassifier(booster='gbtree',
objective='binary:logistic', max_depth=6,
learning_rate=0.1, n_estimators=100,
random_state=2, n_jobs=-1, scale_pos_weight=13, use_label_encoder=False, eval_metric='logloss')

xgb.fit(X_train, y_train)

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, enable_categorical=False,
              eval_metric='logloss', gamma=0, gpu_id=-1, importance_type=None,
              interaction_constraints='', learning_rate=0.1, max_delta_step=0,
              max_depth=6, min_child_weight=1, missing=nan,
              monotone_constraints='()', n_estimators=100, n_jobs=-1,
              num_parallel_tree=1, predictor='auto', random_state=2,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=13, subsample=1,
              tree_method='exact', use_label_encoder=False,
              validate_parameters=1, verbosity=None)

y_pred = xgb.predict(X_test)

score = accuracy_score(y_pred, y_test)

print('Score: ' + str(score))

Score: 0.8936605316973415

Accuracy of 89% kind of meaningless in such an unbalanced dataset.

# Confusion matrix code
def make_confusion_matrix(cf,
                          group_names=None,
                          categories='auto',
                          count=True,
                          percent=True,
                          cbar=True,
                          xyticks=True,
                          xyplotlabels=True,
                          sum_stats=True,
                          figsize=None,
                          cmap='Blues',
                          title=None):
    '''
    This function will make a pretty plot of an sklearn Confusion Matrix cm using a Seaborn heatmap visualization.
    Arguments
    '''


    # CODE TO GENERATE TEXT INSIDE EACH SQUARE
    blanks = ['' for i in range(cf.size)]

    if group_names and len(group_names)==cf.size:
        group_labels = ["{}\n".format(value) for value in group_names]
    else:
        group_labels = blanks

    if count:
        group_counts = ["{0:0.0f}\n".format(value) for value in cf.flatten()]
    else:
        group_counts = blanks

    if percent:
        group_percentages = ["{0:.2%}".format(value) for value in cf.flatten()/np.sum(cf)]
    else:
        group_percentages = blanks

    box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
    box_labels = np.asarray(box_labels).reshape(cf.shape[0],cf.shape[1])


    # CODE TO GENERATE SUMMARY STATISTICS & TEXT FOR SUMMARY STATS
    if sum_stats:
        #Accuracy is sum of diagonal divided by total observations
        accuracy  = np.trace(cf) / float(np.sum(cf))

        #if it is a binary confusion matrix, show some more stats
        if len(cf)==2:
            #Metrics for Binary Confusion Matrices
            precision = cf[1,1] / sum(cf[:,1])
            recall    = cf[1,1] / sum(cf[1,:])
            f1_score  = 2*precision*recall / (precision + recall)
            stats_text = "\n\nAccuracy={:0.3f}\nPrecision={:0.3f}\nRecall={:0.3f}\nF1 Score={:0.3f}".format(
                accuracy,precision,recall,f1_score)
        else:
            stats_text = "\n\nAccuracy={:0.3f}".format(accuracy)
    else:
        stats_text = ""


    # SET FIGURE PARAMETERS ACCORDING TO OTHER ARGUMENTS
    if figsize==None:
        #Get default figure size if not set
        figsize = plt.rcParams.get('figure.figsize')

    if xyticks==False:
        #Do not show categories if xyticks is False
        categories=False
# MAKE THE HEATMAP VISUALIZATION
    plt.figure(figsize=figsize)
    sns.heatmap(cf,annot=box_labels,fmt="",cmap=cmap,cbar=cbar,xticklabels=categories,yticklabels=categories)

    if xyplotlabels:
        plt.ylabel('True label')
        plt.xlabel('Predicted label' + stats_text)
    else:
        plt.xlabel(stats_text)
    
    if title:
        plt.title(title)

Confusion Matrix

cm1=confusion_matrix(y_test, y_pred)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'not profitable > 10%','profitable > 10%']
make_confusion_matrix(cm1, 
                      group_names=labels,
                      categories=categories,
                      figsize=(10,8), 
                      cmap='Blues')

As expected, poor metrics with this limited S3 data and the unbalance.

What we are really interested in is the feature importance that our model came up with.

Feature importance

xgb.feature_importances_

array([0.04025586, 0.07743476, 0.03004097, 0.05922084, 0.02786616,
       0.04044219, 0.03646403, 0.04996768, 0.05421074, 0.05780815,
       0.        , 0.03577696, 0.24009986, 0.06576711, 0.04685961,
       0.03901406, 0.09877095], dtype=float32)

sorted_idx = xgb.feature_importances_.argsort()
plt.figure(figsize=(10,8))
plt.barh(np.array(list(X))[sorted_idx], xgb.feature_importances_[sorted_idx])
plt.xlabel("Xgboost Feature Importance");

In this preliminary model, DaysToCover10Day is the most important feature by far. More than double the Squeeze Risk importance which would be expected at the top.

Would need to explore this further.

S3SIPctFloat has 0 importance, must be highly correlated to another feature in our top 10 Squeeze Risk dataset.

X[['S3SIPctFloat','ShortInterestPct']].corr()

Confirmed our tree model trigerred on the ShortInterestPct feature which is 99% correlated with S3SIPPctFloat.

Conclusion

S3 data is a valuable dataset. The S3 short data could be used in alpha factor research and contribute to models. The accurate float data by itself is valuable as float affects the way a stock trades.

Finviz Elite short interest data seem to be highly correlated to S3 short data and the stated increased accuracy is not obvious. S3 data includes other proprietary features like Squeeze Risk which provide additional information value.

This is just an initial look at the S3 data, more analysis is warranted.

Introduction

In this project we will be comparing the results from a sentiment analysis model using a RNN implemented directly in PyTorch with another using the Fastai library with the ULMFit (Universal Language Model Fine-tuning) approach from this paper from Jeremy Howard and Sebastian Ruder.

The ULMFit approach involves using a language model first and then train it with the new vocabulary from the new application. Once our language model has been trained with the added new data, we use it as the base from our classification problem. A language model is a model that has been trained to guess the next word coming in a text from having seen the words that came before. This kind of process is called self supervized learning. In this case, no labels are provided to our model.

Lets move onto the actual classification problem we are going to solve.

When deciding the value of a company, it's important to follow the news. For example, a product recall or natural disaster in a company's product chain. You want to be able to turn this information into a signal.

We will be using posts from the social media site StockTwits. The community on StockTwits is full of investors, traders, and entrepreneurs. Each message posted is called a Twit. This is similar to Twitter's version of a post, called a Tweet. We will build a model around these twits that generate a sentiment score.

The data is a large collection of messages that were hand labeled with the sentiment of each. The degree of sentiment is a five-point scale: very negative, negative, neutral, positive, very positive. Each twit is labeled -2 to 2 in steps of 1, from very negative to very positive respectively.

The first thing we should do, is load the data.

PyTorch NLP model

Load Twits Data

This JSON file contains a list of objects for each twit in the 'data' field:

{'data':
  {'message_body': 'Neutral twit body text here',
   'sentiment': 0},
  {'message_body': 'Happy twit body text here',
   'sentiment': 1},
   ...
}

The fields represent the following:

'message_body': The text of the twit.
'sentiment': Sentiment score for the twit, ranges from -2 to 2 in steps of 1, with 0 being neutral.

To see what the data look like by printing the first 10 twits from the list.

from fastai.text.all import *

with open(os.path.join('data', 'stocktwits_sentiment', 'twits.json'), 'r') as f:
    twits = json.load(f)

Length of data

Lets look at the length of our data:

print(len(twits['data']))

1548010

And a couple of messages

print(twits['data'][:5])

[{'message_body': '$FITB great buy at 26.00...ill wait', 'sentiment': 2, 'timestamp': '2018-07-01T00:00:09Z'}, {'message_body': '@StockTwits $MSFT', 'sentiment': 1, 'timestamp': '2018-07-01T00:00:42Z'}, {'message_body': '#STAAnalystAlert for $TDG : Jefferies Maintains with a rating of Hold setting target price at USD 350.00. Our own verdict is Buy  http://www.stocktargetadvisor.com/toprating', 'sentiment': 2, 'timestamp': '2018-07-01T00:01:24Z'}, {'message_body': '$AMD I heard there’s a guy who knows someone who thinks somebody knows something - on StockTwits.', 'sentiment': 1, 'timestamp': '2018-07-01T00:01:47Z'}, {'message_body': '$AMD reveal yourself!', 'sentiment': 0, 'timestamp': '2018-07-01T00:02:13Z'}]

Lets split the messages and the labels

messages = [twit['message_body'] for twit in twits['data']]
# Since the sentiment scores are discrete, we'll scale the sentiments to 0 to 4 for use in our network
sentiments = [twit['sentiment'] + 2 for twit in twits['data']]

messages[45], sentiments[45]

('$NFLX just noticed they have the last jedi on stream. Love this stock', 2)

Preprocessing the Data

With our data in hand we need to preprocess our text. These twits are collected by filtering on ticker symbols where these are denoted with a leader $ symbol in the twit itself. For example,

{'message_body': 'RT @google Our annual look at the year in Google blogging (and beyond) http://t.co/sptHOAh8 $GOOG', 'sentiment': 0}

The ticker symbols don't provide information on the sentiment, and they are in every twit, so we should remove them. This twit also has the @google username, again not providing sentiment information, so we should also remove it. We also see a URL http://t.co/sptHOAh8. Let's remove these too.

import json
import nltk
import os
import random
import re
import torch

from torch import nn, optim
import torch.nn.functional as F

nltk.download('wordnet')

def preprocess_1(message):
    """
    This function takes a string as input, then performs these operations: 
        - lowercase
        - remove URLs
        - remove ticker symbols 
        - removes punctuation
        - tokenize by splitting the string on whitespace 
        - removes any single character tokens
    
    Parameters
    ----------
        message : The text message to be preprocessed.
        
    Returns
    -------
        tokens: The preprocessed text into tokens.
    """ 
    # Lowercase the twit message
    text = message.lower()
    
    # Replace URLs with a space in the message
    text = re.sub(r'https?://[^\s]+', ' ', text)
    
    # Replace ticker symbols with a space. The ticker symbols are any stock symbol that starts with $.
    text = re.sub(r'\$[a-z]*\b', ' ', text)
    
    # Replace StockTwits usernames with a space. The usernames are any word that starts with @.
    text = re.sub(r'@\w*\b', ' ', text)

    # Replace everything not a letter with a space
    text = re.sub(r'[^a-z]', ' ', text)
    
    # Tokenize by splitting the string on whitespace into a list of words
    tokens = text.split()

    # Lemmatize words using the WordNetLemmatizer. You can ignore any word that is not longer than one character.
    wnl = nltk.stem.WordNetLemmatizer()
    tokens = [wnl.lemmatize(t) for t in tokens if len(t) > 1]
    
    return tokens

[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Unzipping corpora/wordnet.zip.

Preprocess All the Twits

tokenized = [preprocess_1(m) for m in messages]

tokenized[:1], messages[:1]

([['great', 'buy', 'at', 'ill', 'wait']],
 ['$FITB great buy at 26.00...ill wait'])

Looking good

Lets check for empty tokens

len([token for token in tokenized if len(token) == 0])

48528

Clean that up

good_tokens = [idx for idx, token in enumerate(tokenized) if len(token) > 0]
tokenized = [tokenized[idx] for idx in good_tokens]
sentiments = [sentiments[idx] for idx in good_tokens]

Bag of Words

Now with all of our messages tokenized, we want to create a vocabulary and count up how often each word appears in our entire corpus.

from collections import Counter
"""
Create a vocabulary by using Bag of words
"""
stacked_tokens = [word for twit in tokenized for word in twit]

bow = Counter(stacked_tokens)
# sort by decreasing order
sorted_bow = sorted(bow, key=bow.get, reverse=True)

Frequency of Words Appearing in Message

With our vocabulary parsed, lets remove some of the most common words such as 'the', 'and', 'it', etc. These words don't contribute to identifying sentiment and are really common, resulting in a lot of noise in our input. If we can filter these out, then our network should have an easier time learning.

We also want to remove really rare words that show up only in a few twits. Here you'll want to divide the count of each word by the number of messages. Then remove words that only appear in some small fraction of the messages.

# The key is the token and the value is the frequency of that word in the corpus.
total_words = len(bow)

freqs = {word: count/total_words for word, count in bow.items()}

# Float that is the frequency cutoff. Drop words with a frequency that is lower or equal to this number.
low_cutoff = 1e-5

# Integer that is the cut off for most common words. Drop words that are the `high_cutoff` most common words.
high_cutoff = 15

# The k most common words in the corpus. Use `high_cutoff` as the k.
K_most_common = [word[0] for word in bow.most_common(high_cutoff)]


filtered_words = [word for word in freqs if (freqs[word] > low_cutoff and word not in K_most_common)]
print(K_most_common)
len(filtered_words)

['the', 'to', 'is', 'for', 'on', 'of', 'and', 'in', 'this', 'it', 'at', 'will', 'up', 'are', 'you']

98448

Remove Filtered Words from Vocabulary

from tqdm import tqdm

# A dictionary for the `filtered_words`. The key is the word and value is an id that represents the word.
vocab = {word: i for i, word in enumerate(filtered_words, 1)}

# Reverse of the `vocab` dictionary. The key is word id and value is the word. 
id2vocab = {i: word for i, word in enumerate(filtered_words, 1)}

# tokenized with the words not in `filtered_words` removed.
filtered = [ [w for w in msg if w in filtered_words] for msg in tqdm(tokenized) ]

100%|██████████| 1499482/1499482 [2:13:51<00:00, 186.69it/s]

Balancing the classes

If we look at how our twits are labeled, we'll find that 45% of them are neutral. This means that our network will be 45% accurate just by guessing 0 every single time. To help our network learn appropriately, we'll want to balance our classes. That is, make sure each of our different sentiment scores show up roughly as frequently in the data.

What we can do here is go through each of our examples and randomly drop twits with neutral sentiment. We want to get around 20% neutral twits starting from 50% neutral.

for i in range(0,5):
    print(f'{i}: {sentiments.count(i)/len(sentiments)}')

0: 0.08710474683924181
1: 0.11403071193918966
2: 0.44561321843143165
3: 0.20476004380179288
4: 0.14849127898834397

balanced = {'messages': [], 'sentiments':[]}

n_neutral = sum(1 for each in sentiments if each == 2)
N_examples = len(sentiments)
keep_prob = (N_examples - n_neutral)/4/n_neutral
print(f'keep_prob: {keep_prob}')

for idx, sentiment in enumerate(sentiments):
    message = filtered[idx]
    if sentiment != 2 or random.random() < keep_prob:
        balanced['messages'].append(message)
        balanced['sentiments'].append(sentiment)

keep_prob: 0.31102465021124265

Lets check that we are balanced:

n_neutral = sum(1 for each in balanced['sentiments'] if each == 2)
N_examples = len(balanced['sentiments'])
n_neutral/N_examples

0.1998550428903639

Looks good, 1/5th of our samples are neutral now.

Let's convert our tokens into integer ids which we can pass to the network.

token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
sentiments = balanced['sentiments']

Neural Network

With our vocabulary mapped to interger ids, we are now ready to build our neural network.

Here is a diagram showing the network:

Embed -> RNN -> Dense -> Softmax

class TextClassifier(nn.Module):
    def __init__(self, vocab_size, embed_size, lstm_size, output_size, lstm_layers=1, dropout=0.1):
        """
        Initialize the model by setting up the layers.
        
        Parameters
        ----------
            vocab_size : The vocabulary size.
            embed_size : The embedding layer size.
            lstm_size : The LSTM layer size.
            output_size : The output size.
            lstm_layers : The number of LSTM layers.
            dropout : The dropout probability.
        """
        
        super().__init__()
        self.vocab_size = vocab_size
        self.embed_size = embed_size
        self.lstm_size = lstm_size
        self.output_size = output_size
        self.lstm_layers = lstm_layers
        self.dropout = dropout

        # Setup embedding layer
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.lstm = nn.LSTM(embed_size, lstm_size, lstm_layers, 
                            dropout=dropout, batch_first=False)
        
        # Setup additional layers
        self.dropout = nn.Dropout(dropout)
        self.fc = nn.Linear(lstm_size, output_size)
        self.logsoftmax = nn.LogSoftmax(dim=1)


    def init_hidden(self, batch_size):
        """ 
        Initializes hidden state
        
        Parameters
        ----------
            batch_size : The size of batches.
        
        Returns
        -------
            hidden_state
            
        """
        
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data
        
        hidden = (weight.new(self.lstm_layers, batch_size, self.lstm_size).zero_(),
                  weight.new(self.lstm_layers, batch_size, self.lstm_size).zero_())
        
        return hidden


    def forward(self, nn_input, hidden_state):
        """
        Perform a forward pass of our model on nn_input.
        
        Parameters
        ----------
            nn_input : The batch of input to the NN.
            hidden_state : The LSTM hidden state.

        Returns
        -------
            logps: log softmax output
            hidden_state: The new hidden state.

        """
        # embeddings and lstm_out
        nn_input = nn_input.long()
        embeds = self.embedding(nn_input)
        lstm_out, hidden_state = self.lstm(embeds, hidden_state)
        
        lstm_out = lstm_out[-1, : , :]
        
        # dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)
        
        logps = self.logsoftmax(out)
        
        return logps, hidden_state

View Model

model = TextClassifier(len(vocab), 10, 6, 5, dropout=0.1, lstm_layers=2)
model.embedding.weight.data.uniform_(-1, 1)
input = torch.randint(0, 1000, (5, 4), dtype=torch.int64)
hidden = model.init_hidden(4)

logps, _ = model.forward(input, hidden)
print(logps)
print(model)

tensor([[-2.0672, -1.8099, -1.3161, -1.5152, -1.5058],
        [-2.0718, -1.7388, -1.3464, -1.5252, -1.5115],
        [-2.0702, -1.7568, -1.3375, -1.5287, -1.5054],
        [-2.0329, -1.9265, -1.2760, -1.5094, -1.4997]], grad_fn=<LogSoftmaxBackward>)
TextClassifier(
  (embedding): Embedding(98448, 10)
  (lstm): LSTM(10, 6, num_layers=2, dropout=0.1)
  (dropout): Dropout(p=0.1, inplace=False)
  (fc): Linear(in_features=6, out_features=5, bias=True)
  (logsoftmax): LogSoftmax(dim=1)
)

Training

DataLoaders and Batching

Now we should build a generator that we can use to loop through our data. It'll be more efficient if we can pass our sequences in as batches. Our input tensors should look like (sequence_length, batch_size). So if our sequences are 40 tokens long and we pass in 25 sequences, then we'd have an input size of (40, 25).

If we set our sequence length to 40, what do we do with messages that are more or less than 40 tokens? For messages with fewer than 40 tokens, we will pad the empty spots with zeros. We should be sure to left pad so that the RNN starts from nothing before going through the data. If the message has 20 tokens, then the first 20 spots of our 40 long sequence will be 0. If a message has more than 40 tokens, we'll just keep the first 40 tokens.

def dataloader(messages, labels, sequence_length=30, batch_size=32, shuffle=False):
    """ 
    Build a dataloader.
    """
    if shuffle:
        indices = list(range(len(messages)))
        random.shuffle(indices)
        messages = [messages[idx] for idx in indices]
        labels = [labels[idx] for idx in indices]

    total_sequences = len(messages)

    for ii in range(0, total_sequences, batch_size):
        batch_messages = messages[ii: ii+batch_size]
        
        # First initialize a tensor of all zeros
        batch = torch.zeros((sequence_length, len(batch_messages)), dtype=torch.int64)
        for batch_num, tokens in enumerate(batch_messages):
            token_tensor = torch.tensor(tokens)
            # Left pad!
            start_idx = max(sequence_length - len(token_tensor), 0)
            batch[start_idx:, batch_num] = token_tensor[:sequence_length]
        
        label_tensor = torch.tensor(labels[ii: ii+len(batch_messages)])
        
        yield batch, label_tensor

Training and Validation

Split it into training and validation sets.

split_frac = 0.8
split_idx = int(len(token_ids)*split_frac)

train_features, valid_features = token_ids[:split_idx], token_ids[split_idx:]
train_labels, valid_labels = sentiments[:split_idx], sentiments[split_idx:]

text_batch, labels = next(iter(dataloader(train_features, train_labels, sequence_length=20, batch_size=64)))
model = TextClassifier(len(vocab)+1, 200, 128, 5, dropout=0.)
hidden = model.init_hidden(64)
logps, hidden = model.forward(text_batch, hidden)

Train model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = TextClassifier(len(vocab)+1, 1024, 512, 5, lstm_layers=2, dropout=0.2)
model.embedding.weight.data.uniform_(-1, 1)
model.to(device)

TextClassifier(
  (embedding): Embedding(98449, 1024)
  (lstm): LSTM(1024, 512, num_layers=2, dropout=0.2)
  (dropout): Dropout(p=0.2, inplace=False)
  (fc): Linear(in_features=512, out_features=5, bias=True)
  (logsoftmax): LogSoftmax(dim=1)
)

"""
Train your model with dropout. Make sure to clip your gradients.
Print the training loss, validation loss, and validation accuracy for every 100 steps.
"""

epochs = 3
batch_size = 1024
learning_rate = .001
clip = 5


print_every = 100
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
model.train()

train_losses = []
valid_losses = []
valid_accs = []

for epoch in range(epochs):
    print('Starting epoch {}'.format(epoch + 1))
    
    steps = 0
    for text_batch, labels in dataloader(
            train_features, train_labels, batch_size=batch_size, sequence_length=20, shuffle=True):
        steps += 1
        # initialize hidden state
        hidden = model.init_hidden(batch_size=labels.shape[0])
        
        # Set Device
        text_batch, labels = text_batch.to(device), labels.to(device)
        for each in hidden:
            each.to(device)
        
        # reset gradients
        model.zero_grad()
        
        # get output from model
        log_probs, hidden = model(text_batch, hidden)
        
        # calculate the loss and perform backprop
        loss = criterion(log_probs, labels)
        loss.backward()
        
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(model.parameters(), clip)
        optimizer.step()
        
        if steps % print_every == 0:
            model.eval()
            
            # Get validation loss
            val_losses = []
            for text_batch, labels in dataloader(valid_features, valid_labels,
                                                batch_size=batch_size, sequence_length=20,
                                                shuffle=True):
                val_hidden = model.init_hidden(labels.shape[0])
                
                text_batch, labels = text_batch.to(device), labels.to(device)
                for each in val_hidden:
                    each.to(device)
                    
                val_log_probs, test_hidden = model(text_batch, val_hidden)
                valid_loss = criterion(val_log_probs, labels)
                
                # Accuracy
                probs = torch.exp(val_log_probs)
                top_prob, top_class = probs.topk(1)
                equality = top_class == labels.view(*top_class.shape)
                valid_acc = torch.mean(equality.type(torch.FloatTensor))
            
            train_losses.append(loss.item())
            valid_losses.append(valid_loss.item())
            valid_accs.append(valid_acc.item())

            model.train()
            
            print(f'Epoch: {epoch+1} / {epochs} \tStep: {steps}',
                  f'\n  Train Loss: {loss.item():.3f}',
                  f'  Validation Loss: {valid_loss.item():.3f}',
                  f'  Validation Accy: {valid_acc.item():.3f}')

Starting epoch 1
Epoch: 1 / 3 	Step: 100 
  Train Loss: 0.913   Validation Loss: 0.956   Validation Accy: 0.634
Epoch: 1 / 3 	Step: 200 
  Train Loss: 0.856   Validation Loss: 0.877   Validation Accy: 0.658
Epoch: 1 / 3 	Step: 300 
  Train Loss: 0.781   Validation Loss: 0.841   Validation Accy: 0.670
Epoch: 1 / 3 	Step: 400 
  Train Loss: 0.783   Validation Loss: 0.724   Validation Accy: 0.716
Epoch: 1 / 3 	Step: 500 
  Train Loss: 0.745   Validation Loss: 0.761   Validation Accy: 0.713
Epoch: 1 / 3 	Step: 600 
  Train Loss: 0.738   Validation Loss: 0.706   Validation Accy: 0.725
Epoch: 1 / 3 	Step: 700 
  Train Loss: 0.722   Validation Loss: 0.688   Validation Accy: 0.746
Epoch: 1 / 3 	Step: 800 
  Train Loss: 0.730   Validation Loss: 0.752   Validation Accy: 0.714
Starting epoch 2
Epoch: 2 / 3 	Step: 100 
  Train Loss: 0.640   Validation Loss: 0.747   Validation Accy: 0.738
Epoch: 2 / 3 	Step: 200 
  Train Loss: 0.652   Validation Loss: 0.780   Validation Accy: 0.706
Epoch: 2 / 3 	Step: 300 
  Train Loss: 0.689   Validation Loss: 0.756   Validation Accy: 0.694
Epoch: 2 / 3 	Step: 400 
  Train Loss: 0.686   Validation Loss: 0.753   Validation Accy: 0.722
Epoch: 2 / 3 	Step: 500 
  Train Loss: 0.679   Validation Loss: 0.722   Validation Accy: 0.722
Epoch: 2 / 3 	Step: 600 
  Train Loss: 0.674   Validation Loss: 0.746   Validation Accy: 0.701
Epoch: 2 / 3 	Step: 700 
  Train Loss: 0.680   Validation Loss: 0.720   Validation Accy: 0.715
Epoch: 2 / 3 	Step: 800 
  Train Loss: 0.683   Validation Loss: 0.683   Validation Accy: 0.729
Starting epoch 3
Epoch: 3 / 3 	Step: 100 
  Train Loss: 0.600   Validation Loss: 0.756   Validation Accy: 0.704
Epoch: 3 / 3 	Step: 200 
  Train Loss: 0.602   Validation Loss: 0.733   Validation Accy: 0.731
Epoch: 3 / 3 	Step: 300 
  Train Loss: 0.615   Validation Loss: 0.744   Validation Accy: 0.722
Epoch: 3 / 3 	Step: 400 
  Train Loss: 0.625   Validation Loss: 0.716   Validation Accy: 0.719
Epoch: 3 / 3 	Step: 500 
  Train Loss: 0.563   Validation Loss: 0.758   Validation Accy: 0.694
Epoch: 3 / 3 	Step: 600 
  Train Loss: 0.663   Validation Loss: 0.760   Validation Accy: 0.716
Epoch: 3 / 3 	Step: 700 
  Train Loss: 0.587   Validation Loss: 0.698   Validation Accy: 0.719
Epoch: 3 / 3 	Step: 800 
  Train Loss: 0.666   Validation Loss: 0.689   Validation Accy: 0.716

So we get about 72% accuracy on the validation set. Lets try the ULMFit approach

ULMFit approach

from fastai.text.all import *

Import Twits

with open(os.path.join('data', 'stocktwits_sentiment', 'twits.json'), 'r') as f:
    twits = json.load(f)

messages = [twit['message_body'] for twit in twits['data']]
# Since the sentiment scores are discrete, we'll scale the sentiments to 0 to 4 for use in our network
sentiments = [twit['sentiment'] + 2 for twit in twits['data']]

Preprocessing the Data

For the preprocess step, fastai does a lot of the work for us using the Spacy tokenizer by default. We keep track of uppercase letters so we do not lowercase everything. Our preprocess function is therefore simplified.

def preprocess(message):
 
    text = message
    
    # Replace URLs with a space in the message
    text = re.sub(r'https?://[^\s]+', ' ', text)
    
    # Replace ticker symbols with a space. The ticker symbols are any stock symbol that starts with $.
    text = re.sub(r'\$[a-zA-Z]*\b', ' ', text)
    
    # Replace StockTwits usernames with a space. The usernames are any word that starts with @.
    text = re.sub(r'@\w*\b', ' ', text)

    # Replace everything not a letter with a space
    text = re.sub(r'[^a-zA-Z]', ' ', text)
    
    # Remove multiple spaces
    text = ' '.join(text.split())
    # Tokenize by splitting the string on whitespace into a list of words
    #tokens = text.split()

    # Lemmatize words using the WordNetLemmatizer. You can ignore any word that is not longer than one character.
    # wnl = nltk.stem.WordNetLemmatizer()
    # tokens = [wnl.lemmatize(t) for t in tokens if len(t) > 1]
    
    return text

messages_clean = [preprocess(m) for m in messages]

messages_clean[:10], sentiments[:10]

(['great buy at ill wait',
  '',
  'STAAnalystAlert for Jefferies Maintains with a rating of Hold setting target price at USD Our own verdict is Buy',
  'I heard there s a guy who knows someone who thinks somebody knows something on StockTwits',
  'reveal yourself',
  'Why the drop I warren Buffet taking out his position',
  'bears have reason on to pay more attention',
  'ok good we re not dropping in price over the weekend lol',
  'Daily Chart we need to get back to above',
  'drop per week after spike if no news in months back to s if BO then bingo what is the odds'],
 [4, 3, 4, 3, 2, 3, 0, 3, 4, 0])

defaults.text_proc_rules

[<function fastai.text.core.fix_html>,
 <function fastai.text.core.replace_rep>,
 <function fastai.text.core.replace_wrep>,
 <function fastai.text.core.spec_add_spaces>,
 <function fastai.text.core.rm_useless_spaces>,
 <function fastai.text.core.replace_all_caps>,
 <function fastai.text.core.replace_maj>,
 <function fastai.text.core.lowercase>]

Build dataframe

Again neutral messages make up around 45% of our sample data. So guessing neutral alone would give us an accuracy of 45%. We can randomly drop neutral labelled rows so that neutral data makes up around 1/5 of the total data.

balanced = {'messages': [], 'sentiments':[]}

n_neutral = sum(1 for each in sentiments if each == 2)
N_examples = len(sentiments)
keep_prob = (N_examples - n_neutral)/4/n_neutral
print(f'keep_prob: {keep_prob}')

for idx, sentiment in enumerate(sentiments):
    message = messages_clean[idx]
    if sentiment != 2 or random.random() < keep_prob:
        balanced['messages'].append(message)
        balanced['sentiments'].append(sentiment)

keep_prob: 0.3016022730997995

Check that neutral data is balanced

n_neutral = sum(1 for each in balanced['sentiments'] if each == 2)
N_examples = len(balanced['sentiments'])
n_neutral/N_examples

0.19986973504599923

data = {'messages': balanced['messages'], 'sentiments': balanced['sentiments']}

df = pd.DataFrame(data); df.head()

Remove blank messages

df = df[df['messages']!='']

df.head()

df.shape

(1500631, 2)

Self supervized language model

The first step of the ULMFit approach is to build a language model. A model that will predict the next word given the previous words. This is self supervized learning. So in a batch we have a text buffer as input and then the same buffer plus the next word as a target.

Text data loader

dls = TextDataLoaders.from_df(df, text_col='messages', label_col='sentiments', is_lm=True)
dls.show_batch(max_n=3)

Look at the data

dls.show_batch(max_n=3)

Create language learner

With this data we can now fine tune the language model. We will be using a recurrent neural network (RNN) using an architecture called AWD-LSTM

learn = language_model_learner(
    dls, AWD_LSTM, drop_mult=0.3, 
    metrics=[accuracy, Perplexity()]).to_fp16()

Fine-Tuning the Language Model

Fit_one_cycle calls freeze when using a pretrained model so we are only training embeddings for words that are in our Stocktwits vocab, but aren't in the pretrained model vocab.

learn.fit_one_cycle(1, 2e-2)

Continue fine-tuning the model after unfreezing

learn.unfreeze()
learn.fit_one_cycle(10, 2e-3)

Display the RNN layers

learn.summary()

SequentialRNN (Input shape: ['64 x 72'])
================================================================
Layer (type)         Output Shape         Param #    Trainable 
================================================================
RNNDropout           64 x 72 x 400        0          False     
________________________________________________________________
RNNDropout           64 x 72 x 1152       0          False     
________________________________________________________________
RNNDropout           64 x 72 x 1152       0          False     
________________________________________________________________
Linear               64 x 72 x 39784      15,953,384 True      
________________________________________________________________
RNNDropout           64 x 72 x 400        0          False     
________________________________________________________________

Total params: 15,953,384
Total trainable params: 15,953,384
Total non-trainable params: 0

Optimizer used: <function Adam at 0x7fc7a15836a8>
Loss function: FlattenedLoss of CrossEntropyLoss()

Model frozen up to parameter group #3

Callbacks:
  - ModelResetter
  - RNNRegularizer
  - ModelToHalf
  - TrainEvalCallback
  - Recorder
  - ProgressCallback
  - MixedPrecision

Test our basic language model

We now have a language model originally trained on Wikipedia data trained with Stocktwits data. The goal is to predict the sentiment of messages but lets look at what this basic language model has learned

TEXT = "Earnings were above expectations. This stock should be trending up."
N_WORDS = 40
N_SENTENCES = 2
preds = [learn.predict(TEXT, N_WORDS, temperature=0.75) 
         for _ in range(N_SENTENCES)]

print("\n".join(preds))

Earnings were above expectations xxunk This stock should be trending up xxunk Update Aug Calls Up to per contract since alerted on Jul days to expire Some of todays top open interest changes Update Sep Puts Up sincealerted on
Earnings were above expectations xxunk This stock should be trending up xxunk Lot of Buying New Insider Filing On THICC JOHN days Transaction Code Both the short term and long term trends are positive This is a very

Classifier for Stocktwits sentiments

We now train our model with the sentiment label using our language model as a starting point.

Create the data loader with sentiment labels

dls_clas = TextDataLoaders.from_df(df, text_col='messages', label_col='sentiments', is_lm=False)
dls_clas.show_batch(max_n=3)

Create the learner to classify our "Twits"

learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5, 
                                metrics=accuracy).to_fp16()

Fine tuning the classifier

We now train our model with discriminative learning rates and gradual unfreezing. For NLP classifiers, the fastai library authors found that unfreezing a few layers at a time makes a real difference

learn.fit_one_cycle(1, 2e-2)

learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2))

learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3))

learn.unfreeze()
learn.fit_one_cycle(10, slice(1e-3/(2.6**4),1e-3))

Model results comparison

Around 75% accuracy which is 3% more than our basic PyTorch model. This is slightly better than our straight PyTorch model. 75% accuracy on a 5 sentiment level classifier is quite impressive. The fastai implementation is also based on PyTorch so it makes sense that the results are relatively close. The straight PyTorch implementation does require more work.

Predictions with the ULMFit model

0: very negative
1: negative
2: neutral
3: positive
4: very positive

Very positive

learn.predict(preprocess('$AAPL doing my part. Just bought my first iPad'))

('4',
 tensor(4),
 tensor([    0.0003,     0.0176,     0.0395,     0.0209,     0.9217])

Very negative

learn.predict(preprocess('$AAPL historic reversal selloff on news , historic crash news'))

('0',
 tensor(0),
 tensor([    0.8399,     0.0051,     0.0703,     0.0847,     0.0000]))

Very Positive

learn_inf.predict(preprocess('$TSLA if you haven’t seen it yet, Tsla had a double bottom and has upward momentum. $410 is strong support and this might close the week above $440. Only positive news from now till Election Day'))

('4',
 tensor(4),
 tensor([    0.0001,     0.0003,     0.0320,     0.0357,     0.9319]))

Negative

learn.predict(preprocess('Sold my $AAPL 10/16 $120c this morning and MAN I&#39;m glad I&#39;m out of there!\n\nAlthough, this was one of my favorite trades,  holding through the volatility last week was ROUGH and I didn&#39;t want to be caught holding the bag (after the iPhone event)\n\nWhat&#39;s next? since $SPY  making a new high... Might looks for some shorts now 🐻🐻🐻'))

('1',
 tensor(1),
 tensor([    0.0524,     0.7317,     0.0754,     0.1390,     0.0015]))

Neutral

learn.predict(preprocess('$AAPL the news already leaked. Nothing new just a battery that last an extra hour and the 5G. They are going to launch 4 phones. Probably this time they will introduce the middle finger option to unlock the phone. To make the idiots happy that they brought something new. Lol'))

('2',
 tensor(2),
 tensor([    0.0001,     0.0303,     0.7729,     0.1964,     0.0002]))

epoch	train_loss	valid_loss	error_rate	time
0	0.713224	0.594963	0.272222	00:16
1	0.614766	0.535685	0.251389	00:15
2	0.540522	0.429647	0.209722	00:15
3	0.445284	0.406699	0.204167	00:15
4	0.368155	0.399672	0.181944	00:15
5	0.312639	0.393371	0.177778	00:15

	BBGID	Name	Offer Rate	Bid Rate	Last Rate	Short Momentum	Short Interest	ShortInterestNotional	ShortInterestPct	S3Float	S3SIPctFloat	IndicativeAvailability	Utilization	DaysToCover10Day	DaysToCover30Day	DaysToCover90Day	Crowded Score	Squeeze Risk	symbol
0	FLWS US	1-800-FLOWERS.COM	0.30	0.15	0.30	0.0	7146702.0	1.858143e+08	0.2628	34338966.0	0.2081	1010980.0	NaN	10.339198	10.152532	8.014981	72.5	82.5	FLWS
1	VCVCU US	10X Capital Venture Acquisition Corp...	0.83	0.53	0.30	0.0	22353.0	2.380595e+05	0.0011	19729035.0	0.0011	5906030.0	NaN	0.281057	0.064992	0.064992	5.0	5.0	VCVCU
2	TXG US	10X GENOMICS INC	0.30	0.15	0.30	0.0	960805.0	1.360500e+08	0.0157	62019955.0	0.0155	17396300.0	NaN	1.244221	1.140568	1.185829	10.0	10.0	TXG
3	YI US	111 INC	0.95	0.61	0.30	-4.3	59512.0	4.136084e+05	0.0013	46418902.0	0.0013	13848300.0	NaN	0.184862	0.379655	0.437930	5.0	5.0	YI
4	YSGG US	1399 Internet Technology Application...	3.58	2.53	3.75	1.5	398.0	2.208900e+02	0.0031	128398.0	0.0031	30322.0	NaN	3.289256	0.802419	1.922705	10.0	0.0	YSGG

	No.	Ticker	Company	Sector	Industry	Country	Market Cap	P/E	Shares Float	Float Short	Short Ratio	Average True Range	Volatility (Week)	Volatility (Month)	Relative Strength Index (14)	Average Volume	Price	Change	Volume
0	1	A	Agilent Technologies, Inc.	Healthcare	Diagnostics & Research	USA	47573.54	39.97	302.58	1.44%	2.78	4.13	2.22%	2.73%	55.64	1566.50	157.80	0.65%	1409105
1	2	AA	Alcoa Corporation	Basic Materials	Aluminum	USA	10870.60	13.82	184.14	5.28%	1.13	2.79	5.31%	5.45%	71.62	8625.14	59.36	0.08%	7509823
2	3	AAL	American Airlines Group Inc.	Industrials	Airlines	USA	11523.70	NaN	641.17	15.42%	2.84	0.92	5.21%	5.62%	51.22	34775.22	18.26	0.00%	32205886
3	4	AAP	Advance Auto Parts, Inc.	Consumer Cyclical	Specialty Retail	USA	14867.93	23.49	62.08	4.89%	4.77	5.68	1.94%	2.36%	50.53	636.40	232.13	-0.28%	492798
4	5	AAPL	Apple Inc.	Technology	Consumer Electronics	USA	2911047.36	31.44	16394.79	0.69%	1.25	4.74	1.91%	2.78%	62.88	89972.33	176.28	0.36%	68138164

	_seq	timestamp	muts	Business Date	Sedol	ISIN	Cusip	BBGID	Name	Offer Rate	Bid Rate	Last Rate	Short Interest	ShortInterestNotional	ShortInterestPct	S3Float	S3SIPctFloat	IndicativeAvailability	Utilization	DaysToCover10Day	DaysToCover30Day	DaysToCover90Day	Crowded Score	Squeeze Risk	symbol
4809929	13	2021-12-23 00:00:00.000	1640235600000000	12/23/2021	2444123	US68243Q1067	<NA>	FLWS US	1-800-FLOWERS.COM	0.3	0.15	0.44	4525039.0	1.007274e+08	0.1608	32668564.0	0.1385	9044225.0	NaN	4.17	6.81	8.86	52.5	32.5	FLWS
4809933	18	2021-12-23 00:00:00.000	1640235600000000	12/23/2021	BKS3RS7	US88025U1097	<NA>	TXG US	10X GENOMICS INC	0.3	0.15	0.30	3119063.0	4.563813e+08	0.0378	85584279.0	0.0364	32485436.0	NaN	5.04	3.79	4.62	25.0	35.0	TXG
4809947	37	2021-12-23 00:00:00.000	1640235600000000	12/23/2021	BKMG1N5	US68269G1076	<NA>	ONEM US	1LIFE HEALTHCARE INC	0.3	0.15	0.32	11572770.0	1.980101e+08	0.0817	153159369.0	0.0756	42193022.0	NaN	4.41	4.44	5.86	32.5	22.5	ONEM
4809953	48	2021-12-23 00:00:00.000	1640235600000000	12/23/2021	2341439	US3208671046	<NA>	FMBI US	1ST MIDWEST BANCORP	0.3	0.15	0.30	2078927.0	4.137065e+07	0.0184	114859045.0	0.0181	59276849.0	NaN	2.40	3.23	3.98	12.5	12.5	FMBI
4809969	73	2021-12-23 00:00:00.000	1640235600000000	12/23/2021	BMTXV88	US9013841070	<NA>	TSVT US	2seventy Bio Inc.	0.3	0.15	0.31	924742.0	2.906464e+07	0.0405	23784666.0	0.0389	8079652.0	NaN	1.46	1.77	1.75	12.5	22.5	TSVT

	symbol	ShortInterestPct	S3SIPctFloat	Crowded Score	Squeeze Risk
4812347	BFRI	0.8828	0.4689	80.0	100.0
4814459	CRTX	0.5510	0.3553	85.0	100.0
4818535	HRTX	0.3298	0.2480	80.0	100.0
4819384	ICPT	0.3583	0.2638	82.5	100.0
4824790	PRLD	0.3444	0.2562	82.5	100.0
4811269	ARCH	0.2729	0.2144	75.0	95.0
4817755	GOGO	0.3251	0.2453	82.5	92.5
4825582	RLAY	0.1957	0.1637	72.5	92.5
4820441	JAMF	0.2026	0.1685	60.0	90.0
4827953	TTCF	0.3294	0.2478	90.0	90.0

	Ticker	Shares Float	Float Short
150	BFRI	3.60	84.91%
299	CRTX	20.02	45.87%
687	LGVN	2.91	42.40%
1152	TTCF	41.42	39.05%
173	BLNK	36.34	34.35%
601	ICPT	24.38	34.23%
693	LMND	39.98	34.01%
201	BYND	56.50	33.96%
584	HRTX	101.77	30.89%
152	BGFV	20.86	27.91%
1002	SAVA	37.46	27.35%
94	ARCH	14.99	26.96%
360	DMTK	18.80	25.35%
492	FSR	157.79	24.81%
1079	STAR	68.68	23.72%
796	NKLA	218.81	23.41%
543	GTHX	38.72	23.33%
948	RAD	52.62	23.22%
973	RLAY	61.35	22.48%
1230	W	72.17	22.32%

	_seq	muts	Business Date	Sedol	ISIN	Cusip	BBGID	Name	Offer Rate	Bid Rate	Last Rate	Short Momentum	Short Interest	ShortInterestNotional	ShortInterestPct	S3Float	S3SIPctFloat	IndicativeAvailability	Utilization	DaysToCover10Day	DaysToCover30Day	DaysToCover90Day	Crowded Score	Squeeze Risk	symbol
Date
2021-01-04	5	1609736400000000	01/04/2021	2444123	US68243Q1067	68243Q106	FLWS US	1-800-FLOWERS.COM	0.3	0.15	0.3	0.0	7146702.0	1.858143e+08	0.2628	34338966.0	0.2081	1010980.0	NaN	10.339198	10.152532	8.014981	72.5	82.5	FLWS
2021-01-04	9	1609736400000000	01/04/2021	BKS3RS7	US88025U1097	88025U109	TXG US	10X GENOMICS INC	0.3	0.15	0.3	0.0	960805.0	1.360500e+08	0.0157	62019955.0	0.0155	17396300.0	NaN	1.244221	1.140568	1.185829	10.0	10.0	TXG
2021-01-04	25	1609736400000000	01/04/2021	BKMG1N5	US68269G1076	68269G107	ONEM US	1LIFE HEALTHCARE INC	0.3	0.15	0.3	0.0	8892859.0	3.881733e+08	0.1064	92461255.0	0.0962	16177700.0	NaN	10.620926	6.566751	5.781965	47.5	67.5	ONEM
2021-01-04	35	1609736400000000	01/04/2021	2341439	US3208671046	320867104	FMBI US	1ST MIDWEST BANCORP	0.3	0.15	0.3	0.0	3071098.0	4.889188e+07	0.0272	116034821.0	0.0265	30818000.0	NaN	3.148213	4.196131	4.725530	15.0	25.0	FMBI
2021-01-04	53	1609736400000000	01/04/2021	BKWBZZ0	US90214J1016	90214J101	TWOU US	2U INC	0.3	0.15	0.3	0.0	12178811.0	4.872742e+08	0.1741	82125089.0	0.1483	8719930.0	NaN	10.451798	11.460037	9.689524	65.0	65.0	TWOU

	squeeze_return
symbol
AMC	17
GME	11
BBBY	11
PLBY	9
SAVA	8
BLNK	8
BFRI	7
BGFV	7
UPST	6
TDUP	5
RIOT	5
FSR	4
BKKT	4
LGVN	4
DDD	4
PUBM	4
MQ	4
APPN	4
NTLA	3
ISIG	3

	squeeze_return
date
2021-01-27	6
2021-06-02	5
2021-10-28	4
2021-05-25	4
2021-01-25	4
2021-01-26	4
2021-03-10	3
2021-03-08	3
2021-10-27	3
2021-11-15	3
2021-01-07	3
2021-11-11	3
2021-01-22	3
2021-06-30	3
2021-07-13	2
2021-09-15	2
2021-04-05	2
2021-05-27	2
2021-11-01	2
2021-03-12	2

	count
Squeeze Risk
100.0	143
97.5	10
90.0	9
92.5	4
80.0	3
85.0	3
87.5	2
75.0	1
77.5	1
82.5	1
95.0	1

	squeeze_return
Squeeze Risk
75.0	0.485000
77.5	0.133000
80.0	0.208000
82.5	0.135000
85.0	0.170333
87.5	0.144000
90.0	0.138222
92.5	0.165500
95.0	0.143000
97.5	0.145200
100.0	0.180797

	Open	prev_close	prev_vwap	Offer Rate	Bid Rate	Last Rate	Short Momentum	Short Interest	ShortInterestNotional	ShortInterestPct	S3Float	S3SIPctFloat	IndicativeAvailability	DaysToCover10Day	DaysToCover30Day	DaysToCover90Day	Crowded Score	Squeeze Risk
0	21.160000	21.190001	21.149408	11.33	8.54	13.88	0.00	9575717.6	2.029095e+08	0.4609	30351942.6	0.3155	0.0	7.460554	6.776367	7.998541	90.0	100.0
1	17.059999	17.120001	17.185627	13.58	10.40	13.75	2.96	19792992.7	3.388560e+08	0.2857	89083420.7	0.2222	0.0	7.981142	9.032790	7.064581	92.5	100.0
2	15.450000	15.380000	14.997252	3.33	2.40	2.02	4.50	1038111.0	1.596615e+07	0.2083	6021385.0	0.1724	10563000.0	0.887518	1.354603	1.632401	57.5	100.0
3	114.690002	113.360001	114.886139	58.83	47.00	58.02	1.26	8307969.5	9.417914e+08	0.3323	33309156.5	0.2494	0.0	1.885515	2.233313	4.117996	87.5	100.0
4	41.599998	41.880001	41.894508	0.30	0.15	0.30	0.00	5843457.4	2.447240e+08	0.2678	27664096.4	0.2112	39771.0	14.457705	18.382069	18.579619	80.0	100.0

	prev_close	prev_vwap	Offer Rate	Bid Rate	Last Rate	Short Momentum	Short Interest	ShortInterestNotional	ShortInterestPct	S3Float	S3SIPctFloat	IndicativeAvailability	DaysToCover10Day	DaysToCover30Day	DaysToCover90Day	Crowded Score	Squeeze Risk
0	1.001418	0.999499	11.33	8.54	13.88	0.00	9575717.6	2.029095e+08	46.09	30351942.6	31.55	0.0	7.460554	6.776367	7.998541	90.0	100.0
1	1.003517	1.007364	13.58	10.40	13.75	2.96	19792992.7	3.388560e+08	28.57	89083420.7	22.22	0.0	7.981142	9.032790	7.064581	92.5	100.0
2	0.995469	0.970696	3.33	2.40	2.02	4.50	1038111.0	1.596615e+07	20.83	6021385.0	17.24	10563000.0	0.887518	1.354603	1.632401	57.5	100.0
3	0.988404	1.001710	58.83	47.00	58.02	1.26	8307969.5	9.417914e+08	33.23	33309156.5	24.94	0.0	1.885515	2.233313	4.117996	87.5	100.0
4	1.006731	1.007080	0.30	0.15	0.30	0.00	5843457.4	2.447240e+08	26.78	27664096.4	21.12	39771.0	14.457705	18.382069	18.579619	80.0	100.0

	S3SIPctFloat	ShortInterestPct
S3SIPctFloat	1.000000	0.986361
ShortInterestPct	0.986361	1.000000

	messages	sentiments
0	great buy at ill wait	4
1		3
2	STAAnalystAlert for Jefferies Maintains with a rating of Hold setting target price at USD Our own verdict is Buy	4
3	I heard there s a guy who knows someone who thinks somebody knows something on StockTwits	3
4	reveal yourself	2

	text	text_
0	xxbos funny how these bears comes out today where were you yesterday when i was all alone getting flamed by these young bulls xxbos holey moley batman xxbos holding my waiting for kiki to tell me she loves me i mean jcpenney xxbos xxmaj china s xxmaj zhoushan city woos xxmaj exxon xxmaj mobil for a billion ethylene plant xxbos is max pain xxbos they are overselling again bring that xxup rsi	funny how these bears comes out today where were you yesterday when i was all alone getting flamed by these young bulls xxbos holey moley batman xxbos holding my waiting for kiki to tell me she loves me i mean jcpenney xxbos xxmaj china s xxmaj zhoushan city woos xxmaj exxon xxmaj mobil for a billion ethylene plant xxbos is max pain xxbos they are overselling again bring that xxup rsi down
1	xxmaj volatility is expensive to forecast xxbos this is looking good xxbos wth xxbos about to break the daily high xxbos xxmaj all top ranks are xxup nvidia no others brands xxmaj see bounds back to xxbos people can run their fingers on the keyboard xxmaj xxunk and watch the chart you could learn a thing or two xxmaj now let s see premarket xxbos xxmaj update xxmaj aug xxmaj puts xxmaj	volatility is expensive to forecast xxbos this is looking good xxbos wth xxbos about to break the daily high xxbos xxmaj all top ranks are xxup nvidia no others brands xxmaj see bounds back to xxbos people can run their fingers on the keyboard xxmaj xxunk and watch the chart you could learn a thing or two xxmaj now let s see premarket xxbos xxmaj update xxmaj aug xxmaj puts xxmaj up
2	unitedhealth xxmaj group s xxup pt raised by xxmaj raymond xxmaj james to strong buy rating xxbos xxmaj great summary of xxmaj boyar xxmaj value xxmaj group s xxmaj letter stay clear of xxup faang xxbos xxmaj absolutely ridiculous xxbos pump fake xxbos waiting to head to xxbos xxmaj it amuses me when bears here think they control the price with xxup st messages xxbos if i m wrong i m wrong	xxmaj group s xxup pt raised by xxmaj raymond xxmaj james to strong buy rating xxbos xxmaj great summary of xxmaj boyar xxmaj value xxmaj group s xxmaj letter stay clear of xxup faang xxbos xxmaj absolutely ridiculous xxbos pump fake xxbos waiting to head to xxbos xxmaj it amuses me when bears here think they control the price with xxup st messages xxbos if i m wrong i m wrong i

epoch	train_loss	valid_loss	accuracy	perplexity	time
0	3.667701	3.579307	0.391628	35.848701	22:17
1	3.556901	3.493224	0.401178	32.891827	22:18
2	3.501796	3.440667	0.407269	31.207756	22:08
3	3.429217	3.405697	0.411324	30.135296	21:58
4	3.346377	3.383271	0.414011	29.466999	22:09
5	3.297989	3.370040	0.416521	29.079689	22:22
6	3.222261	3.362387	0.418393	28.858006	22:10
7	3.179726	3.362380	0.419272	28.857779	22:16
8	3.155103	3.368545	0.419497	29.036251	22:22
9	3.091218	3.374530	0.419274	29.210562	22:23

	text	category
0	xxbos i xxup want xxup him xxup to xxup feel xxup max xxup pain xxup the xxup rest xxup ill xxup play xxup by xxup play xxup it xxup same xxup as xxup amd xxup at xxup exp xxup why xxup it xxup had xxup to xxup hit xxup then xxup it xxup did xxup lulu xxup wk b xxup ibm b	0
1	xxbos xxup the xxup xxunk xxup fucks xxup are xxup waiting xxup for xxup mu xxup to xxup go xxup below xxup to xxup buy xxup it xxup back xxup lets xxup just xxup all xxup max xxup out xxup our xxup credit xxup cards xxup and xxup buy xxup more xxup mu xxup at xxup this xxup low xxpad xxpad xxpad	4
2	xxbos lookslike xxmaj about xxup mm s xxmaj trying xxmaj to xxmaj hold xxup bid xxmaj in xxmaj low s xxmaj but xxup mm xxmaj forcing xxmaj trading xxmaj in xxmaj low s xxup jpm xxmaj tusa lol xxmaj glta xxmaj bulls xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad	4

epoch	train_loss	valid_loss	accuracy	time
0	0.862675	0.810372	0.678238	11:56
1	0.831596	0.728818	0.714281	11:56
2	0.751489	0.685342	0.732929	11:56
3	0.694547	0.662922	0.742413	12:00
4	0.706285	0.652665	0.747716	12:01
5	0.665387	0.639441	0.750803	12:02
6	0.658407	0.631364	0.753900	11:57
7	0.673705	0.632880	0.753232	11:50
8	0.626122	0.632878	0.754316	11:56
9	0.634495	0.630918	0.754664	12:02

AI quant musings

Breast Cancer CT Scan dataset with fastai

We have 3600 Breast CT scans size 384 x 256 with labels and a mask

Lets look at an image

And the corresponding mask

DataBlock

S3 partners short interest dataset initial analysis

S3 partners short interest

S3 Data Exploratory Data Analysis

xgboost model

Confusion Matrix

Feature importance

Conclusion

fastai ULMFit versus a basic NLP Pytorch classification model for StockTwits messages

Introduction

PyTorch NLP model

Load Twits Data

Length of data

Preprocessing the Data

Preprocess All the Twits

Bag of Words

Frequency of Words Appearing in Message

Remove Filtered Words from Vocabulary

Balancing the classes

Neural Network

Embed -> RNN -> Dense -> Softmax

View Model

Training

DataLoaders and Batching

Training and Validation

Train model

ULMFit approach

Import Twits

Preprocessing the Data

Build dataframe

Self supervized language model

Text data loader

Look at the data

Create language learner

Fine-Tuning the Language Model

Continue fine-tuning the model after unfreezing

Display the RNN layers

Test our basic language model

Classifier for Stocktwits sentiments

Create the data loader with sentiment labels

Create the learner to classify our "Twits"

Fine tuning the classifier

Model results comparison

Predictions with the ULMFit model

Very positive

Very negative

Very Positive

Negative

Neutral