from transformers import AutoModelForSequenceClassification
from fastai.text.all import *
from fastai.callback.wandb import *

from fasthugs.learner import TransLearner, default_splitter
from fasthugs.data import TransformersTextBlock

def read_text(fn):
    return open(fn).read()

path = untar_data(URLs.IMDB)
path.ls()

(#7) [Path('/root/.fastai/data/imdb/README'),Path('/root/.fastai/data/imdb/imdb.vocab'),Path('/root/.fastai/data/imdb/test'),Path('/root/.fastai/data/imdb/tmp_clas'),Path('/root/.fastai/data/imdb/tmp_lm'),Path('/root/.fastai/data/imdb/train'),Path('/root/.fastai/data/imdb/unsup')]

Setup

Let's define main settings for the run in one place:

ds_name = 'imdb'
model_name = "distilroberta-base"

max_len = 512
bs = 8
val_bs = bs*2

lr = 2e-5

Dataloaders

dblock = DataBlock(
    blocks = [TransformersTextBlock(pretrained_model_name=model_name),
              CategoryBlock()],
    get_items=get_text_files,
    get_x=read_text,
    get_y=parent_label,
    splitter=GrandparentSplitter(valid_name='test'))

%%time
dls = dblock.dataloaders(path, bs=bs, val_bs=val_bs)

CPU times: user 34.1 s, sys: 2.21 s, total: 36.3 s
Wall time: 36.4 s

dls.show_batch(max_n=4)

	text	category
0	Match 1: Tag Team Table Match Bubba Ray and Spike Dudley vs Eddie Guerrero and Chris Benoit Bubba Ray and Spike Dudley started things off with a Tag Team Table Match against Eddie Guerrero and Chris Benoit. According to the rules of the match, both opponents have to go through tables in order to get the win. Benoit and Guerrero heated up early on by taking turns hammering first Spike and then Bubba Ray. A German suplex by Benoit to Bubba took the wind out of the Dudley brother. Spike tried to help his brother, but the referee restrained him while Benoit and Guerrero ganged up on him in the corner. With Benoit stomping away on Bubba, Guerrero set up a table outside. Spike dashed into the ring and somersaulted over the top rope onto Guerrero on the outside! After recovering and taking care of Spike, Guerrero slipped a table	pos
1	Now that Che(2008) has finished its relatively short Australian cinema run (extremely limited release:1 screen in Sydney, after 6wks), I can guiltlessly join both hosts of "At The Movies" in taking Steven Soderbergh to task.<br /><br />It's usually satisfying to watch a film director change his style/subject, but Soderbergh's most recent stinker, The Girlfriend Experience(2009), was also missing a story, so narrative (and editing?) seem to suddenly be Soderbergh's main challenge. Strange, after 20-odd years in the business. He was probably never much good at narrative, just hid it well inside "edgy" projects.<br /><br />None of this excuses him this present, almost diabolical failure. As David Stratton warns, "two parts of Che don't (even) make a whole". <br /><br />Epic biopic in name only, Che(2008) barely qualifies as a feature film! It certainly has no legs, inasmuch as except for its uncharacteristic ultimate resolution forced upon it by history, Soderbergh's	neg
2	I did not know for some time in my youth all that could in general be known about this film however the ways of making a film was not what in fact drew my attention, what made this motion picture one the most liked films even to this very day that I have ever seen was of the Heroism,bravery and the Honor to have served in Her Majestys Service.This film is not always what it seems and that is perhaps as it should be,however I cant say enough for the courage exhibited by Sgt.Cutter in defense of The Uniform that he too would of sacrificed his life to save from peril of the sort that they and the troop were threatened with the emergence of this thugee group.<br /><br />To be certain Sgt. Cutter is the kind of individual you might suggest something about and then you watch this unequivocal	pos
3	On Sunday July 27, 1997, the first episode of a new science fiction series called "Stargate SG-1" was broadcast on Showtime. A spin-off of and sequel to the 1994 film "Stargate" starring Kurt Russell and James Spader, the series begins approximately one year after the events portrayed in the film. For ten seasons, it chronicled the adventures and misadventures of an intrepid team of explorers known as SG-1. Originally, the series starred Richard Dean Anderson as Colonel Jack O'Neill (two "l"s!), Michael Shanks as Dr. Daniel Jackson, Amanda Tapping as Captain Samantha Carter, Christopher Judge as Teal'c and Don S. Davis as Major General George S. Hammond. For ten long years, we watched the team battle against the Goa'uld, the Replicators, the Ori and many other aggressors. At the same time, they forged alliances with the Asgard, the Tok'ra, the rebel Jaffa, the Nox and the Tollan. They saved the	pos

Tracking with W&B

Here comes some details on w&b tracking and the leaderboard to be established...

import wandb

WANDB_NAME = f'{ds_name}-{model_name}'
GROUP = f'{ds_name}-{model_name}-simple-{lr:.0e}'
NOTES = f'finetuning {model_name} with RAdam lr={lr:.0e}'
CONFIG = {}
TAGS =[model_name, ds_name, 'radam']

wandb.init(reinit=True, project="fasthugs", entity="fastai_community",
           name=WANDB_NAME, group=GROUP, notes=NOTES, tags=TAGS, config=CONFIG);

Training

model = AutoModelForSequenceClassification.from_pretrained(model_name)
learn = TransLearner(dls, model, metrics=accuracy).to_fp16()

Some weights of the model checkpoint at distilroberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at distilroberta-base and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

learn.fit_one_cycle(4, lr, cbs=[WandbCallback(log_preds=False, log_model=False), SaveModelCallback(monitor='accuracy')])

Could not gather input dimensions

epoch	train_loss	valid_loss	accuracy	time
0	0.204557	0.181656	0.929080	13:16
1	0.135426	0.169763	0.937880	13:19
2	0.078452	0.178007	0.941720	13:20
3	0.030883	0.201435	0.945680	13:18

Better model found at epoch 0 with accuracy value: 0.9290800094604492.
Better model found at epoch 1 with accuracy value: 0.9378799796104431.
Better model found at epoch 2 with accuracy value: 0.9417200088500977.
Better model found at epoch 3 with accuracy value: 0.9456800222396851.

learn.show_results()

	text	category	category_
0	There's a sign on The Lost Highway that says:<br /><br />*MAJOR SPOILERS AHEAD*<br /><br />(but you already knew that, didn't you?)<br /><br />Since there's a great deal of people that apparently did not get the point of this movie, I'd like to contribute my interpretation of why the plot makes perfect sense. As others have pointed out, one single viewing of this movie is not sufficient. If you have the DVD of MD, you can "cheat" by looking at David Lynch's "Top 10 Hints to Unlocking MD" (but only upon second or third viewing, please.) ;)<br /><br />First of all, Mulholland Drive is downright brilliant. A masterpiece. This is the kind of movie that refuse to leave your head. Not often are the comments on the DVDs very accurate, but Vogue's "It gets inside your head and stays there" really hit the mark.<br /><br />David Lynch deserves praise for creating	pos	pos
1	Yeah, what did I expect? I thought this would be a film about young adults at their turning-point in life, something like "Sonnenallee" or "American Pie", which I liked a lot. I wanted to see a funny film, perhaps with an ironic look on idyllic Wuerzburg. And what did I get?<br /><br />Attention, spoilers ahead!<br /><br />This film starts with a lengthy dialogue which gives you a good hint of what will inevitably follow: more lengthy dialogues. Sometimes I thought Moritz Bleibtreu might have forgotten his text and trying to hide that fact by improvising and just repeating what he was saying before. But as I think of Bleibtreu as one of the better german actors, I believe that this effect really was intended. I think the author wanted to show how boring talking to close friends can be - especially when they are stoned. But really, I don't need	neg	neg
2	This was one of the most dishonest, meaningless, and non-peaceful of the films I have ever seen. The representation of the other, of the Israelis, was racist, backward, and unfair. For one, the song played on E.S' car radio when pulled up alongside a very right-wing Israeli driver was "I put a spell on you" by Natacha Atlas. The song's style is quite Arabic, but it was released on an Israeli compilation CD, and I have even heard it on the radio in Israel. Many Israeli songs (as well as architecture, foods, and slang) are influenced by Arabic culture, and there is no reason an Israeli Jew would be offended or angered by a nearby car playing that song. The way E.S. appears so calm and collected with his sunglasses and cool glare, via a long, still shot, is meant to force the viewer into seeing the Jew as haggard	neg	neg
3	Pistol-packing Pam Grier takes names and kicks butt as the heroine in "Asylum of Satan" director William Girdler's entertaining blaxploitation actioneer "Sheba Baby," co-starring D'Urville Martin and Austin Stoker. "Sheba Baby" is one of several tough chick flicks that Grier appeared in during the 1970s, including "Coffy," "Foxy Brown," and "Friday Foster." The short-lived Girdler co-wrote this thoroughly routine private eye potboiler with producer David Shelton in one night and it features a headstrong female shamus that refuses to rely on a man to help her take care of business. Unfortunately, "Sheba Baby" isn't nearly as good as the blaxploitation movies that Grier made under the supervision of director Jack Hill. Hill helmed the African-American North Carolina native in "Coffy," "Foxy Brown," "The Big Bird Cage," and "The Big Doll House." Anybody that analyzes images of African-American women in cinema should be familiar with these epics. The chief problem with	pos	pos
4	Once again I must play something of the contrarian. Most of the reviews for Ab Tak Chappan have been extremely positive. Mine is positive, but only slightly. A 7 out of 10 is equivalent to a "C" letter grade from me.<br /><br />It seems that a lot of the praise is rooted in two factors: One, that Ab Tak Chappan is more realistic than the typical Bollywood film, and two, that it is trying to do things differently.<br /><br />The first point I couldn't care less about. I'm not looking for realism in films, and so I do not score higher for a film that shows a story and characters closer to how I believe the real world to be--I'm a big fan of surrealism, fantasy, absurdism, and so on, although I do not dislike realist films merely for the fact that they're realist.<br /><br />For the second point, I	pos	pos
5	Master of Italian horror Dario Argento is called a lot of bad things by non-fans. And is deserving of absolutely none of the backlash. In fact, every time I hear something bad about Argento- I think they're really talking about Michele Soavi. He just doesn't get the same amount of attention because his films were never as successful in theaters. In fact, his best film - 1994's Cemetery Man - was probably his least successful. Or just didn't get the attention he felt it deserved, because after that, he left film and went into directing television. He's never gone back. So people really don't know how inferior his other films are because by the time they've seen them, they're already fans of the Italian horror aesthetic. Which means you have to accept the fact that they make almost zero sense and are usually very unattractive films. This is where The	neg	pos
6	Wallace and Gromit are the main characters in some of the best cartoons ever crafted. The excellent mix of visual humor and claymation makes "A Grand Day Out," "The Wrong Trousers," and also "A Close Shave" some of the best animated footage ever put on television. Winning several Oscars and also countless other awards, Nick Park became quite the popular man in the U.K., yet his impact on the United States has not been big. After the third Wallace and Gromit short, there was all this speculation about a full-length Wallace and Gromit movie, yet for years nothing had happened. Then in 2000 instead of a full-length Wallace and Gromit film, we get another brilliant claymation film from Nick Park, which was Chicken Run, which almost got nominated for best picture in the Academy Awards. Perhaps it was the success of this film that ultimately drove Park to finally work	pos	pos
7	It's almost impossible for me to sit down and write a conscientious review of THREE COLORS: RED without letting people in on some of the ideas that Krysztoff Kieszlowski has explored in the previous two entries to this fascinating trilogy. The more I see them and think of them, and imagine myself in their world, the more I get its theme: that we are more linked to each other than we would want to think ourselves, and all it takes is a little hand of fate to set some events in motion. In BLUE, Juliette Binoche played a grieving widow whose plan to live her life without connections to the past had her meet someone unexpected. In WHITE, an act of cruelty spawns an unlikely friendship between two men who will, against the odds, conspire to bring the perpetrator to justice and full circle. And now, in RED, all the	pos	pos
8	A blaxploitation classic, this movie was terribly influential in rap music for the "toasts" that Rudy Ray Moore performs. Toasts are long rhyming stories that are funny and deliver a point, and you can see how they would naturally evolve into rap. For more on toasts, Rudy Ray Moore, and why this movie is important, go to Dolemite.com.<br /><br />Which leaves us just to talk about the movie itself. This movie packs in a great deal of "laugh-at-the-funny-outfits-and-hairstyles" bang for the buck, as nearly every shot has some sort of outrageous element or dialogue. It starts as Dolemite is being released from prison in order to find out who framed him and bring him to justice. I was unaware that prisons release people so they can prove their own innocence, but that's me, I'm a neophyte in the prison scene. He is helped in this by Queen Bee, who is	pos	neg