Setup

ds_name = "swag"
model_name = "google/electra-base-discriminator"

max_len = 512
bs = 16
val_bs = bs*2

lr = 3e-5
ds = load_dataset(ds_name, "regular")
# train_ds = load_dataset(ds_name, "regular", split='validation')

Dataloaders

splits = get_splits(ds)
ds = concatenate_datasets([ds['train'], ds['validation']])
ds[0]
{'ending0': 'passes by walking down the street playing their instruments.',
 'ending1': 'has heard approaching them.',
 'ending2': "arrives and they're outside dancing and asleep.",
 'ending3': 'turns the lead singer watches the performance.',
 'fold-ind': '3416',
 'gold-source': 'gold',
 'label': 0,
 'sent1': 'Members of the procession walk down the street holding small horn brass instruments.',
 'sent2': 'A drum line',
 'startphrase': 'Members of the procession walk down the street holding small horn brass instruments. A drum line',
 'video-id': 'anetv_jkn6uvmqwh4'}
text_fields = [
    'sent1',
    'sent2',
    'ending0',
    'ending1',
    'ending2',
    'ending3'
    ]
dblock = DataBlock(
    blocks = [MultiChoiceBlock(text_fields[:2], text_fields[2:], pretrained_model_name=model_name),
              CategoryBlock()],
    get_x=KeyGetter(text_fields),
    get_y=ItemGetter('label'),
    splitter=IndexSplitter(splits[1]))
%%time
dls = dblock.dataloaders(ds, bs=bs, val_bs=val_bs, num_workes=2)
CPU times: user 1min 1s, sys: 2.41 s, total: 1min 3s
Wall time: 1min 2s

The inputs are shape as bs x n_choices x seq_len:

b = dls.one_batch()
b[0]['input_ids'].shape
torch.Size([16, 4, 65])
dls.show_batch(max_n=4)
sent1 ending0 ending1 ending2 ending3 category
0 members of the procession walk down the street holding small horn brass instruments. a drum line passes by walking down the street playing their instruments. a drum line has heard approaching them. a drum line arrives and they're outside dancing and asleep. a drum line turns the lead singer watches the performance. 0
1 a chain is wrapped around a tree stump. a man looks at him on the ground. a man uses the tool to wrap the ink off the tree. a man is lifted into the tree. a man holds the roof of the terraced house. 0
2 someone appears to remind someone of her appointment. someone stays behind for another moment. someone leans down and places his hand on her lower back. someone pitches a billy bag with a pie. someone shoots shafts of light into someone's face. 0
3 father and daughter go into their flat. the daughter sprays buckets on her head, slides tension into the upper corner of the apartment. the daughter feeds him phone. the daughter subdues a smile at her feet. the daughter peeps back out and waves. 3
wandb.init(reinit=True, project="fasthugs", entity="fastai_community",
           name=WANDB_NAME, group=GROUP, notes=NOTES, tags=TAGS, config=CONFIG);

Training

model = AutoModelForMultipleChoice.from_pretrained(model_name)
learn = TransLearner(dls, model, metrics=accuracy)

The final layers of the model are initialized with random weights, we can varufy that the performance is as good as random choice:

learn.validate()
(#2) [1.3858414888381958,0.2647205889225006]
learn.fit_one_cycle(2, lr, wd=0.01, cbs=WandbCallback(log_preds=False, log_model=False))
Could not gather input dimensions
epoch train_loss valid_loss accuracy time
0 0.605681 0.375084 0.857193 52:49
1 0.401334 0.316267 0.882435 52:49
learn.show_results()
sent1 ending0 ending1 ending2 ending3 category category_
0 students lower their eyes nervously. she pats her shoulder, then saunters toward someone. she turns with two students. she walks slowly towards someone. she wheels around as her dog thunders out. 2 2
1 as somone charges, he delivers a mighty upward blow. spraying blood gushes through the giant. spraying blood, he lies on a floor that bears the silverware and treatment. spraying blood soaks half his face. spraying blood from someone's leg, someone swings him back down. 2 3
2 the player runs past the first base. the player then serves the ball with the invite. the player then comes back to the first base and puts his foot on it. the player then shows on it for a second and then throws it on the field. the player then hits one and turns back with the shield. 1 1
3 when they are done, they hold hands as the crowd cheers. they continue drumming with each other while people cheer for the match. they walk over to the woman in a green dress, hugging her before leaving the stage. they look into the dance screen and continue to play. they do a routine on the platform. 1 1
4 the woman hold the center of the plastic. the woman positions the carpet with a squeegee. the woman puts the woman's fingers in the cutting pin, and the woman puts the black dot on the finger. the woman uses a piece of steel on the rope. the woman tied the ribbon on the wrapper and cut the excess ribbon. 3 3
5 he runs fast fast down the strip and quickly leaps into the sand. then he runs all over the room and runs, almost rolling one foot after another. then he runs off the road and getting excited. then he follows current movements before leaving only the other corporate breathless. then he goes back to start over and does it a few more times. 3 3
6 someone follows someone into the leeds dressing room. someone notes on an just nearby cabinet feature, its shadows at its sides. someone takes a note from his blazer pocket. someone smiles, who stares forward, horrified. someone crosses to a small waiting cafe where its left door stands. 1 1
7 once she is on, she begins kicking her legs up and down and doing several flips along the bar. for her final trick, she flips off the bar and her foot goes further than expected but she steps into the ropes. for her final trick, she flips off the bar and her foot goes further than expected but she barely jumps up into the acetone net. for her final trick, she flips off the bar and her foot goes further than expected but she quickly gathers herself and sticks her landing. for her final trick, she flips off the bar and her foot goes further than expected but she flops on a board multiple times. 2 2
8 the club owner affords a slight nod. now someone rubs her hands over her head and body. now someone draws a white frozen mirror. now someone opens a stairwell as someone shuffles layering creamy fluid. now someone pulls a sausage out of a large bottle on the stairs. 0 0