TODOs:
- [x] exclude modules w/o params
- [ ] add layerwise splitter for Transfomers
default_splitter[source]
default_splitter(model)
def layerwise_splitter(model):
raise NotImplementedError('use default_splitter for now')
device = torch.device('cuda:0')
d = {'a':tensor([1,2,3])}
d_cuda = to_device(d)
assert d_cuda['a'].device == device
class TransCallback[source]
TransCallback(model) ::Callback
Handles HuggingFace model inputs and outputs
class GeneratePreds[source]
GeneratePreds(input_ids:Optional[LongTensor]=None,max_length:Optional[int]=None,min_length:Optional[int]=None,do_sample:Optional[bool]=None,early_stopping:Optional[bool]=None,num_beams:Optional[int]=None,temperature:Optional[float]=None,top_k:Optional[int]=None,top_p:Optional[float]=None,repetition_penalty:Optional[float]=None,bad_words_ids:Optional[Iterable[int]]=None,bos_token_id:Optional[int]=None,pad_token_id:Optional[int]=None,eos_token_id:Optional[int]=None,length_penalty:Optional[float]=None,no_repeat_ngram_size:Optional[int]=None,encoder_no_repeat_ngram_size:Optional[int]=None,num_return_sequences:Optional[int]=None,max_time:Optional[float]=None,decoder_start_token_id:Optional[int]=None,use_cache:Optional[bool]=None,num_beam_groups:Optional[int]=None,diversity_penalty:Optional[float]=None,prefix_allowed_tokens_fn:Optional[Callable[int,Tensor,List[int]]]=None,output_attentions:Optional[bool]=None,output_hidden_states:Optional[bool]=None,output_scores:Optional[bool]=None,return_dict_in_generate:Optional[bool]=None,forced_bos_token_id:Optional[int]=None,forced_eos_token_id:Optional[int]=None,remove_invalid_values:Optional[bool]=None,synced_gpus:Optional[bool]=None) ::Callback
Produces generated_tokens which can be used for metrics computation
class TransLearner[source]
TransLearner(dls,model:PreTrainedModel,predict_with_generate:bool=False,loss_func=None,opt_func=Adam,lr=0.001,splitter=trainable_params,cbs=None,metrics=None,path=None,model_dir='models',wd=None,wd_bn_bias=False,train_bn=True,moms=(0.95, 0.85, 0.95)) ::Learner
Learner for training transformers from HuggingFace
path = untar_data(URLs.IMDB_SAMPLE)
texts = pd.read_csv(path/'texts.csv')
model_name = 'google/electra-small-discriminator'
max_len = 128
bs = 8
val_bs = 16
tokenizer = AutoTokenizer.from_pretrained(model_name)
dblock = DataBlock(blocks = [TransformersTextBlock(tokenizer=tokenizer), CategoryBlock()],
get_x=ItemGetter('text'),
get_y=ItemGetter('label'),
splitter=ColSplitter())
dls = dblock.dataloaders(texts, bs=bs, val_bs=val_bs)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
learn = TransLearner(dls, model, metrics=accuracy).to_fp16()
learn.fit(2, 5e-5)
Some weights of the model checkpoint at google/electra-small-discriminator were not used when initializing ElectraForSequenceClassification: ['discriminator_predictions.dense.bias', 'discriminator_predictions.dense.weight', 'discriminator_predictions.dense_prediction.bias', 'discriminator_predictions.dense_prediction.weight'] - This IS expected if you are initializing ElectraForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing ElectraForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of ElectraForSequenceClassification were not initialized from the model checkpoint at google/electra-small-discriminator and are newly initialized: ['classifier.out_proj.bias', 'classifier.out_proj.weight', 'classifier.dense.bias', 'classifier.dense.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.597513 | 0.440168 | 0.840000 | 00:39 |
| 1 | 0.423275 | 0.334340 | 0.890000 | 00:39 |
learn.show_results()
| text | category | category_ | |
|---|---|---|---|
| 0 | the trouble with the book, " memoirs of a geisha " is that it had japanese surfaces but underneath the surfaces it was all an american man's way of thinking. reading the book is like watching a magnificent ballet with great music, sets, and costumes yet performed by barnyard animals dressed in those costumesso far from japanese ways of thinking were the characters. < br / > < br / > the movie isn't about japan or real geisha. it is a story about a few american men's mistaken ideas about japan and geisha filtered through their own ignorance and misconceptions. so what is this movie if it isn't about japan or geisha? is it pure fantasy as so many people have said? yes, but then why make it into an american fantasy? < br / > < br / > there were so many missed opportunities. imagine a culture | negative | negative |
| 1 | < br / > < br / > i'm sure things didn't exactly go the same way in the real life of homer hickam as they did in the film adaptation of his book, rocket boys, but the movie " october sky " ( an anagram of the book's title ) is good enough to stand alone. i have not read hickam's memoirs, but i am still able to enjoy and understand their film adaptation. the film, directed by joe johnston and written by lewis colick, records the story of teenager homer hickam ( jake gyllenhaal ), beginning in october of 1957. it opens with the sound of a radio broadcast, bringing news of the russian satellite sputnik, the first artificial satellite in orbit. we see a images of a blue - gray town and its people : mostly miners working for the olga coal company. one of the miners | positive | positive |
| 2 | how viewers react to this new " adaption " of shirley jackson's book, which was promoted as not being a remake of the original 1963 movie ( true enough ), will be based, i suspect, on the following : those who were big fans of either the book or original movie are not going to think much of this one... and those who have never been exposed to either, and who are big fans of hollywood's current trend towards " special effects " being the first and last word in how " good " a film is, are going to love it. < br / > < br / > things i did not like about this adaption : < br / > < br / > 1. it was not a true adaption of the book. from the articles i had read, this movie was supposed to cover other | negative | negative |
| 3 | to review this movie, i without any doubt would have to quote that memorable scene in tarantino's " pulp fiction " ( 1994 ) when jules and vincent are talking about mia wallace and what she does for a living. jules tells vincent that the " only thing she did worthwhile was pilot ". vincent asks " what the hell is a pilot? " and jules goes into a very well description of what a tv pilot is : " well, the way they make shows is, they make one show. that show's called a'pilot '. then they show that show to the people who make shows, and on the strength of that one show they decide if they're going to make more shows. some pilots get picked and become television programs. some don't, become nothing. she starred in one of the ones that became nothing. " now to stretch | negative | negative |
| 4 | bonanza had a great cast of wonderful actors. lorne greene, pernell whitaker, michael landon, dan blocker, and even guy williams ( as the cousin who was brought in for several episodes during 1964 to replace adam when he was leaving the series ). the cast had chemistry, and they seemed to genuinely like each other. that made many of their weakest stories work a lot better than they should have. it also made many of their best stories into great western drama. < br / > < br / > like any show that was shooting over thirty episodes every season, there are bound to be some weak ones. however, most of the time each episode had an interesting story, some kind of conflict, and a resolution that usually did not include violence. while bonanza was a western, the gunfighting was never featured as the main attraction. while i am | positive | positive |
| 5 | we saw the silent version of this film, and it is quite simply shimmeringly beautiful. it's quite hard to see how a sound version could have been created, since it is shot with pure silent technique, long wordless sweeps of narrative without a single intertitle - - save for a few disconcerting sequences where louise brooks, playing a french typist, is quite visibly speaking in english... the only section that obviously cries out for sound is the final scene, where brooks is watching the rushes for her test'for a sound film': footage which plays constantly in the background as the action unfolds, with her mouth moving in ceaseless soundless song. i was unsurprised to learn afterwards that this passage alone in the talkie version had been hailed as an exemplar of new technique! < br / > < br / > in the sunny beauty of its opening scenes and | positive | positive |
| 6 | i couldn't believe that this movie dates from 2007, it had all the looks of a below - average seventies horror - flick. didn't they have any knowledge of modern special effects or cgi?!? didn't they know that in the post - millennium the violence in a supposed horror - and / or scifi - movie should at least be a little bit graphic? or did i get the purpose wrong, was it supposed to be a deep and meaningful story of man and animal, bound together in the big cycle of life, or a warning to mankind not to mess with nature, or something like that?? it doesn't really matter, either way it turned out wrong and to me this movie failed on all accounts. < br / > < br / > first of all : the premise is very improbable. if at a given time you're capable | negative | negative |
| 7 | for me the only reason for having a look at this remake was to see how bad and funny it could be. there was no doubt about it being funny and bad, because i had seen " voyna i mir " ( 1968 ). shall we begin? here we go... < br / > < br / > robert dornhelm & brendan donnison's pierre bezukhov - a lean fellow that lacks the depth of the original ; robert dornhelm & brendan donnison's natasha rostova - a scarecrow, her image can cause insomnia ; robert dornhelm & brendan donnison's andrej bolkonsky - an ok incarnation which, like the lean fellow ( cf. above ), lacks depth of a russian soul and " struggle within " ; robert dornhelm & brendan donnison's napoleon - a rather unimpressive leader ; robert dornhelm & brendan donnison's prince bolkonsky - a turd with an english | negative | negative |
| 8 | i don't usually write a comment when there are so many others but this time i feel i have to. i have spoken of taste in another review, saying it's all in the eye of the beholder but when it comes to this film, if you like it, it simply means you have bad taste. < br / > < br / > i love films. i loved " isle of the dead " which is pretty much an unknown b & w film. i even liked " scream " and " scary movie " i liked these films because they have, if not a lot, at least something good about them. i appreciate 99. 9 % of the films i've seen because they tell a story which i haven't heard before, and most directors only make films with a good storyline. throughout this film i was thinking " where | negative | negative |