Very low accuracy

[ratom]

I trained CIFAR10 for 100 epochs but got accuracy around 86-88%. Again I trained for CIFAR100 the accuary of resnet50 is 58%. i think there is some issues in the code.

[DenseLance]

I noticed that for CIFAR100 if you were to stick to the original parameters (i.e. batch size, network architecture, loss function, and optimizer learning rate, weight decay and scheduler), overfitting of the model is very likely to occur. This is the output that I received when training with CIFAR100 data, and although the accuracy when using training data approaches 100%, but the accuracy when using testing data hovers around 40% (overfitting, and by a very large margin at that):

Loss [1, 100](epoch, minibatch):  9.65963740825653
Loss [1, 200](epoch, minibatch):  5.152797379493713
Loss [1, 300](epoch, minibatch):  4.661489753723145
Loss [2, 100](epoch, minibatch):  4.46427243232727
Loss [2, 200](epoch, minibatch):  4.324779586791992
Loss [2, 300](epoch, minibatch):  4.255856695175171
Loss [3, 100](epoch, minibatch):  4.171083827018737
Loss [3, 200](epoch, minibatch):  4.088866672515869
Loss [3, 300](epoch, minibatch):  4.03771208524704
Loss [4, 100](epoch, minibatch):  3.97960834980011
Loss [4, 200](epoch, minibatch):  3.8966946840286254
Loss [4, 300](epoch, minibatch):  3.843762333393097
Loss [5, 100](epoch, minibatch):  3.8005256271362304
Loss [5, 200](epoch, minibatch):  3.733882863521576
Loss [5, 300](epoch, minibatch):  3.6684217929840086
Loss [6, 100](epoch, minibatch):  3.638233540058136
Loss [6, 200](epoch, minibatch):  3.5818062448501586
Loss [6, 300](epoch, minibatch):  3.5721277141571046
Loss [7, 100](epoch, minibatch):  3.523845181465149
Loss [7, 200](epoch, minibatch):  3.468423018455505
Loss [7, 300](epoch, minibatch):  3.45445018529892
Loss [8, 100](epoch, minibatch):  3.4095366621017456
Loss [8, 200](epoch, minibatch):  3.382539267539978
Loss [8, 300](epoch, minibatch):  3.3518160343170167
Loss [9, 100](epoch, minibatch):  3.3134766006469727
Loss [9, 200](epoch, minibatch):  3.2694722747802736
Loss [9, 300](epoch, minibatch):  3.2539067530632018
Loss [10, 100](epoch, minibatch):  3.2150195121765135
Loss [10, 200](epoch, minibatch):  3.172513723373413
Loss [10, 300](epoch, minibatch):  3.1698724937438967
Accuracy on 50,000 train images:  23.79 %
Accuracy on 10,000 test images:  23.98 %
Loss [11, 100](epoch, minibatch):  3.111788070201874
Loss [11, 200](epoch, minibatch):  3.08822633266449
Loss [11, 300](epoch, minibatch):  3.086936044692993
Loss [12, 100](epoch, minibatch):  3.030677797794342
Loss [12, 200](epoch, minibatch):  3.041368067264557
Loss [12, 300](epoch, minibatch):  3.0079576206207275
Loss [13, 100](epoch, minibatch):  2.9415133023262023
Loss [13, 200](epoch, minibatch):  2.936082601547241
Loss [13, 300](epoch, minibatch):  2.930698275566101
Loss [14, 100](epoch, minibatch):  2.8895335507392885
Loss [14, 200](epoch, minibatch):  2.8928070712089538
Loss [14, 300](epoch, minibatch):  2.8562344455718995
Loss [15, 100](epoch, minibatch):  2.792374005317688
Loss [15, 200](epoch, minibatch):  2.7952922105789186
Loss [15, 300](epoch, minibatch):  2.78404337644577
Loss [16, 100](epoch, minibatch):  2.7345845246315004
Loss [16, 200](epoch, minibatch):  2.743603892326355
Loss [16, 300](epoch, minibatch):  2.734344184398651
Loss [17, 100](epoch, minibatch):  2.6538477873802186
Loss [17, 200](epoch, minibatch):  2.689882161617279
Loss [17, 300](epoch, minibatch):  2.7104456353187563
Loss [18, 100](epoch, minibatch):  2.5949219131469725
Loss [18, 200](epoch, minibatch):  2.585297725200653
Loss [18, 300](epoch, minibatch):  2.603741648197174
Loss [19, 100](epoch, minibatch):  2.5306129002571107
Loss [19, 200](epoch, minibatch):  2.552508237361908
Loss [19, 300](epoch, minibatch):  2.5481868815422057
Loss [20, 100](epoch, minibatch):  2.542559084892273
Loss [20, 200](epoch, minibatch):  2.514786593914032
Loss [20, 300](epoch, minibatch):  2.529491856098175
Accuracy on 50,000 train images:  37.204 %
Accuracy on 10,000 test images:  34.78 %
Loss [21, 100](epoch, minibatch):  2.391527605056763
Loss [21, 200](epoch, minibatch):  2.4492400741577147
Loss [21, 300](epoch, minibatch):  2.4479135918617247
Loss [22, 100](epoch, minibatch):  2.419568040370941
Loss [22, 200](epoch, minibatch):  2.402209030389786
Loss [22, 300](epoch, minibatch):  2.388232374191284
Loss [23, 100](epoch, minibatch):  2.2863242518901825
Loss [23, 200](epoch, minibatch):  2.331242425441742
Loss [23, 300](epoch, minibatch):  2.343509953022003
Loss [24, 100](epoch, minibatch):  2.220913870334625
Loss [24, 200](epoch, minibatch):  2.2798406672477722
Loss [24, 300](epoch, minibatch):  2.281049385070801
Loss [25, 100](epoch, minibatch):  2.2344158911705017
Loss [25, 200](epoch, minibatch):  2.4615078103542327
Loss [25, 300](epoch, minibatch):  2.312665514945984
Loss [26, 100](epoch, minibatch):  2.1725078797340394
Loss [26, 200](epoch, minibatch):  2.173547308444977
Loss [26, 300](epoch, minibatch):  2.263216755390167
Loss [27, 100](epoch, minibatch):  2.369027678966522
Loss [27, 200](epoch, minibatch):  2.2575604581832884
Loss [27, 300](epoch, minibatch):  2.25222852230072
Loss [28, 100](epoch, minibatch):  2.1128001129627227
Loss [28, 200](epoch, minibatch):  2.1400400364398955
Loss [28, 300](epoch, minibatch):  2.1222088396549226
Loss [29, 100](epoch, minibatch):  2.003260705471039
Loss [29, 200](epoch, minibatch):  2.0435080063343047
Loss [29, 300](epoch, minibatch):  2.082212600708008
Loss [30, 100](epoch, minibatch):  1.9611615526676178
Loss [30, 200](epoch, minibatch):  2.0490327322483064
Loss [30, 300](epoch, minibatch):  1.999650387763977
Accuracy on 50,000 train images:  50.104000000000006 %
Accuracy on 10,000 test images:  40.01 %
Loss [31, 100](epoch, minibatch):  1.8840069663524628
Loss [31, 200](epoch, minibatch):  1.9233769023418426
Loss [31, 300](epoch, minibatch):  1.9300939416885377
Loss [32, 100](epoch, minibatch):  1.8145395934581756
Loss [32, 200](epoch, minibatch):  1.866343650817871
Loss [32, 300](epoch, minibatch):  1.8943881750106812
Loss [33, 100](epoch, minibatch):  1.7512195098400116
Loss [33, 200](epoch, minibatch):  1.8067417454719543
Loss [33, 300](epoch, minibatch):  1.8578553473949433
Loss [34, 100](epoch, minibatch):  1.710550389289856
Loss [34, 200](epoch, minibatch):  1.7830115115642549
Loss [34, 300](epoch, minibatch):  1.8077159166336059
Loss [35, 100](epoch, minibatch):  1.680847693681717
Loss [35, 200](epoch, minibatch):  1.7217409110069275
Loss [35, 300](epoch, minibatch):  1.757295436859131
Loss [36, 100](epoch, minibatch):  1.6003015065193176
Loss [36, 200](epoch, minibatch):  1.6890592110157012
Loss [36, 300](epoch, minibatch):  1.6676046061515808
Loss [37, 100](epoch, minibatch):  1.5702935373783111
Loss [37, 200](epoch, minibatch):  1.6037746381759643
Loss [37, 300](epoch, minibatch):  1.6565000808238983
Loss [38, 100](epoch, minibatch):  1.4921616160869597
Loss [38, 200](epoch, minibatch):  1.5678565645217895
Loss [38, 300](epoch, minibatch):  1.6005572295188903
Loss [39, 100](epoch, minibatch):  1.4282839703559875
Loss [39, 200](epoch, minibatch):  1.5206191146373749
Loss [39, 300](epoch, minibatch):  1.579311866760254
Loss [40, 100](epoch, minibatch):  1.4037487483024598
Loss [40, 200](epoch, minibatch):  1.4998678135871888
Loss [40, 300](epoch, minibatch):  1.529771227836609
Accuracy on 50,000 train images:  61.873999999999995 %
Accuracy on 10,000 test images:  41.92 %
Loss [41, 100](epoch, minibatch):  1.3598134469985963
Loss [41, 200](epoch, minibatch):  1.4289080786705017
Loss [41, 300](epoch, minibatch):  1.4801931047439576
Loss [42, 100](epoch, minibatch):  1.3017448097467423
Loss [42, 200](epoch, minibatch):  1.3791745674610139
Loss [42, 300](epoch, minibatch):  1.4164017021656037
Loss [43, 100](epoch, minibatch):  1.2288037472963333
Loss [43, 200](epoch, minibatch):  1.3382691752910614
Loss [43, 300](epoch, minibatch):  1.3812544226646424
Loss [44, 100](epoch, minibatch):  1.2031632536649703
Loss [44, 200](epoch, minibatch):  1.279058056473732
Loss [44, 300](epoch, minibatch):  1.359627227783203
Loss [45, 100](epoch, minibatch):  1.1387980622053147
Loss [45, 200](epoch, minibatch):  1.2400834155082703
Loss [45, 300](epoch, minibatch):  1.3010744595527648
Loss [46, 100](epoch, minibatch):  1.102361086010933
Loss [46, 200](epoch, minibatch):  1.185320331454277
Loss [46, 300](epoch, minibatch):  1.2688620758056641
Loss [47, 100](epoch, minibatch):  1.0469347548484802
Loss [47, 200](epoch, minibatch):  1.1268413329124451
Loss [47, 300](epoch, minibatch):  1.2060655468702317
Loss [48, 100](epoch, minibatch):  0.9994390034675598
Loss [48, 200](epoch, minibatch):  1.120630054473877
Loss [48, 300](epoch, minibatch):  1.1641131913661957
Loss [49, 100](epoch, minibatch):  0.9625856906175614
Loss [49, 200](epoch, minibatch):  1.0851884579658508
Loss [49, 300](epoch, minibatch):  1.1149670660495759
Loss [50, 100](epoch, minibatch):  0.9201913815736771
Loss [50, 200](epoch, minibatch):  0.9872671216726303
Loss [50, 300](epoch, minibatch):  1.0967929524183273
Accuracy on 50,000 train images:  73.61 %
Accuracy on 10,000 test images:  42.4 %
Loss [51, 100](epoch, minibatch):  0.8666902154684066
Loss [51, 200](epoch, minibatch):  0.9561692970991135
Loss [51, 300](epoch, minibatch):  1.0483973532915116
Loss [52, 100](epoch, minibatch):  0.844135627746582
Loss [52, 200](epoch, minibatch):  0.9048478132486344
Loss [52, 300](epoch, minibatch):  0.9972473722696304
Loss [53, 100](epoch, minibatch):  0.793890820145607
Loss [53, 200](epoch, minibatch):  0.9245178890228272
Loss [53, 300](epoch, minibatch):  0.9795035910606384
Loss [54, 100](epoch, minibatch):  0.7595633751153946
Loss [54, 200](epoch, minibatch):  0.827938329577446
Loss [54, 300](epoch, minibatch):  0.9306302487850189
Loss [55, 100](epoch, minibatch):  0.7217686468362808
Loss [55, 200](epoch, minibatch):  0.7902770930528641
Loss [55, 300](epoch, minibatch):  0.8844201076030731
Loss [56, 100](epoch, minibatch):  0.7119229081273079
Loss [56, 200](epoch, minibatch):  0.7813915836811066
Loss [56, 300](epoch, minibatch):  0.8706304621696472
Loss [57, 100](epoch, minibatch):  0.635873698592186
Loss [57, 200](epoch, minibatch):  0.7318183141946792
Loss [57, 300](epoch, minibatch):  0.8401437389850617
Loss [58, 100](epoch, minibatch):  0.6574186837673187
Loss [58, 200](epoch, minibatch):  0.6844675147533417
Loss [58, 300](epoch, minibatch):  0.7758466893434525
Loss [59, 100](epoch, minibatch):  0.6229745650291443
Loss [59, 200](epoch, minibatch):  0.6680485486984253
Loss [59, 300](epoch, minibatch):  0.7453454726934433
Loss [60, 100](epoch, minibatch):  0.5780613473057747
Loss [60, 200](epoch, minibatch):  0.6367659533023834
Loss [60, 300](epoch, minibatch):  0.7241937392950057
Accuracy on 50,000 train images:  81.63 %
Accuracy on 10,000 test images:  41.449999999999996 %
Loss [61, 100](epoch, minibatch):  0.6036755788326263
Loss [61, 200](epoch, minibatch):  0.6008390259742736
Loss [61, 300](epoch, minibatch):  0.6836028742790222
Loss [62, 100](epoch, minibatch):  0.5663443025946617
Loss [62, 200](epoch, minibatch):  0.6225888797640801
Loss [62, 300](epoch, minibatch):  0.7028672048449516
Loss [63, 100](epoch, minibatch):  0.5240642940998077
Loss [63, 200](epoch, minibatch):  0.5685034781694412
Loss [63, 300](epoch, minibatch):  0.6544782996177674
Loss [64, 100](epoch, minibatch):  0.5082533964514733
Loss [64, 200](epoch, minibatch):  0.5259771725535393
Loss [64, 300](epoch, minibatch):  0.6148519089818001
Loss [65, 100](epoch, minibatch):  0.503084518313408
Loss [65, 200](epoch, minibatch):  0.5306018209457397
Loss [65, 300](epoch, minibatch):  0.5980738776922226
Loss [66, 100](epoch, minibatch):  0.5035534173250198
Loss [66, 200](epoch, minibatch):  0.5231514617800712
Loss [66, 300](epoch, minibatch):  0.5927589553594589
Loss [67, 100](epoch, minibatch):  0.48718121826648714
Loss [67, 200](epoch, minibatch):  0.5093998727202416
Loss [67, 300](epoch, minibatch):  0.5642876991629601
Loss [68, 100](epoch, minibatch):  0.4540174850821495
Loss [68, 200](epoch, minibatch):  0.48130774438381196
Loss [68, 300](epoch, minibatch):  0.5540426418185234
Loss [69, 100](epoch, minibatch):  0.4433350026607513
Loss [69, 200](epoch, minibatch):  0.49269282519817353
Loss [69, 300](epoch, minibatch):  0.5352348229289055
Loss [70, 100](epoch, minibatch):  0.44320369228720663
Loss [70, 200](epoch, minibatch):  0.4563912069797516
Loss [70, 300](epoch, minibatch):  0.5178181174397468
Accuracy on 50,000 train images:  85.74000000000001 %
Accuracy on 10,000 test images:  42.26 %
Loss [71, 100](epoch, minibatch):  0.43476847127079965
Loss [71, 200](epoch, minibatch):  0.4434332972764969
Loss [71, 300](epoch, minibatch):  0.5168307721614838
Loss [72, 100](epoch, minibatch):  0.4067448100447655
Loss [72, 200](epoch, minibatch):  0.43131724774837493
Loss [72, 300](epoch, minibatch):  0.46041554674506185
Loss [73, 100](epoch, minibatch):  0.42382754176855086
Loss [73, 200](epoch, minibatch):  0.4410082796216011
Loss [73, 300](epoch, minibatch):  0.4618587934970856
Loss [74, 100](epoch, minibatch):  0.3798394650220871
Loss [74, 200](epoch, minibatch):  0.4143345159292221
Loss [74, 300](epoch, minibatch):  0.46268628478050233
Loss [75, 100](epoch, minibatch):  0.4129736241698265
Loss [75, 200](epoch, minibatch):  0.4040052953362465
Loss [75, 300](epoch, minibatch):  0.4664993447065353
Loss [76, 100](epoch, minibatch):  0.36193560481071474
Loss [76, 200](epoch, minibatch):  0.39976647615432737
Loss [76, 300](epoch, minibatch):  0.4552112281322479
Loss [77, 100](epoch, minibatch):  0.3622706100344658
Loss [77, 200](epoch, minibatch):  0.38268182560801506
Loss [77, 300](epoch, minibatch):  0.4734111547470093
Loss [78, 100](epoch, minibatch):  0.34385020986199377
Loss [78, 200](epoch, minibatch):  0.39410021930933
Loss [78, 300](epoch, minibatch):  0.449415143430233
Loss [79, 100](epoch, minibatch):  0.3591036696732044
Loss [79, 200](epoch, minibatch):  0.36555811315774916
Loss [79, 300](epoch, minibatch):  0.43289698094129564
Loss [80, 100](epoch, minibatch):  0.35678920581936835
Loss [80, 200](epoch, minibatch):  0.40643581449985505
Loss [80, 300](epoch, minibatch):  0.439931660592556
Accuracy on 50,000 train images:  88.986 %
Accuracy on 10,000 test images:  43.05 %
Loss [81, 100](epoch, minibatch):  0.3422529332339764
Loss [81, 200](epoch, minibatch):  0.3847895050048828
Loss [81, 300](epoch, minibatch):  0.40207163140177726
Loss [82, 100](epoch, minibatch):  0.34096994027495386
Loss [82, 200](epoch, minibatch):  0.3318336267769337
Loss [82, 300](epoch, minibatch):  0.4037871979176998
Loss [83, 100](epoch, minibatch):  0.33820448353886606
Loss [83, 200](epoch, minibatch):  0.364043854624033
Loss [83, 300](epoch, minibatch):  0.4216237026453018
Loss [84, 100](epoch, minibatch):  0.35130114808678625
Loss [84, 200](epoch, minibatch):  0.34772142484784124
Loss [84, 300](epoch, minibatch):  0.40137705728411677
Loss [85, 100](epoch, minibatch):  0.3453549598157406
Loss [85, 200](epoch, minibatch):  0.3451563359797001
Loss [85, 300](epoch, minibatch):  0.4073307874798775
Loss [86, 100](epoch, minibatch):  0.32240747585892676
Loss [86, 200](epoch, minibatch):  0.34613521173596384
Loss [86, 300](epoch, minibatch):  0.4118900798261166
Loss [87, 100](epoch, minibatch):  0.316205488294363
Loss [87, 200](epoch, minibatch):  0.3182556135952473
Loss [87, 300](epoch, minibatch):  0.37135674014687536
Loss [88, 100](epoch, minibatch):  0.31342229768633845
Loss [88, 200](epoch, minibatch):  0.3408366958796978
Loss [88, 300](epoch, minibatch):  0.3774087509512901
Loss [89, 100](epoch, minibatch):  0.32998675644397735
Loss [89, 200](epoch, minibatch):  0.3199284952878952
Loss [89, 300](epoch, minibatch):  0.35286944001913073
Loss [90, 100](epoch, minibatch):  0.30694174125790596
Loss [90, 200](epoch, minibatch):  0.3487551236152649
Loss [90, 300](epoch, minibatch):  0.38789851605892184
Accuracy on 50,000 train images:  91.264 %
Accuracy on 10,000 test images:  43.51 %
Loss [91, 100](epoch, minibatch):  0.3080439759790897
Loss [91, 200](epoch, minibatch):  0.3216807597875595
Loss [91, 300](epoch, minibatch):  0.36734491020441057
Loss [92, 100](epoch, minibatch):  0.3106425814330578
Loss [92, 200](epoch, minibatch):  0.34122798264026644
Loss [92, 300](epoch, minibatch):  0.3857238844037056
Loss [93, 100](epoch, minibatch):  0.3123758891224861
Loss [93, 200](epoch, minibatch):  0.3492915810644627
Loss [93, 300](epoch, minibatch):  0.3989545039832592
Loss [94, 100](epoch, minibatch):  0.3037424553930759
Loss [94, 200](epoch, minibatch):  0.31953591272234916
Loss [94, 300](epoch, minibatch):  0.35063293933868406
Loss [95, 100](epoch, minibatch):  0.2614819672703743
Loss [95, 200](epoch, minibatch):  0.30220710441470144
Loss [95, 300](epoch, minibatch):  0.3532290717959404
Loss [96, 100](epoch, minibatch):  0.2901356229186058
Loss [96, 200](epoch, minibatch):  0.2984791699051857
Loss [96, 300](epoch, minibatch):  0.37546130642294884
Loss [97, 100](epoch, minibatch):  0.30284454226493834
Loss [97, 200](epoch, minibatch):  0.3390818251669407
Loss [97, 300](epoch, minibatch):  0.3947117929160595
Loss [98, 100](epoch, minibatch):  0.30256054416298866
Loss [98, 200](epoch, minibatch):  0.29690241158008573
Loss [98, 300](epoch, minibatch):  0.3335610783100128
Loss [99, 100](epoch, minibatch):  0.28193055853247645
Loss [99, 200](epoch, minibatch):  0.28917603701353073
Loss [99, 300](epoch, minibatch):  0.3730214349925518
Loss [100, 100](epoch, minibatch):  0.2929776883125305
Loss [100, 200](epoch, minibatch):  0.30315176501870156
Loss [100, 300](epoch, minibatch):  0.344363958388567
Accuracy on 50,000 train images:  92.196 %
Accuracy on 10,000 test images:  43.519999999999996 %

To solve the issue of overfitting, you can try to implement certain ideas as described in this answer here. In the following case I attempted to reduce overfitting by adding a dropout layer before the linear layer under the ResNet class:

self.fc = nn.Sequential(
            nn.Dropout(p=0.5),
            nn.Linear(512*ResBlock.expansion, num_classes)
        )

Just a note that p = 0.5 is merely an arbitrary value that I used, and I have not tested manipulating the value for said probability. Do also note that I have also added net.train() and net.eval() before training and evaluation respectively, which leads to slower decrease in training loss (as nodes are trained in a way that they are less codependent on each other) but this also causes the model to be less accurate:

Loss [1, 100](epoch, minibatch):  12.75592649936676
Loss [1, 200](epoch, minibatch):  5.553812828063965
Loss [1, 300](epoch, minibatch):  5.045726490020752
Loss [2, 100](epoch, minibatch):  4.630871663093567
Loss [2, 200](epoch, minibatch):  4.461605010032653
Loss [2, 300](epoch, minibatch):  4.3488942861557005
Loss [3, 100](epoch, minibatch):  4.296987271308899
Loss [3, 200](epoch, minibatch):  4.238659825325012
Loss [3, 300](epoch, minibatch):  4.218650751113891
Loss [4, 100](epoch, minibatch):  4.168693675994873
Loss [4, 200](epoch, minibatch):  4.087853887081146
Loss [4, 300](epoch, minibatch):  4.058581359386444
Loss [5, 100](epoch, minibatch):  4.042889981269837
Loss [5, 200](epoch, minibatch):  3.9707986092567444
Loss [5, 300](epoch, minibatch):  3.945917134284973
Loss [6, 100](epoch, minibatch):  3.9540729475021363
Loss [6, 200](epoch, minibatch):  3.885404748916626
Loss [6, 300](epoch, minibatch):  3.865490424633026
Loss [7, 100](epoch, minibatch):  3.8600299096107484
Loss [7, 200](epoch, minibatch):  3.8075525617599486
Loss [7, 300](epoch, minibatch):  3.8140141940116883
Loss [8, 100](epoch, minibatch):  3.79449773311615
Loss [8, 200](epoch, minibatch):  3.7404275870323183
Loss [8, 300](epoch, minibatch):  3.7279130816459656
Loss [9, 100](epoch, minibatch):  3.724336361885071
Loss [9, 200](epoch, minibatch):  3.680591278076172
Loss [9, 300](epoch, minibatch):  3.684523892402649
Loss [10, 100](epoch, minibatch):  3.6346549654006957
Loss [10, 200](epoch, minibatch):  3.626280491352081
Loss [10, 300](epoch, minibatch):  3.5994439911842346
Accuracy on 50,000 train images:  15.476 %
Accuracy on 10,000 test images:  16.2 %
Loss [11, 100](epoch, minibatch):  3.5805077958106994
Loss [11, 200](epoch, minibatch):  3.5562810945510863
Loss [11, 300](epoch, minibatch):  3.5434193325042727
Loss [12, 100](epoch, minibatch):  3.5191331195831297
Loss [12, 200](epoch, minibatch):  3.49624436378479
Loss [12, 300](epoch, minibatch):  3.47701229095459
Loss [13, 100](epoch, minibatch):  3.4776473689079284
Loss [13, 200](epoch, minibatch):  3.448890993595123
Loss [13, 300](epoch, minibatch):  3.4246092653274536
Loss [14, 100](epoch, minibatch):  3.433450655937195
Loss [14, 200](epoch, minibatch):  3.4000335121154786
Loss [14, 300](epoch, minibatch):  3.3767191052436827
Loss [15, 100](epoch, minibatch):  3.385192196369171
Loss [15, 200](epoch, minibatch):  3.5351124668121336
Loss [15, 300](epoch, minibatch):  3.410601947307587
Loss [16, 100](epoch, minibatch):  3.341110429763794
Loss [16, 200](epoch, minibatch):  3.3177233076095582
Loss [16, 300](epoch, minibatch):  3.3922006630897523
Loss [17, 100](epoch, minibatch):  4.3483661890029905
Loss [17, 200](epoch, minibatch):  4.142327001094818
Loss [17, 300](epoch, minibatch):  4.070472881793976
Loss [18, 100](epoch, minibatch):  3.9750861525535583
Loss [18, 200](epoch, minibatch):  3.908917305469513
Loss [18, 300](epoch, minibatch):  3.86131865978241
Loss [19, 100](epoch, minibatch):  3.8199827027320863
Loss [19, 200](epoch, minibatch):  3.7647355937957765
Loss [19, 300](epoch, minibatch):  3.7343397092819215
Loss [20, 100](epoch, minibatch):  3.683690595626831
Loss [20, 200](epoch, minibatch):  3.6460893416404723
Loss [20, 300](epoch, minibatch):  3.6093356680870055
Accuracy on 50,000 train images:  16.508 %
Accuracy on 10,000 test images:  17.25 %
Loss [21, 100](epoch, minibatch):  3.533077619075775
Loss [21, 200](epoch, minibatch):  3.4658457374572755
Loss [21, 300](epoch, minibatch):  3.419013419151306
Loss [22, 100](epoch, minibatch):  3.4167966723442076
Loss [22, 200](epoch, minibatch):  3.3863394141197203
Loss [22, 300](epoch, minibatch):  3.371549484729767
Loss [23, 100](epoch, minibatch):  3.404439876079559
Loss [23, 200](epoch, minibatch):  3.3427918577194213
Loss [23, 300](epoch, minibatch):  3.366339828968048
Loss [24, 100](epoch, minibatch):  3.358658628463745
Loss [24, 200](epoch, minibatch):  3.3394760608673097
Loss [24, 300](epoch, minibatch):  3.338911259174347
Loss [25, 100](epoch, minibatch):  3.3358917570114137
Loss [25, 200](epoch, minibatch):  3.2831122779846194
Loss [25, 300](epoch, minibatch):  3.3232077836990355
Loss [26, 100](epoch, minibatch):  3.3298849511146544
Loss [26, 200](epoch, minibatch):  3.28564998626709
Loss [26, 300](epoch, minibatch):  3.2988738226890564
Loss [27, 100](epoch, minibatch):  3.3032293009757994
Loss [27, 200](epoch, minibatch):  3.246927161216736
Loss [27, 300](epoch, minibatch):  3.249611258506775
Loss [28, 100](epoch, minibatch):  3.2842838668823244
Loss [28, 200](epoch, minibatch):  3.240279905796051
Loss [28, 300](epoch, minibatch):  3.23079705953598
Loss [29, 100](epoch, minibatch):  3.264514548778534
Loss [29, 200](epoch, minibatch):  3.2432462692260744
Loss [29, 300](epoch, minibatch):  3.225596034526825
Loss [30, 100](epoch, minibatch):  3.23151073217392
Loss [30, 200](epoch, minibatch):  3.2022443532943727
Loss [30, 300](epoch, minibatch):  3.204464430809021
Accuracy on 50,000 train images:  23.824 %
Accuracy on 10,000 test images:  23.549999999999997 %
Loss [31, 100](epoch, minibatch):  3.2093122029304504
Loss [31, 200](epoch, minibatch):  3.1696739053726195
Loss [31, 300](epoch, minibatch):  3.1839580225944517
Loss [32, 100](epoch, minibatch):  3.197716860771179
Loss [32, 200](epoch, minibatch):  3.1467290043830873
Loss [32, 300](epoch, minibatch):  3.1509727668762206
Loss [33, 100](epoch, minibatch):  3.1838720178604127
Loss [33, 200](epoch, minibatch):  3.149483563899994
Loss [33, 300](epoch, minibatch):  3.13321076631546
Loss [34, 100](epoch, minibatch):  3.140736300945282
Loss [34, 200](epoch, minibatch):  3.13158097743988
Loss [34, 300](epoch, minibatch):  3.134981529712677
Loss [35, 100](epoch, minibatch):  3.1370452213287354
Loss [35, 200](epoch, minibatch):  3.099466679096222
Loss [35, 300](epoch, minibatch):  3.1189078998565676
Loss [36, 100](epoch, minibatch):  3.1227260613441468
Loss [36, 200](epoch, minibatch):  3.075305902957916
Loss [36, 300](epoch, minibatch):  3.101146197319031
Loss [37, 100](epoch, minibatch):  3.0873507952690122
Loss [37, 200](epoch, minibatch):  3.0730795192718507
Loss [37, 300](epoch, minibatch):  3.063821270465851
Loss [38, 100](epoch, minibatch):  3.065966258049011
Loss [38, 200](epoch, minibatch):  3.056092345714569
Loss [38, 300](epoch, minibatch):  3.0576061153411866
Loss [39, 100](epoch, minibatch):  3.0591069507598876
Loss [39, 200](epoch, minibatch):  3.046183214187622
Loss [39, 300](epoch, minibatch):  3.0269343781471254
Loss [40, 100](epoch, minibatch):  3.030228650569916
Loss [40, 200](epoch, minibatch):  3.0018977618217466
Loss [40, 300](epoch, minibatch):  3.0360106110572813
Accuracy on 50,000 train images:  27.772000000000002 %
Accuracy on 10,000 test images:  26.31 %
Loss [41, 100](epoch, minibatch):  3.016771080493927
Loss [41, 200](epoch, minibatch):  2.9758677244186402
Loss [41, 300](epoch, minibatch):  3.0148168635368346
Loss [42, 100](epoch, minibatch):  3.0094384765625
Loss [42, 200](epoch, minibatch):  2.975652689933777
Loss [42, 300](epoch, minibatch):  2.971101121902466
Loss [43, 100](epoch, minibatch):  2.9761976981163025
Loss [43, 200](epoch, minibatch):  2.962677822113037
Loss [43, 300](epoch, minibatch):  2.9771375942230223
Loss [44, 100](epoch, minibatch):  2.9565117883682253
Loss [44, 200](epoch, minibatch):  2.9341182041168214
Loss [44, 300](epoch, minibatch):  2.9539233016967774
Loss [45, 100](epoch, minibatch):  2.920460674762726
Loss [45, 200](epoch, minibatch):  2.9314848661422728
Loss [45, 300](epoch, minibatch):  2.9309214329719544
Loss [46, 100](epoch, minibatch):  2.9271196722984314
Loss [46, 200](epoch, minibatch):  2.89759202003479
Loss [46, 300](epoch, minibatch):  2.9082726073265075
Loss [47, 100](epoch, minibatch):  2.8976655912399294
Loss [47, 200](epoch, minibatch):  2.8934041357040403
Loss [47, 300](epoch, minibatch):  2.886395950317383
Loss [48, 100](epoch, minibatch):  2.879007992744446
Loss [48, 200](epoch, minibatch):  2.847784676551819
Loss [48, 300](epoch, minibatch):  2.8551166677474975
Loss [49, 100](epoch, minibatch):  2.8555069160461426
Loss [49, 200](epoch, minibatch):  2.8562831044197083
Loss [49, 300](epoch, minibatch):  2.8593265509605406
Loss [50, 100](epoch, minibatch):  2.8305810713768005
Loss [50, 200](epoch, minibatch):  2.85518030166626
Loss [50, 300](epoch, minibatch):  2.818350613117218
Accuracy on 50,000 train images:  31.622 %
Accuracy on 10,000 test images:  29.73 %
Loss [51, 100](epoch, minibatch):  2.809882607460022
Loss [51, 200](epoch, minibatch):  2.805054123401642
Loss [51, 300](epoch, minibatch):  2.8140970492362976
Loss [52, 100](epoch, minibatch):  2.795890061855316
Loss [52, 200](epoch, minibatch):  2.794620597362518
Loss [52, 300](epoch, minibatch):  2.8058355689048766
Loss [53, 100](epoch, minibatch):  2.8162489914894104
Loss [53, 200](epoch, minibatch):  2.756137025356293
Loss [53, 300](epoch, minibatch):  2.7801968264579773
Loss [54, 100](epoch, minibatch):  2.7783587050437926
Loss [54, 200](epoch, minibatch):  2.740168581008911
Loss [54, 300](epoch, minibatch):  2.7879208159446716
Loss [55, 100](epoch, minibatch):  2.739133563041687
Loss [55, 200](epoch, minibatch):  2.753390200138092
Loss [55, 300](epoch, minibatch):  2.7266321420669555
Loss [56, 100](epoch, minibatch):  2.7074920654296877
Loss [56, 200](epoch, minibatch):  2.721222631931305
Loss [56, 300](epoch, minibatch):  2.7426643896102907
Loss [57, 100](epoch, minibatch):  2.7279270195961
Loss [57, 200](epoch, minibatch):  2.717345864772797
Loss [57, 300](epoch, minibatch):  2.7055483508110045
Loss [58, 100](epoch, minibatch):  2.7074980211257933
Loss [58, 200](epoch, minibatch):  2.683807199001312
Loss [58, 300](epoch, minibatch):  2.697851402759552
Loss [59, 100](epoch, minibatch):  2.6557422423362733
Loss [59, 200](epoch, minibatch):  2.6635415506362916
Loss [59, 300](epoch, minibatch):  2.6622504663467406
Loss [60, 100](epoch, minibatch):  2.6504904413223267
Loss [60, 200](epoch, minibatch):  2.621088263988495
Loss [60, 300](epoch, minibatch):  2.657397434711456
Accuracy on 50,000 train images:  36.198 %
Accuracy on 10,000 test images:  32.14 %
Loss [61, 100](epoch, minibatch):  2.6381341361999513
Loss [61, 200](epoch, minibatch):  2.617038195133209
Loss [61, 300](epoch, minibatch):  2.660598554611206
Loss [62, 100](epoch, minibatch):  2.616093678474426
Loss [62, 200](epoch, minibatch):  2.619141170978546
Loss [62, 300](epoch, minibatch):  2.6151106262207033
Loss [63, 100](epoch, minibatch):  2.586893284320831
Loss [63, 200](epoch, minibatch):  2.5722559976577757
Loss [63, 300](epoch, minibatch):  2.6138885521888735
Loss [64, 100](epoch, minibatch):  2.569895031452179
Loss [64, 200](epoch, minibatch):  2.5521742630004884
Loss [64, 300](epoch, minibatch):  2.566419734954834
Loss [65, 100](epoch, minibatch):  2.5596899008750915
Loss [65, 200](epoch, minibatch):  2.543582353591919
Loss [65, 300](epoch, minibatch):  2.5472596383094785
Loss [66, 100](epoch, minibatch):  2.5270086479187013
Loss [66, 200](epoch, minibatch):  2.5221631717681885
Loss [66, 300](epoch, minibatch):  2.5481902456283567
Loss [67, 100](epoch, minibatch):  2.5235697555541994
Loss [67, 200](epoch, minibatch):  2.478102970123291
Loss [67, 300](epoch, minibatch):  2.5102244353294374
Loss [68, 100](epoch, minibatch):  2.4962460708618166
Loss [68, 200](epoch, minibatch):  2.4808554100990294
Loss [68, 300](epoch, minibatch):  2.4872255539894104
Loss [69, 100](epoch, minibatch):  2.468422236442566
Loss [69, 200](epoch, minibatch):  2.4642954993247987
Loss [69, 300](epoch, minibatch):  2.4694756078720093
Loss [70, 100](epoch, minibatch):  2.446262605190277
Loss [70, 200](epoch, minibatch):  2.448289120197296
Loss [70, 300](epoch, minibatch):  2.4500590109825136
Accuracy on 50,000 train images:  40.846 %
Accuracy on 10,000 test images:  34.48 %
Loss [71, 100](epoch, minibatch):  2.424196779727936
Loss [71, 200](epoch, minibatch):  2.425877095460892
Loss [71, 300](epoch, minibatch):  2.4276933383941652
Loss [72, 100](epoch, minibatch):  2.4282083249092103
Loss [72, 200](epoch, minibatch):  2.3904894375801087
Loss [72, 300](epoch, minibatch):  2.413998849391937
Loss [73, 100](epoch, minibatch):  2.382257902622223
Loss [73, 200](epoch, minibatch):  2.383921809196472
Loss [73, 300](epoch, minibatch):  2.3937970924377443
Loss [74, 100](epoch, minibatch):  2.369750074148178
Loss [74, 200](epoch, minibatch):  2.3759262657165525
Loss [74, 300](epoch, minibatch):  2.370888600349426
Loss [75, 100](epoch, minibatch):  2.3595095503330232
Loss [75, 200](epoch, minibatch):  2.3573951268196107
Loss [75, 300](epoch, minibatch):  2.3798316752910615
Loss [76, 100](epoch, minibatch):  2.351377034187317
Loss [76, 200](epoch, minibatch):  2.3069503796100617
Loss [76, 300](epoch, minibatch):  2.346709803342819
Loss [77, 100](epoch, minibatch):  2.3017871356010438
Loss [77, 200](epoch, minibatch):  2.2962716257572175
Loss [77, 300](epoch, minibatch):  2.332345823049545
Loss [78, 100](epoch, minibatch):  2.275312652587891
Loss [78, 200](epoch, minibatch):  2.2777904725074767
Loss [78, 300](epoch, minibatch):  2.29882924079895
Loss [79, 100](epoch, minibatch):  2.2641587817668913
Loss [79, 200](epoch, minibatch):  2.2501710271835327
Loss [79, 300](epoch, minibatch):  2.266463428735733
Loss [80, 100](epoch, minibatch):  2.2315343737602236
Loss [80, 200](epoch, minibatch):  2.2358140563964843
Loss [80, 300](epoch, minibatch):  2.2785512351989747
Accuracy on 50,000 train images:  46.338 %
Accuracy on 10,000 test images:  36.46 %
Loss [81, 100](epoch, minibatch):  2.199248116016388
Loss [81, 200](epoch, minibatch):  2.2026746666431425
Loss [81, 300](epoch, minibatch):  2.2113017630577088
Loss [82, 100](epoch, minibatch):  2.2012501990795137
Loss [82, 200](epoch, minibatch):  2.1782724392414092
Loss [82, 300](epoch, minibatch):  2.2366762709617616
Loss [83, 100](epoch, minibatch):  2.1413865315914156
Loss [83, 200](epoch, minibatch):  2.1800301492214205
Loss [83, 300](epoch, minibatch):  2.1856681621074676
Loss [84, 100](epoch, minibatch):  2.1111997258663178
Loss [84, 200](epoch, minibatch):  2.1532635509967806
Loss [84, 300](epoch, minibatch):  2.190418838262558
Loss [85, 100](epoch, minibatch):  2.101057668924332
Loss [85, 200](epoch, minibatch):  2.1402593874931335
Loss [85, 300](epoch, minibatch):  2.145553067922592
Loss [86, 100](epoch, minibatch):  2.077761994600296
Loss [86, 200](epoch, minibatch):  2.1201197361946105
Loss [86, 300](epoch, minibatch):  2.1325294399261474
Loss [87, 100](epoch, minibatch):  2.0966448283195493
Loss [87, 200](epoch, minibatch):  2.077999233007431
Loss [87, 300](epoch, minibatch):  2.1170987927913667
Loss [88, 100](epoch, minibatch):  2.0586745715141297
Loss [88, 200](epoch, minibatch):  2.0484184658527376
Loss [88, 300](epoch, minibatch):  2.077878919839859
Loss [89, 100](epoch, minibatch):  2.0097897243499756
Loss [89, 200](epoch, minibatch):  2.0640410232543944
Loss [89, 300](epoch, minibatch):  2.067508339881897
Loss [90, 100](epoch, minibatch):  1.9726059067249297
Loss [90, 200](epoch, minibatch):  2.0085206854343416
Loss [90, 300](epoch, minibatch):  2.046379854679108
Accuracy on 50,000 train images:  52.30200000000001 %
Accuracy on 10,000 test images:  36.41 %
Loss [91, 100](epoch, minibatch):  1.9885798799991607
Loss [91, 200](epoch, minibatch):  1.9949439644813538
Loss [91, 300](epoch, minibatch):  1.9925900292396546
Loss [92, 100](epoch, minibatch):  1.9526666486263276
Loss [92, 200](epoch, minibatch):  1.9651919245719909
Loss [92, 300](epoch, minibatch):  1.9859992837905884
Loss [93, 100](epoch, minibatch):  1.9037964606285096
Loss [93, 200](epoch, minibatch):  1.9484059762954713
Loss [93, 300](epoch, minibatch):  1.9679171359539032
Loss [94, 100](epoch, minibatch):  1.8865438532829284
Loss [94, 200](epoch, minibatch):  1.9356673133373261
Loss [94, 300](epoch, minibatch):  1.936594215631485
Loss [95, 100](epoch, minibatch):  1.8533212983608245
Loss [95, 200](epoch, minibatch):  1.8828663098812104
Loss [95, 300](epoch, minibatch):  1.9003119111061095
Loss [96, 100](epoch, minibatch):  1.8494437646865844
Loss [96, 200](epoch, minibatch):  1.8643197083473206
Loss [96, 300](epoch, minibatch):  1.863225429058075
Loss [97, 100](epoch, minibatch):  1.7769045448303222
Loss [97, 200](epoch, minibatch):  1.8413770079612732
Loss [97, 300](epoch, minibatch):  1.8432961761951447
Loss [98, 100](epoch, minibatch):  1.7616633236408235
Loss [98, 200](epoch, minibatch):  1.8109859728813171
Loss [98, 300](epoch, minibatch):  1.840243912935257
Loss [99, 100](epoch, minibatch):  1.7263727378845215
Loss [99, 200](epoch, minibatch):  1.8012216186523438
Loss [99, 300](epoch, minibatch):  1.7945610070228577
Loss [100, 100](epoch, minibatch):  1.7223280477523804
Loss [100, 200](epoch, minibatch):  1.744649795293808
Loss [100, 300](epoch, minibatch):  1.752061471939087
Accuracy on 50,000 train images:  60.529999999999994 %
Accuracy on 10,000 test images:  37.1 %

I then set track_running_stats = False for all of the BatchNorm2d that were used so that the normalization layer will use the batch statistics to normalize the activations even during evaluation. Validation acccuracy did improve, though I have to mention that after 100 epochs, overfitting seems to happen again, with large training loss fluctuations of +-0.3:

Loss [1, 100](epoch, minibatch):  11.6797882938385
Loss [1, 200](epoch, minibatch):  6.862401247024536
Loss [1, 300](epoch, minibatch):  4.945142064094544
Loss [2, 100](epoch, minibatch):  4.521172280311585
Loss [2, 200](epoch, minibatch):  4.397140665054321
Loss [2, 300](epoch, minibatch):  4.324621558189392
Loss [3, 100](epoch, minibatch):  4.266140768527984
Loss [3, 200](epoch, minibatch):  4.1990881085395815
Loss [3, 300](epoch, minibatch):  4.17779137134552
Loss [4, 100](epoch, minibatch):  4.16957403421402
Loss [4, 200](epoch, minibatch):  4.132833144664764
Loss [4, 300](epoch, minibatch):  4.085322487354278
Loss [5, 100](epoch, minibatch):  4.112524454593658
Loss [5, 200](epoch, minibatch):  4.055501687526703
Loss [5, 300](epoch, minibatch):  4.031656720638275
Loss [6, 100](epoch, minibatch):  4.0305104517936705
Loss [6, 200](epoch, minibatch):  3.9817501091957093
Loss [6, 300](epoch, minibatch):  3.9928100323677063
Loss [7, 100](epoch, minibatch):  3.9925182104110717
Loss [7, 200](epoch, minibatch):  3.9390996742248534
Loss [7, 300](epoch, minibatch):  3.90317617893219
Loss [8, 100](epoch, minibatch):  3.9239475321769715
Loss [8, 200](epoch, minibatch):  3.866264841556549
Loss [8, 300](epoch, minibatch):  3.8617013764381407
Loss [9, 100](epoch, minibatch):  3.8461808657646177
Loss [9, 200](epoch, minibatch):  3.8157106161117555
Loss [9, 300](epoch, minibatch):  3.795797472000122
Loss [10, 100](epoch, minibatch):  3.8110691142082214
Loss [10, 200](epoch, minibatch):  3.7199524736404417
Loss [10, 300](epoch, minibatch):  3.75013644695282
Accuracy on 50,000 train images:  13.474 %
Accuracy on 10,000 test images:  13.489999999999998 %
Loss [11, 100](epoch, minibatch):  3.731899538040161
Loss [11, 200](epoch, minibatch):  3.711401278972626
Loss [11, 300](epoch, minibatch):  3.6856098532676698
Loss [12, 100](epoch, minibatch):  3.692992055416107
Loss [12, 200](epoch, minibatch):  3.6443198990821837
Loss [12, 300](epoch, minibatch):  3.61707147359848
Loss [13, 100](epoch, minibatch):  3.6287704491615296
Loss [13, 200](epoch, minibatch):  3.5686117386817933
Loss [13, 300](epoch, minibatch):  3.560962431430817
Loss [14, 100](epoch, minibatch):  3.5653439021110533
Loss [14, 200](epoch, minibatch):  3.5096245884895323
Loss [14, 300](epoch, minibatch):  3.5334311151504516
Loss [15, 100](epoch, minibatch):  3.486849076747894
Loss [15, 200](epoch, minibatch):  3.484344460964203
Loss [15, 300](epoch, minibatch):  3.4602121186256407
Loss [16, 100](epoch, minibatch):  3.4546516942977905
Loss [16, 200](epoch, minibatch):  3.409707827568054
Loss [16, 300](epoch, minibatch):  3.4198642492294313
Loss [17, 100](epoch, minibatch):  3.4034258913993836
Loss [17, 200](epoch, minibatch):  3.3402858209609985
Loss [17, 300](epoch, minibatch):  3.3683947634696962
Loss [18, 100](epoch, minibatch):  3.343637857437134
Loss [18, 200](epoch, minibatch):  3.2920667004585265
Loss [18, 300](epoch, minibatch):  3.3088834381103513
Loss [19, 100](epoch, minibatch):  3.255146317481995
Loss [19, 200](epoch, minibatch):  3.246616635322571
Loss [19, 300](epoch, minibatch):  3.2081560635566713
Loss [20, 100](epoch, minibatch):  3.220287058353424
Loss [20, 200](epoch, minibatch):  3.180156772136688
Loss [20, 300](epoch, minibatch):  3.209682357311249
Accuracy on 50,000 train images:  24.576 %
Accuracy on 10,000 test images:  24.64 %
Loss [21, 100](epoch, minibatch):  3.1672178959846495
Loss [21, 200](epoch, minibatch):  3.1473090553283694
Loss [21, 300](epoch, minibatch):  3.102460367679596
Loss [22, 100](epoch, minibatch):  3.0955756640434267
Loss [22, 200](epoch, minibatch):  3.0603551387786867
Loss [22, 300](epoch, minibatch):  3.090216100215912
Loss [23, 100](epoch, minibatch):  3.0295106983184814
Loss [23, 200](epoch, minibatch):  3.0485816860198973
Loss [23, 300](epoch, minibatch):  3.0057256603240967
Loss [24, 100](epoch, minibatch):  2.981288845539093
Loss [24, 200](epoch, minibatch):  3.035719873905182
Loss [24, 300](epoch, minibatch):  3.006225419044495
Loss [25, 100](epoch, minibatch):  2.9726898169517515
Loss [25, 200](epoch, minibatch):  2.9337098002433777
Loss [25, 300](epoch, minibatch):  2.9429150795936585
Loss [26, 100](epoch, minibatch):  2.912439534664154
Loss [26, 200](epoch, minibatch):  2.8810079526901244
Loss [26, 300](epoch, minibatch):  2.8708882570266723
Loss [27, 100](epoch, minibatch):  2.821021330356598
Loss [27, 200](epoch, minibatch):  2.83649936914444
Loss [27, 300](epoch, minibatch):  2.860845012664795
Loss [28, 100](epoch, minibatch):  2.8113577795028686
Loss [28, 200](epoch, minibatch):  2.7664765238761904
Loss [28, 300](epoch, minibatch):  2.769034595489502
Loss [29, 100](epoch, minibatch):  2.7837938046455384
Loss [29, 200](epoch, minibatch):  2.7684893465042113
Loss [29, 300](epoch, minibatch):  2.719820215702057
Loss [30, 100](epoch, minibatch):  2.707117199897766
Loss [30, 200](epoch, minibatch):  2.7073365592956544
Loss [30, 300](epoch, minibatch):  2.7337183690071107
Accuracy on 50,000 train images:  34.75 %
Accuracy on 10,000 test images:  33.23 %
Loss [31, 100](epoch, minibatch):  2.708650312423706
Loss [31, 200](epoch, minibatch):  2.698123731613159
Loss [31, 300](epoch, minibatch):  2.653499438762665
Loss [32, 100](epoch, minibatch):  2.6093307447433474
Loss [32, 200](epoch, minibatch):  2.6364089894294738
Loss [32, 300](epoch, minibatch):  2.632340476512909
Loss [33, 100](epoch, minibatch):  2.597180120944977
Loss [33, 200](epoch, minibatch):  2.582633752822876
Loss [33, 300](epoch, minibatch):  2.5806027173995973
Loss [34, 100](epoch, minibatch):  2.516182005405426
Loss [34, 200](epoch, minibatch):  2.5566463351249693
Loss [34, 300](epoch, minibatch):  2.5344914507865908
Loss [35, 100](epoch, minibatch):  2.5091000962257386
Loss [35, 200](epoch, minibatch):  2.5224377727508545
Loss [35, 300](epoch, minibatch):  2.5290846705436705
Loss [36, 100](epoch, minibatch):  2.4851724910736084
Loss [36, 200](epoch, minibatch):  2.455410920381546
Loss [36, 300](epoch, minibatch):  2.450762628316879
Loss [37, 100](epoch, minibatch):  2.440768909454346
Loss [37, 200](epoch, minibatch):  2.544754960536957
Loss [37, 300](epoch, minibatch):  2.4912730765342714
Loss [38, 100](epoch, minibatch):  2.380314340591431
Loss [38, 200](epoch, minibatch):  2.3607143998146056
Loss [38, 300](epoch, minibatch):  2.40353052854538
Loss [39, 100](epoch, minibatch):  2.3349947118759156
Loss [39, 200](epoch, minibatch):  2.3727056074142454
Loss [39, 300](epoch, minibatch):  2.336453821659088
Loss [40, 100](epoch, minibatch):  2.3237900948524475
Loss [40, 200](epoch, minibatch):  2.33398463845253
Loss [40, 300](epoch, minibatch):  2.350702464580536
Accuracy on 50,000 train images:  41.046 %
Accuracy on 10,000 test images:  38.18 %
Loss [41, 100](epoch, minibatch):  2.3021518635749816
Loss [41, 200](epoch, minibatch):  2.293010370731354
Loss [41, 300](epoch, minibatch):  2.312970930337906
Loss [42, 100](epoch, minibatch):  2.2531477069854735
Loss [42, 200](epoch, minibatch):  2.268412197828293
Loss [42, 300](epoch, minibatch):  2.2586595022678377
Loss [43, 100](epoch, minibatch):  2.1815492379665375
Loss [43, 200](epoch, minibatch):  2.244083278179169
Loss [43, 300](epoch, minibatch):  2.2349623143672943
Loss [44, 100](epoch, minibatch):  2.200046036243439
Loss [44, 200](epoch, minibatch):  2.202470157146454
Loss [44, 300](epoch, minibatch):  2.2121153330802916
Loss [45, 100](epoch, minibatch):  2.1468175935745237
Loss [45, 200](epoch, minibatch):  2.1768783211708067
Loss [45, 300](epoch, minibatch):  2.180906013250351
Loss [46, 100](epoch, minibatch):  2.107428126335144
Loss [46, 200](epoch, minibatch):  2.124142565727234
Loss [46, 300](epoch, minibatch):  2.129796557426453
Loss [47, 100](epoch, minibatch):  2.0854806292057035
Loss [47, 200](epoch, minibatch):  2.0849017667770386
Loss [47, 300](epoch, minibatch):  2.082968372106552
Loss [48, 100](epoch, minibatch):  2.051953226327896
Loss [48, 200](epoch, minibatch):  2.0438814580440523
Loss [48, 300](epoch, minibatch):  2.0948777949810027
Loss [49, 100](epoch, minibatch):  2.0264714479446413
Loss [49, 200](epoch, minibatch):  2.012107743024826
Loss [49, 300](epoch, minibatch):  2.0383155488967897
Loss [50, 100](epoch, minibatch):  2.0081615447998047
Loss [50, 200](epoch, minibatch):  2.004052840471268
Loss [50, 300](epoch, minibatch):  2.031334534883499
Accuracy on 50,000 train images:  49.634 %
Accuracy on 10,000 test images:  43.4 %
Loss [51, 100](epoch, minibatch):  1.9695109927654266
Loss [51, 200](epoch, minibatch):  1.993222301006317
Loss [51, 300](epoch, minibatch):  2.0040572512149812
Loss [52, 100](epoch, minibatch):  1.9340404617786406
Loss [52, 200](epoch, minibatch):  1.9578441536426545
Loss [52, 300](epoch, minibatch):  1.9730224061012267
Loss [53, 100](epoch, minibatch):  1.9299118161201476
Loss [53, 200](epoch, minibatch):  1.9314423155784608
Loss [53, 300](epoch, minibatch):  1.9817650783061982
Loss [54, 100](epoch, minibatch):  1.9091873037815095
Loss [54, 200](epoch, minibatch):  1.909456170797348
Loss [54, 300](epoch, minibatch):  1.9445743870735168
Loss [55, 100](epoch, minibatch):  1.8734123265743257
Loss [55, 200](epoch, minibatch):  1.9072880494594573
Loss [55, 300](epoch, minibatch):  1.9319496548175812
Loss [56, 100](epoch, minibatch):  1.8636007511615753
Loss [56, 200](epoch, minibatch):  1.8716809391975402
Loss [56, 300](epoch, minibatch):  1.905283054113388
Loss [57, 100](epoch, minibatch):  1.8506467187404632
Loss [57, 200](epoch, minibatch):  1.8516873657703399
Loss [57, 300](epoch, minibatch):  1.8857384824752808
Loss [58, 100](epoch, minibatch):  1.8130059707164765
Loss [58, 200](epoch, minibatch):  1.8469235956668855
Loss [58, 300](epoch, minibatch):  1.8609120094776153
Loss [59, 100](epoch, minibatch):  1.8134743797779083
Loss [59, 200](epoch, minibatch):  1.821519889831543
Loss [59, 300](epoch, minibatch):  1.840588766336441
Loss [60, 100](epoch, minibatch):  1.7754599010944367
Loss [60, 200](epoch, minibatch):  1.8018251824378968
Loss [60, 300](epoch, minibatch):  1.8132909953594207
Accuracy on 50,000 train images:  54.944 %
Accuracy on 10,000 test images:  46.79 %
Loss [61, 100](epoch, minibatch):  1.77632404088974
Loss [61, 200](epoch, minibatch):  1.8043932676315309
Loss [61, 300](epoch, minibatch):  1.8177869522571564
Loss [62, 100](epoch, minibatch):  1.7330296123027802
Loss [62, 200](epoch, minibatch):  1.753926419019699
Loss [62, 300](epoch, minibatch):  1.7861640024185181
Loss [63, 100](epoch, minibatch):  1.6997134983539581
Loss [63, 200](epoch, minibatch):  1.7587981200218201
Loss [63, 300](epoch, minibatch):  1.789086035490036
Loss [64, 100](epoch, minibatch):  1.6887178444862365
Loss [64, 200](epoch, minibatch):  1.7446004951000214
Loss [64, 300](epoch, minibatch):  1.7653974556922913
Loss [65, 100](epoch, minibatch):  1.6807984387874604
Loss [65, 200](epoch, minibatch):  1.7270735895633698
Loss [65, 300](epoch, minibatch):  1.7433596634864807
Loss [66, 100](epoch, minibatch):  1.6510264229774476
Loss [66, 200](epoch, minibatch):  1.6930538547039031
Loss [66, 300](epoch, minibatch):  1.7444598305225372
Loss [67, 100](epoch, minibatch):  1.653894486427307
Loss [67, 200](epoch, minibatch):  1.682239224910736
Loss [67, 300](epoch, minibatch):  1.7101531374454497
Loss [68, 100](epoch, minibatch):  1.654067758321762
Loss [68, 200](epoch, minibatch):  1.6684845280647278
Loss [68, 300](epoch, minibatch):  1.7082119762897492
Loss [69, 100](epoch, minibatch):  1.613350238800049
Loss [69, 200](epoch, minibatch):  1.6274965822696685
Loss [69, 300](epoch, minibatch):  1.6839677286148071
Loss [70, 100](epoch, minibatch):  1.608758009672165
Loss [70, 200](epoch, minibatch):  1.6475862979888916
Loss [70, 300](epoch, minibatch):  1.6679020357131957
Accuracy on 50,000 train images:  58.982 %
Accuracy on 10,000 test images:  48.6 %
Loss [71, 100](epoch, minibatch):  1.6069848263263702
Loss [71, 200](epoch, minibatch):  1.6251343011856079
Loss [71, 300](epoch, minibatch):  1.637300021648407
Loss [72, 100](epoch, minibatch):  1.5962118244171142
Loss [72, 200](epoch, minibatch):  1.59470516204834
Loss [72, 300](epoch, minibatch):  1.6158559334278106
Loss [73, 100](epoch, minibatch):  1.5582088470458983
Loss [73, 200](epoch, minibatch):  1.5753504359722137
Loss [73, 300](epoch, minibatch):  1.6236006331443786
Loss [74, 100](epoch, minibatch):  1.5499393105506898
Loss [74, 200](epoch, minibatch):  1.5606462454795837
Loss [74, 300](epoch, minibatch):  1.6240913724899293
Loss [75, 100](epoch, minibatch):  1.5169697439670562
Loss [75, 200](epoch, minibatch):  1.5806949317455292
Loss [75, 300](epoch, minibatch):  1.612876009941101
Loss [76, 100](epoch, minibatch):  1.5416342830657959
Loss [76, 200](epoch, minibatch):  1.547739051580429
Loss [76, 300](epoch, minibatch):  1.569955427646637
Loss [77, 100](epoch, minibatch):  1.5000427794456481
Loss [77, 200](epoch, minibatch):  1.5526274597644807
Loss [77, 300](epoch, minibatch):  1.5538803517818451
Loss [78, 100](epoch, minibatch):  1.4705273401737213
Loss [78, 200](epoch, minibatch):  1.525465008020401
Loss [78, 300](epoch, minibatch):  1.5489486360549927
Loss [79, 100](epoch, minibatch):  1.4894460511207581
Loss [79, 200](epoch, minibatch):  1.5166389846801758
Loss [79, 300](epoch, minibatch):  1.5081677877902984
Loss [80, 100](epoch, minibatch):  1.4864945769309998
Loss [80, 200](epoch, minibatch):  1.4859675252437592
Loss [80, 300](epoch, minibatch):  1.5384834039211273
Accuracy on 50,000 train images:  62.364 %
Accuracy on 10,000 test images:  49.69 %
Loss [81, 100](epoch, minibatch):  1.439107004404068
Loss [81, 200](epoch, minibatch):  1.485583438873291
Loss [81, 300](epoch, minibatch):  1.5344973981380463
Loss [82, 100](epoch, minibatch):  1.4400836396217347
Loss [82, 200](epoch, minibatch):  1.4772904646396636
Loss [82, 300](epoch, minibatch):  1.4863573813438415
Loss [83, 100](epoch, minibatch):  1.401202689409256
Loss [83, 200](epoch, minibatch):  1.4739128243923187
Loss [83, 300](epoch, minibatch):  1.5179744005203246
Loss [84, 100](epoch, minibatch):  1.4219323194026947
Loss [84, 200](epoch, minibatch):  1.4304572021961213
Loss [84, 300](epoch, minibatch):  1.486987464427948
Loss [85, 100](epoch, minibatch):  1.4016130292415618
Loss [85, 200](epoch, minibatch):  1.4420735585689544
Loss [85, 300](epoch, minibatch):  1.456790348291397
Loss [86, 100](epoch, minibatch):  1.390570502281189
Loss [86, 200](epoch, minibatch):  1.4386502301692963
Loss [86, 300](epoch, minibatch):  1.4593178343772888
Loss [87, 100](epoch, minibatch):  1.3768282079696654
Loss [87, 200](epoch, minibatch):  1.4241219806671142
Loss [87, 300](epoch, minibatch):  1.4204120790958406
Loss [88, 100](epoch, minibatch):  1.3420930314064026
Loss [88, 200](epoch, minibatch):  1.4098711848258971
Loss [88, 300](epoch, minibatch):  1.418105251789093
Loss [89, 100](epoch, minibatch):  1.354487556219101
Loss [89, 200](epoch, minibatch):  1.4077046120166778
Loss [89, 300](epoch, minibatch):  1.4332940101623535
Loss [90, 100](epoch, minibatch):  1.338451938033104
Loss [90, 200](epoch, minibatch):  1.3776389348506928
Loss [90, 300](epoch, minibatch):  1.407549957036972
Accuracy on 50,000 train images:  66.39 %
Accuracy on 10,000 test images:  51.25999999999999 %
Loss [91, 100](epoch, minibatch):  1.3209140372276307
Loss [91, 200](epoch, minibatch):  1.37977933883667
Loss [91, 300](epoch, minibatch):  1.4204512250423431
Loss [92, 100](epoch, minibatch):  1.3338573312759399
Loss [92, 200](epoch, minibatch):  1.3543418806791305
Loss [92, 300](epoch, minibatch):  1.3911315619945526
Loss [93, 100](epoch, minibatch):  1.3024685257673263
Loss [93, 200](epoch, minibatch):  1.3610780072212219
Loss [93, 300](epoch, minibatch):  1.3893763399124146
Loss [94, 100](epoch, minibatch):  1.2955530858039856
Loss [94, 200](epoch, minibatch):  1.314615049958229
Loss [94, 300](epoch, minibatch):  1.3681304931640625
Loss [95, 100](epoch, minibatch):  1.2738198506832124
Loss [95, 200](epoch, minibatch):  1.3740097677707672
Loss [95, 300](epoch, minibatch):  1.3490469133853913
Loss [96, 100](epoch, minibatch):  1.2964045369625092
Loss [96, 200](epoch, minibatch):  1.3104093426465988
Loss [96, 300](epoch, minibatch):  1.3573172187805176
Loss [97, 100](epoch, minibatch):  1.2452750480175019
Loss [97, 200](epoch, minibatch):  1.2975342845916749
Loss [97, 300](epoch, minibatch):  1.3351542913913728
Loss [98, 100](epoch, minibatch):  1.2630453395843506
Loss [98, 200](epoch, minibatch):  1.2939193320274354
Loss [98, 300](epoch, minibatch):  1.3520437824726104
Loss [99, 100](epoch, minibatch):  1.237801432609558
Loss [99, 200](epoch, minibatch):  1.2943899953365325
Loss [99, 300](epoch, minibatch):  1.3005889236927033
Loss [100, 100](epoch, minibatch):  1.227265843153
Loss [100, 200](epoch, minibatch):  1.2851003110408783
Loss [100, 300](epoch, minibatch):  1.290888456106186
Accuracy on 50,000 train images:  68.53399999999999 %
Accuracy on 10,000 test images:  50.82 %

I don't know how you managed to get 58% accuracy with 100 epochs and the initial parameters, but the above may give you some insights on what went wrong, because I don't think I saw any glaring problems regarding the network architecture.

[teatonedev]

Can you also share the way you implement test images? I ran many epochs and still not very satisfied with the result.

[DenseLance]

Can you also share the way you implement test images? I ran many epochs and still not very satisfied with the result.

Do you mean how I evaluated the test images? Also, what are your results like currently?

[teatonedev]

With the code, I get around (tilde)70 percents of accuracy. However these are training results, I did not use anything to compare with them. I wonder how can I achieve such thing?

[DenseLance]

The last cell in ResNet/CIFAR10_ResNet50.ipynb, as seen below, shows the accuracy of your trained model using 10000 test images in CIFAR10. Not sure if this is what you meant.

correct = 0
total = 0

with torch.no_grad():
    for data in testloader:
        images, labels = data
        images, labels = images.to('cuda'), labels.to('cuda')
        outputs = net(images)
        
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print('Accuracy on 10,000 test images: ', 100*(correct/total), '%')

[teatonedev]

This is what I meant. Thanks for your contribution and help!