Inference on my machine differs from other machines I tested

[webstruck]

Hi, I trained the model on AWS (GPU instance) for 60K steps and got the model. I then tested it on several GPU/CPU instance and results are consistent. When I deploy it locally on my Ubuntu desktop (CPU only), the inferences are totally off. I tested on AWS GPU instance (p2.xlarge), AWS CPU instance (c5d.4xlarge) and also on Colab. All three show consistent answers for a given context and questions. Only on my desktop the answers are way off. Any inputs as to why this could be happening would help. Thanks!

[webstruck]

I further tested this on a windows machine (CPU). Initially I used pre-processed data from my Ubuntu desktop and it did not work well. Then I performed data pre-processing on windows machine and it worked like a charm. I then copied the data from windows to ubuntu and there too it worked well. So looks like something to do with data pre-processing on my Ubuntu machine. Any hints?

[theSage21]

One simple way of seeing what could be going wrong is to see which preprocess files differ from each other. They are json so should be easy enough to compare.

[webstruck]

One simple way of seeing what could be going wrong is to see which preprocess files differ from each other. They are json so should be easy enough to compare.

Yeah, so basically it comes down to char and word dictionaries and embeddings. On all the machines it worked well, the dictionary file follow the sequence (e.g. sequence of tokens in glove embeddings)

e.g.
word dictionary always starts with

{",":2,".":3,"the":4,"and":5,"to":6,"of":7,"a":8,"in":9}

On my ubuntu desktop this sequence is totally different.
{ "Gasteyer": 2, "Naturalists": 3, "Newlywed": 4, "reliability": 5, "homophobia": 27, "gracefully": 45837, "Factual": 8, "Jankowski": 9,}

Not that the sequence matters as long are indexes are pointing to right vector. But due to this, text comparison between files is not easy.