Scoring code

[ankit-vaghela30]

In Features.py, Loader class with read method is there which reads and transforms the data into required RDDs. I ran below snippet:

import hydrus.features as hf

data, labels = hf.Loader(ctx).read(args.data_path, args.label_path)
testData, testLabel = hf.Loader(ctx).read(args.test_data_path, args.test_label_path)
print('training doc length: ', len(data.map(lambda x: (x[0][0], x[1])).reduceByKey(lambda x,y: x+y).collect()))
print('training label length: ', len(labels.collect()))

data = hf.TfIdfTransformer(ctx).fit(data).transform(data)
testData = hf.TfIdfTransformer(ctx).fit(testData).transform(testData)
print('testing doc length: ', len(testData.map(lambda x: (x[0][0], x[1])).reduceByKey(lambda x,y: x+y).collect()))
print('testing labels length: ', len(testLabel.collect()))

Now, for training data I got same size as 77, 77 while for testing data i got size as 13 and 14.

@cbarrick , Can you please take a look at this?

[cbarrick]

What are args.data_path and args.label_path? If they are the same, then this is expected behavior.

Consider the case where document 1 has the label C12,CCAT,MCAT. In this case, we have been instructed to ignore the C12 label and duplicate the document for the CCAT and MCAT labels. So that one documents become two: 1_CCAT and 1_MCAT.

Similarly, if document 2 only has the label C12 then we are instructed to ignore it.

If you don't provide a label file, none of this ignoring or duplicating happens (because there are no labels to ignore or duplicate from). So the number of docs cannot be guaranteed to be the same.

The upshot is that your training data must be loaded with a label file and your test data must be loaded without a label file. Even if you're training and testing on the same set, you still have to load the data twice.

[cbarrick]

From the docs

If a label file is specified and a document does not have a 'CAT' label, it is ignored. On the other hand, If a label file is specified and a document has multiple 'CAT' labels, the document is duplicated under both labels. If no label file is given, documents are neither ignored nor duplicated.

[ankit-vaghela30]

How to check the accuracy in this case?
I created a map out of my model having similar structure as labels which is returned from the preprocessing. I am currently comparing labels with my map and they are not matching simply because they have different length and w is coming out to be low.

Can't all document names from labels match the document names in features RDD-- data?

[cbarrick]

You'll have to manually write a routine that loads in the labels for a test set.

I was hoping someone other than me would write that (since I originally committed to neither ML algorithms nor scoring). But since I'm working on an logistic regression now, I will need such functionality eventually.

I think @vamsi3309 was supposed to write the scoring code.

Let's refocus this issue to be about getting scoring code ready, and assign to Vamsi.

[cbarrick]

The logistic regression model (#18) has it's own accuracy code. And it looks like the NB model (#8) is growing it's own accuracy code.

So the focus of this issue should be a general metric library. We could use functions like accuracy, f-score, and r^2. The API would be simple: just bare functions that take predicted labels and true labels and return the score.

Building off of the algorithm in the logistic regression would be a good start.