Provides changes necessary to run evaluation scripts.

ISWC-reproducibility.md

@@ -75,8 +75,7 @@ We'll use the official python scorer for Clef'18 Task 2, so we need to generate
 
 ``` shell
 export CLEF18_TE_DIR=data/evaluation/clef18_fact_checking_lab_submissions_and_scores_and_combinations
-mkdir ${CLEF18_TE_DIR}/task2_submissions/acred/
-mkdir ${CLEF18_TE_DIR}/task2_submissions/acred/acred_task2_en 
+mkdir -p ${CLEF18_TE_DIR}/task2_submissions/acred/acred_task2_en 
 ```
 To generate the predictions, we provide `scripts/pred_clef2018.py` which reads the input files and sends them to an `acred` REST API. By default, we assume this is deployed on `localhost`, if this is not the case, please edit `scripts/pred_clef2018-config.json` to point to the correct ip and port. Then you can run the following command to generate the credibility reviews and Clef'18 label predictions (`TRUE`, `FALSE` or `HALF-TRUE`; this should take about 3 minutes, depending on your server hardware):
 
@@ -154,11 +153,23 @@ Scoring credibility accuracy of 248 coinform250 tweets
 First, you need to follow the instructions at the [FakeNewsNet GitHub repo](https://github.com/KaiDMML/FakeNewsNet) to download the data. Since the dataset published only contains the gold label and URLs of the articles, you need to run a script to crawl the URLs. In our ISWC evaluation we only used:
  * `news_source`: `politifact`
  * `label`: `fake` and `real`
- * `data_features_to_collect`: `news_articles** only
+ * `data_features_to_collect`: **news_articles** only
 
-**TODO**: provide instructions for running the prediction and scoring scripts for this dataset.
+You can directly generate the credibility reviews and label predictions by executing the following script:
+
+``` shell
+mkdir data/evaluation/fakeNewsnet_reviews
+python scripts/pred_fakeNewsNet.py -fakeNewsNetFolder data/evaluation/fakeNewsNet -output_dir data/evaluation/fakeNewsnet_reviews/ -acredapi_url https://ACREDAPI
+```
+
+Once this is finished, you should be able to execute the scoring script by executing:
+
+``` shell
+python scripts/score_fakeNewsNet.py -fakeNewsNetFolder data/evaluation/fakeNewsNet -predictions_csv data/evaluation/fakeNewsnet_reviews/predictions.csv
+```
+
 
 # Limitations
-Reproducibility limitation: some of code in the version of `acred` we used during experimentation relied on proprietary code from Expert System which had to be removed from the code released. This code is used during the analysis of long texts like articles, as it identifies sentences which may be claims based on its relevance to topics and the presence of named entities. We plan to replace this code with sentence detection via NLTK and a custom checkworthiness detection model. This should mainly affect results on `FakeNewsNet` (articles), but not `Clef'18` (claims) or `coinform250` (tweets, most of which do not contain links to other web pages**.
+Reproducibility limitation: some of code in the version of `acred` we used during experimentation relied on proprietary code from Expert System which had to be removed from the code released. This code is used during the analysis of long texts like articles, as it identifies sentences which may be claims based on its relevance to topics and the presence of named entities. We plan to replace this code with sentence detection via NLTK and a custom checkworthiness detection model. This should mainly affect results on `FakeNewsNet` (articles), but not `Clef'18` (claims) or `coinform250` (tweets, most of which do not contain links to other web pages).
 
 

acred.ini

@@ -42,6 +42,8 @@ fnc_test_batch_size = 64
 acred_factchecker_urls_path = factchecker_urls.txt
 acred_pred_claim_search_url = http://localhost:8070/test/api/v1/claim/internal-search
 acred_search_verify = True
+acred_search_auth_user = testuser
+acred_search_auth_pwrd = testpass
 # thresholds
 # default cut-off point between not-verifiable and the rest
 cred_conf_threshold = 0.75

claimneuralindex/run_faiss_index.py

@@ -6,7 +6,7 @@
 
 logger = logging.getLogger(__name__)
 
-claim_embeddings = '/opt/model/claim-embeddings/claim_embs.tsv'
+claim_embeddings = '/opt/data/model/claim-embeddings/claim_embs.tsv'
 
 def normalize(vectors, norm='l2'):
     if len(vectors.shape) == 2:

data/fnc1-classifier.json

@@ -0,0 +1 @@
+{"class": "RoBERTa", "batch_size": 64, "finetuned_from_layer": 8, "train_val_result": {"loss": 0.28, "metrics": {"acc": 0.923, "f1_weighted": 0.9186, "n": 25413}}, "stance2i": {"agree": 1, "disagree": 2, "discuss": 3, "unrelated": 0}, "seq_len": 128}

docker-compose.yml

@@ -35,7 +35,7 @@ services:
     command: su -c '/usr/local/bin/uwsgi --ini /opt/acred/wsgi-acred-api.ini' acred
     volumes:
      - ${ACRED_docker_logdir}:/var/log/acred
-     - ${ACRED_docker_acred_model_dir}:/opt/model/
+     - ${ACRED_docker_acred_model_dir}:/opt/data/model/
      - ${ACRED_docker_acred_data_dir}:/opt/data/     
     depends_on:
      - claimneuralindex
@@ -49,10 +49,10 @@ services:
       dockerfile: docker/Dockerfile-claimneuralindex
     expose:
       - "9001"
-    command: su -c '/usr/local/bin/uwsgi --ini /opt/claimneuralindex/wsgi-claimneuralindex-api.ini' acred
+    command: su -c '/opt/conda/bin/uwsgi --ini /opt/claimneuralindex/wsgi-claimneuralindex-api.ini' acred
     volumes:
      - ${ACRED_docker_logdir}:/var/log/claimneuralindex
-     - ${ACRED_docker_acred_model_dir}:/opt/model/
+     - ${ACRED_docker_acred_model_dir}:/opt/data/model/
     depends_on:
      - claimencoder
 
@@ -67,7 +67,7 @@ services:
     command: su -c '/usr/local/bin/uwsgi --ini /opt/claimencoder/wsgi-claimencoder-api.ini' acred
     volumes:
      - ${ACRED_docker_logdir}:/var/log/claimencoder
-     - ${ACRED_docker_acred_model_dir}:/opt/model/
+     - ${ACRED_docker_acred_model_dir}:/opt/data/model/
 
   worthinesschecker:
     image: acred/worthinesschecker
@@ -80,4 +80,4 @@ services:
     command: su -c '/usr/local/bin/uwsgi --ini /opt/worthinesschecker/wsgi-worthinesschecker-api.ini' acred
     volumes:
       - ${ACRED_docker_logdir}:/var/log/worthinesschecker
-      - ${ACRED_docker_acred_model_dir}:/opt/model/
+      - ${ACRED_docker_acred_model_dir}:/opt/data/model/

docker/Dockerfile-acredapi

@@ -16,7 +16,7 @@ WORKDIR /opt/acred
 
 # use pip to install all the requirements
 COPY requirements.txt ./
-COPY ./requirements/ ./requirements/
+
 RUN pip3 install --upgrade pip \
 &&  pip3 install -r requirements.txt
 

docker/Dockerfile-claimencoder

@@ -17,7 +17,6 @@ WORKDIR /opt/claimencoder
 # use pip to install all the requirements
 COPY requirements.txt ./
 COPY requirements-torch.txt ./
-COPY ./requirements/ ./requirements/
 
 RUN pip3 install --upgrade pip \
 &&  pip3 install -r requirements.txt \

docker/Dockerfile-claimneuralindex

@@ -1,4 +1,4 @@
-FROM conda/miniconda3
+FROM continuumio/miniconda3
 MAINTAINER Ronald Denaux <rdenaux@expertsystem.com>
 COPY docker/timezone /etc/timezone
 RUN dpkg-reconfigure -f noninteractive tzdata
@@ -15,14 +15,13 @@ ENV PATH="/opt/claimneuralindex:${PATH}"
 WORKDIR /opt/claimneuralindex
 
 RUN conda update -n base -c defaults conda \
-    && conda install faiss-cpu -c pytorch \
+    && conda install -c pytorch faiss-cpu \
     && conda install -c conda-forge uwsgi \
     && conda install -c anaconda pip    
 
 # use pip to install the other claimneuralindex requirements
 COPY claimneuralindex/requirements-claimneuralindex.txt ./
 COPY requirements-torch.txt ./
-COPY ./requirements/ ./requirements/
 
 RUN pip install -r requirements-torch.txt -f https://download.pytorch.org/whl/torch_stable.html \
  && pip install -r requirements-claimneuralindex.txt

docker/Dockerfile-worthinesschecker

@@ -17,7 +17,6 @@ WORKDIR /opt/worthinesschecker
 # use pip to install all the requirements
 COPY requirements.txt ./
 COPY requirements-torch.txt ./
-COPY ./requirements/ ./requirements/
 
 RUN pip3 install --upgrade pip \
 &&  pip3 install -r requirements.txt \

docker/env-prod

@@ -39,12 +39,12 @@ ACRED_acredapi_stance_pred_url=http://nginx-api:9080/claimneuralindex
 
 ACRED_claimneuralindex_semencoder_url=http://nginx-api:9080/claimencoder
 
-ACRED_claimneuralindex_claim_embeddings_path=/opt/model/claim-embeddings/claim_embs.tsv
-ACRED_claimencoder_semantic_encoder_dir=/opt/model/semantic_encoder/
+ACRED_claimneuralindex_claim_embeddings_path=/opt/data/model/claim-embeddings/claim_embs.tsv
+ACRED_claimencoder_semantic_encoder_dir=/opt/data/model/semantic_encoder/
 
-ACRED_stance_fnc1_model_path=/opt/model/stance/saved_fnc1_classifier_acc_0.92
+ACRED_stance_fnc1_model_path=/opt/data/model/stance/saved_fnc1_classifier_acc_0.92
 
-ACRED_worthinesschecker_check_worthiness_model_path=/opt/model/check_worthiness
+ACRED_worthinesschecker_check_worthiness_model_path=/opt/data/model/check_worthiness
 ACRED_worthinesschecker_clef_test_batch_size=64
 ACRED_worthinesschecker_clef_test_worth_samples=64
 

docker/start-env.sh

@@ -17,15 +17,15 @@ fi
 export ACRED_docker_env=.env
 export ACRED_docker_logdir=./log/
 
-export DL_MODEL=data/model
-export DATA=data
+export DATA=./data
+export DL_MODEL=${DATA}/model
 
-export COINFOAPY_docker_acred_data_dir=${DATA}/
+export ACRED_docker_acred_data_dir=${DATA}/
 
 if [ -d ${DL_MODEL}/semantic_encoder ] && [ -L ${DL_MODEL}/claim-embeddings ] && [ -L ${DL_MODEL}/check_worthiness ];
 then
     echo "Linking to models in prod environment"
-    export ACRED_docker_model_dir=${DL_MODEL}/
+    export ACRED_docker_acred_model_dir=${DL_MODEL}/
 else
     echo "*** Could not find expected models in ${DL_MODEL}"
 fi

requirements-torch.txt

@@ -1,2 +1,2 @@
-torch==1.3.0+cpu 
-torchvision==0.4.1+cpu
+torch==1.7.1+cpu
+torchvision==0.8.2+cpu

scripts/fetch_data.sh

@@ -26,7 +26,7 @@ fi
 mkdir data
 echo "Fetching DB of claims"
 wget -O data/claims-from-ClaimReviews-45K.csv https://zenodo.org/record/4030305/files/claims-from-ClaimReviews-45K.csv?download=1
-wget -O data/sentences-extratedFrom-Articles-40K.csv https://zenodo.org/record/4030305/files/sentences-extractedFrom-Articles-40K.csv?download=1
+wget -O data/sentences-extractedFrom-Articles-40K.csv https://zenodo.org/record/4030305/files/sentences-extractedFrom-Articles-40K.csv?download=1
 wget -O data/claimReviews-pruned-45K.jsonl https://zenodo.org/record/4030305/files/claimReviews-pruned-45K.jsonl?download=1
 
 mkdir data/model
@@ -46,9 +46,11 @@ tar -xzf data/model/saved_fnc1_classifier_acc_0.92.tar.gz -C data/model/stance
 echo "Fetching checkworthiness model"
 wget -O data/model/check_worthiness_acc_0.95.zip https://zenodo.org/record/4030305/files/check_worthiness_acc_0.95.zip?download=1
 unzip data/model/check_worthiness_acc_0.95.zip -d data/model/
-ln -s data/model/check_worthiness_acc_0.95 data/model/check_worthiness 
+ln -sr data/model/check_worthiness_acc_0.95 data/model/check_worthiness 
 
 echo "Fetching embeddings for sentences in DB "
 wget -O data/model/claim_dev_embs_85K_20200426.tar.gz https://zenodo.org/record/4030305/files/claim_dev_embs_85K_20200426.tar.gz?download=1
 tar -zxf data/model/claim_dev_embs_85K_20200426.tar.gz -C data/model/
-ln -s data/model/claim_dev_embs_85K_20200426 data/model/claim-embeddings 
+ln -sr data/model/claim_dev_embs_85K_20200426 data/model/claim-embeddings
+
+cp data/fnc1-classifier.json data/model/stance/saved_fnc1_classifier_acc_0.92

scripts/fetch_eval_data.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # This script downloads and unpacks data required for running acred
-#
+
 echo "This script downloads and unpacks data required for evaluating acred"
 echo " "
 
@@ -32,15 +32,14 @@ wget -O data/evaluation/clef2018-factchecking-v1.0.tar.gz https://github.com/cle
 echo "Unpacking Clef'18 training data"
 tar -xzf data/evaluation/clef2018-factchecking-v1.0.tar.gz -C data/evaluation/
 
-echo "Adding acred-iswc predictions as task2 submission for Clef'18"
-mkdir data/evaluation/clef18_fact_checking_lab_submissions_and_scores_and_combinations/task2_submissions/acred-iswc
-cp data/acred-iswc__task2_en.zip data/evaluation/clef18_fact_checking_lab_submissions_and_scores_and_combinations/task2_submissions/acred-iswc
-
-echo "Fetching Clef'18 testing data, submissions and scores from http://alt.qcri.org/clef2018-factcheck/"
-wget -O data/evaluation/clef18_submissions_and_scores.zip http://alt.qcri.org/clef2018-factcheck/data/uploads/clef18_fact_checking_lab_submissions_and_scores_and_combinations.zip
+echo "Copying Clef'18 testing data, submissions and scores"
+cp data/clef18_submissions_and_scores.zip data/evaluation/
 echo "Unpacking Clef'18 testing data"
 unzip data/evaluation/clef18_submissions_and_scores.zip -d data/evaluation/
 
+echo "Adding acred-iswc predictions as task2 submission for Clef'18"
+mkdir data/evaluation/clef18_fact_checking_lab_submissions_and_scores_and_combinations/task2_submissions/acred-iswc
+cp data/acred-iswc___task2_en.zip data/evaluation/clef18_fact_checking_lab_submissions_and_scores_and_combinations/task2_submissions/acred-iswc
 
 echo "Fetching coinfo250 testing data from GitHub"
 wget -O data/evaluation/coinform250.json https://github.com/co-inform/Datasets/raw/master/coinform250.json

scripts/pred_clef2018.py

@@ -18,11 +18,27 @@
 import json
 import pandas as pd
 import logging
-from esiutils import dictu
 
 logger = logging.getLogger(__name__)
 
 
+def get_in(dct, path, default_val=None):
+    """Gets a nested value in a dict by following the path
+
+    :param dct: a python dictionary
+    :param path: a list of keys pointing to a node in dct
+    :returns: the value at the specified path
+    :rtype: any
+    """
+    if dct is None:
+        return default_val
+    assert len(path) > 0
+    next_dct = dct.get(path[0], None)
+    if len(path) == 1:
+        return next_dct
+    return get_in(next_dct, path[1:], default_val=default_val)
+
+
 def read_all_factuality_claims(folder):
   files = [f for f in os.listdir(folder) if 'README' not in f]
   columns = ['line_number', 'speaker', 'text', 'claim_number', 'normalized_claim', 'label']
@@ -108,7 +124,7 @@ def setup_logging():
             resp = requests.get(url, verify=False)
             resp.raise_for_status()
             claimcreds = resp.json()
-            credRating = dictu.get_in(claimcreds[0], cred_path)
+            credRating = get_in(claimcreds[0], cred_path)
             clef_pred.append({
                 'id': cid,
                 'label': acred_as_clef_label(

scripts/pred_coinfo250.py

@@ -41,7 +41,7 @@ def acred_as_coinfo_label(credreview, thresh=0.4):
     if val >= -0.25:
         return 'credible_uncertain'
     if val >= -0.5:
-        return 'credible_uncertain'
+        return 'mostly_not_credible'
     return 'not_credible'
 
 
@@ -74,7 +74,7 @@ def exec_req(i, req, args):
     return result
 
 
-def as_acred_requests(tweets, batchSize=5):
+def as_acred_requests(tweets, batchSize=1):
     batch = []
     for i, t in enumerate(tweets):
         batch.append({
@@ -101,7 +101,7 @@ def as_acred_requests(tweets, batchSize=5):
         help='Path to the coinform250.json file',
         required=True)
     parser.add_argument(
-        '-batchSize', type=int, default=5,
+        '-batchSize', type=int, default=1,
         help='Number of tweets to send per request to acred endpoint')
     parser.add_argument(
         '-outDir',

scripts/pred_fakeNewsNet.py

@@ -22,6 +22,8 @@
 import logging
 import sys
 import datetime
+from os.path import exists
+
 
 logger = logging.getLogger(__name__)
 
@@ -173,17 +175,22 @@ def try_write_review_to_outdir(review, doc, args):
         print('Failed to write %s %s' % (out_path, e))
 
 def review_article(ditem, args):
-    url = '%s/acred/api/v1/acred/reviewer/credibility/webpage' % args.acredapi_url
-    req = {
-        'webpages': [ditem]
-    }
-    resp = requests.post(url, verify=False, json=req)
-    resp.raise_for_status()
-    respd = resp.json()
-    assert type(respd) is list
-    assert len(respd) == 1  # single doc requested
-    review = respd[0]
-    try_write_review_to_outdir(review, ditem, args)
+    try:
+        url = '%s/acred/api/v1/acred/reviewer/credibility/webpage' % args.acredapi_url
+        req = {
+            'webpages': [ditem]
+        }
+        resp = requests.post(url, verify=False, json=req)
+        resp.raise_for_status()
+        respd = resp.json()
+        assert type(respd) is list
+        assert len(respd) == 1  # single doc requested
+        review = respd[0]
+        try_write_review_to_outdir(review, ditem, args)
+    except:
+        review = {
+                'item_id': ditem["id"], 'label': 'not_verifiable',
+                'explanation': 'Error from acred endpoint'}
     return review
 
 def acred_rating_as_acred_label(rating):
@@ -229,6 +236,17 @@ def acred_rating_as_fakeNewsNet_label(rating):
     print('base_dirs %s' % base_dirs)
     preds = []
     for item_id, json_path in gen_data_feature_files(base_dirs, args.data_feature):
+        if exists(f"{args.output_dir}{item_id}.json"):
+            print(f"Review for file with id {item_id} already exists")
+            with open(f"{args.output_dir}{item_id}.json", 'r', encoding='utf-8') as in_file:
+                review = json.load(in_file)
+                preds.append({
+                    'item_id': item_id,
+                    'label': acred_rating_as_fakeNewsNet_label(review['reviewRating']),
+                    'acred_label': acred_rating_as_acred_label(review['reviewRating']),
+                    'explanation': review['text']
+                })
+            continue
         if not os.path.exists(json_path):
             preds.append({
                 'item_id': item_id, 'label': 'not_verifiable',
@@ -242,12 +260,15 @@ def acred_rating_as_fakeNewsNet_label(rating):
             }
             review = review_article(in_doc, args)
             print('review keys', list(review.keys()))
-            preds.append({
-                'item_id': item_id,
-                'label': acred_rating_as_fakeNewsNet_label(review['reviewRating']),
-                'acred_label': acred_rating_as_acred_label(review['reviewRating']),
-                'explanation': review['text']
-            })
+            if review.get('reviewRating'):
+                preds.append({
+                    'item_id': item_id,
+                    'label': acred_rating_as_fakeNewsNet_label(review['reviewRating']),
+                    'acred_label': acred_rating_as_acred_label(review['reviewRating']),
+                    'explanation': review['text']
+                })
+            else:
+                preds.append(review)
 
     path = '%s/predictions.csv' % (args.output_dir)
     pd.DataFrame(preds).to_csv(path)