why in test.py we transform each frame of a video 3 times?

[JavanehBahrami]

Hi all, I can not understand why for each frame of a video we do redundant transformation (3 times)?

                    for i in range(3):
                        image = cv2.cvtColor(src, cv2.COLOR_BGR2RGB)
                        image = transform(image)
                        image = image.unsqueeze(0)
                        feature = network.forward(image)
                        feature = F.normalize(feature, dim=1)
                        score = F.softmax(loss.amsm_loss.s * loss.amsm_loss.fc(feature.squeeze()), dim=0)
                        score_lst.append(score)

the score_lst is always the same at the end of the loop. I don't know why? please could you explain it?

[zero-suger]

For your question : In that code means i is for set of images and for 3 images you r going to inference.
Open image -> apply transform as the same in validation -> now you have transformed image with 3D and batch with unsqueeze and make it 4D, bcz model needs 4D not 3D -> sent to network (can be DC-CDN,Swin_base ...) -> normalize and like [0,1] something like this -> and apply softmax to get your final decision -> if you apply predicted_index = torch.argmax(score).item() you will get model decision 0 for fake and 1 to real. In that case loop is just used to number of frames (fps) in your case may be images and score_lst is append with score in this case something like [1. 0.]

GL

[JavanehBahrami]

Thanks @zero-suger for your answer,

Do we need to use that loop, while testing only the directory of images, not videos?

I changed the code of test to apply on images not videos. Afterwards, I feed each image three times to the transformer, all output images values are the same and does not impact on prediction (it's look like that I copied the transfromed image 3 times), and the prediction of 3 transformed images always is the same.

this is my code for testing images (I remove the loop ->> for i in range(3):):

if __name__ == '__main__':
    cfg = read_cfg(cfg_file='config/config.yaml')

    device = get_device(cfg)

    network = build_network(cfg, device)
    network = network.to(device)
    loss = PatchLoss().to(device)
    saved_name = cfg['test']['checkpoint_path']
    state = torch.load(saved_name)

    class_names = cfg["test"]["classes"]

    network.load_state_dict(state['state_dict'])
    loss.load_state_dict(state["loss"])
    logger.info("loading model is done!")
    network.eval()

    test_transform = transforms.Compose([
        transforms.Resize(cfg['model']['image_size']),
        transforms.RandomCrop(cfg['dataset']['augmentation']['rand_crop_size']),
        transforms.ToTensor(),
        transforms.Normalize(cfg['dataset']['mean'], cfg['dataset']['sigma'])
    ])


    root_path = cfg['dataset']['root']
    csv_file = os.path.join(root_path, cfg['dataset']['test_set'])

    df = pd.read_csv(csv_file, header=None)

    with torch.no_grad():
        with open(csv_file) as file_obj:
            y_pred_list = []
            for idx, row in enumerate(df.iterrows()):
                if idx > 0:
                    img_path = row[1][0]
                    y_true = int(row[1][1])
                    img = Image.open(img_path)
                    image = test_transform(img)
                    image = image.unsqueeze(0)
                    feature = network.forward(image)
                    feature = F.normalize(feature, dim=1)
                    score = F.softmax(loss.amsm_loss.s * loss.amsm_loss.fc(feature.squeeze()), dim=0)
                    pred = torch.argmax(score, dim=0)
                    y_pred = pred.item()

                    logger.info(f' idx = {idx}, img_path = {img_path}, GT = {y_true}, Y_Pred = {y_pred}')
                    y_pred_list.append(y_pred)

[zero-suger]

Javaneh, I'm sorry for replying too late. Let me explain something to you... 1. a = 1 for a in range(10) means you are printing only 1, 10 times. This is the core of your previous question. 2. In face of anti spoofing tasks, a lot of datasets like 'Oulu', 'CASIA', ... have nearly the same images per video. For example, you are trying to use 'OULU' and there is video in 1_01_01_1 and if you want to extract frames from that video (usually 32 fps frames per second) and videos are short, so images and results are too close, if you forget to use shuffle before test 3. When you are training and testing, please don't forget to use cropped faces from RetinaFace or MTCNN. 4. The original code uses this strategy: video -> take frames -> detect face MTCNN -> crop 256 256 -> train and testing they used only 3 frames for each video set 5. There is no problem with your code in test.py, but I can't check your csv file and config file (please check your config file 'data'). 6. The patchnet final score is not like other FAS(Face Anti Spoofing) models map_score like 0.42, 0.21, 00.03, etc. Score can be 1 or 0 because of score = F.softmax(loss.amsm_loss.s * loss.amsm_loss.fc(feature.squeeze()), dim=0) this calculation . 7. I am sending my test.py file which opens 'image_folder' (in my case just 256 256 size face cropped images) and checks real or fake based on score 1 or 0. My model trained swinbase, please check and change pretrained weights and yaml file paths. Change paths, upload your 10 or 20 images (cropped fake and real face images) into image_folder path and check results. Hopefully you will get a 1 or 0 score based on your images. Good Luck (:

…

On Fri, Nov 24, 2023 at 7:49 PM Javaneh Bahrami ***@***.***> wrote: Thanks @zero-suger <https://github.com/zero-suger>, I changed the code of test to apply on images not videos. Afterwards, I feed each image three times to the transformer, all output images values are the same and does not impact on prediction, because the prediction of 3 transformed images always is the same. *Do we need to use that loop, while testing only the directory of images, not videos?* this is my code for testing images (I remove the loop ->> for i in range(3):): if __name__ == '__main__': os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:4024" cfg = read_cfg(cfg_file='/home/javaneh/javaneh_files/projects/face_anti_spoofing_project/Implementation-patchnet/config/config.yaml') device = get_device(cfg) network = build_network(cfg, device) network = network.to(device) loss = PatchLoss().to(device) saved_name = cfg['test']['checkpoint_path'] state = torch.load(saved_name) class_names = cfg["test"]["classes"] network.load_state_dict(state['state_dict']) loss.load_state_dict(state["loss"]) logger.info("loading model is done!") network.eval() test_transform = transforms.Compose([ transforms.Resize(cfg['model']['image_size']), transforms.RandomCrop(cfg['dataset']['augmentation']['rand_crop_size']), transforms.ToTensor(), transforms.Normalize(cfg['dataset']['mean'], cfg['dataset']['sigma']) ]) root_path = cfg['dataset']['root'] csv_file = os.path.join(root_path, cfg['dataset']['test_set']) df = pd.read_csv(csv_file, header=None) with torch.no_grad(): with open(csv_file) as file_obj: GroundTruth, pred_list, img_path_list = [], [], [] for idx, row in enumerate(df.iterrows()): score_lst = [] if idx > 0: img_path = row[1][0] gt = int(row[1][1]) # img = cv2.imread(img_path) img = Image.open(img_path) image = test_transform(img) image = image.unsqueeze(0) feature = network.forward(image) feature = F.normalize(feature, dim=1) score = F.softmax(loss.amsm_loss.s * loss.amsm_loss.fc(feature.squeeze()), dim=0) pred = torch.argmax(score, dim=0) pred = pred.item() logger.info(f' idx = {idx}, img_path = {img_path}, gt = {gt}, pred = {pred}') pred_list.append(pred) GroundTruth.append(gt) img_path_list.append(img_path) — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APDGECEB6MV2G7JMAOCL3Z3YGB3SDAVCNFSM6AAAAAA7XOKP46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRVGQ4TAMBWHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JavanehBahrami]

Thanks alot @zero-suger
my csv file contains cropped face images, not the raw images.

image_path                                                                                         | label |
CelebA/pre_processed_cropped_faces/test/6187/live/499699.png    |   1    |  
CelebA/pre_processed_cropped_faces/test/6187/live/500490.png    |   1    |  
CelebA/pre_processed_cropped_faces/test/6187/spoof/537370.png |   0    |  
CelebA/pre_processed_cropped_faces/test/6187/spoof/505132.png |   0    |  

I would appreciate if you send the code

[zero-suger]

Thanks alot @zero-suger my csv file contains cropped face images, not the raw images.

image_path                                                                                         | label |
CelebA/pre_processed_cropped_faces/test/6187/live/499699.png    |   1    |  
CelebA/pre_processed_cropped_faces/test/6187/live/500490.png    |   1    |  
CelebA/pre_processed_cropped_faces/test/6187/spoof/537370.png |   0    |  
CelebA/pre_processed_cropped_faces/test/6187/spoof/505132.png |   0    |  

I would appreciate if you send the code

I sent to you gmail

[azadehashouri]

@JavanehBahrami

Hi. Could you help me find the pretrained model?

[zero-suger]

Hi ~

First of all this is not official repo. So if you want to obtain pre trained weights, there is only one way : you must send official requests to each dataset and you can download them with provided links. Good luck...