GSoC 2019 Phase 1, 2 & 3 work submission

README.md

@@ -10,36 +10,98 @@ If you want to contribute to this project, there are several things that you hav
 2. Exactly Java 1.8 
 3. MySQL & Tomcat
 4. Gradle
-5. Slack account
+5. Slack/Telegram/Facebook account
+
+And for NLU, you are required to have
+1. Python 3.6.1 (as NOW Rasa NLU is having dependency issues with Python 3.7)
+2. Microsoft VC++ Compiler
+3. Rasa (for offline training and testing models)
+
+```
+# install Rasa NLU using these commands
+
+pip install rasa[spacy]
+python -m spacy download en_core_web_md
+python -m spacy link en_core_web_md en
+```
 
 ## Getting started
 
-### Installing on Mac OS
-Java: <br>
+### Setting up
+##### Java:
 There are two options. 
 1. Download Java 1.8 from [official website](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html), install it and configure it on class path
 2. If you have `brew` installed in your system, you can also download from homebrew. `brew cask install java8` to install Java 8 on Mac OS. 
 
-MySQL:
+##### MySQL:
 1. Download from [official website](https://dev.mysql.com/downloads/mysql/), install it and configure it. During installation, if the user does not set the initial password, then the initial password should be set in the System Preferences -> MySQL -> set initial password. Then configure mysql in the classpath. 
+1.1. If you want to use Cloud Service for database, you are free to do that. Remember to update `application.properties` file. 
+2. Create table `user_creds` using following MySQL command
+```
+create table users_creds
+(
+	username varchar(255) not null,
+	secret_key varchar(255) not null,
+	fb_userid varchar(255) null,
+	slack_userid varchar(255) null,
+	telegram_userid varchar(255) null,
+	skype_userid varchar(255) null
+);
+```
 
-2. If you want to use Cloud Service instead of local installation of database, you can use Docker to create a virtual database. 
-
-Gradle <br>
+##### Gradle:
 Download Gradle from [official website](https://gradle.org/install/). 
 
-Slack
-1. Join Mifos workspace and search for `chatbot` in the application page to access service. 
-2. If you want to have your own Slack bot, apply from [here](https://api.slack.com/apps?new_app=1). 
-
-### Playing with chatbot
-After forking and cloning chatbot project , do a gradle import run the unit tests in those modules to understand the function of each module.<br>
-
-Key in text in the chatbot and read the response to understand more.
-
-For example, you can type `Check my arrear day`, `Check my due interest`, `Check my next due date` in the Slack interface. Then you will receive response from the Slack bot, like "478 days" as the response of `arrear day`, "1150.0" as the response of `due interest` and `Sat Apr 29 00:00:00 SGT 2017` as the response of `next due date`. 
+Now we need a platform to run your chatbot, we have integration with Slack, Telegram and Facebook messenger (atleast one is required to setup).
+##### Slack:
+1. Deploy your own Slack bot, apply from [here](https://api.slack.com/apps?new_app=1).
+2. Get `Bot User OAuth Access Token` from OAuth & Permission.
+3. Set token to environment variable with name `MIFOS_SLACK_API_TOKEN`.
+
+Now you are all setup to test bot on Slack.
+##### Telegram:
+1. Search for the “botfather” telegram bot on either [web.telegram.org](https://web.telegram.org/) or on telegram app.
+2. Create bot using `/newbot` command.
+3. Set username of bot to name `TELEGRAMBOT_USERNAME` and token to `TELEGRAMBOT_TOKEN` environment variables.
+
+Now you are all setup to test bot on telegram.
+##### Facebook Messenger:
+1. Create a facebook page (here we will deploy our bot) and set page access token to `MESSENGER_PAGE_ACCESS_TOKEN` environment variable.
+2. Create facebook app from [here](https://developers.facebook.com/).
+3. Set messenger app token to `MESSENGER_APP_SECRET` and verify token (will be used to setup callback) to `MESSENGER_VERIFY_TOKEN` environment variables.
+4. Set callback url to endpoint where chatbot is deployed and messenger verify token on Webhooks section (by clicking on "Edit Subscription" button).
+
+Now you are all setup to test bot on Facebook Messenger.
+### Training your NLU own model
+Use the following command to train Rasa NLU model. Make sure you have prerequisites for Rasa and Rasa is installed on your local system. 
+```
+rasa train nlu
+```
+Remember to use following structure for Rasa, else provide training data and other configuration files manually.
+###### Rasa structure
+```
+mifos-chatbot\nlp\src\main\resources\rasa/
+|-- data/
+|    |
+|    |-- nlu.md
+|
+|-- models
+|    |
+|    |-- nlu-XXXXXXXX-XXXXXX.tar.gz
+|
+|-- config.yml
+``` 
+
+### Deploying and testing your own model
+Use the following command to run Rasa NLU model locally. 
+```
+# Start a server with your NLU model as web service
+rasa run --enable-api -m models/nlu-XXXXXXXX-XXXXXX.tar.gz
+# Test your model on shell 
+rasa shell -m models/nlu-XXXXXXXX-XXXXXX.tar.gz
+```
 
-### Playing with Fineract and Chatbot
+### Playing with Fineract
 
 As you may know, the Chatbot project will query information from Fineract. Hence, we are supposed to have a public instance or a local instance of Fineract running on Tomcat server.
 
@@ -49,28 +111,45 @@ Set up database using mysql-first-time.sql to populate sample data into local da
 
 For the public instance, [open demo org] is available for use. One minor issue about this instance is that the authority level is that everyone involved in Fineract related project can change it. Hence, if someone push the code with bugs to the server, then it may cause errors.
 
-After settling down issues about Fineract, configure mifos.url in application.properties in server package. Apply for your own bot user in the application page of Slack and put the token in application.properties file also.
+### Playing with chatbot
 
-The sample application properties file is like that:
-```
-server.port=10000
+Fork and clone chatbot project, do a gradle import and run the unit tests in those modules to understand the function of each module.<br>
 
-mifos.apiUrl=https://demo.openmf.org/fineract-provider/api/v1
+Run chatbot and type your queries on respective chat platform. For example, you can type `Check my arrear day`, `Check my due interest`, `Check my next due date`. Then you will receive response from the Mifos chatbot, like "478 days" as the response of `arrear day`, "1150.0" as the response of `due interest` and `Sat Apr 29 00:00:00 SGT 2017` as the response of `next due date`. 
 
-#Slack
-mifos.slackApiToken=xoxb-15186592304-405342609110-7PHj7JTsvcQ3wlGrGb8XLMjW
+Remember to configure application.properties in server package.
 
-mifos.username=mifos
-mifos.password=password
+The sample application properties file is like that:
+```
+server.port="port where chatbot will run"
+mifos.apiUrl="fineract endpoint"
+mifos.slackApiToken=${MIFOS_SLACK_API_TOKEN}
+mifos.username=${MIFOS_USERNAME}
+mifos.password=${MIFOS_PASSWORD}
+messenger4j.appSecret=${MESSENGER_APP_SECRET}
+messenger4j.verifyToken=${MESSENGER_VERIFY_TOKEN}
+messenger4j.pageAccessToken=${MESSENGER_PAGE_ACCESS_TOKEN}
+telegram.botUsername=${TELEGRAMBOT_USERNAME}
+telegram.botToken=${TELEGRAMBOT_TOKEN}
+developer.metadata="developer defined Metadata"
+# configured for mysql
+spring.jpa.hibernate.ddl-auto=none
+spring.datasource.driverClassName = com.mysql.jdbc.Driver
+spring.datasource.url = "database url"
+spring.datasource.username = "database username"
+spring.datasource.password = "database password"
+rasa.nlu.service.url = "Rasa NLU model endpoint"
+logging.level.org.springframework=DEBUG
 ```
 
 Run `./gradlew server:bootRun` to startup the application and input text in the Slack interface.
 
 ## Authors
-[Zhao Dingfan](https://github.com/ZhaoDingfan)
+- [Zhao Dingfan](https://github.com/ZhaoDingfan) (GSoC 2018)
+- [Anshul Singh](https://github/com/iosdev747) (GSoC 2019)
 
 ## Mentor
-Aleksandar
+- Aleksandar
 
 ## License
 This project is licensed under Apache License Version 2.0.

adapter/src/main/java/org/mifos/chatbot/adapter/MifosChatbotAdapterService.java

@@ -18,12 +18,14 @@
 import lombok.extern.slf4j.Slf4j;
 import org.mifos.chatbot.core.AdapterService;
 import org.mifos.chatbot.core.IntentHandler;
+import org.mifos.chatbot.core.SmallTalkService;
 import org.mifos.chatbot.core.model.Intent;
 import org.mifos.chatbot.core.model.MifosResponse;
-import org.mifos.chatbot.nlp.OpenNLPService;
+import org.mifos.chatbot.nlp.RasaNLUService;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.stream.Collectors;
@@ -32,38 +34,42 @@
 @Service
 public class MifosChatbotAdapterService implements AdapterService {
     @Autowired
-    private OpenNLPService openNLPService;
+    private RasaNLUService rasaNLUService;
 
     @Autowired
     private List<IntentHandler> handlers;
 
+    @Autowired
+    private SmallTalkService smallTalkService;
+
     @Override
     public List<MifosResponse> handle(Intent intent) {
         return handlers
-            .stream()
-            .filter(handler -> handler.canHandle(intent))
-            .map(handler -> handler.handle(intent))
-            .collect(Collectors.toList());
+                .stream()
+                .filter(handler -> handler.canHandle(intent))
+                .map(handler -> handler.handle(intent))
+                .collect(Collectors.toList());
     }
 
     @Override
     public List<MifosResponse> handle(String input) {
-        List<MifosResponse> results = new ArrayList<>();
-        Intent[] intents = openNLPService.recognize(input);
-
-        Long id = 1L; // default id, set to 1
-
-        for(Intent intent : intents) {
-            results.addAll(handle(intent));
+        Intent intent = null;
+        try {
+            intent = rasaNLUService.recognize(input);
+        } catch (IOException e) {
+            e.printStackTrace();
         }
-        return results;
+        return new ArrayList<>(handle(intent));
     }
 
-    /*
-
-
-      [Slack/Viber/XMPP/IRC/Skype] -> NLP -> Intent1, Intent2, Intent3 -> AdapterService(Intent) -> responses -> [Slack/Viber/XMPP/IRC/Skype]
-
-
-     */
+    @Override
+    public boolean isSmallTalkRequest(String messageText) {
+        Intent intent = null;
+        try {
+            intent = rasaNLUService.recognize(messageText);
+        } catch (IOException e) {
+            e.printStackTrace();
+        }
+        return smallTalkService.isSmallTalkRequest(intent);
+    }
 }

adapter/src/main/java/org/mifos/chatbot/adapter/handler/loan/ArrearDayHandler.java

@@ -20,6 +20,7 @@
 import org.apache.commons.lang3.time.FastDateFormat;
 import org.mifos.chatbot.adapter.handler.HandlerUtils;
 import org.mifos.chatbot.client.ApiException;
+import org.mifos.chatbot.client.Configuration;
 import org.mifos.chatbot.client.api.LoansApi;
 import org.mifos.chatbot.client.model.GetLoansLoanIdResponse;
 import org.mifos.chatbot.core.model.Intent;
@@ -35,13 +36,14 @@
 @Slf4j
 @Component
 public class ArrearDayHandler extends BaseLoanIntentHandler {
-    private static final String[] INTENT_KEYWORDS = {"arrear", "day"};
+    private static final String[] INTENT_KEYWORDS = {"arrear_day"};
 
     @Autowired
     private LoansApi loansApi;
 
     @Override
     public Boolean canHandle(Intent intent) {
+        // not removing this logic because it is possible that multiple intent can be handled with same handler
         for(String intent_keyword : INTENT_KEYWORDS) {
             if (!intent.getKeyword().toLowerCase().contains(intent_keyword.toLowerCase())) {
                 return false;
@@ -53,10 +55,15 @@ public Boolean canHandle(Intent intent) {
 
     @Override
     public MifosResponse handle(Intent intent) {
+        loansApi.setApiClient(Configuration.getDefaultApiClient());
         MifosResponse response = new MifosResponse();
         try {
             GetLoansLoanIdResponse result = loansApi.retrieveLoan(intent.getParameterAsLong("ID"), false);
             List<Long> overdueSinceDate = result.getSummary().getOverdueSinceDate();
+            if(overdueSinceDate == null) {
+                response.setContent("No data found for the given id.");
+                return response;
+            }
 //
 //            StringBuffer sb = new StringBuffer();
 //            sb.append(String.format("%04d", overdueSinceDate.get(0)));
@@ -74,7 +81,7 @@ public MifosResponse handle(Intent intent) {
 
             long diffDays = TimeUnit.DAYS.convert(diff, TimeUnit.MILLISECONDS);
 
-            response.setContent(String.valueOf(diffDays) + " days");
+            response.setContent(diffDays + " days");
         } catch (ApiException e) {
             log.info("Error", e);
             response.setContent(e.getMessage());

adapter/src/main/java/org/mifos/chatbot/adapter/handler/loan/ClientActivationDateHandler.java

@@ -18,6 +18,7 @@
 import lombok.extern.slf4j.Slf4j;
 import org.mifos.chatbot.adapter.handler.HandlerUtils;
 import org.mifos.chatbot.client.ApiException;
+import org.mifos.chatbot.client.Configuration;
 import org.mifos.chatbot.client.api.ClientApi;
 import org.mifos.chatbot.client.model.GetClientsClientIdResponse;
 import org.mifos.chatbot.core.model.Intent;
@@ -30,7 +31,7 @@
 @Slf4j
 @Component
 public class ClientActivationDateHandler extends BaseLoanIntentHandler {
-    private static final String[] INTENT_KEYWORDS = {"client", "activation", "date"};
+    private static final String[] INTENT_KEYWORDS = {"client_activation_date"};
 
     @Autowired
     private ClientApi clientApi;
@@ -48,6 +49,7 @@ public Boolean canHandle(Intent intent) {
 
     @Override
     public MifosResponse handle(Intent intent) {
+        clientApi.setApiClient(Configuration.getDefaultApiClient());
         MifosResponse response = new MifosResponse();
         try {
             GetClientsClientIdResponse result = clientApi.retrieveOne(intent.getParameterAsLong("ID"), false);

adapter/src/main/java/org/mifos/chatbot/adapter/handler/loan/DueInterestHandler.java

@@ -17,6 +17,7 @@
 
 import lombok.extern.slf4j.Slf4j;
 import org.mifos.chatbot.client.ApiException;
+import org.mifos.chatbot.client.Configuration;
 import org.mifos.chatbot.client.api.LoansApi;
 import org.mifos.chatbot.client.model.GetLoansLoanIdResponse;
 import org.mifos.chatbot.core.model.Intent;
@@ -27,7 +28,7 @@
 @Slf4j
 @Component
 public class DueInterestHandler extends BaseLoanIntentHandler {
-    private static final String[] INTENT_KEYWORDS = {"due", "Interest"};
+    private static final String[] INTENT_KEYWORDS = {"due_interest"};
 
     @Autowired
     private LoansApi loansApi;
@@ -45,6 +46,7 @@ public Boolean canHandle(Intent intent) {
 
     @Override
     public MifosResponse handle(Intent intent) {
+        loansApi.setApiClient(Configuration.getDefaultApiClient());
         MifosResponse response = new MifosResponse();
         try {
             GetLoansLoanIdResponse result = loansApi.retrieveLoan(intent.getParameterAsLong("ID"), false);

adapter/src/main/java/org/mifos/chatbot/adapter/handler/loan/DuePrincipalHandler.java

@@ -17,15 +17,18 @@
 
 import lombok.extern.slf4j.Slf4j;
 import org.mifos.chatbot.client.ApiException;
+import org.mifos.chatbot.client.Configuration;
 import org.mifos.chatbot.client.api.LoansApi;
 import org.mifos.chatbot.client.model.GetLoansLoanIdResponse;
 import org.mifos.chatbot.core.model.Intent;
 import org.mifos.chatbot.core.model.MifosResponse;
 import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
 
 @Slf4j
+@Component
 public class DuePrincipalHandler extends BaseLoanIntentHandler {
-    private static final String[] INTENT_KEYWORDS = {"due", "principal"};
+    private static final String[] INTENT_KEYWORDS = {"due_principal"};
 
     @Autowired
     private LoansApi loansApi;
@@ -43,6 +46,7 @@ public Boolean canHandle(Intent intent) {
 
     @Override
     public MifosResponse handle(Intent intent) {
+        loansApi.setApiClient(Configuration.getDefaultApiClient());
         MifosResponse response = new MifosResponse();
         try {
             GetLoansLoanIdResponse result = loansApi.retrieveLoan(intent.getParameterAsLong("ID"), false);