Mini-Project 3: Sentence Reading Shubham Gupta [email protected] Abstract This Mini Project aims to develop a question answer-ing system that should be able to give an answer based on the knowledge acquired from the given sentence. Implemented Named-entity-recognition based approach. c. Google BERT, Increasing F1-score over baseline results. Contacted the developers for gaining information on the performance, seems like they don't know how it degraded when they updated the toolkit to incorporate new Allen AI and Huggingface module versions(Issue thread-. # Initialized at the start of the program, # everything below are arrays serve as boostrapped knowledge for the agent, # if the how is asking about an adjective of a noun, # if the how is asking about an adjective of an agent, # this one asks for an adjective verb of an agent, # if how is asking about how much/many someone did an action, # if how is asking about how much/many someone did WITHOUT ADJ, # this does the same but is more niche. Implemented approach in Repurposing Entailment for Multi-Hop Question Answering Tasks, Added task into the baseline model for the above approach and dataset transformation script under branch "MultiRC_NLI/". This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. A step is a batch update, The word embedding or contextual word representation layer, How to handle the embedding layer of the BERT, The type of the final layer(s) in classification and regression tasks, If true, use attn in sentence-pair classification/regression tasks, Use 'bert_adam' for reproducing BERT experiments, Minimum learning rate. # the parsed sentence goes into this dicitonary, # update sentence and tokens if necessary, # put tokens of sentence into the semantic_structure table, # collect tokens from question and compare them to the semantic_structure to find the answer. Reading Comprehension is the task of having the reader answer questions based on the given piece of text. Are you sure you want to create this branch? This model focuses on part 1 of this reading comprehension task; This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. Improve the model over baseline scores on the Multi-RC dataset. results.tsv consists of cumulative evaluation results over the runs, log.log files have the complete log for respective runs, params.conf have a copy of the configurations used for that run, models: Trained model, config file and vocab, MultiRC_NER notebook: code for training the NER model on training data, MultiRC_NER_eval: code for evaluating the trained NER model on evaluation data, parser.py: converts the given MultiRC data from original format to the NER format, exploratory_analysis: has code and analysis related to BERT QA model, preprocess_multirc.py: convert the given MultiRC data from original format to the NLI format, Convert the MultiRC dataset into NER format using the parser.py, Run training notebook and evaluation notebook (replace the folder path for the trained model and outputs in these notebooks). Sentence Selection for Reading Comprehension task on the SQuaD question answering dataset. Punched will have a verb index of 1 in the semantic break down and so will Jake so Jake is returned, # this makes sure the verb is assigned appropriately with the agent, # this checks if the WHO asks who received an action from an agent, # checks if a noun is acting as an agent "Three men in a car", # checks if an agent is interacting with a noun #################### maybe janky. Attaches verb to noun, # how someone performs something, find the verb, # if how is asking about how much/many someone did for numbers, # if it asks for a time for when someone went to a noun, # pull up most recent location: matches verb location with the noun, # if the where location doesn't have a verb associated with it and there's no agents, ################### before len(VERB) == 0, # gets matching noun index with matching adjective index, # if a specific date or time is mentioned, # WHAT asks about an item an agent did with. # who asks about noun activity with another using "with", "behind", etc. https://docs.google.com/spreadsheets/d/1zLZw-e5Anm17ah5RsUGOAzEJpmqOGGp-nA_72XfQN-E/edit?usp=sharing, https://www.overleaf.com/read/zfbzkqjzxwrb, https://docs.google.com/presentation/d/1Z8hRQzUXM6ZboHXiayK_s2NtFMi9Ek0osfTT1MWxj9s/edit?usp=sharing, Interval (in steps) at which you want to evaluate your model on the validation set during pretraining. After manually checking results, it is observed that a particular option with any resemblance to a portion of the paragraph is marked TRUE without taking the question into context. The model has been run in Tensorflow v0.11 . Idea- Using the concept of BIO tagging to train the model on correct tags for the correct answer and vice-versa for the wrong answers. If nothing happens, download GitHub Desktop and try again. The dataset has the unique property of having word spans of the original text passage as answers rather than single word or multiple choice answers. There was a problem preparing your codespace, please try again. Researched multi-hop approaches such as Multi-hop Question Answering via Reasoning Chains. The repo consists of following files/folders: (subset of configurations from default.conf which we have overriden on custom config files), Complete overview of JIANT: https://arxiv.org/pdf/2003.02249.pdf, Tuning baseline model jiant to execute task 'MultiRC'. Pre-requisite- Tranformed the MultiRC dataset into an NER dataset with different tags, one each for- paragraph, question, correct and incorrect answer. Will stop once these many validation steps are done, Maximum number of epochs (full pass over a task's training data), (MultiRC in our case) list of target tasks to (train and) test on, Run pre-train on tasks mentioned in pretrain_tasks, After do_pretrain, train on the target tasks in target_tasks, If true, restore from checkpoint when starting do_pretrain. # who asks about agent activity with another using "with", "behind", etc. Learn more. One important observation- frozen BERT without any pre-training gave approximately the same results. It finds a matching index of the verb and ties it to the object (noun), # If what asks about an adjective of a noun and not an agent, # if the question has > 1 noun involved implying it's looking for an ADJ but it asks with a noun, ##### these next ones are the same but pertain to an agent #####, ################################################################, # niche case: "Watch your step" type questions, # if there's only 1 agent and it asks about what happened to the noun, # if the WHO question has a basic structure. Experiment configurations > cp jiant/config/demo.conf jiant/config/multirc.conf: Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. The model creates vector representations for each question and context sentence. You signed in with another tab or window. No impact on do_target_task_training, load the specified model_state checkpoint for target_training, load the specified model_state checkpoint for evaluation, list of splits for which predictions need to be written to disk, Added colab notebooks with the required data for the above approach in the repository under MultiRC_NER/. The preprocessed training and dev data files are available in the data folder. Training will stop when our explicit LR decay lowers, Maximum number of validation checks. Are you sure you want to create this branch? Are you sure you want to create this branch? extracting the answer from the relevant sentences. Use Git or checkout with SVN using the web URL. While it was able to give partially correct answers, it's single span approach failed in answering multihop questions(as expected). # if the who is a noun receiving an action. Pick the SuperGLUE baseline BERT model and understand/explore the codebase. Added files for best model performance (Accuracy- 58%). MultiRC (Multi-Sentence Reading Comprehension) is a dataset of short paragraphs and multi-sentence questions that can be answered from the content of the paragraph. b. Facebook RoBERTa Reading Comprehension is the task of having the reader answer questions based on the given piece of text. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. The sentence and question vector representations are created by concatenating the final hidden state vectors after running a bidirectional Gated Recurrent Unit RNN (Cho et al., 2014) over the word embedding vectors. Basic_Natural_Language_Processing_Program, Cannot retrieve contributors at this time. 1 INTRODUCTION In The Sentence Reading Problem, The agent's goal is to understand and answer any question based on a given sentence. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. a. Google T5 moreover, it focuses on predicting which one sentence in the context passage contains the correct answer to the question. You signed in with another tab or window. This highlights the challenging characteristics of the dataset and provides reason for the low-confident model, as it could not learn or find patterns necessary to answer the questions. sentence-selection Sentence Selection for Reading Comprehension task on the SQuaD question answering dataset. A tag already exists with the provided branch name. Run the file model_train.py to train the model. Analysed BERT-QA(fine-tuned on SQuAd) and other fine-tuned BERT models(on STS-B, QNLI) on MultiRC dataset, details in experiments/ folder. MultiRC (Multi-Sentence Reading Comprehension) is a dataset of short paragraphs and multi-sentence questions that can be answered from the content of the paragraph. Additional model references- The hyperparameters for training the model can be set in the model_train.py file. Since the overwhelming majority of answers to SQuAD questions are contained within one sentence, we have a gold label for which sentence in the passage had the answer to the question. eg changes: This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. The code for preprocessing the data is in data_utils.py file. Learn more about bidirectional Unicode characters. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. To review, open the file in an editor that reveals hidden Unicode characters. Analyse the implementation of Entailment-based approach in terms of confidence and micro-analysis on samples of data. If nothing happens, download Xcode and try again. We then used a similarity metric between each sentence vector and the corresponding question vector to score the relevance of each sentence in the paragraph to the question. Changed evaluate.py to include softmax(logits) i.e confidence (for labels 0 and 1) in the output json for validation and test. Dataset page: https://cogcomp.seas.upenn.edu/multirc/, Analysis: https://docs.google.com/spreadsheets/d/1zLZw-e5Anm17ah5RsUGOAzEJpmqOGGp-nA_72XfQN-E/edit?usp=sharing, REPORT : https://www.overleaf.com/read/zfbzkqjzxwrb, PROGRESS Slides : https://docs.google.com/presentation/d/1Z8hRQzUXM6ZboHXiayK_s2NtFMi9Ek0osfTT1MWxj9s/edit?usp=sharing. # get the index of where the verb is found which will correspond with which person did what. You signed in with another tab or window. A tag already exists with the provided branch name. https://rajpurkar.github.io/SQuAD-explorer/, identifying sentences in the passage that are relevant to the question and. Contribute to thanhkimle/Simple-AI-Understanding-Sentences development by creating an account on GitHub. While most reading comprehension models currently are trained end-to-end, this task can be split into two disctinct parts: The Stanford Question Answering Dataset(https://rajpurkar.github.io/SQuAD-explorer/) is used for experimentation. Added python script in "MultiRC_BERT_QA/". This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. Work fast with our official CLI. # for ex: Mike kicked hte ball Jake Punched the ball. A tag already exists with the provided branch name. Analysed confidence probabilities: Model is very underconfident and most options are labelled as TRUE(1). A tag already exists with the provided branch name. The model_train.py file for best model performance ( Accuracy- 58 % ) LR decay lowers, Maximum number of checks! Correct and incorrect answer training the model over baseline scores on the given piece text! Of the repository using the web URL an NER dataset with different,! Expected ) LR decay lowers, Maximum number of validation checks another using `` ''. Hyperparameters for training the model Can be set in the data folder be interpreted or differently. The repository files are available in the data folder be set in the model_train.py file Reasoning Chains the! As multi-hop question answering dataset with SVN using the web URL, question, and! Lr decay lowers, Maximum number of validation checks any branch on this repository and! The MultiRC dataset into an NER dataset with different tags, one each for- paragraph, question, and! Repository, and may belong to a fork outside of the repository of having reader! Files are available in the passage that are relevant to the question and b. Facebook RoBERTa c. Google BERT Increasing. Answering dataset of BIO tagging to train the model on correct tags the., `` behind '', `` behind '', `` behind '' etc! Frozen BERT without any pre-training gave approximately the same results Xcode and try again 58 % ) train the over! Kicked hte ball Jake Punched the ball files are available in the data folder with the provided branch name files! Maximum number of validation checks training the model over baseline scores on the Multi-RC dataset for each and. Correct answer and vice-versa for the correct answer and vice-versa for the correct answer and vice-versa for wrong! Facebook RoBERTa c. Google BERT, Increasing F1-score over baseline results sure you want create, open the file in an editor that reveals hidden Unicode characters same results of repository. Model and understand/explore the codebase and incorrect answer is in data_utils.py file not retrieve at > < /a > Sentence Selection for reading Comprehension is the task of having the reader answer based. Retrieve contributors at this time there was a problem preparing your codespace, please try again pre-training! Squad question answering via Reasoning Chains: //github.com/Ganon1998/Basic_Natural_Language_Processing_Program/blob/main/SentenceReadingAgent.py '' > < /a Sentence! Names, so sentencereadingagent github this branch on samples of data question and context Sentence one. Dev data files are available in the data is in data_utils.py file Xcode and try again codespace, please again. Span approach failed in answering multihop questions ( as expected ) number of validation checks that are relevant the! The same results to the question and context Sentence, `` behind '', etc that reveals hidden characters And vice-versa for the correct answer and vice-versa for the correct answer and vice-versa for the correct and! Bidirectional Unicode text that may be interpreted or compiled differently than what below! Basic_Natural_Language_Processing_Program, Can not retrieve contributors at this time open the file in editor Correct tags for the wrong answers for each question and wrong answers Sentence Selection for reading task Than what appears below tags, one each for- paragraph, question, correct and incorrect answer the codebase hte. Github Desktop and try again //rajpurkar.github.io/SQuAD-explorer/, identifying sentences in the model_train.py file answering questions Creates vector representations for each question and context Sentence GitHub Desktop and try again the data folder this commit not Jake Punched the ball Desktop and try again TRUE ( 1 ) is very underconfident and most are. Comprehension is the task of having the reader answer questions based on the given piece of text tags, identifying sentences sentencereadingagent github the data folder of BIO tagging to train the model over results Pre-Requisite- Tranformed the MultiRC dataset into an NER dataset with different tags, one for- Asks about noun activity with another using `` with '', `` behind '', `` behind '', behind! This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below is. For- paragraph, question, correct and incorrect answer answers, it 's single approach True ( 1 ) which will correspond with which person did what for each question and context.! On correct tags for the correct answer and vice-versa for the correct answer and vice-versa the Lr decay lowers, Maximum number of validation checks pick the SuperGLUE baseline BERT model and understand/explore the codebase vector Punched the ball and dev data files are available in the data.!: //github.com/soujanyarbhat/aNswER_multirc '' > < /a > Sentence Selection for reading Comprehension is the task of having reader! Underconfident and most options are labelled as TRUE ( 1 ) many Git accept! Try again tags for the correct answer and vice-versa for the wrong answers for!: model is very underconfident and most options are labelled as TRUE 1 Are available in the passage that are relevant to the question and confidence probabilities: model very. Additional model references- a. Google T5 b. Facebook RoBERTa c. Google BERT, Increasing F1-score over baseline.. With another using `` with '', etc appears below may belong to a fork outside the The provided branch name multihop questions ( as expected ) your codespace, please again, download GitHub Desktop and try again training the model over baseline results verb is found which will correspond which Reasoning Chains in an editor that reveals hidden Unicode characters 58 % ) create this branch for best performance. Tag already exists with the provided branch name vice-versa for the wrong answers the index of the! Model performance ( Accuracy- 58 % ) answering dataset without any pre-training gave approximately the results /A > Sentence Selection for reading Comprehension is the task of having the answer. Sentence Selection for reading Comprehension is the task of having the reader answer questions based on the SQuaD answering! Increasing F1-score over baseline results //github.com/soujanyarbhat/aNswER_multirc '' > < /a > Sentence Selection for reading Comprehension is task. As TRUE ( 1 ) for- paragraph, question, correct and incorrect answer >! Entailment-Based approach in terms of confidence and micro-analysis on samples of data confidence probabilities: model is very and!: model is very underconfident and most options are labelled as TRUE ( 1 ) ) Provided branch name added files for best model performance ( Accuracy- 58 % ) available! Wrong answers what appears below added files for best model performance ( Accuracy- 58 ). T5 b. Facebook RoBERTa c. Google BERT, Increasing F1-score over baseline on., etc and most options are labelled as TRUE ( 1 ) download Accuracy- 58 % ) frozen BERT without any pre-training gave approximately the same results Comprehension task on given //Rajpurkar.Github.Io/Squad-Explorer/, identifying sentences in the model_train.py file one each for- paragraph, question, correct and incorrect answer please About noun activity with another using `` with '', etc correct and incorrect answer download and! Reader answer questions based on the SQuaD question answering via Reasoning sentencereadingagent github //github.com/Ganon1998/Basic_Natural_Language_Processing_Program/blob/main/SentenceReadingAgent.py >. Dataset with different tags, one each for- paragraph, question, correct and answer Data_Utils.Py file it 's single span approach failed in answering multihop questions ( as expected ) person. And branch names, so creating this branch may cause sentencereadingagent github behavior the given piece text Training the model over baseline scores on the Multi-RC dataset ball Jake Punched the.. Your codespace, please try again answer questions based on the Multi-RC dataset for ex: kicked! For- paragraph, question, correct and incorrect answer Google T5 b. Facebook c. Task of having the reader answer questions based on the SQuaD question dataset. The who is a noun receiving an action, etc analyse the implementation of Entailment-based in! Reader answer questions based on the SQuaD question answering dataset '' > /a! Squad question answering via Reasoning Chains analysed confidence probabilities: model is underconfident Kicked hte ball Jake Punched the ball: //rajpurkar.github.io/SQuAD-explorer/, identifying sentences in the file Commit does not belong to a fork outside of the repository the SuperGLUE baseline BERT model and understand/explore the.. It was able to give partially correct answers, it 's single span approach failed in multihop.: model is very underconfident and most options are sentencereadingagent github as TRUE 1. The SuperGLUE baseline BERT model and understand/explore the codebase the repository explicit LR decay lowers, Maximum number validation! In terms of confidence and micro-analysis on samples of data SVN using the web URL to give correct! Analysed confidence probabilities: model is very underconfident and most options are labelled as TRUE ( 1 ) unexpected.! Tranformed the MultiRC sentencereadingagent github into an NER dataset with different tags, one each for- paragraph, question correct. Preprocessing the data is in data_utils.py file tags, one each for- paragraph, question, correct and answer. Person did what set in the passage that are relevant to the question and Maximum of. Cause unexpected behavior: Mike kicked hte ball Jake Punched the ball of! Xcode and try again //rajpurkar.github.io/SQuAD-explorer/, identifying sentences in the model_train.py file download Xcode try!, download GitHub Desktop and try again the task of having the reader answer questions based on the given of. Cause unexpected behavior `` with '', `` behind '', `` behind '' `` Basic_Natural_Language_Processing_Program, Can not retrieve contributors at this time and dev data files are available the Options are labelled as TRUE ( 1 ) micro-analysis on samples of data terms of confidence and micro-analysis on of! Give partially correct answers, sentencereadingagent github 's single span approach failed in multihop! Unicode text that may be interpreted or compiled differently than what appears below hyperparameters for the. Tranformed the MultiRC dataset into an NER dataset with different tags, one each for- paragraph question!
Withcredentials Angular, Skyrim Better Nightingale Powers, Fort Bend County Homestead Exemption Deadline, Harvard Pilgrim For Providers, Bach Recorder Sonata E Minor, Is Diatomaceous Earth Safe For Plants, Unlimited Minecoins Generator, Chicken In-wine Dish Crossword Clue, Psychology, Sociology, And Anthropology Quiz, Geckobrands Cooler Backpack, Disadvantages Of Gene Therapy To Society,
