Prompt-learning is the latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks

An Open-Source Framework for Prompt-learning.


OverviewInstallationHow To UseDocs

Overview

Prompt-learning is the latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks, which modifies the input text with a textual template and directly uses PLMs to conduct pre-trained tasks. This library provides a standard, flexible and extensible framework to deploy the prompt-learning pipeline. OpenPrompt supports loading PLMs directly from huggingface transformers. In the future, we will also support PLMs implemented by other libraries.

What Can You Do via OpenPrompt?

demo

  • Use the implementations of current prompt-learning approaches.* We have implemented various of prompting methods, including templating, verbalizing and optimization strategies under a unified standard. You can easily call and understand these methods.
  • Design your own prompt-learning work. With the extensibility of OpenPrompt, you can quickly practice your prompt-learning ideas.

Installation

Using Git

Clone the repository from github:

git clone https://github.com/thunlp/OpenPrompt.git
cd OpenPrompt
pip install -r requirements.txt
python setup.py install

Modify the code

python setup.py develop

Use OpenPrompt

Base Concepts

A Prompt class contains a (or multiple) Template and a (or multiple) Verbalizer, where the Template class is defined to wrap the original input with templates, and the Verbalizer class is to construct a projection between labels and target words in the current vocabulary.

A PromptModel class combines the Template, Verbalizer and PLM, practically participating in training and inference.

Introduction by a Simple Example

With the modularity and flexibility of OpenPrompt, you can easily develop a prompt-learning pipeline.

Step 1: Define a task

The first step is to determine the current NLP task, think about what’s your data looks like and what do you want from the data! That is, the essence of this step is to determine the classses and the InputExample of the task. For simplicity, we use Sentiment Analysis as an example. tutorial_task.

from openprompt.data_utils import InputExample
classes = [ # There are two classes in Sentiment Analysis, one for negative and one for positive
    "negative",
    "positive"
]
dataset = [ # For simplicity, there's only two examples
    # text_a is the input text of the data, some other datasets may have multiple input sentences in one example.
    InputExample(
        guid = 0,
        text_a = "Albert Einstein was one of the greatest intellects of his time.",
    ),
    InputExample(
        guid = 1,
        text_a = "The film was badly made.",
    ),
]

Step 2: Define a Pre-trained Language Models (PLMs) as backbone.

Choose a PLM to support your task. Different models have different attributes, we encourge you to use OpenPrompt to explore the potential of various PLMs. OpenPrompt is compatible with models on huggingface.

from openprompt.plms import get_model_class
model_class = get_model_class(plm_type = "bert")
model_path = "bert-base-cased"
bertConfig = model_class.config.from_pretrained(model_path)
bertTokenizer = model_class.tokenizer.from_pretrained(model_path)
bertModel = model_class.model.from_pretrained(model_path)

Step 3: Define a Template.

Template is a modifier of the original input text, which is also one of the most important modules in prompt-learning. 

", "It", "was", " "], tokenizer = bertTokenizer, ) ">
from openprompt.prompts import ManualTemplate
promptTemplate = ManualTemplate(
    text = ["
     
      "
     , "It", "was", "
     
      "
     ],
    tokenizer = bertTokenizer,
)

Step 4: Define a Verbalizer

Verbalizer is another important (but not neccessary) in prompt-learning,which projects the original labels (we have defined them as classes, remember?) to a set of label words. Here is an example that we project the negative class to the word bad, and project the positive class to the words good, wonderful, great.

from openprompt.prompts import ManualVerbalizer
promptVerbalizer = ManualVerbalizer(
    classes = classes,
    label_words = {
        "negative": ["bad"],
        "positive": ["good", "wonderful", "great"],
    },
    tokenizer = bertTokenizer,
)

Step 5: Combine them into a PromptModel

Given the task, now we have a PLM, a Template and a Verbalizer, we combine them into a PromptModel. Note that although the example naively combine the three modules, you can actually define some complicated interactions among them.

from openprompt import PromptForClassification
promptModel = PromptForClassification(
    template = promptTemplate,
    model = bertModel,
    verbalizer = promptVerbalizer,
)

Please refer to our documentation for more details.

Datasets

We provide a series of download scripts in the dataset/ folder, feel free to use them to download benchmarks.

Citation

We are working on the technical report...

Contributors

We thank all the contributors to this project, more contributors are welcome!

Ning Ding, Shengding Hu, Weilin Zhao.

Owner
THUNLP
Natural Language Processing Lab at Tsinghua University
THUNLP
Comments
  • An error occurred while using the latest version (1.0.0)

    An error occurred while using the latest version (1.0.0)

    When I use ptr_template in the latest version.The following error occurred. TypeError: __init__() got an unexpected keyword argument 'placeholder_mapping' Version 0.1.1 does not have this problem.

  • Failed to run the demo in `tutorial`

    Failed to run the demo in `tutorial`

    command: python tutorial/1.1_mixed_template.py

    output:

      File "tutorial/1.1_mixed_template.py", line 94, in <module>
        logits = prompt_model(inputs)
      File "/home/h/anaconda3/envs/openprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "/home/h/work/OpenPrompt/openprompt/pipeline_base.py", line 241, in forward
        outputs = self.verbalizer.gather_outputs(outputs)
    TypeError: gather_outputs() takes 1 positional argument but 2 were given
    
  • no attribute 'tokenize_one_example'

    no attribute 'tokenize_one_example'

    Hi,

    thank you for your amazing work to ease the users of prompt learning.

    I tried to implement the LMBFF tutorial and happened to meet this error:

    Traceback (most recent call last):
      File "run_lmbff.py", line 116, in <module>
        dataloader = PromptDataLoader(dataset['train'], template, template_generate_tokenizer, template_tokenizer_wrapper, batch_size=len(dataset['train']), decoder_max_length=128) # register all data at once
      File "/home/oryza/playground/OpenPrompt/openprompt/pipeline_base.py", line 101, in __init__
        self.tokenize()
      File "/home/oryza/playground/OpenPrompt/openprompt/pipeline_base.py", line 137, in tokenize
        inputfeatures = InputFeatures(**self.tokenizer_wrapper.tokenize_one_example(wrapped_example, self.teacher_forcing), **wrapped_example[1]).to_tensor()
    AttributeError: 'T5Tokenizer' object has no attribute 'tokenize_one_example'
    

    This is my pip list:

    Package            Version   Editable project location
    ------------------ --------- ---------------------------------
    aiohttp            3.8.1
    aiosignal          1.2.0
    async-timeout      4.0.2
    asynctest          0.13.0
    attrs              21.4.0
    certifi            2021.10.8
    charset-normalizer 2.0.12
    click              8.1.2
    datasets           2.0.0
    dill               0.3.4
    filelock           3.6.0
    frozenlist         1.3.0
    fsspec             2022.3.0
    huggingface-hub    0.5.1
    idna               3.3
    importlib-metadata 4.11.3
    joblib             1.1.0
    multidict          6.0.2
    multiprocess       0.70.12.2
    nltk               3.7
    numpy              1.21.5
    openprompt         1.0.0     /home/oryza/playground/OpenPrompt
    packaging          21.3
    pandas             1.3.5
    pip                22.0.4
    protobuf           3.20.0
    pyarrow            7.0.0
    pyparsing          3.0.8
    python-dateutil    2.8.2
    pytz               2022.1
    PyYAML             6.0
    regex              2022.3.15
    requests           2.27.1
    responses          0.18.0
    rouge              1.0.0
    sacremoses         0.0.49
    scikit-learn       1.0.2
    scipy              1.7.3
    sentencepiece      0.1.96
    setuptools         41.2.0
    six                1.16.0
    sklearn            0.0
    tensorboardX       2.5
    threadpoolctl      3.1.0
    tokenizers         0.10.3
    torch              1.11.0
    tqdm               4.64.0
    transformers       4.10.0
    typing_extensions  4.1.1
    urllib3            1.26.9
    xxhash             3.0.0
    yacs               0.1.8
    yarl               1.7.2
    zipp               3.8.0
    

    Do you have any idea about the error? I read in another thread about installing SentencePiece to solve this problem but my sentencepiece is already there.

    Thank you in advance!

    Best, Oryza

  • bug:TypeError: _forward_unimplemented() got an unexpected keyword argument 'output_hidden_states'

    bug:TypeError: _forward_unimplemented() got an unexpected keyword argument 'output_hidden_states'

    File "/workspace/knowledgegraphcommon/business/text_classification/prompt/text_classification_prompt.py", line 185, in train logits = self.prompt_model(inputs) File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/workspace/OpenPrompt/openprompt/pipeline_base.py", line 263, in forward outputs = self.prompt_model(batch) File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/workspace/OpenPrompt/openprompt/pipeline_base.py", line 185, in forward outputs = self.plm(**input_batch, output_hidden_states=True) File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) TypeError: _forward_unimplemented() got an unexpected keyword argument 'output_hidden_states'

  • Question about updating the repository

    Question about updating the repository

    How do you keep this repository up to date? Do I need to clone the entire library every time and update it again as follows:

    git clone https://github.com/thunlp/OpenPrompt.git
    cd OpenPrompt
    pip install -r requirements.txt
    python setup.py install
    
  • TypeError: __init__() missing 1 required positional argument: 'tokenizer_wrapper_class'

    TypeError: __init__() missing 1 required positional argument: 'tokenizer_wrapper_class'

    When going through the tutorial

    In step 6, raised the errors below:

    data_loader = PromptDataLoader( ... dataset = dataset, ... tokenizer = bertTokenizer, ... template = promptTemplate, ... ) Traceback (most recent call last): File "", line 1, in TypeError: init() missing 1 required positional argument: 'tokenizer_wrapper_class'

  • Shape mismatch in auto_v - tutorial/ 3.1_LMBFF.py

    Shape mismatch in auto_v - tutorial/ 3.1_LMBFF.py

    When running this tutorial script as is, a shape mismatch appears when trying to auto generate the verbaliser. The code run is as it is on github. Error output below. Without knowing the fine details of the auto verbalizer, its hard to deduce where exactly this is going wrong. Any thoughs?

    load model... [[{'text': 'nothing happens , and it happens to flat characters .', 'loss_ids': 0, 'shortenable_ids': 1}, {'text': '<extra_id_0>', 'loss_ids': 1, 'shortenable_ids': 0}, {'text': ' terrible', 'loss_ids': 0, 'shortenable_ids': 0}, {'text': '<extra_id_1>', 'loss_ids': 1, 'shortenable_ids': 0}, {'text': '.', 'loss_ids': 0, 'shortenable_ids': 0}], {'guid': 'train-0', 'label': 0}] performing auto_v... tokenizing: 32it [00:00, 798.56it/s] Traceback (most recent call last): File "tutorial/3.1_LMBFF.py", line 186, in label_words_list = verbalizer_generator.generate() File "/home/niallt/venvs/nlp_prompts/lib/python3.6/site-packages/openprompt/prompts/prompt_generator.py", line 447, in generate self.label_words_ids = self._find_verbalizer() File "/home/niallt/venvs/nlp_prompts/lib/python3.6/site-packages/openprompt/prompts/prompt_generator.py", line 392, in _find_verbalizer label_words = self._get_top_words() File "/home/niallt/venvs/nlp_prompts/lib/python3.6/site-packages/openprompt/prompts/prompt_generator.py", line 414, in _get_top_words scores = self.probs_buffer[self.labels_buffer==label_id].mean(axis=0).cpu().numpy() IndexError: The shape of the mask [32] at index 0 does not match the shape of the indexed tensor [448, 50265] at index 0

  • Model output without gard.

    Model output without gard.

    I followed the tutorial code 0_basic.py and modified the Template, Verbalizer, and then defined the model, but in the end my model output had no gradient. The code show as below:

    plm, tokenizer, model_config, WrapperClass = load_plm("t5", "t5-base")
    
    promptTemplate = ManualTemplate(
        text = '{"placeholder": "text_a"} {"text": "In this sentence,"} {"placeholder": "text_b"} {"text": "is a"} {"mask"} ',
        tokenizer = tokenizer,
    )
    
    promptVerbalizer = ManualVerbalizer(tokenizer,
        num_classes=2,
        #classes = classes,
        label_words = [
            ["person"],
            ["NA"]
        ]
    )
    
    prompt_model = PromptForClassification(plm=plm,template=promptTemplate, verbalizer=promptVerbalizer, freeze_plm=False)
    

    the output of the model:

    loss_func = torch.nn.CrossEntropyLoss()
    logits = prompt_model(inputs)
    labels = inputs['label']
    loss = loss_func(logits, labels)
    
    
    ---> 34         loss.backward()
         35         tot_loss += loss.item()
         36         optimizer.step()
    
    ~/anaconda3/envs/prompt-dev/lib/python3.7/site-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
        305                 create_graph=create_graph,
        306                 inputs=inputs)
    --> 307         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
        308 
        309     def register_hook(self, hook):
    
    ~/anaconda3/envs/prompt-dev/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
        154     Variable._execution_engine.run_backward(
        155         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    --> 156         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
        157 
        158 
    
    RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
    
  • detail about prefix tuning

    detail about prefix tuning

    In prefix_tuning_template.py file https://github.com/thunlp/OpenPrompt/blob/675545ce1f946aa186efda8e8640dbc29fd1159f/openprompt/prompts/prefix_tuning_template.py#L207

    The above code pad the attention_mask for extra prompt tokens.

    Why the function 'torch.zeros' is used here? Should we use 'torch.ones' here?

  • How to config `num_classes` parameter of Verbalizer?

    How to config `num_classes` parameter of Verbalizer?

    In 0_basic.py:

    myverbalizer = ManualVerbalizer(tokenizer, num_classes=2, 
                            label_words=[["yes"], ["no"], ["maybe"]])
    

    Why could len(label_words) > num_classes? what if len(label_words) < num_classes?

  • how to use bart on openprompt?

    how to use bart on openprompt?

    I'd like to use prefix tuning template and BARTmodel (As its paper does) , but found this template only supports T5 and GPT2. How to use BART in this work? And I found that the attention module of the transformer BART has been modified in original paper. If I want to use BART here, do I also need to modify it?

  • Applying OpenPrompt

    Applying OpenPrompt

    Hi, I intend to apply the generation framework of OpenPrompt to NER task, based on your newly released dataset: https://ningding97.github.io/fewnerd/

    Is there any avaliable example or related papers within this framework ? I havn't found yet.

    Thanks in advance.

  • Multiple mask

    Multiple mask

    In generation_verbalizer.py, there is an example says that we can define a multiple-mask template and verbalizer:

        >>> input_example = InputExample(text_a = "Can fish run?", meta={"answer":"no", "explanation": "The fish have no legs"}, label=0)
        >>> template = "{'placeholder':'text_a'} answer: {'mask'} explanation: {'mask'}"
        The verbalizer can be:
        >>> label_words = {0:["no", "{'meta':'explanation'}"], 1:["yes", "{'meta':'explanation'}"]}
        >>> verbalizer = GenerationVerbalizer(tokenizer, classes=None, is_rule=True, label_words=label_words)
    

    however, in the on_label_words_set function (in generation_verbalizer.py 102-115 link), it seems like only the first element of each class (i.e., no and yes in this case) would be pass into the self.label_words:

            if isinstance(self.label_words[0], list):
                self.label_words = [x[0] for x in self.label_words]
    

    In this case, the tgt_text will only contain the first element, then it would raise an error when constructing data_loader:

    tokenizing: 0it [00:00, ?it/s]
    Traceback (most recent call last):
      File "prompt_learning_5-2-2-1.py", line 135, in <module>
        truncate_method="tail")
      File "/home/cike/anaconda3/envs/logic2nl/lib/python3.6/site-packages/openprompt/pipeline_base.py", line 101, in __init__
        self.tokenize()
      File "/home/cike/anaconda3/envs/logic2nl/lib/python3.6/site-packages/openprompt/pipeline_base.py", line 137, in tokenize
        inputfeatures = InputFeatures(**self.tokenizer_wrapper.tokenize_one_example(wrapped_example, self.teacher_forcing), **wrapped_example[1]).to_tensor()
      File "/home/cike/anaconda3/envs/logic2nl/lib/python3.6/site-packages/openprompt/plms/seq2seq.py", line 77, in tokenize_one_example
        print(tgt_text[num_mask_token_used])
    IndexError: list index out of range
    

    I am not familiar with prompt learning, What do I miss? Can you help me out please? Thanks a lot.

  • Use 2.1_conditional_generation.py , after fine-tuning, it only generates the same char.  Why ?

    Use 2.1_conditional_generation.py , after fine-tuning, it only generates the same char. Why ?

    use 2.1_conditional_generation.py in datasets/CondGen/webnlg_2017/

    generated txt: ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

  • SoftVerbalizer not frozen in PromptForClassification pipeline

    SoftVerbalizer not frozen in PromptForClassification pipeline

    According to the SoftVerbalizer script and my general understanding of what is desired in a frozen PLM training setting, the grouped_parameters_1 of the SoftVerbalizer should be frozen. However, in the current pipeline this does not happen.

    I will do my best to showcase what I believe to be a bug, whereby the PLM is frozen but the SoftVerbalizer is not.

    Below follows most of the OpenPrompt tutorial on the repos readme page.

    from openprompt.data_utils import InputExample
    classes = [ # There are two classes in Sentiment Analysis, one for negative and one for positive
        "negative",
        "positive"
    ]
    dataset = [ # For simplicity, there's only two examples
        # text_a is the input text of the data, some other datasets may have multiple input sentences in one example.
        InputExample(
            guid = 0,
            text_a = "Albert Einstein was one of the greatest intellects of his time.",
        ),
        InputExample(
            guid = 1,
            text_a = "The film was badly made.",
        ),
    ]
    
    
    from openprompt.prompts import ManualTemplate, SoftVerbalizer
    promptTemplate = ManualTemplate(
        text = '{"placeholder":"text_a"} It was {"mask"}',
        tokenizer = tokenizer,
    )
    
    # setup the soft verbalizer
    promptVerbalizer = SoftVerbalizer(tokenizer, plm, num_classes=2)
    
    
    # instantiate the PromptForClassification model
    
    from openprompt import PromptForClassification
    
    # model with no freezing of the plm
    promptModel = PromptForClassification(
        template = promptTemplate,
        plm = plm,
        verbalizer = promptVerbalizer,
        freeze_plm = True
    )
    

    After instantiating the promptModel, the PLM will have its paramters require_grad set to False. Now we can look at the number of tunable parameters - i.e. those in the whole prompt model that require_grad

    # check number of params that require_grad
    
    def get_n_trainable_params(model):    
    
        
        # all trainable
        num_total_trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
        
        # split into the plm and classisifcation head
        num_plm_trainable = sum(p.numel() for p in model.plm.parameters() if p.requires_grad)
        
        # template trainable
        try:
            num_template_trainable = sum(p.numel() for p in model.template.soft_embedding.parameters() if p.requires_grad)
        except:
            num_template_trainable = 0
        
        # verbalizer trainable 
        num_verbalizer_trainable = sum(p.numel() for p in model.verbalizer.parameters() if p.requires_grad)
        
        # assert sum of the two = total
        assert num_plm_trainable+num_template_trainable+num_verbalizer_trainable == num_total_trainable
        
        print(f"Number of trainable parameters of PLM: {num_plm_trainable}\n")
        print('#'*50)
        print(f"Number of trainable parameters of template: {num_template_trainable}\n")
        print('#'*50)
        print(f"Number of trainable parameters of verbalizer: {num_verbalizer_trainable}\n")
        print('#'*50)
        print(f"Total number of trainable parameters of whole model: {num_total_trainable}")
        print(f"Verbalizer grouped_parameters_1 require_grad: {model.verbalizer.group_parameters_1[0].requires_grad}")
    
    
    
    get_n_trainable_params(promptModel)
    
    Number of trainable parameters of PLM: 0
    
    ##################################################
    Number of trainable parameters of template: 0
    
    ##################################################
    Number of trainable parameters of verbalizer: 622660
    
    ##################################################
    Total number of trainable parameters of whole model: 622660
    
    Verbalizer grouped_parameters_1 require_grad: True
    
    

    But the verbalizer in this case has all parameters with requires_grad = True - including grouped_parameters_1.

    If you then re-intilialize the SoftVerbalizer and promptModel, with the now frozen PLM, it appears how it should be.

    
    promptVerbalizer_2 = SoftVerbalizer(tokenizer, plm, num_classes=2)
    promptModel_2 = PromptForClassification(
        template = promptTemplate,
        plm = plm,
        verbalizer = promptVerbalizer_2,
        freeze_plm = True
    )
    
    get_n_trainable_params(promptModel_2)
    
    Number of trainable parameters of PLM: 0
    
    ##################################################
    Number of trainable parameters of template: 0
    
    ##################################################
    Number of trainable parameters of verbalizer: 1536
    
    ##################################################
    Total number of trainable parameters of whole model: 1536
    
    Verbalizer grouped_parameters_1 require_grad: False
    

    This is a huge difference in the number of trainable parameters and I wonder if the latter scenario is actually the desired one in the frozen PLM setting?

    See: https://github.com/thunlp/OpenPrompt/blob/4ba7cb380e7b42c19d566e9836dce7efdb2cc235/openprompt/prompts/soft_verbalizer.py#L82

BMInf (Big Model Inference) is a low-resource inference package for large-scale pretrained language models (PLMs).
BMInf (Big Model Inference) is a low-resource inference package for large-scale pretrained language models (PLMs).

BMInf (Big Model Inference) is a low-resource inference package for large-scale pretrained language models (PLMs).

May 16, 2022
May 11, 2022
BPEmb is a collection of pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) and trained on Wikipedia.

BPEmb is a collection of pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) and trained on Wikipedia. Its intended use is as input for neural models in natural language processing.

May 10, 2022
RoNER is a Named Entity Recognition model based on a pre-trained BERT transformer model trained on RONECv2

RoNER RoNER is a Named Entity Recognition model based on a pre-trained BERT transformer model trained on RONECv2. It is meant to be an easy to use, hi

Apr 28, 2022
Must-read papers on improving efficiency for pre-trained language models.

Must-read papers on improving efficiency for pre-trained language models.

Apr 21, 2022
The repository for the paper: Multilingual Translation via Grafting Pre-trained Language Models

Graformer The repository for the paper: Multilingual Translation via Grafting Pre-trained Language Models Graformer (also named BridgeTransformer in t

May 20, 2022
Chinese Pre-Trained Language Models (CPM-LM) Version-I

CPM-Generate 为了促进中文自然语言处理研究的发展,本项目提供了 CPM-LM (2.6B) 模型的文本生成代码,可用于文本生成的本地测试,并以此为基础进一步研究零次学习/少次学习等场景。[项目首页] [模型下载] [技术报告] 若您想使用CPM-1进行推理,我们建议使用高效推理工具BMI

May 21, 2022
PyTorch Implementation of "Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging" (Findings of ACL 2022)

Feature_CRF_AE Feature_CRF_AE provides a implementation of Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

Apr 29, 2022
Guide to using pre-trained large language models of source code
Guide to using pre-trained large language models of source code

Large Models of Source Code I occasionally train and publicly release large neural language models on programs, including PolyCoder. Here, I describe

May 23, 2022
Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple
Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple

Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple

May 17, 2022
[Preprint] Escaping the Big Data Paradigm with Compact Transformers, 2021
[Preprint] Escaping the Big Data Paradigm with Compact Transformers, 2021

Compact Transformers Preprint Link: Escaping the Big Data Paradigm with Compact Transformers By Ali Hassani[1]*, Steven Walton[1]*, Nikhil Shah[1], Ab

May 12, 2022
A design of MIDI language for music generation task, specifically for Natural Language Processing (NLP) models.

MIDI Language Introduction Reference Paper: Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions: code This

Dec 27, 2021
DziriBERT: a Pre-trained Language Model for the Algerian Dialect
DziriBERT: a Pre-trained Language Model for the Algerian Dialect

DziriBERT is the first Transformer-based Language Model that has been pre-trained specifically for the Algerian Dialect.

May 18, 2022
Implementation of Natural Language Code Search in the project CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

CodeBERT-Implementation In this repo we have replicated the paper CodeBERT: A Pre-Trained Model for Programming and Natural Languages. We are interest

May 3, 2022
An easy-to-use framework for BERT models, with trainers, various NLP tasks and detailed annonations

FantasyBert English | 中文 Introduction An easy-to-use framework for BERT models, with trainers, various NLP tasks and detailed annonations. You can imp

May 20, 2022
Code associated with the "Data Augmentation using Pre-trained Transformer Models" paper

Data Augmentation using Pre-trained Transformer Models Code associated with the Data Augmentation using Pre-trained Transformer Models paper Code cont

May 13, 2022
A high-level yet extensible library for fast language model tuning via automatic prompt search

ruPrompts ruPrompts is a high-level yet extensible library for fast language model tuning via automatic prompt search, featuring integration with Hugg

Apr 27, 2022
:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

(Framework for Adapting Representation Models) What is it? FARM makes Transfer Learning with BERT & Co simple, fast and enterprise-ready. It's built u

May 14, 2022