Open Source Language Model Named Dolly 2.0 Trained Similarly To ChatGPT

Apr 15, 2023 | SEO News Feeds | 0 comments



SEO Content Writing Service

Databricks announced the release of the first open source instruction-tuned language model, called Dolly 2.0. It was trained using similar methodology as InstructGPT but with a claimed higher quality dataset that is 100% open source.

This model is free to use, including for commercial purposes, because every part of the model is 100% open source.

Open Source Instruction Training

What makes ChatGPT able to follow directions is the training it receives using techniques outlined in the InstructGPT research paper.

The breakthrough discovered with InstructGPT is that language models don’t need larger and larger training sets.

By using human evaluated question and answer training, OpenAI was able to train a better language model using one hundred times fewer parameters than the previous model, GPT-3.

Databricks used a similar approach to create prompt and response dataset called they call databricks-dolly-15k.

Their prompt/response dataset was created without scraping web forums or Reddit.

databricks-dolly-15k is a dataset created by Databricks employees, a 100% original, human generated 15,000 prompt and response pairs designed to train the Dolly 2.0 language model in the same way that ChatGPT model was created with InstructGPT.

The GitHub page for the dataset explains how they did it:

“databricks-dolly-15k is an open source dataset of instruction-following records used in training databricks/dolly-v2-12b that was generated by thousands of Databricks employees in several of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.

…Databricks employees were invited to create prompt / response pairs in each of eight different instruction categories, including the seven outlined in the InstructGPT paper, as well as an open-ended free-form category.

Attorney Websites For Sale 4ebusiness Media Group

The contributors were instructed to avoid using information from any source on the web with the exception of Wikipedia (for particular subsets of instruction categories), and explicitly instructed to avoid using generative AI in formulating instructions or responses. Examples of each behavior were provided to motivate the types of questions and instructions appropriate to each category.

Halfway through the data generation process, contributors were given the option of answering questions posed by other contributors. They were asked to rephrase the original question and only select questions they could be reasonably expected to answer correctly.”

Databricks claims that this may be the very first human generated instruction dataset created to train a language model to follow instructions, just like ChatGPT does.

The challenge was to create a 100% original dataset that had zero ties to ChatGPT or any other source with a restrictive license.

Employees were incentivized by a contest to contribute to generating the 15,000 prompt/responses along seven categories of tasks such as brainstorming, classification, and creative writing.

Databricks asserts that the databricks-dolly-15k training set may be superior to the dataset used to train ChatGPT.

They note that although their dataset is smaller than the one used to train the Stanford Alpaca model, their model performed better because their data is higher quality.

They write:

“Dolly 2.0 model, based on EleutherAI’s pythia-12b, exhibited high-quality instruction following behavior. In hindsight, this isn’t surprising.

Many of the instruction tuning datasets released in recent months contain synthesized data, which often contains hallucinations and factual errors.

databricks-dolly-15k, on the other hand, is generated by professionals, is high quality, and contains long answers to most tasks.

…we don’t expect Dolly to be state-of-the-art in terms of effectiveness.

However, we do expect Dolly and the open source dataset will act as the seed for a multitude of follow-on works, which may serve to bootstrap even more powerful language models.”

Limitations to the Dataset

The GitHub page for the dataset acknowledges that there may be some shortcomings to the dataset.

Wikipedia data was used for some of the training in the context of creating prompts and responses. Thus, it’s possible that whatever bias contained in Wikipedia may end up reflected within the resulting dataset.

Some of the employees who worked to create the dataset were not native speakers of English, which could introduce some anomalies in the dataset.

The demographic makeup of the employees who created the dataset may itself influence the dataset to contain biases that are peculiar to those employees.

Despite those possible shortcomings in the dataset, Databricks expressed that theirs is of a higher quality.

Additionally, Dolly 2.0 is meant to serve as a starting point for others to create and innovate even better versions.

Databricks Insists that Open Source AI Is Better

One of the motivations behind creating Dolly 2.0 is that users of the data can own the models they created and can better safeguard their data by not having to share it with a third party.

They also believe that AI safety should not be concentrated in the hands of three large corporations but spread out among all the stakeholders.

Open source is picking up momentum and it will be interesting to see where this industry is at within the next two years.

More information on where to download the Dolly 2.0 model and how to use it can be found in their announcement.

Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM

Featured image by Shutterstock/Kamil Macniak

Source link


Anxiety Stress Management

Live a Life of Contentment eBook We all want to be satisfied, even though we know some people who will never be that way, and others who see satisfaction as a foreign emotion that they can’t hope to ever feel.

Newspaper Ads Canyon Crest CA

Click To See Full Page Ads

Click To See Half Page Ads

Click To See Quarter Page Ads

Click To See Business Card Size Ads

If you have questions before you order, give me a call @ 951-235-3518 or email @ canyoncrestnewspaper@gmail.com Like us on Facebook Here

You May Also Like

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Contact Us

Contact Us

Personal Injury Attorney

Websites For Sale Personal Injury Attorneys

Criminal Defense Attorneys

Websites For Sale Criminal Defense Attorney

Bankruptcy Attorneys

Websites For Sale Bankruptcy Attorneys

General Practice Attorneys

Websites For Sale General Practice Attorneys

Family Attorneys

Websites For Sale Family Attorneys

Corporate Attorneys

Websites For Sale Corporate Attorneys

Home Privacy Policy Terms Of Use Anti Spam Policy Contact Us Affiliate Disclosure Amazon Affiliate Disclaimer DMCA Earnings Disclaimer