OpenAI Codex - The Model behind GitHub Copilot

OpenAI has released the research paper,

"Evaluating Large Language Models Trained on Code"

Which powers Codex, a highly improved Autoregressive language model than the 3rd generation Generative Pre-trained Transformer (GPT-3) language model.

Codex is much improved than GPT-3 due to its model been trained on a dataset that includes a much larger concentration of public source code from GitHub.

Codex has been fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities.

An evaluation harness for the HumanEval problem solving dataset from the research paper is also available on OpenAI Github repository.

"A distinct production version of Codex powers GitHub Copilot" says the paper

Which explains that the sucessor of Codex is used to train on code dataset from GitHub repositories for the GitHub Copilot project.

The paper concludes,

"We found that our models displayed strong performance on a dataset of human-written problems with difficulty level comparable to easy interview problems"

About data collection, the paper says,

"Our training dataset was collected in May 2020 from 54 million public software repositories hosted on GitHub, containing 179 GB of unique Python files under 1 MB. We filtered out files which were likely auto-generated, had average line length greater than 100, had maximum line length greater than 1000, or contained a small percentage of alphanumeric characters. After filtering, our final dataset totaled 159 GB"

Check out the following resources,

Link to the paper : https://arxiv.org/abs/2107.03374

OpenAI Github repository : https://github.com/openai/human-eval

Personal Blog @ danyson.github.io