A LLM for Code Generation using the Glaive Code Assistant dataset
LLM trains sucessfully, trianing loss is at a 1.1, code generated sucessfully completes prompted task
Use custom tokenizer instead of the gpt2 tokenizer.
This project was done in part by Andrej Karpathy's lecture on GPT architecture, as well as A Survey of Large Language Models (Wayne Xin Zhao et al.) and Attention Is All You Need (Ashish Vaswani et al.)
-
✅ Code Generation with LLM: Use an LLM to generate code based on natural language descriptions. The LLM can be trained or fine-tuned on a dataset of code snippets paired with their descriptions to understand how to translate user input into syntactically correct code.
-
Code Refinement using GANs: Employ a GAN where the generator is used to produce variations of the initial code generated by the LLM, and the discriminator is used to evaluate the quality of the code. The discriminator checks for functionality, optimization, and how well the code meets the requirements specified in the user's description.