GPT 4

[cameronfen]

52 minutes ago, ValueArb said:

100% agreed. But the effective models are huge and require mass scale in service and storage. I am a developer but not an AI expert so I don’t really understand how to solve the problem.
 

Can you build a proprietary model with your own data, that relies on the huge public model to fill in gaps in your proprietary model?  Or can it extend the public model without writing back to it?

Well, all you need to do is copy the GPT-3 model both the neural network architecture and the weights and install it in an on-prem GPU.  I couldn't find the number for GPT-4, but you can fine-tune and run a GPT-3 model on as few as 1 NVIDIA A100 GPUs: https://blogs.oracle.com/research/post/oracle-first-to-finetune-gpt3-sized-ai-models-with-nvidia-a100-gpu#:~:text=Achieving GPT3 finetuning with as,3.9 tokens%2Fsec%2FGPU.  So openAI will train their model, give you the weights of the model to be used over 2 or 3 A100 GPUs, and whenever a developer needs some code completion, you run the on-prem GPT, which is no longer being trained but also OpenAI has no access to.  Every year when OpenAI comes out with a new model, the on-prem version gets updated.  Fine-tuning on custom data is doable, but is probably not necessary for models that are able to do writing completion or code completion.  

 

From the text, there are other startups that can run GPT-3 scale inference on as few as 8 A100s:

Quote

In terms of serving, DeepSpeed MII and Hugging Face are among the first to release their support for GPT3-sized model inference with a minimal hardware requirement of 8 NVIDIA A100 GPUs with 80GB memory in Sept 2022. In the case of 8 NVIDIA A100s at 40GB, an efficient inference further requires INT8 precision to meet the prohibitive memory need.