How to Run Llama 3.1 Locally Using Python and Hugging Face Dev Community

How to Run Llama-3.1🦙 Locally Using Python🐍 and Hugging Face 🤗 - DEV Community #

Excerpt #
Introduction The latest Llama🦙 (Large Language Model Meta AI) 3.1 is a powerful AI model…

Introduction #

The latest Llama🦙 (Large Language Model Meta AI) 3.1 is a powerful AI model developed by Meta AI that has gained significant attention in the natural language processing (NLP) community. It is the most capable open-source llm till date. In this blog, I will guide you through the process of cloning the Llama 3.1 model from Hugging Face🤗 and running it on your local machine using Python. After which you can integrate it in any AI project.

Prerequisites #

Python 3.8 or higher installed on your local machine
Hugging Face Transformers library installed (pip install transformers)
Git installed on your local machine
A Hugging Face account

Step 1: Get access to the model #

Click here https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct to open the official hugging face repository of Meta’s Llama-3.1-8B-Instruct (you can use other llama 3.1 models in the same way).

At the beginning you should be seeing this:

Submit the form below to get access of the model

Once you see “You have been granted access to this model”, you are good to go…

Step 2: Create an ACCESS_TOKEN #

Go to “Settings” (Bottom right corner of the below image):

Go to “Access Tokens” click “Create new token”(upper right corner of the image):

Give read and write permissions and select the repo as shown:

Copy the token and place it somewhere safe and secure as it will be needed in the future.(note: once you copy it you cannot copy it again, so if you anyhow forget the key, you have to create a new one to begin with :))

Step 3: Clone the LLaMA 3.1 Model #

Now run the following command on your favorite terminal.
The ACCESS_TOKEN is the one you copied and the <huggingface-user-name> is the username of your hugging face account.

git clone https://&lt;huggingface-user-name&gt;:&lt;ACCESS_TOKEN&gt;@huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct

This can take a lot of time depending on your internet speed.

Step 4: Install Required Libraries #

Once the cloning is done, go to the cloned folder and install all the dependencies from the requirements.txt. (you can create an virtual-environment using conda(recommended) or virtualenv)

Using conda:

<span>cd </span>Meta-Llama-3.1-8B-Instruct
conda <span>install</span> <span>--yes</span> <span>--file</span> requirements.txt

Using pip:

<span>cd </span>Meta-Llama-3.1-8B-Instruct
pip <span>install</span> <span>-r</span> requirements.txt

Step 5: Run the Llama 3.1 Model #

Create a new Python file (e.g., test.py) and paste the location of the model repository you just cloned as the model_id (such as, "D:\\Codes\\NLP\\Meta-Llama-3.1-8B-Instruct"). Here is an example:

<span>import</span> <span>transformers</span>
<span>import</span> <span>torch</span>

<span>## Here you paste your cloned repos location
</span><span>model_id</span> <span>=</span> <span>"</span><span>D:</span><span>\\</span><span>Codes</span><span>\\</span><span>NLP</span><span>\\</span><span>Meta-Llama-3.1-8B-Instruct</span><span>"</span> 

<span>pipeline</span> <span>=</span> <span>transformers</span><span>.</span><span>pipeline</span><span>(</span>
    <span>"</span><span>text-generation</span><span>"</span><span>,</span>
    <span>model</span><span>=</span><span>model_id</span><span>,</span>
    <span>model_kwargs</span><span>=</span><span>{</span><span>"</span><span>torch_dtype</span><span>"</span><span>:</span> <span>torch</span><span>.</span><span>bfloat16</span><span>},</span>
    <span>device_map</span><span>=</span><span>"</span><span>auto</span><span>"</span><span>,</span>
<span>)</span>

<span>messages</span> <span>=</span> <span>[</span>
    <span>{</span><span>"</span><span>role</span><span>"</span><span>:</span> <span>"</span><span>user</span><span>"</span><span>,</span> <span>"</span><span>content</span><span>"</span><span>:</span> <span>"</span><span>Who are you?</span><span>"</span><span>},</span>
<span>]</span>

<span>outputs</span> <span>=</span> <span>pipeline</span><span>(</span>
    <span>messages</span><span>,</span>
    <span>max_new_tokens</span><span>=</span><span>256</span><span>,</span>
<span>)</span>
<span>print</span><span>(</span><span>outputs</span><span>[</span><span>0</span><span>][</span><span>"</span><span>generated_text</span><span>"</span><span>][</span><span>-</span><span>1</span><span>])</span>

You can set device_map=cuda if you want use the gpu also.

Step 6: Run the Python Script

Output #

Issues you can face #

OSError: [WinError 126] fbgemm.dll
- To solve this error make sure you have Visual Studio installed.
  - In case you don’t have it, click here and install it.
  - Then restart the computer.
If there is still any errors with pytorch versions, use anaconda or miniconda to configure a new environment with suitable python version and dependencies.
If you are facing any other issue or error feel free to comment below.

Resources #
For more details on llama 3.1 check out: https://ai.meta.com/blog/meta-llama-3-1/

Conclusion #

In this blog, we have successfully cloned the LLaMA-3.1-8B-Instruct model from Hugging Face and run it on our local machine using Python. You can now experiment with the model by modifying the prompt, adjusting hyperparameters, or integrate with your upcoming projects. Happy coding!

📓 Cabinet of Ideas