agiresearch/AIOS

Before launching AIOS, it is required to set up configurations. AIOS provides two ways of setting up configurations, one is to set up by directly modifying the configuration file, another is to set up interactively.

Set up configuration file directly (Recommended)

You need API keys for services like OpenAI, Anthropic, Groq and HuggingFace. The simplest way to configure them is to edit the aios/config/config.yaml.

[!TIP] It is important to mention that, we strongly recommend using the aios/config/config.yaml file to set up your API keys. This method is straightforward and helps avoid potential sychronization issues with environment variables.

A simple example to set up your API keys in aios/config/config.yaml is shown below:

api_keys:
  openai: "your-openai-key"    
  gemini: "your-gemini-key"    
  groq: "your-groq-key"      
  anthropic: "your-anthropic-key" 
  huggingface:
    auth_token: "your-huggingface-token-for-authorized-models"  
    cache_dir: "your-cache-dir-for-saving-models"
  novita: "your-novita-api-key"

To obtain these API keys:

1. Deepseek API: Visit https://api-docs.deepseek.com/
2. OpenAI API: Visit https://platform.openai.com/api-keys
3. Google Gemini API: Visit https://makersuite.google.com/app/apikey
4. Groq API: Visit https://console.groq.com/keys
5. HuggingFace Token: Visit https://huggingface.co/settings/tokens
6. Anthropic API: Visit https://console.anthropic.com/keys
7. Novita AI API: Visit https://novita.ai/api-keys

Configure LLM Models

You can configure which LLM models to use in the same aios/config/config.yaml file. Here's an example configuration:

llms:
  models:
    # Ollama Models
    - name: "qwen2.5:7b"
      backend: "ollama"
      hostname: "http://localhost:11434"  # Make sure to run ollama server

    # vLLM Models
    - name: "meta-llama/Llama-3.1-8B-Instruct"
      backend: "vllm"
      hostname: "http://localhost:8091/v1"  # Make sure to run vllm server

Using Ollama Models:

1. First, download ollama from https://ollama.com/
2. Start the ollama server in a separate terminal:

ollama serve

3. Pull your desired models from https://ollama.com/library:

ollama pull qwen2.5:7b  # example model

[!TIP] Ollama supports both CPU-only and GPU environments. For more details about ollama usage, visit ollama documentation

Using vLLM Models:

1. Install vLLM following their installation guide
2. Start the vLLM server in a separate terminal:

vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8091

[!NOTE] vLLM currently only supports Linux and GPU-enabled environments. If you don't have a compatible environment, please choose other backend options. To enable the tool calling feature of vllm, refer to https://docs.vllm.ai/en/latest/features/tool_calling.html

Using HuggingFace Models: You can configure HuggingFace models with specific GPU memory allocation:

- name: "meta-llama/Llama-3.1-8B-Instruct"
  backend: "huggingface"
  max_gpu_memory: {0: "24GB", 1: "24GB"}  # GPU memory allocation
  eval_device: "cuda:0"  # Device for model evaluation

Set up interactively

Alternatively, you can set up aios configurations interactively by using the following command.

• aios env list: Show current environment variables, or show available API keys if no variables are set
• aios env set: Show current environment variables, or show available API keys if no variables are set
• aios refresh: Refresh AIOS configuration. Reloads the configuration from aios/config/config.yaml. Reinitializes all components without restarting the server. The server must be running.

When no environment variables are set, the following API keys will be shown:

• DEEPSEEK_API_KEY: Deepseek API key for accessing Deepseek services
• OPENAI_API_KEY: OpenAI API key for accessing OpenAI services
• GEMINI_API_KEY: Google Gemini API key for accessing Google's Gemini services
• GROQ_API_KEY: Groq API key for accessing Groq services
• HF_AUTH_TOKEN: HuggingFace authentication token for accessing models
• HF_HOME: Optional path to store HuggingFace models
• NOVITA_API_KEY: Novita AI API key for accessing Novita AI services

Launch AIOS

After you setup your keys or environment parameters, then you can follow the instructions below to start.

Run:

bash runtime/launch_kernel.sh

Or if you need to explicity set the Python version by running python3.10, python3.11, python3, etc. run the command below:

python3.x -m uvicorn runtime.launch:app --host 0.0.0.0 --port 8000 # replace the port with your own port

You also need to set up the host and port in the configuration of Cerebrum (AIOS SDK) to make sure it is consistent with the configurations of AIOS.

You can also force the kernel to run in the background with:

python3.x -m uvicorn runtime.la

…