Android / llama.cpp / GGUF

LLM AI Server with llama.cpp

LLM AI Server with llama.cpp is an Android local-LLM testing app that lets you load GGUF models, tune inference and prompt-template settings, manage shared MCP / Function Definitions settings, inspect logs, and expose an Ollama/OpenAI/Anthropic-compatible API plus the bundled WebUI from one app.

Docs: User Manual | API Reference | Technical Specification | llama.cpp / JNI / CMake Deep Dive | Privacy Policy

        Supports model downloads from URLs, local .gguf imports, and LoRA adapters (scale 0.3–1.0).
Lets you combine generation settings, Think behavior, custom chat templates, shared MCP settings, and Function Definitions JSON.
On-device Ollama/OpenAI/Anthropic/Gemini-compatible API and WebUI on the same port. Browse all endpoints in the built-in Swagger UI (/swagger), with a CORS toggle in Settings.

      

Screenshots

Main screen. API/WebUI status (1936.8 tok/s · CPU), Web AI Chat section, and Direct Run with profile picker and prompt field — all in one view.

Model Loading settings. Select a LoRA adapter file and adjust its scale (strength 0.3–1.0). mmproj (Multimodal Projector) is managed on the same screen.

Built-in WebUI. Chat from any browser on the same Wi-Fi. The bundled "LLM AI Server Guide" profile answers questions about how to use the app.

WebUI Function Calling — Function Calling in WebUI. Asking "What time is it?" triggers the get_time tool, which returns the current time (9:34:53).

WebUI Vision — Vision in WebUI. An uploaded image is described in detail by the model. Available when a model with an mmproj (vision projector) is loaded.

API server settings. Manage port, timeout, auto-restart, a direct link to the Swagger UI, and connection checks (127.0.0.1 / LAN / etc.).

Parameter settings. MTP Settings, Model Parameters, Penalty Parameters, Mirostat, and Additional Sampling in collapsible sections. "Generate Model" links to GGUF Generator.

Key Features

On-device local inference: Runs GGUF models directly on Android with llama.cpp.
Flexible model loading: Supports downloadable model URLs and local .gguf imports. LoRA adapters (.gguf format) can be selected independently and applied with an adjustable scale (strength 0.3–1.0). mmproj (Multimodal Projector) management is on the same screen.
Profile descriptions & card picker: Each profile now has a description field. The main-screen profile selector has been redesigned from a dropdown into a card-based dialog showing name and description side by side. Existing profiles are auto-patched with descriptions from the bundled asset on first launch after upgrade.
Deep inference controls: Adjust n_ctx, n_threads, compute backend (CPU / GPU), offload layer count, Top-p, Top-k, penalties, Mirostat, DRY, Think behavior, and custom chat templates.
Vision & automatic context expansion: The default vision model is Qwen3-VL. If an image or long prompt exceeds n_ctx, the app auto-expands n_ctx (within device memory) and reloads; if it still does not fit, API clients get a proper server error. GPU init failures fall back to CPU automatically.
Smarter model search: The in-app Hugging Face search filters by model name, parameter size and quantization via pull-downs (each with a one-line guide), shows a device-suitability rating, and can download mmproj (projector) files directly.
Upgraded inference engine: Updated to llama.cpp b10621 (v0.3.0) / ggml 0.22 — newer model support and Adreno OpenCL GPU.
MTP speculative decoding (experimental): For models that embed an MTP head (Qwen3.5-MTP, Gemma 4, etc.), enable MTP (draft-mtp) per model profile. Off by default, adjustable n_draft, no extra file needed (own head). Generation can be faster on supported setups.
Shared MCP / function-calling settings: Save MCP Config JSON and Function Definitions JSON separately from model profiles, then optionally enable them for the main prompt input, /api/chat, /api/generate, and /v1/chat/completions.
Built-in Ollama/OpenAI/Anthropic/Gemini-compatible API and WebUI: Provides /api/chat, /api/generate, /api/tags, /api/diagnostics, /v1/chat/completions, /v1/completions, /v1/messages, /v1/messages/count_tokens, /v1/responses (OpenAI Responses API), /v1/models, /props, /slots, and /v1beta/models/{model}:generateContent (Gemini-compatible) on the same port. Browse all endpoints in the built-in Swagger UI at /swagger. A CORS toggle in Settings controls cross-origin access. One generation runs at a time; queue depth is 10, default timeout 600 s (configurable).
Anthropic Messages API compatible: Provides /v1/messages (streaming via named SSE events) and /v1/messages/count_tokens, supporting system prompts, multi-turn, thinking blocks, image/audio input, and basic tool_use / tool_result. Point an official Anthropic SDK's base URL at this server.
Embeddings, token counts, and model control: Get embedding vectors from /api/embed, /api/embeddings, and /v1/embeddings; token counts from /api/tokenize and /v1/responses/input_tokens; and control model residency with /models/load and /models/unload.
Structured output and token statistics: Constrain output with format / grammar / response_format (JSON Schema, GBNF), and get Ollama-style eval_count fields plus OpenAI-style usage so existing clients work unchanged.
Multimodal API inputs: /api/chat and /v1/chat/completions accept image_url / input_audio, and /v1/messages accepts image / audio blocks, when a model with an mmproj is loaded (vision/audio).

Operational Notes

Model downloads can be several gigabytes. Wi‑Fi is strongly recommended.
The local API server is intended for the same device or a local network. Android 13+ may require notification permission.
If you configure MCP servers for the main prompt input, API integrations, or the WebUI, parts of conversation content or tool inputs may be sent to those MCP servers.