Android / llama.cpp / GGUF

LLM AI Server with llama.cpp

LLM AI Server with llama.cpp is an Android local-LLM testing app that lets you load GGUF models, tune inference and prompt-template settings, manage shared MCP / Function Definitions settings, inspect logs, and expose an Ollama/OpenAI/Anthropic-compatible API plus the bundled WebUI from one app.

Docs: User Manual | API Reference | Technical Specification | llama.cpp / JNI / CMake Deep Dive | Privacy Policy

        Supports both model downloads from a URL and importing local .gguf files from the device.
Lets you combine generation settings, Think behavior, custom chat templates, shared MCP settings, and Function Definitions JSON.
Can start an on-device Ollama/OpenAI/Anthropic-compatible API and WebUI on the same port, including endpoints such as /api/chat, /v1/chat/completions, /v1/completions, and /v1/messages.

      

Screenshots

LLM AI Server with llama.cpp main screen — Main screen with status header (speed, temperature, backend), API/WebUI controls, Direct Run section, and processing logs.

LLM AI Server with llama.cpp settings screen — Settings screen for model URLs, local GGUF import, GPU backend selection, mmproj settings, and inference parameters.

LLM AI Server with llama.cpp template settings — Template, API, MCP, language, and log settings, including System Prompt, custom chat template, and Function Definitions JSON fields.

Key Features

On-device local inference: Runs GGUF models directly on Android with llama.cpp.
Flexible model loading: Supports downloadable model URLs and local .gguf imports.
Deep inference controls: Adjust n_ctx, n_threads, compute backend (CPU / GPU), offload layer count, Top-p, Top-k, penalties, Mirostat, DRY, Think behavior, and custom chat templates.
Vision & automatic context expansion: The default vision model is Qwen3-VL. If an image or long prompt exceeds n_ctx, the app auto-expands n_ctx (within device memory) and reloads; if it still does not fit, API clients get a proper server error. GPU init failures fall back to CPU automatically.
Smarter model search: The in-app Hugging Face search filters by model name, parameter size and quantization via pull-downs (each with a one-line guide), shows a device-suitability rating, and can download mmproj (projector) files directly.
Upgraded inference engine: Updated to llama.cpp b10621 (v0.3.0) / ggml 0.22 — newer model support and Adreno OpenCL GPU.
MTP speculative decoding (experimental): For models that embed an MTP head (Qwen3.5-MTP, Gemma 4, etc.), enable MTP (draft-mtp) per model profile. Off by default, adjustable n_draft, no extra file needed (own head). Generation can be faster on supported setups.
Shared MCP / function-calling settings: Save MCP Config JSON and Function Definitions JSON separately from model profiles, then optionally enable them for the main prompt input, /api/chat, /api/generate, and /v1/chat/completions.
Built-in Ollama/OpenAI/Anthropic-compatible API and WebUI: Provides /api/chat, /api/generate, /api/tags, /api/diagnostics, /v1/chat/completions, /v1/completions, /v1/messages, /v1/messages/count_tokens, /v1/models, /props, and /slots on the same port; only one generation runs at a time, with a queue of up to 10 requests for up to 600 seconds by default (configurable).
Anthropic Messages API compatible: Provides /v1/messages (streaming via named SSE events) and /v1/messages/count_tokens, supporting system prompts, multi-turn, thinking blocks, image/audio input, and basic tool_use / tool_result. Point an official Anthropic SDK's base URL at this server.
Embeddings, token counts, and model control: Get embedding vectors from /api/embed, /api/embeddings, and /v1/embeddings; token counts from /api/tokenize and /v1/responses/input_tokens; and control model residency with /models/load and /models/unload.
Structured output and token statistics: Constrain output with format / grammar / response_format (JSON Schema, GBNF), and get Ollama-style eval_count fields plus OpenAI-style usage so existing clients work unchanged.
Multimodal API inputs: /api/chat and /v1/chat/completions accept image_url / input_audio, and /v1/messages accepts image / audio blocks, when a model with an mmproj is loaded (vision/audio).

Operational Notes

Model downloads can be several gigabytes. Wi‑Fi is strongly recommended.
The local API server is intended for the same device or a local network. Android 13+ may require notification permission.
If you configure MCP servers for the main prompt input, API integrations, or the WebUI, parts of conversation content or tool inputs may be sent to those MCP servers.