User Manual
GGUF Generator lets you run the full LLM workflow — interactive learning, LoRA adaptation, training-pair generation, and quantization conversion — entirely on your smartphone. The four main features are each accessible from a tile on the home screen.
1. Common Settings
- Set the LLM AI server URL (default
http://127.0.0.1:11434) and tap "Fetch" to load the model list (also fetched automatically at launch). - Select the generation model (LLM) and embedding model (RAG). "Start Server" can launch an external local API server; the status tile shows connectivity.
- Choose the default save location (Downloads / Documents). Individual file references use the standard system dialog.
- Use input profiles to save and restore named configurations.
2. Interactive Learning (New GGUF)
Teach a small LLM from scratch through chat — instruction by instruction, response by response. The learning state is saved and resumed as a Retrainable GGUF.
Basic flow
- Tap "New" to initialize a model. Set the config (dModel / ff / layers / ctx) and learning rate.
- Type an instruction → tap "New Learn" → type the response → tap "New Learn". The model is trained and the result is shown immediately.
- After training, the button becomes "Re-learn". Tap to reinforce the last pair with repeated training passes.
- "Response Learn" uses the model's last reply as the instruction and your input as the response — useful for teaching conversation flow.
- "Try Inference" tests the model's current response at any time.
- Tap "Export GGUF" to save a Retrainable GGUF in F32 / F16 / Q8_0 precision.
Continuous Learning (tap to expand)
- Import pairs: Bulk-import a JSONL file, rebuild the BPE vocabulary, and retrain on the entire corpus. Runs in the background — continues even after the app is closed.
- Target loss: Train until the slider loss value is reached. Also applies during pair import — auto-stops when the target is hit.
- Regenerate model: Rebuild the vocabulary from all data, expand embeddings, and retrain on all pairs.
Resuming from a saved GGUF
Tap "Load GGUF" to reload a previously saved Retrainable GGUF. The training corpus history and Adam optimizer state are restored, letting you continue exactly where you left off. See Retrainable GGUF for details.
3. Interactive Learning (LoRA)
Layer LoRA adapters on top of an existing GGUF model through chat. The base weights are frozen; only the small adapter matrices are trained, keeping things lightweight.
Supported architectures
llama / gemma3 / qwen3 / LFM2 (under 1B recommended)
Basic flow
- Tap "Load model" to load the base GGUF.
- Use "New Learn" and "Response Learn" to train the adapter interactively (same steps as New GGUF).
- Tap "Export adapter" to save the LoRA adapter GGUF.
- In-progress state (training pairs + optimizer) is embedded in the adapter GGUF. Resume via "Load adapter".
Continuous LoRA Learning
- Import pairs: Bulk-import JSONL and retrain all pairs for LoRA. Runs in the background.
- Retrain all: Re-run the full corpus with updated settings (LR, epochs).
LoRA settings (tap to expand)
| Parameter | Description | Guide |
|---|---|---|
| rank (r) | Adapter capacity (low-rank dimension) | 4–16 (8 recommended) |
| alpha (α) | Effective scale = α ÷ r | Same as rank or 1–2× |
| Learning rate | Adam update step size | 0.001–0.003 |
| Steps/pair | Max epochs per pair | 1–3 |
| targets | Target projections (q,v by default) | q,k,v,o,gate,up,down |
4. Pair Edit & Generation
Use an LLM server to auto-generate and edit training pairs. Saved pairs feed directly into Interactive Learning batch import.
Pair list operations
- Tap a row to edit, long-press to copy or delete. Check boxes to multi-select.
- "Load" reads a JSONL file; "Save" writes one.
LLM operations
- Add new pairs: Generate N pairs from a prompt instruction, optionally using source material (URL / one Wikipedia result).
- Edit pairs: Transform selected pairs one by one with the LLM (translate, rephrase, rewrite, etc.).
- Derive: Use selected pairs as seeds and generate N new pairs from each one.
LLM operations require an Ollama-compatible server. Set the server URL and model in Settings → "Fetch" first.
5. Quantization Conversion
Convert any existing GGUF file to a different quantization type on-device.
- Tap "Pick source GGUF" to select the source file.
- Select the target quantization type (F32 > F16 > Q8_0 > Q6_K > Q5_K_M > Q4_K_M > Q3_K_M > Q2_K — progressively smaller, faster, and lower precision).
- Tap "Run quantization".
- 1-D tensors (norm gains) are always kept at F32.
- Matrices whose row length is not a multiple of the block size fall back to F16 automatically.
Early Stopping & Evaluation (optional, shared across all training modes)
- When "Enable early stopping / evaluation" is on, training stops automatically once the pass rate reaches the threshold (default 60%).
- Choose how to specify expected values: manual keywords ("input => keywordA|keywordB", one per line) or auto token extraction (top-surprisal).
- Set the pass threshold, evaluation interval, max generated tokens, and number of evaluation cases.
Tips
- For staged target losses, larger values mean weaker training (generalization, more judgment room) and smaller values mean stronger training (memorization).
- Interactive learning retrains on the full corpus every time a new pair is added. This prevents forgetting but gets heavier as pairs accumulate.
- On-device training is CPU- and memory-intensive. For long runs, grant notification permission and disable battery optimization.
- LLM operations (pair edit & gen) need an Ollama-compatible server on the same device or local network.
- For a deep dive into the Retrainable GGUF format, see Retrainable GGUF.