SpaceLLM Single Turn QA

SpaceLLM Single-Turn QA

A LoRA adapter for openai/gpt-oss-20b, fine-tuned for single-turn question answering over space-agency-derived text. This repository contains adapter weights only; the base model must be loaded separately.

• Developed by: Aditya Pratap Singh, Shivam Kumar, Dr. Saurabh Srivastava.
• Model type: PEFT LoRA adapter (CAUSAL_LM)
• Language: English
• Base model: openai/gpt-oss-20b, revision 6cee5e81ee83917806bbde320786a8fb61efebee
• Companion model: SpaceLLM Multi-Turn QA — a separate adapter trained on a different data format and evaluated on a different test set. The two adapters' scores are not a head-to-head comparison.

Please note

• This adapter uses no retrieval. There is no RAG pipeline, no index, and nothing looked up at inference time.
• LoRA targets attention projections only — q_proj, k_proj, v_proj, o_proj — not lm_head.
• Training and evaluation references were generated by a teacher model and have not been independently fact-checked. Reported scores measure similarity to those references, not verified factual accuracy.
• This is a distinct artifact from the older AdityaPS/SpaceLLM_v1 checkpoint, which used a different configuration (lm_head-only, rank 32) and different data/evaluation. If SpaceLLM_v1 remains public, treat it as a separate, earlier experiment — its card and weights should not be conflated with this one.

---

Uses

Intended:

• Research on parameter-efficient domain adaptation for space-agency-style QA.
• Serving as a single-turn baseline in comparative dialogue/QA evaluations.

Out of scope:

• Mission-critical operations, spacecraft control, orbital mechanics execution, or any safety-critical or autonomous decision system.
• Any use where an unverified or incorrect answer could cause harm.
• Multi-turn dialogue (use the companion multi-turn adapter, evaluated separately, for that).

Expert verification is recommended before relying on any specific technical claim this model produces.

---

How to get started

The example loads the pinned base revision and this adapter, applies the Harmony chat template, forces the assistant final channel, and extracts only that channel's text — the same approach used in evaluation. Relying on skip_special_tokens=True alone is avoided because it can mix analysis-channel and final-channel text.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, Mxfp4Config
from peft import PeftModel

BASE_MODEL = "openai/gpt-oss-20b"
BASE_REVISION = "6cee5e81ee83917806bbde320786a8fb61efebee"
ADAPTER_REPO = "AdityaPS/SpaceLLM_Single_Turn_QA"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    revision=BASE_REVISION,
    torch_dtype=torch.bfloat16,
    quantization_config=Mxfp4Config(dequantize=True),
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).eval()

question = "What was the primary objective of the Mars Science Laboratory mission?"
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": question}],
    tokenize=False,
    add_generation_prompt=True,
)
prompt += "<|channel|>final<|message|>"  # force the final channel

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,  # inference-example setting; evaluation used 2048 (see below)
        do_sample=False,
        repetition_penalty=1.05,
        no_repeat_ngram_size=6,
    )

generation = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
for stop_token in ("<|return|>", "<|end|>"):
    generation = generation.split(stop_token)[0]

print("Answer:", generation.strip())

Note: this example uses max_new_tokens=512 for a quick, practical demo. This differs from the 2,048-token limit used during evaluation (see Evaluation) — treat this as an inference-example setting, not the evaluated protocol.

---

Training details

Training data

Data was built offline, with no retrieval stage used anywhere in training or inference:

1. Space-agency text was scraped and stored as JSON.
2. Text was normalized and split into chunks.
3. Each chunk was passed directly to an Ollama mistral:7b teacher model.
4. The teacher was prompted to generate five standalone single-turn question–answer pairs per chunk, with answers requested to be grounded in the supplied chunk.
5. The generation pipeline did not independently verify that answers were entailed by their source chunk — some answers may be unsupported or incorrect.

Split	Usable examples
Train	14,395
Validation	1,695
Test (evaluated)	845

These are the only counts supported for this artifact. A previously circulated figure of "3,428 documents / 16,831 training samples" does not match this adapter's training/evaluation split counts or file hashes and should not be cited for this repository.

Training hyperparameters

Setting	Value
LoRA target modules	q_proj, k_proj, v_proj, o_proj
LoRA rank ($r$) / alpha ($\alpha$)	16 / 32
LoRA dropout	0.05
Task type	CAUSAL_LM
Max sequence length	2,048 tokens
Base weights	MXFP4, dequantized at load time
Compute precision	BF16

The base model is the published MXFP4 gpt-oss-20b checkpoint, loaded with MXFP4 dequantization and run in BF16 compute — this is a loading/compute configuration, not a separately distributed "BF16 version" of the model.

---

Evaluation

Test set: 845 single-turn records. Test file SHA-256: 337a80d2375a053ab7d8e9e1c5ee2ec7b975f74bfe16db13bf0d1c5a60aa91f6 Adapter config SHA-256: ef641c55f5f961a86be20f364b39b80f7777053959caff1523b2348f00432748

Protocol: Base and adapter used identical tokenized prompts. All 845 records produced a final answer; none were skipped. Generation was deterministic (do_sample=False) with max_seq_len=2048, max_new_tokens=2048, repetition_penalty=1.05, and no_repeat_ngram_size=6, applied to both base and adapter. Only text from the Harmony final channel was scored; analysis-channel text was excluded.

Metrics: BERTScore computed with roberta-large; token F1 computed on lowercased whitespace tokens; exact match computed after stripping surrounding whitespace. A missing answer would score zero on all metrics while remaining in the denominator — no answers were missing for this test set.

References were produced by the same teacher-based generation process used for training data and have not received independent factuality verification.

Results

Model	BERTScore P	BERTScore R	BERTScore F1	Token F1	Exact match	Hit 2,048-token cap
Base (gpt-oss-20b)	0.781081	0.869184	0.822320	0.122433	0 / 845	3 / 845
+ SpaceLLM Single-Turn Adapter	0.905112	0.890707	0.897506	0.371768	2 / 845	1 / 845

BERTScore F1 change: an absolute gain of 0.075186. This is not a "6% improvement," and it is not evidence of improved factual accuracy or domain understanding — it reflects greater semantic/stylistic similarity to teacher-generated references under this specific protocol.

Reading the weaker metrics honestly

• Exact match is very low (2/845, ≈0.24%) for both models, which is expected for free-form generative QA — a correct answer rarely matches a reference string verbatim in wording, order, or length. This metric is reported for completeness, not as a proxy for correctness rate.
• Token F1 (0.372) is modest in absolute terms, even though it nearly tripled over the base model's 0.122. Token overlap penalizes any answer that is phrased differently from the reference, including answers that may be factually equivalent. Treat the relative change as a signal of closer stylistic alignment, not the absolute value as an accuracy score.
• BERTScore is the most informative similarity signal here, but it has the same ceiling as its references: since references are teacher-generated and unverified, a higher BERTScore partly reflects the adapter reproducing the teacher's phrasing habits — including any errors or unsupported claims the teacher made — rather than confirmed improvement in factual correctness.

---

Generation stability

An earlier unguarded diagnostic run at the same 2,048-token maximum, without repetition controls, produced outputs that fell into exact phrase loops. The final evaluation protocol added repetition_penalty=1.05 and 6-token n-gram blocking (no_repeat_ngram_size=6) to both base and adapter generations to address this.

Model	Unguarded diagnostic (cap events)	Guarded final run (cap events)	Reduction
Base	35	3	91.4%
Adapter	28	1	96.4%

This is reported as an engineering diagnostic, not a controlled ablation: prompt hashes changed across rebuilt evaluation images between the unguarded and guarded runs, so the reduction cannot be attributed to the decoding settings alone. Additionally, the guarded settings removed the conspicuous exact-repetition loops but did not eliminate semantic degeneration generally — a small number of guarded outputs still reached the token cap while producing drifting numerical lists or otherwise unrelated text.

---

Bias, risks, and limitations

• The supported conclusion is narrow: these results show improved similarity to synthetic, teacher-generated references under this specific evaluation protocol. They do not establish improved factual accuracy, reasoning ability, or general domain understanding.
• Exact match (2/845) and token F1 (0.372) are low in absolute terms. This is a known property of free-form generative QA evaluated against a single reference string, not necessarily a sign the model is "wrong" most of the time — but it also means these numbers cannot be read as an accuracy rate.
• No independent factuality check exists. References were generated by the same mistral:7b teacher process used for training data; higher similarity scores may partly reflect the adapter learning to imitate the teacher's style, length, and any of its errors, rather than confirmed improvements in correctness.
• The exact training/evaluation data snapshot has not yet been recovered and published; the dataset manifest, source-agency distribution, and dedup/filtering details are pending.
• This adapter has not been validated for operational, mission-critical, or safety-critical use of any kind, including spacecraft control, mission planning, or autonomous decision-making.
• There is no retrieval and no source citation — answers cannot be traced back to a specific source document by the model itself.
• no_repeat_ngram_size=6 can suppress legitimate repeated technical phrases (e.g., a mission name or unit repeated for clarity), which may slightly affect fluency on some answers.
• Generation...