If you’re an ML engineer or a technical lead planning to pre-train a large language model (LLM) for Tagalog or Taglish, how do you start? In this article, we walk through the process of defining a target voice, building a data pipeline, running a QLoRA training experiment, and evaluating results with native speakers, so a team can test the idea with a small, well-scoped proof of concept before committing to a larger build. 

Why natural Tagalog is harder than correct Tagalog for LLM

A general-purpose LLM may produce Tagalog that is grammatically acceptable yet still sound overly formal, translated, or socially out of place. Everyday speech often includes shortened phrases, implied subjects, regional vocabulary, humor, politeness markers, and code-switching, which means moving between Tagalog and English within one exchange. A model can understand each word and still miss the rhythm that makes the reply feel local.

The target is therefore not “more Tagalog” in every response. It is the right language mix, level of formality, emotional register, and reply length for the situation. A friend asking “Okay ka lang?” needs a different answer from a customer asking about a delayed payment, even when both conversations use casual language.

Prompt instructions can help, but they have limits. A system prompt, the instruction that sets behavior for a chat session, may tell the model to be informal while its learned habits still pull it toward long explanations or textbook phrasing. Supervised fine-tuning trains on input-and-response examples, so it can address that deeper pattern.

Naturalness also varies by audience. Speech used in a large city, a provincial community, an online gaming group, and a family chat may differ in vocabulary and code-switching. Before collecting training examples, define whose speech the system should reflect and which varieties it should handle without stereotyping or flattening regional differences.

Fine-tune an existing model instead of pretraining one

Pretraining creates a foundation model by teaching it broad language patterns and general knowledge from a very large collection of text. That process requires extensive data, computing systems, methods for splitting text into tokens, and repeated experiments. A conversational Tagalog project usually does not need to rebuild those capabilities.

Fine-tuning starts with a pretrained model and adjusts its behavior for a narrower goal. The base model already knows how to follow instructions, generate text, and handle many common topics. The new dataset teaches it how to respond in a chosen voice, not how to learn language from zero.

An open-weight model is a model whose trained parameters can be downloaded and run under its license. “Open-weight” does not automatically mean that every part of the training data, code, or license is open, so teams must still review usage terms. For a first experiment, compare several suitable multilingual models with roughly 7 billion to 9 billion parameters. Parameters are the learned numeric weights that store a model’s patterns, and this size range is often practical for an initial test.

Use real prompts and native-speaker reviews for that comparison. A model with strong scores in math or English reasoning may still produce stiff Tagalog, weak Taglish, or inconsistent use of politeness markers such as po and opo. The best starting point is the candidate that already sounds closest to the target voice. Fine-tuning should refine a workable starting point, not try to repair a poor one.

For a broader planning framework, the Tutorials Dojo course on planning a generative AI project connects model choice with goals, data readiness, and review. It can help a team set scope before it commits to a training run. (Tutorials Dojo)

Define the target voice and benchmark before training

Write a short style guide before building the dataset. It should state the intended users, supported language varieties, acceptable code-switching, preferred response length, level of casualness, safety limits, and cases that require a more formal tone. Without this document, reviewers may disagree because they are judging against different ideas of “natural.”

Create a baseline benchmark in parallel. A benchmark is a fixed set of prompts and evaluation criteria used to compare models and training runs. Include short replies, multi-turn chats, emotional messages, ambiguous inputs, common service questions, slang, misspellings, and Taglish prompts that reflect real use.

Keep the benchmark separate from training data. When benchmark conversations appear in the training set, the system may memorize them and produce results that look better than its true performance. Use conversation-level grouping when creating the training set, the validation set used during development, and the final test set. Different turns from one thread must not leak across those sets.

A useful first benchmark may contain a few hundred carefully selected prompts, but coverage matters more than a specific count. Record the source category, language mix, topic, risk level, and expected tone for each item. These tags make it easier to see whether a candidate improves on casual conversation while getting worse on safety, helpfulness, or formal tasks.

Review the benchmark with native speakers before training begins. This step catches unclear prompts and confirms that the scoring guide measures the behavior the team actually wants. It also creates a stable baseline for deciding whether later training-compute spending is justified.

Choose data sources for conversation, coverage, and quality

Conversation logs are the most direct source for learning short replies, turn-taking, casual phrasing, and code-switching. They are also the noisiest source because they may contain automated messages, incomplete threads, repeated templates, abuse, account details, and language that is natural between people but unsuitable for an assistant. Use only data covered by a documented permission and privacy review for model training.

Licensed transcripts, blogs, and creator content can add vocabulary, cultural references, and longer explanations. They should normally support rather than dominate a conversational dataset because monologues do not teach back-and-forth interaction. Before adding public content, confirm that the data owner and applicable terms permit the intended training use.

A corpus is a collection of text used for training. Treat each source as a separate corpus with its own weight. A high-level mixture might give the largest share to reviewed conversations, a smaller share to licensed spoken transcripts, and a smaller share to edited long-form text. The exact ratio should come from evaluation rather than a fixed recipe.

Do not take the first block of an unordered dataset and call it representative. Index the full collection, score the examples, and sample across speakers, topics, time periods, thread lengths, and language patterns. Stratified sampling means selecting examples from defined groups so one source or speaking style does not overwhelm the rest.

Start with a smaller, high-quality subset that is large enough to test the idea but inexpensive enough to rebuild. When the first model improves, expand the corpus and compare new mixtures. When it does not, investigate the labels and data quality before adding more tokens.

Map roles to the behavior you want

Supervised fine-tuning trains a model on input-and-target pairs. In chat data, those pairs are usually represented with system, user, and assistant roles. The content assigned to the assistant role becomes the behavior to reproduce, so role mapping is a training decision, not a clerical detail.

Do not automatically map every human message to the assistant role and every bot message to the user role. Instead, identify which response is a safe, helpful example of the intended assistant behavior. A casual human reply may sound natural but still be too vague, rude, private, or dependent on missing context for a general assistant.

Incorrect mapping can reverse the goal. For example, a support log may contain an automated greeting followed by a short customer response such as “Need ko lang mag-reset ng PIN.” When the greeting is placed in the assistant field, the fine-tuned system may learn repetitive scripts. When the customer message is used without adaptation, it may learn to speak like a requester rather than a helper.

Build rules that group messages by conversation, sort them by time, identify the speaker, and preserve enough earlier turns to make the target response understandable. Flag uncertain cases for human review instead of forcing a label. You can still learn from naturally written human messages, but the final target should match the role the deployed system will play.

This is the main human-in-the-loop decision in the pipeline. People do not need to label every line manually, but they must define the mapping rules, review difficult examples, and audit whether the resulting targets are appropriate.

Build a data refinery, not a manual review queue

Large chat collections cannot be reviewed line by line. A data refinery is an automated pipeline that parses, cleans, scores, samples, and validates examples while sending only uncertain or high-risk cases to people. The pipeline should be reproducible so a later dataset version can be rebuilt from the original source and rules.

A practical workflow includes the stages below. Each stage should record its decision so the dataset can be reproduced.

• Parse conversation identifiers, timestamps, speakers, and message order.
• Normalize encoding and whitespace without removing meaningful slang, punctuation, or emoji.
• Detect language and code-switching patterns, then filter unrelated language data.
• Remove empty messages, obvious auto-replies, spam, URL-only responses, and corrupted text.
• Detect exact and near duplicates so repeated templates do not dominate training.
• Identify and redact personally identifiable information, or PII, such as names, phone numbers, email addresses, account numbers, and street addresses.
• Score quality, safety, role confidence, naturalness, and source reliability.
• Split complete conversations into training, validation, and held-out test sets.

Not every unusual message should be deleted. Emoji-heavy replies, short acknowledgments, and mixed-language sentences are part of real conversation, but they need balanced sampling so they do not become overrepresented in the model’s replies. Keep the raw source unchanged, store each filtering decision, and assign a version number to every cleaned output.

Quality scoring should guide sampling rather than hide uncertainty. A simple score can combine language match, reply completeness, duplication risk, privacy risk, harmful-language risk, role confidence, and whether the message needs missing context. Train first on the clearest examples, then add lower-scoring categories only when the evaluation set shows a specific gap.

Use a focused toolchain for preparation and review

Python is a general-purpose programming language often used for data work. The pandas library adds table-like structures that help group messages by conversation, sort turns, clean fields, and apply rules. For larger collections, Hugging Face Datasets stores and transforms machine learning data with operations such as map() and filter(). (Hugging Face)

Label Studio is an open-source annotation platform that can present uncertain examples to reviewers and export labels such as valid_target, auto_reply, unsafe, or role_uncertain. It is most useful for auditing a sample and resolving difficult cases, not for asking people to inspect every message. 

Presidio is an open-source framework for detecting and anonymizing sensitive information in text and images. It supports built-in and custom recognizers, but automated PII detection can miss unusual names, local identifiers, or context-specific account formats, so manual audits and additional controls remain necessary.

For duplicates, cryptographic hashes create fixed fingerprints that can find exact matches. The Python library datasketch implements MinHash and locality-sensitive hashing, or LSH, which can find near-duplicate text without comparing every pair directly. Pandera checks table schemas, which define the required structure. It can verify columns, role values, unique identifiers, and nonempty target responses.

A transformer is the neural network design behind most modern LLMs. During training, Hugging Face Transformers loads these models and their tokenizers, which turn text into model tokens. The TRL SFTTrainer, where SFT means supervised fine-tuning, trains conversational data or prompt-completion pairs with an input and target output. PEFT, short for Parameter-Efficient Fine-Tuning, supplies methods that update small trainable layers called adapters instead of every model weight.

Format each example as a clear chat exchange

Most chat models expect a chat template, which is a structured sequence of role-labeled messages. The exact tokens differ by model family, but the conceptual format uses a system instruction, one or more user turns, and the assistant response that should be learned. Always apply the selected model’s official chat template instead of inventing model-specific markers.

JSON is a common text format for structured data. This example teaches the assistant to respond to an emotional message with reassurance and a practical next step. The target is conversational without being careless, and the system message defines a broad style without forcing slang into every answer.

Use multi-turn examples when earlier context changes the correct reply, but avoid adding unrelated history simply to fill the context window. When a human message is natural but unsuitable as an assistant response, edit or exclude it under a documented policy rather than silently treating all human text as high quality.

Store the source, license, language mix, quality score, and transformation history beside each example. That metadata makes error analysis possible later. It also allows the team to remove a source or rebuild the dataset when consent, licensing, or quality assumptions change.

Start with QLoRA and conservative settings

LoRA, or Low-Rank Adaptation, freezes most of the base model and trains small adapter matrices that change its behavior. Quantization stores model weights with fewer bits; QLoRA uses a 4-bit frozen base while training the LoRA adapters. The original QLoRA paper describes this approach, and the Hugging Face PEFT LoRA documentation explains the main adapter settings.

A starting settings file may look like this. Keep it in version control with the dataset and code.

YAML is a human-readable format often used for settings files. This example loads a selected model in a 4-bit format, trains LoRA adapters, and begins with one pass through the dataset. NF4 is a 4-bit data type introduced for quantized model weights, while bfloat16, or BF16, is a 16-bit compute format that keeps a wide numerical range.

These values are starting points, not guarantees. Test the learning rate, which controls update size; the adapter rank, which controls adapter capacity; the batch size, or examples processed together; and the layers that receive adapters. A checkpoint is a saved copy of the training state, so save several because the best conversational result may appear before the final step.

Keep the first comparison simple. Change one major factor at a time. Record the dataset version and random seed, the number used to initialize random sampling, then test every candidate against one held-out benchmark. This makes it easier to identify whether an improvement came from the data, settings, or chance.

Plan context length, epochs, hardware, and cost together

A token is a small unit of text processed by an LLM, and context length is the maximum number of tokens in one training example. A 4,096-token window is often enough for short and medium conversations, while longer windows use more memory and training time. Increase context only when the evaluation set includes long threads that need it.

An epoch is one complete pass through the training dataset. Begin with one epoch for a large, diverse corpus, then test a second only when held-out evaluation continues to improve. More passes can increase memorization, repeated phrasing, and overfitting, where results improve on familiar data but get worse on new cases. This risk grows when the corpus contains near duplicates or narrow source patterns.

BF16 is a practical default when the hardware and software support it. FP8, an 8-bit floating-point format, can increase speed on compatible systems, but it adds setup and testing work. For the first proof of concept, or small feasibility test, prefer stable training and clean comparisons over the newest numeric format.

A graphics processing unit, or GPU, is a processor suited to the parallel math used in model training. GPU models differ in memory, speed, availability, and price, so the fastest chip is not always the least expensive option. Compare measured tokens per second, memory limits, checkpoint overhead, queue time, and the provider’s current rate under one fixed setup. A small pilot on balanced data gives a better estimate than a vendor chart.

Managed services can simplify storage, training jobs, and hosted model APIs. The Tutorials Dojo Amazon SageMaker cheat sheet provides an overview of the AWS service, but this experiment design applies to other clouds or self-managed infrastructure.

Evaluate naturalness with native speakers

Validation loss measures how well a candidate predicts held-out text, and perplexity is a related measure of prediction uncertainty. Both are useful for detecting training problems, but neither can decide whether a Tagalog reply sounds socially natural. The main evaluation should therefore compare outputs through blind review by native speakers.

Show reviewers an identical prompt with responses from the base model and the fine-tuned model in random order. Do not reveal which system produced each answer. Ask for a preference and separate scores for naturalness, helpfulness, local fit, clarity, safety, formality, and code-switching.

The evaluation set should cover the categories below. Keep the category tags so results can be compared by use case.

• Short casual replies and acknowledgments
• Multi-turn conversations that depend on earlier context
• Tagalog-English code-switching
• Emotional, informal, or humorous inputs
• Common service and instructional questions
• Misspellings, fragments, and naturally messy messages
• Safety-sensitive or privacy-sensitive situations

Define success before training. For example, require enough reviews to show that a preference over the base model is unlikely to be random. Do not declare success from a small change in average score, and do not accept a drop in safety or helpfulness. Report results by category because a single overall number can hide failures on formal requests or regional language.

Automated checks should support human review. Track response length, repeated phrases, language distribution, unsafe content, PII leakage, refusal behavior, and similarity to training examples. When reviewers disagree, inspect the prompt and scoring guide instead of averaging away a real difference in language preference.

Address privacy, security, legal, and quality risks

The most immediate risk is training on private information. A conversation log may contain a phone number, full name, address, or account reference; a memorizing model could later reproduce that detail after a similar prompt. Use redaction, duplicate removal, access controls, data retention limits, and targeted leakage tests. No single detector catches every case. Presidio’s documentation also warns that automated detection cannot guarantee that all sensitive information will be found. (Data Privacy Stack)

Bot contamination is another practical failure. When thousands of automated greetings remain in the assistant targets, the fine-tuned system may begin many replies with one scripted sentence even when it does not fit. Review frequent phrases before and after training, and down-weight or remove templates.

Unsafe or abusive text can also become a learned style. For example, raw chats may contain insults between users that sound locally authentic but are not acceptable assistant behavior. Separate conversational naturalness from permission to imitate every observed expression, and keep safety examples in the training and evaluation sets.

Licensing and privacy reviews need source-level records. For scraped videos, blogs, or social posts, document the license or other basis that permits reuse. Store the source, legal basis for reuse, collection date, and removal process. These records let the team audit or revise the corpus later.

Finally, watch for overfitting to one community, organization, or region. A model that overuses one catchphrase or local slang may appear natural to one group and strange to another. Balanced sampling, category-level evaluation, and limited beta testing are better safeguards than a larger training run.

Follow a staged roadmap and use clear decision rules

A proof of concept should answer whether high-quality fine-tuning improves the target behavior, not attempt to produce the final production model. Use a staged plan so each step creates evidence for the next investment.

1. Define the target voice. Write the language, tone, safety, and use-case guide.
2. Build the benchmark. Freeze a held-out set and reviewer rubric before training.
3. Create the data refinery. Parse, map, filter, redact, deduplicate, score, and version the data.
4. Compare base models. Test several multilingual candidates with a shared prompt set.
5. Run a small QLoRA experiment. Train on the cleanest representative sample and save multiple checkpoints.
6. Conduct blind human evaluation. Compare the adapted model with the unchanged baseline.
7. Scale selectively. Add data or change settings only to address measured weaknesses.
8. Deploy to a limited beta. Test with a small user group, add monitoring and safety filters, and keep a rollback path that can restore the previous model.

Use decision rules instead of intuition. When naturalness does not improve, inspect role mapping and data quality before increasing model size. When naturalness improves but helpfulness falls, add reviewed assistant-style examples and rebalance the mixture. When slang becomes excessive, broaden the speaker and topic sample.

When results are unclear, expand or repair the benchmark before spending more on training. Teams building these skills on AWS can also use the Tutorials Dojo AWS Certified Machine Learning Engineer Associate exam guide to review data preparation, modeling, deployment, and monitoring concepts. (Tutorials Dojo)

Build the evaluation asset before the trained model

Natural Tagalog fine-tuning succeeds when the adapted system learns appropriate conversational choices, not when it merely produces more local vocabulary. That requires licensed data, careful target selection, efficient adapter training, and native-speaker review tied to real use cases.

The most durable project asset is the versioned benchmark and review process. Base models, GPU options, and training libraries will change. A trusted set of prompts, risk cases, and human judgments lets the team test each new option without starting over. Build that evaluation asset first, then scale the training data only when the results show what needs to improve.

A practical next step is to create the style guide and a small blind benchmark before cleaning the full corpus. Those two items expose disagreements early, guide the data pipeline, and provide a clear stopping rule for the first fine-tuning run.

Treat each new base model as a challenger, not a reset. Run it against the established benchmark, record the data and settings used, and keep a short model card that states known strengths, limits, and risks. This practice makes future upgrades faster without weakening review or oversight.

Keep reviewer notes and failure examples beside the scores. They often show which data category or policy should change before the next training run.

References

• QLoRA: Efficient Finetuning of Quantized LLMs
• Hugging Face PEFT LoRA documentation
• Hugging Face Transformers quickstart
• Hugging Face TRL SFTTrainer documentation
• Hugging Face Datasets processing guide
• Presidio Analyzer documentation
• Label Studio labeling guide
• Pandera DataFrame schema documentation
• datasketch MinHash LSH documentation