Multilingual Synthetic Data Generator

I needed sentence datasets in Indian languages and kept running into free-tier limits on hosted APIs. This tool runs entirely on a local model through Ollama: pick any installed model (or type a name to download one), give it a topic and languages, and get a clean, deduplicated dataset as TXT, JSON, CSV, PDF or MP3.

Supported languages: English, Hindi, Bengali, Urdu, Tamil, Telugu, Marathi, Gujarati, Punjabi, Malayalam, Kannada, Nepali and Sinhala.

• Hosted APIs charge per token and cap free usage, which makes large dataset runs expensive.
• LLM output for lower-resource scripts is noisy: numbering, intro lines, repeated sentences, and sentences in the wrong script.
• Mixed-language datasets need exact counts, not approximate ratios.

Topic + languages + model
  -> Prompt   the local LLM in each language, in that language
  -> Clean    strip numbering, bullets and intro lines
  -> Check    keep only sentences written in the right script
  -> Dedupe   drop repeats, then ask again until there are enough
  -> Mix      split the samples by percentage and shuffle
  -> Export   TXT, JSON, CSV, PDF or MP3

Each stage is its own module: generator.py prompts, cleans and checks; scripts.py detects Unicode scripts; mixer.py allocates and shuffles; output.py and media.py export; models.py lists, picks and downloads Ollama models.

Script validation. A sentence is kept only if at least 70% of its letters belong to the target language's script, and it's rejected if it contains letters from any third script (Latin letters are allowed).

def script_share(text, language_code):
    letters = [c for c in text if c.isalpha()]

    if not letters:
        return 0.0

    return sum(in_script(c, language_code) for c in letters) / len(letters)

Exact mixing. Percentages are allocated with the largest-remainder method, so a 60/40 split always adds up to exactly the number of samples requested.

def allocate_counts(total_samples, mix_ratio):
    counts = {lang: total_samples * percent // 100 for lang, percent in mix_ratio.items()}
    leftover = total_samples - sum(counts.values())

    by_remainder = sorted(
        mix_ratio,
        key=lambda lang: total_samples * mix_ratio[lang] % 100,
        reverse=True
    )

    for lang in by_remainder[:leftover]:
        counts[lang] += 1

    return counts

• Deduplication with re-prompting: repeats are dropped and the model is asked again, within a fixed call budget. If it still falls short, the tool warns and returns a smaller dataset rather than padding it with repeats.
• Rendering every script: PDFs use a Noto font per script, with proper letter shaping and right-to-left Urdu.
• Scriptable: passing the settings as command-line flags skips the interactive questions, so runs can be automated.
• Document generator: a sub-tool writes a complete article with headings, paragraphs, bullet lists and tables in English or Hindi, renders it as a PDF, and saves JSON recording exactly what the PDF contains. Each pair works as test data for document parsers, text extraction and RAG pipelines.

This was a single comparison of 27 sentences, not a benchmark. When a language comes up short, the tool warns, and the README recommends trying a different model. The test suite has 37 pytest tests for the sentence generator and 11 for the document generator, and GitHub Actions runs both on every push to main and every pull request.

• The script check validates the writing system, not grammar, fluency or factual accuracy.
• MP3 export uses Google's online text-to-speech (gTTS), so audio isn't fully offline.
• Only Ollama is supported as a model server for now.

• Offline text-to-speech, so audio datasets don't need the internet.
• Support for other local servers such as LM Studio and llama.cpp.
• More South Asian languages, such as Odia, Assamese and Kashmiri.