2. Build-Corpus.md

1. Build the Corpus

This demo shows how to use cntext to read multiple CSV files, clean Chinese user-generated text, save a merged CSV file, and write a plain-text corpus file for embedding training.

The example assumes that each raw CSV file contains one text column, such as text, content, comment, or post_text. The examples use Chinese text because the paper workflow is platform-specific and uses lang="chinese". English translations are provided in comments.

Folder Layout

Recommended working layout:

WEPA-DEMO/
  raw_csv/
    keep_posts_001.csv
    keep_posts_002.csv
    keep_posts_003.csv
  outputs/

Each CSV can look like this:

user_id,created_at,text
u001,2024-01-01,今天完成了5公里跑步，坚持训练很有成就感。
u002,2024-01-01,最近有点累，但还是想继续打卡。
u003,2024-01-02,目标还没有完成，明天继续努力。

Translations:

• 今天完成了5公里跑步，坚持训练很有成就感。 means "I completed a 5 km run today and feel accomplished for keeping up with training."
• 最近有点累，但还是想继续打卡。 means "I have been a little tired recently, but I still want to keep checking in."
• 目标还没有完成，明天继续努力。 means "The goal is not finished yet; I will keep trying tomorrow."

Read Multiple CSV Files

Use ct.read_files to read all CSV files matching a path pattern.

from pathlib import Path

import cntext as ct
import pandas as pd

RAW_DIR = Path("WEPA-DEMO/raw_csv")
OUT_DIR = Path("WEPA-DEMO/outputs")
OUT_DIR.mkdir(parents=True, exist_ok=True)

# Read all CSV files. Each CSV should contain a text column.
df = ct.read_files(
    fformat=str(RAW_DIR / "*.csv"),
    encoding="utf-8",
    recursive=False,
)

print(df.head())
print(df.shape)

Select the Text Column

Different platforms or preprocessing scripts may use different column names. Choose the column that contains the raw text.

candidate_columns = ["text", "content", "comment", "post_text", "body"]
text_col = next((col for col in candidate_columns if col in df.columns), None)

if text_col is None:
    raise ValueError(
        f"No text column found. Expected one of: {candidate_columns}. "
        f"Available columns: {list(df.columns)}"
    )

df = df.copy()
df["raw_text"] = df[text_col].astype(str)

Clean Chinese Text with cntext

Use cntext's built-in cleaning function. The lang="chinese" setting keeps Chinese text and normalizes common noise such as URLs and digits.

df["clean_text"] = df["raw_text"].map(
    lambda x: ct.clean_text(x, lang="chinese")
)

# Remove empty strings after cleaning.
df = df[df["clean_text"].str.len() > 0].reset_index(drop=True)

Example:

example = "今天完成了5公里跑步，坚持训练很有成就感。https://example.com"
print(ct.clean_text(example, lang="chinese"))

Possible output:

今天完成了数字公里跑步，坚持训练很有成就感。

Translation: "I completed a NUMBER km run today and feel accomplished for keeping up with training."

Save the Merged CSV

The merged CSV is useful for later document-level scoring and diagnostics.

merged_csv = OUT_DIR / "all_texts.csv"
df.to_csv(merged_csv, index=False, encoding="utf-8-sig")

print(f"Saved merged CSV to: {merged_csv}")

Write corpus.txt

Embedding training functions expect a plain-text corpus file. Write one cleaned document per line. Keeping line breaks helps cntext process large corpora safely.

corpus_file = OUT_DIR / "corpus.txt"

with corpus_file.open("w", encoding="utf-8") as f:
    for text in df["clean_text"]:
        f.write(text.replace("\n", " ").strip() + "\n")

print(f"Saved corpus file to: {corpus_file}")
print(f"Number of documents: {len(df)}")

Quality Checks Before Training

Run basic checks before training embeddings:

print(df["clean_text"].head(10).to_string(index=False))
print(df["clean_text"].str.len().describe())

Recommended checks:

• Make sure the text column is correct.
• Make sure rows are not accidentally duplicated.
• Make sure the cleaned corpus still contains meaningful Chinese text.
• Keep user privacy protections in place. Do not publish raw user-generated text unless you have appropriate permission.

Output files from this step:

WEPA-DEMO/outputs/all_texts.csv
WEPA-DEMO/outputs/corpus.txt