NER on Code-Switched Spanish-English Tweets

Code-switched text is where multilingual NER models struggle most: entity boundaries often sit exactly where the language flips, and a token near a Spanish-English switch can be wrongly pulled into an entity span. The research question was whether an explicit language signal can fix that.

The dataset is the LINCE SpaEng benchmark: 67,223 tweets (33,611 train / 10,085 validation / 23,527 test) with IOB2 tags over nine entity types. It is heavily imbalanced — only ~2.4% of tokens are entities — and informally written, with slang and missing capitalization.

The core contribution is architectural but deliberately lightweight: the transformer encoder is untouched. Each token's LID label gets a small trainable embedding; a gating network decides, per entity type, how strongly language should matter; and a learned bias table capturing entity-type-language associations is added to the classifier logits.

Everything was trained on an HPC cluster via SLURM, with a one-at-a-time hyperparameter exploration around a fixed baseline configuration, and evaluated with strict span-level F1 (seqeval).

Multilingual transformers are pre-trained on formal, mostly monolingual text; tweets that mix Spanish and English mid-sentence break their assumptions.

Entity boundaries frequently coincide with language switch points, where models incorrectly extend or cut entity spans.

Entities are rare (~2.4% of tokens) and the nine classes are heavily imbalanced — PER has 940 test-support entities, EVENT just 46.

Prior work often attacks this with expensive secondary pre-training on translation pairs; the goal here was a cheap architectural alternative.

A four-person course project — designed and decided together, with shared responsibility for the results and the paper.

My hands-on focus was the engineering: I built most of the codebase — the training and evaluation pipeline, data loading with IOB2 validation and sub-word label masking, and the prediction/export tooling.

Implemented the language-bias architecture: LID embeddings, the gating network, and the entity-type-language bias table on top of the frozen classifier design.

Set up the SLURM cluster workflow for environment setup, training, and prediction, plus the analysis scripts behind the confusion matrices, per-class F1 breakdowns, and learning curves.

Fine-tuned two multilingual backbones — mBERT and XLM-RoBERTa — as sequence labelers with cross-entropy over BIO tags, aligning labels to first sub-tokens and masking the rest.

Added the language-aware variant: token-level LID labels enter through trainable embeddings, a gating network scales their influence per entity type, and a learned bias table adjusts the final logits — no change to the encoder, no second transformer.

Trained all four configurations identically (AdamW, lr 3e-5, batch 8, 5 epochs, max length 128) so the comparison isolates the architecture change.

Evaluated with strict span-level micro F1 via seqeval — both entity type and exact boundaries must match — and analyzed errors through confusion matrices, per-class F1, learning curves, and manual reading of failing tweets.

The language bias improved both backbones: mBERT went from span F1 0.6198 to 0.6329, XLM-RoBERTa from 0.6358 to 0.6487 — the best model overall.

The gain came mainly from precision (0.6352 → 0.6541 for XLM-R): explicit language information helps the model reject tokens that don't belong in an entity span, exactly the failure mode near switch points.

LID-equipped models also trained better — reaching their best F1 by around epoch 3 with visibly smoother loss curves than the baselines.

High-support classes (PER 0.786, LOC 0.734) were strong and stable; rare classes (EVENT, TIME) stayed hard, confirming support as the binding constraint.

The full study is written up in a course paper, hosted here, with the code public on GitHub.