Skip to content
Guide

Using an LLM to classify Arabic customer messages

By Fayez AlshamiPublished: Updated: 6 min read
TL;DR

Let the model label intent, language, and whether digits look like a transaction ID. Marking an order paid stays in ordinary code - transaction ID plus amount via the gateway, with no memo field on the wallet transfer.

A Syrian shop inbox is messy in a particular way. One customer writes «تم التحويل». The next pastes a long number and nothing else. Someone asks «وين طلبتي» with a typo. A fourth writes tm alt7weel in Arabizi. If your bot treats every number as a payment, or waits for a person to read every line, the wrong order gets marked paid, or a paid order sits untouched until morning.

I use an LLM to classify those messages. I do not use it to decide that a payment is valid. The model answers a narrower question: is this a new order, a transaction ID, a status question, a complaint, a cancellation, or something else? Payment matching stays deterministic. The bot takes a transaction ID and an amount to a hosted gateway, the same way I describe in How to build a Telegram bot that accepts Syriatel Cash and in Syriatel Cash API: Accept Payments on Your Site. The model does not get a vote on whether the money arrived.

The Arabic walkthrough of a full support bot, including the replies I show the customer, is بوت دعم عربي للزبائن. This article stays on the classification layer: labels, extraction, dialect, and the boundary around payments.

What I ask the model to return

A short list of intents, in JSON, every time. For the shops I wire this into, the useful set is usually: new_order, transaction_id, payment_claim, order_status, complaint, cancel, greeting, other. I also ask for a confidence score, a language tag (ar, arabizi, mixed, or en), a transaction ID if one is actually present, and a needs_human flag.

payment_claim matters. Plenty of people write «دفعت» or «تم» and attach no number. That is not a transaction ID. The classifier should say so, and the bot should ask for رقم العملية. Syriatel Cash and Sham Cash transfers have no memo field, so the customer cannot put an order code inside the transfer itself. The ID appears after the payment, in the app or the SMS, and that is what they paste into the chat.

I keep the taxonomy small on purpose. A list of twenty intents looks thorough and then collapses in production, because the model starts inventing fine shades that your code does not handle.

What the model is not allowed to do

It does not mark an order paid. It does not invent or "correct" a transaction ID when the digits are messy. It does not treat the sender's phone number as proof of payment. The gateway returns the sender number on every transfer, and I store it with the order, but matching is still transaction ID plus amount.

If a message is a complaint that happens to include a number, the intent is complaint and needs_human is true. A support reply can go out later. The payment path does not run just because digits appeared.

Incoming confirmations also arrive as webhooks from the gateway, with no customer message at all. That path is covered in Designing reliable payment webhooks. Both paths must share the same rule: one transaction ID is applied once, and the amount has to match the open order.

Arabic that breaks a keyword list

A list of words such as «دفعت» and «تم» fails quickly. People write Syrian colloquial, Modern Standard Arabic, Arabizi, or a mix in the same line. They paste the operator SMS, wrap the ID in a sentence, use Arabic-Indic digits (١٢٣٤), or write rqm al3amleye and put the digits on the next line.

A message that is only a long digit run can go to verification without the model. The model is there for the buried number, the dialect, and the cases where a number is not a payment at all.

The pipeline

I normalize the text: Unicode compatibility form, Arabic-Indic digits to ASCII, collapsed whitespace. I collect long digit runs as candidates. I send the normalized message and those candidates to a model with a strict JSON schema. After the response, I apply my own rules: if intent is not transaction_id, drop any ID the model filled in; if confidence is below the threshold I set for that shop, I send it to a person.

The threshold is a setting. I start around 0.7 and change it after I read that shop's real messages.

```python import json import os import re import unicodedata from openai import OpenAI

INTENTS = ( "new_order", "transaction_id", "payment_claim", "order_status", "complaint", "cancel", "greeting", "other", )

# Long digit runs only; no official wallet ID length is assumed. DIGIT_RUN = re.compile(r"\d{8,24}") ARABIC_INDIC = str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789") EASTERN_ARABIC = str.maketrans("۰۱۲۳۴۵۶۷۸۹", "0123456789")

SCHEMA = { "type": "object", "additionalProperties": False, "properties": { "intent": {"type": "string", "enum": list(INTENTS)}, "confidence": {"type": "number"}, "language": { "type": "string", "enum": ["ar", "arabizi", "mixed", "en"], }, "transaction_id": {"type": ["string", "null"]}, "needs_human": {"type": "boolean"}, }, "required": ["intent", "confidence", "language", "transaction_id", "needs_human"], }

SYSTEM = """Classify shop messages (Syrian Arabic, MSA, Arabizi, or mixed). JSON only. Do not decide if a payment is valid or mark an order paid. transaction_id is the pasted operation number, or null. If they say they paid but gave no number, intent is payment_claim. needs_human is true for complaints, or when unsure. """

client = OpenAI( api_key=os.environ["LLM_API_KEY"], base_url=os.environ.get("LLM_BASE_URL"), )

def normalize(text: str) -> str: text = unicodedata.normalize("NFKC", text) text = text.translate(ARABIC_INDIC).translate(EASTERN_ARABIC) return " ".join(text.split())

def classify(message: str, min_confidence: float = 0.7) -> dict: normalized = normalize(message) candidates = DIGIT_RUN.findall(normalized) completion = client.chat.completions.create( model=os.environ["LLM_MODEL"], temperature=0, response_format={ "type": "json_schema", "json_schema": { "name": "classification", "strict": True, "schema": SCHEMA, }, }, messages=[ {"role": "system", "content": SYSTEM}, { "role": "user", "content": json.dumps( {"message": normalized, "digit_candidates": candidates}, ensure_ascii=False, ), }, ], ) result = json.loads(completion.choices[0].message.content) if result["intent"] != "transaction_id": result["transaction_id"] = None if result["confidence"] < min_confidence: result["needs_human"] = True return result

```

After classification, transaction_id goes to ordinary code: ask the gateway for that ID, check the amount against an open order, and refuse to reuse an ID that already paid a different order. The LLM is not in that function. A payment_claim with no ID gets a reply asking for رقم العملية, because the transfer itself has no memo field. Complaints go to a person.

If a provider does not accept json_schema, I use json_object mode and validate the object in Python. The model and base URL stay in environment variables.

Mistakes I watch for when I hook this up

The costly one is letting the model "fix" a typo inside a transaction ID. A single wrong digit will either miss the payment or, worse, collide with a different transfer if you are sloppy about amount checks. If the digits are doubtful, I ask the customer to paste the ID again, or I wait for the gateway webhook.

The other mistake is answering the customer with policy the model invented. This layer classifies. Replies for complaints and refunds go to a person, or to copy you have actually written.

Before going live I run the classifier on a sample from that shop's real chat, including Arabizi and pasted SMS text, and I look at the misses by hand. I do not publish a generic accuracy rate.

Frequently asked questions

Do I need an LLM if most people just paste the number?

Not for those messages. A standalone long digit run can go straight to verification. The model helps when the number sits inside a sentence, when the text is dialect or Arabizi, and when a number is part of a complaint rather than a payment.

Can the classifier mark the order as paid?

No. Matching is transaction ID plus amount, through the gateway. The sender number is stored as extra information. There is no memo field on the transfer.

What about screenshots and voice notes?

This classifier is text. Transcribe a voice note first. For a screenshot, still get the transaction ID from the customer or from the gateway. I do not mark an order paid from a receipt image alone.

Guides & comparisons

Need this built or connected for your business?

See services