Cikin Byte-Pair Encoding (BPE): Yadda LLMs ke Raba Haruffa zuwa Ƙananan Alamomi
Tsarin gine-ginen mai sauyawa na zamani—ciki har da jerin GPT-5 da GPT-4.1 na OpenAI, Claude 4.6 na Anthropic, Gemini 3.1 da 2.5 na Google, Grok 4 na xAI, Llama 4 na Meta, da DeepSeek V3/R1—suna sarrafa rubutu ta hanyar ƙididdigar raba alama ta ƙaramin kalma kamar Byte-Pair Encoding (BPE) da SentencePiece. Maimakon kiyaye ƙamus marasa iyaka na kowane kalma a kowane harshe, BPE yana farawa da ɗaiɗaikun bayis ɗin UTF-8 kuma yana haɗa ma'auratan harafi mafi yawanci kusa da juna zuwa ƙananan alamomi.
Sakamakon haka, kalmar Turanci ta kowa tare da sarari na gaba kamar ` intelligence` ana ɓoye ta azaman alamar lamba guda ɗaya, yayin da kalmar haɗin gwiwa da ba a saba gani ba, haxadecimal hash, ko toshe na Python mai zurfi ana raba ta zuwa ɗaruruwan alamomi ɗaya ɗaya. Mai raba alama ta gani na Multinook yana sake gina waɗannan iyakokin ƙananan alama a cikin lokaci na ainihi a cikin RAM na mai binciken ku.

