Preface to translation

This is a translation of the explanatory part of the proposal Intl.Segmenter, which is likely to be added to the next ECMAScript specification.

The proposal has already been implemented in V8 and without the flag can be used in version 8.7 (more precisely, in 8.7.38and above), so it can be tested in Google Chrome Canary (starting from version 87.0.4252.0) or in Node.js V8 Canary (starting from version v15.0.0-v8-canary202009025a2ca762b8; for Windows binaries are available v15.0.0-v8-canary202009173b56586162).

If you test in earlier versions with the flag --harmony-intl-segmenter, be careful as the specification has changed and the implementation under the flag may be out of date. Check by output in code examples.

After the translation, links are provided to materials about the reasons for the problems that this proposal solves.

`Intl.Segmenter`: Unicode segmentation in JavaScript

Proposal is in Stage 3 with support from Richard Gibson.

Motivation

(code point) «» . , (, ). , . , .

, CLDR (Common Locale Data Repository, ) (, locales). , , , .

, UAX 29. , JavaScript .

Chrome API Intl.v8BreakIterator. API . API, API JavaScript — , ES2015.

, segment(), Intl.Segmenter, Iterable.

//      .
let segmenter = new Intl.Segmenter("fr", {granularity: "word"});

//       .
let input = "Moi?  N'est-ce pas.";
let segments = segmenter.segment(input);

//    !
for (let {segment, index, isWordLike} of segments) {
  console.log("segment at code units [%d, %d): «%s»%s",
    index, index + segment.length,
    segment,
    isWordLike ? " (word-like)" : ""
  );
}

//  console.log:
// segment at code units [0, 3): «Moi» (word-like)
// segment at code units [3, 4): «?»
// segment at code units [4, 6): «  »
// segment at code units [6, 11): «N'est» (word-like)
// segment at code units [11, 12): «-»
// segment at code units [12, 14): «ce» (word-like)
// segment at code units [14, 15): « »
// segment at code units [15, 18): «pas» (word-like)
// segment at code units [18, 19): «.»

, API .

// ┃0 1 2 3 4 5┃6┃7┃8┃9
// ┃A l l o n s┃-┃y┃!┃
let input = "Allons-y!";

let segmenter = new Intl.Segmenter("fr", {granularity: "word"});
let segments = segmenter.segment(input);
let current = undefined;

current = segments.containing(0)
// → { index: 0, segment: "Allons", isWordLike: true }

current = segments.containing(5)
// → { index: 0, segment: "Allons", isWordLike: true }

current = segments.containing(6)
// → { index: 6, segment: "-", isWordLike: false }

current = segments.containing(current.index + current.segment.length)
// → { index: 7, segment: "y", isWordLike: true }

current = segments.containing(current.index + current.segment.length)
// → { index: 8, segment: "!", isWordLike: false }

current = segments.containing(current.index + current.segment.length)
// → undefined

API

`new Intl.Segmenter(locale, options)`

options , granularity, ("grapheme" ( ), "word" ( ) "sentence" ( ); — "grapheme").

`Intl.Segmenter.prototype.segment(string)`

%Segments% Iterable .

segment — .
index — (code unit index) , .
input — .
isWordLike — true, "word" ( ) ( /// ..); false, "word" ( // ..); undefined, "word".

%Segments%.prototype:

`%Segments%.prototype.containing(index)`

, , (code unit) , undefined, .

`%Segments%.prototype[Symbol.iterator]`

%SegmentIterator%, "" (lazy, ) , .

%SegmentIterator%.prototype:

`%SegmentIterator%.prototype.next()`

next() Iterator, IteratorResult, value , .

FAQ

? ?

— , . . . CLDR. , CLDR/ICU , .

API ?

, 3- , . TC39 . ; , , .

?

API, , API : , API (, ). API CSS Houdini.

?

API:

.
.
, (.. Web API (Web Platform), ECMAScript).
, . CLDR ICU . CSS, . . , , , ; .

?

%SegmentIterator%.prototype, (, seek([inclusiveStartIndex = thisIterator.index + 1]) seekBefore([exclusiveLastIndex = thisIterator.index]), . ECMA-262 ( ). , , .

API Intl, String?

, . segment() SegmentIterator. , API Intl, ECMA-402. , . String, , .

?

n (code unit), . , "Hello, world\u{1F499}" ( , - — ), 0, 5, 6, 7 12. : ┃Hello┃,┃ ┃world┃\u{1F499}┃, (code units), (code point). , .

?

, next().

, ?

, - QA ;)

Number: null 0, — 0 1, , , Symbol BigInt, undefined NaN *. , ( , ).

* . "fail". Chrome Canary, Symbol BigInt TypeError, undefined NaN , 0.

JavaScript.

Intl.Segmenter: Unicode segmentation in JavaScript