Recommendations for extended identifier characters for C and C++

Introduction

In response to their 2010 FCD ballot, WG21 received the following comment (designated as CA 24) from the Canadian national body:

A list of issues related TR 10176:2003

1. "Combining characters should not appear as the first character of an identifier." Reference: ISO/IEC TR 10176:2003 (Annex A) This is not reflected in FCD.
2. Restrictions on the first character of an identifier are not observed as recommended in TR 10176:2003. The inclusion of digits (outside of those in the basic character set) under identifer-nondigit is implied by FCD.
3. It is implied that only the "main listing" from Annex A is included for C++. That is, the list ends with the Special Characters section. This is not made explicit in FCD. Existing practice in C++03 as well as WG 14 (C, as of N1425) and WG 4 (COBOL, as of N4315) is to include a list in a normative Annex.
4. Specify width sensitivity as implied by C++03: \uFF21 is not the same as A. Case sensitivity is already stated in [lex.name].

It is reasonable to expect that WG14 would receive a very similar comment in response to an upcoming ballot.

Background

In investigating what various standards say about extended characters in identifiers, the following facts came to light.

General

1. The C++ WD now cites TR 10176:2003 for the specification of extended identifier characters.
2. The C standard (and WD) incorporates the lists from TR 10176:1998 for the specification of valid extended identifier characters.
3. There are differences between the lists in the C standard and in TR 10176:2003.
4. The lists in TR 10176:2003 are recommended as the minimum set of characters that an implementation should allow in identifiers.
5. The C standard explicitly allows implementations to accept other characters (implementation-defined) outside the basic source character set in identifiers.
6. The C++ standard (and WD) gives no such permission.
7. In C, a UCN not in the set of those specified as allowed is undefined behavior.
8. In C++, a UCN not in the set of those specified as allowed requires a diagnostic.
9. TR 10176 is based on the same principles as the "Default Identifier Syntax" defined in Unicode UAX#31 (see [DefId]): very roughly, characters defined to be letters are allowed initially, characters defined to be digits and combining marks are allowed non-initially.
10. The identifier character set in XML 1.0 was originally defined using the same principles.
11. For the sake of stability in the presence of an expanding character set, UAX#31 also defines an "Alternative Identifier Syntax" (see [AltId]), in which anything is allowed but: white space, "syntax" characters, private use characters, surrogates, control characters, and non-characters.
12. Unicode defines "syntax" characters to include most punctuation and symbols (including mathematical operators).
13. XML 1.1 also defines the identifier character set generously, excluding only certain characters and character ranges.
14. The latest edition of XML 1.0 has adopted the identifier specification from XML 1.1.

Combining characters

1. The C standard (and presumably TR 10176:1998) does not list combining characters, compatibility presentation forms, or fullwidth or halfwidth forms as valid in identifiers.
2. TR 10176:2003 has a separate list, also in Annex A, for combining characters, compatibility presentation forms, fullwidth & halfwidth forms, etc.
3. The C++ standard references Annex A, without even acknowledging that it has two parts.
4. TR 10176:2003 recommends that a combining character should not appear as the first character of an identifier.

Digits

1. TR 10176:2003 gives digits as an example of a kind of character often not allowed initially, and calls them out as a separate category in Annex A, but makes no actual recommendation.
2. In C, a UCN representing a digit is not allowed to start an identifier.
3. The C++ standard has no such restriction.

Halfwidth and fullwidth variants

1. TR 10176:2003 makes no recommendation with respect to halfwidth or fullwidth variants, but notes that COBOL (in particular) considers halfwidth and fullwidth variants to be equivalent to the original character (at least in some contexts).
2. The C standard does not explicitly include halfwidth or fullwidth variants as identifier characters.
3. The C++ standard is silent on this topic.

Discussion

General

WG14 and WG21 have experienced that trying to keep a language standard in synch with an expanding character set definition can be problematic. The Unicode Consortium and the World Wide Web Consortium have both acknowledged this, and provided (normative) guidance to avoid the problem. Although TR 10176 is based on problematic principles, the recommendation that at least the specified characters should be accepted in identifiers can be completely satisfied using the [AltId] approach.Therefore, it seems reasonable to abandon the [DefId] and TR 10176 approach, in favor of something simpler and more stable.

In a sane world, C and C++ would use the same definition of valid extended identifier characters. In an ideal world, the definition would appear in the C standard, and be referenced by the C++ standard. The publication schedules of WG14 and WG21 would appear to put an ideal world out of reach. But putting textually identical specifications in annexes of both C and C++, based on the [AltId] principles, would seem to be feasible.

Specifics

Estalishing the principle by which the set of valid extended identifier characters will be defined is clearly not enough; it is also necessary to select the exact definition.

In an ideal world, it would be possible to cite a definition from a different standard (as C++ was going to attempt to do by referencing [TR2003]). Citing the identifier syntax from [XML2008] would be problematic, for several reasons:

1. It's not an ISO standard.
2. It covers so much more than just the identifier syntax.
3. Its identifier syntax allows some inappropriate characters (including the basic source characters "-", "." and ":", and several punctuation marks used in CJK text), and disallows some inappropriate characters (including "ª", "µ" and "º", which were allowed in C99).

The content and organization of UAX#31 would make it easier to cite. It's still not an ISO standard, but that may not be an insuperable problem. In general, the current definition of [AltId] seems to be better than [XML2008] at tracking recent assignments of blocks in Unicode. But the exact definition of [AltId] has what appears to be a very serious flaw.

[AltId] defines a property called "Pattern_White_Space"; characters having that property are disallowed as identifier characters.Several other categories of characters are also disallowed, including "Pattern_Syntax" characters and control characters. The "Pattern_White_Space" category includes the ASCII SPACE character, several control characters (including HT, LF and CR), and a few others (including the Unicode LINE SEPARATOR and PARAGRAPH SEPARATOR characters) — but it does not include several other characters defined as spaces, including the Latin-1 NO-BREAK SPACE (NBSP).

In addition, it should be noted that [AltId] seems to be unique among standards and recommendations for identifiers, in having no restriction on characters allowed initially.

Combining characters

There is a fairly serious technical reason why an identifier should not start with a combining character: a combining character combines, semantically and visually, with the character preceding it. So disallowing this would prevent potentially serious, gratuitous confusion. Thus TR 10176 (reasonably) recommends that it be disallowed.

But [AltId] imposes no restrictions on the first character of an identifier. On the other hand, [XML2008] disallows (some) general combining characters initially, but not any script-specific combining character.

It should be noted that the C and C++ standards published to date appear to scrupulously disallow all combining characters in identifiers — both general and script-specific.Therefore, it has not previously been necessary to place restrictions on whether to allow them initially. It is difficult to imagine how to continue to avoid this problem using an extended identifier character definition based on [AltId].

Digits

While an identifier starting with a script-specific digit might be confused with a number by a human (who recognizes the digit as such), it will not be so confused by a C or C++ compiler — unless the compiler has been specifically extended to recognize script-specific numerical literals. Thus the problem of initial script-specific digits is less severe than the problem of initial combining characters (which is inherent in the structure of Unicode). TR 10176 does not go so far as to actually recommend that they be disallowed — it only mentions that they might be disallowed.

To date, C++ has never allowed script-specific digits as identifier characters, initially or otherwise. C allows them. While C disallows them initially, the "shall" imposing the requirement is not in a "Constraint" section, which of course means that implementations have not, strictly speaking, been required to diagnose them; instead, an identifier starting with a script-specific digit yields undefined behavior.

Halfwidth and fullwidth variants

This is the tip of a very large iceberg; see UAX#15 (if you dare) for much, much more information. Basically, width variations are just one of a dozen or so ways in which one character, or sequence of characters, can be confused for another. Unicode defines four different normalization forms, which can be used to resolve these confusions. The fact that there are multiple normalization forms can reasonably be taken as an indication of the complexity of the situation.

Any standard (including COBOL and TR 10176) that talks about width variations, and no other form of canonical or compatibility equivalence, is very likely demonstrating a potent combination of hubris and ignorance.

I think the C and C++ standards should be silent on this whole topic. An mplementer should be able to decide whether his implementation should normalize or not, and if so which normalization form should be used, based on his understanding of the needs of his customers. The implication of that would be that users should never name different things using identifiers that would normalize to the same string, nor attempt to reference something using anything but its exact name (for example, by using a name that would normalize to the same string as the original name).

Recommendations

The definition of the ranges of UCNs allowed in an identifier should appear in an annex in each standard; the text of these two annexes should be identical. (The wording of the citations of these annexes will be different between the two standards.)

The set of UCNs disallowed in identifiers in C and C++ should exactly match the specification in [AltId], with the following additions: all characters in the Basic Latin (i.e. ASCII, basic source character) block, and all characters in the Unicode General Category "Separator, space".

General combining characters, appearing in blocks dedicated to that purpose, should be disallowed as the initial character of an identifier. (Script-specific combining characters would be allowed initially, only because of the additional complexity and instability of specifying them.)

There should be no restriction on script-specific digits initally in an identifier.

Proposed wording

The annex specifying the ranges of allowed identifier characters should be Annex D in the C standard, and Annex E in the C++ standard. It should have two sub-clauses: one for allowed identifier characters, and one for characters disallowed initially.

Annex

X. Universal character names for identifier characters (normative)

X.1 Ranges of characters allowed

00A8, 00AA, 00AD, 00AF, 00B2-00B5, 00B7-00BA, 00BC-00BE, 00C0-00D6, 00D8-00F6, 00F8-00FF

0100-167F, 1681-180D, 180F-1FFF

200B-200D, 202A-202E, 203F-2040, 2054, 2060-206F

2070-218F, 2460-24FF, 2776-2793, 2C00-2DFF, 2E80-2FFF

3004-3007, 3021-302F, 3031-303F

3040-D7FF

F900-FD3D, FD40-FDCF, FDF0-FE44, FE47-FFFD

10000-1FFFD, 20000-2FFFD, 30000-3FFFD, 40000-4FFFD, 50000-5FFFD, 60000-6FFFD, 70000-7FFFD, 80000-8FFFD, 90000-9FFFD, A0000-AFFFD, B0000-BFFFD, C0000-CFFFD, D0000-DFFFD, E0000-EFFFD

X.2 Ranges of characters disallowed initially

0300-036F, 1DC0-1DFF, 20D0-20FF, FE20-FE2F

Citing the annex from C

Change 6.2.4.1p3:

Each universal character name in an identifier shall designate a character whose encoding in ISO/IEC 10646 falls into one of the ranges specified in annex D, subclause D.1.⁷¹⁾ The initial character shall not be a universal character name designating a digit character whose encoding falls into one of the ranges specified in subclause D.2. An implementation may allow multibyte characters that are not part of the basic source character set to appear in identifiers; which characters and their correspondence to universal character names is implementation-defined.

Citing the annex from C++

Change 2.11p1:

An identifier is an arbitrarily long sequence of letters and digits. Each universal-character-name in an identifier shall designate a character whose encoding in ISO 10646 falls into one of the ranges specified in Annex A of TR 10176:2003 annex E, subclause E.1. The initial element shall not be a universal-character-name designating a character whose encoding falls into one of the ranges specified in subclause E.2. Upper- and lower-case letters are different. All characters are significant.¹⁹

It may also be appropriate to delete the normative reference to TR 10176:2003. Even though the standard will (substantially) follow its recommendations for extended identifier characters, there will remain no actual reference to it.

References

[AltId]

Unicode Standard Annex #31: Unicode Identifier and Pattern Syntax, "Alternative Identifier Syntax", http://www.unicode.org/reports/tr31/tr31-11.html#Alternative_Identifier_Syntax

[DefId]

Unicode Standard Annex #31: Unicode Identifier and Pattern Syntax, "Default Identifier Syntax", http://www.unicode.org/reports/tr31/tr31-11.html#Default_Identifier_Syntax

[TR2003]

ISO/IEC TR 10176:2003 (WG 20 DTR ballot draft), http://www.open-std.org/JTC1/sc22/WG20/docs/n970-tr10176-2002.pdf

[TR2003a]

Online text of identifier character repertoire recommended by [TR2003], http://www.iso.org/ittf/ISOIEC_TR_10176_2003_Table.txt

[XML2006]

Extensible Markup Language (XML) 1.0 (Fourth Edition), "Common Syntactic Constructs", http://www.w3.org/TR/2006/REC-xml-20060816/#sec-common-syn

[XML2008]

Extensible Markup Language (XML) 1.0 (Fifth Edition), "Common Syntactic Constructs", http://www.w3.org/TR/2008/REC-xml-20081126/#sec-common-syn

Appendix: [AltId] and [XML2008] illustrated

The Basic Multilingual Plane

Legend: Characters disallowed by [AltId] are indicated by a box. Characters disallowed by [XML2008] are indicated by a gray background.

Every block is presented, most only by name. Certain blocks (especially including those corresponding to the ASCII and Latin-1 "legacy encodings") have significant numbers of punctuation and symbol characters; these are presented character by character. Each assigned, non-control character appears in the HTML source as the appropriate character entity; if it doesn't display correctly in your browser, the fault is almost certainly in your browser setup. The formal name of each character is also present an HTML title, which will hopefully pop up if you hover your mouse pointer over the character.

It should be noted that [AltId] is unique in having no distinction between characters allowed initially and non-initially in an identifier. Where [XML2008] makes such a distinction, it is indicated below as a note. There are also notes for a few isolated (plausible) non-identifier characters.

Beyond the BMP

The Supplementary Private Use Area extends from F0000 through 10FFFF; both [AltId] and [XML2008] disallow characters in that range.

In addition, [AltId] disallows, as non-characters, the last two code positions of each plane, i.e. every position of the form PFFFE or PFFFF, for any value of P.

Otherwise, no character outside the BMP is disallowed as an identifier character by either specification.