Q: Lonely surrogates and unicode regexps

[André Bargull]

On 1/28/2015 3:36 PM, Marja Hölttä wrote:
> Based on Ex1, looks like the input string is not read as a sequence of code points when we try to
> find a match for \1. So it's mostly read as a sequence of code points except when it's not. :/

Yep, back references are matched as a sequence of code units. The first link I've posted points to 
the relevant method in java.util.regex.Pattern. I've got no idea why it's implemented that way, for 
example when you enable case-insensitive matching, back references are no longer matched as a 
sequence of code units:

---
int[] flags = { 0, Pattern.CASE_INSENSITIVE, Pattern.UNICODE_CASE,
         Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE };

// Prints true, false, true, false
Arrays.stream(flags).mapToObj(f -> Pattern.compile("foo(.+)bar\\1", f))
         .map(p -> p.matcher("foo\uD834bar\uD834\uDC00").find())
         .forEach(System.out::println);
---



>
> On Wed, Jan 28, 2015 at 3:11 PM, André Bargull <andre.bargull at udo.edu
> <mailto:andre.bargull at udo.edu>> wrote:
>
>     On 1/28/2015 2:51 PM, André Bargull wrote:
>
>             For a reference, here's how Java (tried w/ Oracle 1.8.0_31 and openjdk
>             1.7.0_65) Pattern.UNICODE_CHARACTER___CLASS works:
>
>             foo\uD834bar and foo\uDC00bar match ^foo[^a]bar$ and ^foo.bar$, so,
>             generally, lonely surrogates match /./.
>
>             Backreferences are allowed to consume the leading surrogate of a valid
>             surrogate pair:
>
>             Ex1: foo\uD834bar\uD834\uDC00 matches foo(.+)bar\1
>
>             But surprisingly:
>
>             Ex2: \uDC00foobar\uD834\__uDC00foobar\uD834 doesn't match ^(.+)\1$
>
>             ... So Ex2 works as if the input string was converted to UTF-32 before
>             matching, but Ex1 works as if it was def not. Idk what's the correct mental
>             model where both Ex1 and Ex2 would make sense.
>
>
>         java.util.regex.Pattern matches back references by comparing (Java) chars [1], but reads
>         patterns as a sequence of code points [2]. That should help to explain the differences
>         between ex1 and ex2.
>
>         [1]
>         http://hg.openjdk.java.net/__jdk8u/jdk8u/jdk/file/__c46daef6edb5/src/share/__classes/java/util/regex/__Pattern.java#l4890
>         <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/c46daef6edb5/src/share/classes/java/util/regex/Pattern.java#l4890>
>         [2]
>         http://hg.openjdk.java.net/__jdk8u/jdk8u/jdk/file/__c46daef6edb5/src/share/__classes/java/util/regex/__Pattern.java#l1671
>         <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/c46daef6edb5/src/share/classes/java/util/regex/Pattern.java#l1671>
>
>
>     Err, the part about how patterns are read is not important here. What I should have written is
>     that the input string is (also) read as a sequence of code points [3]. So in ex2 `\uD834\uDC00`
>     is read as a single code point (and not split into \uD834 and \uDC00 during backtracking).
>
>     [3]
>     http://hg.openjdk.java.net/__jdk8u/jdk8u/jdk/file/__c46daef6edb5/src/share/__classes/java/util/regex/__Pattern.java#l3773
>     <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/c46daef6edb5/src/share/classes/java/util/regex/Pattern.java#l3773>
>
>