I made a comment yesterday on an answer where someone had used [0123456789] in a regular expression rather than [0-9] or \d. I said it was probably more efficient to use a range or digit specifier than a character set.
I decided to test that out today and found out to my surprise that (in the C# regex engine at least) \d appears to be less efficient than either of the other two which don't seem to differ much. Here is my test output over 10000 random strings of 1000 random characters with 5077 actually containing a digit:
Regular expression \d took 00:00:00.2141226 result: 5077/10000
Regular expression [0-9] took 00:00:00.1357972 result: 5077/10000 63.42 % of first
Regular expression [0123456789] took 00:00:00.1388997 result: 5077/10000 64.87 % of first
It's a surprise to me for two reasons:
- I would have thought the range would be implemented much more efficiently than the set.
- I can't understand why
\dis worse than[0-9]. Is there more to\dthan simply shorthand for[0-9]?
Here is the test code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using System.Text.RegularExpressions;
namespace SO_RegexPerformance
{
class Program
{
static void Main(string[] args)
{
var rand = new Random(1234);
var strings = new List<string>();
//10K random strings
for (var i = 0; i < 10000; i++)
{
//Generate random string
var sb = new StringBuilder();
for (var c = 0; c < 1000; c++)
{
//Add a-z randomly
sb.Append((char)('a' + rand.Next(26)));
}
//In roughly 50% of them, put a digit
if (rand.Next(2) == 0)
{
//Replace one character with a digit, 0-9
sb[rand.Next(sb.Length)] = (char)('0' + rand.Next(10));
}
strings.Add(sb.ToString());
}
var baseTime = testPerfomance(strings, @"\d");
Console.WriteLine();
var testTime = testPerfomance(strings, "[0-9]");
Console.WriteLine(" {0:P2} of first", testTime.TotalMilliseconds / baseTime.TotalMilliseconds);
testTime = testPerfomance(strings, "[0123456789]");
Console.WriteLine(" {0:P2} of first", testTime.TotalMilliseconds / baseTime.TotalMilliseconds);
}
private static TimeSpan testPerfomance(List<string> strings, string regex)
{
var sw = new Stopwatch();
int successes = 0;
var rex = new Regex(regex);
sw.Start();
foreach (var str in strings)
{
if (rex.Match(str).Success)
{
successes++;
}
}
sw.Stop();
Console.Write("Regex {0,-12} took {1} result: {2}/{3}", regex, sw.Elapsed, successes, strings.Count);
return sw.Elapsed;
}
}
}
\d checks all Unicode digits, while [0-9] is limited to these 10 characters. For example, Persian digits, ۱۲۳۴۵۶۷۸۹, are an example of Unicode digits which are matched with \d, but not [0-9].
You can generate a list of all such characters using the following code:
var sb = new StringBuilder();
for(UInt16 i = 0; i < UInt16.MaxValue; i++)
{
string str = Convert.ToChar(i).ToString();
if (Regex.IsMatch(str, @"\d"))
sb.Append(str);
}
Console.WriteLine(sb.ToString());
Which generates:
0123456789٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹߀߁߂߃߄߅߆߇߈߉०१२३४५६७८९০১২৩৪৫৬৭৮৯੦੧੨੩੪੫੬੭੮੯૦૧૨૩૪૫૬૭૮૯୦୧୨୩୪୫୬୭୮୯௦௧௨௩௪௫௬௭௮௯౦౧౨౩౪౫౬౭౮౯೦೧೨೩೪೫೬೭೮೯൦൧൨൩൪൫൬൭൮൯๐๑๒๓๔๕๖๗๘๙໐໑໒໓໔໕໖໗໘໙༠༡༢༣༤༥༦༧༨༩၀၁၂၃၄၅၆၇၈၉႐႑႒႓႔႕႖႗႘႙០១២៣៤៥៦៧៨៩᠐᠑᠒᠓᠔᠕᠖᠗᠘᠙᥆᥇᥈᥉᥊᥋᥌᥍᥎᥏᧐᧑᧒᧓᧔᧕᧖᧗᧘᧙᭐᭑᭒᭓᭔᭕᭖᭗᭘᭙᮰᮱᮲᮳᮴᮵᮶᮷᮸᮹᱀᱁᱂᱃᱄᱅᱆᱇᱈᱉᱐᱑᱒᱓᱔᱕᱖᱗᱘᱙꘠꘡꘢꘣꘤꘥꘦꘧꘨꘩꣐꣑꣒꣓꣔꣕꣖꣗꣘꣙꤀꤁꤂꤃꤄꤅꤆꤇꤈꤉꩐꩑꩒꩓꩔꩕꩖꩗꩘꩙０１２３４５６７８９
Credit to ByteBlast for noticing this in the docs. Just changing the regex constructor:
var rex = new Regex(regex, RegexOptions.ECMAScript);
Gives new timings:
Regex \d took 00:00:00.1355787 result: 5077/10000
Regex [0-9] took 00:00:00.1360403 result: 5077/10000 100.34 % of first
Regex [0123456789] took 00:00:00.1362112 result: 5077/10000 100.47 % of first
From Does “\d” in regex mean a digit?:
[0-9]isn't equivalent to\d.[0-9]matches only0123456789characters, while\dmatches[0-9]and other digit characters, for example Eastern Arabic numerals٠١٢٣٤٥٦٧٨٩
An addition to top answer from Sina Iravianian, here is a .NET 4.5 version (since only that version supports UTF16 output, c.f. the first three lines) of his code, using the full range of Unicode code points. Due to the lack of proper support for higher Unicode planes, many people are not aware of always checking for and including the upper Unicode planes. Nevertheless they sometimes do contain some important characters.
public static void Main()
{
var unicodeEncoding = new UnicodeEncoding(!BitConverter.IsLittleEndian, false);
Console.InputEncoding = unicodeEncoding;
Console.OutputEncoding = unicodeEncoding;
var sb = new StringBuilder();
for (var codePoint = 0; codePoint < 0x10ffff; codePoint++)
{
var isSurrogateCodePoint =
codePoint <= UInt16.MaxValue
&& ( char.IsLowSurrogate((char) codePoint)
|| char.IsHighSurrogate((char) codePoint)
);
if (isSurrogateCodePoint)
continue;
var codePointString = char.ConvertFromUtf32(codePoint);
if (Regex.IsMatch(codePointString, @"\d"))
sb.AppendFormat("{0} ", codePointString);
}
Console.WriteLine(sb.ToString());
Console.ReadKey();
}
Yielding the following output:
0 1 2 3 4 5 6 7 8 9 ٠ ١ ٢ ٣ ٤ ٥ ٦ ٧ ٨ ٩ ۰ ۱ ۲ ۳ ۴ ۵ ۶ ۷ ۸ ۹ ߀ ߁ ߂ ߃ ߄ ߅ ߆ ߇ ߈ ߉ ० १ २ ३ ४ ५ ६ ७ ८ ९
০ ১ ২ ৩ ৪ ৫ ৬ ৭ ৮ ৯ ੦ ੧ ੨ ੩ ੪ ੫ ੬ ੭ ੮ ੯ ૦ ૧ ૨ ૩ ૪ ૫ ૬ ૭ ૮ ૯ ୦ ୧ ୨ ୩ ୪ ୫ ୬ ୭ ୮ ୯ ௦ ௧ ௨ ௩ ௪ ௫ ௬ ௭ ௮ ௯
౦ ౧ ౨ ౩ ౪ ౫ ౬ ౭ ౮ ౯ ೦ ೧ ೨ ೩ ೪ ೫ ೬ ೭ ೮ ೯ ൦ ൧ ൨ ൩ ൪ ൫ ൬ ൭ ൮ ൯
๐ ๑ ๒ ๓ ๔ ๕ ๖ ๗ ๘ ๙ ໐ ໑ ໒ ໓ ໔ ໕ ໖ ໗ ໘ ໙ ༠ ༡ ༢ ༣ ༤ ༥ ༦ ༧ ༨ ༩ ၀ ၁ ၂ ၃ ၄ ၅ ၆ ၇ ၈ ၉ ႐ ႑ ႒ ႓ ႔ ႕ ႖ ႗ ႘ ႙
០ ១ ២ ៣ ៤ ៥ ៦ ៧ ៨ ៩ ᠐ ᠑ ᠒ ᠓ ᠔ ᠕ ᠖ ᠗ ᠘ ᠙ ᥆ ᥇ ᥈ ᥉ ᥊ ᥋ ᥌ ᥍ ᥎ ᥏ ᧐ ᧑ ᧒ ᧓ ᧔ ᧕ ᧖ ᧗ ᧘ ᧙ ᭐ ᭑ ᭒ ᭓ ᭔ ᭕ ᭖ ᭗ ᭘ ᭙ ᮰ ᮱ ᮲ ᮳ ᮴ ᮵ ᮶
᮷ ᮸ ᮹ ᱀ ᱁ ᱂ ᱃ ᱄ ᱅ ᱆ ᱇ ᱈ ᱉ ᱐ ᱑ ᱒ ᱓ ᱔ ᱕ ᱖ ᱗ ᱘ ᱙ ꘠ ꘡ ꘢ ꘣ ꘤ ꘥ ꘦ ꘧ ꘨ ꘩ ꣐ ꣑ ꣒ ꣓ ꣔ ꣕ ꣖ ꣗ ꣘ ꣙ ꤀ ꤁ ꤂ ꤃ ꤄
꤅ ꤆ ꤇ ꤈ ꤉ ꩐ ꩑ ꩒ ꩓ ꩔ ꩕ ꩖ ꩗ ꩘ ꩙ ０ １ ２ ３ ４ ５ ６ ７ ８ ９
I made a comment yesterday on an answer where someone had used [0123456789] in a regular expression rather than [0-9] or \d. I said it was probably more efficient to use a range or digit specifier than a character set.
I decided to test that out today and found out to my surprise that (in the C# regex engine at least) \d appears to be less efficient than either of the other two which don't seem to differ much. Here is my test output over 10000 random strings of 1000 random characters with 5077 actually containing a digit:
Regular expression \d took 00:00:00.2141226 result: 5077/10000
Regular expression [0-9] took 00:00:00.1357972 result: 5077/10000 63.42 % of first
Regular expression [0123456789] took 00:00:00.1388997 result: 5077/10000 64.87 % of first
It's a surprise to me for two reasons:
- I would have thought the range would be implemented much more efficiently than the set.
- I can't understand why
\dis worse than[0-9]. Is there more to\dthan simply shorthand for[0-9]?
Here is the test code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using System.Text.RegularExpressions;
namespace SO_RegexPerformance
{
class Program
{
static void Main(string[] args)
{
var rand = new Random(1234);
var strings = new List<string>();
//10K random strings
for (var i = 0; i < 10000; i++)
{
//Generate random string
var sb = new StringBuilder();
for (var c = 0; c < 1000; c++)
{
//Add a-z randomly
sb.Append((char)('a' + rand.Next(26)));
}
//In roughly 50% of them, put a digit
if (rand.Next(2) == 0)
{
//Replace one character with a digit, 0-9
sb[rand.Next(sb.Length)] = (char)('0' + rand.Next(10));
}
strings.Add(sb.ToString());
}
var baseTime = testPerfomance(strings, @"\d");
Console.WriteLine();
var testTime = testPerfomance(strings, "[0-9]");
Console.WriteLine(" {0:P2} of first", testTime.TotalMilliseconds / baseTime.TotalMilliseconds);
testTime = testPerfomance(strings, "[0123456789]");
Console.WriteLine(" {0:P2} of first", testTime.TotalMilliseconds / baseTime.TotalMilliseconds);
}
private static TimeSpan testPerfomance(List<string> strings, string regex)
{
var sw = new Stopwatch();
int successes = 0;
var rex = new Regex(regex);
sw.Start();
foreach (var str in strings)
{
if (rex.Match(str).Success)
{
successes++;
}
}
sw.Stop();
Console.Write("Regex {0,-12} took {1} result: {2}/{3}", regex, sw.Elapsed, successes, strings.Count);
return sw.Elapsed;
}
}
}
\d checks all Unicode digits, while [0-9] is limited to these 10 characters. For example, Persian digits, ۱۲۳۴۵۶۷۸۹, are an example of Unicode digits which are matched with \d, but not [0-9].
You can generate a list of all such characters using the following code:
var sb = new StringBuilder();
for(UInt16 i = 0; i < UInt16.MaxValue; i++)
{
string str = Convert.ToChar(i).ToString();
if (Regex.IsMatch(str, @"\d"))
sb.Append(str);
}
Console.WriteLine(sb.ToString());
Which generates:
0123456789٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹߀߁߂߃߄߅߆߇߈߉०१२३४५६७८९০১২৩৪৫৬৭৮৯੦੧੨੩੪੫੬੭੮੯૦૧૨૩૪૫૬૭૮૯୦୧୨୩୪୫୬୭୮୯௦௧௨௩௪௫௬௭௮௯౦౧౨౩౪౫౬౭౮౯೦೧೨೩೪೫೬೭೮೯൦൧൨൩൪൫൬൭൮൯๐๑๒๓๔๕๖๗๘๙໐໑໒໓໔໕໖໗໘໙༠༡༢༣༤༥༦༧༨༩၀၁၂၃၄၅၆၇၈၉႐႑႒႓႔႕႖႗႘႙០១២៣៤៥៦៧៨៩᠐᠑᠒᠓᠔᠕᠖᠗᠘᠙᥆᥇᥈᥉᥊᥋᥌᥍᥎᥏᧐᧑᧒᧓᧔᧕᧖᧗᧘᧙᭐᭑᭒᭓᭔᭕᭖᭗᭘᭙᮰᮱᮲᮳᮴᮵᮶᮷᮸᮹᱀᱁᱂᱃᱄᱅᱆᱇᱈᱉᱐᱑᱒᱓᱔᱕᱖᱗᱘᱙꘠꘡꘢꘣꘤꘥꘦꘧꘨꘩꣐꣑꣒꣓꣔꣕꣖꣗꣘꣙꤀꤁꤂꤃꤄꤅꤆꤇꤈꤉꩐꩑꩒꩓꩔꩕꩖꩗꩘꩙０１２３４５６７８９
Credit to ByteBlast for noticing this in the docs. Just changing the regex constructor:
var rex = new Regex(regex, RegexOptions.ECMAScript);
Gives new timings:
Regex \d took 00:00:00.1355787 result: 5077/10000
Regex [0-9] took 00:00:00.1360403 result: 5077/10000 100.34 % of first
Regex [0123456789] took 00:00:00.1362112 result: 5077/10000 100.47 % of first
From Does “\d” in regex mean a digit?:
[0-9]isn't equivalent to\d.[0-9]matches only0123456789characters, while\dmatches[0-9]and other digit characters, for example Eastern Arabic numerals٠١٢٣٤٥٦٧٨٩
An addition to top answer from Sina Iravianian, here is a .NET 4.5 version (since only that version supports UTF16 output, c.f. the first three lines) of his code, using the full range of Unicode code points. Due to the lack of proper support for higher Unicode planes, many people are not aware of always checking for and including the upper Unicode planes. Nevertheless they sometimes do contain some important characters.
public static void Main()
{
var unicodeEncoding = new UnicodeEncoding(!BitConverter.IsLittleEndian, false);
Console.InputEncoding = unicodeEncoding;
Console.OutputEncoding = unicodeEncoding;
var sb = new StringBuilder();
for (var codePoint = 0; codePoint < 0x10ffff; codePoint++)
{
var isSurrogateCodePoint =
codePoint <= UInt16.MaxValue
&& ( char.IsLowSurrogate((char) codePoint)
|| char.IsHighSurrogate((char) codePoint)
);
if (isSurrogateCodePoint)
continue;
var codePointString = char.ConvertFromUtf32(codePoint);
if (Regex.IsMatch(codePointString, @"\d"))
sb.AppendFormat("{0} ", codePointString);
}
Console.WriteLine(sb.ToString());
Console.ReadKey();
}
Yielding the following output:
0 1 2 3 4 5 6 7 8 9 ٠ ١ ٢ ٣ ٤ ٥ ٦ ٧ ٨ ٩ ۰ ۱ ۲ ۳ ۴ ۵ ۶ ۷ ۸ ۹ ߀ ߁ ߂ ߃ ߄ ߅ ߆ ߇ ߈ ߉ ० १ २ ३ ४ ५ ६ ७ ८ ९
০ ১ ২ ৩ ৪ ৫ ৬ ৭ ৮ ৯ ੦ ੧ ੨ ੩ ੪ ੫ ੬ ੭ ੮ ੯ ૦ ૧ ૨ ૩ ૪ ૫ ૬ ૭ ૮ ૯ ୦ ୧ ୨ ୩ ୪ ୫ ୬ ୭ ୮ ୯ ௦ ௧ ௨ ௩ ௪ ௫ ௬ ௭ ௮ ௯
౦ ౧ ౨ ౩ ౪ ౫ ౬ ౭ ౮ ౯ ೦ ೧ ೨ ೩ ೪ ೫ ೬ ೭ ೮ ೯ ൦ ൧ ൨ ൩ ൪ ൫ ൬ ൭ ൮ ൯
๐ ๑ ๒ ๓ ๔ ๕ ๖ ๗ ๘ ๙ ໐ ໑ ໒ ໓ ໔ ໕ ໖ ໗ ໘ ໙ ༠ ༡ ༢ ༣ ༤ ༥ ༦ ༧ ༨ ༩ ၀ ၁ ၂ ၃ ၄ ၅ ၆ ၇ ၈ ၉ ႐ ႑ ႒ ႓ ႔ ႕ ႖ ႗ ႘ ႙
០ ១ ២ ៣ ៤ ៥ ៦ ៧ ៨ ៩ ᠐ ᠑ ᠒ ᠓ ᠔ ᠕ ᠖ ᠗ ᠘ ᠙ ᥆ ᥇ ᥈ ᥉ ᥊ ᥋ ᥌ ᥍ ᥎ ᥏ ᧐ ᧑ ᧒ ᧓ ᧔ ᧕ ᧖ ᧗ ᧘ ᧙ ᭐ ᭑ ᭒ ᭓ ᭔ ᭕ ᭖ ᭗ ᭘ ᭙ ᮰ ᮱ ᮲ ᮳ ᮴ ᮵ ᮶
᮷ ᮸ ᮹ ᱀ ᱁ ᱂ ᱃ ᱄ ᱅ ᱆ ᱇ ᱈ ᱉ ᱐ ᱑ ᱒ ᱓ ᱔ ᱕ ᱖ ᱗ ᱘ ᱙ ꘠ ꘡ ꘢ ꘣ ꘤ ꘥ ꘦ ꘧ ꘨ ꘩ ꣐ ꣑ ꣒ ꣓ ꣔ ꣕ ꣖ ꣗ ꣘ ꣙ ꤀ ꤁ ꤂ ꤃ ꤄
꤅ ꤆ ꤇ ꤈ ꤉ ꩐ ꩑ ꩒ ꩓ ꩔ ꩕ ꩖ ꩗ ꩘ ꩙ ０ １ ２ ３ ４ ５ ６ ７ ８ ９
0 commentaires:
Enregistrer un commentaire