Adds a C++ tokenizer along with unit tests.
This tokenizer will replace the current C implementation
which does a poor job of keeping track of pointers.
This CL is a prerequisite for up coming changes to
the parser. This CL does not wire up this tokenizer and
changes no exsiting code. All that builds is the unit tests.
Change-Id: Iec3740bce7153640adc5e5bbdc57e644cedf0038
TEST: Unit tests all pass. No leaks under valgrind
BUG: 22843198
Lee Campbell