SILUBS.ScriptureChecks.UnitTestTokenizer.Tokenize C# (CSharp) Method

UnitTestTokenizer Class Documentation Mostrar archivo Open project: sillsdev/FieldWorks

Tokenize() public method

Split text for book into TextTokens. Populate Tokens and chapters.
public Tokenize ( string text ) : List
text string
return List 
		public List<UnitTestUSFMTextToken> Tokenize(string text)
		{
			List<UnitTestUSFMTextToken> tokens = DivideText(text);
			CategorizeTokens(tokens);
			return tokens;
		}
UnitTestTokenizer
CategorizeTokens
DivideText
GetCVNumber
IsParagraphStart
NextToken
Tokenize