Lucene.Net.Analysis.Cjk.CJKBigramFilter.FlushUnigram C# (CSharp) Method

CJKBigramFilter Class Documentation Show file Open project: apache/lucenenet

FlushUnigram() private method

Flushes a unigram token to output from our buffer. This happens when we encounter isolated CJK characters, either the whole CJK string is a single character, or we encounter a CJK character surrounded by space, punctuation, english, etc, but not beside any other CJK.

private FlushUnigram ( ) : void
return	void

        private void FlushUnigram()
        {
            ClearAttributes();
            char[] termBuffer = termAtt.ResizeBuffer(2); // maximum unigram length (2 surrogates)
            int len = Character.ToChars(buffer[index], termBuffer, 0);
            termAtt.Length = len;
            offsetAtt.SetOffset(startOffset[index], endOffset[index]);
            typeAtt.Type = SINGLE_TYPE;
            index++;
        }

CJKBigramFilter

DoNext

FlushBigram

FlushUnigram

IncrementToken

Refill

Reset