基于Lucene3.5.0如何从TokenStream获得Token

全栈程序员-站长 • 2022年7月22日下午2:46 • 未分类 • 阅读 6

大家好，又见面了，我是你们的朋友全栈君。通过学习Lucene3.5.0的doc文档，对不同release版本 lucene版本的API改动做分析。最后找到了有价值的改动信息。

LUCENE-2302: Deprecated TermAttribute and replaced by a new CharTermAttribute. The change is backwards compatible, so mixed new/old TokenStreams all work on the same char[] buffer independent of which interface they use. CharTermAttribute has shorter method names and implements CharSequence and Appendable. This allows usage like Java’s StringBuilder in addition to direct char[] access. Also terms can directly be used in places where CharSequence is allowed (e.g. regular expressions). (Uwe Schindler, Robert Muir)
以上信息可以知道，原来的通过的方法已经不能够提取响应的Token了
[java]
view plain
copy
print
?
1. StringReader reader = new StringReader(s);
2. TokenStream ts =analyzer.tokenStream(s, reader);
3. TermAttribute ta = ts.getAttribute(TermAttribute.class);
StringReader reader = new StringReader(s);<br />TokenStream ts =analyzer.tokenStream(s, reader);<br />TermAttribute ta = ts.getAttribute(TermAttribute.class);
通过分析Api文档信息可知，CharTermAttribute已经成为替换TermAttribute的接口
因此我编写了一个例子来更好的从TokenStream中提取Token
[html]
view plain
copy
print
?
1. package com.segment;
3. import java.io.StringReader;
4. import org.apache.lucene.analysis.Analyzer;
5. import org.apache.lucene.analysis.Token;
6. import org.apache.lucene.analysis.TokenStream;
7. import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
8. import org.apache.lucene.analysis.tokenattributes.TermAttribute;
9. import org.apache.lucene.util.AttributeImpl;
10. import org.wltea.analyzer.lucene.IKAnalyzer;
13. public class Segment {
14. public static String show(Analyzer a, String s) throws Exception {
16. StringReader reader = new StringReader(s);
17. TokenStream ts = a.tokenStream(s, reader);
18. String s1 = “”, s2 = “”;
19. boolean hasnext= ts.incrementToken();
20. //Token t = ts.next();
21. while (hasnext) {
22. //AttributeImpl ta = new AttributeImpl();
23. CharTermAttribute ta = ts.getAttribute(CharTermAttribute.class);
24. //TermAttribute ta = ts.getAttribute(TermAttribute.class);
26. s2 = ta.toString() + ” “;
27. s1 += s2;
28. hasnext = ts.incrementToken();
29. }
30. return s1;
31. }
33. public String segment(String s) throws Exception {
34. Analyzer a = new IKAnalyzer();
35. return show(a, s);
36. }
37. public static void main(String args[])
38. {
39. String name = “我是俊杰，我爱编程，我的测试用例”;
40. Segment s = new Segment();
41. String test = “”;
42. try {
43. System.out.println(test+s.segment(name));
44. } catch (Exception e) {
45. // TODO Auto-generated catch block
46. e.printStackTrace();
47. }
48. }
50. }