Lucene IndexWriter 添加文档速度慢

Lucene IndexWriter slow to add documents(Lucene IndexWriter 添加文档速度慢)

问题描述

我编写了一个小循环，将 10,000 个文档添加到 IndexWriter 中，并且花了很长时间才完成.

I wrote a small loop which added 10,000 documents into the IndexWriter and it took for ever to do it.

还有其他方法可以索引大量文档吗?

Is there another way to index large volumes of documents?

我问是因为当它上线时，它必须加载 15,000 条记录.

I ask because when this goes live it has to load in 15,000 records.

另一个问题是如何避免在重新启动 Web 应用程序时再次加载所有记录?

The other question is how do I prevent having to load in all the records again when the web application is restarted?

编辑

这是我使用的代码；

for (int t = 0; t < 10000; t++){
    doc = new Document();
    text = "Value" + t.toString();
    doc.Add(new Field("Value", text, Field.Store.YES, Field.Index.TOKENIZED));
    iwriter.AddDocument(doc);
};

编辑 2

        Analyzer analyzer = new StandardAnalyzer();
        Directory directory = new RAMDirectory();

        IndexWriter iwriter = new IndexWriter(directory, analyzer, true);

        iwriter.SetMaxFieldLength(25000);

然后是添加文档的代码，然后；

then the code to add the documents, then;

        iwriter.Close();

推荐答案

只是检查，但你在运行时没有附加调试器有吗?

Just checking, but you haven't got the debugger attached when you're running it have you?

这会严重影响添加文档时的性能.

This severely affects performance when adding documents.

在我的机器上(Lucene 2.0.0.4):

On my machine (Lucene 2.0.0.4):

使用平台目标 x86 构建:

Built with platform target x86:

• 无调试器 - 5.2 秒

• No debugger - 5.2 seconds

附加调试器 - 113.8 秒

Debugger attached - 113.8 seconds

使用平台目标 x64 构建:

Built with platform target x64:

• 无调试器 - 6.0 秒

• No debugger - 6.0 seconds

附加调试器 - 171.4 秒

Debugger attached - 171.4 seconds

在 RAMDirectory 中保存和加载索引的粗略示例:

Rough example of saving and loading an index to and from a RAMDirectory:

const int DocumentCount = 10 * 1000;
const string IndexFilePath = @"X:Temp	mp.idx";

Analyzer analyzer = new StandardAnalyzer();
Directory ramDirectory = new RAMDirectory();

IndexWriter indexWriter = new IndexWriter(ramDirectory, analyzer, true);

for (int i = 0; i < DocumentCount; i++)
{
    Document doc = new Document();
    string text = "Value" + i;
    doc.Add(new Field("Value", text, Field.Store.YES, Field.Index.TOKENIZED));
    indexWriter.AddDocument(doc);
}

indexWriter.Close();

//Save index
FSDirectory fileDirectory = FSDirectory.GetDirectory(IndexFilePath, true);
IndexWriter fileIndexWriter = new IndexWriter(fileDirectory, analyzer, true);
fileIndexWriter.AddIndexes(new[] { ramDirectory });
fileIndexWriter.Close();

//Load index
FSDirectory newFileDirectory = FSDirectory.GetDirectory(IndexFilePath, false);
Directory newRamDirectory = new RAMDirectory();
IndexWriter newIndexWriter = new IndexWriter(newRamDirectory, analyzer, true);
newIndexWriter.AddIndexes(new[] { newFileDirectory });

Console.WriteLine("New index writer document count:{0}.", newIndexWriter.DocCount());

这篇关于Lucene IndexWriter 添加文档速度慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！