BookKeeper全解（1）-BookKeeper简介和快速上手

什么是BookKeeper

BookKeeper是一个提供日志条目流存储持久化的服务框架。特别适合日志流存储，一个比较经典的应用是作为消息队列Pulsar的持久框架。

那么BookKeeper是怎样产生的呢？

• 高效写
• 基于复制的高容错（消息在ensembles之间复制，这个概念之后会讲）
• 高吞吐量写

快速部署

git clone https://github.com/apache/bookkeeper.git 

之后，我们在conf目录下可以看到：

conf └─bk_cli_env.sh └─bk_server.conf └─bkenv.sh └─jaas_example.conf └─log4j.properties └─log4j.cli.properties └─log4j.shell.properties └─nettyenv.sh └─standalone.conf └─zookeeper.conf └─zookeeper.conf.dynamic 

Bookeeper提供了本地启动测试的类org.apache.bookkeeper.util.LocalBookKeeper，但是这个不太好调试，我们想搞清楚实际BookKeeper的工作模式还有部署，最好做一个相对完整的本地环境出来

编译代码，之后先在本地启用一个ZooKeeper，这个直接下载最新版的，直接默认配置启动即可（默认端口是2181）

 Server parameters bookiePort=3181 extraServerComponents= server settings httpServerEnabled=false httpServerPort=8080 httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer # Bookie Storage # Journal settings journalDirectories=d:/tmp1/bk-txn Ledger storage settings ledgerDirectories=d:/tmp1/bk-data indexDirectories=d:/tmp1/bk-data # Metadata Services # Metadata Service settings metadataServiceUri=zk+hierarchical://localhost:2181/ledgers ZooKeeper Metadata Service settings zkServers=localhost:2181 zkTimeout=10000 zkEnableSecurity=false # Settings below are used by stream/table service Grpc Server storageserver.grpc.port=4181 Dlog Settings for table service # Replication Settings dlog.bkcEnsembleSize=3 dlog.bkcWriteQuorumSize=2 dlog.bkcAckQuorumSize=2 Storage # local storage directories for storing table ranges data (e.g. rocksdb sst files) storage.range.store.dirs=d:/tmp1/bookkeeper/ranges # whether the storage server capable of serving readonly tables. default is false. storage.serve.readonly.tables=false # the cluster controller schedule interval, in milliseconds. default is 30 seconds. storage.cluster.controller.schedule.interval.ms=30000 

Bookie2：

 Server parameters bookiePort=3182 extraServerComponents= server settings httpServerEnabled=false httpServerPort=8081 httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer # Bookie Storage # Journal settings journalDirectories=d:/tmp2/bk-txn Ledger storage settings ledgerDirectories=d:/tmp2/bk-data indexDirectories=d:/tmp2/bk-data # Metadata Services # Metadata Service settings metadataServiceUri=zk+hierarchical://localhost:2181/ledgers ZooKeeper Metadata Service settings zkServers=localhost:2181 zkTimeout=10000 zkEnableSecurity=false # Settings below are used by stream/table service Grpc Server storageserver.grpc.port=4182 Dlog Settings for table service # Replication Settings dlog.bkcEnsembleSize=3 dlog.bkcWriteQuorumSize=2 dlog.bkcAckQuorumSize=2 Storage # local storage directories for storing table ranges data (e.g. rocksdb sst files) storage.range.store.dirs=d:/tmp2/bookkeeper/ranges # whether the storage server capable of serving readonly tables. default is false. storage.serve.readonly.tables=false # the cluster controller schedule interval, in milliseconds. default is 30 seconds. storage.cluster.controller.schedule.interval.ms=30000 

Bookie3：

 Server parameters bookiePort=3183 extraServerComponents= server settings httpServerEnabled=false httpServerPort=8082 httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer # Bookie Storage # Journal settings journalDirectories=d:/tmp3/bk-txn Ledger storage settings ledgerDirectories=d:/tmp3/bk-data indexDirectories=d:/tmp3/bk-data # Metadata Services # Metadata Service settings metadataServiceUri=zk+hierarchical://localhost:2181/ledgers ZooKeeper Metadata Service settings zkServers=localhost:2181 zkTimeout=10000 zkEnableSecurity=false # Settings below are used by stream/table service Grpc Server storageserver.grpc.port=4182 Dlog Settings for table service # Replication Settings dlog.bkcEnsembleSize=3 dlog.bkcWriteQuorumSize=2 dlog.bkcAckQuorumSize=2 Storage # local storage directories for storing table ranges data (e.g. rocksdb sst files) storage.range.store.dirs=d:/tmp3/bookkeeper/ranges # whether the storage server capable of serving readonly tables. default is false. storage.serve.readonly.tables=false # the cluster controller schedule interval, in milliseconds. default is 30 seconds. storage.cluster.controller.schedule.interval.ms=30000 

我用的IDE是IDEA，这里配置BookieShell运行初始化：

• 主类选择org.apache.bookkeeper.bookie.BookieShell
• VM options填写：-DentryFormatterClass=${ENTRY_FORMATTER_CLASS:-org.apache.bookkeeper.util.StringEntryFormatter}
• Program aruguments填写：--conf ./conf/bk_server_1.conf metaformat
  
  
  

之后启动，可以看到日志：

2018-10-16 15:43:14,766 - INFO - [main:MetadataDrivers@107] - BookKeeper metadata driver manager initialized 

然后，开始配置三个Bookie的启动：

• 主类选择org.apache.bookkeeper.server.Main
• Program aruguments填写：--conf ./conf/bk_server_1.conf, 另外两个分别填写--conf ./conf/bk_server_2.conf和--conf ./conf/bk_server_3.conf

编写配置好后，启动三个Bookie（为何启动三个，默认配置情况下，一个ledger至少要写到两个Bokkie中，Bookies需要选举出最合适的，这个选举算法至少需要三个Bookie）：

2018-10-16 16:00:01,923 - INFO - [main:Main@110] - Using configuration file ./conf/bk_server_1.conf 2018-10-16 16:00:01,935 - INFO - [main:Main@267] - Hello, I'm your bookie, listening on port 3181. Metadata service uri is zk+hierarchical://localhost:2181/ledgers. Journals are in [d:/tmp1/bk-txn]. Ledgers are stored in d:/tmp1/bk-data. 2018-10-16 16:00:02,045 - INFO - [main:Main@296] - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService 2018-10-16 16:00:02,304 - INFO - [main:BookieServer@94] - { "storage.cluster.controller.schedule.interval.ms" : "30000", "zkEnableSecurity" : "false", "dlog.bkcAckQuorumSize" : "2", "indexDirectories" : "d:/tmp1/bk-data", "zkServers" : "localhost:2181", "storage.range.store.dirs" : "d:/tmp1/bookkeeper/ranges", "httpServerPort" : "8080", "dlog.bkcWriteQuorumSize" : "2", "bookiePort" : "3181", "storage.serve.readonly.tables" : "false", "ledgerDirectories" : "d:/tmp1/bk-data", "zkTimeout" : "10000", "httpServerClass" : "org.apache.bookkeeper.http.vertx.VertxHttpServer", "httpServerEnabled" : "false", "metadataServiceUri" : "zk+hierarchical://localhost:2181/ledgers", "dlog.bkcEnsembleSize" : "3", "journalDirectories" : "d:/tmp1/bk-txn", "storageserver.grpc.port" : "4181", "extraServerComponents" : "" } 2018-10-16 16:00:03,032 - INFO - [main:MetadataDrivers@107] - BookKeeper metadata driver manager initialized 2018-10-16 16:00:03,033 - INFO - [main:MetadataDrivers@107] - BookKeeper metadata driver manager initialized 2018-10-16 16:00:03,033 - INFO - [main:MetadataDrivers@107] - BookKeeper metadata driver manager initialized 2018-10-16 16:00:03,167 - INFO - [main:Bookie@399] - Stamping new cookies on all dirs [d:\tmp1\bk-txn\current] [d:\tmp1\bk-data\current, d:\tmp1\bk-data\current] 2018-10-16 16:00:03,368 - WARN - [main:Bookie@302] - Dirs: [d:\tmp1\bk-data\current, d:\tmp1\bk-data\current] are in same DiskPartition/FileSystem: (d:) 2018-10-16 16:00:03,373 - INFO - [main:Bookie@644] - instantiate ledger manager org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory 2018-10-16 16:00:03,415 - ERROR - [main:Journal$LastLogMark@244] - Problems reading from d:\tmp1\bk-data\current\lastMark (this is okay if it is the first time starting this bookie 2018-10-16 16:00:03,420 - INFO - [main:Bookie@700] - Using ledger storage: org.apache.bookkeeper.bookie.SortedLedgerStorage 2018-10-16 16:00:03,500 - INFO - [main:IndexPersistenceMgr@107] - openFileLimit = 20000 2018-10-16 16:00:03,539 - INFO - [main:IndexInMemPageMgr@377] - maxDirectMemory = , pageSize = 8192, pageLimit = 10560 2018-10-16 16:00:03,594 - INFO - [main:ScanAndCompareGarbageCollector@107] - Over Replicated Ledger Deletion : enabled=true, interval= 2018-10-16 16:00:03,603 - INFO - [main:GarbageCollectorThread@253] - Minor Compaction : enabled=true, threshold=0.023224, interval= 2018-10-16 16:00:03,603 - INFO - [main:GarbageCollectorThread@255] - Major Compaction : enabled=true, threshold=0.0929, interval= 2018-10-16 16:00:03,734 - INFO - [main:Main@303] - Load lifecycle component : org.apache.bookkeeper.server.service.BookieService 2018-10-16 16:00:03,748 - INFO - [main:ComponentStarter@79] - Starting component bookie-server. 2018-10-16 16:00:03,753 - INFO - [main:Bookie@853] - Finished replaying journal in 3 ms. 2018-10-16 16:00:03,756 - INFO - [SyncThread-7-1:SyncThread@136] - Flush ledger storage at checkpoint CheckpointList{checkpoints=[LogMark: logFileId - 0 , logFileOffset - 0]}. 2018-10-16 16:00:03,786 - INFO - [main:Bookie@892] - Finished reading journal, starting bookie 2018-10-16 16:00:03,787 - INFO - [BookieJournal-3181:Journal@932] - Starting journal on d:\tmp1\bk-txn\current 2018-10-16 16:00:03,789 - INFO - [ForceWriteThread:Journal$ForceWriteThread@471] - ForceWrite Thread started 2018-10-16 16:00:03,828 - INFO - [BookieJournal-3181:JournalChannel@154] - Opening journal d:\tmp1\bk-txn\current\1667be392ce.txn 2018-10-16 16:00:04,078 - INFO - [main:ComponentStarter@81] - Started component bookie-server. 2018-10-16 16:00:04,309 - INFO - [BookieJournal-3181:NativeIO@48] - Unable to link C library. Native methods will be disabled. 

之后我们简单编写个测试程序，来看下日志写入读取效果：

public class TestClient { 
    public static void main(String[] args) throws InterruptedException, BKException, IOException { 
    BookKeeper bkc = new BookKeeper("localhost:2181"); // A password for the new ledger byte[] ledgerPassword = "test".getBytes(); // Create a new ledger and fetch its identifier LedgerHandle lh = bkc.createLedger(BookKeeper.DigestType.MAC, ledgerPassword); long ledgerId = lh.getId(); // Create a buffer for four-byte entries ByteBuffer entry = ByteBuffer.allocate(4); int numberOfEntries = 10; // Add entries to the ledger, then close it for (int i = 0; i < numberOfEntries; i++) { 
    entry.putInt(i); entry.position(0); lh.addEntry(entry.array()); } lh.close(); // Open the ledger for reading lh = bkc.openLedger(ledgerId, BookKeeper.DigestType.MAC, ledgerPassword); // Read all available entries Enumeration<LedgerEntry> entries = lh.readEntries(0, numberOfEntries - 1); while (entries.hasMoreElements()) { 
    ByteBuffer result = ByteBuffer.wrap(entries.nextElement().getEntry()); Integer retrEntry = result.getInt(); // Print the integer stored in each entry System.out.println(String.format("Result: %s", retrEntry)); } // Close the ledger and the client lh.close(); bkc.close(); } } 

运行，可以看到输出：

Result: 0 Result: 1 Result: 2 Result: 3 Result: 4 Result: 5 Result: 6 Result: 7 Result: 8 Result: 9 

BookKeeper基本概念

在BookKeeper中：

• 每一块日志是一个entry
• 日志流被定义为ledgers
• 每个独立的保存ledges的服务器叫做bookies

Entries

Ledgers

Bookies 和 Ensemble

Bookie是一个独立的BookKeeper服务器,处理Ledgers片段（因为每个Bookie保存的是每个Ledgers的片段，不是完整的Ledgers）。

一个Ensemble是一个Bookies集合，他们共同保存着一个Ledger的所有entries。通常一个Ensemble是整个Bookies集群的子集。

BookKeeper API简介

BookKeeper的客户端主要负责创建删除ledgers，并且从ledgers中读取entries。

BookKeeper提供两种API，Ledger API还有DistributedLog API；顾名思义，Ledger API提供了直接与Ledger交互的接口，DistributedLog API不直接与ledger交互而是与Bookeeper集群交互。

BookKeeper 元数据存储