spanner讨论
Published in the Proceedings of OSDI 2012
OSDI是计算机学界最顶级学术会议之一，全称本来是USENIX Symposium on Operating Systems Design and Implementation，简称OSDI
问题：
如何理解：
schema：
external consistency：
consistency ：
linearizability ：
serializability：
commit wait：
什么是关系型数据库、结构化、半结构化、nosql，键值数据库？
schema：模式，因为不使用Bigtable，就是因为其不能适配很多模式
external consistency：客户端访问spanner集群中任一数据库（spanner server）得到的结果是一致的
section4: Spanner uses TrueTime to imple-ment externally-consistent distributed transactions, lock-free read-only transactions, and atomic schema updates.
commit wait：
The coordinator leader ensures that clients cannot see any data committed by Ti until TT.after(si) is true. Commit wait ensures thats i is less than the absolute commit time of Ti, or si
Like Bigtable, writes that occur in a transaction are buffered at the client until commit.
Q: How does external consistency relate to linearizability and serializability?
A: External consistency seems to be equivalent to linearizability, but applied to entire transactions rather than individual reads and writes. External consistency also seems equivalent to strict serializability, which is serializability with the added constraint that the equivalent serial order must obey real time order. The critical property is that if transaction T1 completes, and then(afterwards in real time) transaction T2 starts, T2 must see T1’s writes.
外部一致性似乎等效于线性化，但适用于整个事务而不是单个读写。外部一致性似乎也等同于严格的可序列化性，即具有可等效性的串行顺序必须服从实时顺序的附加限制。关键特性是，如果事务T1完成，然后（随后实时）事务T2开始，则T2必须看到T1的写操作。
Q: Why is external consistency desirable(被需要)?
A: Suppose Hatshepsut（法老） changes the password on an account shared by her work group, via a web server in a datacenter in San Jose. She whispers the new password over the cubicle（隔间） wall to her colleage Cassandra（facebook的数据库）.Cassandra logs into the account via a web server in a different datacenter, in San Mateo. External consistency guarantees that Cassandra will observe the change to the password, and not, for example, see a stale replica.
假设Hatshepsut通过一个位于圣何塞（San Jose）数据中心的Web服务器更改了由她的工作组共享的帐户的密码。她在隔壁墙上对同事Cassandra窃窃私语新密码。Cassandra通过位于圣马特奥的另一个数据中心中的Web服务器登录该帐户。外部一致性保证Cassandra将遵守密码的更改，例如，不会看到陈旧的副本。
Hatshepsut(哈特谢普苏特)是埃及第十八王朝的第五任法老。她是历史上第二位确认的女法老，第一位是索贝涅菲鲁。哈特谢普苏特于公元前1478年登上埃及的王位
Apache Cassandra是一个开源的分布式NoSQL数据库。 它提供了具有最终一致语义的分区宽列存储模型。
Q: What is the purpose of Spanner’s commit wait?
A: Commit wait ensures that a read/write transaction does not completeuntil the time in its timestamp is guaranteed to have passed. Thatmeans that a read/only transaction that starts after the read/writetransaction completes is guaranteed to have a higher timestamp, andthus to see the read/write transaction’s writes. This helps fulfil theguarantee of external consistency: if T1 completes before T2 starts,T2 will come after T1 in the equivalent serial order (i.e. T2 will seeT1’s writes).
提交等待可确保读/写事务在完成其时间戳之前一直未完成。这意味着在读/写事务完成之后开始的读/只读事务被保证具有更高的时间戳，从而可以看到读/写事务的写操作。这有助于确保外部一致性：如果T1在T2开始之前完成，则T2将以等效的顺序在T1之后出现（即T2将看到T1的写入）。
-
介绍
产生背景：F1之前使用mysql，但在实际使用时，随着业务的变化，mysql已经不再能满足需求，至此，flanner诞生。
可伸缩、多版本、全球分布和同步复制
基于paxos选主
这是第一个在全球范围内分发数据并支持外部一致的分布式事务的系统
新的时间API与下面的特性相关：
non-blocking reads
lock-free read-only transac-tions,
atomic schema changes
是一个跨分布在世界各地的数据中心中的许多Paxos状态机集对数据进行分片的数据库。
clients auto-matically failover（故障转移） between replicas.
当数据量或服务器数量发生变化时，spanner会自动跨机器进行数据分片（reshards data ）
and it automatically migrates data across machines (even across datacenters)to balance load（还可以平衡负载） and in response to failures.
high availability
Spanner’s main focus is managing cross-datacenter replicated data
不使用Bigtable的原因：
对于某些类型的应用程序，Bigtable可能很难使用:那些具有复杂的、不断发展的模式的应用程序，或者那些希望在存在广域复制时保持强一致性的应用程序。
Megastore：半关系数据模型 同步复制
Spanner已经从一个类似于bigtable的键值存储演化为一个临时的多版本数据库
Data is stored in schematized semi-relationaltables; data is versioned, and each version is automati-cally timestamped with its commit time; old versions ofdata are subject to configurable garbage-collection poli-cies; and applications can read data at old timestamps.Spanner supports general-purpose transactions, and pro-vides a SQL-based query language
（解释了Spanner中多版本是数据的版本，通过给每个数据加时间戳的方式设置版本）
Spanner的特点：（以下特性通过时间戳实现）
- 首先，应用程序可以在一个粒度上动态地控制数据的复制配置。应用程序可以指定控制权限来控制哪些数据中心包含哪些数据、数据与其用户之间的距离(以控制读取延迟)、副本之间的距离(以控制写操作)以及维护了多少副本(以控制持久性、可用性和读取性能)。系统还可以动态地、透明地在数据中心之间移动数据，以平衡数据中心之间的资源。
- 它提供外部一致的[16]读写，以及按时间戳跨数据库进行全局一致的读取。这些特性使Spanner能够在全局范围内支持连续备份、一致的MapReduce执行[12]和原子模式更新，甚至在存在正在进行的事务时也支持这些功能。
时间戳反映序列化顺序。此外，序列化顺序满足外部一致性(或者说线性化)
Spanner如何防止时间戳不一致：
This implementation keeps uncertainty small (gen-erally less than 10ms) by using multiple modern clockreferences (GPS and atomic clocks).
TrueTime API可以直接暴露时钟不确定性，Spanner时间戳的保证就是取决于这个API实现的界限。如果这个不确定性很大，Spanner就降低速度来等待这个大的不确定性结束。
2、实现
类似于Bigtable
Spanner的部署被称为universe。由于Spanner管理的是全局数据，所以运行的universe只有少数几个。
Zones are the unit of administrative deploy-ment. The set of zones is also the set of locations acrosswhich data can be replicated.
zone也是物理隔离的单位
图1展示了一个universe中的服务器。一个zone有一个zonemaster，每个zone有100到几千个spanserver。前者将数据分配给spanserver;后者向客户端提供数据。客户端使用区域定位代理来定位分配给它们的数据的spanservers。uni-versemaster和placementdriver现在是sin-gleton。universe master主要是一个控制台，它显示用于交互活动调试的所有区域的状态信息。placementdriver程序处理跨区域的数据自动移动。placementdiver定期与spanservers通信，查找需要删除的数据，以满足更新的复制约束或平衡负载。
2.1 基于Bigtable的spanserver软件栈
each spanserver is responsible for between 100and 1000 instances of a data structure called a tablet. A tablet is similar to Bigtable’s tablet abstraction, in that it implements a bag of the following mappings:
(key:string, timestamp:int64)→string
Unlike Bigtable, Spanner assigns timestamps to data,which is an important way in which Spanner is morelike a multi-version database than a key-value store.（spanner给每个数据加了时间戳）
与BigTable不同的是，Spanner会把时间戳分配给数据，这种非常重要的方式，使得Spanner更像一个多版本数据库，而不是一个键值存储。一个tablet的状态是存储在类似于B-树的文件集合和写前(write-ahead)的日志中，所有这些都会被保存到一个分布式的文件系统中，这个分布式文件系统被称为Colossus，它继承自Google File System。
tablet的状态存储在类b树文件和预写日志中
Paxos implementation supports long-lived leaders withtime-based leader leases, whose length defaults to 10seconds.
The current Spanner implementation logs every Paxos write twice: once in the tablet’s log, and once in the Paxos log.
Our implementation of Paxos is pipelined, so as to improveSpanner’s throughput in the presence of WAN latencies;
Writes must initiate the Paxos protocol at the leader; reads access state directly from the underlying tablet at any replica that is sufficiently up-to-date.
Spanner support for syn-chronous replication across datacenters. (Bigtable onlysupports eventually-consistent replication across data-centers.
3、TrueTime
truetime使用的底层时间引用是GPS和原子钟。
F1选择Spanner主要出于一下几点原因：
- Spanner不需要手动重新切分。
- Spanner提供同步复制和自动故障转移。
- F1需要强大的事务语义，这使得使用其他NoSQL系统是不现实的。应用程序语义要求跨任意数据的事务处理和一致的读取。
4、未来工作
in-memory database
New SQL
the database com-munity, a familiar, easy-to-use, semi-relational interface,transactions, and an SQL-based query language; fromthe systems community, scalability, automatic sharding,fault tolerance, consistent replication, external consis-tency, and wide-area distribution.
笔记：
from // https://www.youtube.com/watch?v=NthK17nbpYs
Distributed multiversion database
General-purpose transactions (ACID）
SQL query language
Schematized tables
Semi-relational data model
Running in production
Storage for Google’s ad data
Replaced a sharded MySQL databse
Overview
Feature: Lock-free distributed read transactions
Property: External consistency of distributed transactions
– First system at global scale
Implementation: Integration of concurrency control, replication, and 2PC
-Correctness and performance
Enabling technology: TrueTime
– Interval-based global time
Synchronizing Snapshots
Global wall-clock time
==
External Consistency:
Commit order respects global wall-time order
==
Timestamp order resspects global wall-time order given
timestamp order == commit order
Strict two-phase locking for write transactions
Assign timestamp while locks are held
now = reference now + loal-clock offset
e = reference e + worst-case local-clock drift
Schematized tables
from //https://baike.baidu.com/item/spanner/?fr=aladdin
可扩展的、多版本、全球分布式、同步复制数据库
支持外部一致性的分布式事务
时间API（可以暴露时钟的不确定性，这个时间API对于支持外部一致性、非阻塞读、不采用锁机制的只读事务、原子变更非常重要）
paxos状态机
from pingcap
// https://pingcap.com/blog-cn/pax/
mit6.824
// https://pdos.csail.mit.edu/6.824/papers/spanner-faq.txt
Q: What is time?
Q: What time is it?
Q: What is an atomic clock?
Q: What kind of atomic clock does Spanner use?
Q: How does external consistency relate to linearizability and serializability?
Q: Why is external consistency desirable?
Q: Could Spanner use Raft rather than Paxos?
Q: What is the purpose of Spanner’s commit wait?
Q: Does anyone use Spanner?
MIT spanner课件
// https://pdos.csail.mit.edu/6.824/notes/l-spanner.txt
2PC将分布式事务分成了两个阶段，两个阶段分别为提交请求（投票）和提交（执行）。协调者根据参与者的响应来决定是否需要真正地执行事务，具体流程如下。
官方PPT
发布者：全栈程序员-站长，转载请注明出处：https://javaforall.net/229665.html原文链接：https://javaforall.net
