书籍详情
精通Apache Pulsar：可伸缩云原生事件流实践
作者：（美）Jowanza Joseph（乔万扎·约瑟夫）
出版社：电子工业出版社
出版时间：2023-12-01
ISBN：9787121467172
定价：¥89.00
购买这本书可以去
内容简介
本书专为想要学习并掌握Apache Pulsar的读者打造，是一本入门图书。全书共有13章内容。第1章和第2章，从生活实例中引入实时消息系统和事件驱动的概念，帮助大家理解Apache Pulsar系统存在的必要性，以及Pulsar的独特性和价值定位。第3章介绍了Pulsar在雅虎的起源故事、设计理念与应用场景。第4章探究了Pulsar内部的主要组件及其工作原理，帮助大家构建对Pulsar的整体架构认知。第5章到第10章，详细介绍了Pulsar消费者、生产者的工作原理，应用于生产和消费消息且与第三方系统交互的框架Pulsar IO，以及Pulsar Functions、分层存储、Pulsar SQL等Pulsar的独特理念和关键特性。第11章与第12章，展示了如何在诸多环境中部署Pulsar，以及在运维Pulsar的过程中需要关注的指标。在第13章中，作者结合个人经验阐述了Pulsar未来值得探索的领域和方向。 本书配图生动形象，实例丰富，不仅提供了很多代码示例，还提供了多篇附录，以作参考。
作者简介
Jowanza Joseph 是一位就职于 Finicity 的软件工程师，他负责领导 Finicity 开放银行平台的网格开发。Jowanza 曾在多个项目中使用 Apache Pulsar，并利用基于 Pulsar 构建的全托管消息和流处理平台实现了日均处理十亿量级的消息。近十年中， Jowanza 在 Pluralsight 等公司的工作中都使用过Apache Kafka、Akka、Kubernetes 及其他流技术与消息技术。他还曾在 Strange Loop、Abstractions、开源峰会及 O’Reilly 的 Strata Data & AI 大会上做过技术分享。蔡正昕，曾在亚马逊工作， Pulsar 活跃贡献者。后来加入了 StreamNative，成为 Cloud 工程师。蔡正昕，软件工程师。曾在亚马逊，Streamnative等公司任职，从事数据处理系统和云服务控制面的开发。19年开始关注Apache Pulsar社区并参与讨论，漏洞修复和特性开发。对Apache Pulsar，Apache BookKeeper，以及在K8S环境中运维有状态系统有比较深入的理解。王殿进，目前在 StreamNative 负责 Apache Pulsar 社区与市场工作。曾在云计算厂商、SaaS 公司从事过开发者社区运营与用户增长工作。徐文涛，曾任Talkdesk 知识管理工程师，为多个企业级 SaaS、CCaaS 与 PaaS 产品撰写并审核英文文档。曾任某世界五百强本地化团队终审负责人，原 KubeSphere 开源容器平台本地化、文档与内容策划负责人。曾获美国 DZone 专栏银牌作家，人事部 CATTI 二级口译、二级笔译，联合国 UNLPP 二级笔译，Linux 基金会认证 Kubernetes 管理员 （CKA）、应用开发者 （CKAD） 与安全专家 （CKS）等行业资格或荣誉。
目录
前言 ....................................................................................................xvii
第 1 章 实时消息的价值 ........................................................................ 1
流动的数据 ............................................................................................................1
资源效率 ................................................................................................................3
使用场景 ................................................................................................................4
银行 .................................................................................................................5
医疗 .................................................................................................................6
安全 .................................................................................................................7
物联网 .............................................................................................................8
小结 .....................................................................................................................10
第 2 章 事件流和事件代理 ................................................................... 11
发布 / 订阅 ........................................................................................................... 11
队列 .....................................................................................................................13
故障类型 ..............................................................................................................15
推送与轮询 ..........................................................................................................15
为什么需要 Pulsar ................................................................................................16
流和队列的统一 ............................................................................................16
模块化 ...........................................................................................................16
性能 ...............................................................................................................18
小结 .....................................................................................................................19
第 3 章 认识 Pulsar ............................................................................ 20
Pulsar 的起源 .......................................................................................................20
Pulsar 的设计理念 ................................................................................................21
多租户 ...........................................................................................................22
跨地域复制 ...................................................................................................23
性能 ...............................................................................................................27
模块化 ...........................................................................................................27
Pulsar 生态 ...........................................................................................................28
Pulsar Functions ............................................................................................28
Pulsar IO .......................................................................................................29
Pulsar SQL ....................................................................................................30
Pulsar 成功案例 ...................................................................................................30
雅虎日本 .......................................................................................................30
Splunk ...........................................................................................................31
Iterable ..........................................................................................................32
小结 .....................................................................................................................33
第 4 章 Pulsar 内部组件 ..................................................................... 34
Broker ..................................................................................................................34
消息缓存 .......................................................................................................35
与 BookKeeper、ZooKeeper 交互 .................................................................37
Schema 校验 ..................................................................................................38
Broker 间的通信 ............................................................................................38
Pulsar Functions 和 Pulsar IO ........................................................................39
Apache BookKeeper .............................................................................................39
预写式日志 ...................................................................................................45
消息存储 .......................................................................................................45
对象 /Blob 存储 .............................................................................................46
Pravega ..........................................................................................................47
Majordodo .....................................................................................................47
Apache ZooKeeper ...............................................................................................48
命名服务 .......................................................................................................49
配置管理 .......................................................................................................49
选主 ...............................................................................................................50
通知系统 .......................................................................................................50
Apache Kafka ................................................................................................50
Apache Druid .................................................................................................51
Pulsar Proxy .........................................................................................................52
Java 虚拟机（JVM） .............................................................................................53
Netty ..............................................................................................................54
Apache Spark .................................................................................................55
Apache Lucene ..............................................................................................55
小结 .....................................................................................................................55
第 5 章 消费者 .................................................................................... 56
成为消费者意味着什么 ........................................................................................56
订阅 .....................................................................................................................57
独占 ...............................................................................................................60
共享 ...............................................................................................................61
键共享 ...........................................................................................................62
故障转移 .......................................................................................................65
确认 .....................................................................................................................66
单条确认 .......................................................................................................66
累积确认 .......................................................................................................67
Schema .................................................................................................................67
消费者 Schema 管理 .....................................................................................68
消费模式 ..............................................................................................................68
批处理 ...........................................................................................................68
分块 ...............................................................................................................69
高级配置 ..............................................................................................................70
延迟消息 .......................................................................................................70
保留策略 .......................................................................................................70
堆积限额 .......................................................................................................71
配置消费者 ..........................................................................................................72
回放 ...............................................................................................................73
死信主题 .......................................................................................................75
重试主题 .......................................................................................................76
小结 .....................................................................................................................76
第 6 章 生产者 .................................................................................... 77
同步生产者 ..........................................................................................................77
异步生产者 ..........................................................................................................78
生产者路由 ..........................................................................................................79
轮询路由 .......................................................................................................79
单分区路由 ...................................................................................................80
定制分区路由 ................................................................................................81
生产者配置 ..........................................................................................................83
topicName .....................................................................................................83
producerName ...............................................................................................83
sendTimeoutMs .............................................................................................83
blockIfQueueFull ...........................................................................................84
maxPendingMessages ....................................................................................84
maxPendingMessagesAcrossPartitions ...........................................................84
messageRoutingMode ....................................................................................84
hashingScheme ..............................................................................................84
cryptoFailureAction .......................................................................................85
batchingMaxPublishDelayMicros ..................................................................85
batchingMaxMessages ...................................................................................85
batchingEnabled ............................................................................................85
compressionType ...........................................................................................85
Schema on Write ..................................................................................................86
使用 Schema Registry ....................................................................................86
非持久化主题 ......................................................................................................89
使用场景 .......................................................................................................90
使用非持久化主题 ........................................................................................90
事务 .....................................................................................................................91
小结 .....................................................................................................................92
第 7 章 Pulsar IO ............................................................................... 93
Pulsar IO 架构 ......................................................................................................93
运行时 ...........................................................................................................94
性能考量 .......................................................................................................94
使用场景 ..............................................................................................................95
简单事件处理管道 ........................................................................................95
变化数据捕获 ................................................................................................96
考量 .....................................................................................................................97
消息序列化 ...................................................................................................97
管道稳定性 ...................................................................................................97
故障处理 .......................................................................................................98
示例 .....................................................................................................................99
Elasticsearch ..................................................................................................99
Netty ............................................................................................................ 100
创建自己的连接器 ............................................................................................. 101
TimescaleDB ............................................................................................... 101
小结 ................................................................................................................... 102
第 8 章 Pulsar Functions .................................................................. 103
流处理 ................................................................................................................ 103
Pulsar Functions 架构 ......................................................................................... 105
运行时 ......................................................................................................... 105
隔离性 ......................................................................................................... 106
在 Kubernetes 上部署 Pulsar Functions 的隔离性 .............................................. 107
使用场景 ............................................................................................................ 107
创建 Pulsar Functions .................................................................................. 108
简单事件处理 .............................................................................................. 108
主题清理 ..................................................................................................... 110
主题数据统计 .............................................................................................. 113
小结 ................................................................................................................... 115
第 9 章 分层存储 .............................................................................. 116
在云端存储数据 ................................................................................................. 117
对象存储 ..................................................................................................... 118
使用场景 ............................................................................................................ 120
复制 ............................................................................................................. 120
CQRS .......................................................................................................... 121
灾难恢复 ..................................................................................................... 122
卸载数据 ............................................................................................................ 122
Pulsar Offloader ........................................................................................... 122
检索已卸载的数据 ............................................................................................. 129
与对象存储数据交互 ................................................................................... 129
重填主题 ..................................................................................................... 129
使用 Pulsar 客户端 ...................................................................................... 130
小结 ................................................................................................................... 131
第 10 章 Pulsar SQL ........................................................................ 132
流即表 ................................................................................................................ 133
SQL-on-Anything 引擎 ...................................................................................... 135
Apache Flink: 另一种视角 ........................................................................... 136
Presto/Trino ................................................................................................. 137
Pulsar SQL 的工作原理 ...................................................................................... 138
配置 Pulsar SQL ................................................................................................. 140
性能考量 ............................................................................................................ 142
小结 ................................................................................................................... 143
第 11 章 部署 Pulsar ........................................................................ 144
Docker ................................................................................................................ 144
裸金属 ................................................................................................................ 146
最低要求 ..................................................................................................... 146
开始安装 ..................................................................................................... 146
部署 ZooKeeper........................................................................................... 147
启动 BookKeeper ........................................................................................ 148
启动 Pulsar .................................................................................................. 148
公有云厂商 ........................................................................................................ 149
AWS ............................................................................................................ 151
Azure ........................................................................................................... 152
Google Cloud Platform ................................................................................ 153
Kubernetes ......................................................................................................... 153
小结 ................................................................................................................... 155
第 12 章 运维 Pulsar ........................................................................ 156
Apache BookKeeper 指标 .................................................................................. 157
服务器指标 ................................................................................................. 157
Journal 指标 ................................................................................................ 158
存储指标 ..................................................................................................... 159
Apache ZooKeeper 指标 ..................................................................................... 159
服务器指标 ................................................................................................. 160
请求指标 ..................................................................................................... 161
主题指标 ............................................................................................................ 161
消费者指标 ........................................................................................................ 163
Pulsar 事务指标 ................................................................................................. 163
Pulsar Functions 指标 ......................................................................................... 164
高级运维技巧 .................................................................................................... 165
拦截器和追踪机制 ...................................................................................... 165
Pulsar SQL 指标 .......................................................................................... 167
指标转发 ............................................................................................................ 169
监控面板 ..................................................................................................... 169
小结 ................................................................................................................... 170
第 13 章 未来 ................................................................................... 171
编程语言支持 .................................................................................................... 171
扩展接口 ..................................................................................................... 171
增强 Pulsar Functions .................................................................................. 172
架构简化 / 扩展 ........................................................................................... 175
消息平台桥接器 .......................................................................................... 177
小结 ................................................................................................................... 180
附录 A Pulsar Admin API ................................................................. 181
附录 B Pulsar Admin CLI ................................................................. 189
附录 C 跨地域复制 ........................................................................... 192
附录 D Pulsar 中的安全、认证和鉴权 ............................................... 198
关于作者 ............................................................................................ 202
封面介绍 ............................................................................................ 202
第 1 章 实时消息的价值 ........................................................................ 1
流动的数据 ............................................................................................................1
资源效率 ................................................................................................................3
使用场景 ................................................................................................................4
银行 .................................................................................................................5
医疗 .................................................................................................................6
安全 .................................................................................................................7
物联网 .............................................................................................................8
小结 .....................................................................................................................10
第 2 章 事件流和事件代理 ................................................................... 11
发布 / 订阅 ........................................................................................................... 11
队列 .....................................................................................................................13
故障类型 ..............................................................................................................15
推送与轮询 ..........................................................................................................15
为什么需要 Pulsar ................................................................................................16
流和队列的统一 ............................................................................................16
模块化 ...........................................................................................................16
性能 ...............................................................................................................18
小结 .....................................................................................................................19
第 3 章 认识 Pulsar ............................................................................ 20
Pulsar 的起源 .......................................................................................................20
Pulsar 的设计理念 ................................................................................................21
多租户 ...........................................................................................................22
跨地域复制 ...................................................................................................23
性能 ...............................................................................................................27
模块化 ...........................................................................................................27
Pulsar 生态 ...........................................................................................................28
Pulsar Functions ............................................................................................28
Pulsar IO .......................................................................................................29
Pulsar SQL ....................................................................................................30
Pulsar 成功案例 ...................................................................................................30
雅虎日本 .......................................................................................................30
Splunk ...........................................................................................................31
Iterable ..........................................................................................................32
小结 .....................................................................................................................33
第 4 章 Pulsar 内部组件 ..................................................................... 34
Broker ..................................................................................................................34
消息缓存 .......................................................................................................35
与 BookKeeper、ZooKeeper 交互 .................................................................37
Schema 校验 ..................................................................................................38
Broker 间的通信 ............................................................................................38
Pulsar Functions 和 Pulsar IO ........................................................................39
Apache BookKeeper .............................................................................................39
预写式日志 ...................................................................................................45
消息存储 .......................................................................................................45
对象 /Blob 存储 .............................................................................................46
Pravega ..........................................................................................................47
Majordodo .....................................................................................................47
Apache ZooKeeper ...............................................................................................48
命名服务 .......................................................................................................49
配置管理 .......................................................................................................49
选主 ...............................................................................................................50
通知系统 .......................................................................................................50
Apache Kafka ................................................................................................50
Apache Druid .................................................................................................51
Pulsar Proxy .........................................................................................................52
Java 虚拟机（JVM） .............................................................................................53
Netty ..............................................................................................................54
Apache Spark .................................................................................................55
Apache Lucene ..............................................................................................55
小结 .....................................................................................................................55
第 5 章 消费者 .................................................................................... 56
成为消费者意味着什么 ........................................................................................56
订阅 .....................................................................................................................57
独占 ...............................................................................................................60
共享 ...............................................................................................................61
键共享 ...........................................................................................................62
故障转移 .......................................................................................................65
确认 .....................................................................................................................66
单条确认 .......................................................................................................66
累积确认 .......................................................................................................67
Schema .................................................................................................................67
消费者 Schema 管理 .....................................................................................68
消费模式 ..............................................................................................................68
批处理 ...........................................................................................................68
分块 ...............................................................................................................69
高级配置 ..............................................................................................................70
延迟消息 .......................................................................................................70
保留策略 .......................................................................................................70
堆积限额 .......................................................................................................71
配置消费者 ..........................................................................................................72
回放 ...............................................................................................................73
死信主题 .......................................................................................................75
重试主题 .......................................................................................................76
小结 .....................................................................................................................76
第 6 章 生产者 .................................................................................... 77
同步生产者 ..........................................................................................................77
异步生产者 ..........................................................................................................78
生产者路由 ..........................................................................................................79
轮询路由 .......................................................................................................79
单分区路由 ...................................................................................................80
定制分区路由 ................................................................................................81
生产者配置 ..........................................................................................................83
topicName .....................................................................................................83
producerName ...............................................................................................83
sendTimeoutMs .............................................................................................83
blockIfQueueFull ...........................................................................................84
maxPendingMessages ....................................................................................84
maxPendingMessagesAcrossPartitions ...........................................................84
messageRoutingMode ....................................................................................84
hashingScheme ..............................................................................................84
cryptoFailureAction .......................................................................................85
batchingMaxPublishDelayMicros ..................................................................85
batchingMaxMessages ...................................................................................85
batchingEnabled ............................................................................................85
compressionType ...........................................................................................85
Schema on Write ..................................................................................................86
使用 Schema Registry ....................................................................................86
非持久化主题 ......................................................................................................89
使用场景 .......................................................................................................90
使用非持久化主题 ........................................................................................90
事务 .....................................................................................................................91
小结 .....................................................................................................................92
第 7 章 Pulsar IO ............................................................................... 93
Pulsar IO 架构 ......................................................................................................93
运行时 ...........................................................................................................94
性能考量 .......................................................................................................94
使用场景 ..............................................................................................................95
简单事件处理管道 ........................................................................................95
变化数据捕获 ................................................................................................96
考量 .....................................................................................................................97
消息序列化 ...................................................................................................97
管道稳定性 ...................................................................................................97
故障处理 .......................................................................................................98
示例 .....................................................................................................................99
Elasticsearch ..................................................................................................99
Netty ............................................................................................................ 100
创建自己的连接器 ............................................................................................. 101
TimescaleDB ............................................................................................... 101
小结 ................................................................................................................... 102
第 8 章 Pulsar Functions .................................................................. 103
流处理 ................................................................................................................ 103
Pulsar Functions 架构 ......................................................................................... 105
运行时 ......................................................................................................... 105
隔离性 ......................................................................................................... 106
在 Kubernetes 上部署 Pulsar Functions 的隔离性 .............................................. 107
使用场景 ............................................................................................................ 107
创建 Pulsar Functions .................................................................................. 108
简单事件处理 .............................................................................................. 108
主题清理 ..................................................................................................... 110
主题数据统计 .............................................................................................. 113
小结 ................................................................................................................... 115
第 9 章 分层存储 .............................................................................. 116
在云端存储数据 ................................................................................................. 117
对象存储 ..................................................................................................... 118
使用场景 ............................................................................................................ 120
复制 ............................................................................................................. 120
CQRS .......................................................................................................... 121
灾难恢复 ..................................................................................................... 122
卸载数据 ............................................................................................................ 122
Pulsar Offloader ........................................................................................... 122
检索已卸载的数据 ............................................................................................. 129
与对象存储数据交互 ................................................................................... 129
重填主题 ..................................................................................................... 129
使用 Pulsar 客户端 ...................................................................................... 130
小结 ................................................................................................................... 131
第 10 章 Pulsar SQL ........................................................................ 132
流即表 ................................................................................................................ 133
SQL-on-Anything 引擎 ...................................................................................... 135
Apache Flink: 另一种视角 ........................................................................... 136
Presto/Trino ................................................................................................. 137
Pulsar SQL 的工作原理 ...................................................................................... 138
配置 Pulsar SQL ................................................................................................. 140
性能考量 ............................................................................................................ 142
小结 ................................................................................................................... 143
第 11 章 部署 Pulsar ........................................................................ 144
Docker ................................................................................................................ 144
裸金属 ................................................................................................................ 146
最低要求 ..................................................................................................... 146
开始安装 ..................................................................................................... 146
部署 ZooKeeper........................................................................................... 147
启动 BookKeeper ........................................................................................ 148
启动 Pulsar .................................................................................................. 148
公有云厂商 ........................................................................................................ 149
AWS ............................................................................................................ 151
Azure ........................................................................................................... 152
Google Cloud Platform ................................................................................ 153
Kubernetes ......................................................................................................... 153
小结 ................................................................................................................... 155
第 12 章 运维 Pulsar ........................................................................ 156
Apache BookKeeper 指标 .................................................................................. 157
服务器指标 ................................................................................................. 157
Journal 指标 ................................................................................................ 158
存储指标 ..................................................................................................... 159
Apache ZooKeeper 指标 ..................................................................................... 159
服务器指标 ................................................................................................. 160
请求指标 ..................................................................................................... 161
主题指标 ............................................................................................................ 161
消费者指标 ........................................................................................................ 163
Pulsar 事务指标 ................................................................................................. 163
Pulsar Functions 指标 ......................................................................................... 164
高级运维技巧 .................................................................................................... 165
拦截器和追踪机制 ...................................................................................... 165
Pulsar SQL 指标 .......................................................................................... 167
指标转发 ............................................................................................................ 169
监控面板 ..................................................................................................... 169
小结 ................................................................................................................... 170
第 13 章 未来 ................................................................................... 171
编程语言支持 .................................................................................................... 171
扩展接口 ..................................................................................................... 171
增强 Pulsar Functions .................................................................................. 172
架构简化 / 扩展 ........................................................................................... 175
消息平台桥接器 .......................................................................................... 177
小结 ................................................................................................................... 180
附录 A Pulsar Admin API ................................................................. 181
附录 B Pulsar Admin CLI ................................................................. 189
附录 C 跨地域复制 ........................................................................... 192
附录 D Pulsar 中的安全、认证和鉴权 ............................................... 198
关于作者 ............................................................................................ 202
封面介绍 ............................................................................................ 202
猜您喜欢
