Channy's blog
//Description:
//Create Date: 2020-01-18 23:21:08
//Author: channy
postgres WAL
复制流
WALSender: 收集需要发送的WAL日志信息，在walwriter同期性刷新page的同时把page发送出去
WALReceiver: 把收到的WAL日志直接更新到本地磁盘的同时startup根据日志更新数据
共享内存：
MyWalSnd：已经发送出去日志的最新位置
output_message：要发送的WAL日志
通过socket交互
基本流程
主机
DDL -> WaitLatchOrSocket (in WalSndLoop) -> WaitEventSetWait -> vWaitEventSetWaitBlock -> epoll_wait
PostgresMain
exec_replication_command (walsender.c)
replication_scanner_init
StartReplication
WalSndLoop
ProcessRepliesIfAny (获取从client来的回复，更新共享内存中的各种位置、标识)
send_data(XLogSendPhysical, walsender.c，回调函数)
XLogRead (读取WAL日志到共享内存output_message中)
WalReceiverMain (walreceiver.c)
load_file("libpqwalreceiver", false); (walrcv_xxx函数所在地)
walrcv_connect
walrcv_identify_system (对应sender中exec_replication_command接收到的IdentifySystemCmd
walrcv_startstreaming
walrcv_receive (循环收取msg，接收WAL日志)
XLogWalRcvProcessMsg (更新共享内存，写的位置、时间)
appendBinaryStringInfo (把收到的信息增加到共享内存incoming_message中)
XLogWalRcvWrite (写WAL日志到磁盘文件，并更新共享内存中的recvOff和LogstreamResult.Write等)
XLogWalRcvSendReply
walrcv_send (回复wirtePtr/flushPtr/applyPtr的最新位置及当前时间)
walrcv_endstreaming
XLogWalRcvFlush
WakeupRecovery (唤醒startup进程)
XLogArchiveForceDone/XLogArchiveNotify (创建.done/.ready文件归档)
StartupProcessMain (startup.c)
StartupXLOG (xlog.c)
readRecoverySignalFile (旧的recovery.conf在这里被抛弃，读取standby.signal/recovery.signal)
XLogReaderAllocate
read_backup_label (backup_lable文件)
ReadCheckpointRecord (读取WAL record)
ReadRecord
XLogReadRecord
ReadPageInternal
XLogPageHeaderSize
(根据RedoStartLSN更新page)
UpdateControlFile (更新pg_control系统表，)
//貌似没走到
maybe_start_bgworkers
do_start_bgworker
StartBackgroundWorker (bgworker.c)
ApplyWorkerMain (worker.c)
walrcv_connect
walrcv_identify_system
walrcv_startstreaming
LogicalRepApplyLoop
walrcv_receive
apply_dispatch (处理各种信息：commit/insert/update/delete/relation...)
apply_handle_insert
logicalrep_read_insert (从获取的msg中读取数据到LogicalRepTupleData中)
slot_store_cstrings (把上一步LogicalRepTupleData的value填到slot中
ExecSimpleRelationInsert (pg12+ExecComputeStoredGenerated)
simple_table_tuple_insert (后同table_tuple_insert (tableam.h)正常insert数据)
send_feedback
get_flush_position
walrcv_send (回复wirte/flush/apply的最新位置及当前时间)
walrcv_endstreaming
//walrcv_receive等在库中实现
void
_PG_init(void)
{
if (WalReceiverFunctions != NULL)
elog(ERROR, "libpqwalreceiver already loaded");
WalReceiverFunctions = &PQWalReceiverFunctions;
}
PS: pg12中配置文件recovery.conf不再支持，直接修改配置postgres.conf就好
复制流配置
pg12配置主从设备时需要先配置好主设备再pg_basebackup
./pg_basebackup -h 127.0.0.1 -p 5432 -U repl -W -Fp -Xs -Pv -R -D ../data-standby01
//从机没有walwriter、autovacuum launcher和logical replication launcher,多了startup和walreceiver
3732 ? S 0:00 sshd: postgres@pts/0
4493 ? Ss 0:00 /usr/local/pgsql/bin/postgres -D /usr/local/pgsql/cpydata
4494 ? Ss 0:00 postgres: logger
4495 ? Ss 0:00 postgres: startup recovering 000000010000000000000005
4496 ? Ss 0:00 postgres: checkpointer
4497 ? Ss 0:00 postgres: background writer
4498 ? Ss 0:00 postgres: stats collector
4499 ? Ss 0:00 postgres: walreceiver streaming 0/5000060
4501 pts/0 S+ 0:00 grep --color=auto postgres
//主机多了archiver和walsender
2479 ? S 0:02 sshd: postgres@pts/0
4646 ? Ss 0:00 /usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
4647 ? Ss 0:00 postgres: logger
4649 ? Ss 0:00 postgres: checkpointer
4650 ? Ss 0:00 postgres: background writer
4651 ? Ss 0:00 postgres: walwriter
4652 ? Ss 0:00 postgres: autovacuum launcher
4653 ? Ss 0:00 postgres: archiver
4654 ? Ss 0:00 postgres: stats collector
4655 ? Ss 0:00 postgres: logical replication launcher
4656 ? Ss 0:00 postgres: walsender postgres 10.1.0.23(39154) streaming 0/5000330
4696 pts/0 S+ 0:00 grep --color=auto postgres
本机WAL日志
每次insert/update等操作时都会触发WAL日志写
heap_insert (heapam.c)
RelationGetBufferForTuple
RelationPutHeapTuple
(准备工作，register buffer、data等)
XLogInsert(RM_HEAP_ID, info)
GetFullPageWriteInfo
XLogRecordAssemble (填充WAL头信息)
XLogInsertRecord
CopyXLogRecordToWAL (写到缓存)
(更新LogwrtRqst.Write位置)
PageSetLSN(page, recptr);
//XLogInsert
- 后台定时flush
WalWriterMain (walwriter.c, 变量init->信号函数register->环境init->异常处理register->loop) XLogBackgroundFlush (xlog.c) XLogWrite (write写) - 有insert/update时flush
exec_simple_query (postgres.c) finish_xact_command CommitTransactionCommand (xact.c) CommitTransaction RecordTransactionCommit XactLogCommitRecord XLogInsert (xloginsert.c) XLogFlush (xlog.c) XLogWrite (写WAL到磁盘)
其它工具类
pg_dump是WAL日志导出工具，用于备份
getopt_long (解析命令行参数)
CreateArchive (根据指定的参数创建初始化相应格式的文档)
ConnectDatabase (连接数据库)
setup_connection (执行一堆set语句)
getTableData
getDependencies
......
RestoreArchive
概况的说，pg_dump导出的内容可以分为数据库对象的定义和数据。数据库对象的定义导出时通过查询系统表把对应元数据信息读取出来后，把该对象的各类信息置于一个链表上包括其依赖对象的oid。而具体的数据，也就是每个数据包的数据也被抽象为一个数据库对象，保存在此链表中。通过调节导出顺序把数据库对象的定义导出然后导出数据，置于通过链表中对应数据对象节点的信息，执行相应的SQL语句，从表中读出数据然后导出写出去。所以，在内存中只是链表上对象的定义，数据是边读边写出的，可以使用流式读出。
gdb 调试信息
//主机DDL操作，唤醒walsender
Program received signal SIGUSR1, User defined signal 1.
0x00007f1c48673b77 in epoll_wait (epfd=7, events=0x140ef30, maxevents=1, timeout=29999) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30 in ../sysdeps/unix/sysv/linux/epoll_wait.c
(gdb) where
#0 0x00007f1c48673b77 in epoll_wait (epfd=7, events=0x140ef30, maxevents=1, timeout=29999) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00000000008d0d59 in WaitEventSetWaitBlock (set=0x140eeb8, cur_timeout=29999, occurred_events=0x7ffd120d83c0, nevents=1) at latch.c:1080
#2 0x00000000008d0c34 in WaitEventSetWait (set=0x140eeb8, timeout=29999, occurred_events=0x7ffd120d83c0, nevents=1, wait_event_info=83886092) at latch.c:1032
#3 0x00000000008d03b9 in WaitLatchOrSocket (latch=0x7f1c478e91f4, wakeEvents=43, sock=10, timeout=29999, wait_event_info=83886092) at latch.c:407
#4 0x000000000088e57e in WalSndLoop (send_data=0x88ee23 <XLogSendPhysical>) at walsender.c:2270
#5 0x000000000088c035 in StartReplication (cmd=0x140de00) at walsender.c:697
#6 0x000000000088d73e in exec_replication_command (cmd_string=0x138e648 "START_REPLICATION 0/5000000 TIMELINE 1") at walsender.c:1544
#7 0x0000000000903c01 in PostgresMain (argc=1, argv=0x13bc2b0, dbname=0x13bc178 "", username=0x13bc150 "postgres") at postgres.c:4232
#8 0x0000000000856983 in BackendRun (port=0x13b34b0) at postmaster.c:4437
#9 0x0000000000856103 in BackendStartup (port=0x13b34b0) at postmaster.c:4128
#10 0x0000000000852486 in ServerLoop () at postmaster.c:1704
#11 0x0000000000851ce4 in PostmasterMain (argc=3, argv=0x1388fa0) at postmaster.c:1377
#12 0x0000000000770684 in main (argc=3, argv=0x1388fa0) at main.c:228