摘要
Canal，让数据流动起来！它是一款强大的数据同步工具，让你轻松实现数据的实时同步和监控。快来尝试吧！
正文
Canal详尽新手入门实战演练(应用汇总)
Canal详细介绍
Canal介绍
canal [kə’næl]，译意为水路/管路/水渠，适用范围是根据 MySQL 数据库查询增加量日志分析，给予增加量数据信息定阅和消費
初期阿里由于杭州市和英国双主机房布署，存有跨主机房同歩的业务流程要求，完成方法主要是根据业务流程 trigger 获得增加量变动。从 2010 年逐渐，业务流程逐渐试着数据库查询日志分析获得增加量变动开展同歩，从而衍化出了很多的数据库查询增加量定阅和消費业务流程。
根据日志增加量定阅和消費的业务流程包含
- 数据库查询镜像系统
- 数据库查询即时备份数据
- 数据库索引搭建和即时维护保养(分拆对映异构数据库索引、全文索引等)
- 业务流程 cache 更新
- 带领域模型的增加量数据处理方法
当今的 canal 适用源端 MySQL 版本号包含 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
MySQL原理
- MySQL master 将数据信息变动载入二进制日志( binary log, 在其中纪录称为二进制日志事情binary log events，能够根据 show binlog events 开展查询)
- MySQL slave 将 master 的 binary log events 拷到它的无线中继日志(relay log)
- MySQL slave 播放 relay log 中事情，将数据信息变动体现它自身的数据信息
Canal原理
- canal 仿真模拟 MySQL slave 的互动协议书，掩藏自身为 MySQL slave ，向 MySQL master 推送dump 协议书
- MySQL master 接到 dump 要求，逐渐消息推送 binary log 给 slave (即 canal )
- canal 分析 binary log 目标(初始为 byte 流)
Canal应用提前准备
MySQL提前准备
1、针对建造 MySQL , 必须先打开 Binlog 载入作用，配备 binlog-format 为 ROW 方式，my.cnf 中配备以下
[mysqld]
log-bin=mysql-bin # 打开binlog
binlog-format=ROW # 挑选ROW方式
server_id=1 # 配备MySQL replaction必须界定，不必和canal的slaveId反复
2、受权 canal 连接 MySQL 账户具备做为 MySQL slave 的管理权限, 假如现有帐户可立即 grant
-- 应用指令登陆：mysql -u root -p
-- 建立mysql客户 登录名：canal 登陆密码：canal
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
3、重新启动数据库查询，查询配备是不是起效
mysql> show variables like 'binlog_format';
--------------- -------
| Variable_name | Value |
--------------- -------
| binlog_format | ROW |
--------------- -------
mysql> show variables like 'log_bin';
--------------- -------
| Variable_name | Value |
--------------- -------
| log_bin | ON |
--------------- -------
Canal应用提前准备
Canal免费下载，这儿应用1.1.5版本号为例子
GitHub连接：https://github.com/alibaba/canal/releases
必须免费下载(别的可依据自主必须开展免费下载)：
- canal.deployer-1.1.5.tar.gz
- canal.adapter-1.1.5.tar.gz
免费下载后缓解压力
1.1.5缓解压力后文件目录构造以下
bin ：canal运行、重新启动、终止文档
conf ：canal环境变量
lib ：canal运作需要的jar包，留意数据库查询驱动包版本号，可手动式拆换
logs ：canal运作日志
plugin ：一些拓展包，如消息队列
建立数据库和表
踩坑纪录：数据库查询名不必应用下横线，一开始用canal_test_01的方法取名，发觉扫描仪不上数据库查询
-- 源数据库查询：canal01
CREATE DATABASE `canal01` CHARACTER SET 'utf8mb4';
-- 建立user01表
CREATE TABLE `user01` (
`id` int(64) NOT NULL AUTO_INCREMENT,
`username` varchar(64) DEFAULT NULL,
`password` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;
-- 总体目标数据库查询：canal02
CREATE DATABASE `canal02` CHARACTER SET 'utf8mb4';
-- 建立user02表
CREATE TABLE `user02` (
`id` int(64) NOT NULL AUTO_INCREMENT,
`username` varchar(64) DEFAULT NULL,
`password` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;
Canal-deployer应用
1、conf\canal.properties环境变量(这儿只详细介绍我采用的配备)
# canal-deployer的ip详细地址，不配备默认设置是该设备
canal.ip =
# canal-deployer的端口
canal.port = 11111
# 方式
canal.serverMode = tcp
# 载入好几个环境变量，需要在canal.deployer-1.1.5\conf下建立相匹配的文件夹名称，并再加上instance.properties环境变量
canal.destinations = example
# 配备好几个应用英文逗号分隔
# canal.destinations = example,example2
2、conf\example\instance.properties环境变量(这儿只详细介绍我采用的配备)
# 不可以与my.ini下的server_id=1同样
canal.instance.mysql.slaveId=1234
# mysql详细地址和端口号
canal.instance.master.address=127.0.0.1:3306
# 数据库查询账户密码，能够应用root账户或刚建立的canal账户
canal.instance.dbUsername=root
canal.instance.dbPassword=root
canal.instance.connectionCharset=UTF-8
# 配备监视，适用正则表达式
canal.instance.filter.regex=canal01\\..*
# 配备不监视，适用正则表达式
canal.instance.filter.black.regex=mysql\\.slave_.*
3、查验lib文件目录下的jar包，一定要有相匹配的数据库查询驱动包，且版本号要对得上
4、在bin文件目录中运行canal-deployer，运行取得成功以下輸出
# 运作后輸出
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128米;
support was removed in 8.0 Listening for transport dt_socket at address: 9099
# 在logs\canal文件目录下，会纪录canal.log
# 有以下纪录表明canal-deployer运作取得成功
2021-04-28 09:26:57.194 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2021-04-28 09:26:57.244 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2021-04-28 09:26:57.257 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2021-04-28 09:26:57.399 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.3.51(192.168.3.51):11111]
2021-04-28 09:26:59.440 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
# 在logs\example文件目录下，会纪录example.log
# 有以下纪录表明canal-deployer与连接数据库成功了
2021-04-28 09:30:37.159 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2021-04-28 09:30:37.175 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^fooddb\..*$
2021-04-28 09:30:37.175 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2021-04-28 09:30:37.270 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2021-04-28 09:30:37.288 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2021-04-28 09:30:37.305 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
{"identity":{"slaveId":-1,"sourceAddress":{"address":"ieonline.microsoft.com","port":3306}},"postion":{"gtid":"","included":false,"journalName":"binlog.000068","position":14636,"serverId":1,"timestamp":1619408047000}}
2021-04-28 09:30:37.744 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=binlog.000068,position=14636,serverId=1,gtid=,timestamp=1619408047000] cost : 466ms , the next step is binlog dump
5、建立springboot工程项目监视canal-deployer(可绕过)
导进maven座标(pom.xml)
<!--Canal-->
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.1.4</version>
</dependency>
建立CanalClient类，并交到spring管理方法
import java.net.InetSocketAddress;
import java.util.List;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;
@Component
public class CanalClient implements InitializingBean {
private final static int BATCH_SIZE = 1000;
@Override
public void afterPropertiesSet() throws Exception {
// 建立连接
CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress("127.0.0.1", 11111), "example", "", "");
try {
//开启联接
connector.connect();
//定阅数据库表,所有表
connector.subscribe(".*\\..*");
//回退到未开展ack的地区，下一次fetch的情况下，能够从最后一个沒有ack的地区逐渐拿
connector.rollback();
while (true) {
// 获得特定总数的数据信息
Message message = connector.getWithoutAck(BATCH_SIZE);
//获得大批量ID
long batchId = message.getId();
//获得大批量的总数
int size = message.getEntries().size();
//要是没有数据信息
if (batchId == -1 || size == 0) {
try {
//进程休眠状态2秒
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
//如果有数据信息,解决数据信息
printEntry(message.getEntries());
}
//开展 batch id 的确定。确定以后，不大于此 batchId 的 Message 都是会被确定。
connector.ack(batchId);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
connector.disconnect();
}
}
/**
* 打印出canal server分析binlog得到的dao层信息内容
*/
private static void printEntry(List<Entry> entrys) {
for (Entry entry : entrys) {
if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
//打开/关掉事务管理的实体线种类，绕过
continue;
}
//RowChange目标，包括了一行数据信息转变的全部特点
//例如isDdl 是不是ddl变动实际操作 sql 实际的ddl sql beforeColumns afterColumns 变动前后左右的数据字段这些
RowChange rowChage;
try {
rowChage = RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" entry.toString(), e);
}
//获得实际操作种类：insert/update/delete种类
EventType eventType = rowChage.getEventType();
//打印出Header信息内容
System.out.println(String.format("================》; binlog[%s:%s] , name[%s,%s] , eventType : %s",
entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
eventType));
//分辨是不是DDL句子
if (rowChage.getIsDdl()) {
System.out.println("================》;isDdl: true,sql:" rowChage.getSql());
}
//获得RowChange目标里的每一行数据信息，打印出出去
for (RowData rowData : rowChage.getRowDatasList()) {
//如果是删除语句
if (eventType == EventType.DELETE) {
printColumn(rowData.getBeforeColumnsList());
//如果是增加句子
} else if (eventType == EventType.INSERT) {
printColumn(rowData.getAfterColumnsList());
//如果是升级的句子
} else {
//变动前的数据信息
System.out.println("------->; before");
printColumn(rowData.getBeforeColumnsList());
//变动后的数据信息
System.out.println("------->; after");
printColumn(rowData.getAfterColumnsList());
}
}
}
}
private static void printColumn(List<Column> columns) {
for (Column column : columns) {
System.out.println(column.getName() " : " column.getValue() " update=" column.getUpdated());
}
}
}
运行springboot新项目，在user01表格中加上数据信息，控制面板以下輸出，监视取得成功
================》; binlog[binlog.000070:5154] , name[canal01,user01] , eventType : INSERT
id : 1 update=true
username : a update=true
password : a update=true
Canal-adapter应用
1、conf\application.yml环境变量(这儿只详细介绍我采用的配备)
server:
port: 8081 # canal-adapter运作端口
canal.conf:
mode: tcp # canal client的方式: tcp kafka rocketMQ rabbitMQ
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 127.0.0.1:11111
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
srcDataSources: # 源数据库查询
canal01: # 自定名字
# 这儿应用的是mysql8.0，必须再加上serverTimezone=UTC
url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
username: root
password: root
# 配备好几个源数据库查询
# canal02:
# url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
# username: root
# password: root
canalAdapters: # 电源适配器目录
- instance: example # canal 实例名或是 MQ topic 名
groups: # 排序目录
- groupId: # 排序id, 如果是MQ方式将采用该值
outerAdapters: # 排序内电源适配器目录
# - name: logger # 日志打印出电源适配器
- name: rdb # 特定为rdb种类同歩
key: mysql1 # 特定adapter的唯一key, 与表投射配备中outerAdapterKey相匹配
# 总体目标数据库查询配备
properties:
jdbc.driverClassName: com.mysql.jdbc.Driver
jdbc.url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
jdbc.username: root
jdbc.password: root
threads: 5 # 并行执行的线程数,可以不改动，默认设置为1
# 配备好几个总体目标数据库查询
# - instance: example2 # canal instance Name or mq topic name
# groups:
# - groupId: g2
# outerAdapters:
# - name: rdb
# key: mysql2
# properties:
# jdbc.driverClassName: com.mysql.jdbc.Driver
# jdbc.url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
# jdbc.username: root
# jdbc.password: root
2、配备RDB表投射文档(conf/rdb/mytest_user.yml文件)
dataSourceKey: canal01 # 源数据库的key, 相匹配上边配备的srcDataSources中的值
destination: example # cannal的instance或是MQ的topic
groupId: g1 # 相匹配MQ方式下的groupId, 总是同歩相匹配groupId的数据信息
outerAdapterKey: mysql1 # adapter key, 相匹配上边配备outAdapters中的key
concurrent: true # 是不是按外键约束hash并行处理同歩, 并行处理同歩的表务必确保外键约束不容易变更及外键约束不可以为别的同歩表的外键约束!!
dbMapping:
database: canal01 # 源数据库的database/shcema
table: user01 # 源数据库表名
targetTable: user02 # 立即写表名
targetPk: # 外键约束投射
id: id # 如果是复合型外键约束能够自动换行投射好几个
mapAll: true # 是不是整表投射, 规定源表和总体目标表字段一模一样 (假如targetColumns也配备了投射,则以targetColumns配备为标准)
# targetColumns: # 字段名投射, 文件格式: 总体目标表字段: 源表字段, 假如字段一样源表字段可不填
# id:
# name:
# role_id:
# c_time:
# test1:
etlCondition: "where c_time>={}"
commitBatch: 3000 # 大批量递交的尺寸
3、查验lib文件目录下的jar包，一定要有相匹配的数据库查询驱动包，且版本号要对得上
4、在bin文件目录中运行canal-adapter，运行取得成功以下輸出
# 运作后輸出
2021-04-28 11:12:38.281 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServe
rApplicationContext - Refreshing org.springframework.boot.web.servlet.context.An
notationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr
28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.Annotation
ConfigApplicationContext@1a677343
2021-04-28 11:12:39.012 [main] INFO org.springframework.cloud.context.scope.Gen
ericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2021-04-28 11:12:39.108 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$B
eanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.Configur
ationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.aut
oconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGL
IB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (f
or example: not eligible for auto-proxying)
2021-04-28 11:12:39.610 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServe
r - Tomcat initialized with port(s): 8081 (http)
2021-04-28 11:12:39.632 [main] INFO org.apache.coyote.http11.Http11NioProtocol
- Initializing ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:39.646 [main] INFO org.apache.catalina.core.StandardService -
Starting service [Tomcat]
2021-04-28 11:12:39.647 [main] INFO org.apache.catalina.core.StandardEngine - S
tarting Servlet Engine: Apache Tomcat/8.5.29
2021-04-28 11:12:39.660 [localhost-startStop-1] INFO org.apache.catalina.core.A
prLifecycleListener - The APR based Apache Tomcat Native library which allows op
timal performance in production environments was not found on the java.library.p
ath: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\sys
tem32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Wind
ows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java
\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\
bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Com
mon;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\User
s\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\Je
tBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4
\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C
:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.]
2021-04-28 11:12:39.830 [localhost-startStop-1] INFO o.a.catalina.core.Containe
rBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationCont
ext
2021-04-28 11:12:39.831 [localhost-startStop-1] INFO org.springframework.web.co
ntext.ContextLoader - Root WebApplicationContext: initialization completed in 15
50 ms
2021-04-28 11:12:39.990 [localhost-startStop-1] INFO o.s.boot.web.servlet.Servl
etRegistrationBean - Servlet dispatcherServlet mapped to [/]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2021-04-28 11:12:40.001 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2021-04-28 11:12:40.002 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class
is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SP
I and manual loading of the driver class is generally unnecessary.
2021-04-28 11:12:40.597 [main] INFO com.alibaba.druid.pool.DruidDataSource - {d
ataSource-1} inited
2021-04-28 11:12:40.849 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springf
ramework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.143 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.serv
let.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup
date [Wed Apr 28 11:12:38 GMT 08:00 2021]; parent: org.springframework.context.a
nnotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:41.245 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto publi
c java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adap
ter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.S
tring)
2021-04-28 11:12:41.247 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java
.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.la
uncher.rest.CommonRest.count(java.lang.String,java.lang.String)
2021-04-28 11:12:41.248 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" on
to public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otte
r.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public
java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapt
er.launcher.rest.CommonRest.etl(java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public
com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.cana
l.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.la
ng.String,java.lang.String)
2021-04-28 11:12:41.250 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.a
libaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adap
ter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.Str
ing)
2021-04-28 11:12:41.251 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.L
ist<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.a
dapter.launcher.rest.CommonRest.destinations()
2021-04-28 11:12:41.256 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/error]}" onto public org.springframework.http.Respons
eEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.b
oot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.htt
p.HttpServletRequest)
2021-04-28 11:12:41.258 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springf
ramework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.ser
vlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse)
2021-04-28 11:12:41.298 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/webjars/**] onto handler of type [class org.springframe
work.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.299 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/**] onto handler of type [class org.springframework.web
.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.604 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Registering beans for JMX exposure on startup
2021-04-28 11:12:41.618 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2021-04-28 11:12:41.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'configurationPropertiesRebinder' has been autodetected
for JMX exposure
2021-04-28 11:12:41.622 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'environmentManager' has been autodetected for JMX expos
ure
2021-04-28 11:12:41.625 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'environmentManager': registering with JMX server
as MBean [org.springframework.cloud.context.environment:name=environmentManager,
type=EnvironmentManager]
2021-04-28 11:12:41.644 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'refreshScope': registering with JMX server as MBe
an [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=Refre
shScope]
2021-04-28 11:12:41.659 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'configurationPropertiesRebinder': registering wit
h JMX server as MBean [org.springframework.cloud.context.properties:name=configu
rationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder]
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## syncSwitch refreshed.
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## start the canal client adapters.
2021-04-28 11:12:41.678 [main] INFO c.a.otter.canal.client.adapter.support.Exte
nsionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools
\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:41.777 [main] INFO c.a.otter.canal.client.adapter.rdb.config.C
onfigLoader - ## Start loading rdb mapping config ...
2021-04-28 11:12:41.837 [main] INFO c.a.otter.canal.client.adapter.rdb.config.C
onfigLoader - ## Rdb mapping config loaded
2021-04-28 11:12:42.224 [main] INFO com.alibaba.druid.pool.DruidDataSource - {d
ataSource-2} inited
2021-04-28 11:12:42.234 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterLoader - Load canal adapter: rdb succeed
2021-04-28 11:12:42.246 [main] INFO c.alibaba.otter.canal.connector.core.spi.Ex
tensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Too
ls\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:42.275 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-04-28 11:12:42.276 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader
.AdapterProcessor - =============> Start to connect destination: example <======
=======
2021-04-28 11:12:42.276 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## the canal client adapters are running now ......
2021-04-28 11:12:42.286 [main] INFO org.apache.coyote.http11.Http11NioProtocol
- Starting ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:42.296 [main] INFO org.apache.tomcat.util.net.NioSelectorPool
- Using a shared selector for servlet write/read
2021-04-28 11:12:42.316 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServe
r - Tomcat started on port(s): 8081 (http) with context path ''
2021-04-28 11:12:42.320 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapt
erApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running fo
r 6.186)
2021-04-28 11:12:42.354 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader
.AdapterProcessor - =============> Subscribe destination: example succeed <=====
========
# logs\adapter文件目录下，会纪录adapter.log
# 有以下纪录表明canal-adapter取得成功运作了
2021-04-28 11:12:37.612 [main] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343: startup date [Wed Apr 28 11:12:37 CST 2021]; root of context hierarchy
2021-04-28 11:12:38.027 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-28 11:12:38.254 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default
2021-04-28 11:12:38.281 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:39.012 [main] INFO org.springframework.cloud.context.scope.GenericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2021-04-28 11:12:39.108 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-28 11:12:39.610 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http)
2021-04-28 11:12:39.632 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:39.646 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat]
2021-04-28 11:12:39.647 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
2021-04-28 11:12:39.660 [localhost-startStop-1] INFO org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\Users\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\JetBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.]
2021-04-28 11:12:39.830 [localhost-startStop-1] INFO o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2021-04-28 11:12:39.831 [localhost-startStop-1] INFO org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 1550 ms
2021-04-28 11:12:39.990 [localhost-startStop-1] INFO o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2021-04-28 11:12:40.001 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2021-04-28 11:12:40.002 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2021-04-28 11:12:40.597 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2021-04-28 11:12:40.849 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.143 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 GMT 08:00 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:41.245 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.247 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String)
2021-04-28 11:12:41.248 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.250 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.251 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations()
2021-04-28 11:12:41.256 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2021-04-28 11:12:41.258 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2021-04-28 11:12:41.298 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.299 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.604 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2021-04-28 11:12:41.618 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2021-04-28 11:12:41.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2021-04-28 11:12:41.622 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure
2021-04-28 11:12:41.625 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2021-04-28 11:12:41.644 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2021-04-28 11:12:41.659 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder]
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed.
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2021-04-28 11:12:41.678 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:41.777 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ...
2021-04-28 11:12:41.837 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
2021-04-28 11:12:42.224 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited
2021-04-28 11:12:42.234 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
2021-04-28 11:12:42.246 [main] INFO c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:42.275 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-04-28 11:12:42.276 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <=============
2021-04-28 11:12:42.276 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2021-04-28 11:12:42.286 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:42.296 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2021-04-28 11:12:42.316 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2021-04-28 11:12:42.320 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running for 6.186)
2021-04-28 11:12:42.354 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============
5、向user01表格中加上数据信息，user02表也是有相匹配数据信息，同歩取得成功
2021-04-28 11:27:59.619 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.rdb.s
ervice.RdbSyncService - DML: {"data":{"id":1,"username":"a","password":"a"},"dat
abase":"canal01","destination":"example","old":null,"table":"user01","type":"INS
ERT"}
如需user02同歩到user01：
（1）加上相匹配的canal-deployer中加上相匹配的example
（2）canal-adapter的application.yml环境变量中添加相匹配的源数据库查询和总体目标数据库查询(实际见上边配备)
（3）新创建RDB表投射文档，在rdb文件目录下，canal-adapter会默认设置扫描仪rdb文件目录下全部的yml文件
同歩好几张表：
（1）在2个数据库查询中提升表（这儿以person为例子）
（2）配备RDB表投射文档，将mytest_user.yml拷贝一份，重新命名为mytest_person.yml
（3）改动mytest_person.yml中的table(源表)、targetTable(总体目标表)
6、假如必须应用Java工程项目，可自主在github的canal中免费下载canal-adapter的工程项目，配备与上边彻底一样
关注不迷路
扫码下方二维码，关注宇凡盒子公众号，免费获取最新技术内幕！
温馨提示：如果您访问和下载本站资源，表示您已同意只将下载文件用于研究、学习而非其他用途。
评论0