浙江龙网

  1. 首页 > 生活百科 > >

Mysql通过Canal同步Elasticsearch( 二 )

生活百科

编写配置文件# 切换到挂载目录$ cd /usr/kang/docker-canal# 切换到需要修改的配置文件所在目录$ cd canal-server/conf/example/# 修改文件$ vim instance.properties更改 slaveId=10 ，记得和数据库的 server-id 不一样
更改数据库地址
canal.instance.mysql.slaveId=0canal.instance.master.address=192.168.80.80:3306canal.instance.dbUsername=rootcanal.instance.dbPassword=123456789

Mysql通过Canal同步Elasticsearch

文章插图
################################################### mysql serverId , v1.0.26+ will autoGen# canal.instance.mysql.slaveId=0# enable gtid use true/falsecanal.instance.gtidon=false# position info# 数据库地址canal.instance.master.address=192.168.118.130:3306# 当前正在写入的 binlog文件 ， 第二部分中mysql命令可查询canal.instance.master.journal.name=binlog.000001# 正在写入的偏移量canal.instance.master.position=156canal.instance.master.timestamp=canal.instance.master.gtid=# rds oss binlogcanal.instance.rds.secretkey=canal.instance.rds.instanceId=# table meta tsdb infocanal.instance.tsdb.enable=true#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb#canal.instance.tsdb.dbUsername=canal#canal.instance.tsdb.dbPassword=canal#canal.instance.standby.address =#canal.instance.standby.journal.name =#canal.instance.standby.position =#canal.instance.standby.timestamp =#canal.instance.standby.gtid=# username/password# mysql 用户，若之前创建过可不用修改canal.instance.dbUsername=canalcanal.instance.dbPassword=canalcanal.instance.connectionCharset = UTF-8# enable druid Decrypt database passwordcanal.instance.enableDruid=false#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==# table regexcanal.instance.filter.regex=.*\\..*# table black regexcanal.instance.filter.black.regex=mysql\\.slave_.*# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch# mq configcanal.mq.topic=example# dynamic topic route by schema or table regex#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*canal.mq.partition=0# hash partition config#canal.mq.partitionsNum=3#canal.mq.partitionHash=test.table:id^name,.*\\..*#canal.mq.dynamicTopicPartitionNum=test.*:4,mycanal:6#################################################
Mysql通过Canal同步Elasticsearch

文章插图
问题：# 下面是挂在的目录 ， 需要授权，不然挂在出来，里面拒绝访问chmod 777 /home/haha/docker-canal/
Mysql通过Canal同步Elasticsearch

文章插图
PUT /area_city{"mappings": {"properties": {"id":{"type": "long"},"pid":{"type": "integer"},"deep":{"type": "integer"},"name":{"type": "text"},"pinyin_prefix":{"type": "text"},"pinyin":{"type": "text"},"ext_id":{"type": "text"},"ext_name":{"type": "text"}}}}安装 Canal-Adapter安装镜像BASH# 拉取镜像$ docker pull slpcat/canal-adapter:v1.1.5# 随意启动，用于挂载配置文件$ docker run --name canal-adapter -p 8081:8081 -d slpcat/canal-adapter:v1.1.5# 创建存储挂载的目录$ mkdir /usr/kang/docker-canal-adapter# 挂载文件$ docker cp canal-adapter:/opt/canal-adapter /home/haha/docker-canal-adapter# 删除容器$ docker stop canal-adapter$ docker rm canal-adapter# 建议添加权限chmod 777 /home/haha/docker-canal-adapter# 启动新的容器$ docker run --name canal-adapter -p 8081:8081 -v /home/haha/docker-canal-adapter/canal-adapter:/opt/canal-adapter -d slpcat/canal-adapter:v1.1.5配置canal-adapter 文件'
Mysql通过Canal同步Elasticsearch

文章插图
需要修改的是，conf目录下的 applicatiopn.yml 、es7目录下的 mytest_user.yml
编辑 application.yml:
Mysql通过Canal同步Elasticsearch

文章插图
BASH# 切换到挂载目录$ cd /home/haha/docker-canal-adapter/canal-adapter$ cd conf/$ vim application.ymlYAMLserver:port: 8081spring:jackson:date-format: yyyy-MM-dd HH:mm:sstime-zone: GMT+8default-property-inclusion: non_nullcanal.conf:mode: tcp #tcp kafka rocketMQ rabbitMQflatMessage: truezookeeperHosts:syncBatchSize: 1000retries: 0timeout:accessKey:secretKey:consumerProperties:# canal tcp consumer编辑此处为 canal.deployer 的地址，canal.deployer的默认端口为 11111# 用于获取 canal 的数据进行实时同步canal.tcp.server.host: 192.168.80.80:11111canal.tcp.zookeeper.hosts:canal.tcp.batch.size: 500canal.tcp.username:canal.tcp.password:# kafka consumer#kafka.bootstrap.servers: 127.0.0.1:9092#kafka.enable.auto.commit: false#kafka.auto.commit.interval.ms: 1000#kafka.auto.offset.reset: latest#kafka.request.timeout.ms: 40000#kafka.session.timeout.ms: 30000#kafka.isolation.level: read_committed#kafka.max.poll.records: 1000# rocketMQ consumer#rocketmq.namespace:#rocketmq.namesrv.addr: 127.0.0.1:9876#rocketmq.batch.size: 1000#rocketmq.enable.message.trace: false#rocketmq.customized.trace.topic:#rocketmq.access.channel:#rocketmq.subscribe.filter:# rabbitMQ consumer#rabbitmq.host:#rabbitmq.virtual.host:#rabbitmq.username:#rabbitmq.password:#rabbitmq.resource.ownerId:srcDataSources:defaultDS: # 修改此处为 数据库信息url: jdbc:mysql://192.168.80.80:3306/estest?useUnicode=trueusername: rootpassword: 123456789canalAdapters:- instance: example # canal instance Name or mq topic namegroups:- groupId: g1outerAdapters:- name: logger#- name: rdb#key: mysql1#properties:#jdbc.driverClassName: com.mysql.jdbc.Driver#jdbc.url: jdbc:mysql://127.0.0.1:3306/mytest2?useUnicode=true#jdbc.username: root#jdbc.password: 121212#- name: rdb#key: oracle1#properties:#jdbc.driverClassName: oracle.jdbc.OracleDriver#jdbc.url: jdbc:oracle:thin:@localhost:49161:XE#jdbc.username: mytest#jdbc.password: m121212#- name: rdb#key: postgres1#properties:#jdbc.driverClassName: org.postgresql.Driver#jdbc.url: jdbc:postgresql://localhost:5432/postgres#jdbc.username: postgres#jdbc.password: 121212#threads: 1#commitSize: 3000#- name: hbase#properties:#hbase.zookeeper.quorum: 127.0.0.1#hbase.zookeeper.property.clientPort: 2181#zookeeper.znode.parent: /hbase- name: es7 # es7文件夹配置,还有es6#key: fgnKeyhosts: 192.168.80.80:9200 # 127.0.0.1:9200 for rest mode es 集群地址, 逗号分隔properties:mode: rest # or rest 可指定transport模式或者rest模式## security.auth: test:123456 #only used for rest modecluster.name: elasticsearch #指定es的集群名称#- name: kudu#key: kudu#properties:#kudu.master.address: 127.0.0.1 # ',' split multi address

推荐阅读


上一篇：商品期货通用模型JF1

下一篇：Vue3.x+element-plus+ts踩坑笔记