同步Mysql数据库到Elasticsearch中

下载安装

• 在官网下载logstash对应的tar.gz压缩文件，使用tar -xzvf进行解压
• 解压之后进入文件夹，安装插件logstash-input-jdbc

./bin/logstash-plugin install logstash-input-jdbc

• 准备对应的mysql-connector-java.jar，可以在官网下载

配置

创建配置文件logstash.conf并在里面写入配置

input {
        jdbc {
                jdbc_driver_library => "/home/mysql-connector-java-8.0.16.jar"
                jdbc_driver_class => "com.mysql.jdbc.Driver"
                jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test"
                jdbc_user => "root"
                jdbc_password => "123"
                jdbc_paging_enabled => "true"
                jdbc_page_size => "50000"
                use_column_value => true
                tracking_column => "id"  # 追踪主键id
                tracking_column_type => numeric
                statement => "SELECT id, name, address FROM employee WHERE id > :sql_last_value"  #sql_last_value为上次最后id，以免插入重复
                schedule => "* * * * *"
        }
}

filter {
        json{
                source => "message"
                remove_field => ["message"]
        }
}

output {
        stdout {
                codec => rubydebug
        }
        elasticsearch {
                hosts => "127.0.0.1:9200"
                index => "logstash_test"  # 要写入的index
                document_id => "%{id}"
        }
}

启动

./bin/logstash -f ./logstash.conf

启动logstash后将会每隔一分钟追踪mysql中数据，同步到elasticsearch之中