| ... | ... | @@ -24,7 +24,7 @@ ic_ar --> table_update_redis: 入库时将变化记录的主键写入redis |
|
|
|
ic_base --> table_update_redis: 入库时将变化记录的主键写入redis
|
|
|
|
ic_biz --> table_update_redis: 入库时将变化记录的主键写入redis
|
|
|
|
table_update_redis --> update_data_json: redis_reader,udm_filter,file_writer
|
|
|
|
update_data_json --> update_data_json_province: 按省份分目录
|
|
|
|
update_data_json --> update_data_json_province: 按省份分目录,/data_227/data_sync/lake_ic/province/province_short/
|
|
|
|
|
|
|
|
@enduml
|
|
|
|
```
|
| ... | ... | @@ -38,12 +38,14 @@ data_pump配置文件： |
|
|
|
/home/collie/product/app_online_lake/data_pump/new_online/all_spider_update_lake.yml
|
|
|
|
supervisor配置：(29个进程)
|
|
|
|
/home/collie/product/app_online_lake/supervisor/ic_spider_sync_lake.conf
|
|
|
|
/etc/supervisord.d/ic_spider_sync_lake.conf
|
|
|
|
2.消费kafka更新mysql并写入redis：
|
|
|
|
部署地址：10.8.6.84
|
|
|
|
data_pump配置文件：
|
|
|
|
/home/collie/product/app_online_lake/data_pump/new_online/collie_all_spider_update_lake_kafka.yml
|
|
|
|
supervisor配置：(72个进程)
|
|
|
|
/home/collie/product/app_online_lake/supervisor/ic_spider_update_lake_kafka.conf
|
|
|
|
/etc/supervisord.d/ic_spider_update_lake_kafka.conf
|
|
|
|
```
|
|
|
|
|
|
|
|
| data_type | topic | group | num_procs |
|
| ... | ... | @@ -57,8 +59,18 @@ supervisor配置：(72个进程) |
|
|
|
/home/collie/product/app_online_lake/data_pump/new_online/collie_all_spider_update_lake_redis.yml
|
|
|
|
supervisor配置：(37个进程)
|
|
|
|
/home/collie/product/app_online_lake/supervisor/ic_spider_update_lake_redis.conf
|
|
|
|
/etc/supervisord.d/ic_spider_update_lake_redis.conf
|
|
|
|
redis:bdp-mq-001.redis.rds.aliyuncs.com
|
|
|
|
db:1
|
|
|
|
```
|
|
|
|
|
|
|
|
## 按省份分目录
|
|
|
|
```
|
|
|
|
部署地址：10.8.6.84
|
|
|
|
/home/collie/product/app_online_lake/data_pump/lake_province/ic_province_split.yml
|
|
|
|
supervisor配置：(37个进程)
|
|
|
|
/home/collie/product/app_online_lake/supervisor/ic_lake_update_province.conf
|
|
|
|
/etc/supervisord.d/ic_lake_update_province.conf
|
|
|
|
```
|
|
|
|
|
|
|
|
## |
|
|
\ No newline at end of file |
