Linux grep技巧 提取log中的json数据
一. 前提
1.1 数据准备
545-1 2024/07/20 18:20:21 [ERROR] MPX001 eventController=aupay transactionId=A545 {"event":"ERROR","auid":"kddi_XXXasd1","category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
545-2 2024/07/20 18:20:22 [INFO] MPX001 eventController=alipay transactionId=Bljk {"event":"INFO","auid":"kddi_XXXasd2","category":{"id":"111","name":"APPLE 2","amout":11,"price":13}}
545-3 2024/07/20 18:20:23 [ERROR] MPX001 eventController=paypay transactionId=Ijkhjk {"event":"ERROR","auid":"kddi_XXXasd3","category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}
545-4 2024/07/20 18:20:24 [INFO] MPX001 eventController=alipay transactionId=C9joj {"event":"INFO","auid":"kddi_XXXasd4","category":{"id":"113","name":"APPLE 4","amout":13,"price":15}}
1.2 需求
- 从log中提取出
[ERROR]相关的日志 - 然后再从日志中进一步提取
eventController，transactionId，JSON数据中的auid和name - 服务器中没有安装
jq库，无法通过此种方式来处理json数据。
1.3 分析
- ①先提取
[ERROR]相关的日志，过滤掉INFO的日志 - ②再提取出
eventController和transactionId字段 - ③然后可以先尝试提取出所有的json数据
- 使用
{.*}正则表达式
- 使用
- ④然后可以进一步缩小范围，提取
category相关的json数据- 使用
"category":{.*}正则表达式
- 使用
- ⑤然后可以进一步提取
auid和name所对应的值auid":"[^"]*"name":"[^"]*"
- ⑥最后再通过sed命令将各字段转置到一行上
sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'
二. 数据提取
2.1 提取所有的json数据
{.*}
grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e "{.*}" | \
sed ':loop; N; $!b loop; ;s/\n\([t{]\)/ \1/g'
⏹效果如下
fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e "{.*}" | \
> sed ':loop; N; $!b loop; ;s/\n\([t{]\)/ \1/g'
eventController=aupay transactionId=A545 {"event":"ERROR","auid":"kddi_XXXasd1","category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
eventController=paypay transactionId=Ijkhjk {"event":"ERROR","auid":"kddi_XXXasd3","category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}
2.2 提取子项目的全部json数据
"category":{.*}
grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e '"category":{.*}' | \
sed ':loop; N; $!b loop; ;s/\n\([t"]\)/ \1/g'
⏹效果如下
fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e '"category":{.*}' | \
> sed ':loop; N; $!b loop; ;s/\n\([t"]\)/ \1/g'
eventController=aupay transactionId=A545 "category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
eventController=paypay transactionId=Ijkhjk "category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}
2.3 提取指定项目的json数据
auid":"[^"]*"name":"[^"]*"[和]：定义一个字符类。^：在字符类中表示否定，意味着字符类匹配所有不在括号内的字符。"：字符类中唯一被否定的字符。*：表示前面的模式（即 [^"]）可以出现零次或多次。
因此，[^"]* 表示匹配由任意数量的非双引号字符组成的字符串，包括零个字符的情况。
grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e 'auid":"[^"]*"' -e 'name":"[^"]*"' | \
sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'
⏹效果如下
fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e 'auid":"[^"]*"' -e 'name":"[^"]*"' | \
> sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'
eventController=aupay transactionId=A545 auid":"kddi_XXXasd1" name":"APPLE 1"
eventController=paypay transactionId=Ijkhjk auid":"kddi_XXXasd3" name":"APPLE 3"
原文地址：https://blog.csdn.net/feyehong/article/details/140582225
免责声明：本站文章内容转载自网络资源，如本站内容侵犯了原著者的合法权益，可联系本站删除。更多内容请关注自学内容网（zxcms.com）！