自学内容网 自学内容网

Linux grep技巧 提取log中的json数据


一. 前提

1.1 数据准备

545-1 2024/07/20 18:20:21 [ERROR] MPX001 eventController=aupay transactionId=A545 {"event":"ERROR","auid":"kddi_XXXasd1","category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
545-2 2024/07/20 18:20:22 [INFO] MPX001 eventController=alipay transactionId=Bljk {"event":"INFO","auid":"kddi_XXXasd2","category":{"id":"111","name":"APPLE 2","amout":11,"price":13}}
545-3 2024/07/20 18:20:23 [ERROR] MPX001 eventController=paypay transactionId=Ijkhjk {"event":"ERROR","auid":"kddi_XXXasd3","category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}
545-4 2024/07/20 18:20:24 [INFO] MPX001 eventController=alipay transactionId=C9joj {"event":"INFO","auid":"kddi_XXXasd4","category":{"id":"113","name":"APPLE 4","amout":13,"price":15}}

1.2 需求

  • 从log中提取出[ERROR]相关的日志
  • 然后再从日志中进一步提取eventControllertransactionId,JSON数据中的auidname
  • 服务器中没有安装jq库,无法通过此种方式来处理json数据。

1.3 分析

  • ①先提取[ERROR]相关的日志,过滤掉INFO的日志
  • ②再提取出eventControllertransactionId字段
  • ③然后可以先尝试提取出所有的json数据
    • 使用{.*}正则表达式
  • ④然后可以进一步缩小范围,提取category相关的json数据
    • 使用"category":{.*}正则表达式
  • ⑤然后可以进一步提取auidname所对应的值
    • auid":"[^"]*"
    • name":"[^"]*"
  • ⑥最后再通过sed命令将各字段转置到一行上
    • sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'

二. 数据提取

2.1 提取所有的json数据

  • {.*}
grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e "{.*}" | \
sed ':loop; N; $!b loop; ;s/\n\([t{]\)/ \1/g'

⏹效果如下

fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e "{.*}" | \
> sed ':loop; N; $!b loop; ;s/\n\([t{]\)/ \1/g'

eventController=aupay transactionId=A545 {"event":"ERROR","auid":"kddi_XXXasd1","category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
eventController=paypay transactionId=Ijkhjk {"event":"ERROR","auid":"kddi_XXXasd3","category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}

2.2 提取子项目的全部json数据

  • "category":{.*}
grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e '"category":{.*}' | \
sed ':loop; N; $!b loop; ;s/\n\([t"]\)/ \1/g'

⏹效果如下

fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e '"category":{.*}' | \
> sed ':loop; N; $!b loop; ;s/\n\([t"]\)/ \1/g'

eventController=aupay transactionId=A545 "category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
eventController=paypay transactionId=Ijkhjk "category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}

2.3 提取指定项目的json数据

  • auid":"[^"]*"
  • name":"[^"]*"
    • []:定义一个字符类。
    • ^:在字符类中表示否定,意味着字符类匹配所有不在括号内的字符。
    • ":字符类中唯一被否定的字符。
    • *:表示前面的模式(即 [^"])可以出现零次或多次。

因此,[^"]* 表示匹配由任意数量的非双引号字符组成的字符串,包括零个字符的情况。

grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e 'auid":"[^"]*"' -e 'name":"[^"]*"' | \
sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'

⏹效果如下

fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e 'auid":"[^"]*"' -e 'name":"[^"]*"' | \
> sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'

eventController=aupay transactionId=A545 auid":"kddi_XXXasd1" name":"APPLE 1"
eventController=paypay transactionId=Ijkhjk auid":"kddi_XXXasd3" name":"APPLE 3"

原文地址:https://blog.csdn.net/feyehong/article/details/140582225

免责声明:本站文章内容转载自网络资源,如本站内容侵犯了原著者的合法权益,可联系本站删除。更多内容请关注自学内容网(zxcms.com)!