基于torch-pruning库对resnet18在cifar100数据集上进行剪枝实验

torch-pruning库是一个基于torch的剪枝库，其使用可以参考https://hpg123.blog.csdn.net/article/details/140421837。在此之外，该项目本身也提供了一些基础代码。本博文主要以benchmarks目录下的main.py代码为运行对象。对比了全局剪枝、局部剪枝、稀疏化训练的效果，训练目的是模型加速3倍。首先进行fineturn，训练的resnet18在cifar100的精度为0.7788，在全局剪枝下，group_norm、l1、l2、lamp、random剪枝方法中，以random剪枝效果最好，精度可达0.7530；局部剪枝下，所有方法的精度都比全局剪枝效果好，其中最佳的是l1剪枝方法，精度可达0.7604。

1、关键知识

在torch-pruning库中有不少知识没有在文档中介绍，需要基于官方的使用代码进行抽取。这里主要抽取benchmarks目录下main.py中关键代码进行说明。

1.1 按加速比进行模型剪枝

torch-pruning库只提供了按通道数量剪枝+设定剪枝步骤的接口。
基于progressive_pruning函数可以将模型剪枝到指定加速比例的状态。其核心是将剪枝step数设置的特别大，例如200，将剪枝ratio设置为1，每执行一次pruner.step()，模型减少0.005的channel，然后检验flop优化速度是否达到预期值，若未达到则进行剪枝。若达到，则终止while训练。

iterative_steps=200
pruner = pruner_entry(
        model,
        example_inputs,
        importance=imp,
        iterative_steps=iterative_steps,
        pruning_ratio=1.0,
        pruning_ratio_dict=pruning_ratio_dict,
        max_pruning_ratio=args.max_pruning_ratio,
        ignored_layers=ignored_layers,
        unwrapped_parameters=unwrapped_parameters,
    )
def progressive_pruning(pruner, model, speed_up, example_inputs, train_loader=None):
    model.eval()
    base_ops, _ = tp.utils.count_ops_and_params(model, example_inputs=example_inputs)
    current_speed_up = 1
    while current_speed_up < speed_up:
        if args.method == "obdc":
            model.zero_grad()
            imp=pruner.importance
            imp._prepare_model(model, pruner)
            for k, (imgs, lbls) in enumerate(train_loader):
                if k>=10: break
                imgs = imgs.to(args.device)
                lbls = lbls.to(args.device)
                output = model(imgs)
                sampled_y = torch.multinomial(torch.nn.functional.softmax(output.cpu().data, dim=1),
                                                  1).squeeze().to(args.device)
                loss_sample = F.cross_entropy(output, sampled_y)
                loss_sample.backward()
                imp.step()
            pruner.step()
            imp._rm_hooks(model)
            imp._clear_buffer()
        else:
            pruner.step()
        pruned_ops, _ = tp.utils.count_ops_and_params(model, example_inputs=example_inputs)
        current_speed_up = float(base_ops) / pruned_ops
        if pruner.current_step == pruner.iterative_steps:
            break
    return current_speed_up

1.2 剪枝评价器与稀疏训练的关系

在torch-pruning库的文档里写到了稀疏训练，但没有介绍使用哪些评价器需要进行稀疏训练。通过对get_pruner函数进行分析可以看到，只有BNScaleImportance与GroupNormImportance在特定参数情况下才将 sparsity_learning设置为ture。

def get_pruner(model, example_inputs):
    args.sparsity_learning = False
    if args.method == "random":
        imp = tp.importance.RandomImportance()
        pruner_entry = partial(tp.pruner.MagnitudePruner, global_pruning=args.global_pruning)
    elif args.method == "l1":
        imp = tp.importance.MagnitudeImportance(p=1)
        pruner_entry = partial(tp.pruner.MagnitudePruner, global_pruning=args.global_pruning)
    elif args.method == "l2":
        imp = tp.importance.MagnitudeImportance(p=2)
        pruner_entry = partial(tp.pruner.MagnitudePruner, global_pruning=args.global_pruning)
    elif args.method == "fpgm":
        imp = tp.importance.FPGMImportance(p=2)
        pruner_entry = partial(tp.pruner.MagnitudePruner, global_pruning=args.global_pruning)
    elif args.method == "obdc":
        imp = tp.importance.OBDCImportance(group_reduction='mean', num_classes=args.num_classes)
        pruner_entry = partial(tp.pruner.MagnitudePruner, global_pruning=args.global_pruning)
    elif args.method == "lamp":
        imp = tp.importance.LAMPImportance(p=2)
        pruner_entry = partial(tp.pruner.MagnitudePruner, global_pruning=args.global_pruning)
    elif args.method == "slim":
        args.sparsity_learning = True
        imp = tp.importance.BNScaleImportance()
        pruner_entry = partial(tp.pruner.BNScalePruner, reg=args.reg, global_pruning=args.global_pruning)
    elif args.method == "group_slim":
        args.sparsity_learning = True
        imp = tp.importance.BNScaleImportance()
        pruner_entry = partial(tp.pruner.BNScalePruner, reg=args.reg, global_pruning=args.global_pruning, group_lasso=True)
    elif args.method == "group_norm":
        imp = tp.importance.GroupNormImportance(p=2)
        pruner_entry = partial(tp.pruner.GroupNormPruner, global_pruning=args.global_pruning)
    elif args.method == "group_sl":
        args.sparsity_learning = True
        imp = tp.importance.GroupNormImportance(p=2, normalizer='max') # normalized by the maximum score for CIFAR
        pruner_entry = partial(tp.pruner.GroupNormPruner, reg=args.reg, global_pruning=args.global_pruning)
    elif args.method == "growing_reg":
        args.sparsity_learning = True
        imp = tp.importance.GroupNormImportance(p=2)
        pruner_entry = partial(tp.pruner.GrowingRegPruner, reg=args.reg, delta_reg=args.delta_reg, global_pruning=args.global_pruning)
    else:
        raise NotImplementedError
    
    #args.is_accum_importance = is_accum_importance
    unwrapped_parameters = []
    ignored_layers = []
    pruning_ratio_dict = {}
    # ignore output layers
    for m in model.modules():
        if isinstance(m, torch.nn.Linear) and m.out_features == args.num_classes:
            ignored_layers.append(m)
        elif isinstance(m, torch.nn.modules.conv._ConvNd) and m.out_channels == args.num_classes:
            ignored_layers.append(m)
    
    # Here we fix iterative_steps=200 to prune the model progressively with small steps 
    # until the required speed up is achieved.
    pruner = pruner_entry(
        model,
        example_inputs,
        importance=imp,
        iterative_steps=args.iterative_steps,
        pruning_ratio=1.0,
        pruning_ratio_dict=pruning_ratio_dict,
        max_pruning_ratio=args.max_pruning_ratio,
        ignored_layers=ignored_layers,
        unwrapped_parameters=unwrapped_parameters,
    )
    return pruner

1.3 稀疏训练的用法

通过对main.py代码的分析，可以发现是稀疏训练是属于剪枝训练的一部分，进行稀疏训练的目的是为了使部分channel对应的kernel为0，以便于后续剪枝时选择出不重要的参数。

        pruner = get_pruner(model, example_inputs=example_inputs)
        # 0. Sparsity Learning
        if args.sparsity_learning:
            reg_pth = "reg_{}_{}_{}_{}.pth".format(args.dataset, args.model, args.method, args.reg)
            reg_pth = os.path.join( os.path.join(args.output_dir, reg_pth) )
            if not args.sl_restore:
                args.logger.info("Regularizing...")
                train_model(
                    model,
                    train_loader=train_loader,
                    test_loader=test_loader,
                    epochs=args.sl_total_epochs,
                    lr=args.sl_lr,
                    lr_decay_milestones=args.sl_lr_decay_milestones,
                    lr_decay_gamma=args.lr_decay_gamma,
                    pruner=pruner,
                    save_state_dict_only=True,
                    save_as = reg_pth,
                )
            args.logger.info("Loading the sparse model from {}...".format(reg_pth))
            model.load_state_dict( torch.load( reg_pth, map_location=args.device) )
        
        # 1. Pruning
        model.eval()
        ori_ops, ori_size = tp.utils.count_ops_and_params(model, example_inputs=example_inputs)
        ori_acc, ori_val_loss = eval(model, test_loader, device=args.device)
        args.logger.info("Pruning...")
        progressive_pruning(pruner, model, speed_up=args.speed_up, example_inputs=example_inputs, train_loader=train_loader)

在train_model函数中可以发现稀疏剪枝的用法，使用sgd优化器，但weight_decay被设置为0，该步骤应该是非必要的。

    optimizer = torch.optim.SGD(
        model.parameters(),
        lr=lr,
        momentum=0.9,
        weight_decay=weight_decay if pruner is None else 0,
    )

然后最关键的部分是在每一个iter的loss.backward()与optimizer.step()中间插入pruner.regularize(model)

    for epoch in range(epochs):
        model.train()
    
        for i, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
            optimizer.zero_grad()
            out = model(data)
            loss = F.cross_entropy(out, target)
            loss.backward()
            if pruner is not None:
                pruner.regularize(model) # for sparsity learning
            optimizer.step()
            if i % 10 == 0 and args.verbose:
                args.logger.info(
                    "Epoch {:d}/{:d}, iter {:d}/{:d}, loss={:.4f}, lr={:.4f}".format(
                        epoch,
                        epochs,
                        i,
                        len(train_loader),
                        loss.item(),
                        optimizer.param_groups[0]["lr"],
                    )
                )

        if pruner is not None and isinstance(pruner, tp.pruner.GrowingRegPruner):
            pruner.update_reg() # increase the strength of regularization
            #print(pruner.group_reg[pruner._groups[0]])

2、剪枝前信息

2.1 命令行参数

通过对代码进行分析，可以看到main.py支持对[‘cifar10’, ‘cifar100’, ‘modelnet40’]三个数据进行训练，支持[“pretrain”, “prune”, “test”]三种模式。其中有基础参数与剪枝参数，最关键的是finetune参数，只有设置finetune才可以在模型剪枝后进行二次训练。

# Basic options
parser.add_argument("--mode", type=str, required=True, choices=["pretrain", "prune", "test"])
parser.add_argument("--model", type=str, required=True)
parser.add_argument("--verbose", action="store_true", default=False)
parser.add_argument("--dataset", type=str, default="cifar100", choices=['cifar10', 'cifar100', 'modelnet40'])
parser.add_argument('--dataroot', default='data', help='path to your datasets')
parser.add_argument("--batch-size", type=int, default=128)
parser.add_argument("--total-epochs", type=int, default=100)
parser.add_argument("--lr-decay-milestones", default="60,80", type=str, help="milestones for learning rate decay")
parser.add_argument("--lr-decay-gamma", default=0.1, type=float)
parser.add_argument("--lr", default=0.01, type=float, help="learning rate")
parser.add_argument("--restore", type=str, default=None)
parser.add_argument('--output-dir', default='run', help='path where to save')
parser.add_argument("--finetune", action="store_true", default=False, help='whether finetune or not')

# For pruning
parser.add_argument("--method", type=str, default=None)
parser.add_argument("--speed-up", type=float, default=2)
parser.add_argument("--max-pruning-ratio", type=float, default=1.0)
parser.add_argument("--soft-keeping-ratio", type=float, default=0.0)
parser.add_argument("--reg", type=float, default=5e-4)
parser.add_argument("--delta_reg", type=float, default=1e-4, help='for growing regularization')
parser.add_argument("--weight-decay", type=float, default=5e-4)

parser.add_argument("--seed", type=int, default=None)
parser.add_argument("--global-pruning", action="store_true", default=False)
parser.add_argument("--sl-total-epochs", type=int, default=100, help="epochs for sparsity learning")
parser.add_argument("--sl-lr", default=0.01, type=float, help="learning rate for sparsity learning")
parser.add_argument("--sl-lr-decay-milestones", default="60,80", type=str, help="milestones for sparsity learning")
parser.add_argument("--sl-reg-warmup", type=int, default=0, help="epochs for sparsity learning")
parser.add_argument("--sl-restore", type=str, default=None)
parser.add_argument("--iterative-steps", default=400, type=int)

2.2 基础训练

以下命令是基于resnet18 模型在cifar100 数据集上进行finetune

python main.py --mode pretrain --dataset cifar100 --model resnet18 --lr 0.1 --total-epochs 200 --lr-decay-milestones 120,150,180 

从相关代码中可以看出，基础训练是是加载了模型在imagenet上的预训练权重的。

model = registry.get_model(args.model, num_classes=num_classes, %pretrained=True, target_dataset=args.dataset)

令执行后的输出如下所示，可以看到训练后的模型精度达到了77.88%，明显比相关剪枝论文中的其他更深resnet模型要高2~4个点。很有可能他们使用的模型是没有进行预训练的

[07/19 13:17:39] cifar100-resnet18 INFO: mode: pretrain
[07/19 13:17:39] cifar100-resnet18 INFO: model: resnet18
[07/19 13:17:39] cifar100-resnet18 INFO: verbose: False
[07/19 13:17:39] cifar100-resnet18 INFO: dataset: cifar100
[07/19 13:17:39] cifar100-resnet18 INFO: dataroot: data
[07/19 13:17:39] cifar100-resnet18 INFO: batch_size: 128
[07/19 13:17:39] cifar100-resnet18 INFO: total_epochs: 200
[07/19 13:17:39] cifar100-resnet18 INFO: lr_decay_milestones: 120,150,180
[07/19 13:17:39] cifar100-resnet18 INFO: lr_decay_gamma: 0.1
[07/19 13:17:39] cifar100-resnet18 INFO: lr: 0.1
[07/19 13:17:39] cifar100-resnet18 INFO: restore: None
[07/19 13:17:39] cifar100-resnet18 INFO: output_dir: run\cifar100\pretrain
[07/19 13:17:39] cifar100-resnet18 INFO: finetune: False
[07/19 13:17:39] cifar100-resnet18 INFO: method: None
[07/19 13:17:39] cifar100-resnet18 INFO: speed_up: 2
[07/19 13:17:39] cifar100-resnet18 INFO: max_pruning_ratio: 1.0
[07/19 13:17:39] cifar100-resnet18 INFO: soft_keeping_ratio: 0.0
[07/19 13:17:39] cifar100-resnet18 INFO: reg: 0.0005
[07/19 13:17:39] cifar100-resnet18 INFO: delta_reg: 0.0001
[07/19 13:17:39] cifar100-resnet18 INFO: weight_decay: 0.0005
[07/19 13:17:39] cifar100-resnet18 INFO: seed: None
[07/19 13:17:39] cifar100-resnet18 INFO: global_pruning: False
[07/19 13:17:39] cifar100-resnet18 INFO: sl_total_epochs: 100
[07/19 13:17:39] cifar100-resnet18 INFO: sl_lr: 0.01
[07/19 13:17:39] cifar100-resnet18 INFO: sl_lr_decay_milestones: 60,80
[07/19 13:17:39] cifar100-resnet18 INFO: sl_reg_warmup: 0
[07/19 13:17:39] cifar100-resnet18 INFO: sl_restore: None
[07/19 13:17:39] cifar100-resnet18 INFO: iterative_steps: 400
[07/19 13:17:39] cifar100-resnet18 INFO: logger: <Logger cifar100-resnet18 (DEBUG)>
[07/19 13:17:39] cifar100-resnet18 INFO: device: cuda
[07/19 13:17:39] cifar100-resnet18 INFO: num_classes: 100
[07/19 13:18:07] cifar100-resnet18 INFO: Params: 11.22 M
[07/19 13:18:07] cifar100-resnet18 INFO: ops: 556.70 M
[07/19 13:19:12] cifar100-resnet18 INFO: Epoch 0/200, Acc=0.1337, Val Loss=3.6320, lr=0.1000
[07/19 13:20:08] cifar100-resnet18 INFO: Epoch 1/200, Acc=0.1978, Val Loss=3.2920, lr=0.1000
[07/19 13:21:03] cifar100-resnet18 INFO: Epoch 2/200, Acc=0.2845, Val Loss=2.7975, lr=0.1000
[07/19 13:22:07] cifar100-resnet18 INFO: Epoch 3/200, Acc=0.3560, Val Loss=2.4438, lr=0.1000
[07/19 13:23:02] cifar100-resnet18 INFO: Epoch 4/200, Acc=0.4420, Val Loss=2.0798, lr=0.1000
[07/19 13:23:56] cifar100-resnet18 INFO: Epoch 5/200, Acc=0.4567, Val Loss=2.0327, lr=0.1000
[07/19 13:24:50] cifar100-resnet18 INFO: Epoch 6/200, Acc=0.4865, Val Loss=1.9149, lr=0.1000
[07/19 13:25:43] cifar100-resnet18 INFO: Epoch 7/200, Acc=0.4918, Val Loss=1.8947, lr=0.1000
[07/19 13:26:36] cifar100-resnet18 INFO: Epoch 8/200, Acc=0.4963, Val Loss=1.8719, lr=0.1000
[07/19 13:27:29] cifar100-resnet18 INFO: Epoch 9/200, Acc=0.4924, Val Loss=2.0017, lr=0.1000
[07/19 13:28:22] cifar100-resnet18 INFO: Epoch 10/200, Acc=0.4955, Val Loss=1.9902, lr=0.1000
[07/19 13:29:16] cifar100-resnet18 INFO: Epoch 11/200, Acc=0.5408, Val Loss=1.6740, lr=0.1000
[07/19 13:30:09] cifar100-resnet18 INFO: Epoch 12/200, Acc=0.5514, Val Loss=1.6730, lr=0.1000
[07/19 13:31:03] cifar100-resnet18 INFO: Epoch 13/200, Acc=0.5330, Val Loss=1.7489, lr=0.1000
[07/19 13:31:56] cifar100-resnet18 INFO: Epoch 14/200, Acc=0.5632, Val Loss=1.6066, lr=0.1000
[07/19 13:32:49] cifar100-resnet18 INFO: Epoch 15/200, Acc=0.5268, Val Loss=1.8168, lr=0.1000
[07/19 13:33:42] cifar100-resnet18 INFO: Epoch 16/200, Acc=0.4921, Val Loss=2.0397, lr=0.1000
[07/19 13:34:35] cifar100-resnet18 INFO: Epoch 17/200, Acc=0.5124, Val Loss=1.8411, lr=0.1000
[07/19 13:35:28] cifar100-resnet18 INFO: Epoch 18/200, Acc=0.5377, Val Loss=1.8048, lr=0.1000
[07/19 13:36:21] cifar100-resnet18 INFO: Epoch 19/200, Acc=0.5568, Val Loss=1.6569, lr=0.1000
[07/19 13:37:14] cifar100-resnet18 INFO: Epoch 20/200, Acc=0.5819, Val Loss=1.5656, lr=0.1000
[07/19 13:38:07] cifar100-resnet18 INFO: Epoch 21/200, Acc=0.5501, Val Loss=1.6694, lr=0.1000
[07/19 13:39:00] cifar100-resnet18 INFO: Epoch 22/200, Acc=0.5678, Val Loss=1.6209, lr=0.1000
[07/19 13:39:53] cifar100-resnet18 INFO: Epoch 23/200, Acc=0.5515, Val Loss=1.7096, lr=0.1000
[07/19 13:40:47] cifar100-resnet18 INFO: Epoch 24/200, Acc=0.5583, Val Loss=1.6427, lr=0.1000
[07/19 13:41:40] cifar100-resnet18 INFO: Epoch 25/200, Acc=0.5716, Val Loss=1.6022, lr=0.1000
[07/19 13:42:33] cifar100-resnet18 INFO: Epoch 26/200, Acc=0.5715, Val Loss=1.6044, lr=0.1000
[07/19 13:43:26] cifar100-resnet18 INFO: Epoch 27/200, Acc=0.5486, Val Loss=1.8042, lr=0.1000
[07/19 13:44:19] cifar100-resnet18 INFO: Epoch 28/200, Acc=0.5877, Val Loss=1.5403, lr=0.1000
[07/19 13:45:12] cifar100-resnet18 INFO: Epoch 29/200, Acc=0.6059, Val Loss=1.4532, lr=0.1000
[07/19 13:46:05] cifar100-resnet18 INFO: Epoch 30/200, Acc=0.4891, Val Loss=2.0751, lr=0.1000
[07/19 13:46:58] cifar100-resnet18 INFO: Epoch 31/200, Acc=0.5607, Val Loss=1.6768, lr=0.1000
[07/19 13:47:52] cifar100-resnet18 INFO: Epoch 32/200, Acc=0.5620, Val Loss=1.6878, lr=0.1000
[07/19 13:48:45] cifar100-resnet18 INFO: Epoch 33/200, Acc=0.5602, Val Loss=1.6448, lr=0.1000
[07/19 13:49:38] cifar100-resnet18 INFO: Epoch 34/200, Acc=0.5919, Val Loss=1.5419, lr=0.1000
[07/19 13:50:31] cifar100-resnet18 INFO: Epoch 35/200, Acc=0.5663, Val Loss=1.6725, lr=0.1000
[07/19 13:51:24] cifar100-resnet18 INFO: Epoch 36/200, Acc=0.5538, Val Loss=1.7597, lr=0.1000
[07/19 13:52:18] cifar100-resnet18 INFO: Epoch 37/200, Acc=0.5823, Val Loss=1.5669, lr=0.1000
[07/19 13:53:10] cifar100-resnet18 INFO: Epoch 38/200, Acc=0.5545, Val Loss=1.7044, lr=0.1000
[07/19 13:54:04] cifar100-resnet18 INFO: Epoch 39/200, Acc=0.5600, Val Loss=1.7275, lr=0.1000
[07/19 13:54:57] cifar100-resnet18 INFO: Epoch 40/200, Acc=0.5373, Val Loss=1.7365, lr=0.1000
[07/19 13:55:50] cifar100-resnet18 INFO: Epoch 41/200, Acc=0.5482, Val Loss=1.7472, lr=0.1000
[07/19 13:56:43] cifar100-resnet18 INFO: Epoch 42/200, Acc=0.5849, Val Loss=1.6076, lr=0.1000
[07/19 13:57:36] cifar100-resnet18 INFO: Epoch 43/200, Acc=0.5817, Val Loss=1.6058, lr=0.1000
[07/19 13:58:29] cifar100-resnet18 INFO: Epoch 44/200, Acc=0.5781, Val Loss=1.6527, lr=0.1000
[07/19 13:59:22] cifar100-resnet18 INFO: Epoch 45/200, Acc=0.5926, Val Loss=1.5427, lr=0.1000
[07/19 14:00:15] cifar100-resnet18 INFO: Epoch 46/200, Acc=0.5474, Val Loss=1.7240, lr=0.1000
[07/19 14:01:09] cifar100-resnet18 INFO: Epoch 47/200, Acc=0.5714, Val Loss=1.6210, lr=0.1000
[07/19 14:02:02] cifar100-resnet18 INFO: Epoch 48/200, Acc=0.5696, Val Loss=1.6508, lr=0.1000
[07/19 14:02:55] cifar100-resnet18 INFO: Epoch 49/200, Acc=0.5818, Val Loss=1.6023, lr=0.1000
[07/19 14:03:48] cifar100-resnet18 INFO: Epoch 50/200, Acc=0.6086, Val Loss=1.4458, lr=0.1000
[07/19 14:04:41] cifar100-resnet18 INFO: Epoch 51/200, Acc=0.5992, Val Loss=1.5147, lr=0.1000
[07/19 14:05:34] cifar100-resnet18 INFO: Epoch 52/200, Acc=0.6055, Val Loss=1.4631, lr=0.1000
[07/19 14:06:28] cifar100-resnet18 INFO: Epoch 53/200, Acc=0.6013, Val Loss=1.5639, lr=0.1000
[07/19 14:07:21] cifar100-resnet18 INFO: Epoch 54/200, Acc=0.5817, Val Loss=1.5981, lr=0.1000
[07/19 14:08:14] cifar100-resnet18 INFO: Epoch 55/200, Acc=0.5864, Val Loss=1.6201, lr=0.1000
[07/19 14:09:07] cifar100-resnet18 INFO: Epoch 56/200, Acc=0.5787, Val Loss=1.6410, lr=0.1000
[07/19 14:10:00] cifar100-resnet18 INFO: Epoch 57/200, Acc=0.5782, Val Loss=1.5940, lr=0.1000
[07/19 14:10:53] cifar100-resnet18 INFO: Epoch 58/200, Acc=0.5766, Val Loss=1.5852, lr=0.1000
[07/19 14:11:46] cifar100-resnet18 INFO: Epoch 59/200, Acc=0.5866, Val Loss=1.5699, lr=0.1000
[07/19 14:12:39] cifar100-resnet18 INFO: Epoch 60/200, Acc=0.5987, Val Loss=1.5052, lr=0.1000
[07/19 14:13:33] cifar100-resnet18 INFO: Epoch 61/200, Acc=0.5785, Val Loss=1.5786, lr=0.1000
[07/19 14:14:26] cifar100-resnet18 INFO: Epoch 62/200, Acc=0.5674, Val Loss=1.6427, lr=0.1000
[07/19 14:15:19] cifar100-resnet18 INFO: Epoch 63/200, Acc=0.5952, Val Loss=1.5210, lr=0.1000
[07/19 14:16:12] cifar100-resnet18 INFO: Epoch 64/200, Acc=0.5622, Val Loss=1.6683, lr=0.1000
[07/19 14:17:05] cifar100-resnet18 INFO: Epoch 65/200, Acc=0.6139, Val Loss=1.4194, lr=0.1000
[07/19 14:17:58] cifar100-resnet18 INFO: Epoch 66/200, Acc=0.5861, Val Loss=1.5526, lr=0.1000
[07/19 14:18:51] cifar100-resnet18 INFO: Epoch 67/200, Acc=0.5486, Val Loss=1.7676, lr=0.1000
[07/19 14:19:44] cifar100-resnet18 INFO: Epoch 68/200, Acc=0.5140, Val Loss=1.9901, lr=0.1000
[07/19 14:20:37] cifar100-resnet18 INFO: Epoch 69/200, Acc=0.5799, Val Loss=1.6832, lr=0.1000
[07/19 14:21:30] cifar100-resnet18 INFO: Epoch 70/200, Acc=0.6032, Val Loss=1.4905, lr=0.1000
[07/19 14:22:23] cifar100-resnet18 INFO: Epoch 71/200, Acc=0.5923, Val Loss=1.4987, lr=0.1000
[07/19 14:23:16] cifar100-resnet18 INFO: Epoch 72/200, Acc=0.6084, Val Loss=1.4722, lr=0.1000
[07/19 14:24:09] cifar100-resnet18 INFO: Epoch 73/200, Acc=0.5728, Val Loss=1.6187, lr=0.1000
[07/19 14:25:02] cifar100-resnet18 INFO: Epoch 74/200, Acc=0.5615, Val Loss=1.7310, lr=0.1000
[07/19 14:25:55] cifar100-resnet18 INFO: Epoch 75/200, Acc=0.5863, Val Loss=1.6054, lr=0.1000
[07/19 14:26:48] cifar100-resnet18 INFO: Epoch 76/200, Acc=0.5820, Val Loss=1.6254, lr=0.1000
[07/19 14:27:41] cifar100-resnet18 INFO: Epoch 77/200, Acc=0.5897, Val Loss=1.5598, lr=0.1000
[07/19 14:28:34] cifar100-resnet18 INFO: Epoch 78/200, Acc=0.5764, Val Loss=1.6959, lr=0.1000
[07/19 14:29:28] cifar100-resnet18 INFO: Epoch 79/200, Acc=0.5809, Val Loss=1.6789, lr=0.1000
[07/19 14:30:21] cifar100-resnet18 INFO: Epoch 80/200, Acc=0.5940, Val Loss=1.5114, lr=0.1000
[07/19 14:31:14] cifar100-resnet18 INFO: Epoch 81/200, Acc=0.5828, Val Loss=1.6427, lr=0.1000
[07/19 14:32:07] cifar100-resnet18 INFO: Epoch 82/200, Acc=0.5820, Val Loss=1.6309, lr=0.1000
[07/19 14:33:00] cifar100-resnet18 INFO: Epoch 83/200, Acc=0.5965, Val Loss=1.5206, lr=0.1000
[07/19 14:33:53] cifar100-resnet18 INFO: Epoch 84/200, Acc=0.5991, Val Loss=1.4937, lr=0.1000
[07/19 14:34:46] cifar100-resnet18 INFO: Epoch 85/200, Acc=0.5952, Val Loss=1.5800, lr=0.1000
[07/19 14:35:39] cifar100-resnet18 INFO: Epoch 86/200, Acc=0.5908, Val Loss=1.5794, lr=0.1000
[07/19 14:36:32] cifar100-resnet18 INFO: Epoch 87/200, Acc=0.5892, Val Loss=1.5879, lr=0.1000
[07/19 14:37:25] cifar100-resnet18 INFO: Epoch 88/200, Acc=0.5593, Val Loss=1.7714, lr=0.1000
[07/19 14:38:19] cifar100-resnet18 INFO: Epoch 89/200, Acc=0.5796, Val Loss=1.6677, lr=0.1000
[07/19 14:39:12] cifar100-resnet18 INFO: Epoch 90/200, Acc=0.6030, Val Loss=1.5269, lr=0.1000
[07/19 14:40:05] cifar100-resnet18 INFO: Epoch 91/200, Acc=0.5534, Val Loss=1.8252, lr=0.1000
[07/19 14:40:58] cifar100-resnet18 INFO: Epoch 92/200, Acc=0.5590, Val Loss=1.6851, lr=0.1000
[07/19 14:41:51] cifar100-resnet18 INFO: Epoch 93/200, Acc=0.5736, Val Loss=1.7152, lr=0.1000
[07/19 14:42:44] cifar100-resnet18 INFO: Epoch 94/200, Acc=0.5661, Val Loss=1.7252, lr=0.1000
[07/19 14:43:37] cifar100-resnet18 INFO: Epoch 95/200, Acc=0.5867, Val Loss=1.5912, lr=0.1000
[07/19 14:44:31] cifar100-resnet18 INFO: Epoch 96/200, Acc=0.5942, Val Loss=1.5521, lr=0.1000
[07/19 14:45:24] cifar100-resnet18 INFO: Epoch 97/200, Acc=0.5882, Val Loss=1.5931, lr=0.1000
[07/19 14:46:17] cifar100-resnet18 INFO: Epoch 98/200, Acc=0.5900, Val Loss=1.5818, lr=0.1000
[07/19 14:47:10] cifar100-resnet18 INFO: Epoch 99/200, Acc=0.5748, Val Loss=1.6500, lr=0.1000
[07/19 14:48:03] cifar100-resnet18 INFO: Epoch 100/200, Acc=0.5521, Val Loss=1.7498, lr=0.1000
[07/19 14:48:56] cifar100-resnet18 INFO: Epoch 101/200, Acc=0.5702, Val Loss=1.7116, lr=0.1000
[07/19 14:49:49] cifar100-resnet18 INFO: Epoch 102/200, Acc=0.5616, Val Loss=1.7329, lr=0.1000
[07/19 14:50:42] cifar100-resnet18 INFO: Epoch 103/200, Acc=0.5805, Val Loss=1.6120, lr=0.1000
[07/19 14:51:35] cifar100-resnet18 INFO: Epoch 104/200, Acc=0.5783, Val Loss=1.6734, lr=0.1000
[07/19 14:52:28] cifar100-resnet18 INFO: Epoch 105/200, Acc=0.5886, Val Loss=1.5711, lr=0.1000
[07/19 14:53:21] cifar100-resnet18 INFO: Epoch 106/200, Acc=0.5894, Val Loss=1.5403, lr=0.1000
[07/19 14:54:14] cifar100-resnet18 INFO: Epoch 107/200, Acc=0.5889, Val Loss=1.5727, lr=0.1000
[07/19 14:55:07] cifar100-resnet18 INFO: Epoch 108/200, Acc=0.6010, Val Loss=1.5296, lr=0.1000
[07/19 14:56:00] cifar100-resnet18 INFO: Epoch 109/200, Acc=0.5281, Val Loss=1.9403, lr=0.1000
[07/19 14:56:53] cifar100-resnet18 INFO: Epoch 110/200, Acc=0.5691, Val Loss=1.6697, lr=0.1000
[07/19 14:57:46] cifar100-resnet18 INFO: Epoch 111/200, Acc=0.5903, Val Loss=1.5517, lr=0.1000
[07/19 14:58:39] cifar100-resnet18 INFO: Epoch 112/200, Acc=0.5606, Val Loss=1.7267, lr=0.1000
[07/19 14:59:32] cifar100-resnet18 INFO: Epoch 113/200, Acc=0.5819, Val Loss=1.6348, lr=0.1000
[07/19 15:00:25] cifar100-resnet18 INFO: Epoch 114/200, Acc=0.5780, Val Loss=1.6177, lr=0.1000
[07/19 15:01:18] cifar100-resnet18 INFO: Epoch 115/200, Acc=0.5946, Val Loss=1.5640, lr=0.1000
[07/19 15:02:11] cifar100-resnet18 INFO: Epoch 116/200, Acc=0.5755, Val Loss=1.6689, lr=0.1000
[07/19 15:03:05] cifar100-resnet18 INFO: Epoch 117/200, Acc=0.5847, Val Loss=1.6159, lr=0.1000
[07/19 15:03:58] cifar100-resnet18 INFO: Epoch 118/200, Acc=0.5687, Val Loss=1.7235, lr=0.1000
[07/19 15:04:51] cifar100-resnet18 INFO: Epoch 119/200, Acc=0.6083, Val Loss=1.5001, lr=0.1000
[07/19 15:05:44] cifar100-resnet18 INFO: Epoch 120/200, Acc=0.7436, Val Loss=0.9004, lr=0.0100
[07/19 15:06:37] cifar100-resnet18 INFO: Epoch 121/200, Acc=0.7530, Val Loss=0.8757, lr=0.0100
[07/19 15:07:30] cifar100-resnet18 INFO: Epoch 122/200, Acc=0.7557, Val Loss=0.8734, lr=0.0100
[07/19 15:08:24] cifar100-resnet18 INFO: Epoch 123/200, Acc=0.7590, Val Loss=0.8795, lr=0.0100
[07/19 15:09:17] cifar100-resnet18 INFO: Epoch 124/200, Acc=0.7641, Val Loss=0.8816, lr=0.0100
[07/19 15:10:10] cifar100-resnet18 INFO: Epoch 125/200, Acc=0.7615, Val Loss=0.8882, lr=0.0100
[07/19 15:11:03] cifar100-resnet18 INFO: Epoch 126/200, Acc=0.7596, Val Loss=0.9029, lr=0.0100
[07/19 15:11:56] cifar100-resnet18 INFO: Epoch 127/200, Acc=0.7611, Val Loss=0.9056, lr=0.0100
[07/19 15:12:49] cifar100-resnet18 INFO: Epoch 128/200, Acc=0.7619, Val Loss=0.9054, lr=0.0100
[07/19 15:13:42] cifar100-resnet18 INFO: Epoch 129/200, Acc=0.7648, Val Loss=0.9136, lr=0.0100
[07/19 15:14:36] cifar100-resnet18 INFO: Epoch 130/200, Acc=0.7587, Val Loss=0.9422, lr=0.0100
[07/19 15:15:29] cifar100-resnet18 INFO: Epoch 131/200, Acc=0.7609, Val Loss=0.9416, lr=0.0100
[07/19 15:16:22] cifar100-resnet18 INFO: Epoch 132/200, Acc=0.7625, Val Loss=0.9530, lr=0.0100
[07/19 15:17:15] cifar100-resnet18 INFO: Epoch 133/200, Acc=0.7589, Val Loss=0.9444, lr=0.0100
[07/19 15:18:08] cifar100-resnet18 INFO: Epoch 134/200, Acc=0.7601, Val Loss=0.9602, lr=0.0100
[07/19 15:19:01] cifar100-resnet18 INFO: Epoch 135/200, Acc=0.7604, Val Loss=0.9652, lr=0.0100
[07/19 15:19:54] cifar100-resnet18 INFO: Epoch 136/200, Acc=0.7574, Val Loss=0.9684, lr=0.0100
[07/19 15:20:47] cifar100-resnet18 INFO: Epoch 137/200, Acc=0.7595, Val Loss=0.9691, lr=0.0100
[07/19 15:21:40] cifar100-resnet18 INFO: Epoch 138/200, Acc=0.7598, Val Loss=0.9702, lr=0.0100
[07/19 15:22:33] cifar100-resnet18 INFO: Epoch 139/200, Acc=0.7608, Val Loss=0.9752, lr=0.0100
[07/19 15:23:27] cifar100-resnet18 INFO: Epoch 140/200, Acc=0.7604, Val Loss=0.9751, lr=0.0100
[07/19 15:24:20] cifar100-resnet18 INFO: Epoch 141/200, Acc=0.7602, Val Loss=0.9895, lr=0.0100
[07/19 15:25:13] cifar100-resnet18 INFO: Epoch 142/200, Acc=0.7619, Val Loss=0.9738, lr=0.0100
[07/19 15:26:06] cifar100-resnet18 INFO: Epoch 143/200, Acc=0.7593, Val Loss=0.9943, lr=0.0100
[07/19 15:26:59] cifar100-resnet18 INFO: Epoch 144/200, Acc=0.7587, Val Loss=0.9876, lr=0.0100
[07/19 15:27:52] cifar100-resnet18 INFO: Epoch 145/200, Acc=0.7576, Val Loss=0.9939, lr=0.0100
[07/19 15:28:45] cifar100-resnet18 INFO: Epoch 146/200, Acc=0.7587, Val Loss=0.9855, lr=0.0100
[07/19 15:29:38] cifar100-resnet18 INFO: Epoch 147/200, Acc=0.7605, Val Loss=0.9873, lr=0.0100
[07/19 15:30:31] cifar100-resnet18 INFO: Epoch 148/200, Acc=0.7608, Val Loss=0.9878, lr=0.0100
[07/19 15:31:24] cifar100-resnet18 INFO: Epoch 149/200, Acc=0.7546, Val Loss=0.9936, lr=0.0100
[07/19 15:32:17] cifar100-resnet18 INFO: Epoch 150/200, Acc=0.7664, Val Loss=0.9508, lr=0.0010
[07/19 15:33:10] cifar100-resnet18 INFO: Epoch 151/200, Acc=0.7694, Val Loss=0.9407, lr=0.0010
[07/19 15:34:03] cifar100-resnet18 INFO: Epoch 152/200, Acc=0.7697, Val Loss=0.9410, lr=0.0010
[07/19 15:34:57] cifar100-resnet18 INFO: Epoch 153/200, Acc=0.7705, Val Loss=0.9337, lr=0.0010
[07/19 15:35:50] cifar100-resnet18 INFO: Epoch 154/200, Acc=0.7718, Val Loss=0.9300, lr=0.0010
[07/19 15:36:43] cifar100-resnet18 INFO: Epoch 155/200, Acc=0.7717, Val Loss=0.9249, lr=0.0010
[07/19 15:37:36] cifar100-resnet18 INFO: Epoch 156/200, Acc=0.7739, Val Loss=0.9323, lr=0.0010
[07/19 15:38:30] cifar100-resnet18 INFO: Epoch 157/200, Acc=0.7734, Val Loss=0.9309, lr=0.0010
[07/19 15:39:24] cifar100-resnet18 INFO: Epoch 158/200, Acc=0.7737, Val Loss=0.9291, lr=0.0010
[07/19 15:40:17] cifar100-resnet18 INFO: Epoch 159/200, Acc=0.7751, Val Loss=0.9269, lr=0.0010
[07/19 15:41:10] cifar100-resnet18 INFO: Epoch 160/200, Acc=0.7727, Val Loss=0.9230, lr=0.0010
[07/19 15:42:03] cifar100-resnet18 INFO: Epoch 161/200, Acc=0.7734, Val Loss=0.9262, lr=0.0010
[07/19 15:42:56] cifar100-resnet18 INFO: Epoch 162/200, Acc=0.7738, Val Loss=0.9245, lr=0.0010
[07/19 15:43:49] cifar100-resnet18 INFO: Epoch 163/200, Acc=0.7740, Val Loss=0.9221, lr=0.0010
[07/19 15:44:43] cifar100-resnet18 INFO: Epoch 164/200, Acc=0.7752, Val Loss=0.9205, lr=0.0010
[07/19 15:45:36] cifar100-resnet18 INFO: Epoch 165/200, Acc=0.7747, Val Loss=0.9213, lr=0.0010
[07/19 15:46:29] cifar100-resnet18 INFO: Epoch 166/200, Acc=0.7737, Val Loss=0.9225, lr=0.0010
[07/19 15:47:23] cifar100-resnet18 INFO: Epoch 167/200, Acc=0.7763, Val Loss=0.9223, lr=0.0010
[07/19 15:48:16] cifar100-resnet18 INFO: Epoch 168/200, Acc=0.7747, Val Loss=0.9194, lr=0.0010
[07/19 15:49:09] cifar100-resnet18 INFO: Epoch 169/200, Acc=0.7759, Val Loss=0.9212, lr=0.0010
[07/19 15:50:02] cifar100-resnet18 INFO: Epoch 170/200, Acc=0.7755, Val Loss=0.9205, lr=0.0010
[07/19 15:50:55] cifar100-resnet18 INFO: Epoch 171/200, Acc=0.7769, Val Loss=0.9170, lr=0.0010
[07/19 15:51:48] cifar100-resnet18 INFO: Epoch 172/200, Acc=0.7753, Val Loss=0.9184, lr=0.0010
[07/19 15:52:41] cifar100-resnet18 INFO: Epoch 173/200, Acc=0.7764, Val Loss=0.9195, lr=0.0010
[07/19 15:53:34] cifar100-resnet18 INFO: Epoch 174/200, Acc=0.7757, Val Loss=0.9167, lr=0.0010
[07/19 15:54:28] cifar100-resnet18 INFO: Epoch 175/200, Acc=0.7761, Val Loss=0.9169, lr=0.0010
[07/19 15:55:21] cifar100-resnet18 INFO: Epoch 176/200, Acc=0.7776, Val Loss=0.9164, lr=0.0010
[07/19 15:56:14] cifar100-resnet18 INFO: Epoch 177/200, Acc=0.7767, Val Loss=0.9140, lr=0.0010
[07/19 15:57:08] cifar100-resnet18 INFO: Epoch 178/200, Acc=0.7760, Val Loss=0.9149, lr=0.0010
[07/19 15:58:01] cifar100-resnet18 INFO: Epoch 179/200, Acc=0.7773, Val Loss=0.9148, lr=0.0010
[07/19 15:58:54] cifar100-resnet18 INFO: Epoch 180/200, Acc=0.7766, Val Loss=0.9156, lr=0.0001
[07/19 15:59:48] cifar100-resnet18 INFO: Epoch 181/200, Acc=0.7783, Val Loss=0.9143, lr=0.0001
[07/19 16:00:41] cifar100-resnet18 INFO: Epoch 182/200, Acc=0.7766, Val Loss=0.9115, lr=0.0001
[07/19 16:01:34] cifar100-resnet18 INFO: Epoch 183/200, Acc=0.7769, Val Loss=0.9154, lr=0.0001
[07/19 16:02:27] cifar100-resnet18 INFO: Epoch 184/200, Acc=0.7770, Val Loss=0.9147, lr=0.0001
[07/19 16:03:20] cifar100-resnet18 INFO: Epoch 185/200, Acc=0.7775, Val Loss=0.9132, lr=0.0001
[07/19 16:04:14] cifar100-resnet18 INFO: Epoch 186/200, Acc=0.7768, Val Loss=0.9145, lr=0.0001
[07/19 16:05:07] cifar100-resnet18 INFO: Epoch 187/200, Acc=0.7782, Val Loss=0.9115, lr=0.0001
[07/19 16:06:00] cifar100-resnet18 INFO: Epoch 188/200, Acc=0.7774, Val Loss=0.9126, lr=0.0001
[07/19 16:06:53] cifar100-resnet18 INFO: Epoch 189/200, Acc=0.7787, Val Loss=0.9132, lr=0.0001
[07/19 16:07:46] cifar100-resnet18 INFO: Epoch 190/200, Acc=0.7787, Val Loss=0.9123, lr=0.0001
[07/19 16:08:40] cifar100-resnet18 INFO: Epoch 191/200, Acc=0.7779, Val Loss=0.9142, lr=0.0001
[07/19 16:09:33] cifar100-resnet18 INFO: Epoch 192/200, Acc=0.7780, Val Loss=0.9133, lr=0.0001
[07/19 16:10:26] cifar100-resnet18 INFO: Epoch 193/200, Acc=0.7773, Val Loss=0.9148, lr=0.0001
[07/19 16:11:19] cifar100-resnet18 INFO: Epoch 194/200, Acc=0.7788, Val Loss=0.9135, lr=0.0001
[07/19 16:12:12] cifar100-resnet18 INFO: Epoch 195/200, Acc=0.7775, Val Loss=0.9153, lr=0.0001
[07/19 16:13:05] cifar100-resnet18 INFO: Epoch 196/200, Acc=0.7773, Val Loss=0.9114, lr=0.0001
[07/19 16:13:58] cifar100-resnet18 INFO: Epoch 197/200, Acc=0.7770, Val Loss=0.9137, lr=0.0001
[07/19 16:14:51] cifar100-resnet18 INFO: Epoch 198/200, Acc=0.7772, Val Loss=0.9135, lr=0.0001
[07/19 16:15:44] cifar100-resnet18 INFO: Epoch 199/200, Acc=0.7780, Val Loss=0.9125, lr=0.0001
[07/19 16:15:44] cifar100-resnet18 INFO: Best Acc=0.7788

从上面数据中可以看到，在学习率下降后模型精度大幅度提升，很明显是在高学习率下，模型参数一直在狭窄的loss鞍部震荡无法跳出，学习率调小后，loss可以继续沿梯度方向下降了。

训练后的模型存储在 run/cifar100/pretrain目录下

3、模型剪枝

3.1 全局剪枝训练

一共使用以下命令进行剪枝训练，对应了好几种剪枝训练方法，不同方法得到的剪枝结构是不一样的。

python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method group_norm --speed-up 3 --global-pruning --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method l1 --speed-up 3 --global-pruning --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method l2 --speed-up 3 --global-pruning --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method random --speed-up 3 --global-pruning --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method lamp --speed-up 3 --global-pruning --finetune

随机剪枝 Acc: 0.7788 =prune=> 0.0160 =fineturn=> 0.7530
lamp剪枝方法 Acc: 0.7788 =prune=> 0.0127=finetrun=>0.7414
l2剪枝训练 Acc: 0.7788 =prune=> 0.0210 =finetrun=>0.7371
l1剪枝训练 Acc: 0.7788 =prune=> 0.0146=finetrun=>0.7478
group_norm剪枝训练 Acc: 0.7788 =prune=> 0.0210=finetrun=>0.7404
可以看到在全局剪枝方法下，居然是随机剪枝效果最好。

随机剪枝输出的信息如下，其他剪枝的日志不在列出了。

[07/19 20:34:40] cifar100-global-random-resnet18 INFO: mode: prune
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: model: resnet18
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: verbose: False
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: dataset: cifar100
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: dataroot: data
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: batch_size: 128
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: total_epochs: 100
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: lr_decay_milestones: 60,80
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: lr_decay_gamma: 0.1
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: lr: 0.01
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: restore: run/cifar100/pretrain/cifar100_resnet18.pth
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: output_dir: run\cifar100\prune\cifar100-global-random-resnet18
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: finetune: False
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: method: random
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: speed_up: 3.0
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: max_pruning_ratio: 1.0
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: soft_keeping_ratio: 0.0
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: reg: 0.0005
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: delta_reg: 0.0001
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: weight_decay: 0.0005
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: seed: None
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: global_pruning: True
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: sl_total_epochs: 100
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: sl_lr: 0.01
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: sl_lr_decay_milestones: 60,80
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: sl_reg_warmup: 0
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: sl_restore: None
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: iterative_steps: 400
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: logger: <Logger cifar100-global-random-resnet18 (DEBUG)>
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: device: cuda
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: num_classes: 100
[07/19 20:34:40] cifar100-global-random-resnet18 INFO: Loading model from run/cifar100/pretrain/cifar100_resnet18.pth
[07/19 20:34:57] cifar100-global-random-resnet18 INFO: Pruning...
[07/19 20:35:04] cifar100-global-random-resnet18 INFO: ResNet(
  (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(32, 38, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(38, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(38, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
    (1): BasicBlock(
      (conv1): Conv2d(32, 38, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(38, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(38, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(32, 85, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(85, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(85, 69, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(69, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(32, 69, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(69, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(69, 72, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(72, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(72, 69, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(69, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(69, 156, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(156, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(156, 163, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(163, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(69, 163, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(163, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(163, 147, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(147, 163, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(163, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(163, 297, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(297, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(297, 305, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(305, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(163, 305, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(305, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(305, 275, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(275, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(275, 305, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(305, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (linear): Linear(in_features=305, out_features=100, bias=True)
)
[07/19 20:35:17] cifar100-global-random-resnet18 INFO: Params: 11.22 M => 3.83 M (34.12%)
[07/19 20:35:17] cifar100-global-random-resnet18 INFO: FLOPs: 556.70 M => 183.80 M (33.02%, 3.03X )
[07/19 20:35:17] cifar100-global-random-resnet18 INFO: Acc: 0.7788 => 0.0160
[07/19 20:35:17] cifar100-global-random-resnet18 INFO: Val Loss: 0.9135 => 5.0128
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: mode: prune
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: model: resnet18
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: verbose: False
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: dataset: cifar100
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: dataroot: data
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: batch_size: 128
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: total_epochs: 100
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: lr_decay_milestones: 60,80
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: lr_decay_gamma: 0.1
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: lr: 0.01
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: restore: run/cifar100/pretrain/cifar100_resnet18.pth
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: output_dir: run\cifar100\prune\cifar100-global-random-resnet18
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: finetune: True
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: method: random
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: speed_up: 3.0
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: max_pruning_ratio: 1.0
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: soft_keeping_ratio: 0.0
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: reg: 0.0005
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: delta_reg: 0.0001
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: weight_decay: 0.0005
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: seed: None
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: global_pruning: True
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: sl_total_epochs: 100
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: sl_lr: 0.01
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: sl_lr_decay_milestones: 60,80
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: sl_reg_warmup: 0
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: sl_restore: None
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: iterative_steps: 400
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: logger: <Logger cifar100-global-random-resnet18 (DEBUG)>
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: device: cuda
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: num_classes: 100
[07/19 23:42:41] cifar100-global-random-resnet18 INFO: Loading model from run/cifar100/pretrain/cifar100_resnet18.pth
[07/19 23:42:57] cifar100-global-random-resnet18 INFO: Pruning...
[07/19 23:43:05] cifar100-global-random-resnet18 INFO: ResNet(
  (conv1): Conv2d(3, 45, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(45, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(45, 37, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(37, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(37, 45, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(45, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
    (1): BasicBlock(
      (conv1): Conv2d(45, 33, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(33, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(33, 45, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(45, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(45, 74, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(74, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(74, 75, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(75, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(45, 75, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(75, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(75, 65, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(65, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(65, 75, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(75, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(75, 138, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(138, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(138, 133, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(133, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(75, 133, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(133, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(133, 130, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(130, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(130, 133, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(133, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(133, 291, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(291, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(291, 292, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(292, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(133, 292, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(292, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(292, 285, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(285, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(285, 292, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(292, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (linear): Linear(in_features=292, out_features=100, bias=True)
)
[07/19 23:43:18] cifar100-global-random-resnet18 INFO: Params: 11.22 M => 3.49 M (31.13%)
[07/19 23:43:18] cifar100-global-random-resnet18 INFO: FLOPs: 556.70 M => 183.35 M (32.93%, 3.04X )
[07/19 23:43:18] cifar100-global-random-resnet18 INFO: Acc: 0.7788 => 0.0157
[07/19 23:43:18] cifar100-global-random-resnet18 INFO: Val Loss: 0.9135 => 5.2573
[07/19 23:43:18] cifar100-global-random-resnet18 INFO: Finetuning...
[07/19 23:44:04] cifar100-global-random-resnet18 INFO: Epoch 0/100, Acc=0.6224, Val Loss=1.3806, lr=0.0100
[07/19 23:44:48] cifar100-global-random-resnet18 INFO: Epoch 1/100, Acc=0.6254, Val Loss=1.4088, lr=0.0100
[07/19 23:45:33] cifar100-global-random-resnet18 INFO: Epoch 2/100, Acc=0.6656, Val Loss=1.2237, lr=0.0100
[07/19 23:46:18] cifar100-global-random-resnet18 INFO: Epoch 3/100, Acc=0.6640, Val Loss=1.2629, lr=0.0100
[07/19 23:47:03] cifar100-global-random-resnet18 INFO: Epoch 4/100, Acc=0.6732, Val Loss=1.2316, lr=0.0100
[07/19 23:47:47] cifar100-global-random-resnet18 INFO: Epoch 5/100, Acc=0.6630, Val Loss=1.2843, lr=0.0100
[07/19 23:48:32] cifar100-global-random-resnet18 INFO: Epoch 6/100, Acc=0.6721, Val Loss=1.2675, lr=0.0100
[07/19 23:49:17] cifar100-global-random-resnet18 INFO: Epoch 7/100, Acc=0.6745, Val Loss=1.2316, lr=0.0100
[07/19 23:50:02] cifar100-global-random-resnet18 INFO: Epoch 8/100, Acc=0.6815, Val Loss=1.2473, lr=0.0100
[07/19 23:50:47] cifar100-global-random-resnet18 INFO: Epoch 9/100, Acc=0.6932, Val Loss=1.1952, lr=0.0100
[07/19 23:51:31] cifar100-global-random-resnet18 INFO: Epoch 10/100, Acc=0.6881, Val Loss=1.2146, lr=0.0100
[07/19 23:52:16] cifar100-global-random-resnet18 INFO: Epoch 11/100, Acc=0.6820, Val Loss=1.2688, lr=0.0100
[07/19 23:53:01] cifar100-global-random-resnet18 INFO: Epoch 12/100, Acc=0.6714, Val Loss=1.3546, lr=0.0100
[07/19 23:53:46] cifar100-global-random-resnet18 INFO: Epoch 13/100, Acc=0.7007, Val Loss=1.2171, lr=0.0100
[07/19 23:54:31] cifar100-global-random-resnet18 INFO: Epoch 14/100, Acc=0.6960, Val Loss=1.2253, lr=0.0100
[07/19 23:55:16] cifar100-global-random-resnet18 INFO: Epoch 15/100, Acc=0.7021, Val Loss=1.2148, lr=0.0100
[07/19 23:56:00] cifar100-global-random-resnet18 INFO: Epoch 16/100, Acc=0.6922, Val Loss=1.2675, lr=0.0100
[07/19 23:56:45] cifar100-global-random-resnet18 INFO: Epoch 17/100, Acc=0.6892, Val Loss=1.2843, lr=0.0100
[07/19 23:57:30] cifar100-global-random-resnet18 INFO: Epoch 18/100, Acc=0.6863, Val Loss=1.2947, lr=0.0100
[07/19 23:58:14] cifar100-global-random-resnet18 INFO: Epoch 19/100, Acc=0.6920, Val Loss=1.2611, lr=0.0100
[07/19 23:58:59] cifar100-global-random-resnet18 INFO: Epoch 20/100, Acc=0.6862, Val Loss=1.3248, lr=0.0100
[07/19 23:59:44] cifar100-global-random-resnet18 INFO: Epoch 21/100, Acc=0.6957, Val Loss=1.2437, lr=0.0100
[07/20 00:00:29] cifar100-global-random-resnet18 INFO: Epoch 22/100, Acc=0.6918, Val Loss=1.2900, lr=0.0100
[07/20 00:01:13] cifar100-global-random-resnet18 INFO: Epoch 23/100, Acc=0.6975, Val Loss=1.2818, lr=0.0100
[07/20 00:01:58] cifar100-global-random-resnet18 INFO: Epoch 24/100, Acc=0.6904, Val Loss=1.3153, lr=0.0100
[07/20 00:02:43] cifar100-global-random-resnet18 INFO: Epoch 25/100, Acc=0.7028, Val Loss=1.2621, lr=0.0100
[07/20 00:03:28] cifar100-global-random-resnet18 INFO: Epoch 26/100, Acc=0.7003, Val Loss=1.2839, lr=0.0100
[07/20 00:04:12] cifar100-global-random-resnet18 INFO: Epoch 27/100, Acc=0.7024, Val Loss=1.2656, lr=0.0100
[07/20 00:04:57] cifar100-global-random-resnet18 INFO: Epoch 28/100, Acc=0.6914, Val Loss=1.3054, lr=0.0100
[07/20 00:05:42] cifar100-global-random-resnet18 INFO: Epoch 29/100, Acc=0.7011, Val Loss=1.2801, lr=0.0100
[07/20 00:06:27] cifar100-global-random-resnet18 INFO: Epoch 30/100, Acc=0.6927, Val Loss=1.3250, lr=0.0100
[07/20 00:07:12] cifar100-global-random-resnet18 INFO: Epoch 31/100, Acc=0.7050, Val Loss=1.2780, lr=0.0100
[07/20 00:07:57] cifar100-global-random-resnet18 INFO: Epoch 32/100, Acc=0.6959, Val Loss=1.3227, lr=0.0100
[07/20 00:08:41] cifar100-global-random-resnet18 INFO: Epoch 33/100, Acc=0.7025, Val Loss=1.2571, lr=0.0100
[07/20 00:09:26] cifar100-global-random-resnet18 INFO: Epoch 34/100, Acc=0.6942, Val Loss=1.2734, lr=0.0100
[07/20 00:10:10] cifar100-global-random-resnet18 INFO: Epoch 35/100, Acc=0.6983, Val Loss=1.2837, lr=0.0100
[07/20 00:10:55] cifar100-global-random-resnet18 INFO: Epoch 36/100, Acc=0.7003, Val Loss=1.3126, lr=0.0100
[07/20 00:11:40] cifar100-global-random-resnet18 INFO: Epoch 37/100, Acc=0.6871, Val Loss=1.3638, lr=0.0100
[07/20 00:12:24] cifar100-global-random-resnet18 INFO: Epoch 38/100, Acc=0.6872, Val Loss=1.3551, lr=0.0100
[07/20 00:13:09] cifar100-global-random-resnet18 INFO: Epoch 39/100, Acc=0.6974, Val Loss=1.2831, lr=0.0100
[07/20 00:13:54] cifar100-global-random-resnet18 INFO: Epoch 40/100, Acc=0.6932, Val Loss=1.3359, lr=0.0100
[07/20 00:14:39] cifar100-global-random-resnet18 INFO: Epoch 41/100, Acc=0.6946, Val Loss=1.3712, lr=0.0100
[07/20 00:15:24] cifar100-global-random-resnet18 INFO: Epoch 42/100, Acc=0.7078, Val Loss=1.2807, lr=0.0100
[07/20 00:16:09] cifar100-global-random-resnet18 INFO: Epoch 43/100, Acc=0.6971, Val Loss=1.2942, lr=0.0100
[07/20 00:16:54] cifar100-global-random-resnet18 INFO: Epoch 44/100, Acc=0.7045, Val Loss=1.2728, lr=0.0100
[07/20 00:17:38] cifar100-global-random-resnet18 INFO: Epoch 45/100, Acc=0.7007, Val Loss=1.3086, lr=0.0100
[07/20 00:18:23] cifar100-global-random-resnet18 INFO: Epoch 46/100, Acc=0.6947, Val Loss=1.3201, lr=0.0100
[07/20 00:19:08] cifar100-global-random-resnet18 INFO: Epoch 47/100, Acc=0.6960, Val Loss=1.3086, lr=0.0100
[07/20 00:19:53] cifar100-global-random-resnet18 INFO: Epoch 48/100, Acc=0.7004, Val Loss=1.2941, lr=0.0100
[07/20 00:20:37] cifar100-global-random-resnet18 INFO: Epoch 49/100, Acc=0.6951, Val Loss=1.3098, lr=0.0100
[07/20 00:21:22] cifar100-global-random-resnet18 INFO: Epoch 50/100, Acc=0.6984, Val Loss=1.2900, lr=0.0100
[07/20 00:22:07] cifar100-global-random-resnet18 INFO: Epoch 51/100, Acc=0.6923, Val Loss=1.3532, lr=0.0100
[07/20 00:22:51] cifar100-global-random-resnet18 INFO: Epoch 52/100, Acc=0.6938, Val Loss=1.3474, lr=0.0100
[07/20 00:23:36] cifar100-global-random-resnet18 INFO: Epoch 53/100, Acc=0.6995, Val Loss=1.2996, lr=0.0100
[07/20 00:24:21] cifar100-global-random-resnet18 INFO: Epoch 54/100, Acc=0.6976, Val Loss=1.3236, lr=0.0100
[07/20 00:25:05] cifar100-global-random-resnet18 INFO: Epoch 55/100, Acc=0.6921, Val Loss=1.3354, lr=0.0100
[07/20 00:25:50] cifar100-global-random-resnet18 INFO: Epoch 56/100, Acc=0.6907, Val Loss=1.3083, lr=0.0100
[07/20 00:26:35] cifar100-global-random-resnet18 INFO: Epoch 57/100, Acc=0.6920, Val Loss=1.3144, lr=0.0100
[07/20 00:27:20] cifar100-global-random-resnet18 INFO: Epoch 58/100, Acc=0.6981, Val Loss=1.3068, lr=0.0100
[07/20 00:28:05] cifar100-global-random-resnet18 INFO: Epoch 59/100, Acc=0.6950, Val Loss=1.2745, lr=0.0100
[07/20 00:28:50] cifar100-global-random-resnet18 INFO: Epoch 60/100, Acc=0.7409, Val Loss=1.1193, lr=0.0010
[07/20 00:29:34] cifar100-global-random-resnet18 INFO: Epoch 61/100, Acc=0.7431, Val Loss=1.1033, lr=0.0010
[07/20 00:30:19] cifar100-global-random-resnet18 INFO: Epoch 62/100, Acc=0.7453, Val Loss=1.0978, lr=0.0010
[07/20 00:31:04] cifar100-global-random-resnet18 INFO: Epoch 63/100, Acc=0.7454, Val Loss=1.0950, lr=0.0010
[07/20 00:31:48] cifar100-global-random-resnet18 INFO: Epoch 64/100, Acc=0.7454, Val Loss=1.0948, lr=0.0010
[07/20 00:32:33] cifar100-global-random-resnet18 INFO: Epoch 65/100, Acc=0.7453, Val Loss=1.0866, lr=0.0010
[07/20 00:33:18] cifar100-global-random-resnet18 INFO: Epoch 66/100, Acc=0.7469, Val Loss=1.0868, lr=0.0010
[07/20 00:34:03] cifar100-global-random-resnet18 INFO: Epoch 67/100, Acc=0.7468, Val Loss=1.0825, lr=0.0010
[07/20 00:34:48] cifar100-global-random-resnet18 INFO: Epoch 68/100, Acc=0.7473, Val Loss=1.0820, lr=0.0010
[07/20 00:35:33] cifar100-global-random-resnet18 INFO: Epoch 69/100, Acc=0.7493, Val Loss=1.0815, lr=0.0010
[07/20 00:36:18] cifar100-global-random-resnet18 INFO: Epoch 70/100, Acc=0.7479, Val Loss=1.0811, lr=0.0010
[07/20 00:37:02] cifar100-global-random-resnet18 INFO: Epoch 71/100, Acc=0.7492, Val Loss=1.0849, lr=0.0010
[07/20 00:37:47] cifar100-global-random-resnet18 INFO: Epoch 72/100, Acc=0.7502, Val Loss=1.0753, lr=0.0010
[07/20 00:38:31] cifar100-global-random-resnet18 INFO: Epoch 73/100, Acc=0.7497, Val Loss=1.0806, lr=0.0010
[07/20 00:39:16] cifar100-global-random-resnet18 INFO: Epoch 74/100, Acc=0.7495, Val Loss=1.0748, lr=0.0010
[07/20 00:40:01] cifar100-global-random-resnet18 INFO: Epoch 75/100, Acc=0.7495, Val Loss=1.0787, lr=0.0010
[07/20 00:40:45] cifar100-global-random-resnet18 INFO: Epoch 76/100, Acc=0.7491, Val Loss=1.0734, lr=0.0010
[07/20 00:41:30] cifar100-global-random-resnet18 INFO: Epoch 77/100, Acc=0.7486, Val Loss=1.0745, lr=0.0010
[07/20 00:42:15] cifar100-global-random-resnet18 INFO: Epoch 78/100, Acc=0.7506, Val Loss=1.0678, lr=0.0010
[07/20 00:43:00] cifar100-global-random-resnet18 INFO: Epoch 79/100, Acc=0.7513, Val Loss=1.0644, lr=0.0010
[07/20 00:43:45] cifar100-global-random-resnet18 INFO: Epoch 80/100, Acc=0.7508, Val Loss=1.0646, lr=0.0001
[07/20 00:44:30] cifar100-global-random-resnet18 INFO: Epoch 81/100, Acc=0.7500, Val Loss=1.0674, lr=0.0001
[07/20 00:45:14] cifar100-global-random-resnet18 INFO: Epoch 82/100, Acc=0.7510, Val Loss=1.0638, lr=0.0001
[07/20 00:45:59] cifar100-global-random-resnet18 INFO: Epoch 83/100, Acc=0.7518, Val Loss=1.0616, lr=0.0001
[07/20 00:46:43] cifar100-global-random-resnet18 INFO: Epoch 84/100, Acc=0.7529, Val Loss=1.0613, lr=0.0001
[07/20 00:47:28] cifar100-global-random-resnet18 INFO: Epoch 85/100, Acc=0.7530, Val Loss=1.0634, lr=0.0001
[07/20 00:48:13] cifar100-global-random-resnet18 INFO: Epoch 86/100, Acc=0.7509, Val Loss=1.0630, lr=0.0001
[07/20 00:48:58] cifar100-global-random-resnet18 INFO: Epoch 87/100, Acc=0.7515, Val Loss=1.0625, lr=0.0001
[07/20 00:49:42] cifar100-global-random-resnet18 INFO: Epoch 88/100, Acc=0.7524, Val Loss=1.0659, lr=0.0001
[07/20 00:50:28] cifar100-global-random-resnet18 INFO: Epoch 89/100, Acc=0.7522, Val Loss=1.0640, lr=0.0001
[07/20 00:51:13] cifar100-global-random-resnet18 INFO: Epoch 90/100, Acc=0.7503, Val Loss=1.0614, lr=0.0001
[07/20 00:51:57] cifar100-global-random-resnet18 INFO: Epoch 91/100, Acc=0.7497, Val Loss=1.0642, lr=0.0001
[07/20 00:52:42] cifar100-global-random-resnet18 INFO: Epoch 92/100, Acc=0.7522, Val Loss=1.0639, lr=0.0001
[07/20 00:53:27] cifar100-global-random-resnet18 INFO: Epoch 93/100, Acc=0.7527, Val Loss=1.0638, lr=0.0001
[07/20 00:54:12] cifar100-global-random-resnet18 INFO: Epoch 94/100, Acc=0.7506, Val Loss=1.0606, lr=0.0001
[07/20 00:54:57] cifar100-global-random-resnet18 INFO: Epoch 95/100, Acc=0.7522, Val Loss=1.0625, lr=0.0001
[07/20 00:55:41] cifar100-global-random-resnet18 INFO: Epoch 96/100, Acc=0.7517, Val Loss=1.0644, lr=0.0001
[07/20 00:56:26] cifar100-global-random-resnet18 INFO: Epoch 97/100, Acc=0.7523, Val Loss=1.0593, lr=0.0001
[07/20 00:57:11] cifar100-global-random-resnet18 INFO: Epoch 98/100, Acc=0.7525, Val Loss=1.0563, lr=0.0001
[07/20 00:57:56] cifar100-global-random-resnet18 INFO: Epoch 99/100, Acc=0.7516, Val Loss=1.0635, lr=0.0001
[07/20 00:57:56] cifar100-global-random-resnet18 INFO: Best Acc=0.7530

3.2 局部剪枝训练

剪枝相关命令如下，一共进行了5中剪枝方法的对比。由于是局部剪枝，每种剪枝方法得到的模型结构都是一样的。

python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method group_norm --speed-up 3  --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method l1 --speed-up 3 --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method l2 --speed-up 3 --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method random --speed-up 3 --finetune
python main.py --mode prune --model resnet18 --batch-size 128 --restore run/cifar100/pretrain/cifar100_resnet18.pth --dataset cifar100  --method lamp --speed-up 3 --finetune

local-l1剪枝 Acc: 0.7788 =prune=> 0.0276=fineturn=> 0.7604
local-group_norm剪枝 Acc: 0.7788 =prune=> 0.0306 =fineturn=> 0.7523
local-l2剪枝 Acc: 0.7788 =prune=> 0.0363 =fineturn=> 0.7572
local-lamp剪枝 Acc: 0.7788 =prune=> 0.0363=fineturn=> 0.7585
local-random剪枝 Acc: 0.7788 =prune=> 0.0113 =fineturn=> 0.7567

local-l1剪枝输出的信息如下所示

[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: mode: prune
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: model: resnet18
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: verbose: False
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: dataset: cifar100
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: dataroot: data
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: batch_size: 128
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: total_epochs: 100
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: lr_decay_milestones: 60,80
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: lr_decay_gamma: 0.1
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: lr: 0.01
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: restore: run/cifar100/pretrain/cifar100_resnet18.pth
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: output_dir: run\cifar100\prune\cifar100-local-l1-resnet18
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: finetune: True
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: method: l1
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: speed_up: 3.0
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: max_pruning_ratio: 1.0
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: soft_keeping_ratio: 0.0
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: reg: 0.0005
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: delta_reg: 0.0001
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: weight_decay: 0.0005
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: seed: None
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: global_pruning: False
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: sl_total_epochs: 100
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: sl_lr: 0.01
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: sl_lr_decay_milestones: 60,80
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: sl_reg_warmup: 0
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: sl_restore: None
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: iterative_steps: 400
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: logger: <Logger cifar100-local-l1-resnet18 (DEBUG)>
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: device: cuda
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: num_classes: 100
[07/20 14:19:16] cifar100-local-l1-resnet18 INFO: Loading model from run/cifar100/pretrain/cifar100_resnet18.pth
[07/20 14:19:33] cifar100-local-l1-resnet18 INFO: Pruning...
[07/20 14:19:43] cifar100-local-l1-resnet18 INFO: ResNet(
  (conv1): Conv2d(3, 36, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(36, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(36, 36, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(36, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(36, 36, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(36, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
    (1): BasicBlock(
      (conv1): Conv2d(36, 36, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(36, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(36, 36, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(36, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(36, 73, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(73, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(73, 73, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(73, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(36, 73, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(73, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(73, 73, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(73, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(73, 73, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(73, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(73, 147, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(147, 147, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(73, 147, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(147, 147, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(147, 147, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(147, 295, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(295, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(295, 295, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(295, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(147, 295, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(295, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(295, 295, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(295, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(295, 295, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(295, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (linear): Linear(in_features=295, out_features=100, bias=True)
)
[07/20 14:19:56] cifar100-local-l1-resnet18 INFO: Params: 11.22 M => 3.73 M (33.22%)
[07/20 14:19:56] cifar100-local-l1-resnet18 INFO: FLOPs: 556.70 M => 181.80 M (32.66%, 3.06X )
[07/20 14:19:56] cifar100-local-l1-resnet18 INFO: Acc: 0.7788 => 0.0276
[07/20 14:19:56] cifar100-local-l1-resnet18 INFO: Val Loss: 0.9135 => 5.3336
[07/20 14:19:56] cifar100-local-l1-resnet18 INFO: Finetuning...
[07/20 14:20:40] cifar100-local-l1-resnet18 INFO: Epoch 0/100, Acc=0.6558, Val Loss=1.2705, lr=0.0100
[07/20 14:21:24] cifar100-local-l1-resnet18 INFO: Epoch 1/100, Acc=0.6563, Val Loss=1.2655, lr=0.0100
[07/20 14:22:08] cifar100-local-l1-resnet18 INFO: Epoch 2/100, Acc=0.6777, Val Loss=1.1976, lr=0.0100
[07/20 14:22:52] cifar100-local-l1-resnet18 INFO: Epoch 3/100, Acc=0.6808, Val Loss=1.2337, lr=0.0100
[07/20 14:23:36] cifar100-local-l1-resnet18 INFO: Epoch 4/100, Acc=0.6804, Val Loss=1.2215, lr=0.0100
[07/20 14:24:20] cifar100-local-l1-resnet18 INFO: Epoch 5/100, Acc=0.6854, Val Loss=1.2443, lr=0.0100
[07/20 14:25:04] cifar100-local-l1-resnet18 INFO: Epoch 6/100, Acc=0.6931, Val Loss=1.1990, lr=0.0100
[07/20 14:25:48] cifar100-local-l1-resnet18 INFO: Epoch 7/100, Acc=0.6893, Val Loss=1.1900, lr=0.0100
[07/20 14:26:33] cifar100-local-l1-resnet18 INFO: Epoch 8/100, Acc=0.6825, Val Loss=1.2586, lr=0.0100
[07/20 14:27:17] cifar100-local-l1-resnet18 INFO: Epoch 9/100, Acc=0.6826, Val Loss=1.2758, lr=0.0100
[07/20 14:28:01] cifar100-local-l1-resnet18 INFO: Epoch 10/100, Acc=0.6942, Val Loss=1.2044, lr=0.0100
[07/20 14:28:45] cifar100-local-l1-resnet18 INFO: Epoch 11/100, Acc=0.6926, Val Loss=1.2756, lr=0.0100
[07/20 14:29:29] cifar100-local-l1-resnet18 INFO: Epoch 12/100, Acc=0.6919, Val Loss=1.2782, lr=0.0100
[07/20 14:30:13] cifar100-local-l1-resnet18 INFO: Epoch 13/100, Acc=0.6893, Val Loss=1.2574, lr=0.0100
[07/20 14:30:57] cifar100-local-l1-resnet18 INFO: Epoch 14/100, Acc=0.6860, Val Loss=1.3108, lr=0.0100
[07/20 14:31:41] cifar100-local-l1-resnet18 INFO: Epoch 15/100, Acc=0.7011, Val Loss=1.2248, lr=0.0100
[07/20 14:32:25] cifar100-local-l1-resnet18 INFO: Epoch 16/100, Acc=0.6933, Val Loss=1.2636, lr=0.0100
[07/20 14:33:09] cifar100-local-l1-resnet18 INFO: Epoch 17/100, Acc=0.7055, Val Loss=1.2492, lr=0.0100
[07/20 14:33:53] cifar100-local-l1-resnet18 INFO: Epoch 18/100, Acc=0.6937, Val Loss=1.2649, lr=0.0100
[07/20 14:34:36] cifar100-local-l1-resnet18 INFO: Epoch 19/100, Acc=0.6909, Val Loss=1.3085, lr=0.0100
[07/20 14:35:20] cifar100-local-l1-resnet18 INFO: Epoch 20/100, Acc=0.6952, Val Loss=1.2462, lr=0.0100
[07/20 14:36:04] cifar100-local-l1-resnet18 INFO: Epoch 21/100, Acc=0.6922, Val Loss=1.2832, lr=0.0100
[07/20 14:36:48] cifar100-local-l1-resnet18 INFO: Epoch 22/100, Acc=0.7004, Val Loss=1.2691, lr=0.0100
[07/20 14:37:32] cifar100-local-l1-resnet18 INFO: Epoch 23/100, Acc=0.6987, Val Loss=1.2666, lr=0.0100
[07/20 14:38:15] cifar100-local-l1-resnet18 INFO: Epoch 24/100, Acc=0.7031, Val Loss=1.2924, lr=0.0100
[07/20 14:39:00] cifar100-local-l1-resnet18 INFO: Epoch 25/100, Acc=0.6985, Val Loss=1.2654, lr=0.0100
[07/20 14:39:43] cifar100-local-l1-resnet18 INFO: Epoch 26/100, Acc=0.6938, Val Loss=1.2897, lr=0.0100
[07/20 14:40:27] cifar100-local-l1-resnet18 INFO: Epoch 27/100, Acc=0.6930, Val Loss=1.3100, lr=0.0100
[07/20 14:41:11] cifar100-local-l1-resnet18 INFO: Epoch 28/100, Acc=0.6970, Val Loss=1.3200, lr=0.0100
[07/20 14:41:55] cifar100-local-l1-resnet18 INFO: Epoch 29/100, Acc=0.6971, Val Loss=1.3157, lr=0.0100
[07/20 14:42:39] cifar100-local-l1-resnet18 INFO: Epoch 30/100, Acc=0.6993, Val Loss=1.2963, lr=0.0100
[07/20 14:43:22] cifar100-local-l1-resnet18 INFO: Epoch 31/100, Acc=0.6879, Val Loss=1.3422, lr=0.0100
[07/20 14:44:07] cifar100-local-l1-resnet18 INFO: Epoch 32/100, Acc=0.7059, Val Loss=1.2621, lr=0.0100
[07/20 14:44:51] cifar100-local-l1-resnet18 INFO: Epoch 33/100, Acc=0.6988, Val Loss=1.2761, lr=0.0100
[07/20 14:45:34] cifar100-local-l1-resnet18 INFO: Epoch 34/100, Acc=0.7057, Val Loss=1.2563, lr=0.0100
[07/20 14:46:18] cifar100-local-l1-resnet18 INFO: Epoch 35/100, Acc=0.7067, Val Loss=1.2822, lr=0.0100
[07/20 14:47:02] cifar100-local-l1-resnet18 INFO: Epoch 36/100, Acc=0.6963, Val Loss=1.2966, lr=0.0100
[07/20 14:47:46] cifar100-local-l1-resnet18 INFO: Epoch 37/100, Acc=0.6997, Val Loss=1.3047, lr=0.0100
[07/20 14:48:30] cifar100-local-l1-resnet18 INFO: Epoch 38/100, Acc=0.7008, Val Loss=1.2907, lr=0.0100
[07/20 14:49:14] cifar100-local-l1-resnet18 INFO: Epoch 39/100, Acc=0.6968, Val Loss=1.3164, lr=0.0100
[07/20 14:49:58] cifar100-local-l1-resnet18 INFO: Epoch 40/100, Acc=0.7088, Val Loss=1.2493, lr=0.0100
[07/20 14:50:42] cifar100-local-l1-resnet18 INFO: Epoch 41/100, Acc=0.7077, Val Loss=1.2828, lr=0.0100
[07/20 14:51:26] cifar100-local-l1-resnet18 INFO: Epoch 42/100, Acc=0.6890, Val Loss=1.3539, lr=0.0100
[07/20 14:52:10] cifar100-local-l1-resnet18 INFO: Epoch 43/100, Acc=0.6919, Val Loss=1.3022, lr=0.0100
[07/20 14:52:54] cifar100-local-l1-resnet18 INFO: Epoch 44/100, Acc=0.7021, Val Loss=1.3283, lr=0.0100
[07/20 14:53:38] cifar100-local-l1-resnet18 INFO: Epoch 45/100, Acc=0.7024, Val Loss=1.2758, lr=0.0100
[07/20 14:54:22] cifar100-local-l1-resnet18 INFO: Epoch 46/100, Acc=0.6950, Val Loss=1.2936, lr=0.0100
[07/20 14:55:07] cifar100-local-l1-resnet18 INFO: Epoch 47/100, Acc=0.7005, Val Loss=1.2721, lr=0.0100
[07/20 14:55:51] cifar100-local-l1-resnet18 INFO: Epoch 48/100, Acc=0.6889, Val Loss=1.3612, lr=0.0100
[07/20 14:56:35] cifar100-local-l1-resnet18 INFO: Epoch 49/100, Acc=0.7046, Val Loss=1.2468, lr=0.0100
[07/20 14:57:19] cifar100-local-l1-resnet18 INFO: Epoch 50/100, Acc=0.6925, Val Loss=1.3648, lr=0.0100
[07/20 14:58:03] cifar100-local-l1-resnet18 INFO: Epoch 51/100, Acc=0.6969, Val Loss=1.3272, lr=0.0100
[07/20 14:58:47] cifar100-local-l1-resnet18 INFO: Epoch 52/100, Acc=0.6884, Val Loss=1.3682, lr=0.0100
[07/20 14:59:30] cifar100-local-l1-resnet18 INFO: Epoch 53/100, Acc=0.6999, Val Loss=1.3116, lr=0.0100
[07/20 15:00:15] cifar100-local-l1-resnet18 INFO: Epoch 54/100, Acc=0.6934, Val Loss=1.3236, lr=0.0100
[07/20 15:00:59] cifar100-local-l1-resnet18 INFO: Epoch 55/100, Acc=0.7049, Val Loss=1.2843, lr=0.0100
[07/20 15:01:43] cifar100-local-l1-resnet18 INFO: Epoch 56/100, Acc=0.6957, Val Loss=1.3413, lr=0.0100
[07/20 15:02:27] cifar100-local-l1-resnet18 INFO: Epoch 57/100, Acc=0.6883, Val Loss=1.3933, lr=0.0100
[07/20 15:03:10] cifar100-local-l1-resnet18 INFO: Epoch 58/100, Acc=0.7019, Val Loss=1.2966, lr=0.0100
[07/20 15:03:54] cifar100-local-l1-resnet18 INFO: Epoch 59/100, Acc=0.6891, Val Loss=1.3715, lr=0.0100
[07/20 15:04:38] cifar100-local-l1-resnet18 INFO: Epoch 60/100, Acc=0.7424, Val Loss=1.0840, lr=0.0010
[07/20 15:05:22] cifar100-local-l1-resnet18 INFO: Epoch 61/100, Acc=0.7477, Val Loss=1.0738, lr=0.0010
[07/20 15:06:06] cifar100-local-l1-resnet18 INFO: Epoch 62/100, Acc=0.7502, Val Loss=1.0661, lr=0.0010
[07/20 15:06:50] cifar100-local-l1-resnet18 INFO: Epoch 63/100, Acc=0.7507, Val Loss=1.0676, lr=0.0010
[07/20 15:07:34] cifar100-local-l1-resnet18 INFO: Epoch 64/100, Acc=0.7535, Val Loss=1.0560, lr=0.0010
[07/20 15:08:18] cifar100-local-l1-resnet18 INFO: Epoch 65/100, Acc=0.7529, Val Loss=1.0535, lr=0.0010
[07/20 15:09:02] cifar100-local-l1-resnet18 INFO: Epoch 66/100, Acc=0.7535, Val Loss=1.0491, lr=0.0010
[07/20 15:09:46] cifar100-local-l1-resnet18 INFO: Epoch 67/100, Acc=0.7538, Val Loss=1.0554, lr=0.0010
[07/20 15:10:30] cifar100-local-l1-resnet18 INFO: Epoch 68/100, Acc=0.7536, Val Loss=1.0513, lr=0.0010
[07/20 15:11:14] cifar100-local-l1-resnet18 INFO: Epoch 69/100, Acc=0.7542, Val Loss=1.0468, lr=0.0010
[07/20 15:11:58] cifar100-local-l1-resnet18 INFO: Epoch 70/100, Acc=0.7549, Val Loss=1.0490, lr=0.0010
[07/20 15:12:42] cifar100-local-l1-resnet18 INFO: Epoch 71/100, Acc=0.7565, Val Loss=1.0459, lr=0.0010
[07/20 15:13:26] cifar100-local-l1-resnet18 INFO: Epoch 72/100, Acc=0.7541, Val Loss=1.0449, lr=0.0010
[07/20 15:14:10] cifar100-local-l1-resnet18 INFO: Epoch 73/100, Acc=0.7545, Val Loss=1.0434, lr=0.0010
[07/20 15:14:54] cifar100-local-l1-resnet18 INFO: Epoch 74/100, Acc=0.7565, Val Loss=1.0437, lr=0.0010
[07/20 15:15:38] cifar100-local-l1-resnet18 INFO: Epoch 75/100, Acc=0.7550, Val Loss=1.0397, lr=0.0010
[07/20 15:16:22] cifar100-local-l1-resnet18 INFO: Epoch 76/100, Acc=0.7576, Val Loss=1.0345, lr=0.0010
[07/20 15:17:06] cifar100-local-l1-resnet18 INFO: Epoch 77/100, Acc=0.7554, Val Loss=1.0356, lr=0.0010
[07/20 15:17:50] cifar100-local-l1-resnet18 INFO: Epoch 78/100, Acc=0.7575, Val Loss=1.0318, lr=0.0010
[07/20 15:18:34] cifar100-local-l1-resnet18 INFO: Epoch 79/100, Acc=0.7579, Val Loss=1.0332, lr=0.0010
[07/20 15:19:18] cifar100-local-l1-resnet18 INFO: Epoch 80/100, Acc=0.7577, Val Loss=1.0355, lr=0.0001
[07/20 15:20:02] cifar100-local-l1-resnet18 INFO: Epoch 81/100, Acc=0.7573, Val Loss=1.0321, lr=0.0001
[07/20 15:20:47] cifar100-local-l1-resnet18 INFO: Epoch 82/100, Acc=0.7584, Val Loss=1.0318, lr=0.0001
[07/20 15:21:31] cifar100-local-l1-resnet18 INFO: Epoch 83/100, Acc=0.7564, Val Loss=1.0345, lr=0.0001
[07/20 15:22:15] cifar100-local-l1-resnet18 INFO: Epoch 84/100, Acc=0.7589, Val Loss=1.0335, lr=0.0001
[07/20 15:22:59] cifar100-local-l1-resnet18 INFO: Epoch 85/100, Acc=0.7586, Val Loss=1.0358, lr=0.0001
[07/20 15:23:43] cifar100-local-l1-resnet18 INFO: Epoch 86/100, Acc=0.7579, Val Loss=1.0306, lr=0.0001
[07/20 15:24:27] cifar100-local-l1-resnet18 INFO: Epoch 87/100, Acc=0.7589, Val Loss=1.0312, lr=0.0001
[07/20 15:25:12] cifar100-local-l1-resnet18 INFO: Epoch 88/100, Acc=0.7583, Val Loss=1.0340, lr=0.0001
[07/20 15:25:55] cifar100-local-l1-resnet18 INFO: Epoch 89/100, Acc=0.7581, Val Loss=1.0324, lr=0.0001
[07/20 15:26:40] cifar100-local-l1-resnet18 INFO: Epoch 90/100, Acc=0.7577, Val Loss=1.0306, lr=0.0001
[07/20 15:27:23] cifar100-local-l1-resnet18 INFO: Epoch 91/100, Acc=0.7568, Val Loss=1.0331, lr=0.0001
[07/20 15:28:07] cifar100-local-l1-resnet18 INFO: Epoch 92/100, Acc=0.7571, Val Loss=1.0316, lr=0.0001
[07/20 15:28:51] cifar100-local-l1-resnet18 INFO: Epoch 93/100, Acc=0.7595, Val Loss=1.0299, lr=0.0001
[07/20 15:29:35] cifar100-local-l1-resnet18 INFO: Epoch 94/100, Acc=0.7576, Val Loss=1.0319, lr=0.0001
[07/20 15:30:19] cifar100-local-l1-resnet18 INFO: Epoch 95/100, Acc=0.7604, Val Loss=1.0301, lr=0.0001
[07/20 15:31:03] cifar100-local-l1-resnet18 INFO: Epoch 96/100, Acc=0.7599, Val Loss=1.0317, lr=0.0001
[07/20 15:31:47] cifar100-local-l1-resnet18 INFO: Epoch 97/100, Acc=0.7574, Val Loss=1.0329, lr=0.0001
[07/20 15:32:30] cifar100-local-l1-resnet18 INFO: Epoch 98/100, Acc=0.7572, Val Loss=1.0322, lr=0.0001
[07/20 15:33:14] cifar100-local-l1-resnet18 INFO: Epoch 99/100, Acc=0.7585, Val Loss=1.0320, lr=0.0001
[07/20 15:33:14] cifar100-local-l1-resnet18 INFO: Best Acc=0.7604

原文地址：https://blog.csdn.net/a486259/article/details/140568152

免责声明：本站文章内容转载自网络资源，如本站内容侵犯了原著者的合法权益，可联系本站删除。更多内容请关注自学内容网（zxcms.com）！