Day 062 — LightGBM 入門 — leaf-wise成長・カテゴリカル処理・XGBoostとの速度比較

📚 背景知識（読んでから問題へ）

XGBoost vs LightGBM — 根本的な違い

なぜLightGBMは速いのか — 3つの技術

🌳 level-wise vs leaf-wise の成長比較

⚡ 速度・精度比較（n=100,000 の場合）

🏷️ カテゴリカル変数の直接処理

カテゴリカル変数の渡し方（3通り）

⚙️ 主要パラメータ

🗂️ データスキーマ（Titanic 拡張版 n=10,000）

📝 問題

セットアップコード（最初に実行）

📋 コピーimport numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.preprocessing import LabelEncoder
import time
import warnings
warnings.filterwarnings('ignore')

try:
    import lightgbm as lgb
    print(f"LightGBM version: {lgb.__version__}")
except ImportError:
    print("pip install lightgbm")

try:
    import xgboost as xgb
    print(f"XGBoost version: {xgb.__version__}")
except ImportError:
    print("pip install xgboost")

np.random.seed(42)
n = 10000
data = pd.DataFrame({
    'Pclass':   np.random.choice([1, 2, 3], n, p=[0.24, 0.21, 0.55]),
    'Sex':      np.random.choice(['male', 'female'], n, p=[0.65, 0.35]),
    'Age':      np.random.normal(29.7, 14.5, n).clip(1, 80),
    'SibSp':    np.random.choice(range(9), n, p=[0.68,0.23,0.04,0.02,0.01,0.01,0.005,0.005,0.01]),
    'Parch':    np.random.choice(range(7), n, p=[0.76,0.13,0.09,0.005,0.005,0.005,0.005]),
    'Fare':     np.abs(np.random.normal(32, 50, n)),
    'Embarked': np.random.choice(['S', 'C', 'Q'], n, p=[0.72, 0.19, 0.09]),
})
data['Survived'] = (
    (data['Sex'] == 'female') * 0.55 + (data['Pclass'] == 1) * 0.25 +
    (data['Age'] < 16) * 0.2 + np.random.normal(0, 0.12, n)
) > 0.4
data['Survived'] = data['Survived'].astype(int)
data['Sex_enc']      = LabelEncoder().fit_transform(data['Sex'])
data['Embarked_enc'] = LabelEncoder().fit_transform(data['Embarked'].fillna('S'))
features_enc = ['Pclass', 'Sex_enc', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked_enc']
X_enc = data[features_enc].values
y     = data['Survived'].values
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

問1 — XGBoost vs LightGBM の基本比較

1. XGBClassifier(n_estimators=200) と LGBMClassifier(n_estimators=200) を同じデータ（エンコード済み）で StratifiedKFold(5) 比較せよ
2. 学習時間・CV精度・訓練精度の過学習度を記録せよ
3. どちらが速く、どちらが精度が高かったか説明せよ

問2 — LightGBMのカテゴリカル変数直接処理

1. dtype='category' に変換した Sex・Embarked を LGBMClassifier に渡して学習せよ
2. ラベルエンコード済みバージョンとCV精度を比較せよ
3. カテゴリカルを直接渡す際の注意点を説明せよ

問3 — leaf-wise vs level-wise の挙動確認

1. num_leaves を [8, 31, 64, 128, 256] で変化させながらCV精度と過学習度を記録せよ
2. num_leaves が大きいと何が起こるか説明せよ
3. max_depth=-1 がデフォルトである理由を leaf-wise の観点から説明せよ

問4 — 主要パラメータの効果実験

1. learning_rate: [0.001, 0.01, 0.05, 0.1, 0.3]（n_estimators=200固定）でCV精度を確認せよ
2. min_child_samples: [5, 20, 50, 100, 200] で過学習度がどう変わるか確認せよ
3. subsample: [0.5, 0.6, 0.8, 1.0] で変化させよ（subsample_freq=1 も忘れずに）

問5 — 特徴量重要度と大規模データでの速度比較

1. LightGBMの feature_importances_（importance_type='gain'）を取得・表示せよ
2. n=100,000 のデータを生成し、XGBoost と LightGBM の学習時間を比較せよ
3. 大規模データで LightGBM が速い理由を2点述べよ

💡 ヒント

📋 コピーimport lightgbm as lgb

# sklearn API（XGBoostと同じ使い方）
clf_lgb = lgb.LGBMClassifier(
    n_estimators=200,
    random_state=42,
    verbose=-1  # ログ非表示
)

# カテゴリカル直接処理
features_cat = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
X_cat = data[features_cat].copy()
X_cat['Sex']      = X_cat['Sex'].astype('category')
X_cat['Embarked'] = X_cat['Embarked'].astype('category')

clf_lgb.fit(X_cat, y)
cv_s = cross_val_score(clf_lgb, X_cat, y, cv=cv, scoring='accuracy').mean()
print(f"カテゴリカル直接: CV={cv_s:.4f}")

📋 コピー# 問3: num_leaves の効果
for n_leaves in [8, 31, 64, 128, 256]:
    clf = lgb.LGBMClassifier(
        n_estimators=200, num_leaves=n_leaves,
        random_state=42, verbose=-1
    )
    clf.fit(X_enc, y)
    tr   = clf.score(X_enc, y)
    cv_s = cross_val_score(clf, X_enc, y, cv=cv, scoring='accuracy').mean()
    print(f"num_leaves={n_leaves:3d}: 訓練={tr:.4f}, CV={cv_s:.4f}, 差={tr-cv_s:.4f}")

# 問5: 大規模データ速度比較
n_large = 100_000
X_large = np.random.randn(n_large, 7)
y_large = (X_large[:, 0] + X_large[:, 1] > 0).astype(int)

for name, clf in [
    ("XGBoost ", xgb.XGBClassifier(n_estimators=200, eval_metric='logloss', random_state=42)),
    ("LightGBM", lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1)),
]:
    t0 = time.time()
    clf.fit(X_large, y_large)
    print(f"[{name}] 学習時間: {time.time()-t0:.2f}s")

✅ 模範解答

📋 コピーimport numpy as np
import pandas as pd
import time
import lightgbm as lgb
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.preprocessing import LabelEncoder
import warnings
warnings.filterwarnings('ignore')

# ── データ準備 ──
np.random.seed(42)
n = 10000
data = pd.DataFrame({
    'Pclass':   np.random.choice([1, 2, 3], n, p=[0.24, 0.21, 0.55]),
    'Sex':      np.random.choice(['male', 'female'], n, p=[0.65, 0.35]),
    'Age':      np.random.normal(29.7, 14.5, n).clip(1, 80),
    'SibSp':    np.random.choice(range(9), n, p=[0.68,0.23,0.04,0.02,0.01,0.01,0.005,0.005,0.01]),
    'Parch':    np.random.choice(range(7), n, p=[0.76,0.13,0.09,0.005,0.005,0.005,0.005]),
    'Fare':     np.abs(np.random.normal(32, 50, n)),
    'Embarked': np.random.choice(['S', 'C', 'Q'], n, p=[0.72, 0.19, 0.09]),
})
data['Survived'] = (
    (data['Sex'] == 'female') * 0.55 + (data['Pclass'] == 1) * 0.25 +
    (data['Age'] < 16) * 0.2 + np.random.normal(0, 0.12, n)
) > 0.4
data['Survived'] = data['Survived'].astype(int)
data['Sex_enc']      = LabelEncoder().fit_transform(data['Sex'])
data['Embarked_enc'] = LabelEncoder().fit_transform(data['Embarked'].fillna('S'))
features_enc = ['Pclass', 'Sex_enc', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked_enc']
X_enc = data[features_enc].values
y     = data['Survived'].values
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# ── 問1: XGBoost vs LightGBM 基本比較 ──
print("=" * 65)
print("問1: XGBoost vs LightGBM 基本比較 (n=10000, n_estimators=200)")
print("=" * 65)
for name, clf in [
    ("XGBoost ", xgb.XGBClassifier(n_estimators=200, eval_metric='logloss', random_state=42)),
    ("LightGBM", lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1)),
]:
    t0 = time.time()
    clf.fit(X_enc, y)
    tr    = clf.score(X_enc, y)
    t_fit = time.time() - t0
    cv_s  = cross_val_score(clf, X_enc, y, cv=cv, scoring='accuracy').mean()
    print(f"[{name}] 訓練:{tr:.4f}, CV:{cv_s:.4f}, 過学習:{tr-cv_s:.4f}, fit:{t_fit:.2f}s")

# ── 問2: カテゴリカル変数の直接処理 ──
print("\n" + "=" * 65)
print("問2: カテゴリカル変数の直接処理")
print("=" * 65)
features_cat = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
X_cat = data[features_cat].copy()
X_cat['Sex']      = X_cat['Sex'].astype('category')
X_cat['Embarked'] = X_cat['Embarked'].astype('category')

cv_cat = cross_val_score(lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1),
                          X_cat, y, cv=cv, scoring='accuracy').mean()
cv_enc = cross_val_score(lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1),
                          X_enc, y, cv=cv, scoring='accuracy').mean()
print(f"  カテゴリカル直接処理: CV={cv_cat:.4f}")
print(f"  ラベルエンコード済み: CV={cv_enc:.4f}")
print(f"  差: {cv_cat - cv_enc:+.4f}")

# ── 問3: num_leaves の効果 ──
print("\n" + "=" * 65)
print("問3: num_leaves の効果（leaf-wise 成長）")
print("=" * 65)
for n_leaves in [8, 31, 64, 128, 256]:
    clf = lgb.LGBMClassifier(n_estimators=200, num_leaves=n_leaves,
                              random_state=42, verbose=-1)
    clf.fit(X_enc, y)
    tr   = clf.score(X_enc, y)
    cv_s = cross_val_score(clf, X_enc, y, cv=cv, scoring='accuracy').mean()
    star = " ★デフォルト" if n_leaves == 31 else ""
    print(f"  num_leaves={n_leaves:3d}: 訓練={tr:.4f}, CV={cv_s:.4f}, 過学習={tr-cv_s:.4f}{star}")

# ── 問4: 主要パラメータ実験 ──
print("\n" + "=" * 65)
print("問4: 主要パラメータの効果")
print("=" * 65)

print("  [4-1] learning_rate:")
for lr in [0.001, 0.01, 0.05, 0.1, 0.3]:
    cv_s = cross_val_score(
        lgb.LGBMClassifier(n_estimators=200, learning_rate=lr, random_state=42, verbose=-1),
        X_enc, y, cv=cv, scoring='accuracy').mean()
    bar = "█" * int(cv_s * 40)
    print(f"    lr={lr:.3f}: CV={cv_s:.4f} {bar}")

print("  [4-2] min_child_samples:")
for mcs in [5, 20, 50, 100, 200]:
    clf = lgb.LGBMClassifier(n_estimators=200, min_child_samples=mcs,
                              random_state=42, verbose=-1)
    clf.fit(X_enc, y)
    tr   = clf.score(X_enc, y)
    cv_s = cross_val_score(clf, X_enc, y, cv=cv, scoring='accuracy').mean()
    print(f"    min_child_samples={mcs:3d}: 訓練={tr:.4f}, CV={cv_s:.4f}")

print("  [4-3] subsample:")
for ss in [0.5, 0.6, 0.8, 1.0]:
    freq = 1 if ss < 1.0 else 0
    cv_s = cross_val_score(
        lgb.LGBMClassifier(n_estimators=200, subsample=ss, subsample_freq=freq,
                            random_state=42, verbose=-1),
        X_enc, y, cv=cv, scoring='accuracy').mean()
    print(f"    subsample={ss}: CV={cv_s:.4f}")

# ── 問5: 特徴量重要度と大規模データ速度比較 ──
print("\n" + "=" * 65)
print("問5: 特徴量重要度 + 大規模データ速度比較")
print("=" * 65)
clf_lgb = lgb.LGBMClassifier(n_estimators=200, random_state=42,
                               verbose=-1, importance_type='gain')
clf_lgb.fit(X_enc, y)
print("  特徴量重要度 (gain):")
for f, imp in sorted(zip(features_enc, clf_lgb.feature_importances_), key=lambda x: -x[1]):
    bar = "█" * int(imp / max(clf_lgb.feature_importances_) * 30)
    print(f"    {f:12s}: {imp:8.1f} {bar}")

n_large = 100_000
X_large = np.random.randn(n_large, 7)
y_large = (X_large[:, 0] + X_large[:, 1] > 0).astype(int)
print("\n  大規模データ (n=100,000) 学習時間:")
for name, clf in [
    ("XGBoost ", xgb.XGBClassifier(n_estimators=200, eval_metric='logloss', random_state=42)),
    ("LightGBM", lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1)),
]:
    t0 = time.time()
    clf.fit(X_large, y_large)
    print(f"    [{name}] {time.time()-t0:.2f}s")

🪜 Step-by-Step 解説

1leaf-wise 成長の直感的理解

📋 コピー# leaf-wise を手動でシミュレーション
# 「最も損失を下げる葉を選んで分割」を確認する

import numpy as np

# 仮想: 3枚の葉の残差（二乗誤差）
leaves = {
    'leaf_A': {'n': 100, 'residual_sq': 500.0},  # 損失大
    'leaf_B': {'n': 50,  'residual_sq':  80.0},  # 損失中
    'leaf_C': {'n': 200, 'residual_sq': 120.0},  # 損失小
}

print("現在の葉の損失:")
for name, leaf in leaves.items():
    print(f"  {name}: n={leaf['n']}, 損失={leaf['residual_sq']:.1f}")

# leaf-wise: 損失が最大の leaf_A を選ぶ（level-wiseは全部分割する）
best_leaf = max(leaves, key=lambda k: leaves[k]['residual_sq'])
print(f"\n→ leaf-wise が選ぶのは: {best_leaf}（損失が最大）")
print("  level-wise はこのラウンドで3枚全部を分割する（効率が悪い）")

# num_leaves の意味
print("\num_leaves の意味:")
print("  num_leaves=31: 最大31枚の葉まで成長できる")
print("  num_leaves=8:  最大8枚 → 単純なモデル（アンダーフィット気味）")
print("  num_leaves=256: 最大256枚 → 複雑なモデル（オーバーフィット気味）")

2カテゴリカル変数の最適分割を確認

📋 コピーimport pandas as pd
import lightgbm as lgb
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np

# LightGBMのカテゴリカル最適分割を確認
# カテゴリカルをターゲットでソートして最良の二分割を探す仕組み

np.random.seed(42)
n = 1000
df = pd.DataFrame({
    'Pclass': np.random.choice([1,2,3], n),
    'Sex':    np.random.choice(['male','female'], n, p=[0.65,0.35]),
    'Age':    np.random.normal(29.7, 14.5, n).clip(1, 80),
})
df['Survived'] = ((df['Sex'] == 'female') * 0.55 + (df['Pclass'] == 1) * 0.3
                  + np.random.normal(0, 0.1, n)) > 0.4
df['Survived'] = df['Survived'].astype(int)
y = df['Survived'].values

# 文字列: エラーになる
# lgb.LGBMClassifier().fit(df[['Sex']], y)  # ValueError

# dtype='category': 正常に動く
df_cat = df[['Pclass', 'Sex', 'Age']].copy()
df_cat['Sex'] = df_cat['Sex'].astype('category')

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_s = cross_val_score(
    lgb.LGBMClassifier(n_estimators=100, random_state=42, verbose=-1),
    df_cat, y, cv=cv, scoring='accuracy').mean()
print(f"カテゴリカル直接処理: CV={cv_s:.4f}")

# category コードの確認
print(f"\nSexのカテゴリ: {df_cat['Sex'].cat.categories.tolist()}")
print(f"内部コード例: {df_cat['Sex'].cat.codes.head(5).tolist()}")

3early_stopping の使い方（LightGBM版）

📋 コピーimport lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X_train, X_val, y_train, y_val = train_test_split(
    X_enc, y, test_size=0.2, random_state=42, stratify=y
)

# sklearn API での early_stopping（LightGBM 4.x 以降）
clf = lgb.LGBMClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    num_leaves=31,
    callbacks=[
        lgb.early_stopping(stopping_rounds=30),
        lgb.log_evaluation(period=-1),   # ログ非表示
    ],
    random_state=42,
    verbose=-1
)
clf.fit(X_train, y_train, eval_set=[(X_val, y_val)])
print(f"最適な n_estimators: {clf.best_iteration_}")
print(f"バリデーション精度: {accuracy_score(y_val, clf.predict(X_val)):.4f}")

4Kaggle標準 LightGBM テンプレート

📋 コピーfrom sklearn.model_selection import cross_val_predict
from sklearn.metrics import accuracy_score

lgb_params = {
    'n_estimators':      1000,
    'learning_rate':     0.05,
    'num_leaves':        31,
    'min_child_samples': 20,
    'subsample':         0.8,
    'subsample_freq':    1,     # subsampleを毎ラウンド適用（必須）
    'colsample_bytree':  0.8,
    'reg_alpha':         0.1,
    'reg_lambda':        1.0,
    'random_state':      42,
    'verbose':          -1,
    'n_jobs':           -1,
}

clf = lgb.LGBMClassifier(**lgb_params)
cv  = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof = cross_val_predict(clf, X_enc, y, cv=cv)
print(f"OOF精度: {accuracy_score(y, oof):.4f}")

clf.fit(X_enc, y)
for f, imp in sorted(zip(features_enc, clf.feature_importances_), key=lambda x: -x[1]):
    print(f"  {f}: {imp:.1f}")

📐 数学・統計の補足（文系向け）

ヒストグラムアルゴリズムを試験で例える

🏆 Kaggleでの実践的な使い方

📋 コピー# Optuna で LightGBM をチューニングする例
import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    params = {
        'n_estimators':      trial.suggest_int('n_estimators', 100, 1000),
        'learning_rate':     trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'num_leaves':        trial.suggest_int('num_leaves', 10, 100),
        'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
        'subsample':         trial.suggest_float('subsample', 0.5, 1.0),
        'subsample_freq':    1,
        'colsample_bytree':  trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'random_state': 42, 'verbose': -1,
    }
    clf = lgb.LGBMClassifier(**params)
    return cross_val_score(clf, X_enc, y, cv=cv, scoring='accuracy').mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)
print(f"最良スコア: {study.best_value:.4f}")
print(f"最良パラメータ: {study.best_params}")

🚫 よくある誤解・ミス

🚀 次のステップ

• 発展: CatBoostを試す（カテゴリカル処理がさらに強力・ordered boosting）。early_stopping の callbacks API を使う
• 次回予告（Day 063）: CatBoost 入門 — 順序付きブースティング・カテゴリカル処理の仕組み・3大GBDTの最終比較（XGBoost / LightGBM / CatBoost）

📋 自己評価（解いた後に記入）