Day 063 — CatBoost 入門 — 順序付きブースティング・カテゴリカル処理・3大GBDT最終比較

📚 背景知識（読んでから問題へ）

CatBoost が解決した2つの問題

🔬 3大GBDT 総合比較

🔄 Ordered Boosting — 情報リークを防ぐ仕組み

🏷️ Ordered Target Statistics — カテゴリカル処理の仕組み

CatBoostへのカテゴリカル変数の渡し方

🌲 対称木（Symmetric Tree）の構造

⚙️ 主要パラメータ

🗂️ データスキーマ（Titanic 拡張版 n=5,000）

📝 問題

セットアップコード（最初に実行）

📋 コピーimport numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.preprocessing import LabelEncoder
import time
import warnings
warnings.filterwarnings('ignore')

try:
    from catboost import CatBoostClassifier, Pool
    print("CatBoost: OK")
except ImportError:
    print("pip install catboost")

try:
    import lightgbm as lgb
    print("LightGBM: OK")
except ImportError:
    print("pip install lightgbm")

try:
    import xgboost as xgb
    print("XGBoost: OK")
except ImportError:
    print("pip install xgboost")

np.random.seed(42)
n = 5000
data = pd.DataFrame({
    'Pclass':     np.random.choice([1, 2, 3], n, p=[0.24, 0.21, 0.55]),
    'Sex':        np.random.choice(['male', 'female'], n, p=[0.65, 0.35]),
    'Age':        np.random.normal(29.7, 14.5, n).clip(1, 80),
    'SibSp':      np.random.choice(range(9), n, p=[0.68,0.23,0.04,0.02,0.01,0.01,0.005,0.005,0.01]),
    'Parch':      np.random.choice(range(7), n, p=[0.76,0.13,0.09,0.005,0.005,0.005,0.005]),
    'Fare':       np.abs(np.random.normal(32, 50, n)),
    'Embarked':   np.random.choice(['S', 'C', 'Q'], n, p=[0.72, 0.19, 0.09]),
    'Cabin_type': np.random.choice(['A','B','C','D','E','None'], n, p=[0.05,0.08,0.12,0.1,0.07,0.58]),
})
data['Survived'] = (
    (data['Sex'] == 'female') * 0.55 + (data['Pclass'] == 1) * 0.25 +
    (data['Age'] < 16) * 0.2 + np.random.normal(0, 0.12, n)
) > 0.4
data['Survived'] = data['Survived'].astype(int)

cat_features = ['Sex', 'Embarked', 'Cabin_type']
data_enc = data.copy()
for col in cat_features:
    data_enc[col] = LabelEncoder().fit_transform(data_enc[col])

features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked', 'Cabin_type']
X_raw = data[features]
X_enc = data_enc[features].values
y = data['Survived'].values
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

問1 — CatBoost の基本使い方（文字列のまま渡す）

1. CatBoostClassifier(iterations=200, verbose=0) を使い、cat_features=['Sex','Embarked','Cabin_type'] を指定して学習せよ
2. 文字列カテゴリカルのままデータを渡せることを確認せよ（エンコード不要）
3. CV精度と訓練精度を出力せよ

問2 — 3大GBDT（XGBoost / LightGBM / CatBoost）の総合比較

1. 学習時間（time.time() で計測）を比較せよ
2. CV精度（StratifiedKFold 5分割）を比較せよ
3. 訓練精度（過学習度 = 訓練精度 - CV精度）を比較せよ
4. どのモデルがこのデータセットに最も向いているか理由も述べよ

問3 — CatBoost の主要パラメータ実験

1. depth: [3, 5, 6, 8, 10] で CV精度と過学習度を確認せよ
2. learning_rate: [0.01, 0.05, 0.1, 0.3]（iterations=200 固定）で CV精度を確認せよ
3. l2_leaf_reg: [1, 3, 10, 30] で過学習度がどう変わるか確認せよ

問4 — Pool オブジェクトの使い方

1. Pool(X_raw, label=y, cat_features=['Sex','Embarked','Cabin_type']) を作成せよ
2. CatBoostClassifier.fit(pool) で学習せよ
3. get_feature_importance(pool) で特徴量重要度を取得・表示せよ
4. Pool を使う利点を2点説明せよ

問5 — Ordered vs Plain Boosting

1. boosting_type='Ordered' と boosting_type='Plain' を比較せよ
2. 小規模データ（n=200）と通常データ（n=5000）でどちらの差が大きいか確認せよ
3. なぜ小規模データで Ordered が有利かを説明せよ

💡 ヒント

📋 コピーfrom catboost import CatBoostClassifier, Pool

# 基本的な使い方
clf_cat = CatBoostClassifier(
    iterations=200,
    depth=6,
    learning_rate=0.1,
    l2_leaf_reg=3,
    cat_features=['Sex', 'Embarked', 'Cabin_type'],
    random_seed=42,
    verbose=0
)

# 文字列のまま渡せる！
clf_cat.fit(X_raw, y)
print(f"訓練精度: {clf_cat.score(X_raw, y):.4f}")

# Pool オブジェクト
pool = Pool(X_raw, label=y, cat_features=['Sex', 'Embarked', 'Cabin_type'])
clf_cat.fit(pool)
importances = clf_cat.get_feature_importance(pool)

📋 コピー# DataFrameにも配列にも対応した手動CV
from sklearn.metrics import accuracy_score

def manual_cv(clf, X, y, cv):
    scores, tr_scores = [], []
    for tr_idx, val_idx in cv.split(X, y):
        if isinstance(X, pd.DataFrame):
            X_tr, X_val = X.iloc[tr_idx], X.iloc[val_idx]
        else:
            X_tr, X_val = X[tr_idx], X[val_idx]
        y_tr, y_val = y[tr_idx], y[val_idx]
        clf.fit(X_tr, y_tr)
        scores.append(accuracy_score(y_val, clf.predict(X_val)))
        tr_scores.append(accuracy_score(y_tr, clf.predict(X_tr)))
    return np.mean(scores), np.mean(tr_scores)

# 問5: Ordered vs Plain
for size, label in [(200, "n=200"), (5000, "n=5000")]:
    idx = np.random.choice(n, size, replace=False)
    X_sub, y_sub = X_raw.iloc[idx], y[idx]
    for btype in ['Ordered', 'Plain']:
        cv_s, tr_s = manual_cv(
            CatBoostClassifier(iterations=200, boosting_type=btype,
                               cat_features=['Sex','Embarked','Cabin_type'],
                               random_seed=42, verbose=0),
            X_sub, y_sub, cv
        )
        print(f"[{label}] {btype}: CV={cv_s:.4f}, 過学習={tr_s-cv_s:.4f}")

✅ 模範解答

📋 コピーimport numpy as np
import pandas as pd
import time
from catboost import CatBoostClassifier, Pool
import lightgbm as lgb
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import accuracy_score
import warnings
warnings.filterwarnings('ignore')

# ── データ準備 ──
np.random.seed(42)
n = 5000
data = pd.DataFrame({
    'Pclass':     np.random.choice([1, 2, 3], n, p=[0.24, 0.21, 0.55]),
    'Sex':        np.random.choice(['male', 'female'], n, p=[0.65, 0.35]),
    'Age':        np.random.normal(29.7, 14.5, n).clip(1, 80),
    'SibSp':      np.random.choice(range(9), n, p=[0.68,0.23,0.04,0.02,0.01,0.01,0.005,0.005,0.01]),
    'Parch':      np.random.choice(range(7), n, p=[0.76,0.13,0.09,0.005,0.005,0.005,0.005]),
    'Fare':       np.abs(np.random.normal(32, 50, n)),
    'Embarked':   np.random.choice(['S', 'C', 'Q'], n, p=[0.72, 0.19, 0.09]),
    'Cabin_type': np.random.choice(['A','B','C','D','E','None'], n, p=[0.05,0.08,0.12,0.1,0.07,0.58]),
})
data['Survived'] = (
    (data['Sex'] == 'female') * 0.55 + (data['Pclass'] == 1) * 0.25 +
    (data['Age'] < 16) * 0.2 + np.random.normal(0, 0.12, n)
) > 0.4
data['Survived'] = data['Survived'].astype(int)

cat_features = ['Sex', 'Embarked', 'Cabin_type']
data_enc = data.copy()
for col in cat_features:
    data_enc[col] = LabelEncoder().fit_transform(data_enc[col])

features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked', 'Cabin_type']
X_raw = data[features]
X_enc = data_enc[features].values
y = data['Survived'].values
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

def manual_cv(clf, X, y, cv):
    scores, tr_scores = [], []
    for tr_idx, val_idx in cv.split(X, y):
        if isinstance(X, pd.DataFrame):
            X_tr, X_val = X.iloc[tr_idx], X.iloc[val_idx]
        else:
            X_tr, X_val = X[tr_idx], X[val_idx]
        y_tr, y_val = y[tr_idx], y[val_idx]
        clf.fit(X_tr, y_tr)
        scores.append(accuracy_score(y_val, clf.predict(X_val)))
        tr_scores.append(accuracy_score(y_tr, clf.predict(X_tr)))
    return np.mean(scores), np.mean(tr_scores)

# ── 問1: CatBoost 基本使い方 ──
print("=" * 65)
print("問1: CatBoost 基本使い方（文字列カテゴリカルをそのまま渡す）")
print("=" * 65)
clf_cat = CatBoostClassifier(
    iterations=200, depth=6, learning_rate=0.1,
    cat_features=['Sex', 'Embarked', 'Cabin_type'],
    random_seed=42, verbose=0
)
clf_cat.fit(X_raw, y)
tr = clf_cat.score(X_raw, y)
cv_s, _ = manual_cv(
    CatBoostClassifier(iterations=200, depth=6, learning_rate=0.1,
                       cat_features=['Sex','Embarked','Cabin_type'],
                       random_seed=42, verbose=0),
    X_raw, y, cv
)
print(f"  訓練精度: {tr:.4f}")
print(f"  CV精度:   {cv_s:.4f}")
print(f"  過学習度: {tr - cv_s:.4f}")
print("  → 文字列カテゴリカルをそのまま渡せることを確認!")

# ── 問2: 3大GBDT 総合比較 ──
print("\n" + "=" * 65)
print("問2: 3大GBDT 総合比較（XGBoost / LightGBM / CatBoost）")
print("=" * 65)
models = [
    ("XGBoost ", xgb.XGBClassifier(n_estimators=200, eval_metric='logloss', random_state=42), X_enc),
    ("LightGBM", lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1), X_enc),
    ("CatBoost", CatBoostClassifier(iterations=200, cat_features=['Sex','Embarked','Cabin_type'],
                                    random_seed=42, verbose=0), X_raw),
]
for name, clf, X_use in models:
    t0 = time.time()
    cv_s, tr_s = manual_cv(clf.__class__(**clf.get_params()), X_use, y, cv)
    t_cv = time.time() - t0
    clf.fit(X_use, y)
    tr = clf.score(X_use, y) if hasattr(clf, 'score') else accuracy_score(y, clf.predict(X_use))
    print(f"[{name}] CV={cv_s:.4f}, 訓練={tr:.4f}, 過学習={tr-cv_s:.4f}, 時間={t_cv:.2f}s")

# ── 問3: パラメータ実験 ──
print("\n" + "=" * 65)
print("問3: 主要パラメータ実験")
print("=" * 65)

print("  [3-1] depth:")
for d in [3, 5, 6, 8, 10]:
    cv_s, tr_s = manual_cv(
        CatBoostClassifier(iterations=200, depth=d,
                           cat_features=['Sex','Embarked','Cabin_type'],
                           random_seed=42, verbose=0),
        X_raw, y, cv
    )
    star = " ★デフォルト" if d == 6 else ""
    print(f"    depth={d:2d}: CV={cv_s:.4f}, 訓練={tr_s:.4f}, 過学習={tr_s-cv_s:.4f}{star}")

print("  [3-2] learning_rate:")
for lr in [0.01, 0.05, 0.1, 0.3]:
    cv_s, _ = manual_cv(
        CatBoostClassifier(iterations=200, learning_rate=lr,
                           cat_features=['Sex','Embarked','Cabin_type'],
                           random_seed=42, verbose=0),
        X_raw, y, cv
    )
    print(f"    lr={lr:.2f}: CV={cv_s:.4f}")

print("  [3-3] l2_leaf_reg:")
for l2 in [1, 3, 10, 30]:
    cv_s, tr_s = manual_cv(
        CatBoostClassifier(iterations=200, l2_leaf_reg=l2,
                           cat_features=['Sex','Embarked','Cabin_type'],
                           random_seed=42, verbose=0),
        X_raw, y, cv
    )
    star = " ★デフォルト" if l2 == 3 else ""
    print(f"    l2={l2:2d}: CV={cv_s:.4f}, 過学習={tr_s-cv_s:.4f}{star}")

# ── 問4: Pool ──
print("\n" + "=" * 65)
print("問4: Pool オブジェクト")
print("=" * 65)
pool = Pool(X_raw, label=y, cat_features=['Sex', 'Embarked', 'Cabin_type'])
clf_pool = CatBoostClassifier(iterations=200, random_seed=42, verbose=0)
clf_pool.fit(pool)
print(f"  Pool で学習完了: 訓練精度={clf_pool.score(pool):.4f}")
print("\n  特徴量重要度:")
importances = clf_pool.get_feature_importance(pool)
for feat, imp in sorted(zip(features, importances), key=lambda x: -x[1]):
    bar = "█" * int(imp / max(importances) * 30)
    print(f"    {feat:12s}: {imp:6.2f} {bar}")
print("\n  Pool の利点:")
print("    1. cat_features をデータとセットで管理（渡し忘れ防止）")
print("    2. 大規模データの前処理をPoolに任せて効率化")

# ── 問5: Ordered vs Plain ──
print("\n" + "=" * 65)
print("問5: Ordered vs Plain Boosting")
print("=" * 65)
for size, label in [(200, "小規模 n=200"), (5000, "通常 n=5000")]:
    idx = np.random.choice(n, size, replace=False)
    X_sub, y_sub = X_raw.iloc[idx], y[idx]
    cv_sub = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    results = {}
    for btype in ['Ordered', 'Plain']:
        cv_s, tr_s = manual_cv(
            CatBoostClassifier(iterations=200, boosting_type=btype,
                               cat_features=['Sex','Embarked','Cabin_type'],
                               random_seed=42, verbose=0),
            X_sub, y_sub, cv_sub
        )
        results[btype] = (cv_s, tr_s)
    print(f"\n  [{label}]")
    for btype, (cv_s, tr_s) in results.items():
        print(f"    {btype:7s}: CV={cv_s:.4f}, 過学習={tr_s-cv_s:.4f}")
    diff = results['Ordered'][0] - results['Plain'][0]
    print(f"    Ordered の優位性: {diff:+.4f}")
print("\n  考察: 小規模データほど Ordered が有利（n が小さいほど1点の情報漏洩の割合が大）")

🪜 Step-by-Step 解説

1CatBoost のインストールと基本確認

📋 コピー# pip install catboost

from catboost import CatBoostClassifier

# sklearn API は XGBoost / LightGBM と同じインターフェース
clf = CatBoostClassifier(
    iterations=200,      # n_estimators 相当
    depth=6,             # 対称木の深さ（デフォルト=6）
    learning_rate=0.1,
    l2_leaf_reg=3,       # L2正則化（デフォルト=3）
    cat_features=['Sex', 'Embarked'],
    random_seed=42,
    verbose=0            # ログ非表示（必須）
)

import pandas as pd
X_demo = pd.DataFrame({'Sex': ['male','female','male'], 'Age': [25, 30, 22]})
y_demo = [0, 1, 0]
clf_demo = CatBoostClassifier(iterations=10, cat_features=['Sex'], verbose=0)
clf_demo.fit(X_demo, y_demo)
print("文字列カテゴリカルをそのまま学習できた!")

2Pool の詳細な使い方

📋 コピーfrom catboost import Pool

pool_train = Pool(
    data=X_raw,
    label=y,
    cat_features=['Sex', 'Embarked', 'Cabin_type'],
)

clf = CatBoostClassifier(iterations=200, verbose=0)
clf.fit(pool_train)

preds = clf.predict(pool_train)
importances = clf.get_feature_importance(pool_train)
print("特徴量重要度:", importances)

3Ordered Boosting の直感をシミュレーション

📋 コピーimport numpy as np

np.random.seed(42)
n_small = 10
y_small = np.array([0,1,0,1,1,0,1,0,1,0])

# 通常のTarget Encoding（情報リーク）
target_mean_all = y_small.mean()
print("全データ平均（リークあり）:", target_mean_all)

# Ordered Target Statistics（リークなし）
print("\nOrdered Target Statistics（リークなし）:")
for i in range(n_small):
    if i == 0:
        ts = 0.5  # 最初は事前分布
    else:
        ts = y_small[:i].mean()
    print(f"  データ点{i}: y={y_small[i]}, TS={ts:.3f}")

43大GBDT アンサンブルの標準パターン

📋 コピーfrom sklearn.model_selection import cross_val_predict
from sklearn.metrics import accuracy_score

oof_xgb = cross_val_predict(xgb.XGBClassifier(n_estimators=200, random_state=42), X_enc, y, cv=cv)
oof_lgb = cross_val_predict(lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1), X_enc, y, cv=cv)

oof_cat = np.zeros(len(y))
cat_clf = CatBoostClassifier(iterations=200, cat_features=['Sex','Embarked','Cabin_type'],
                              random_seed=42, verbose=0)
for tr_idx, val_idx in cv.split(X_raw, y):
    cat_clf.fit(X_raw.iloc[tr_idx], y[tr_idx])
    oof_cat[val_idx] = cat_clf.predict(X_raw.iloc[val_idx])

oof_ensemble = ((oof_xgb + oof_lgb + oof_cat) >= 2).astype(int)
print(f"XGBoost OOF:   {accuracy_score(y, oof_xgb):.4f}")
print(f"LightGBM OOF:  {accuracy_score(y, oof_lgb):.4f}")
print(f"CatBoost OOF:  {accuracy_score(y, oof_cat):.4f}")
print(f"アンサンブル:   {accuracy_score(y, oof_ensemble):.4f}")

📐 数学・統計の補足（文系向け）

Target Statistics の計算式（数式を日本語で）

対称木を選挙で例える

🏆 Kaggleでの実践的な使い方

📋 コピー# Kaggle典型: Optuna で CatBoost をチューニング
import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    params = {
        'iterations':   trial.suggest_int('iterations', 100, 1000),
        'depth':        trial.suggest_int('depth', 4, 10),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'l2_leaf_reg':  trial.suggest_float('l2_leaf_reg', 1.0, 30.0, log=True),
        'cat_features': ['Sex', 'Embarked', 'Cabin_type'],
        'random_seed':  42,
        'verbose':      0,
    }
    clf = CatBoostClassifier(**params)
    cv_s, _ = manual_cv(clf, X_raw, y, cv)
    return cv_s

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=30)
print(f"最良スコア: {study.best_value:.4f}")
print(f"最良パラメータ: {study.best_params}")

🚫 よくある誤解・ミス

🚀 次のステップ

• 発展: Optuna で3大GBDTを同時チューニング。use_best_model=True, eval_set=... で CatBoost の early_stopping を活用する
• 次回予告（Day 064）: Optuna によるハイパーパラメータ最適化 — GBDTのチューニングを自動化する（XGBoost / LightGBM / CatBoost 対応）

📋 自己評価（解いた後に記入）