{
"cells": [
{
"cell_type": "markdown",
"id": "c2be1906-f144-410e-b65e-35538dded4f7",
"metadata": {},
"source": [
"# Learning Rate Schedulers\n",
"\n",
"**Welcome**\n",
"\n",
"Welcome to the Learning Rate Schedulers tutorial. Learning rate schedulers can help us dynamically adjust the learning rate of the Adam optimization algorithm. That way, we can decrease the learning rate as we approach the minima of the cost function.\n",
"\n",
"Run this notebook on Google Colab:\n",
"\n",
"[](https://colab.research.google.com/github/AG-Peter/encodermap/blob/main/tutorials/notebooks_customization/04_learning_rate_schedulers.ipynb)\n",
"\n",
"Find the documentation of EncoderMap:\n",
"\n",
"https://ag-peter.github.io/encodermap\n",
"\n",
"**Goals:**\n",
"\n",
"In this tutorial you will learn:\n",
"\n",
"* [Why we can profit from learning rate schedulers](#why)\n",
"* [How to log the current learning rate to TensorBoard](#log_to_tb)\n",
"* [How to implement a learning rate scheduler with an exponentially decaying learning rate](#lr_implementation)"
]
},
{
"cell_type": "markdown",
"id": "65568a51-1a34-4727-938b-955e135f94ce",
"metadata": {},
"source": [
"### For Google colab only:\n",
"\n",
"If you're on Google colab, please uncomment these lines and install EncoderMap."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "408f06e7",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:15.540794Z",
"iopub.status.busy": "2024-12-29T12:56:15.540426Z",
"iopub.status.idle": "2024-12-29T12:56:15.543339Z",
"shell.execute_reply": "2024-12-29T12:56:15.542642Z"
}
},
"outputs": [],
"source": [
"# !wget https://gist.githubusercontent.com/kevinsawade/deda578a3c6f26640ae905a3557e4ed1/raw/b7403a37710cb881839186da96d4d117e50abf36/install_encodermap_google_colab.sh\n",
"# !sudo bash install_encodermap_google_colab.sh"
]
},
{
"cell_type": "markdown",
"id": "016b78d1",
"metadata": {},
"source": [
"If you're on Google Colab, you also want to download the data we will use:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "16654191",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:15.545091Z",
"iopub.status.busy": "2024-12-29T12:56:15.544991Z",
"iopub.status.idle": "2024-12-29T12:56:15.546994Z",
"shell.execute_reply": "2024-12-29T12:56:15.546688Z"
}
},
"outputs": [],
"source": [
"# !wget https://raw.githubusercontent.com/AG-Peter/encodermap/main/tutorials/notebooks_starter/asp7.csv"
]
},
{
"cell_type": "markdown",
"id": "43decefb",
"metadata": {},
"source": [
"## Import Libraries\n",
"\n",
"Before we can start exploring the learning rate scheduler, we need to import some libraries."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c6be3e13",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:15.548627Z",
"iopub.status.busy": "2024-12-29T12:56:15.548495Z",
"iopub.status.idle": "2024-12-29T12:56:19.262440Z",
"shell.execute_reply": "2024-12-29T12:56:19.261663Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/kevin/git/encoder_map_private/encodermap/__init__.py:194: GPUsAreDisabledWarning: EncoderMap disables the GPU per default because most tensorflow code runs with a higher compatibility when the GPU is disabled. If you want to enable GPUs manually, set the environment variable 'ENCODERMAP_ENABLE_GPU' to 'True' before importing EncoderMap. To do this in python you can run:\n",
"\n",
"import os; os.environ['ENCODERMAP_ENABLE_GPU'] = 'True'\n",
"\n",
"before importing encodermap.\n",
" _warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "83407c56601845439856f7e7f528fa77",
"version_major": 2,
"version_minor": 0
},
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import os\n",
"import numpy as np\n",
"import encodermap as em\n",
"import tensorflow as tf\n",
"import pandas as pd\n",
"from pathlib import Path\n",
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "markdown",
"id": "05def998",
"metadata": {},
"source": [
"We wil work in the directory `runs/lr_scheduler`. We will create it now."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "234a1751-d9f5-4e8c-8a7f-cef281c2117c",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:19.264779Z",
"iopub.status.busy": "2024-12-29T12:56:19.264426Z",
"iopub.status.idle": "2024-12-29T12:56:19.286073Z",
"shell.execute_reply": "2024-12-29T12:56:19.285209Z"
}
},
"outputs": [],
"source": [
"(Path.cwd() / \"runs/lr_scheduler\").mkdir(parents=True, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"id": "674b6f11-95d6-4634-ba56-be87a5fb6677",
"metadata": {},
"source": [
"\n",
"\n",
"## Why learning rate schedulers? A linear regression example"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "809d7c7e-f9ed-4b79-beda-3293c96263c9",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "c1063547",
"metadata": {},
"source": [
"\n",
"\n",
"## Log the current learning rate to Tensorboard\n",
"\n",
"Before we implement some dynamic learning rates we want to find a way to log the learning rate to tensorboard.\n",
"\n",
"### Running tensorboard on Google colab\n",
"\n",
"To use tensorboard in google colabs notebooks, you neet to first load the tensorboard extension\n",
"\n",
"```python\n",
"%load_ext tensorboard\n",
"```\n",
"\n",
"And then activate it with:\n",
"\n",
"```python\n",
"%tensorboard --logdir .\n",
"```\n",
"\n",
"The next code cell contains these commands. Uncomment them and then continue.\n",
"\n",
"### Running tensorboard locally\n",
"\n",
"TensorBoard is a visualization tool from the machine learning library TensorFlow which is used by the EncoderMap package. During the dimensionality reduction step, when the neural network autoencoder is trained, several readings are saved in a TensorBoard format. All output files are saved to the path defined in `parameters.main_path`. Navigate to this location in a shell and start TensorBoard. Change the paramter Tensorboard to `True` to make Encodermap log to Tensorboard.\n",
"\n",
"In case you run this tutorial in the provided Docker container you can open a new console inside the container by typing the following command in a new system shell.\n",
"```shell\n",
"docker exec -it emap bash\n",
"```\n",
"Navigate to the location where all the runs are saved. e.g.:\n",
"```shell\n",
"cd notebooks_easy/runs/asp7/\n",
"```\n",
"Start TensorBoard in this directory with:\n",
"```shell\n",
"tensorboard --logdir .\n",
"```\n",
"\n",
"You should now be able to open TensorBoard in your webbrowser on port 6006. \n",
"`0.0.0.0:6006` or `127.0.0.1:6006`\n",
"\n",
"In the SCALARS tab of TensorBoard you should see among other values the overall cost and different contributions to the cost. The two most important contributions are `auto_cost` and `distance_cost`. `auto_cost` indicates differences between the inputs and outputs of the autoencoder. `distance_cost` is the part of the cost function which compares pairwise distances in the input space and the low-dimensional (latent) space.\n",
"\n",
"**Fixing Reloading issues**\n",
"Using Tensorboard we often encountered some issues while training multiple models and writing mutliple runs to Tensorboard's logdir. Reloading the data and event refreshing the web page did not display the data of the current run. We needed to kill tensorboard and restart it in order to see the new data. This issue was fixed by setting `reload_multifile` `True`.\n",
"\n",
"```bash\n",
"tensorboard --logdir . --reload_multifile True\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "dbd7f3c8-2502-4c18-a40e-78cbf138909d",
"metadata": {},
"source": [
"**When you're on Goole Colab, you can load the Tensorboard extension with:**"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "7b5bcabb",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:19.288031Z",
"iopub.status.busy": "2024-12-29T12:56:19.287899Z",
"iopub.status.idle": "2024-12-29T12:56:19.307782Z",
"shell.execute_reply": "2024-12-29T12:56:19.307194Z"
}
},
"outputs": [],
"source": [
"# %load_ext tensorboard\n",
"# %tensorboard --logdir ."
]
},
{
"cell_type": "markdown",
"id": "abe6f9ac-3f9f-4a63-b3da-eeb95ae97602",
"metadata": {},
"source": [
"### Sublcassing EncoderMap's `EncoderMapBaseCallback`\n",
"\n",
"The easiest way to implement and log a new variable to TensorBorard is by subclassing EncoderMap's `EncodeMapBaseCallback` from the `callbacks` submodule."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e188df5c-47a3-4bca-86d8-cb8911d31fea",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:19.309556Z",
"iopub.status.busy": "2024-12-29T12:56:19.309428Z",
"iopub.status.idle": "2024-12-29T12:56:19.346237Z",
"shell.execute_reply": "2024-12-29T12:56:19.345709Z"
}
},
"outputs": [],
"source": [
"?em.callbacks.EncoderMapBaseCallback"
]
},
{
"cell_type": "markdown",
"id": "563c9b43",
"metadata": {},
"source": [
"As per the docstring of the `EncoderMapBaseCallback` class, we create the `LearningRateLogger` class and implement a piece of code in the `on_summary_step` method."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "3c34f6e7",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:19.348257Z",
"iopub.status.busy": "2024-12-29T12:56:19.348126Z",
"iopub.status.idle": "2024-12-29T12:56:19.368466Z",
"shell.execute_reply": "2024-12-29T12:56:19.367959Z"
}
},
"outputs": [],
"source": [
"class LearningRateLogger(em.callbacks.EncoderMapBaseCallback):\n",
" def on_summary_step(self, step, logs=None):\n",
" with tf.name_scope(\"Learning Rate\"):\n",
" tf.summary.scalar('current learning rate', self.model.optimizer.lr, step=step)"
]
},
{
"cell_type": "markdown",
"id": "9f53d462",
"metadata": {},
"source": [
"We can now create an `EncoderMap` class and add our new callback with the `add_callback` method."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "9bf99bcc",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:19.370301Z",
"iopub.status.busy": "2024-12-29T12:56:19.370189Z",
"iopub.status.idle": "2024-12-29T12:56:19.589619Z",
"shell.execute_reply": "2024-12-29T12:56:19.588583Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output files are saved to runs/lr_scheduler/run0 as defined in 'main_path' in the parameters.\n",
"Saved a text-summary of the model and an image in runs/lr_scheduler/run0, as specified in 'main_path' in the parameters.\n"
]
}
],
"source": [
"df = pd.read_csv('asp7.csv')\n",
"dihedrals = df.iloc[:,:-1].values.astype(np.float32)\n",
"cluster_ids = df.iloc[:,-1].values\n",
"\n",
"parameters = em.Parameters(\n",
"tensorboard=True,\n",
"periodicity=2*np.pi,\n",
"main_path=em.misc.run_path('runs/lr_scheduler'),\n",
"n_steps=100,\n",
"summary_step=5\n",
")\n",
"\n",
"# create an instance of EncoderMap\n",
"e_map = em.EncoderMap(parameters, dihedrals)\n",
"\n",
"# Add an instance of the new Callback\n",
"e_map.add_callback(LearningRateLogger)"
]
},
{
"cell_type": "markdown",
"id": "e214e7f1",
"metadata": {},
"source": [
"We train the Model."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "96569684",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:19.591820Z",
"iopub.status.busy": "2024-12-29T12:56:19.591621Z",
"iopub.status.idle": "2024-12-29T12:56:23.469511Z",
"shell.execute_reply": "2024-12-29T12:56:23.469112Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 0%| | 0/100 [00:00\n",
"\n",
"A constant learning rate of 0.001"
]
},
{
"cell_type": "markdown",
"id": "598068c0",
"metadata": {},
"source": [
"\n",
"\n",
"## Write a learning rate scheduler\n",
"\n",
"We can write a learning rate scheduler either by providing intervals of training steps and the associated learning rate:\n",
"\n",
"```python\n",
"def lr_schedule(step):\n",
" \"\"\"\n",
" Returns a custom learning rate that decreases as steps progress.\n",
" \"\"\"\n",
" learning_rate = 0.2\n",
" if step > 10:\n",
" learning_rate = 0.02\n",
" if step > 20:\n",
" learning_rate = 0.01\n",
" if step > 50:\n",
" learning_rate = 0.005\n",
"```\n",
"\n",
"Or by using a function that gives us a learning rate:\n",
"\n",
"```python\n",
"def scheduler(step, lr=1, n_steps=1000):\n",
" \"\"\"\n",
" Returns a custom learning rate that decreases based on an exp function as steps progress.\n",
" \"\"\"\n",
" if step < 10:\n",
" return lr\n",
" else:\n",
" return lr * tf.math.exp(-step / n_steps)\n",
"```\n",
"\n",
"Below, is an example combining both:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "1a37f62f",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:23.471558Z",
"iopub.status.busy": "2024-12-29T12:56:23.471416Z",
"iopub.status.idle": "2024-12-29T12:56:23.494937Z",
"shell.execute_reply": "2024-12-29T12:56:23.494439Z"
}
},
"outputs": [],
"source": [
"def scheduler(step, lr=1):\n",
" \"\"\"\n",
" Returns a custom learning rate that decreases based on an exp function as steps progress.\n",
" \"\"\"\n",
" if step < 10:\n",
" return lr\n",
" else:\n",
" return lr * tf.math.exp(-0.1)"
]
},
{
"cell_type": "markdown",
"id": "a848aa1d",
"metadata": {},
"source": [
"This scheduler function can simply be provided to the builtin `keras.callbacks.LearningRateScheduler` callback."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "332faaec",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:23.496615Z",
"iopub.status.busy": "2024-12-29T12:56:23.496491Z",
"iopub.status.idle": "2024-12-29T12:56:23.517579Z",
"shell.execute_reply": "2024-12-29T12:56:23.516848Z"
}
},
"outputs": [],
"source": [
"callback = tf.keras.callbacks.LearningRateScheduler(scheduler)"
]
},
{
"cell_type": "markdown",
"id": "e5f22dc5",
"metadata": {},
"source": [
"And appended to the list of `callbacks` in the EncoderMap class."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "9087259e",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:23.519613Z",
"iopub.status.busy": "2024-12-29T12:56:23.519498Z",
"iopub.status.idle": "2024-12-29T12:56:23.688602Z",
"shell.execute_reply": "2024-12-29T12:56:23.688124Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output files are saved to runs/lr_scheduler/run1 as defined in 'main_path' in the parameters.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Saved a text-summary of the model and an image in runs/lr_scheduler/run1, as specified in 'main_path' in the parameters.\n"
]
}
],
"source": [
"parameters = em.Parameters(\n",
"tensorboard=True,\n",
"periodicity=2*np.pi,\n",
"main_path=em.misc.run_path('runs/lr_scheduler'),\n",
"n_steps=50,\n",
"summary_step=1\n",
")\n",
"\n",
"e_map = em.EncoderMap(parameters, dihedrals)\n",
"e_map.add_callback(LearningRateLogger)\n",
"e_map.add_callback(callback)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "6b026be4",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:23.690845Z",
"iopub.status.busy": "2024-12-29T12:56:23.690702Z",
"iopub.status.idle": "2024-12-29T12:56:26.958840Z",
"shell.execute_reply": "2024-12-29T12:56:26.958236Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 0%| | 0/50 [00:00"
]
},
{
"cell_type": "markdown",
"id": "2372c499-3c60-420c-9e3f-c0f67a08e268",
"metadata": {},
"source": [
"And here's the learning rate plotted from the history."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "669a5b1b-4ea2-4652-a039-0ce23381ef4a",
"metadata": {
"execution": {
"iopub.execute_input": "2024-12-29T12:56:26.960835Z",
"iopub.status.busy": "2024-12-29T12:56:26.960632Z",
"iopub.status.idle": "2024-12-29T12:56:27.345400Z",
"shell.execute_reply": "2024-12-29T12:56:27.344833Z"
}
},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"hovertemplate": "variable=0
index=%{x}
value=%{y}",
"legendgroup": "0",
"line": {
"color": "#636efa",
"dash": "solid"
},
"marker": {
"symbol": "circle"
},
"mode": "lines",
"name": "0",
"orientation": "v",
"showlegend": true,
"type": "scatter",
"x": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49
],
"xaxis": "x",
"y": [
0.001,
0.001,
0.001,
0.001,
0.001,
0.001,
0.001,
0.001,
0.001,
0.001,
0.00090483745,
0.0008187308,
0.0007408183,
0.0006703201,
0.00060653075,
0.00054881175,
0.0004965854,
0.00044932903,
0.0004065697,
0.0003678795,
0.00033287113,
0.00030119426,
0.00027253185,
0.00024659702,
0.0002231302,
0.00020189656,
0.00018268357,
0.00016529893,
0.00014956866,
0.00013533531,
0.00012245646,
0.000110803194,
0.000100258876,
9.0717986e-05,
8.2085025e-05,
7.42736e-05,
6.720553e-05,
6.081008e-05,
5.502324e-05,
4.9787086e-05,
4.504922e-05,
4.076222e-05,
3.6883182e-05,
3.3373282e-05,
3.0197394e-05,
2.7323733e-05,
2.4723537e-05,
2.2370781e-05,
2.024192e-05,
1.8315646e-05
],
"yaxis": "y"
}
],
"layout": {
"legend": {
"title": {
"text": "variable"
},
"tracegroupgap": 0
},
"margin": {
"t": 60
},
"template": {
"data": {
"bar": [
{
"error_x": {
"color": "#2a3f5f"
},
"error_y": {
"color": "#2a3f5f"
},
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
},
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "bar"
}
],
"barpolar": [
{
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
},
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "barpolar"
}
],
"carpet": [
{
"aaxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"baxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"type": "carpet"
}
],
"choropleth": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "choropleth"
}
],
"contour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"type": "contour"
}
],
"contourcarpet": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "contourcarpet"
}
],
"heatmap": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"type": "heatmap"
}
],
"heatmapgl": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"type": "heatmapgl"
}
],
"histogram": [
{
"marker": {
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "histogram"
}
],
"histogram2d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"type": "histogram2d"
}
],
"histogram2dcontour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"type": "histogram2dcontour"
}
],
"mesh3d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "mesh3d"
}
],
"parcoords": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "parcoords"
}
],
"pie": [
{
"automargin": true,
"type": "pie"
}
],
"scatter": [
{
"fillpattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
},
"type": "scatter"
}
],
"scatter3d": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter3d"
}
],
"scattercarpet": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattercarpet"
}
],
"scattergeo": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergeo"
}
],
"scattergl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergl"
}
],
"scattermapbox": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattermapbox"
}
],
"scatterpolar": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolar"
}
],
"scatterpolargl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolargl"
}
],
"scatterternary": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterternary"
}
],
"surface": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"type": "surface"
}
],
"table": [
{
"cells": {
"fill": {
"color": "#EBF0F8"
},
"line": {
"color": "white"
}
},
"header": {
"fill": {
"color": "#C8D4E3"
},
"line": {
"color": "white"
}
},
"type": "table"
}
]
},
"layout": {
"annotationdefaults": {
"arrowcolor": "#2a3f5f",
"arrowhead": 0,
"arrowwidth": 1
},
"autotypenumbers": "strict",
"coloraxis": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"colorscale": {
"diverging": [
[
0,
"#8e0152"
],
[
0.1,
"#c51b7d"
],
[
0.2,
"#de77ae"
],
[
0.3,
"#f1b6da"
],
[
0.4,
"#fde0ef"
],
[
0.5,
"#f7f7f7"
],
[
0.6,
"#e6f5d0"
],
[
0.7,
"#b8e186"
],
[
0.8,
"#7fbc41"
],
[
0.9,
"#4d9221"
],
[
1,
"#276419"
]
],
"sequential": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
],
"sequentialminus": [
[
0.0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1.0,
"#f0f921"
]
]
},
"colorway": [
"#636efa",
"#EF553B",
"#00cc96",
"#ab63fa",
"#FFA15A",
"#19d3f3",
"#FF6692",
"#B6E880",
"#FF97FF",
"#FECB52"
],
"font": {
"color": "#2a3f5f"
},
"geo": {
"bgcolor": "white",
"lakecolor": "white",
"landcolor": "#E5ECF6",
"showlakes": true,
"showland": true,
"subunitcolor": "white"
},
"hoverlabel": {
"align": "left"
},
"hovermode": "closest",
"mapbox": {
"style": "light"
},
"paper_bgcolor": "white",
"plot_bgcolor": "#E5ECF6",
"polar": {
"angularaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"radialaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"scene": {
"xaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"yaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"zaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
}
},
"shapedefaults": {
"line": {
"color": "#2a3f5f"
}
},
"ternary": {
"aaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"baxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"caxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"title": {
"x": 0.05
},
"xaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
},
"yaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
}
}
},
"xaxis": {
"anchor": "y",
"domain": [
0.0,
1.0
],
"title": {
"text": "index"
}
},
"yaxis": {
"anchor": "x",
"domain": [
0.0,
1.0
],
"title": {
"text": "value"
}
}
}
},
"text/html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import plotly.express as px\n",
"\n",
"px.line(history.history[\"lr\"])"
]
},
{
"cell_type": "markdown",
"id": "23fc18f6",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"Learning rate schedulers are helpful to prevent overtraining, but still slightly increase the predictive power of your NN model. EncoderMap's modularity allows for them to be simple Plug-In solutions."
]
}
],
"metadata": {
"emap": "run",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {
"3e07b669e4fa4f37b939b29f2b9c4bf6": {
"model_module": "ipycanvas",
"model_module_version": "^0.13",
"model_name": "CanvasManagerModel",
"state": {
"_model_module": "ipycanvas",
"_model_module_version": "^0.13",
"_model_name": "CanvasManagerModel",
"_view_count": null,
"_view_module": null,
"_view_module_version": "",
"_view_name": null
}
},
"83407c56601845439856f7e7f528fa77": {
"model_module": "nglview-js-widgets",
"model_module_version": "3.0.8",
"model_name": "ColormakerRegistryModel",
"state": {
"_dom_classes": [],
"_model_module": "nglview-js-widgets",
"_model_module_version": "3.0.8",
"_model_name": "ColormakerRegistryModel",
"_msg_ar": [],
"_msg_q": [],
"_ready": false,
"_view_count": null,
"_view_module": "nglview-js-widgets",
"_view_module_version": "3.0.8",
"_view_name": "ColormakerRegistryView",
"layout": "IPY_MODEL_df23820780134dacbbc539925e36fee9",
"tabbable": null,
"tooltip": null
}
},
"df23820780134dacbbc539925e36fee9": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}