{ "cells": [ { "cell_type": "markdown", "id": "683108f1", "metadata": { "id": "683108f1" }, "source": [ "# SQL analysis with differential privacy guarantees\n", "[![View On GitHub](https://img.shields.io/badge/View_in_Github-grey?logo=github)](https://github.com/Qrlew/pyqrlew/blob/main/examples/rewrite_with_dp.ipynb)\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Qrlew/pyqrlew/blob/main/examples/rewrite_with_dp.ipynb)" ] }, { "cell_type": "markdown", "id": "c1cdff07", "metadata": { "id": "c1cdff07" }, "source": [ "The purpose of this notebook is to illustrate the process of *transforming a standard SQL query into a differentially private SQL query*.\n", "\n", "Our goal here is to demonstrate how this complex transformation of an SQL query into a differentially private version can be achieved with ease using Qrlew." ] }, { "cell_type": "markdown", "id": "ec43846d", "metadata": { "id": "ec43846d" }, "source": [ "The main idea of Qrlew is to parse and rewrite SQL queries into *Relations*: an [Intermediate Representation](https://en.wikipedia.org/wiki/Intermediate_representation) (IR) that is well-suited for various rewriting tasks.\n", "\n", "*Relations* may be of different kinds:\n", "- `Map`s typically represent the transform of an input *Relation* row by row, therefore, naturaly preserving the *privacy unit*,\n", "- `Reduce`s represent aggregations that can be turned into DP equivalent\n", "- `Join`s combine two input *Relations*\n", "- `Table`s are simply sources.\n", "\n", "This representation simplifies the process of rewriting queries and reduces dependencies on the diverse range of syntactic constructs present in SQL." ] }, { "cell_type": "markdown", "id": "792fe5b0", "metadata": { "id": "792fe5b0" }, "source": [ "## Install the database and packages\n" ] }, { "cell_type": "code", "execution_count": 29, "id": "08f55d69", "metadata": { "id": "08f55d69" }, "outputs": [], "source": [ "%%capture\n", "# Load the database\n", "# Inspired by https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/postgresql.ipynb#scrollTo=YUj0878jPyz7\n", "!sudo apt-get -y -qq update\n", "!sudo apt-get -y -qq install postgresql-14 graphviz\n", "# Start postgresql server\n", "!sudo sed -i \"s/port = 5432/port = 5433/g\" /etc/postgresql/14/main/postgresql.conf\n", "!sudo service postgresql start\n", "# Set password\n", "!sudo -u postgres psql -U postgres -c \"ALTER USER postgres PASSWORD 'pyqrlew-db'\"\n", "!pip install -U pyqrlew numpy pandas matplotlib graphviz" ] }, { "cell_type": "code", "execution_count": 30, "id": "6770c3e8", "metadata": { "id": "6770c3e8" }, "outputs": [], "source": [ "from pyqrlew.io import PostgreSQL\n", "from pyqrlew import Dataset\n", "# Read data\n", "database = PostgreSQL()\n", "dataset = database.retail()" ] }, { "cell_type": "markdown", "id": "98161de3", "metadata": { "id": "98161de3" }, "source": [ "For our analysis, we will use the Qrlew built-in sample datasets `retail` and an Postgres connection.\n", "\n", "Note that you can also load one or several csv files or use another SQL engine." ] }, { "cell_type": "markdown", "id": "e542593d", "metadata": { "id": "e542593d" }, "source": [ "## The `Relation`: an intermediate represention for SQL rewritting" ] }, { "cell_type": "markdown", "id": "ada2d2e9", "metadata": { "id": "ada2d2e9" }, "source": [ "The `retail` dataset contains 3 tables, each can be converted into a `Relation::Table`:" ] }, { "cell_type": "code", "execution_count": 31, "id": "9936b983", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 451 }, "id": "9936b983", "outputId": "9c52bd01-6b0d-4709-f78f-4fb95e556899" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_pv7m\n", "\n", "\n", "\n", "graph_pv7m\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3, 4, 5}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[3.879 8.623]\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_6cje\n", "\n", "\n", "\n", "graph_6cje\n", "\n", "RETAIL_SALES size ∈ int{715}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3, 4, 5}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2012-10-26 00:00:00]\n", "weekly_sales = weekly_sales ∈ float[3440.69 91965.85]\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_rdzg\n", "\n", "\n", "\n", "graph_rdzg\n", "\n", "RETAIL_STORES size ∈ int{5}\n", "store = store ∈ str{1, 2, 3, 4, 5}\n", "type = type ∈ str{A, B}\n", "size = size ∈ int{34875, 37392, 151315, 202307, 205863}\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import graphviz\n", "display(graphviz.Source(dataset.retail.features.relation().dot()))\n", "display(graphviz.Source(dataset.retail.sales.relation().dot()))\n", "display(graphviz.Source(dataset.retail.stores.relation().dot()))" ] }, { "cell_type": "markdown", "id": "0561bdfe", "metadata": { "id": "0561bdfe" }, "source": [ "Each `Relation::Table` contains the size of the table (which can be approximate with DP) and the columns in the tables with their name and type." ] }, { "cell_type": "markdown", "id": "a9316957", "metadata": { "id": "a9316957" }, "source": [ "Before using any differential privacy mechanism, it is essential to determine:\n", "- the *privacy unit*, i.e. the *user* identifier,\n", "- the *sensitivity of the aggregation functions*, i.e. the maximal contribution a user may have on the result,\n", "- the values of the `GROUP BY` columns that can be safely released" ] }, { "cell_type": "markdown", "id": "5b4e7000", "metadata": { "id": "5b4e7000" }, "source": [ "With Qrlew is very easy to set ranges, unique constraints and distinct possible values for specific columns" ] }, { "cell_type": "code", "execution_count": 32, "id": "1ddfd0ec", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 201 }, "id": "1ddfd0ec", "outputId": "a26e542d-121a-458d-f0c6-01d8f57bed9e" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "dataset = dataset.retail.features.unemployment.with_range(0, 20)\n", "dataset = dataset.retail.features.unemployment.with_unique_constraint()\n", "dataset = dataset.retail.features.store.with_possible_values([\"1\", \"2\", \"3\"])\n", "display(graphviz.Source(dataset.retail.features.relation().dot()))" ] }, { "cell_type": "markdown", "id": "7de6c738", "metadata": { "id": "7de6c738" }, "source": [ "## The privacy unit" ] }, { "cell_type": "markdown", "id": "30077766", "metadata": { "id": "30077766" }, "source": [ "In this example, let's consider that we want to protect the `id` column of the tables `features` and `sales`.\n", "\n", "The tables `store` is linked to `features` via foreign keys.\n", "\n", "Then each table have a privacy unit (even if it is not directly in the table):\n", "- `features`: column `id`\n", "- `stores`: the privacy unit is the column `id` of the `feature` table. It is obtained by joining the tables `features` and `stores` with the condition `features.store = stores.store`\n", "- `sales`: column `id`\n", "\n", "We code the privacy unit as:" ] }, { "cell_type": "code", "execution_count": 33, "id": "b96421f2", "metadata": { "id": "b96421f2" }, "outputs": [], "source": [ "privacy_unit = [\n", " (\"features\", [], \"id\"),\n", " (\"stores\", [(\"store\", \"features\", \"store\")], \"id\"),\n", " (\"sales\", [], \"id\")\n", "]\n", "# Other arguments that will be explained later\n", "budget = {\"epsilon\": 1.0, \"delta\": 5e-4}\n", "synthetic_data = [\n", " ([\"retail\", \"features\"], [\"retail\", \"features_sd\"]),\n", "]" ] }, { "cell_type": "markdown", "id": "dddeb53d", "metadata": { "id": "dddeb53d" }, "source": [ "The `privacy_unit` is a vector whose each element is a tuple with the convention: `(table_name, join_path, id_column)` with:\n", "- `table_name` the name of the table,\n", "- `join_path` a vector storing the joins whose each item `(column_name, table_name, column)` represents a foreign key,\n", "- `id_column` the name of the column which is the privacy unit." ] }, { "cell_type": "markdown", "id": "0aa67b12", "metadata": { "id": "0aa67b12" }, "source": [ "The `Relation` method `protect` allows to propagate the **privacy unit** through all the relations:" ] }, { "cell_type": "code", "execution_count": 34, "id": "0ce5df6d", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 438 }, "id": "0ce5df6d", "outputId": "0b076114-67bc-467e-b6fd-a4c4f92b4935" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_wsfm\n", "\n", "\n", "\n", "graph_wsfm\n", "\n", "MAP_2P3W size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = md5(cast_as_text(id)) ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = 1 ∈ int{1}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph_wsfm->graph_5k33\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "pup_features = dataset.retail.features.relation().rewrite_as_privacy_unit_preserving(\n", " dataset=dataset,\n", " privacy_unit=privacy_unit,\n", " epsilon_delta=budget,\n", " synthetic_data=synthetic_data,\n", ")\n", "display(graphviz.Source(pup_features.relation().dot()))" ] }, { "cell_type": "code", "execution_count": 35, "id": "d975df77", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 632 }, "id": "d975df77", "outputId": "f229bac4-bfbe-4705-8010-6acf3883b3f0" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_eqvx\n", "\n", "\n", "\n", "graph_eqvx\n", "\n", "MAP_40SW size ∈ int[0 4550]\n", "_PRIVACY_UNIT_ = md5(cast_as_text(field_tv1c)) ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = 1 ∈ int{1}\n", "store = field_xh5r ∈ str{1, 2, 3}\n", "type = field_mzk8 ∈ str{A, B}\n", "size = field_1p2r ∈ int{34875, 37392, 151315, 202307, 205863}\n", "\n", "\n", "\n", "graph_k6gc\n", "\n", "JOIN_2FIX size ∈ int[0 4550]\n", "field_tv1c = _LEFT_.id ∈ str\n", "field_ay_7 = _LEFT_.store ∈ str{1, 2, 3}\n", "field_djqk = _LEFT_.date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "field_x8ew = _LEFT_.temperature ∈ float[28.84 93.34]\n", "field_z1o7 = _LEFT_.fuel_price ∈ float[2.514 3.907]\n", "field_vto2 = _LEFT_.cpi ∈ float[126.064 228.7298638]\n", "field_hftr = _LEFT_.unemployment ∈ float[0 20]\n", "field_nl9d = _LEFT_.isholiday ∈ bool\n", "field_xh5r = _RIGHT_.store ∈ str{1, 2, 3}\n", "field_mzk8 = _RIGHT_.type ∈ str{A, B}\n", "field_1p2r = _RIGHT_.size ∈ int{34875, 37392, 151315, 202307, 205863}\n", "INNER ON (_RIGHT_.store = _LEFT_.store)\n", "\n", "\n", "\n", "graph_eqvx->graph_k6gc\n", "\n", "\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph_k6gc->graph_5k33\n", "\n", "\n", "\n", "\n", "\n", "graph_rdzg\n", "\n", "RETAIL_STORES size ∈ int{5}\n", "store = store ∈ str{1, 2, 3, 4, 5}\n", "type = type ∈ str{A, B}\n", "size = size ∈ int{34875, 37392, 151315, 202307, 205863}\n", "\n", "\n", "\n", "graph_k6gc->graph_rdzg\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "pup_stores = dataset.retail.stores.relation().rewrite_as_privacy_unit_preserving(\n", " dataset=dataset,\n", " privacy_unit=privacy_unit,\n", " epsilon_delta=budget,\n", " synthetic_data=synthetic_data,\n", ")\n", "display(graphviz.Source(pup_stores.relation().dot()))" ] }, { "cell_type": "markdown", "id": "f232bac1", "metadata": { "id": "f232bac1" }, "source": [ "We notice that to protect the `features` table we just have to hash the values of the `id` column and store them into the `_PRIVACY_UNIT_` while to protect the `store` table we had to make a join before." ] }, { "cell_type": "markdown", "id": "b5badc08", "metadata": { "id": "b5badc08" }, "source": [ "## The sensitivity" ] }, { "cell_type": "markdown", "id": "9de52758", "metadata": { "id": "9de52758" }, "source": [ "In the case we want to relase the result of query : `SELECT SUM(3 * fuel_price + temperature / 10) FROM retail.features`, we have to compute the sensitivity of the `SUM(3 * fuel_price + temperature / 10)` aggregation.\n", "\n", "The user provides the ranges of the inputs column in the `WHERE` clause of the query, then these values are propagated through all the relations." ] }, { "cell_type": "code", "execution_count": 36, "id": "4ca7732e", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 426 }, "id": "4ca7732e", "outputId": "40b8758b-5c05-450b-f983-4666faff4fcd" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_05e5\n", "\n", "\n", "\n", "graph_05e5\n", "\n", "REDUCE_YK9H size ∈ int[0 910]\n", "field_qq4j = sum(field_38yt) ∈ float[0 19110]\n", "\n", "\n", "\n", "graph_b3gz\n", "\n", "MAP_27MX size ∈ int[0 910]\n", "field_38yt = ((3 * fuel_price) + (temperature / 10)) ∈ float[10.426 21]\n", "WHERE ((((fuel_price > 2.5) and (fuel_price < 3.9)) and (temperature > 28)) and (temperature < 93))\n", "\n", "\n", "\n", "graph_05e5->graph_b3gz\n", "\n", "\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph_b3gz->graph_5k33\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "relation = dataset.relation('SELECT SUM(3 * fuel_price + temperature / 10) FROM retail.features WHERE fuel_price > 2.5 AND fuel_price < 3.9 AND temperature > 28 AND temperature < 93')\n", "display(graphviz.Source(relation.dot()))" ] }, { "cell_type": "markdown", "id": "48d3a1dd", "metadata": { "id": "48d3a1dd" }, "source": [ "More information about the ranges propagation can be found [here](https://github.com/Qrlew/pyqrlew/blob/main/examples/range_propagation.ipynb)." ] }, { "cell_type": "markdown", "id": "1fc2f206", "metadata": { "id": "1fc2f206" }, "source": [ "## The protection of the GROUP BY keys" ] }, { "cell_type": "markdown", "id": "88ad8146", "metadata": { "id": "88ad8146" }, "source": [ "As explained in [Wilson et al. (2019)](https://arxiv.org/abs/1909.01917) and [this post](https://www.sarus.tech/post/the-protection-of-grouping-keys-in-the-context-of-differentially-private-sql), releasing the values of the grouping columns may leak some sensitive information.\n", "\n", "To avoid that, two methods can be used:\n", "- if the grouping keys have public values, we can release the propagated public values,\n", "- otherwise we can use the tau-thresholding mechanism.\n", "\n", "In the following cell, we show how the public values are propagated and then can be used in the DP rewriting." ] }, { "cell_type": "code", "execution_count": 37, "id": "f591c869", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 590 }, "id": "f591c869", "outputId": "12050901-1d86-4209-9f58-28c3c6406269" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_sxmy\n", "\n", "\n", "\n", "graph_sxmy\n", "\n", "MAP_JRFK size ∈ int[0 910]\n", "store = field_8tj1 ∈ ∅ UNIQUE\n", "field_dtwp = field_7hx6 ∈ float[0 3555.37]\n", "\n", "\n", "\n", "graph_irrh\n", "\n", "REDUCE_4FNE size ∈ int[0 910]\n", "field_8tj1 = first(field_8tj1) ∈ ∅ UNIQUE\n", "field_7hx6 = sum(field__6vl) ∈ float[0 3555.37]\n", "GROUP BY (field_8tj1)\n", "\n", "\n", "\n", "graph_sxmy->graph_irrh\n", "\n", "\n", "\n", "\n", "\n", "graph_owle\n", "\n", "MAP_PXMR size ∈ int[0 910]\n", "field__6vl = fuel_price ∈ float[2.514 3.907]\n", "field_8tj1 = store ∈ ∅\n", "WHERE (store in (a, b))\n", "\n", "\n", "\n", "graph_irrh->graph_owle\n", "\n", "\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph_owle->graph_5k33\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "relation = dataset.relation(\"SELECT store, SUM(fuel_price) FROM retail.features WHERE store IN ('a', 'b') GROUP BY store \")\n", "display(graphviz.Source(relation.dot()))" ] }, { "cell_type": "markdown", "id": "5b28a481", "metadata": { "id": "5b28a481" }, "source": [ "## Rewritting with DP" ] }, { "cell_type": "markdown", "id": "f03b57f8", "metadata": { "id": "f03b57f8" }, "source": [ "The rewritting with DP is done in 3 steps:\n", "- **Protect** the input `Relation` as described before\n", "- **DP-rewritting of the grouping keys**: a grouping key can be released if already public or if it appears in enough users rows,\n", "- **Add noise** scaled by the privacy parameters to each aggregation." ] }, { "cell_type": "markdown", "id": "fac10e08", "metadata": { "id": "fac10e08" }, "source": [ "\n", "We use the `rewrite_with_differential_privacy` method that transforms a `Relation` into its differentially private equivalent.\n", "\n", "It inputs:\n", "- `dataset`: the `Dataset` we want to query,\n", "- `privacy_unit`: the privacy unit described previously,\n", "- `privacy parameters`: the $(\\varepsilon, \\delta)$ privacy parameters,\n", "- `synthetic_data`: an **optional** list giving the correpondance between the original tables and their synthetic version. Each table must be specified. The list is made of two-element tuples whose first element is a list representing the path to the original table (e.g.: `[\"retail\", \"features\"]` for \"retail.features\" table or `[\"features\"]` for the \"feature\" table) and the second element, the path to the synthetic table." ] }, { "cell_type": "code", "execution_count": 38, "id": "9b989849", "metadata": { "id": "9b989849" }, "outputs": [], "source": [ "privacy_unit = [\n", " (\"stores\", [(\"store\", \"features\", \"store\")], \"id\"),\n", " (\"features\", [], \"id\"),\n", " (\"sales\", [], \"id\")\n", "]\n", "budget = {\"epsilon\": 1.0, \"delta\": 5e-4}\n", "synthetic_data = None\n", "\n", "query = \"SELECT SUM(fuel_price) AS my_sum FROM retail.features WHERE fuel_price > 2.514 AND fuel_price < 3.907\"\n", "relation = dataset.relation(query)\n", "\n", "relation_with_dp_event = relation.rewrite_with_differential_privacy(\n", " dataset=dataset,\n", " privacy_unit=privacy_unit,\n", " epsilon_delta=budget,\n", " synthetic_data=synthetic_data,\n", ")\n", "dp_relation = relation_with_dp_event.relation()\n", "mechanisms_used = relation_with_dp_event.dp_event()" ] }, { "cell_type": "markdown", "id": "r_POfPfFXk21", "metadata": { "id": "r_POfPfFXk21" }, "source": [ "`dp_events` are compatible with Google's dp_accounting" ] }, { "cell_type": "code", "execution_count": 39, "id": "6nMvN25zYJoC", "metadata": { "id": "6nMvN25zYJoC" }, "outputs": [], "source": [ "%%capture\n", "!pip install dp_accounting==0.4.1" ] }, { "cell_type": "code", "execution_count": 40, "id": "b5eec893", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "b5eec893", "outputId": "3c5e8375-7c3e-4098-9ddd-f2417c140ddd" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "DpEvent(Gaussian { noise_multiplier: 1406.4215077657532 })\n", "GaussianDpEvent(noise_multiplier=1406.4215077657532)\n" ] } ], "source": [ "from dp_accounting import DpEvent\n", "print(mechanisms_used)\n", "print(DpEvent.from_named_tuple(mechanisms_used.to_named_tuple()))" ] }, { "cell_type": "markdown", "id": "b58c4ff3", "metadata": { "id": "b58c4ff3" }, "source": [ "The ouput is a tuple made of two elements:\n", "- The first element is the `Relation` rewritten with dp,\n", "- the second element is a `DpEvent` which stores the DP mechanims invoked during the rewritting.\n", "\n", "In the present case, we have used only a gaussian mechanim since our query contain only a simple aggregate and no `GROUP BY`." ] }, { "cell_type": "code", "execution_count": 41, "id": "fd1670d3", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "fd1670d3", "outputId": "113820de-f515-470d-b357-09c996065cf8" }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "graph_nk1f\n", "\n", "\n", "\n", "graph_nk1f\n", "\n", "MAP_5KNY size ∈ int[0 910]\n", "my_sum = my_sum ∈ float[0 3555.37]\n", "\n", "\n", "\n", "graph_9ys5\n", "\n", "MAP_815Z size ∈ int[0 910]\n", "my_sum = _SUM_field__6vl ∈ float[0 3555.37]\n", "\n", "\n", "\n", "graph_nk1f->graph_9ys5\n", "\n", "\n", "\n", "\n", "\n", "graph_vvic\n", "\n", "MAP_C_77 size ∈ int[0 910]\n", "_SUM_field__6vl = least(3555.37, greatest(0, (coalesce(_SUM_field__6vl, 0) + (1406.4215077657532 * (sqrt((-2 * ln(random())))...\n", "\n", "\n", "\n", "graph_9ys5->graph_vvic\n", "\n", "\n", "\n", "\n", "\n", "graph_fkpc\n", "\n", "REDUCE_8IQS size ∈ int[0 910]\n", "_SUM_field__6vl = sum(field_q9u_) ∈ float[0 3555.37]\n", "\n", "\n", "\n", "graph_vvic->graph_fkpc\n", "\n", "\n", "\n", "\n", "\n", "graph_t375\n", "\n", "MAP_NLZ1 size ∈ int[0 910]\n", "field_q9u_ = _CLIPPED_field__6vl ∈ float[0.008333828849736673 3.907]\n", "\n", "\n", "\n", "graph_fkpc->graph_t375\n", "\n", "\n", "\n", "\n", "\n", "graph_v3fs\n", "\n", "MAP_82AH size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = _PRIVACY_UNIT_ ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = _PRIVACY_UNIT_WEIGHT_ ∈ int{1}\n", "field__6vl = field__6vl ∈ float[2.514 3.907]\n", "_SCALE_FACTOR__PRIVACY_UNIT_ = _SCALE_FACTOR__PRIVACY_UNIT_ ∈ str\n", "_SCALE_FACTOR_field__6vl = _SCALE_FACTOR_field__6vl ∈ float[0.003314967720658979 1]\n", "_CLIPPED_field__6vl = (field__6vl * _SCALE_FACTOR_field__6vl) ∈ float[0.008333828849736673 3.907]\n", "\n", "\n", "\n", "graph_t375->graph_v3fs\n", "\n", "\n", "\n", "\n", "\n", "graph_k_22\n", "\n", "JOIN_8DFM size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = _LEFT_._PRIVACY_UNIT_ ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = _LEFT_._PRIVACY_UNIT_WEIGHT_ ∈ int{1}\n", "field__6vl = _LEFT_.field__6vl ∈ float[2.514 3.907]\n", "_SCALE_FACTOR__PRIVACY_UNIT_ = _RIGHT_._PRIVACY_UNIT_ ∈ str\n", "_SCALE_FACTOR_field__6vl = _RIGHT_.field__6vl ∈ float[0.003314967720658979 1]\n", "INNER ON (_LEFT_._PRIVACY_UNIT_ = _RIGHT_._PRIVACY_UNIT_)\n", "\n", "\n", "\n", "graph_v3fs->graph_k_22\n", "\n", "\n", "\n", "\n", "\n", "graph_7y6n\n", "\n", "MAP_GQZ_ size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = _PRIVACY_UNIT_ ∈ str UNIQUE\n", "field__6vl = (1 / greatest(1, (sqrt(field__6vl) / 355.537))) ∈ float[0.003314967720658979 1]\n", "\n", "\n", "\n", "graph_k_22->graph_7y6n\n", "\n", "\n", "\n", "\n", "\n", "graph_8uo9\n", "\n", "MAP_6C_B size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = coalesce(cast_as_text(_PRIVACY_UNIT_), _PRIVACY_UNIT_DEFAULT_) ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = coalesce(_PRIVACY_UNIT_WEIGHT_, 0) ∈ int{1}\n", "field__6vl = field__6vl ∈ float[2.514 3.907]\n", "\n", "\n", "\n", "graph_k_22->graph_8uo9\n", "\n", "\n", "\n", "\n", "\n", "graph_clv9\n", "\n", "REDUCE_K2TQ size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = first(field_grep) ∈ str UNIQUE\n", "field__6vl = sum(field_czvy) ∈ float[0 11502996811.579]\n", "GROUP BY (field_grep)\n", "\n", "\n", "\n", "graph_7y6n->graph_clv9\n", "\n", "\n", "\n", "\n", "\n", "graph_ood0\n", "\n", "MAP_C59Q size ∈ int[0 910]\n", "field_czvy = _NORM_field__6vl ∈ float[0 12640655.8369]\n", "field_grep = _PRIVACY_UNIT_ ∈ str UNIQUE\n", "\n", "\n", "\n", "graph_clv9->graph_ood0\n", "\n", "\n", "\n", "\n", "\n", "graph_qzw2\n", "\n", "MAP_SULF size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = _PRIVACY_UNIT_ ∈ str UNIQUE\n", "_NORM_field__6vl = abs((_NORM_field__6vl * _NORM_field__6vl)) ∈ float[0 12640655.8369]\n", "\n", "\n", "\n", "graph_ood0->graph_qzw2\n", "\n", "\n", "\n", "\n", "\n", "graph_4kkz\n", "\n", "REDUCE_Q7ZO size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = first(field_grep) ∈ str UNIQUE\n", "_NORM_field__6vl = sum(field_3d25) ∈ float[0 3555.37]\n", "GROUP BY (field_grep)\n", "\n", "\n", "\n", "graph_qzw2->graph_4kkz\n", "\n", "\n", "\n", "\n", "\n", "graph_bvr1\n", "\n", "MAP_PIL5 size ∈ int[0 910]\n", "field_3d25 = field__6vl ∈ float[2.514 3.907]\n", "field_grep = _PRIVACY_UNIT_ ∈ str\n", "\n", "\n", "\n", "graph_4kkz->graph_bvr1\n", "\n", "\n", "\n", "\n", "\n", "graph_bvr1->graph_8uo9\n", "\n", "\n", "\n", "\n", "\n", "graph__cln\n", "\n", "MAP_Y19B size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = _PRIVACY_UNIT_ ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = _PRIVACY_UNIT_WEIGHT_ ∈ int{1}\n", "field__6vl = fuel_price ∈ float[2.514 3.907]\n", "WHERE ((fuel_price > 2.514) and (fuel_price < 3.907))\n", "\n", "\n", "\n", "graph_8uo9->graph__cln\n", "\n", "\n", "\n", "\n", "\n", "graph_wsfm\n", "\n", "MAP_2P3W size ∈ int[0 910]\n", "_PRIVACY_UNIT_ = md5(cast_as_text(id)) ∈ str\n", "_PRIVACY_UNIT_WEIGHT_ = 1 ∈ int{1}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph__cln->graph_wsfm\n", "\n", "\n", "\n", "\n", "\n", "graph_5k33\n", "\n", "RETAIL_FEATURES size ∈ int{910}\n", "id = id ∈ str\n", "store = store ∈ str{1, 2, 3}\n", "date = date ∈ datetime[2010-02-05 00:00:00 2013-07-26 00:00:00]\n", "temperature = temperature ∈ float[28.84 93.34]\n", "fuel_price = fuel_price ∈ float[2.514 3.907]\n", "cpi = cpi ∈ float[126.064 228.7298638]\n", "unemployment = unemployment ∈ float[0 20] UNIQUE\n", "isholiday = isholiday ∈ bool\n", "\n", "\n", "\n", "graph_wsfm->graph_5k33\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(graphviz.Source(dp_relation.dot()))" ] }, { "cell_type": "markdown", "id": "e254bca1", "metadata": { "id": "e254bca1" }, "source": [ "The `Relation` can be translated into an SQL query:" ] }, { "cell_type": "code", "execution_count": 42, "id": "e17064c4", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "e17064c4", "outputId": "4413dd75-d41c-43fc-efe0-82e72d7aa7c3" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[34mWITH\u001b[0m\n", " \"map_2p3w\" (\"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\", \"id\", \"store\", \"date\", \"temperature\", \"fuel_price\", \"cpi\", \"unemployment\", \"isholiday\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m MD5(C\u001b[35mAS\u001b[0mT(\"id\" \u001b[35mAS\u001b[0m TEXT)) \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", 1 \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_WEIGHT_\", \"id\" \u001b[35mAS\u001b[0m \"id\", \"store\" \u001b[35mAS\u001b[0m \"store\", \"date\" \u001b[35mAS\u001b[0m \"date\", \"temperature\" \u001b[35mAS\u001b[0m \"temperature\", \"fuel_price\" \u001b[35mAS\u001b[0m \"fuel_price\", \"cpi\" \u001b[35mAS\u001b[0m \"cpi\", \"unemployment\" \u001b[35mAS\u001b[0m \"unemployment\", \"isholiday\" \u001b[35mAS\u001b[0m \"isholiday\" FROM \"retail\".\"features\"),\n", " \"map_y19b\" (\"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\", \"field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\" \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_WEIGHT_\", \"fuel_price\" \u001b[35mAS\u001b[0m \"field__6vl\" FROM \"map_2p3w\" \u001b[35mWHERE\u001b[0m ((\"fuel_price\") > (2.514)) AND ((\"fuel_price\") < (3.907))),\n", " \"map_6c_b\" (\"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\", \"field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m COALESCE(C\u001b[35mAS\u001b[0mT(\"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m TEXT),\n", " '_PRIVACY_UNIT_DEFAULT_') \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", COALESCE(\"_PRIVACY_UNIT_WEIGHT_\", 0) \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_WEIGHT_\", \"field__6vl\" \u001b[35mAS\u001b[0m \"field__6vl\" FROM \"map_y19b\"),\n", " \"map_pil5\" (\"field_3d25\", \"field_grep\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"field__6vl\" \u001b[35mAS\u001b[0m \"field_3d25\", \"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"field_grep\" FROM \"map_6c_b\"),\n", " \"reduce_q7zo\" (\"_PRIVACY_UNIT_\", \"_NORM_field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m field_grep \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", SUM(\"field_3d25\") \u001b[35mAS\u001b[0m \"_NORM_field__6vl\" FROM \"map_pil5\" \u001b[35mGROUP BY\u001b[0m \"field_grep\"),\n", " \"map_sulf\" (\"_PRIVACY_UNIT_\", \"_NORM_field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", ABS((\"_NORM_field__6vl\") * (\"_NORM_field__6vl\")) \u001b[35mAS\u001b[0m \"_NORM_field__6vl\" FROM \"reduce_q7zo\"),\n", " \"map_c59q\" (\"field_czvy\", \"field_grep\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_NORM_field__6vl\" \u001b[35mAS\u001b[0m \"field_czvy\", \"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"field_grep\" FROM \"map_sulf\"),\n", " \"reduce_k2tq\" (\"_PRIVACY_UNIT_\", \"field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m field_grep \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", SUM(\"field_czvy\") \u001b[35mAS\u001b[0m \"field__6vl\" FROM \"map_c59q\" \u001b[35mGROUP BY\u001b[0m \"field_grep\"),\n", " \"map_gqz_\" (\"_PRIVACY_UNIT_\", \"field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", (1) / (GREATEST(1, (SQRT(\"field__6vl\")) / (355.537))) \u001b[35mAS\u001b[0m \"field__6vl\" FROM \"reduce_k2tq\"),\n", " \"join_8dfm\" (\"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\", \"field__6vl\", \"_SCALE_FACTOR__PRIVACY_UNIT_\", \"_SCALE_FACTOR_field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m * FROM \"map_6c_b\" \u001b[35mAS\u001b[0m \"_LEFT_\" JOIN \"map_gqz_\" \u001b[35mAS\u001b[0m \"_RIGHT_\" ON (\"_LEFT_\".\"_PRIVACY_UNIT_\") = (\"_RIGHT_\".\"_PRIVACY_UNIT_\")),\n", " \"map_82ah\" (\"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\", \"field__6vl\", \"_SCALE_FACTOR__PRIVACY_UNIT_\", \"_SCALE_FACTOR_field__6vl\", \"_CLIPPED_field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_\", \"_PRIVACY_UNIT_WEIGHT_\" \u001b[35mAS\u001b[0m \"_PRIVACY_UNIT_WEIGHT_\", \"field__6vl\" \u001b[35mAS\u001b[0m \"field__6vl\", \"_SCALE_FACTOR__PRIVACY_UNIT_\" \u001b[35mAS\u001b[0m \"_SCALE_FACTOR__PRIVACY_UNIT_\", \"_SCALE_FACTOR_field__6vl\" \u001b[35mAS\u001b[0m \"_SCALE_FACTOR_field__6vl\", (\"field__6vl\") * (\"_SCALE_FACTOR_field__6vl\") \u001b[35mAS\u001b[0m \"_CLIPPED_field__6vl\" FROM \"join_8dfm\"),\n", " \"map_nlz1\" (\"field_q9u_\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_CLIPPED_field__6vl\" \u001b[35mAS\u001b[0m \"field_q9u_\" FROM \"map_82ah\"),\n", " \"reduce_8iqs\" (\"_SUM_field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m SUM(\"field_q9u_\") \u001b[35mAS\u001b[0m \"_SUM_field__6vl\" FROM \"map_nlz1\"),\n", " \"map_c_77\" (\"_SUM_field__6vl\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m LE\u001b[35mAS\u001b[0mT(3555.37, GREATEST(0, (COALESCE(\"_SUM_field__6vl\", 0)) + ((1406.4215077657532) * ((SQRT((-2) * (LN(RANDOM())))) * (COS((6.283185307179586) * (RANDOM()))))))) \u001b[35mAS\u001b[0m \"_SUM_field__6vl\" FROM \"reduce_8iqs\"),\n", " \"map_815z\" (\"my_sum\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"_SUM_field__6vl\" \u001b[35mAS\u001b[0m \"my_sum\" FROM \"map_c_77\"),\n", " \"map_5kny\" (\"my_sum\") \u001b[35mAS\u001b[0m (\u001b[35mSELECT\u001b[0m \"my_sum\" \u001b[35mAS\u001b[0m \"my_sum\" FROM \"map_815z\")\n", "\u001b[35mSELECT\u001b[0m * FROM \"map_5kny\"\n" ] } ], "source": [ "MAGENTA_COLOR = '\\033[35m'\n", "BLUE_COLOR = '\\033[34m'\n", "RESET_COLOR = '\\033[0m'\n", "\n", "def print_query(query: str):\n", " keywords = [\"SELECT\", \"AS\", \"GROUP BY\", \"LIMIT\", \"ORDER BY\", \"WHERE\"]\n", " colored_query = query\n", " colored_query = colored_query.replace(\"WITH\", \"WITH\\n \")\n", " colored_query = colored_query.replace(\" SELECT\", \"\\nSELECT\")\n", " colored_query = colored_query.replace(\"),\", \"),\\n \")\n", " for word in keywords:\n", " colored_query = colored_query.replace(word, MAGENTA_COLOR + word + RESET_COLOR)\n", " colored_query = colored_query.replace(\"WITH\", BLUE_COLOR + \"WITH\" + RESET_COLOR)\n", " print(colored_query)\n", "\n", "\n", "dp_query = dp_relation.to_query()\n", "print_query(dp_query)" ] }, { "cell_type": "markdown", "id": "f626ef40", "metadata": { "id": "f626ef40" }, "source": [ "Then the query is sent to the databqe:" ] }, { "cell_type": "code", "execution_count": 43, "id": "30d8afd9", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "30d8afd9", "outputId": "f28e1ebe-eb92-4f08-c88d-e06fba341831" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initial query: 2939.4329999999995\n", "DP query: 3555.37\n" ] } ], "source": [ "import pandas as pd\n", "true_res = pd.read_sql(query, database.engine())\n", "dp_res = pd.read_sql(dp_query, database.engine())\n", "\n", "print(f\"Initial query: {true_res.iloc[0].iloc[0]}\")\n", "print(\"DP query: \", dp_res[\"my_sum\"][0])" ] }, { "cell_type": "markdown", "id": "c4b5c4ac", "metadata": { "id": "c4b5c4ac" }, "source": [ "We can sent the DP query multiple times wo see the variantions among the results:" ] }, { "cell_type": "code", "execution_count": 44, "id": "a3cc993b", "metadata": { "id": "a3cc993b" }, "outputs": [], "source": [ "N_RUNS = 300\n", "\n", "def run(dataset, query, epsilons):\n", " data = {}\n", " privacy_unit = [\n", " (\"stores\", [(\"store\", \"features\", \"store\")], \"id\"),\n", " (\"features\", [], \"id\"),\n", " (\"sales\", [], \"id\")\n", " ]\n", " synthetic_data = None\n", "\n", " delta = 1e-3\n", " relation = dataset.relation(query)\n", "\n", " for epsilon in epsilons:\n", " budget = {\"epsilon\": epsilon, \"delta\": delta}\n", " rel_with_pq = relation.rewrite_with_differential_privacy(\n", " dataset=dataset,\n", " privacy_unit=privacy_unit,\n", " epsilon_delta=budget,\n", " synthetic_data=synthetic_data,\n", " )\n", " dp_query = rel_with_pq.relation().to_query()\n", " data[epsilon] = [\n", " pd.read_sql(dp_query, database.engine())[\"my_sum\"][0] for _ in range(N_RUNS)\n", " ]\n", " return data" ] }, { "cell_type": "code", "execution_count": 45, "id": "66b29b60", "metadata": { "id": "66b29b60" }, "outputs": [], "source": [ "true_value = pd.read_sql(query, database.engine())[\"my_sum\"][0]\n", "data = run(dataset, query, [0.5, 1., 5.])" ] }, { "cell_type": "code", "execution_count": 46, "id": "d3762dcc", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "d3762dcc", "outputId": "e85757f7-6eaf-4b13-f171-6f0c1731ebb8" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "SELECT SUM(fuel_price) AS my_sum FROM retail.features WHERE fuel_price > 2.514 AND fuel_price < 3.907\n" ] } ], "source": [ "print(query)" ] }, { "cell_type": "code", "execution_count": 47, "id": "c698def3", "metadata": { "id": "c698def3" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import matplotlib as mpl\n", "mpl.rcParams['text.usetex'] = False\n", "\n", "def plot_hist(true_value, data, query):\n", " plt.axvline(true_value, color='red', label=\"True\")\n", " for e, d in data.items():\n", " plt.hist(d, bins=10, alpha=0.5, label = f\"eps = {e}\")\n", " plt.legend()\n", " plt.title(query.encode('unicode_escape').decode().replace(\"<\", \"$<$\").replace(\">\", \"$>$\"))" ] }, { "cell_type": "code", "execution_count": 48, "id": "a875ac4e", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 453 }, "id": "a875ac4e", "outputId": "c0e78e12-5726-4a17-8f1a-e61f23422f6b" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6gAAAG0CAYAAAA/ygrhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABekklEQVR4nO3de5xNZf//8fc2zMGMmXGYI4NxPpNDjBQyhUoUiciQHApRN4kScdegu1IS6S6HIt25kVJ+IRINiZRDOYcw4zgzGHMwc/3+6J79tc1pz5jDMl7Px2M/HjNrXftan2uvda21P3tday2bMcYIAAAAAIAiVqKoAwAAAAAAQCJBBQAAAABYBAkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkhQi6np06erTp06SktLc5i+bds2tW7dWp6enrLZbNq5c2eBLH/+/Pmy2Wz6888/C6T+olrunDlzVLlyZSUlJRVI/cDNLrM+mJt+WVj7KBQfHNeAvCmMvkO/QV7kKUHdtWuXevTooSpVqsjd3V0VK1bUPffco5kzZ9rLpG8YWb22bNmSoezPP/+c5TJzU58kHTp0SEOGDFG1atXk7u4ub29v3XHHHXr77bd15coVScq2vmtfGzZsuKHPQpImTZokm82ms2fPZlpPgwYN1K5du0zbu2nTpgzljTEKCQmRzWbTAw884DAvPj5e06ZN09ixY1WixP+t4pSUFD3yyCM6f/683nrrLX388ceqUqVKlm1DRv3791dycrLef//9XL/3vffek81mU8uWLbMs4+z2hBuT3f7khRdeyLRMyZIlVbFiRfXv318nTpzIsu49e/aob9++qlixotzc3BQcHKw+ffpoz5492caR235eUH788UdNmjRJsbGxhbK8axXWPqoo23ij/vOf/8hms2n58uUZ5jVu3Fg2m03r16/PMK9y5cpq3bq1/f+cjrvt2rVTgwYNMpTPzTHdmb6T22P79Tiu3dy2bdum4cOHq379+vL09FTlypXVs2dP7d+/36n3b9iwwalt59KlS5o4caI6deqkcuXKyWazaf78+U4t49VXX5XNZnPoD87K6bifvv27u7tnelzJqR+6u7srODhYHTt21DvvvKOLFy86HRt95+a1Z88ePfLII6pWrZpKly6tChUq6K677tKXX37pdB3bt29Xp06d5O3trTJlyujee+/N8geKpKQkjR07VsHBwfLw8FDLli21Zs0ahzL9+/fPdl+e3femzJTMVWn9fWBv3769KleurEGDBikwMFDHjx/Xli1b9Pbbb2vEiBEO5SdPnqzQ0NAM9dSoUSO3i3a6vlWrVumRRx6Rm5ub+vXrpwYNGig5OVmbNm3SmDFjtGfPHs2dO1cff/yxQx0LFy7UmjVrMkyvW7duprHk9rPIC3d3dy1evFht2rRxmP7999/rr7/+kpubW4b3fPTRR7p69ap69+7tMP3QoUM6evSoPvjgAz355JM3HJsVPf744+rVq1emn0t+cHd3V0REhN58802NGDFCNpvN6fcuWrRIVatW1U8//aSDBw9m6AOFsT3BUWb7k+u/hKSXSUxM1JYtWzR//nxt2rRJu3fvlru7u0PZZcuWqXfv3ipXrpwGDhyo0NBQ/fnnn/rwww+1dOlSLVmyRA899FCGOPLSzwvKjz/+qFdeeUX9+/eXr69vrt9/I32wsPZRN9rGopS+jWzatMlhW4qPj9fu3btVsmRJbd68We3bt7fPO378uI4fP65evXrd8PJzc0zPTd/J63cFjms3t2nTpmnz5s165JFH1KhRI0VHR+vdd99V06ZNtWXLFqeTwmeeeUYtWrRwmHbttnP27FlNnjxZlStXVuPGjbM98XCtv/76S6+99po8PT2dbtO1cjrup0tKStLUqVOd/jE6vb+kpKQoOjpaGzZs0KhRo/Tmm29q5cqVatSoUY51FPe+U5z7zdGjR3Xx4kVFREQoODhYCQkJ+u9//6sHH3xQ77//vgYPHpzt+3fs2KE2bdooJCREEydOVFpamt577z21bdtWP/30k2rXru1Qvn///lq6dKlGjRqlmjVrav78+brvvvu0fv16+zFpyJAhCg8Pd3ifMUZDhw5V1apVVbFixdw10uTSfffdZ/z8/MyFCxcyzIuJibH/PW/ePCPJbNu2Lcc6nSnrbH2HDx82Xl5epk6dOubkyZMZ5h84cMDMmDEj0/cOGzbM5OYjcfazMMaYiRMnGknmzJkzmdZVv35907ZtW/v/6e19+OGHTYUKFUxKSopD+UGDBplmzZqZKlWqmPvvv99hXqNGjUzfvn0zLOP77783ksznn3/uZAvzLj3+I0eOFPiyjDHm0qVLhbIcY4z5+eefjSSzbt06p99z+PBhI8ksW7bM+Pn5mUmTJmUok5vtCTfmRvY5Y8eONZLMZ5995jD94MGDpnTp0qZOnTrm9OnTDvPOnDlj6tSpYzw9Pc2hQ4cyLCMv/dxZue0br7/+er73XWf3B4W1jyqINhpTePuh0NBQc/vttztMW716tbHZbKZ3796mY8eODvMWL15sJJkvvvjCPi2nPtC2bVtTv359p8tfKzd9Jzf1ZobjWuE4c+ZMpt+pbtTmzZtNUlKSw7T9+/cbNzc306dPnxzfv379eqfWf2Jiojl16pQxxpht27YZSWbevHk51v/oo4+au+++O0N/cIYzx/30bapJkybGzc3NnDhxwmF+bvrhunXrjIeHh6lSpYpJSEjIMb7C6juF3W+MsVbfccbRo0cz/e6XG1evXjWNGzc2tWvXzrHsfffdZ8qWLWvOnj1rn3by5Enj5eVlHn74YYeyW7duNZLM66+/bp925coVU716dRMWFpbtcn744Qcjybz66qu5bI0xuR7ie+jQIdWvXz/TX539/f1zW12+mz59ui5duqQPP/xQQUFBGebXqFFDI0eOzJdlFcZn0bt3b507d87hVHpycrKWLl2qxx57LEP5I0eO6LfffsvwK0b//v3Vtm1bSdIjjzwim81mH1Lcv39/Va1aNUNd6cOSr3XixAk98cQTCggIkJubm+rXr6+PPvroBlv5f8v6448/1LNnT3l7e6t8+fIaOXKkEhMTMy27d+9ePfbYYypbtqz9F5zMrjk4ceKEBg4cqODgYLm5uSk0NFRPPfWUkpOT89S2Zs2aqVy5cvriiy+cbt+iRYtUtmxZ3X///erRo4cWLVqUocyNbk/pn8v+/fvVt29f+fj4yM/PTxMmTJAxRsePH1fXrl3l7e2twMBAvfHGG/b3rl+/Psthg4sXL5bNZlNUVJRTbb148aJGjRqlqlWrys3NTf7+/rrnnnu0Y8cOexlnt7kbaVNBufPOOyX9vb6u9frrryshIUFz586Vn5+fw7wKFSro/fff1+XLlzV9+vQMdea2n2clu74h5byNT5o0SWPGjJEkhYaG2ofm/Pnnnzp69Kiefvpp1a5dWx4eHipfvrweeeSRDNf35PW6n+z2Uc7ELsmpGLNrY3ocudk28/pZS871lcy0adNGv/zyi/1yFUnavHmz6tevr86dO2vLli0O9x/YvHmzbDab7rjjjmzrLWhZ9Z28svpx7drl5XRsy+1xLT3unI5t+dW23bt3q3LlyuratatWrlypq1ev5u0DuU7r1q3l6urqMK1mzZqqX7++fv/991zVdfHixSzjcnNzU2BgYK7q27hxo5YuXaoZM2bk6n3pnDnupxs/frxSU1M1derUPC1Lku6++25NmDBBR48e1SeffJJt2az6Tm76jWTt74RW6DfZST/Gd+rUyT7a6ka4uLgoJCTEqUtXfvjhB4WHh6t8+fL2aUFBQWrbtq2++uorXbp0yT596dKlcnFxcTgr6+7uroEDByoqKkrHjx/Pcjnp3x9z8z0mXa6H+FapUkVRUVHavXu3U0Mv4uLiMlx3abPZHD6U3Mipvi+//FLVqlVzuNamoOT2s8iLqlWrKiwsTJ9++qk6d+4sSfrmm28UFxenXr166Z133nEo/+OPP0qSmjZt6jB9yJAhqlixol577TX7UJiAgIBcxRITE6NWrVrJZrNp+PDh8vPz0zfffKOBAwcqPj5eo0aNyntD/6dnz56qWrWqIiMjtWXLFr3zzju6cOGCFi5cmKHsI488opo1a+q1116TMSbT+k6ePKnbb79dsbGxGjx4sOrUqaMTJ05o6dKlSkhIsB8Yc9u2pk2bavPmzU63a9GiRXr44Yfl6uqq3r17a/bs2dq2bZvDkKT82p4effRR1a1bV1OnTtWqVav0z3/+U+XKldP777+vu+++W9OmTdOiRYs0evRotWjRQnfddZfatWunkJAQLVq0KMMQ1EWLFql69eoKCwtzavlDhw7V0qVLNXz4cNWrV0/nzp3Tpk2b9Pvvv2fYLguyTc7IbH9SoUKFbN+TfhApW7asw/Qvv/xSVatWtX8Jv95dd92lqlWratWqVRnm5baf5ySzvuHMNv7www9r//79+vTTT/XWW2/ZP4v0sj/++KN69eqlSpUq6c8//9Ts2bPVrl077d27V6VLl85VjNfLbh/lbP/ctm1bjjFm18a8yOtnLeW9r7Rp00Yff/yxtm7dak/INm/erNatW6t169aKi4vT7t277cP8Nm/erDp16mR63M2sD0h/X5+WmRs5pmfVd/Ja781yXJOcP7Y5c1yTnDu25WfbmjRpogkTJmj+/Pnq2rWrgoKCFBERoSeeeEI1a9bM68eSKWOMYmJiVL9+faffM2DAAF26dEkuLi6688479frrr6t58+Z5jiE1NVUjRozQk08+qYYNG+apDmeO++lCQ0PVr18/ffDBB3rhhRcUHBycp2U+/vjjGj9+vL799lsNGjQoy3JZ9Z3skujrFdfvhAXdrj179ujDDz/Uxx9/rLNnz6p27dp67bXX8tSPLl++rCtXriguLk4rV67UN998o0cffTTH9yUlJcnDwyPD9NKlSys5OVm7d+9Wq1atJEm//PKLatWqJW9vb4eyt99+uyRp586dCgkJyVBXSkqK/vOf/6h169aZ/uiRo9yecv3222+Ni4uLcXFxMWFhYeb55583/+///T+TnJzsUC79lH5mLzc3t0zLOjPcLrv64uLijCTTtWvX3DbLGJP7Ib7OfhbG5H2I77Zt28y7775rypQpYx+y8cgjj5j27dsbY0yGoX8vvfSSkWQuXryYYRlZDYWJiIgwVapUyTLmdAMHDjRBQUEOQwKMMaZXr17Gx8fHYUhJbod0pC/rwQcfdJj+9NNPG0nm119/zVC2d+/eGeq5frn9+vUzJUqUyHTbSktLy1PbjDFm8ODBxsPDw6m2pQ8JXrNmjX25lSpVMiNHjnQol5vtKTPpn8vgwYPt065evWoqVapkbDabmTp1qn36hQsXjIeHh4mIiLBPGzdunHFzczOxsbH2aadPnzYlS5Y0EydOdCoGY4zx8fExw4YNy7aMs9vcjbYpK9ntT64vs3btWnPmzBlz/Phxs3TpUuPn52fc3NzM8ePH7WVjY2Od2vc8+OCDRpKJj493WEZu+3lWsusbzm7jWQ1/zWzIWFRUlJFkFi5caJ+WWd93dn+Q1T7K2didjTG7Ib653TZv5LN2pq9kZs+ePUaSmTJlijHGmJSUFOPp6WkWLFhgjDEmICDAzJo1yxhjTHx8vHFxcTGDBg1yqCO7PpD+ymxoYW6O6c70ndzUmxkrH9euXV5Ox7bcHNeMce7YltvjmjPS0tLMd999Z/r27Ws8PDyMJHPXXXeZBQsW5Km+zHz88cdGkvnwww9zLLt582bTvXt38+GHH5ovvvjCREZGmvLlyxt3d3ezY8eOTN/jzBDfd9991/j4+Ngv18jtEF9nj/vXHgMOHTpkSpYsaZ555hn7/LwMtffx8TG33XZbjjFm1nec7TfGOL99FWS/ubbs9X3HSv0mPj7efPDBB6Zly5ZGkilTpowZOHCg2bx5c67rutaQIUPs+8sSJUqYHj16mPPnz+f4voYNG5patWqZq1ev2qclJSWZypUrG0lm6dKl9un169c3d999d4Y60o9Dc+bMyXQZX375pZFk3nvvvTy0LA9DfO+55x5FRUXpwQcf1K+//qrp06erY8eOqlixolauXJmh/KxZs7RmzRqH1zfffJPbxTpVX3x8vCSpTJkyea4/N3L7WeRVz549deXKFX311Ve6ePGivvrqqyxPl587d04lS5aUl5dXvi1f+vsXzf/+97/q0qWLjDE6e/as/dWxY0fFxcXlOCzNGcOGDXP4P/3GQF9//XWGskOHDs22rrS0NK1YsUJdunTJ9JfU9OEqeWlb2bJldeXKFSUkJOTYpkWLFikgIMB+0xKbzaZHH31US5YsUWpqqr1cfm1P197swMXFRc2bN5cxRgMHDrRP9/X1Ve3atXX48GH7tH79+ikpKUlLly61T/vss8909epV9e3b1+nl+/r6auvWrTp58qTT78lJXtuUk8z2J9cLDw+Xn5+fQkJC1KNHD3l6emrlypWqVKmSvUz6nRNz2vekz0/fV10rN/08J9f3jfzov9f+2pqSkqJz586pRo0a8vX1zZe+n5XcxF4UMd7IZ53XvlK3bl2VL1/efufnX3/9VZcvX7aPHGrdurV9hEdUVJRSU1Mz3IArXWZ9YM2aNVneZCU3x3Rn+k5e6s0vhXVck5w/tuV0XJOcO7YVVNtsNpvat2+vjz/+WNHR0ZozZ46SkpIUERGhoKAgPfXUU7pw4UKu6033xx9/aNiwYQoLC1NERESO5Vu3bq2lS5fqiSee0IMPPqgXXnhBW7Zskc1m07hx4/IUw7lz5/Tyyy9rwoQJeR5Z4exx/1rVqlXT448/rrlz5+rUqVN5Wq4keXl55epuvnlRXL8T5ne7oqOj9cQTTygoKEiDBw+Wu7u75s+fr+joaP373/++4dGeo0aN0po1a7RgwQJ17txZqampGS5fy8zTTz+t/fv3a+DAgdq7d692796tfv362be7ay8fuXLlSqY3mkq/0d21Za+1ePFilSpVSj179sxL03I/xFeSWrRooWXLlik5OVm//vqrli9frrfeeks9evTQzp07Va9ePXvZ22+//YaGWVwvu/rSTz8XdMe8Vm4+i5xkdUdYPz8/hYeHa/HixUpISFBqaqp69OiRX01wypkzZxQbG6u5c+dq7ty5mZY5ffr0DS/n+iEO1atXV4kSJTIdm5/ZHR+vdebMGcXHx+c4XDYvbTP/Gz6S0118U1NTtWTJErVv315HjhyxT2/ZsqXeeOMNrVu3Tvfee699en5sT5UrV3b438fHR+7u7hmGrvr4+OjcuXP2/+vUqaMWLVpo0aJF9sRv0aJFatWqVa7uuj19+nRFREQoJCREzZo103333ad+/fqpWrVqTteRX23KiTP7p1mzZqlWrVqKi4vTRx99pI0bN2bYWacnnjnte7JLZPOzn1/fN/Kj/165ckWRkZGaN2+eTpw44TCEKi4uLk9xOiM3sRdFjDfyWee1r9hsNrVu3VobN25UWlqaNm/eLH9/f3s/bd26td59911JsieqWSWoWfWBsmXLZjr0NzfHdGf6Tl7qzS+FdVyTnD+25XRck5w7thVG27y9vTVkyBBFRETo1Vdf1auvvqo5c+ZoyJAhmQ7jzkl0dLTuv/9++fj42K97y4saNWqoa9euWrZsmVJTU3Ndz0svvaRy5crl+c75uT3uX7/sjz/+WFOnTtXbb7+dp+VfunSpwO8JU1y/E+Z3u/744w/NmzdPJUuW1PTp0zVy5EiVKlXK6ffnpE6dOqpTp46kv08y3HvvverSpYu2bt2a7ffToUOH6vjx43r99de1YMECSVLz5s31/PPP69VXX3U4yeXh4aGkpKQMdaRfC5zZUOFLly7piy++UMeOHfN8SWeeEtR0rq6uatGihVq0aKFatWppwIAB+vzzzzVx4sQbqTbPvL29FRwcrN27dxf6snP6LHL6pSEhISHDbfev9dhjj2nQoEGKjo5W586ds3w0Qvny5XX16lVdvHjR6TPJWW3E1/7Kl37Djb59+2b5q6YztzXPrew6WGadIi/y0rYLFy6odOnSOcbw3Xff6dSpU1qyZImWLFmSYf6iRYsyPVDdSN/K7GCc1QH62i/w0t87uJEjR+qvv/5SUlKStmzZYv+i66yePXvqzjvv1PLly/Xtt9/q9ddf17Rp07Rs2TL79ZXObHM5xe9sm27UtV+cu3XrpjZt2uixxx7Tvn377DtxHx8fBQUF6bfffsu2rt9++00VK1bMcC1HOmf7eU6u3y7zo/+OGDFC8+bN06hRoxQWFiYfHx/ZbDb16tXL4YY8+S03sedHjLndNm/ks3amr2SlTZs2+vLLL7Vr1y779afpWrdurTFjxujEiRPatGmTgoODb+gHorxypu8UFCsf16Ss4yvK41pubdu2TR999JGWLFmi2NhYtWzZUgMHDszy0XzZiYuLU+fOnRUbG6sffvghz9dgpgsJCVFycrIuX76c5f42MwcOHNDcuXM1Y8YMh5ENiYmJSklJ0Z9//ilvb2+VK1cuyzryetyX/j6L2rdvX82dO9f+TO7c+OuvvxQXF5fnRzk6u/8rrt8J87tdLVq00LvvvqsPP/xQY8aM0bRp09S3b18NGDCgQD6fHj16aMiQIdq/f3+GR8Vc79VXX9Xo0aO1Z88e+fj4qGHDhho/frwkqVatWvZyQUFBmT7DNP1sa2Z9dcWKFUpISFCfPn3y3JYbSlCvlX4QupFhCfnhgQce0Ny5cxUVFeX0TV3yW2afRfrDj/ft25fhYuKEhAQdP348yx2WJD300EMaMmSItmzZos8++yzLcum/pBw5csTpjb9s2bKZ3vXr6NGj9r/9/PxUpkwZpaamZrhDcH46cOCAw69gBw8eVFpaWp4usPbz85O3t3eOP1jkpW1Hjhxx6iC8aNEi+fv7a9asWRnmLVu2TMuXL9ecOXOy3bEWZt/q1auXnnvuOX366ae6cuWKSpUq5dQF99cLCgrS008/raefflqnT59W06ZN9eqrr9q/dDuzzVmRi4uLIiMj1b59e7377rsOXyAeeOABffDBB9q0aVOmZ6t++OEH/fnnnxoyZEiW9Tvbz3MrN9t4Vl8Ali5dqoiICIc7JScmJjp1x8AbkZvYnY0xuy85N7pt5nZ/klNfycq1z0PdvHmzw407mjVrJjc3N23YsEFbt27Vfffd51TsBSm7vlMQrHRckwr/2FZQbTt9+rQ+/vhjzZs3T3v27FH58uXVv39/DRw4MM8390tMTFSXLl20f/9+rV27Nlcjz7Jy+PBhubu75/qHkBMnTigtLU3PPPOMnnnmmQzzQ0NDNXLkyGzv7Hujx/2XXnpJn3zyiaZNm5ar2CXp448/liR17Ngx1++VnN//FdfvhPndLk9PTw0bNkzDhg3Tjh079O9//1vz5s3TjBkz1LRpUw0YMECPPfZYtj945Eb6STBnRwxdf/f5tWvXqlKlSvZcQvr7Bmnr169XfHy8w489W7dutc+/3qJFi+Tl5aUHH3wwL82QJOX6GtT169dneoYifTx4Thl7QXv++efl6empJ598UjExMRnmHzp0KM/DJq6Xm8+iQ4cOcnV11ezZszP8kj937lxdvXo12y8kXl5emj17tiZNmqQuXbpkWS49Kf/555+dbkf16tUVFxfncPbn1KlTDo8ccXFxUffu3fXf//4308595swZp5eXnet36OkPrc7py1pmSpQooW7duunLL7/M9PNIX3d5aduOHTtyvHbgypUrWrZsmR544AH16NEjw2v48OG6ePGi/fpSK/StChUqqHPnzvrkk0+0aNEiderUKce72l4rNTU1w47R399fwcHBDkNEnNnmrKpdu3a6/fbbNWPGDIfb3Y8ZM0YeHh4aMmRIhmHG58+f19ChQ1W6dGn7I04y42w/z63cbOPpD6S//kuKi4tLhu1z5syZWZ5ZzElCQoL++OOPTIeR5jV2Z2PMqo3SjW+bzsbrbF/JSvPmzeXu7q5FixbpxIkTDvsjNzc3NW3aVLNmzdLly5ezHN5b2LLqOwXBSsc1qfCPbfndtuPHj6tbt26qWLGixowZo6CgIC1ZskQnT57UW2+9lefkNDU1VY8++qiioqL0+eefZ3tiIbN9Rmbt+PXXX7Vy5Urde++9KlEid19zGzRooOXLl2d41a9fX5UrV9by5csd7n1wvdwe9zNTvXp19e3bV++//76io6Odjv27777TlClTFBoamuczV87u/4rrd8KCbFfTpk313nvv6dSpU1qwYIG8vLw0YsQIBQcHq2fPnrmqO7NhxikpKVq4cKE8PDwcfuRx9lj72Wefadu2bRo1apRDv+nRo4dSU1MdhjwnJSVp3rx5atmyZYaTbmfOnNHatWv10EMP3dDd/XN9BnXEiBFKSEjQQw89pDp16ig5OVk//vijPvvsM1WtWlUDBgxwKP/NN9/ojz/+yFBP69atMww5+uijj7R69eoMZa99bmlO9VWvXl2LFy+2P5aiX79+atCggT3Ozz//XP37989tszOVm8/C399fL7/8sl566SXdddddevDBB1W6dGn9+OOP+vTTT+3jxrPjzA0DqlWrpgYNGmjt2rV64oknnGpHr169NHbsWD300EN65plnlJCQoNmzZ6tWrVoOF4NPnTpV69evV8uWLTVo0CDVq1dP58+f144dO7R27VqdP3/eqeVl58iRI3rwwQfVqVMnRUVF6ZNPPtFjjz2mxo0b56m+1157Td9++63atm2rwYMHq27dujp16pQ+//xzbdq0yT6EMjdt2759u86fP6+uXbtmu+yVK1fq4sWLWf6C1KpVK/n5+WnRokV69NFHc923Ckq/fv3s1z5OmTIlV++9ePGiKlWqpB49eqhx48by8vLS2rVrtW3bNoezWs5uc1Y1ZswYPfLII5o/f779xgw1a9bUggUL1KdPHzVs2FADBw60P9vsww8/1NmzZ/Xpp5+qevXq2dbtTD/PC2e38WbNmkmSXnzxRfXq1UulSpVSly5d9MADD+jjjz+Wj4+P6tWrp6ioKK1duzbP15f89NNPat++vSZOnKhJkyblS+zOxphVGz09PfNl23QmXmf7SlbSLwP44Ycf5ObmZm9TutatW9vryc8ENTfH9Mxk1nfyo97rWem4JhXNsS0/23bo0CHt2LFD48aN0xNPPJG3x0Zk4h//+IdWrlypLl266Pz58xme33ntDfoy22c8+uij8vDwUOvWreXv76+9e/dq7ty5Kl26dIZnir777ruKjY21D9398ssv9ddff0n6+/ucj4+PKlSooG7dumWIM/2MaWbzrpXb435WXnzxRX388cfat29fpo/bSe8vV69eVUxMjL777jutWbNGVapU0cqVK7O9ZCw7udn/FdfvhAXdLg8PD/Xr10/9+vXTgQMH9OGHH2rBggU6ceKE0zflGjJkiOLj43XXXXepYsWKio6O1qJFi/THH3/ojTfecBg5kFm/2bhxoyZPnqx7771X5cuX15YtWzRv3jx16tTJIeeS/r52+pFHHtG4ceN0+vRp1ahRQwsWLLB/t7le+s01b2R4r6TcP2bmm2++MU888YSpU6eO8fLyMq6urqZGjRpmxIgRJiYmxl4up1vYX3t775zKHj9+PFf1GWPM/v37zaBBg0zVqlWNq6urKVOmjLnjjjvMzJkzTWJiYqZty+1jZpz9LK71ySefmFatWhlPT0/j5uZm6tSpY1555ZUMMTlzK3FjMn/8xJtvvmm8vLwy3Ao7q9vxG/P3I04aNGhgXF1dTe3atc0nn3yS6W3FY2JizLBhw0xISIgpVaqUCQwMNB06dDBz587NNP7cPmZm7969pkePHqZMmTKmbNmyZvjw4ebKlSuZls3skT2ZLffo0aOmX79+9kccVKtWzQwbNswkJSXlqW1jx441lStXdnhMTWa6dOli3N3dzeXLl7Ms079/f1OqVClz9uzZPG1PznwuERERxtPTM0P5rG6bn5SUZMqWLWt8fHwyfPY5SUpKMmPGjDGNGzc2ZcqUMZ6enqZx48aZ3mbcmW0uv9p0vdw82iqzMqmpqaZ69eqmevXqDrdpN8aY3377zfTu3dsEBQXZt6PevXubXbt25SkOY3L/mJmsHmfl7DY+ZcoUU7FiRVOiRAl7f7pw4YIZMGCAqVChgvHy8jIdO3Y0f/zxh6lSpYrDo32cfcxM+v7o2kcYZbePciZ2Z2PMqo3pbmTbdDbe3PSVrIwbN85IMq1bt84wb9myZfbHGVy/jRqT87aX1eMtcnNMd6bv5PbYfj0rH9eMcf7YltvjmjHOHducbVtOEhISTGpqaq7e44y2bdtmu/6vldk+4+233za33367KVeunClZsqQJCgoyffv2NQcOHMiwrCpVqmS5nJzWqbPHltwe97PrKxEREUbK/nFPrq6uJjAw0Nxzzz3m7bfftj/CzBlZ9R1n+40xzm1fBdlvri17fd+xQr9xVkpKSpa5SWY+/fRTEx4ebgICAkzJkiVN2bJlTXh4uPniiy8ylM2s3xw8eNDce++9pkKFCvZcJDIyMsP34nRXrlwxo0ePNoGBgcbNzc20aNHCrF69OtOyrVq1Mv7+/pked3LDZkw+31EERS4uLk7VqlXT9OnTsx2KYiWTJk3SK6+8ojNnzuRqSGlhS0pKUtWqVfXCCy9k+JWpuLh69aqCg4PVpUuXTH8dAwA452Y5tgFWQr9Brq9BhfX5+Pjo+eef1+uvv16gd9e8Fc2bN0+lSpVy6ll1N6sVK1bozJkz6tevX1GHAgAAgFtMvt3FF9YyduxYjR07tqjDUGpqao4Xfhf04wby09ChQ4ttcrp161b99ttvmjJlim677Ta1bdvWYb6z6/JmWp8AkFvsC4G8KW7fCVFwSFBRoI4fP57jw5OL6rm5cDR79mx98sknatKkiebPn59hvrPrMqcb3gDAzYx9IZA3fCeEs7gGFQUqMTFRmzZtyrZMtWrViuQh8sgd1iUAsC8E8oq+A2eRoAIAAAAALIEhvkAhSUtL08mTJ1WmTBnZbLaiDgcAADjBGKOLFy8qODhYJUpwf1GgoJGgAoXk5MmTCgkJKeowAABAHhw/flyVKlUq6jCAYo8EFSgkZcqUkfT3Ac7b27uIowEAWM7ly1Jw8N9/nzwpeXoWbTyQJMXHxyskJMR+HAdQsEhQgUKSPqzX29ubBBUAkJGLy//97e1NgmoxXJ4DFA4G0gMAAAAALIEEFQAAAABgCSSoAAAAAABL4BpUwEKMMbp69apSU1OLOhTcABcXF5UsWZLrlQAAAHKJBBWwiOTkZJ06dUoJCQlFHQryQenSpRUUFCRXV9eiDgUAAOCmQYIKWEBaWpqOHDkiFxcXBQcHy9XVlbNvNyljjJKTk3XmzBkdOXJENWvW5MHuAAAATiJBBSwgOTlZaWlpCgkJUenSpYs6HNwgDw8PlSpVSkePHlVycrLc3d2LOiQAAICbAj/rAxbCmbbig3UJAACQe3yDAgAAAABYAgkqAAAAAMASSFABAAAAAJZAggogT2w2W7avSZMmFXWIAAAAuMlwF18AeXLq1Cn735999plefvll7du3zz7Ny8vL/rcxRqmpqSpZkl0OAAAAssYZVMCqjJEuXy78lzFOhRcYGGh/+fj4yGaz2f//448/VKZMGX3zzTdq1qyZ3NzctGnTJvXv31/dunVzqGfUqFFq166d/f+0tDRFRkYqNDRUHh4eaty4sZYuXZqPHywAAACsitMZgFUlJEjXnIUsNJcuSZ6e+VLVCy+8oH/961+qVq2aypYt69R7IiMj9cknn2jOnDmqWbOmNm7cqL59+8rPz09t27bNl7gAANl7a83+og4h1569p1ZRhwAgH5CgAigwkydP1j333ON0+aSkJL322mtau3atwsLCJEnVqlXTpk2b9P7775OgAgAAFHMkqIBVlS7999nMolhuPmnevHmuyh88eFAJCQkZktrk5GTddttt+RYXAAAArIkEFbAqmy3fhtoWFc/r4i9RooTMdde4pqSk2P++9L+EfNWqVapYsaJDOTc3twKKEgAAAFZBggqg0Pj5+Wn37t0O03bu3KlSpUpJkurVqyc3NzcdO3aM4bwAAAC3IBJUAIXm7rvv1uuvv66FCxcqLCxMn3zyiXbv3m0fvlumTBmNHj1azz77rNLS0tSmTRvFxcVp8+bN8vb2VkRERBG3AAAAAAWJBBVAoenYsaMmTJig559/XomJiXriiSfUr18/7dq1y15mypQp8vPzU2RkpA4fPixfX181bdpU48ePL8LIAQAAUBhs5voLwgAUiPj4ePn4+CguLk7e3t4O8xITE3XkyBGFhobK3d29iCJEfmKdAsi1y5f/7/Fi+fjIr7zgMTP/J7vjN4D8V6KoAwAAAAAAQCJBBQAAAABYBAkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsoWdQBAMjeW2v2F9qynr2nVqEtCwAAALgeZ1AB3FKMMXr55ZcVFBQkDw8PhYeH68CBA9m+Z9KkSbLZbA6vOnXqFFLEAAAAtw4SVAC3lOnTp+udd97RnDlztHXrVnl6eqpjx45KTEzM9n3169fXqVOn7K9NmzYVUsQAAAC3DhJUAHmWlpamyMhIhYaGysPDQ40bN9bSpUvt8zds2CCbzaZVq1apUaNGcnd3V6tWrbR79257maNHj6pLly4qW7asPD09Vb9+fX399dcFEq8xRjNmzNBLL72krl27qlGjRlq4cKFOnjypFStWZPvekiVLKjAw0P6qUKFCgcQIAABwKyNBBZBnkZGRWrhwoebMmaM9e/bo2WefVd++ffX99987lBszZozeeOMNbdu2TX5+furSpYtSUlIkScOGDVNSUpI2btyoXbt2adq0afLy8spymUOHDpWXl1e2r6wcOXJE0dHRCg8Pt0/z8fFRy5YtFRUVlW1bDxw4oODgYFWrVk19+vTRsWPHnPmIAAAAkAvcJAlAniQlJem1117T2rVrFRYWJkmqVq2aNm3apPfff19t27a1l504caLuueceSdKCBQtUqVIlLV++XD179tSxY8fUvXt3NWzY0F5HdiZPnqzRo0fnKebo6GhJUkBAgMP0gIAA+7zMtGzZUvPnz1ft2rV16tQpvfLKK7rzzju1e/dulSlTJk+xAAAAICMSVAB5cvDgQSUkJNgTz3TJycm67bbbHKalJ7CSVK5cOdWuXVu///67JOmZZ57RU089pW+//Vbh4eHq3r27GjVqlOVy/f395e/vn48tyVnnzp3tfzdq1EgtW7ZUlSpV9J///EcDBw4s1FgAAACKM4b4AsiTS5cuSZJWrVqlnTt32l979+51uA41J08++aQOHz6sxx9/XLt27VLz5s01c+bMLMvfyBDfwMBASVJMTIzD9JiYGPs8Z/j6+qpWrVo6ePCg0+8BAABAzjiDCiBP6tWrJzc3Nx07dsxhOG9mtmzZosqVK0uSLly4oP3796tu3br2+SEhIRo6dKiGDh2qcePG6YMPPtCIESMyretGhviGhoYqMDBQ69atU5MmTSRJ8fHx2rp1q5566imn67l06ZIOHTqkxx9/PE9xAAAAIHMkqADypEyZMho9erSeffZZpaWlqU2bNoqLi9PmzZvl7e2tiIgIe9nJkyerfPnyCggI0IsvvqgKFSqoW7dukqRRo0apc+fOqlWrli5cuKD169c7JK/Xu5EhvjabTaNGjdI///lP1axZU6GhoZowYYKCg4Pt8UhShw4d9NBDD2n48OGSpNGjR6tLly6qUqWKTp48qYkTJ8rFxUW9e/fOUxwAAADIHAkqYHHP3lOrqEPI0pQpU+Tn56fIyEgdPnxYvr6+atq0qcaPH+9QburUqRo5cqQOHDigJk2a6Msvv5Srq6skKTU1VcOGDdNff/0lb29vderUSW+99VaBxfz888/r8uXLGjx4sGJjY9WmTRutXr1a7u7u9jKHDh3S2bNn7f//9ddf6t27t86dOyc/Pz+1adNGW7ZskZ+fX4HFCQAAcCuyGWNMUQcB3Ari4+Pl4+OjuLg4eXt7O8xLTEzUkSNHFBoa6pAo3ew2bNig9u3b68KFC/L19S3qcApVcV2nAArQ5ctS+nX0ly5Jnp5FFspba/YX2bLzqqB+0M3u+A0g/3GTJAAAAACAJZCgAgAAAAAsgQQVxd7GjRvVpUsXBQcHy2azacWKFfZ5KSkpGjt2rBo2bChPT08FBwerX79+OnnypEMd58+fV58+feTt7S1fX18NHDjQ/pgVZK1du3Yyxtxyw3sBAACQNySoKPYuX76sxo0ba9asWRnmJSQkaMeOHZowYYJ27NihZcuWad++fXrwwQcdyvXp00d79uzRmjVr9NVXX2njxo0aPHhwYTUBAAAAuCVwF18Ue507d1bnzp0znefj46M1a9Y4THv33Xd1++2369ixY6pcubJ+//13rV69Wtu2bVPz5s0lSTNnztR9992nf/3rXwoODi7wNgAAAAC3As6gAteJi4uTzWazD0uNioqSr6+vPTmVpPDwcJUoUUJbt27Nsp6kpCTFx8c7vAAAAABkjQQVuEZiYqLGjh2r3r17228lHx0dLX9/f4dyJUuWVLly5RQdHZ1lXZGRkfLx8bG/QkJCCjR2AAAA4GZHggr8T0pKinr27CljjGbPnn3D9Y0bN05xcXH21/Hjx/MhSgAAAKD44hpUQP+XnB49elTfffedw4O4AwMDdfr0aYfyV69e1fnz5xUYGJhlnW5ubnJzcyuwmAEAAIDihgQVt7z05PTAgQNav369ypcv7zA/LCxMsbGx2r59u5o1ayZJ+u6775SWlqaWLVsWfIDrIwt+Genajyu8ZQEAAADXYYgvir1Lly5p586d2rlzpyTpyJEj2rlzp44dO6aUlBT16NFDP//8sxYtWqTU1FRFR0crOjpaycnJkqS6deuqU6dOGjRokH766Sdt3rxZw4cPV69evbiD701o2bJluvfee1W+fHnZbDb7dpGTzz//XHXq1JG7u7saNmyor7/+umADBQAAuAVxBhXF3s8//6z27dvb/3/uueckSREREZo0aZJWrlwpSWrSpInD+9avX6927dpJkhYtWqThw4erQ4cOKlGihLp376533nmnUOJH/rp8+bLatGmjnj17atCgQU6958cff1Tv3r0VGRmpBx54QIsXL1a3bt20Y8cONWjQoIAjBoDC1+rY3KIOIQ/+VdQBAMgHnEFFsdeuXTsZYzK85s+fr6pVq2Y6zxhjT04lqVy5clq8eLEuXryouLg4ffTRR/Ly8iq6RllEWlqaIiMjFRoaKg8PDzVu3FhLly61z9+wYYNsNptWrVqlRo0ayd3dXa1atdLu3bvtZY4ePaouXbqobNmy8vT0VP369Qv07OTjjz+ul19+WeHh4U6/5+2331anTp00ZswY1a1bV1OmTFHTpk317rvvFlicAAAAtyISVAB5FhkZqYULF2rOnDnas2ePnn32WfXt21fff/+9Q7kxY8bojTfe0LZt2+Tn56cuXbooJSVFkjRs2DAlJSVp48aN2rVrl6ZNm5Zt8j906FB5eXll+8pvUVFRGRLajh07KioqKt+XBQAAcCtjiC+APElKStJrr72mtWvXKiwsTJJUrVo1bdq0Se+//77atm1rLztx4kTdc889kqQFCxaoUqVKWr58uXr27Kljx46pe/fuatiwob2O7EyePFmjR48uoFZlLjo6WgEBAQ7TAgICsn0OLgAAAHKPBBVAnhw8eFAJCQn2xDNdcnKybrvtNodp6Qms9Pdw6dq1a+v333+XJD3zzDN66qmn9O233yo8PFzdu3dXo0aNslyuv7+//P3987ElAAAAsAqG+ALIk0uXLkmSVq1aZb9L8s6dO7V3716H61Bz8uSTT+rw4cN6/PHHtWvXLjVv3lwzZ87MsnxRDPENDAxUTEyMw7SYmJhsn4MLAACA3OMMKoA8qVevntzc3HTs2DGH4byZ2bJliypXrixJunDhgvbv36+6deva54eEhGjo0KEaOnSoxo0bpw8++EAjRozItK6iGOIbFhamdevWadSoUfZpa9ascTgzDAAAgBtHggogT8qUKaPRo0fr2WefVVpamtq0aaO4uDht3rxZ3t7eioiIsJedPHmyypcvr4CAAL344ouqUKGCunXrJkkaNWqUOnfurFq1aunChQtav369Q/J6vRsd4nv+/HkdO3ZMJ0+elCTt27dP0t9nSdPPiPbr108VK1ZUZGSkJGnkyJFq27at3njjDd1///1asmSJfv75Z82dezM+hgEAAMC6SFABq2s/rqgjyNKUKVPk5+enyMhIHT58WL6+vmratKnGjx/vUG7q1KkaOXKkDhw4oCZNmujLL7+Uq6urJCk1NVXDhg3TX3/9JW9vb3Xq1ElvvfVWgcW8cuVKDRgwwP5/r169JP19I6dJkyZJko4dO6YSJf7vCojWrVtr8eLFeumllzR+/HjVrFlTK1as4BmoAAAA+cxmjDFFHQRwK4iPj5ePj4/i4uLk7e3tMC8xMVFHjhxRaGio3N3diyjC/Ldhwwa1b99eFy5ckK+vb1GHU6iK6zoFUIAuX5bSr6O/dEny9CyyUKI+LNxLKfJD2MB/FUi92R2/AeQ/bpIEAAAAALAEElQAAAAAgCVwDSqAAtOuXTtxFQEAAACcxRlUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASeMwMYHHv7Xyv0Jb1dJOnC21ZAAAAwPU4gwrgltK/f3/ZbDaHV6dOnXJ836xZs1S1alW5u7urZcuW+umnnwohWgAAgFsLCSqAW06nTp106tQp++vTTz/Ntvxnn32m5557ThMnTtSOHTvUuHFjdezYUadPny6kiAEAAG4NJKgA8iwtLU2RkZEKDQ2Vh4eHGjdurKVLl9rnb9iwQTabTatWrVKjRo3k7u6uVq1aaffu3fYyR48eVZcuXVS2bFl5enqqfv36+vrrrws0bjc3NwUGBtpfZcuWzbb8m2++qUGDBmnAgAGqV6+e5syZo9KlS+ujjz4q0DgBAABuNSSoAPIsMjJSCxcu1Jw5c7Rnzx49++yz6tu3r77//nuHcmPGjNEbb7yhbdu2yc/PT126dFFKSookadiwYUpKStLGjRu1a9cuTZs2TV5eXlkuc+jQofLy8sr2lZMNGzbI399ftWvX1lNPPaVz585lWTY5OVnbt29XeHi4fVqJEiUUHh6uqKioHJcFAAAA53GTJAB5kpSUpNdee01r165VWFiYJKlatWratGmT3n//fbVt29ZeduLEibrnnnskSQsWLFClSpW0fPly9ezZU8eOHVP37t3VsGFDex3ZmTx5skaPHp3nuDt16qSHH35YoaGhOnTokMaPH6/OnTsrKipKLi4uGcqfPXtWqampCggIcJgeEBCgP/74I89xAAAAICMSVAB5cvDgQSUkJNgTz3TJycm67bbbHKalJ7CSVK5cOdWuXVu///67JOmZZ57RU089pW+//Vbh4eHq3r27GjVqlOVy/f395e/vn+e4e/XqZf+7YcOGatSokapXr64NGzaoQ4cOea4XAAAAN44hvgDy5NKlS5KkVatWaefOnfbX3r17Ha5DzcmTTz6pw4cP6/HHH9euXbvUvHlzzZw5M8vy+THE91rVqlVThQoVdPDgwUznV6hQQS4uLoqJiXGYHhMTo8DAwFwtCwAAANnjDCqAPKlXr57c3Nx07Ngxh+G8mdmyZYsqV64sSbpw4YL279+vunXr2ueHhIRo6NChGjp0qMaNG6cPPvhAI0aMyLSuGx3ie72//vpL586dU1BQUKbzXV1d1axZM61bt07dunWT9PfNodatW6fhw4fnWxwAAAAgQQWQR2XKlNHo0aP17LPPKi0tTW3atFFcXJw2b94sb29vRURE2MtOnjxZ5cuXV0BAgF588UVVqFDBnuyNGjVKnTt3Vq1atXThwgWtX7/eIXm93o0M8b106ZJeeeUVde/eXYGBgTp06JCef/551ahRQx07drSX69Chgx566CF7Avrcc88pIiJCzZs31+23364ZM2bo8uXLGjBgQJ7iAAAAQOZIUAGLe7rJ00UdQpamTJkiPz8/RUZG6vDhw/L19VXTpk01fvx4h3JTp07VyJEjdeDAATVp0kRffvmlXF1dJUmpqakaNmyY/vrrL3l7e6tTp0566623CiReFxcX/fbbb1qwYIFiY2MVHByse++9V1OmTJGbm5u93KFDh3T27Fn7/48++qjOnDmjl19+WdHR0WrSpIlWr16d4cZJAAAAuDE2Y4wp6iCAW0F8fLx8fHwUFxcnb29vh3mJiYk6cuSIQkND5e7uXkQR5r8NGzaoffv2unDhgnx9fYs6nEJVXNcpgAJ0+bKUfh39pUuSp2eRhRL1Yf5dSlFYwgb+q0Dqze74DSD/cZMkAAAAAIAlkKACAAAAACyBa1ABFJh27dqJqwgAAADgLM6gAgAAAAAsgQQVsBDONhYfrEsAAIDcI0EFLKBUqVKSpISEhCKOBPklfV2mr1sAAADkjGtQAQtwcXGRr6+vTp8+LUkqXbq0bDZbEUeFvDDGKCEhQadPn5avr69cXFyKOiQAAICbBgkqYBGBgYGSZE9ScXPz9fW1r1MAAAA4hwQVsAibzaagoCD5+/srJSWlqMPBDShVqhRnTgEAAPKABBWwGBcXF5IbAAAA3JK4SRIAAAAAwBJIUAEAAAAAlkCCCgAAAACwBBJUFHsbN25Uly5dFBwcLJvNphUrVjjMN8bo5ZdfVlBQkDw8PBQeHq4DBw44lDl//rz69Okjb29v+fr6auDAgbp06VIhtgIAAAAo/khQUexdvnxZjRs31qxZszKdP336dL3zzjuaM2eOtm7dKk9PT3Xs2FGJiYn2Mn369NGePXu0Zs0affXVV9q4caMGDx5cWE0AAAAAbgncxRfFXufOndW5c+dM5xljNGPGDL300kvq2rWrJGnhwoUKCAjQihUr1KtXL/3+++9avXq1tm3bpubNm0uSZs6cqfvuu0//+te/FBwcXGhtAQAAAIozzqDilnbkyBFFR0crPDzcPs3Hx0ctW7ZUVFSUJCkqKkq+vr725FSSwsPDVaJECW3dujXLupOSkhQfH+/wAgAAAJA1ElTc0qKjoyVJAQEBDtMDAgLs86Kjo+Xv7+8wv2TJkipXrpy9TGYiIyPl4+Njf4WEhORz9AAAAEDxQoIKFJBx48YpLi7O/jp+/HhRhwQAAABYGgkqbmmBgYGSpJiYGIfpMTEx9nmBgYE6ffq0w/yrV6/q/Pnz9jKZcXNzk7e3t8MLAAAAQNZIUHFLCw0NVWBgoNatW2efFh8fr61btyosLEySFBYWptjYWG3fvt1e5rvvvlNaWppatmxZ6DEDAAAAxRV38UWxd+nSJR08eND+/5EjR7Rz506VK1dOlStX1qhRo/TPf/5TNWvWVGhoqCZMmKDg4GB169ZNklS3bl116tRJgwYN0pw5c5SSkqLhw4erV69e3MEXAAAAyEckqCj2fv75Z7Vv397+/3PPPSdJioiI0Pz58/X888/r8uXLGjx4sGJjY9WmTRutXr1a7u7u9vcsWrRIw4cPV4cOHVSiRAl1795d77zzTqG3BQAAACjObMYYU9RBALeC+Ph4+fj4KC4ujutRAQAZXb4seXn9/felS5KnZ5GFEvXh6CJbdl6FDfxXgdTL8RsoXFyDCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkhQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAIAAAAALIEEFQAAAABgCSSoAAAAAABLIEEFAAAAAFgCCSoAAAAAwBJIUAEAAAAAlkCCCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkhQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAKSUlNTNWHCBIWGhsrDw0PVq1fXlClTZIyxlzHG6OWXX1ZQUJA8PDwUHh6uAwcOFGHUAAAAQPFCggpImjZtmmbPnq13331Xv//+u6ZNm6bp06dr5syZ9jLTp0/XO++8ozlz5mjr1q3y9PRUx44dlZiYWISRAwAAAMVHyaIOALCCH3/8UV27dtX9998vSapatao+/fRT/fTTT5L+Pns6Y8YMvfTSS+rataskaeHChQoICNCKFSvUq1evIosdAAAAKC44gwpIat26tdatW6f9+/dLkn799Vdt2rRJnTt3liQdOXJE0dHRCg8Pt7/Hx8dHLVu2VFRUVKZ1JiUlKT4+3uEFAAAAIGucQQUkvfDCC4qPj1edOnXk4uKi1NRUvfrqq+rTp48kKTo6WpIUEBDg8L6AgAD7vOtFRkbqlVdeKdjAAQAAgGKEM6iApP/85z9atGiRFi9erB07dmjBggX617/+pQULFuS5znHjxikuLs7+On78eD5GDAAAABQ/nEEFJI0ZM0YvvPCC/VrShg0b6ujRo4qMjFRERIQCAwMlSTExMQoKCrK/LyYmRk2aNMm0Tjc3N7m5uRV47AAAAEBxwRlUQFJCQoJKlHDsDi4uLkpLS5MkhYaGKjAwUOvWrbPPj4+P19atWxUWFlaosQIAAADFFWdQAUldunTRq6++qsqVK6t+/fr65Zdf9Oabb+qJJ56QJNlsNo0aNUr//Oc/VbNmTYWGhmrChAkKDg5Wt27dijZ4AAAAoJggQQUkzZw5UxMmTNDTTz+t06dPKzg4WEOGDNHLL79sL/P888/r8uXLGjx4sGJjY9WmTRutXr1a7u7uRRg5AAAAUHzYjDGmqIMAbgXx8fHy8fFRXFycvL29izocAIDVXL4seXn9/felS5KnZ5GFEvXh6CJbdl6FDfxXgdTL8RsoXFyDCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkhQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAIAAAAALIEEFQAAAABgCSSoAAAAAABLIEEFAAAAAFgCCSoAAAAAwBJIUAEAAAAAlkCCCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZQsqgDAAAAsKr3dr6XL/U83eTpfKkHAIo7zqACAAAAACyBBBUAAAAAYAkM8QUAALhJMOQYQHHHGVQAAAAAgCWQoAIAAAAALIEEFQAAAABgCSSoAAAAAABLIEEFAAAAAFgCCSoAAAAAwBJIUIH/OXHihPr27avy5cvLw8NDDRs21M8//2yfb4zRyy+/rKCgIHl4eCg8PFwHDhwowogBAACA4oUEFZB04cIF3XHHHSpVqpS++eYb7d27V2+88YbKli1rLzN9+nS98847mjNnjrZu3SpPT0917NhRiYmJRRg5AAAAUHyULOoAACuYNm2aQkJCNG/ePPu00NBQ+9/GGM2YMUMvvfSSunbtKklauHChAgICtGLFCvXq1StDnUlJSUpKSrL/Hx8fX4AtAAAAAG5+nEEFJK1cuVLNmzfXI488In9/f91222364IMP7POPHDmi6OhohYeH26f5+PioZcuWioqKyrTOyMhI+fj42F8hISEF3g4AAADgZkaCCkg6fPiwZs+erZo1a+r//b//p6eeekrPPPOMFixYIEmKjo6WJAUEBDi8LyAgwD7veuPGjVNcXJz9dfz48YJtBAAAAHCTY4gvICktLU3NmzfXa6+9Jkm67bbbtHv3bs2ZM0cRERF5qtPNzU1ubm75GSYAAABQrHEGFZAUFBSkevXqOUyrW7eujh07JkkKDAyUJMXExDiUiYmJsc8DAAAAcGNIUAFJd9xxh/bt2+cwbf/+/apSpYqkv2+YFBgYqHXr1tnnx8fHa+vWrQoLCyvUWAEAAIDiiiG+gKRnn31WrVu31muvvaaePXvqp59+0ty5czV37lxJks1m06hRo/TPf/5TNWvWVGhoqCZMmKDg4GB169ataIMHAAAAigkSVEBSixYttHz5co0bN06TJ09WaGioZsyYoT59+tjLPP/887p8+bIGDx6s2NhYtWnTRqtXr5a7u3sRRg4AAAAUHySowP888MADeuCBB7Kcb7PZNHnyZE2ePLkQowIAAABuHVyDCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZQsqgDAJA/3lqzv6hDyLVn76lV1CEAAADAQjiDCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkoWdQAAAADF3Xs738uxTMkrSRr8v7/n/jZXVz3cCjYoALAgzqACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkhQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAIAAAAALIEEFbjO1KlTZbPZNGrUKPu0xMREDRs2TOXLl5eXl5e6d++umJiYogsSAAAAKIZIUIFrbNu2Te+//74aNWrkMP3ZZ5/Vl19+qc8//1zff/+9Tp48qYcffriIogQAAACKJxJU4H8uXbqkPn366IMPPlDZsmXt0+Pi4vThhx/qzTff1N13361mzZpp3rx5+vHHH7Vly5Ys60tKSlJ8fLzDCwAAAEDWSFCB/xk2bJjuv/9+hYeHO0zfvn27UlJSHKbXqVNHlStXVlRUVJb1RUZGysfHx/4KCQkpsNgBAACA4oAEFZC0ZMkS7dixQ5GRkRnmRUdHy9XVVb6+vg7TAwICFB0dnWWd48aNU1xcnP11/Pjx/A4bAAAAKFZKFnUAQFE7fvy4Ro4cqTVr1sjd3T3f6nVzc5Obm1u+1QcAAAAUd5xBxS1v+/btOn36tJo2baqSJUuqZMmS+v777/XOO++oZMmSCggIUHJysmJjYx3eFxMTo8DAwKIJGgAAACiGOIOKW16HDh20a9cuh2kDBgxQnTp1NHbsWIWEhKhUqVJat26dunfvLknat2+fjh07prCwsKIIGQAAACiWSFBxyytTpowaNGjgMM3T01Ply5e3Tx84cKCee+45lStXTt7e3hoxYoTCwsLUqlWroggZAAAAKJZIUAEnvPXWWypRooS6d++upKQkdezYUe+9915RhwUAAAAUKySoQCY2bNjg8L+7u7tmzZqlWbNmFU1AAAAAwC2AmyQBAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZQs6gAAAACctj6ycJcX+9uN1xF6543XAQC3CM6gAgAAAAAsgQQVAAAAAGAJDPEFAAC4xby3871s5x8vcdCpeh5Mq5Ef4QCAHWdQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAIAAAAALIGbJAEAABSkIz84Vy7x6v/9/eePkjtf0wDcejiDCgAAAACwBBJUAAAAAIAlMHYEAAAUO+/F/lbUIQAA8oAzqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJbATZIAALhVrY8s6ggAAHDAGVQAAAAAgCWQoAIAAAAALIEhvgAA5AeGywIAcMM4gwoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWAIJKgAAAADAEkhQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAIAAAAALIEEFQAAAABgCSSoAAAAAABLIEEFAAAAAFgCCSogKTIyUi1atFCZMmXk7++vbt26ad++fQ5lEhMTNWzYMJUvX15eXl7q3r27YmJiiihiAAAAoPgpWdQBAFbw/fffa9iwYWrRooWuXr2q8ePH695779XevXvl6ekpSXr22We1atUqff755/Lx8dHw4cP18MMPa/PmzUUcPQAARWNliYP5Us+DaTXypR4ANz8SVEDS6tWrHf6fP3++/P39tX37dt11112Ki4vThx9+qMWLF+vuu++WJM2bN09169bVli1b1KpVqwx1JiUlKSkpyf5/fHx8wTYCAAAAuMkxxBfIRFxcnCSpXLlykqTt27crJSVF4eHh9jJ16tRR5cqVFRUVlWkdkZGR8vHxsb9CQkIKPnAAAADgJkaCClwnLS1No0aN0h133KEGDRpIkqKjo+Xq6ipfX1+HsgEBAYqOjs60nnHjxikuLs7+On78eEGHDgAAANzUGOILXGfYsGHavXu3Nm3adEP1uLm5yc3NLZ+iAgAAAIo/zqAC1xg+fLi++uorrV+/XpUqVbJPDwwMVHJysmJjYx3Kx8TEKDAwsJCjBAAAAIonElRAkjFGw4cP1/Lly/Xdd98pNDTUYX6zZs1UqlQprVu3zj5t3759OnbsmMLCwgo7XAAAAKBYYogvoL+H9S5evFhffPGFypQpY7+u1MfHRx4eHvLx8dHAgQP13HPPqVy5cvL29taIESMUFhaW6R18AQAAAOQeCSogafbs2ZKkdu3aOUyfN2+e+vfvL0l66623VKJECXXv3l1JSUnq2LGj3nvvvUKOFMiD9ZFFHUHutR9X1BHAoqIOn3Oq3PESVwo4kvxXKumq/e+/4q4oJZGvaQBuPez5AP09xDcn7u7umjVrlmbNmlUIEQEAAAC3HhJUAID13IxnfQEAwA3jJkkAAAAAAEsgQQUAAAAAWAJDfAEAyAfO3rzHSsKqlS/qEAAAcMAZVAAAAACAJZCgAgAAAAAsgSG+AIrMW2v2F3UIufbsPbWKOgQAAIBiizOoAAAAAABL4AwqAOQGz+dEMWLFGzutLHEw+wL8tA4AxRq7eQAAAACAJZCgAgAAAAAsgSG+AAAAKFI5Du12Qlg+xAGg6HEGFQAAAABgCSSoAAAAAABLIEEFAAAAAFgCCSoAAAAAwBK4SRIA5IIVnxtZHIVVK1/UIQAAgCLAGVQAAAAAgCWQoAIAAAAALIEhvgAAy2EoNQAAtybOoAIAAAAALIEzqAAA4IatLHGwqEMAABQDnEEFAAAAAFgCCSoAAAAAwBJIUAEAAAAAlkCCCgAAAACwBBJUAAAAAIAlcBdfoJhodWxuUYeQa1sqDy7qEAAAAGAhnEEFAAAAAFgCZ1ABFJmb8awvAAAACg5nUAEAAAAAlkCCCgAAAACwBBJUAAAAAIAlkKACAAAAACyBBBUAAAAAYAkkqAAAAAAASyBBBQAAAABYAs9BBQDgFrayxMGiDgEAADvOoAIAAAAALIEEFQAAAABgCSSoAAAAAABLIEEFAAAAAFgCCSqQC7NmzVLVqlXl7u6uli1b6qeffirqkAAAAIBigwQVcNJnn32m5557ThMnTtSOHTvUuHFjdezYUadPny7q0AAAAIBigcfMAE568803NWjQIA0YMECSNGfOHK1atUofffSRXnjhhQzlk5KSlJSUZP8/Li5OkhQfH18g8V2+kpRzIQC4TlKJlKIOAf+TlnRV6UeIpCspSkkzRRrPzaagjq/p9RrD+gAKg83Q24AcJScnq3Tp0lq6dKm6detmnx4REaHY2Fh98cUXGd4zadIkvfLKK4UYJQAAKCjHjx9XpUqVijoMoNjjDCrghLNnzyo1NVUBAQEO0wMCAvTHH39k+p5x48bpueees/+flpam8+fPq3z58rLZbPkaX3x8vEJCQnT8+HF5e3vna91WR9tvzbZLt3b7b+W2S7d2+2l74bfdGKOLFy8qODi40JYJ3MpIUIEC4ubmJjc3N4dpvr6+BbpMb2/vW+4LSzrafmu2Xbq1238rt126tdtP2wu37T4+PoW6POBWxk2SACdUqFBBLi4uiomJcZgeExOjwMDAIooKAAAAKF5IUAEnuLq6qlmzZlq3bp19WlpamtatW6ewsLAijAwAAAAoPhjiCzjpueeeU0REhJo3b67bb79dM2bM0OXLl+139S1Kbm5umjhxYoYhxbcC2n5rtl26tdt/K7ddurXbT9tvzbYDtxLu4gvkwrvvvqvXX39d0dHRatKkid555x21bNmyqMMCAAAAigUSVAAAAACAJXANKgAAAADAEkhQAQAAAACWQIIKAAAAALAEElQAAAAAgCWQoAI3uVmzZqlq1apyd3dXy5Yt9dNPPxV1SDds0qRJstlsDq86derY5ycmJmrYsGEqX768vLy81L17d8XExDjUcezYMd1///0qXbq0/P39NWbMGF29erWwm5KjjRs3qkuXLgoODpbNZtOKFSsc5htj9PLLLysoKEgeHh4KDw/XgQMHHMqcP39effr0kbe3t3x9fTVw4EBdunTJocxvv/2mO++8U+7u7goJCdH06dMLumlOyan9/fv3z7AtdOrUyaHMzdr+yMhItWjRQmXKlJG/v7+6deumffv2OZTJr219w4YNatq0qdzc3FSjRg3Nnz+/oJuXLWfa3q5duwzrfujQoQ5lbsa2S9Ls2bPVqFEjeXt7y9vbW2FhYfrmm2/s84vrepdybntxXu8AnGQA3LSWLFliXF1dzUcffWT27NljBg0aZHx9fU1MTExRh3ZDJk6caOrXr29OnTplf505c8Y+f+jQoSYkJMSsW7fO/Pzzz6ZVq1amdevW9vlXr141DRo0MOHh4eaXX34xX3/9talQoYIZN25cUTQnW19//bV58cUXzbJly4wks3z5cof5U6dONT4+PmbFihXm119/NQ8++KAJDQ01V65csZfp1KmTady4sdmyZYv54YcfTI0aNUzv3r3t8+Pi4kxAQIDp06eP2b17t/n000+Nh4eHef/99wurmVnKqf0RERGmU6dODtvC+fPnHcrcrO3v2LGjmTdvntm9e7fZuXOnue+++0zlypXNpUuX7GXyY1s/fPiwKV26tHnuuefM3r17zcyZM42Li4tZvXp1obb3Ws60vW3btmbQoEEO6z4uLs4+/2ZtuzHGrFy50qxatcrs37/f7Nu3z4wfP96UKlXK7N692xhTfNe7MTm3vTivdwDOIUEFbmK33367GTZsmP3/1NRUExwcbCIjI4swqhs3ceJE07hx40znxcbGmlKlSpnPP//cPu333383kkxUVJQx5u+kp0SJEiY6OtpeZvbs2cbb29skJSUVaOw34voELS0tzQQGBprXX3/dPi02Nta4ubmZTz/91BhjzN69e40ks23bNnuZb775xthsNnPixAljjDHvvfeeKVu2rEPbx44da2rXrl3ALcqdrBLUrl27Zvme4tT+06dPG0nm+++/N8bk37b+/PPPm/r16zss69FHHzUdO3Ys6CY57fq2G/N3ojJy5Mgs31Nc2p6ubNmy5t///vcttd7TpbfdmFtvvQPIiCG+wE0qOTlZ27dvV3h4uH1aiRIlFB4erqioqCKMLH8cOHBAwcHBqlatmvr06aNjx45JkrZv366UlBSHdtepU0eVK1e2tzsqKkoNGzZUQECAvUzHjh0VHx+vPXv2FG5DbsCRI0cUHR3t0FYfHx+1bNnSoa2+vr5q3ry5vUx4eLhKlCihrVu32svcddddcnV1tZfp2LGj9u3bpwsXLhRSa/Juw4YN8vf3V+3atfXUU0/p3Llz9nnFqf1xcXGSpHLlyknKv209KirKoY70MlbaT1zf9nSLFi1ShQoV1KBBA40bN04JCQn2ecWl7ampqVqyZIkuX76ssLCwW2q9X9/2dLfCegeQtZJFHQCAvDl79qxSU1MdDtKSFBAQoD/++KOIosofLVu21Pz581W7dm2dOnVKr7zyiu68807t3r1b0dHRcnV1la+vr8N7AgICFB0dLUmKjo7O9HNJn3ezSI81s7Zc21Z/f3+H+SVLllS5cuUcyoSGhmaoI31e2bJlCyT+/NCpUyc9/PDDCg0N1aFDhzR+/Hh17txZUVFRcnFxKTbtT0tL06hRo3THHXeoQYMGkpRv23pWZeLj43XlyhV5eHgURJOcllnbJemxxx5TlSpVFBwcrN9++01jx47Vvn37tGzZMkk3f9t37dqlsLAwJSYmysvLS8uXL1e9evW0c+fOYr/es2q7VPzXO4CckaACsJzOnTvb/27UqJFatmypKlWq6D//+Q9fLG4xvXr1sv/dsGFDNWrUSNWrV9eGDRvUoUOHIowsfw0bNky7d+/Wpk2bijqUQpdV2wcPHmz/u2HDhgoKClKHDh106NAhVa9evbDDzHe1a9fWzp07FRcXp6VLlyoiIkLff/99UYdVKLJqe7169Yr9egeQM4b4AjepChUqyMXFJcOdHWNiYhQYGFhEURUMX19f1apVSwcPHlRgYKCSk5MVGxvrUObadgcGBmb6uaTPu1mkx5rdOg4MDNTp06cd5l+9elXnz58vdp+HJFWrVk0VKlTQwYMHJRWP9g8fPlxfffWV1q9fr0qVKtmn59e2nlUZb2/vIv/BJ6u2Z6Zly5aS5LDub+a2u7q6qkaNGmrWrJkiIyPVuHFjvf3227fEes+q7ZkpbusdQM5IUIGblKurq5o1a6Z169bZp6WlpWndunUO1/IUB5cuXdKhQ4cUFBSkZs2aqVSpUg7t3rdvn44dO2Zvd1hYmHbt2uWQuKxZs0be3t72YWQ3g9DQUAUGBjq0NT4+Xlu3bnVoa2xsrLZv324v89133yktLc3+xS4sLEwbN25USkqKvcyaNWtUu3ZtSwxvzY2//vpL586dU1BQkKSbu/3GGA0fPlzLly/Xd999l2EYcn5t62FhYQ51pJcpyv1ETm3PzM6dOyXJYd3fjG3PSlpampKSkor1es9KetszU9zXO4BMFPVdmgDk3ZIlS4ybm5uZP3++2bt3rxk8eLDx9fV1uLvhzegf//iH2bBhgzly5IjZvHmzCQ8PNxUqVDCnT582xvz9CIbKlSub7777zvz8888mLCzMhIWF2d+f/hiCe++91+zcudOsXr3a+Pn5WfIxMxcvXjS//PKL+eWXX4wk8+abb5pffvnFHD161Bjz92NmfH19zRdffGF+++0307Vr10wfM3PbbbeZrVu3mk2bNpmaNWs6PGYlNjbWBAQEmMcff9zs3r3bLFmyxJQuXbrIH7NiTPbtv3jxohk9erSJiooyR44cMWvXrjVNmzY1NWvWNImJifY6btb2P/XUU8bHx8ds2LDB4ZEaCQkJ9jL5sa2nP3JjzJgx5vfffzezZs0q8kdu5NT2gwcPmsmTJ5uff/7ZHDlyxHzxxRemWrVq5q677rLXcbO23RhjXnjhBfP999+bI0eOmN9++8288MILxmazmW+//dYYU3zXuzHZt724r3cAziFBBW5yM2fONJUrVzaurq7m9ttvN1u2bCnqkG7Yo48+aoKCgoyrq6upWLGiefTRR83Bgwft869cuWKefvppU7ZsWVO6dGnz0EMPmVOnTjnU8eeff5rOnTsbDw8PU6FCBfOPf/zDpKSkFHZTcrR+/XojKcMrIiLCGPP3o2YmTJhgAgICjJubm+nQoYPZt2+fQx3nzp0zvXv3Nl5eXsbb29sMGDDAXLx40aHMr7/+atq0aWPc3NxMxYoVzdSpUwuridnKrv0JCQnm3nvvNX5+fqZUqVKmSpUqZtCgQRl+gLlZ259ZuyWZefPm2cvk17a+fv1606RJE+Pq6mqqVavmsIyikFPbjx07Zu666y5Trlw54+bmZmrUqGHGjBnj8DxMY27OthtjzBNPPGGqVKliXF1djZ+fn+nQoYM9OTWm+K53Y7Jve3Ff7wCcYzPGmMI7XwsAAAAAQOa4BhUAAAAAYAkkqAAAAAAASyBBBQAAAABYAgkqAAAAAMASSFABAAAAAJZAggoAAAAAsAQSVAAAAACAJZCgAgAAAAAsgQQVAAAAAGAJJKgAAAAAAEsgQQUAAAAAWML/B6TaC5Wu0IhZAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_hist(true_value, data, query)" ] }, { "cell_type": "code", "execution_count": 48, "id": "cb9bd5bb", "metadata": { "id": "cb9bd5bb" }, "outputs": [], "source": [] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 5 }