Contents

• 1. Overview
  • 1.1. How to Apply PilotScope?
    • 1.1.1. Supported Tasks and Algorithms
      • 1.1.1.1. Knob Tuning
      • 1.1.1.2. Index Recommendation
      • 1.1.1.3. Cardinality Estimation
      • 1.1.1.4. End-to-end Learned Query Optimizer
      • 1.1.1.5. More AI4DB Methods …
      • 1.1.1.6. Usage Details
  • 1.2. How to Develop in PilotScope?
    • 1.2.1. Developing Details
  • 1.3. What to Do in PilotScope?
  • 1.4. License
  • 1.5. References
• 2. Installation
  • 2.1. Install from Pre-built Docker Image
    • 2.1.1. Start a Container from the Image
      • 2.1.1.1. Step 1: Build an Image
      • 2.1.1.2. Step 2: Start a Container from the Image
      • 2.1.1.3. Step 3: Log into the Container
    • 2.1.2. Start PostgreSQL in the Container
    • 2.1.3. Start Spark in the Container
    • 2.1.4. Get Started in the Container
  • 2.2. Install All Components from Scratch
    • 2.2.1. Prepare the Environment and Configure the PilotScope User
      • 2.2.1.1. Prepare the Environment
      • 2.2.1.2. Configure the PilotScope User
    • 2.2.2. Install PilotScope Package
    • 2.2.3. Install PostgreSQL
      • 2.2.3.1. Step 1: Prepare the Environment
      • 2.2.3.2. Step 2: Install PostgreSQL
      • 2.2.3.3. Step 3: Install the Extension
      • 2.2.3.4. Step 4: Configure Database Connection Settings
      • 2.2.3.5. Verify the connection to PostgreSQL via psql
      • 2.2.3.6. Update the Latest PilotScope Patch for PostgreSQL
      • 2.2.3.7. A Guideline for Installing PilotScope Patch into Your PostgreSQL Database.
    • 2.2.4. Install Spark
      • 2.2.4.1. Step 1: Download PilotScope Patch for Spark
      • 2.2.4.2. Step 2: Download Spark and Apply Patch for Spark
      • 2.2.4.3. Step 3: Configuring Java Environment Variables
      • 2.2.4.4. Step 4: Compile and Build PilotScope and Spark
      • 2.2.4.5. Step 5: Install PySpark
      • 2.2.4.6. Update the Latest PilotScope Patch for Spark
    • 2.2.5. Get Started
  • 2.3. Load Test Databases
• 3. System Pipeline
• 4. Core Components (ML Side)
  • 4.1. Components of PilotScope for Data Exchange and Control over Database
  • 4.2. PilotConfig
  • 4.3. PilotDataInteractor
  • 4.4. DBController
  • 4.5. DataManager
  • 4.6. Components of PilotScope for Deployment of Custom Algorithms
  • 4.7. PilotModel
  • 4.8. PushHandler (Driver): Applying custom algorithms to incoming SQL queries automatically
  • 4.9. PilotScheduler
  • 4.10. Event
    • 4.10.1. PretrainingModelEvent
    • 4.10.2. PeriodicModelUpdateEvent
    • 4.10.3. Registering Events into PilotScheduler
  • 4.11. Dataset
• 5. Example
  • 5.1. Knob Tuning Task Example
  • 5.2. Index Recommendation Task Example
  • 5.3. Cardinality Estimation Task Example
  • 5.4. Query Optimizer Task Example
• 6. API (ML Side)
  • 6.1. pilotscope.Anchor
    • 6.1.1. pilotscope.Anchor.BaseAnchor
      • 6.1.1.1. pilotscope.Anchor.BaseAnchor.BasePushHandler
  • 6.2. pilotscope.PilotConfig
  • 6.3. pilotscope.DBInteractor
    • 6.3.1. pilotscope.DBInteractor.PilotDataInteractor
  • 6.4. pilotscope.PilotModel
  • 6.5. pilotscope.PilotScheduler
  • 6.6. pilotscope.PilotEvent
  • 6.7. pilotscope.DataManager
    • 6.7.1. pilotscope.DataManager.DataManager
  • 6.8. pilotscope.PilotTransData
  • 6.9. pilotscope.Factory
    • 6.9.1. pilotscope.Factory.DBControllerFectory
    • 6.9.2. pilotscope.Factory.SchedulerFactory
  • 6.10. pilotscope.DBController
    • 6.10.1. pilotscope.DBController.BaseDBController
    • 6.10.2. pilotscope.DBController.PostgreSQLController
    • 6.10.3. pilotscope.DBController.SparkSQLController
  • 6.11. pilotscope.PilotEnum
• 7. Database Side Develop
  • 7.1. Overview
    • 7.1.1. Extended SQL
      • 7.1.1.1. Anchor Information
      • 7.1.1.2. Transmission information
    • 7.1.2. Parse module
    • 7.1.3. The pipeline of query optimizer with anchors
    • 7.1.4. Data transmission module
    • 7.1.5. PilotTransData
  • 7.2. Implement in PostgreSQL
    • 7.2.1. Parse Module
    • 7.2.2. The pipeline of query optimizer with anchors
    • 7.2.3. Data Transmission Module
• 8. Annotations