← Back to Docs

BPD Scraper - Rocky Linux 10 Deployment Guide

Last Updated: May 16, 2026
Target OS: Rocky Linux 10
Author: Tyler Lehane

For day-to-day deploys: push to master — GitLab CI/CD handles it automatically.
This guide covers manual/from-scratch setup. See setup/README.md for the quick-reference version.

Table of Contents

  1. Prerequisites
  2. System Setup
  3. Database Installation
  4. Application Deployment
  5. Web Server Configuration
  6. Daily Automation
  7. Troubleshooting
  8. Verification Checklist

Prerequisites

Hardware Requirements

Software Requirements

Network Ports

System Setup

1. Update System

sudo dnf update -y
sudo dnf install -y git wget curl vim htop

2. Create Application User

sudo useradd -m -s /bin/bash flaskapp
sudo usermod -aG wheel flaskapp  # Add to sudoers if needed

3. Set Up SSH Key Access (Optional)

sudo -u flaskapp mkdir -p /home/flaskapp/.ssh
sudo chown flaskapp:flaskapp /home/flaskapp/.ssh
sudo chmod 700 /home/flaskapp/.ssh

# Copy public key to authorized_keys
# sudo vi /home/flaskapp/.ssh/authorized_keys

4. Create Application Directory

sudo mkdir -p /opt/flaskapp
sudo chown flaskapp:flaskapp /opt/flaskapp
sudo chmod 755 /opt/flaskapp

5. Clone/Copy Application Code

cd /opt/flaskapp
# Option A: Git clone
git clone <repo-url> .

# Option B: Copy from local
scp -r ./* [email protected]:/opt/flaskapp/

# Set permissions
sudo chown -R flaskapp:flaskapp /opt/flaskapp

Database Installation

1. Install PostgreSQL 15

sudo dnf install -y postgresql15-server postgresql15-contrib postgresql15-libs

2. Initialize Database Cluster

sudo /usr/pgsql-15/bin/postgresql-15-setup initdb
sudo systemctl enable postgresql-15
sudo systemctl start postgresql-15

3. Configure PostgreSQL Network Access

Edit /var/lib/pgsql/15/data/postgresql.conf:

sudo vi /var/lib/pgsql/15/data/postgresql.conf

Find and modify:

listen_addresses = '*'  # Listen on all interfaces
port = 5432
shared_buffers = 256MB  # Adjust for 4GB RAM systems

4. Configure pg_hba.conf (Authentication)

Edit /var/lib/pgsql/15/data/pg_hba.conf:

sudo vi /var/lib/pgsql/15/data/pg_hba.conf

Add this line to allow network connections from your IP:

host    all             all             192.168.68.0/24         md5
host    all             all             0.0.0.0/0               md5  # For development only

5. Restart PostgreSQL

sudo systemctl restart postgresql-15

6. Create Database and User

sudo -u postgres psql << 'EOF'
CREATE DATABASE bpd;
CREATE USER bpd_user WITH PASSWORD 'YOUR_SECURE_PASSWORD_HERE';
ALTER ROLE bpd_user SET client_encoding TO 'utf8';
ALTER ROLE bpd_user SET default_transaction_isolation TO 'read committed';
ALTER ROLE bpd_user SET default_transaction_deferrable TO on;
ALTER ROLE bpd_user SET default_transaction_read_committed TO on;
GRANT ALL PRIVILEGES ON DATABASE bpd TO bpd_user;
\c bpd
GRANT ALL PRIVILEGES ON SCHEMA public TO bpd_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO bpd_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO bpd_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO bpd_user;
EOF

7. Create Database Schema

sudo -u postgres psql -d bpd << 'EOF'
CREATE TABLE incidents (
    id SERIAL PRIMARY KEY,
    incident_date TEXT,
    incident_time TEXT,
    case_number TEXT,
    location TEXT,
    incident_type TEXT,
    description TEXT,
    scraped_at TEXT,
    query_start TEXT,
    query_end TEXT,
    parse_status TEXT,
    raw_extra TEXT,
    location_lat NUMERIC(11,8),
    location_lon NUMERIC(11,8),
    geocode_status VARCHAR(20) DEFAULT 'unknown',
    UNIQUE(case_number, incident_date)
);

CREATE INDEX idx_incident_date ON incidents(incident_date);
CREATE INDEX idx_case_number ON incidents(case_number);
CREATE INDEX idx_geocode_status ON incidents(geocode_status);
EOF

8. Verify PostgreSQL Connection

PGPASSWORD='YOUR_SECURE_PASSWORD_HERE' psql -U bpd_user -d bpd -h localhost -c "SELECT version();"

Application Deployment

1. Install Python 3.12

sudo dnf install -y python3.12 python3.12-devel python3.12-pip

2. Create Virtual Environment

cd /opt/flaskapp
sudo -u flaskapp python3.12 -m venv .venv
sudo -u flaskapp .venv/bin/pip install --upgrade pip setuptools wheel

3. Install Python Dependencies

cd /opt/flaskapp
sudo -u flaskapp .venv/bin/pip install -r requirements.txt

Important: Create requirements.txt if it doesn't exist:

Flask==2.3.0
psycopg2-binary==2.9.0
python-dotenv==1.0.0
gunicorn==25.0.0
playwright>=1.40.0
beautifulsoup4>=4.12.0
lxml>=5.0.0
requests>=2.28.0

4. Install Playwright Browsers

sudo -u flaskapp .venv/bin/playwright install chromium

5. Create Environment Configuration

sudo -u flaskapp tee /opt/flaskapp/.env > /dev/null << 'EOF'
PG_HOST=localhost
PG_PORT=5432
PG_DB=bpd
PG_USER=bpd_user
PG_PASSWORD=YOUR_SECURE_PASSWORD_HERE
FLASK_ENV=production
FLASK_DEBUG=0
EOF

sudo chmod 600 /opt/flaskapp/.env

6. Set Up Data Directories

sudo -u flaskapp mkdir -p /opt/flaskapp/data/{raw,parsed,output}
sudo -u flaskapp mkdir -p /opt/flaskapp/logs
sudo chmod 755 /opt/flaskapp/data/{raw,parsed,output}
sudo chmod 777 /opt/flaskapp/logs

Web Server Configuration

1. Install Nginx

sudo dnf install -y nginx
sudo systemctl enable nginx

2. Create Nginx Configuration

sudo tee /etc/nginx/conf.d/flaskapp.conf > /dev/null << 'EOF'
upstream flask_app {
    server 127.0.0.1:8000;
}

server {
    listen 80;
    server_name _;

    client_max_body_size 10M;

    location / {
        proxy_pass http://flask_app;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    location /static/ {
        alias /opt/flaskapp/bpd_dashboard/static/;
        expires 30d;
    }
}
EOF

3. Test Nginx Configuration

sudo nginx -t

4. Start Nginx

sudo systemctl restart nginx

Flask Application Service

1. Create Systemd Service File

sudo tee /etc/systemd/system/flaskapp.service > /dev/null << 'EOF'
[Unit]
Description=Flask app via gunicorn (TCP port 8000)
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=flaskapp
Group=flaskapp
WorkingDirectory=/opt/flaskapp
UMask=0002
Environment="PG_HOST=localhost"
Environment="PG_PORT=5432"
Environment="PG_DB=bpd"
Environment="PG_USER=bpd_user"
Environment="PG_PASSWORD=YOUR_SECURE_PASSWORD_HERE"
Environment="SECRET_KEY=REPLACE_WITH_64_CHAR_HEX"
Environment="TURNSTILE_SITEKEY=0x4AAAAAADQNyPv4m4YYHmbx"
Environment="TURNSTILE_SECRET=REPLACE_WITH_REAL_SECRET"
ExecStart=/opt/flaskapp/.venv/bin/gunicorn --workers 3 --bind 127.0.0.1:8000 --env SCRIPT_NAME=/dashboard dashboard.app:app
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable flaskapp

Generate SECRET_KEY if needed:

python3 -c "import secrets; print(secrets.token_hex(32))"

2. Start Flask Service

sudo systemctl start flaskapp
sudo systemctl status flaskapp

3. Verify Service

curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/dashboard/incidents
# Returns 302 (redirect to Turnstile gate) — this is correct

Daily Automation (Cron)

run_daily_win.py in bpd_scraper/ is the cron entry point — it runs scraper → parse → geocode in sequence and logs to bpd_scraper/logs/cron.log.

1. Add Crontab Entry

sudo -u flaskapp crontab -e

Add this line (runs daily at 7:30 AM):

30 7 * * * /opt/flaskapp/.venv/bin/python /opt/flaskapp/bpd_scraper/run_daily_win.py >> /opt/flaskapp/bpd_scraper/logs/cron.log 2>&1

2. Verify Cron Setup

sudo -u flaskapp crontab -l
tail -f /opt/flaskapp/bpd_scraper/logs/cron.log

SELinux Configuration (Production)

Option A: Disable SELinux (Development/Testing)

sudo setenforce 0

To make permanent, edit /etc/selinux/config:

sudo vi /etc/selinux/config
# Set: SELINUX=disabled

Option B: Configure SELinux Policy (Production)

# Allow Nginx to connect to Flask app
sudo setsebool -P httpd_can_network_connect on

# Allow PostgreSQL network access
sudo semanage port -a -t postgresql_port_t -p tcp 5432

Firewall Configuration

1. Open Firewall Ports

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

2. Verify Firewall Rules

sudo firewall-cmd --list-all

Verification Checklist

Run through this after deployment:

Troubleshooting

Flask Service Won't Start

# Check service logs
sudo journalctl -u flaskapp -n 50
sudo systemctl status flaskapp -l

# Check Flask logs
tail -50 /opt/flaskapp/logs/error.log

# Common issues:
# - PostgreSQL not running: sudo systemctl start postgresql-15
# - .env file permissions: sudo chmod 600 /opt/flaskapp/.env
# - Port already in use: sudo lsof -i :8000

Database Connection Fails

# Test connection directly
PGPASSWORD='PASSWORD' psql -U bpd_user -d bpd -h localhost -c "SELECT 1;"

# Check PostgreSQL is listening
sudo netstat -tlnp | grep 5432

# Check pg_hba.conf for connection rules
sudo cat /var/lib/pgsql/15/data/pg_hba.conf

Nginx 502 Bad Gateway

# Verify Flask is running
curl http://127.0.0.1:8000/

# Check Nginx logs
sudo tail -50 /var/log/nginx/error.log

# Check Flask is binding to correct port/address
sudo lsof -i :8000

Cron Job Not Running

# Check cron logs
sudo journalctl -u crond -n 50

# Verify crontab is set
sudo -u flaskapp crontab -l

# Test script manually
sudo -u flaskapp /opt/flaskapp/.venv/bin/python /opt/flaskapp/bpd_scraper/run_daily_win.py

Playwright Browser Not Found / Scraper Records 0 Incidents

Symptom in bpd_scraper/logs/scraper_YYYY-MM-DD.log:

Playwright failed: BrowserType.launch: Executable doesn't exist at .../chromium_headless

The scraper silently falls back and writes a 0-record JSONL file — no parse errors, but no data.

Fix:

sudo -u flaskapp /opt/flaskapp/.venv/bin/playwright install chromium

Verify the binary works afterward:

/opt/flaskapp/.cache/ms-playwright/chromium_headless_shell-*/chrome-headless-shell-linux64/chrome-headless-shell --version

The CI/CD pipeline (playwright install chromium step) handles this automatically on future deploys.

Performance Tuning

For 5,000+ Records

Increase PostgreSQL Buffer

sudo vi /var/lib/pgsql/15/data/postgresql.conf
# Increase: shared_buffers = 512MB (for 4GB RAM)
# Then restart: sudo systemctl restart postgresql-15

Reduce Gunicorn Workers if Memory Constrained

# Edit /etc/systemd/system/flaskapp.service
# Change: --workers 1 (instead of 2)
# Then: sudo systemctl daemon-reload && sudo systemctl restart flaskapp

Add Database Indexes

sudo -u postgres psql -d bpd << 'EOF'
CREATE INDEX idx_location ON incidents(location);
CREATE INDEX idx_location_lat ON incidents(location_lat);
CREATE INDEX idx_incident_type ON incidents(incident_type);
VACUUM ANALYZE;
EOF

Backup & Recovery

Backup Database

sudo -u postgres pg_dump -d bpd > /backup/bpd_$(date +%Y-%m-%d).sql

Restore Database

sudo -u postgres psql -d bpd < /backup/bpd_YYYY-MM-DD.sql

Backup Application Code

sudo tar -czf /backup/flaskapp_$(date +%Y-%m-%d).tar.gz /opt/flaskapp/

Next Steps

  1. Enable HTTPS — Use Let's Encrypt with certbot
  2. Set up Monitoring — Monitor Flask logs and PostgreSQL performance
  3. Add Alerting — Email/Slack notifications for cron job failures
  4. Scale Database — Optimize queries and add proper indexing
  5. Load Balancing — Run multiple Flask workers if needed

Support

For issues, check: - Flask logs: /opt/flaskapp/logs/error.log - Cron logs: /opt/flaskapp/logs/cron.log - Nginx logs: /var/log/nginx/error.log - PostgreSQL logs: sudo journalctl -u postgresql-15 -n 50

For detailed service status:

# All services
sudo systemctl status postgresql-15 flaskapp nginx

# Full logs
sudo journalctl -u flaskapp --all -n 100