BPD Scraper — Setup & Deployment

Current Production State

Deploying Code Changes

Push to master — the GitLab CI/CD pipeline handles everything:

git push origin master

The pipeline (.gitlab-ci.yml) runs on the bpd-production shell runner: 1. rsync the repo to /opt/flaskapp/ (skips .env, .venv, logs, data) 2. pip install from both requirements files 3. playwright install chromium — keeps browser binaries current after playwright version bumps 4. systemctl restart flaskapp 5. Health check: curl http://localhost:8000/dashboard/incidents — expects 200 or 302 (302 = Turnstile gate redirect, which is correct) 6. rsync landing/ to /var/www/landing/ (updates landing page and openapi.yaml)

Monitor at: GitLab → Build → Pipelines

Setting Up the GitLab Runner (one-time)

1. GitLab: Project → Settings → CI/CD → Runners → New project runner
   Tag: bpd-production | Uncheck "Run untagged jobs" | Click Create
   Copy the glrt-... token.

2. Server: bash scp -i ~/.ssh/id_rsa setup/install_gitlab_runner.sh [email protected]:/tmp/ ssh -i ~/.ssh/id_rsa [email protected] \ "RUNNER_TOKEN=glrt-xxxx sudo -E bash /tmp/install_gitlab_runner.sh"

3. Confirm the runner shows a green dot in GitLab, then push any commit to trigger the first deploy.

Note: The runner executes as the gitlab-runner system user. It needs passwordless sudo — this is granted by /etc/sudoers.d/gitlab-runner on the server (gitlab-runner ALL=(ALL) NOPASSWD: ALL).

Fresh Server Provisioning (from scratch)

If setting up a new Rocky Linux 10 server:

Step 1 — Bootstrap infrastructure

scp -i ~/.ssh/id_rsa setup/bootstrap_rocky_v2.sh localadmin@<server-ip>:/tmp/
ssh -i ~/.ssh/id_rsa localadmin@<server-ip> "sudo bash /tmp/bootstrap_rocky_v2.sh"

This installs Python, PostgreSQL, nginx, gunicorn, creates the flaskapp user and systemd service, and configures firewall rules. It is idempotent.

Step 2 — Set Turnstile and secret-key env vars

The bootstrap script leaves TURNSTILE_SITEKEY and TURNSTILE_SECRET as REPLACE_ME placeholders in the systemd service. Set the real values after provisioning:

ssh -i ~/.ssh/id_rsa localadmin@<server-ip>
sudo sed -i \
  's/TURNSTILE_SITEKEY=REPLACE_ME/TURNSTILE_SITEKEY=0x4AAAAAADQNyPv4m4YYHmbx/' \
  /etc/systemd/system/flaskapp.service
sudo sed -i \
  's/TURNSTILE_SECRET=REPLACE_ME/TURNSTILE_SECRET=<your-secret>/' \
  /etc/systemd/system/flaskapp.service
# Generate a real SECRET_KEY if not already set:
SK=$(python3 -c "import secrets; print(secrets.token_hex(32))")
sudo sed -i "s/SECRET_KEY=REPLACE_ME/SECRET_KEY=$SK/" \
  /etc/systemd/system/flaskapp.service
sudo systemctl daemon-reload && sudo systemctl restart flaskapp

Step 3 — Set up GitLab CI/CD runner

Follow the "Setting Up the GitLab Runner" steps above. Once the runner is registered, push to master to deploy the code.

Step 4 — Restore database (if migrating)

# Copy backup to server
scp -i ~/.ssh/id_rsa bpd_database_backup_YYYY-MM-DD.sql localadmin@<server-ip>:/tmp/

# Restore
ssh -i ~/.ssh/id_rsa localadmin@<server-ip>
sudo -u postgres psql -d bpd < /tmp/bpd_database_backup_YYYY-MM-DD.sql

Step 5 — Set up cron job (daily scraping)

ssh -i ~/.ssh/id_rsa localadmin@<server-ip>
sudo -u flaskapp crontab -e

Add:

30 7 * * * /opt/flaskapp/.venv/bin/python /opt/flaskapp/bpd_scraper/run_daily_win.py >> /opt/flaskapp/bpd_scraper/logs/cron.log 2>&1

Scripts Reference

Archived (do not use)

• bootstrap.sh, bootstrap_rocky.sh — superseded by v2
• deploy_complete.sh, deploy_rocky.sh, deploy_ubuntu.sh — superseded
• deploy.ps1 — Windows Task Scheduler (local scraper runs only)

Verification

# All services healthy?
ssh -i ~/.ssh/id_rsa [email protected] \
  "sudo systemctl is-active flaskapp nginx postgresql"

# App responds?
curl -s -o /dev/null -w "%{http_code}" http://192.168.68.100/dashboard/incidents

# Row count and freshness?
ssh -i ~/.ssh/id_rsa [email protected] \
  "PGPASSWORD=\$(sudo cat /root/.flaskapp_db_password) \
   psql -U bpd_user -d bpd -h localhost -tAc \
   'SELECT COUNT(*), MAX(incident_date) FROM incidents;'"

Architecture

  Internet
     │
     ▼
Cloudflare Tunnel (apps.tlehane.com)
     │
     ▼
Rocky Linux 10 — 192.168.68.100
  ┌──────────────────────────────────┐
  │  Nginx :80                       │
  │    └─► Gunicorn :8000            │
  │          └─► Flask (SCRIPT_NAME  │
  │                = /dashboard)     │
  │                  │               │
  │                  ▼               │
  │           PostgreSQL :5432       │
  │                  ▲               │
  │                  │               │
  │           Cron 7:30 AM           │
  │       (scrape → parse → geocode) │
  └──────────────────────────────────┘
     │
     ▼
GitLab (push to master)
  └─► bpd-production runner
        └─► rsync + pip + restart

Troubleshooting

Service not starting:

sudo journalctl -xeu flaskapp.service -n 50

Nginx 502:

curl http://127.0.0.1:8000/dashboard/incidents   # test gunicorn directly
sudo nginx -t                                     # test nginx config

Cron not running:

sudo -u flaskapp crontab -l
tail -50 /opt/flaskapp/bpd_scraper/logs/cron.log
# Manually trigger:
sudo -u flaskapp /opt/flaskapp/.venv/bin/python /opt/flaskapp/bpd_scraper/run_daily_win.py

GitLab pipeline fails: - Check /home/gitlab-runner/builds/.../ for the build workspace - Runner logs: sudo journalctl -xeu gitlab-runner -n 50 - Sudo permissions: sudo -l -U gitlab-runner

PostgreSQL auth failure:

sudo cat /root/.flaskapp_db_password         # get password
sudo cat /var/lib/pgsql/data/pg_hba.conf     # check auth rules
sudo systemctl restart postgresql