DorkScan - Automating Dorked Search Queries

Table of Contents

With DorkScan being my first distributed project it’s time I set up continuous integration. Seeing as how my code is hosted on GitLab Ill be using their CI.

DorkScan GitLab Pipeline

Summary of My Initial CI Pipeline

My initial pipeline is simple. It consists of 3 stages and 5 jobs.

Lint, Type Check, Vulnerability Scan and, Build

.gitlab-ci.yml

image: python:3.10-buster

stages:
- code-analysis
- build
- test

before_script:
  - pip install -U pip setuptools wheel
  - pip install -r requirements.txt

lint-black:
  stage: code-analysis
  script:
    - pip install black
    - echo linting code using black...
    - black dorkscan/
bandit-scan:
  stage: code-analysis
  needs:
    - job: lint-black
  script:
    - pip install bandit
    - bandit -ll -r dorkscan/
mypy-typing:
  stage: code-analysis
  needs:
    - job: lint-black
  script:
    - pip install mypy
    - mypy --install-types --non-interactive dorkscan/
    - echo type checking using mypy...
    - mypy dorkscan/

build_sdist-bdist:
  stage: build
  script:
    - python3 setup.py sdist bdist_wheel

runTest:
  stage: test
  artifacts:
    paths:
      - dist/
    expire_in: 30 days
  script:
    - pip install -e .

A breif aside to explain the above which will explain some of thr chsnges required and my basic understanding.

Breakdown Scripts, Stages & Jobs

image - python 3.10 on debian buster

before_script - installs build packages and rewuirements for dorkscan

  1. code-analysis
    • lints code with black
    • checks typing with mypy and a bandit scan for medium or above issues. Both require linting done first.
  2. build - runs setup.py with sdist and bdist_wheel
  3. test - Currently just attempts a local pip install of dorkscan.

dorkscan gitlab pipeline results

Changes Based on Pipeline Results

Most notable is the lack of actually running a test. I’ve built the source and wheel package and even installed them; but, I never actually run a test of DorkScan. Next revision I’ll be running the dorkscan demonstration (calling the module python3 -m dorkscan).

Next I see some simple performance improvements like eliminating pip installs from job themselves and having everything installed in the before_script section.

Stage Revision

before_script

  1. install all required pip modules. Includes build, code analysis and whatever is in requirements.txt.

code-review

  1. linting -
  2. typing -
  3. safety -

build

rules: # only run if commit is tagged release
	if: $CI_COMMIT_TAG
  1. python3 setup.py sdist bdsit_wheel
  2. python3 -m pip install -e .

test

rules: # only run if on main branch
	if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  1. python3 -m dorkscan

Posts in this series