gyulaid
/
alice
1
0
Fork 0
Code Issues 7 Pull Requests Projects Releases 5 Wiki Activity
Browse Source

7_runner-interface (#12)
continuous-integration/drone/push Build is passing
Details

Fix #7

Co-authored-by: gyulaid <gyulaid@gyulai.cloud>
Reviewed-on: #12
Co-authored-by: Daniel Gyulai <gyulaid@gyulai.cloud>
Co-committed-by: Daniel Gyulai <gyulaid@gyulai.cloud>
pull/16/head
Daniel Gyulai 3 years ago
parent
10af87bc19
commit
9e4b61225c
23 changed files with 774 additions and 695 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 11
      .devcontainer/Dockerfile
  2. 30
      .devcontainer/devcontainer.json
  3. 62
      .drone.yml
  4. 280
      .gitignore
  5. 18
      LICENSE
  6. 22
      README.md
  7. 30
      alice-ci/README.md
  8. 10
      alice-ci/pyproject.toml
  9. 54
      alice-ci/setup.cfg
  10. 18
      alice-ci/src/alice/__init__.py
  11. 6
      alice-ci/src/alice/__main__.py
  12. 125
      alice-ci/src/alice/cli.py
  13. 20
      alice-ci/src/alice/exceptions.py
  14. 94
      alice-ci/src/alice/runnerfactory.py
  15. 5
      alice-ci/src/alice/runners/__init__.py
  16. 2
      alice-ci/src/alice/runners/dockerrunner.py
  17. 233
      alice-ci/src/alice/runners/pythonrunner.py
  18. 174
      alice-ci/src/alice/utils.py
  19. 64
      ci-examples/full.yaml
  20. 34
      ci-examples/python1.yaml
  21. 46
      docs/examples.md
  22. 53
      docs/runners.md
  23. 78
      docs/syntax.md

11
.devcontainer/Dockerfile
View File

@ -0,0 +1,11 @@
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.231.3/containers/ubuntu/.devcontainer/base.Dockerfile
# [Choice] Ubuntu version (use hirsuite or bionic on local arm64/Apple Silicon): hirsute, focal, bionic
ARG VARIANT="hirsute"
FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}
# [Optional] Uncomment this section to install additional OS packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
# && apt-get -y install --no-install-recommends <your-package-list-here>

30
.devcontainer/devcontainer.json
View File

@ -0,0 +1,30 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.231.3/containers/ubuntu
{
"name": "Ubuntu",
"build": {
"dockerfile": "Dockerfile",
// Update 'VARIANT' to pick an Ubuntu version: hirsute, focal, bionic
// Use hirsute or bionic on local arm64/Apple Silicon.
"args": { "VARIANT": "focal" }
},
// Set *default* container specific settings.json values on container create.
"settings": {},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [],
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "uname -a",
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "vscode",
"features": {
"python": "latest"
}
}

62
.drone.yml
View File

@ -1,32 +1,32 @@
kind: pipeline kind: pipeline
type: docker type: docker
name: default name: default
steps: steps:
- name: static-test - name: static-test
image: alpine/flake8 image: alpine/flake8
commands: commands:
- python3 -m flake8 --ignore E501,W503 - python3 -m flake8 --ignore E501,W503
- name: build - name: build
image: python image: python
commands: commands:
- python3 -m pip install build - python3 -m pip install build
- python3 -m build alice-ci - python3 -m build alice-ci
- name: publish - name: publish
image: python image: python
environment: environment:
TWINE_PASSWORD: TWINE_PASSWORD:
from_secret: pypi_username from_secret: pypi_username
TWINE_USERNAME: TWINE_USERNAME:
from_secret: pypi_password from_secret: pypi_password
commands: commands:
- python3 -m pip install twine - python3 -m pip install twine
- python3 -m twine upload --verbose alice-ci/dist/* - python3 -m twine upload --verbose alice-ci/dist/*
when: when:
branch: branch:
- master - master
event: event:
exclude: exclude:
- pull_request - pull_request

280
.gitignore
View File

@ -1,140 +1,140 @@
# ---> Python # ---> Python
# Byte-compiled / optimized / DLL files # Byte-compiled / optimized / DLL files
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
*$py.class *$py.class
# C extensions # C extensions
*.so *.so
# Distribution / packaging # Distribution / packaging
.Python .Python
build/ build/
develop-eggs/ develop-eggs/
dist/ dist/
downloads/ downloads/
eggs/ eggs/
.eggs/ .eggs/
lib/ lib/
lib64/ lib64/
parts/ parts/
sdist/ sdist/
var/ var/
wheels/ wheels/
share/python-wheels/ share/python-wheels/
*.egg-info/ *.egg-info/
.installed.cfg .installed.cfg
*.egg *.egg
MANIFEST MANIFEST
# PyInstaller # PyInstaller
# Usually these files are written by a python script from a template # Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it. # before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest *.manifest
*.spec *.spec
# Installer logs # Installer logs
pip-log.txt pip-log.txt
pip-delete-this-directory.txt pip-delete-this-directory.txt
# Unit test / coverage reports # Unit test / coverage reports
htmlcov/ htmlcov/
.tox/ .tox/
.nox/ .nox/
.coverage .coverage
.coverage.* .coverage.*
.cache .cache
nosetests.xml nosetests.xml
coverage.xml coverage.xml
*.cover *.cover
*.py,cover *.py,cover
.hypothesis/ .hypothesis/
.pytest_cache/ .pytest_cache/
cover/ cover/
# Translations # Translations
*.mo *.mo
*.pot *.pot
# Django stuff: # Django stuff:
*.log *.log
local_settings.py local_settings.py
db.sqlite3 db.sqlite3
db.sqlite3-journal db.sqlite3-journal
# Flask stuff: # Flask stuff:
instance/ instance/
.webassets-cache .webassets-cache
# Scrapy stuff: # Scrapy stuff:
.scrapy .scrapy
# Sphinx documentation # Sphinx documentation
docs/_build/ docs/_build/
# PyBuilder # PyBuilder
.pybuilder/ .pybuilder/
target/ target/
# Jupyter Notebook # Jupyter Notebook
.ipynb_checkpoints .ipynb_checkpoints
# IPython # IPython
profile_default/ profile_default/
ipython_config.py ipython_config.py
# pyenv # pyenv
# For a library or package, you might want to ignore these files since the code is # For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in: # intended to run in multiple environments; otherwise, check them in:
# .python-version # .python-version
# pipenv # pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies # However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not # having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies. # install all needed dependencies.
#Pipfile.lock #Pipfile.lock
# PEP 582; used by e.g. github.com/David-OConnor/pyflow # PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/ __pypackages__/
# Celery stuff # Celery stuff
celerybeat-schedule celerybeat-schedule
celerybeat.pid celerybeat.pid
# SageMath parsed files # SageMath parsed files
*.sage.py *.sage.py
# Environments # Environments
.env .env
.venv .venv
env/ env/
venv/ venv/
ENV/ ENV/
env.bak/ env.bak/
venv.bak/ venv.bak/
# Spyder project settings # Spyder project settings
.spyderproject .spyderproject
.spyproject .spyproject
# Rope project settings # Rope project settings
.ropeproject .ropeproject
# mkdocs documentation # mkdocs documentation
/site /site
# mypy # mypy
.mypy_cache/ .mypy_cache/
.dmypy.json .dmypy.json
dmypy.json dmypy.json
# Pyre type checker # Pyre type checker
.pyre/ .pyre/
# pytype static type analyzer # pytype static type analyzer
.pytype/ .pytype/
# Cython debug symbols # Cython debug symbols
cython_debug/ cython_debug/

18
LICENSE
View File

@ -1,9 +1,9 @@
MIT License MIT License
Copyright (c) 2022 Daniel Gyulai Copyright (c) 2022 Daniel Gyulai
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

22
README.md
View File

@ -1,12 +1,12 @@
# alice # alice
CI framework with support for local running. CI framework with support for local running.
Main repo [here](https://git.gyulai.cloud/gyulaid/alice). Main repo [here](https://git.gyulai.cloud/gyulaid/alice).
[![Build Status](https://ci.gyulai.cloud/api/badges/gyulaid/alice/status.svg)](https://ci.gyulai.cloud/gyulaid/alice) [![Build Status](https://ci.gyulai.cloud/api/badges/gyulaid/alice/status.svg)](https://ci.gyulai.cloud/gyulaid/alice)
[![PyPI version](https://badge.fury.io/py/alice-ci.svg)](https://badge.fury.io/py/alice-ci) [![PyPI version](https://badge.fury.io/py/alice-ci.svg)](https://badge.fury.io/py/alice-ci)
* [Basic usage](alice-ci/README.md) * [Basic usage](alice-ci/README.md)
* [Runners](docs/runners.md) * [Runners](docs/runners.md)
* [CI syntax](docs/syntax.md) * [CI syntax](docs/syntax.md)

30
alice-ci/README.md
View File

@ -1,16 +1,16 @@
# Alice-CI # Alice-CI
Continous Integration framework with the goal of using the exact same code in CI and local env. Steps can be defined in yaml files, for syntax see the docs. Runs on LInux and Windows, Mac should work too, but not yet tested. Continous Integration framework with the goal of using the exact same code in CI and local env. Steps can be defined in yaml files, for syntax see the docs. Runs on LInux and Windows, Mac should work too, but not yet tested.
## Usage ## Usage
Install with pip: Install with pip:
``` ```
pythom3 -m pip install alice-ci pythom3 -m pip install alice-ci
``` ```
To run: To run:
``` ```
pythom3 -m alice [-i <ci.yaml>] STEPS pythom3 -m alice [-i <ci.yaml>] STEPS
``` ```

10
alice-ci/pyproject.toml
View File

@ -1,6 +1,6 @@
[build-system] [build-system]
requires = [ requires = [
"setuptools>=42", "setuptools>=42",
"wheel" "wheel"
] ]
build-backend = "setuptools.build_meta" build-backend = "setuptools.build_meta"

54
alice-ci/setup.cfg
View File

@ -1,26 +1,30 @@
[metadata] [metadata]
name = alice-ci name = alice-ci
version = 0.0.6 version = 0.0.7
author = Daniel Gyulai author = Daniel Gyulai
description = Alice CI framework description = Alice CI framework
long_description = file: README.md long_description = file: README.md
long_description_content_type = text/markdown long_description_content_type = text/markdown
url = https://git.gyulai.cloud/gyulaid/alice url = https://git.gyulai.cloud/gyulaid/alice
project_urls = project_urls =
Bug Tracker = https://git.gyulai.cloud/gyulaid/alice/issues Bug Tracker = https://git.gyulai.cloud/gyulaid/alice/issues
classifiers = classifiers =
Programming Language :: Python :: 3 Programming Language :: Python :: 3
License :: OSI Approved :: MIT License License :: OSI Approved :: MIT License
Operating System :: OS Independent Operating System :: OS Independent
[options] [options]
package_dir = package_dir =
= src = src
packages = find: packages = alice
python_requires = >=3.6 python_requires = >=3.6
install_requires = install_requires =
PyYAML==6.0 PyYAML==6.0
virtualenv==20.14.0 virtualenv==20.14.0
[options.packages.find] [options.entry_points]
console_scripts =
alice = alice.cli:main
[options.packages.find]
where = src where = src

18
alice-ci/src/alice/__init__.py
View File

@ -1,10 +1,10 @@
# flake8: noqa F401 # flake8: noqa F401
from alice.utils import ConfigParser from alice.utils import ConfigParser
from alice.exceptions import NonZeroRetcode from alice.exceptions import NonZeroRetcode
from alice.runnerfactory import Factory from alice.runnerfactory import Factory
from alice.runners.pythonrunner import PythonRunner from alice.runners.pythonrunner import PythonRunner
from alice.exceptions import NonZeroRetcode from alice.exceptions import NonZeroRetcode
from alice.exceptions import RunnerError from alice.exceptions import RunnerError
from alice.exceptions import ConfigException from alice.exceptions import ConfigException
name = "alice" name = "alice"

6
alice-ci/src/alice/__main__.py
View File

@ -1,3 +1,3 @@
from alice.cli import main from alice.cli import main
main() main()

125
alice-ci/src/alice/cli.py
View File

@ -1,63 +1,62 @@
import os import os
import argparse import argparse
from alice.utils import ConfigParser from alice.utils import ConfigParser
from alice.runnerfactory import Factory from alice.runnerfactory import Factory
from alice.exceptions import ConfigException, NonZeroRetcode, RunnerError from alice.exceptions import ConfigException, NonZeroRetcode, RunnerError
def gen_env(self, param_list): def gen_env(param_list):
env_vars = {} env_vars = {}
for item in param_list: for item in param_list:
item_parts = item.split("=") item_parts = item.split("=")
if len(item_parts) == 2: if len(item_parts) == 2:
env_vars[item_parts[0]] = item_parts[1] env_vars[item_parts[0]] = item_parts[1]
else: else:
raise ConfigException(f"Invalid parameter: {item}") raise ConfigException(f"Invalid parameter: {item}")
return env_vars return env_vars
def parse_jobs(args): def parse_jobs(args):
try: try:
factory = Factory(args.verbose) factory = Factory(args.verbose)
if len(args.env) > 0: if len(args.env) > 0:
envs = gen_env(args.env) envs = gen_env(args.env)
if args.verbose: if args.verbose:
print(f"[Alice] Env vars from CLI: {envs}") print(f"[Alice] Env vars from CLI: {envs}")
factory.update_runners({"env": envs}) jobParser = ConfigParser(args.input, factory, gen_env(args.env), args.verbose)
jobParser = ConfigParser(args.input, factory, args.verbose)
print("[Alice] Begin pipeline steps")
print("Begin pipeline steps...") for step in args.steps:
for step in args.steps: if step in jobParser.jobs:
if step in jobParser.jobs: status = jobParser.execute_job(step)
status = jobParser.execute_job(step) print(f"[Alice][Step] {step}: {status}")
print(f"[Step] {step}: {status}") else:
else: raise ConfigException(f"Step {step} not found in {args.input}")
print(f"Step {step} not found in {args.input}") exit(1)
exit(1) except ConfigException as e:
except ConfigException as e: print(f"Configuration error-> {e}")
print(f"Configuration error-> {e}") exit(1)
exit(1) except NonZeroRetcode:
except NonZeroRetcode: print("[Alice] FAILED")
print("FAILED") exit(1)
exit(1) except RunnerError as e:
except RunnerError as e: print(f"RunnerError-> {e}")
print(f"RunnerError-> {e}")
def main():
def main(): parser = argparse.ArgumentParser(prog="alice")
parser = argparse.ArgumentParser(prog="alice") parser.add_argument("steps", nargs='+')
parser.add_argument("steps", nargs='+') parser.add_argument("-i", "--input", default="alice-ci.yaml")
parser.add_argument("-i", "--input", default="alice-ci.yaml") parser.add_argument("-e", "--env", nargs='*', default=[])
parser.add_argument("-e", "--env", nargs='*', default=[]) parser.add_argument("-a", "--addrunner", nargs='*', default=[])
parser.add_argument("-a", "--addrunner", nargs='*', default=[]) parser.add_argument("-v", "--verbose", action='store_true')
parser.add_argument("-v", "--verbose", action='store_true') args = parser.parse_args()
args = parser.parse_args() if not os.path.isfile(args.input):
if not os.path.isfile(args.input): print(f"No such file: {args.input}")
print(f"No such file: {args.input}") exit(1)
exit(1) parse_jobs(args)
parse_jobs(args)
if __name__ == "__main__":
if __name__ == "__main__": main()
main()

20
alice-ci/src/alice/exceptions.py
View File

@ -1,10 +1,10 @@
class NonZeroRetcode(Exception): class NonZeroRetcode(Exception):
pass pass
class RunnerError(Exception): class RunnerError(Exception):
pass pass
class ConfigException(Exception): class ConfigException(Exception):
pass pass

94
alice-ci/src/alice/runnerfactory.py
View File

@ -1,49 +1,45 @@
from os import getcwd from alice.runners.pythonrunner import PythonRunner
from alice.exceptions import ConfigException
from alice.runners.pythonrunner import PythonRunner
from alice.exceptions import ConfigException
class Factory():
def __init__(self, verbose) -> None:
class Factory(): self.verbose = verbose
def __init__(self, verbose) -> None: self.runnertypes = self.__load_runners()
self.verbose = verbose self.runner_configs = {}
self.runnertypes = self.__load_runners() self.runners = {}
self.runners = {} self.globals = {}
self.workdir = getcwd()
self.globals = {} def __load_runners(self):
# TODO: Runners can be imported via cli too
def __load_runners(self): # https://git.gyulai.cloud/gyulaid/alice/issues/4
# TODO: Runners can be imported via cli too # module = __import__("module_file")
# module = __import__("module_file") # my_class = getattr(module, "class_name")
# my_class = getattr(module, "class_name") runners = {"python": PythonRunner}
runners = {"python": PythonRunner}
if (self.verbose):
if (self.verbose): print(f"[Alice] Available runners: {'|'.join(runners.keys())}")
print(f"[Alice] Available runners: {'|'.join(runners.keys())}") return runners
return runners
def set_globals(self, globals):
def set_globals(self, globals): self.globals = globals
self.globals = globals
def update_runners(self, config):
def update_globals(self, update): for runnertype, runnerconfig in config.items():
if "env" in update: if runnertype != "global":
self.globals["env"].update(update["env"]) if (self.verbose):
print(f"[Alice] Configuring runner: {runnertype}")
def update_runners(self, config): self.get_runner(runnertype).update_config(runnerconfig)
for runnertype, runnerconfig in config.items():
if runnertype != "global": def get_runner(self, runnertype):
if (self.verbose): if runnertype not in self.runners:
print(f"[Alice] Configuring runner {runnertype}") if runnertype in self.runnertypes:
self.get_runner(runnertype).update_config(runnerconfig) if (self.verbose):
print(f"[Alice] Initializing runner: {runnertype}")
def get_runner(self, runnertype): params = {
if runnertype not in self.runners: "verbose": self.verbose
if runnertype in self.runnertypes: }
if (self.verbose): self.runners[runnertype] = self.runnertypes[runnertype](params, self.globals)
print(f"[Alice] Initializing runner: {runnertype}") else:
self.runners[runnertype] = self.runnertypes[runnertype](self.workdir, raise ConfigException(f"Invalid runner type: {runnertype}")
self.globals, return self.runners[runnertype]
self.verbose)
else:
raise ConfigException(f"Invalid runner type: {runnertype}")
return self.runners[runnertype]

5
alice-ci/src/alice/runners/__init__.py
View File

@ -1,3 +1,2 @@
from alice.runners.pythonrunner import PythonRunner # flake8: noqa F401
from alice.runners.pythonrunner import PythonRunner
__all__ = ["PythonRunner"]

2
alice-ci/src/alice/runners/dockerrunner.py
View File

@ -1 +1 @@
# TODO Implement # TODO Implement

233
alice-ci/src/alice/runners/pythonrunner.py
View File

@ -1,116 +1,117 @@
import subprocess import subprocess
import os import os
import sys import sys
import shlex import shlex
from tabnanny import verbose
from alice.exceptions import NonZeroRetcode, RunnerError, ConfigException
from alice.exceptions import NonZeroRetcode, RunnerError, ConfigException
class PythonRunner():
# same venv across all runs! def __init__(self, params, user_defaults) -> None:
class PythonRunner(): self.verbose = params["verbose"]
def __init__(self, workdir, defaults, verbose) -> None: if self.verbose:
self.workdir = workdir print("[PythonRunner] Initializing")
self.virtual_dir = os.path.abspath(os.path.join(workdir, "venv")) self.workdir = user_defaults["workdir"]
self.config = defaults self.virtual_dir = os.path.abspath(os.path.join(self.workdir, "venv"))
self.env_vars = os.environ.copy() self.config = user_defaults
for env_var in defaults["env"]:
self.env_vars[env_var["name"]] = env_var["value"] self.__init_venv()
self.verbose = verbose
def __init_venv(self):
self.__init_venv() if os.name == "nt": # Windows
self.vpython = os.path.join(self.virtual_dir, "Scripts", "python.exe")
def __init_venv(self): else: # Linux & Mac
if os.name == "nt": # Windows self.vpython = os.path.join(self.virtual_dir, "bin", "python3")
self.vpython = os.path.join(self.virtual_dir, "Scripts", "python.exe")
else: # Linux & Mac if not os.path.exists(self.vpython):
self.vpython = os.path.join(self.virtual_dir, "bin", "python3") print("[PythonRunner] Initializing venv")
with subprocess.Popen([sys.executable, "-m", "virtualenv", self.virtual_dir],
if not os.path.exists(self.vpython): stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p:
with subprocess.Popen([sys.executable, "-m", "virtualenv", self.virtual_dir], p.wait()
stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p: if p.returncode != 0:
p.wait() sys.stdout.buffer.write(p.stderr.read())
if p.returncode != 0: raise RunnerError("[PythonRunner] Could not create virtualenv")
sys.stdout.buffer.write(p.stderr.read()) else:
raise RunnerError("[PythonRunner] Could not create virtualenv") if self.verbose:
else: print(f"[PythonRunner] Virtualenv initialized at {self.virtual_dir}")
if self.verbose: else:
print(f"[PythonRunner] Virtualenv initialized at {self.virtual_dir}") if self.verbose:
else: print(f"[PythonRunner] Found virtualenv at {self.virtual_dir}")
if self.verbose:
print(f"[PythonRunner] Found virtualenv at {self.virtual_dir}") # Stores common defaults for all jobs - all types!
# Also - dependency install by config is only allowed in this step
# Stores common defaults for all jobs - all types! def update_config(self, config):
# Also - dependency install by config is only allowed in this step if "dependencies" in config:
def update_config(self, config): for dependency in config["dependencies"]:
if "dependencies" in config: # TODO: Check what happens with fixed version
for dependency in config["dependencies"]: command = [self.vpython, "-m", "pip", "install", dependency, "--upgrade"]
# TODO: Check what happens with fixed version with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p:
command = [self.vpython, "-m", "pip", "install", dependency, "--upgrade"] p.wait()
with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p: if p.returncode != 0:
p.wait() sys.stdout.buffer.write(p.stderr.read())
if p.returncode != 0: raise(RunnerError(f"[PythonRunner] Could not install dependency: {dependency} ({p.returncode})"))
sys.stdout.buffer.write(p.stderr.read()) if "env" in config:
raise(RunnerError(f"[PythonRunner] Could not install dependency: {dependency} ({p.returncode})")) for env_var in config["env"]:
if "env" in config: self.config["env"][env_var["name"]] = env_var["value"]
for env_var in config["env"]: if "workdir" in config and config["workdir"] is not None:
self.env_vars[env_var["name"]] = env_var["value"] self.workdir = os.path.join(self.workdir, config["workdir"])
if "workdir" in config and config["workdir"] is not None:
self.workdir = os.path.join(self.workdir, config["workdir"]) def __ghetto_glob(self, command, workdir):
if self.verbose:
def __ghetto_glob(self, command, workdir): print(f"[PythonRunner][Globbing] Starting command: {' '.join(command)}")
if self.verbose: new_command = []
print(f"[PythonRunner][Globbing] Starting command: {' '.join(command)}") for item in command:
new_command = [] if "*" in item:
for item in command: if self.verbose:
if "*" in item: print(f"[PythonRunner][Globbing] Found item: [{item}]")
if self.verbose: dir = os.path.abspath(os.path.join(workdir, os.path.dirname(item)))
print(f"[PythonRunner][Globbing] Found item: [{item}]") base_name = os.path.basename(item)
dir = os.path.abspath(os.path.join(workdir, os.path.dirname(item))) if os.path.isdir(dir):
base_name = os.path.basename(item) item_parts = base_name.split("*")
if os.path.isdir(dir): for file in os.listdir(dir):
item_parts = base_name.split("*") # TODO: Fix ordering! A*B = B*A = AB*
for file in os.listdir(dir): if item_parts[0] in file and item_parts[1] in file:
# TODO: Fix ordering! A*B = B*A = AB* new_item = os.path.join(dir, file)
if item_parts[0] in file and item_parts[1] in file: if self.verbose:
new_item = os.path.join(dir, file) print(f"[PythonRunner][Globbing] Substitute: {new_item}")
if self.verbose: new_command.append(new_item)
print(f"[PythonRunner][Globbing] Substitute: {new_item}") else:
new_command.append(new_item) if self.verbose:
else: print(f"[PythonRunner][Globbing] Dir not exists: {dir}")
if self.verbose: else:
print(f"[PythonRunner][Globbing] Dir not exists: {dir}") new_command.append(item)
else: return new_command
new_command.append(item)
return new_command # Executes the given job in the one and only venv
# parameter shall be the raw jobscpec
# Executes the given job in the one and only venv def run(self, job_spec):
# parameter shall be the raw jobscpec if "workdir" in job_spec:
def run(self, job_spec): pwd = os.path.abspath(os.path.join(self.workdir, job_spec["workdir"]))
if "workdir" in job_spec: else:
pwd = os.path.abspath(os.path.join(self.workdir, job_spec["workdir"])) pwd = self.workdir
else: run_env = self.config["env"].copy()
pwd = self.workdir if "env" in job_spec:
run_env = self.env_vars.copy() for env_var in job_spec["env"]:
if "env" in job_spec: run_env[env_var["name"]] = env_var["value"]
for env_var in job_spec["env"]: if "commands" in job_spec:
run_env[env_var["name"]] = env_var["value"] commands = job_spec["commands"]
if "commands" in job_spec: for command in commands:
commands = job_spec["commands"] if self.verbose:
for command in commands: print(f"[PythonRunner] Raw command: {command}")
if self.verbose: # TODO: only split if command is not an array
print(f"[PythonRunner] Raw command: {command}") if "*" in command:
# TODO: only split if command is not an array run_command = self.__ghetto_glob(shlex.split(command), pwd)
run_command = self.__ghetto_glob(shlex.split(command), pwd) else:
if self.verbose: run_command = shlex.split(command)
print(f"[PythonRunner] Command to execute: {run_command}") if self.verbose:
print(f"[PythonRunner] Workdir: {pwd}") print(f"[PythonRunner] Command to execute: {run_command}")
if os.path.isdir(pwd): print(f"[PythonRunner] Workdir: {pwd}")
with subprocess.Popen([self.vpython] + run_command, cwd=pwd, env=run_env) as p: if os.path.isdir(pwd):
p.wait() with subprocess.Popen([self.vpython] + run_command, cwd=pwd, env=run_env) as p:
if p.returncode != 0: p.wait()
raise NonZeroRetcode(f"Command {command} returned code {p.returncode}") if p.returncode != 0:
else: raise NonZeroRetcode(f"Command {command} returned code {p.returncode}")
raise RunnerError(f"[PythonRunner] Invalid path for shell command: {pwd}") else:
else: raise RunnerError(f"[PythonRunner] Invalid path for shell command: {pwd}")
raise ConfigException(f"[PythonRunner] No commands specified in step {job_spec['name']}") else:
raise ConfigException(f"[PythonRunner] No commands specified in step {job_spec['name']}")

174
alice-ci/src/alice/utils.py
View File

@ -1,86 +1,88 @@
import os from os import getcwd, path, environ
import subprocess import subprocess
import yaml import yaml
from alice.exceptions import ConfigException from alice.exceptions import ConfigException
class ConfigParser: class ConfigParser:
def __init__(self, file_path, factory, verbose=False) -> None: def __init__(self, file_path, factory, cli_env_vars, verbose=False) -> None:
self.verbose = verbose self.verbose = verbose
with open(file_path) as f: with open(file_path) as f:
self.config = yaml.safe_load(f) self.config = yaml.safe_load(f)
self.factory = factory self.factory = factory
self.factory.set_globals(self.__gen_globals()) self.factory.set_globals(self.__gen_globals(cli_env_vars))
if "runners" in self.config: if "runners" in self.config:
self.factory.update_runners(self.config["runners"]) self.factory.update_runners(self.config["runners"])
self.jobs = self.__get_jobs() self.jobs = self.__get_jobs()
# Initialize env, workdir if not present # Initialize env and workdir if not present in global
def __gen_globals(self): def __gen_globals(self, cli_vars):
globals = { env_vars = environ.copy()
"env": [], env_vars.update(cli_vars)
"workdir": None globals = {
} "env": env_vars,
if "runners" in self.config: "workdir": getcwd()
if "global" in self.config["runners"]: }
if "env" in self.config["runners"]["global"]: if "runners" in self.config:
globals["env"] = self.config["runners"]["global"]["env"] if "global" in self.config["runners"]:
if "workdir" in self.config["runners"]["global"]: if "env" in self.config["runners"]["global"]:
globals["workdir"] = self.config["runners"]["global"]["workdir"] for var in self.config["runners"]["global"]["env"]:
globals["env"][var["name"]] = var["value"]
if (self.verbose): if "workdir" in self.config["runners"]["global"]:
print(f"[Alice] Configured globals: {globals}") globals["workdir"] = self.config["runners"]["global"]["workdir"]
return globals
if (self.verbose):
def __get_jobs(self): print(f"[Alice] Configured globals: {globals}")
if "jobs" in self.config: return globals
jobs = {}
for job_spec in self.config["jobs"]: def __get_jobs(self):
name = job_spec["name"] if "jobs" in self.config:
if name in jobs: jobs = {}
raise ConfigException(f"Job with name {name} already exists!") for job_spec in self.config["jobs"]:
name = job_spec["name"]
jobs[name] = job_spec if name in jobs:
if (self.verbose): raise ConfigException(f"Job with name {name} already exists!")
print(f"[Alice] Parsed jobs: {', '.join(jobs.keys())}")
return jobs jobs[name] = job_spec
else: if (self.verbose):
raise ConfigException("No jobs defined in config") print(f"[Alice] Parsed jobs: {', '.join(jobs.keys())}")
return jobs
def __is_changed(self, changes): else:
try: raise ConfigException("No jobs defined in config")
target = changes["branch"]
paths = [] def __is_changed(self, changes):
for path in changes["paths"]: try:
paths.append(os.path.abspath(path)) target = changes["branch"]
# TODO: Error handling paths = []
command = ["git", "diff", "--name-only", target] for _path in changes["paths"]:
with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p: paths.append(path.abspath(_path))
p.wait() # TODO: Error handling
for line in p.stdout: command = ["git", "diff", "--name-only", target]
change_path = os.path.abspath(line.decode("UTF-8").strip()) with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p:
for path in paths: p.wait()
spec_path = os.path.abspath(path) for line in p.stdout:
if change_path.startswith(spec_path): change_path = path.abspath(line.decode("UTF-8").strip())
if self.verbose: for _path in paths:
print(f"[Alice] Modified file: {change_path}") spec_path = path.abspath(_path)
print(f"[Alice] Path match: {path}") if change_path.startswith(spec_path):
return True if self.verbose:
except KeyError: print(f"[Alice] Modified file: {change_path}")
raise ConfigException(f"Invalid 'changes' config: {changes}") print(f"[Alice] Path match: {_path}")
return False return True
except KeyError:
def execute_job(self, job_name): raise ConfigException(f"Invalid 'changes' config: {changes}")
if job_name in self.jobs: return False
job_spec = self.jobs[job_name]
should_run = True def execute_job(self, job_name):
if "changes" in job_spec: if job_name in self.jobs:
should_run = self.__is_changed(job_spec["changes"]) job_spec = self.jobs[job_name]
if should_run: should_run = True
runner = self.factory.get_runner(job_spec["type"]) if "changes" in job_spec:
runner.run(job_spec) should_run = self.__is_changed(job_spec["changes"])
return "SUCCESS" if should_run:
else: runner = self.factory.get_runner(job_spec["type"])
return "SKIP, no change detected" runner.run(job_spec)
return "SUCCESS"
else:
return "SKIP, no change detected"

64
ci-examples/full.yaml
View File

@ -1,33 +1,33 @@
runners: runners:
global: global:
env: env:
- name: A - name: A
value: A value: A
- name: B - name: B
value: B value: B
- name: C - name: C
value: C value: C
workdir: packages workdir: packages
python: python:
env: env:
- name: A - name: A
value: D value: D
dependencies: dependencies:
- flake8 - flake8
- build - build
jobs: jobs:
- name: env - name: env
type: python type: python
changes: changes:
branch: origin/master branch: origin/master
paths: paths:
- "docs" - "docs"
env: env:
- name: B - name: B
value: E value: E
commands: commands:
- "-c \"import os; print(os.environ)\"" - "-c \"import os; print(os.environ)\""
- name: lint - name: lint
workdir: alice-ci workdir: alice-ci
commands: commands:
- "-m flake8 --ignore E501" - "-m flake8 --ignore E501"

34
ci-examples/python1.yaml
View File

@ -1,18 +1,18 @@
runners: runners:
python: python:
dependencies: dependencies:
- flake8 - flake8
- build - build
- twine - twine
jobs: jobs:
- name: selfcheck - name: selfcheck
type: python type: python
workdir: ci workdir: ci
commands: commands:
- "-m flake8 --ignore E501 --exclude venv" - "-m flake8 --ignore E501 --exclude venv"
- name: lint - name: lint
type: python type: python
workdir: alice-ci/src workdir: alice-ci/src
commands: commands:
- "-m flake8 --ignore E501" - "-m flake8 --ignore E501"

46
docs/examples.md
View File

@ -1,24 +1,24 @@
# alice-ci.yaml examples # alice-ci.yaml examples
## Python lint ## Python lint
Installes flake8 package in a virtual elvironment, then lints the contents of the packages directory in the current working dir. Installes flake8 package in a virtual elvironment, then lints the contents of the packages directory in the current working dir.
``` ```
runners: runners:
python: python:
dependencies: dependencies:
- name: flake8 - name: flake8
jobs: jobs:
- name: lint - name: lint
type: python type: python
workdir: packages workdir: packages
commands: commands:
- "-m flake8" - "-m flake8"
``` ```
To run this job: To run this job:
``` ```
pythom3 -m alice lint pythom3 -m alice lint
``` ```

53
docs/runners.md
View File

@ -1,8 +1,45 @@
# Runners # Runners
Runners are responsible to execute a list of commands in a set environment defined in the CI yaml file. Runners are responsible to execute a list of commands in a set environment defined in the CI yaml file.
## List of runners ## List of runners
* Python - executes python commands in a virtual environment * Python - executes python commands in a virtual environment
* Docker - executes each job in a separate Docker container - unimplemented * Docker - executes each job in a separate Docker container - unimplemented
## Import schema
What you need to do to make Alice recognise and import your custom Runners
TODO
## Runner API
Each runner has to support the following functions:
### __init__(params, user_defaults)
* params: dict of runtime variables for the program itself.
* user_defaults: raw data from the CI file's global dict, augmented with an "env" dict, which contains environment variables from the host sytem, the CLI params and the pipeline global config, and the "workdir" value, which is the absolute path of the directory that the runner shall recognize as the current working directory.
#### Params
Currently the only param used is the dict is "verbose", whichis a boolean. The intended purpose is to show debug output if set to True.
#### Workdir
workdir can be assigned at CI yaml level as global
Order:
By default: os.cwd()
if overwritten in global
------------------------------- Below this level is the runner's responsibility
if owerwritten in runner config
if overwritten in job
Runner shall receive the current working directory, unless stated otherwise in global config
### update_config(config)
The function takes the raw data from the parsed yaml under runners.(runnername). Handling its own config is the sole responsibility of the runner. This function may be called at any point of the running lifecycle, so the runner has to support changing its own configuration.
### run(job_spec)
This function executes one job attributed ith the type of the runner called. As the only hard requirement for Alice is the "type" field in a job (or the optional "changes"), everything else is handled by the runner.

78
docs/syntax.md
View File

@ -1,39 +1,39 @@
# alice-ci.yaml # alice-ci.yaml
This yaml file defines the job steps executed by Alice. The jobs are called by names for each passed parameter on CLI. For example the following command searches for a job called lint defined in the `alice-ci.yaml` file in the current working directory, then runs it. This yaml file defines the job steps executed by Alice. The jobs are called by names for each passed parameter on CLI. For example the following command searches for a job called lint defined in the `alice-ci.yaml` file in the current working directory, then runs it.
``` ```
pythom3 -m alice lint pythom3 -m alice lint
``` ```
[Example configs](examples.md) [Example configs](examples.md)
## runners ## runners
Contains global configuration for various runners. Currently the only supported runner is `python`. Contains global configuration for various runners. Currently the only supported runner is `python`.
### Python ### Python
#### Dependencies #### Dependencies
List of dependencies installed in the virtual environment. Each dependency has a `name` and an `import_name`, as Alice checks the availability of each package by trying to import `import_name`, and if it fails, calls pip to install `name`. List of dependencies installed in the virtual environment. Each dependency has a `name` and an `import_name`, as Alice checks the availability of each package by trying to import `import_name`, and if it fails, calls pip to install `name`.
## jobs ## jobs
List of jobs. Each job has a mandatory name, type and a list of commands, optional parameter is workdir. List of jobs. Each job has a mandatory name, type and a list of commands, optional parameter is workdir.
### name ### name
Mandatory value, string. Has to be unique in the current file. Mandatory value, string. Has to be unique in the current file.
### type ### type
Job type, selects the runner executing the commands. Currently the only supported type is `python`. Job type, selects the runner executing the commands. Currently the only supported type is `python`.
### comands ### comands
List of strings, each executed one by one from top to bottom in the current context. List of strings, each executed one by one from top to bottom in the current context.
### workdir ### workdir
Optional, defines Working directory relative to PWD. The default working directory is the current directory. Optional, defines Working directory relative to PWD. The default working directory is the current directory.

Write Preview
Loading…
Cancel
Save