❤
npm
Sign UpSign In

@ai-em/python
TypeScript icon, indicating that this package has built-in type declarations

16.0.3 • Public • Published

@ai-em/python

Coverage Badge

@ai-em/python plugin is designed to extend the Nx features to work with Python projects based on Poetry.

Check this article for more details: https://lucasvieirasilva.medium.com/poetry-python-nx-monorepo-5750d8627024

What is @ai-em/python

🔎 An Nx Custom Plugin to generate Python projects using Poetry, Tox and a custom dependency tree plugin

Getting Started

Add to an existing Nx Workspace

Install the npm dependency

npm install @ai-em/python --save-dev

Usage

  1. Update nx.json to add the property plugins with @ai-em/python value.

Example:

{
  ...
  "plugins": [
    "@ai-em/python"
  ]
  ...
}

Add a new Python Project

nx generate @ai-em/python:project myproject

Options

Option Type Description Required Default
--type string Project type application or library true application
--description string Project description false
--directory string A directory where the project is placed false
--packageName string Package name true
--publishable boolean Speficies if the project is publishable or not false true
--buildLockedVersions boolean Use locked versions for build dependencies false true
--buildBundleLocalDependencies boolean Bundle local dependencies false true
--customSource boolean Speficies if the project uses custom PyPi registry false false
--sourceName string Custom PyPi registry name only if the --customSource is true
--sourceUrl string Custom PyPi registry url only if the --customSource is true
--sourceSecondary boolean Custom PyPi registry secondary flag only if the --customSource is true true
--tags string Add tags to the project false

Add a new dependency to a project

nx run {project}:add --name {projectName} --local

Add an external dependency to the project

To add a new dependency to the project use the nx run {project}:add command detailed below. This ensures that any dependent projects are updated.

nx run {project}:add --name {dependencyName}

Executors

sls-deploy

The @ai-em/python:sls-deploy executor handles npx sls deploy command for serverless framework projects.

This executor uses the @ai-em/python:build artifacts to generate a requirements.txt and to be used with serverless-python-requirements plugin.

Serverless YAML example:

service: myservice

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    usePoetry: false

The property usePoetry must be false, so, the serverless-python-requirements uses the requirements.txt file generated by this executor, this is required when the project has more than 2 levels of local dependencies.

Example:

- root:
  - sls-app
   - local-lib1
    - local-lib2

Using the native serverless-python-requirements plugin with poetry the 2 levels of local dependencies are not supported.

project.json example:

{
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "projectType": "application",
  "sourceRoot": "apps/myapp/lambda_functions",
  "targets": {
    "deploy": {
      "executor": "@ai-em/python:sls-deploy",
      "dependsOn": ["build"],
      "options": {}
    },
    "package": {
      "executor": "@ai-em/python:sls-package",
      "dependsOn": ["build"],
      "options": {}
    },
    ...
    "build": {
      "executor": "@ai-em/python:build",
      "outputs": ["apps/myapp/dist"],
      "options": {
        "outputPath": "apps/myapp/dist",
        "publish": false
      }
    },
  }
}
Options
Option Type Description Required Default
--stage string Serverless Framework stahe name true
--verbose boolean Serverless Framework CLI verbose flag false true
--force boolean Serverless Framework CLI force flag false false

add

The @ai-em/python:add executor handles poetry add command to provide a level of abstraction and control in the monorepo projects.

Features
  • Add new external dependencies
  • Add local dependencies

Both features updates the local workspace dependency tree to keep the lock/venv updated.

Options
Option Type Description Required Default
--name string Dependency name (if local dependency use the Nx project name) true
--args string Custom args to be used in the poetry add command false
--local boolean Specifies if the dependency is local false (only if the --name is a local dependency)

update

The @ai-em/python:update executor handles poetry update command to provide a level of abstraction and control in the monorepo projects.

Features
  • Update external dependencies
  • Update local dependencies

Both features updates the local workspace dependency tree to keep the lock/venv updated.

Options
Option Type Description Required Default
--name string Dependency name (if local dependency use the Nx project name) false
--args string Custom args to be used in the poetry update command false
--local boolean Specifies if the dependency is local false (only if the --name is a local dependency)

remove

The @ai-em/python:remove executor handles poetry remove command to provide a level of abstraction and control in the monorepo projects.

Features
  • Remove external dependencies
  • Remove local dependencies

Both features updates the local workspace dependency tree to keep the lock/venv updated.

Options
Option Type Description Required Default
--name string Dependency name (if local dependency use the Nx project name) true
--args string Custom args to be used in the poetry remove command false
--local boolean Specifies if the dependency is local false (only if the --name is a local dependency)

build

The @ai-em/python:build command handles the sdist and wheel build generation. When the project has local dependencies the executor copies the package/dependencies recursively.

Options
Option Type Description Required Default
--silent boolean Hide output text false false
--outputPath string Output path for the python tar/whl files true
--keepBuildFolder boolean Keep build folder false false
--lockedVersions boolean Build with locked versions false true
--bundleLocalDependencies boolean Bundle local dependencies false true
--ignorePaths array Ignore folder/files on build process false [".venv", ".tox", "tests"]
Locked Versions Build

Using the default (lockedVersions and bundleLocalDependencies) options, the executor uses the locked versions across all the dependencies and bundles the local dependencies in the same wheel file.

packages/proj1/pyproject.toml

[tool.poetry]
name = "pymonorepo-proj1"

  [[tool.poetry.packages]]
  include = "pymonorepo_proj1"

  [tool.poetry.dependencies]
  python = ">=3.8,<3.10"
  pendulum = "^2.1.2"

    [tool.poetry.dependencies.pymonorepo-lib1]
    path = "../lib1"
    develop = true

packages/lib1/pyproject.toml

[tool.poetry]
name = "pymonorepo-lib1"
version = "1.0.0"

  [[tool.poetry.packages]]
  include = "pymonorepo_lib1"

  [tool.poetry.dependencies]
  python = ">=3.8,<3.10"
  numpy = "^1.24.1"

When the build is executed in the proj1 package, the dist tar/whl file will contain the lib1 package and all dependencies in the poetry.lock file.

packages/proj1/dist/pymonorepo-proj1-1.0.0.tar.gz/pyproject.toml

[tool.poetry]
name = "pymonorepo-proj1"
version = "1.0.0"

  [[tool.poetry.packages]]
  include = "pymonorepo_proj1"

  [[tool.poetry.packages]]
  include = "pymonorepo_lib1"

  [tool.poetry.dependencies]
  python = ">=3.8,<3.10"

    [tool.poetry.dependencies.numpy]
    version = "1.24.1 "
    markers = 'python_version >= "3.8" and python_version < "3.10"'
    optional = false

    [tool.poetry.dependencies.pendulum]
    version = "2.1.2 "
    markers = 'python_version >= "3.8" and python_version < "3.10"'
    optional = false

    [tool.poetry.dependencies.python-dateutil]
    version = "2.8.2 "
    markers = 'python_version >= "3.8" and python_version < "3.10"'
    optional = false

    [tool.poetry.dependencies.pytzdata]
    version = "2020.1 "
    markers = 'python_version >= "3.8" and python_version < "3.10"'
    optional = false

    [tool.poetry.dependencies.six]
    version = "1.16.0 "
    markers = 'python_version >= "3.8" and python_version < "3.10"'
    optional = false

Note, that python-dateutil is a dependency of pendulum, and the pymonorepo_lib1 is now part of the project instead of a dependency.

Non-Locked Versions Build

Using the --lockedVersions=false option, the executor uses the versions from the pyproject.toml file across all the dependencies and bundles the local dependencies in the same wheel file.

packages/proj1/dist/pymonorepo-proj1-1.0.0.tar.gz/pyproject.toml

[tool.poetry]
name = "pymonorepo-proj1"
version = "1.0.0"

  [[tool.poetry.packages]]
  include = "pymonorepo_proj1"

  [[tool.poetry.packages]]
  include = "pymonorepo_lib1"

  [tool.poetry.dependencies]
  python = ">=3.8,<3.10"
  numpy = "^1.24.1"
  pendulum = "^2.1.2"

Note, the pymonorepo_lib1 still bundled in the project but the dependencies are listed in the same way it is on the pyproject (using ^).

Non-Bundled Local Dependencies Build

Using the --bundleLocalDependencies=false and --lockedVersions=false options, the executor checks if the local dependency is publishable and uses the version from the pyproject.toml file, instead of bundling the package.

packages/proj1/dist/pymonorepo-proj1-1.0.0.tar.gz/pyproject.toml

[tool.poetry]
name = "pymonorepo-proj1"
version = "1.0.0"

  [[tool.poetry.packages]]
  include = "pymonorepo_proj1"

  [tool.poetry.dependencies]
  python = ">=3.8,<3.10"
  pendulum = "^2.1.2"
  pymonorepo-lib1 = "1.0.0"

To identify if the package is publishable, the executor checks project.json file, property targets.build.options.publish.

If the publish option is set to false and the --bundleLocalDependencies=false option is used, the executor will bundle the package.

Custom source specification

In addition when adding dependencies in this way its also possible to configure a custom source for a package. This works similar to the publish option in that its specified on the target dependencies build options. To use this set the customSourceName and customSourceUrl to valid values for the source to retrieve the package from for each package stored on a custom Pypi.

project.json example:

{
  ...
  "targets": {
    ...
    "build": {
      "executor": "@ai-em/python:build",
      "outputs": ["apps/myapp/dist"],
      "options": {
        "outputPath": "apps/myapp/dist",
        "publish": false,
        "customSourceName": "example",
        "customSourceUrl": "http://example.com/"
      }
    },
  }
}

Alternatively its also possible to configured it within the nx.json as targetDefaults across the whole repository.

flake8

The @ai-em/python:flake8 handles the flake8 linting tasks and reporting generator.

Options
Option Type Description Required Default
--silent boolean Hide output text false false
--outputFile string Output pylint file path true

install

The @ai-em/python:install handles the poetry install command for a project.

Options

Option Type Description Required Default
--silent boolean Hide output text false false
--args string Custom arguments (e.g --group dev) false
--cacheDir string Custom poetry install cache directory false
--verbose boolean Use verbose mode in the install poetry install -vv false false
--debug boolean Use debug mode in the install poetry install -vvv false false

tox

The @ai-em/python:tox handles tox executions for a project.

Options

Option Type Description Required Default
--silent boolean Hide output text false false
--args string Custom arguments (e.g -e py38) false

Shared Virtual Environment

By default, the @ai-em/python manages the projects individually, so, all the projects have their one set of dependencies and virtual environments.

However, In some cases, we want to use a shared virtual environment for the entire workspace to save some installation time in your local environment and CI tool, we use this mode when the workspace contains many projects with the same dependencies and versions that don't conflict in the workspace level.

To migrate to this mode, run the following command:

npx nx generate @ai-em/python:migrate-to-shared-venv

Options

Option Type Description Required Default
--moveDevDependencies boolean Specifies if migration moves the dev dependencies from the projects to the root pyproject.toml true true

Readme

Keywords

none

Package Sidebar

Install

npm i @ai-em/python

Repository

github.com/lucasvieirasilva/nx-plugins

Homepage

github.com/lucasvieirasilva/nx-plugins#readme

Weekly Downloads

0

Version

16.0.3

License

MIT

Unpacked Size

195 kB

Total Files

108

Last publish

Collaborators

  • ai-energy-miracle
Try on RunKit
Report malware