@@ -26,7 +26,7 @@ Currently, the flask backend provides mostly/only headless api routes which are
Still, the tabs and links that appear in the menu bar (e.g., "Dashboards") are defined in the backend init.
API endpoints are generally defined in the following structure: `superset/<resource_name_plural>/api.py`.
- Note the route method whitelist `include_route_methods`. Otherwise, the route will not be accessible, even if it is defined with `@expose`. This gives 404s.
- Note the route method whitelist `include_route_methods` class attribute. Otherwise, the route will not be accessible, even if it is defined with `@expose`. This gives 404s.
- (At least) when using the `@protect` decorator, the route method may have to be added in `method_permission_name`. When testing a newly added route, consider that per-method permissions may not be automatically added (even for the admin) until full restart, or even recreation of the mounted superset_db volume. This may lead to 405s.
The MitM routes are defined in `superset/customization/`.
...
...
@@ -85,7 +85,7 @@ Without manually adding a dialect-implementing package, having a sqlalchemy uri
## MitM Additions
The mitm support touches all parts of the superset architecture.
The MitM support touches all parts of the superset architecture.
It is gated with the feature flag "MITM_SUPPORT" in the superset config.
### MitMDataset Asset
...
...
@@ -104,7 +104,7 @@ Ordinarily, superset just manages links to external databases that live outside
For seamless integration, the MitM files (e.g., .maed/.zip) themselves need to be handled.
While superset has some support for uploading (csv) files to databases that are configured to permit it, this is insufficient for proper MitM support.
To have a separation of concerns, I decided to implement MitM supporting functionality as (tightly-coupled) "microservices".
For example, logic relating to the handling of MitM files (e.g., .maed/.zip), including upload and insertion into a managed relational database, is implemented in a separate microservice: `superset-mitm-service`.
For example, logic relating to the handling of MitM files (e.g., .maed/.zip), including upload and insertion into a managed relational database, is implemented in a separate microservice: `superset-mitm-service` ([repository](https://git-ce.rwth-aachen.de/machine-data/superset-mitm-service)).
### External Service Support
...
...
@@ -116,3 +116,14 @@ In addition to full request forwarding, the external service manager can be used
For example, the aforementioned upload of MitM files makes multiple requests to the supporting service in the `upload_mitm_dataset_task`.
To make passing around large results more feasible, there are some new (streaming) write to/read from cache commands.
## Kubernetes Helm Chart
In addition to the docker compose setup, Superset comes with a highly configurable helm chart (`helm/superset`).
I have modified the chart to depend on my `superset-mitm-service` chart, which comes with its own database. (currently also postgres)
Since the MitM support relies on the async query extension and preferably makes use of the bundled websocket server, config relating to these components has been modified.
Consider in particular the key `.Values.configOverrides`, and added templated environment variables in `templates/secret-env.yaml`.
To update Superset deployment, the image needs to be built and pushed to the configured registry (currently: registry.git-ce.rwth-aachen.de/machine-data/mitm-superset:release).
Specifically, the `mitm-superset` layer defined in the `Dockerfile` is needed so that the optional python package dependency group `mitm` is included.
When intending to deploy with other such dependency groups, such as specific database dialect support, they must be added here as well.
For convenient command recipes for docker and kubernetes deployment, consider the `justfile` (and imported `kube.just` file).
