In this article I want to share my experience of working with NJS, a JavaScript interpreter for Nginx developed by Nginx Inc, describing its main features using a real example. NJS is a subset of JavaScript that allows you to extend the functionality of Nginx. When asked why own interpreter ??? Dmitry Volyntsev answered in detail. In short: NJS is nginx-way, and JavaScript is more progressive, "native" and without GC, unlike Lua.
A long time ago ...
At my last job, I inherited gitlab with a number of motley CI / CD pipelines with docker-compose, dind and other delights that were transferred to the kaniko rails. The images that were previously used in CI have moved in their original form. They worked fine until the day when our gitlab changed its IP and CI turned into a pumpkin. The problem was that one of the docker images that participated in the CI contained git, which pulled Python modules via ssh. Ssh needs a private key and ... it was in the image along with the known_hosts. And any CI failed to verify the key due to a mismatch between the real IP and the one specified in the known_hosts. A new image was quickly built from the existing Dockfiles and an option was addedStrictHostKeyChecking no... But the unpleasant aftertaste remained and there was a desire to transfer it to a private PyPI repository. An additional bonus, after switching to a private PyPI, became a simpler pipeline and a normal description of requirements.txt
The choice is made, gentlemen!
We spin everything in the clouds and Kubernetes, and in the end we wanted to get a small service that was a stateless container with external storage. Well, since we use S3, then it was the priority. And, if possible, with authentication in gitlab (you can add it yourself if necessary).
A quick search yielded several results for s3pypi, pypicloud, and an option to "manually" generate html files for the turnip. The last option disappeared by itself.
s3pypi: This is the cli for using S3 hosting. We upload files, generate html and fill in the same bucket. Suitable for home use.
pypicloud: , . , . , , 3-5 . . , .
Nginx, ngx_aws_auth. XML , S3. , , . .
PEP-503 , XML HTML pip. Nginx S3 S3 JS Nginx. NJS.
, XML, ngx_aws_auth, JS.
nginx . - , - Nginx ( ), - Nginx, . , Python Go ( ), nexus.
TL;DR 2 PyPi CI.
?
Nginx ngx_http_js_module, docker-. c js_import Nginx. js_content. js_set, . NJS Nginx, XMLHttpRequest. Nginx . (subrequest) . Nginx, export default.
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import imported_name from script.js;
server {
listen 8080;
...
location = /sub-query {
internal;
proxy_pass http://upstream;
}
location / {
js_content imported_name.request;
}
}script.js
function request(r) {
function call_back(resp) {
// handler's code
r.return(resp.status, resp.responseBody);
}
r.subrequest('/sub-query', { method: r.method }, call_back);
}
export default {request} http://localhost:8080/ location / js_content request script.js. request location = /sub-query, ( GET) (r), . call_back.
S3
S3-, :
ACCESS_KEY
SECRET_KEY
S3_BUCKET
http-, /, S3_NAME URI , (HMAC_SHA1) SECRET_KEY. , AWS $ACCESS_KEY:$HASH, . /, , X-amz-date. :
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import s3 from s3.js;
js_set $s3_datetime s3.date_now;
js_set $s3_auth s3.s3_sign;
server {
listen 8080;
...
location ~* /s3-query/(?<s3_path>.*) {
internal;
proxy_set_header X-amz-date $s3_datetime;
proxy_set_header Authorization $s3_auth;
proxy_pass $s3_endpoint/$s3_path;
}
location ~ "^/(?<prefix>[\w-]*)[/]?(?<postfix>[\w-\.]*)$" {
js_content s3.request;
}
}s3.js( AWS Sign v2, deprecated)
var crypt = require('crypto');
var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:\-]|\.\d{3}/g, '');
function date_now() {
return _datetime
}
function s3_sign(r) {
var s2s = r.method + '\n\n\n\n';
s2s += `x-amz-date:${date_now()}\n`;
s2s += '/' + s3_bucket;
s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;
return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}
function request(r) {
var v = r.variables;
function call_back(resp) {
r.return(resp.status, resp.responseBody);
}
var _subrequest_uri = r.uri;
if (r.uri === '/') {
// root
_subrequest_uri = '/?delimiter=/';
} else if (v.prefix !== '' && v.postfix === '') {
// directory
var slash = v.prefix.endsWith('/') ? '' : '/';
_subrequest_uri = '/?prefix=' + v.prefix + slash;
}
r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}
export default {request, s3_sign, date_now} _subrequest_uri: uri S3. «», uri- delimiter, xml- CommonPrefixes, ( PyPI, ). ( ), uri- prefix () /. , . aiohttp-request aiohttp-requests /?prefix=aiohttp-request, . , /?prefix=aiohttp-request/, . , uri .
, Nginx. Nginx, XML, :
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>myback-space</Name>
<Prefix></Prefix>
<Marker></Marker>
<MaxKeys>10000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>new/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>old/</Prefix>
</CommonPrefixes>
</ListBucketResult> CommonPrefixes.
, , , XML:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name> myback-space</Name>
<Prefix>old/</Prefix>
<Marker></Marker>
<MaxKeys>10000</MaxKeys>
<Delimiter></Delimiter>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>old/giphy.mp4</Key>
<LastModified>2020-08-21T20:27:46.000Z</LastModified>
<ETag>"00000000000000000000000000000000-1"</ETag>
<Size>1350084</Size>
<Owner>
<ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
<DisplayName></DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>old/hsd-k8s.jpg</Key>
<LastModified>2020-08-31T16:40:01.000Z</LastModified>
<ETag>"b2d76df4aeb4493c5456366748218093"</ETag>
<Size>93183</Size>
<Owner>
<ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
<DisplayName></DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult> Key.
XML HTML, Content-Type text/html.
function request(r) {
var v = r.variables;
function call_back(resp) {
var body = resp.responseBody;
if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
r.headersOut['Content-Type'] = "text/html; charset=utf-8";
body = toHTML(body);
}
r.return(resp.status, body);
}
var _subrequest_uri = r.uri;
...
}
function toHTML(xml_str) {
var keysMap = {
'CommonPrefixes': 'Prefix',
'Contents': 'Key',
};
var pattern = `<k>(?<v>.*?)<\/k>`;
var out = [];
for(var group_key in keysMap) {
var reS;
var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');
while(reS = reGroup.exec(xml_str)) {
var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
var reValue = data.exec(reS);
var a_text = '';
if (group_key === 'CommonPrefixes') {
a_text = reValue.groups.v.replace(/\//g, '');
} else {
a_text = reValue.groups.v.split('/').slice(-1);
}
out.push(`<a href="/${reValue.groups.v}">${a_text}</a>`);
}
}
return '<html><body>\n' + out.join('</br>\n') + '\n</html></body>'
}PyPI
, .
#
python3 -m venv venv
. ./venv/bin/activate
# .
pip download aiohttp
#
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
rm -f *.whl
#
pip install aiohttp -i http://localhost:8080.
#
python3 -m venv venv
. ./venv/bin/activate
pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
pip install our_pkg --extra-index-url http://localhost:8080CI, :
pip install setuptools wheel
python setup.py bdist_wheel
curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"Gitlab JWT / . auth_request Nginx, . url Gitlab- , Gitlab 200 / . Gitlab? , Nginx , - , . , Kubernetes read-only root filesystem, nginx.conf configmap. Nginx configmap (pvc) read-only root filesystem ( ).
NJS, nginx - (, URL).
nginx.conf
location = /auth-provider {
internal;
proxy_pass $auth_url;
}
location = /auth {
internal;
proxy_set_header Content-Length "";
proxy_pass_request_body off;
js_content auth.auth;
}
location ~ "^/(?<prefix>[\w-]*)[/]?(?<postfix>[\w-\.]*)$" {
auth_request /auth;
js_content s3.request;
}s3.js
var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;
function url() {
return `${gitlab_url}/jwt/auth?service=container_registry`
}
function auth(r) {
if (auth_disabled) {
r.return(202, '{"auth": "disabled"}');
return null
}
r.subrequest('/auth-provider',
{method: 'GET', body: ''},
function(res) {
r.return(res.status, "");
});
}
export default {auth, url}: - ? ! , var AWS = require('aws-sdk') "" S3-!
, JS-, , . require('crypto'), build-in- require . - . , - .
Nginx gzip off;
, gzip- NJS , . , . , . , .
«» error.log. info, warn error 3 r.log, r.warn, r.error . Chrome (v8) njs, . , , history :
docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginx.
, . IDE . , .
ES6.
- , . NJS.
NJS - open-source , Nginx JavaScript. . , . , - NJS , Nginx . NGINX Plus - !
Examples of using NJS from Dmitry Volyntsev
njs - native JavaScript scripting in nginx / Dmitry Volnyev's speech at Saint HighLoad ++ 2019
NJS in production / Vasily Soshnikov's speech at HighLoad ++ 2019