Skip to content

DOMA Token-based AuthZ Testbed

Joining testbed

Send mail to wlcg-doma-tpc e-group if you want to join Rucio DOMA Functional Tests OIDC or advertise your CE that support job submission with tokens. Please include necessary to access your SE/CE endpoint, as an example you can use entries that were already filled in tables with available resources that to some degree already supports tokens.


To be able to use tokens it is necessary to know which token issuer is used by individual VOs. For tests we use WLCG IAM token issuer which provides services for "wlcg" VO, e.g.

VO Issuer

OSG have token isser details stored with all VO detail in their virtual-organization topology files. WLCG/EGI currently doesn't provide common (trusted) place with token isser details, but naturally this could become part of EGI VO ID Cards (briefly touched this topic here).



ARC-CE 6.12 still has limited support for WLCG JWT tokens and they can be only used to submit jobs with HTTP based protocols (EMI-ES and REST interface). With current limitations configuration close to expectations from WLCG JWT profile could look like (replace with your ARC-CE hostname or if you don't use standard HTTPS port 443):

# ...

[authgroup: wlcg_iam]
# capability based authorization that use compute.* scopes
authtokens = * compute.create *
authtokens = * *
authtokens = * compute.modify *
authtokens = * compute.cancel *
# group based authorization that use /wlcg/pilots group (LHC experiments prefer capabilities)
authtokens = * * /wlcg/pilots

# accept token issued by EGI Check-in for job submission (both old MitreID and new Keycloak issuer)
[authgroup: egi_aai]
# very rough / DANGEROUS configuration - accepting all tokens without restrictions
#authtokens = * * * *
#authtokens = * * * *
# it is possible to restrict job submission to the specific EGI user
authtokens = * * *
authtokens = * * *

# just an example for ARC-CE running on
# recommendation for ATLAS configuration may change in fugure
# (this is not the official ATLAS site configuration documentation)
[authgroup: atlas_iam_prd]
authtokens = 7dee38a3-6ab8-4fe2-9e4c-58039c21d817 compute.create *
authtokens = 7dee38a3-6ab8-4fe2-9e4c-58039c21d817 *
authtokens = 7dee38a3-6ab8-4fe2-9e4c-58039c21d817 compute.modify *
authtokens = 7dee38a3-6ab8-4fe2-9e4c-58039c21d817 compute.cancel *
[authgroup: atlas_iam_plt]
authtokens = 750e9609-485a-4ed4-bf16-d5cc46c71024 compute.create *
authtokens = 750e9609-485a-4ed4-bf16-d5cc46c71024 *
authtokens = 750e9609-485a-4ed4-bf16-d5cc46c71024 compute.modify *
authtokens = 750e9609-485a-4ed4-bf16-d5cc46c71024 compute.cancel *
[authgroup: atlas_iam_sgm]
authtokens = 5c5d2a4d-9177-3efa-912f-1b4e5c9fb660 compute.create *
authtokens = 5c5d2a4d-9177-3efa-912f-1b4e5c9fb660 *
authtokens = 5c5d2a4d-9177-3efa-912f-1b4e5c9fb660 compute.modify *
authtokens = 5c5d2a4d-9177-3efa-912f-1b4e5c9fb660 compute.cancel *

# again, just an example for ARC-CE running on
# (this is not the official CMS site configuration documentation)
[authgroup: cms_iam_pilot]
authtokens = bad55f4e-602c-4e8d-a5c5-bd8ffb762113 compute.create *
authtokens = bad55f4e-602c-4e8d-a5c5-bd8ffb762113 *
authtokens = bad55f4e-602c-4e8d-a5c5-bd8ffb762113 compute.modify *
authtokens = bad55f4e-602c-4e8d-a5c5-bd8ffb762113 compute.cancel *
[authgroup: cms_iam_test]
authtokens = 08ca855e-d715-410e-a6ff-ad77306e1763 compute.create *
authtokens = 08ca855e-d715-410e-a6ff-ad77306e1763 *
authtokens = 08ca855e-d715-410e-a6ff-ad77306e1763 compute.modify *
authtokens = 08ca855e-d715-410e-a6ff-ad77306e1763 compute.cancel *
[authgroup: cms_iam_itb]
authtokens = 490a9a36-0268-4070-8813-65af031be5a3 compute.create *
authtokens = 490a9a36-0268-4070-8813-65af031be5a3 *
authtokens = 490a9a36-0268-4070-8813-65af031be5a3 compute.modify *
authtokens = 490a9a36-0268-4070-8813-65af031be5a3 compute.cancel *

# this assumes existence of local users wlcg, egi, atlasprd, atlasplt, atlassgm, cmsplt, cmstest and cmsitb with corresponding groups
map_to_user = wlcg_iam wlcg:wlcg
map_to_user = egi_aai egi:egi
map_to_user = atlas_iam_prd atlasprd:atlasprd
map_to_user = atlas_iam_plt atlasplt:atlasplt
map_to_user = atlas_iam_sgm atlassgm:atlassgm
map_to_user = cms_iam_pilot cmsplt:cmsplt
map_to_user = cms_iam_test cmstest:cmstest
map_to_user = cms_iam_itb cmsitb:cmsitb

# ...

Token implementation in ARC-CE is still preliminary and it seems possible there will be changes even in the structure of configuration file. You should follow official documentation if you install more recent ARC-CE version.

Even though job submission is possible with tokens it is still necessary to have existing X.509 proxy, because current arcsub version still unconditionally verify proxy presence. On the other hand this proxy is delegated to ARC-CE regardless of auth mechanism used to submit jobs.

With token in BEARER_TOKEN environement variable (ARC_OTOKEN for older ARC-CE 6.7 clients) token'll be automatically used for authorization with HTTP interface, e.g.

cat > test.xrsl <<EOF
(jobname="ARC-CE test")
(runtimeenvironment = "ENV/PROXY")
# following line is necessary for ARC-CE 6.7 client
# newer version use BEARER_TOKEN or token discovery
#export ARC_OTOKEN=$(oidc-token --aud= --scope=compute.modify --scope=compute.cancel --scope=compute.create --scope=wlcg.groups:/wlcg/pilots OIDC_STORE_NAME)
export BEARER_OTOKEN=$(oidc-token --aud= --scope=compute.modify --scope=compute.cancel --scope=compute.create --scope=wlcg.groups:/wlcg/pilots OIDC_STORE_NAME)
arcsub --debug=DEBUG --info-endpoint-type arcrest --submission-endpoint-type arcrest --computing-element test.xrsl

ARC-CE support token discovery as described in WLCG Bearer Token Discovery

Configured endpoints

Site Host VO Audience Group
praguelcg2, wlcg * /wlcg/pilots
praguelcg2, atlas, dune * *


It is possible to submit jobs with WLCG JWT token if HTCondor accepts SCITOKENS. By default HTCondor-CE is configured with non-standard aud claim, but this can be changed by adding following line in /etc/condor-ce/config.d/99-local file

# special "any" audience, not recommended for production use

All other configuration and identity mapping is described in the documentation. Currently identity mapping is limited to sub+iss which is not sufficient for IAM shared by multiple groups (e.g. Fermialab), but there are plans to implement callout similar to the GSI lcmaps.

Follow these steps to submit job with token to HTCondor-CE

cat > test.sub <<EOF
executable = /bin/env
output = env.out
error = env.err
log = env.log

# get access token, e.g. from oidc-agent (see bellow) and store its
# content in a file used by WLCG bearer token discovery mechanism
oidc-token --aud=condor:// -s compute.create -s -s compute.modify -s compute.cancel wlcg-compute > $XDG_RUNTIME_DIR/bt_u$(id -u)

export CONDOR_CONFIG=/dev/null
export GSI_AUTHZ_CONF=/dev/null
export _condor_AUTH_SSL_CLIENT_CADIR=/etc/grid-security/certificates

condor_ping -verbose -pool -name WRITE
condor_submit -pool -remote test.sub
condor_q -pool -name

Obtain access token with compute.* scopes

  • oidc-agent
# start oidc-agent (if it is not already running)
eval $(oidc-gen)
# register new client in oidc-agent with compute.* scopes (one time
# action, but not all IAM allows user to apply for compute.* scopes)
oidc-gen --iss= --scope="openid offline_access wlcg.groups compute.create compute.modify compute.cancel" wlcg-compute
# obtain token with all necessary scopes and right audience
oidc-token --aud=condor:// --scope "compute.create compute.modify compute.cancel" wlcg-compute > $XDG_RUNTIME_DIR/bt_u$(id -u)
  • client credentials grant (client registered in IAM with compute.* scopes, grant type "client credentials" and response type "token")
curl -s --data "grant_type=client_credentials&client_id=client_id_from_iam_registration&client_secret=client_secret_from_iam_registration&audience=condor://" | jq -r .access_token > $XDG_RUNTIME_DIR/bt_u$(id -u)

Configured endpoints

All OSG sites should now support job submission with token. You can test your HTCondor-CE configuration with ATLAS jobs submission.

Site Host VO Audience
osg wlcg
praguelcg2 wlcg, atlas, dune, condor://,



Configured endpoints:

Site Implementation Host VO Audience
praguelcg2 DPM 1.15.0 wlcg
UNL XRootD wlcg
DESY Devel dCache 7.1 wlcg
CNAF Prod StoRM wlcg
CNAF Devel StoRM wlcg
CERN Devel EOS wlcg
RAL Echo wlcg
Manchester Test DPM wlcg


CERN Devel FTS with 3.10.x provides JWT support for WLCG and XDC.

fts-rest-whoami --access-token=<token> -s
fts-rest-transfer-submit --access-token=<token> -s <src_url> <dst_url>
fts-rest-transfer-status --access-token=<token> -s

Be aware that for sucessfull FTS transfer submission with OIDC you also need recent 3.10.x FTS rest client.


Install Rucio client with one method described in documentation, e.g.

# latest rucio client works only with python3
virtualenv-3 rucio
source rucio/bin/activate
pip3 install rucio-clients
pip3 install gfal2-python

and save following configuration file in rucio/etc/rucio.cfg for WLCG DOMA Rucio OIDC tests

rucio_host =
auth_host =
auth_type = oidc
account = wlcg_doma
oidc_issuer = wlcg
ca_cert = /etc/grid-security/certificates
#ca_cert = /etc/pki/tls/certs/CERN-bundle.pem

Setup environment for Rucio client installed in virtualenv

source rucio/bin/activate

or if you use different installation method just set RUCIO_HOME environment variable to the base directory with etc/rucio.cfg file

export RUCIO_HOME=/base/path/to/rucio

WLCG IAM account is necessary to access WLCG DOMA Rucio instance and user sub claim (WLCG IAM uuid identity) must be associated with wlcg_doma Rucio account by DOMA Rucio ADMIN. It is also possible to associate user certificate subject with wlcg_doma Rucio account to provide access with WLCG VO X.509 certificate proxy, but for different authorization type it is necessary to update auth_type = x509_proxy in rucio.cfg or setting environment variable RUCIO_AUTH_TYPE=x509_proxy.