invitations: a bad INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS value creates the Keycloak guest and then returns 500 #13253

[OCISDEV-1593]

I was doing confirmatory test of PR #12467 against the new release v9.0.0-rc.1. Sorry in advance for the long issue.

Short version: The main changes work, but a few statements in the PR description do not match the code, and a failed invitation leaves a guest account in Keycloak that cannot be invited again.

Versions

ocis_full cannot be used for this test as shipped, see Problem 4, so I used a small lab with only the three containers that matter. OR if my below setup seems to be missing something then please provide me the details. 😄

The lab

lab/
  compose.yml
  keycloak/ocis-realm.json

The realm file is copied unchanged. The name inside the container must be oCIS-realm.json (capital C), which is why the mount target below is spelled that way. The host file name does not matter.

mkdir -p lab/keycloak
cp deployments/examples/ocis_full/config/keycloak/ocis-realm.dist.json \
   lab/keycloak/ocis-realm.json
sha256sum lab/keycloak/ocis-realm.json
# 4e8b15473ac72002240441f280048d17f913db584d196cb0d3c65485a162809f

lab/compose.yml:

services:
  ocis:
    image: owncloud/ocis:9.0.0-rc.1
    entrypoint: /bin/sh
    command: ['-c', 'ocis init || true && ocis server']
    environment:
      OCIS_LOG_LEVEL: 'error'
      OCIS_LOG_PRETTY: true
      OCIS_LOG_COLOR: true
      OCIS_INSECURE: true
      IDM_ADMIN_PASSWORD: 'admin'
      PROXY_ENABLE_BASIC_AUTH: true
      OCIS_URL: 'https://localhost:9200'
      OCIS_ADD_RUN_SERVICES: 'notifications,invitations'
      NOTIFICATIONS_SMTP_HOST: mailpit
      NOTIFICATIONS_SMTP_PORT: 1025
      NOTIFICATIONS_SMTP_INSECURE: true
      NOTIFICATIONS_SMTP_SENDER: 'owncloud <noreply@example.com>'
      # invitations -> Keycloak
      INVITATIONS_KEYCLOAK_BASE_PATH: 'http://keycloak:8080'
      INVITATIONS_KEYCLOAK_CLIENT_ID: 'invitations'
      INVITATIONS_KEYCLOAK_CLIENT_SECRET: 'invitations-secret'
      INVITATIONS_KEYCLOAK_CLIENT_REALM: 'oCIS'
      INVITATIONS_KEYCLOAK_USER_REALM: 'oCIS'
      INVITATIONS_KEYCLOAK_INSECURE_SKIP_VERIFY: 'true'
      # empty means "use UPDATE_PASSWORD,VERIFY_EMAIL"
      INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS: ${INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS:-}
    ports:
      - 9200:9200
      - 5200:5200
    depends_on:
      - keycloak
      - mailpit

  mailpit:
    image: axllent/mailpit:v1.22.3
    networks:
      default:
        # the realm in this PR ships smtpServer.host = "mailserver"
        aliases:
          - mailserver
    ports:
      - 8025:8025

  keycloak:
    image: quay.io/keycloak/keycloak:26.2.5
    command: ['start-dev', '--import-realm']
    environment:
      KC_BOOTSTRAP_ADMIN_USERNAME: admin
      KC_BOOTSTRAP_ADMIN_PASSWORD: admin
      KC_HTTP_ENABLED: 'true'
      # links in the execute-actions mail have to point at the published port
      KC_HOSTNAME: 'http://localhost:18080'
      KC_HOSTNAME_STRICT: 'false'
    ports:
      - 18080:8080
    volumes:
      - ./keycloak/ocis-realm.json:/opt/keycloak/data/import/oCIS-realm.json:ro

cd lab
docker compose up -d

Wait for Realm 'oCIS' imported in docker compose logs keycloak before continuing.

Create the invitations client

The realm in the PR has no invitations client, so the service cannot talk to Keycloak without this:

docker compose exec -T keycloak /opt/keycloak/bin/kcadm.sh config credentials \
  --server http://localhost:8080 --realm master --user admin --password admin

docker compose exec -T keycloak /opt/keycloak/bin/kcadm.sh create clients -r oCIS \
  -s clientId=invitations \
  -s enabled=true \
  -s publicClient=false \
  -s clientAuthenticatorType=client-secret \
  -s secret=invitations-secret \
  -s serviceAccountsEnabled=true \
  -s standardFlowEnabled=false \
  -s directAccessGrantsEnabled=false

for role in manage-users view-users query-users view-realm; do
  docker compose exec -T keycloak /opt/keycloak/bin/kcadm.sh add-roles -r oCIS \
    --uusername service-account-invitations \
    --cclientid realm-management \
    --rolename "$role"
done

Check the credentials:

curl -s -d client_id=invitations -d client_secret=invitations-secret \
  -d grant_type=client_credentials \
  http://localhost:18080/realms/oCIS/protocol/openid-connect/token \
  | jq -r '.token_type // .error'

Expected output is Bearer.

The tests I ran

Keycloak on its own, as in the PR description

KC=http://localhost:18080
TOK=$(curl -s -d client_id=admin-cli -d username=admin -d password=admin \
  -d grant_type=password \
  $KC/realms/master/protocol/openid-connect/token | jq -r .access_token)

EMAIL=e2e-$(date +%s)@example.org

curl -s -o /dev/null -w 'create user: %{http_code}\n' \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -X POST $KC/admin/realms/oCIS/users \
  -d "{\"username\":\"$EMAIL\",\"email\":\"$EMAIL\",\"enabled\":true}"

KUID=$(curl -s -H "Authorization: Bearer $TOK" \
  --get --data-urlencode "email=$EMAIL" --data-urlencode exact=true \
  $KC/admin/realms/oCIS/users | jq -r '.[0].id')

curl -s -o /dev/null -w 'execute-actions-email: %{http_code}\n' \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -X PUT $KC/admin/realms/oCIS/users/$KUID/execute-actions-email \
  -d '["UPDATE_PASSWORD","VERIFY_EMAIL"]'

create user: 201
execute-actions-email: 204

mailpit shows a mail from keycloak@owncloud.test with subject Update Your Account, and the link opens a Keycloak page listing "Update Password, Verify Email". The imported realm contains the PR values:

{
  "starttls": "false",
  "port": "1025",
  "auth": "false",
  "host": "mailserver",
  "from": "keycloak@owncloud.test",
  "fromDisplayName": "oCIS",
  "ssl": "false"
}

Through oCIS

EMAIL=guest-$(date +%s)@example.org

curl -sk -u admin:admin -X POST https://localhost:9200/graph/v1.0/invitations \
  -H 'Content-Type: application/json' \
  -d "{\"invitedUserDisplayName\":\"CF Guest\",\"invitedUserEmailAddress\":\"$EMAIL\",\"inviteRedirectUrl\":\"https://localhost:9200\"}" | jq

One of the responses, with the long token link removed:

{
  "invitedUserDisplayName": "CF Guest",
  "invitedUserEmailAddress": "guest-1793099880@example.org",
  "inviteRedirectUrl": "https://localhost:9200",
  "invitedUser": {
    "accountEnabled": true,
    "displayName": "",
    "id": "407c2a79-7240-4c80-9e2a-4cb89ac25f55",
    "mail": "guest-1793099880@example.org",
    "onPremisesSamAccountName": "guest-1793099880@example.org",
    "userType": "Guest"
  }
}

And in Keycloak:

TOK=$(curl -s -d client_id=admin-cli -d username=admin -d password=admin \
  -d grant_type=password \
  $KC/realms/master/protocol/openid-connect/token | jq -r .access_token)

curl -s -H "Authorization: Bearer $TOK" \
  --get --data-urlencode "email=$EMAIL" --data-urlencode exact=true \
  $KC/admin/realms/oCIS/users | jq '.[0] | {username, requiredActions, attributes}'

{
  "username": "guest-1793099880@example.org",
  "requiredActions": ["UPDATE_PASSWORD", "VERIFY_EMAIL"],
  "attributes": {
    "OWNCLOUD_ID": ["407c2a79-7240-4c80-9e2a-4cb89ac25f55"],
    "OWNCLOUD_USER_TYPE": ["Guest"]
  }
}

invitedUser.id matches OWNCLOUD_ID, and the mail arrives in mailpit.

What works

• INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS is honoured: not set gives UPDATE_PASSWORD,VERIFY_EMAIL, UPDATE_PASSWORD gives only that action, UPDATE_PASSWORD, VERIFY_EMAIL works with the space, and " , " falls back to the defaults.
• Only ocis_full has the new smtpServer block (ocis_keycloak and ocis_multi still have {}), and Keycloak has no KC_SMTP_* option, so putting SMTP in the realm was the right call.
• SMTP is connected lazily. Keycloak started fine with mailpit stopped; sending returned 500; after mailpit came back the next send returned 204 without a Keycloak restart.

Problem 1: an unknown action creates the guest, fails the request, sends no mail, and blocks retries

The PR description says:

Unknown actions are logged and skipped; if none remain valid the service falls back to the defaults.

That is not what happens. parseUserActions (services/invitations/pkg/backends/keycloak/backend.go:120) splits and trims, then passes the value to Keycloak. There is no validation, no logging and no fallback.

INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS=FOO docker compose up -d ocis

# the proxy answers 502 until the invitations service registers itself
until [ "$(curl -sk -o /dev/null -w '%{http_code}' -u admin:admin -X POST \
  https://localhost:9200/graph/v1.0/invitations \
  -H 'Content-Type: application/json' -d '{}')" != 502 ]; do
  sleep 2
done

EMAIL=fooguest2-$(date +%s)@example.org

curl -sk -o /tmp/invite-foo.out -w 'invitation HTTP %{http_code}\n' -u admin:admin \
  -X POST https://localhost:9200/graph/v1.0/invitations \
  -H 'Content-Type: application/json' \
  -d "{\"invitedUserEmailAddress\":\"$EMAIL\"}"

echo "response body: $(cat /tmp/invite-foo.out)"

invitation HTTP 500
response body: backend error: 400 Bad Request: Provided invalid required actions

The user was still created:

TOK=$(curl -s -d client_id=admin-cli -d username=admin -d password=admin \
  -d grant_type=password \
  $KC/realms/master/protocol/openid-connect/token | jq -r .access_token)

curl -s -H "Authorization: Bearer $TOK" \
  --get --data-urlencode "email=$EMAIL" --data-urlencode exact=true \
  $KC/admin/realms/oCIS/users | jq '.[0] | {username, requiredActions, attributes}'

{
  "username": "fooguest2-1793102061@example.org",
  "requiredActions": [],
  "attributes": {
    "OWNCLOUD_ID": ["fdb04bc1-6bbf-4dd3-bd73-72ec08e716a2"],
    "OWNCLOUD_USER_TYPE": ["Guest"]
  }
}

No mail was sent:

curl -s "http://localhost:8025/api/v1/messages?limit=200" \
  | jq --arg e "$EMAIL" '[.messages[] | select(any(.To[]?; .Address==$e))] | length'
# 0

So CreateUser (backend.go:73) succeeds, SendMail (backend.go:110) fails, the error goes back to the caller, and nothing is rolled back.

The retry is the bad part. After restoring the default config, the same address cannot be invited again:

docker compose up -d ocis
# wait for the route again, then:

curl -sk -u admin:admin -X POST https://localhost:9200/graph/v1.0/invitations \
  -H 'Content-Type: application/json' \
  -d "{\"invitedUserEmailAddress\":\"$EMAIL\"}"

HTTP 500
backend error: 409 Conflict: User exists with same email

The user has to be deleted in Keycloak before the invite can be sent again.

Expected, based on the description: FOO is skipped, the defaults are used, the mail is sent, and the call returns 201.

Problem 2: the promised unit tests for string parsing do not exist

The description says:

New unit tests cover the default actions, a configured subset, and the string parsing (known / unknown / case-sensitive).

Default and subset are real (backend_test.go:45 and :74), but they pass []keycloak.UserAction straight into the mock, so the string path is never executed:

grep -rn 'parseUserActions\|ExecuteActions' --include='*_test.go' \
  services/invitations ocis-pkg/keycloak
# no output

There is no unknown or case-sensitive test either. The unknown action bug above is exactly what such a test would catch.

Problem 3: "Keycloak silently drops every mail" is not what happens

With smtpServer set to {}, Keycloak returns an explicit error:

execute-actions-email with empty smtpServer: HTTP 500
body: {"errorMessage":"Failed to send execute actions email: No sender address configured in the realm settings for emails"}

Through oCIS it is also a clear 500, not a lost mail:

invitation via oCIS: 500
body: backend error: 500 Internal Server Error: Failed to send execute actions email: No sender address configured in the realm settings for emails

With mailpit stopped the error is Error when attempting to send the email to the server. More information is available in the server log. I could not find a case where the mail is dropped silently, so I think the wording in the description should change.

Problem 4: ocis_full still cannot invite guests, so the SMTP fix is dormant there

The example does not start the invitations service and has no credentials for it:

grep -n 'START_ADDITIONAL_SERVICES' deployments/examples/ocis_full/.env
# 135:START_ADDITIONAL_SERVICES="notifications"

grep -n 'OCIS_ADD_RUN_SERVICES' deployments/examples/ocis_full/ocis.yml
# 22:      OCIS_ADD_RUN_SERVICES: ${START_ADDITIONAL_SERVICES}

grep -rn 'INVITATIONS_' deployments/
# no output

The realm also has no invitations client (the client list starts at ocis-realm.dist.json:828). As shipped, /graph/v1.0/invitations returns 502 because the proxy has no backend. I had to add the run service, the env vars and the client before anything worked. Either the example should be wired up, or the PR text should say that invitations still need manual setup.

Problem 5: CONFIGURE_RECOVERY_AUTHN_CODES is not a valid required action on Keycloak 26.2.5

I sent each new constant through execute-actions-email. VERIFY_PROFILE, webauthn-register, webauthn-register-passwordless, UPDATE_PROFILE, CONFIGURE_TOTP, TERMS_AND_CONDITIONS, delete_account and update_user_locale return 204. CONFIGURE_RECOVERY_AUTHN_CODES returns 400 Provided invalid required actions, so using it in INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS produces the orphaned user from Problem 1.

Possible fixes

I am not sure which direction you would prefer, so these are options rather than a plan:

1. Validate INVITATIONS_KEYCLOAK_EXECUTE_ACTIONS before CreateUser: either skip unknown values with a log line, which matches the description, or reject the request with 400 before the guest is created. Skipping is more forgiving, failing fast is easier to debug.
2. Delete the Keycloak user when the mail step fails, so a retry does not hit 409.
3. Add the parsing tests the description claims, including an unknown value and a case-sensitive one.
4. Wire invitations into ocis_full, or document the env vars and client it needs.

Two open questions:

• Is skipping unknown actions what you want, or would a hard 400 be better?
• Does CONFIGURE_RECOVERY_AUTHN_CODES need to stay for older Keycloak versions?

References

• services/invitations/pkg/backends/keycloak/backend.go:120 (parseUserActions), :73 (CreateUser), :110 (SendMail)
• services/invitations/pkg/backends/keycloak/backend_test.go:45 and :74
• deployments/examples/ocis_full/config/keycloak/ocis-realm.dist.json:1967 (smtpServer)
• deployments/examples/ocis_full/.env:135, deployments/examples/ocis_full/ocis.yml:22

Teardown

cd lab
docker compose down