Can you post the self signed certificate here so I can see the details?
I created a self signed certificate using openssl that is not CA capable and loaded it into a FIP and it works fine. Here's what I did:
1. Create the key pair (see attached):
$ openssl req -x509 -newkey rsa:2048 -keyout test1.key.pem -out test1.cer.pem -days 365
...
$ openssl x509 -in test1.cer.pem -text -noout
Certificate:
Data:
Version: 1 (0x0)
Serial Number:
5f:17:b7:18:14:bf:ae:85:1c:b5:7c:29:72:05:9a:02:fb:19:f3:db
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = CA, ST = British Columbia, L = North Vancouver, O = Tennyson Hollow Studios, CN = test1
Validity
Not Before: Feb 25 19:29:46 2021 GMT
Not After : Feb 25 19:29:46 2022 GMT
Subject: C = CA, ST = British Columbia, L = North Vancouver, O = Tennyson Hollow Studios, CN = test1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:cf:5b:e0:3d:07:ee:02:ff:6a:61:5c:51:99:14:
cd:b1:8b:4d:76:9a:9d:d6:d2:02:dc:a1:6c:8c:c7:
da:40:1d:12:88:96:fe:19:bb:1b:7f:4c:c9:ae:b8:
72:15:e4:71:81:6e:ff:75:7b:45:46:2e:9d:3e:e6:
91:b7:7b:71:f5:54:17:f3:2f:57:8d:c5:c6:75:dd:
73:17:ad:56:d3:f7:21:bd:aa:ab:01:61:01:f4:3d:
aa:4a:4b:0c:c5:29:c8:26:97:f6:30:1c:5d:7b:28:
d0:b1:f0:4c:1c:fe:a3:61:f6:c4:03:ef:e7:e1:ae:
f0:d4:cf:b2:64:e5:95:df:44:d1:56:e9:e6:33:0f:
62:06:36:31:03:a6:ce:c4:f4:89:69:53:e1:46:7e:
57:db:94:74:d1:d7:5d:ad:c1:ae:03:1e:98:eb:c7:
98:c1:eb:59:d8:9e:1f:da:c3:6a:e0:01:89:ed:e4:
a9:16:c0:5f:1b:17:32:82:5b:47:41:0c:29:a9:0f:
78:75:29:c6:75:ca:39:d2:d0:bb:0b:55:ec:91:bd:
4d:cb:90:07:4e:39:51:96:94:98:8d:85:05:66:d0:
2c:80:30:73:de:6d:34:fd:f4:17:6a:66:38:21:1c:
01:31:4b:59:ba:64:58:1e:1e:c4:6c:9d:9d:34:b6:
7e:4d
Exponent: 65537 (0x10001)
Signature Algorithm: sha256WithRSAEncryption
50:b2:cb:e6:7a:70:1b:84:3e:5f:9e:e8:29:b9:c4:b4:e2:2b:
0f:80:70:49:ad:1e:e3:0d:fd:78:bb:d4:8f:e7:4c:99:87:8b:
a3:23:be:45:1d:b0:d1:d1:c1:40:65:13:e2:a5:ca:f9:f1:be:
c8:82:78:2c:6d:c6:61:d8:76:67:8c:62:63:0c:8a:09:6c:c0:
30:71:15:bb:da:da:e8:2d:2e:6a:3f:7d:fd:4c:a9:4d:0f:31:
74:5e:75:c3:36:01:7e:1f:90:2c:36:c2:86:9b:fc:87:4e:f3:
53:7d:1d:8c:a3:de:01:b4:df:f6:2e:a8:fc:c1:c0:0f:9a:2f:
d1:ef:a6:b1:54:b9:19:68:ef:4c:32:61:73:15:23:d6:27:dd:
d3:90:f0:4d:46:fe:8c:fb:4f:c1:ea:42:f7:2a:6c:43:09:f0:
20:ac:4a:bf:dd:8b:9c:1a:49:6c:e4:a2:fd:5f:3a:95:c1:de:
4f:c1:f4:4e:f4:ed:80:0a:2c:c2:cc:a2:b0:12:25:ea:11:9d:
c8:23:b6:87:02:10:4a:35:a3:97:19:86:c0:43:ba:9d:8a:62:
a9:ba:74:ad:bd:cd:b1:d1:00:32:62:1d:77:17:2d:ae:7c:11:
a7:8b:c2:33:92:43:e4:27:96:2c:aa:5d:97:a8:20:ec:a3:86:
78:da:ce:fc
$
2. Create a FIP with no signing certificates
3. Create a user in the FIP and load the test1.cer.pem certificate into that user
4. Publish a Web API service with "Require SSL or TLS with Client Certificate Authentication", "Authenticate Against FIP" and a template response (see attached)
5. Hit the service with curl
$ curl -v --cacert ssg100.l7tech.com.pem --cert test1.pem:7layer https://ssg100.l7tech.com:8443/test
* Trying 10.7.50.100:8443...
* TCP_NODELAY set
* Connected to ssg100.l7tech.com (10.7.50.100) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: ssg100.l7tech.com.pem
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=ssg100.l7tech.com
* start date: Sep 4 16:17:34 2020 GMT
* expire date: Sep 2 16:17:34 2030 GMT
* common name: ssg100.l7tech.com (matched)
* issuer: CN=ssg100.l7tech.com
* SSL certificate verify ok.
> GET /test HTTP/1.1
> Host: ssg100.l7tech.com:8443
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 32
< Date: Mon, 01 Mar 2021 15:46:32 GMT
< Server: Layer7-API-Gateway
<
request.authenticateduser=test1
* Connection #0 to host ssg100.l7tech.com left intact
jay@jaymac-laptop:~$
Note that for this test I did not add the certificate to the trust store and declare it as a trust anchor and for signing certificates, although this may be necessary for some clients in order to see an advertised list of authorities. After running this I did add it to the trust store as an anchor and for signing client certificates and recreated the user with the same results.