1. Securing gRPC services from unauthorized access#


Activity name	Securing gRPC services from unauthorized access
Activity ID	3
Short Description	Secure access to the SR Linux file system by preventing unauthorized 3^rd party gRPC clients from transferring corrupted files to the router
Difficulty	Beginner
Tools used	gNMIc, gNOIc, gNSIc (beta)
Topology Nodes	leaf11
References	SR Linux documentation gRPC website gNxI protobuf reference

A daily scheduled backup of the configuration file on leaf11 is taken by a remote file server using the gRPC gNOI File service with client1 as the username . A 3^rd party got access to this file server and they use the gNOI File service to push a compromised configuration file to the SR Linux device. Using other unspecified methods, they reboot the router resulting in the compromised configuration to become active and thereby opening up the entire network for the attacker. Your task is to stop such an attack by denying the client1 user from writing to the local filesystem of the router.

1.1 Objective#

On your groups hackathon VM, create a remote configuration file backup for leaf11 using client1 as the username and then use authorization policies on leaf11 to prevent client1 from pushing a corrupted file from the remote server to the router.

In this activity, we will:

Use gNOI to backup the configuration file for a SR Linux device.
Simulate how an attacker who got access to the VM can use gNOI to transfer a corrupted file to the router.
Use gNSI to deny access to gNOI requests for writing files on the router.
Repeat attacker simulation to verify if the authorization policy is working.

1.2 Technology explanation#

1.2.1 gRPC#

gRPC is an open source Remote Procedure Call (RPC) framework that implements gRPC services. To use a gRPC service, a gRPC client and a gRPC server are required. In this activity, the gRPC server is the router and the gRPC client is the server. The client sends a gRPC request to the server who will then process the request and respond back to the client.

gRPC (client) --------(Request)-----------> gRPC (server)
gRPC (client) <-------(Response)----------- gRPC (server)

Examples of gRPC servers include SR OS and SR Linux. Examples of gRPC clients include gNMIc, gNOIc and gNSIc.

1.2.2 gNOI#

gNOI (gRPC Network Operations Interface) is a gRPC service for executing operational commands on a network device. gNOI supports many services for different operations including, File Get, File Put, System Ping, System Traceroute and others. Refer to the gNxI reference to see the details of all the gNOI RPCs supported on SR Linux.

For this activity, our focus will be on gNOI File service. This service has the following RPCs:

Stat - list a file or directory on a remote router (similar to ls command on Linux)
Get - transfer a file from a remote router (similar to scp)
Put - transfer a file to a remote router (similar to scp)
Remove - delete a file on the remote router (similar to rm command on Linux)
Transfer - transfer a file on a remote router to a remote destination

1.2.3 gNSI#

gNSI (gRPC Network Security Interface) is a gRPC service for defining and retrieving security configuration on network devices. gNSI supports functions like TLS certificate management, gRPC service authorization and accounting to name a few. Refer to the gNxI reference to see the details of all the gNSI RPCs supported on SR Linux.

For this activity, our focus will be on gNSI Authz service. This service has the following RPCs:

Get - list the active gNSI Authz policy on the remote router
Probe - test if a particular gRPC call is permitted or denied for a specific user
Rotate - install and overwrite existing active Authz policy on the remote router. Only one policy can be installed which is always active.

1.3 Preparing the router#

1.3.1 gRPC configuration on SR Linux#

Before starting the tasks, let's verify the gRPC configuration on leaf11.

gRPC is enabled with TLS under gRPC server called mgmt and relevant gRPC services for this activity (gNMI, gNOI and gNSI) are also enabled under this gRPC server.

Verify the config of this grpc server.

cmdexpected output

info flat system grpc-server secure-mgmt

set / system grpc-server secure-mgmt admin-state enable
set / system grpc-server secure-mgmt rate-limit 65535
set / system grpc-server secure-mgmt session-limit 1024
set / system grpc-server secure-mgmt default-tls-profile true
set / system grpc-server secure-mgmt network-instance mgmt
set / system grpc-server secure-mgmt port 57401
set / system grpc-server secure-mgmt services [ gnmi gnoi gnsi p4rt ]

Based on the config, you will notice that the secure gRPC server is listening on port 57401.

1.3.2 User configuration on SR Linux#

Create 2 users described below on leaf11. These users will be used by gRPC clients for remote access to the router.

Username	Password	Role
client1	client1	gNOI service
grclient1	grclient1	gNMI service

cmdverify after commitexpected output

set / system aaa authorization role ext-clients services [ gnoi ]
set / system aaa authentication user client1 password client1
set / system aaa authentication user client1 role [ ext-clients ]
set / system aaa authorization role gnmi-clients services [ gnmi ]
set / system aaa authentication user grclient1 password grclient1 role [ gnmi-clients ]
set / system configuration role gnmi-clients rule / action write

info flat from running system aaa | grep client

set / system aaa authentication user client1 password $y$j9T$c0b094a538ddae13$GtCdwrCxDIrMhva6AtwPrXBR9YKFj4Gkr3RhaqRBstB
set / system aaa authentication user client1 role [ ext-clients ]
set / system aaa authentication user grclient1 password $y$j9T$720580ea832aa30d$AP.4Qi6e1kFXyU6TG82/v1xs99r4tCk/H8kBmvPenpB
set / system aaa authentication user grclient1 role [ gnmi-clients ]
set / system aaa authorization role ext-clients services [ gnoi ]
set / system aaa authorization role gnmi-clients services [ gnmi ]

1.3.3 Client Verification#

We will be using gNOIc and gNSIc as our clients for gNOI and gNSI services respectively.

Verify these clients are installed on your group's hackathon VM.

gNSIc

The gNSIc client is in beta status at time of writing this activity.

gnoicgnoic expected outputgnsicgnsic expected output

gnoic version

❯ gnoic version
version : 0.1.0
 commit : a5e9584
   date : 2024-12-18T19:04:06Z
 gitURL : https://github.com/karimra/gnoic
   docs : https://gnoic.kmrd.dev

gnsic version

❯ gnsic version
version : dev
 commit : none
   date : unknown
 gitURL :

It is also acceptable to have newer versions of these clients.

If clients are not installed or if you need to update the clients to the latest version, refer to the sources listed below.

1.3.4 Test gNOIc connectivity to SR Linux device#

Let's do a connectivity check for gNOI to ensure gRPC is running on leaf11. We will use the client1 user for all gNOI operations.

For this test, we will be using the gNOI System service Time RPC to get the current timestamp on leaf11.

cmdexpected output

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1 --skip-verify system time

+-----------------------------+------------------------------------------+---------------------+
|         Target Name         |                   Time                   |      Timestamp      |
+-----------------------------+------------------------------------------+---------------------+
| clab-srexperts-leaf11:57401 | 2025-04-25 22:32:58.749420494 +0300 EEST | 1745609578749420494 |
+-----------------------------+------------------------------------------+---------------------+

We received the current system timestamp from leaf11 and this confirms that gNOI is enabled on the router.

1.4 Tasks#

You should read these tasks from top-to-bottom before beginning the activity.

It is tempting to skip ahead but tasks may require you to have completed previous tasks before tackling them.

1.4.1 Use gNOIc to get the configuration file from SR Linux#

Our first task is to setup a remote configuration file backup service on the host VM.

Before we get the configuration file from leaf11, let's verify from the host using gNOI that the configuration exists on the router.

We will be using gNOI File Stat RPC for this purpose. The default configuration file for SR Linux is located at /etc/opt/srlinux/config.json.

cmdexpected output

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1 --skip-verify file stat --path /etc/opt/srlinux/config.json

+-----------------------------+------------------------------+---------------------------+------------+------------+--------+
|         Target Name         |             Path             |       LastModified        |    Perm    |   Umask    |  Size  |
+-----------------------------+------------------------------+---------------------------+------------+------------+--------+
| clab-srexperts-leaf11:57401 | /etc/opt/srlinux/config.json | 2025-04-24T19:16:13+03:00 | -rw-rw-r-- | -----w--w- | 121885 |
+-----------------------------+------------------------------+---------------------------+------------+------------+--------+

We confirmed that the configuration file exists and we also got information on the last modified timestamp and the size of the file.

Now let's retrieve this file to our host.

Create a directory called my-backups on your host to store the configuration backup file.

mkdir ~/my-backups

We will be using gNOI File Get RPC to retrieve the file from leaf11.

cmdexpected output

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1 --skip-verify file get --file /etc/opt/srlinux/config.json --dst ~/my-backups

INFO[0000] "clab-srexperts-leaf11:57401" received 64000 bytes 
INFO[0000] "clab-srexperts-leaf11:57401" received 57885 bytes 
INFO[0000] "clab-srexperts-leaf11:57401" file "/etc/opt/srlinux/config.json" saved

Let's check that the file get operation was successful and the file exists on your groups hackathon VM.

cmdexpected output

ls -lrt ~/my-backups/etc/opt/srlinux

❯ ls -lrt ~/my-backups/etc/opt/srlinux
total 120
-rw-r--r-- 1 nokia nokia 121885 Apr 25 23:00 config.json

Note

The destination file is created on the same path as the origin. In this case /etc/opt/srlinux/

You may open the configuration backup to verify the contents.

Note

This gnoic command can be added to a cron job on the host for taking regular configuration backups from your devices. This is outside the scope of this activity.

1.4.2 Simulate unauthorized gRPC access from host#

Let us now simulate how an unauthorized user who got access to the VM can use gNOI service to transfer a corrupt file or image to the router.

Create a file on your host that we will consider as a compromised file.

echo "This is a corrupted file" > ~/corrupt.img

We will use the gNOI File Put RPC to transfer this file to leaf11 under the directory /var/log/srlinux/.

cmdexpected output

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1  --skip-verify file put --file ~/corrupt.img --dst /var/log/srlinux/SRL-25-3.img

INFO[0000] "clab-srexperts-leaf11:57401" sending file="/home/nokia/corrupt.img" hash 
INFO[0000] "clab-srexperts-leaf11:57401" file "/home/nokia/corrupt.img" written successfully

Let's verify that the file is now present on leaf11. We will use the gNOI File Stat service we used earlier.

cmdexpected output

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1 --skip-verify file stat --path /var/log/srlinux/SRL-25-3.img

+-----------------------------+-------------------------------+---------------------------+------------+------------+------+
|         Target Name         |             Path              |       LastModified        |    Perm    |   Umask    | Size |
+-----------------------------+-------------------------------+---------------------------+------------+------------+------+
| clab-srexperts-leaf11:57401 | /var/log/srlinux/SRL-25-3.img | 2025-04-25T23:11:43+03:00 | -rwxrwxrwx | -----w--w- | 25   |
+-----------------------------+-------------------------------+---------------------------+------------+------------+------+

We confirmed that the file is present on leaf11.

Before we move on, let's delete that file on leaf11 using gNOI File Remove RPC.

cmd

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1 --skip-verify file remove --path /var/log/srlinux/SRL-25-3.img

1.4.3 Secure gRPC access using gNSI#

SR Linux has a default Authz policy that allows all gRPC operations. This can be verified from leaf11 CLI.

cmdexpected output

info flat from state /system aaa authorization authz-policy

/ system aaa authorization authz-policy version 2023-06-01
/ system aaa authorization authz-policy created-on "2025-04-28T13:52:58.067Z (4 minutes ago)"
/ system aaa authorization authz-policy policy "{
  \"name\": \"Default policy\",
  \"allow_rules\": [
    {
      \"name\": \"admin-access\",
      \"source\": {
        \"principals\": [
          \"*\"
        ]
      },
      \"request\": {
        \"paths\": [
          \"/*\",
          \"\"
        ]
      }
    }
  ]
}
"

This default policy is allowing all gNOI operations including the Put RPC that was used to transfer the compromised file to the router.

Only one gNSI Authz policy can be configured on the router and it is always active. The policy is displayed in JSON format with escape characters (\) for each ".

The following are the components of an Authz policy:

name - Authz policy name
allow_rules - List of users/roles allowed access to the list of gRPC RPCs
1. name - Name for allow rules
2. principals - list of usernames and roles applicable to this allow rule
3. paths - list of gRPC RPCs applicable to this allow rule
deny_rules - List of users/roles denied access to the list of gRPC RPCs
1. name - Name for deny rules
2. principals - list of usernames and roles applicable to this deny rule
3. paths - list of gRPC RPCs applicable to this deny rule

The paths in the policy are obtained from the gNOI proto file from the Openconfig git repo. For example, for gNOI File Get, the package name is gnoi.file, the service is File and the RPC is Get.

Refer to SR Linux Documentation for more details.

Your task is to overwrite this default Authz policy with another policy that will have the following rules:

Allow gNOI File Stat and gNOI File Get for client1 and role ext-clients
Deny gNOI File Put RPC for client1 and role ext-clients

Once your Authz policy is ready to be configured, use Authz Rotate RPC in the gNSIc command to push the policy to leaf11. Here's an example. Replace the json payload part with your Authz policy in JSON format (remember to use the escape character \ for the " character).

cmdexpected output

gnsic -a clab-srexperts-leaf11:57401 -u admin -p $EVENT_PASSWORD --skip-verify authz rotate --policy "json payload"

INFO[0000] targets: map[clab-srexperts-leaf11:57401:0xc00034c4e0] 
INFO[0000] "clab-srexperts-leaf11:57401": got UploadResponse 
INFO[0001] "clab-srexperts-leaf11:57401": sending finalize request 
INFO[0001] "clab-srexperts-leaf11:57401": closing stream

1.4.4 Repeat unauthorized access simulation#

After the Authz policy is installed on leaf11, try to transfer the same compromised file to leaf11.

WIth the configured Authz policy, the system will deny the request to write a file on the local filesystem.

cmdexpected output with Authz policy

gnoic -a clab-srexperts-leaf11:57401 -u client1 -p client1 --skip-verify file put --file ~/corrupt.img --dst /var/log/srlinux/SRL-25-3.img

INFO[0000] "clab-srexperts-leaf11:57401" sending file="corrupt.img" hash  
ERRO[0000] "clab-srexperts-leaf11:57401" File Put failed: rpc error: code = PermissionDenied desc = User 'client1' is not authorized to use rpc '/gnoi.file.File/Put' 
Error: there was 1 error(s)

Check the Authz policy counters to verify that the reject counter has incremented. Repeat the above command and check the counter.

cmdexpected output

info flat from state system aaa authorization authz-policy counters | grep Put

/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put access-rejects 1
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put last-access-reject "2025-04-29T02:16:31.582Z (3 minutes ago)"
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put access-accepts 1
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put last-access-accept "2025-04-29T01:57:08.730Z (22 minutes ago)"

1.5 Solution#

If you need help, refer to the Authz policy below.

gNSI Authz policy

{
  "name": "Ext-clients",
  "allow_rules": [
    {
      "name": "backup-access",
      "source": {
        "principals": [
          "client1","ext-clients"
        ]
      },
      "request": {
        "paths": [
          "/gnoi.file.File/Get",
          "/gnoi.file.File/Stat"
        ]
      }
    }
  ],
  "deny_rules": [
    {
      "name": "backup-access",
      "source": {
        "principals": [
          "client1","ext-clients"
        ]
      },
      "request": {
        "paths": [
          "/gnoi.file.File/Put"
        ]
      }
    }
  ]
}

Here's the command to push this policy to leaf11 using Authz Rotate RPC.

cmdexpected output

gnsic -a clab-srexperts-leaf11:57401 -u admin -p $EVENT_PASSWORD --skip-verify authz rotate --policy "{\"name\":\"Ext-clients\",\"allow_rules\":[{\"name\":\"backup-access\",\"source\":{\"principals\":[\"client1\",\"ext-clients\"]},\"request\":{\"paths\":[\"/gnoi.file.File/Get\",\"/gnoi.file.File/Stat\"]}}],\"deny_rules\":[{\"name\":\"backup-access\",\"source\":{\"principals\":[\"client1\",\"ext-clients\"]},\"request\":{\"paths\":[\"/gnoi.file.File/Put\"]}}]}"

INFO[0000] targets: map[clab-srexperts-leaf11:57401:0xc00034c4e0] 
INFO[0000] "clab-srexperts-leaf11:57401": got UploadResponse 
INFO[0001] "clab-srexperts-leaf11:57401": sending finalize request 
INFO[0001] "clab-srexperts-leaf11:57401": closing stream

Using an input file

Although not officially supported, you could try putting the Authz payload in a file with the following workaround:\n

gnsic -a clab-srexperts-leaf11:57401 -u admin -p $EVENT_PASSWORD --skip-verify authz rotate --policy "$(jq . authz.json)"

where authz.json is the file with the Authz payload.

Verify the configured Authz policy on leaf11 CLI.

cmdexpected output

info flat from state /system aaa authorization authz-policy

/ system aaa authorization authz-policy version ""
/ system aaa authorization authz-policy created-on "2417-02-18T14:54:24.465Z (392 years from now)"
/ system aaa authorization authz-policy policy "{\"name\":\"Ext-clients\",\"allow_rules\":[{\"name\":\"backup-access\",\"source\":{\"principals\":[\"client1\",\"ext-clients\"]},\"request\":{\"paths\":[\"/gnoi.file.File/Get\",\"/gnoi.file.File/Stat\"]}}],\"deny_rules\":[{\"name\":\"backup-access\",\"source\":{\"principals\":[\"client1\",\"ext-clients\"]},\"request\":{\"paths\":[\"/gnoi.file.File/Put\"]}}]}"
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Get access-rejects 0
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Get access-accepts 1
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Get last-access-accept "2025-04-29T01:56:45.028Z (6 minutes ago)"
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put access-rejects 0
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put access-accepts 1
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Put last-access-accept "2025-04-29T01:57:08.730Z (5 minutes ago)"
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Stat access-rejects 0
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Stat access-accepts 5
/ system aaa authorization authz-policy counters rpc /gnoi.file.File/Stat last-access-accept "2025-04-29T01:57:15.326Z (5 minutes ago)"
/ system aaa authorization authz-policy counters rpc /gnoi.system.System/Time access-rejects 0
/ system aaa authorization authz-policy counters rpc /gnoi.system.System/Time access-accepts 1
/ system aaa authorization authz-policy counters rpc /gnoi.system.System/Time last-access-accept "2025-04-29T01:55:42.746Z (7 minutes ago)"
/ system aaa authorization authz-policy counters rpc /gnsi.authz.v1.Authz/Rotate access-rejects 0
/ system aaa authorization authz-policy counters rpc /gnsi.authz.v1.Authz/Rotate access-accepts 3
/ system aaa authorization authz-policy counters rpc /gnsi.authz.v1.Authz/Rotate last-access-accept "2025-04-29T02:03:00.794Z (4 seconds ago)"

1.6 Delete Authz policy#

Login to leaf11 and remove the configured Authz policy.

cmdexpected output

tools system aaa authorization authz-policy remove

/system/aaa:
    Authz authorization policy has been removed

1.7 Additional Task#

Streaming Telemetry is being sent from leaf11 with grclient1 username to an open source stats collector. An unauthorized 3^rd party gets access to the CLI of this stats collector and uses gNMI Set RPC to disable BGP on leaf11. Your task is to secure leaf11 from such an attack using gNSI Authz policies.

1.7.1 gNMI commands#

The gnmic client is installed on your host VM. This can be verified using:

cmdexpected output

gnmic version

version : 0.40.0
 commit : 3f13e44d
   date : 2025-01-27T22:03:58Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net

If gnmic client is not installed, refer to gNMIc page for installation instructions.

Here are the commands to test Get, Set and Subscribe.

gNMI GetgNMI SetgNMI Subscribe

gnmic -a clab-srexperts-leaf11:57401 -u grclient1 -p grclient1 --skip-verify get --type state --path "/interface[name=ethernet-1/1]/statistics/in-octets/" -e json_ietf

gnmic -a clab-srexperts-leaf11:57401 -u grclient1 -p grclient1 --skip-verify set --update-path "/interface[name=ethernet-1/1]/description" --update-value "gnmi-test"

gnmic -a clab-srexperts-leaf11:57401 -u grclient1 -p grclient1 --skip-verify sub --path "/interface[name=ethernet-1/1]/statistics/in-octets/" --mode once

Test the above commands and verify the outputs.

Then apply an Authz policy to allow Get and Subscribe RPCs while denying Set RPC for user grclient1 and role gnmi-clients.

Both the username and the role name should be added to the principals section.

Refer to the gNMI proto file here to get the RPC name to include in the policy. The format is package_name.servce_name/RPC.

1.7.2 Solution#

If you need help, refer to the solution below.

Solution for gNMI Authz policy

{
  "name": "srex-gnmi",
  "allow_rules": [
    {
      "name": "gnmi-access",
      "source": {
        "principals": [
          "grclient1","gnmi-clients"
        ]
      },
      "request": {
        "paths": [
          "/gnmi.gNMI/Get",
          "/gnmi.gNMI/Subscribe"
        ]
      }
    }
  ],
  "deny_rules": [
    {
      "name": "gnmi-access",
      "source": {
        "principals": [
          "grclient1","gnmi-clients"
        ]
      },
      "request": {
        "paths": [
          "/gnmi.gNMI/Set"
        ]
      }
    }
  ]
}

gNSI request for gNMI Authz policy

gnsic -a clab-srexperts-leaf11:57401 -u admin -p $EVENT_PASSWORD --skip-verify authz rotate --policy "{\"name\":\"gnmi-access\",\"allow_rules\":[{\"name\":\"gnmi-access\",\"source\":{\"principals\":[\"grclient1\",\"gnmi-clients\"]},\"request\":{\"paths\":[\"/gnmi.gNMI/Get\",\"/gnmi.gNMI/Subscribe\"]}}],\"deny_rules\":[{\"name\":\"gnmi-access\",\"source\":{\"principals\":[\"grclient1\",\"gnmi-clients\"]},\"request\":{\"paths\":[\"/gnmi.gNMI/Set\"]}}]}"

1.8 Summary#

Congratulations! By this point you should have successfully completed the activity and restricted the rogue actor from inflitrating your network. With the work you have put in, you should now have a basic understanding of the following topics:

Using gRPC to perform activities on a remote router
Using the gNOI service to manipulate the file system
Using the gNSI service to secure network services on a router

Good job in completing all the tasks in this activity!

Feedback


Do you feel you have achieved something?	Yes No
Was the difficulty level graded appropriately?	Yes No
How do you rate this activity?