Querying Network and Storage Configuration and Capabilities of the DTN

DTN Agent

Sajith Sasidharan

Introduction

DTN Agents “manage” data transfer nodes (DTNs); each node runs its own unique instance of DTN Agent. This is the component of Big Data Express that is responsible for:

•Querying network and storage configuration and capabilities of the DTN.

•Registering itself with the database, thus advertising the availability, configuration, and capabilities of the DTN to BDE Server.

•Launching, monitoring, and re-starting the data transfer application (such as MdtmFTP or GridFTP) as necessary.

•Initializing Storage Agents, based on configuration.

•Acting upon, and responding to, commands from BDE Server.

•Setting up the data transfer node with job-specific configurations, such as adding virtual circuits or routes, and verifying that the configuration works.

•Monitoring and rate control data transfer.

•Tearing down job-specific configuration once the job is complete.

In the following sections, we discuss the design and implementation of DTN Agent.

General Execution Pattern

DTN Agent instances are created by BDE Agent daemons running in the DTN. Upon starting up, DTN Agent will do the following:

1. Validate its own configuration block.
2. Next is the bootstrap step: in this step, DTN Agent will generate a unique name and ID to identify itself. Name is based on hostname of the DTN, and ID is MAC address of one of the active NIC devices present on the system.
3. In the initialization step, DTN Agent will enumerate network interfaces present in the node, and verify that they correspond to configuration.
4. Depending on configuration, DTN Agent will bring up one or more Storage Agents.
5. In the registration step, DTN Agent will report its status to BDE Server by adding an entry the database.
6. Finally, it enters an event handling loop:
7. Listen on the message queue for incoming commands from BDE Server
8. Act on these commands
9. Send responses with results to BDE Server

In addition to the above, DTN Agent will periodically refresh its database entry. BDE Server will use this information for monitoring and scheduling tasks.

Configuration

DTN Agent requires some configuration to be able to operate correctly. BDE Agent provides this configuration in a JSON-encoded buffer:

{
"modules":{
"m1":{
"type":"DTN",
"data_interfaces":["enp4s0f0"],
"management_interface":"enp4s0f0",
"storage_interfaces":"ib0",
"data_folders":{
"path":["/data1"],
"scan":true
},
"local_storage_agent":{
"type":"LocalStoageAgent",
"iozone-executable":"/usr/local/bin/iozone"
}
}
}
}

When data_folders configuration item is present in the configuration, DTN Agent will initialize Local Storage Agents. While doing so, DTN Agent will find the storage devices associated with the data folders, and then construct a JSON object with a generated ID, name, and type fields, which will in turn be used to initialize the Local Storage Agent.

{
"name":"Local Storage Agent on host.fnal.gov, device /dev/sda1",
"id":"00:11:22:33:44:55 |/dev/sda1",
"type":"LocalStorageAgent",
"device":"/dev/sda1",
"root-folders":["/path/one","/path/two"],
"iozone-executable":"/usr/local/bin/iozone",
"iozone-work-folder":"/path/to/work-folder"
}

Registration

During the registration step, DTN Agent will add two entries in the database:

1. A description of itself, named dtn.
2. Storage-to-DTN map, named sdmap.

Both entries are refreshed periodically. They both carry an expiry timestamp, which indicates their current-nessto BDE Server.

The dtn entry will describe the DTN Agents name, ID, message queue where it is listening, and control, data and storage interfaces of the DTN:

{
"id":"0c:c4:7a:ab:63:7e",
"name":"host.fnal.gov",
"expire_at":1493406516524,
"queue_name":"amq.gen-dsJJaHl39eOrsAuylX0X0w",
"ctrl_interface":
{"name":"enp11s0f0",
"ip":"107.0.1.10",
"rate":1000000
},
"data_interfaces":
[{"name":"enp11s0f1",
"ip":"107.0.1.11",
"rate":40000000
}],
"storage_interfaces":
[{"name":"ib0",
"rate":40000000,
"type":"local"}],
"data_folder":["/data1"]
}

The sdmap entries describes storage agents present in the DTN:

{
"dtn":"00:25:90:fd:1b:24",
"expire_at":1493406516524,
"storage":"00:25:90:fd:1b:24|/dev/sda2",
"storage_interface":"local"
}

Commands and responses

As described in [sec-2], DTN Agent listens on a RabbitMQ message queue for incoming commands from BDE Server, acts upon those commands, and reports back with the results. Both commands and responses are JSON-encoded.

Preparing for a task

“setup” command

Setup command instructs DTN Agent to prepare the DTN for a data transfer task. This preparation includes setting up the appropriate virtual circuits, and maintaining an internal “task” structure that may later be used for further actions.

{
"cmd":"dtn_setup",
"params":
{
"task":"test-task-bde1",
"direction":"in-bound",
"target_site":"fnal.gov",
"target_dtn":"bde1.fnal.gov",
"rate":1000,
"interface":"enp4s0f0",
"vlan":5,
"ip":"192.5.5.1"
}
}

In general, responses to setup command will be of the below format:

{
"code":0,
"error":"OK"
}

In the event of errors, code will be non-zero, and a descriptive error message will be provided in errorfield.

“teardown” command

The teardown command signals that a task has ended, and it may proceed to remove all configurations (such as virtual network devices and their configuration) related to that task.

{
"cmd":"dtn_teardown",
"params":
{
"task":"test-task-bde1"
}
}

Responses to teardown command will be similar to those sent in response to setup command.

“add route” command

The add route command is meant to use in situations where BDE may not need to set up a dedicated virtual circuit to initiate a data transfer task; simply adding a route between DTNs involved could be sufficient.

{
"cmd":"dtn_add_route",
"params":
{
"dest":"192.5.5.1",
"device":"enp4s0f0",
"via":"192.5.5.254"
}
}

Here device parameter is optional; responses are similar to “setup” responses.

“delete route” command

The delete route command will have the opposite effect of add route command, and follows the same request/response format and parameters.

{
"cmd":"dtn_del_route",
"params":
{
"dest":"192.5.5.1",
"device":"enp4s0f0",
"via":"192.5.5.254"
}
}

Verifying connectiviy

Once a virtual circuit or route is set up between two DTNs, BDE will need to verify the connectivity between them. Commands to send and receive our custom version of “ping” are used to accomplish this.

Instead of the standard ping utility that uses ICMP packets (which may be blocked in some network paths), we attempt to create TCP connections between the DTNs involved in a data transfer task. We have three separate commands for this:

“send ping” command

The send ping command, which will instruct a DTN which will act as the source system: on this command, DTN Agent will attempt to transmit a known string to the specified target DTN.

{
"cmd":"dtn_send_ping",
"params":
{
"source":
{
"ip":"192.5.5.1"
},
"target":
{
"ip":"192.5.5.3",
"port":5555
}
}
}

“wait for ping” command

The wait for ping command, which will instruct target DTN to open a TCP port and send appropriate responses to pings it has received:

{
"cmd":"dtn_start_pong",
"params":
{
"ip":"192.5.5.1",
"port":5555
}
}

“stop waiting for ping” command

The stop waiting for ping command, which will ask the target system to close the port:

{
"cmd":"dtn_stop_pong"
}

By design, A DTN Agent can run only one ping responder can run at a time; responses follow the same form as “setup” responses.

Checking the DTN’s status

“get overall status” command

This command is used to query the DTN Agent about the status of network interfaces:

{
"cmd":"dtn_status",
"params":{
"interface":"enp4s0f0"
}
}

The interface parameter is optional. If an interface is specified in the parameters, response will return transmit/receive statistics for that interface from the last sampling period. If no interface was provided, statistics for all network interfaces will be returned in the response in a JSON array.

Response to get overall status command will be of the following form:

{
"code":0,
"status":{
"interface":"enp4s0f0",
"rate":{
"rx-bytes":606,
"rx-dropped":0,
"rx-errors":0,
"rx-packets":6,
"tx-bytes":102,
"tx-dropped":0,
"tx-errors":0,
"tx-packets":0
}
}
}

“get task status” command

The get task status command is used to query the status of an ongoing task.

{
"cmd":"dtn_task_status",
"params":{
"task":"test-task-bde1"
}
}

Responses will be of the following form:

{
"code":0,
"error":"OK",
"status":{
"direction":"in-bound",
"interface":"enp4s0f0",
"ip":"192.5.5.1",
"rate":1000,
"state":"set up",
"target_dtn":"host.fnal.gov",
"target_site":"fnal.gov",
"task":"test-task-bde1",
"virtual_interface":"enp4s0f0.5",
"vlan":5
}
}

Resetting the agent

“reset all” command

This command is used to bring the DTN Agent to a “fresh” state; it will clear all ongoing tasks and their associated configurations.

{
"cmd":"dtn_reset_all"
}

Responses are similar to setup responses.

Listing and grouping files

“expand a path” command

This command is used by BDE to list all files under a folder in preparation for a data transfer job.

{
"cmd":"dtn_expand",
"params":{
"files":["/data/10G/"]
}
}

The command can have an optional user field; if this field is present, DTN Agent will check that the user has permission to access the entire directory tree under files.

In the response, DTN Agent will send a recursively expanded list of files under the specified path; response will be of the following format:

{
"code":0,
"error":"OK",
"files":[
{
"name":"/data1/10G/f2",
"size":10737418240
},
{
"name":"/data1/10G/f3",
"size":10737418240
},
{
"name":"/data1/10G/f5",
"size":10737418240
},
{
"name":"/data1/10G/f1",
"size":10737418240
},
{
"name":"/data1/10G/f4",
"size":10737418240
}
],
"total_size":53687091200

“expand and group a path” command

This command is used by BDE to expand a source path and partition it into groups of a specified size, in preparation for a data transfer task. Just like expand command, an optional user parameter will be honored. The dst\_path parameter specifies the target folder where files will be transferred; this command will also return a constructed target path for each file.

{
"cmd":"dtn_expand_and_group",
"params":{
"src_path":["/data1/10G"],
"dst_path":"/data2",
"group_size":32212254720
}
}

The response will contain recursively expanded list of files under the specified paths; and they will be groups of size bytes (or the closest we can fit) as specified in the command:

{
"code":0,
"error":"OK",
"files":{
"groups":[
[
{
"files":[
["/data1/10G/f2","/data2/10G",10737418240,0],
["/data1/10G/f3","/data2/10G",10737418240,0],
["/data1/10G/f5","/data2/10G",10737418240,0]
],
"size":32212254720
},
{
"files":[
["/data1/10G/f1","/data2/10G",10737418240,0],
["/data1/10G/f4","/data2/10G",10737418240,0]
],
"size":21474836480
}
]
]
},
"total_size":53687091200
}

Running the agent

Big Data Express server project is hosted in Fermilab Redmine instance, and the code is available in a Git repository. If you have Fermilab Kerberos tokens, you can check out, build and run the agent code in a GNU/Linux host (that has fairly recent C++ compiler, CMake, GNU Make, and GNU Bash) with these commands:

$ git clone ssh:///cvs/projects/bigdata-express-server
$ cd bigdata-express-server
$ ./bootstrap_libraries.sh
$ mkdir build;cd build
$ cmake -DCMAKE_BUILD_TYPE=Debug ..
$ make
$ cd agent
$ ./bdeagent -c config/bdeagent.dtn1.conf

Note that the configuration file will need some changes, depending on the host where it is run. DTN Agent instances are usually spawned by BDE Agent, based on its configuration. The bdeagent program supports some options:

$ ./bdeagent -?
Usage:
./bdeagent[-i|-d] [-c config file] [-l log file]
-i to run in foreground
-d to run as daemon in background

bdeagent accepts configuration file name and log file name as parameters, and it can run as a daemon or as a foreground process. If the configuration is correct, and when running in the foreground, BDE Agent will run like so:

______
| _ ) \| __| /_\ ______| |_
| _ \ |) | _| / _ \/ _` / -_) ' \ _|
|___/___/|___| /_/ \_\__, \___|_||_\__|
|___/

-- Starting BDE Agent as a foreground process.
-- Reading configuration from ../../agent/config/bdeagent.bde1.json.
-- Logging to console.
[2017-05-10 18:12:23] [Main] Starting BDE agent...
[2017-05-10 18:12:23] [Main] bde agent listening on queue amq.gen-XZt6xmWxXH-vyjrT9pG0xg
[2017-05-10 18:12:23] [AgentManager] bootstrapping agent modules
[2017-05-10 18:12:23] [DTN Agent] Using MAC address 0c:c4:7a:ab:63:7e as DTN ID
[2017-05-10 18:12:23] [DTN Agent] using bde1.fnal.gov as DTN name
[2017-05-10 18:12:23] [DTN Agent] scanning data folders..
[2017-05-10 18:12:23] [DTN Agent] finding storage device for path /data1, scanning: yes
[2017-05-10 18:12:23] [DTN Agent] adding device: /dev/nvme0n1, folders: /data1,
[2017-05-10 18:12:23] [DTN Agent] Adding path /data1 to storage agent folders
[2017-05-10 18:12:23] [DTN Agent] Created work folder for /data1 at /data1/bde-iozone-work-xOoAta
[2017-05-10 18:12:23] [DTN Agent] Bringing up LocalStorageAgent for device /dev/nvme0n1, conf: {
"device" : "/dev/nvme0n1",
"id" : "0c:c4:7a:ab:63:7e|/dev/nvme0n1",
"iozone-executable" : "/usr/local/bin/iozone",
"iozone-work-folder" : "/data1/bde-iozone-work-xOoAta",
"name" : "Local Storage on DTN: bde1.fnal.gov, device: /dev/nvme0n1",
"root-folders" :
[
"/data1"
],
"type" : "LocalStorage"
}
[2017-05-10 18:12:23] [DTN Agent] Brought up LocalStorageAgent for device /dev/nvme0n1
[2017-05-10 18:12:23] [DTN Agent] finished bootstrap step.
[2017-05-10 18:12:23] [AgentManager] bootstrapping done.
[2017-05-10 18:12:23] [AgentManager] initializing agent module 0c:c4:7a:ab:63:7e
[2017-05-10 18:12:23] [AgentManager] 0c:c4:7a:ab:63:7e init...
[2017-05-10 18:12:23] [AgentManager] 0c:c4:7a:ab:63:7e registration...
[2017-05-10 18:12:23] [DTN Agent] finished registration step.
[2017-05-10 18:12:23] [AgentManager] initializing agent module 0c:c4:7a:ab:63:7e|/dev/nvme0n1
[2017-05-10 18:12:23] [AgentManager] 0c:c4:7a:ab:63:7e|/dev/nvme0n1 init...
[2017-05-10 18:12:36] [AgentManager] 0c:c4:7a:ab:63:7e|/dev/nvme0n1 registration...
[2017-05-10 18:12:36] [Main] BDE agent has been started. Press Ctrl-c to exit running

Manual testing

To manually test BDE Agent, we have created a small utility called rpctest. It accepts a RabbitMQ host, port, queue name, and a JSON file that contains the command to be sent to the target BDE agent:

./rpctest -?
Usage: ./rpctest [OPTIONS] [PARAM JSON FILE]
OPTIONS are:
-h [rabbitmq host] (default: yosemite.fnal.gov)
-p [rabbitmq port] (default: 5671)
-q [rabbitmq queue] (default: daemon)
-t [rabbitmq timeout] (default: 5s)
-c [cacert file] (default: cert.pem)
Example: rpctest -h yosemite.fnal.gov -p 5671 -c ./cacert.pem \
-q storage ./params.json

rpctest and the parameter files be found under the folder rpctest. You can run the program like below (queue name is a randomly generated string when BDE Agent starts up, and it is printed on the agent’s console):

$ ./rpctest -q amq.gen-Z5oGBoD_uhQ71_rbXzaIhg \
params/dtnagent/bde3/expand-good.json

And it should print the response it has received:

{
"code":0,
"error":"OK",
"files":[
{
"name":"/data1/users/user13",
"size":10737418240
},
[...]
],
"total_size":214748364800
}

Test suite

The test suite is present under tests folder. These programs are in general used to test and debug various components in isolation, to run them under memory checkers (such as valgrind), etc.

$ cd bigdata-express-server/build/tests; make; ctest
Test project [...]/bigdata-express-server/build/tests
Start 1: dtnagent-test
1/13 Test #1: dtnagent-test ...... Passed 0.01 sec
[...]
7/13 Test #7: vlan-test ...... ***Failed 0.01 sec
Start 8: certssh-test
[...]
Start 13: pathgroups-cli-test
13/13 Test #13: pathgroups-cli-test ...... Passed 0.03 sec
92% tests passed, 1 tests failed out of 13
Total Test time (real) = 0.55 sec
The following tests FAILED:
7 - vlan-test (Failed)
Errors while running CTest

(The above run has a failing test because vlan-test needs root privileges. This is an expected failure when running with normal user privileges.)

Final remarks

(TODO: need feedback here.)