Here we go, it’s time. Welcome to the new PandApache3 release, version 3.3! (Technically, it’s 3.3.1 because a minor fix was added on day one, similar to recent AAA game updates). I hope you have enjoyed this deep dive series on PandApache3 so far. Each major release will now be accompanied by a technical article discussing the features and their implementation.
Such exciting times we are living in! If you haven’t read or need a refresher on what we discussed last time, you can check our previous article:
PandApache3, the Apache killer
Deep Dive into PandApache3: Launch Code
Deep Dive into PandApache3: Understanding Connection Management and Response Generation
Deep Dive into PandApache3: Implementation of Authentication and Security
I will do my best to explain everything here, so no prior background is necessary.
Ready? Let’s go!
The release note
First, let’s discuss what we aimed to achieve with this release. The goal was to extend the administrative capabilities of PandApache3. By administrative capabilities, we mean functions such as:
Changing and updating the server configuration
Retrieving the status, restarting, or stopping services
Executing various operations to maintain service health
Usually, these actions are performed directly on the server, requiring an administrator to log in and execute commands. While this is doable for a few services and servers, it becomes challenging at scale. We cannot connect to 10.000 servers to update a configuration.
To address this problem, we added administrative endpoints to PandApache3. These endpoints allow you to execute the same actions across your entire fleet effortlessly. Let’s explore the different available endpoints:
/admin/status: Get the current status of your server
/admin/reload: Reload the configuration file
/admin/config: Get or update specific configuration fields
/admin/stop: Stop the service
/admin/restart: Restart the service
/admin/script: Get, upload, or execute scripts
Grab a snack and a coffee, and let’s dive into how some of these endpoints were implemented!
Between Us
We already have tools like Ansible to perform identical actions on multiple servers. So why provide API endpoints to a service? Tools like Ansible are indeed powerful, but having administrative endpoints available to perform admin tasks is simpler to integrate with other services. It’s also more flexible and lighter to integrate into scripts than Ansible playbooks.
This API can also be used to build a user interface, which is not possible with Ansible.
Get the status of the server
We will start with this endpoint because it is quite simple and will allow us to focus on the architecture rather than the result of the endpoint itself.
As you probably guessed, to implement this new administration endpoint in our web server, we will again use and rely on the middleware. In case you missed some previous articles, a middleware is a component of our architecture that is in charge of executing some processing on a request.
All admin requests, like all requests, must pass through the authentication and directory middleware to gain access to the endpoint. Then the admin middleware will be responsible for generating the response.
{
Logger.LogDebug(“Admin middleware”);
Request request = context.Request;
if (request.Verb == “GET”)
{
string adminURL = ServerConfiguration.Instance.AdminDirectory.URL;
if (request.Path.ToLower().Equals(adminURL + “/status”))
{
context.Response = new HttpResponse(200)
{
Body = new MemoryStream(Encoding.UTF8.GetBytes(Server.STATUS))
};
}
else
{
context.Response = new HttpResponse(404);
}
}
else
{
context.Response = new HttpResponse(404);
}
await _next(context);
}
Here a schema for visual people
Stopping or restarting the service
These two endpoints are more interesting to examine. Stopping, and especially restarting, the service is an important feature. Most services need an orchestrator to restart. But before looking at this complex scenario, let’s simplify it. A restart is a stop action followed by a start. We have already seen the PandApache3 start action Deep Dive into PandApache3: Launch Code ; keeping this in mind will help us understand the stop function.
Here is the function called when the endpoint admin/stop is hit and how the response is generated:
response = new HttpResponse(200)
{
Body = new MemoryStream(Encoding.UTF8.GetBytes(“Stopping….”))
};
You can already notice something different compared to the status endpoint. Stopping the server is not a synchronous call. Unlike the status endpoint, where the server executes all operations needed before returning the answer, this time a new independent task is created, and the OK response is sent. This makes sense, as if you stop the server, it will be unable to answer clients. The response here is an acknowledgment that the requested task will be performed, but it is not complete when the response arrives to the client.
Another part of the stop server action code is unique in PandApache3. Let’s check directly in the function StoppingServerAsynccalled by our new thread:
{
if (!Monitor.TryEnter(_lock))
{
Logger.LogDebug($”Thread (Thread ID: {Thread.CurrentThread.ManagedThreadId}) could not acquire the lock and will exit.”);
Logger.LogInfo(“Server is already restarting”);
return;
}
lock (_lock)
{
List<CancellationTokenSource> cancellationTokens = new List<CancellationTokenSource>();
cancellationTokens.Add(_ConnectionManager._cancellationTokenSource);
StopServerAsync(cancellationTokens).Wait();
}
Logger.LogDebug(“Get out of the lock”);
Monitor.Exit(_lock);
}
The real function that will stop the server is StopServerAsync, but you probably noticed that this function is inside a lock block. A lock ensures that your code will be executed only in one thread. Some actions, like stopping the server, can be run only once. It would not make sense to ask the server to stop twice, and it could generate many issues. In general, threads can generate many concurrency issues. Using the lock ensures that the function will not be called multiple times by different clients.
In this lock statement, we do not just stop the server. You can also see that we have cancellation tokens. Let’s see the purpose of this token that we saved in a list.
Do you remember the way the loop in charge of accepting and managing new connections was presented in the article Deep Dive into PandApache3: Understanding Connection Management and Response Generation?
I told you that the server is executed in an infinite while loop because we are always accepting connections. Well, it was a simplification, because of course the server cannot continuously accept connections in some situations, like being terminated. We must be sure that no new connections will be attempted. And a way to stop this “infinite” loop at a distance is the token. Here is the correct loop:
{
if (listener.Pending())
{
ISocketWrapper client = new SocketWrapper(listener.AcceptSocket());
await connectionManager.AcceptConnectionsAsync(client);
}
} while (connectionManager._cancellationTokenSource.IsCancellationRequested == false);
Let’s address the elephant in the room. Yes, it’s a do while and not a while loop. But you can see in the condition that to exit the loop, the cancellationTokenSource, a property of our connectionManager, must be canceled. That is how we will stop accepting new connections.
Now we can really stop the server. Here is the last function for this use case, StopServerAsync:
{
int retry = 5;
Server.STATUS = “PandApache3 is stopping”;
Logger.LogInfo($”{Server.STATUS}“);
if (cancellationTokens != null)
{
foreach (CancellationTokenSource token in cancellationTokens)
{
token.Cancel();
}
}
_ConnectionManager.Listener.Stop();
for (int i = 0; i < retry; retry–)
{
if (_ConnectionManager.clients.Count > 0)
{
Thread.Sleep(1000);
Logger.LogDebug(“There are still active connections…”);
}
else
{
break;
}
}
if (retry == 0)
{
Logger.LogInfo(“Force server to stop”);
}
else
{
Logger.LogInfo(“Server stopped”);
}
Server.STATUS = “PandApache3 is stopped”;
Logger.LogInfo($”{Server.STATUS}“);
}
We start by canceling all tokens held by the connectionManager:
Then we stop the listener:
At this point, even if we block all new connections, we could still have some open and active ones waiting for a response from the server. Being considerate, we give them 5 seconds before terminating them (currently we don’t, but we should).
Finally, no more child threads will be running. One of our first server functions, RunServerAsync, will finish now that the infinite loop is terminated. The program will continue until the end of its main code and say goodbye:
Between Us
Why do we have a list of tokens? We have just one connectionManager, right? It’s correct. The list for the moment is optional, and the code could work without it by just passing our token. But, wait for it… We will address this point later.
Restarting the Server
Now that we have covered the stop function, let’s examine the restart process. The restart uses the same functions as the stop, with an additional Boolean parameter to indicate a restart. To restart the server, we need to stop it and then start it again from the beginning. How do we achieve this? With a loop and a token, of course!
Here is the main function:
{
await StartServerAsync();
await RunServerAsync();
}
Logger.LogInfo(“La revedere!”);
As discussed in the previous article, we start the server, which then runs continuously. The RunServerAsyncfunction never ends until we hit the administration endpoint to stop the server. When the server stops, the process ends.
However, to perform a restart, we need to stop the server and then return to the beginning of the main function to execute StartServerAsyncand RunServerAsyncagain.
To achieve this, we use a token, but this time at the server level rather than the connectionManagerlevel. When we want to restart the server, we execute the function described in the previous section. Our StopServer function essentially becomes a RestartServerfunction. To effectively stop the server, we use a flag to determine whether to cancel the server token.
Logger.LogInfo($”{Server.STATUS}“);
if (isRestart == false)
{
_cancellationTokenSourceServer.Cancel();
}
That’s it! While the previous section was detailed in explaining how the server stops, this effort was not in vain because understanding how the server restarts is now much easier.
Query String
Okay, we still have one interesting endpoint to look at: /admin/script, which allows you to get, upload, or execute scripts. This endpoint has something different compared to the others; it accepts what we call a query string.
Here’s an example of a URL with a query string: http://pandapache3/admin/script?name=hello_world.ps1 The query string is what follows the ?. In this example, we have one parameter, “name,” associated with the value “hello_world.ps1.” This request contains only one parameter, but there could be many more.
As you might imagine, this means our request object will continue to evolve to store this query string. Our new attribute will be a dictionary of strings:
Here’s a simple function to parse and store the values:
{
var parameters = new Dictionary<string, string>();
if (!string.IsNullOrEmpty(QueryString))
{
var keyValuePairs = QueryString.Split(‘&’);
foreach (var pair in keyValuePairs)
{
var keyValue = pair.Split(‘=’);
Logger.LogDebug($”KeyValue: {keyValue}“);
if (keyValue.Length == 2)
{
var key = Uri.UnescapeDataString(keyValue[0]);
var value = Uri.UnescapeDataString(keyValue[1]);
parameters[key] = value;
}
}
}
return parameters;
}
Now, what happens when our endpoint is hit? There are three use cases:
You call the endpoint with a POST request: This means you want to upload a script to the server. Since the first release the server handles the upload, the process here didn’t change and the script is saved in the admin directory.
{
if (context.Request.Headers[“Content-Type”] != null && context.Request.Headers[“Content-Type”].StartsWith(“multipart/form-data”))
{
string adminURL = ServerConfiguration.Instance.AdminDirectory.URL;
string scriptsDirectory = ServerConfiguration.Instance.AdminDirectory.Path;
if (context.Request.Path.ToLower().StartsWith(adminURL + “/script”))
{
if (ServerConfiguration.Instance.AdminScript)
return RequestParser.UploadHandler(context.Request, true);
else
return new HttpResponse(403);
}
}
return new HttpResponse(404);
}
You call the endpoint with a GET request without a query string: In this case, your URL might be something like http://pandapache3/admin/script. We will consider that you want to know what scripts are present on the server that you can execute. This is managed by a simple request:
foreach (string script in Directory.GetFiles(scriptsDirectory))
{
FileInfo fileInfo = new FileInfo(script);
bodyScript += $”t- {fileInfo.Name}n”;
}
response = new HttpResponse(200)
{
Body = new MemoryStream(Encoding.UTF8.GetBytes(bodyScript))
};
You use a query string: This means you want to execute a specific script on the server. Let’s see what happens:
{
HttpResponse response = null;
string terminal = string.Empty;
if (ServerConfiguration.Instance.Platform.Equals(“WIN”))
terminal = “powershell.exe”;
else
terminal = “/bin/bash”;
string argumentList = $”{Path.Combine(scriptDirectory, queryParameters[“name”])}“;
foreach (var item in queryParameters)
{
if (item.Key != “name”)
{
argumentList += $” {item.Value}“;
}
}
var processInfo = new ProcessStartInfo
{
FileName = terminal,
Arguments = argumentList,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
UseShellExecute = false,
};
try
{
using (var process = new Process { StartInfo = processInfo })
{
process.Start();
process.WaitForExit();
string standardOutput = process.StandardOutput.ReadToEnd();
string standardError = process.StandardError.ReadToEnd();
ScriptResult scriptResult = new ScriptResult
{
ExitCode = process.ExitCode,
StandardOutput = standardOutput,
ErrorOutput = standardError
};
response = new HttpResponse(200)
{
Body = new MemoryStream(Encoding.UTF8.GetBytes(Newtonsoft.Json.JsonConvert.SerializeObject(scriptResult)))
};
}
}
catch (Exception ex)
{
Logger.LogError($”Error with script execution {ex.Message}“);
response = new HttpResponse(500);
}
return response;
}
Let’s go through this function. First, the parameters are the script directory (which is also the admin directory) and the query string that contains the script we want to execute.
The next step is to determine the current platform to execute our script with the appropriate program (PowerShell for Windows and Bash for Linux)
if (ServerConfiguration.Instance.Platform.Equals(“WIN”))
terminal = “powershell.exe”;
else
terminal = “/bin/bash”;
With all the information, we build the command we want to execute, including the script’s full path and potential arguments:
foreach (var item in queryParameters)
{
if (item.Key != “name”)
{
argumentList += $” {item.Value}“;
}
}
To run the script, we need a ProcessStartInfoto specify options for the execution:
{
FileName = terminal,
Arguments = argumentList,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
UseShellExecute = false,
};
We can now run it and save its output to generate a response for our client:
{
process.Start();
process.WaitForExit();
string standardOutput = process.StandardOutput.ReadToEnd();
string standardError = process.StandardError.ReadToEnd();
ScriptResult scriptResult = new ScriptResult
{
ExitCode = process.ExitCode,
StandardOutput = standardOutput,
ErrorOutput = standardError
};
response = new HttpResponse(200)
{
Body = new MemoryStream(Encoding.UTF8.GetBytes(Newtonsoft.Json.JsonConvert.SerializeObject(scriptResult)))
};
}
The response will be structured in JSON format with the exit code, standard output, and error output. This format is designed to be easily used by other programs rather than by humans.
Between Us
The /admin/config endpoint can also work with query parameters to change a specific parameter in the configuration. This part functions exactly like the script endpoint at this level, so we won’t cover it in this article. However, you now have all the knowledge to explore the code yourself!
Still Between Us
You might notice that this time, the HTTP request is synchronous for the client. This is quite different compared to the action of stopping the server, which returns a response before the end of the action. Here, we’ve chosen to block the client until they receive a response. This could be an issue if your script performs heavy processing. We will address this in the future by offering both synchronous and asynchronous script execution options on the server.
Protect our administration endpoint
It’s time to discuss the final aspect of this release. We have seen that the administration endpoints are quite powerful and can be used to create CLI tools or a web administration console. Their power is such that, if an administration action does not exist, you can create it yourself through scripts.
However, if you are a good system administrator, you might recognize that the ability to upload and execute remote code on a server can be extremely dangerous! And you are correct!
This feature, while very useful for managing a PaaS, is also risky if not configured correctly. That’s why your administration endpoints do not run on the same site as your PandApache site!
Currently, my site runs at http://127.0.0.1:8080/. When I access this address, my connection manager handles the connection, passes the request through the middleware pipeline, and displays my resources. However, my admin endpoint is not located at this address. You can configure it to run on a different port or even on another network interface! This site has its own connection manager, meaning it also has its own middleware pipeline.
That’s why, when restarting the server, we use a list of tokens to cancel connections. This is because we are stopping not just one ConnectionManager with its TCPListener, but two!
The ability to separate the two sites at the network level is a good way to protect your server from attacks. It allows your customers to access content through one path and your administration team to manage your service through another.
Between Us
If you are still not convinced about the security measures, there are additional steps you can take. First, you can disable script execution. Your administration endpoint will still be available, but you won’t be able to execute actions that have not already been implemented through the API. You can also completely disable the admin site and continue managing your service as you do currently. We will continue to enhance security in the future, offering even greater isolation between the public and administrative parts of PandApache3.
In this article, we delved into the new features of PandApache3, covering how administration endpoint was implemented to be secure and useful
The next topic that we will cover in this series of article is an important part for system administrator, the logging, the telemetry and the monitoring.
Stay tuned!
Thank you so much for exploring the inner workings of PandApache3 with me! Your thoughts and support are crucial in advancing this project. 🚀
Feel free to share your ideas and impressions in the comments below. I look forward to hearing from you!
Follow my adventures on Twitter @pykpyky to stay updated on all the news.
You can also explore the full project on GitHub and join me for live coding sessions on Twitch for exciting and interactive sessions. See you soon behind the screen!
