Plugin Architecture: How to Handle Databases and Logging?

Question

I am in the process of designing an architecture for a plugin based application and I have some questions about how to handle database access and logging. The goal of my application is to allow a device my company is manufacturing to send messages over any number of third-party protocols and to any number of third party devices. Because of the nature of my application, I need to log every message that is sent via my system in reasonable detail.

Here's what I am thinking on the Database side: There will be a global message table which stores all of the implementation agnostic data we have, and then there will be a table (or set of tables) for each messaging plugin as defined by the plugin writer which references the Global Message table via a foreign key.

It is handling these other tables that I am having a hard time figuring out how to design safely. On the one hand, I don't particularly want to give the plugins full access to the database (I want them to have the fewest possible privileges), on the other hand I want the plugin developers to have full reign to log any data they need to tables that they define similarly to how plugins work in Wordpress. Additionally, I don't want logging to the Database to be optional at all: Good logging is crucial to my application.

Things I have considered:

• Designing a logging infrastructure which allows a plugin to define (maybe in an XML file) the tables it needs in the DB and a mapping between those tables and objects that they will return, and using those mappings and table definitions to write to the DB the results from method calls.
• Making a naming convention for Stored Procedures in the DB that each plugin must define when defining their tables and automatically calling those stored procedures after sending a message.
• Allowing each plugin to implement a logging method which returns a String holding the proper SQL query.

Each of the above solutions makes me a little uncomfortable, but I'm not sure what other approaches might exist. Any Suggestions?

Implementation Details: I am writing my application in Java, and I am using a hand-rolled plugin solution, so I have a pretty decent amount of flexibility. Since none of the actual code has been written (other than some proof-of concept code which is really just for reference), I can make fairly major changes to my plan without great cost.

Answer 1

Why does it have to be a database? Why not just have a generic "data sink" and write streams of data to it. Plugins are responsible for serializing and deserializing their own data, and you provide a simple read/write facility. They give you a reference to a message stream, then you write it and hand them back a GUID (or some other token) to identify the data.

Answer 2

How are you planning to implement the "sending" of the messages ? There are different approaches based on this answer. For example, are you whipping up your custom implementation of Apache James Server ? You haven't mentioned if you are working on any available frameworks. But I think one other approach you can consider is to use AOP for this. You can weave your aspects around each sender function and whenever such a function is called logging statements can be prepared and documented in a db.

Answer 3

What is the purpose of the messages? It sounds like they can be schema-less/unstructured. Hard-coding the implementation to the application database is probably going to come back to haunt you. If the logging is unstructured anyway, why not just do like/use log4j or similar? One option is to just write a thin wrapper API around log4j. Presumably you want to maintain logging context such that you can retrieve all logs from all plugins related to a single action/user. If so, then you need a way to tie the different logs together.

AppLogger.logMessage(AppContext context, PluginContext plugin, string log)

your storage is abstracted from the plugins, who shouldn't really care where the data is logged. This allows you to easily move it to FS, Queues, separate DB, etc without the plugins knowing/caring.

If you want to keep logging in the main DB, just store the logs in the master table with the log message stored as string/blob/xml/json, whatever the DB supports easily.

Answer 4

What you are describing is a situation in which you are designing APIs - between your application and the plug-ins. At this level you should focus on designing the interfaces and the data that needs to be exchanged without worrying about any specific persistence strategy. Once you nail that part, then part of the implementation in your application can handle the data any way it wants.

Since you are using Java, take a look at any high-level API you already use for the design cues - like Spring Data, Collections (e.g. Map, List) etc. They all hide the details behind Interfaces, Abstract classes and Concrete classes that implement the desired functionality.

At a very high level, I can think of the following (very simplistic approach, not knowing the exact details):

1. An Interface that allows a plugin to register itself to your application. Say you call it PluginRegistry. Then the method in it should be relevant for any plugin to register itself to the application and establish the communication links (like message listeners, callbacks etc). Your interface can dictate what method(s) the plugin needs to implement.
2. Say another interface defines the message exchange. It could be a high-level Interface coupled with an Abstract class (for default implementation) and one or more Concrete classes for specific implementations. You can leave the implementation of concrete classes to the plugins.

Your abstract classes can hide the details of logging in the database. Since you have the flexibility to design how the message should look like (plug-in will inherit from and/or implement the necessary interface), the abstract class with the default implementation can write the necessary data to the database (I'd prefer a log file) without the plugin knowing too much about it. This way you do not need to expose this functionality to plugins and can later change your implementations as needed.

The reason I mentioned that a log file is preferred over a relational-db is that you will have flexibility in logging any way you want in a file. A NoSQL db would also be a good choice here as it wont force you into hard relationships between your data early on. Then write some scripts (e.g. in Python, Perl etc) to analyze the log files and gather the necessary data and put in the relational-db as needed for reporting. But your main raw data store will be log files. As your application matures, you will get a good idea of the data's structure, important elements and automate the analysis scripts. As you log more type of data, you will need to update the scripts as needed, but hey you wont be changing the db tables where and breaking your application.