Autoencoder for anomaly detection from feature vectors

Question

I am trying to use an autoencoder (as described here https://blog.keras.io/building-autoencoders-in-keras.html#) for anomaly detection. I am using a ~1700 feature vector (rather than images, which were used in the example) with each vector describing a different protein interaction. I have a "normal" category of interactions on which I train the AE, then I feed it new vectors and use reconstruction error to detect anomalous interactions.

Adjusting my threshold so I get a true positive rate of 0.95, I get a false positive rate of 0.15, which is rather high. When I trained xgboost on the normal and anomalous vectors (using both types of interactions in training and testing) I was able to get precision of 0.98 **.

Does that mean that my model (or indeed my approach of using an AE) is ineffective, or maybe this is the best I could hope for when training an anomaly detector rather than a 2 category classifier (that is, xgboost in my case)? How should I proceed?

** Of course, this is merely a sanity check, and cannot be used as the solution. I need the model to detect anomalies that can be very different from those I currently have - thus I need to train it on the normal interaction set, and leave anomalies for testing alone.

Answer 1

Does that mean that my model (or indeed my approach of using an AE) is ineffective

Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss function, optimizer, epochs.

How should I proceed?

From my gut feeling I would say that you don't have enough data to train the AE properly. Keep in Mind, the MNIST database contains 50,000 image. And you need enough variance in order to not overfit your training data. Tree based approaches are, at least in my experience, easier to train. If you like to stick at the anomaly detection part, which I recommend since you don't know what anomalies you will face, try the Isolation Forest Algorithm. But for a solid recommendation I would need to know how your data looks.

Btw, A good metric to use in such a case is the ROC score, which basically tells you how likely it is that your model will classify new data points correctly. Check out the link for an visual explanationROC explained

So Baseline is, try less complex approaches until you a certain that they are not sufficient enough.

Answer 2

Adding StandardScaler from sklearn.preprocessing improved the results somewhat, as did (in this case) making the net deeper.